From python-checkins at python.org Tue May 1 00:05:29 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 00:05:29 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Make_it_clear_that_only_finder?= =?utf8?q?s_change=2C_not_loaders=2E?= Message-ID: http://hg.python.org/peps/rev/02805cdc0604 changeset: 4326:02805cdc0604 user: Eric V. Smith date: Mon Apr 30 18:05:25 2012 -0400 summary: Make it clear that only finders change, not loaders. files: pep-0420.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -159,6 +159,7 @@ the string that will be recorded and later used as a component of the namespace module's __path__, as described above. +There is no impact on PEP 302 "loaders". Discussion ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 01:04:51 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 01:04:51 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_No_trailing_separators_on_stri?= =?utf8?q?ngs_returned_by_find=5Fmodule=2E?= Message-ID: http://hg.python.org/peps/rev/c833ec103ecf changeset: 4327:c833ec103ecf user: Eric V. Smith date: Mon Apr 30 19:04:39 2012 -0400 summary: No trailing separators on strings returned by find_module. files: pep-0420.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -157,7 +157,8 @@ "loader" object or None. For a finder to contribute to namespace packages, ``find_module`` will return a third type: a string. This is the string that will be recorded and later used as a component of the -namespace module's __path__, as described above. +namespace module's ``__path__``, as described above. This string must +not contain a trailing path separator. There is no impact on PEP 302 "loaders". -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 01:29:03 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 01:29:03 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Discuss_implications_on_existi?= =?utf8?q?ng_finders=2E?= Message-ID: http://hg.python.org/peps/rev/93e760149759 changeset: 4328:93e760149759 user: Eric V. Smith date: Mon Apr 30 19:28:54 2012 -0400 summary: Discuss implications on existing finders. files: pep-0420.txt | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -162,6 +162,10 @@ There is no impact on PEP 302 "loaders". +If an existing finder is not updated to support returning a string +from ``find_module``, the only impact is that such a loader will be +unable to provide portions of a namespace package. + Discussion ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 02:35:51 2012 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 01 May 2012 02:35:51 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Handle_a_possible_race_cond?= =?utf8?q?ition?= Message-ID: http://hg.python.org/cpython/rev/b3aeaef6c315 changeset: 76675:b3aeaef6c315 user: Raymond Hettinger date: Mon Apr 30 14:14:28 2012 -0700 summary: Handle a possible race condition files: Lib/functools.py | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -241,6 +241,12 @@ return result result = user_function(*args, **kwds) with lock: + if key in cache: + # getting here means that this same key was added to the + # cache while the lock was released. since the link + # update is already done, we need only return the + # computed result and update the count of misses. + pass if currsize < maxsize: # put result in a new link at the front of the queue last = root[PREV] -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Tue May 1 04:21:48 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 1 May 2012 12:21:48 +1000 Subject: [Python-checkins] cpython: Handle a possible race condition In-Reply-To: References: Message-ID: On Tue, May 1, 2012 at 10:35 AM, raymond.hettinger wrote: > http://hg.python.org/cpython/rev/b3aeaef6c315 > changeset: ? 76675:b3aeaef6c315 > user: ? ? ? ?Raymond Hettinger > date: ? ? ? ?Mon Apr 30 14:14:28 2012 -0700 > summary: > ?Handle a possible race condition > > files: > ?Lib/functools.py | ?6 ++++++ > ?1 files changed, 6 insertions(+), 0 deletions(-) > > > diff --git a/Lib/functools.py b/Lib/functools.py > --- a/Lib/functools.py > +++ b/Lib/functools.py > @@ -241,6 +241,12 @@ > ? ? ? ? ? ? ? ? ? ? ? ? return result > ? ? ? ? ? ? ? ? result = user_function(*args, **kwds) > ? ? ? ? ? ? ? ? with lock: > + ? ? ? ? ? ? ? ? ? ?if key in cache: > + ? ? ? ? ? ? ? ? ? ? ? ?# getting here means that this same key was added to the > + ? ? ? ? ? ? ? ? ? ? ? ?# cache while the lock was released. ?since the link > + ? ? ? ? ? ? ? ? ? ? ? ?# update is already done, we need only return the > + ? ? ? ? ? ? ? ? ? ? ? ?# computed result and update the count of misses. > + ? ? ? ? ? ? ? ? ? ? ? ?pass > ? ? ? ? ? ? ? ? ? ? if currsize < maxsize: > ? ? ? ? ? ? ? ? ? ? ? ? # put result in a new link at the front of the queue > ? ? ? ? ? ? ? ? ? ? ? ? last = root[PREV] To get the desired effect, I believe you also need s/if currsize/elif currsize/ Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python-checkins at python.org Tue May 1 04:37:56 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 01 May 2012 04:37:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_issue13183_-_Fi?= =?utf8?q?x_pdb_skipping_frames_after_hitting_a_breakpoint_and_running?= Message-ID: http://hg.python.org/cpython/rev/96cb47f8142e changeset: 76676:96cb47f8142e branch: 3.2 parent: 76673:5c801899cd6d user: Senthil Kumaran date: Tue May 01 10:07:49 2012 +0800 summary: issue13183 - Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye files: Lib/bdb.py | 15 ++++++++- Lib/test/test_pdb.py | 51 ++++++++++++++++++++++++++++++++ Misc/NEWS | 3 + 3 files changed, 68 insertions(+), 1 deletions(-) diff --git a/Lib/bdb.py b/Lib/bdb.py --- a/Lib/bdb.py +++ b/Lib/bdb.py @@ -22,6 +22,7 @@ self.skip = set(skip) if skip else None self.breaks = {} self.fncache = {} + self.frame_returning = None def canonic(self, filename): if filename == "<" + filename[1:-1] + ">": @@ -80,7 +81,11 @@ def dispatch_return(self, frame, arg): if self.stop_here(frame) or frame == self.returnframe: - self.user_return(frame, arg) + try: + self.frame_returning = frame + self.user_return(frame, arg) + finally: + self.frame_returning = None if self.quitting: raise BdbQuit return self.trace_dispatch @@ -186,6 +191,14 @@ def set_step(self): """Stop after one line of code.""" + # Issue #13183: pdb skips frames after hitting a breakpoint and running + # step commands. + # Restore the trace function in the caller (that may not have been set + # for performance reasons) when returning from the current frame. + if self.frame_returning: + caller_frame = self.frame_returning.f_back + if caller_frame and not caller_frame.f_trace: + caller_frame.f_trace = self.trace_dispatch self._set_stopinfo(None, None) def set_next(self, frame): diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -5,6 +5,7 @@ import sys import unittest import subprocess +import textwrap from test import support # This little helper class is essential for testing pdb under doctest. @@ -595,6 +596,22 @@ class PdbTestCase(unittest.TestCase): + def run_pdb(self, script, commands): + """Run 'script' lines with pdb and the pdb 'commands'.""" + filename = 'main.py' + with open(filename, 'w') as f: + f.write(textwrap.dedent(script)) + cmd = [sys.executable, '-m', 'pdb', filename] + stdout = stderr = None + with subprocess.Popen(cmd, stdout=subprocess.PIPE, + stdin=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) as proc: + stdout, stderr = proc.communicate(str.encode(commands)) + stdout = stdout and bytes.decode(stdout) + stderr = stderr and bytes.decode(stderr) + return stdout, stderr + def test_issue7964(self): # open the file as binary so we can force \r\n newline with open(support.TESTFN, 'wb') as f: @@ -610,6 +627,40 @@ self.assertNotIn(b'SyntaxError', stdout, "Got a syntax error running test script under PDB") + def test_issue13183(self): + script = """ + from bar import bar + + def foo(): + bar() + + def nope(): + pass + + def foobar(): + foo() + nope() + + foobar() + """ + commands = """ + from bar import bar + break bar + continue + step + step + quit + """ + bar = """ + def bar(): + print('1') + """ + with open('bar.py', 'w') as f: + f.write(textwrap.dedent(bar)) + stdout, stderr = self.run_pdb(script, commands) + self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3], + 'Fail to step into the caller after a return') + def tearDown(self): support.unlink(support.TESTFN) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -59,6 +59,9 @@ Library ------- +- Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running + step. Patch by Xavier de Gaye. + - Issue #14696: Fix parser module to understand 'nonlocal' declarations. - Issue #10941: Fix imaplib.Internaldate2tuple to produce correct result near -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 04:37:57 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 01 May 2012 04:37:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_issue13183_-_Fi?= =?utf8?q?x_pdb_skipping_frames_after_hitting_a_breakpoint_and_running?= Message-ID: http://hg.python.org/cpython/rev/5ea23739e9ba changeset: 76677:5ea23739e9ba branch: 2.7 parent: 76672:2468b58f7fce user: Senthil Kumaran date: Tue May 01 10:36:28 2012 +0800 summary: issue13183 - Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye files: Lib/bdb.py | 11 ++++++ Lib/test/test_pdb.py | 58 +++++++++++++++++++++++++++++++- Misc/NEWS | 3 + 3 files changed, 71 insertions(+), 1 deletions(-) diff --git a/Lib/bdb.py b/Lib/bdb.py --- a/Lib/bdb.py +++ b/Lib/bdb.py @@ -24,6 +24,7 @@ self.skip = set(skip) if skip else None self.breaks = {} self.fncache = {} + self.frame_returning = None def canonic(self, filename): if filename == "<" + filename[1:-1] + ">": @@ -82,7 +83,9 @@ def dispatch_return(self, frame, arg): if self.stop_here(frame) or frame == self.returnframe: + self.frame_returning = frame self.user_return(frame, arg) + self.frame_returning = None if self.quitting: raise BdbQuit return self.trace_dispatch @@ -186,6 +189,14 @@ def set_step(self): """Stop after one line of code.""" + # Issue #13183: pdb skips frames after hitting a breakpoint and running + # step commands. + # Restore the trace function in the caller (that may not have been set + # for performance reasons) when returning from the current frame. + if self.frame_returning: + caller_frame = self.frame_returning.f_back + if caller_frame and not caller_frame.f_trace: + caller_frame.f_trace = self.trace_dispatch self._set_stopinfo(None, None) def set_next(self, frame): diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -6,12 +6,66 @@ import os import unittest import subprocess +import textwrap from test import test_support # This little helper class is essential for testing pdb under doctest. from test_doctest import _FakeInput +class PdbTestCase(unittest.TestCase): + + def run_pdb(self, script, commands): + """Run 'script' lines with pdb and the pdb 'commands'.""" + filename = 'main.py' + with open(filename, 'w') as f: + f.write(textwrap.dedent(script)) + cmd = [sys.executable, '-m', 'pdb', filename] + stdout = stderr = None + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, + stdin=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) + stdout, stderr = proc.communicate(commands) + proc.stdout.close() + proc.stdin.close() + return stdout, stderr + + def test_issue13183(self): + script = """ + from bar import bar + + def foo(): + bar() + + def nope(): + pass + + def foobar(): + foo() + nope() + + foobar() + """ + commands = """ + from bar import bar + break bar + continue + step + step + quit + """ + bar = """ + def bar(): + print('1') + """ + with open('bar.py', 'w') as f: + f.write(textwrap.dedent(bar)) + stdout, stderr = self.run_pdb(script, commands) + self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3], + 'Fail to step into the caller after a return') + + class PdbTestInput(object): """Context manager that makes testing Pdb in doctests easier.""" @@ -309,7 +363,9 @@ def test_main(): from test import test_pdb test_support.run_doctest(test_pdb, verbosity=True) - test_support.run_unittest(ModuleInitTester) + test_support.run_unittest( + PdbTestCase, + ModuleInitTester) if __name__ == '__main__': test_main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -56,6 +56,9 @@ Library ------- +- Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running + step. Patch by Xavier de Gaye. + - Issue #14664: It is now possible to use @unittest.skip{If,Unless} on a test class that doesn't inherit from TestCase (i.e. a mixin). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 04:37:58 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 01 May 2012 04:37:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_issue13183_-_Fix_pdb_skipping_frames_after_hitting_a_breakpo?= =?utf8?q?int_and_running?= Message-ID: http://hg.python.org/cpython/rev/ab63e874265e changeset: 76678:ab63e874265e parent: 76675:b3aeaef6c315 parent: 76676:96cb47f8142e user: Senthil Kumaran date: Tue May 01 10:37:11 2012 +0800 summary: issue13183 - Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye files: Lib/bdb.py | 15 ++++++++- Lib/test/test_pdb.py | 51 ++++++++++++++++++++++++++++++++ Misc/NEWS | 3 + 3 files changed, 68 insertions(+), 1 deletions(-) diff --git a/Lib/bdb.py b/Lib/bdb.py --- a/Lib/bdb.py +++ b/Lib/bdb.py @@ -22,6 +22,7 @@ self.skip = set(skip) if skip else None self.breaks = {} self.fncache = {} + self.frame_returning = None def canonic(self, filename): if filename == "<" + filename[1:-1] + ">": @@ -80,7 +81,11 @@ def dispatch_return(self, frame, arg): if self.stop_here(frame) or frame == self.returnframe: - self.user_return(frame, arg) + try: + self.frame_returning = frame + self.user_return(frame, arg) + finally: + self.frame_returning = None if self.quitting: raise BdbQuit return self.trace_dispatch @@ -186,6 +191,14 @@ def set_step(self): """Stop after one line of code.""" + # Issue #13183: pdb skips frames after hitting a breakpoint and running + # step commands. + # Restore the trace function in the caller (that may not have been set + # for performance reasons) when returning from the current frame. + if self.frame_returning: + caller_frame = self.frame_returning.f_back + if caller_frame and not caller_frame.f_trace: + caller_frame.f_trace = self.trace_dispatch self._set_stopinfo(None, None) def set_next(self, frame): diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -5,6 +5,7 @@ import sys import unittest import subprocess +import textwrap from test import support # This little helper class is essential for testing pdb under doctest. @@ -598,6 +599,22 @@ class PdbTestCase(unittest.TestCase): + def run_pdb(self, script, commands): + """Run 'script' lines with pdb and the pdb 'commands'.""" + filename = 'main.py' + with open(filename, 'w') as f: + f.write(textwrap.dedent(script)) + cmd = [sys.executable, '-m', 'pdb', filename] + stdout = stderr = None + with subprocess.Popen(cmd, stdout=subprocess.PIPE, + stdin=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) as proc: + stdout, stderr = proc.communicate(str.encode(commands)) + stdout = stdout and bytes.decode(stdout) + stderr = stderr and bytes.decode(stderr) + return stdout, stderr + def test_issue7964(self): # open the file as binary so we can force \r\n newline with open(support.TESTFN, 'wb') as f: @@ -613,6 +630,40 @@ self.assertNotIn(b'SyntaxError', stdout, "Got a syntax error running test script under PDB") + def test_issue13183(self): + script = """ + from bar import bar + + def foo(): + bar() + + def nope(): + pass + + def foobar(): + foo() + nope() + + foobar() + """ + commands = """ + from bar import bar + break bar + continue + step + step + quit + """ + bar = """ + def bar(): + print('1') + """ + with open('bar.py', 'w') as f: + f.write(textwrap.dedent(bar)) + stdout, stderr = self.run_pdb(script, commands) + self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3], + 'Fail to step into the caller after a return') + def tearDown(self): support.unlink(support.TESTFN) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -87,6 +87,9 @@ Library ------- +- Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running + step. Patch by Xavier de Gaye. + - Issue #14696: Fix parser module to understand 'nonlocal' declarations. - Issue #10941: Fix imaplib.Internaldate2tuple to produce correct result near -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 04:47:22 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 01 May 2012 04:47:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Have_Bdb_frame?= =?utf8?q?=5Freturning_in_the_finally_clause?= Message-ID: http://hg.python.org/cpython/rev/4c3c4891fd6a changeset: 76679:4c3c4891fd6a branch: 2.7 parent: 76677:5ea23739e9ba user: Senthil Kumaran date: Tue May 01 10:46:59 2012 +0800 summary: Have Bdb frame_returning in the finally clause files: Lib/bdb.py | 8 +++++--- 1 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Lib/bdb.py b/Lib/bdb.py --- a/Lib/bdb.py +++ b/Lib/bdb.py @@ -83,9 +83,11 @@ def dispatch_return(self, frame, arg): if self.stop_here(frame) or frame == self.returnframe: - self.frame_returning = frame - self.user_return(frame, arg) - self.frame_returning = None + try: + self.frame_returning = frame + self.user_return(frame, arg) + finally: + self.frame_returning = None if self.quitting: raise BdbQuit return self.trace_dispatch -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue May 1 05:37:41 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 01 May 2012 05:37:41 +0200 Subject: [Python-checkins] Daily reference leaks (b3aeaef6c315): sum=0 Message-ID: results for b3aeaef6c315 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogUiz3yj', '-x'] From python-checkins at python.org Tue May 1 05:49:05 2012 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 01 May 2012 05:49:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_a_flag_to_indicate_when?= =?utf8?q?_the_circular_queue_is_fully_populated_and_stable=2E?= Message-ID: http://hg.python.org/cpython/rev/f5bb290c751a changeset: 76680:f5bb290c751a parent: 76678:ab63e874265e user: Raymond Hettinger date: Mon Apr 30 20:48:55 2012 -0700 summary: Use a flag to indicate when the circular queue is fully populated and stable. files: Lib/functools.py | 21 ++++++++++++--------- 1 files changed, 12 insertions(+), 9 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -176,6 +176,7 @@ cache = {} hits = misses = currsize = 0 + full = False cache_get = cache.get # bound method to lookup a key or return None lock = Lock() # because linkedlist updates aren't threadsafe root = [] # root of the circular doubly linked list @@ -224,7 +225,7 @@ def wrapper(*args, **kwds): # size limited caching that tracks accesses by recency - nonlocal root, hits, misses, currsize + nonlocal root, hits, misses, currsize, full key = make_key(args, kwds, typed) if kwds or typed else args with lock: link = cache_get(key) @@ -247,13 +248,7 @@ # update is already done, we need only return the # computed result and update the count of misses. pass - if currsize < maxsize: - # put result in a new link at the front of the queue - last = root[PREV] - link = [last, root, key, result] - cache[key] = last[NEXT] = root[PREV] = link - currsize += 1 - else: + elif full: # use root to store the new key and result root[KEY] = key root[RESULT] = result @@ -262,6 +257,13 @@ root = root[NEXT] del cache[root[KEY]] root[KEY] = root[RESULT] = None + else: + # put result in a new link at the front of the queue + last = root[PREV] + link = [last, root, key, result] + cache[key] = last[NEXT] = root[PREV] = link + currsize += 1 + full = (currsize == maxsize) misses += 1 return result @@ -272,11 +274,12 @@ def cache_clear(): """Clear the cache and cache statistics""" - nonlocal hits, misses, currsize + nonlocal hits, misses, currsize, full with lock: cache.clear() root[:] = [root, root, None, None] hits = misses = currsize = 0 + full = False wrapper.cache_info = cache_info wrapper.cache_clear = cache_clear -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 07:32:36 2012 From: python-checkins at python.org (raymond.hettinger) Date: Tue, 01 May 2012 07:32:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_make=5Fkey=28=29_out_o?= =?utf8?q?f_the_decorator_body=2E_Make_keys_that_only_need_to_be?= Message-ID: http://hg.python.org/cpython/rev/f981fe3b8bf7 changeset: 76681:f981fe3b8bf7 user: Raymond Hettinger date: Mon Apr 30 22:32:16 2012 -0700 summary: Move make_key() out of the decorator body. Make keys that only need to be hashed once. files: Lib/functools.py | 44 ++++++++++++++++++++++------------- 1 files changed, 27 insertions(+), 17 deletions(-) diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -142,6 +142,30 @@ _CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"]) +class _CacheKey(list): + 'Make a cache key from optionally typed positional and keyword arguments' + + __slots__ = 'hashvalue' + + def __init__(self, args, kwds, typed, + kwd_mark = (object(),), + sorted=sorted, tuple=tuple, type=type, hash=hash): + key = args + if kwds: + sorted_items = sorted(kwds.items()) + key += kwd_mark + for item in sorted_items: + key += item + if typed: + key += tuple(type(v) for v in args) + if kwds: + key += tuple(type(v) for k, v in sorted_items) + self[:] = key + self.hashvalue = hash(key) # so we only have to hash just once + + def __hash__(self): + return self.hashvalue + def lru_cache(maxsize=100, typed=False): """Least-recently-used cache decorator. @@ -168,8 +192,8 @@ # to allow the implementation to change (including a possible C version). # Constants shared by all lru cache instances: - kwd_mark = (object(),) # separate positional and keyword args sentinel = object() # unique object used to signal cache misses + make_key = _CacheKey # build a key from the function arguments PREV, NEXT, KEY, RESULT = 0, 1, 2, 3 # names for the link fields def decorating_function(user_function): @@ -182,20 +206,6 @@ root = [] # root of the circular doubly linked list root[:] = [root, root, None, None] # initialize by pointing to self - def make_key(args, kwds, typed, tuple=tuple, sorted=sorted, type=type): - # build a cache key from positional and keyword args - key = args - if kwds: - sorted_items = tuple(sorted(kwds.items())) - key += kwd_mark - key += tuple(k for k, v in sorted_items) - key += tuple(v for k, v in sorted_items) - if typed: - key += tuple(type(v) for v in args) - if kwds: - key += tuple(type(v) for k, v in sorted_items) - return key - if maxsize == 0: def wrapper(*args, **kwds): @@ -210,7 +220,7 @@ def wrapper(*args, **kwds): # simple caching without ordering or size limit nonlocal hits, misses, currsize - key = make_key(args, kwds, typed) if kwds or typed else args + key = make_key(args, kwds, typed) result = cache_get(key, sentinel) if result is not sentinel: hits += 1 @@ -226,7 +236,7 @@ def wrapper(*args, **kwds): # size limited caching that tracks accesses by recency nonlocal root, hits, misses, currsize, full - key = make_key(args, kwds, typed) if kwds or typed else args + key = make_key(args, kwds, typed) with lock: link = cache_get(key) if link is not None: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 07:58:53 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 07:58:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314371=3A_Support_b?= =?utf8?q?zip2_in_zipfile_module=2E?= Message-ID: http://hg.python.org/cpython/rev/028e8e0b03e8 changeset: 76682:028e8e0b03e8 user: Martin v. L?wis date: Tue May 01 07:58:44 2012 +0200 summary: Issue #14371: Support bzip2 in zipfile module. Patch by Serhiy Storchaka. files: Doc/library/zipfile.rst | 28 ++- Lib/test/support.py | 9 +- Lib/test/test_zipfile.py | 126 ++++++++++++- Lib/zipfile.py | 260 +++++++++++++++++--------- Misc/NEWS | 3 + 5 files changed, 331 insertions(+), 95 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -87,7 +87,22 @@ .. data:: ZIP_DEFLATED The numeric constant for the usual ZIP compression method. This requires the - zlib module. No other compression methods are currently supported. + zlib module. + + +.. data:: ZIP_BZIP2 + + The numeric constant for the BZIP2 compression method. This requires the + bz2 module. + + .. versionadded:: 3.3 + + .. note:: + + The ZIP file format specification has included support for bzip2 compression + since 2001. However, some tools (including older Python releases) do not + support it, and may either refuse to process the ZIP file altogether, or + fail to extract individual files. .. seealso:: @@ -118,9 +133,11 @@ adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. *compression* is the ZIP compression method to use when writing the archive, - and should be :const:`ZIP_STORED` or :const:`ZIP_DEFLATED`; unrecognized - values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` - is specified but the :mod:`zlib` module is not available, :exc:`RuntimeError` + and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`; or + :const:`ZIP_DEFLATED`; unrecognized + values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` or + :const:`ZIP_BZIP2` is specified but the corresponded module + (:mod:`zlib` or :mod:`bz2`) is not available, :exc:`RuntimeError` is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` zipfile will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GB. If it is false (the default) :mod:`zipfile` @@ -143,6 +160,9 @@ .. versionadded:: 3.2 Added the ability to use :class:`ZipFile` as a context manager. + .. versionchanged:: 3.3 + Added support for :mod:`bzip2` compression. + .. method:: ZipFile.close() diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -40,6 +40,11 @@ except ImportError: zlib = None +try: + import bz2 +except ImportError: + bz2 = None + __all__ = [ "Error", "TestFailed", "ResourceDenied", "import_module", "verbose", "use_resources", "max_memuse", "record_original_stdout", @@ -57,7 +62,7 @@ "get_attribute", "swap_item", "swap_attr", "requires_IEEE_754", "TestHandler", "Matcher", "can_symlink", "skip_unless_symlink", "import_fresh_module", "requires_zlib", "PIPE_MAX_SIZE", "failfast", - "anticipate_failure", "run_with_tz" + "anticipate_failure", "run_with_tz", "requires_bz2" ] class Error(Exception): @@ -506,6 +511,8 @@ requires_zlib = unittest.skipUnless(zlib, 'requires zlib') +requires_bz2 = unittest.skipUnless(bz2, 'requires bz2') + is_jython = sys.platform.startswith('java') # Filename used for testing diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -13,7 +13,7 @@ from random import randint, random from unittest import skipUnless -from test.support import TESTFN, run_unittest, findfile, unlink, requires_zlib +from test.support import TESTFN, run_unittest, findfile, unlink, requires_zlib, requires_bz2 TESTFN2 = TESTFN + "2" TESTFNDIR = TESTFN + "d" @@ -313,6 +313,54 @@ self.assertEqual(openobj.read(1), b'1') self.assertEqual(openobj.read(1), b'2') + @requires_bz2 + def test_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_open_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_open_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_random_open_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_random_open_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_readline_read_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_readline_read_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_readline_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_readline_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_readlines_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_readlines_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_iterlines_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_iterlines_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_low_compression_bzip2(self): + """Check for cases where compressed data is larger than original.""" + # Create the ZIP archive + with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_BZIP2) as zipfp: + zipfp.writestr("strfile", '12') + + # Get an open object for strfile + with zipfile.ZipFile(TESTFN2, "r", zipfile.ZIP_BZIP2) as zipfp: + with zipfp.open("strfile") as openobj: + self.assertEqual(openobj.read(1), b'1') + self.assertEqual(openobj.read(1), b'2') + def test_absolute_arcnames(self): with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_STORED) as zipfp: zipfp.write(TESTFN, "/absolute") @@ -453,6 +501,13 @@ info = zipfp.getinfo('b.txt') self.assertEqual(info.compress_type, zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_writestr_compression_bzip2(self): + zipfp = zipfile.ZipFile(TESTFN2, "w") + zipfp.writestr("b.txt", "hello world", compress_type=zipfile.ZIP_BZIP2) + info = zipfp.getinfo('b.txt') + self.assertEqual(info.compress_type, zipfile.ZIP_BZIP2) + def zip_test_writestr_permissions(self, f, compression): # Make sure that writestr creates files with mode 0600, # when it is passed a name rather than a ZipInfo instance. @@ -626,6 +681,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_test(f, zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_test(f, zipfile.ZIP_BZIP2) + def test_absolute_arcnames(self): with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_STORED, allowZip64=True) as zipfp: @@ -754,6 +814,18 @@ b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00' b'\x00afilePK\x05\x06\x00\x00\x00\x00\x01\x00' b'\x01\x003\x00\x00\x003\x00\x00\x00\x00\x00'), + zipfile.ZIP_BZIP2: ( + b'PK\x03\x04\x14\x03\x00\x00\x0c\x00nu\x0c=FA' + b'KE8\x00\x00\x00n\x00\x00\x00\x05\x00\x00\x00af' + b'ileBZh91AY&SY\xd4\xa8\xca' + b'\x7f\x00\x00\x0f\x11\x80@\x00\x06D\x90\x80 \x00 \xa5' + b'P\xd9!\x03\x03\x13\x13\x13\x89\xa9\xa9\xc2u5:\x9f' + b'\x8b\xb9"\x9c(HjTe?\x80PK\x01\x02\x14' + b'\x03\x14\x03\x00\x00\x0c\x00nu\x0c=FAKE8' + b'\x00\x00\x00n\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00' + b'\x00 \x80\x80\x81\x00\x00\x00\x00afilePK' + b'\x05\x06\x00\x00\x00\x00\x01\x00\x01\x003\x00\x00\x00[\x00' + b'\x00\x00\x00\x00'), } def test_unicode_filenames(self): @@ -1007,6 +1079,10 @@ def test_testzip_with_bad_crc_deflated(self): self.check_testzip_with_bad_crc(zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_testzip_with_bad_crc_bzip2(self): + self.check_testzip_with_bad_crc(zipfile.ZIP_BZIP2) + def check_read_with_bad_crc(self, compression): """Tests that files with bad CRCs raise a BadZipFile exception when read.""" zipdata = self.zips_with_bad_crc[compression] @@ -1035,6 +1111,10 @@ def test_read_with_bad_crc_deflated(self): self.check_read_with_bad_crc(zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_read_with_bad_crc_bzip2(self): + self.check_read_with_bad_crc(zipfile.ZIP_BZIP2) + def check_read_return_size(self, compression): # Issue #9837: ZipExtFile.read() shouldn't return more bytes # than requested. @@ -1055,6 +1135,10 @@ def test_read_return_size_deflated(self): self.check_read_return_size(zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_read_return_size_bzip2(self): + self.check_read_return_size(zipfile.ZIP_BZIP2) + def test_empty_zipfile(self): # Check that creating a file in 'w' or 'a' mode and closing without # adding any files to the archives creates a valid empty ZIP file @@ -1196,6 +1280,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_test(f, zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_test(f, zipfile.ZIP_BZIP2) + def zip_open_test(self, f, compression): self.make_test_archive(f, compression) @@ -1236,6 +1325,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_open_test(f, zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_open_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_open_test(f, zipfile.ZIP_BZIP2) + def zip_random_open_test(self, f, compression): self.make_test_archive(f, compression) @@ -1264,6 +1358,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_random_open_test(f, zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_random_open_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_random_open_test(f, zipfile.ZIP_BZIP2) + @requires_zlib class TestsWithMultipleOpens(unittest.TestCase): @@ -1483,6 +1582,31 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.iterlines_test(f, zipfile.ZIP_DEFLATED) + @requires_bz2 + def test_read_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.read_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_readline_read_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.readline_read_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_readline_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.readline_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_readlines_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.readlines_test(f, zipfile.ZIP_BZIP2) + + @requires_bz2 + def test_iterlines_bzip2(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.iterlines_test(f, zipfile.ZIP_BZIP2) + def tearDown(self): for sep, fn in self.arcfiles.items(): os.remove(fn) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -22,7 +22,13 @@ zlib = None crc32 = binascii.crc32 -__all__ = ["BadZipFile", "BadZipfile", "error", "ZIP_STORED", "ZIP_DEFLATED", +try: + import bz2 # We may need its compression method +except ImportError: + bz2 = None + +__all__ = ["BadZipFile", "BadZipfile", "error", + "ZIP_STORED", "ZIP_DEFLATED", "ZIP_BZIP2" "is_zipfile", "ZipInfo", "ZipFile", "PyZipFile", "LargeZipFile"] class BadZipFile(Exception): @@ -45,8 +51,13 @@ # constants for Zip file compression methods ZIP_STORED = 0 ZIP_DEFLATED = 8 +ZIP_BZIP2 = 12 # Other ZIP compression methods not supported +DEFAULT_VERSION = 20 +ZIP64_VERSION = 45 +BZIP2_VERSION = 46 + # Below are some formats and associated data for reading/writing headers using # the struct module. The names and structures of headers/records are those used # in the PKWARE description of the ZIP file format: @@ -313,8 +324,8 @@ else: # Assume everything else is unix-y self.create_system = 3 # System which created ZIP archive - self.create_version = 20 # Version which created ZIP archive - self.extract_version = 20 # Version needed to extract archive + self.create_version = DEFAULT_VERSION # Version which created ZIP archive + self.extract_version = DEFAULT_VERSION # Version needed to extract archive self.reserved = 0 # Must be zero self.flag_bits = 0 # ZIP flag bits self.volume = 0 # Volume number of file header @@ -341,6 +352,7 @@ extra = self.extra + min_version = 0 if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT: # File is larger than what fits into a 4 byte integer, # fall back to the ZIP64 extension @@ -349,9 +361,13 @@ 1, struct.calcsize(fmt)-4, file_size, compress_size) file_size = 0xffffffff compress_size = 0xffffffff - self.extract_version = max(45, self.extract_version) - self.create_version = max(45, self.extract_version) + min_version = ZIP64_VERSION + if self.compress_type == ZIP_BZIP2: + min_version = max(BZIP2_VERSION, min_version) + + self.extract_version = max(min_version, self.extract_version) + self.create_version = max(min_version, self.create_version) filename, flag_bits = self._encodeFilenameFlags() header = struct.pack(structFileHeader, stringFileHeader, self.extract_version, self.reserved, flag_bits, @@ -461,6 +477,41 @@ self._UpdateKeys(c) return c + +def _check_compression(compression): + if compression == ZIP_STORED: + pass + elif compression == ZIP_DEFLATED: + if not zlib: + raise RuntimeError( + "Compression requires the (missing) zlib module") + elif compression == ZIP_BZIP2: + if not bz2: + raise RuntimeError( + "Compression requires the (missing) bz2 module") + else: + raise RuntimeError("That compression method is not supported") + + +def _get_compressor(compress_type): + if compress_type == ZIP_DEFLATED: + return zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, + zlib.DEFLATED, -15) + elif compress_type == ZIP_BZIP2: + return bz2.BZ2Compressor() + else: + return None + + +def _get_decompressor(compress_type): + if compress_type == ZIP_DEFLATED: + return zlib.decompressobj(-15) + elif compress_type == ZIP_BZIP2: + return bz2.BZ2Decompressor() + else: + return None + + class ZipExtFile(io.BufferedIOBase): """File-like object for reading an archive member. Is returned by ZipFile.open(). @@ -482,13 +533,12 @@ self._close_fileobj = close_fileobj self._compress_type = zipinfo.compress_type - self._compress_size = zipinfo.compress_size self._compress_left = zipinfo.compress_size + self._left = zipinfo.file_size - if self._compress_type == ZIP_DEFLATED: - self._decompressor = zlib.decompressobj(-15) - self._unconsumed = b'' + self._decompressor = _get_decompressor(self._compress_type) + self._eof = False self._readbuffer = b'' self._offset = 0 @@ -563,7 +613,11 @@ """Returns buffered bytes without advancing the position.""" if n > len(self._readbuffer) - self._offset: chunk = self.read(n) - self._offset -= len(chunk) + if len(chunk) > self._offset: + self._readbuffer = chunk + self._readbuffer[self._offset:] + self._offset = 0 + else: + self._offset -= len(chunk) # Return up to 512 bytes to reduce allocation overhead for tight loops. return self._readbuffer[self._offset: self._offset + 512] @@ -575,80 +629,121 @@ """Read and return up to n bytes. If the argument is omitted, None, or negative, data is read and returned until EOF is reached.. """ - buf = b'' - if n is None: - n = -1 - while True: - if n < 0: - data = self.read1(n) - elif n > len(buf): - data = self.read1(n - len(buf)) - else: - return buf - if len(data) == 0: - return buf + if n is None or n < 0: + buf = self._readbuffer[self._offset:] + self._readbuffer = b'' + self._offset = 0 + while not self._eof: + buf += self._read1(self.MAX_N) + return buf + + n -= len(self._readbuffer) - self._offset + if n < 0: + buf = self._readbuffer[self._offset:n] + self._offset += len(buf) + return buf + + buf = self._readbuffer[self._offset:] + self._readbuffer = b'' + self._offset = 0 + while n > 0 and not self._eof: + data = self._read1(n) + if n < len(data): + self._readbuffer = data + self._offset = n + buf += data[:n] + break buf += data + n -= len(data) + return buf - def _update_crc(self, newdata, eof): + def _update_crc(self, newdata): # Update the CRC using the given data. if self._expected_crc is None: # No need to compute the CRC if we don't have a reference value return self._running_crc = crc32(newdata, self._running_crc) & 0xffffffff # Check the CRC if we're at the end of the file - if eof and self._running_crc != self._expected_crc: + if self._eof and self._running_crc != self._expected_crc: raise BadZipFile("Bad CRC-32 for file %r" % self.name) def read1(self, n): """Read up to n bytes with at most one read() system call.""" - # Simplify algorithm (branching) by transforming negative n to large n. - if n < 0 or n is None: - n = self.MAX_N + if n is None or n < 0: + buf = self._readbuffer[self._offset:] + self._readbuffer = b'' + self._offset = 0 + data = self._read1(self.MAX_N) + buf += data + return buf - # Bytes available in read buffer. - len_readbuffer = len(self._readbuffer) - self._offset + n -= len(self._readbuffer) - self._offset + if n < 0: + buf = self._readbuffer[self._offset:n] + self._offset += len(buf) + return buf + + buf = self._readbuffer[self._offset:] + self._readbuffer = b'' + self._offset = 0 + if n > 0: + data = self._read1(n) + if n < len(data): + self._readbuffer = data + self._offset = n + data = data[:n] + buf += data + return buf + + def _read1(self, n): + # Read up to n compressed bytes with at most one read() system call, + # decrypt and decompress them. + if self._eof or n <= 0: + return b'' # Read from file. - if self._compress_left > 0 and n > len_readbuffer + len(self._unconsumed): - nbytes = n - len_readbuffer - len(self._unconsumed) - nbytes = max(nbytes, self.MIN_READ_SIZE) - nbytes = min(nbytes, self._compress_left) + if self._compress_type == ZIP_DEFLATED: + ## Handle unconsumed data. + data = self._decompressor.unconsumed_tail + if n > len(data): + data += self._read2(n - len(data)) + else: + data = self._read2(n) - data = self._fileobj.read(nbytes) - self._compress_left -= len(data) + if self._compress_type == ZIP_STORED: + self._eof = self._compress_left <= 0 + elif self._compress_type == ZIP_DEFLATED: + n = max(n, self.MIN_READ_SIZE) + data = self._decompressor.decompress(data, n) + self._eof = (self._decompressor.eof or + self._compress_left <= 0 and + not self._decompressor.unconsumed_tail) + if self._eof: + data += self._decompressor.flush() + else: + data = self._decompressor.decompress(data) + self._eof = self._decompressor.eof or self._compress_left <= 0 - if data and self._decrypter is not None: - data = bytes(map(self._decrypter, data)) + data = data[:self._left] + self._left -= len(data) + if self._left <= 0: + self._eof = True + self._update_crc(data) + return data - if self._compress_type == ZIP_STORED: - self._update_crc(data, eof=(self._compress_left==0)) - self._readbuffer = self._readbuffer[self._offset:] + data - self._offset = 0 - else: - # Prepare deflated bytes for decompression. - self._unconsumed += data + def _read2(self, n): + if self._compress_left <= 0: + return b'' - # Handle unconsumed data. - if (len(self._unconsumed) > 0 and n > len_readbuffer and - self._compress_type == ZIP_DEFLATED): - data = self._decompressor.decompress( - self._unconsumed, - max(n - len_readbuffer, self.MIN_READ_SIZE) - ) + n = max(n, self.MIN_READ_SIZE) + n = min(n, self._compress_left) - self._unconsumed = self._decompressor.unconsumed_tail - eof = len(self._unconsumed) == 0 and self._compress_left == 0 - if eof: - data += self._decompressor.flush() + data = self._fileobj.read(n) + self._compress_left -= len(data) - self._update_crc(data, eof=eof) - self._readbuffer = self._readbuffer[self._offset:] + data - self._offset = 0 - - # Read from buffer. - data = self._readbuffer[self._offset: self._offset + n] - self._offset += len(data) + if self._decrypter is not None: + data = bytes(map(self._decrypter, data)) return data def close(self): @@ -667,7 +762,8 @@ file: Either the path to the file, or a file-like object. If it is a path, the file will be opened and closed by ZipFile. mode: The mode can be either read "r", write "w" or append "a". - compression: ZIP_STORED (no compression) or ZIP_DEFLATED (requires zlib). + compression: ZIP_STORED (no compression), ZIP_DEFLATED (requires zlib) or + ZIP_BZIP2 (requires bz2). allowZip64: if True ZipFile will create files with ZIP64 extensions when needed, otherwise it will raise an exception when this would be necessary. @@ -681,14 +777,7 @@ if mode not in ("r", "w", "a"): raise RuntimeError('ZipFile() requires mode "r", "w", or "a"') - if compression == ZIP_STORED: - pass - elif compression == ZIP_DEFLATED: - if not zlib: - raise RuntimeError( - "Compression requires the (missing) zlib module") - else: - raise RuntimeError("That compression method is not supported") + _check_compression(compression) self._allowZip64 = allowZip64 self._didModify = False @@ -1067,11 +1156,7 @@ if not self.fp: raise RuntimeError( "Attempt to write ZIP archive that was already closed") - if zinfo.compress_type == ZIP_DEFLATED and not zlib: - raise RuntimeError( - "Compression requires the (missing) zlib module") - if zinfo.compress_type not in (ZIP_STORED, ZIP_DEFLATED): - raise RuntimeError("That compression method is not supported") + _check_compression(zinfo.compress_type) if zinfo.file_size > ZIP64_LIMIT: if not self._allowZip64: raise LargeZipFile("Filesize would require ZIP64 extensions") @@ -1122,17 +1207,13 @@ self.fp.write(zinfo.FileHeader()) return + cmpr = _get_compressor(zinfo.compress_type) with open(filename, "rb") as fp: # Must overwrite CRC and sizes with correct data later zinfo.CRC = CRC = 0 zinfo.compress_size = compress_size = 0 zinfo.file_size = file_size = 0 self.fp.write(zinfo.FileHeader()) - if zinfo.compress_type == ZIP_DEFLATED: - cmpr = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, - zlib.DEFLATED, -15) - else: - cmpr = None while 1: buf = fp.read(1024 * 8) if not buf: @@ -1189,9 +1270,8 @@ self._writecheck(zinfo) self._didModify = True zinfo.CRC = crc32(data) & 0xffffffff # CRC-32 checksum - if zinfo.compress_type == ZIP_DEFLATED: - co = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, - zlib.DEFLATED, -15) + co = _get_compressor(zinfo.compress_type) + if co: data = co.compress(data) + co.flush() zinfo.compress_size = len(data) # Compressed size else: @@ -1243,18 +1323,20 @@ header_offset = zinfo.header_offset extra_data = zinfo.extra + min_version = 0 if extra: # Append a ZIP64 field to the extra's extra_data = struct.pack( ' http://hg.python.org/cpython/rev/596b0eaeece8 changeset: 76683:596b0eaeece8 user: Martin v. L?wis date: Tue May 01 08:38:01 2012 +0200 summary: Detect unsupported compression types. files: Lib/test/test_zipfile.py | 11 +++++++++++ Lib/zipfile.py | 21 +++++++++++++++++++-- 2 files changed, 30 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -992,6 +992,17 @@ caught.""" self.assertRaises(RuntimeError, zipfile.ZipFile, TESTFN, "w", -1) + def test_unsupported_compression(self): + # data is declared as shrunk, but actually deflated + data = (b'PK\x03\x04.\x00\x00\x00\x01\x00\xe4C\xa1@\x00\x00\x00' + b'\x00\x02\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00x\x03\x00PK\x01' + b'\x02.\x03.\x00\x00\x00\x01\x00\xe4C\xa1@\x00\x00\x00\x00\x02\x00\x00' + b'\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' + b'\x80\x01\x00\x00\x00\x00xPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00' + b'/\x00\x00\x00!\x00\x00\x00\x00\x00') + with zipfile.ZipFile(io.BytesIO(data), 'r') as zipf: + self.assertRaises(NotImplementedError, zipf.open, 'x') + def test_null_byte_in_filename(self): """Check that a filename containing a null byte is properly terminated.""" diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -504,12 +504,29 @@ def _get_decompressor(compress_type): - if compress_type == ZIP_DEFLATED: + if compress_type == ZIP_STORED: + return None + elif compress_type == ZIP_DEFLATED: return zlib.decompressobj(-15) elif compress_type == ZIP_BZIP2: return bz2.BZ2Decompressor() else: - return None + unknown_compressors = { + 1: 'shrink', + 2: 'reduce', + 3: 'reduce', + 4: 'reduce', + 5: 'reduce', + 6: 'implode', + 9: 'enhanced deflate', + 10: 'implode', + 14: 'lzma', + } + descr = unknown_compressors.get(compress_type) + if descr: + raise NotImplementedError("compression type %d (%s)" % (compress_type, descr)) + else: + raise NotImplementedError("compression type %d" % (compress_type,)) class ZipExtFile(io.BufferedIOBase): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 08:44:19 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 08:44:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Recognize_unsupported_featu?= =?utf8?q?re_=22compressed_patch_data_set=22_from_zip_2=2E7=2E?= Message-ID: http://hg.python.org/cpython/rev/e2ff6113ec6c changeset: 76684:e2ff6113ec6c user: Martin v. L?wis date: Tue May 01 08:44:08 2012 +0200 summary: Recognize unsupported feature "compressed patch data set" from zip 2.7. files: Lib/zipfile.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1053,6 +1053,10 @@ if fheader[_FH_EXTRA_FIELD_LENGTH]: zef_file.read(fheader[_FH_EXTRA_FIELD_LENGTH]) + if zinfo.flag_bits & 0x20: + # Zip 2.7: compressed patched data + raise NotImplementedError("compressed patched data (flag bit 5)") + if zinfo.flag_bits & 0x800: # UTF-8 filename fname_str = fname.decode("utf-8") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 08:50:51 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 08:50:51 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_421_changes_by_Eric_+_form?= =?utf8?q?atting/spelling_fixes=2E?= Message-ID: http://hg.python.org/peps/rev/3e5accbd68f4 changeset: 4329:3e5accbd68f4 user: Georg Brandl date: Tue May 01 08:51:22 2012 +0200 summary: PEP 421 changes by Eric + formatting/spelling fixes. files: pep-0421.txt | 371 ++++++++++++++++++++++++++------------ 1 files changed, 251 insertions(+), 120 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -13,11 +13,11 @@ Abstract ======== -This PEP introduces a new variable for the sys module: ``sys.implementation``. -The variable holds consolidated information about the implementation of -the running interpreter. Thus ``sys.implementation`` is the source to -which the standard library may look for implementation-specific -information. +This PEP introduces a new attribute for the ``sys`` module: +``sys.implementation``. The attribute holds consolidated information +about the implementation of the running interpreter. Thus +``sys.implementation`` is the source to which the standard library may +look for implementation-specific information. The proposal in this PEP is in line with a broader emphasis on making Python friendlier to alternate implementations. It describes the new @@ -34,121 +34,211 @@ viable alternate implementations of Python. Consider, however, the nearly two decades of CPython-centric Python -(i.e. most of its existance). That focus had understandably contributed -to quite a few CPython-specific artifacts both in the standard library -and exposed in the interpreter. Though the core developers have made an -effort in recent years to address this, quite a few of the artifacts -remain. +(i.e. most of its existence). That focus had understandably +contributed to quite a few CPython-specific artifacts both in the +standard library and exposed in the interpreter. Though the core +developers have made an effort in recent years to address this, quite +a few of the artifacts remain. -Part of the solution is presented in this PEP: a single namespace on +Part of the solution is presented in this PEP: a single namespace in which to consolidate implementation specifics. This will help focus -efforts to differentiate the implementation specifics from the language. -Additionally, it will foster a multiple-implementation mindset. +efforts to differentiate the implementation specifics from the +language. Additionally, it will foster a multiple-implementation +mindset. Proposal ======== -We will add ``sys.implementation``, in the sys module, as a namespace to -contain implementation-specific information. +We will add a new variable to the ``sys`` module, called +``sys.implementation``, as a mapping to contain +implementation-specific information. -The contents of this namespace will remain fixed during interpreter +The contents of this mapping will remain fixed during interpreter execution and through the course of an implementation version. This -ensures behaviors don't change between versions which depend on variables -in ``sys.implementation``. +ensures behaviors don't change between versions which depend on +variables in ``sys.implementation``. -``sys.implementation`` is a dictionary, as opposed to any form of "named" -tuple (a la ``sys.version_info``). This is partly because it doesn't -have meaning as a sequence, and partly because it's a potentially more -variable data structure. - -The namespace will contain at least the variables described in the -`Required Variables`_ section below. However, implementations are free -to add other implementation information there. Some possible extra -variables are described in the `Other Possible Variables`_ section. +The mapping will contain at least the values described in the +`Required Variables`_ section below. However, implementations are +free to add other implementation information there. Some +*conceivable* extra variables are described in the `Other Possible +Variables`_ section. This proposal takes a conservative approach in requiring only two -variables. As more become appropriate, they may be added with discretion. +variables. As more become appropriate, they may be added with +discretion. Required Variables -------------------- These are variables in ``sys.implementation`` on which the standard -library would rely, meaning they would need to be defined: +library would rely, meaning implementers must define them: -name - the name of the implementation (case sensitive). +**name** + This is the name of the implementation (case sensitive). Examples + include 'PyPy', 'Jython', 'IronPython', and 'CPython'. -version - the version of the implementation, as opposed to the version of the - language it implements. This would use a standard format, similar to - ``sys.version_info`` (see `Version Format`_). +**version** + This is the version of the implementation, as opposed to the + version of the language it implements. This value conforms to the + format described in `Version Format`_. Other Possible Variables ------------------------ -These variables could be useful, but don't necessarily have a clear use -case presently: +These variables could be useful, but don't necessarily have a clear +use case presently. They are listed here as values to consider in the +future, with appropriate discussion, relative to clear use-cases. +Their descriptions are therefore intentionally unhindered by details. -cache_tag - a string used for the PEP 3147 cache tag (e.g. 'cpython33' for - CPython 3.3). The name and version from above could be used to - compose this, though an implementation may want something else. - However, module caching is not a requirement of implementations, nor - is the use of cache tags. +**cache_tag** + A string used for the PEP 3147 cache tag (e.g. 'cpython33' for + CPython 3.3). The name and version from above could be used to + compose this, though an implementation may want something else. + However, module caching is not a requirement of implementations, + nor is the use of cache tags. -repository - the implementation's repository URL. +**vcs_url** + The URL pointing to the main VCS repository for the implementation + project. -repository_revision - the revision identifier for the implementation. +**vcs_revision_id** + A value that identifies the VCS revision of the implementation + that is currently running. -build_toolchain - identifies the tools used to build the interpreter. +**build_toolchain** + Identifies the tools used to build the interpreter. -url (or website) - the URL of the implementation's site. +**homepage** + The URL of the implementation's website. -site_prefix - the preferred site prefix for this implementation. +**site_prefix** + The preferred site prefix for this implementation. -runtime - the run-time environment in which the interpreter is running. +**runtime** + The run-time environment in which the interpreter is running, as + in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* + Executable". -gc_type - the type of garbage collection used. +**gc_type** + The type of garbage collection used, like "reference counting" or + "mark and sweep". Version Format -------------- -XXX same as sys.version_info? +A main point of ``sys.implementation`` is to contain information that +will be used internally in the standard library. In order to +facilitate the usefulness of a version variable, its value should be +in a consistent format across implementations. + +XXX Subject to feedback + +As such, the format of ``sys.implementation['version']`` must follow +that of ``sys.version_info``, which is effectively a named tuple. It +is a familiar format and generally consistent with normal version +format conventions. + +Keep in mind, however, that ``sys.implementation['version']`` is the +version of the Python *implementation*, while ``sys.version_info`` +(and friends) is the version of the Python language. Rationale ========= The status quo for implementation-specific information gives us that -information in a more fragile, harder to maintain way. It's spread out -over different modules or inferred from other information, as we see with -``platform.python_implementation()``. +information in a more fragile, harder to maintain way. It's spread +out over different modules or inferred from other information, as we +see with ``platform.python_implementation()``. -This PEP is the main alternative to that approach. It consolidates the -implementation-specific information into a single namespace and makes -explicit that which was implicit. +This PEP is the main alternative to that approach. It consolidates +the implementation-specific information into a single namespace and +makes explicit that which was implicit. -With the single-namespace-under-sys so straightforward, no alternatives -have been considered for this PEP. + +Why a Dictionary? +----------------- + +A dictionary reflects a simple namespace. It maps names to values and +that's it. Really that's all we need. + +The alternatives to a dictionary are considered separately here: + +**"Named" Tuple** + +The first alternative is a namedtuple or a structseq or some other +tuple type with dotted access (a la ``sys.version_info``). This type +is immutable and simple. It is a well established pattern for +implementation- specific variables in Python. Dotted access on a +namespace is also very convenient. + +However, sys.implementation does not have meaning as a sequence. +Also, unlike other such variables, it has the potential for more +variation over time. Finally, generic lookup may favor dicts:: + + cache_tag = sys.implementation.get('cache_tag') + + vs. + + cache_tag = getattr(sys.implementation.get, 'cache_tag', None) + +If a named tuple were used, we'd be very clear in the documentation +that the length and order of the value are not reliable. Iterability +would not be guaranteed. + +**Concrete Class** + +Another option would be to have a dedicated class, of which +``sys.implementation`` is an instance. This would facilitate the +dotted access of a "named" tuple, without exposing any confusion about +ordering and iteration. + +One downside is that you lose the immutable aspect of a tuple, making +it less clear that ``sys.implementation`` should not be manipulated. +Another downside is that classes often imply the presence (or +possibility) of methods, which may be misleading in this case. + +**Module** + +Using a module instead of a dict is another option. It has similar +characteristics to an instance, but with a slight hint of immutability +(at least by convention). Such a module could be a stand-alone sub- +module of ``sys`` or added on, like ``os.path``. Unlike a concrete +class, no new type would be necessary. + +The downsides are similar to those of a concrete class. + + +Why a Part of ``sys``? +---------------------- + +The ``sys`` module should hold the new namespace because ``sys`` is +the depot for interpreter-centric variables and functions. Many +implementation-specific variables are already found in ``sys``. + + +Why Strict Constraints on Any of the Values? +-------------------------------------------- + +As already noted in `Version Format`_, values in +``sys.implementation`` are intended for use by the standard library. +Constraining those values, essentially specifying an API for them, +allows them to be used consistently, regardless of how they are +implemented otherwise. + Discussion ========== -The topic of ``sys.implementation`` came up on the python-ideas list in -2009, where the reception was broadly positive [1]_. I revived the -discussion recently while working on a pure-python ``imp.get_tag()`` [2]_. -The messages in `issue #14673`_ are also relevant. +The topic of ``sys.implementation`` came up on the python-ideas list +in 2009, where the reception was broadly positive [1]_. I revived the +discussion recently while working on a pure-python ``imp.get_tag()`` +[2]_. The messages in `issue #14673`_ are also relevant. Use-cases @@ -159,62 +249,66 @@ "explicit is better than implicit" -The platform module guesses the python implementation by looking for -clues in a couple different sys variables [3]_. However, this approach -is fragile. Beyond that, it's limited to those implementations that core -developers have blessed by special-casing them in the platform module. +The ``platform`` module guesses the python implementation by looking +for clues in a couple different ``sys`` variables [3]_. However, this +approach is fragile. Beyond that, it's limited to those +implementations that core developers have blessed by special-casing +them in the ``platform`` module. With ``sys.implementation`` the various implementations would -*explicitly* set the values in their own version of the sys module. +*explicitly* set the values in their own version of the ``sys`` +module. -Aside from the guessing, another concern is that the platform module is -part of the stdlib, which ideally would minimize implementation details -such as would be moved to ``sys.implementation``. +Aside from the guessing, another concern is that the ``platform`` +module is part of the stdlib, which ideally would minimize +implementation details such as would be moved to +``sys.implementation``. -Any overlap between ``sys.implementation`` and the platform module would -simply defer to ``sys.implementation`` (with the same interface in -platform wrapping it). +Any overlap between ``sys.implementation`` and the ``platform`` module +would simply defer to ``sys.implementation`` (with the same interface +in ``platform`` wrapping it). Cache Tag Generation in Frozen Importlib ---------------------------------------- -PEP 3147 defined the use of a module cache and cache tags for file names. -The importlib bootstrap code, frozen into the Python binary as of 3.3, -uses the cache tags during the import process. Part of the project to -bootstrap importlib has been to clean out of Lib/import.c any code that -did not need to be there. +PEP 3147 defined the use of a module cache and cache tags for file +names. The importlib bootstrap code, frozen into the Python binary as +of 3.3, uses the cache tags during the import process. Part of the +project to bootstrap importlib has been to clean out of +`Python/import.c` any code that did not need to be there. -The cache tag defined in Lib/import.c was hard-coded to +The cache tag defined in `Python/import.c` was hard-coded to ``"cpython" MAJOR MINOR`` [4]_. For importlib the options are either hard-coding it in the same way, or guessing the implementation in the same way as does ``platform.python_implementation()``. -As long as the hard-coded tag is limited to CPython-specific code, it's -livable. However, inasmuch as other Python implementations use the -importlib code to work with the module cache, a hard-coded tag would -become a problem.. +As long as the hard-coded tag is limited to CPython-specific code, +it's livable. However, inasmuch as other Python implementations use +the importlib code to work with the module cache, a hard-coded tag +would become a problem.. -Directly using the platform module in this case is a non-starter. Any -module used in the importlib bootstrap must be built-in or frozen, -neither of which apply to the platform module. This is the point that -led to the recent interest in ``sys.implementation``. +Directly using the ``platform`` module in this case is a non-starter. +Any module used in the importlib bootstrap must be built-in or frozen, +neither of which apply to the ``platform`` module. This is the point +that led to the recent interest in ``sys.implementation``. -Regardless of how the implementation name is gotten, the version to use -for the cache tag is more likely to be the implementation version rather -than the language version. That implementation version is not readily +Regardless of the outcome for the implementation name used, another +problem relates to the version used in the cache tag. That version is +likely to be the implementation version rather than the language +version. However, the implementation version is not readily identified anywhere in the standard library. Implementation-Specific Tests ----------------------------- -XXX - -http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509 -http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246 -http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252 -http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275 +Currently there are a number of implementation-specific tests in the +test suite under ``Lib/test``. The test support module +(`Lib/test/support.py`_) provides some functionality for dealing with +these tests. However, like the ``platform`` module, ``test.support`` +must do some guessing that ``sys.implementation`` would render +unnecessary. Jython's ``os.name`` Hack @@ -225,14 +319,8 @@ http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512 -Impact on CPython -================= - -XXX - - -Feedback From Other Python Implementators -========================================= +Feedback From Other Python Implementers +======================================= IronPython ---------- @@ -253,28 +341,62 @@ Past Efforts ============ -XXX PEP 3139 -XXX PEP 399 +PEP 3139 +-------- + +This PEP from 2008 recommended a clean-up of the ``sys`` module in +part by extracting implementation-specific variables and functions +into a separate module. PEP 421 is a much lighter version of that +idea. While PEP 3139 was rejected, its goals are reflected in PEP 421 +to a large extent, though with a much lighter approach. + + +PEP 399 +------- + +This informational PEP dictates policy regarding the standard library, +helping to make it friendlier to alternate implementations. PEP 421 +is proposed in that same spirit. + + +Alternatives +============ + +With the single-namespace-under-sys so straightforward, no +alternatives have been considered for this PEP. Open Issues =========== -* What are the long-term objectives for sys.implementation? +* What are the long-term objectives for ``sys.implementation``? - - pull in implementation detail from the main sys namespace and - elsewhere (PEP 3137 lite). + - possibly pull in implementation details from the main ``sys`` + namespace and elsewhere (PEP 3137 lite). * Alternatives to the approach dictated by this PEP? -* ``sys.implementation`` as a proper namespace rather than a dict. It - would be it's own module or an instance of a concrete class. +* Do we really want to commit to using a dict for + ``sys.implementation``? + +Backward compatibility issues will make it difficult to change our +minds later. + +The type we use ultimately depends on how general we expect the +consumption of ``sys.implementation`` to be. If its practicality is +oriented toward internal use then the data structure is not as +critical. However, ``sys.implementation`` is intended to have a +non-localized impact across the standard library and the interpreter. +It's better to *not* make hacking it become an attractive nuisance, +regardless of our intentions for usage. + +* use (immutable?) nameddict (analogous to namedtuple/structseq)? Implementation ============== -The implementatation of this PEP is covered in `issue #14673`_. +The implementation of this PEP is covered in `issue #14673`_. References @@ -288,8 +410,17 @@ .. [4] http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c#l121 +.. [5] Examples of implementation-specific handling in test.support: + +| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509 +| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246 +| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252 +| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275 + .. _issue #14673: http://bugs.python.org/issue14673 +.. _Lib/test/support.py: http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 08:54:24 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 08:54:24 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_=22local_variables_section?= =?utf8?b?Ii4=?= Message-ID: http://hg.python.org/peps/rev/0ce76199d9f3 changeset: 4330:0ce76199d9f3 user: Georg Brandl date: Tue May 01 08:54:55 2012 +0200 summary: Fix "local variables section". files: pep-0421.txt | 35 ++++++++++++++++++----------------- 1 files changed, 18 insertions(+), 17 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -379,16 +379,16 @@ * Do we really want to commit to using a dict for ``sys.implementation``? -Backward compatibility issues will make it difficult to change our -minds later. + Backward compatibility issues will make it difficult to change our + minds later. -The type we use ultimately depends on how general we expect the -consumption of ``sys.implementation`` to be. If its practicality is -oriented toward internal use then the data structure is not as -critical. However, ``sys.implementation`` is intended to have a -non-localized impact across the standard library and the interpreter. -It's better to *not* make hacking it become an attractive nuisance, -regardless of our intentions for usage. + The type we use ultimately depends on how general we expect the + consumption of ``sys.implementation`` to be. If its practicality is + oriented toward internal use then the data structure is not as + critical. However, ``sys.implementation`` is intended to have a + non-localized impact across the standard library and the + interpreter. It's better to *not* make hacking it become an + attractive nuisance, regardless of our intentions for usage. * use (immutable?) nameddict (analogous to namedtuple/structseq)? @@ -425,14 +425,15 @@ Copyright ========= - This document has been placed in the public domain. +This document has been placed in the public domain. -Local Variables: -mode: indented-text -indent-tabs-mode: nil -sentence-end-double-space: t -fill-column: 70 -coding: utf-8 -End: +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 09:00:30 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:00:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_missing_comma_in_=5F=5F?= =?utf8?b?YWxsX18gbGlzdC4=?= Message-ID: http://hg.python.org/cpython/rev/7bdac82c16ab changeset: 76685:7bdac82c16ab user: Georg Brandl date: Tue May 01 09:00:59 2012 +0200 summary: Add missing comma in __all__ list. files: Lib/zipfile.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -28,7 +28,7 @@ bz2 = None __all__ = ["BadZipFile", "BadZipfile", "error", - "ZIP_STORED", "ZIP_DEFLATED", "ZIP_BZIP2" + "ZIP_STORED", "ZIP_DEFLATED", "ZIP_BZIP2", "is_zipfile", "ZipInfo", "ZipFile", "PyZipFile", "LargeZipFile"] class BadZipFile(Exception): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 09:20:50 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:20:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_test=5Fpdb=3A_fix_failure_o?= =?utf8?q?f_test=5Fissue13183_in_debug_mode=2C_and_make_sure_files_are?= Message-ID: http://hg.python.org/cpython/rev/1b174a117e19 changeset: 76686:1b174a117e19 user: Georg Brandl date: Tue May 01 09:21:16 2012 +0200 summary: test_pdb: fix failure of test_issue13183 in debug mode, and make sure files are cleaned up. files: Lib/test/test_pdb.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -604,6 +604,7 @@ filename = 'main.py' with open(filename, 'w') as f: f.write(textwrap.dedent(script)) + self.addCleanup(support.unlink, filename) cmd = [sys.executable, '-m', 'pdb', filename] stdout = stderr = None with subprocess.Popen(cmd, stdout=subprocess.PIPE, @@ -660,9 +661,11 @@ """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) + self.addCleanup(support.unlink, 'bar.py') stdout, stderr = self.run_pdb(script, commands) - self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3], - 'Fail to step into the caller after a return') + self.assertTrue( + any('main.py(5)foo()->None' in l for l in stdout.splitlines()), + 'Fail to step into the caller after a return') def tearDown(self): support.unlink(support.TESTFN) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 09:29:27 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:29:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Regenerate_pydoc_topics=2E?= Message-ID: http://hg.python.org/cpython/rev/3de108ef63f0 changeset: 76687:3de108ef63f0 user: Georg Brandl date: Tue May 01 09:26:47 2012 +0200 summary: Regenerate pydoc topics. files: Lib/pydoc_data/topics.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Sun Apr 1 13:46:17 2012 +# Autogenerated by Sphinx on Tue May 1 09:25:14 2012 topics = {'assert': '\nThe ``assert`` statement\n************************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, ``assert expression``, is equivalent to\n\n if __debug__:\n if not expression: raise AssertionError\n\nThe extended form, ``assert expression1, expression2``, is equivalent\nto\n\n if __debug__:\n if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that ``__debug__`` and ``AssertionError``\nrefer to the built-in variables with those names. In the current\nimplementation, the built-in variable ``__debug__`` is ``True`` under\nnormal circumstances, ``False`` when optimization is requested\n(command line option -O). The current code generator emits no code\nfor an assert statement when optimization is requested at compile\ntime. Note that it is unnecessary to include the source code for the\nexpression that failed in the error message; it will be displayed as\npart of the stack trace.\n\nAssignments to ``__debug__`` are illegal. The value for the built-in\nvariable is determined when the interpreter starts.\n', 'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n target_list ::= target ("," target)* [","]\n target ::= identifier\n | "(" target_list ")"\n | "[" target_list "]"\n | attributeref\n | subscription\n | slicing\n | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable. The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n that target.\n\n* If the target list is a comma-separated list of targets: The object\n must be an iterable with the same number of items as there are\n targets in the target list, and the items are assigned, from left to\n right, to the corresponding targets.\n\n * If the target list contains one target prefixed with an asterisk,\n called a "starred" target: The object must be a sequence with at\n least as many items as there are targets in the target list, minus\n one. The first items of the sequence are assigned, from left to\n right, to the targets before the starred target. The final items\n of the sequence are assigned to the targets after the starred\n target. A list of the remaining items in the sequence is then\n assigned to the starred target (the list can be empty).\n\n * Else: The object must be a sequence with the same number of items\n as there are targets in the target list, and the items are\n assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n * If the name does not occur in a ``global`` or ``nonlocal``\n statement in the current code block: the name is bound to the\n object in the current local namespace.\n\n * Otherwise: the name is bound to the object in the global namespace\n or the outer namespace determined by ``nonlocal``, respectively.\n\n The name is rebound if it was already bound. This may cause the\n reference count for the object previously bound to the name to reach\n zero, causing the object to be deallocated and its destructor (if it\n has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in square\n brackets: The object must be an iterable with the same number of\n items as there are targets in the target list, and its items are\n assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n the reference is evaluated. It should yield an object with\n assignable attributes; if this is not the case, ``TypeError`` is\n raised. That object is then asked to assign the assigned object to\n the given attribute; if it cannot perform the assignment, it raises\n an exception (usually but not necessarily ``AttributeError``).\n\n Note: If the object is a class instance and the attribute reference\n occurs on both sides of the assignment operator, the RHS expression,\n ``a.x`` can access either an instance attribute or (if no instance\n attribute exists) a class attribute. The LHS target ``a.x`` is\n always set as an instance attribute, creating it if necessary.\n Thus, the two occurrences of ``a.x`` do not necessarily refer to the\n same attribute: if the RHS expression refers to a class attribute,\n the LHS creates a new instance attribute as the target of the\n assignment:\n\n class Cls:\n x = 3 # class variable\n inst = Cls()\n inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3\n\n This description does not necessarily apply to descriptor\n attributes, such as properties created with ``property()``.\n\n* If the target is a subscription: The primary expression in the\n reference is evaluated. It should yield either a mutable sequence\n object (such as a list) or a mapping object (such as a dictionary).\n Next, the subscript expression is evaluated.\n\n If the primary is a mutable sequence object (such as a list), the\n subscript must yield an integer. If it is negative, the sequence\'s\n length is added to it. The resulting value must be a nonnegative\n integer less than the sequence\'s length, and the sequence is asked\n to assign the assigned object to its item with that index. If the\n index is out of range, ``IndexError`` is raised (assignment to a\n subscripted sequence cannot add new items to a list).\n\n If the primary is a mapping object (such as a dictionary), the\n subscript must have a type compatible with the mapping\'s key type,\n and the mapping is then asked to create a key/datum pair which maps\n the subscript to the assigned object. This can either replace an\n existing key/value pair with the same key value, or insert a new\n key/value pair (if no key with the same value existed).\n\n For user-defined objects, the ``__setitem__()`` method is called\n with appropriate arguments.\n\n* If the target is a slicing: The primary expression in the reference\n is evaluated. It should yield a mutable sequence object (such as a\n list). The assigned object should be a sequence object of the same\n type. Next, the lower and upper bound expressions are evaluated,\n insofar they are present; defaults are zero and the sequence\'s\n length. The bounds should evaluate to integers. If either bound is\n negative, the sequence\'s length is added to it. The resulting\n bounds are clipped to lie between zero and the sequence\'s length,\n inclusive. Finally, the sequence object is asked to replace the\n slice with the items of the assigned sequence. The length of the\n slice may be different from the length of the assigned sequence,\n thus changing the length of the target sequence, if the object\n allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample ``a, b = b, a`` swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe! For instance, the\nfollowing program prints ``[0, 2]``:\n\n x = [0, 1]\n i = 0\n i, x[i] = 1, 2\n print(x)\n\nSee also:\n\n **PEP 3132** - Extended Iterable Unpacking\n The specification for the ``*target`` feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n augtarget ::= identifier | attributeref | subscription | slicing\n augop ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like ``x += 1`` can be rewritten as\n``x = x + 1`` to achieve a similar, but not exactly equal effect. In\nthe augmented version, ``x`` is only evaluated once. Also, when\npossible, the actual operation is performed *in-place*, meaning that\nrather than creating a new object and assigning that to the target,\nthe old object is modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n', 'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name. See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a ``NameError`` exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them. The transformation inserts the\nclass name in front of the name, with leading underscores removed, and\na single underscore inserted in front of the class name. For example,\nthe identifier ``__spam`` occurring in a class named ``Ham`` will be\ntransformed to ``_Ham__spam``. This transformation is independent of\nthe syntactical context in which the identifier is used. If the\ntransformed name is extremely long (longer than 255 characters),\nimplementation defined truncation may happen. If the class name\nconsists only of underscores, no transformation is done.\n', @@ -41,7 +41,7 @@ 'identifiers': '\nIdentifiers and keywords\n************************\n\nIdentifiers (also referred to as *names*) are described by the\nfollowing lexical definitions.\n\nThe syntax of identifiers in Python is based on the Unicode standard\nannex UAX-31, with elaboration and changes as defined below; see also\n**PEP 3131** for further details.\n\nWithin the ASCII range (U+0001..U+007F), the valid characters for\nidentifiers are the same as in Python 2.x: the uppercase and lowercase\nletters ``A`` through ``Z``, the underscore ``_`` and, except for the\nfirst character, the digits ``0`` through ``9``.\n\nPython 3.0 introduces additional characters from outside the ASCII\nrange (see **PEP 3131**). For these characters, the classification\nuses the version of the Unicode Character Database as included in the\n``unicodedata`` module.\n\nIdentifiers are unlimited in length. Case is significant.\n\n identifier ::= xid_start xid_continue*\n id_start ::= \n id_continue ::= \n xid_start ::= \n xid_continue ::= \n\nThe Unicode category codes mentioned above stand for:\n\n* *Lu* - uppercase letters\n\n* *Ll* - lowercase letters\n\n* *Lt* - titlecase letters\n\n* *Lm* - modifier letters\n\n* *Lo* - other letters\n\n* *Nl* - letter numbers\n\n* *Mn* - nonspacing marks\n\n* *Mc* - spacing combining marks\n\n* *Nd* - decimal numbers\n\n* *Pc* - connector punctuations\n\n* *Other_ID_Start* - explicit list of characters in PropList.txt to\n support backwards compatibility\n\n* *Other_ID_Continue* - likewise\n\nAll identifiers are converted into the normal form NFKC while parsing;\ncomparison of identifiers is based on NFKC.\n\nA non-normative HTML file listing all valid identifier characters for\nUnicode 4.1 can be found at http://www.dcl.hpi.uni-\npotsdam.de/home/loewis/table-3131.html.\n\n\nKeywords\n========\n\nThe following identifiers are used as reserved words, or *keywords* of\nthe language, and cannot be used as ordinary identifiers. They must\nbe spelled exactly as written here:\n\n False class finally is return\n None continue for lambda try\n True def from nonlocal while\n and del global not with\n as elif if or yield\n assert else import pass\n break except in raise\n\n\nReserved classes of identifiers\n===============================\n\nCertain classes of identifiers (besides keywords) have special\nmeanings. These classes are identified by the patterns of leading and\ntrailing underscore characters:\n\n``_*``\n Not imported by ``from module import *``. The special identifier\n ``_`` is used in the interactive interpreter to store the result of\n the last evaluation; it is stored in the ``builtins`` module. When\n not in interactive mode, ``_`` has no special meaning and is not\n defined. See section *The import statement*.\n\n Note: The name ``_`` is often used in conjunction with\n internationalization; refer to the documentation for the\n ``gettext`` module for more information on this convention.\n\n``__*__``\n System-defined names. These names are defined by the interpreter\n and its implementation (including the standard library). Current\n system names are discussed in the *Special method names* section\n and elsewhere. More will likely be defined in future versions of\n Python. *Any* use of ``__*__`` names, in any context, that does\n not follow explicitly documented use, is subject to breakage\n without warning.\n\n``__*``\n Class-private names. Names in this category, when used within the\n context of a class definition, are re-written to use a mangled form\n to help avoid name clashes between "private" attributes of base and\n derived classes. See section *Identifiers (Names)*.\n', 'if': '\nThe ``if`` statement\n********************\n\nThe ``if`` statement is used for conditional execution:\n\n if_stmt ::= "if" expression ":" suite\n ( "elif" expression ":" suite )*\n ["else" ":" suite]\n\nIt selects exactly one of the suites by evaluating the expressions one\nby one until one is found to be true (see section *Boolean operations*\nfor the definition of true and false); then that suite is executed\n(and no other part of the ``if`` statement is executed or evaluated).\nIf all expressions are false, the suite of the ``else`` clause, if\npresent, is executed.\n', 'imaginary': '\nImaginary literals\n******************\n\nImaginary literals are described by the following lexical definitions:\n\n imagnumber ::= (floatnumber | intpart) ("j" | "J")\n\nAn imaginary literal yields a complex number with a real part of 0.0.\nComplex numbers are represented as a pair of floating point numbers\nand have the same restrictions on their range. To create a complex\nnumber with a nonzero real part, add a floating point number to it,\ne.g., ``(3+4j)``. Some examples of imaginary literals:\n\n 3.14j 10.j 10j .001j 1e100j 3.14e-10j\n', - 'import': '\nThe ``import`` statement\n************************\n\n import_stmt ::= "import" module ["as" name] ( "," module ["as" name] )*\n | "from" relative_module "import" identifier ["as" name]\n ( "," identifier ["as" name] )*\n | "from" relative_module "import" "(" identifier ["as" name]\n ( "," identifier ["as" name] )* [","] ")"\n | "from" module "import" "*"\n module ::= (identifier ".")* identifier\n relative_module ::= "."* module | "."+\n name ::= identifier\n\nImport statements are executed in two steps: (1) find a module, and\ninitialize it if necessary; (2) define a name or names in the local\nnamespace (of the scope where the ``import`` statement occurs). The\nstatement comes in two forms differing on whether it uses the ``from``\nkeyword. The first form (without ``from``) repeats these steps for\neach identifier in the list. The form with ``from`` performs step (1)\nonce, and then performs step (2) repeatedly. For a reference\nimplementation of step (1), see the ``importlib`` module.\n\nTo understand how step (1) occurs, one must first understand how\nPython handles hierarchical naming of modules. To help organize\nmodules and provide a hierarchy in naming, Python has a concept of\npackages. A package can contain other packages and modules while\nmodules cannot contain other modules or packages. From a file system\nperspective, packages are directories and modules are files. The\noriginal specification for packages is still available to read,\nalthough minor details have changed since the writing of that\ndocument.\n\nOnce the name of the module is known (unless otherwise specified, the\nterm "module" will refer to both packages and modules), searching for\nthe module or package can begin. The first place checked is\n``sys.modules``, the cache of all modules that have been imported\npreviously. If the module is found there then it is used in step (2)\nof import unless ``None`` is found in ``sys.modules``, in which case\n``ImportError`` is raised.\n\nIf the module is not found in the cache, then ``sys.meta_path`` is\nsearched (the specification for ``sys.meta_path`` can be found in\n**PEP 302**). The object is a list of *finder* objects which are\nqueried in order as to whether they know how to load the module by\ncalling their ``find_module()`` method with the name of the module. If\nthe module happens to be contained within a package (as denoted by the\nexistence of a dot in the name), then a second argument to\n``find_module()`` is given as the value of the ``__path__`` attribute\nfrom the parent package (everything up to the last dot in the name of\nthe module being imported). If a finder can find the module it returns\na *loader* (discussed later) or returns ``None``.\n\nIf none of the finders on ``sys.meta_path`` are able to find the\nmodule then some implicitly defined finders are queried.\nImplementations of Python vary in what implicit meta path finders are\ndefined. The one they all do define, though, is one that handles\n``sys.path_hooks``, ``sys.path_importer_cache``, and ``sys.path``.\n\nThe implicit finder searches for the requested module in the "paths"\nspecified in one of two places ("paths" do not have to be file system\npaths). If the module being imported is supposed to be contained\nwithin a package then the second argument passed to ``find_module()``,\n``__path__`` on the parent package, is used as the source of paths. If\nthe module is not contained in a package then ``sys.path`` is used as\nthe source of paths.\n\nOnce the source of paths is chosen it is iterated over to find a\nfinder that can handle that path. The dict at\n``sys.path_importer_cache`` caches finders for paths and is checked\nfor a finder. If the path does not have a finder cached then\n``sys.path_hooks`` is searched by calling each object in the list with\na single argument of the path, returning a finder or raises\n``ImportError``. If a finder is returned then it is cached in\n``sys.path_importer_cache`` and then used for that path entry. If no\nfinder can be found but the path exists then a value of ``None`` is\nstored in ``sys.path_importer_cache`` to signify that an implicit,\nfile-based finder that handles modules stored as individual files\nshould be used for that path. If the path does not exist then a finder\nwhich always returns ``None`` is placed in the cache for the path.\n\nIf no finder can find the module then ``ImportError`` is raised.\nOtherwise some finder returned a loader whose ``load_module()`` method\nis called with the name of the module to load (see **PEP 302** for the\noriginal definition of loaders). A loader has several responsibilities\nto perform on a module it loads. First, if the module already exists\nin ``sys.modules`` (a possibility if the loader is called outside of\nthe import machinery) then it is to use that module for initialization\nand not a new module. But if the module does not exist in\n``sys.modules`` then it is to be added to that dict before\ninitialization begins. If an error occurs during loading of the module\nand it was added to ``sys.modules`` it is to be removed from the dict.\nIf an error occurs but the module was already in ``sys.modules`` it is\nleft in the dict.\n\nThe loader must set several attributes on the module. ``__name__`` is\nto be set to the name of the module. ``__file__`` is to be the "path"\nto the file unless the module is built-in (and thus listed in\n``sys.builtin_module_names``) in which case the attribute is not set.\nIf what is being imported is a package then ``__path__`` is to be set\nto a list of paths to be searched when looking for modules and\npackages contained within the package being imported. ``__package__``\nis optional but should be set to the name of package that contains the\nmodule or package (the empty string is used for module not contained\nin a package). ``__loader__`` is also optional but should be set to\nthe loader object that is loading the module.\n\nIf an error occurs during loading then the loader raises\n``ImportError`` if some other exception is not already being\npropagated. Otherwise the loader returns the module that was loaded\nand initialized.\n\nWhen step (1) finishes without raising an exception, step (2) can\nbegin.\n\nThe first form of ``import`` statement binds the module name in the\nlocal namespace to the module object, and then goes on to import the\nnext identifier, if any. If the module name is followed by ``as``,\nthe name following ``as`` is used as the local name for the module.\n\nThe ``from`` form does not bind the module name: it goes through the\nlist of identifiers, looks each one of them up in the module found in\nstep (1), and binds the name in the local namespace to the object thus\nfound. As with the first form of ``import``, an alternate local name\ncan be supplied by specifying "``as`` localname". If a name is not\nfound, ``ImportError`` is raised. If the list of identifiers is\nreplaced by a star (``\'*\'``), all public names defined in the module\nare bound in the local namespace of the ``import`` statement.\n\nThe *public names* defined by a module are determined by checking the\nmodule\'s namespace for a variable named ``__all__``; if defined, it\nmust be a sequence of strings which are names defined or imported by\nthat module. The names given in ``__all__`` are all considered public\nand are required to exist. If ``__all__`` is not defined, the set of\npublic names includes all names found in the module\'s namespace which\ndo not begin with an underscore character (``\'_\'``). ``__all__``\nshould contain the entire public API. It is intended to avoid\naccidentally exporting items that are not part of the API (such as\nlibrary modules which were imported and used within the module).\n\nThe ``from`` form with ``*`` may only occur in a module scope. The\nwild card form of import --- ``import *`` --- is only allowed at the\nmodule level. Attempting to use it in class or function definitions\nwill raise a ``SyntaxError``.\n\nWhen specifying what module to import you do not have to specify the\nabsolute name of the module. When a module or package is contained\nwithin another package it is possible to make a relative import within\nthe same top package without having to mention the package name. By\nusing leading dots in the specified module or package after ``from``\nyou can specify how high to traverse up the current package hierarchy\nwithout specifying exact names. One leading dot means the current\npackage where the module making the import exists. Two dots means up\none package level. Three dots is up two levels, etc. So if you execute\n``from . import mod`` from a module in the ``pkg`` package then you\nwill end up importing ``pkg.mod``. If you execute ``from ..subpkg2\nimport mod`` from within ``pkg.subpkg1`` you will import\n``pkg.subpkg2.mod``. The specification for relative imports is\ncontained within **PEP 328**.\n\n``importlib.import_module()`` is provided to support applications that\ndetermine which modules need to be loaded dynamically.\n\n\nFuture statements\n=================\n\nA *future statement* is a directive to the compiler that a particular\nmodule should be compiled using syntax or semantics that will be\navailable in a specified future release of Python. The future\nstatement is intended to ease migration to future versions of Python\nthat introduce incompatible changes to the language. It allows use of\nthe new features on a per-module basis before the release in which the\nfeature becomes standard.\n\n future_statement ::= "from" "__future__" "import" feature ["as" name]\n ("," feature ["as" name])*\n | "from" "__future__" "import" "(" feature ["as" name]\n ("," feature ["as" name])* [","] ")"\n feature ::= identifier\n name ::= identifier\n\nA future statement must appear near the top of the module. The only\nlines that can appear before a future statement are:\n\n* the module docstring (if any),\n\n* comments,\n\n* blank lines, and\n\n* other future statements.\n\nThe features recognized by Python 3.0 are ``absolute_import``,\n``division``, ``generators``, ``unicode_literals``,\n``print_function``, ``nested_scopes`` and ``with_statement``. They\nare all redundant because they are always enabled, and only kept for\nbackwards compatibility.\n\nA future statement is recognized and treated specially at compile\ntime: Changes to the semantics of core constructs are often\nimplemented by generating different code. It may even be the case\nthat a new feature introduces new incompatible syntax (such as a new\nreserved word), in which case the compiler may need to parse the\nmodule differently. Such decisions cannot be pushed off until\nruntime.\n\nFor any given release, the compiler knows which feature names have\nbeen defined, and raises a compile-time error if a future statement\ncontains a feature not known to it.\n\nThe direct runtime semantics are the same as for any import statement:\nthere is a standard module ``__future__``, described later, and it\nwill be imported in the usual way at the time the future statement is\nexecuted.\n\nThe interesting runtime semantics depend on the specific feature\nenabled by the future statement.\n\nNote that there is nothing special about the statement:\n\n import __future__ [as name]\n\nThat is not a future statement; it\'s an ordinary import statement with\nno special semantics or syntax restrictions.\n\nCode compiled by calls to the built-in functions ``exec()`` and\n``compile()`` that occur in a module ``M`` containing a future\nstatement will, by default, use the new syntax or semantics associated\nwith the future statement. This can be controlled by optional\narguments to ``compile()`` --- see the documentation of that function\nfor details.\n\nA future statement typed at an interactive interpreter prompt will\ntake effect for the rest of the interpreter session. If an\ninterpreter is started with the *-i* option, is passed a script name\nto execute, and the script includes a future statement, it will be in\neffect in the interactive session started after the script is\nexecuted.\n\nSee also:\n\n **PEP 236** - Back to the __future__\n The original proposal for the __future__ mechanism.\n', + 'import': '\nThe ``import`` statement\n************************\n\n import_stmt ::= "import" module ["as" name] ( "," module ["as" name] )*\n | "from" relative_module "import" identifier ["as" name]\n ( "," identifier ["as" name] )*\n | "from" relative_module "import" "(" identifier ["as" name]\n ( "," identifier ["as" name] )* [","] ")"\n | "from" module "import" "*"\n module ::= (identifier ".")* identifier\n relative_module ::= "."* module | "."+\n name ::= identifier\n\nImport statements are executed in two steps: (1) find a module, and\ninitialize it if necessary; (2) define a name or names in the local\nnamespace (of the scope where the ``import`` statement occurs). The\nstatement comes in two forms differing on whether it uses the ``from``\nkeyword. The first form (without ``from``) repeats these steps for\neach identifier in the list. The form with ``from`` performs step (1)\nonce, and then performs step (2) repeatedly. For a reference\nimplementation of step (1), see the ``importlib`` module.\n\nTo understand how step (1) occurs, one must first understand how\nPython handles hierarchical naming of modules. To help organize\nmodules and provide a hierarchy in naming, Python has a concept of\npackages. A package can contain other packages and modules while\nmodules cannot contain other modules or packages. From a file system\nperspective, packages are directories and modules are files. The\noriginal specification for packages is still available to read,\nalthough minor details have changed since the writing of that\ndocument.\n\nOnce the name of the module is known (unless otherwise specified, the\nterm "module" will refer to both packages and modules), searching for\nthe module or package can begin. The first place checked is\n``sys.modules``, the cache of all modules that have been imported\npreviously. If the module is found there then it is used in step (2)\nof import unless ``None`` is found in ``sys.modules``, in which case\n``ImportError`` is raised.\n\nIf the module is not found in the cache, then ``sys.meta_path`` is\nsearched (the specification for ``sys.meta_path`` can be found in\n**PEP 302**). The object is a list of *finder* objects which are\nqueried in order as to whether they know how to load the module by\ncalling their ``find_module()`` method with the name of the module. If\nthe module happens to be contained within a package (as denoted by the\nexistence of a dot in the name), then a second argument to\n``find_module()`` is given as the value of the ``__path__`` attribute\nfrom the parent package (everything up to the last dot in the name of\nthe module being imported). If a finder can find the module it returns\na *loader* (discussed later) or returns ``None``.\n\nIf none of the finders on ``sys.meta_path`` are able to find the\nmodule then some implicitly defined finders are queried.\nImplementations of Python vary in what implicit meta path finders are\ndefined. The one they all do define, though, is one that handles\n``sys.path_hooks``, ``sys.path_importer_cache``, and ``sys.path``.\n\nThe implicit finder searches for the requested module in the "paths"\nspecified in one of two places ("paths" do not have to be file system\npaths). If the module being imported is supposed to be contained\nwithin a package then the second argument passed to ``find_module()``,\n``__path__`` on the parent package, is used as the source of paths. If\nthe module is not contained in a package then ``sys.path`` is used as\nthe source of paths.\n\nOnce the source of paths is chosen it is iterated over to find a\nfinder that can handle that path. The dict at\n``sys.path_importer_cache`` caches finders for paths and is checked\nfor a finder. If the path does not have a finder cached then\n``sys.path_hooks`` is searched by calling each object in the list with\na single argument of the path, returning a finder or raises\n``ImportError``. If a finder is returned then it is cached in\n``sys.path_importer_cache`` and then used for that path entry. If no\nfinder can be found but the path exists then a value of ``None`` is\nstored in ``sys.path_importer_cache`` to signify that an implicit,\nfile-based finder that handles modules stored as individual files\nshould be used for that path. If the path does not exist then a finder\nwhich always returns ``None`` is placed in the cache for the path.\n\nIf no finder can find the module then ``ImportError`` is raised.\nOtherwise some finder returned a loader whose ``load_module()`` method\nis called with the name of the module to load (see **PEP 302** for the\noriginal definition of loaders). A loader has several responsibilities\nto perform on a module it loads. First, if the module already exists\nin ``sys.modules`` (a possibility if the loader is called outside of\nthe import machinery) then it is to use that module for initialization\nand not a new module. But if the module does not exist in\n``sys.modules`` then it is to be added to that dict before\ninitialization begins. If an error occurs during loading of the module\nand it was added to ``sys.modules`` it is to be removed from the dict.\nIf an error occurs but the module was already in ``sys.modules`` it is\nleft in the dict.\n\nThe loader must set several attributes on the module. ``__name__`` is\nto be set to the name of the module. ``__file__`` is to be the "path"\nto the file unless the module is built-in (and thus listed in\n``sys.builtin_module_names``) in which case the attribute is not set.\nIf what is being imported is a package then ``__path__`` is to be set\nto a list of paths to be searched when looking for modules and\npackages contained within the package being imported. ``__package__``\nis optional but should be set to the name of package that contains the\nmodule or package (the empty string is used for module not contained\nin a package). ``__loader__`` is also optional but should be set to\nthe loader object that is loading the module. While loaders are\nrequired to return the module they loaded, import itself always\nretrieves any modules it returns from ``sys.modules``.\n\nIf an error occurs during loading then the loader raises\n``ImportError`` if some other exception is not already being\npropagated. Otherwise the loader returns the module that was loaded\nand initialized.\n\nWhen step (1) finishes without raising an exception, step (2) can\nbegin.\n\nThe first form of ``import`` statement binds the module name in the\nlocal namespace to the module object, and then goes on to import the\nnext identifier, if any. If the module name is followed by ``as``,\nthe name following ``as`` is used as the local name for the module.\n\nThe ``from`` form does not bind the module name: it goes through the\nlist of identifiers, looks each one of them up in the module found in\nstep (1), and binds the name in the local namespace to the object thus\nfound. As with the first form of ``import``, an alternate local name\ncan be supplied by specifying "``as`` localname". If a name is not\nfound, ``ImportError`` is raised. If the list of identifiers is\nreplaced by a star (``\'*\'``), all public names defined in the module\nare bound in the local namespace of the ``import`` statement.\n\nThe *public names* defined by a module are determined by checking the\nmodule\'s namespace for a variable named ``__all__``; if defined, it\nmust be a sequence of strings which are names defined or imported by\nthat module. The names given in ``__all__`` are all considered public\nand are required to exist. If ``__all__`` is not defined, the set of\npublic names includes all names found in the module\'s namespace which\ndo not begin with an underscore character (``\'_\'``). ``__all__``\nshould contain the entire public API. It is intended to avoid\naccidentally exporting items that are not part of the API (such as\nlibrary modules which were imported and used within the module).\n\nThe ``from`` form with ``*`` may only occur in a module scope. The\nwild card form of import --- ``import *`` --- is only allowed at the\nmodule level. Attempting to use it in class or function definitions\nwill raise a ``SyntaxError``.\n\nWhen specifying what module to import you do not have to specify the\nabsolute name of the module. When a module or package is contained\nwithin another package it is possible to make a relative import within\nthe same top package without having to mention the package name. By\nusing leading dots in the specified module or package after ``from``\nyou can specify how high to traverse up the current package hierarchy\nwithout specifying exact names. One leading dot means the current\npackage where the module making the import exists. Two dots means up\none package level. Three dots is up two levels, etc. So if you execute\n``from . import mod`` from a module in the ``pkg`` package then you\nwill end up importing ``pkg.mod``. If you execute ``from ..subpkg2\nimport mod`` from within ``pkg.subpkg1`` you will import\n``pkg.subpkg2.mod``. The specification for relative imports is\ncontained within **PEP 328**.\n\n``importlib.import_module()`` is provided to support applications that\ndetermine which modules need to be loaded dynamically.\n\n\nFuture statements\n=================\n\nA *future statement* is a directive to the compiler that a particular\nmodule should be compiled using syntax or semantics that will be\navailable in a specified future release of Python. The future\nstatement is intended to ease migration to future versions of Python\nthat introduce incompatible changes to the language. It allows use of\nthe new features on a per-module basis before the release in which the\nfeature becomes standard.\n\n future_statement ::= "from" "__future__" "import" feature ["as" name]\n ("," feature ["as" name])*\n | "from" "__future__" "import" "(" feature ["as" name]\n ("," feature ["as" name])* [","] ")"\n feature ::= identifier\n name ::= identifier\n\nA future statement must appear near the top of the module. The only\nlines that can appear before a future statement are:\n\n* the module docstring (if any),\n\n* comments,\n\n* blank lines, and\n\n* other future statements.\n\nThe features recognized by Python 3.0 are ``absolute_import``,\n``division``, ``generators``, ``unicode_literals``,\n``print_function``, ``nested_scopes`` and ``with_statement``. They\nare all redundant because they are always enabled, and only kept for\nbackwards compatibility.\n\nA future statement is recognized and treated specially at compile\ntime: Changes to the semantics of core constructs are often\nimplemented by generating different code. It may even be the case\nthat a new feature introduces new incompatible syntax (such as a new\nreserved word), in which case the compiler may need to parse the\nmodule differently. Such decisions cannot be pushed off until\nruntime.\n\nFor any given release, the compiler knows which feature names have\nbeen defined, and raises a compile-time error if a future statement\ncontains a feature not known to it.\n\nThe direct runtime semantics are the same as for any import statement:\nthere is a standard module ``__future__``, described later, and it\nwill be imported in the usual way at the time the future statement is\nexecuted.\n\nThe interesting runtime semantics depend on the specific feature\nenabled by the future statement.\n\nNote that there is nothing special about the statement:\n\n import __future__ [as name]\n\nThat is not a future statement; it\'s an ordinary import statement with\nno special semantics or syntax restrictions.\n\nCode compiled by calls to the built-in functions ``exec()`` and\n``compile()`` that occur in a module ``M`` containing a future\nstatement will, by default, use the new syntax or semantics associated\nwith the future statement. This can be controlled by optional\narguments to ``compile()`` --- see the documentation of that function\nfor details.\n\nA future statement typed at an interactive interpreter prompt will\ntake effect for the rest of the interpreter session. If an\ninterpreter is started with the *-i* option, is passed a script name\nto execute, and the script includes a future statement, it will be in\neffect in the interactive session started after the script is\nexecuted.\n\nSee also:\n\n **PEP 236** - Back to the __future__\n The original proposal for the __future__ mechanism.\n', 'in': '\nComparisons\n***********\n\nUnlike C, all comparison operations in Python have the same priority,\nwhich is lower than that of any arithmetic, shifting or bitwise\noperation. Also unlike C, expressions like ``a < b < c`` have the\ninterpretation that is conventional in mathematics:\n\n comparison ::= or_expr ( comp_operator or_expr )*\n comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="\n | "is" ["not"] | ["not"] "in"\n\nComparisons yield boolean values: ``True`` or ``False``.\n\nComparisons can be chained arbitrarily, e.g., ``x < y <= z`` is\nequivalent to ``x < y and y <= z``, except that ``y`` is evaluated\nonly once (but in both cases ``z`` is not evaluated at all when ``x <\ny`` is found to be false).\n\nFormally, if *a*, *b*, *c*, ..., *y*, *z* are expressions and *op1*,\n*op2*, ..., *opN* are comparison operators, then ``a op1 b op2 c ... y\nopN z`` is equivalent to ``a op1 b and b op2 c and ... y opN z``,\nexcept that each expression is evaluated at most once.\n\nNote that ``a op1 b op2 c`` doesn\'t imply any kind of comparison\nbetween *a* and *c*, so that, e.g., ``x < y > z`` is perfectly legal\n(though perhaps not pretty).\n\nThe operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare\nthe values of two objects. The objects need not have the same type.\nIf both are numbers, they are converted to a common type. Otherwise,\nthe ``==`` and ``!=`` operators *always* consider objects of different\ntypes to be unequal, while the ``<``, ``>``, ``>=`` and ``<=``\noperators raise a ``TypeError`` when comparing objects of different\ntypes that do not implement these operators for the given pair of\ntypes. You can control comparison behavior of objects of non-built-in\ntypes by defining rich comparison methods like ``__gt__()``, described\nin section *Basic customization*.\n\nComparison of objects of the same type depends on the type:\n\n* Numbers are compared arithmetically.\n\n* The values ``float(\'NaN\')`` and ``Decimal(\'NaN\')`` are special. The\n are identical to themselves, ``x is x`` but are not equal to\n themselves, ``x != x``. Additionally, comparing any value to a\n not-a-number value will return ``False``. For example, both ``3 <\n float(\'NaN\')`` and ``float(\'NaN\') < 3`` will return ``False``.\n\n* Bytes objects are compared lexicographically using the numeric\n values of their elements.\n\n* Strings are compared lexicographically using the numeric equivalents\n (the result of the built-in function ``ord()``) of their characters.\n [3] String and bytes object can\'t be compared!\n\n* Tuples and lists are compared lexicographically using comparison of\n corresponding elements. This means that to compare equal, each\n element must compare equal and the two sequences must be of the same\n type and have the same length.\n\n If not equal, the sequences are ordered the same as their first\n differing elements. For example, ``[1,2,x] <= [1,2,y]`` has the\n same value as ``x <= y``. If the corresponding element does not\n exist, the shorter sequence is ordered first (for example, ``[1,2] <\n [1,2,3]``).\n\n* Mappings (dictionaries) compare equal if and only if they have the\n same ``(key, value)`` pairs. Order comparisons ``(\'<\', \'<=\', \'>=\',\n \'>\')`` raise ``TypeError``.\n\n* Sets and frozensets define comparison operators to mean subset and\n superset tests. Those relations do not define total orderings (the\n two sets ``{1,2}`` and {2,3} are not equal, nor subsets of one\n another, nor supersets of one another). Accordingly, sets are not\n appropriate arguments for functions which depend on total ordering.\n For example, ``min()``, ``max()``, and ``sorted()`` produce\n undefined results given a list of sets as inputs.\n\n* Most other objects of built-in types compare unequal unless they are\n the same object; the choice whether one object is considered smaller\n or larger than another one is made arbitrarily but consistently\n within one execution of a program.\n\nComparison of objects of the differing types depends on whether either\nof the types provide explicit support for the comparison. Most\nnumeric types can be compared with one another, but comparisons of\n``float`` and ``Decimal`` are not supported to avoid the inevitable\nconfusion arising from representation issues such as ``float(\'1.1\')``\nbeing inexactly represented and therefore not exactly equal to\n``Decimal(\'1.1\')`` which is. When cross-type comparison is not\nsupported, the comparison method returns ``NotImplemented``. This can\ncreate the illusion of non-transitivity between supported cross-type\ncomparisons and unsupported comparisons. For example, ``Decimal(2) ==\n2`` and ``2 == float(2)`` but ``Decimal(2) != float(2)``.\n\nThe operators ``in`` and ``not in`` test for membership. ``x in s``\nevaluates to true if *x* is a member of *s*, and false otherwise. ``x\nnot in s`` returns the negation of ``x in s``. All built-in sequences\nand set types support this as well as dictionary, for which ``in``\ntests whether a the dictionary has a given key. For container types\nsuch as list, tuple, set, frozenset, dict, or collections.deque, the\nexpression ``x in y`` is equivalent to ``any(x is e or x == e for e in\ny)``.\n\nFor the string and bytes types, ``x in y`` is true if and only if *x*\nis a substring of *y*. An equivalent test is ``y.find(x) != -1``.\nEmpty strings are always considered to be a substring of any other\nstring, so ``"" in "abc"`` will return ``True``.\n\nFor user-defined classes which define the ``__contains__()`` method,\n``x in y`` is true if and only if ``y.__contains__(x)`` is true.\n\nFor user-defined classes which do not define ``__contains__()`` but do\ndefine ``__iter__()``, ``x in y`` is true if some value ``z`` with ``x\n== z`` is produced while iterating over ``y``. If an exception is\nraised during the iteration, it is as if ``in`` raised that exception.\n\nLastly, the old-style iteration protocol is tried: if a class defines\n``__getitem__()``, ``x in y`` is true if and only if there is a non-\nnegative integer index *i* such that ``x == y[i]``, and all lower\ninteger indices do not raise ``IndexError`` exception. (If any other\nexception is raised, it is as if ``in`` raised that exception).\n\nThe operator ``not in`` is defined to have the inverse true value of\n``in``.\n\nThe operators ``is`` and ``is not`` test for object identity: ``x is\ny`` is true if and only if *x* and *y* are the same object. ``x is\nnot y`` yields the inverse truth value. [4]\n', 'integers': '\nInteger literals\n****************\n\nInteger literals are described by the following lexical definitions:\n\n integer ::= decimalinteger | octinteger | hexinteger | bininteger\n decimalinteger ::= nonzerodigit digit* | "0"+\n nonzerodigit ::= "1"..."9"\n digit ::= "0"..."9"\n octinteger ::= "0" ("o" | "O") octdigit+\n hexinteger ::= "0" ("x" | "X") hexdigit+\n bininteger ::= "0" ("b" | "B") bindigit+\n octdigit ::= "0"..."7"\n hexdigit ::= digit | "a"..."f" | "A"..."F"\n bindigit ::= "0" | "1"\n\nThere is no limit for the length of integer literals apart from what\ncan be stored in available memory.\n\nNote that leading zeros in a non-zero decimal number are not allowed.\nThis is for disambiguation with C-style octal literals, which Python\nused before version 3.0.\n\nSome examples of integer literals:\n\n 7 2147483647 0o177 0b100110111\n 3 79228162514264337593543950336 0o377 0x100000000\n 79228162514264337593543950336 0xdeadbeef\n', 'lambda': '\nLambdas\n*******\n\n lambda_form ::= "lambda" [parameter_list]: expression\n lambda_form_nocond ::= "lambda" [parameter_list]: expression_nocond\n\nLambda forms (lambda expressions) have the same syntactic position as\nexpressions. They are a shorthand to create anonymous functions; the\nexpression ``lambda arguments: expression`` yields a function object.\nThe unnamed object behaves like a function object defined with\n\n def (arguments):\n return expression\n\nSee section *Function definitions* for the syntax of parameter lists.\nNote that functions created with lambda forms cannot contain\nstatements or annotations.\n', @@ -68,7 +68,7 @@ 'try': '\nThe ``try`` statement\n*********************\n\nThe ``try`` statement specifies exception handlers and/or cleanup code\nfor a group of statements:\n\n try_stmt ::= try1_stmt | try2_stmt\n try1_stmt ::= "try" ":" suite\n ("except" [expression ["as" target]] ":" suite)+\n ["else" ":" suite]\n ["finally" ":" suite]\n try2_stmt ::= "try" ":" suite\n "finally" ":" suite\n\nThe ``except`` clause(s) specify one or more exception handlers. When\nno exception occurs in the ``try`` clause, no exception handler is\nexecuted. When an exception occurs in the ``try`` suite, a search for\nan exception handler is started. This search inspects the except\nclauses in turn until one is found that matches the exception. An\nexpression-less except clause, if present, must be last; it matches\nany exception. For an except clause with an expression, that\nexpression is evaluated, and the clause matches the exception if the\nresulting object is "compatible" with the exception. An object is\ncompatible with an exception if it is the class or a base class of the\nexception object or a tuple containing an item compatible with the\nexception.\n\nIf no except clause matches the exception, the search for an exception\nhandler continues in the surrounding code and on the invocation stack.\n[1]\n\nIf the evaluation of an expression in the header of an except clause\nraises an exception, the original search for a handler is canceled and\na search starts for the new exception in the surrounding code and on\nthe call stack (it is treated as if the entire ``try`` statement\nraised the exception).\n\nWhen a matching except clause is found, the exception is assigned to\nthe target specified after the ``as`` keyword in that except clause,\nif present, and the except clause\'s suite is executed. All except\nclauses must have an executable block. When the end of this block is\nreached, execution continues normally after the entire try statement.\n(This means that if two nested handlers exist for the same exception,\nand the exception occurs in the try clause of the inner handler, the\nouter handler will not handle the exception.)\n\nWhen an exception has been assigned using ``as target``, it is cleared\nat the end of the except clause. This is as if\n\n except E as N:\n foo\n\nwas translated to\n\n except E as N:\n try:\n foo\n finally:\n del N\n\nThis means the exception must be assigned to a different name to be\nable to refer to it after the except clause. Exceptions are cleared\nbecause with the traceback attached to them, they form a reference\ncycle with the stack frame, keeping all locals in that frame alive\nuntil the next garbage collection occurs.\n\nBefore an except clause\'s suite is executed, details about the\nexception are stored in the ``sys`` module and can be access via\n``sys.exc_info()``. ``sys.exc_info()`` returns a 3-tuple consisting of\nthe exception class, the exception instance and a traceback object\n(see section *The standard type hierarchy*) identifying the point in\nthe program where the exception occurred. ``sys.exc_info()`` values\nare restored to their previous values (before the call) when returning\nfrom a function that handled an exception.\n\nThe optional ``else`` clause is executed if and when control flows off\nthe end of the ``try`` clause. [2] Exceptions in the ``else`` clause\nare not handled by the preceding ``except`` clauses.\n\nIf ``finally`` is present, it specifies a \'cleanup\' handler. The\n``try`` clause is executed, including any ``except`` and ``else``\nclauses. If an exception occurs in any of the clauses and is not\nhandled, the exception is temporarily saved. The ``finally`` clause is\nexecuted. If there is a saved exception, it is re-raised at the end\nof the ``finally`` clause. If the ``finally`` clause raises another\nexception or executes a ``return`` or ``break`` statement, the saved\nexception is set as the context of the new exception. The exception\ninformation is not available to the program during execution of the\n``finally`` clause.\n\nWhen a ``return``, ``break`` or ``continue`` statement is executed in\nthe ``try`` suite of a ``try``...``finally`` statement, the\n``finally`` clause is also executed \'on the way out.\' A ``continue``\nstatement is illegal in the ``finally`` clause. (The reason is a\nproblem with the current implementation --- this restriction may be\nlifted in the future).\n\nAdditional information on exceptions can be found in section\n*Exceptions*, and information on using the ``raise`` statement to\ngenerate exceptions may be found in section *The raise statement*.\n', 'types': '\nThe standard type hierarchy\n***************************\n\nBelow is a list of the types that are built into Python. Extension\nmodules (written in C, Java, or other languages, depending on the\nimplementation) can define additional types. Future versions of\nPython may add types to the type hierarchy (e.g., rational numbers,\nefficiently stored arrays of integers, etc.), although such additions\nwill often be provided via the standard library instead.\n\nSome of the type descriptions below contain a paragraph listing\n\'special attributes.\' These are attributes that provide access to the\nimplementation and are not intended for general use. Their definition\nmay change in the future.\n\nNone\n This type has a single value. There is a single object with this\n value. This object is accessed through the built-in name ``None``.\n It is used to signify the absence of a value in many situations,\n e.g., it is returned from functions that don\'t explicitly return\n anything. Its truth value is false.\n\nNotImplemented\n This type has a single value. There is a single object with this\n value. This object is accessed through the built-in name\n ``NotImplemented``. Numeric methods and rich comparison methods may\n return this value if they do not implement the operation for the\n operands provided. (The interpreter will then try the reflected\n operation, or some other fallback, depending on the operator.) Its\n truth value is true.\n\nEllipsis\n This type has a single value. There is a single object with this\n value. This object is accessed through the literal ``...`` or the\n built-in name ``Ellipsis``. Its truth value is true.\n\n``numbers.Number``\n These are created by numeric literals and returned as results by\n arithmetic operators and arithmetic built-in functions. Numeric\n objects are immutable; once created their value never changes.\n Python numbers are of course strongly related to mathematical\n numbers, but subject to the limitations of numerical representation\n in computers.\n\n Python distinguishes between integers, floating point numbers, and\n complex numbers:\n\n ``numbers.Integral``\n These represent elements from the mathematical set of integers\n (positive and negative).\n\n There are two types of integers:\n\n Integers (``int``)\n\n These represent numbers in an unlimited range, subject to\n available (virtual) memory only. For the purpose of shift\n and mask operations, a binary representation is assumed, and\n negative numbers are represented in a variant of 2\'s\n complement which gives the illusion of an infinite string of\n sign bits extending to the left.\n\n Booleans (``bool``)\n These represent the truth values False and True. The two\n objects representing the values False and True are the only\n Boolean objects. The Boolean type is a subtype of the integer\n type, and Boolean values behave like the values 0 and 1,\n respectively, in almost all contexts, the exception being\n that when converted to a string, the strings ``"False"`` or\n ``"True"`` are returned, respectively.\n\n The rules for integer representation are intended to give the\n most meaningful interpretation of shift and mask operations\n involving negative integers.\n\n ``numbers.Real`` (``float``)\n These represent machine-level double precision floating point\n numbers. You are at the mercy of the underlying machine\n architecture (and C or Java implementation) for the accepted\n range and handling of overflow. Python does not support single-\n precision floating point numbers; the savings in processor and\n memory usage that are usually the reason for using these is\n dwarfed by the overhead of using objects in Python, so there is\n no reason to complicate the language with two kinds of floating\n point numbers.\n\n ``numbers.Complex`` (``complex``)\n These represent complex numbers as a pair of machine-level\n double precision floating point numbers. The same caveats apply\n as for floating point numbers. The real and imaginary parts of a\n complex number ``z`` can be retrieved through the read-only\n attributes ``z.real`` and ``z.imag``.\n\nSequences\n These represent finite ordered sets indexed by non-negative\n numbers. The built-in function ``len()`` returns the number of\n items of a sequence. When the length of a sequence is *n*, the\n index set contains the numbers 0, 1, ..., *n*-1. Item *i* of\n sequence *a* is selected by ``a[i]``.\n\n Sequences also support slicing: ``a[i:j]`` selects all items with\n index *k* such that *i* ``<=`` *k* ``<`` *j*. When used as an\n expression, a slice is a sequence of the same type. This implies\n that the index set is renumbered so that it starts at 0.\n\n Some sequences also support "extended slicing" with a third "step"\n parameter: ``a[i:j:k]`` selects all items of *a* with index *x*\n where ``x = i + n*k``, *n* ``>=`` ``0`` and *i* ``<=`` *x* ``<``\n *j*.\n\n Sequences are distinguished according to their mutability:\n\n Immutable sequences\n An object of an immutable sequence type cannot change once it is\n created. (If the object contains references to other objects,\n these other objects may be mutable and may be changed; however,\n the collection of objects directly referenced by an immutable\n object cannot change.)\n\n The following types are immutable sequences:\n\n Strings\n A string is a sequence of values that represent Unicode\n codepoints. All the codepoints in range ``U+0000 - U+10FFFF``\n can be represented in a string. Python doesn\'t have a\n ``chr`` type, and every character in the string is\n represented as a string object with length ``1``. The built-\n in function ``ord()`` converts a character to its codepoint\n (as an integer); ``chr()`` converts an integer in range ``0 -\n 10FFFF`` to the corresponding character. ``str.encode()`` can\n be used to convert a ``str`` to ``bytes`` using the given\n encoding, and ``bytes.decode()`` can be used to achieve the\n opposite.\n\n Tuples\n The items of a tuple are arbitrary Python objects. Tuples of\n two or more items are formed by comma-separated lists of\n expressions. A tuple of one item (a \'singleton\') can be\n formed by affixing a comma to an expression (an expression by\n itself does not create a tuple, since parentheses must be\n usable for grouping of expressions). An empty tuple can be\n formed by an empty pair of parentheses.\n\n Bytes\n A bytes object is an immutable array. The items are 8-bit\n bytes, represented by integers in the range 0 <= x < 256.\n Bytes literals (like ``b\'abc\'`` and the built-in function\n ``bytes()`` can be used to construct bytes objects. Also,\n bytes objects can be decoded to strings via the ``decode()``\n method.\n\n Mutable sequences\n Mutable sequences can be changed after they are created. The\n subscription and slicing notations can be used as the target of\n assignment and ``del`` (delete) statements.\n\n There are currently two intrinsic mutable sequence types:\n\n Lists\n The items of a list are arbitrary Python objects. Lists are\n formed by placing a comma-separated list of expressions in\n square brackets. (Note that there are no special cases needed\n to form lists of length 0 or 1.)\n\n Byte Arrays\n A bytearray object is a mutable array. They are created by\n the built-in ``bytearray()`` constructor. Aside from being\n mutable (and hence unhashable), byte arrays otherwise provide\n the same interface and functionality as immutable bytes\n objects.\n\n The extension module ``array`` provides an additional example of\n a mutable sequence type, as does the ``collections`` module.\n\nSet types\n These represent unordered, finite sets of unique, immutable\n objects. As such, they cannot be indexed by any subscript. However,\n they can be iterated over, and the built-in function ``len()``\n returns the number of items in a set. Common uses for sets are fast\n membership testing, removing duplicates from a sequence, and\n computing mathematical operations such as intersection, union,\n difference, and symmetric difference.\n\n For set elements, the same immutability rules apply as for\n dictionary keys. Note that numeric types obey the normal rules for\n numeric comparison: if two numbers compare equal (e.g., ``1`` and\n ``1.0``), only one of them can be contained in a set.\n\n There are currently two intrinsic set types:\n\n Sets\n These represent a mutable set. They are created by the built-in\n ``set()`` constructor and can be modified afterwards by several\n methods, such as ``add()``.\n\n Frozen sets\n These represent an immutable set. They are created by the\n built-in ``frozenset()`` constructor. As a frozenset is\n immutable and *hashable*, it can be used again as an element of\n another set, or as a dictionary key.\n\nMappings\n These represent finite sets of objects indexed by arbitrary index\n sets. The subscript notation ``a[k]`` selects the item indexed by\n ``k`` from the mapping ``a``; this can be used in expressions and\n as the target of assignments or ``del`` statements. The built-in\n function ``len()`` returns the number of items in a mapping.\n\n There is currently a single intrinsic mapping type:\n\n Dictionaries\n These represent finite sets of objects indexed by nearly\n arbitrary values. The only types of values not acceptable as\n keys are values containing lists or dictionaries or other\n mutable types that are compared by value rather than by object\n identity, the reason being that the efficient implementation of\n dictionaries requires a key\'s hash value to remain constant.\n Numeric types used for keys obey the normal rules for numeric\n comparison: if two numbers compare equal (e.g., ``1`` and\n ``1.0``) then they can be used interchangeably to index the same\n dictionary entry.\n\n Dictionaries are mutable; they can be created by the ``{...}``\n notation (see section *Dictionary displays*).\n\n The extension modules ``dbm.ndbm`` and ``dbm.gnu`` provide\n additional examples of mapping types, as does the\n ``collections`` module.\n\nCallable types\n These are the types to which the function call operation (see\n section *Calls*) can be applied:\n\n User-defined functions\n A user-defined function object is created by a function\n definition (see section *Function definitions*). It should be\n called with an argument list containing the same number of items\n as the function\'s formal parameter list.\n\n Special attributes:\n\n +---------------------------+---------------------------------+-------------+\n | Attribute | Meaning | |\n +===========================+=================================+=============+\n | ``__doc__`` | The function\'s documentation | Writable |\n | | string, or ``None`` if | |\n | | unavailable | |\n +---------------------------+---------------------------------+-------------+\n | ``__name__`` | The function\'s name | Writable |\n +---------------------------+---------------------------------+-------------+\n | ``__qualname__`` | The function\'s *qualified name* | Writable |\n | | New in version 3.3. | |\n +---------------------------+---------------------------------+-------------+\n | ``__module__`` | The name of the module the | Writable |\n | | function was defined in, or | |\n | | ``None`` if unavailable. | |\n +---------------------------+---------------------------------+-------------+\n | ``__defaults__`` | A tuple containing default | Writable |\n | | argument values for those | |\n | | arguments that have defaults, | |\n | | or ``None`` if no arguments | |\n | | have a default value | |\n +---------------------------+---------------------------------+-------------+\n | ``__code__`` | The code object representing | Writable |\n | | the compiled function body. | |\n +---------------------------+---------------------------------+-------------+\n | ``__globals__`` | A reference to the dictionary | Read-only |\n | | that holds the function\'s | |\n | | global variables --- the global | |\n | | namespace of the module in | |\n | | which the function was defined. | |\n +---------------------------+---------------------------------+-------------+\n | ``__dict__`` | The namespace supporting | Writable |\n | | arbitrary function attributes. | |\n +---------------------------+---------------------------------+-------------+\n | ``__closure__`` | ``None`` or a tuple of cells | Read-only |\n | | that contain bindings for the | |\n | | function\'s free variables. | |\n +---------------------------+---------------------------------+-------------+\n | ``__annotations__`` | A dict containing annotations | Writable |\n | | of parameters. The keys of the | |\n | | dict are the parameter names, | |\n | | or ``\'return\'`` for the return | |\n | | annotation, if provided. | |\n +---------------------------+---------------------------------+-------------+\n | ``__kwdefaults__`` | A dict containing defaults for | Writable |\n | | keyword-only parameters. | |\n +---------------------------+---------------------------------+-------------+\n\n Most of the attributes labelled "Writable" check the type of the\n assigned value.\n\n Function objects also support getting and setting arbitrary\n attributes, which can be used, for example, to attach metadata\n to functions. Regular attribute dot-notation is used to get and\n set such attributes. *Note that the current implementation only\n supports function attributes on user-defined functions. Function\n attributes on built-in functions may be supported in the\n future.*\n\n Additional information about a function\'s definition can be\n retrieved from its code object; see the description of internal\n types below.\n\n Instance methods\n An instance method object combines a class, a class instance and\n any callable object (normally a user-defined function).\n\n Special read-only attributes: ``__self__`` is the class instance\n object, ``__func__`` is the function object; ``__doc__`` is the\n method\'s documentation (same as ``__func__.__doc__``);\n ``__name__`` is the method name (same as ``__func__.__name__``);\n ``__module__`` is the name of the module the method was defined\n in, or ``None`` if unavailable.\n\n Methods also support accessing (but not setting) the arbitrary\n function attributes on the underlying function object.\n\n User-defined method objects may be created when getting an\n attribute of a class (perhaps via an instance of that class), if\n that attribute is a user-defined function object or a class\n method object.\n\n When an instance method object is created by retrieving a user-\n defined function object from a class via one of its instances,\n its ``__self__`` attribute is the instance, and the method\n object is said to be bound. The new method\'s ``__func__``\n attribute is the original function object.\n\n When a user-defined method object is created by retrieving\n another method object from a class or instance, the behaviour is\n the same as for a function object, except that the ``__func__``\n attribute of the new instance is not the original method object\n but its ``__func__`` attribute.\n\n When an instance method object is created by retrieving a class\n method object from a class or instance, its ``__self__``\n attribute is the class itself, and its ``__func__`` attribute is\n the function object underlying the class method.\n\n When an instance method object is called, the underlying\n function (``__func__``) is called, inserting the class instance\n (``__self__``) in front of the argument list. For instance,\n when ``C`` is a class which contains a definition for a function\n ``f()``, and ``x`` is an instance of ``C``, calling ``x.f(1)``\n is equivalent to calling ``C.f(x, 1)``.\n\n When an instance method object is derived from a class method\n object, the "class instance" stored in ``__self__`` will\n actually be the class itself, so that calling either ``x.f(1)``\n or ``C.f(1)`` is equivalent to calling ``f(C,1)`` where ``f`` is\n the underlying function.\n\n Note that the transformation from function object to instance\n method object happens each time the attribute is retrieved from\n the instance. In some cases, a fruitful optimization is to\n assign the attribute to a local variable and call that local\n variable. Also notice that this transformation only happens for\n user-defined functions; other callable objects (and all non-\n callable objects) are retrieved without transformation. It is\n also important to note that user-defined functions which are\n attributes of a class instance are not converted to bound\n methods; this *only* happens when the function is an attribute\n of the class.\n\n Generator functions\n A function or method which uses the ``yield`` statement (see\n section *The yield statement*) is called a *generator function*.\n Such a function, when called, always returns an iterator object\n which can be used to execute the body of the function: calling\n the iterator\'s ``__next__()`` method will cause the function to\n execute until it provides a value using the ``yield`` statement.\n When the function executes a ``return`` statement or falls off\n the end, a ``StopIteration`` exception is raised and the\n iterator will have reached the end of the set of values to be\n returned.\n\n Built-in functions\n A built-in function object is a wrapper around a C function.\n Examples of built-in functions are ``len()`` and ``math.sin()``\n (``math`` is a standard built-in module). The number and type of\n the arguments are determined by the C function. Special read-\n only attributes: ``__doc__`` is the function\'s documentation\n string, or ``None`` if unavailable; ``__name__`` is the\n function\'s name; ``__self__`` is set to ``None`` (but see the\n next item); ``__module__`` is the name of the module the\n function was defined in or ``None`` if unavailable.\n\n Built-in methods\n This is really a different disguise of a built-in function, this\n time containing an object passed to the C function as an\n implicit extra argument. An example of a built-in method is\n ``alist.append()``, assuming *alist* is a list object. In this\n case, the special read-only attribute ``__self__`` is set to the\n object denoted by *alist*.\n\n Classes\n Classes are callable. These objects normally act as factories\n for new instances of themselves, but variations are possible for\n class types that override ``__new__()``. The arguments of the\n call are passed to ``__new__()`` and, in the typical case, to\n ``__init__()`` to initialize the new instance.\n\n Class Instances\n Instances of arbitrary classes can be made callable by defining\n a ``__call__()`` method in their class.\n\nModules\n Modules are imported by the ``import`` statement (see section *The\n import statement*). A module object has a namespace implemented by\n a dictionary object (this is the dictionary referenced by the\n __globals__ attribute of functions defined in the module).\n Attribute references are translated to lookups in this dictionary,\n e.g., ``m.x`` is equivalent to ``m.__dict__["x"]``. A module object\n does not contain the code object used to initialize the module\n (since it isn\'t needed once the initialization is done).\n\n Attribute assignment updates the module\'s namespace dictionary,\n e.g., ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``.\n\n Special read-only attribute: ``__dict__`` is the module\'s namespace\n as a dictionary object.\n\n **CPython implementation detail:** Because of the way CPython\n clears module dictionaries, the module dictionary will be cleared\n when the module falls out of scope even if the dictionary still has\n live references. To avoid this, copy the dictionary or keep the\n module around while using its dictionary directly.\n\n Predefined (writable) attributes: ``__name__`` is the module\'s\n name; ``__doc__`` is the module\'s documentation string, or ``None``\n if unavailable; ``__file__`` is the pathname of the file from which\n the module was loaded, if it was loaded from a file. The\n ``__file__`` attribute is not present for C modules that are\n statically linked into the interpreter; for extension modules\n loaded dynamically from a shared library, it is the pathname of the\n shared library file.\n\nCustom classes\n Custom class types are typically created by class definitions (see\n section *Class definitions*). A class has a namespace implemented\n by a dictionary object. Class attribute references are translated\n to lookups in this dictionary, e.g., ``C.x`` is translated to\n ``C.__dict__["x"]`` (although there are a number of hooks which\n allow for other means of locating attributes). When the attribute\n name is not found there, the attribute search continues in the base\n classes. This search of the base classes uses the C3 method\n resolution order which behaves correctly even in the presence of\n \'diamond\' inheritance structures where there are multiple\n inheritance paths leading back to a common ancestor. Additional\n details on the C3 MRO used by Python can be found in the\n documentation accompanying the 2.3 release at\n http://www.python.org/download/releases/2.3/mro/.\n\n When a class attribute reference (for class ``C``, say) would yield\n a class method object, it is transformed into an instance method\n object whose ``__self__`` attributes is ``C``. When it would yield\n a static method object, it is transformed into the object wrapped\n by the static method object. See section *Implementing Descriptors*\n for another way in which attributes retrieved from a class may\n differ from those actually contained in its ``__dict__``.\n\n Class attribute assignments update the class\'s dictionary, never\n the dictionary of a base class.\n\n A class object can be called (see above) to yield a class instance\n (see below).\n\n Special attributes: ``__name__`` is the class name; ``__module__``\n is the module name in which the class was defined; ``__dict__`` is\n the dictionary containing the class\'s namespace; ``__bases__`` is a\n tuple (possibly empty or a singleton) containing the base classes,\n in the order of their occurrence in the base class list;\n ``__doc__`` is the class\'s documentation string, or None if\n undefined.\n\nClass instances\n A class instance is created by calling a class object (see above).\n A class instance has a namespace implemented as a dictionary which\n is the first place in which attribute references are searched.\n When an attribute is not found there, and the instance\'s class has\n an attribute by that name, the search continues with the class\n attributes. If a class attribute is found that is a user-defined\n function object, it is transformed into an instance method object\n whose ``__self__`` attribute is the instance. Static method and\n class method objects are also transformed; see above under\n "Classes". See section *Implementing Descriptors* for another way\n in which attributes of a class retrieved via its instances may\n differ from the objects actually stored in the class\'s\n ``__dict__``. If no class attribute is found, and the object\'s\n class has a ``__getattr__()`` method, that is called to satisfy the\n lookup.\n\n Attribute assignments and deletions update the instance\'s\n dictionary, never a class\'s dictionary. If the class has a\n ``__setattr__()`` or ``__delattr__()`` method, this is called\n instead of updating the instance dictionary directly.\n\n Class instances can pretend to be numbers, sequences, or mappings\n if they have methods with certain special names. See section\n *Special method names*.\n\n Special attributes: ``__dict__`` is the attribute dictionary;\n ``__class__`` is the instance\'s class.\n\nI/O objects (also known as file objects)\n A *file object* represents an open file. Various shortcuts are\n available to create file objects: the ``open()`` built-in function,\n and also ``os.popen()``, ``os.fdopen()``, and the ``makefile()``\n method of socket objects (and perhaps by other functions or methods\n provided by extension modules).\n\n The objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are\n initialized to file objects corresponding to the interpreter\'s\n standard input, output and error streams; they are all open in text\n mode and therefore follow the interface defined by the\n ``io.TextIOBase`` abstract class.\n\nInternal types\n A few types used internally by the interpreter are exposed to the\n user. Their definitions may change with future versions of the\n interpreter, but they are mentioned here for completeness.\n\n Code objects\n Code objects represent *byte-compiled* executable Python code,\n or *bytecode*. The difference between a code object and a\n function object is that the function object contains an explicit\n reference to the function\'s globals (the module in which it was\n defined), while a code object contains no context; also the\n default argument values are stored in the function object, not\n in the code object (because they represent values calculated at\n run-time). Unlike function objects, code objects are immutable\n and contain no references (directly or indirectly) to mutable\n objects.\n\n Special read-only attributes: ``co_name`` gives the function\n name; ``co_argcount`` is the number of positional arguments\n (including arguments with default values); ``co_nlocals`` is the\n number of local variables used by the function (including\n arguments); ``co_varnames`` is a tuple containing the names of\n the local variables (starting with the argument names);\n ``co_cellvars`` is a tuple containing the names of local\n variables that are referenced by nested functions;\n ``co_freevars`` is a tuple containing the names of free\n variables; ``co_code`` is a string representing the sequence of\n bytecode instructions; ``co_consts`` is a tuple containing the\n literals used by the bytecode; ``co_names`` is a tuple\n containing the names used by the bytecode; ``co_filename`` is\n the filename from which the code was compiled;\n ``co_firstlineno`` is the first line number of the function;\n ``co_lnotab`` is a string encoding the mapping from bytecode\n offsets to line numbers (for details see the source code of the\n interpreter); ``co_stacksize`` is the required stack size\n (including local variables); ``co_flags`` is an integer encoding\n a number of flags for the interpreter.\n\n The following flag bits are defined for ``co_flags``: bit\n ``0x04`` is set if the function uses the ``*arguments`` syntax\n to accept an arbitrary number of positional arguments; bit\n ``0x08`` is set if the function uses the ``**keywords`` syntax\n to accept arbitrary keyword arguments; bit ``0x20`` is set if\n the function is a generator.\n\n Future feature declarations (``from __future__ import\n division``) also use bits in ``co_flags`` to indicate whether a\n code object was compiled with a particular feature enabled: bit\n ``0x2000`` is set if the function was compiled with future\n division enabled; bits ``0x10`` and ``0x1000`` were used in\n earlier versions of Python.\n\n Other bits in ``co_flags`` are reserved for internal use.\n\n If a code object represents a function, the first item in\n ``co_consts`` is the documentation string of the function, or\n ``None`` if undefined.\n\n Frame objects\n Frame objects represent execution frames. They may occur in\n traceback objects (see below).\n\n Special read-only attributes: ``f_back`` is to the previous\n stack frame (towards the caller), or ``None`` if this is the\n bottom stack frame; ``f_code`` is the code object being executed\n in this frame; ``f_locals`` is the dictionary used to look up\n local variables; ``f_globals`` is used for global variables;\n ``f_builtins`` is used for built-in (intrinsic) names;\n ``f_lasti`` gives the precise instruction (this is an index into\n the bytecode string of the code object).\n\n Special writable attributes: ``f_trace``, if not ``None``, is a\n function called at the start of each source code line (this is\n used by the debugger); ``f_lineno`` is the current line number\n of the frame --- writing to this from within a trace function\n jumps to the given line (only for the bottom-most frame). A\n debugger can implement a Jump command (aka Set Next Statement)\n by writing to f_lineno.\n\n Traceback objects\n Traceback objects represent a stack trace of an exception. A\n traceback object is created when an exception occurs. When the\n search for an exception handler unwinds the execution stack, at\n each unwound level a traceback object is inserted in front of\n the current traceback. When an exception handler is entered,\n the stack trace is made available to the program. (See section\n *The try statement*.) It is accessible as the third item of the\n tuple returned by ``sys.exc_info()``. When the program contains\n no suitable handler, the stack trace is written (nicely\n formatted) to the standard error stream; if the interpreter is\n interactive, it is also made available to the user as\n ``sys.last_traceback``.\n\n Special read-only attributes: ``tb_next`` is the next level in\n the stack trace (towards the frame where the exception\n occurred), or ``None`` if there is no next level; ``tb_frame``\n points to the execution frame of the current level;\n ``tb_lineno`` gives the line number where the exception\n occurred; ``tb_lasti`` indicates the precise instruction. The\n line number and last instruction in the traceback may differ\n from the line number of its frame object if the exception\n occurred in a ``try`` statement with no matching except clause\n or with a finally clause.\n\n Slice objects\n Slice objects are used to represent slices for ``__getitem__()``\n methods. They are also created by the built-in ``slice()``\n function.\n\n Special read-only attributes: ``start`` is the lower bound;\n ``stop`` is the upper bound; ``step`` is the step value; each is\n ``None`` if omitted. These attributes can have any type.\n\n Slice objects support one method:\n\n slice.indices(self, length)\n\n This method takes a single integer argument *length* and\n computes information about the slice that the slice object\n would describe if applied to a sequence of *length* items.\n It returns a tuple of three integers; respectively these are\n the *start* and *stop* indices and the *step* or stride\n length of the slice. Missing or out-of-bounds indices are\n handled in a manner consistent with regular slices.\n\n Static method objects\n Static method objects provide a way of defeating the\n transformation of function objects to method objects described\n above. A static method object is a wrapper around any other\n object, usually a user-defined method object. When a static\n method object is retrieved from a class or a class instance, the\n object actually returned is the wrapped object, which is not\n subject to any further transformation. Static method objects are\n not themselves callable, although the objects they wrap usually\n are. Static method objects are created by the built-in\n ``staticmethod()`` constructor.\n\n Class method objects\n A class method object, like a static method object, is a wrapper\n around another object that alters the way in which that object\n is retrieved from classes and class instances. The behaviour of\n class method objects upon such retrieval is described above,\n under "User-defined methods". Class method objects are created\n by the built-in ``classmethod()`` constructor.\n', 'typesfunctions': '\nFunctions\n*********\n\nFunction objects are created by function definitions. The only\noperation on a function object is to call it: ``func(argument-list)``.\n\nThere are really two flavors of function objects: built-in functions\nand user-defined functions. Both support the same operation (to call\nthe function), but the implementation is different, hence the\ndifferent object types.\n\nSee *Function definitions* for more information.\n', - 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects. There is currently only one standard\nmapping type, the *dictionary*. (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values. Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys. Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry. (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n Return a new dictionary initialized from an optional positional\n argument or from a set of keyword arguments. If no arguments are\n given, return a new empty dictionary. If the positional argument\n *arg* is a mapping object, return a dictionary mapping the same\n keys to the same values as does the mapping object. Otherwise the\n positional argument must be a sequence, a container that supports\n iteration, or an iterator object. The elements of the argument\n must each also be of one of those kinds, and each must in turn\n contain exactly two objects. The first is used as a key in the new\n dictionary, and the second as the key\'s value. If a given key is\n seen more than once, the last value associated with it is retained\n in the new dictionary.\n\n If keyword arguments are given, the keywords themselves with their\n associated values are added as items to the dictionary. If a key\n is specified both in the positional argument and as a keyword\n argument, the value associated with the keyword is retained in the\n dictionary. For example, these all return a dictionary equal to\n ``{"one": 1, "two": 2}``:\n\n * ``dict(one=1, two=2)``\n\n * ``dict({\'one\': 1, \'two\': 2})``\n\n * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n The first example only works for keys that are valid Python\n identifiers; the others work with any valid keys.\n\n These are the operations that dictionaries support (and therefore,\n custom mapping types should support too):\n\n len(d)\n\n Return the number of items in the dictionary *d*.\n\n d[key]\n\n Return the item of *d* with key *key*. Raises a ``KeyError`` if\n *key* is not in the map.\n\n If a subclass of dict defines a method ``__missing__()``, if the\n key *key* is not present, the ``d[key]`` operation calls that\n method with the key *key* as argument. The ``d[key]`` operation\n then returns or raises whatever is returned or raised by the\n ``__missing__(key)`` call if the key is not present. No other\n operations or methods invoke ``__missing__()``. If\n ``__missing__()`` is not defined, ``KeyError`` is raised.\n ``__missing__()`` must be a method; it cannot be an instance\n variable:\n\n >>> class Counter(dict):\n ... def __missing__(self, key):\n ... return 0\n >>> c = Counter()\n >>> c[\'red\']\n 0\n >>> c[\'red\'] += 1\n >>> c[\'red\']\n 1\n\n See ``collections.Counter`` for a complete implementation\n including other methods helpful for accumulating and managing\n tallies.\n\n Changed in version 3.3: If the dict is modified during the\n lookup, a ``RuntimeError`` exception is now raised.\n\n d[key] = value\n\n Set ``d[key]`` to *value*.\n\n del d[key]\n\n Remove ``d[key]`` from *d*. Raises a ``KeyError`` if *key* is\n not in the map.\n\n key in d\n\n Return ``True`` if *d* has a key *key*, else ``False``.\n\n key not in d\n\n Equivalent to ``not key in d``.\n\n iter(d)\n\n Return an iterator over the keys of the dictionary. This is a\n shortcut for ``iter(d.keys())``.\n\n clear()\n\n Remove all items from the dictionary.\n\n copy()\n\n Return a shallow copy of the dictionary.\n\n classmethod fromkeys(seq[, value])\n\n Create a new dictionary with keys from *seq* and values set to\n *value*.\n\n ``fromkeys()`` is a class method that returns a new dictionary.\n *value* defaults to ``None``.\n\n get(key[, default])\n\n Return the value for *key* if *key* is in the dictionary, else\n *default*. If *default* is not given, it defaults to ``None``,\n so that this method never raises a ``KeyError``.\n\n items()\n\n Return a new view of the dictionary\'s items (``(key, value)``\n pairs). See below for documentation of view objects.\n\n keys()\n\n Return a new view of the dictionary\'s keys. See below for\n documentation of view objects.\n\n pop(key[, default])\n\n If *key* is in the dictionary, remove it and return its value,\n else return *default*. If *default* is not given and *key* is\n not in the dictionary, a ``KeyError`` is raised.\n\n popitem()\n\n Remove and return an arbitrary ``(key, value)`` pair from the\n dictionary.\n\n ``popitem()`` is useful to destructively iterate over a\n dictionary, as often used in set algorithms. If the dictionary\n is empty, calling ``popitem()`` raises a ``KeyError``.\n\n setdefault(key[, default])\n\n If *key* is in the dictionary, return its value. If not, insert\n *key* with a value of *default* and return *default*. *default*\n defaults to ``None``.\n\n update([other])\n\n Update the dictionary with the key/value pairs from *other*,\n overwriting existing keys. Return ``None``.\n\n ``update()`` accepts either another dictionary object or an\n iterable of key/value pairs (as tuples or other iterables of\n length two). If keyword arguments are specified, the dictionary\n is then updated with those key/value pairs: ``d.update(red=1,\n blue=2)``.\n\n values()\n\n Return a new view of the dictionary\'s values. See below for\n documentation of view objects.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*. They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n Return the number of entries in the dictionary.\n\niter(dictview)\n\n Return an iterator over the keys, values or items (represented as\n tuples of ``(key, value)``) in the dictionary.\n\n Keys and values are iterated over in an arbitrary order which is\n non-random, varies across Python implementations, and depends on\n the dictionary\'s history of insertions and deletions. If keys,\n values and items views are iterated over with no intervening\n modifications to the dictionary, the order of items will directly\n correspond. This allows the creation of ``(value, key)`` pairs\n using ``zip()``: ``pairs = zip(d.values(), d.keys())``. Another\n way to create the same list is ``pairs = [(v, k) for (k, v) in\n d.items()]``.\n\n Iterating views while adding or deleting entries in the dictionary\n may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n Return ``True`` if *x* is in the underlying dictionary\'s keys,\n values or items (in the latter case, *x* should be a ``(key,\n value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like. (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n >>> keys = dishes.keys()\n >>> values = dishes.values()\n\n >>> # iteration\n >>> n = 0\n >>> for val in values:\n ... n += val\n >>> print(n)\n 504\n\n >>> # keys and values are iterated over in the same order\n >>> list(keys)\n [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n >>> list(values)\n [2, 1, 1, 500]\n\n >>> # view objects are dynamic and reflect dict changes\n >>> del dishes[\'eggs\']\n >>> del dishes[\'sausage\']\n >>> list(keys)\n [\'spam\', \'bacon\']\n\n >>> # set operations\n >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n {\'bacon\'}\n >>> keys ^ {\'sausage\', \'juice\'}\n {\'juice\', \'sausage\', \'bacon\', \'spam\'}\n', + 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects. There is currently only one standard\nmapping type, the *dictionary*. (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values. Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys. Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry. (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n Return a new dictionary initialized from an optional positional\n argument or from a set of keyword arguments. If no arguments are\n given, return a new empty dictionary. If the positional argument\n *arg* is a mapping object, return a dictionary mapping the same\n keys to the same values as does the mapping object. Otherwise the\n positional argument must be a sequence, a container that supports\n iteration, or an iterator object. The elements of the argument\n must each also be of one of those kinds, and each must in turn\n contain exactly two objects. The first is used as a key in the new\n dictionary, and the second as the key\'s value. If a given key is\n seen more than once, the last value associated with it is retained\n in the new dictionary.\n\n If keyword arguments are given, the keywords themselves with their\n associated values are added as items to the dictionary. If a key\n is specified both in the positional argument and as a keyword\n argument, the value associated with the keyword is retained in the\n dictionary. For example, these all return a dictionary equal to\n ``{"one": 1, "two": 2}``:\n\n * ``dict(one=1, two=2)``\n\n * ``dict({\'one\': 1, \'two\': 2})``\n\n * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n The first example only works for keys that are valid Python\n identifiers; the others work with any valid keys.\n\n These are the operations that dictionaries support (and therefore,\n custom mapping types should support too):\n\n len(d)\n\n Return the number of items in the dictionary *d*.\n\n d[key]\n\n Return the item of *d* with key *key*. Raises a ``KeyError`` if\n *key* is not in the map.\n\n If a subclass of dict defines a method ``__missing__()``, if the\n key *key* is not present, the ``d[key]`` operation calls that\n method with the key *key* as argument. The ``d[key]`` operation\n then returns or raises whatever is returned or raised by the\n ``__missing__(key)`` call if the key is not present. No other\n operations or methods invoke ``__missing__()``. If\n ``__missing__()`` is not defined, ``KeyError`` is raised.\n ``__missing__()`` must be a method; it cannot be an instance\n variable:\n\n >>> class Counter(dict):\n ... def __missing__(self, key):\n ... return 0\n >>> c = Counter()\n >>> c[\'red\']\n 0\n >>> c[\'red\'] += 1\n >>> c[\'red\']\n 1\n\n See ``collections.Counter`` for a complete implementation\n including other methods helpful for accumulating and managing\n tallies.\n\n Changed in version 3.3: If the dict is modified during the\n lookup, a ``RuntimeError`` exception is now raised.\n\n d[key] = value\n\n Set ``d[key]`` to *value*.\n\n del d[key]\n\n Remove ``d[key]`` from *d*. Raises a ``KeyError`` if *key* is\n not in the map.\n\n key in d\n\n Return ``True`` if *d* has a key *key*, else ``False``.\n\n key not in d\n\n Equivalent to ``not key in d``.\n\n iter(d)\n\n Return an iterator over the keys of the dictionary. This is a\n shortcut for ``iter(d.keys())``.\n\n clear()\n\n Remove all items from the dictionary.\n\n copy()\n\n Return a shallow copy of the dictionary.\n\n classmethod fromkeys(seq[, value])\n\n Create a new dictionary with keys from *seq* and values set to\n *value*.\n\n ``fromkeys()`` is a class method that returns a new dictionary.\n *value* defaults to ``None``.\n\n get(key[, default])\n\n Return the value for *key* if *key* is in the dictionary, else\n *default*. If *default* is not given, it defaults to ``None``,\n so that this method never raises a ``KeyError``.\n\n items()\n\n Return a new view of the dictionary\'s items (``(key, value)``\n pairs). See the *documentation of view objects*.\n\n keys()\n\n Return a new view of the dictionary\'s keys. See the\n *documentation of view objects*.\n\n pop(key[, default])\n\n If *key* is in the dictionary, remove it and return its value,\n else return *default*. If *default* is not given and *key* is\n not in the dictionary, a ``KeyError`` is raised.\n\n popitem()\n\n Remove and return an arbitrary ``(key, value)`` pair from the\n dictionary.\n\n ``popitem()`` is useful to destructively iterate over a\n dictionary, as often used in set algorithms. If the dictionary\n is empty, calling ``popitem()`` raises a ``KeyError``.\n\n setdefault(key[, default])\n\n If *key* is in the dictionary, return its value. If not, insert\n *key* with a value of *default* and return *default*. *default*\n defaults to ``None``.\n\n update([other])\n\n Update the dictionary with the key/value pairs from *other*,\n overwriting existing keys. Return ``None``.\n\n ``update()`` accepts either another dictionary object or an\n iterable of key/value pairs (as tuples or other iterables of\n length two). If keyword arguments are specified, the dictionary\n is then updated with those key/value pairs: ``d.update(red=1,\n blue=2)``.\n\n values()\n\n Return a new view of the dictionary\'s values. See the\n *documentation of view objects*.\n\nSee also:\n\n ``types.MappingProxyType`` can be used to create a read-only view\n of a ``dict``.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*. They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n Return the number of entries in the dictionary.\n\niter(dictview)\n\n Return an iterator over the keys, values or items (represented as\n tuples of ``(key, value)``) in the dictionary.\n\n Keys and values are iterated over in an arbitrary order which is\n non-random, varies across Python implementations, and depends on\n the dictionary\'s history of insertions and deletions. If keys,\n values and items views are iterated over with no intervening\n modifications to the dictionary, the order of items will directly\n correspond. This allows the creation of ``(value, key)`` pairs\n using ``zip()``: ``pairs = zip(d.values(), d.keys())``. Another\n way to create the same list is ``pairs = [(v, k) for (k, v) in\n d.items()]``.\n\n Iterating views while adding or deleting entries in the dictionary\n may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n Return ``True`` if *x* is in the underlying dictionary\'s keys,\n values or items (in the latter case, *x* should be a ``(key,\n value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like. (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n >>> keys = dishes.keys()\n >>> values = dishes.values()\n\n >>> # iteration\n >>> n = 0\n >>> for val in values:\n ... n += val\n >>> print(n)\n 504\n\n >>> # keys and values are iterated over in the same order\n >>> list(keys)\n [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n >>> list(values)\n [2, 1, 1, 500]\n\n >>> # view objects are dynamic and reflect dict changes\n >>> del dishes[\'eggs\']\n >>> del dishes[\'sausage\']\n >>> list(keys)\n [\'spam\', \'bacon\']\n\n >>> # set operations\n >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n {\'bacon\'}\n >>> keys ^ {\'sausage\', \'juice\'}\n {\'juice\', \'sausage\', \'bacon\', \'spam\'}\n', 'typesmethods': "\nMethods\n*******\n\nMethods are functions that are called using the attribute notation.\nThere are two flavors: built-in methods (such as ``append()`` on\nlists) and class instance methods. Built-in methods are described\nwith the types that support them.\n\nIf you access a method (a function defined in a class namespace)\nthrough an instance, you get a special object: a *bound method* (also\ncalled *instance method*) object. When called, it will add the\n``self`` argument to the argument list. Bound methods have two\nspecial read-only attributes: ``m.__self__`` is the object on which\nthe method operates, and ``m.__func__`` is the function implementing\nthe method. Calling ``m(arg-1, arg-2, ..., arg-n)`` is completely\nequivalent to calling ``m.__func__(m.__self__, arg-1, arg-2, ...,\narg-n)``.\n\nLike function objects, bound method objects support getting arbitrary\nattributes. However, since method attributes are actually stored on\nthe underlying function object (``meth.__func__``), setting method\nattributes on bound methods is disallowed. Attempting to set a method\nattribute results in a ``TypeError`` being raised. In order to set a\nmethod attribute, you need to explicitly set it on the underlying\nfunction object:\n\n class C:\n def method(self):\n pass\n\n c = C()\n c.method.__func__.whoami = 'my name is c'\n\nSee *The standard type hierarchy* for more information.\n", 'typesmodules': "\nModules\n*******\n\nThe only special operation on a module is attribute access:\n``m.name``, where *m* is a module and *name* accesses a name defined\nin *m*'s symbol table. Module attributes can be assigned to. (Note\nthat the ``import`` statement is not, strictly speaking, an operation\non a module object; ``import foo`` does not require a module object\nnamed *foo* to exist, rather it requires an (external) *definition*\nfor a module named *foo* somewhere.)\n\nA special attribute of every module is ``__dict__``. This is the\ndictionary containing the module's symbol table. Modifying this\ndictionary will actually change the module's symbol table, but direct\nassignment to the ``__dict__`` attribute is not possible (you can\nwrite ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but\nyou can't write ``m.__dict__ = {}``). Modifying ``__dict__`` directly\nis not recommended.\n\nModules built into the interpreter are written like this: ````. If loaded from a file, they are written as\n````.\n", 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects. For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters. Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``. See *String and\nBytes literals* for more about string literals. In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence. Bytes objects can\nbe constructed the constructor, ``bytes()``, and from literals; use a\n``b`` prefix with normal string syntax: ``b\'xyzzy\'``. To construct\nbyte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes. That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1. The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``. You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts. There\'s no implicit en-/decoding if you pass an object of\nthe wrong type. A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``. Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``. A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations. The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations. The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority). In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation | Result | Notes |\n+====================+==================================+============+\n| ``x in s`` | ``True`` if an item of *s* is | (1) |\n| | equal to *x*, else ``False`` | |\n+--------------------+----------------------------------+------------+\n| ``x not in s`` | ``False`` if an item of *s* is | (1) |\n| | equal to *x*, else ``True`` | |\n+--------------------+----------------------------------+------------+\n| ``s + t`` | the concatenation of *s* and *t* | (6) |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s`` | *n* shallow copies of *s* | (2) |\n| | concatenated | |\n+--------------------+----------------------------------+------------+\n| ``s[i]`` | *i*th item of *s*, origin 0 | (3) |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]`` | slice of *s* from *i* to *j* | (3)(4) |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]`` | slice of *s* from *i* to *j* | (3)(5) |\n| | with step *k* | |\n+--------------------+----------------------------------+------------+\n| ``len(s)`` | length of *s* | |\n+--------------------+----------------------------------+------------+\n| ``min(s)`` | smallest item of *s* | |\n+--------------------+----------------------------------+------------+\n| ``max(s)`` | largest item of *s* | |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)`` | index of the first occurence of | |\n| | *i* in *s* | |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)`` | total number of occurences of | |\n| | *i* in *s* | |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons. In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements. This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length. (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n empty sequence of the same type as *s*). Note also that the copies\n are shallow; nested structures are not copied. This often haunts\n new Python programmers; consider:\n\n >>> lists = [[]] * 3\n >>> lists\n [[], [], []]\n >>> lists[0].append(3)\n >>> lists\n [[3], [3], [3]]\n\n What has happened is that ``[[]]`` is a one-element list containing\n an empty list, so all three elements of ``[[]] * 3`` are (pointers\n to) this single empty list. Modifying any of the elements of\n ``lists`` modifies this single list. You can create a list of\n different lists this way:\n\n >>> lists = [[] for i in range(3)]\n >>> lists[0].append(3)\n >>> lists[1].append(5)\n >>> lists[2].append(7)\n >>> lists\n [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n string: ``len(s) + i`` or ``len(s) + j`` is substituted. But note\n that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n items with index *k* such that ``i <= k < j``. If *i* or *j* is\n greater than ``len(s)``, use ``len(s)``. If *i* is omitted or\n ``None``, use ``0``. If *j* is omitted or ``None``, use\n ``len(s)``. If *i* is greater than or equal to *j*, the slice is\n empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n sequence of items with index ``x = i + n*k`` such that ``0 <= n <\n (j-i)/k``. In other words, the indices are ``i``, ``i+k``,\n ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n never including *j*). If *i* or *j* is greater than ``len(s)``,\n use ``len(s)``. If *i* or *j* are omitted or ``None``, they become\n "end" values (which end depends on the sign of *k*). Note, *k*\n cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. Concatenating immutable strings always results in a new object.\n This means that building up a string by repeated concatenation will\n have a quadratic runtime cost in the total string length. To get a\n linear runtime cost, you must switch to one of the alternatives\n below:\n\n * if concatenating ``str`` objects, you can build a list and use\n ``str.join()`` at the end;\n\n * if concatenating ``bytes`` objects, you can similarly use\n ``bytes.join()``, or you can do in-place concatenation with a\n ``bytearray`` object. ``bytearray`` objects are mutable and have\n an efficient overallocation mechanism.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n Return a copy of the string with its first character capitalized\n and the rest lowercased.\n\nstr.casefold()\n\n Return a casefolded copy of the string. Casefolded strings may be\n used for caseless matching.\n\n Casefolding is similar to lowercasing but more aggressive because\n it is intended to remove all case distinctions in a string. For\n example, the German lowercase letter ``\'\xc3\x9f\'`` is equivalent to\n ``"ss"``. Since it is already lowercase, ``lower()`` would do\n nothing to ``\'\xc3\x9f\'``; ``casefold()`` converts it to ``"ss"``.\n\n The casefolding algorithm is described in section 3.13 of the\n Unicode Standard.\n\n New in version 3.3.\n\nstr.center(width[, fillchar])\n\n Return centered in a string of length *width*. Padding is done\n using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n Return the number of non-overlapping occurrences of substring *sub*\n in the range [*start*, *end*]. Optional arguments *start* and\n *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n Return an encoded version of the string as a bytes object. Default\n encoding is ``\'utf-8\'``. *errors* may be given to set a different\n error handling scheme. The default for *errors* is ``\'strict\'``,\n meaning that encoding errors raise a ``UnicodeError``. Other\n possible values are ``\'ignore\'``, ``\'replace\'``,\n ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n Return ``True`` if the string ends with the specified *suffix*,\n otherwise return ``False``. *suffix* can also be a tuple of\n suffixes to look for. With optional *start*, test beginning at\n that position. With optional *end*, stop comparing at that\n position.\n\nstr.expandtabs([tabsize])\n\n Return a copy of the string where all tab characters are replaced\n by zero or more spaces, depending on the current column and the\n given tab size. The column number is reset to zero after each\n newline occurring in the string. If *tabsize* is not given, a tab\n size of ``8`` characters is assumed. This doesn\'t understand other\n non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n Return the lowest index in the string where substring *sub* is\n found, such that *sub* is contained in the slice ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` if *sub* is not found.\n\n Note: The ``find()`` method should be used only if you need to know the\n position of *sub*. To check if *sub* is a substring or not, use\n the ``in`` operator:\n\n >>> \'Py\' in \'Python\'\n True\n\nstr.format(*args, **kwargs)\n\n Perform a string formatting operation. The string on which this\n method is called can contain literal text or replacement fields\n delimited by braces ``{}``. Each replacement field contains either\n the numeric index of a positional argument, or the name of a\n keyword argument. Returns a copy of the string where each\n replacement field is replaced with the string value of the\n corresponding argument.\n\n >>> "The sum of 1 + 2 is {0}".format(1+2)\n \'The sum of 1 + 2 is 3\'\n\n See *Format String Syntax* for a description of the various\n formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n Similar to ``str.format(**mapping)``, except that ``mapping`` is\n used directly and not copied to a ``dict`` . This is useful if for\n example ``mapping`` is a dict subclass:\n\n >>> class Default(dict):\n ... def __missing__(self, key):\n ... return key\n ...\n >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n \'Guido was born in country\'\n\n New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n Like ``find()``, but raise ``ValueError`` when the substring is not\n found.\n\nstr.isalnum()\n\n Return true if all characters in the string are alphanumeric and\n there is at least one character, false otherwise. A character\n ``c`` is alphanumeric if one of the following returns ``True``:\n ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n ``c.isnumeric()``.\n\nstr.isalpha()\n\n Return true if all characters in the string are alphabetic and\n there is at least one character, false otherwise. Alphabetic\n characters are those characters defined in the Unicode character\n database as "Letter", i.e., those with general category property\n being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is\n different from the "Alphabetic" property defined in the Unicode\n Standard.\n\nstr.isdecimal()\n\n Return true if all characters in the string are decimal characters\n and there is at least one character, false otherwise. Decimal\n characters are those from general category "Nd". This category\n includes digit characters, and all characters that can be used to\n form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.\n\nstr.isdigit()\n\n Return true if all characters in the string are digits and there is\n at least one character, false otherwise. Digits include decimal\n characters and digits that need special handling, such as the\n compatibility superscript digits. Formally, a digit is a character\n that has the property value Numeric_Type=Digit or\n Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n Return true if the string is a valid identifier according to the\n language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n Return true if all cased characters [4] in the string are lowercase\n and there is at least one cased character, false otherwise.\n\nstr.isnumeric()\n\n Return true if all characters in the string are numeric characters,\n and there is at least one character, false otherwise. Numeric\n characters include digit characters, and all characters that have\n the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n ONE FIFTH. Formally, numeric characters are those with the\n property value Numeric_Type=Digit, Numeric_Type=Decimal or\n Numeric_Type=Numeric.\n\nstr.isprintable()\n\n Return true if all characters in the string are printable or the\n string is empty, false otherwise. Nonprintable characters are\n those characters defined in the Unicode character database as\n "Other" or "Separator", excepting the ASCII space (0x20) which is\n considered printable. (Note that printable characters in this\n context are those which should not be escaped when ``repr()`` is\n invoked on a string. It has no bearing on the handling of strings\n written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n Return true if there are only whitespace characters in the string\n and there is at least one character, false otherwise. Whitespace\n characters are those characters defined in the Unicode character\n database as "Other" or "Separator" and those with bidirectional\n property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n Return true if the string is a titlecased string and there is at\n least one character, for example uppercase characters may only\n follow uncased characters and lowercase characters only cased ones.\n Return false otherwise.\n\nstr.isupper()\n\n Return true if all cased characters [4] in the string are uppercase\n and there is at least one cased character, false otherwise.\n\nstr.join(iterable)\n\n Return a string which is the concatenation of the strings in the\n *iterable* *iterable*. A ``TypeError`` will be raised if there are\n any non-string values in *iterable*, including ``bytes`` objects.\n The separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n Return the string left justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.lower()\n\n Return a copy of the string with all the cased characters [4]\n converted to lowercase.\n\n The lowercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.lstrip([chars])\n\n Return a copy of the string with leading characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a prefix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.lstrip()\n \'spacious \'\n >>> \'www.example.com\'.lstrip(\'cmowz.\')\n \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n This static method returns a translation table usable for\n ``str.translate()``.\n\n If there is only one argument, it must be a dictionary mapping\n Unicode ordinals (integers) or characters (strings of length 1) to\n Unicode ordinals, strings (of arbitrary lengths) or None.\n Character keys will then be converted to ordinals.\n\n If there are two arguments, they must be strings of equal length,\n and in the resulting dictionary, each character in x will be mapped\n to the character at the same position in y. If there is a third\n argument, it must be a string, whose characters will be mapped to\n None in the result.\n\nstr.partition(sep)\n\n Split the string at the first occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing the string itself, followed by\n two empty strings.\n\nstr.replace(old, new[, count])\n\n Return a copy of the string with all occurrences of substring *old*\n replaced by *new*. If the optional argument *count* is given, only\n the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n Return the highest index in the string where substring *sub* is\n found, such that *sub* is contained within ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n is not found.\n\nstr.rjust(width[, fillchar])\n\n Return the string right justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.rpartition(sep)\n\n Split the string at the last occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing two empty strings, followed by\n the string itself.\n\nstr.rsplit(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n are done, the *rightmost* ones. If *sep* is not specified or\n ``None``, any whitespace string is a separator. Except for\n splitting from the right, ``rsplit()`` behaves like ``split()``\n which is described in detail below.\n\nstr.rstrip([chars])\n\n Return a copy of the string with trailing characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a suffix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.rstrip()\n \' spacious\'\n >>> \'mississippi\'.rstrip(\'ipz\')\n \'mississ\'\n\nstr.split(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit*\n splits are done (thus, the list will have at most ``maxsplit+1``\n elements). If *maxsplit* is not specified, then there is no limit\n on the number of splits (all possible splits are made).\n\n If *sep* is given, consecutive delimiters are not grouped together\n and are deemed to delimit empty strings (for example,\n ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``). The *sep*\n argument may consist of multiple characters (for example,\n ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n an empty string with a specified separator returns ``[\'\']``.\n\n If *sep* is not specified or is ``None``, a different splitting\n algorithm is applied: runs of consecutive whitespace are regarded\n as a single separator, and the result will contain no empty strings\n at the start or end if the string has leading or trailing\n whitespace. Consequently, splitting an empty string or a string\n consisting of just whitespace with a ``None`` separator returns\n ``[]``.\n\n For example, ``\' 1 2 3 \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n and ``\' 1 2 3 \'.split(None, 1)`` returns ``[\'1\', \'2 3 \']``.\n\nstr.splitlines([keepends])\n\n Return a list of the lines in the string, breaking at line\n boundaries. Line breaks are not included in the resulting list\n unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n Return ``True`` if string starts with the *prefix*, otherwise\n return ``False``. *prefix* can also be a tuple of prefixes to look\n for. With optional *start*, test string beginning at that\n position. With optional *end*, stop comparing string at that\n position.\n\nstr.strip([chars])\n\n Return a copy of the string with the leading and trailing\n characters removed. The *chars* argument is a string specifying the\n set of characters to be removed. If omitted or ``None``, the\n *chars* argument defaults to removing whitespace. The *chars*\n argument is not a prefix or suffix; rather, all combinations of its\n values are stripped:\n\n >>> \' spacious \'.strip()\n \'spacious\'\n >>> \'www.example.com\'.strip(\'cmowz.\')\n \'example\'\n\nstr.swapcase()\n\n Return a copy of the string with uppercase characters converted to\n lowercase and vice versa. Note that it is not necessarily true that\n ``s.swapcase().swapcase() == s``.\n\nstr.title()\n\n Return a titlecased version of the string where words start with an\n uppercase character and the remaining characters are lowercase.\n\n The algorithm uses a simple language-independent definition of a\n word as groups of consecutive letters. The definition works in\n many contexts but it means that apostrophes in contractions and\n possessives form word boundaries, which may not be the desired\n result:\n\n >>> "they\'re bill\'s friends from the UK".title()\n "They\'Re Bill\'S Friends From The Uk"\n\n A workaround for apostrophes can be constructed using regular\n expressions:\n\n >>> import re\n >>> def titlecase(s):\n return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n lambda mo: mo.group(0)[0].upper() +\n mo.group(0)[1:].lower(),\n s)\n\n >>> titlecase("they\'re bill\'s friends.")\n "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n Return a copy of the *s* where all characters have been mapped\n through the *map* which must be a dictionary of Unicode ordinals\n (integers) to Unicode ordinals, strings or ``None``. Unmapped\n characters are left untouched. Characters mapped to ``None`` are\n deleted.\n\n You can use ``str.maketrans()`` to create a translation map from\n character-to-character mappings in different formats.\n\n Note: An even more flexible approach is to create a custom character\n mapping codec using the ``codecs`` module (see\n ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n Return a copy of the string with all the cased characters [4]\n converted to uppercase. Note that ``str.upper().isupper()`` might\n be ``False`` if ``s`` contains uncased characters or if the Unicode\n category of the resulting character(s) is not "Lu" (Letter,\n uppercase), but e.g. "Lt" (Letter, titlecase).\n\n The uppercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.zfill(width)\n\n Return the numeric string left filled with zeros in a string of\n length *width*. A sign prefix is handled correctly. The original\n string is returned if *width* is less than or equal to ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are modelled on C\'s\n printf() syntax. They only support formatting of certain builtin\n types. The use of a binary operator means that care may be needed\n in order to format tuples and dictionaries correctly. As the new\n *String Formatting* syntax is more flexible and handles tuples and\n dictionaries naturally, it is recommended for new code. However,\n there are no current plans to deprecate printf-style formatting.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [5] Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n conversion types.\n\n4. Minimum field width (optional). If specified as an ``\'*\'``\n (asterisk), the actual width is read from the next element of the\n tuple in *values*, and the object to convert comes after the\n minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n precision. If specified as ``\'*\'`` (an asterisk), the actual\n precision is read from the next element of the tuple in *values*,\n and the value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n... {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag | Meaning |\n+===========+=======================================================================+\n| ``\'#\'`` | The value conversion will use the "alternate form" (where defined |\n| | below). |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'`` | The conversion will be zero padded for numeric values. |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'`` | The converted value is left adjusted (overrides the ``\'0\'`` |\n| | conversion if both are given). |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'`` | (a space) A blank should be left before a positive number (or empty |\n| | string) produced by a signed conversion. |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'`` | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion |\n| | (overrides a "space" flag). |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion | Meaning | Notes |\n+==============+=======================================================+=========+\n| ``\'d\'`` | Signed integer decimal. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'`` | Signed integer decimal. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'`` | Signed octal value. | (1) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'`` | Obsolete type -- it is identical to ``\'d\'``. | (7) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'`` | Signed hexadecimal (lowercase). | (2) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'`` | Signed hexadecimal (uppercase). | (2) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'`` | Floating point exponential format (lowercase). | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'`` | Floating point exponential format (uppercase). | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'`` | Floating point decimal format. | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'`` | Floating point decimal format. | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'`` | Floating point format. Uses lowercase exponential | (4) |\n| | format if exponent is less than -4 or not less than | |\n| | precision, decimal format otherwise. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'`` | Floating point format. Uses uppercase exponential | (4) |\n| | format if exponent is less than -4 or not less than | |\n| | precision, decimal format otherwise. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'`` | Single character (accepts integer or single character | |\n| | string). | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'`` | String (converts any Python object using ``repr()``). | (5) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'`` | String (converts any Python object using ``str()``). | (5) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'a\'`` | String (converts any Python object using | (5) |\n| | ``ascii()``). | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'`` | No argument is converted, results in a ``\'%\'`` | |\n| | character in the result. | |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n between left-hand padding and the formatting of the number if the\n leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n between left-hand padding and the formatting of the number if the\n leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n point, even if no digits follow it.\n\n The precision determines the number of digits after the decimal\n point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n point, and trailing zeroes are not removed as they would otherwise\n be.\n\n The precision determines the number of significant digits before\n and after the decimal point and defaults to 6.\n\n5. If precision is ``N``, the output is truncated to ``N`` characters.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping. The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n Return the number of *i*\'s for which ``s[i] == x``.\n\n New in version 3.2.\n\nrange.index(x)\n\n Return the smallest *i* such that ``s[i] == x``. Raises\n ``ValueError`` when *x* is not in the range.\n\n New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object. Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation | Result | Notes |\n+================================+==================================+=======================+\n| ``s[i] = x`` | item *i* of *s* is replaced by | |\n| | *x* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t`` | slice of *s* from *i* to *j* is | |\n| | replaced by the contents of the | |\n| | iterable *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]`` | same as ``s[i:j] = []`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t`` | the elements of ``s[i:j:k]`` are | (1) |\n| | replaced by those of *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]`` | removes the elements of | |\n| | ``s[i:j:k]`` from the list | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)`` | same as ``s[len(s):len(s)] = | |\n| | [x]`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)`` | same as ``s[len(s):len(s)] = x`` | (2) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.clear()`` | remove all items from ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.copy()`` | return a shallow copy of ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)`` | return number of *i*\'s for which | |\n| | ``s[i] == x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])`` | return smallest *k* such that | (3) |\n| | ``s[k] == x`` and ``i <= k < j`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)`` | same as ``s[i:i] = [x]`` | (4) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])`` | same as ``x = s[i]; del s[i]; | (5) |\n| | return x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)`` | same as ``del s[s.index(x)]`` | (3) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()`` | reverses the items of *s* in | (6) |\n| | place | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])`` | sort the items of *s* in place | (6), (7), (8) |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n index is passed as the second or third parameter to the ``index()``\n method, the sequence length is added, as for slice indices. If it\n is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n ``insert()`` method, the sequence length is added, as for slice\n indices. If it is still negative, it is truncated to zero, as for\n slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n place for economy of space when sorting or reversing a large\n sequence. To remind you that they operate by side effect, they\n don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n comparisons. Each must be specified as a keyword argument.\n\n *key* specifies a function of one argument that is used to extract\n a comparison key from each list element: ``key=str.lower``. The\n default value is ``None``. Use ``functools.cmp_to_key()`` to\n convert an old-style *cmp* function to a *key* function.\n\n *reverse* is a boolean value. If set to ``True``, then the list\n elements are sorted as if each comparison were reversed.\n\n The ``sort()`` method is guaranteed to be stable. A sort is stable\n if it guarantees not to change the relative order of elements that\n compare equal --- this is helpful for sorting in multiple passes\n (for example, sort by department, then by salary grade).\n\n **CPython implementation detail:** While a list is being sorted,\n the effect of attempting to mutate, or even inspect, the list is\n undefined. The C implementation of Python makes the list appear\n empty for the duration, and raises ``ValueError`` if it can detect\n that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n New in version 3.3: ``clear()`` and ``copy()`` methods.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types. For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNew in version 3.3: The functions ``count()``, ``find()``,\n``index()``, ``rfind()`` and ``rindex()`` have additional semantics\ncompared to the corresponding string functions: They also accept an\ninteger in range 0 to 255 (a byte) as their first argument.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n their arguments, just as the methods on strings don\'t accept bytes\n as their arguments. For example, you have to write\n\n a = "abc"\n b = a.replace("a", "f")\n\n and\n\n a = b"abc"\n b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n Return a string decoded from the given bytes. Default encoding is\n ``\'utf-8\'``. *errors* may be given to set a different error\n handling scheme. The default for *errors* is ``\'strict\'``, meaning\n that encoding errors raise a ``UnicodeError``. Other possible\n values are ``\'ignore\'``, ``\'replace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n This ``bytes`` class method returns a bytes or bytearray object,\n decoding the given string object. The string must contain two\n hexadecimal digits per byte, spaces are ignored.\n\n >>> bytes.fromhex(\'f0 f1f2 \')\n b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n Return a copy of the bytes or bytearray object where all bytes\n occurring in the optional argument *delete* are removed, and the\n remaining bytes have been mapped through the given translation\n table, which must be a bytes object of length 256.\n\n You can use the ``bytes.maketrans()`` method to create a\n translation table.\n\n Set the *table* argument to ``None`` for translations that only\n delete characters:\n\n >>> b\'read this short text\'.translate(None, b\'aeiou\')\n b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n This static method returns a translation table usable for\n ``bytes.translate()`` that will map each character in *from* into\n the character at the same position in *to*; *from* and *to* must be\n bytes objects and have the same length.\n\n New in version 3.1.\n', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 09:29:28 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:29:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Suspicious_markup_check=2E?= Message-ID: http://hg.python.org/cpython/rev/9509ae977985 changeset: 76688:9509ae977985 user: Georg Brandl date: Tue May 01 09:29:56 2012 +0200 summary: Suspicious markup check. files: Doc/howto/logging-cookbook.rst | 2 +- Doc/tools/sphinxext/susp-ignored.csv | 1 + 2 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -1599,7 +1599,7 @@ 'ASCII section\ufeffUnicode section' - The Unicode code point ``'\feff```, when encoded using UTF-8, will be + The Unicode code point ``'\feff'``, when encoded using UTF-8, will be encoded as a UTF-8 BOM -- the byte-string ``b'\xef\xbb\xbf'``. #. Replace the ASCII section with whatever placeholders you like, but make sure diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -87,6 +87,7 @@ library/functions,,:stop,a[start:stop:step] library/hotshot,,:lineno,"ncalls tottime percall cumtime percall filename:lineno(function)" library/http.client,52,:port,host:port +library/http.cookies,,`,!#$%&'*+-.^_`|~: library/httplib,,:port,host:port library/imaplib,,:MM,"""DD-Mmm-YYYY HH:MM:SS" library/imaplib,,:SS,"""DD-Mmm-YYYY HH:MM:SS" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 09:34:27 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:34:27 +0200 Subject: [Python-checkins] =?utf8?q?release=3A_Remove_=22next=5Ftext=22=2C?= =?utf8?q?_which_is_confusing_and_unneeded=2E?= Message-ID: http://hg.python.org/release/rev/a40c9afb6da4 changeset: 59:a40c9afb6da4 user: Georg Brandl date: Tue May 01 09:34:58 2012 +0200 summary: Remove "next_text", which is confusing and unneeded. files: release.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/release.py b/release.py --- a/release.py +++ b/release.py @@ -155,7 +155,7 @@ print('Updating Lib/idlelib/idlever.py...', end=' ') with open('Lib/idlelib/idlever.py', 'w', encoding="ascii") as fp: - new = 'IDLE_VERSION = "%s"\n' % tag.next_text + new = 'IDLE_VERSION = "%s"\n' % tag.text fp.write(new) print('done') @@ -353,7 +353,6 @@ self.text = "{}.{}.{}".format(self.major, self.minor, self.patch) if self.level != "f": self.text += self.level + str(self.serial) - self.next_text = tag_name self.basic_version = '%s.%s' % (self.major, self.minor) def __str__(self): -- Repository URL: http://hg.python.org/release From python-checkins at python.org Tue May 1 09:34:51 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:34:51 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Bump_to_3=2E3=2E0a3=2E?= Message-ID: http://hg.python.org/cpython/rev/0b4308118618 changeset: 76689:0b4308118618 user: Georg Brandl date: Tue May 01 09:35:18 2012 +0200 summary: Bump to 3.3.0a3. files: Include/patchlevel.h | 4 ++-- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 2 +- Misc/RPM/python-3.3.spec | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 3 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA -#define PY_RELEASE_SERIAL 2 +#define PY_RELEASE_SERIAL 3 /* Version as a string */ -#define PY_VERSION "3.3.0a2+" +#define PY_VERSION "3.3.0a3" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.0a2" +__version__ = "3.3.0a3" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.0a2" +IDLE_VERSION = "3.3.0a3" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -5,7 +5,7 @@ What's New in Python 3.3.0 Alpha 3? =================================== -*Release date: XXXX-XX-XX* +*Release date: 01-May-2012* Core and Builtins ----------------- diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.0a2 +%define version 3.3.0a3 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,4 +1,4 @@ -This is Python version 3.3.0 alpha 2 +This is Python version 3.3.0 alpha 3 ==================================== Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 09:57:16 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:57:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Disable_test=5F13183_tempor?= =?utf8?q?arily_on_Windows_for_3=2E3a3_release=2E?= Message-ID: http://hg.python.org/cpython/rev/0b53b70a40a0 changeset: 76690:0b53b70a40a0 tag: v3.3.0a3 user: Georg Brandl date: Tue May 01 09:57:34 2012 +0200 summary: Disable test_13183 temporarily on Windows for 3.3a3 release. files: Lib/test/test_pdb.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -1,5 +1,5 @@ # A test suite for pdb; not very comprehensive at the moment. - +import os import imp import pdb import sys @@ -631,6 +631,7 @@ self.assertNotIn(b'SyntaxError', stdout, "Got a syntax error running test script under PDB") + @unittest.skipIf(os.name == 'nt', "temporarily disabled on Windows") def test_issue13183(self): script = """ from bar import bar -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 09:57:16 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 09:57:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Added_tag_v3=2E3=2E0a3_for_?= =?utf8?q?changeset_0b53b70a40a0?= Message-ID: http://hg.python.org/cpython/rev/0d8cbfe8d88b changeset: 76691:0d8cbfe8d88b user: Georg Brandl date: Tue May 01 09:57:42 2012 +0200 summary: Added tag v3.3.0a3 for changeset 0b53b70a40a0 files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -101,3 +101,4 @@ 3d0686d90f55a78f96d9403da2c52dc2411419d0 v3.2.3 f1a9a6505731714f0e157453ff850e3b71615c45 v3.3.0a1 2f69db52d6de306cdaef0a0cc00cc823fb350b01 v3.3.0a2 +0b53b70a40a00013505eb35e3660057b62be77be v3.3.0a3 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 11:55:54 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 11:55:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_an_option_to_timeit_to_?= =?utf8?q?use_time=2Eprocess=5Ftime=28=29_and_mark_-t_and_-c_as?= Message-ID: http://hg.python.org/cpython/rev/d43a8aa9dbef changeset: 76692:d43a8aa9dbef user: Georg Brandl date: Tue May 01 11:56:22 2012 +0200 summary: Add an option to timeit to use time.process_time() and mark -t and -c as deprecated. files: Lib/timeit.py | 14 +++++++++----- 1 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Lib/timeit.py b/Lib/timeit.py --- a/Lib/timeit.py +++ b/Lib/timeit.py @@ -9,14 +9,15 @@ Library usage: see the Timer class. Command line usage: - python timeit.py [-n N] [-r N] [-s S] [-t] [-c] [-h] [--] [statement] + python timeit.py [-n N] [-r N] [-s S] [-t] [-c] [-p] [-h] [--] [statement] Options: -n/--number N: how many times to execute 'statement' (default: see below) -r/--repeat N: how many times to repeat the timer (default 3) -s/--setup S: statement to be executed once initially (default 'pass') - -t/--time: use time.time() - -c/--clock: use time.clock() + -p/--process: use time.process_time() (default is time.perf_counter()) + -t/--time: use time.time() (deprecated) + -c/--clock: use time.clock() (deprecated) -v/--verbose: print raw timing results; repeat for more digits precision -h/--help: print this usage message and exit --: separate options from statement, use when statement starts with - @@ -249,9 +250,10 @@ args = sys.argv[1:] import getopt try: - opts, args = getopt.getopt(args, "n:s:r:tcvh", + opts, args = getopt.getopt(args, "n:s:r:tcpvh", ["number=", "setup=", "repeat=", - "time", "clock", "verbose", "help"]) + "time", "clock", "process", + "verbose", "help"]) except getopt.error as err: print(err) print("use -h/--help for command line help") @@ -276,6 +278,8 @@ timer = time.time if o in ("-c", "--clock"): timer = time.clock + if o in ("-p", "--process"): + timer = time.process_time if o in ("-v", "--verbose"): if verbose: precision += 1 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 11:59:11 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 11:59:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_timeit_documentation?= =?utf8?q?_w=2Er=2Et_default_timer_changes=2E?= Message-ID: http://hg.python.org/cpython/rev/552c207f65e4 changeset: 76693:552c207f65e4 user: Georg Brandl date: Tue May 01 11:59:36 2012 +0200 summary: Update timeit documentation w.r.t default timer changes. files: Doc/library/timeit.rst | 30 +++++++++++++++--------------- 1 files changed, 15 insertions(+), 15 deletions(-) diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -104,13 +104,7 @@ .. function:: default_timer() - Define a default timer, in a platform specific manner. On Windows, - :func:`time.clock` has microsecond granularity but :func:`time.time`'s - granularity is 1/60th of a second; on Unix, :func:`time.clock` has 1/100th of - a second granularity and :func:`time.time` is much more precise. On either - platform, :func:`default_timer` measures wall clock time, not the CPU - time. This means that other processes running on the same computer may - interfere with the timing. + The default timer, which is always :func:`time.perf_counter`. .. function:: repeat(stmt='pass', setup='pass', timer=, repeat=3, number=1000000) @@ -149,13 +143,20 @@ statement to be executed once initially (default ``pass``) +.. cmdoption:: -p, --process + + measure process time, not wallclock time, using :func:`time.process_time` + instead of :func:`time.perf_counter`, which is the default + + .. versionadded:: 3.3 + .. cmdoption:: -t, --time - use :func:`time.time` (default on all platforms but Windows) + use :func:`time.time` (deprecated) .. cmdoption:: -c, --clock - use :func:`time.clock` (default on Windows) + use :func:`time.clock` (deprecated) .. cmdoption:: -v, --verbose @@ -173,12 +174,11 @@ If :option:`-n` is not given, a suitable number of loops is calculated by trying successive powers of 10 until the total time is at least 0.2 seconds. -:func:`default_timer` measurations can be affected by other programs running on -the same machine, so -the best thing to do when accurate timing is necessary is to repeat -the timing a few times and use the best time. The :option:`-r` option is good -for this; the default of 3 repetitions is probably enough in most cases. On -Unix, you can use :func:`time.clock` to measure CPU time. +:func:`default_timer` measurements can be affected by other programs running on +the same machine, so the best thing to do when accurate timing is necessary is +to repeat the timing a few times and use the best time. The :option:`-r` +option is good for this; the default of 3 repetitions is probably enough in +most cases. You can use :func:`time.process_time` to measure CPU time. .. note:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 13:39:42 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:39:42 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_3=2E3_release_schedule_update?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/peps/rev/6a05bfdc339f changeset: 4331:6a05bfdc339f user: Georg Brandl date: Tue May 01 13:40:12 2012 +0200 summary: 3.3 release schedule update. files: pep-0398.txt | 22 ++++++++++++---------- 1 files changed, 12 insertions(+), 10 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -35,8 +35,8 @@ The current schedule is: - 3.3.0 alpha 1: March 3, 2012 -- 3.3.0 alpha 2: March 31, 2012 -- 3.3.0 alpha 3: April 28, 2012 +- 3.3.0 alpha 2: April 1, 2012 +- 3.3.0 alpha 3: May 1, 2012 - 3.3.0 alpha 4: May 26, 2012 - 3.3.0 beta 1: June 23, 2012 @@ -45,13 +45,13 @@ - 3.3.0 beta 2: July 14, 2012 - 3.3.0 candidate 1: July 28, 2012 - 3.3.0 candidate 2: August 11, 2012 -- 3.3.0 final: August 18, 2012 +- 3.3.0 final: August 25, 2012 .. don't forget to update final date above as well -Every release day listed here is a Saturday; the actual availability -of the release for download on python.org will depend on the schedules -of the crew. +Every release day listed here is the prospective day of tagging the release; +the actual availability of the release for download on python.org will depend +on the schedules of the crew. Features for 3.3 @@ -63,7 +63,9 @@ * PEP 393: Flexible String Representation * PEP 399: Pure Python/C Accelerator Module Compatibility Requirements * PEP 409: Suppressing exception context +* PEP 412: Key-Sharing Dictionary * PEP 414: Explicit Unicode Literal for Python 3.3 +* PEP 418: Add monotonic time, performance counter, and process time functions * PEP 3151: Reworking the OS and IO exception hierarchy * PEP 3155: Qualified name for classes and functions @@ -72,6 +74,9 @@ * Addition of the "packaging" module, deprecating "distutils" * Addition of the "faulthandler" module * Addition of the "lzma" module, and lzma/xz support in tarfile +* Implementing ``__import__`` using importlib +* Addition of the "unittest.mock" library +* Addition of the C decimal implementation Candidate PEPs: @@ -80,7 +85,7 @@ * PEP 397: Python launcher for Windows * PEP 402: Simplified Package Layout (likely a new PEP derived from it) * PEP 405: Python Virtual Environments -* PEP 412: Key-Sharing Dictionary +* PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3144: IP Address manipulation library @@ -89,11 +94,8 @@ Other planned large-scale changes: -* Addition of the "mock" library -* Addition of the C decimal implementation * Addition of the "regex" module * Email version 6 -* Implementing ``__import__`` using importlib * A standard event-loop interface (PEP by Jim Fulton pending) * Breaking out standard library and docs in separate repos? -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 13:47:17 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:47:17 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_This_PEP_should_be_final=2C_as?= =?utf8?q?_the_feature_has_been_implemented_long_ago=2E?= Message-ID: http://hg.python.org/peps/rev/51119569c556 changeset: 4332:51119569c556 user: Georg Brandl date: Tue May 01 13:47:47 2012 +0200 summary: This PEP should be final, as the feature has been implemented long ago. files: pep-0274.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0274.txt b/pep-0274.txt --- a/pep-0274.txt +++ b/pep-0274.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Barry Warsaw -Status: Accepted +Status: Final Type: Standards Track Created: 25-Oct-2001 Python-Version: 2.7, 3.0 (originally 2.3) -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 13:48:17 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:48:17 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_409_is_implemented_and_fin?= =?utf8?b?YWwu?= Message-ID: http://hg.python.org/peps/rev/79aece959b29 changeset: 4333:79aece959b29 user: Georg Brandl date: Tue May 01 13:48:42 2012 +0200 summary: PEP 409 is implemented and final. files: pep-0409.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0409.txt b/pep-0409.txt --- a/pep-0409.txt +++ b/pep-0409.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 26-Jan-2012 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 13:54:08 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:54:08 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Mock_is_actually_a_PEP=2E?= Message-ID: http://hg.python.org/peps/rev/0b512cc80c43 changeset: 4334:0b512cc80c43 user: Georg Brandl date: Tue May 01 13:54:18 2012 +0200 summary: Mock is actually a PEP. files: pep-0398.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -65,6 +65,7 @@ * PEP 409: Suppressing exception context * PEP 412: Key-Sharing Dictionary * PEP 414: Explicit Unicode Literal for Python 3.3 +* PEP 417: Including mock in the Standard Library * PEP 418: Add monotonic time, performance counter, and process time functions * PEP 3151: Reworking the OS and IO exception hierarchy * PEP 3155: Qualified name for classes and functions @@ -75,7 +76,6 @@ * Addition of the "faulthandler" module * Addition of the "lzma" module, and lzma/xz support in tarfile * Implementing ``__import__`` using importlib -* Addition of the "unittest.mock" library * Addition of the C decimal implementation Candidate PEPs: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 13:54:09 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:54:09 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_418_is_accepted_and_implem?= =?utf8?q?ented=2C_therefore_Final=2E?= Message-ID: http://hg.python.org/peps/rev/8535e3040db1 changeset: 4335:8535e3040db1 user: Georg Brandl date: Tue May 01 13:54:39 2012 +0200 summary: PEP 418 is accepted and implemented, therefore Final. files: pep-0418.txt | 15 ++++++++++++++- 1 files changed, 14 insertions(+), 1 deletions(-) diff --git a/pep-0418.txt b/pep-0418.txt --- a/pep-0418.txt +++ b/pep-0418.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Cameron Simpson , Jim Jewett , Stephen J. Turnbull , Victor Stinner -Status: Draft +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 26-March-2012 @@ -1622,6 +1622,19 @@ `_ by Joe Korty (2010-04) +Acceptance +========== + +The PEP was accepted on 2012-04-28 by Guido van Rossum [1]_. The PEP +implementation has since been committed to the repository. + + +References +========== + +.. [1] http://mail.python.org/pipermail/python-dev/2012-April/119094.html + + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 13:55:59 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:55:59 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_PEP_395_name=2E?= Message-ID: http://hg.python.org/peps/rev/7a5d28c11698 changeset: 4336:7a5d28c11698 user: Georg Brandl date: Tue May 01 13:56:30 2012 +0200 summary: Fix PEP 395 name. files: pep-0395.txt | 2 +- pep-0398.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0395.txt b/pep-0395.txt --- a/pep-0395.txt +++ b/pep-0395.txt @@ -1,5 +1,5 @@ PEP: 395 -Title: Qualifed Names for Modules +Title: Qualified Names for Modules Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -81,7 +81,7 @@ Candidate PEPs: * PEP 362: Function Signature Object -* PEP 395: Module Aliasing +* PEP 395: Qualified Names for Modules * PEP 397: Python launcher for Windows * PEP 402: Simplified Package Layout (likely a new PEP derived from it) * PEP 405: Python Virtual Environments -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 13:57:09 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 13:57:09 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_PEP_3154=2E?= Message-ID: http://hg.python.org/peps/rev/3fe9b5978b54 changeset: 4337:3fe9b5978b54 user: Georg Brandl date: Tue May 01 13:57:40 2012 +0200 summary: Add PEP 3154. files: pep-0398.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -88,6 +88,7 @@ * PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3144: IP Address manipulation library +* PEP 3154: Pickle protocol version 4 (Note that these are not accepted yet and even if they are, they might not be finished in time for Python 3.3.) -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 14:08:47 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 14:08:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Check_extract=5Fversion_whe?= =?utf8?q?n_opening_a_zipfile=2E?= Message-ID: http://hg.python.org/cpython/rev/66741dd4fabf changeset: 76694:66741dd4fabf user: Martin v. L?wis date: Tue May 01 14:08:22 2012 +0200 summary: Check extract_version when opening a zipfile. files: Lib/test/test_zipfile.py | 10 ++++++++++ Lib/zipfile.py | 5 +++++ 2 files changed, 15 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -828,6 +828,16 @@ b'\x00\x00\x00\x00'), } + def test_unsupported_version(self): + # File has an extract_version of 120 + data = (b'PK\x03\x04x\x00\x00\x00\x00\x00!p\xa1@\x00\x00\x00\x00\x00\x00' + b'\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00xPK\x01\x02x\x03x\x00\x00\x00\x00' + b'\x00!p\xa1@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00' + b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01\x00\x00\x00\x00xPK\x05\x06' + b'\x00\x00\x00\x00\x01\x00\x01\x00/\x00\x00\x00\x1f\x00\x00\x00\x00\x00') + self.assertRaises(NotImplementedError, zipfile.ZipFile, + io.BytesIO(data), 'r') + def test_unicode_filenames(self): with zipfile.ZipFile(TESTFN, "w") as zf: zf.writestr("foo.txt", "Test for unicode filename") diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -57,6 +57,8 @@ DEFAULT_VERSION = 20 ZIP64_VERSION = 45 BZIP2_VERSION = 46 +# we recognize (but not necessarily support) all features up to that version +MAX_EXTRACT_VERSION = 46 # Below are some formats and associated data for reading/writing headers using # the struct module. The names and structures of headers/records are those used @@ -920,6 +922,9 @@ (x.create_version, x.create_system, x.extract_version, x.reserved, x.flag_bits, x.compress_type, t, d, x.CRC, x.compress_size, x.file_size) = centdir[1:12] + if x.extract_version > MAX_EXTRACT_VERSION: + raise NotImplementedError("zip file version %.1f" % + (x.extract_version / 10)) x.volume, x.internal_attr, x.external_attr = centdir[15:18] # Convert date/time code to (year, month, day, hour, min, sec) x._raw_time = t -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 14:23:00 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 14:23:00 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Clarifications=2E?= Message-ID: http://hg.python.org/peps/rev/377fe8cb57a0 changeset: 4338:377fe8cb57a0 user: Eric V. Smith date: Tue May 01 08:22:49 2012 -0400 summary: Clarifications. files: pep-0420.txt | 23 ++++++++++++++--------- 1 files changed, 14 insertions(+), 9 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -37,9 +37,9 @@ * "portion" refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. * "regular package" refers to packages as they are implemented in - Python 3.2. + Python 3.2 and earlier. -This PEP describes a new type of package, the "namespace package". +This PEP defines a new type of package, the "namespace package". Namespace packages today ======================== @@ -169,6 +169,9 @@ Discussion ========== +At PyCon 2012, we had a discussion about namespace packages at which +PEP 382 and PEP 402 were rejected, to be replaced by this PEP [2]_. + There is no intention to remove support of regular packages. If there is no intention that a package is a namespace package, then there is a performance advantage to it being a regular package. Creation and @@ -181,9 +184,6 @@ imported as a namespace package, whereas in prior Python versions an ImportWarning would be raised. -At PyCon 2012, we had a discussion about namespace packages at which -PEP 382 and PEP 402 were rejected, to be replaced by this PEP [2]_. - Nick Coglan presented a list of his objections to this proposal [3]_. They are: @@ -217,19 +217,24 @@ If these are installed in the same location, a single directory "foo" would be in a directory that is on ``sys.path``. Inside "foo" would be two directories, "bar" and "baz". If "foo.bar" is removed (perhaps -by an automatic packager), care must be taken not to remove the +by an OS package manager), care must be taken not to remove the "foo/baz" or "foo" directories. Note that in this case "foo" will be a namespace package (because it lacks an ``__init__.py``), even though all of its portions are in the same directory. +Note that "foo.bar" and "foo.baz" can be installed into the same "foo" +directory because they will not have any files in common. + If the portions are installed in different locations, two different "foo" directories would be in directories that are on ``sys.path``. "foo/bar" would be in one of these sys.path entries, and "foo/baz" would be in the other. Upon removal of "foo.bar", the "foo/bar" and -corresonding "foo" directories can be completely removed. +corresonding "foo" directories can be completely removed. But +"foo/baz" and its corresponding "foo" directory cannot be removed. -Note that even if they are installed in the same directory, "foo.bar" -and "foo.baz" would not have any files in common. +It is also possible to have the "foo.bar" portion installed in a +directory on ``sys.path``, and have the "foo.baz" portion provided in +a zip file, also on ``sys.path``. References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 15:37:45 2012 From: python-checkins at python.org (nick.coghlan) Date: Tue, 01 May 2012 15:37:45 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Move_the_mock_and_buffer_proto?= =?utf8?q?col_PEPs_to_Final?= Message-ID: http://hg.python.org/peps/rev/eba6bcdb19eb changeset: 4339:eba6bcdb19eb user: Nick Coghlan date: Tue May 01 23:37:34 2012 +1000 summary: Move the mock and buffer protocol PEPs to Final files: pep-0398.txt | 1 + pep-0417.txt | 2 +- pep-3118.txt | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -67,6 +67,7 @@ * PEP 414: Explicit Unicode Literal for Python 3.3 * PEP 417: Including mock in the Standard Library * PEP 418: Add monotonic time, performance counter, and process time functions +* PEP 3118: Revising the buffer protocol (protocol semantics finalised) * PEP 3151: Reworking the OS and IO exception hierarchy * PEP 3155: Qualified name for classes and functions diff --git a/pep-0417.txt b/pep-0417.txt --- a/pep-0417.txt +++ b/pep-0417.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Michael Foord -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 12-Mar-2012 diff --git a/pep-3118.txt b/pep-3118.txt --- a/pep-3118.txt +++ b/pep-3118.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Travis Oliphant , Carl Banks -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 28-Aug-2006 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 15:38:41 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 15:38:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_strip_is=5F_prefixes_on_clo?= =?utf8?q?ck=5Finfo_fields?= Message-ID: http://hg.python.org/cpython/rev/47c84e3f08ae changeset: 76695:47c84e3f08ae user: Benjamin Peterson date: Tue May 01 09:38:34 2012 -0400 summary: strip is_ prefixes on clock_info fields files: Doc/library/time.rst | 4 +- Include/pytime.h | 4 +- Lib/test/test_time.py | 22 +++++----- Modules/timemodule.c | 62 +++++++++++++++--------------- Python/pytime.c | 18 ++++---- 5 files changed, 55 insertions(+), 55 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -168,11 +168,11 @@ The name of the underlying C function used to get the clock value. - .. attribute:: is_monotonic + .. attribute:: monotonic ``True`` if the clock cannot go backward, ``False`` otherwise. - .. attribute:: is_adjusted + .. attribute:: adjusted ``True`` if the clock can be adjusted (e.g. by a NTP daemon), ``False`` otherwise. diff --git a/Include/pytime.h b/Include/pytime.h --- a/Include/pytime.h +++ b/Include/pytime.h @@ -25,8 +25,8 @@ /* Structure used by time.get_clock_info() */ typedef struct { const char *implementation; - int is_monotonic; - int is_adjusted; + int monotonic; + int adjusted; double resolution; } _Py_clock_info_t; diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -30,16 +30,16 @@ def test_time(self): time.time() info = time.get_clock_info('time') - self.assertEqual(info.is_monotonic, False) + self.assertEqual(info.monotonic, False) if sys.platform != 'win32': - self.assertEqual(info.is_adjusted, True) + self.assertEqual(info.adjusted, True) def test_clock(self): time.clock() info = time.get_clock_info('clock') - self.assertEqual(info.is_monotonic, True) - self.assertEqual(info.is_adjusted, False) + self.assertEqual(info.monotonic, True) + self.assertEqual(info.adjusted, False) @unittest.skipUnless(hasattr(time, 'clock_gettime'), 'need time.clock_gettime()') @@ -370,11 +370,11 @@ self.assertAlmostEqual(dt, 0.1, delta=0.2) info = time.get_clock_info('monotonic') - self.assertEqual(info.is_monotonic, True) + self.assertEqual(info.monotonic, True) if sys.platform == 'linux': - self.assertEqual(info.is_adjusted, True) + self.assertEqual(info.adjusted, True) else: - self.assertEqual(info.is_adjusted, False) + self.assertEqual(info.adjusted, False) def test_perf_counter(self): time.perf_counter() @@ -386,8 +386,8 @@ self.assertLess(stop - start, 0.01) info = time.get_clock_info('process_time') - self.assertEqual(info.is_monotonic, True) - self.assertEqual(info.is_adjusted, False) + self.assertEqual(info.monotonic, True) + self.assertEqual(info.adjusted, False) @unittest.skipUnless(hasattr(time, 'monotonic'), 'need time.monotonic') @@ -433,12 +433,12 @@ #self.assertIsInstance(info, dict) self.assertIsInstance(info.implementation, str) self.assertNotEqual(info.implementation, '') - self.assertIsInstance(info.is_monotonic, bool) + self.assertIsInstance(info.monotonic, bool) self.assertIsInstance(info.resolution, float) # 0.0 < resolution <= 1.0 self.assertGreater(info.resolution, 0.0) self.assertLessEqual(info.resolution, 1.0) - self.assertIsInstance(info.is_adjusted, bool) + self.assertIsInstance(info.adjusted, bool) self.assertRaises(ValueError, time.get_clock_info, 'xxx') diff --git a/Modules/timemodule.c b/Modules/timemodule.c --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -95,8 +95,8 @@ if (info) { info->implementation = "clock()"; info->resolution = 1.0 / (double)CLOCKS_PER_SEC; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; } return PyFloat_FromDouble((double)value / CLOCKS_PER_SEC); } @@ -131,8 +131,8 @@ if (info) { info->implementation = "QueryPerformanceCounter()"; info->resolution = 1.0 / (double)cpu_frequency; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; } *result = PyFloat_FromDouble(diff / (double)cpu_frequency); return 0; @@ -874,7 +874,7 @@ info->implementation = "GetTickCount64()"; else info->implementation = "GetTickCount()"; - info->is_monotonic = 1; + info->monotonic = 1; ok = GetSystemTimeAdjustment(&timeAdjustment, &timeIncrement, &isTimeAdjustmentDisabled); if (!ok) { @@ -882,7 +882,7 @@ return NULL; } info->resolution = timeIncrement * 1e-7; - info->is_adjusted = 0; + info->adjusted = 0; } return PyFloat_FromDouble(result); @@ -902,8 +902,8 @@ if (info) { info->implementation = "mach_absolute_time()"; info->resolution = (double)timebase.numer / timebase.denom * 1e-9; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; } return PyFloat_FromDouble(secs); @@ -924,14 +924,14 @@ if (info) { struct timespec res; - info->is_monotonic = 1; + info->monotonic = 1; info->implementation = function; #if (defined(linux) || defined(__linux) || defined(__linux__)) \ && !defined(CLOCK_HIGHRES) /* CLOCK_MONOTONIC is adjusted on Linux */ - info->is_adjusted = 1; + info->adjusted = 1; #else - info->is_adjusted = 0; + info->adjusted = 0; #endif if (clock_getres(clk_id, &res) == 0) info->resolution = res.tv_sec + res.tv_nsec * 1e-9; @@ -1023,8 +1023,8 @@ if (info) { info->implementation = "GetProcessTimes()"; info->resolution = 1e-7; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; } return PyFloat_FromDouble(total * 1e-7); #else @@ -1052,8 +1052,8 @@ if (info) { struct timespec res; info->implementation = function; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; if (clock_getres(clk_id, &res) == 0) info->resolution = res.tv_sec + res.tv_nsec * 1e-9; else @@ -1070,8 +1070,8 @@ total += ru.ru_stime.tv_sec + ru.ru_stime.tv_usec * 1e-6; if (info) { info->implementation = "getrusage(RUSAGE_SELF)"; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; info->resolution = 1e-6; } return PyFloat_FromDouble(total); @@ -1099,8 +1099,8 @@ total += (double)t.tms_stime / ticks_per_second; if (info) { info->implementation = "times()"; - info->is_monotonic = 1; - info->is_adjusted = 0; + info->monotonic = 1; + info->adjusted = 0; info->resolution = 1.0 / ticks_per_second; } return PyFloat_FromDouble(total); @@ -1132,8 +1132,8 @@ static PyStructSequence_Field ClockInfo_fields[] = { {"implementation", "name of the underlying C function " "used to get the clock value"}, - {"is_monotonic", "True if the clock cannot go backward, False otherwise"}, - {"is_adjusted", "True if the clock can be adjusted " + {"monotonic", "True if the clock cannot go backward, False otherwise"}, + {"adjusted", "True if the clock can be adjusted " "(e.g. by a NTP daemon), False otherwise"}, {"resolution", "resolution of the clock in seconds"}, {NULL, NULL} @@ -1159,13 +1159,13 @@ #ifdef Py_DEBUG info.implementation = NULL; - info.is_monotonic = -1; - info.is_adjusted = -1; + info.monotonic = -1; + info.adjusted = -1; info.resolution = -1.0; #else info.implementation = ""; - info.is_monotonic = 0; - info.is_adjusted = 0; + info.monotonic = 0; + info.adjusted = 0; info.resolution = 1.0; #endif @@ -1201,14 +1201,14 @@ goto error; PyStructSequence_SET_ITEM(result, 0, obj); - assert(info.is_monotonic != -1); - obj = PyBool_FromLong(info.is_monotonic); + assert(info.monotonic != -1); + obj = PyBool_FromLong(info.monotonic); if (obj == NULL) goto error; PyStructSequence_SET_ITEM(result, 1, obj); - assert(info.is_adjusted != -1); - obj = PyBool_FromLong(info.is_adjusted); + assert(info.adjusted != -1); + obj = PyBool_FromLong(info.adjusted); if (obj == NULL) goto error; PyStructSequence_SET_ITEM(result, 2, obj); @@ -1487,8 +1487,8 @@ if (info) { struct timespec res; info->implementation = "clock_gettime(CLOCK_REALTIME)"; - info->is_monotonic = 0; - info->is_adjusted = 1; + info->monotonic = 0; + info->adjusted = 1; if (clock_getres(CLOCK_REALTIME, &res) == 0) info->resolution = res.tv_sec + res.tv_nsec * 1e-9; else diff --git a/Python/pytime.c b/Python/pytime.c --- a/Python/pytime.c +++ b/Python/pytime.c @@ -40,14 +40,14 @@ BOOL isTimeAdjustmentDisabled; info->implementation = "GetSystemTimeAsFileTime()"; - info->is_monotonic = 0; + info->monotonic = 0; (void) GetSystemTimeAdjustment(&timeAdjustment, &timeIncrement, &isTimeAdjustmentDisabled); info->resolution = timeIncrement * 1e-7; if (isTimeAdjustmentDisabled) - info->is_adjusted = 0; + info->adjusted = 0; else - info->is_adjusted = 1; + info->adjusted = 1; } #else /* There are three ways to get the time: @@ -70,8 +70,8 @@ if (info) { info->implementation = "gettimeofday()"; info->resolution = 1e-6; - info->is_monotonic = 0; - info->is_adjusted = 1; + info->monotonic = 0; + info->adjusted = 1; } return; } @@ -86,8 +86,8 @@ if (info) { info->implementation = "ftime()"; info->resolution = 1e-3; - info->is_monotonic = 0; - info->is_adjusted = 1; + info->monotonic = 0; + info->adjusted = 1; } } #else /* !HAVE_FTIME */ @@ -96,8 +96,8 @@ if (info) { info->implementation = "time()"; info->resolution = 1.0; - info->is_monotonic = 0; - info->is_adjusted = 1; + info->monotonic = 0; + info->adjusted = 1; } #endif /* !HAVE_FTIME */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 15:56:15 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 15:56:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fix_calling_the?= =?utf8?q?_classmethod_descriptor_directly_=28closes_=2314699=29?= Message-ID: http://hg.python.org/cpython/rev/eab5120cc208 changeset: 76696:eab5120cc208 branch: 3.2 parent: 76676:96cb47f8142e user: Benjamin Peterson date: Tue May 01 09:51:09 2012 -0400 summary: fix calling the classmethod descriptor directly (closes #14699) files: Lib/test/test_descr.py | 16 ++++++++++ Misc/NEWS | 2 + Objects/descrobject.c | 44 +++++++++++++++++++++++++++-- 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -1458,6 +1458,22 @@ self.assertEqual(x, spam.spamlist) self.assertEqual(a, a1) self.assertEqual(d, d1) + spam_cm = spam.spamlist.__dict__['classmeth'] + x2, a2, d2 = spam_cm(spam.spamlist, *a, **d) + self.assertEqual(x2, spam.spamlist) + self.assertEqual(a2, a1) + self.assertEqual(d2, d1) + class SubSpam(spam.spamlist): pass + x2, a2, d2 = spam_cm(SubSpam, *a, **d) + self.assertEqual(x2, SubSpam) + self.assertEqual(a2, a1) + self.assertEqual(d2, d1) + with self.assertRaises(TypeError): + spam_cm() + with self.assertRaises(TypeError): + spam_cm(spam.spamlist()) + with self.assertRaises(TypeError): + spam_cm(list) def test_staticmethods(self): # Testing static methods... diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #14699: Fix calling the classmethod descriptor directly. + - Issue #14433: Prevent msvcrt crash in interactive prompt when stdin is closed. diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -256,14 +256,52 @@ classmethoddescr_call(PyMethodDescrObject *descr, PyObject *args, PyObject *kwds) { - PyObject *func, *result; + Py_ssize_t argc; + PyObject *self, *func, *result; - func = PyCFunction_New(descr->d_method, (PyObject *)PyDescr_TYPE(descr)); + /* Make sure that the first argument is acceptable as 'self' */ + assert(PyTuple_Check(args)); + argc = PyTuple_GET_SIZE(args); + if (argc < 1) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' of '%.100s' " + "object needs an argument", + descr_name((PyDescrObject *)descr), "?", + PyDescr_TYPE(descr)->tp_name); + return NULL; + } + self = PyTuple_GET_ITEM(args, 0); + if (!PyType_Check(self)) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' requires a type " + "but received a '%.100s'", + descr_name((PyDescrObject *)descr), "?", + PyDescr_TYPE(descr)->tp_name, + self->ob_type->tp_name); + return NULL; + } + if (!PyType_IsSubtype((PyTypeObject *)self, PyDescr_TYPE(descr))) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' " + "requires a subtype of '%.100s' " + "but received '%.100s", + descr_name((PyDescrObject *)descr), "?", + PyDescr_TYPE(descr)->tp_name, + self->ob_type->tp_name); + return NULL; + } + + func = PyCFunction_New(descr->d_method, self); if (func == NULL) return NULL; - + args = PyTuple_GetSlice(args, 1, argc); + if (args == NULL) { + Py_DECREF(func); + return NULL; + } result = PyEval_CallObjectWithKeywords(func, args, kwds); Py_DECREF(func); + Py_DECREF(args); return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 15:56:15 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 15:56:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?b?OiBtZXJnZSAzLjIgKCMxNDY5OSk=?= Message-ID: http://hg.python.org/cpython/rev/e1a200dfd5db changeset: 76697:e1a200dfd5db parent: 76695:47c84e3f08ae parent: 76696:eab5120cc208 user: Benjamin Peterson date: Tue May 01 09:51:46 2012 -0400 summary: merge 3.2 (#14699) files: Lib/test/test_descr.py | 16 ++++++++++ Misc/NEWS | 2 + Objects/descrobject.c | 44 +++++++++++++++++++++++++++-- 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -1466,6 +1466,22 @@ self.assertEqual(x, spam.spamlist) self.assertEqual(a, a1) self.assertEqual(d, d1) + spam_cm = spam.spamlist.__dict__['classmeth'] + x2, a2, d2 = spam_cm(spam.spamlist, *a, **d) + self.assertEqual(x2, spam.spamlist) + self.assertEqual(a2, a1) + self.assertEqual(d2, d1) + class SubSpam(spam.spamlist): pass + x2, a2, d2 = spam_cm(SubSpam, *a, **d) + self.assertEqual(x2, SubSpam) + self.assertEqual(a2, a1) + self.assertEqual(d2, d1) + with self.assertRaises(TypeError): + spam_cm() + with self.assertRaises(TypeError): + spam_cm(spam.spamlist()) + with self.assertRaises(TypeError): + spam_cm(list) def test_staticmethods(self): # Testing static methods... diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #14699: Fix calling the classmethod descriptor directly. + - Issue #14433: Prevent msvcrt crash in interactive prompt when stdin is closed. diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -257,14 +257,52 @@ classmethoddescr_call(PyMethodDescrObject *descr, PyObject *args, PyObject *kwds) { - PyObject *func, *result; + Py_ssize_t argc; + PyObject *self, *func, *result; - func = PyCFunction_New(descr->d_method, (PyObject *)PyDescr_TYPE(descr)); + /* Make sure that the first argument is acceptable as 'self' */ + assert(PyTuple_Check(args)); + argc = PyTuple_GET_SIZE(args); + if (argc < 1) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' of '%.100s' " + "object needs an argument", + descr_name((PyDescrObject *)descr), "?", + PyDescr_TYPE(descr)->tp_name); + return NULL; + } + self = PyTuple_GET_ITEM(args, 0); + if (!PyType_Check(self)) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' requires a type " + "but received a '%.100s'", + descr_name((PyDescrObject *)descr), "?", + PyDescr_TYPE(descr)->tp_name, + self->ob_type->tp_name); + return NULL; + } + if (!PyType_IsSubtype((PyTypeObject *)self, PyDescr_TYPE(descr))) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' " + "requires a subtype of '%.100s' " + "but received '%.100s", + descr_name((PyDescrObject *)descr), "?", + PyDescr_TYPE(descr)->tp_name, + self->ob_type->tp_name); + return NULL; + } + + func = PyCFunction_New(descr->d_method, self); if (func == NULL) return NULL; - + args = PyTuple_GetSlice(args, 1, argc); + if (args == NULL) { + Py_DECREF(func); + return NULL; + } result = PyEval_CallObjectWithKeywords(func, args, kwds); Py_DECREF(func); + Py_DECREF(args); return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 15:56:16 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 15:56:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_fix_calling_the?= =?utf8?q?_classmethod_descriptor_directly_=28closes_=2314699=29?= Message-ID: http://hg.python.org/cpython/rev/6484f5a51285 changeset: 76698:6484f5a51285 branch: 2.7 parent: 76679:4c3c4891fd6a user: Benjamin Peterson date: Tue May 01 09:51:09 2012 -0400 summary: fix calling the classmethod descriptor directly (closes #14699) files: Lib/test/test_descr.py | 16 ++++++++++ Misc/NEWS | 2 + Objects/descrobject.c | 44 +++++++++++++++++++++++++++-- 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -1419,6 +1419,22 @@ self.assertEqual(x, spam.spamlist) self.assertEqual(a, a1) self.assertEqual(d, d1) + spam_cm = spam.spamlist.__dict__['classmeth'] + x2, a2, d2 = spam_cm(spam.spamlist, *a, **d) + self.assertEqual(x2, spam.spamlist) + self.assertEqual(a2, a1) + self.assertEqual(d2, d1) + class SubSpam(spam.spamlist): pass + x2, a2, d2 = spam_cm(SubSpam, *a, **d) + self.assertEqual(x2, SubSpam) + self.assertEqual(a2, a1) + self.assertEqual(d2, d1) + with self.assertRaises(TypeError): + spam_cm() + with self.assertRaises(TypeError): + spam_cm(spam.spamlist()) + with self.assertRaises(TypeError): + spam_cm(list) def test_staticmethods(self): # Testing static methods... diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,8 @@ Core and Builtins ----------------- +- Issue #14699: Fix calling the classmethod descriptor directly. + - Issue #11603 (again): Setting __repr__ to __str__ now raises a RuntimeError when repr() or str() is called on such an object. diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -254,14 +254,52 @@ classmethoddescr_call(PyMethodDescrObject *descr, PyObject *args, PyObject *kwds) { - PyObject *func, *result; + Py_ssize_t argc; + PyObject *self, *func, *result; - func = PyCFunction_New(descr->d_method, (PyObject *)descr->d_type); + /* Make sure that the first argument is acceptable as 'self' */ + assert(PyTuple_Check(args)); + argc = PyTuple_GET_SIZE(args); + if (argc < 1) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' of '%.100s' " + "object needs an argument", + descr_name((PyDescrObject *)descr), "?", + descr->d_type->tp_name); + return NULL; + } + self = PyTuple_GET_ITEM(args, 0); + if (!PyType_Check(self)) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' requires a type " + "but received a '%.100s'", + descr_name((PyDescrObject *)descr), "?", + descr->d_type->tp_name, + self->ob_type->tp_name); + return NULL; + } + if (!PyType_IsSubtype((PyTypeObject *)self, descr->d_type)) { + PyErr_Format(PyExc_TypeError, + "descriptor '%V' " + "requires a subtype of '%.100s' " + "but received '%.100s", + descr_name((PyDescrObject *)descr), "?", + descr->d_type->tp_name, + self->ob_type->tp_name); + return NULL; + } + + func = PyCFunction_New(descr->d_method, self); if (func == NULL) return NULL; - + args = PyTuple_GetSlice(args, 1, argc); + if (args == NULL) { + Py_DECREF(func); + return NULL; + } result = PyEval_CallObjectWithKeywords(func, args, kwds); Py_DECREF(func); + Py_DECREF(args); return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 16:16:38 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 16:16:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_use_formats_tha?= =?utf8?q?t_work_on_2=2E7?= Message-ID: http://hg.python.org/cpython/rev/6ff4440b5fa2 changeset: 76699:6ff4440b5fa2 branch: 2.7 user: Benjamin Peterson date: Tue May 01 10:16:33 2012 -0400 summary: use formats that work on 2.7 files: Objects/descrobject.c | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-) diff --git a/Objects/descrobject.c b/Objects/descrobject.c --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -262,28 +262,27 @@ argc = PyTuple_GET_SIZE(args); if (argc < 1) { PyErr_Format(PyExc_TypeError, - "descriptor '%V' of '%.100s' " + "descriptor '%s' of '%.100s' " "object needs an argument", - descr_name((PyDescrObject *)descr), "?", + descr_name((PyDescrObject *)descr), descr->d_type->tp_name); return NULL; } self = PyTuple_GET_ITEM(args, 0); if (!PyType_Check(self)) { PyErr_Format(PyExc_TypeError, - "descriptor '%V' requires a type " + "descriptor '%s' requires a type " "but received a '%.100s'", - descr_name((PyDescrObject *)descr), "?", - descr->d_type->tp_name, + descr_name((PyDescrObject *)descr), self->ob_type->tp_name); return NULL; } if (!PyType_IsSubtype((PyTypeObject *)self, descr->d_type)) { PyErr_Format(PyExc_TypeError, - "descriptor '%V' " + "descriptor '%s' " "requires a subtype of '%.100s' " "but received '%.100s", - descr_name((PyDescrObject *)descr), "?", + descr_name((PyDescrObject *)descr), descr->d_type->tp_name, self->ob_type->tp_name); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 16:55:40 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 16:55:40 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Improved_wording_on_regular_pa?= =?utf8?q?ckage_performance=2E?= Message-ID: http://hg.python.org/peps/rev/4f0334a2e532 changeset: 4340:4f0334a2e532 user: Eric V. Smith date: Tue May 01 10:55:21 2012 -0400 summary: Improved wording on regular package performance. files: pep-0420.txt | 13 +++++++------ 1 files changed, 7 insertions(+), 6 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -172,12 +172,13 @@ At PyCon 2012, we had a discussion about namespace packages at which PEP 382 and PEP 402 were rejected, to be replaced by this PEP [2]_. -There is no intention to remove support of regular packages. If there -is no intention that a package is a namespace package, then there is a -performance advantage to it being a regular package. Creation and -loading of the package can take place once it is located along the -path. With namespace packages, all entries in the path must be -scanned. +There is no intention to remove support of regular packages. If a +developer knows that her package will never be a portion of a +namespace package, then there is a performance advantage to it being a +regular package (with an ``__init__.py``). Creation and loading of a +regular package can take place immediately when it is located along +the path. With namespace packages, all entries in the path must be +scanned before the package is created. Note that an ImportWarning will no longer be raised for a directory lacking an ``__init__.py`` file. Such a directory will now be -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 17:14:37 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 17:14:37 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_don=27t_use_assertEqual_for?= =?utf8?q?_test_for_bool_equality?= Message-ID: http://hg.python.org/cpython/rev/bec267540811 changeset: 76700:bec267540811 parent: 76697:e1a200dfd5db user: Benjamin Peterson date: Tue May 01 11:14:32 2012 -0400 summary: don't use assertEqual for test for bool equality files: Lib/test/test_time.py | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -30,16 +30,16 @@ def test_time(self): time.time() info = time.get_clock_info('time') - self.assertEqual(info.monotonic, False) + self.assertFalse(info.monotonic) if sys.platform != 'win32': - self.assertEqual(info.adjusted, True) + self.assertTrue(info.adjusted) def test_clock(self): time.clock() info = time.get_clock_info('clock') - self.assertEqual(info.monotonic, True) - self.assertEqual(info.adjusted, False) + self.assertTrue(info.monotonic) + self.assertFalse(info.adjusted) @unittest.skipUnless(hasattr(time, 'clock_gettime'), 'need time.clock_gettime()') @@ -370,11 +370,11 @@ self.assertAlmostEqual(dt, 0.1, delta=0.2) info = time.get_clock_info('monotonic') - self.assertEqual(info.monotonic, True) + self.assertTrue(info.monotonic) if sys.platform == 'linux': - self.assertEqual(info.adjusted, True) + self.assertTrue(info.adjusted) else: - self.assertEqual(info.adjusted, False) + self.assertFalse(info.adjusted) def test_perf_counter(self): time.perf_counter() @@ -386,8 +386,8 @@ self.assertLess(stop - start, 0.01) info = time.get_clock_info('process_time') - self.assertEqual(info.monotonic, True) - self.assertEqual(info.adjusted, False) + self.assertTrue(info.monotonic) + self.assertFalse(info.adjusted) @unittest.skipUnless(hasattr(time, 'monotonic'), 'need time.monotonic') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 18:01:06 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 18:01:06 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_398_update=3A_deferring_so?= =?utf8?q?me_changes=2C_adding_new_changes_from_python-dev?= Message-ID: http://hg.python.org/peps/rev/0b9ffda7c4ac changeset: 4341:0b9ffda7c4ac user: Georg Brandl date: Tue May 01 18:00:57 2012 +0200 summary: PEP 398 update: deferring some changes, adding new changes from python-dev files: pep-0398.txt | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -82,10 +82,10 @@ Candidate PEPs: * PEP 362: Function Signature Object -* PEP 395: Qualified Names for Modules * PEP 397: Python launcher for Windows -* PEP 402: Simplified Package Layout (likely a new PEP derived from it) * PEP 405: Python Virtual Environments +* PEP 415: Implementing PEP 409 differently +* PEP 420: Implicit Namespace Packages * PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3144: IP Address manipulation library @@ -99,8 +99,13 @@ * Addition of the "regex" module * Email version 6 * A standard event-loop interface (PEP by Jim Fulton pending) -* Breaking out standard library and docs in separate repos? +* Switch of Windows build toolchain to VS 2010 or 2012 (depending on + availability and platform support) +Deferred to post-3.3: + +* PEP 395: Qualified Names for Modules +* Breaking out standard library and docs in separate repos Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 18:15:59 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 18:15:59 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Include_micro_version_even_?= =?utf8?q?if_it_is_0=2E?= Message-ID: http://hg.python.org/cpython/rev/5ba6124da155 changeset: 76701:5ba6124da155 parent: 76691:0d8cbfe8d88b user: Martin v. L?wis date: Tue May 01 16:27:55 2012 +0200 summary: Include micro version even if it is 0. files: Doc/tools/sphinxext/patchlevel.py | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Doc/tools/sphinxext/patchlevel.py b/Doc/tools/sphinxext/patchlevel.py --- a/Doc/tools/sphinxext/patchlevel.py +++ b/Doc/tools/sphinxext/patchlevel.py @@ -34,8 +34,7 @@ release = version = '%s.%s' % (d['PY_MAJOR_VERSION'], d['PY_MINOR_VERSION']) micro = int(d['PY_MICRO_VERSION']) - if micro != 0: - release += '.' + str(micro) + release += '.' + str(micro) level = d['PY_RELEASE_LEVEL'] suffixes = { @@ -51,8 +50,7 @@ def get_sys_version_info(): major, minor, micro, level, serial = sys.version_info release = version = '%s.%s' % (major, minor) - if micro: - release += '.%s' % micro + release += '.%s' % micro if level != 'final': release += '%s%s' % (level[0], serial) return version, release -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 18:15:59 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 18:15:59 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Include_micro_version_even_?= =?utf8?q?if_it_is_0=2E?= Message-ID: http://hg.python.org/cpython/rev/a8ac318da01b changeset: 76702:a8ac318da01b user: Martin v. L?wis date: Tue May 01 16:31:18 2012 +0200 summary: Include micro version even if it is 0. files: Lib/idlelib/EditorWindow.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -28,8 +28,7 @@ "Format sys.version_info to produce the Sphinx version string used to install the chm docs" major, minor, micro, level, serial = sys.version_info release = '%s%s' % (major, minor) - if micro: - release += '%s' % (micro,) + release += '%s' % (micro,) if level == 'candidate': release += 'rc%s' % (serial,) elif level != 'final': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 18:16:00 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 18:16:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Include_micro_version_even_?= =?utf8?q?if_it_is_0=2E?= Message-ID: http://hg.python.org/cpython/rev/8513a7fbd301 changeset: 76703:8513a7fbd301 user: Martin v. L?wis date: Tue May 01 16:37:44 2012 +0200 summary: Include micro version even if it is 0. files: Tools/msi/msi.py | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -122,9 +122,7 @@ } [major+minor] # Compute the name that Sphinx gives to the docfile -docfile = "" -if int(micro): - docfile = micro +docfile = micro if level < 0xf: if level == 0xC: docfile += "rc%s" % (serial,) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 18:16:01 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 01 May 2012 18:16:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge_sphinx_changes?= Message-ID: http://hg.python.org/cpython/rev/9ea07e65ed2a changeset: 76704:9ea07e65ed2a parent: 76700:bec267540811 parent: 76703:8513a7fbd301 user: Martin v. L?wis date: Tue May 01 18:15:13 2012 +0200 summary: Merge sphinx changes files: Doc/tools/sphinxext/patchlevel.py | 6 ++---- Lib/idlelib/EditorWindow.py | 3 +-- Tools/msi/msi.py | 4 +--- 3 files changed, 4 insertions(+), 9 deletions(-) diff --git a/Doc/tools/sphinxext/patchlevel.py b/Doc/tools/sphinxext/patchlevel.py --- a/Doc/tools/sphinxext/patchlevel.py +++ b/Doc/tools/sphinxext/patchlevel.py @@ -34,8 +34,7 @@ release = version = '%s.%s' % (d['PY_MAJOR_VERSION'], d['PY_MINOR_VERSION']) micro = int(d['PY_MICRO_VERSION']) - if micro != 0: - release += '.' + str(micro) + release += '.' + str(micro) level = d['PY_RELEASE_LEVEL'] suffixes = { @@ -51,8 +50,7 @@ def get_sys_version_info(): major, minor, micro, level, serial = sys.version_info release = version = '%s.%s' % (major, minor) - if micro: - release += '.%s' % micro + release += '.%s' % micro if level != 'final': release += '%s%s' % (level[0], serial) return version, release diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -28,8 +28,7 @@ "Format sys.version_info to produce the Sphinx version string used to install the chm docs" major, minor, micro, level, serial = sys.version_info release = '%s%s' % (major, minor) - if micro: - release += '%s' % (micro,) + release += '%s' % (micro,) if level == 'candidate': release += 'rc%s' % (serial,) elif level != 'final': diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -122,9 +122,7 @@ } [major+minor] # Compute the name that Sphinx gives to the docfile -docfile = "" -if int(micro): - docfile = micro +docfile = micro if level < 0xf: if level == 0xC: docfile += "rc%s" % (serial,) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 1 18:17:58 2012 From: python-checkins at python.org (georg.brandl) Date: Tue, 01 May 2012 18:17:58 +0200 Subject: [Python-checkins] =?utf8?q?release=3A_Remove_new_=2Ehgtouch_file?= =?utf8?q?=2C_and_also_=2Ehgtags_from_archive?= Message-ID: http://hg.python.org/release/rev/d8b32a40ce66 changeset: 60:d8b32a40ce66 user: Georg Brandl date: Tue May 01 18:13:38 2012 +0200 summary: Remove new .hgtouch file, and also .hgtags from archive files: release.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/release.py b/release.py --- a/release.py +++ b/release.py @@ -263,8 +263,9 @@ archivename = 'Python-%s' % tag.text run_cmd(['hg', 'archive', '-r', tag.hgname, archivename]) with changed_dir(archivename): - print('Removing VCS .*ignore and .hgeol') - for name in ('.hgignore', '.hgeol', '.bzrignore', '.gitignore'): + print('Removing VCS .*ignore and .hg*') + for name in ('.hgignore', '.hgeol', '.hgtags', '.hgtouch', + '.bzrignore', '.gitignore'): try: os.unlink(name) except OSError: -- Repository URL: http://hg.python.org/release From python-checkins at python.org Tue May 1 18:38:09 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 18:38:09 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Record_response_=28by_Nick=29_?= =?utf8?q?to_Nick=27s_initial_objections=2E_I_hope_I=27ve_summarized?= Message-ID: http://hg.python.org/peps/rev/be131c928b5c changeset: 4342:be131c928b5c user: Eric V. Smith date: Tue May 01 12:37:58 2012 -0400 summary: Record response (by Nick) to Nick's initial objections. I hope I've summarized correctly. files: pep-0420.txt | 32 ++++++++++++++++++++++---------- 1 files changed, 22 insertions(+), 10 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -188,21 +188,30 @@ Nick Coglan presented a list of his objections to this proposal [3]_. They are: - * Implicit package directories go against the Zen of Python + 1. Implicit package directories go against the Zen of Python. - * Implicit package directories pose awkward backwards compatibility - challenges + 2. Implicit package directories pose awkward backwards compatibility + challenges. - * Implicit package directories introduce ambiguity into filesystem - layouts + 3. Implicit package directories introduce ambiguity into filesystem + layouts. - * Implicit package directories will permanently entrench current - newbie-hostile behaviour in __main__ + 4. Implicit package directories will permanently entrench current + newbie-hostile behaviour in ``__main__``. -(These need to be addressed here.) +Nick gave a detailed response [4]_, which is summarized here: + + 1. Practicality beats purity. + + 2. Minor backward compatibility issues are okay, as long as they are + properly documented. + + 3. This will be addressed in PEP 395. + + 4. This will also be addressed in PEP 395. Phillip Eby asked about auto-updating of ``__path__``, instead of it -being a simple list [4]_. It is the intent of this PEP to get the +being a simple list [5]_. It is the intent of this PEP to get the simplest possible solution working. It will be possible at a later date to add such features. Several possible ways to do so were discussed in the referenced email thread. @@ -248,7 +257,10 @@ .. [3] Nick Coglan's objection to the lack of marker files or directories (http://mail.python.org/pipermail/import-sig/2012-March/000423.html) -.. [4] Phillip Eby's question about auto-updating __path__ +.. [4] Nick Coglan's response to his initial objections + (http://mail.python.org/pipermail/import-sig/2012-April/000464.html) + +.. [5] Phillip Eby's question about auto-updating __path__ (http://mail.python.org/pipermail/import-sig/2012-April/000468.html) Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 18:41:43 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 18:41:43 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Improvement_in_wording=2E?= Message-ID: http://hg.python.org/peps/rev/490f4ad443cd changeset: 4343:490f4ad443cd user: Eric V. Smith date: Tue May 01 12:41:37 2012 -0400 summary: Improvement in wording. files: pep-0420.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -201,7 +201,8 @@ Nick gave a detailed response [4]_, which is summarized here: - 1. Practicality beats purity. + 1. The practicality of this PEP wins over other proposals and the + status quo. 2. Minor backward compatibility issues are okay, as long as they are properly documented. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 18:45:39 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 01 May 2012 18:45:39 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Moved_sections_around_so_that_?= =?utf8?q?the_discussion_comes_after_all_relevant_details?= Message-ID: http://hg.python.org/peps/rev/d06e6520e92e changeset: 4344:d06e6520e92e user: Eric V. Smith date: Tue May 01 12:45:32 2012 -0400 summary: Moved sections around so that the discussion comes after all relevant details are presented. files: pep-0420.txt | 60 ++++++++++++++++++++-------------------- 1 files changed, 30 insertions(+), 30 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -166,6 +166,36 @@ from ``find_module``, the only impact is that such a loader will be unable to provide portions of a namespace package. +Packaging Implications +====================== + +Multiple portions of a namespace package can be installed into the +same directory, or into separate directories. For this section, +suppose there are two portions which define "foo.bar" and "foo.baz". +"foo" itself is a namespace package. + +If these are installed in the same location, a single directory "foo" +would be in a directory that is on ``sys.path``. Inside "foo" would +be two directories, "bar" and "baz". If "foo.bar" is removed (perhaps +by an OS package manager), care must be taken not to remove the +"foo/baz" or "foo" directories. Note that in this case "foo" will be +a namespace package (because it lacks an ``__init__.py``), even though +all of its portions are in the same directory. + +Note that "foo.bar" and "foo.baz" can be installed into the same "foo" +directory because they will not have any files in common. + +If the portions are installed in different locations, two different +"foo" directories would be in directories that are on ``sys.path``. +"foo/bar" would be in one of these sys.path entries, and "foo/baz" +would be in the other. Upon removal of "foo.bar", the "foo/bar" and +corresonding "foo" directories can be completely removed. But +"foo/baz" and its corresponding "foo" directory cannot be removed. + +It is also possible to have the "foo.bar" portion installed in a +directory on ``sys.path``, and have the "foo.baz" portion provided in +a zip file, also on ``sys.path``. + Discussion ========== @@ -217,36 +247,6 @@ date to add such features. Several possible ways to do so were discussed in the referenced email thread. -Packaging Implications -====================== - -Multiple portions of a namespace package can be installed into the -same directory, or into separate directories. For this section, -suppose there are two portions which define "foo.bar" and "foo.baz". -"foo" itself is a namespace package. - -If these are installed in the same location, a single directory "foo" -would be in a directory that is on ``sys.path``. Inside "foo" would -be two directories, "bar" and "baz". If "foo.bar" is removed (perhaps -by an OS package manager), care must be taken not to remove the -"foo/baz" or "foo" directories. Note that in this case "foo" will be -a namespace package (because it lacks an ``__init__.py``), even though -all of its portions are in the same directory. - -Note that "foo.bar" and "foo.baz" can be installed into the same "foo" -directory because they will not have any files in common. - -If the portions are installed in different locations, two different -"foo" directories would be in directories that are on ``sys.path``. -"foo/bar" would be in one of these sys.path entries, and "foo/baz" -would be in the other. Upon removal of "foo.bar", the "foo/bar" and -corresonding "foo" directories can be completely removed. But -"foo/baz" and its corresponding "foo" directory cannot be removed. - -It is also possible to have the "foo.bar" portion installed in a -directory on ``sys.path``, and have the "foo.baz" portion provided in -a zip file, also on ``sys.path``. - References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 1 19:44:28 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 01 May 2012 19:44:28 +0200 Subject: [Python-checkins] =?utf8?q?release=3A_don=27t_produce_archive_met?= =?utf8?q?a_data?= Message-ID: http://hg.python.org/release/rev/2a005c3e8fb9 changeset: 61:2a005c3e8fb9 user: Benjamin Peterson date: Tue May 01 13:44:24 2012 -0400 summary: don't produce archive meta data files: release.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/release.py b/release.py --- a/release.py +++ b/release.py @@ -261,7 +261,8 @@ with changed_dir(tag.text): print('Exporting tag:', tag.text) archivename = 'Python-%s' % tag.text - run_cmd(['hg', 'archive', '-r', tag.hgname, archivename]) + run_cmd(['hg', 'archive', '--config', 'ui.archivemeta=off', + '-r', tag.hgname, archivename]) with changed_dir(archivename): print('Removing VCS .*ignore and .hg*') for name in ('.hgignore', '.hgeol', '.hgtags', '.hgtouch', -- Repository URL: http://hg.python.org/release From python-checkins at python.org Wed May 2 01:19:09 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 02 May 2012 01:19:09 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314687=3A_Optimize_?= =?utf8?q?str=25args?= Message-ID: http://hg.python.org/cpython/rev/4b98ce6ef95e changeset: 76705:4b98ce6ef95e user: Victor Stinner date: Wed May 02 00:29:36 2012 +0200 summary: Issue #14687: Optimize str%args * formatfloat() uses unicode_fromascii() instead of PyUnicode_DecodeASCII() to not have to check characters, we know that it is really ASCII * Use PyUnicode_FromOrdinal() instead of _PyUnicode_FromUCS4() to format a character: if avoids a call to ucs4lib_find_max_char() to compute the maximum character (whereas we already know it, it is just the character itself) files: Objects/unicodeobject.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13431,7 +13431,7 @@ (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL); if (p == NULL) return NULL; - result = PyUnicode_DecodeASCII(p, strlen(p), NULL); + result = unicode_fromascii((unsigned char*)p, strlen(p)); PyMem_Free(p); return result; } @@ -13947,7 +13947,7 @@ Py_UCS4 ch = formatchar(v); if (ch == (Py_UCS4) -1) goto onError; - temp = _PyUnicode_FromUCS4(&ch, 1); + temp = PyUnicode_FromOrdinal(ch); break; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 01:19:10 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 02 May 2012 01:19:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314687=3A_Cleanup_P?= =?utf8?q?yUnicode=5FFormat=28=29?= Message-ID: http://hg.python.org/cpython/rev/a966f9311ebb changeset: 76706:a966f9311ebb user: Victor Stinner date: Wed May 02 00:41:57 2012 +0200 summary: Issue #14687: Cleanup PyUnicode_Format() files: Objects/unicodeobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13719,7 +13719,8 @@ PyObject *signobj = NULL, *fillobj = NULL; fmtpos++; - if (PyUnicode_READ(fmtkind, fmt, fmtpos) == '(') { + c = PyUnicode_READ(fmtkind, fmt, fmtpos); + if (c == '(') { Py_ssize_t keystart; Py_ssize_t keylen; PyObject *key; @@ -13765,7 +13766,8 @@ argidx = -2; } while (--fmtcnt >= 0) { - switch (c = PyUnicode_READ(fmtkind, fmt, fmtpos++)) { + c = PyUnicode_READ(fmtkind, fmt, fmtpos++); + switch (c) { case '-': flags |= F_LJUST; continue; case '+': flags |= F_SIGN; continue; case ' ': flags |= F_BLANK; continue; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 01:19:11 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 02 May 2012 01:19:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_unicodeobject=2Ec=3A_Add_MA?= =?utf8?q?X=5FMAXCHAR=28=29_macro_to_=28micro-=29optimize_the_computation?= Message-ID: http://hg.python.org/cpython/rev/359b1a6a3836 changeset: 76707:359b1a6a3836 user: Victor Stinner date: Wed May 02 01:15:40 2012 +0200 summary: unicodeobject.c: Add MAX_MAXCHAR() macro to (micro-)optimize the computation of the second argument of PyUnicode_New(). * Create also align_maxchar() function * Optimize fix_decimal_and_space_to_ascii(): don't compute the maximum character when ch <= 127 (it is ASCII) files: Objects/unicodeobject.c | 99 ++++++++++++++-------------- 1 files changed, 50 insertions(+), 49 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -111,6 +111,11 @@ #define _PyUnicode_DATA_ANY(op) \ (((PyUnicodeObject*)(op))->data.any) +/* Optimized version of Py_MAX() to compute the maximum character: + use it when your are computing the second argument of PyUnicode_New() */ +#define MAX_MAXCHAR(maxchar1, maxchar2) \ + ((maxchar1) | (maxchar2)) + #undef PyUnicode_READY #define PyUnicode_READY(op) \ (assert(_PyUnicode_CHECK(op)), \ @@ -1867,6 +1872,19 @@ } } +Py_LOCAL_INLINE(Py_UCS4) +align_maxchar(Py_UCS4 maxchar) +{ + if (maxchar <= 127) + return 127; + else if (maxchar <= 255) + return 255; + else if (maxchar <= 65535) + return 65535; + else + return MAX_UNICODE; +} + static PyObject* _PyUnicode_FromUCS1(const unsigned char* u, Py_ssize_t size) { @@ -2439,7 +2457,7 @@ case 'c': { Py_UCS4 ordinal = va_arg(count, int); - maxchar = Py_MAX(maxchar, ordinal); + maxchar = MAX_MAXCHAR(maxchar, ordinal); n++; break; } @@ -2535,7 +2553,7 @@ /* since PyUnicode_DecodeUTF8 returns already flexible unicode objects, there is no need to call ready on them */ argmaxchar = PyUnicode_MAX_CHAR_VALUE(str); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(str); /* Remember the str and switch to the next slot */ *callresult++ = str; @@ -2548,7 +2566,7 @@ if (PyUnicode_READY(obj) == -1) goto fail; argmaxchar = PyUnicode_MAX_CHAR_VALUE(obj); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(obj); break; } @@ -2563,7 +2581,7 @@ if (PyUnicode_READY(obj) == -1) goto fail; argmaxchar = PyUnicode_MAX_CHAR_VALUE(obj); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(obj); *callresult++ = NULL; } @@ -2576,7 +2594,7 @@ goto fail; } argmaxchar = PyUnicode_MAX_CHAR_VALUE(str_obj); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(str_obj); *callresult++ = str_obj; } @@ -2595,7 +2613,7 @@ goto fail; } argmaxchar = PyUnicode_MAX_CHAR_VALUE(str); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(str); /* Remember the str and switch to the next slot */ *callresult++ = str; @@ -2614,7 +2632,7 @@ goto fail; } argmaxchar = PyUnicode_MAX_CHAR_VALUE(repr); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(repr); /* Remember the repr and switch to the next slot */ *callresult++ = repr; @@ -2633,7 +2651,7 @@ goto fail; } argmaxchar = PyUnicode_MAX_CHAR_VALUE(ascii); - maxchar = Py_MAX(maxchar, argmaxchar); + maxchar = MAX_MAXCHAR(maxchar, argmaxchar); n += PyUnicode_GET_LENGTH(ascii); /* Remember the repr and switch to the next slot */ *callresult++ = ascii; @@ -5563,14 +5581,14 @@ maxch = (Py_UCS2)(block & 0xFFFF); #if SIZEOF_LONG == 8 ch = (Py_UCS2)((block >> 16) & 0xFFFF); - maxch = Py_MAX(maxch, ch); + maxch = MAX_MAXCHAR(maxch, ch); ch = (Py_UCS2)((block >> 32) & 0xFFFF); - maxch = Py_MAX(maxch, ch); + maxch = MAX_MAXCHAR(maxch, ch); ch = (Py_UCS2)(block >> 48); - maxch = Py_MAX(maxch, ch); + maxch = MAX_MAXCHAR(maxch, ch); #else ch = (Py_UCS2)(block >> 16); - maxch = Py_MAX(maxch, ch); + maxch = MAX_MAXCHAR(maxch, ch); #endif if (maxch > PyUnicode_MAX_CHAR_VALUE(unicode)) { if (unicode_widen(&unicode, maxch) < 0) @@ -8987,7 +9005,7 @@ const Py_ssize_t len = PyUnicode_GET_LENGTH(self); const int kind = PyUnicode_KIND(self); void *data = PyUnicode_DATA(self); - Py_UCS4 maxchar = 0, ch, fixed; + Py_UCS4 maxchar = 127, ch, fixed; int modified = 0; Py_ssize_t i; @@ -9004,15 +9022,12 @@ } if (fixed != 0) { modified = 1; - if (fixed > maxchar) - maxchar = fixed; + maxchar = MAX_MAXCHAR(maxchar, fixed); PyUnicode_WRITE(kind, data, i, fixed); } - else if (ch > maxchar) - maxchar = ch; - } - else if (ch > maxchar) - maxchar = ch; + else + maxchar = MAX_MAXCHAR(maxchar, ch); + } } return (modified) ? maxchar : 0; @@ -9052,7 +9067,7 @@ int decimal = Py_UNICODE_TODECIMAL(ch); if (decimal >= 0) ch = '0' + decimal; - maxchar = Py_MAX(maxchar, ch); + maxchar = MAX_MAXCHAR(maxchar, ch); } } @@ -9293,8 +9308,8 @@ if (unicode == NULL) { *maxchar = 127; if (len != n_digits) { - *maxchar = Py_MAX(*maxchar, - PyUnicode_MAX_CHAR_VALUE(thousands_sep)); + *maxchar = MAX_MAXCHAR(*maxchar, + PyUnicode_MAX_CHAR_VALUE(thousands_sep)); } } return len; @@ -9591,14 +9606,7 @@ return u; } - if (maxchar_new <= 127) - maxchar_new = 127; - else if (maxchar_new <= 255) - maxchar_new = 255; - else if (maxchar_new <= 65535) - maxchar_new = 65535; - else - maxchar_new = MAX_UNICODE; + maxchar_new = align_maxchar(maxchar_new); if (maxchar_new == maxchar_old) return u; @@ -9695,16 +9703,14 @@ c = PyUnicode_READ(kind, data, 0); n_res = _PyUnicode_ToUpperFull(c, mapped); for (j = 0; j < n_res; j++) { - if (mapped[j] > *maxchar) - *maxchar = mapped[j]; + *maxchar = MAX_MAXCHAR(*maxchar, mapped[j]); res[k++] = mapped[j]; } for (i = 1; i < length; i++) { c = PyUnicode_READ(kind, data, i); n_res = lower_ucs4(kind, data, length, i, c, mapped); for (j = 0; j < n_res; j++) { - if (mapped[j] > *maxchar) - *maxchar = mapped[j]; + *maxchar = MAX_MAXCHAR(*maxchar, mapped[j]); res[k++] = mapped[j]; } } @@ -9729,8 +9735,7 @@ mapped[0] = c; } for (j = 0; j < n_res; j++) { - if (mapped[j] > *maxchar) - *maxchar = mapped[j]; + *maxchar = MAX_MAXCHAR(*maxchar, mapped[j]); res[k++] = mapped[j]; } } @@ -9751,8 +9756,7 @@ else n_res = _PyUnicode_ToUpperFull(c, mapped); for (j = 0; j < n_res; j++) { - if (mapped[j] > *maxchar) - *maxchar = mapped[j]; + *maxchar = MAX_MAXCHAR(*maxchar, mapped[j]); res[k++] = mapped[j]; } } @@ -9781,8 +9785,7 @@ Py_UCS4 mapped[3]; int j, n_res = _PyUnicode_ToFoldedFull(c, mapped); for (j = 0; j < n_res; j++) { - if (mapped[j] > *maxchar) - *maxchar = mapped[j]; + *maxchar = MAX_MAXCHAR(*maxchar, mapped[j]); res[k++] = mapped[j]; } } @@ -9807,8 +9810,7 @@ n_res = _PyUnicode_ToTitleFull(c, mapped); for (j = 0; j < n_res; j++) { - if (mapped[j] > *maxchar) - *maxchar = mapped[j]; + *maxchar = MAX_MAXCHAR(*maxchar, mapped[j]); res[k++] = mapped[j]; } @@ -9965,7 +9967,7 @@ goto onError; sz += PyUnicode_GET_LENGTH(item); item_maxchar = PyUnicode_MAX_CHAR_VALUE(item); - maxchar = Py_MAX(maxchar, item_maxchar); + maxchar = MAX_MAXCHAR(maxchar, item_maxchar); if (i != 0) sz += seplen; if (sz < old_sz || sz > PY_SSIZE_T_MAX) { @@ -10127,8 +10129,7 @@ return NULL; } maxchar = PyUnicode_MAX_CHAR_VALUE(self); - if (fill > maxchar) - maxchar = fill; + maxchar = MAX_MAXCHAR(maxchar, fill); u = PyUnicode_New(left + _PyUnicode_LENGTH(self) + right, maxchar); if (!u) return NULL; @@ -10442,7 +10443,7 @@ /* Replacing str1 with str2 may cause a maxchar reduction in the result string. */ mayshrink = (maxchar_str2 < maxchar); - maxchar = Py_MAX(maxchar, maxchar_str2); + maxchar = MAX_MAXCHAR(maxchar, maxchar_str2); if (len1 == len2) { /* same length */ @@ -11027,7 +11028,7 @@ maxchar = PyUnicode_MAX_CHAR_VALUE(u); maxchar2 = PyUnicode_MAX_CHAR_VALUE(v); - maxchar = Py_MAX(maxchar, maxchar2); + maxchar = MAX_MAXCHAR(maxchar, maxchar2); /* Concat the two Unicode strings */ w = PyUnicode_New(new_len, maxchar); @@ -11114,7 +11115,7 @@ else { maxchar = PyUnicode_MAX_CHAR_VALUE(left); maxchar2 = PyUnicode_MAX_CHAR_VALUE(right); - maxchar = Py_MAX(maxchar, maxchar2); + maxchar = MAX_MAXCHAR(maxchar, maxchar2); /* Concat the two Unicode strings */ res = PyUnicode_New(new_len, maxchar); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 02:01:15 2012 From: python-checkins at python.org (senthil.kumaran) Date: Wed, 02 May 2012 02:01:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_fix_windows_tes?= =?utf8?q?t_failure_-_issue13183?= Message-ID: http://hg.python.org/cpython/rev/3b2aa777b725 changeset: 76708:3b2aa777b725 branch: 2.7 parent: 76699:6ff4440b5fa2 user: Senthil Kumaran date: Wed May 02 07:59:36 2012 +0800 summary: fix windows test failure - issue13183 files: Lib/test/test_pdb.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -57,7 +57,7 @@ """ bar = """ def bar(): - print('1') + pass """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 02:01:16 2012 From: python-checkins at python.org (senthil.kumaran) Date: Wed, 02 May 2012 02:01:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fix_windows_tes?= =?utf8?q?t_failure_-_issue13183?= Message-ID: http://hg.python.org/cpython/rev/d17ecee3f752 changeset: 76709:d17ecee3f752 branch: 3.2 parent: 76696:eab5120cc208 user: Senthil Kumaran date: Wed May 02 08:00:22 2012 +0800 summary: fix windows test failure - issue13183 files: Lib/test/test_pdb.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -653,7 +653,7 @@ """ bar = """ def bar(): - print('1') + pass """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 02:01:19 2012 From: python-checkins at python.org (senthil.kumaran) Date: Wed, 02 May 2012 02:01:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_fix_closes_issue13183_-_windows_test_failure?= Message-ID: http://hg.python.org/cpython/rev/0269c592c8b1 changeset: 76710:0269c592c8b1 parent: 76707:359b1a6a3836 parent: 76709:d17ecee3f752 user: Senthil Kumaran date: Wed May 02 08:01:02 2012 +0800 summary: fix closes issue13183 - windows test failure files: Lib/test/test_pdb.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -658,7 +658,7 @@ """ bar = """ def bar(): - print('1') + pass """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed May 2 05:39:34 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 02 May 2012 05:39:34 +0200 Subject: [Python-checkins] Daily reference leaks (0269c592c8b1): sum=2 Message-ID: results for 0269c592c8b1 on branch "default" -------------------------------------------- test_dbm leaked [2, 0, 0] references, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog9a2olP', '-x'] From python-checkins at python.org Wed May 2 07:41:29 2012 From: python-checkins at python.org (martin.v.loewis) Date: Wed, 02 May 2012 07:41:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313183=3A_Revert_0b?= =?utf8?q?53b70a40a0_=28reenable_test_on_windows=29?= Message-ID: http://hg.python.org/cpython/rev/6c9ce7e34511 changeset: 76711:6c9ce7e34511 user: Martin v. L?wis date: Wed May 02 07:41:22 2012 +0200 summary: Issue #13183: Revert 0b53b70a40a0 (reenable test on windows) files: Lib/test/test_pdb.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -1,5 +1,5 @@ # A test suite for pdb; not very comprehensive at the moment. -import os + import imp import pdb import sys @@ -631,7 +631,6 @@ self.assertNotIn(b'SyntaxError', stdout, "Got a syntax error running test script under PDB") - @unittest.skipIf(os.name == 'nt', "temporarily disabled on Windows") def test_issue13183(self): script = """ from bar import bar -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 14:00:33 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 02 May 2012 14:00:33 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_418=3A_Update_time=2Eget?= =?utf8?q?=5Fclock=5Finfo=28=29_according_to_the_implementation?= Message-ID: http://hg.python.org/peps/rev/4f33e5ec6145 changeset: 4345:4f33e5ec6145 user: Victor Stinner date: Wed May 02 14:00:30 2012 +0200 summary: PEP 418: Update time.get_clock_info() according to the implementation files: pep-0418.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0418.txt b/pep-0418.txt --- a/pep-0418.txt +++ b/pep-0418.txt @@ -86,7 +86,7 @@ * time.monotonic() and time.perf_counter() may or may not be adjusted. For example, ``CLOCK_MONOTONIC`` is slewed on Linux, whereas ``GetTickCount()`` is not adjusted on Windows. - ``time.get_clock_info('monotonic')['is_adjusted']`` can be used to check + ``time.get_clock_info('monotonic')['adjusted']`` can be used to check if the monotonic clock is adjusted or not. * No time.thread_time() function is proposed by this PEP because it is not needed by Python standard library nor a common asked feature. @@ -119,8 +119,8 @@ * ``implementation`` (str): name of the underlying operating system function. Examples: ``"QueryPerformanceCounter()"``, ``"clock_gettime(CLOCK_REALTIME)"``. - * ``is_monotonic`` (bool): True if the clock cannot go backward. - * ``is_adjusted`` (bool): True if the clock is adjusted (e.g. by a + * ``monotonic`` (bool): True if the clock cannot go backward. + * ``adjusted`` (bool): True if the clock can be adjusted (e.g. by a NTP daemon). * ``resolution`` (float): resolution in seconds of the clock. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 2 17:39:17 2012 From: python-checkins at python.org (richard.oudkerk) Date: Wed, 02 May 2012 17:39:17 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzk0MDA6?= =?utf8?q?_Partial_backport_of_fix_for_=239244?= Message-ID: http://hg.python.org/cpython/rev/26bbff4562a7 changeset: 76712:26bbff4562a7 branch: 2.7 parent: 76708:3b2aa777b725 user: Richard Oudkerk date: Wed May 02 16:36:26 2012 +0100 summary: Issue #9400: Partial backport of fix for #9244 In multiprocessing, a pool worker process would die if the result/error could not be pickled. This could cause pool methods to hang. In 3.x this was fixed by 0aa8af79359d (which also added an error_callback argument to some methods), but the fix was not back ported. files: Lib/multiprocessing/pool.py | 25 +++++++++++++++++++- Lib/test/test_multiprocessing.py | 18 ++++++++++++++ 2 files changed, 42 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -68,6 +68,23 @@ # Code run by worker processes # +class MaybeEncodingError(Exception): + """Wraps possible unpickleable errors, so they can be + safely sent through the socket.""" + + def __init__(self, exc, value): + self.exc = repr(exc) + self.value = repr(value) + super(MaybeEncodingError, self).__init__(self.exc, self.value) + + def __str__(self): + return "Error sending result: '%s'. Reason: '%s'" % (self.value, + self.exc) + + def __repr__(self): + return "" % str(self) + + def worker(inqueue, outqueue, initializer=None, initargs=(), maxtasks=None): assert maxtasks is None or (type(maxtasks) == int and maxtasks > 0) put = outqueue.put @@ -96,7 +113,13 @@ result = (True, func(*args, **kwds)) except Exception, e: result = (False, e) - put((job, i, result)) + try: + put((job, i, result)) + except Exception as e: + wrapped = MaybeEncodingError(e, result[1]) + debug("Possible encoding error while sending result: %s" % ( + wrapped)) + put((job, i, (False, wrapped))) completed += 1 debug('worker exiting after %d tasks' % completed) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -1152,6 +1152,24 @@ join() self.assertTrue(join.elapsed < 0.2) +def unpickleable_result(): + return lambda: 42 + +class _TestPoolWorkerErrors(BaseTestCase): + ALLOWED_TYPES = ('processes', ) + + def test_unpickleable_result(self): + from multiprocessing.pool import MaybeEncodingError + p = multiprocessing.Pool(2) + + # Make sure we don't lose pool processes because of encoding errors. + for iteration in range(20): + res = p.apply_async(unpickleable_result) + self.assertRaises(MaybeEncodingError, res.get) + + p.close() + p.join() + class _TestPoolWorkerLifetime(BaseTestCase): ALLOWED_TYPES = ('processes', ) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 17:57:35 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 02 May 2012 17:57:35 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Clarify_path_scanning_logic=2E?= Message-ID: http://hg.python.org/peps/rev/95cc28d462fd changeset: 4346:95cc28d462fd parent: 4344:d06e6520e92e user: Eric V. Smith date: Wed May 02 11:16:48 2012 -0400 summary: Clarify path scanning logic. files: pep-0420.txt | 27 +++++++++++++++++++-------- 1 files changed, 19 insertions(+), 8 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -114,19 +114,30 @@ for specifing a namespace package. During import processing, the import machinery will continue to -iterate over the parent path as it does in Python 3.2. While looking -for a module or package named "foo": +iterate over each directory in the parent path as it does in Python +3.2. While looking for a module or package named "foo", for each +directory in the parent path: - * If ``foo/__init__.py`` is found, a regular package is imported. - * If not, but ``foo.{py,pyc,so,pyd}`` is found, a module is imported. - * If not, but ``foo`` is found and is a directory, it is recorded. + * If ``/foo/__init__.py`` is found, a regular package is + imported and returned. -If the scan along the parent path completes without finding a module -or package and at least one directory was recorded, then a namespace -package is created. The new namespace package: + * If not, but ``/foo.{py,pyc,so,pyd}`` is found, a module + is imported and returned. + + * If not, but ``/foo`` is found and is a directory, it is + recorded and the scan continues with the next directory in the + parent path. + + * Otherwise the scan continues with the next directory in the parent + path. + +If the scan completes without returning a module or package, and at +least one directory was recorded, then a namespace package is created. +The new namespace package: * Has a ``__file__`` attribute set to the first directory that was found during the scan, including the trailing path separator. + * Has a ``__path__`` attribute set to the list of directories there were found and recorded during the scan. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 2 17:57:36 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 02 May 2012 17:57:36 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merge=2E?= Message-ID: http://hg.python.org/peps/rev/6edfeda4d49f changeset: 4347:6edfeda4d49f parent: 4346:95cc28d462fd parent: 4345:4f33e5ec6145 user: Eric V. Smith date: Wed May 02 11:57:17 2012 -0400 summary: Merge. files: pep-0418.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0418.txt b/pep-0418.txt --- a/pep-0418.txt +++ b/pep-0418.txt @@ -86,7 +86,7 @@ * time.monotonic() and time.perf_counter() may or may not be adjusted. For example, ``CLOCK_MONOTONIC`` is slewed on Linux, whereas ``GetTickCount()`` is not adjusted on Windows. - ``time.get_clock_info('monotonic')['is_adjusted']`` can be used to check + ``time.get_clock_info('monotonic')['adjusted']`` can be used to check if the monotonic clock is adjusted or not. * No time.thread_time() function is proposed by this PEP because it is not needed by Python standard library nor a common asked feature. @@ -119,8 +119,8 @@ * ``implementation`` (str): name of the underlying operating system function. Examples: ``"QueryPerformanceCounter()"``, ``"clock_gettime(CLOCK_REALTIME)"``. - * ``is_monotonic`` (bool): True if the clock cannot go backward. - * ``is_adjusted`` (bool): True if the clock is adjusted (e.g. by a + * ``monotonic`` (bool): True if the clock cannot go backward. + * ``adjusted`` (bool): True if the clock can be adjusted (e.g. by a NTP daemon). * ``resolution`` (float): resolution in seconds of the clock. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 2 18:56:14 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 02 May 2012 18:56:14 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Typo=2E_Thanks_Terry_Reedy=2E?= Message-ID: http://hg.python.org/peps/rev/1bc475324033 changeset: 4348:1bc475324033 user: Eric V. Smith date: Wed May 02 12:56:07 2012 -0400 summary: Typo. Thanks Terry Reedy. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -138,7 +138,7 @@ * Has a ``__file__`` attribute set to the first directory that was found during the scan, including the trailing path separator. - * Has a ``__path__`` attribute set to the list of directories there + * Has a ``__path__`` attribute set to the list of directories that were found and recorded during the scan. There is no mechanism to automatically recompute the ``__path__`` if -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 2 20:05:27 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:05:27 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0Njk4?= =?utf8?q?=3A_Make_test=5Fposix_more_robust_when_the_current_UID_doesn=27t?= =?utf8?q?_have_an?= Message-ID: http://hg.python.org/cpython/rev/45f0272f5296 changeset: 76713:45f0272f5296 branch: 2.7 user: Charles-Fran?ois Natali date: Wed May 02 20:00:37 2012 +0200 summary: Issue #14698: Make test_posix more robust when the current UID doesn't have an associated pwd entry. files: Lib/test/test_posix.py | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -108,7 +108,11 @@ # If a non-privileged user invokes it, it should fail with OSError # EPERM. if os.getuid() != 0: - name = pwd.getpwuid(posix.getuid()).pw_name + try: + name = pwd.getpwuid(posix.getuid()).pw_name + except KeyError: + # the current UID may not have a pwd entry + raise unittest.SkipTest("need a pwd entry") try: posix.initgroups(name, 13) except OSError as e: @@ -418,8 +422,9 @@ def test_getgroups(self): with os.popen('id -G') as idg: groups = idg.read().strip() + ret = idg.close() - if not groups: + if ret != 0 or not groups: raise unittest.SkipTest("need working 'id -G'") # 'id -G' and 'os.getgroups()' should return the same -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:05:29 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:05:29 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0Njk4?= =?utf8?q?=3A_Make_test=5Fposix_more_robust_when_the_current_UID_doesn=27t?= =?utf8?q?_have_an?= Message-ID: http://hg.python.org/cpython/rev/93424b084d08 changeset: 76714:93424b084d08 branch: 3.2 parent: 76709:d17ecee3f752 user: Charles-Fran?ois Natali date: Wed May 02 20:01:38 2012 +0200 summary: Issue #14698: Make test_posix more robust when the current UID doesn't have an associated pwd entry. files: Lib/test/test_posix.py | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -107,7 +107,11 @@ # If a non-privileged user invokes it, it should fail with OSError # EPERM. if os.getuid() != 0: - name = pwd.getpwuid(posix.getuid()).pw_name + try: + name = pwd.getpwuid(posix.getuid()).pw_name + except KeyError: + # the current UID may not have a pwd entry + raise unittest.SkipTest("need a pwd entry") try: posix.initgroups(name, 13) except OSError as e: @@ -421,8 +425,9 @@ def test_getgroups(self): with os.popen('id -G') as idg: groups = idg.read().strip() + ret = idg.close() - if not groups: + if ret != 0 or not groups: raise unittest.SkipTest("need working 'id -G'") # 'id -G' and 'os.getgroups()' should return the same -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:05:29 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:05:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314698=3A_Make_test=5Fposix_more_robust_when_the_cur?= =?utf8?q?rent_UID_doesn=27t_have_an?= Message-ID: http://hg.python.org/cpython/rev/982c30064bc9 changeset: 76715:982c30064bc9 parent: 76711:6c9ce7e34511 parent: 76714:93424b084d08 user: Charles-Fran?ois Natali date: Wed May 02 20:04:40 2012 +0200 summary: Issue #14698: Make test_posix more robust when the current UID doesn't have an associated pwd entry. files: Lib/test/test_posix.py | 12 +++++++++--- 1 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -108,7 +108,11 @@ # If a non-privileged user invokes it, it should fail with OSError # EPERM. if os.getuid() != 0: - name = pwd.getpwuid(posix.getuid()).pw_name + try: + name = pwd.getpwuid(posix.getuid()).pw_name + except KeyError: + # the current UID may not have a pwd entry + raise unittest.SkipTest("need a pwd entry") try: posix.initgroups(name, 13) except OSError as e: @@ -624,8 +628,9 @@ def test_getgrouplist(self): with os.popen('id -G') as idg: groups = idg.read().strip() + ret = idg.close() - if not groups: + if ret != 0 or not groups: raise unittest.SkipTest("need working 'id -G'") self.assertEqual( @@ -637,8 +642,9 @@ def test_getgroups(self): with os.popen('id -G') as idg: groups = idg.read().strip() + ret = idg.close() - if not groups: + if ret != 0 or not groups: raise unittest.SkipTest("need working 'id -G'") # 'id -G' and 'os.getgroups()' should return the same -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:45:06 2012 From: python-checkins at python.org (richard.oudkerk) Date: Wed, 02 May 2012 20:45:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_test=5Fmultiprocessing?= =?utf8?q?_cleanup_properly?= Message-ID: http://hg.python.org/cpython/rev/e54adf13e7a6 changeset: 76716:e54adf13e7a6 user: Richard Oudkerk date: Wed May 02 19:36:11 2012 +0100 summary: Make test_multiprocessing cleanup properly Previously, when an error was encountered some processes would not be stopped until atexit callbacks were run. On Windows that was too late to prevent a PermissionError when regrtest tried to remove the temp directory it ran the tests in. files: Lib/test/test_multiprocessing.py | 21 ++++++++++++------- 1 files changed, 13 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2555,6 +2555,8 @@ def pool_in_process(): pool = multiprocessing.Pool(processes=4) x = pool.map(_afunc, [1, 2, 3, 4, 5, 6, 7]) + pool.close() + pool.join() class _file_like(object): def __init__(self, delegate): @@ -2808,14 +2810,17 @@ loadTestsFromTestCase = unittest.defaultTestLoader.loadTestsFromTestCase suite = unittest.TestSuite(loadTestsFromTestCase(tc) for tc in testcases) - run(suite) - - ThreadsMixin.pool.terminate() - ProcessesMixin.pool.terminate() - ManagerMixin.pool.terminate() - ManagerMixin.manager.shutdown() - - del ProcessesMixin.pool, ThreadsMixin.pool, ManagerMixin.pool + try: + run(suite) + finally: + ThreadsMixin.pool.terminate() + ProcessesMixin.pool.terminate() + ManagerMixin.pool.terminate() + ManagerMixin.pool.join() + ManagerMixin.manager.shutdown() + ThreadsMixin.pool.join() + ProcessesMixin.pool.join() + del ProcessesMixin.pool, ThreadsMixin.pool, ManagerMixin.pool def main(): test_main(unittest.TextTestRunner(verbosity=2).run) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:52:15 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:52:15 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogb3MucG9wZW4oKS5j?= =?utf8?q?lose=28=29_returns_None_on_success=2C_not_0=2E=2E=2E?= Message-ID: http://hg.python.org/cpython/rev/67b19fe51a65 changeset: 76717:67b19fe51a65 branch: 2.7 parent: 76713:45f0272f5296 user: Charles-Fran?ois Natali date: Wed May 02 20:48:21 2012 +0200 summary: os.popen().close() returns None on success, not 0... files: Lib/test/test_posix.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -424,7 +424,7 @@ groups = idg.read().strip() ret = idg.close() - if ret != 0 or not groups: + if ret != None or not groups: raise unittest.SkipTest("need working 'id -G'") # 'id -G' and 'os.getgroups()' should return the same -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:52:16 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:52:16 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogb3MucG9wZW4oKS5j?= =?utf8?q?lose=28=29_returns_None_on_success=2C_not_0=2E=2E=2E?= Message-ID: http://hg.python.org/cpython/rev/26c1a1f29e1e changeset: 76718:26c1a1f29e1e branch: 3.2 parent: 76714:93424b084d08 user: Charles-Fran?ois Natali date: Wed May 02 20:49:14 2012 +0200 summary: os.popen().close() returns None on success, not 0... files: Lib/test/test_posix.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -427,7 +427,7 @@ groups = idg.read().strip() ret = idg.close() - if ret != 0 or not groups: + if ret != None or not groups: raise unittest.SkipTest("need working 'id -G'") # 'id -G' and 'os.getgroups()' should return the same -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:52:17 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:52:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_os=2Epopen=28=29=2Eclose=28=29_returns_None_on_success=2C_no?= =?utf8?b?dCAwLi4u?= Message-ID: http://hg.python.org/cpython/rev/d7779f783ae2 changeset: 76719:d7779f783ae2 parent: 76715:982c30064bc9 parent: 76718:26c1a1f29e1e user: Charles-Fran?ois Natali date: Wed May 02 20:50:13 2012 +0200 summary: os.popen().close() returns None on success, not 0... files: Lib/test/test_posix.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -630,7 +630,7 @@ groups = idg.read().strip() ret = idg.close() - if ret != 0 or not groups: + if ret != None or not groups: raise unittest.SkipTest("need working 'id -G'") self.assertEqual( @@ -644,7 +644,7 @@ groups = idg.read().strip() ret = idg.close() - if ret != 0 or not groups: + if ret != None or not groups: raise unittest.SkipTest("need working 'id -G'") # 'id -G' and 'os.getgroups()' should return the same -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 2 20:52:18 2012 From: python-checkins at python.org (charles-francois.natali) Date: Wed, 02 May 2012 20:52:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?b?KTogTWVyZ2Uu?= Message-ID: http://hg.python.org/cpython/rev/8635825b9734 changeset: 76720:8635825b9734 parent: 76719:d7779f783ae2 parent: 76716:e54adf13e7a6 user: Charles-Fran?ois Natali date: Wed May 02 20:51:59 2012 +0200 summary: Merge. files: Lib/test/test_multiprocessing.py | 21 ++++++++++++------- 1 files changed, 13 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2555,6 +2555,8 @@ def pool_in_process(): pool = multiprocessing.Pool(processes=4) x = pool.map(_afunc, [1, 2, 3, 4, 5, 6, 7]) + pool.close() + pool.join() class _file_like(object): def __init__(self, delegate): @@ -2808,14 +2810,17 @@ loadTestsFromTestCase = unittest.defaultTestLoader.loadTestsFromTestCase suite = unittest.TestSuite(loadTestsFromTestCase(tc) for tc in testcases) - run(suite) - - ThreadsMixin.pool.terminate() - ProcessesMixin.pool.terminate() - ManagerMixin.pool.terminate() - ManagerMixin.manager.shutdown() - - del ProcessesMixin.pool, ThreadsMixin.pool, ManagerMixin.pool + try: + run(suite) + finally: + ThreadsMixin.pool.terminate() + ProcessesMixin.pool.terminate() + ManagerMixin.pool.terminate() + ManagerMixin.pool.join() + ManagerMixin.manager.shutdown() + ThreadsMixin.pool.join() + ProcessesMixin.pool.join() + del ProcessesMixin.pool, ThreadsMixin.pool, ManagerMixin.pool def main(): test_main(unittest.TextTestRunner(verbosity=2).run) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 01:46:27 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 01:46:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314687=3A_Optimize_?= =?utf8?q?str=25tuple_for_the_=22=25=28name=29s=22_syntax?= Message-ID: http://hg.python.org/cpython/rev/90b4c2d7c90d changeset: 76721:90b4c2d7c90d user: Victor Stinner date: Thu May 03 01:44:59 2012 +0200 summary: Issue #14687: Optimize str%tuple for the "%(name)s" syntax Avoid an useless and expensive call to PyUnicode_READ(). files: Objects/unicodeobject.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13737,9 +13737,10 @@ keystart = fmtpos; /* Skip over balanced parentheses */ while (pcount > 0 && --fmtcnt >= 0) { - if (PyUnicode_READ(fmtkind, fmt, fmtpos) == ')') + c = PyUnicode_READ(fmtkind, fmt, fmtpos); + if (c == ')') --pcount; - else if (PyUnicode_READ(fmtkind, fmt, fmtpos) == '(') + else if (c == '(') ++pcount; fmtpos++; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 02:33:56 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 02:33:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Unicode=3A_optimize_creatin?= =?utf8?q?g_of_1-character_strings?= Message-ID: http://hg.python.org/cpython/rev/3d83e2297166 changeset: 76722:3d83e2297166 user: Victor Stinner date: Thu May 03 02:17:04 2012 +0200 summary: Unicode: optimize creating of 1-character strings files: Objects/unicodeobject.c | 60 ++++++++++++++++++++++++---- 1 files changed, 51 insertions(+), 9 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1919,8 +1919,18 @@ return unicode_empty; } assert(size > 0); - if (size == 1 && u[0] < 256) - return get_latin1_char((unsigned char)u[0]); + if (size == 1) { + Py_UCS4 ch = u[0]; + if (ch < 256) + return get_latin1_char((unsigned char)ch); + + res = PyUnicode_New(1, ch); + if (res == NULL) + return NULL; + PyUnicode_WRITE(PyUnicode_KIND(res), PyUnicode_DATA(res), 0, ch); + assert(_PyUnicode_CheckConsistency(res, 1)); + return res; + } max_char = ucs2lib_find_max_char(u, u + size); res = PyUnicode_New(size, max_char); @@ -1947,8 +1957,18 @@ return unicode_empty; } assert(size > 0); - if (size == 1 && u[0] < 256) - return get_latin1_char((unsigned char)u[0]); + if (size == 1) { + Py_UCS4 ch = u[0]; + if (ch < 256) + return get_latin1_char((unsigned char)ch); + + res = PyUnicode_New(1, ch); + if (res == NULL) + return NULL; + PyUnicode_WRITE(PyUnicode_KIND(res), PyUnicode_DATA(res), 0, ch); + assert(_PyUnicode_CheckConsistency(res, 1)); + return res; + } max_char = ucs4lib_find_max_char(u, u + size); res = PyUnicode_New(size, max_char); @@ -11368,10 +11388,33 @@ static PyObject * unicode_getitem(PyObject *self, Py_ssize_t index) { - Py_UCS4 ch = PyUnicode_ReadChar(self, index); - if (ch == (Py_UCS4)-1) - return NULL; - return PyUnicode_FromOrdinal(ch); + void *data; + enum PyUnicode_Kind kind; + Py_UCS4 ch; + PyObject *res; + + if (!PyUnicode_Check(self) || PyUnicode_READY(self) == -1) { + PyErr_BadArgument(); + return NULL; + } + if (index < 0 || index >= PyUnicode_GET_LENGTH(self)) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return NULL; + } + kind = PyUnicode_KIND(self); + data = PyUnicode_DATA(self); + ch = PyUnicode_READ(kind, data, index); + if (ch < 256) + return get_latin1_char(ch); + + res = PyUnicode_New(1, ch); + if (res == NULL) + return NULL; + kind = PyUnicode_KIND(res); + data = PyUnicode_DATA(res); + PyUnicode_WRITE(kind, data, 0, ch); + assert(_PyUnicode_CheckConsistency(res, 1)); + return res; } /* Believe it or not, this produces the same value for ASCII strings @@ -12039,7 +12082,6 @@ } if (PyUnicode_IS_ASCII(self)) { - kind = PyUnicode_KIND(self); data = PyUnicode_1BYTE_DATA(self); return unicode_fromascii(data + start, length); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 02:33:57 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 02:33:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_PyUnicode=5FSubstring?= =?utf8?q?=28=29_for_start_=3E=3D_length_and_start_=3E_end?= Message-ID: http://hg.python.org/cpython/rev/99be985edeca changeset: 76723:99be985edeca user: Victor Stinner date: Thu May 03 02:32:34 2012 +0200 summary: Fix PyUnicode_Substring() for start >= length and start > end Remove the fast-path for 1-character string: unicode_fromascii() and _PyUnicode_FromUCS*() now have their own fast-path for 1-character strings. files: Objects/unicodeobject.c | 18 ++++++++++-------- 1 files changed, 10 insertions(+), 8 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -12067,20 +12067,22 @@ if (PyUnicode_READY(self) == -1) return NULL; - end = Py_MIN(end, PyUnicode_GET_LENGTH(self)); - - if (start == 0 && end == PyUnicode_GET_LENGTH(self)) + length = PyUnicode_GET_LENGTH(self); + end = Py_MIN(end, length); + + if (start == 0 && end == length) return unicode_result_unchanged(self); - length = end - start; - if (length == 1) - return unicode_getitem(self, start); - if (start < 0 || end < 0) { PyErr_SetString(PyExc_IndexError, "string index out of range"); return NULL; } - + if (start >= length || end < start) { + assert(end == length); + return PyUnicode_New(0, 0); + } + + length = end - start; if (PyUnicode_IS_ASCII(self)) { data = PyUnicode_1BYTE_DATA(self); return unicode_fromascii(data + start, length); -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Thu May 3 02:53:38 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 May 2012 10:53:38 +1000 Subject: [Python-checkins] cpython: Fix PyUnicode_Substring() for start >= length and start > end In-Reply-To: References: Message-ID: On Thu, May 3, 2012 at 10:33 AM, victor.stinner wrote: > + ? ?if (start >= length || end < start) { > + ? ? ? ?assert(end == length); > + ? ? ? ?return PyUnicode_New(0, 0); > + ? ?} That assert doesn't look right. Consider: "abc"[4:1] Unless I'm missing something, "end" will be 1, but "length" will be 3 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python-checkins at python.org Thu May 3 03:37:48 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 03:37:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_buggy_assertion_in_P?= =?utf8?q?yUnicode=5FSubstring=28=29?= Message-ID: http://hg.python.org/cpython/rev/69ed936deb0f changeset: 76724:69ed936deb0f user: Victor Stinner date: Thu May 03 03:36:40 2012 +0200 summary: Remove buggy assertion in PyUnicode_Substring() Use also directly unicode_empty, instead of PyUnicode_New(0,0). files: Objects/unicodeobject.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -12078,8 +12078,8 @@ return NULL; } if (start >= length || end < start) { - assert(end == length); - return PyUnicode_New(0, 0); + Py_INCREF(unicode_empty); + return unicode_empty; } length = end - start; -- Repository URL: http://hg.python.org/cpython From victor.stinner at gmail.com Thu May 3 03:38:33 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 3 May 2012 03:38:33 +0200 Subject: [Python-checkins] cpython: Fix PyUnicode_Substring() for start >= length and start > end In-Reply-To: References: Message-ID: >> + ? ?if (start >= length || end < start) { >> + ? ? ? ?assert(end == length); >> + ? ? ? ?return PyUnicode_New(0, 0); >> + ? ?} > > That assert doesn't look right. Oh, you're right. I added it for the first case: start>=length. But the assertion is really useless, I removed it. Thanks! Victor From solipsis at pitrou.net Thu May 3 05:36:41 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 03 May 2012 05:36:41 +0200 Subject: [Python-checkins] Daily reference leaks (99be985edeca): sum=0 Message-ID: results for 99be985edeca on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloguWY0qX', '-x'] From python-checkins at python.org Thu May 3 09:31:05 2012 From: python-checkins at python.org (larry.hastings) Date: Thu, 03 May 2012 09:31:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314127=3A_Add_ns=3D?= =?utf8?q?_parameter_to_utime=2C_futimes=2C_and_lutimes=2E?= Message-ID: http://hg.python.org/cpython/rev/bba131e48852 changeset: 76725:bba131e48852 user: Larry Hastings date: Thu May 03 00:30:07 2012 -0700 summary: Issue #14127: Add ns= parameter to utime, futimes, and lutimes. Removed futimens as it is now redundant. Changed shutil.copystat to use st_atime_ns and st_mtime_ns from os.stat and ns= parameter to utime--it once again preserves exact metadata on Linux! files: Doc/library/os.rst | 54 ++- Include/pytime.h | 4 + Lib/shutil.py | 2 +- Lib/test/test_os.py | 86 ++++- Modules/posixmodule.c | 457 +++++++++++++++-------------- Python/pytime.c | 2 +- 6 files changed, 357 insertions(+), 248 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -934,13 +934,11 @@ .. versionadded:: 3.3 -.. function:: futimes(fd[, times]) +.. function:: futimes(fd[, times, *, ns=times]) Set the access and modified time of the file specified by the file - descriptor *fd* to the given values. *atimes* must be a 2-tuple of numbers, - of the form ``(atime, mtime)``, or None. If no second argument is used, - set the access and modified times to the current time. - + descriptor *fd* to the given values. See :func:`utime` for proper + use of the *times* and *ns* arguments. Availability: Unix. .. versionadded:: 3.3 @@ -1762,12 +1760,11 @@ Added support for Windows 6.0 (Vista) symbolic links. -.. function:: lutimes(path[, times]) +.. function:: lutimes(path[, times, *, ns=times]) Like :func:`utime`, but if *path* is a symbolic link, it is not - dereferenced. *times* must be a 2-tuple of numbers, of the form - ``(atime, mtime)``, or None. - + dereferenced. See :func:`utime` for proper use of the + *times* and *ns* arguments. Availability: Unix. @@ -2226,22 +2223,43 @@ Availability: Unix, Windows. -.. function:: utime(path[, times]) - - Set the access and modified times of the file specified by *path*. If *times* - is ``None`` or not specified, then the file's access and modified times are - set to the current time. (The effect is similar to running the Unix program - :program:`touch` on the path.) Otherwise, *times* must be a 2-tuple of - numbers, of the form ``(atime, mtime)`` which is used to set the access and - modified times, respectively. Whether a directory can be given for *path* +.. function:: utime(path[, times, *, ns=(atime_ns, mtime_ns)]) + + Set the access and modified times of the file specified by *path*. + + :func:`utime` takes two optional parameters, *times* and *ns*. + These specify the times set on *path* and are used as follows: + + - If *ns* is specified, + it must be a 2-tuple of the form ``(atime_ns, mtime_ns)`` + where each member is an int expressing nanoseconds. + - If *times* is specified and is not ``None``, + it must be a 2-tuple of the form ``(atime, mtime)`` + where each member is an int or float expressing seconds. + - If *times* is specified as ``None``, + this is equivalent to specifying an ``(atime, mtime)`` + where both times are the current time. + (The effect is similar to running the Unix program + :program:`touch` on *path*.) + - If neither *ns* nor *times* is specified, this is + equivalent to specifying *times* as ``None``. + + Specifying both *times* and *ns* simultaneously is an error. + + Whether a directory can be given for *path* depends on whether the operating system implements directories as files (for example, Windows does not). Note that the exact times you set here may not be returned by a subsequent :func:`~os.stat` call, depending on the resolution with which your operating system records access and modification - times; see :func:`~os.stat`. + times; see :func:`~os.stat`. The best way to preserve exact times is to + use the *st_atime_ns* and *st_mtime_ns* fields from the :func:`os.stat` + result object with the *ns* parameter to `utime`. Availability: Unix, Windows. + .. versionadded:: 3.3 + The :attr:`ns` keyword parameter. + .. function:: walk(top, topdown=True, onerror=None, followlinks=False) diff --git a/Include/pytime.h b/Include/pytime.h --- a/Include/pytime.h +++ b/Include/pytime.h @@ -62,6 +62,10 @@ PyAPI_FUNC(PyObject *) _PyLong_FromTime_t( time_t sec); +/* Convert a PyLong to a time_t. */ +PyAPI_FUNC(time_t) _PyLong_AsTime_t( + PyObject *obj); + /* Convert a number of seconds, int or float, to a timeval structure. usec is in the range [0; 999999] and rounded towards zero. For example, -1.2 is converted to (-2, 800000). */ diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -154,7 +154,7 @@ st = stat_func(src) mode = stat.S_IMODE(st.st_mode) - utime_func(dst, (st.st_atime, st.st_mtime)) + utime_func(dst, ns=(st.st_atime_ns, st.st_mtime_ns)) chmod_func(dst, mode) if hasattr(st, 'st_flags'): try: diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -192,11 +192,11 @@ self.assertIn(attr, members) # Make sure that the st_?time and st_?time_ns fields roughly agree - # (they should always agree up to the tens-of-microseconds magnitude) + # (they should always agree up to around tens-of-microseconds) for name in 'st_atime st_mtime st_ctime'.split(): floaty = int(getattr(result, name) * 100000) nanosecondy = getattr(result, name + "_ns") // 10000 - self.assertEqual(floaty, nanosecondy) + self.assertAlmostEqual(floaty, nanosecondy, delta=2) try: result[200] @@ -303,20 +303,80 @@ st2 = os.stat(support.TESTFN) self.assertEqual(st2.st_mtime, int(st.st_mtime-delta)) - def test_utime_noargs(self): + def _test_utime(self, filename, attr, utime, delta): # Issue #13327 removed the requirement to pass None as the # second argument. Check that the previous methods of passing # a time tuple or None work in addition to no argument. - st = os.stat(support.TESTFN) + st0 = os.stat(filename) # Doesn't set anything new, but sets the time tuple way - os.utime(support.TESTFN, (st.st_atime, st.st_mtime)) + utime(filename, (attr(st0, "st_atime"), attr(st0, "st_mtime"))) + # Setting the time to the time you just read, then reading again, + # should always return exactly the same times. + st1 = os.stat(filename) + self.assertEqual(attr(st0, "st_mtime"), attr(st1, "st_mtime")) + self.assertEqual(attr(st0, "st_atime"), attr(st1, "st_atime")) # Set to the current time in the old explicit way. - os.utime(support.TESTFN, None) - st1 = os.stat(support.TESTFN) + os.utime(filename, None) + st2 = os.stat(support.TESTFN) # Set to the current time in the new way - os.utime(support.TESTFN) - st2 = os.stat(support.TESTFN) - self.assertAlmostEqual(st1.st_mtime, st2.st_mtime, delta=10) + os.utime(filename) + st3 = os.stat(filename) + self.assertAlmostEqual(attr(st2, "st_mtime"), attr(st3, "st_mtime"), delta=delta) + + def test_utime(self): + def utime(file, times): + return os.utime(file, times) + self._test_utime(self.fname, getattr, utime, 10) + self._test_utime(support.TESTFN, getattr, utime, 10) + + + def _test_utime_ns(self, set_times_ns, test_dir=True): + def getattr_ns(o, attr): + return getattr(o, attr + "_ns") + ten_s = 10 * 1000 * 1000 * 1000 + self._test_utime(self.fname, getattr_ns, set_times_ns, ten_s) + if test_dir: + self._test_utime(support.TESTFN, getattr_ns, set_times_ns, ten_s) + + def test_utime_ns(self): + def utime_ns(file, times): + return os.utime(file, ns=times) + self._test_utime_ns(utime_ns) + + requires_lutimes = unittest.skipUnless(hasattr(os, 'lutimes'), + "os.lutimes required for this test.") + requires_futimes = unittest.skipUnless(hasattr(os, 'futimes'), + "os.futimes required for this test.") + + @requires_lutimes + def test_lutimes_ns(self): + def lutimes_ns(file, times): + return os.lutimes(file, ns=times) + self._test_utime_ns(lutimes_ns) + + @requires_futimes + def test_futimes_ns(self): + def futimes_ns(file, times): + with open(file, "wb") as f: + os.futimes(f.fileno(), ns=times) + self._test_utime_ns(futimes_ns, test_dir=False) + + def _utime_invalid_arguments(self, name, arg): + with self.assertRaises(RuntimeError): + getattr(os, name)(arg, (5, 5), ns=(5, 5)) + + def test_utime_invalid_arguments(self): + self._utime_invalid_arguments('utime', self.fname) + + @requires_lutimes + def test_lutimes_invalid_arguments(self): + self._utime_invalid_arguments('lutimes', self.fname) + + @requires_futimes + def test_futimes_invalid_arguments(self): + with open(self.fname, "wb") as f: + self._utime_invalid_arguments('futimes', f.fileno()) + @unittest.skipUnless(stat_supports_subsecond, "os.stat() doesn't has a subsecond resolution") @@ -338,8 +398,7 @@ os.utime(filename, (atime, mtime)) self._test_utime_subsecond(set_time) - @unittest.skipUnless(hasattr(os, 'futimes'), - "os.futimes required for this test.") + @requires_futimes def test_futimes_subsecond(self): def set_time(filename, atime, mtime): with open(filename, "wb") as f: @@ -375,8 +434,7 @@ os.close(dirfd) self._test_utime_subsecond(set_time) - @unittest.skipUnless(hasattr(os, 'lutimes'), - "os.lutimes required for this test.") + @requires_lutimes def test_lutimes_subsecond(self): def set_time(filename, atime, mtime): os.lutimes(filename, (atime, mtime)) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3572,28 +3572,194 @@ #endif /* HAVE_UNAME */ +static int +split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns) +{ + int result = 0; + PyObject *divmod; + divmod = PyNumber_Divmod(py_long, billion); + if (!divmod) + goto exit; + *s = _PyLong_AsTime_t(PyTuple_GET_ITEM(divmod, 0)); + if ((*s == -1) && PyErr_Occurred()) + goto exit; + *ns = PyLong_AsLong(PyTuple_GET_ITEM(divmod, 1)); + if ((*s == -1) && PyErr_Occurred()) + goto exit; + + result = 1; +exit: + Py_XDECREF(divmod); + return result; +} + + +typedef int (*parameter_converter_t)(PyObject *, void *); + +typedef struct { + /* input only */ + char path_format; + parameter_converter_t converter; + char *function_name; + char *first_argument_name; + PyObject *args; + PyObject *kwargs; + + /* input/output */ + PyObject **path; + + /* output only */ + int now; + time_t atime_s; + long atime_ns; + time_t mtime_s; + long mtime_ns; +} utime_arguments; + +#define DECLARE_UA(ua, fname) \ + utime_arguments ua; \ + memset(&ua, 0, sizeof(ua)); \ + ua.function_name = fname; \ + ua.args = args; \ + ua.kwargs = kwargs; \ + ua.first_argument_name = "path"; \ + +/* UA_TO_FILETIME doesn't declare atime and mtime for you */ +#define UA_TO_FILETIME(ua, atime, mtime) \ + time_t_to_FILE_TIME(ua.atime_s, ua.atime_ns, &atime); \ + time_t_to_FILE_TIME(ua.mtime_s, ua.mtime_ns, &mtime) + +/* the rest of these macros declare the output variable for you */ +#define UA_TO_TIMESPEC(ua, ts) \ + struct timespec ts[2]; \ + ts[0].tv_sec = ua.atime_s; \ + ts[0].tv_nsec = ua.atime_ns; \ + ts[1].tv_sec = ua.mtime_s; \ + ts[1].tv_nsec = ua.mtime_ns + +#define UA_TO_TIMEVAL(ua, tv) \ + struct timeval tv[2]; \ + tv[0].tv_sec = ua.atime_s; \ + tv[0].tv_usec = ua.atime_ns / 1000; \ + tv[1].tv_sec = ua.mtime_s; \ + tv[1].tv_usec = ua.mtime_ns / 1000 + +#define UA_TO_UTIMBUF(ua, u) \ + struct utimbuf u; \ + utimbuf.actime = ua.atime_s; \ + utimbuf.modtime = ua.mtime_s + +#define UA_TO_TIME_T(ua, timet) \ + time_t timet[2]; \ + timet[0] = ua.atime_s; \ + timet[1] = ua.mtime_s + + +/* + * utime_read_time_arguments() processes arguments for the utime + * family of functions. + * returns zero on failure. + */ +static int +utime_read_time_arguments(utime_arguments *ua) +{ + PyObject *times = NULL; + PyObject *ns = NULL; + char format[24]; + char *kwlist[4]; + char **kw = kwlist; + int return_value; + + *kw++ = ua->first_argument_name; + *kw++ = "times"; + *kw++ = "ns"; + *kw = NULL; + + sprintf(format, "%c%s|O$O:%s", + ua->path_format, + ua->converter ? "&" : "", + ua->function_name); + + if (ua->converter) + return_value = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, + format, kwlist, ua->converter, ua->path, ×, &ns); + else + return_value = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, + format, kwlist, ua->path, ×, &ns); + + if (!return_value) + return 0; + + if (times && ns) { + PyErr_Format(PyExc_RuntimeError, + "%s: you may specify either 'times'" + " or 'ns' but not both", + ua->function_name); + return 0; + } + + if (times && (times != Py_None)) { + if (!PyTuple_CheckExact(times) || (PyTuple_Size(times) != 2)) { + PyErr_Format(PyExc_TypeError, + "%s: 'time' must be either" + " a valid tuple of two ints or None", + ua->function_name); + return 0; + } + ua->now = 0; + return (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 0), + &(ua->atime_s), &(ua->atime_ns)) != -1) + && (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), + &(ua->mtime_s), &(ua->mtime_ns)) != -1); + } + + if (ns) { + if (!PyTuple_CheckExact(ns) || (PyTuple_Size(ns) != 2)) { + PyErr_Format(PyExc_TypeError, + "%s: 'ns' must be a valid tuple of two ints", + ua->function_name); + return 0; + } + ua->now = 0; + return (split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 0), + &(ua->atime_s), &(ua->atime_ns))) + && (split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 1), + &(ua->mtime_s), &(ua->mtime_ns))); + } + + /* either times=None, or neither times nor ns was specified. use "now". */ + ua->now = 1; + return 1; +} + + PyDoc_STRVAR(posix_utime__doc__, -"utime(path[, (atime, mtime)])\n\ -Set the access and modified time of the file to the given values.\n\ -If no second argument is used, set the access and modified times to\n\ -the current time."); - -static PyObject * -posix_utime(PyObject *self, PyObject *args) +"utime(path[, times=(atime, mtime), *, ns=(atime_ns, mtime_ns)])\n\ +Set the access and modified time of the file.\n\ +If the second argument ('times') is specified,\n\ + the values should be expressed as float seconds since the epoch.\n\ +If the keyword argument 'ns' is specified,\n\ + the values should be expressed as integer nanoseconds since the epoch.\n\ +If neither the second nor the 'ns' argument is specified,\n\ + utime uses the current time.\n\ +Specifying both 'times' and 'ns' is an error."); + +static PyObject * +posix_utime(PyObject *self, PyObject *args, PyObject *kwargs) { #ifdef MS_WINDOWS - PyObject *arg = Py_None; - PyObject *obwpath; - wchar_t *wpath = NULL; - const char *apath; + PyObject *upath; HANDLE hFile; - time_t atimesec, mtimesec; - long ansec, mnsec; + PyObject *result = NULL; FILETIME atime, mtime; - PyObject *result = NULL; - - if (PyArg_ParseTuple(args, "U|O:utime", &obwpath, &arg)) { - wpath = PyUnicode_AsUnicode(obwpath); + + DECLARE_UA(ua, "utime"); + + ua.path_format = 'U'; + ua.path = &upath; + + if (!utime_read_time_arguments(&ua)) { + wchar_t *wpath = PyUnicode_AsUnicode(upath); if (wpath == NULL) return NULL; Py_BEGIN_ALLOW_THREADS @@ -3602,14 +3768,17 @@ FILE_FLAG_BACKUP_SEMANTICS, NULL); Py_END_ALLOW_THREADS if (hFile == INVALID_HANDLE_VALUE) - return win32_error_object("utime", obwpath); + return win32_error_object("utime", upath); } else { + const char *apath; /* Drop the argument parsing error as narrow strings are also valid. */ PyErr_Clear(); - if (!PyArg_ParseTuple(args, "y|O:utime", &apath, &arg)) + ua.path_format = 'y'; + ua.path = (PyObject **)&apath; + if (!utime_read_time_arguments(&ua)) return NULL; if (win32_warn_bytes_api()) return NULL; @@ -3625,7 +3794,7 @@ } } - if (arg == Py_None) { + if (ua.now) { SYSTEMTIME now; GetSystemTime(&now); if (!SystemTimeToFileTime(&now, &mtime) || @@ -3634,20 +3803,8 @@ goto done; } } - else if (!PyTuple_Check(arg) || PyTuple_Size(arg) != 2) { - PyErr_SetString(PyExc_TypeError, - "utime() arg 2 must be a tuple (atime, mtime)"); - goto done; - } else { - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 0), - &atimesec, &ansec) == -1) - goto done; - time_t_to_FILE_TIME(atimesec, ansec, &atime); - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 1), - &mtimesec, &mnsec) == -1) - goto done; - time_t_to_FILE_TIME(mtimesec, mnsec, &mtime); + UA_TO_FILETIME(ua, atime, mtime); } if (!SetFileTime(hFile, NULL, &atime, &mtime)) { /* Avoid putting the file name into the error here, @@ -3663,136 +3820,85 @@ CloseHandle(hFile); return result; #else /* MS_WINDOWS */ - PyObject *opath; char *path; - time_t atime, mtime; - long ansec, mnsec; int res; - PyObject* arg = Py_None; - - if (!PyArg_ParseTuple(args, "O&|O:utime", - PyUnicode_FSConverter, &opath, &arg)) + + DECLARE_UA(ua, "utime"); + + ua.path_format = 'O'; + ua.path = &opath; + ua.converter = PyUnicode_FSConverter; + + if (!utime_read_time_arguments(&ua)) return NULL; path = PyBytes_AsString(opath); - if (arg == Py_None) { - /* optional time values not given */ + if (ua.now) { Py_BEGIN_ALLOW_THREADS res = utime(path, NULL); Py_END_ALLOW_THREADS } - else if (!PyTuple_Check(arg) || PyTuple_Size(arg) != 2) { - PyErr_SetString(PyExc_TypeError, - "utime() arg 2 must be a tuple (atime, mtime)"); - Py_DECREF(opath); - return NULL; - } else { - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 0), - &atime, &ansec) == -1) { - Py_DECREF(opath); - return NULL; - } - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 1), - &mtime, &mnsec) == -1) { - Py_DECREF(opath); - return NULL; - } - Py_BEGIN_ALLOW_THREADS - { #ifdef HAVE_UTIMENSAT - struct timespec buf[2]; - buf[0].tv_sec = atime; - buf[0].tv_nsec = ansec; - buf[1].tv_sec = mtime; - buf[1].tv_nsec = mnsec; + UA_TO_TIMESPEC(ua, buf); res = utimensat(AT_FDCWD, path, buf, 0); #elif defined(HAVE_UTIMES) - struct timeval buf[2]; - buf[0].tv_sec = atime; - buf[0].tv_usec = ansec / 1000; - buf[1].tv_sec = mtime; - buf[1].tv_usec = mnsec / 1000; + UA_TO_TIMEVAL(ua, buf); res = utimes(path, buf); #elif defined(HAVE_UTIME_H) /* XXX should define struct utimbuf instead, above */ - struct utimbuf buf; - buf.actime = atime; - buf.modtime = mtime; + UA_TO_UTIMBUF(ua, buf); res = utime(path, &buf); #else - time_t buf[2]; - buf[0] = atime; - buf[1] = mtime; + UA_TO_TIME_T(ua, buf); res = utime(path, buf); #endif - } Py_END_ALLOW_THREADS } + if (res < 0) { return posix_error_with_allocated_filename(opath); } Py_DECREF(opath); - Py_INCREF(Py_None); - return Py_None; + Py_RETURN_NONE; #undef UTIME_EXTRACT #endif /* MS_WINDOWS */ } #ifdef HAVE_FUTIMES PyDoc_STRVAR(posix_futimes__doc__, -"futimes(fd[, (atime, mtime)])\n\ +"futimes(fd[, times=(atime, mtime), *, ns=(atime_ns, mtime_ns)])\n\ Set the access and modified time of the file specified by the file\n\ -descriptor fd to the given values. If no second argument is used, set the\n\ -access and modified times to the current time."); - -static PyObject * -posix_futimes(PyObject *self, PyObject *args) +descriptor fd. See utime for the semantics of the times and ns parameters."); + +static PyObject * +posix_futimes(PyObject *self, PyObject *args, PyObject *kwargs) { int res, fd; - PyObject* arg = Py_None; - time_t atime, mtime; - long ansec, mnsec; - - if (!PyArg_ParseTuple(args, "i|O:futimes", &fd, &arg)) - return NULL; - - if (arg == Py_None) { - /* optional time values not given */ + + DECLARE_UA(ua, "futimes"); + + ua.path_format = 'i'; + ua.path = (PyObject **)&fd; + ua.first_argument_name = "fd"; + + if (!utime_read_time_arguments(&ua)) + return NULL; + + if (ua.now) { Py_BEGIN_ALLOW_THREADS res = futimes(fd, NULL); Py_END_ALLOW_THREADS } - else if (!PyTuple_Check(arg) || PyTuple_Size(arg) != 2) { - PyErr_SetString(PyExc_TypeError, - "futimes() arg 2 must be a tuple (atime, mtime)"); - return NULL; - } else { - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 0), - &atime, &ansec) == -1) { - return NULL; - } - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 1), - &mtime, &mnsec) == -1) { - return NULL; - } Py_BEGIN_ALLOW_THREADS { #ifdef HAVE_FUTIMENS - struct timespec buf[2]; - buf[0].tv_sec = atime; - buf[0].tv_nsec = ansec; - buf[1].tv_sec = mtime; - buf[1].tv_nsec = mnsec; + UA_TO_TIMESPEC(ua, buf); res = futimens(fd, buf); #else - struct timeval buf[2]; - buf[0].tv_sec = atime; - buf[0].tv_usec = ansec / 1000; - buf[1].tv_sec = mtime; - buf[1].tv_usec = mnsec / 1000; + UA_TO_TIMEVAL(ua, buf); res = futimes(fd, buf); #endif } @@ -3806,61 +3912,40 @@ #ifdef HAVE_LUTIMES PyDoc_STRVAR(posix_lutimes__doc__, -"lutimes(path[, (atime, mtime)])\n\ +"lutimes(path[, times=(atime, mtime), *, ns=(atime_ns, mtime_ns)])\n\ Like utime(), but if path is a symbolic link, it is not dereferenced."); static PyObject * -posix_lutimes(PyObject *self, PyObject *args) +posix_lutimes(PyObject *self, PyObject *args, PyObject *kwargs) { PyObject *opath; - PyObject *arg = Py_None; const char *path; int res; - time_t atime, mtime; - long ansec, mnsec; - - if (!PyArg_ParseTuple(args, "O&|O:lutimes", - PyUnicode_FSConverter, &opath, &arg)) + + DECLARE_UA(ua, "lutimes"); + + ua.path_format = 'O'; + ua.path = &opath; + ua.converter = PyUnicode_FSConverter; + + if (!utime_read_time_arguments(&ua)) return NULL; path = PyBytes_AsString(opath); - if (arg == Py_None) { + + if (ua.now) { /* optional time values not given */ Py_BEGIN_ALLOW_THREADS res = lutimes(path, NULL); Py_END_ALLOW_THREADS } - else if (!PyTuple_Check(arg) || PyTuple_Size(arg) != 2) { - PyErr_SetString(PyExc_TypeError, - "lutimes() arg 2 must be a tuple (atime, mtime)"); - Py_DECREF(opath); - return NULL; - } else { - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 0), - &atime, &ansec) == -1) { - Py_DECREF(opath); - return NULL; - } - if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(arg, 1), - &mtime, &mnsec) == -1) { - Py_DECREF(opath); - return NULL; - } Py_BEGIN_ALLOW_THREADS { #ifdef HAVE_UTIMENSAT - struct timespec buf[2]; - buf[0].tv_sec = atime; - buf[0].tv_nsec = ansec; - buf[1].tv_sec = mtime; - buf[1].tv_nsec = mnsec; + UA_TO_TIMESPEC(ua, buf); res = utimensat(AT_FDCWD, path, buf, AT_SYMLINK_NOFOLLOW); #else - struct timeval buf[2]; - buf[0].tv_sec = atime; - buf[0].tv_usec = ansec / 1000; - buf[1].tv_sec = mtime; - buf[1].tv_usec = mnsec / 1000; + UA_TO_TIMEVAL(ua, buf); res = lutimes(path, buf); #endif } @@ -3873,62 +3958,6 @@ } #endif -#ifdef HAVE_FUTIMENS -PyDoc_STRVAR(posix_futimens__doc__, -"futimens(fd[, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec)])\n\ -Updates the timestamps of a file specified by the file descriptor fd, with\n\ -nanosecond precision.\n\ -If no second argument is given, set atime and mtime to the current time.\n\ -If *_nsec is specified as UTIME_NOW, the timestamp is updated to the\n\ -current time.\n\ -If *_nsec is specified as UTIME_OMIT, the timestamp is not updated."); - -static PyObject * -posix_futimens(PyObject *self, PyObject *args) -{ - int res, fd; - PyObject *atime = Py_None; - PyObject *mtime = Py_None; - struct timespec buf[2]; - - if (!PyArg_ParseTuple(args, "i|OO:futimens", - &fd, &atime, &mtime)) - return NULL; - if (atime == Py_None && mtime == Py_None) { - /* optional time values not given */ - Py_BEGIN_ALLOW_THREADS - res = futimens(fd, NULL); - Py_END_ALLOW_THREADS - } - else if (!PyTuple_Check(atime) || PyTuple_Size(atime) != 2) { - PyErr_SetString(PyExc_TypeError, - "futimens() arg 2 must be a tuple (atime_sec, atime_nsec)"); - return NULL; - } - else if (!PyTuple_Check(mtime) || PyTuple_Size(mtime) != 2) { - PyErr_SetString(PyExc_TypeError, - "futimens() arg 3 must be a tuple (mtime_sec, mtime_nsec)"); - return NULL; - } - else { - if (!PyArg_ParseTuple(atime, "ll:futimens", - &(buf[0].tv_sec), &(buf[0].tv_nsec))) { - return NULL; - } - if (!PyArg_ParseTuple(mtime, "ll:futimens", - &(buf[1].tv_sec), &(buf[1].tv_nsec))) { - return NULL; - } - Py_BEGIN_ALLOW_THREADS - res = futimens(fd, buf); - Py_END_ALLOW_THREADS - } - if (res < 0) - return posix_error(); - Py_RETURN_NONE; -} -#endif - /* Process operations */ PyDoc_STRVAR(posix__exit__doc__, @@ -10619,15 +10648,15 @@ #endif /* HAVE_UNAME */ {"unlink", posix_unlink, METH_VARARGS, posix_unlink__doc__}, {"remove", posix_unlink, METH_VARARGS, posix_remove__doc__}, - {"utime", posix_utime, METH_VARARGS, posix_utime__doc__}, + {"utime", (PyCFunction)posix_utime, + METH_VARARGS | METH_KEYWORDS, posix_utime__doc__}, #ifdef HAVE_FUTIMES - {"futimes", posix_futimes, METH_VARARGS, posix_futimes__doc__}, + {"futimes", (PyCFunction)posix_futimes, + METH_VARARGS | METH_KEYWORDS, posix_futimes__doc__}, #endif #ifdef HAVE_LUTIMES - {"lutimes", posix_lutimes, METH_VARARGS, posix_lutimes__doc__}, -#endif -#ifdef HAVE_FUTIMENS - {"futimens", posix_futimens, METH_VARARGS, posix_futimens__doc__}, + {"lutimes", (PyCFunction)posix_lutimes, + METH_VARARGS | METH_KEYWORDS, posix_lutimes__doc__}, #endif #ifdef HAVE_TIMES {"times", posix_times, METH_NOARGS, posix_times__doc__}, diff --git a/Python/pytime.c b/Python/pytime.c --- a/Python/pytime.c +++ b/Python/pytime.c @@ -123,7 +123,7 @@ "timestamp out of range for platform time_t"); } -static time_t +time_t _PyLong_AsTime_t(PyObject *obj) { #if defined(HAVE_LONG_LONG) && SIZEOF_TIME_T == SIZEOF_LONG_LONG -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 12:34:05 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 12:34:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314624=2C_=2314687?= =?utf8?q?=3A_Optimize_unicode=5Fwiden=28=29?= Message-ID: http://hg.python.org/cpython/rev/830eeff4fe8f changeset: 76726:830eeff4fe8f user: Victor Stinner date: Thu May 03 12:29:04 2012 +0200 summary: Issue #14624, #14687: Optimize unicode_widen() Don't convert uninitialized characters. Patch written by Serhiy Storchaka. files: Objects/unicodeobject.c | 16 +++++++++------- 1 files changed, 9 insertions(+), 7 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1627,18 +1627,19 @@ } static int -unicode_widen(PyObject **p_unicode, unsigned int maxchar) +unicode_widen(PyObject **p_unicode, Py_ssize_t length, + unsigned int maxchar) { PyObject *result; assert(PyUnicode_IS_READY(*p_unicode)); + assert(length <= PyUnicode_GET_LENGTH(*p_unicode)); if (maxchar <= PyUnicode_MAX_CHAR_VALUE(*p_unicode)) return 0; result = PyUnicode_New(PyUnicode_GET_LENGTH(*p_unicode), maxchar); if (result == NULL) return -1; - PyUnicode_CopyCharacters(result, 0, *p_unicode, 0, - PyUnicode_GET_LENGTH(*p_unicode)); + PyUnicode_CopyCharacters(result, 0, *p_unicode, 0, length); Py_DECREF(*p_unicode); *p_unicode = result; return 0; @@ -1649,7 +1650,7 @@ Py_UCS4 ch) { assert(ch <= MAX_UNICODE); - if (unicode_widen(p_unicode, ch) < 0) + if (unicode_widen(p_unicode, *pos, ch) < 0) return -1; PyUnicode_WRITE(PyUnicode_KIND(*p_unicode), PyUnicode_DATA(*p_unicode), @@ -4165,7 +4166,8 @@ if (unicode_resize(output, requiredsize) < 0) goto onError; } - if (unicode_widen(output, PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0) + if (unicode_widen(output, *outpos, + PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0) goto onError; copy_characters(*output, *outpos, repunicode, 0, replen); *outpos += replen; @@ -5611,7 +5613,7 @@ maxch = MAX_MAXCHAR(maxch, ch); #endif if (maxch > PyUnicode_MAX_CHAR_VALUE(unicode)) { - if (unicode_widen(&unicode, maxch) < 0) + if (unicode_widen(&unicode, outpos, maxch) < 0) goto onError; kind = PyUnicode_KIND(unicode); data = PyUnicode_DATA(unicode); @@ -7993,7 +7995,7 @@ goto onError; } } - if (unicode_widen(&v, PyUnicode_MAX_CHAR_VALUE(x)) < 0) + if (unicode_widen(&v, outpos, PyUnicode_MAX_CHAR_VALUE(x)) < 0) goto onError; PyUnicode_CopyCharacters(v, outpos, x, 0, targetsize); outpos += targetsize; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 13:09:52 2012 From: python-checkins at python.org (vinay.sajip) Date: Thu, 03 May 2012 13:09:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Changed_order_o?= =?utf8?q?f_cleanup_operations_to_be_more_sensible=2E?= Message-ID: http://hg.python.org/cpython/rev/e606c123244f changeset: 76727:e606c123244f branch: 2.7 parent: 76717:67b19fe51a65 user: Vinay Sajip date: Thu May 03 12:03:29 2012 +0100 summary: Changed order of cleanup operations to be more sensible. files: Lib/test/test_logging.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1906,8 +1906,8 @@ r = logging.makeLogRecord({'msg': 'testing' }) h.handle(r) finally: + remover.join() h.close() - remover.join() if os.path.exists(fn): os.unlink(fn) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 13:09:53 2012 From: python-checkins at python.org (vinay.sajip) Date: Thu, 03 May 2012 13:09:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Changed_order_o?= =?utf8?q?f_cleanup_operations_to_be_more_sensible=2E?= Message-ID: http://hg.python.org/cpython/rev/c334e90160b9 changeset: 76728:c334e90160b9 branch: 3.2 parent: 76718:26c1a1f29e1e user: Vinay Sajip date: Thu May 03 12:06:52 2012 +0100 summary: Changed order of cleanup operations to be more sensible. files: Lib/test/test_logging.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -2360,8 +2360,8 @@ r = logging.makeLogRecord({'msg': 'testing' }) h.handle(r) finally: + remover.join() h.close() - remover.join() if os.path.exists(fn): os.unlink(fn) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 13:09:53 2012 From: python-checkins at python.org (vinay.sajip) Date: Thu, 03 May 2012 13:09:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merged_hanged_order_of_cleanup_operations_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/ae086d16374e changeset: 76729:ae086d16374e parent: 76726:830eeff4fe8f parent: 76728:c334e90160b9 user: Vinay Sajip date: Thu May 03 12:09:38 2012 +0100 summary: Merged hanged order of cleanup operations from 3.2. files: Lib/test/test_logging.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -611,8 +611,8 @@ r = logging.makeLogRecord({'msg': 'testing' }) h.handle(r) finally: + remover.join() h.close() - remover.join() if os.path.exists(fn): os.unlink(fn) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 13:16:52 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 13:16:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314687=3A_str=25tup?= =?utf8?q?le_now_uses_an_optimistic_=22unicode_writer=22_instead_of_an?= Message-ID: http://hg.python.org/cpython/rev/f1db931b93d3 changeset: 76730:f1db931b93d3 user: Victor Stinner date: Thu May 03 13:10:40 2012 +0200 summary: Issue #14687: str%tuple now uses an optimistic "unicode writer" instead of an accumulator. Directly write characters into the output (don't use a temporary list): resize and widen the string on demand. files: Objects/unicodeobject.c | 272 +++++++++++++++++---------- 1 files changed, 169 insertions(+), 103 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -10074,7 +10074,7 @@ switch ((kind)) { \ case PyUnicode_1BYTE_KIND: { \ unsigned char * to_ = (unsigned char *)((data)) + (start); \ - memset(to_, (unsigned char)value, length); \ + memset(to_, (unsigned char)value, (length)); \ break; \ } \ case PyUnicode_2BYTE_KIND: { \ @@ -13655,56 +13655,133 @@ return (Py_UCS4) -1; } -static int -repeat_accumulate(_PyAccu *acc, PyObject *obj, Py_ssize_t count) -{ - int r; - assert(count > 0); - assert(PyUnicode_Check(obj)); - if (count > 5) { - PyObject *repeated = unicode_repeat(obj, count); - if (repeated == NULL) +struct unicode_writer_t { + PyObject *buffer; + void *data; + enum PyUnicode_Kind kind; + Py_UCS4 maxchar; + Py_ssize_t length; + Py_ssize_t pos; +}; + +Py_LOCAL_INLINE(void) +unicode_writer_update(struct unicode_writer_t *writer) +{ + writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); + writer->data = PyUnicode_DATA(writer->buffer); + writer->kind = PyUnicode_KIND(writer->buffer); +} + +Py_LOCAL_INLINE(int) +unicode_writer_init(struct unicode_writer_t *writer, + Py_ssize_t length, Py_UCS4 maxchar) +{ + writer->pos = 0; + writer->length = length; + writer->buffer = PyUnicode_New(writer->length, maxchar); + if (writer->buffer == NULL) + return -1; + unicode_writer_update(writer); + return 0; +} + +Py_LOCAL_INLINE(int) +unicode_writer_prepare(struct unicode_writer_t *writer, + Py_ssize_t length, Py_UCS4 maxchar) +{ + Py_ssize_t newlen; + + if (length > PY_SSIZE_T_MAX - writer->pos) { + PyErr_NoMemory(); + return -1; + } + newlen = writer->pos + length; + + if (newlen > writer->length && maxchar > writer->maxchar) { + PyObject *newbuffer; + + /* overallocate 25% to limit the number of resize */ + if (newlen > PY_SSIZE_T_MAX - newlen / 4) + writer->length = newlen; + else + writer->length = newlen + newlen / 4; + + /* resize + widen */ + newbuffer = PyUnicode_New(writer->length, maxchar); + if (newbuffer == NULL) return -1; - r = _PyAccu_Accumulate(acc, repeated); - Py_DECREF(repeated); - return r; - } - else { - do { - if (_PyAccu_Accumulate(acc, obj)) - return -1; - } while (--count); + PyUnicode_CopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); + Py_DECREF(writer->buffer); + writer->buffer = newbuffer; + unicode_writer_update(writer); return 0; } + if (newlen > writer->length) { + /* overallocate 25% to limit the number of resize */ + if (newlen > PY_SSIZE_T_MAX - newlen / 4) + writer->length = newlen; + else + writer->length = newlen + newlen / 4; + if (PyUnicode_Resize(&writer->buffer, writer->length) < 0) + return -1; + unicode_writer_update(writer); + } + if (maxchar > writer->maxchar) { + if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) + return -1; + unicode_writer_update(writer); + } + return 0; +} + +Py_LOCAL_INLINE(int) +unicode_writer_write_str( + struct unicode_writer_t *writer, + PyObject *str, Py_ssize_t start, Py_ssize_t length) +{ + Py_UCS4 maxchar; + maxchar = _PyUnicode_FindMaxChar(str, start, start + length); + if (unicode_writer_prepare(writer, length, maxchar) == -1) + return -1; + assert((writer->pos + length) <= writer->length); + copy_characters(writer->buffer, writer->pos, + str, start, length); + writer->pos += length; + return 0; +} + +Py_LOCAL_INLINE(int) +unicode_writer_write_char( + struct unicode_writer_t *writer, + Py_UCS4 ch) +{ + if (unicode_writer_prepare(writer, 1, ch) == -1) + return -1; + assert((writer->pos + 1) <= writer->length); + PyUnicode_WRITE(writer->kind, writer->data, writer->pos, ch); + writer->pos += 1; + return 0; +} + +Py_LOCAL_INLINE(void) +unicode_writer_dealloc(struct unicode_writer_t *writer) +{ + Py_CLEAR(writer->buffer); } PyObject * PyUnicode_Format(PyObject *format, PyObject *args) { - void *fmt; - int fmtkind; - PyObject *result; - int kind; - int r; Py_ssize_t fmtcnt, fmtpos, arglen, argidx; int args_owned = 0; PyObject *dict = NULL; PyObject *temp = NULL; PyObject *second = NULL; PyObject *uformat; - _PyAccu acc; - static PyObject *plus, *minus, *blank, *zero, *percent; - - if (!plus && !(plus = get_latin1_char('+'))) - return NULL; - if (!minus && !(minus = get_latin1_char('-'))) - return NULL; - if (!blank && !(blank = get_latin1_char(' '))) - return NULL; - if (!zero && !(zero = get_latin1_char('0'))) - return NULL; - if (!percent && !(percent = get_latin1_char('%'))) - return NULL; + void *fmt; + enum PyUnicode_Kind kind, fmtkind; + struct unicode_writer_t writer; if (format == NULL || args == NULL) { PyErr_BadInternalCall(); @@ -13715,13 +13792,15 @@ return NULL; if (PyUnicode_READY(uformat) == -1) Py_DECREF(uformat); - if (_PyAccu_Init(&acc)) - goto onError; + fmt = PyUnicode_DATA(uformat); fmtkind = PyUnicode_KIND(uformat); fmtcnt = PyUnicode_GET_LENGTH(uformat); fmtpos = 0; + if (unicode_writer_init(&writer, fmtcnt + 100, 127) < 0) + goto onError; + if (PyTuple_Check(args)) { arglen = PyTuple_Size(args); argidx = 0; @@ -13736,7 +13815,6 @@ while (--fmtcnt >= 0) { if (PyUnicode_READ(fmtkind, fmt, fmtpos) != '%') { - PyObject *nonfmt; Py_ssize_t nonfmtpos; nonfmtpos = fmtpos++; while (fmtcnt >= 0 && @@ -13744,12 +13822,9 @@ fmtpos++; fmtcnt--; } - nonfmt = PyUnicode_Substring(uformat, nonfmtpos, fmtpos); - if (nonfmt == NULL) - goto onError; - r = _PyAccu_Accumulate(&acc, nonfmt); - Py_DECREF(nonfmt); - if (r) + if (fmtcnt < 0) + fmtpos--; + if (unicode_writer_write_str(&writer, uformat, nonfmtpos, fmtpos - nonfmtpos) < 0) goto onError; } else { @@ -13758,12 +13833,13 @@ Py_ssize_t width = -1; int prec = -1; Py_UCS4 c = '\0'; - Py_UCS4 fill, sign; + Py_UCS4 fill; + int sign; + Py_UCS4 signchar; int isnumok; PyObject *v = NULL; void *pbuf = NULL; Py_ssize_t pindex, len; - PyObject *signobj = NULL, *fillobj = NULL; fmtpos++; c = PyUnicode_READ(fmtkind, fmt, fmtpos); @@ -13906,7 +13982,8 @@ } if (c == '%') { - _PyAccu_Accumulate(&acc, percent); + if (unicode_writer_write_char(&writer, '%') < 0) + goto onError; continue; } @@ -13916,8 +13993,8 @@ goto onError; sign = 0; + signchar = '\0'; fill = ' '; - fillobj = blank; switch (c) { case 's': @@ -13972,10 +14049,8 @@ "not %.200s", (char)c, Py_TYPE(v)->tp_name); goto onError; } - if (flags & F_ZERO) { + if (flags & F_ZERO) fill = '0'; - fillobj = zero; - } break; case 'e': @@ -13985,10 +14060,8 @@ case 'g': case 'G': sign = 1; - if (flags & F_ZERO) { + if (flags & F_ZERO) fill = '0'; - fillobj = zero; - } temp = formatfloat(v, flags, prec, c); break; @@ -14029,20 +14102,16 @@ /* pbuf is initialized here. */ pindex = 0; if (sign) { - if (PyUnicode_READ(kind, pbuf, pindex) == '-') { - signobj = minus; + Py_UCS4 ch = PyUnicode_READ(kind, pbuf, pindex); + if (ch == '-' || ch == '+') { + signchar = ch; len--; pindex++; } - else if (PyUnicode_READ(kind, pbuf, pindex) == '+') { - signobj = plus; - len--; - pindex++; - } else if (flags & F_SIGN) - signobj = plus; + signchar = '+'; else if (flags & F_BLANK) - signobj = blank; + signchar = ' '; else sign = 0; } @@ -14050,8 +14119,7 @@ width = len; if (sign) { if (fill != ' ') { - assert(signobj != NULL); - if (_PyAccu_Accumulate(&acc, signobj)) + if (unicode_writer_write_char(&writer, signchar) < 0) goto onError; } if (width > len) @@ -14061,14 +14129,12 @@ assert(PyUnicode_READ(kind, pbuf, pindex) == '0'); assert(PyUnicode_READ(kind, pbuf, pindex + 1) == c); if (fill != ' ') { - second = get_latin1_char( - PyUnicode_READ(kind, pbuf, pindex + 1)); + if (unicode_writer_prepare(&writer, 2, 127) < 0) + goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, '0'); + PyUnicode_WRITE(writer.kind, writer.data, writer.pos+1, c); + writer.pos += 2; pindex += 2; - if (second == NULL || - _PyAccu_Accumulate(&acc, zero) || - _PyAccu_Accumulate(&acc, second)) - goto onError; - Py_CLEAR(second); } width -= 2; if (width < 0) @@ -14076,45 +14142,43 @@ len -= 2; } if (width > len && !(flags & F_LJUST)) { - assert(fillobj != NULL); - if (repeat_accumulate(&acc, fillobj, width - len)) + Py_ssize_t sublen; + sublen = width - len; + if (unicode_writer_prepare(&writer, sublen, fill) < 0) goto onError; + FILL(writer.kind, writer.data, fill, writer.pos, sublen); + writer.pos += sublen; width = len; } if (fill == ' ') { if (sign) { - assert(signobj != NULL); - if (_PyAccu_Accumulate(&acc, signobj)) + if (unicode_writer_write_char(&writer, signchar) < 0) goto onError; } if ((flags & F_ALT) && (c == 'x' || c == 'X' || c == 'o')) { assert(PyUnicode_READ(kind, pbuf, pindex) == '0'); assert(PyUnicode_READ(kind, pbuf, pindex+1) == c); - second = get_latin1_char( - PyUnicode_READ(kind, pbuf, pindex + 1)); + + if (unicode_writer_prepare(&writer, 2, 127) < 0) + goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, '0'); + PyUnicode_WRITE(writer.kind, writer.data, writer.pos+1, c); + writer.pos += 2; + pindex += 2; - if (second == NULL || - _PyAccu_Accumulate(&acc, zero) || - _PyAccu_Accumulate(&acc, second)) - goto onError; - Py_CLEAR(second); - } - } + } + } + /* Copy all characters, preserving len */ - if (pindex == 0 && len == PyUnicode_GET_LENGTH(temp)) { - r = _PyAccu_Accumulate(&acc, temp); - } - else { - v = PyUnicode_Substring(temp, pindex, pindex + len); - if (v == NULL) + if (unicode_writer_write_str(&writer, temp, pindex, len) < 0) + goto onError; + if (width > len) { + Py_ssize_t sublen = width - len; + if (unicode_writer_prepare(&writer, sublen, ' ') < 0) goto onError; - r = _PyAccu_Accumulate(&acc, v); - Py_DECREF(v); - } - if (r) - goto onError; - if (width > len && repeat_accumulate(&acc, blank, width - len)) - goto onError; + FILL(writer.kind, writer.data, ' ', writer.pos, sublen); + writer.pos += sublen; + } if (dict && (argidx < arglen) && c != '%') { PyErr_SetString(PyExc_TypeError, "not all arguments converted during string formatting"); @@ -14129,20 +14193,22 @@ goto onError; } - result = _PyAccu_Finish(&acc); + if (PyUnicode_Resize(&writer.buffer, writer.pos) < 0) + goto onError; + if (args_owned) { Py_DECREF(args); } Py_DECREF(uformat); Py_XDECREF(temp); Py_XDECREF(second); - return result; + return writer.buffer; onError: Py_DECREF(uformat); Py_XDECREF(temp); Py_XDECREF(second); - _PyAccu_Destroy(&acc); + unicode_writer_dealloc(&writer); if (args_owned) { Py_DECREF(args); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 13:43:25 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 13:43:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314687=3A_Cleanup_u?= =?utf8?q?nicode=5Fwriter=5Fprepare=28=29?= Message-ID: http://hg.python.org/cpython/rev/0a9143d7b097 changeset: 76731:0a9143d7b097 user: Victor Stinner date: Thu May 03 13:43:07 2012 +0200 summary: Issue #14687: Cleanup unicode_writer_prepare() "Inline" PyUnicode_Resize(): call directly resize_compact() files: Objects/unicodeobject.c | 44 +++++++++++++--------------- 1 files changed, 20 insertions(+), 24 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -654,6 +654,7 @@ Py_ssize_t new_size; int share_wstr; PyObject *new_unicode; + assert(unicode_modifiable(unicode)); assert(PyUnicode_IS_READY(unicode)); assert(PyUnicode_IS_COMPACT(unicode)); @@ -690,6 +691,7 @@ } PyUnicode_WRITE(PyUnicode_KIND(unicode), PyUnicode_DATA(unicode), length, 0); + assert(_PyUnicode_CheckConsistency(unicode, 0)); return unicode; } @@ -1603,7 +1605,6 @@ if (new_unicode == NULL) return -1; *p_unicode = new_unicode; - assert(_PyUnicode_CheckConsistency(*p_unicode, 0)); return 0; } return resize_inplace(unicode, length); @@ -13690,6 +13691,7 @@ Py_ssize_t length, Py_UCS4 maxchar) { Py_ssize_t newlen; + PyObject *newbuffer; if (length > PY_SSIZE_T_MAX - writer->pos) { PyErr_NoMemory(); @@ -13697,37 +13699,31 @@ } newlen = writer->pos + length; - if (newlen > writer->length && maxchar > writer->maxchar) { - PyObject *newbuffer; - - /* overallocate 25% to limit the number of resize */ - if (newlen > PY_SSIZE_T_MAX - newlen / 4) - writer->length = newlen; - else - writer->length = newlen + newlen / 4; - - /* resize + widen */ - newbuffer = PyUnicode_New(writer->length, maxchar); - if (newbuffer == NULL) - return -1; - PyUnicode_CopyCharacters(newbuffer, 0, - writer->buffer, 0, writer->pos); - Py_DECREF(writer->buffer); - writer->buffer = newbuffer; - unicode_writer_update(writer); - return 0; - } if (newlen > writer->length) { /* overallocate 25% to limit the number of resize */ if (newlen > PY_SSIZE_T_MAX - newlen / 4) writer->length = newlen; else writer->length = newlen + newlen / 4; - if (PyUnicode_Resize(&writer->buffer, writer->length) < 0) - return -1; + + if (maxchar > writer->maxchar) { + /* resize + widen */ + newbuffer = PyUnicode_New(writer->length, maxchar); + if (newbuffer == NULL) + return -1; + PyUnicode_CopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); + Py_DECREF(writer->buffer); + } + else { + newbuffer = resize_compact(writer->buffer, writer->length); + if (newbuffer == NULL) + return -1; + } + writer->buffer = newbuffer; unicode_writer_update(writer); } - if (maxchar > writer->maxchar) { + else if (maxchar > writer->maxchar) { if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) return -1; unicode_writer_update(writer); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 16:39:40 2012 From: python-checkins at python.org (barry.warsaw) Date: Thu, 03 May 2012 16:39:40 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_the_spelling_of_Nick=27s_l?= =?utf8?q?ast_name=2E?= Message-ID: http://hg.python.org/peps/rev/790237ffdf35 changeset: 4349:790237ffdf35 user: Barry Warsaw date: Thu May 03 10:39:30 2012 -0400 summary: Fix the spelling of Nick's last name. files: pep-0420.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -226,7 +226,7 @@ imported as a namespace package, whereas in prior Python versions an ImportWarning would be raised. -Nick Coglan presented a list of his objections to this proposal [3]_. +Nick Coghlan presented a list of his objections to this proposal [3]_. They are: 1. Implicit package directories go against the Zen of Python. @@ -266,10 +266,10 @@ .. [2] PyCon 2012 Namespace Package discussion outcome (http://mail.python.org/pipermail/import-sig/2012-March/000421.html) -.. [3] Nick Coglan's objection to the lack of marker files or directories +.. [3] Nick Coghlan's objection to the lack of marker files or directories (http://mail.python.org/pipermail/import-sig/2012-March/000423.html) -.. [4] Nick Coglan's response to his initial objections +.. [4] Nick Coghlan's response to his initial objections (http://mail.python.org/pipermail/import-sig/2012-April/000464.html) .. [5] Phillip Eby's question about auto-updating __path__ -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 3 17:48:02 2012 From: python-checkins at python.org (barry.warsaw) Date: Thu, 03 May 2012 17:48:02 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_*_reST-ified?= Message-ID: http://hg.python.org/peps/rev/854a044893be changeset: 4350:854a044893be user: Barry Warsaw date: Thu May 03 11:47:57 2012 -0400 summary: * reST-ified * Remove long dead links * Update reference numbers files: pep-0302.txt | 838 +++++++++++++++++++------------------- 1 files changed, 409 insertions(+), 429 deletions(-) diff --git a/pep-0302.txt b/pep-0302.txt --- a/pep-0302.txt +++ b/pep-0302.txt @@ -6,279 +6,267 @@ Paul Moore Status: Final Type: Standards Track -Content-Type: text/plain +Content-Type: text/x-rst Created: 19-Dec-2002 Python-Version: 2.3 Post-History: 19-Dec-2002 Abstract +======== - This PEP proposes to add a new set of import hooks that offer better - customization of the Python import mechanism. Contrary to the - current __import__ hook, a new-style hook can be injected into the - existing scheme, allowing for a finer grained control of how modules - are found and how they are loaded. +This PEP proposes to add a new set of import hooks that offer better +customization of the Python import mechanism. Contrary to the current +``__import__`` hook, a new-style hook can be injected into the existing +scheme, allowing for a finer grained control of how modules are found and how +they are loaded. Motivation +========== - The only way to customize the import mechanism is currently to - override the built-in __import__ function. However, overriding - __import__ has many problems. To begin with: +The only way to customize the import mechanism is currently to override the +built-in ``__import__`` function. However, overriding ``__import__`` has many +problems. To begin with: - - An __import__ replacement needs to *fully* reimplement the entire - import mechanism, or call the original __import__ before or after - the custom code. + * An ``__import__`` replacement needs to *fully* reimplement the entire + import mechanism, or call the original ``__import__`` before or after the + custom code. - - It has very complex semantics and responsibilities. + * It has very complex semantics and responsibilities. - - __import__ gets called even for modules that are already in - sys.modules, which is almost never what you want, unless you're - writing some sort of monitoring tool. + * ``__import__`` gets called even for modules that are already in + ``sys.modules``, which is almost never what you want, unless you're writing + some sort of monitoring tool. - The situation gets worse when you need to extend the import - mechanism from C: it's currently impossible, apart from hacking - Python's import.c or reimplementing much of import.c from scratch. +The situation gets worse when you need to extend the import mechanism from C: +it's currently impossible, apart from hacking Python's ``import.c`` or +reimplementing much of ``import.c`` from scratch. - There is a fairly long history of tools written in Python that allow - extending the import mechanism in various way, based on the - __import__ hook. The Standard Library includes two such tools: - ihooks.py (by GvR) and imputil.py (Greg Stein), but perhaps the most - famous is iu.py by Gordon McMillan, available as part of his - Installer [1] package. Their usefulness is somewhat limited because - they are written in Python; bootstrapping issues need to worked - around as you can't load the module containing the hook with the - hook itself. So if you want the entire Standard Library to be - loadable from an import hook, the hook must be written in C. +There is a fairly long history of tools written in Python that allow extending +the import mechanism in various way, based on the ``__import__`` hook. The +Standard Library includes two such tools: ``ihooks.py`` (by GvR) and +``imputil.py`` [1]_ (Greg Stein), but perhaps the most famous is ``iu.py`` by +Gordon McMillan, available as part of his Installer package. Their usefulness +is somewhat limited because they are written in Python; bootstrapping issues +need to worked around as you can't load the module containing the hook with +the hook itself. So if you want the entire Standard Library to be loadable +from an import hook, the hook must be written in C. Use cases +========= - This section lists several existing applications that depend on - import hooks. Among these, a lot of duplicate work was done that - could have been saved if there had been a more flexible import hook - at the time. This PEP should make life a lot easier for similar - projects in the future. +This section lists several existing applications that depend on import hooks. +Among these, a lot of duplicate work was done that could have been saved if +there had been a more flexible import hook at the time. This PEP should make +life a lot easier for similar projects in the future. - Extending the import mechanism is needed when you want to load - modules that are stored in a non-standard way. Examples include - modules that are bundled together in an archive; byte code that is - not stored in a pyc formatted file; modules that are loaded from a - database over a network. +Extending the import mechanism is needed when you want to load modules that +are stored in a non-standard way. Examples include modules that are bundled +together in an archive; byte code that is not stored in a ``pyc`` formatted +file; modules that are loaded from a database over a network. - The work on this PEP was partly triggered by the implementation of - PEP 273 [2], which adds imports from Zip archives as a built-in - feature to Python. While the PEP itself was widely accepted as a - must-have feature, the implementation left a few things to desire. - For one thing it went through great lengths to integrate itself with - import.c, adding lots of code that was either specific for Zip file - imports or *not* specific to Zip imports, yet was not generally - useful (or even desirable) either. Yet the PEP 273 implementation - can hardly be blamed for this: it is simply extremely hard to do, - given the current state of import.c. +The work on this PEP was partly triggered by the implementation of PEP 273, +which adds imports from Zip archives as a built-in feature to Python. While +the PEP itself was widely accepted as a must-have feature, the implementation +left a few things to desire. For one thing it went through great lengths to +integrate itself with ``import.c``, adding lots of code that was either +specific for Zip file imports or *not* specific to Zip imports, yet was not +generally useful (or even desirable) either. Yet the PEP 273 implementation +can hardly be blamed for this: it is simply extremely hard to do, given the +current state of ``import.c``. - Packaging applications for end users is a typical use case for - import hooks, if not *the* typical use case. Distributing lots of - source or pyc files around is not always appropriate (let alone a - separate Python installation), so there is a frequent desire to - package all needed modules in a single file. So frequent in fact - that multiple solutions have been implemented over the years. +Packaging applications for end users is a typical use case for import hooks, +if not *the* typical use case. Distributing lots of source or ``pyc`` files +around is not always appropriate (let alone a separate Python installation), +so there is a frequent desire to package all needed modules in a single file. +So frequent in fact that multiple solutions have been implemented over the +years. - The oldest one is included with the Python source code: Freeze [3]. - It puts marshalled byte code into static objects in C source code. - Freeze's "import hook" is hard wired into import.c, and has a couple - of issues. Later solutions include Fredrik Lundh's Squeeze [4], - Gordon McMillan's Installer [1] and Thomas Heller's py2exe [5]. - MacPython ships with a tool called BuildApplication. +The oldest one is included with the Python source code: Freeze [2]_. It puts +marshalled byte code into static objects in C source code. Freeze's "import +hook" is hard wired into ``import.c``, and has a couple of issues. Later +solutions include Fredrik Lundh's Squeeze, Gordon McMillan's Installer, and +Thomas Heller's py2exe [3]_. MacPython ships with a tool called +``BuildApplication``. - Squeeze, Installer and py2exe use an __import__ based scheme (py2exe - currently uses Installer's iu.py, Squeeze used ihooks.py), MacPython - has two Mac-specific import hooks hard wired into import.c, that are - similar to the Freeze hook. The hooks proposed in this PEP enables - us (at least in theory; it's not a short term goal) to get rid of - the hard coded hooks in import.c, and would allow the - __import__-based tools to get rid of most of their import.c - emulation code. +Squeeze, Installer and py2exe use an ``__import__`` based scheme (py2exe +currently uses Installer's ``iu.py``, Squeeze used ``ihooks.py``), MacPython +has two Mac-specific import hooks hard wired into ``import.c``, that are +similar to the Freeze hook. The hooks proposed in this PEP enables us (at +least in theory; it's not a short term goal) to get rid of the hard coded +hooks in ``import.c``, and would allow the ``__import__``-based tools to get +rid of most of their ``import.c`` emulation code. - Before work on the design and implementation of this PEP was - started, a new BuildApplication-like tool for MacOS X prompted one - of the authors of this PEP (JvR) to expose the table of frozen - modules to Python, in the imp module. The main reason was to be - able to use the freeze import hook (avoiding fancy __import__ - support), yet to also be able to supply a set of modules at - runtime. This resulted in sf patch #642578 [6], which was - mysteriously accepted (mostly because nobody seemed to care either - way ;-). Yet it is completely superfluous when this PEP gets - accepted, as it offers a much nicer and general way to do the same - thing. +Before work on the design and implementation of this PEP was started, a new +``BuildApplication``-like tool for Mac OS X prompted one of the authors of +this PEP (JvR) to expose the table of frozen modules to Python, in the ``imp`` +module. The main reason was to be able to use the freeze import hook +(avoiding fancy ``__import__`` support), yet to also be able to supply a set +of modules at runtime. This resulted in issue #642578 [4]_, which was +mysteriously accepted (mostly because nobody seemed to care either way ;-). +Yet it is completely superfluous when this PEP gets accepted, as it offers a +much nicer and general way to do the same thing. Rationale +========= - While experimenting with alternative implementation ideas to get - built-in Zip import, it was discovered that achieving this is - possible with only a fairly small amount of changes to import.c. - This allowed to factor out the Zip-specific stuff into a new source - file, while at the same time creating a *general* new import hook - scheme: the one you're reading about now. +While experimenting with alternative implementation ideas to get built-in Zip +import, it was discovered that achieving this is possible with only a fairly +small amount of changes to ``import.c``. This allowed to factor out the +Zip-specific stuff into a new source file, while at the same time creating a +*general* new import hook scheme: the one you're reading about now. - An earlier design allowed non-string objects on sys.path. Such an - object would have the necessary methods to handle an import. This - has two disadvantages: 1) it breaks code that assumes all items on - sys.path are strings; 2) it is not compatible with the PYTHONPATH - environment variable. The latter is directly needed for Zip - imports. A compromise came from Jython: allow string *subclasses* - on sys.path, which would then act as importer objects. This avoids - some breakage, and seems to work well for Jython (where it is used - to load modules from .jar files), but it was perceived as an "ugly - hack". +An earlier design allowed non-string objects on ``sys.path``. Such an object +would have the necessary methods to handle an import. This has two +disadvantages: 1) it breaks code that assumes all items on ``sys.path`` are +strings; 2) it is not compatible with the ``PYTHONPATH`` environment variable. +The latter is directly needed for Zip imports. A compromise came from Jython: +allow string *subclasses* on ``sys.path``, which would then act as importer +objects. This avoids some breakage, and seems to work well for Jython (where +it is used to load modules from ``.jar`` files), but it was perceived as an +"ugly hack". - This lead to a more elaborate scheme, (mostly copied from McMillan's - iu.py) in which each in a list of candidates is asked whether it can - handle the sys.path item, until one is found that can. This list of - candidates is a new object in the sys module: sys.path_hooks. +This lead to a more elaborate scheme, (mostly copied from McMillan's +``iu.py``) in which each in a list of candidates is asked whether it can +handle the ``sys.path`` item, until one is found that can. This list of +candidates is a new object in the ``sys`` module: ``sys.path_hooks``. - Traversing sys.path_hooks for each path item for each new import can - be expensive, so the results are cached in another new object in the - sys module: sys.path_importer_cache. It maps sys.path entries to - importer objects. +Traversing ``sys.path_hooks`` for each path item for each new import can be +expensive, so the results are cached in another new object in the ``sys`` +module: ``sys.path_importer_cache``. It maps ``sys.path`` entries to importer +objects. - To minimize the impact on import.c as well as to avoid adding extra - overhead, it was chosen to not add an explicit hook and importer - object for the existing file system import logic (as iu.py has), but - to simply fall back to the built-in logic if no hook on - sys.path_hooks could handle the path item. If this is the case, a - None value is stored in sys.path_importer_cache, again to avoid - repeated lookups. (Later we can go further and add a real importer - object for the built-in mechanism, for now, the None fallback scheme - should suffice.) +To minimize the impact on ``import.c`` as well as to avoid adding extra +overhead, it was chosen to not add an explicit hook and importer object for +the existing file system import logic (as ``iu.py`` has), but to simply fall +back to the built-in logic if no hook on ``sys.path_hooks`` could handle the +path item. If this is the case, a ``None`` value is stored in +``sys.path_importer_cache``, again to avoid repeated lookups. (Later we can +go further and add a real importer object for the built-in mechanism, for now, +the ``None`` fallback scheme should suffice.) - A question was raised: what about importers that don't need *any* - entry on sys.path? (Built-in and frozen modules fall into that - category.) Again, Gordon McMillan to the rescue: iu.py contains a - thing he calls the "metapath". In this PEP's implementation, it's a - list of importer objects that is traversed *before* sys.path. This - list is yet another new object in the sys.module: sys.meta_path. - Currently, this list is empty by default, and frozen and built-in - module imports are done after traversing sys.meta_path, but still - before sys.path. +A question was raised: what about importers that don't need *any* entry on +``sys.path``? (Built-in and frozen modules fall into that category.) Again, +Gordon McMillan to the rescue: ``iu.py`` contains a thing he calls the +*metapath*. In this PEP's implementation, it's a list of importer objects +that is traversed *before* ``sys.path``. This list is yet another new object +in the ``sys`` module: ``sys.meta_path``. Currently, this list is empty by +default, and frozen and built-in module imports are done after traversing +``sys.meta_path``, but still before ``sys.path``. + Specification part 1: The Importer Protocol +=========================================== - This PEP introduces a new protocol: the "Importer Protocol". It is - important to understand the context in which the protocol operates, - so here is a brief overview of the outer shells of the import - mechanism. +This PEP introduces a new protocol: the "Importer Protocol". It is important +to understand the context in which the protocol operates, so here is a brief +overview of the outer shells of the import mechanism. - When an import statement is encountered, the interpreter looks up - the __import__ function in the built-in name space. __import__ is - then called with four arguments, amongst which are the name of the - module being imported (may be a dotted name) and a reference to the - current global namespace. +When an import statement is encountered, the interpreter looks up the +``__import__`` function in the built-in name space. ``__import__`` is then +called with four arguments, amongst which are the name of the module being +imported (may be a dotted name) and a reference to the current global +namespace. - The built-in __import__ function (known as PyImport_ImportModuleEx - in import.c) will then check to see whether the module doing the - import is a package or a submodule of a package. If it is indeed a - (submodule of a) package, it first tries to do the import relative - to the package (the parent package for a submodule). For example if - a package named "spam" does "import eggs", it will first look for a - module named "spam.eggs". If that fails, the import continues as an - absolute import: it will look for a module named "eggs". Dotted - name imports work pretty much the same: if package "spam" does - "import eggs.bacon" (and "spam.eggs" exists and is itself a - package), "spam.eggs.bacon" is tried. If that fails "eggs.bacon" is - tried. (There are more subtleties that are not described here, but - these are not relevant for implementers of the Importer Protocol.) +The built-in ``__import__`` function (known as ``PyImport_ImportModuleEx()`` +in ``import.c``) will then check to see whether the module doing the import is +a package or a submodule of a package. If it is indeed a (submodule of a) +package, it first tries to do the import relative to the package (the parent +package for a submodule). For example if a package named "spam" does "import +eggs", it will first look for a module named "spam.eggs". If that fails, the +import continues as an absolute import: it will look for a module named +"eggs". Dotted name imports work pretty much the same: if package "spam" does +"import eggs.bacon" (and "spam.eggs" exists and is itself a package), +"spam.eggs.bacon" is tried. If that fails "eggs.bacon" is tried. (There are +more subtleties that are not described here, but these are not relevant for +implementers of the Importer Protocol.) - Deeper down in the mechanism, a dotted name import is split up by - its components. For "import spam.ham", first an "import spam" is - done, and only when that succeeds is "ham" imported as a submodule - of "spam". +Deeper down in the mechanism, a dotted name import is split up by its +components. For "import spam.ham", first an "import spam" is done, and only +when that succeeds is "ham" imported as a submodule of "spam". - The Importer Protocol operates at this level of *individual* - imports. By the time an importer gets a request for "spam.ham", - module "spam" has already been imported. +The Importer Protocol operates at this level of *individual* imports. By the +time an importer gets a request for "spam.ham", module "spam" has already been +imported. - The protocol involves two objects: a finder and a loader. A - finder object has a single method: +The protocol involves two objects: a *finder* and a *loader*. A finder object +has a single method:: - finder.find_module(fullname, path=None) + finder.find_module(fullname, path=None) - This method will be called with the fully qualified name of the - module. If the finder is installed on sys.meta_path, it will - receive a second argument, which is None for a top-level module, or - package.__path__ for submodules or subpackages[7]. It should return - a loader object if the module was found, or None if it wasn't. If - find_module() raises an exception, it will be propagated to the - caller, aborting the import. +This method will be called with the fully qualified name of the module. If +the finder is installed on ``sys.meta_path``, it will receive a second +argument, which is ``None`` for a top-level module, or ``package.__path__`` +for submodules or subpackages [5]_. It should return a loader object if the +module was found, or ``None`` if it wasn't. If ``find_module()`` raises an +exception, it will be propagated to the caller, aborting the import. - A loader object also has one method: +A loader object also has one method:: - loader.load_module(fullname) + loader.load_module(fullname) - This method returns the loaded module or raises an exception, - preferably ImportError if an existing exception is not being - propagated. If load_module() is asked to load a module that it - cannot, ImportError is to be raised. +This method returns the loaded module or raises an exception, preferably +``ImportError`` if an existing exception is not being propagated. If +``load_module()`` is asked to load a module that it cannot, ``ImportError`` is +to be raised. - In many cases the finder and loader can be one and the same - object: finder.find_module() would just return self. +In many cases the finder and loader can be one and the same object: +``finder.find_module()`` would just return ``self``. - The 'fullname' argument of both methods is the fully qualified - module name, for example "spam.eggs.ham". As explained above, when - finder.find_module("spam.eggs.ham") is called, "spam.eggs" has - already been imported and added to sys.modules. However, the - find_module() method isn't necessarily always called during an - actual import: meta tools that analyze import dependencies (such as - freeze, Installer or py2exe) don't actually load modules, so an - finder shouldn't *depend* on the parent package being available in - sys.modules. +The ``fullname`` argument of both methods is the fully qualified module name, +for example "spam.eggs.ham". As explained above, when +``finder.find_module("spam.eggs.ham")`` is called, "spam.eggs" has already +been imported and added to ``sys.modules``. However, the ``find_module()`` +method isn't necessarily always called during an actual import: meta tools +that analyze import dependencies (such as freeze, Installer or py2exe) don't +actually load modules, so a finder shouldn't *depend* on the parent package +being available in ``sys.modules``. - The load_module() method has a few responsibilities that it must - fulfill *before* it runs any code: +The ``load_module()`` method has a few responsibilities that it must fulfill +*before* it runs any code: - - If there is an existing module object named 'fullname' in - sys.modules, the loader must use that existing module. - (Otherwise, the reload() builtin will not work correctly.) - If a module named 'fullname' does not exist in sys.modules, - the loader must create a new module object and add it to - sys.modules. + * If there is an existing module object named 'fullname' in ``sys.modules``, + the loader must use that existing module. (Otherwise, the ``reload()`` + builtin will not work correctly.) If a module named 'fullname' does not + exist in ``sys.modules``, the loader must create a new module object and + add it to ``sys.modules``. - Note that the module object *must* be in sys.modules before the - loader executes the module code. This is crucial because the - module code may (directly or indirectly) import itself; adding - it to sys.modules beforehand prevents unbounded recursion in the - worst case and multiple loading in the best. + Note that the module object *must* be in ``sys.modules`` before the loader + executes the module code. This is crucial because the module code may + (directly or indirectly) import itself; adding it to ``sys.modules`` + beforehand prevents unbounded recursion in the worst case and multiple + loading in the best. - If the load fails, the loader needs to remove any module it may have - inserted into sys.modules. If the module was already in - sys.modules then the loader should leave it alone. + If the load fails, the loader needs to remove any module it may have + inserted into ``sys.modules``. If the module was already in ``sys.modules`` + then the loader should leave it alone. - - The __file__ attribute must be set. This must be a string, but it - may be a dummy value, for example "". The privilege of - not having a __file__ attribute at all is reserved for built-in - modules. + * The ``__file__`` attribute must be set. This must be a string, but it may + be a dummy value, for example "". The privilege of not having a + ``__file__`` attribute at all is reserved for built-in modules. - - The __name__ attribute must be set. If one uses - imp.new_module() then the attribute is set automatically. + * The ``__name__`` attribute must be set. If one uses ``imp.new_module()`` + then the attribute is set automatically. - - If it's a package, the __path__ variable must be set. This must - be a list, but may be empty if __path__ has no further - significance to the importer (more on this later). + * If it's a package, the ``__path__`` variable must be set. This must be a + list, but may be empty if ``__path__`` has no further significance to the + importer (more on this later). - - The __loader__ attribute must be set to the loader object. - This is mostly for introspection and reloading, but can be used - for importer-specific extras, for example getting data associated - with an importer. + * The ``__loader__`` attribute must be set to the loader object. This is + mostly for introspection and reloading, but can be used for + importer-specific extras, for example getting data associated with an + importer. - - The __package__ attribute [10] must be set. + * The ``__package__`` attribute [8]_ must be set. - If the module is a Python module (as opposed to a built-in module or - a dynamically loaded extension), it should execute the module's code - in the module's global name space (module.__dict__). + If the module is a Python module (as opposed to a built-in module or a + dynamically loaded extension), it should execute the module's code in the + module's global name space (``module.__dict__``). - Here is a minimal pattern for a load_module() method: + Here is a minimal pattern for a ``load_module()`` method:: # Consider using importlib.util.module_for_loader() to handle # most of these details for you. @@ -298,294 +286,286 @@ Specification part 2: Registering Hooks +======================================= - There are two types of import hooks: Meta hooks and Path hooks. - Meta hooks are called at the start of import processing, before any - other import processing (so that meta hooks can override sys.path - processing, or frozen modules, or even built-in modules). To - register a meta hook, simply add the finder object to - sys.meta_path (the list of registered meta hooks). +There are two types of import hooks: *Meta hooks* and *Path hooks*. Meta +hooks are called at the start of import processing, before any other import +processing (so that meta hooks can override ``sys.path`` processing, frozen +modules, or even built-in modules). To register a meta hook, simply add the +finder object to ``sys.meta_path`` (the list of registered meta hooks). - Path hooks are called as part of sys.path (or package.__path__) - processing, at the point where their associated path item is - encountered. A path hook is registered by adding an importer - factory to sys.path_hooks. +Path hooks are called as part of ``sys.path`` (or ``package.__path__``) +processing, at the point where their associated path item is encountered. A +path hook is registered by adding an importer factory to ``sys.path_hooks``. - sys.path_hooks is a list of callables, which will be checked in - sequence to determine if they can handle a given path item. The - callable is called with one argument, the path item. The callable - must raise ImportError if it is unable to handle the path item, and - return an importer object if it can handle the path item. Note - that if the callable returns an importer object for a specific - sys.path entry, the builtin import machinery will not be invoked - to handle that entry any longer, even if the importer object later - fails to find a specific module. The callable is typically the - class of the import hook, and hence the class __init__ method is - called. (This is also the reason why it should raise ImportError: - an __init__ method can't return anything. This would be possible - with a __new__ method in a new style class, but we don't want to - require anything about how a hook is implemented.) +``sys.path_hooks`` is a list of callables, which will be checked in sequence +to determine if they can handle a given path item. The callable is called +with one argument, the path item. The callable must raise ``ImportError`` if +it is unable to handle the path item, and return an importer object if it can +handle the path item. Note that if the callable returns an importer object +for a specific ``sys.path`` entry, the builtin import machinery will not be +invoked to handle that entry any longer, even if the importer object later +fails to find a specific module. The callable is typically the class of the +import hook, and hence the class ``__init__()`` method is called. (This is +also the reason why it should raise ``ImportError``: an ``__init__()`` method +can't return anything. This would be possible with a ``__new__()`` method in +a new style class, but we don't want to require anything about how a hook is +implemented.) - The results of path hook checks are cached in - sys.path_importer_cache, which is a dictionary mapping path entries - to importer objects. The cache is checked before sys.path_hooks is - scanned. If it is necessary to force a rescan of sys.path_hooks, it - is possible to manually clear all or part of - sys.path_importer_cache. +The results of path hook checks are cached in ``sys.path_importer_cache``, +which is a dictionary mapping path entries to importer objects. The cache is +checked before ``sys.path_hooks`` is scanned. If it is necessary to force a +rescan of ``sys.path_hooks``, it is possible to manually clear all or part of +``sys.path_importer_cache``. - Just like sys.path itself, the new sys variables must have specific - types: +Just like ``sys.path`` itself, the new ``sys`` variables must have specific +types: - sys.meta_path and sys.path_hooks must be Python lists. - sys.path_importer_cache must be a Python dict. + * ``sys.meta_path`` and ``sys.path_hooks`` must be Python lists. + * ``sys.path_importer_cache`` must be a Python dict. - Modifying these variables in place is allowed, as is replacing them - with new objects. +Modifying these variables in place is allowed, as is replacing them with new +objects. -Packages and the role of __path__ +Packages and the role of ``__path__`` +===================================== - If a module has a __path__ attribute, the import mechanism will - treat it as a package. The __path__ variable is used instead of - sys.path when importing submodules of the package. The rules for - sys.path therefore also apply to pkg.__path__. So sys.path_hooks is - also consulted when pkg.__path__ is traversed. Meta importers don't - necessarily use sys.path at all to do their work and may therefore - ignore the value of pkg.__path__. In this case it is still advised - to set it to list, which can be empty. +If a module has a ``__path__`` attribute, the import mechanism will treat it +as a package. The ``__path__`` variable is used instead of ``sys.path`` when +importing submodules of the package. The rules for ``sys.path`` therefore +also apply to ``pkg.__path__``. So ``sys.path_hooks`` is also consulted when +``pkg.__path__`` is traversed. Meta importers don't necessarily use +``sys.path`` at all to do their work and may therefore ignore the value of +``pkg.__path__``. In this case it is still advised to set it to list, which +can be empty. Optional Extensions to the Importer Protocol +============================================ - The Importer Protocol defines three optional extensions. One is to - retrieve data files, the second is to support module packaging tools - and/or tools that analyze module dependencies (for example Freeze - [3]), while the last is to support execution of modules as scripts. - The latter two categories of tools usually don't actually *load* - modules, they only need to know if and where they are available. - All three extensions are highly recommended for general purpose - importers, but may safely be left out if those features aren't - needed. +The Importer Protocol defines three optional extensions. One is to retrieve +data files, the second is to support module packaging tools and/or tools that +analyze module dependencies (for example Freeze), while the last is to support +execution of modules as scripts. The latter two categories of tools usually +don't actually *load* modules, they only need to know if and where they are +available. All three extensions are highly recommended for general purpose +importers, but may safely be left out if those features aren't needed. - To retrieve the data for arbitrary "files" from the underlying - storage backend, loader objects may supply a method named get_data: +To retrieve the data for arbitrary "files" from the underlying storage +backend, loader objects may supply a method named ``get_data()``:: - loader.get_data(path) + loader.get_data(path) - This method returns the data as a string, or raise IOError if the - "file" wasn't found. The data is always returned as if "binary" mode - was used - there is no CRLF translation of text files, for example. - It is meant for importers that have some file-system-like properties. - The 'path' argument is a path that can be constructed by munging - module.__file__ (or pkg.__path__ items) with the os.path.* functions, - for example: +This method returns the data as a string, or raise ``IOError`` if the "file" +wasn't found. The data is always returned as if "binary" mode was used - +there is no CRLF translation of text files, for example. It is meant for +importers that have some file-system-like properties. The 'path' argument is +a path that can be constructed by munging ``module.__file__`` (or +``pkg.__path__`` items) with the ``os.path.*`` functions, for example:: - d = os.path.dirname(__file__) - data = __loader__.get_data(os.path.join(d, "logo.gif")) + d = os.path.dirname(__file__) + data = __loader__.get_data(os.path.join(d, "logo.gif")) - The following set of methods may be implemented if support for (for - example) Freeze-like tools is desirable. It consists of three - additional methods which, to make it easier for the caller, each of - which should be implemented, or none at all. +The following set of methods may be implemented if support for (for example) +Freeze-like tools is desirable. It consists of three additional methods +which, to make it easier for the caller, each of which should be implemented, +or none at all:: - loader.is_package(fullname) - loader.get_code(fullname) - loader.get_source(fullname) + loader.is_package(fullname) + loader.get_code(fullname) + loader.get_source(fullname) - All three methods should raise ImportError if the module wasn't - found. +All three methods should raise ``ImportError`` if the module wasn't found. - The loader.is_package(fullname) method should return True if the - module specified by 'fullname' is a package and False if it isn't. +The ``loader.is_package(fullname)`` method should return ``True`` if the +module specified by 'fullname' is a package and ``False`` if it isn't. - The loader.get_code(fullname) method should return the code object - associated with the module, or None if it's a built-in or extension - module. If the loader doesn't have the code object but it _does_ - have the source code, it should return the compiled source code. - (This is so that our caller doesn't also need to check get_source() - if all it needs is the code object.) +The ``loader.get_code(fullname)`` method should return the code object +associated with the module, or ``None`` if it's a built-in or extension +module. If the loader doesn't have the code object but it *does* have the +source code, it should return the compiled source code. (This is so that our +caller doesn't also need to check ``get_source()`` if all it needs is the code +object.) - The loader.get_source(fullname) method should return the source code - for the module as a string (using newline characters for line - endings) or None if the source is not available (yet it should still - raise ImportError if the module can't be found by the importer at - all). +The ``loader.get_source(fullname)`` method should return the source code for +the module as a string (using newline characters for line endings) or ``None`` +if the source is not available (yet it should still raise ``ImportError`` if +the module can't be found by the importer at all). - To support execution of modules as scripts [9], the above three - methods for finding the code associated with a module must be - implemented. In addition to those methods, the following method - may be provided in order to allow the ``runpy`` module to correctly - set the ``__file__`` attribute: +To support execution of modules as scripts [6]_, the above three methods for +finding the code associated with a module must be implemented. In addition to +those methods, the following method may be provided in order to allow the +``runpy`` module to correctly set the ``__file__`` attribute:: - loader.get_filename(fullname) + loader.get_filename(fullname) - This method should return the value that ``__file__`` would be set - to if the named module was loaded. If the module is not found, then - ImportError should be raised. +This method should return the value that ``__file__`` would be set to if the +named module was loaded. If the module is not found, then ``ImportError`` +should be raised. Integration with the 'imp' module +================================= - The new import hooks are not easily integrated in the existing - imp.find_module() and imp.load_module() calls. It's questionable - whether it's possible at all without breaking code; it is better to - simply add a new function to the imp module. The meaning of the - existing imp.find_module() and imp.load_module() calls changes from: - "they expose the built-in import mechanism" to "they expose the - basic *unhooked* built-in import mechanism". They simply won't - invoke any import hooks. A new imp module function is proposed (but - not yet implemented) under the name "get_loader", which is used as - in the following pattern: +The new import hooks are not easily integrated in the existing +``imp.find_module()`` and ``imp.load_module()`` calls. It's questionable +whether it's possible at all without breaking code; it is better to simply add +a new function to the ``imp`` module. The meaning of the existing +``imp.find_module()`` and ``imp.load_module()`` calls changes from: "they +expose the built-in import mechanism" to "they expose the basic *unhooked* +built-in import mechanism". They simply won't invoke any import hooks. A new +``imp`` module function is proposed (but not yet implemented) under the name +``get_loader()``, which is used as in the following pattern:: - loader = imp.get_loader(fullname, path) - if loader is not None: - loader.load_module(fullname) + loader = imp.get_loader(fullname, path) + if loader is not None: + loader.load_module(fullname) - In the case of a "basic" import, one the imp.find_module() function - would handle, the loader object would be a wrapper for the current - output of imp.find_module(), and loader.load_module() would call - imp.load_module() with that output. +In the case of a "basic" import, one the `imp.find_module()` function would +handle, the loader object would be a wrapper for the current output of +``imp.find_module()``, and ``loader.load_module()`` would call +``imp.load_module()`` with that output. - Note that this wrapper is currently not yet implemented, although a - Python prototype exists in the test_importhooks.py script (the - ImpWrapper class) included with the patch. +Note that this wrapper is currently not yet implemented, although a Python +prototype exists in the ``test_importhooks.py`` script (the ``ImpWrapper`` +class) included with the patch. Forward Compatibility +===================== - Existing __import__ hooks will not invoke new-style hooks by magic, - unless they call the original __import__ function as a fallback. - For example, ihooks.py, iu.py and imputil.py are in this sense not - forward compatible with this PEP. +Existing ``__import__`` hooks will not invoke new-style hooks by magic, unless +they call the original ``__import__`` function as a fallback. For example, +``ihooks.py``, ``iu.py`` and ``imputil.py`` are in this sense not forward +compatible with this PEP. Open Issues +=========== - Modules often need supporting data files to do their job, - particularly in the case of complex packages or full applications. - Current practice is generally to locate such files via sys.path (or - a package.__path__ attribute). This approach will not work, in - general, for modules loaded via an import hook. +Modules often need supporting data files to do their job, particularly in the +case of complex packages or full applications. Current practice is generally +to locate such files via ``sys.path`` (or a ``package.__path__`` attribute). +This approach will not work, in general, for modules loaded via an import +hook. - There are a number of possible ways to address this problem: +There are a number of possible ways to address this problem: - - "Don't do that". If a package needs to locate data files via its - __path__, it is not suitable for loading via an import hook. The - package can still be located on a directory in sys.path, as at - present, so this should not be seen as a major issue. + * "Don't do that". If a package needs to locate data files via its + ``__path__``, it is not suitable for loading via an import hook. The + package can still be located on a directory in ``sys.path``, as at present, + so this should not be seen as a major issue. - - Locate data files from a standard location, rather than relative - to the module file. A relatively simple approach (which is - supported by distutils) would be to locate data files based on - sys.prefix (or sys.exec_prefix). For example, looking in - os.path.join(sys.prefix, "data", package_name). + * Locate data files from a standard location, rather than relative to the + module file. A relatively simple approach (which is supported by + distutils) would be to locate data files based on ``sys.prefix`` (or + ``sys.exec_prefix``). For example, looking in + ``os.path.join(sys.prefix, "data", package_name)``. - - Import hooks could offer a standard way of getting at data files - relative to the module file. The standard zipimport object - provides a method get_data(name) which returns the content of the - "file" called name, as a string. To allow modules to get at the - importer object, zipimport also adds an attribute "__loader__" - to the module, containing the zipimport object used to load the - module. If such an approach is used, it is important that client - code takes care not to break if the get_data method is not available, - so it is not clear that this approach offers a general answer to the - problem. + * Import hooks could offer a standard way of getting at data files relative + to the module file. The standard ``zipimport`` object provides a method + ``get_data(name)`` which returns the content of the "file" called ``name``, + as a string. To allow modules to get at the importer object, ``zipimport`` + also adds an attribute ``__loader__`` to the module, containing the + ``zipimport`` object used to load the module. If such an approach is used, + it is important that client code takes care not to break if the + ``get_data()`` method is not available, so it is not clear that this + approach offers a general answer to the problem. - It was suggested on python-dev that it would be useful to be able to - receive a list of available modules from an importer and/or a list - of available data files for use with the get_data() method. The - protocol could grow two additional extensions, say list_modules() - and list_files(). The latter makes sense on loader objects with a - get_data() method. However, it's a bit unclear which object should - implement list_modules(): the importer or the loader or both? +It was suggested on python-dev that it would be useful to be able to receive a +list of available modules from an importer and/or a list of available data +files for use with the ``get_data()`` method. The protocol could grow two +additional extensions, say ``list_modules()`` and ``list_files()``. The +latter makes sense on loader objects with a ``get_data()`` method. However, +it's a bit unclear which object should implement ``list_modules()``: the +importer or the loader or both? - This PEP is biased towards loading modules from alternative places: - it currently doesn't offer dedicated solutions for loading modules - from alternative file formats or with alternative compilers. In - contrast, the ihooks module from the standard library does have a - fairly straightforward way to do this. The Quixote project [8] uses - this technique to import PTL files as if they are ordinary Python - modules. To do the same with the new hooks would either mean to add - a new module implementing a subset of ihooks as a new-style - importer, or add a hookable built-in path importer object. +This PEP is biased towards loading modules from alternative places: it +currently doesn't offer dedicated solutions for loading modules from +alternative file formats or with alternative compilers. In contrast, the +``ihooks`` module from the standard library does have a fairly straightforward +way to do this. The Quixote project [7]_ uses this technique to import PTL +files as if they are ordinary Python modules. To do the same with the new +hooks would either mean to add a new module implementing a subset of +``ihooks`` as a new-style importer, or add a hookable built-in path importer +object. - There is no specific support within this PEP for "stacking" hooks. - For example, it is not obvious how to write a hook to load modules - from ..tar.gz files by combining separate hooks to load modules from - .tar and ..gz files. However, there is no support for such stacking - in the existing hook mechanisms (either the basic "replace - __import__" method, or any of the existing import hook modules) and - so this functionality is not an obvious requirement of the new - mechanism. It may be worth considering as a future enhancement, - however. +There is no specific support within this PEP for "stacking" hooks. For +example, it is not obvious how to write a hook to load modules from ``tar.gz`` +files by combining separate hooks to load modules from ``.tar`` and ``.gz`` +files. However, there is no support for such stacking in the existing hook +mechanisms (either the basic "replace ``__import__``" method, or any of the +existing import hook modules) and so this functionality is not an obvious +requirement of the new mechanism. It may be worth considering as a future +enhancement, however. - It is possible (via sys.meta_path) to add hooks which run before - sys.path is processed. However, there is no equivalent way of - adding hooks to run after sys.path is processed. For now, if a hook - is required after sys.path has been processed, it can be simulated - by adding an arbitrary "cookie" string at the end of sys.path, and - having the required hook associated with this cookie, via the normal - sys.path_hooks processing. In the longer term, the path handling - code will become a "real" hook on sys.meta_path, and at that stage - it will be possible to insert user-defined hooks either before or - after it. +It is possible (via ``sys.meta_path``) to add hooks which run before +``sys.path`` is processed. However, there is no equivalent way of adding +hooks to run after ``sys.path`` is processed. For now, if a hook is required +after ``sys.path`` has been processed, it can be simulated by adding an +arbitrary "cookie" string at the end of ``sys.path``, and having the required +hook associated with this cookie, via the normal ``sys.path_hooks`` +processing. In the longer term, the path handling code will become a "real" +hook on ``sys.meta_path``, and at that stage it will be possible to insert +user-defined hooks either before or after it. Implementation +============== - The PEP 302 implementation has been integrated with Python as of - 2.3a1. An earlier version is available as SourceForge patch - #652586, but more interestingly, the SF item contains a fairly - detailed history of the development and design. - http://www.python.org/sf/652586 +The PEP 302 implementation has been integrated with Python as of 2.3a1. An +earlier version is available as patch #652586 [9]_, but more interestingly, +the issue contains a fairly detailed history of the development and design. - PEP 273 has been implemented using PEP 302's import hooks. +PEP 273 has been implemented using PEP 302's import hooks. References and Footnotes +======================== - [1] Installer by Gordon McMillan - http://www.mcmillan-inc.com/install1.html +.. [1] imputil module + http://docs.python.org/library/imputil.html - [2] PEP 273, Import Modules from Zip Archives, Ahlstrom - http://www.python.org/dev/peps/pep-0273/ +.. [2] The Freeze tool. + See also the ``Tools/freeze/`` directory in a Python source distribution - [3] The Freeze tool - Tools/freeze/ in a Python source distribution +.. [3] py2exe by Thomas Heller + http://www.py2exe.org/ - [4] Squeeze - http://starship.python.net/crew/fredrik/ipa/squeeze.htm +.. [4] imp.set_frozenmodules() patch + http://bugs.python.org/issue642578 - [5] py2exe by Thomas Heller - http://py2exe.sourceforge.net/ +.. [5] The path argument to ``finder.find_module()`` is there because the + ``pkg.__path__`` variable may be needed at this point. It may either come + from the actual parent module or be supplied by ``imp.find_module()`` or + the proposed ``imp.get_loader()`` function. - [6] imp.set_frozenmodules() patch - http://www.python.org/sf/642578 +.. [6] PEP 338: Executing modules as scripts + http://www.python.org/dev/peps/pep-0338/ - [7] The path argument to finder.find_module() is there because the - pkg.__path__ variable may be needed at this point. It may either - come from the actual parent module or be supplied by - imp.find_module() or the proposed imp.get_loader() function. +.. [7] Quixote, a framework for developing Web applications + http://www.mems-exchange.org/software/quixote/ - [8] Quixote, a framework for developing Web applications - http://www.mems-exchange.org/software/quixote/ +.. [8] PEP 366: Main module explicit relative imports + http://www.python.org/dev/peps/pep-0366/ - [9] PEP 338: Executing modules as scripts - http://www.python.org/dev/peps/pep-0338/ - - [10] PEP 366: Main module explicit relative imports - http://www.python.org/dev/peps/pep-0366/ +.. [9] New import hooks + Import from Zip files + http://bugs.python.org/issue652586 Copyright +========= - This document has been placed in the public domain. +This document has been placed in the public domain. -Local Variables: -mode: indented-text -indent-tabs-mode: nil -sentence-end-double-space: t -fill-column: 70 -End: +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 3 18:21:58 2012 From: python-checkins at python.org (ezio.melotti) Date: Thu, 03 May 2012 18:21:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Use_Python_3_in?= =?utf8?q?stead_of_3=2E0=2E?= Message-ID: http://hg.python.org/cpython/rev/8aab13719c02 changeset: 76732:8aab13719c02 branch: 2.7 parent: 76727:e606c123244f user: Ezio Melotti date: Thu May 03 19:21:40 2012 +0300 summary: Use Python 3 instead of 3.0. files: Doc/c-api/buffer.rst | 2 +- Doc/c-api/objbuffer.rst | 2 +- Doc/c-api/typeobj.rst | 2 +- Doc/glossary.rst | 2 +- Doc/howto/sorting.rst | 4 ++-- Doc/library/2to3.rst | 2 +- Doc/library/_winreg.rst | 4 ++-- Doc/library/al.rst | 4 ++-- Doc/library/anydbm.rst | 4 ++-- Doc/library/basehttpserver.rst | 4 ++-- Doc/library/bastion.rst | 2 +- Doc/library/binascii.rst | 2 +- Doc/library/bsddb.rst | 4 ++-- Doc/library/carbon.rst | 2 +- Doc/library/cd.rst | 2 +- Doc/library/cgihttpserver.rst | 4 ++-- Doc/library/commands.rst | 2 +- Doc/library/compiler.rst | 2 +- Doc/library/configparser.rst | 4 ++-- Doc/library/cookie.rst | 4 ++-- Doc/library/cookielib.rst | 4 ++-- Doc/library/copy_reg.rst | 4 ++-- Doc/library/dbhash.rst | 2 +- Doc/library/dbm.rst | 4 ++-- Doc/library/dircache.rst | 2 +- Doc/library/dl.rst | 2 +- Doc/library/docxmlrpcserver.rst | 4 ++-- Doc/library/dumbdbm.rst | 4 ++-- Doc/library/dummy_thread.rst | 4 ++-- Doc/library/fl.rst | 6 +++--- Doc/library/fm.rst | 2 +- Doc/library/fpformat.rst | 2 +- Doc/library/gdbm.rst | 4 ++-- Doc/library/gl.rst | 6 +++--- Doc/library/htmllib.rst | 6 +++--- Doc/library/httplib.rst | 4 ++-- Doc/library/imageop.rst | 2 +- Doc/library/imgfile.rst | 2 +- Doc/library/imputil.rst | 2 +- Doc/library/jpeg.rst | 2 +- Doc/library/macostools.rst | 2 +- Doc/library/mailbox.rst | 2 +- Doc/library/mhlib.rst | 2 +- Doc/library/mutex.rst | 2 +- Doc/library/new.rst | 2 +- Doc/library/os.path.rst | 2 +- Doc/library/parser.rst | 2 +- Doc/library/queue.rst | 4 ++-- Doc/library/repr.rst | 4 ++-- Doc/library/rexec.rst | 2 +- Doc/library/rfc822.rst | 2 +- Doc/library/robotparser.rst | 4 ++-- Doc/library/scrolledtext.rst | 4 ++-- Doc/library/sgmllib.rst | 2 +- Doc/library/simplehttpserver.rst | 4 ++-- Doc/library/simplexmlrpcserver.rst | 4 ++-- Doc/library/socketserver.rst | 4 ++-- Doc/library/statvfs.rst | 2 +- Doc/library/stdtypes.rst | 2 +- Doc/library/string.rst | 2 +- Doc/library/sunaudio.rst | 4 ++-- Doc/library/sys.rst | 4 ++-- Doc/library/tarfile.rst | 2 +- Doc/library/thread.rst | 4 ++-- Doc/library/tix.rst | 4 ++-- Doc/library/tkinter.rst | 6 +++--- Doc/library/urllib.rst | 8 ++++---- Doc/library/urllib2.rst | 4 ++-- Doc/library/urlparse.rst | 4 ++-- Doc/library/user.rst | 2 +- Doc/library/userdict.rst | 10 +++++----- Doc/library/warnings.rst | 2 +- Doc/library/whichdb.rst | 4 ++-- Doc/library/xmlrpclib.rst | 4 ++-- Doc/reference/datamodel.rst | 6 +++--- Doc/reference/expressions.rst | 2 +- 76 files changed, 125 insertions(+), 125 deletions(-) diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst --- a/Doc/c-api/buffer.rst +++ b/Doc/c-api/buffer.rst @@ -33,7 +33,7 @@ Starting from version 1.6, Python has been providing Python-level buffer objects and a C-level buffer API so that any built-in or used-defined type can expose its characteristics. Both, however, have been deprecated because of -various shortcomings, and have been officially removed in Python 3.0 in favour +various shortcomings, and have been officially removed in Python 3 in favour of a new C-level buffer API and a new Python-level object named :class:`memoryview`. diff --git a/Doc/c-api/objbuffer.rst b/Doc/c-api/objbuffer.rst --- a/Doc/c-api/objbuffer.rst +++ b/Doc/c-api/objbuffer.rst @@ -8,7 +8,7 @@ This section describes the legacy buffer protocol, which has been introduced in Python 1.6. It is still supported but deprecated in the Python 2.x series. -Python 3.0 introduces a new buffer protocol which fixes weaknesses and +Python 3 introduces a new buffer protocol which fixes weaknesses and shortcomings of the protocol, and has been backported to Python 2.6. See :ref:`bufferobjects` for more information. diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -1227,7 +1227,7 @@ - If the :const:`Py_TPFLAGS_CHECKTYPES` flag is set, binary and ternary functions must check the type of all their operands, and implement the necessary conversions (at least one of the operands is an instance of the - defined type). This is the recommended way; with Python 3.0 coercion will + defined type). This is the recommended way; with Python 3 coercion will disappear completely. If the operation is not defined for the given operands, binary and ternary diff --git a/Doc/glossary.rst b/Doc/glossary.rst --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -80,7 +80,7 @@ classic class Any class which does not inherit from :class:`object`. See - :term:`new-style class`. Classic classes will be removed in Python 3.0. + :term:`new-style class`. Classic classes have been removed in Python 3. coercion The implicit conversion of an instance of one type to another during an diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -210,11 +210,11 @@ arguments. Instead, all of the Py2.x versions supported a *cmp* parameter to handle user specified comparison functions. -In Py3.0, the *cmp* parameter was removed entirely (as part of a larger effort to +In Python 3, the *cmp* parameter was removed entirely (as part of a larger effort to simplify and unify the language, eliminating the conflict between rich comparisons and the :meth:`__cmp__` magic method). -In Py2.x, sort allowed an optional function which can be called for doing the +In Python 2, :meth:`~list.sort` allowed an optional function which can be called for doing the comparisons. That function should take two arguments to be compared and then return a negative value for less-than, return zero if they are equal, or return a positive value for greater-than. For example, we can do: diff --git a/Doc/library/2to3.rst b/Doc/library/2to3.rst --- a/Doc/library/2to3.rst +++ b/Doc/library/2to3.rst @@ -314,7 +314,7 @@ Converts ``raise E, V`` to ``raise E(V)``, and ``raise E, V, T`` to ``raise E(V).with_traceback(T)``. If ``E`` is a tuple, the translation will be - incorrect because substituting tuples for exceptions has been removed in 3.0. + incorrect because substituting tuples for exceptions has been removed in Python 3. .. 2to3fixer:: raw_input diff --git a/Doc/library/_winreg.rst b/Doc/library/_winreg.rst --- a/Doc/library/_winreg.rst +++ b/Doc/library/_winreg.rst @@ -7,9 +7,9 @@ .. sectionauthor:: Mark Hammond .. note:: - The :mod:`_winreg` module has been renamed to :mod:`winreg` in Python 3.0. + The :mod:`_winreg` module has been renamed to :mod:`winreg` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. versionadded:: 2.0 diff --git a/Doc/library/al.rst b/Doc/library/al.rst --- a/Doc/library/al.rst +++ b/Doc/library/al.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`al` module has been deprecated for removal in Python 3.0. + The :mod:`al` module has been removed in Python 3. This module provides access to the audio facilities of the SGI Indy and Indigo @@ -201,7 +201,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`AL` module has been deprecated for removal in Python 3.0. + The :mod:`AL` module has been removed in Python 3. This module defines symbolic constants needed to use the built-in module diff --git a/Doc/library/anydbm.rst b/Doc/library/anydbm.rst --- a/Doc/library/anydbm.rst +++ b/Doc/library/anydbm.rst @@ -6,9 +6,9 @@ .. note:: - The :mod:`anydbm` module has been renamed to :mod:`dbm` in Python 3.0. The + The :mod:`anydbm` module has been renamed to :mod:`dbm` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. index:: module: dbhash diff --git a/Doc/library/basehttpserver.rst b/Doc/library/basehttpserver.rst --- a/Doc/library/basehttpserver.rst +++ b/Doc/library/basehttpserver.rst @@ -6,8 +6,8 @@ .. note:: The :mod:`BaseHTTPServer` module has been merged into :mod:`http.server` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. .. index:: diff --git a/Doc/library/bastion.rst b/Doc/library/bastion.rst --- a/Doc/library/bastion.rst +++ b/Doc/library/bastion.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`Bastion` module has been removed in Python 3.0. + The :mod:`Bastion` module has been removed in Python 3. .. moduleauthor:: Barry Warsaw diff --git a/Doc/library/binascii.rst b/Doc/library/binascii.rst --- a/Doc/library/binascii.rst +++ b/Doc/library/binascii.rst @@ -127,7 +127,7 @@ The return value is in the range [-2**31, 2**31-1] regardless of platform. In the past the value would be signed on some platforms and unsigned on others. Use & 0xffffffff on the - value if you want it to match 3.0 behavior. + value if you want it to match Python 3 behavior. .. versionchanged:: 3.0 The return value is unsigned and in the range [0, 2**32-1] diff --git a/Doc/library/bsddb.rst b/Doc/library/bsddb.rst --- a/Doc/library/bsddb.rst +++ b/Doc/library/bsddb.rst @@ -7,7 +7,7 @@ .. sectionauthor:: Skip Montanaro .. deprecated:: 2.6 - The :mod:`bsddb` module has been deprecated for removal in Python 3.0. + The :mod:`bsddb` module has been removed in Python 3. The :mod:`bsddb` module provides an interface to the Berkeley DB library. Users @@ -86,7 +86,7 @@ This is present *only* to allow backwards compatibility with systems which ship with the old Berkeley DB 1.85 database library. The :mod:`bsddb185` module should never be used directly in new code. The module has been removed in - Python 3.0. If you find you still need it look in PyPI. + Python 3. If you find you still need it look in PyPI. .. seealso:: diff --git a/Doc/library/carbon.rst b/Doc/library/carbon.rst --- a/Doc/library/carbon.rst +++ b/Doc/library/carbon.rst @@ -24,7 +24,7 @@ .. note:: - The Carbon modules have been removed in Python 3.0. + The Carbon modules have been removed in Python 3. :mod:`Carbon.AE` --- Apple Events diff --git a/Doc/library/cd.rst b/Doc/library/cd.rst --- a/Doc/library/cd.rst +++ b/Doc/library/cd.rst @@ -9,7 +9,7 @@ .. deprecated:: 2.6 - The :mod:`cd` module has been deprecated for removal in Python 3.0. + The :mod:`cd` module has been removed in Python 3. This module provides an interface to the Silicon Graphics CD library. It is diff --git a/Doc/library/cgihttpserver.rst b/Doc/library/cgihttpserver.rst --- a/Doc/library/cgihttpserver.rst +++ b/Doc/library/cgihttpserver.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`CGIHTTPServer` module has been merged into :mod:`http.server` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. The :mod:`CGIHTTPServer` module defines a request-handler class, interface diff --git a/Doc/library/commands.rst b/Doc/library/commands.rst --- a/Doc/library/commands.rst +++ b/Doc/library/commands.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`commands` module has been removed in Python 3.0. Use the + The :mod:`commands` module has been removed in Python 3. Use the :mod:`subprocess` module instead. .. sectionauthor:: Sue Williams diff --git a/Doc/library/compiler.rst b/Doc/library/compiler.rst --- a/Doc/library/compiler.rst +++ b/Doc/library/compiler.rst @@ -6,7 +6,7 @@ *********************** .. deprecated:: 2.6 - The :mod:`compiler` package has been removed in Python 3.0. + The :mod:`compiler` package has been removed in Python 3. .. sectionauthor:: Jeremy Hylton diff --git a/Doc/library/configparser.rst b/Doc/library/configparser.rst --- a/Doc/library/configparser.rst +++ b/Doc/library/configparser.rst @@ -12,8 +12,8 @@ .. note:: The :mod:`ConfigParser` module has been renamed to :mod:`configparser` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. .. index:: pair: .ini; file diff --git a/Doc/library/cookie.rst b/Doc/library/cookie.rst --- a/Doc/library/cookie.rst +++ b/Doc/library/cookie.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`Cookie` module has been renamed to :mod:`http.cookies` in Python - 3.0. The :term:`2to3` tool will automatically adapt imports when converting - your sources to 3.0. + 3. The :term:`2to3` tool will automatically adapt imports when converting + your sources to Python 3. **Source code:** :source:`Lib/Cookie.py` diff --git a/Doc/library/cookielib.rst b/Doc/library/cookielib.rst --- a/Doc/library/cookielib.rst +++ b/Doc/library/cookielib.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`cookielib` module has been renamed to :mod:`http.cookiejar` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. .. versionadded:: 2.4 diff --git a/Doc/library/copy_reg.rst b/Doc/library/copy_reg.rst --- a/Doc/library/copy_reg.rst +++ b/Doc/library/copy_reg.rst @@ -5,9 +5,9 @@ :synopsis: Register pickle support functions. .. note:: - The :mod:`copy_reg` module has been renamed to :mod:`copyreg` in Python 3.0. + The :mod:`copy_reg` module has been renamed to :mod:`copyreg` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. index:: module: pickle diff --git a/Doc/library/dbhash.rst b/Doc/library/dbhash.rst --- a/Doc/library/dbhash.rst +++ b/Doc/library/dbhash.rst @@ -6,7 +6,7 @@ .. sectionauthor:: Fred L. Drake, Jr. .. deprecated:: 2.6 - The :mod:`dbhash` module has been deprecated for removal in Python 3.0. + The :mod:`dbhash` module has been removed in Python 3. .. index:: module: bsddb diff --git a/Doc/library/dbm.rst b/Doc/library/dbm.rst --- a/Doc/library/dbm.rst +++ b/Doc/library/dbm.rst @@ -6,9 +6,9 @@ :synopsis: The standard "database" interface, based on ndbm. .. note:: - The :mod:`dbm` module has been renamed to :mod:`dbm.ndbm` in Python 3.0. The + The :mod:`dbm` module has been renamed to :mod:`dbm.ndbm` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. The :mod:`dbm` module provides an interface to the Unix "(n)dbm" library. Dbm diff --git a/Doc/library/dircache.rst b/Doc/library/dircache.rst --- a/Doc/library/dircache.rst +++ b/Doc/library/dircache.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`dircache` module has been removed in Python 3.0. + The :mod:`dircache` module has been removed in Python 3. .. sectionauthor:: Moshe Zadka diff --git a/Doc/library/dl.rst b/Doc/library/dl.rst --- a/Doc/library/dl.rst +++ b/Doc/library/dl.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`dl` module has been removed in Python 3.0. Use the :mod:`ctypes` + The :mod:`dl` module has been removed in Python 3. Use the :mod:`ctypes` module instead. .. sectionauthor:: Moshe Zadka diff --git a/Doc/library/docxmlrpcserver.rst b/Doc/library/docxmlrpcserver.rst --- a/Doc/library/docxmlrpcserver.rst +++ b/Doc/library/docxmlrpcserver.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`DocXMLRPCServer` module has been merged into :mod:`xmlrpc.server` - in Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + in Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. .. versionadded:: 2.3 diff --git a/Doc/library/dumbdbm.rst b/Doc/library/dumbdbm.rst --- a/Doc/library/dumbdbm.rst +++ b/Doc/library/dumbdbm.rst @@ -5,9 +5,9 @@ :synopsis: Portable implementation of the simple DBM interface. .. note:: - The :mod:`dumbdbm` module has been renamed to :mod:`dbm.dumb` in Python 3.0. + The :mod:`dumbdbm` module has been renamed to :mod:`dbm.dumb` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. index:: single: databases diff --git a/Doc/library/dummy_thread.rst b/Doc/library/dummy_thread.rst --- a/Doc/library/dummy_thread.rst +++ b/Doc/library/dummy_thread.rst @@ -6,8 +6,8 @@ .. note:: The :mod:`dummy_thread` module has been renamed to :mod:`_dummy_thread` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0; however, you should consider using the + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3; however, you should consider using the high-lever :mod:`dummy_threading` module instead. **Source code:** :source:`Lib/dummy_thread.py` diff --git a/Doc/library/fl.rst b/Doc/library/fl.rst --- a/Doc/library/fl.rst +++ b/Doc/library/fl.rst @@ -9,7 +9,7 @@ .. deprecated:: 2.6 - The :mod:`fl` module has been deprecated for removal in Python 3.0. + The :mod:`fl` module has been removed in Python 3. .. index:: @@ -487,7 +487,7 @@ .. deprecated:: 2.6 - The :mod:`FL` module has been deprecated for removal in Python 3.0. + The :mod:`FL` module has been removed in Python 3. This module defines symbolic constants needed to use the built-in module @@ -509,7 +509,7 @@ .. deprecated:: 2.6 - The :mod:`flp` module has been deprecated for removal in Python 3.0. + The :mod:`flp` module has been removed in Python 3. This module defines functions that can read form definitions created by the diff --git a/Doc/library/fm.rst b/Doc/library/fm.rst --- a/Doc/library/fm.rst +++ b/Doc/library/fm.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`fm` module has been deprecated for removal in Python 3.0. + The :mod:`fm` module has been removed in Python 3. diff --git a/Doc/library/fpformat.rst b/Doc/library/fpformat.rst --- a/Doc/library/fpformat.rst +++ b/Doc/library/fpformat.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`fpformat` module has been removed in Python 3.0. + The :mod:`fpformat` module has been removed in Python 3. .. sectionauthor:: Moshe Zadka diff --git a/Doc/library/gdbm.rst b/Doc/library/gdbm.rst --- a/Doc/library/gdbm.rst +++ b/Doc/library/gdbm.rst @@ -6,9 +6,9 @@ :synopsis: GNU's reinterpretation of dbm. .. note:: - The :mod:`gdbm` module has been renamed to :mod:`dbm.gnu` in Python 3.0. The + The :mod:`gdbm` module has been renamed to :mod:`dbm.gnu` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. index:: module: dbm diff --git a/Doc/library/gl.rst b/Doc/library/gl.rst --- a/Doc/library/gl.rst +++ b/Doc/library/gl.rst @@ -8,7 +8,7 @@ .. deprecated:: 2.6 - The :mod:`gl` module has been deprecated for removal in Python 3.0. + The :mod:`gl` module has been removed in Python 3. This module provides access to the Silicon Graphics *Graphics Library*. It is @@ -168,7 +168,7 @@ .. deprecated:: 2.6 - The :mod:`DEVICE` module has been deprecated for removal in Python 3.0. + The :mod:`DEVICE` module has been removed in Python 3. This modules defines the constants used by the Silicon Graphics *Graphics @@ -186,7 +186,7 @@ .. deprecated:: 2.6 - The :mod:`GL` module has been deprecated for removal in Python 3.0. + The :mod:`GL` module has been removed in Python 3. This module contains constants used by the Silicon Graphics *Graphics Library* from the C header file ````. Read the module source file for details. diff --git a/Doc/library/htmllib.rst b/Doc/library/htmllib.rst --- a/Doc/library/htmllib.rst +++ b/Doc/library/htmllib.rst @@ -6,7 +6,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`htmllib` module has been removed in Python 3.0. + The :mod:`htmllib` module has been removed in Python 3. .. index:: @@ -162,8 +162,8 @@ .. note:: The :mod:`htmlentitydefs` module has been renamed to :mod:`html.entities` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. **Source code:** :source:`Lib/htmlentitydefs.py` diff --git a/Doc/library/httplib.rst b/Doc/library/httplib.rst --- a/Doc/library/httplib.rst +++ b/Doc/library/httplib.rst @@ -6,8 +6,8 @@ .. note:: The :mod:`httplib` module has been renamed to :mod:`http.client` in Python - 3.0. The :term:`2to3` tool will automatically adapt imports when converting - your sources to 3.0. + 3. The :term:`2to3` tool will automatically adapt imports when converting + your sources to Python 3. .. index:: diff --git a/Doc/library/imageop.rst b/Doc/library/imageop.rst --- a/Doc/library/imageop.rst +++ b/Doc/library/imageop.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`imageop` module has been removed in Python 3.0. + The :mod:`imageop` module has been removed in Python 3. The :mod:`imageop` module contains some useful operations on images. It operates on images consisting of 8 or 32 bit pixels stored in Python strings. This is diff --git a/Doc/library/imgfile.rst b/Doc/library/imgfile.rst --- a/Doc/library/imgfile.rst +++ b/Doc/library/imgfile.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`imgfile` module has been deprecated for removal in Python 3.0. + The :mod:`imgfile` module has been removed in Python 3. diff --git a/Doc/library/imputil.rst b/Doc/library/imputil.rst --- a/Doc/library/imputil.rst +++ b/Doc/library/imputil.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`imputil` module has been removed in Python 3.0. + The :mod:`imputil` module has been removed in Python 3. .. index:: statement: import diff --git a/Doc/library/jpeg.rst b/Doc/library/jpeg.rst --- a/Doc/library/jpeg.rst +++ b/Doc/library/jpeg.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`jpeg` module has been deprecated for removal in Python 3.0. + The :mod:`jpeg` module has been removed in Python 3. diff --git a/Doc/library/macostools.rst b/Doc/library/macostools.rst --- a/Doc/library/macostools.rst +++ b/Doc/library/macostools.rst @@ -15,7 +15,7 @@ .. note:: - This module has been removed in Python 3.0. + This module has been removed in Python 3. diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -1513,7 +1513,7 @@ mailboxes, such as adding or removing message, and do not provide classes to represent format-specific message properties. For backward compatibility, the older mailbox classes are still available, but the newer classes should be used -in preference to them. The old classes will be removed in Python 3.0. +in preference to them. The old classes will be removed in Python 3. Older mailbox objects support only iteration and provide a single public method: diff --git a/Doc/library/mhlib.rst b/Doc/library/mhlib.rst --- a/Doc/library/mhlib.rst +++ b/Doc/library/mhlib.rst @@ -6,7 +6,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`mhlib` module has been removed in Python 3.0. Use the + The :mod:`mhlib` module has been removed in Python 3. Use the :mod:`mailbox` instead. .. sectionauthor:: Skip Montanaro diff --git a/Doc/library/mutex.rst b/Doc/library/mutex.rst --- a/Doc/library/mutex.rst +++ b/Doc/library/mutex.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`mutex` module has been removed in Python 3.0. + The :mod:`mutex` module has been removed in Python 3. .. sectionauthor:: Moshe Zadka diff --git a/Doc/library/new.rst b/Doc/library/new.rst --- a/Doc/library/new.rst +++ b/Doc/library/new.rst @@ -6,7 +6,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`new` module has been removed in Python 3.0. Use the :mod:`types` + The :mod:`new` module has been removed in Python 3. Use the :mod:`types` module's classes instead. .. sectionauthor:: Moshe Zadka diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -331,7 +331,7 @@ .. note:: - This function is deprecated and has been removed in 3.0 in favor of + This function is deprecated and has been removed in Python 3 in favor of :func:`os.walk`. diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -34,7 +34,7 @@ replaced by "ast"; this is a legacy from the time when there was no other AST and has nothing to do with the AST found in Python 2.5. This is also the reason for the functions' keyword arguments being called *ast*, not *st*. - The "ast" functions will be removed in Python 3.0. + The "ast" functions will be removed in Python 3. There are a few things to note about this module which are important to making use of the data structures created. This is not a tutorial on editing the parse diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -5,9 +5,9 @@ :synopsis: A synchronized queue class. .. note:: - The :mod:`Queue` module has been renamed to :mod:`queue` in Python 3.0. The + The :mod:`Queue` module has been renamed to :mod:`queue` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. **Source code:** :source:`Lib/Queue.py` diff --git a/Doc/library/repr.rst b/Doc/library/repr.rst --- a/Doc/library/repr.rst +++ b/Doc/library/repr.rst @@ -6,9 +6,9 @@ .. sectionauthor:: Fred L. Drake, Jr. .. note:: - The :mod:`repr` module has been renamed to :mod:`reprlib` in Python 3.0. The + The :mod:`repr` module has been renamed to :mod:`reprlib` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. **Source code:** :source:`Lib/repr.py` diff --git a/Doc/library/rexec.rst b/Doc/library/rexec.rst --- a/Doc/library/rexec.rst +++ b/Doc/library/rexec.rst @@ -6,7 +6,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`rexec` module has been removed in Python 3.0. + The :mod:`rexec` module has been removed in Python 3. .. versionchanged:: 2.3 Disabled module. diff --git a/Doc/library/rfc822.rst b/Doc/library/rfc822.rst --- a/Doc/library/rfc822.rst +++ b/Doc/library/rfc822.rst @@ -10,7 +10,7 @@ .. deprecated:: 2.3 The :mod:`email` package should be used in preference to the :mod:`rfc822` module. This module is present only to maintain backward compatibility, and - has been removed in 3.0. + has been removed in Python 3. This module defines a class, :class:`Message`, which represents an "email message" as defined by the Internet standard :rfc:`2822`. [#]_ Such messages diff --git a/Doc/library/robotparser.rst b/Doc/library/robotparser.rst --- a/Doc/library/robotparser.rst +++ b/Doc/library/robotparser.rst @@ -16,9 +16,9 @@ .. note:: The :mod:`robotparser` module has been renamed :mod:`urllib.robotparser` in - Python 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when converting - your sources to 3.0. + your sources to Python 3. This module provides a single class, :class:`RobotFileParser`, which answers questions about whether or not a particular user agent can fetch a URL on the diff --git a/Doc/library/scrolledtext.rst b/Doc/library/scrolledtext.rst --- a/Doc/library/scrolledtext.rst +++ b/Doc/library/scrolledtext.rst @@ -16,8 +16,8 @@ .. note:: :mod:`ScrolledText` has been renamed to :mod:`tkinter.scrolledtext` in Python - 3.0. The :term:`2to3` tool will automatically adapt imports when converting - your sources to 3.0. + 3. The :term:`2to3` tool will automatically adapt imports when converting + your sources to Python 3. The text widget and scrollbar are packed together in a :class:`Frame`, and the methods of the :class:`Grid` and :class:`Pack` geometry managers are acquired diff --git a/Doc/library/sgmllib.rst b/Doc/library/sgmllib.rst --- a/Doc/library/sgmllib.rst +++ b/Doc/library/sgmllib.rst @@ -6,7 +6,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`sgmllib` module has been removed in Python 3.0. + The :mod:`sgmllib` module has been removed in Python 3. .. index:: single: SGML diff --git a/Doc/library/simplehttpserver.rst b/Doc/library/simplehttpserver.rst --- a/Doc/library/simplehttpserver.rst +++ b/Doc/library/simplehttpserver.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`SimpleHTTPServer` module has been merged into :mod:`http.server` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. The :mod:`SimpleHTTPServer` module defines a single class, diff --git a/Doc/library/simplexmlrpcserver.rst b/Doc/library/simplexmlrpcserver.rst --- a/Doc/library/simplexmlrpcserver.rst +++ b/Doc/library/simplexmlrpcserver.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`SimpleXMLRPCServer` module has been merged into - :mod:`xmlrpc.server` in Python 3.0. The :term:`2to3` tool will automatically - adapt imports when converting your sources to 3.0. + :mod:`xmlrpc.server` in Python 3. The :term:`2to3` tool will automatically + adapt imports when converting your sources to Python 3. .. versionadded:: 2.2 diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst --- a/Doc/library/socketserver.rst +++ b/Doc/library/socketserver.rst @@ -7,8 +7,8 @@ .. note:: The :mod:`SocketServer` module has been renamed to :mod:`socketserver` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. **Source code:** :source:`Lib/SocketServer.py` diff --git a/Doc/library/statvfs.rst b/Doc/library/statvfs.rst --- a/Doc/library/statvfs.rst +++ b/Doc/library/statvfs.rst @@ -6,7 +6,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`statvfs` module has been deprecated for removal in Python 3.0. + The :mod:`statvfs` module has been removed in Python 3. .. sectionauthor:: Moshe Zadka diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -969,7 +969,7 @@ See :ref:`formatstrings` for a description of the various formatting options that can be specified in format strings. - This method of string formatting is the new standard in Python 3.0, and + This method of string formatting is the new standard in Python 3, and should be preferred to the ``%`` formatting described in :ref:`string-formatting` in new code. diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -793,7 +793,7 @@ The following list of functions are also defined as methods of string and Unicode objects; see section :ref:`string-methods` for more information on those. You should consider these functions as deprecated, although they will -not be removed until Python 3.0. The functions defined in this module are: +not be removed until Python 3. The functions defined in this module are: .. function:: atof(s) diff --git a/Doc/library/sunaudio.rst b/Doc/library/sunaudio.rst --- a/Doc/library/sunaudio.rst +++ b/Doc/library/sunaudio.rst @@ -8,7 +8,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`sunaudiodev` module has been deprecated for removal in Python 3.0. + The :mod:`sunaudiodev` module has been removed in Python 3. @@ -153,7 +153,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`SUNAUDIODEV` module has been deprecated for removal in Python 3.0. + The :mod:`SUNAUDIODEV` module has been removed in Python 3. diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -797,10 +797,10 @@ .. data:: py3kwarning - Bool containing the status of the Python 3.0 warning flag. It's ``True`` + Bool containing the status of the Python 3 warning flag. It's ``True`` when Python is started with the -3 option. (This should be considered read-only; setting it to a different value doesn't have an effect on - Python 3.0 warnings.) + Python 3 warnings.) .. versionadded:: 2.6 diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -143,7 +143,7 @@ .. deprecated:: 2.6 - The :class:`TarFileCompat` class has been deprecated for removal in Python 3.0. + The :class:`TarFileCompat` class has been removed in Python 3. .. exception:: TarError diff --git a/Doc/library/thread.rst b/Doc/library/thread.rst --- a/Doc/library/thread.rst +++ b/Doc/library/thread.rst @@ -5,9 +5,9 @@ :synopsis: Create multiple threads of control within one interpreter. .. note:: - The :mod:`thread` module has been renamed to :mod:`_thread` in Python 3.0. + The :mod:`thread` module has been renamed to :mod:`_thread` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0; however, you should consider using the high-level + sources to Python 3; however, you should consider using the high-level :mod:`threading` module instead. diff --git a/Doc/library/tix.rst b/Doc/library/tix.rst --- a/Doc/library/tix.rst +++ b/Doc/library/tix.rst @@ -24,9 +24,9 @@ .. note:: - :mod:`Tix` has been renamed to :mod:`tkinter.tix` in Python 3.0. The + :mod:`Tix` has been renamed to :mod:`tkinter.tix` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. seealso:: diff --git a/Doc/library/tkinter.rst b/Doc/library/tkinter.rst --- a/Doc/library/tkinter.rst +++ b/Doc/library/tkinter.rst @@ -13,9 +13,9 @@ .. note:: - :mod:`Tkinter` has been renamed to :mod:`tkinter` in Python 3.0. The + :mod:`Tkinter` has been renamed to :mod:`tkinter` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting your - sources to 3.0. + sources to Python 3. .. seealso:: @@ -109,7 +109,7 @@ :mod:`turtle` Turtle graphics in a Tk window. -These have been renamed as well in Python 3.0; they were all made submodules of +These have been renamed as well in Python 3; they were all made submodules of the new ``tkinter`` package. diff --git a/Doc/library/urllib.rst b/Doc/library/urllib.rst --- a/Doc/library/urllib.rst +++ b/Doc/library/urllib.rst @@ -6,11 +6,11 @@ .. note:: The :mod:`urllib` module has been split into parts and renamed in - Python 3.0 to :mod:`urllib.request`, :mod:`urllib.parse`, + Python 3 to :mod:`urllib.request`, :mod:`urllib.parse`, and :mod:`urllib.error`. The :term:`2to3` tool will automatically adapt - imports when converting your sources to 3.0. + imports when converting your sources to Python 3. Also note that the :func:`urllib.urlopen` function has been removed in - Python 3.0 in favor of :func:`urllib2.urlopen`. + Python 3 in favor of :func:`urllib2.urlopen`. .. index:: single: WWW @@ -131,7 +131,7 @@ :envvar:`no_proxy` environment variable. .. deprecated:: 2.6 - The :func:`urlopen` function has been removed in Python 3.0 in favor + The :func:`urlopen` function has been removed in Python 3 in favor of :func:`urllib2.urlopen`. diff --git a/Doc/library/urllib2.rst b/Doc/library/urllib2.rst --- a/Doc/library/urllib2.rst +++ b/Doc/library/urllib2.rst @@ -9,9 +9,9 @@ .. note:: The :mod:`urllib2` module has been split across several modules in - Python 3.0 named :mod:`urllib.request` and :mod:`urllib.error`. + Python 3 named :mod:`urllib.request` and :mod:`urllib.error`. The :term:`2to3` tool will automatically adapt imports when converting - your sources to 3.0. + your sources to Python 3. The :mod:`urllib2` module defines functions and classes which help in opening diff --git a/Doc/library/urlparse.rst b/Doc/library/urlparse.rst --- a/Doc/library/urlparse.rst +++ b/Doc/library/urlparse.rst @@ -13,9 +13,9 @@ pair: relative; URL .. note:: - The :mod:`urlparse` module is renamed to :mod:`urllib.parse` in Python 3.0. + The :mod:`urlparse` module is renamed to :mod:`urllib.parse` in Python 3. The :term:`2to3` tool will automatically adapt imports when converting - your sources to 3.0. + your sources to Python 3. **Source code:** :source:`Lib/urlparse.py` diff --git a/Doc/library/user.rst b/Doc/library/user.rst --- a/Doc/library/user.rst +++ b/Doc/library/user.rst @@ -7,7 +7,7 @@ :deprecated: .. deprecated:: 2.6 - The :mod:`user` module has been removed in Python 3.0. + The :mod:`user` module has been removed in Python 3. .. index:: pair: .pythonrc.py; file diff --git a/Doc/library/userdict.rst b/Doc/library/userdict.rst --- a/Doc/library/userdict.rst +++ b/Doc/library/userdict.rst @@ -114,8 +114,8 @@ .. note:: The :class:`UserList` class has been moved to the :mod:`collections` - module in Python 3.0. The :term:`2to3` tool will automatically adapt - imports when converting your sources to 3.0. + module in Python 3. The :term:`2to3` tool will automatically adapt + imports when converting your sources to Python 3. In addition to supporting the methods and operations of mutable sequences (see @@ -187,8 +187,8 @@ .. note:: The :class:`UserString` class has been moved to the :mod:`collections` - module in Python 3.0. The :term:`2to3` tool will automatically adapt - imports when converting your sources to 3.0. + module in Python 3. The :term:`2to3` tool will automatically adapt + imports when converting your sources to Python 3. @@ -203,7 +203,7 @@ hard to track down. .. deprecated:: 2.6 - The :class:`MutableString` class has been removed in Python 3.0. + The :class:`MutableString` class has been removed in Python 3. In addition to supporting the methods and operations of string and Unicode objects (see section :ref:`string-methods`), :class:`UserString` instances diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -419,7 +419,7 @@ .. note:: - In Python 3.0, the arguments to the constructor for + In Python 3, the arguments to the constructor for :class:`catch_warnings` are keyword-only arguments. .. versionadded:: 2.6 diff --git a/Doc/library/whichdb.rst b/Doc/library/whichdb.rst --- a/Doc/library/whichdb.rst +++ b/Doc/library/whichdb.rst @@ -6,8 +6,8 @@ .. note:: The :mod:`whichdb` module's only function has been put into the :mod:`dbm` - module in Python 3.0. The :term:`2to3` tool will automatically adapt imports - when converting your sources to 3.0. + module in Python 3. The :term:`2to3` tool will automatically adapt imports + when converting your sources to Python 3. The single function in this module attempts to guess which of the several simple diff --git a/Doc/library/xmlrpclib.rst b/Doc/library/xmlrpclib.rst --- a/Doc/library/xmlrpclib.rst +++ b/Doc/library/xmlrpclib.rst @@ -8,8 +8,8 @@ .. note:: The :mod:`xmlrpclib` module has been renamed to :mod:`xmlrpc.client` in - Python 3.0. The :term:`2to3` tool will automatically adapt imports when - converting your sources to 3.0. + Python 3. The :term:`2to3` tool will automatically adapt imports when + converting your sources to Python 3. .. XXX Not everything is documented yet. It might be good to describe diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -573,7 +573,7 @@ :attr:`im_self` used to refer to the class that defined the method. .. versionchanged:: 2.6 - For 3.0 forward-compatibility, :attr:`im_func` is also available as + For Python 3 forward-compatibility, :attr:`im_func` is also available as :attr:`__func__`, and :attr:`im_self` as :attr:`__self__`. .. index:: @@ -1149,7 +1149,7 @@ single: class; classic single: class; old-style -Old-style classes are removed in Python 3.0, leaving only the semantics of +Old-style classes are removed in Python 3, leaving only the semantics of new-style classes. @@ -2235,7 +2235,7 @@ This section used to document the rules for coercion. As the language has evolved, the coercion rules have become hard to document precisely; documenting what one version of one particular implementation does is undesirable. Instead, -here are some informal guidelines regarding coercion. In Python 3.0, coercion +here are some informal guidelines regarding coercion. In Python 3, coercion will not be supported. * diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1392,7 +1392,7 @@ .. [#] In Python 2.3 and later releases, a list comprehension "leaks" the control variables of each ``for`` it contains into the containing scope. However, this - behavior is deprecated, and relying on it will not work in Python 3.0 + behavior is deprecated, and relying on it will not work in Python 3. .. [#] While ``abs(x%y) < abs(y)`` is true mathematically, for floats it may not be true numerically due to roundoff. For example, and assuming a platform on which -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 19:28:25 2012 From: python-checkins at python.org (victor.stinner) Date: Thu, 03 May 2012 19:28:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314687=3A_Remove_re?= =?utf8?q?dundant_length_attribute_of_unicode=5Fwrite=5Ft?= Message-ID: http://hg.python.org/cpython/rev/f58159e5d52f changeset: 76733:f58159e5d52f parent: 76731:0a9143d7b097 user: Victor Stinner date: Thu May 03 19:27:14 2012 +0200 summary: Issue #14687: Remove redundant length attribute of unicode_write_t The length can be read directly from the buffer files: Objects/unicodeobject.c | 20 ++++++++------------ 1 files changed, 8 insertions(+), 12 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13661,7 +13661,6 @@ void *data; enum PyUnicode_Kind kind; Py_UCS4 maxchar; - Py_ssize_t length; Py_ssize_t pos; }; @@ -13678,8 +13677,7 @@ Py_ssize_t length, Py_UCS4 maxchar) { writer->pos = 0; - writer->length = length; - writer->buffer = PyUnicode_New(writer->length, maxchar); + writer->buffer = PyUnicode_New(length, maxchar); if (writer->buffer == NULL) return -1; unicode_writer_update(writer); @@ -13699,16 +13697,14 @@ } newlen = writer->pos + length; - if (newlen > writer->length) { + if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { /* overallocate 25% to limit the number of resize */ - if (newlen > PY_SSIZE_T_MAX - newlen / 4) - writer->length = newlen; - else - writer->length = newlen + newlen / 4; + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + newlen += newlen / 4; if (maxchar > writer->maxchar) { /* resize + widen */ - newbuffer = PyUnicode_New(writer->length, maxchar); + newbuffer = PyUnicode_New(newlen, maxchar); if (newbuffer == NULL) return -1; PyUnicode_CopyCharacters(newbuffer, 0, @@ -13716,7 +13712,7 @@ Py_DECREF(writer->buffer); } else { - newbuffer = resize_compact(writer->buffer, writer->length); + newbuffer = resize_compact(writer->buffer, newlen); if (newbuffer == NULL) return -1; } @@ -13740,7 +13736,7 @@ maxchar = _PyUnicode_FindMaxChar(str, start, start + length); if (unicode_writer_prepare(writer, length, maxchar) == -1) return -1; - assert((writer->pos + length) <= writer->length); + assert((writer->pos + length) <= PyUnicode_GET_LENGTH(writer->buffer)); copy_characters(writer->buffer, writer->pos, str, start, length); writer->pos += length; @@ -13754,7 +13750,7 @@ { if (unicode_writer_prepare(writer, 1, ch) == -1) return -1; - assert((writer->pos + 1) <= writer->length); + assert((writer->pos + 1) <= PyUnicode_GET_LENGTH(writer->buffer)); PyUnicode_WRITE(writer->kind, writer->data, writer->pos, ch); writer->pos += 1; return 0; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 19:34:56 2012 From: python-checkins at python.org (richard.oudkerk) Date: Thu, 03 May 2012 19:34:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_dangling_warning_for_te?= =?utf8?q?st=5Fmultiprocessing?= Message-ID: http://hg.python.org/cpython/rev/5a4276fdf4b7 changeset: 76734:5a4276fdf4b7 user: Richard Oudkerk date: Thu May 03 18:29:02 2012 +0100 summary: Fix dangling warning for test_multiprocessing files: Lib/multiprocessing/managers.py | 5 ++++- Lib/test/test_multiprocessing.py | 3 +++ 2 files changed, 7 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -548,7 +548,10 @@ ''' Join the manager process (if it has been spawned) ''' - self._process.join(timeout) + if self._process is not None: + self._process.join(timeout) + if not self._process.is_alive(): + self._process = None def _debug_info(self): ''' diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2516,6 +2516,7 @@ def tearDown(self): self.mgr.shutdown() + self.mgr.join() def test_manager_initializer(self): m = multiprocessing.managers.SyncManager() @@ -2523,6 +2524,7 @@ m.start(initializer, (self.ns,)) self.assertEqual(self.ns.test, 1) m.shutdown() + m.join() def test_pool_initializer(self): self.assertRaises(TypeError, multiprocessing.Pool, initializer=1) @@ -2818,6 +2820,7 @@ ManagerMixin.pool.terminate() ManagerMixin.pool.join() ManagerMixin.manager.shutdown() + ManagerMixin.manager.join() ThreadsMixin.pool.join() ProcessesMixin.pool.join() del ProcessesMixin.pool, ThreadsMixin.pool, ManagerMixin.pool -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 3 20:30:59 2012 From: python-checkins at python.org (barry.warsaw) Date: Thu, 03 May 2012 20:30:59 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_improve_the_readability_of_one?= =?utf8?q?_section?= Message-ID: http://hg.python.org/peps/rev/5e088d558b88 changeset: 4351:5e088d558b88 user: Barry Warsaw date: Thu May 03 14:30:48 2012 -0400 summary: improve the readability of one section files: pep-0396.txt | 37 ++++++++++++++++++++++++------------- pep-0420.txt | 2 +- 2 files changed, 25 insertions(+), 14 deletions(-) diff --git a/pep-0396.txt b/pep-0396.txt --- a/pep-0396.txt +++ b/pep-0396.txt @@ -102,15 +102,13 @@ module version number when included in the standard library, and SHOULD include a version number when packaged separately. -#. When a module includes a version number, it SHOULD be available in - the ``__version__`` attribute on that module. +#. When a module (or package) includes a version number, the version + SHOULD be available in the ``__version__`` attribute. -#. For modules which are also packages, the module's namespace SHOULD - include the ``__version__`` attribute. - -#. For modules which live inside a namespace package, the sub-package - name SHOULD include the ``__version__`` attribute. The namespace - module itself SHOULD NOT include its own ``__version__`` attribute. +#. For modules which live inside a namespace package, the module + SHOULD include the ``__version__`` attribute. The namespace + package itself SHOULD NOT include its own ``__version__`` + attribute. #. The ``__version__`` attribute's value SHOULD be a string. @@ -218,9 +216,20 @@ Because the distutils2 style ``setup.cfg`` is declarative, we can't run any code to extract the ``__version__`` attribute, either via -import or via parsing. This PEP suggests a special key be added to -the ``[metadata]`` section of the ``setup.cfg`` file to indicate "get -the version from this file". Something like this might work:: +import or via parsing. + +In consultation with the distutils-sig [9]_, two options are +proposed. Both entail containing the version number in a file, and +declaring that file in the ``setup.cfg``. When the entire contents of +the file contains the version number, the ``version-file`` key will be +used:: + + [metadata] + version-file: version.txt + +When the version number is contained within a larger file, e.g. of +Python code, such that the file must be parsed to extract the version, +the key ``version-from-file`` will be used:: [metadata] version-from-file: elle.py @@ -240,7 +249,7 @@ PEP 376 metadata ================ -PEP 376 [9]_ defines a standard for static metadata, but doesn't +PEP 376 [10]_ defines a standard for static metadata, but doesn't describe the process by which this metadata gets created. It is highly desirable for the derived version information to be placed into the PEP 376 ``.dist-info`` metadata at build-time rather than @@ -274,7 +283,9 @@ .. [8] pkgutil - Package utilities (http://distutils2.notmyidea.org/library/pkgutil.html) -.. [9] PEP 376, Database of Installed Python Distributions +.. [9] http://mail.python.org/pipermail/distutils-sig/2011-June/017862.html + +.. [10] PEP 376, Database of Installed Python Distributions (http://www.python.org/dev/peps/pep-0376/) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -174,7 +174,7 @@ There is no impact on PEP 302 "loaders". If an existing finder is not updated to support returning a string -from ``find_module``, the only impact is that such a loader will be +from ``find_module``, the only impact is that such a finder will be unable to provide portions of a namespace package. Packaging Implications -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 3 21:57:39 2012 From: python-checkins at python.org (larry.hastings) Date: Thu, 03 May 2012 21:57:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314127=3A_Fix_no-op?= =?utf8?q?_stub_for_platforms_that_lack_some_=22os=22_functions=2E?= Message-ID: http://hg.python.org/cpython/rev/cdc4e0f8135d changeset: 76735:cdc4e0f8135d user: Larry Hastings date: Thu May 03 12:56:44 2012 -0700 summary: Issue #14127: Fix no-op stub for platforms that lack some "os" functions. files: Lib/shutil.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -138,7 +138,7 @@ only if both `src` and `dst` are symlinks. """ - def _nop(*args): + def _nop(*args, ns=None): pass if symlinks and os.path.islink(src) and os.path.islink(dst): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 00:02:12 2012 From: python-checkins at python.org (victor.stinner) Date: Fri, 04 May 2012 00:02:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_unicode=5Fwriter=3A_add_fin?= =?utf8?q?ish=28=29_method_and_assertions_to_write=5Fstr=28=29_method?= Message-ID: http://hg.python.org/cpython/rev/16e8622068fa changeset: 76736:16e8622068fa user: Victor Stinner date: Thu May 03 23:58:55 2012 +0200 summary: unicode_writer: add finish() method and assertions to write_str() method * The write_str() method does nothing if the length is zero. * Replace "struct unicode_writer_t" with "unicode_writer_t" files: Objects/unicodeobject.c | 46 ++++++++++++++++++++-------- 1 files changed, 33 insertions(+), 13 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13656,16 +13656,16 @@ return (Py_UCS4) -1; } -struct unicode_writer_t { +typedef struct { PyObject *buffer; void *data; enum PyUnicode_Kind kind; Py_UCS4 maxchar; Py_ssize_t pos; -}; +} unicode_writer_t; Py_LOCAL_INLINE(void) -unicode_writer_update(struct unicode_writer_t *writer) +unicode_writer_update(unicode_writer_t *writer) { writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); writer->data = PyUnicode_DATA(writer->buffer); @@ -13673,7 +13673,7 @@ } Py_LOCAL_INLINE(int) -unicode_writer_init(struct unicode_writer_t *writer, +unicode_writer_init(unicode_writer_t *writer, Py_ssize_t length, Py_UCS4 maxchar) { writer->pos = 0; @@ -13685,7 +13685,7 @@ } Py_LOCAL_INLINE(int) -unicode_writer_prepare(struct unicode_writer_t *writer, +unicode_writer_prepare(unicode_writer_t *writer, Py_ssize_t length, Py_UCS4 maxchar) { Py_ssize_t newlen; @@ -13729,13 +13729,26 @@ Py_LOCAL_INLINE(int) unicode_writer_write_str( - struct unicode_writer_t *writer, + unicode_writer_t *writer, PyObject *str, Py_ssize_t start, Py_ssize_t length) { Py_UCS4 maxchar; + + assert(str != NULL); + assert(PyUnicode_Check(str)); + if (PyUnicode_READY(str) == -1) + return -1; + + assert(0 <= start); + assert(0 <= length); + assert(start + length <= PyUnicode_GET_LENGTH(str)); + if (length == 0) + return 0; + maxchar = _PyUnicode_FindMaxChar(str, start, start + length); if (unicode_writer_prepare(writer, length, maxchar) == -1) return -1; + assert((writer->pos + length) <= PyUnicode_GET_LENGTH(writer->buffer)); copy_characters(writer->buffer, writer->pos, str, start, length); @@ -13745,7 +13758,7 @@ Py_LOCAL_INLINE(int) unicode_writer_write_char( - struct unicode_writer_t *writer, + unicode_writer_t *writer, Py_UCS4 ch) { if (unicode_writer_prepare(writer, 1, ch) == -1) @@ -13756,8 +13769,18 @@ return 0; } +Py_LOCAL_INLINE(PyObject *) +unicode_writer_finish(unicode_writer_t *writer) +{ + if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { + Py_DECREF(writer->buffer); + return NULL; + } + return writer->buffer; +} + Py_LOCAL_INLINE(void) -unicode_writer_dealloc(struct unicode_writer_t *writer) +unicode_writer_dealloc(unicode_writer_t *writer) { Py_CLEAR(writer->buffer); } @@ -13773,7 +13796,7 @@ PyObject *uformat; void *fmt; enum PyUnicode_Kind kind, fmtkind; - struct unicode_writer_t writer; + unicode_writer_t writer; if (format == NULL || args == NULL) { PyErr_BadInternalCall(); @@ -14185,16 +14208,13 @@ goto onError; } - if (PyUnicode_Resize(&writer.buffer, writer.pos) < 0) - goto onError; - if (args_owned) { Py_DECREF(args); } Py_DECREF(uformat); Py_XDECREF(temp); Py_XDECREF(second); - return writer.buffer; + return unicode_writer_finish(&writer); onError: Py_DECREF(uformat); -- Repository URL: http://hg.python.org/cpython From benjamin at python.org Fri May 4 00:14:08 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 3 May 2012 18:14:08 -0400 Subject: [Python-checkins] cpython: unicode_writer: add finish() method and assertions to write_str() method In-Reply-To: References: Message-ID: 2012/5/3 victor.stinner : > ?Py_LOCAL_INLINE(void) Do these have to be marked inline? -- Regards, Benjamin From python-checkins at python.org Fri May 4 00:31:12 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 00:31:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_if_the_kind_of_the_string_t?= =?utf8?q?o_count_is_larger_than_the_string_to_search=2C?= Message-ID: http://hg.python.org/cpython/rev/0ea729cc2a0b changeset: 76737:0ea729cc2a0b user: Benjamin Peterson date: Thu May 03 18:31:07 2012 -0400 summary: if the kind of the string to count is larger than the string to search, shortcut to 0 files: Objects/unicodeobject.c | 13 +++---------- 1 files changed, 3 insertions(+), 10 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -11187,20 +11187,15 @@ kind1 = PyUnicode_KIND(self); kind2 = PyUnicode_KIND(substring); - kind = kind1 > kind2 ? kind1 : kind2; + if (kind2 > kind1) + return PyLong_FromLong(0); + kind = kind1; buf1 = PyUnicode_DATA(self); buf2 = PyUnicode_DATA(substring); - if (kind1 != kind) - buf1 = _PyUnicode_AsKind(self, kind); - if (!buf1) { - Py_DECREF(substring); - return NULL; - } if (kind2 != kind) buf2 = _PyUnicode_AsKind(substring, kind); if (!buf2) { Py_DECREF(substring); - if (kind1 != kind) PyMem_Free(buf1); return NULL; } len1 = PyUnicode_GET_LENGTH(self); @@ -11232,8 +11227,6 @@ result = PyLong_FromSsize_t(iresult); - if (kind1 != kind) - PyMem_Free(buf1); if (kind2 != kind) PyMem_Free(buf2); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 00:44:39 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 00:44:39 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogY2xvc2UoKSBkb2Vz?= =?utf8?q?n=27t_take_any_args_=28closes_=2314717=29?= Message-ID: http://hg.python.org/cpython/rev/b8c1cabcd115 changeset: 76738:b8c1cabcd115 branch: 3.2 parent: 76728:c334e90160b9 user: Benjamin Peterson date: Thu May 03 18:44:09 2012 -0400 summary: close() doesn't take any args (closes #14717) files: Objects/genobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -129,7 +129,7 @@ } PyDoc_STRVAR(close_doc, -"close(arg) -> raise GeneratorExit inside generator."); +"close() -> raise GeneratorExit inside generator."); static PyObject * gen_close(PyGenObject *gen, PyObject *args) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 00:44:39 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 00:44:39 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogY2xvc2UoKSBkb2Vz?= =?utf8?q?n=27t_take_any_args_=28closes_=2314717=29?= Message-ID: http://hg.python.org/cpython/rev/b2031eb95dd9 changeset: 76739:b2031eb95dd9 branch: 2.7 parent: 76732:8aab13719c02 user: Benjamin Peterson date: Thu May 03 18:44:09 2012 -0400 summary: close() doesn't take any args (closes #14717) files: Objects/genobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -120,7 +120,7 @@ } PyDoc_STRVAR(close_doc, -"close(arg) -> raise GeneratorExit inside generator."); +"close() -> raise GeneratorExit inside generator."); static PyObject * gen_close(PyGenObject *gen, PyObject *args) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 00:44:40 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 00:44:40 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?b?OiBtZXJnZSAzLjIgKCMxNDcxNyk=?= Message-ID: http://hg.python.org/cpython/rev/da12cb2461d1 changeset: 76740:da12cb2461d1 parent: 76737:0ea729cc2a0b parent: 76738:b8c1cabcd115 user: Benjamin Peterson date: Thu May 03 18:44:33 2012 -0400 summary: merge 3.2 (#14717) files: Objects/genobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/genobject.c b/Objects/genobject.c --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -137,7 +137,7 @@ } PyDoc_STRVAR(close_doc, -"close(arg) -> raise GeneratorExit inside generator."); +"close() -> raise GeneratorExit inside generator."); /* * This helper function is used by gen_close and gen_throw to -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 01:20:31 2012 From: python-checkins at python.org (victor.stinner) Date: Fri, 04 May 2012 01:20:31 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_unicode=5Fwriter=3A_don=27t?= =?utf8?q?_force_inline_when_it_is_not_necessary?= Message-ID: http://hg.python.org/cpython/rev/44699acd28b4 changeset: 76741:44699acd28b4 user: Victor Stinner date: Fri May 04 01:19:15 2012 +0200 summary: unicode_writer: don't force inline when it is not necessary Keep inline for performance critical functions (functions used in loops) files: Objects/unicodeobject.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13665,7 +13665,7 @@ writer->kind = PyUnicode_KIND(writer->buffer); } -Py_LOCAL_INLINE(int) +Py_LOCAL(int) unicode_writer_init(unicode_writer_t *writer, Py_ssize_t length, Py_UCS4 maxchar) { @@ -13762,7 +13762,7 @@ return 0; } -Py_LOCAL_INLINE(PyObject *) +Py_LOCAL(PyObject *) unicode_writer_finish(unicode_writer_t *writer) { if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { @@ -13772,7 +13772,7 @@ return writer->buffer; } -Py_LOCAL_INLINE(void) +Py_LOCAL(void) unicode_writer_dealloc(unicode_writer_t *writer) { Py_CLEAR(writer->buffer); -- Repository URL: http://hg.python.org/cpython From victor.stinner at gmail.com Fri May 4 01:24:13 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 May 2012 01:24:13 +0200 Subject: [Python-checkins] cpython: unicode_writer: add finish() method and assertions to write_str() method In-Reply-To: References: Message-ID: >> ?Py_LOCAL_INLINE(void) > > Do these have to be marked inline? Functions used in loops, yes: the inline keyword *does* impact performances (5% slower). I removed the keyword for the other unicode_writer methods. Victor From solipsis at pitrou.net Fri May 4 05:37:25 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 04 May 2012 05:37:25 +0200 Subject: [Python-checkins] Daily reference leaks (44699acd28b4): sum=30 Message-ID: results for 44699acd28b4 on branch "default" -------------------------------------------- test_import leaked [1, 1, 1] references, sum=3 test_importlib leaked [1, 1, 1] references, sum=3 test_os leaked [2, 2, 2] references, sum=6 test_posix leaked [6, 6, 6] references, sum=18 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogdMBTyT', '-x'] From benjamin at python.org Fri May 4 07:07:04 2012 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 4 May 2012 01:07:04 -0400 Subject: [Python-checkins] cpython: Issue #14127: Add ns= parameter to utime, futimes, and lutimes. In-Reply-To: References: Message-ID: 2012/5/3 larry.hastings : > diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c > --- a/Modules/posixmodule.c > +++ b/Modules/posixmodule.c > @@ -3572,28 +3572,194 @@ > ?#endif /* HAVE_UNAME */ > > > +static int > +split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns) > +{ > + ? ?int result = 0; > + ? ?PyObject *divmod; > + ? ?divmod = PyNumber_Divmod(py_long, billion); > + ? ?if (!divmod) > + ? ? ? ?goto exit; > + ? ?*s = _PyLong_AsTime_t(PyTuple_GET_ITEM(divmod, 0)); > + ? ?if ((*s == -1) && PyErr_Occurred()) > + ? ? ? ?goto exit; > + ? ?*ns = PyLong_AsLong(PyTuple_GET_ITEM(divmod, 1)); > + ? ?if ((*s == -1) && PyErr_Occurred()) > + ? ? ? ?goto exit; > + > + ? ?result = 1; > +exit: > + ? ?Py_XDECREF(divmod); > + ? ?return result; > +} > + > + > +typedef int (*parameter_converter_t)(PyObject *, void *); > + > +typedef struct { > + ? ?/* input only */ > + ? ?char path_format; > + ? ?parameter_converter_t converter; > + ? ?char *function_name; > + ? ?char *first_argument_name; > + ? ?PyObject *args; > + ? ?PyObject *kwargs; > + > + ? ?/* input/output */ > + ? ?PyObject **path; > + > + ? ?/* output only */ > + ? ?int now; > + ? ?time_t atime_s; > + ? ?long ? atime_ns; > + ? ?time_t mtime_s; > + ? ?long ? mtime_ns; > +} utime_arguments; > + > +#define DECLARE_UA(ua, fname) \ > + ? ?utime_arguments ua; \ > + ? ?memset(&ua, 0, sizeof(ua)); \ > + ? ?ua.function_name = fname; \ > + ? ?ua.args = args; \ > + ? ?ua.kwargs = kwargs; \ > + ? ?ua.first_argument_name = "path"; \ > + > +/* UA_TO_FILETIME doesn't declare atime and mtime for you */ > +#define UA_TO_FILETIME(ua, atime, mtime) \ > + ? ?time_t_to_FILE_TIME(ua.atime_s, ua.atime_ns, &atime); \ > + ? ?time_t_to_FILE_TIME(ua.mtime_s, ua.mtime_ns, &mtime) > + > +/* the rest of these macros declare the output variable for you */ > +#define UA_TO_TIMESPEC(ua, ts) \ > + ? ?struct timespec ts[2]; \ > + ? ?ts[0].tv_sec = ua.atime_s; \ > + ? ?ts[0].tv_nsec = ua.atime_ns; \ > + ? ?ts[1].tv_sec = ua.mtime_s; \ > + ? ?ts[1].tv_nsec = ua.mtime_ns > + > +#define UA_TO_TIMEVAL(ua, tv) \ > + ? ?struct timeval tv[2]; \ > + ? ?tv[0].tv_sec = ua.atime_s; \ > + ? ?tv[0].tv_usec = ua.atime_ns / 1000; \ > + ? ?tv[1].tv_sec = ua.mtime_s; \ > + ? ?tv[1].tv_usec = ua.mtime_ns / 1000 > + > +#define UA_TO_UTIMBUF(ua, u) \ > + ? ?struct utimbuf u; \ > + ? ?utimbuf.actime = ua.atime_s; \ > + ? ?utimbuf.modtime = ua.mtime_s > + > +#define UA_TO_TIME_T(ua, timet) \ > + ? ?time_t timet[2]; \ > + ? ?timet[0] = ua.atime_s; \ > + ? ?timet[1] = ua.mtime_s > + > + > +/* > + * utime_read_time_arguments() processes arguments for the utime > + * family of functions. > + * returns zero on failure. > + */ > +static int > +utime_read_time_arguments(utime_arguments *ua) > +{ > + ? ?PyObject *times = NULL; > + ? ?PyObject *ns = NULL; > + ? ?char format[24]; > + ? ?char *kwlist[4]; > + ? ?char **kw = kwlist; > + ? ?int return_value; > + > + ? ?*kw++ = ua->first_argument_name; > + ? ?*kw++ = "times"; > + ? ?*kw++ = "ns"; > + ? ?*kw = NULL; > + > + ? ?sprintf(format, "%c%s|O$O:%s", > + ? ? ? ? ? ?ua->path_format, > + ? ? ? ? ? ?ua->converter ? "&" : "", > + ? ? ? ? ? ?ua->function_name); > + > + ? ?if (ua->converter) > + ? ? ? ?return_value = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, > + ? ? ? ? ? ?format, kwlist, ua->converter, ua->path, ×, &ns); > + ? ?else > + ? ? ? ?return_value = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, > + ? ? ? ? ? ?format, kwlist, ua->path, ×, &ns); > + > + ? ?if (!return_value) > + ? ? ? ?return 0; > + > + ? ?if (times && ns) { > + ? ? ? ?PyErr_Format(PyExc_RuntimeError, Why not a ValueError or TypeError? > + ? ? ? ? ? ? ? ? ? ? "%s: you may specify either 'times'" > + ? ? ? ? ? ? ? ? ? ? " or 'ns' but not both", > + ? ? ? ? ? ? ? ? ? ? ua->function_name); > + ? ? ? ?return 0; > + ? ?} > + > + ? ?if (times && (times != Py_None)) { Conditions in parenthesis like this is not style. > + ? ? ? ?if (!PyTuple_CheckExact(times) || (PyTuple_Size(times) != 2)) { > + ? ? ? ? ? ?PyErr_Format(PyExc_TypeError, > + ? ? ? ? ? ? ? ? ? ? ? ? "%s: 'time' must be either" > + ? ? ? ? ? ? ? ? ? ? ? ? " a valid tuple of two ints or None", > + ? ? ? ? ? ? ? ? ? ? ? ? ua->function_name); > + ? ? ? ? ? ?return 0; > + ? ? ? ?} > + ? ? ? ?ua->now = 0; > + ? ? ? ?return (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 0), > + ? ? ? ? ? ? ? ? ? ?&(ua->atime_s), &(ua->atime_ns)) != -1) > + ? ? ? ? ? ?&& (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), Put && on previous line like Python. > + ? ? ? ? ? ? ? ? ? ?&(ua->mtime_s), &(ua->mtime_ns)) != -1); > + ? ?} > + > + ? ?if (ns) { > + ? ? ? ?if (!PyTuple_CheckExact(ns) || (PyTuple_Size(ns) != 2)) { > + ? ? ? ? ? ?PyErr_Format(PyExc_TypeError, > + ? ? ? ? ? ? ? ? ? ? ? ? "%s: 'ns' must be a valid tuple of two ints", > + ? ? ? ? ? ? ? ? ? ? ? ? ua->function_name); > + ? ? ? ? ? ?return 0; > + ? ? ? ?} > + ? ? ? ?ua->now = 0; > + ? ? ? ?return (split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 0), > + ? ? ? ? ? ? ? ? ? ?&(ua->atime_s), &(ua->atime_ns))) > + ? ? ? ? ? ?&& (split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 1), > + ? ? ? ? ? ? ? ? ? ?&(ua->mtime_s), &(ua->mtime_ns))); > + ? ?} > + > + ? ?/* either times=None, or neither times nor ns was specified. use "now". */ > + ? ?ua->now = 1; > + ? ?return 1; > +} -- Regards, Benjamin From python-checkins at python.org Fri May 4 07:13:14 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 07:13:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_check_correct_variable_for_?= =?utf8?q?error?= Message-ID: http://hg.python.org/cpython/rev/7d29b63bd854 changeset: 76742:7d29b63bd854 user: Benjamin Peterson date: Fri May 04 01:10:59 2012 -0400 summary: check correct variable for error files: Modules/posixmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3584,7 +3584,7 @@ if ((*s == -1) && PyErr_Occurred()) goto exit; *ns = PyLong_AsLong(PyTuple_GET_ITEM(divmod, 1)); - if ((*s == -1) && PyErr_Occurred()) + if ((*ns == -1) && PyErr_Occurred()) goto exit; result = 1; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 07:14:09 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 07:14:09 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_avoid_unitialized_memory?= Message-ID: http://hg.python.org/cpython/rev/b0deafca6c02 changeset: 76743:b0deafca6c02 user: Benjamin Peterson date: Fri May 04 01:14:03 2012 -0400 summary: avoid unitialized memory files: Modules/posixmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3576,7 +3576,7 @@ split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns) { int result = 0; - PyObject *divmod; + PyObject *divmod = NULL; divmod = PyNumber_Divmod(py_long, billion); if (!divmod) goto exit; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 07:31:17 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 07:31:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_clean_up_converted_path_on_?= =?utf8?q?error?= Message-ID: http://hg.python.org/cpython/rev/7d235551439e changeset: 76744:7d235551439e user: Benjamin Peterson date: Fri May 04 01:31:13 2012 -0400 summary: clean up converted path on error files: Modules/posixmodule.c | 31 ++++++++++++++++++++----------- 1 files changed, 20 insertions(+), 11 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3695,7 +3695,7 @@ "%s: you may specify either 'times'" " or 'ns' but not both", ua->function_name); - return 0; + goto fail; } if (times && (times != Py_None)) { @@ -3704,13 +3704,15 @@ "%s: 'time' must be either" " a valid tuple of two ints or None", ua->function_name); - return 0; + goto fail; } ua->now = 0; - return (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 0), - &(ua->atime_s), &(ua->atime_ns)) != -1) - && (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), - &(ua->mtime_s), &(ua->mtime_ns)) != -1); + if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 0), + &ua->atime_s, &ua->atime_ns) == -1 || + _PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), + &ua->mtime_s, &ua->mtime_ns) == -1) + goto fail; + return 1; } if (ns) { @@ -3718,18 +3720,25 @@ PyErr_Format(PyExc_TypeError, "%s: 'ns' must be a valid tuple of two ints", ua->function_name); - return 0; + goto fail; } ua->now = 0; - return (split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 0), - &(ua->atime_s), &(ua->atime_ns))) - && (split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 1), - &(ua->mtime_s), &(ua->mtime_ns))); + if (!split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 0), + &ua->atime_s, &ua->atime_ns) || + !split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 1), + &ua->mtime_s, &ua->mtime_ns)) + goto fail; + return 1; } /* either times=None, or neither times nor ns was specified. use "now". */ ua->now = 1; return 1; + + fail: + if (ua->converter) + Py_DECREF(ua->path); + return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 07:42:45 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 07:42:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_what_is_a_invalid_tuple=3F?= Message-ID: http://hg.python.org/cpython/rev/6951adf604ea changeset: 76745:6951adf604ea user: Benjamin Peterson date: Fri May 04 01:42:41 2012 -0400 summary: what is a invalid tuple? files: Modules/posixmodule.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3702,7 +3702,7 @@ if (!PyTuple_CheckExact(times) || (PyTuple_Size(times) != 2)) { PyErr_Format(PyExc_TypeError, "%s: 'time' must be either" - " a valid tuple of two ints or None", + " a tuple of two ints or None", ua->function_name); goto fail; } @@ -3718,7 +3718,7 @@ if (ns) { if (!PyTuple_CheckExact(ns) || (PyTuple_Size(ns) != 2)) { PyErr_Format(PyExc_TypeError, - "%s: 'ns' must be a valid tuple of two ints", + "%s: 'ns' must be a tuple of two ints", ua->function_name); goto fail; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 10:50:07 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 04 May 2012 10:50:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Give_test=5Fmultiprocessing?= =?utf8?q?_better_chance_of_avoiding_timeout_failures_on_Windows?= Message-ID: http://hg.python.org/cpython/rev/11205ffc537a changeset: 76746:11205ffc537a user: Richard Oudkerk date: Fri May 04 09:44:39 2012 +0100 summary: Give test_multiprocessing better chance of avoiding timeout failures on Windows files: Lib/test/test_multiprocessing.py | 23 +++++++++++++------ 1 files changed, 16 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2699,35 +2699,43 @@ def test_wait_timeout(self): from multiprocessing.connection import wait - expected = 1 + expected = 5 a, b = multiprocessing.Pipe() start = time.time() - res = wait([a, b], 1) + res = wait([a, b], expected) delta = time.time() - start self.assertEqual(res, []) - self.assertLess(delta, expected + 0.5) - self.assertGreater(delta, expected - 0.5) + self.assertLess(delta, expected + 1) + self.assertGreater(delta, expected - 1) b.send(None) start = time.time() - res = wait([a, b], 1) + res = wait([a, b], 20) delta = time.time() - start self.assertEqual(res, [a]) self.assertLess(delta, 0.4) + @classmethod + def signal_and_sleep(cls, sem, period): + sem.release() + time.sleep(period) + def test_wait_integer(self): from multiprocessing.connection import wait - expected = 5 + expected = 3 + sem = multiprocessing.Semaphore(0) a, b = multiprocessing.Pipe() - p = multiprocessing.Process(target=time.sleep, args=(expected,)) + p = multiprocessing.Process(target=self.signal_and_sleep, + args=(sem, expected)) p.start() self.assertIsInstance(p.sentinel, int) + self.assertTrue(sem.acquire(timeout=20)) start = time.time() res = wait([a, p.sentinel, b], expected + 20) @@ -2755,6 +2763,7 @@ self.assertEqual(res, [a, p.sentinel, b]) self.assertLess(delta, 0.4) + p.terminate() p.join() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 11:32:55 2012 From: python-checkins at python.org (larry.hastings) Date: Fri, 04 May 2012 11:32:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314127=3A_Fix_two_b?= =?utf8?q?ugs_with_the_Windows_implementation=2E?= Message-ID: http://hg.python.org/cpython/rev/fc5d2f4291ac changeset: 76747:fc5d2f4291ac user: Larry Hastings date: Fri May 04 02:31:57 2012 -0700 summary: Issue #14127: Fix two bugs with the Windows implementation. files: Modules/posixmodule.c | 63 +++++++++++++++++++++--------- 1 files changed, 43 insertions(+), 20 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3658,9 +3658,17 @@ /* * utime_read_time_arguments() processes arguments for the utime * family of functions. - * returns zero on failure. */ -static int + +typedef enum { + utime_success = 0, + utime_parse_failure = 1, + utime_times_and_ns_collision = 2, + utime_times_conversion_failure = 3, + utime_ns_conversion_failure = 4, +} utime_status; + +static utime_status utime_read_time_arguments(utime_arguments *ua) { PyObject *times = NULL; @@ -3668,7 +3676,8 @@ char format[24]; char *kwlist[4]; char **kw = kwlist; - int return_value; + utime_status return_value; + int parse_result; *kw++ = ua->first_argument_name; *kw++ = "times"; @@ -3681,20 +3690,21 @@ ua->function_name); if (ua->converter) - return_value = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, + parse_result = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, format, kwlist, ua->converter, ua->path, ×, &ns); else - return_value = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, + parse_result = PyArg_ParseTupleAndKeywords(ua->args, ua->kwargs, format, kwlist, ua->path, ×, &ns); - if (!return_value) - return 0; + if (!parse_result) + return utime_parse_failure; if (times && ns) { PyErr_Format(PyExc_RuntimeError, "%s: you may specify either 'times'" " or 'ns' but not both", ua->function_name); + return_value = utime_times_and_ns_collision; goto fail; } @@ -3704,15 +3714,18 @@ "%s: 'time' must be either" " a tuple of two ints or None", ua->function_name); + return_value = utime_times_conversion_failure; goto fail; } ua->now = 0; if (_PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 0), &ua->atime_s, &ua->atime_ns) == -1 || _PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), - &ua->mtime_s, &ua->mtime_ns) == -1) + &ua->mtime_s, &ua->mtime_ns) == -1) { + return_value = utime_times_conversion_failure; goto fail; - return 1; + } + return utime_success; } if (ns) { @@ -3720,25 +3733,28 @@ PyErr_Format(PyExc_TypeError, "%s: 'ns' must be a tuple of two ints", ua->function_name); + return_value = utime_ns_conversion_failure; goto fail; } ua->now = 0; if (!split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 0), &ua->atime_s, &ua->atime_ns) || !split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 1), - &ua->mtime_s, &ua->mtime_ns)) + &ua->mtime_s, &ua->mtime_ns)) { + return_value = utime_ns_conversion_failure; goto fail; - return 1; + } + return utime_success; } /* either times=None, or neither times nor ns was specified. use "now". */ ua->now = 1; - return 1; + return utime_success; fail: if (ua->converter) Py_DECREF(ua->path); - return 0; + return return_value; } @@ -3767,7 +3783,10 @@ ua.path_format = 'U'; ua.path = &upath; - if (!utime_read_time_arguments(&ua)) { + switch (utime_read_time_arguments(&ua)) { + default: + return NULL; + case utime_success: { wchar_t *wpath = PyUnicode_AsUnicode(upath); if (wpath == NULL) return NULL; @@ -3778,8 +3797,9 @@ Py_END_ALLOW_THREADS if (hFile == INVALID_HANDLE_VALUE) return win32_error_object("utime", upath); - } - else { + break; + } + case utime_parse_failure: { const char *apath; /* Drop the argument parsing error as narrow strings are also valid. */ @@ -3787,7 +3807,7 @@ ua.path_format = 'y'; ua.path = (PyObject **)&apath; - if (!utime_read_time_arguments(&ua)) + if (utime_read_time_arguments(&ua) != utime_success) return NULL; if (win32_warn_bytes_api()) return NULL; @@ -3801,6 +3821,9 @@ win32_error("utime", apath); return NULL; } + break; + } + } if (ua.now) { @@ -3839,7 +3862,7 @@ ua.path = &opath; ua.converter = PyUnicode_FSConverter; - if (!utime_read_time_arguments(&ua)) + if (utime_read_time_arguments(&ua) != utime_success) return NULL; path = PyBytes_AsString(opath); if (ua.now) { @@ -3892,7 +3915,7 @@ ua.path = (PyObject **)&fd; ua.first_argument_name = "fd"; - if (!utime_read_time_arguments(&ua)) + if (utime_read_time_arguments(&ua) != utime_success) return NULL; if (ua.now) { @@ -3937,7 +3960,7 @@ ua.path = &opath; ua.converter = PyUnicode_FSConverter; - if (!utime_read_time_arguments(&ua)) + if (utime_read_time_arguments(&ua) != utime_success) return NULL; path = PyBytes_AsString(opath); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 13:02:37 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 04 May 2012 13:02:37 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_for_fatal_errors_in_os?= =?utf8?q?=2E*utime*=28=29?= Message-ID: http://hg.python.org/cpython/rev/4deb7c1f49b9 changeset: 76748:4deb7c1f49b9 user: Richard Oudkerk date: Fri May 04 12:01:31 2012 +0100 summary: Fix for fatal errors in os.*utime*() The address of an object was being decreffed instead of the object. files: Modules/posixmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3753,7 +3753,7 @@ fail: if (ua->converter) - Py_DECREF(ua->path); + Py_DECREF(*ua->path); return return_value; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 16:28:49 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 16:28:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_test_connec?= =?utf8?q?ting_to_sha256=2Etbs-internet=2Ecom=2E?= Message-ID: http://hg.python.org/cpython/rev/683295cf9489 changeset: 76749:683295cf9489 branch: 3.2 parent: 76738:b8c1cabcd115 user: Antoine Pitrou date: Fri May 04 16:26:02 2012 +0200 summary: Fix test connecting to sha256.tbs-internet.com. The certificate has changed and the test now needs SNI to pass. files: Lib/test/sha256.pem | 222 +++++++++++++++--------------- Lib/test/test_ssl.py | 11 +- 2 files changed, 119 insertions(+), 114 deletions(-) diff --git a/Lib/test/sha256.pem b/Lib/test/sha256.pem --- a/Lib/test/sha256.pem +++ b/Lib/test/sha256.pem @@ -1,128 +1,128 @@ # Certificate chain for https://sha256.tbs-internet.com - 0 s:/C=FR/postalCode=14000/ST=Calvados/L=CAEN/street=22 rue de Bretagne/O=TBS INTERNET/OU=0002 440443810/OU=Certificats TBS X509/CN=ecom.tbs-x509.com - i:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA business + 0 s:/C=FR/postalCode=14000/ST=Calvados/L=CAEN/street=22 rue de Bretagne/O=TBS INTERNET/OU=0002 440443810/OU=sha-256 production/CN=sha256.tbs-internet.com + i:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA SGC -----BEGIN CERTIFICATE----- -MIIGTjCCBTagAwIBAgIQOh3d9dNDPq1cSdJmEiMpqDANBgkqhkiG9w0BAQUFADCB -yTELMAkGA1UEBhMCRlIxETAPBgNVBAgTCENhbHZhZG9zMQ0wCwYDVQQHEwRDYWVu -MRUwEwYDVQQKEwxUQlMgSU5URVJORVQxSDBGBgNVBAsTP1Rlcm1zIGFuZCBDb25k -aXRpb25zOiBodHRwOi8vd3d3LnRicy1pbnRlcm5ldC5jb20vQ0EvcmVwb3NpdG9y -eTEYMBYGA1UECxMPVEJTIElOVEVSTkVUIENBMR0wGwYDVQQDExRUQlMgWDUwOSBD -QSBidXNpbmVzczAeFw0xMTAxMjUwMDAwMDBaFw0xMzAyMDUyMzU5NTlaMIHHMQsw -CQYDVQQGEwJGUjEOMAwGA1UEERMFMTQwMDAxETAPBgNVBAgTCENhbHZhZG9zMQ0w -CwYDVQQHEwRDQUVOMRswGQYDVQQJExIyMiBydWUgZGUgQnJldGFnbmUxFTATBgNV -BAoTDFRCUyBJTlRFUk5FVDEXMBUGA1UECxMOMDAwMiA0NDA0NDM4MTAxHTAbBgNV -BAsTFENlcnRpZmljYXRzIFRCUyBYNTA5MRowGAYDVQQDExFlY29tLnRicy14NTA5 -LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKRrlHUnJ++1lpcg -jtYco7cdmRe+EEfTmwPfCdfV3G1QfsTSvY6FfMpm/83pqHfT+4ANwr18wD9ZrAEN -G16mf9VdCGK12+TP7DmqeZyGIqlFFoahQnmb8EarvE43/1UeQ2CV9XmzwZvpqeli -LfXsFonawrY3H6ZnMwS64St61Z+9gdyuZ/RbsoZBbT5KUjDEG844QRU4OT1IGeEI -eY5NM5RNIh6ZNhVtqeeCxMS7afONkHQrOco73RdSTRck/Hj96Ofl3MHNHryr+AMK -DGFk1kLCZGpPdXtkxXvaDeQoiYDlil26CWc+YK6xyDPMdsWvoG14ZLyCpzMXA7/7 -4YAQRH0CAwEAAaOCAjAwggIsMB8GA1UdIwQYMBaAFBoJBMz5CY+7HqDO1KQUf0vV -I1jNMB0GA1UdDgQWBBQgOU8HsWzbmD4WZP5Wtdw7jca2WDAOBgNVHQ8BAf8EBAMC -BaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw -TAYDVR0gBEUwQzBBBgsrBgEEAYDlNwIBATAyMDAGCCsGAQUFBwIBFiRodHRwczov -L3d3dy50YnMtaW50ZXJuZXQuY29tL0NBL0NQUzEwdwYDVR0fBHAwbjA3oDWgM4Yx -aHR0cDovL2NybC50YnMtaW50ZXJuZXQuY29tL1RCU1g1MDlDQWJ1c2luZXNzLmNy -bDAzoDGgL4YtaHR0cDovL2NybC50YnMteDUwOS5jb20vVEJTWDUwOUNBYnVzaW5l -c3MuY3JsMIGwBggrBgEFBQcBAQSBozCBoDA9BggrBgEFBQcwAoYxaHR0cDovL2Ny -dC50YnMtaW50ZXJuZXQuY29tL1RCU1g1MDlDQWJ1c2luZXNzLmNydDA5BggrBgEF -BQcwAoYtaHR0cDovL2NydC50YnMteDUwOS5jb20vVEJTWDUwOUNBYnVzaW5lc3Mu -Y3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC50YnMteDUwOS5jb20wMwYDVR0R -BCwwKoIRZWNvbS50YnMteDUwOS5jb22CFXd3dy5lY29tLnRicy14NTA5LmNvbTAN -BgkqhkiG9w0BAQUFAAOCAQEArT4NHfbY87bGAw8lPV4DmHlmuDuVp/y7ltO3Ynse -3Rz8RxW2AzuO0Oy2F0Cu4yWKtMyEyMXyHqWtae7ElRbdTu5w5GwVBLJHClCzC8S9 -SpgMMQTx3Rgn8vjkHuU9VZQlulZyiPK7yunjc7c310S9FRZ7XxOwf8Nnx4WnB+No -WrfApzhhQl31w+RyrNxZe58hCfDDHmevRvwLjQ785ZoQXJDj2j3qAD4aI2yB8lB5 -oaE1jlCJzC7Kmz/Y9jzfmv/zAs1LQTm9ktevv4BTUFaGjv9jxnQ1xnS862ZiouLW -zZYIlYPf4F6JjXGiIQgQRglILUfq3ftJd9/ok9W9ZF8h8w== +MIIGXDCCBUSgAwIBAgIRAKpVmHgg9nfCodAVwcP4siwwDQYJKoZIhvcNAQELBQAw +gcQxCzAJBgNVBAYTAkZSMREwDwYDVQQIEwhDYWx2YWRvczENMAsGA1UEBxMEQ2Fl +bjEVMBMGA1UEChMMVEJTIElOVEVSTkVUMUgwRgYDVQQLEz9UZXJtcyBhbmQgQ29u +ZGl0aW9uczogaHR0cDovL3d3dy50YnMtaW50ZXJuZXQuY29tL0NBL3JlcG9zaXRv +cnkxGDAWBgNVBAsTD1RCUyBJTlRFUk5FVCBDQTEYMBYGA1UEAxMPVEJTIFg1MDkg +Q0EgU0dDMB4XDTEyMDEwNDAwMDAwMFoXDTE0MDIxNzIzNTk1OVowgcsxCzAJBgNV +BAYTAkZSMQ4wDAYDVQQREwUxNDAwMDERMA8GA1UECBMIQ2FsdmFkb3MxDTALBgNV +BAcTBENBRU4xGzAZBgNVBAkTEjIyIHJ1ZSBkZSBCcmV0YWduZTEVMBMGA1UEChMM +VEJTIElOVEVSTkVUMRcwFQYDVQQLEw4wMDAyIDQ0MDQ0MzgxMDEbMBkGA1UECxMS +c2hhLTI1NiBwcm9kdWN0aW9uMSAwHgYDVQQDExdzaGEyNTYudGJzLWludGVybmV0 +LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKQIX/zdJcyxty0m +PM1XQSoSSifueS3AVcgqMsaIKS/u+rYzsv4hQ/qA6vLn5m5/ewUcZDj7zdi6rBVf +PaVNXJ6YinLX0tkaW8TEjeVuZG5yksGZlhCt1CJ1Ho9XLiLaP4uJ7MCoNUntpJ+E +LfrOdgsIj91kPmwjDJeztVcQCvKzhjVJA/KxdInc0JvOATn7rpaSmQI5bvIjufgo +qVsTPwVFzuUYULXBk7KxRT7MiEqnd5HvviNh0285QC478zl3v0I0Fb5El4yD3p49 +IthcRnxzMKc0UhU5ogi0SbONyBfm/mzONVfSxpM+MlyvZmJqrbuuLoEDzJD+t8PU +xSuzgbcCAwEAAaOCAj4wggI6MB8GA1UdIwQYMBaAFAdEdoWTKLx/bXjSCuv6TEvf +2YIfMB0GA1UdDgQWBBT/qTGYdaj+f61c2IRFL/B1eEsM8DAOBgNVHQ8BAf8EBAMC +BaAwDAYDVR0TAQH/BAIwADA0BgNVHSUELTArBggrBgEFBQcDAQYIKwYBBQUHAwIG +CisGAQQBgjcKAwMGCWCGSAGG+EIEATBLBgNVHSAERDBCMEAGCisGAQQB5TcCBAEw +MjAwBggrBgEFBQcCARYkaHR0cHM6Ly93d3cudGJzLWludGVybmV0LmNvbS9DQS9D +UFM0MG0GA1UdHwRmMGQwMqAwoC6GLGh0dHA6Ly9jcmwudGJzLWludGVybmV0LmNv +bS9UQlNYNTA5Q0FTR0MuY3JsMC6gLKAqhihodHRwOi8vY3JsLnRicy14NTA5LmNv +bS9UQlNYNTA5Q0FTR0MuY3JsMIGmBggrBgEFBQcBAQSBmTCBljA4BggrBgEFBQcw +AoYsaHR0cDovL2NydC50YnMtaW50ZXJuZXQuY29tL1RCU1g1MDlDQVNHQy5jcnQw +NAYIKwYBBQUHMAKGKGh0dHA6Ly9jcnQudGJzLXg1MDkuY29tL1RCU1g1MDlDQVNH +Qy5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnRicy14NTA5LmNvbTA/BgNV +HREEODA2ghdzaGEyNTYudGJzLWludGVybmV0LmNvbYIbd3d3LnNoYTI1Ni50YnMt +aW50ZXJuZXQuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQA0pOuL8QvAa5yksTbGShzX +ABApagunUGoEydv4YJT1MXy9tTp7DrWaozZSlsqBxrYAXP1d9r2fuKbEniYHxaQ0 +UYaf1VSIlDo1yuC8wE7wxbHDIpQ/E5KAyxiaJ8obtDhFstWAPAH+UoGXq0kj2teN +21sFQ5dXgA95nldvVFsFhrRUNB6xXAcaj0VZFhttI0ZfQZmQwEI/P+N9Jr40OGun +aa+Dn0TMeUH4U20YntfLbu2nDcJcYfyurm+8/0Tr4HznLnedXu9pCPYj0TaddrgT +XO0oFiyy7qGaY6+qKh71yD64Y3ycCJ/HR9Wm39mjZYc9ezYwT4noP6r7Lk8YO7/q -----END CERTIFICATE----- - 1 s:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA business + 1 s:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA SGC i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root -----BEGIN CERTIFICATE----- -MIIFPzCCBCegAwIBAgIQDlBz/++iRSmLDeVRHT/hADANBgkqhkiG9w0BAQUFADBv +MIIFVjCCBD6gAwIBAgIQXpDZ0ETJMV02WTx3GTnhhTANBgkqhkiG9w0BAQUFADBv MQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFk ZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBF -eHRlcm5hbCBDQSBSb290MB4XDTA1MTIwMTAwMDAwMFoXDTE5MDcwOTE4MTkyMlow -gckxCzAJBgNVBAYTAkZSMREwDwYDVQQIEwhDYWx2YWRvczENMAsGA1UEBxMEQ2Fl +eHRlcm5hbCBDQSBSb290MB4XDTA1MTIwMTAwMDAwMFoXDTE5MDYyNDE5MDYzMFow +gcQxCzAJBgNVBAYTAkZSMREwDwYDVQQIEwhDYWx2YWRvczENMAsGA1UEBxMEQ2Fl bjEVMBMGA1UEChMMVEJTIElOVEVSTkVUMUgwRgYDVQQLEz9UZXJtcyBhbmQgQ29u ZGl0aW9uczogaHR0cDovL3d3dy50YnMtaW50ZXJuZXQuY29tL0NBL3JlcG9zaXRv -cnkxGDAWBgNVBAsTD1RCUyBJTlRFUk5FVCBDQTEdMBsGA1UEAxMUVEJTIFg1MDkg -Q0EgYnVzaW5lc3MwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDB1PAU -qudCcz3tmyGcf+u6EkZqonKKHrV4gZYbvVkIRojmmlhfi/jwvpHvo8bqSt/9Rj5S -jhCDW0pcbI+IPPtD1Jy+CHNSfnMqVDy6CKQ3p5maTzCMG6ZT+XjnvcND5v+FtaiB -xk1iCX6uvt0jeUtdZvYbyytsSDE6c3Y5//wRxOF8tM1JxibwO3pyER26jbbN2gQz -m/EkdGjLdJ4svPk23WDAvQ6G0/z2LcAaJB+XLfqRwfQpHQvfKa1uTi8PivC8qtip -rmNQMMPMjxSK2azX8cKjjTDJiUKaCb4VHlJDWKEsCFRpgJAoAuX8f7Yfs1M4esGo -sWb3PGspK3O22uIlAgMBAAGjggF6MIIBdjAdBgNVHQ4EFgQUGgkEzPkJj7seoM7U -pBR/S9UjWM0wDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwGAYD -VR0gBBEwDzANBgsrBgEEAYDlNwIBATB7BgNVHR8EdDByMDigNqA0hjJodHRwOi8v -Y3JsLmNvbW9kb2NhLmNvbS9BZGRUcnVzdEV4dGVybmFsQ0FSb290LmNybDA2oDSg -MoYwaHR0cDovL2NybC5jb21vZG8ubmV0L0FkZFRydXN0RXh0ZXJuYWxDQVJvb3Qu -Y3JsMIGGBggrBgEFBQcBAQR6MHgwOwYIKwYBBQUHMAKGL2h0dHA6Ly9jcnQuY29t -b2RvY2EuY29tL0FkZFRydXN0VVROU2VydmVyQ0EuY3J0MDkGCCsGAQUFBzAChi1o -dHRwOi8vY3J0LmNvbW9kby5uZXQvQWRkVHJ1c3RVVE5TZXJ2ZXJDQS5jcnQwEQYJ -YIZIAYb4QgEBBAQDAgIEMA0GCSqGSIb3DQEBBQUAA4IBAQA7mqrMgk/MrE6QnbNA -h4nRCn2ti4bg4w2C3lB6bSvRPnYwuNw9Jb8vuKkNFzRDxNJXqVDZdfFW5CVQJuyd -nfAx83+wk+spzvFaE1KhFYfN9G9pQfXUfvDRoIcJgPEKUXL1wRiOG+IjU3VVI8pg -IgqHkr7ylln5i5zCiFAPuIJmYUSFg/gxH5xkCNcjJqqrHrHatJr6Qrrke93joupw -oU1njfAcZtYp6fbiK6u2b1pJqwkVBE8RsfLnPhRj+SFbpvjv8Od7o/ieJhFIYQNU -k2jX2u8qZnAiNw93LZW9lpYjtuvMXq8QQppENNja5b53q7UwI+lU7ZGjZ7quuESp -J6/5 +cnkxGDAWBgNVBAsTD1RCUyBJTlRFUk5FVCBDQTEYMBYGA1UEAxMPVEJTIFg1MDkg +Q0EgU0dDMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsgOkO3f7wzN6 +rOjg45tR5vjBfzK7qmV9IBxb/QW9EEXxG+E7FNhZqQLtwGBKoSsHTnQqV75wWMk0 +9tinWvftBkSpj5sTi/8cbzJfUvTSVYh3Qxv6AVVjMMH/ruLjE6y+4PoaPs8WoYAQ +ts5R4Z1g8c/WnTepLst2x0/Wv7GmuoQi+gXvHU6YrBiu7XkeYhzc95QdviWSJRDk +owhb5K43qhcvjRmBfO/paGlCliDGZp8mHwrI21mwobWpVjTxZRwYO3bd4+TGcI4G +Ie5wmHwE8F7SK1tgSqbBacKjDa93j7txKkfz/Yd2n7TGqOXiHPsJpG655vrKtnXk +9vs1zoDeJQIDAQABo4IBljCCAZIwHQYDVR0OBBYEFAdEdoWTKLx/bXjSCuv6TEvf +2YIfMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEAMCAGA1UdJQQZ +MBcGCisGAQQBgjcKAwMGCWCGSAGG+EIEATAYBgNVHSAEETAPMA0GCysGAQQBgOU3 +AgQBMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0Fk +ZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k +by5uZXQvQWRkVHJ1c3RFeHRlcm5hbENBUm9vdC5jcmwwgYAGCCsGAQUFBwEBBHQw +cjA4BggrBgEFBQcwAoYsaHR0cDovL2NydC5jb21vZG9jYS5jb20vQWRkVHJ1c3RV +VE5TR0NDQS5jcnQwNgYIKwYBBQUHMAKGKmh0dHA6Ly9jcnQuY29tb2RvLm5ldC9B +ZGRUcnVzdFVUTlNHQ0NBLmNydDARBglghkgBhvhCAQEEBAMCAgQwDQYJKoZIhvcN +AQEFBQADggEBAK2zEzs+jcIrVK9oDkdDZNvhuBYTdCfpxfFs+OAujW0bIfJAy232 +euVsnJm6u/+OrqKudD2tad2BbejLLXhMZViaCmK7D9nrXHx4te5EP8rL19SUVqLY +1pTnv5dhNgEgvA7n5lIzDSYs7yRLsr7HJsYPr6SeYSuZizyX1SNz7ooJ32/F3X98 +RB0Mlc/E0OyOrkQ9/y5IrnpnaSora8CnUrV5XNOg+kyCz9edCyx4D5wXYcwZPVWz +8aDqquESrezPyjtfi4WRO4s/VD3HLZvOxzMrWAVYCDG9FxaOhF0QGuuG1F7F3GKV +v6prNyCl016kRl2j1UT+a7gLd8fA25A4C9E= -----END CERTIFICATE----- 2 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root - i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware + i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC -----BEGIN CERTIFICATE----- -MIIETzCCAzegAwIBAgIQHM5EYpUZep1jUvnyI6m2mDANBgkqhkiG9w0BAQUFADCB -lzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug +MIIEZjCCA06gAwIBAgIQUSYKkxzif5zDpV954HKugjANBgkqhkiG9w0BAQUFADCB +kzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug Q2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho -dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3Qt -SGFyZHdhcmUwHhcNMDUwNjA3MDgwOTEwWhcNMTkwNzA5MTgxOTIyWjBvMQswCQYD -VQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0 -IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5h -bCBDQSBSb290MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt/caM+by -AAQtOeBOW+0fvGwPzbX6I7bO3psRM5ekKUx9k5+9SryT7QMa44/P5W1QWtaXKZRa -gLBJetsulf24yr83OC0ePpFBrXBWx/BPP+gynnTKyJBU6cZfD3idmkA8Dqxhql4U -j56HoWpQ3NeaTq8Fs6ZxlJxxs1BgCscTnTgHhgKo6ahpJhiQq0ywTyOrOk+E2N/O -n+Fpb7vXQtdrROTHre5tQV9yWnEIN7N5ZaRZoJQ39wAvDcKSctrQOHLbFKhFxF0q -fbe01sTurM0TRLfJK91DACX6YblpalgjEbenM49WdVn1zSnXRrcKK2W200JvFbK4 -e/vv6V1T1TRaJwIDAQABo4G9MIG6MB8GA1UdIwQYMBaAFKFyXyYbKJhDlV0HN9WF -lp1L0sNFMB0GA1UdDgQWBBStvZh6NLQm9/rEJlTvA73gJMtUGjAOBgNVHQ8BAf8E -BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zARBglghkgBhvhCAQEEBAMCAQIwRAYDVR0f -BD0wOzA5oDegNYYzaHR0cDovL2NybC51c2VydHJ1c3QuY29tL1VUTi1VU0VSRmly -c3QtSGFyZHdhcmUuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQByQhANOs4kClrwF8BW -onvUOGCSjRK52zYZgDXYNjDtmr5rJ6NyPFDNn+JxkLpjYetIFMTbSRe679Bt8m7a -gIAoQYFQtxMuyLnJegB2aEbQiIxh/tC21UcFF7ktdnDoTlA6w3pLuvunaI84Of3o -2YBrhzkTbCfaYk5JRlTpudW9DkUkHBsyx3nknPKnplkIGaK0jgn8E0n+SFabYaHk -I9LroYT/+JtLefh9lgBdAgVv0UPbzoGfuDsrk/Zh+UrgbLFpHoVnElhzbkh64Z0X -OGaJunQc68cCZu5HTn/aK7fBGMcVflRCXLVEQpU9PIAdGA8Ynvg684t8GMaKsRl1 -jIGZ +dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xGzAZBgNVBAMTElVUTiAtIERBVEFDb3Jw +IFNHQzAeFw0wNTA2MDcwODA5MTBaFw0xOTA2MjQxOTA2MzBaMG8xCzAJBgNVBAYT +AlNFMRQwEgYDVQQKEwtBZGRUcnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0 +ZXJuYWwgVFRQIE5ldHdvcmsxIjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENB +IFJvb3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC39xoz5vIABC05 +4E5b7R+8bA/Ntfojts7emxEzl6QpTH2Tn71KvJPtAxrjj8/lbVBa1pcplFqAsEl6 +2y6V/bjKvzc4LR4+kUGtcFbH8E8/6DKedMrIkFTpxl8PeJ2aQDwOrGGqXhSPnoeh +alDc15pOrwWzpnGUnHGzUGAKxxOdOAeGAqjpqGkmGJCrTLBPI6s6T4TY386f4Wlv +u9dC12tE5Met7m1BX3JacQg3s3llpFmglDf3AC8NwpJy2tA4ctsUqEXEXSp9t7TW +xO6szRNEt8kr3UMAJfphuWlqWCMRt6czj1Z1WfXNKddGtworZbbTQm8Vsrh7++/p +XVPVNFonAgMBAAGjgdgwgdUwHwYDVR0jBBgwFoAUUzLRs89/+uDxoF2FTpLSnkUd +tE8wHQYDVR0OBBYEFK29mHo0tCb3+sQmVO8DveAky1QaMA4GA1UdDwEB/wQEAwIB +BjAPBgNVHRMBAf8EBTADAQH/MBEGCWCGSAGG+EIBAQQEAwIBAjAgBgNVHSUEGTAX +BgorBgEEAYI3CgMDBglghkgBhvhCBAEwPQYDVR0fBDYwNDAyoDCgLoYsaHR0cDov +L2NybC51c2VydHJ1c3QuY29tL1VUTi1EQVRBQ29ycFNHQy5jcmwwDQYJKoZIhvcN +AQEFBQADggEBAMbuUxdoFLJRIh6QWA2U/b3xcOWGLcM2MY9USEbnLQg3vGwKYOEO +rVE04BKT6b64q7gmtOmWPSiPrmQH/uAB7MXjkesYoPF1ftsK5p+R26+udd8jkWjd +FwBaS/9kbHDrARrQkNnHptZt9hPk/7XJ0h4qy7ElQyZ42TCbTg0evmnv3+r+LbPM ++bDdtRTKkdSytaX7ARmjR3mfnYyVhzT4HziS2jamEfpr62vp3EV4FTkG101B5CHI +3C+H0be/SGB1pWLLJN47YaApIKa+xWycxOkKaSLvkTr6Jq/RW0GnOuL4OAdCq8Fb ++M5tug8EPzI0rNwEKNdwMBQmBsTkm5jVz3g= -----END CERTIFICATE----- - 3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware - i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware + 3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC + i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC -----BEGIN CERTIFICATE----- -MIIEdDCCA1ygAwIBAgIQRL4Mi1AAJLQR0zYq/mUK/TANBgkqhkiG9w0BAQUFADCB -lzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug +MIIEXjCCA0agAwIBAgIQRL4Mi1AAIbQR0ypoBqmtaTANBgkqhkiG9w0BAQUFADCB +kzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug Q2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho -dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3Qt -SGFyZHdhcmUwHhcNOTkwNzA5MTgxMDQyWhcNMTkwNzA5MTgxOTIyWjCBlzELMAkG -A1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEe -MBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v -d3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3QtSGFyZHdh -cmUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx98M4P7Sof885glFn -0G2f0v9Y8+efK+wNiVSZuTiZFvfgIXlIwrthdBKWHTxqctU8EGc6Oe0rE81m65UJ -M6Rsl7HoxuzBdXmcRl6Nq9Bq/bkqVRcQVLMZ8Jr28bFdtqdt++BxF2uiiPsA3/4a -MXcMmgF6sTLjKwEHOG7DpV4jvEWbe1DByTCP2+UretNb+zNAHqDVmBe8i4fDidNd -oI6yqqr2jmmIBsX6iSHzCJ1pLgkzmykNRg+MzEk0sGlRvfkGzWitZky8PqxhvQqI -DsjfPe58BEydCl5rkdbux+0ojatNh4lz0G6k0B4WixThdkQDf2Os5M1JnMWS9Ksy -oUhbAgMBAAGjgbkwgbYwCwYDVR0PBAQDAgHGMA8GA1UdEwEB/wQFMAMBAf8wHQYD -VR0OBBYEFKFyXyYbKJhDlV0HN9WFlp1L0sNFMEQGA1UdHwQ9MDswOaA3oDWGM2h0 -dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VVE4tVVNFUkZpcnN0LUhhcmR3YXJlLmNy -bDAxBgNVHSUEKjAoBggrBgEFBQcDAQYIKwYBBQUHAwUGCCsGAQUFBwMGBggrBgEF -BQcDBzANBgkqhkiG9w0BAQUFAAOCAQEARxkP3nTGmZev/K0oXnWO6y1n7k57K9cM -//bey1WiCuFMVGWTYGufEpytXoMs61quwOQt9ABjHbjAbPLPSbtNk28Gpgoiskli -CE7/yMgUsogWXecB5BKV5UU0s4tpvc+0hY91UZ59Ojg6FEgSxvunOxqNDYJAB+gE -CJChicsZUN/KHAG8HQQZexB2lzvukJDKxA4fFm517zP4029bHpbj4HR3dHuKom4t -3XbWOTCC8KucUvIqx69JXn7HaOWCgchqJ/kniCrVWFCVH/A7HFe7fRQ5YiuayZSS -KqMiDP+JJn1fIytH1xUdqWqeUQ0qUZ6B+dQ7XnASfxAynB67nfhmqA== +dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xGzAZBgNVBAMTElVUTiAtIERBVEFDb3Jw +IFNHQzAeFw05OTA2MjQxODU3MjFaFw0xOTA2MjQxOTA2MzBaMIGTMQswCQYDVQQG +EwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYD +VQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cu +dXNlcnRydXN0LmNvbTEbMBkGA1UEAxMSVVROIC0gREFUQUNvcnAgU0dDMIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3+5YEKIrblXEjr8uRgnn4AgPLit6 +E5Qbvfa2gI5lBZMAHryv4g+OGQ0SR+ysraP6LnD43m77VkIVni5c7yPeIbkFdicZ +D0/Ww5y0vpQZY/KmEQrrU0icvvIpOxboGqBMpsn0GFlowHDyUwDAXlCCpVZvNvlK +4ESGoE1O1kduSUrLZ9emxAW5jh70/P/N5zbgnAVssjMiFdC04MwXwLLA9P4yPykq +lXvY8qdOD1R8oQ2AswkDwf9c3V6aPryuvEeKaq5xyh+xKrhfQgUL7EYw0XILyulW +bfXv33i+Ybqypa4ETLyorGkVl73v67SMvzX41MPRKA5cOp9wGDMgd8SirwIDAQAB +o4GrMIGoMAsGA1UdDwQEAwIBxjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRT +MtGzz3/64PGgXYVOktKeRR20TzA9BgNVHR8ENjA0MDKgMKAuhixodHRwOi8vY3Js +LnVzZXJ0cnVzdC5jb20vVVROLURBVEFDb3JwU0dDLmNybDAqBgNVHSUEIzAhBggr +BgEFBQcDAQYKKwYBBAGCNwoDAwYJYIZIAYb4QgQBMA0GCSqGSIb3DQEBBQUAA4IB +AQAnNZcAiosovcYzMB4p/OL31ZjUQLtgyr+rFywJNn9Q+kHcrpY6CiM+iVnJowft +Gzet/Hy+UUla3joKVAgWRcKZsYfNjGjgaQPpxE6YsjuMFrMOoAyYUJuTqXAJyCyj +j98C5OBxOvG0I3KgqgHf35g+FFCgMSa9KOlaMCZ1+XtgHI3zzVAmbQQnmt/VDUVH +KWss5nbZqSl9Mt3JNjy9rjXxEZ4du5A/EkdOjtd+D2JzHVImOBwYSf0wdJrE5SIv +2MCN7ZF6TACPcn9d2t0bi0Vr591pl6jFVkwPDPafepE39peC4N1xaf92P2BNPM/3 +mfnGV/TJVTl4uix5yaaIK/QI -----END CERTIFICATE----- diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -713,13 +713,18 @@ # SHA256 was added in OpenSSL 0.9.8 if ssl.OPENSSL_VERSION_INFO < (0, 9, 8, 0, 15): self.skipTest("SHA256 not available on %r" % ssl.OPENSSL_VERSION) + # sha256.tbs-internet.com needs SNI to use the correct certificate + if not ssl.HAS_SNI: + self.skipTest("SNI needed for this test") # https://sha2.hboeck.de/ was used until 2011-01-08 (no route to host) remote = ("sha256.tbs-internet.com", 443) sha256_cert = os.path.join(os.path.dirname(__file__), "sha256.pem") with support.transient_internet("sha256.tbs-internet.com"): - s = ssl.wrap_socket(socket.socket(socket.AF_INET), - cert_reqs=ssl.CERT_REQUIRED, - ca_certs=sha256_cert,) + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + ctx.verify_mode = ssl.CERT_REQUIRED + ctx.load_verify_locations(sha256_cert) + s = ctx.wrap_socket(socket.socket(socket.AF_INET), + server_hostname="sha256.tbs-internet.com") try: s.connect(remote) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 16:28:50 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 16:28:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Fix_test_connecting_to_sha256=2Etbs-internet=2Ecom=2E?= Message-ID: http://hg.python.org/cpython/rev/f8c9d131eb66 changeset: 76750:f8c9d131eb66 parent: 76748:4deb7c1f49b9 parent: 76749:683295cf9489 user: Antoine Pitrou date: Fri May 04 16:26:56 2012 +0200 summary: Fix test connecting to sha256.tbs-internet.com. The certificate has changed and the test now needs SNI to pass. files: Lib/test/sha256.pem | 222 +++++++++++++++--------------- Lib/test/test_ssl.py | 11 +- 2 files changed, 119 insertions(+), 114 deletions(-) diff --git a/Lib/test/sha256.pem b/Lib/test/sha256.pem --- a/Lib/test/sha256.pem +++ b/Lib/test/sha256.pem @@ -1,128 +1,128 @@ # Certificate chain for https://sha256.tbs-internet.com - 0 s:/C=FR/postalCode=14000/ST=Calvados/L=CAEN/street=22 rue de Bretagne/O=TBS INTERNET/OU=0002 440443810/OU=Certificats TBS X509/CN=ecom.tbs-x509.com - i:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA business + 0 s:/C=FR/postalCode=14000/ST=Calvados/L=CAEN/street=22 rue de Bretagne/O=TBS INTERNET/OU=0002 440443810/OU=sha-256 production/CN=sha256.tbs-internet.com + i:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA SGC -----BEGIN CERTIFICATE----- -MIIGTjCCBTagAwIBAgIQOh3d9dNDPq1cSdJmEiMpqDANBgkqhkiG9w0BAQUFADCB -yTELMAkGA1UEBhMCRlIxETAPBgNVBAgTCENhbHZhZG9zMQ0wCwYDVQQHEwRDYWVu -MRUwEwYDVQQKEwxUQlMgSU5URVJORVQxSDBGBgNVBAsTP1Rlcm1zIGFuZCBDb25k -aXRpb25zOiBodHRwOi8vd3d3LnRicy1pbnRlcm5ldC5jb20vQ0EvcmVwb3NpdG9y -eTEYMBYGA1UECxMPVEJTIElOVEVSTkVUIENBMR0wGwYDVQQDExRUQlMgWDUwOSBD -QSBidXNpbmVzczAeFw0xMTAxMjUwMDAwMDBaFw0xMzAyMDUyMzU5NTlaMIHHMQsw -CQYDVQQGEwJGUjEOMAwGA1UEERMFMTQwMDAxETAPBgNVBAgTCENhbHZhZG9zMQ0w -CwYDVQQHEwRDQUVOMRswGQYDVQQJExIyMiBydWUgZGUgQnJldGFnbmUxFTATBgNV -BAoTDFRCUyBJTlRFUk5FVDEXMBUGA1UECxMOMDAwMiA0NDA0NDM4MTAxHTAbBgNV -BAsTFENlcnRpZmljYXRzIFRCUyBYNTA5MRowGAYDVQQDExFlY29tLnRicy14NTA5 -LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKRrlHUnJ++1lpcg -jtYco7cdmRe+EEfTmwPfCdfV3G1QfsTSvY6FfMpm/83pqHfT+4ANwr18wD9ZrAEN -G16mf9VdCGK12+TP7DmqeZyGIqlFFoahQnmb8EarvE43/1UeQ2CV9XmzwZvpqeli -LfXsFonawrY3H6ZnMwS64St61Z+9gdyuZ/RbsoZBbT5KUjDEG844QRU4OT1IGeEI -eY5NM5RNIh6ZNhVtqeeCxMS7afONkHQrOco73RdSTRck/Hj96Ofl3MHNHryr+AMK -DGFk1kLCZGpPdXtkxXvaDeQoiYDlil26CWc+YK6xyDPMdsWvoG14ZLyCpzMXA7/7 -4YAQRH0CAwEAAaOCAjAwggIsMB8GA1UdIwQYMBaAFBoJBMz5CY+7HqDO1KQUf0vV -I1jNMB0GA1UdDgQWBBQgOU8HsWzbmD4WZP5Wtdw7jca2WDAOBgNVHQ8BAf8EBAMC -BaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw -TAYDVR0gBEUwQzBBBgsrBgEEAYDlNwIBATAyMDAGCCsGAQUFBwIBFiRodHRwczov -L3d3dy50YnMtaW50ZXJuZXQuY29tL0NBL0NQUzEwdwYDVR0fBHAwbjA3oDWgM4Yx -aHR0cDovL2NybC50YnMtaW50ZXJuZXQuY29tL1RCU1g1MDlDQWJ1c2luZXNzLmNy -bDAzoDGgL4YtaHR0cDovL2NybC50YnMteDUwOS5jb20vVEJTWDUwOUNBYnVzaW5l -c3MuY3JsMIGwBggrBgEFBQcBAQSBozCBoDA9BggrBgEFBQcwAoYxaHR0cDovL2Ny -dC50YnMtaW50ZXJuZXQuY29tL1RCU1g1MDlDQWJ1c2luZXNzLmNydDA5BggrBgEF -BQcwAoYtaHR0cDovL2NydC50YnMteDUwOS5jb20vVEJTWDUwOUNBYnVzaW5lc3Mu -Y3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC50YnMteDUwOS5jb20wMwYDVR0R -BCwwKoIRZWNvbS50YnMteDUwOS5jb22CFXd3dy5lY29tLnRicy14NTA5LmNvbTAN -BgkqhkiG9w0BAQUFAAOCAQEArT4NHfbY87bGAw8lPV4DmHlmuDuVp/y7ltO3Ynse -3Rz8RxW2AzuO0Oy2F0Cu4yWKtMyEyMXyHqWtae7ElRbdTu5w5GwVBLJHClCzC8S9 -SpgMMQTx3Rgn8vjkHuU9VZQlulZyiPK7yunjc7c310S9FRZ7XxOwf8Nnx4WnB+No -WrfApzhhQl31w+RyrNxZe58hCfDDHmevRvwLjQ785ZoQXJDj2j3qAD4aI2yB8lB5 -oaE1jlCJzC7Kmz/Y9jzfmv/zAs1LQTm9ktevv4BTUFaGjv9jxnQ1xnS862ZiouLW -zZYIlYPf4F6JjXGiIQgQRglILUfq3ftJd9/ok9W9ZF8h8w== +MIIGXDCCBUSgAwIBAgIRAKpVmHgg9nfCodAVwcP4siwwDQYJKoZIhvcNAQELBQAw +gcQxCzAJBgNVBAYTAkZSMREwDwYDVQQIEwhDYWx2YWRvczENMAsGA1UEBxMEQ2Fl +bjEVMBMGA1UEChMMVEJTIElOVEVSTkVUMUgwRgYDVQQLEz9UZXJtcyBhbmQgQ29u +ZGl0aW9uczogaHR0cDovL3d3dy50YnMtaW50ZXJuZXQuY29tL0NBL3JlcG9zaXRv +cnkxGDAWBgNVBAsTD1RCUyBJTlRFUk5FVCBDQTEYMBYGA1UEAxMPVEJTIFg1MDkg +Q0EgU0dDMB4XDTEyMDEwNDAwMDAwMFoXDTE0MDIxNzIzNTk1OVowgcsxCzAJBgNV +BAYTAkZSMQ4wDAYDVQQREwUxNDAwMDERMA8GA1UECBMIQ2FsdmFkb3MxDTALBgNV +BAcTBENBRU4xGzAZBgNVBAkTEjIyIHJ1ZSBkZSBCcmV0YWduZTEVMBMGA1UEChMM +VEJTIElOVEVSTkVUMRcwFQYDVQQLEw4wMDAyIDQ0MDQ0MzgxMDEbMBkGA1UECxMS +c2hhLTI1NiBwcm9kdWN0aW9uMSAwHgYDVQQDExdzaGEyNTYudGJzLWludGVybmV0 +LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKQIX/zdJcyxty0m +PM1XQSoSSifueS3AVcgqMsaIKS/u+rYzsv4hQ/qA6vLn5m5/ewUcZDj7zdi6rBVf +PaVNXJ6YinLX0tkaW8TEjeVuZG5yksGZlhCt1CJ1Ho9XLiLaP4uJ7MCoNUntpJ+E +LfrOdgsIj91kPmwjDJeztVcQCvKzhjVJA/KxdInc0JvOATn7rpaSmQI5bvIjufgo +qVsTPwVFzuUYULXBk7KxRT7MiEqnd5HvviNh0285QC478zl3v0I0Fb5El4yD3p49 +IthcRnxzMKc0UhU5ogi0SbONyBfm/mzONVfSxpM+MlyvZmJqrbuuLoEDzJD+t8PU +xSuzgbcCAwEAAaOCAj4wggI6MB8GA1UdIwQYMBaAFAdEdoWTKLx/bXjSCuv6TEvf +2YIfMB0GA1UdDgQWBBT/qTGYdaj+f61c2IRFL/B1eEsM8DAOBgNVHQ8BAf8EBAMC +BaAwDAYDVR0TAQH/BAIwADA0BgNVHSUELTArBggrBgEFBQcDAQYIKwYBBQUHAwIG +CisGAQQBgjcKAwMGCWCGSAGG+EIEATBLBgNVHSAERDBCMEAGCisGAQQB5TcCBAEw +MjAwBggrBgEFBQcCARYkaHR0cHM6Ly93d3cudGJzLWludGVybmV0LmNvbS9DQS9D +UFM0MG0GA1UdHwRmMGQwMqAwoC6GLGh0dHA6Ly9jcmwudGJzLWludGVybmV0LmNv +bS9UQlNYNTA5Q0FTR0MuY3JsMC6gLKAqhihodHRwOi8vY3JsLnRicy14NTA5LmNv +bS9UQlNYNTA5Q0FTR0MuY3JsMIGmBggrBgEFBQcBAQSBmTCBljA4BggrBgEFBQcw +AoYsaHR0cDovL2NydC50YnMtaW50ZXJuZXQuY29tL1RCU1g1MDlDQVNHQy5jcnQw +NAYIKwYBBQUHMAKGKGh0dHA6Ly9jcnQudGJzLXg1MDkuY29tL1RCU1g1MDlDQVNH +Qy5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnRicy14NTA5LmNvbTA/BgNV +HREEODA2ghdzaGEyNTYudGJzLWludGVybmV0LmNvbYIbd3d3LnNoYTI1Ni50YnMt +aW50ZXJuZXQuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQA0pOuL8QvAa5yksTbGShzX +ABApagunUGoEydv4YJT1MXy9tTp7DrWaozZSlsqBxrYAXP1d9r2fuKbEniYHxaQ0 +UYaf1VSIlDo1yuC8wE7wxbHDIpQ/E5KAyxiaJ8obtDhFstWAPAH+UoGXq0kj2teN +21sFQ5dXgA95nldvVFsFhrRUNB6xXAcaj0VZFhttI0ZfQZmQwEI/P+N9Jr40OGun +aa+Dn0TMeUH4U20YntfLbu2nDcJcYfyurm+8/0Tr4HznLnedXu9pCPYj0TaddrgT +XO0oFiyy7qGaY6+qKh71yD64Y3ycCJ/HR9Wm39mjZYc9ezYwT4noP6r7Lk8YO7/q -----END CERTIFICATE----- - 1 s:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA business + 1 s:/C=FR/ST=Calvados/L=Caen/O=TBS INTERNET/OU=Terms and Conditions: http://www.tbs-internet.com/CA/repository/OU=TBS INTERNET CA/CN=TBS X509 CA SGC i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root -----BEGIN CERTIFICATE----- -MIIFPzCCBCegAwIBAgIQDlBz/++iRSmLDeVRHT/hADANBgkqhkiG9w0BAQUFADBv +MIIFVjCCBD6gAwIBAgIQXpDZ0ETJMV02WTx3GTnhhTANBgkqhkiG9w0BAQUFADBv MQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFk ZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBF -eHRlcm5hbCBDQSBSb290MB4XDTA1MTIwMTAwMDAwMFoXDTE5MDcwOTE4MTkyMlow -gckxCzAJBgNVBAYTAkZSMREwDwYDVQQIEwhDYWx2YWRvczENMAsGA1UEBxMEQ2Fl +eHRlcm5hbCBDQSBSb290MB4XDTA1MTIwMTAwMDAwMFoXDTE5MDYyNDE5MDYzMFow +gcQxCzAJBgNVBAYTAkZSMREwDwYDVQQIEwhDYWx2YWRvczENMAsGA1UEBxMEQ2Fl bjEVMBMGA1UEChMMVEJTIElOVEVSTkVUMUgwRgYDVQQLEz9UZXJtcyBhbmQgQ29u ZGl0aW9uczogaHR0cDovL3d3dy50YnMtaW50ZXJuZXQuY29tL0NBL3JlcG9zaXRv -cnkxGDAWBgNVBAsTD1RCUyBJTlRFUk5FVCBDQTEdMBsGA1UEAxMUVEJTIFg1MDkg -Q0EgYnVzaW5lc3MwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDB1PAU -qudCcz3tmyGcf+u6EkZqonKKHrV4gZYbvVkIRojmmlhfi/jwvpHvo8bqSt/9Rj5S -jhCDW0pcbI+IPPtD1Jy+CHNSfnMqVDy6CKQ3p5maTzCMG6ZT+XjnvcND5v+FtaiB -xk1iCX6uvt0jeUtdZvYbyytsSDE6c3Y5//wRxOF8tM1JxibwO3pyER26jbbN2gQz -m/EkdGjLdJ4svPk23WDAvQ6G0/z2LcAaJB+XLfqRwfQpHQvfKa1uTi8PivC8qtip -rmNQMMPMjxSK2azX8cKjjTDJiUKaCb4VHlJDWKEsCFRpgJAoAuX8f7Yfs1M4esGo -sWb3PGspK3O22uIlAgMBAAGjggF6MIIBdjAdBgNVHQ4EFgQUGgkEzPkJj7seoM7U -pBR/S9UjWM0wDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwGAYD -VR0gBBEwDzANBgsrBgEEAYDlNwIBATB7BgNVHR8EdDByMDigNqA0hjJodHRwOi8v -Y3JsLmNvbW9kb2NhLmNvbS9BZGRUcnVzdEV4dGVybmFsQ0FSb290LmNybDA2oDSg -MoYwaHR0cDovL2NybC5jb21vZG8ubmV0L0FkZFRydXN0RXh0ZXJuYWxDQVJvb3Qu -Y3JsMIGGBggrBgEFBQcBAQR6MHgwOwYIKwYBBQUHMAKGL2h0dHA6Ly9jcnQuY29t -b2RvY2EuY29tL0FkZFRydXN0VVROU2VydmVyQ0EuY3J0MDkGCCsGAQUFBzAChi1o -dHRwOi8vY3J0LmNvbW9kby5uZXQvQWRkVHJ1c3RVVE5TZXJ2ZXJDQS5jcnQwEQYJ -YIZIAYb4QgEBBAQDAgIEMA0GCSqGSIb3DQEBBQUAA4IBAQA7mqrMgk/MrE6QnbNA -h4nRCn2ti4bg4w2C3lB6bSvRPnYwuNw9Jb8vuKkNFzRDxNJXqVDZdfFW5CVQJuyd -nfAx83+wk+spzvFaE1KhFYfN9G9pQfXUfvDRoIcJgPEKUXL1wRiOG+IjU3VVI8pg -IgqHkr7ylln5i5zCiFAPuIJmYUSFg/gxH5xkCNcjJqqrHrHatJr6Qrrke93joupw -oU1njfAcZtYp6fbiK6u2b1pJqwkVBE8RsfLnPhRj+SFbpvjv8Od7o/ieJhFIYQNU -k2jX2u8qZnAiNw93LZW9lpYjtuvMXq8QQppENNja5b53q7UwI+lU7ZGjZ7quuESp -J6/5 +cnkxGDAWBgNVBAsTD1RCUyBJTlRFUk5FVCBDQTEYMBYGA1UEAxMPVEJTIFg1MDkg +Q0EgU0dDMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsgOkO3f7wzN6 +rOjg45tR5vjBfzK7qmV9IBxb/QW9EEXxG+E7FNhZqQLtwGBKoSsHTnQqV75wWMk0 +9tinWvftBkSpj5sTi/8cbzJfUvTSVYh3Qxv6AVVjMMH/ruLjE6y+4PoaPs8WoYAQ +ts5R4Z1g8c/WnTepLst2x0/Wv7GmuoQi+gXvHU6YrBiu7XkeYhzc95QdviWSJRDk +owhb5K43qhcvjRmBfO/paGlCliDGZp8mHwrI21mwobWpVjTxZRwYO3bd4+TGcI4G +Ie5wmHwE8F7SK1tgSqbBacKjDa93j7txKkfz/Yd2n7TGqOXiHPsJpG655vrKtnXk +9vs1zoDeJQIDAQABo4IBljCCAZIwHQYDVR0OBBYEFAdEdoWTKLx/bXjSCuv6TEvf +2YIfMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEAMCAGA1UdJQQZ +MBcGCisGAQQBgjcKAwMGCWCGSAGG+EIEATAYBgNVHSAEETAPMA0GCysGAQQBgOU3 +AgQBMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0Fk +ZFRydXN0RXh0ZXJuYWxDQVJvb3QuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k +by5uZXQvQWRkVHJ1c3RFeHRlcm5hbENBUm9vdC5jcmwwgYAGCCsGAQUFBwEBBHQw +cjA4BggrBgEFBQcwAoYsaHR0cDovL2NydC5jb21vZG9jYS5jb20vQWRkVHJ1c3RV +VE5TR0NDQS5jcnQwNgYIKwYBBQUHMAKGKmh0dHA6Ly9jcnQuY29tb2RvLm5ldC9B +ZGRUcnVzdFVUTlNHQ0NBLmNydDARBglghkgBhvhCAQEEBAMCAgQwDQYJKoZIhvcN +AQEFBQADggEBAK2zEzs+jcIrVK9oDkdDZNvhuBYTdCfpxfFs+OAujW0bIfJAy232 +euVsnJm6u/+OrqKudD2tad2BbejLLXhMZViaCmK7D9nrXHx4te5EP8rL19SUVqLY +1pTnv5dhNgEgvA7n5lIzDSYs7yRLsr7HJsYPr6SeYSuZizyX1SNz7ooJ32/F3X98 +RB0Mlc/E0OyOrkQ9/y5IrnpnaSora8CnUrV5XNOg+kyCz9edCyx4D5wXYcwZPVWz +8aDqquESrezPyjtfi4WRO4s/VD3HLZvOxzMrWAVYCDG9FxaOhF0QGuuG1F7F3GKV +v6prNyCl016kRl2j1UT+a7gLd8fA25A4C9E= -----END CERTIFICATE----- 2 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root - i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware + i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC -----BEGIN CERTIFICATE----- -MIIETzCCAzegAwIBAgIQHM5EYpUZep1jUvnyI6m2mDANBgkqhkiG9w0BAQUFADCB -lzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug +MIIEZjCCA06gAwIBAgIQUSYKkxzif5zDpV954HKugjANBgkqhkiG9w0BAQUFADCB +kzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug Q2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho -dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3Qt -SGFyZHdhcmUwHhcNMDUwNjA3MDgwOTEwWhcNMTkwNzA5MTgxOTIyWjBvMQswCQYD -VQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0 -IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5h -bCBDQSBSb290MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt/caM+by -AAQtOeBOW+0fvGwPzbX6I7bO3psRM5ekKUx9k5+9SryT7QMa44/P5W1QWtaXKZRa -gLBJetsulf24yr83OC0ePpFBrXBWx/BPP+gynnTKyJBU6cZfD3idmkA8Dqxhql4U -j56HoWpQ3NeaTq8Fs6ZxlJxxs1BgCscTnTgHhgKo6ahpJhiQq0ywTyOrOk+E2N/O -n+Fpb7vXQtdrROTHre5tQV9yWnEIN7N5ZaRZoJQ39wAvDcKSctrQOHLbFKhFxF0q -fbe01sTurM0TRLfJK91DACX6YblpalgjEbenM49WdVn1zSnXRrcKK2W200JvFbK4 -e/vv6V1T1TRaJwIDAQABo4G9MIG6MB8GA1UdIwQYMBaAFKFyXyYbKJhDlV0HN9WF -lp1L0sNFMB0GA1UdDgQWBBStvZh6NLQm9/rEJlTvA73gJMtUGjAOBgNVHQ8BAf8E -BAMCAQYwDwYDVR0TAQH/BAUwAwEB/zARBglghkgBhvhCAQEEBAMCAQIwRAYDVR0f -BD0wOzA5oDegNYYzaHR0cDovL2NybC51c2VydHJ1c3QuY29tL1VUTi1VU0VSRmly -c3QtSGFyZHdhcmUuY3JsMA0GCSqGSIb3DQEBBQUAA4IBAQByQhANOs4kClrwF8BW -onvUOGCSjRK52zYZgDXYNjDtmr5rJ6NyPFDNn+JxkLpjYetIFMTbSRe679Bt8m7a -gIAoQYFQtxMuyLnJegB2aEbQiIxh/tC21UcFF7ktdnDoTlA6w3pLuvunaI84Of3o -2YBrhzkTbCfaYk5JRlTpudW9DkUkHBsyx3nknPKnplkIGaK0jgn8E0n+SFabYaHk -I9LroYT/+JtLefh9lgBdAgVv0UPbzoGfuDsrk/Zh+UrgbLFpHoVnElhzbkh64Z0X -OGaJunQc68cCZu5HTn/aK7fBGMcVflRCXLVEQpU9PIAdGA8Ynvg684t8GMaKsRl1 -jIGZ +dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xGzAZBgNVBAMTElVUTiAtIERBVEFDb3Jw +IFNHQzAeFw0wNTA2MDcwODA5MTBaFw0xOTA2MjQxOTA2MzBaMG8xCzAJBgNVBAYT +AlNFMRQwEgYDVQQKEwtBZGRUcnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0 +ZXJuYWwgVFRQIE5ldHdvcmsxIjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENB +IFJvb3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC39xoz5vIABC05 +4E5b7R+8bA/Ntfojts7emxEzl6QpTH2Tn71KvJPtAxrjj8/lbVBa1pcplFqAsEl6 +2y6V/bjKvzc4LR4+kUGtcFbH8E8/6DKedMrIkFTpxl8PeJ2aQDwOrGGqXhSPnoeh +alDc15pOrwWzpnGUnHGzUGAKxxOdOAeGAqjpqGkmGJCrTLBPI6s6T4TY386f4Wlv +u9dC12tE5Met7m1BX3JacQg3s3llpFmglDf3AC8NwpJy2tA4ctsUqEXEXSp9t7TW +xO6szRNEt8kr3UMAJfphuWlqWCMRt6czj1Z1WfXNKddGtworZbbTQm8Vsrh7++/p +XVPVNFonAgMBAAGjgdgwgdUwHwYDVR0jBBgwFoAUUzLRs89/+uDxoF2FTpLSnkUd +tE8wHQYDVR0OBBYEFK29mHo0tCb3+sQmVO8DveAky1QaMA4GA1UdDwEB/wQEAwIB +BjAPBgNVHRMBAf8EBTADAQH/MBEGCWCGSAGG+EIBAQQEAwIBAjAgBgNVHSUEGTAX +BgorBgEEAYI3CgMDBglghkgBhvhCBAEwPQYDVR0fBDYwNDAyoDCgLoYsaHR0cDov +L2NybC51c2VydHJ1c3QuY29tL1VUTi1EQVRBQ29ycFNHQy5jcmwwDQYJKoZIhvcN +AQEFBQADggEBAMbuUxdoFLJRIh6QWA2U/b3xcOWGLcM2MY9USEbnLQg3vGwKYOEO +rVE04BKT6b64q7gmtOmWPSiPrmQH/uAB7MXjkesYoPF1ftsK5p+R26+udd8jkWjd +FwBaS/9kbHDrARrQkNnHptZt9hPk/7XJ0h4qy7ElQyZ42TCbTg0evmnv3+r+LbPM ++bDdtRTKkdSytaX7ARmjR3mfnYyVhzT4HziS2jamEfpr62vp3EV4FTkG101B5CHI +3C+H0be/SGB1pWLLJN47YaApIKa+xWycxOkKaSLvkTr6Jq/RW0GnOuL4OAdCq8Fb ++M5tug8EPzI0rNwEKNdwMBQmBsTkm5jVz3g= -----END CERTIFICATE----- - 3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware - i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware + 3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC + i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC -----BEGIN CERTIFICATE----- -MIIEdDCCA1ygAwIBAgIQRL4Mi1AAJLQR0zYq/mUK/TANBgkqhkiG9w0BAQUFADCB -lzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug +MIIEXjCCA0agAwIBAgIQRL4Mi1AAIbQR0ypoBqmtaTANBgkqhkiG9w0BAQUFADCB +kzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2Ug Q2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho -dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3Qt -SGFyZHdhcmUwHhcNOTkwNzA5MTgxMDQyWhcNMTkwNzA5MTgxOTIyWjCBlzELMAkG -A1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEe -MBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8v -d3d3LnVzZXJ0cnVzdC5jb20xHzAdBgNVBAMTFlVUTi1VU0VSRmlyc3QtSGFyZHdh -cmUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx98M4P7Sof885glFn -0G2f0v9Y8+efK+wNiVSZuTiZFvfgIXlIwrthdBKWHTxqctU8EGc6Oe0rE81m65UJ -M6Rsl7HoxuzBdXmcRl6Nq9Bq/bkqVRcQVLMZ8Jr28bFdtqdt++BxF2uiiPsA3/4a -MXcMmgF6sTLjKwEHOG7DpV4jvEWbe1DByTCP2+UretNb+zNAHqDVmBe8i4fDidNd -oI6yqqr2jmmIBsX6iSHzCJ1pLgkzmykNRg+MzEk0sGlRvfkGzWitZky8PqxhvQqI -DsjfPe58BEydCl5rkdbux+0ojatNh4lz0G6k0B4WixThdkQDf2Os5M1JnMWS9Ksy -oUhbAgMBAAGjgbkwgbYwCwYDVR0PBAQDAgHGMA8GA1UdEwEB/wQFMAMBAf8wHQYD -VR0OBBYEFKFyXyYbKJhDlV0HN9WFlp1L0sNFMEQGA1UdHwQ9MDswOaA3oDWGM2h0 -dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VVE4tVVNFUkZpcnN0LUhhcmR3YXJlLmNy -bDAxBgNVHSUEKjAoBggrBgEFBQcDAQYIKwYBBQUHAwUGCCsGAQUFBwMGBggrBgEF -BQcDBzANBgkqhkiG9w0BAQUFAAOCAQEARxkP3nTGmZev/K0oXnWO6y1n7k57K9cM -//bey1WiCuFMVGWTYGufEpytXoMs61quwOQt9ABjHbjAbPLPSbtNk28Gpgoiskli -CE7/yMgUsogWXecB5BKV5UU0s4tpvc+0hY91UZ59Ojg6FEgSxvunOxqNDYJAB+gE -CJChicsZUN/KHAG8HQQZexB2lzvukJDKxA4fFm517zP4029bHpbj4HR3dHuKom4t -3XbWOTCC8KucUvIqx69JXn7HaOWCgchqJ/kniCrVWFCVH/A7HFe7fRQ5YiuayZSS -KqMiDP+JJn1fIytH1xUdqWqeUQ0qUZ6B+dQ7XnASfxAynB67nfhmqA== +dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xGzAZBgNVBAMTElVUTiAtIERBVEFDb3Jw +IFNHQzAeFw05OTA2MjQxODU3MjFaFw0xOTA2MjQxOTA2MzBaMIGTMQswCQYDVQQG +EwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYD +VQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cu +dXNlcnRydXN0LmNvbTEbMBkGA1UEAxMSVVROIC0gREFUQUNvcnAgU0dDMIIBIjAN +BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3+5YEKIrblXEjr8uRgnn4AgPLit6 +E5Qbvfa2gI5lBZMAHryv4g+OGQ0SR+ysraP6LnD43m77VkIVni5c7yPeIbkFdicZ +D0/Ww5y0vpQZY/KmEQrrU0icvvIpOxboGqBMpsn0GFlowHDyUwDAXlCCpVZvNvlK +4ESGoE1O1kduSUrLZ9emxAW5jh70/P/N5zbgnAVssjMiFdC04MwXwLLA9P4yPykq +lXvY8qdOD1R8oQ2AswkDwf9c3V6aPryuvEeKaq5xyh+xKrhfQgUL7EYw0XILyulW +bfXv33i+Ybqypa4ETLyorGkVl73v67SMvzX41MPRKA5cOp9wGDMgd8SirwIDAQAB +o4GrMIGoMAsGA1UdDwQEAwIBxjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRT +MtGzz3/64PGgXYVOktKeRR20TzA9BgNVHR8ENjA0MDKgMKAuhixodHRwOi8vY3Js +LnVzZXJ0cnVzdC5jb20vVVROLURBVEFDb3JwU0dDLmNybDAqBgNVHSUEIzAhBggr +BgEFBQcDAQYKKwYBBAGCNwoDAwYJYIZIAYb4QgQBMA0GCSqGSIb3DQEBBQUAA4IB +AQAnNZcAiosovcYzMB4p/OL31ZjUQLtgyr+rFywJNn9Q+kHcrpY6CiM+iVnJowft +Gzet/Hy+UUla3joKVAgWRcKZsYfNjGjgaQPpxE6YsjuMFrMOoAyYUJuTqXAJyCyj +j98C5OBxOvG0I3KgqgHf35g+FFCgMSa9KOlaMCZ1+XtgHI3zzVAmbQQnmt/VDUVH +KWss5nbZqSl9Mt3JNjy9rjXxEZ4du5A/EkdOjtd+D2JzHVImOBwYSf0wdJrE5SIv +2MCN7ZF6TACPcn9d2t0bi0Vr591pl6jFVkwPDPafepE39peC4N1xaf92P2BNPM/3 +mfnGV/TJVTl4uix5yaaIK/QI -----END CERTIFICATE----- diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -830,13 +830,18 @@ # SHA256 was added in OpenSSL 0.9.8 if ssl.OPENSSL_VERSION_INFO < (0, 9, 8, 0, 15): self.skipTest("SHA256 not available on %r" % ssl.OPENSSL_VERSION) + # sha256.tbs-internet.com needs SNI to use the correct certificate + if not ssl.HAS_SNI: + self.skipTest("SNI needed for this test") # https://sha2.hboeck.de/ was used until 2011-01-08 (no route to host) remote = ("sha256.tbs-internet.com", 443) sha256_cert = os.path.join(os.path.dirname(__file__), "sha256.pem") with support.transient_internet("sha256.tbs-internet.com"): - s = ssl.wrap_socket(socket.socket(socket.AF_INET), - cert_reqs=ssl.CERT_REQUIRED, - ca_certs=sha256_cert,) + ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1) + ctx.verify_mode = ssl.CERT_REQUIRED + ctx.load_verify_locations(sha256_cert) + s = ctx.wrap_socket(socket.socket(socket.AF_INET), + server_hostname="sha256.tbs-internet.com") try: s.connect(remote) if support.verbose: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 16:35:21 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 16:35:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Skip_test=5Falg?= =?utf8?q?orithms_=28known_remote_hosts_need_SNI=2C_which_is_only_availabl?= =?utf8?q?e_on?= Message-ID: http://hg.python.org/cpython/rev/8215aaccc9dd changeset: 76751:8215aaccc9dd branch: 2.7 parent: 76739:b2031eb95dd9 user: Antoine Pitrou date: Fri May 04 16:33:30 2012 +0200 summary: Skip test_algorithms (known remote hosts need SNI, which is only available on 3.2+) files: Lib/test/test_ssl.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -355,7 +355,8 @@ # SHA256 was added in OpenSSL 0.9.8 if ssl.OPENSSL_VERSION_INFO < (0, 9, 8, 0, 15): self.skipTest("SHA256 not available on %r" % ssl.OPENSSL_VERSION) - # NOTE: https://sha256.tbs-internet.com is another possible test host + self.skipTest("remote host needs SNI, only available on Python 3.2+") + # NOTE: https://sha2.hboeck.de is another possible test host remote = ("sha256.tbs-internet.com", 443) sha256_cert = os.path.join(os.path.dirname(__file__), "sha256.pem") with test_support.transient_internet("sha256.tbs-internet.com"): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 17:06:14 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 04 May 2012 17:06:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_initialization_not_needed?= Message-ID: http://hg.python.org/cpython/rev/01824bf55376 changeset: 76752:01824bf55376 parent: 76750:f8c9d131eb66 user: Benjamin Peterson date: Fri May 04 11:06:09 2012 -0400 summary: initialization not needed files: Modules/posixmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3576,7 +3576,7 @@ split_py_long_to_s_and_ns(PyObject *py_long, time_t *s, long *ns) { int result = 0; - PyObject *divmod = NULL; + PyObject *divmod; divmod = PyNumber_Divmod(py_long, billion); if (!divmod) goto exit; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 21:20:47 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 21:20:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Properly_mark_names_in_impo?= =?utf8?q?rtlib=2E=5Fbootstrap_as_private=2E?= Message-ID: http://hg.python.org/cpython/rev/59b8a7fcb047 changeset: 76753:59b8a7fcb047 user: Brett Cannon date: Fri May 04 13:52:49 2012 -0400 summary: Properly mark names in importlib._bootstrap as private. files: Lib/imp.py | 12 ++-- Lib/importlib/_bootstrap.py | 62 ++++++++++++------------ 2 files changed, 37 insertions(+), 37 deletions(-) diff --git a/Lib/imp.py b/Lib/imp.py --- a/Lib/imp.py +++ b/Lib/imp.py @@ -15,8 +15,8 @@ # Can (probably) move to importlib from _imp import get_suffixes -from importlib._bootstrap import _new_module as new_module -from importlib._bootstrap import _cache_from_source as cache_from_source +from importlib._bootstrap import new_module +from importlib._bootstrap import cache_from_source from importlib import _bootstrap import os @@ -48,14 +48,14 @@ """ head, pycache_filename = os.path.split(path) head, pycache = os.path.split(head) - if pycache != _bootstrap.PYCACHE: + if pycache != _bootstrap._PYCACHE: raise ValueError('{} not bottom-level directory in ' - '{!r}'.format(_bootstrap.PYCACHE, path)) + '{!r}'.format(_bootstrap._PYCACHE, path)) if pycache_filename.count('.') != 2: raise ValueError('expected only 2 dots in ' '{!r}'.format(pycache_filename)) base_filename = pycache_filename.partition('.')[0] - return os.path.join(head, base_filename + _bootstrap.SOURCE_SUFFIXES[0]) + return os.path.join(head, base_filename + _bootstrap._SOURCE_SUFFIXES[0]) class NullImporter: @@ -185,7 +185,7 @@ for entry in path: package_directory = os.path.join(entry, name) - for suffix in ['.py', _bootstrap.BYTECODE_SUFFIX]: + for suffix in ['.py', _bootstrap._BYTECODE_SUFFIX]: package_file_name = '__init__' + suffix file_path = os.path.join(package_directory, package_file_name) if os.path.isfile(file_path): diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -26,11 +26,11 @@ # Bootstrap-related code ###################################################### -CASE_INSENSITIVE_PLATFORMS = 'win', 'cygwin', 'darwin' +_CASE_INSENSITIVE_PLATFORMS = 'win', 'cygwin', 'darwin' def _make_relax_case(): - if sys.platform.startswith(CASE_INSENSITIVE_PLATFORMS): + if sys.platform.startswith(_CASE_INSENSITIVE_PLATFORMS): def _relax_case(): """True if filenames must be checked case-insensitively.""" return b'PYTHONCASEOK' in _os.environ @@ -179,10 +179,10 @@ new.__dict__.update(old.__dict__) -code_type = type(_wrap.__code__) +_code_type = type(_wrap.__code__) -def _new_module(name): +def new_module(name): """Create a new module. The module is not entered into sys.modules. @@ -193,15 +193,15 @@ # Finder/loader utility code ################################################## -PYCACHE = '__pycache__' +_PYCACHE = '__pycache__' -SOURCE_SUFFIXES = ['.py'] # _setup() adds .pyw as needed. +_SOURCE_SUFFIXES = ['.py'] # _setup() adds .pyw as needed. -DEBUG_BYTECODE_SUFFIX = '.pyc' -OPT_BYTECODE_SUFFIX = '.pyo' -BYTECODE_SUFFIX = DEBUG_BYTECODE_SUFFIX if __debug__ else OPT_BYTECODE_SUFFIX +_DEBUG_BYTECODE_SUFFIX = '.pyc' +_OPT_BYTECODE_SUFFIX = '.pyo' +_BYTECODE_SUFFIX = _DEBUG_BYTECODE_SUFFIX if __debug__ else _OPT_BYTECODE_SUFFIX -def _cache_from_source(path, debug_override=None): +def cache_from_source(path, debug_override=None): """Given the path to a .py file, return the path to its .pyc/.pyo file. The .py file does not need to exist; this simply returns the path to the @@ -213,14 +213,14 @@ """ debug = __debug__ if debug_override is None else debug_override - suffix = DEBUG_BYTECODE_SUFFIX if debug else OPT_BYTECODE_SUFFIX + suffix = _DEBUG_BYTECODE_SUFFIX if debug else _OPT_BYTECODE_SUFFIX head, tail = _path_split(path) base_filename, sep, _ = tail.partition('.') filename = ''.join([base_filename, sep, _TAG, suffix]) - return _path_join(head, PYCACHE, filename) + return _path_join(head, _PYCACHE, filename) -def verbose_message(message, *args): +def _verbose_message(message, *args): """Print the message to stderr if -v/PYTHONVERBOSE is turned on.""" if sys.flags.verbose: if not message.startswith(('#', 'import ')): @@ -277,7 +277,7 @@ # This must be done before open() is called as the 'io' module # implicitly imports 'locale' and would otherwise trigger an # infinite loop. - module = _new_module(fullname) + module = new_module(fullname) sys.modules[fullname] = module module.__loader__ = self try: @@ -472,11 +472,11 @@ raise ImportError(msg, name=fullname, path=bytecode_path) elif len(raw_timestamp) != 4: message = 'bad timestamp in {}'.format(fullname) - verbose_message(message) + _verbose_message(message) raise EOFError(message) elif len(raw_size) != 4: message = 'bad size in {}'.format(fullname) - verbose_message(message) + _verbose_message(message) raise EOFError(message) if source_stats is not None: try: @@ -486,7 +486,7 @@ else: if _r_long(raw_timestamp) != source_mtime: message = 'bytecode is stale for {}'.format(fullname) - verbose_message(message) + _verbose_message(message) raise ImportError(message, name=fullname, path=bytecode_path) try: @@ -510,7 +510,7 @@ code_object = self.get_code(name) module.__file__ = self.get_filename(name) if not sourceless: - module.__cached__ = _cache_from_source(module.__file__) + module.__cached__ = cache_from_source(module.__file__) else: module.__cached__ = module.__file__ module.__package__ = name @@ -573,7 +573,7 @@ """ source_path = self.get_filename(fullname) - bytecode_path = _cache_from_source(source_path) + bytecode_path = cache_from_source(source_path) source_mtime = None if bytecode_path is not None: try: @@ -594,12 +594,12 @@ except (ImportError, EOFError): pass else: - verbose_message('{} matches {}', bytecode_path, + _verbose_message('{} matches {}', bytecode_path, source_path) found = marshal.loads(bytes_data) - if isinstance(found, code_type): + if isinstance(found, _code_type): _imp._fix_co_filename(found, source_path) - verbose_message('code object from {}', + _verbose_message('code object from {}', bytecode_path) return found else: @@ -609,7 +609,7 @@ source_bytes = self.get_data(source_path) code_object = compile(source_bytes, source_path, 'exec', dont_inherit=True) - verbose_message('code object from {}', source_path) + _verbose_message('code object from {}', source_path) if (not sys.dont_write_bytecode and bytecode_path is not None and source_mtime is not None): data = bytearray(_MAGIC_NUMBER) @@ -618,7 +618,7 @@ data.extend(marshal.dumps(code_object)) try: self.set_data(bytecode_path, data) - verbose_message('wrote {!r}', bytecode_path) + _verbose_message('wrote {!r}', bytecode_path) except NotImplementedError: pass return code_object @@ -687,7 +687,7 @@ return try: _write_atomic(path, data) - verbose_message('created {!r}', path) + _verbose_message('created {!r}', path) except (PermissionError, FileExistsError): # Don't worry if you can't write bytecode or someone is writing # it at the same time. @@ -706,8 +706,8 @@ data = self.get_data(path) bytes_data = self._bytes_from_bytecode(fullname, data, path, None) found = marshal.loads(bytes_data) - if isinstance(found, code_type): - verbose_message('code object from {!r}', path) + if isinstance(found, _code_type): + _verbose_message('code object from {!r}', path) return found else: raise ImportError("Non-code object in {}".format(path), @@ -738,7 +738,7 @@ is_reload = fullname in sys.modules try: module = _imp.load_dynamic(fullname, self.path) - verbose_message('extension module loaded from {!r}', self.path) + _verbose_message('extension module loaded from {!r}', self.path) return module except: if not is_reload and fullname in sys.modules: @@ -908,7 +908,7 @@ new_name = name lower_suffix_contents.add(new_name) self._path_cache = lower_suffix_contents - if sys.platform.startswith(CASE_INSENSITIVE_PLATFORMS): + if sys.platform.startswith(_CASE_INSENSITIVE_PLATFORMS): self._relaxed_path_cache = set(fn.lower() for fn in contents) @classmethod @@ -1013,7 +1013,7 @@ elif name not in sys.modules: # The parent import may have already imported this module. loader.load_module(name) - verbose_message('import {!r} # {!r}', name, loader) + _verbose_message('import {!r} # {!r}', name, loader) # Backwards-compatibility; be nicer to skip the dict lookup. module = sys.modules[name] if parent: @@ -1185,7 +1185,7 @@ setattr(self_module, '_MAGIC_NUMBER', _imp_module.get_magic()) setattr(self_module, '_TAG', _imp.get_tag()) if builtin_os == 'nt': - SOURCE_SUFFIXES.append('.pyw') + _SOURCE_SUFFIXES.append('.pyw') def _install(sys_module, _imp_module): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 21:20:48 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 21:20:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313959=3A_Re-implem?= =?utf8?q?ent_imp=2Eget=5Fsuffixes=28=29_in_Lib/imp=2Epy=2E?= Message-ID: http://hg.python.org/cpython/rev/257cbd2fac38 changeset: 76754:257cbd2fac38 user: Brett Cannon date: Fri May 04 15:20:40 2012 -0400 summary: Issue #13959: Re-implement imp.get_suffixes() in Lib/imp.py. This introduces a new function, imp.extension_suffixes(), which is currently undocumented. That is forthcoming once issue #14657 is resolved and how to expose file suffixes is decided. files: Lib/imp.py | 16 +- Lib/importlib/_bootstrap.py | 45 +----- Lib/importlib/test/extension/test_case_sensitivity.py | 2 +- Lib/importlib/test/extension/test_finder.py | 2 +- Lib/importlib/test/extension/test_path_hook.py | 2 +- Lib/importlib/test/source/test_case_sensitivity.py | 4 +- Lib/importlib/test/source/test_finder.py | 8 +- Lib/importlib/test/source/test_path_hook.py | 2 +- Python/dynload_aix.c | 5 +- Python/dynload_dl.c | 5 +- Python/dynload_hpux.c | 5 +- Python/dynload_next.c | 5 +- Python/dynload_os2.c | 6 +- Python/dynload_shlib.c | 20 +- Python/dynload_stub.c | 4 +- Python/dynload_win.c | 8 +- Python/import.c | 72 +-------- Python/importdl.h | 9 +- Python/importlib.h | Bin 19 files changed, 60 insertions(+), 160 deletions(-) diff --git a/Lib/imp.py b/Lib/imp.py --- a/Lib/imp.py +++ b/Lib/imp.py @@ -9,11 +9,9 @@ from _imp import (lock_held, acquire_lock, release_lock, load_dynamic, get_frozen_object, is_frozen_package, init_builtin, init_frozen, is_builtin, is_frozen, - _fix_co_filename) + _fix_co_filename, extension_suffixes) # Could move out of _imp, but not worth the code from _imp import get_magic, get_tag -# Can (probably) move to importlib -from _imp import get_suffixes from importlib._bootstrap import new_module from importlib._bootstrap import cache_from_source @@ -38,6 +36,14 @@ IMP_HOOK = 9 +def get_suffixes(): + extensions = [(s, 'rb', C_EXTENSION) for s in extension_suffixes()] + source = [(s, 'U', PY_SOURCE) for s in _bootstrap._SOURCE_SUFFIXES] + bytecode = [(_bootstrap._BYTECODE_SUFFIX, 'rb', PY_COMPILED)] + + return extensions + source + bytecode + + def source_from_cache(path): """Given the path to a .pyc./.pyo file, return the path to its .py file. @@ -120,8 +126,8 @@ # XXX deprecate def load_package(name, path): if os.path.isdir(path): - extensions = _bootstrap._suffix_list(PY_SOURCE) - extensions += _bootstrap._suffix_list(PY_COMPILED) + extensions = _bootstrap._SOURCE_SUFFIXES + extensions += [_bootstrap._BYTECODE_SUFFIX] for extension in extensions: path = os.path.join(path, '__init__'+extension) if os.path.exists(path): diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -95,16 +95,6 @@ return front, tail -def _path_exists(path): - """Replacement for os.path.exists.""" - try: - _os.stat(path) - except OSError: - return False - else: - return True - - def _path_is_mode_type(path, mode): """Test whether the path is the specified mode type.""" try: @@ -128,28 +118,6 @@ return _path_is_mode_type(path, 0o040000) -def _path_without_ext(path, ext_type): - """Replacement for os.path.splitext()[0].""" - for suffix in _suffix_list(ext_type): - if path.endswith(suffix): - return path[:-len(suffix)] - else: - raise ValueError("path is not of the specified type") - - -def _path_absolute(path): - """Replacement for os.path.abspath.""" - if not path: - path = _os.getcwd() - try: - return _os._getfullpathname(path) - except AttributeError: - if path.startswith('/'): - return path - else: - return _path_join(_os.getcwd(), path) - - def _write_atomic(path, data): """Best-effort function to write data to a path atomically. Be prepared to handle a FileExistsError if concurrent writing of the @@ -338,12 +306,6 @@ return _requires_frozen_wrapper -def _suffix_list(suffix_type): - """Return a list of file suffixes based on the imp file type.""" - return [suffix[0] for suffix in _imp.get_suffixes() - if suffix[2] == suffix_type] - - # Loaders ##################################################################### class BuiltinImporter: @@ -1196,8 +1158,9 @@ """ _setup(sys_module, _imp_module) - supported_loaders = [(ExtensionFileLoader, _suffix_list(3), False), - (SourceFileLoader, _suffix_list(1), True), - (SourcelessFileLoader, _suffix_list(2), True)] + extensions = ExtensionFileLoader, _imp_module.extension_suffixes(), False + source = SourceFileLoader, _SOURCE_SUFFIXES, True + bytecode = SourcelessFileLoader, [_BYTECODE_SUFFIX], True + supported_loaders = [extensions, source, bytecode] sys.path_hooks.extend([FileFinder.path_hook(*supported_loaders)]) sys.meta_path.extend([BuiltinImporter, FrozenImporter, PathFinder]) diff --git a/Lib/importlib/test/extension/test_case_sensitivity.py b/Lib/importlib/test/extension/test_case_sensitivity.py --- a/Lib/importlib/test/extension/test_case_sensitivity.py +++ b/Lib/importlib/test/extension/test_case_sensitivity.py @@ -16,7 +16,7 @@ assert good_name != bad_name finder = _bootstrap.FileFinder(ext_util.PATH, (_bootstrap.ExtensionFileLoader, - _bootstrap._suffix_list(imp.C_EXTENSION), + imp.extension_suffixes(), False)) return finder.find_module(bad_name) diff --git a/Lib/importlib/test/extension/test_finder.py b/Lib/importlib/test/extension/test_finder.py --- a/Lib/importlib/test/extension/test_finder.py +++ b/Lib/importlib/test/extension/test_finder.py @@ -12,7 +12,7 @@ def find_module(self, fullname): importer = _bootstrap.FileFinder(util.PATH, (_bootstrap.ExtensionFileLoader, - _bootstrap._suffix_list(imp.C_EXTENSION), + imp.extension_suffixes(), False)) return importer.find_module(fullname) diff --git a/Lib/importlib/test/extension/test_path_hook.py b/Lib/importlib/test/extension/test_path_hook.py --- a/Lib/importlib/test/extension/test_path_hook.py +++ b/Lib/importlib/test/extension/test_path_hook.py @@ -15,7 +15,7 @@ def hook(self, entry): return _bootstrap.FileFinder.path_hook((_bootstrap.ExtensionFileLoader, - _bootstrap._suffix_list(imp.C_EXTENSION), False))(entry) + imp.extension_suffixes(), False))(entry) def test_success(self): # Path hook should handle a directory where a known extension module diff --git a/Lib/importlib/test/source/test_case_sensitivity.py b/Lib/importlib/test/source/test_case_sensitivity.py --- a/Lib/importlib/test/source/test_case_sensitivity.py +++ b/Lib/importlib/test/source/test_case_sensitivity.py @@ -22,10 +22,10 @@ def find(self, path): finder = _bootstrap.FileFinder(path, (_bootstrap.SourceFileLoader, - _bootstrap._suffix_list(imp.PY_SOURCE), + _bootstrap._SOURCE_SUFFIXES, True), (_bootstrap.SourcelessFileLoader, - _bootstrap._suffix_list(imp.PY_COMPILED), + [_bootstrap._BYTECODE_SUFFIX], True)) return finder.find_module(self.name) diff --git a/Lib/importlib/test/source/test_finder.py b/Lib/importlib/test/source/test_finder.py --- a/Lib/importlib/test/source/test_finder.py +++ b/Lib/importlib/test/source/test_finder.py @@ -37,9 +37,9 @@ def import_(self, root, module): loader_details = [(_bootstrap.SourceFileLoader, - _bootstrap._suffix_list(imp.PY_SOURCE), True), + _bootstrap._SOURCE_SUFFIXES, True), (_bootstrap.SourcelessFileLoader, - _bootstrap._suffix_list(imp.PY_COMPILED), True)] + [_bootstrap._BYTECODE_SUFFIX], True)] finder = _bootstrap.FileFinder(root, *loader_details) return finder.find_module(module) @@ -139,7 +139,7 @@ def test_empty_string_for_dir(self): # The empty string from sys.path means to search in the cwd. finder = _bootstrap.FileFinder('', (_bootstrap.SourceFileLoader, - _bootstrap._suffix_list(imp.PY_SOURCE), True)) + _bootstrap._SOURCE_SUFFIXES, True)) with open('mod.py', 'w') as file: file.write("# test file for importlib") try: @@ -151,7 +151,7 @@ def test_invalidate_caches(self): # invalidate_caches() should reset the mtime. finder = _bootstrap.FileFinder('', (_bootstrap.SourceFileLoader, - _bootstrap._suffix_list(imp.PY_SOURCE), True)) + _bootstrap._SOURCE_SUFFIXES, True)) finder._path_mtime = 42 finder.invalidate_caches() self.assertEqual(finder._path_mtime, -1) diff --git a/Lib/importlib/test/source/test_path_hook.py b/Lib/importlib/test/source/test_path_hook.py --- a/Lib/importlib/test/source/test_path_hook.py +++ b/Lib/importlib/test/source/test_path_hook.py @@ -11,7 +11,7 @@ def path_hook(self): return _bootstrap.FileFinder.path_hook((_bootstrap.SourceFileLoader, - _bootstrap._suffix_list(imp.PY_SOURCE), True)) + _bootstrap._SOURCE_SUFFIXES, True)) def test_success(self): with source_util.create_modules('dummy') as mapping: diff --git a/Python/dynload_aix.c b/Python/dynload_aix.c --- a/Python/dynload_aix.c +++ b/Python/dynload_aix.c @@ -26,10 +26,7 @@ void *entry; } Module, *ModulePtr; -const struct filedescr _PyImport_DynLoadFiletab[] = { - {".so", "rb", C_EXTENSION}, - {0, 0} -}; +const char *_PyImport_DynLoadFiletab[] = {".so", NULL}; static int aix_getoldmodules(void **modlistptr) diff --git a/Python/dynload_dl.c b/Python/dynload_dl.c --- a/Python/dynload_dl.c +++ b/Python/dynload_dl.c @@ -9,10 +9,7 @@ extern char *Py_GetProgramName(void); -const struct filedescr _PyImport_DynLoadFiletab[] = { - {".o", "rb", C_EXTENSION}, - {0, 0} -}; +const char *_PyImport_DynLoadFiletab[] = {".o", NULL}; dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, diff --git a/Python/dynload_hpux.c b/Python/dynload_hpux.c --- a/Python/dynload_hpux.c +++ b/Python/dynload_hpux.c @@ -13,10 +13,7 @@ #define FUNCNAME_PATTERN "PyInit_%.200s" #endif -const struct filedescr _PyImport_DynLoadFiletab[] = { - {SHLIB_EXT, "rb", C_EXTENSION}, - {0, 0} -}; +const char *_PyImport_DynLoadFiletab[] = {SHLIB_EXT, NULL}; dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) diff --git a/Python/dynload_next.c b/Python/dynload_next.c --- a/Python/dynload_next.c +++ b/Python/dynload_next.c @@ -8,10 +8,7 @@ #include -const struct filedescr _PyImport_DynLoadFiletab[] = { - {".so", "rb", C_EXTENSION}, - {0, 0} -}; +const char *_PyImport_DynLoadFiletab[] = {".so", NULL}; /* ** Python modules are Mach-O MH_BUNDLE files. The best way to load these diff --git a/Python/dynload_os2.c b/Python/dynload_os2.c --- a/Python/dynload_os2.c +++ b/Python/dynload_os2.c @@ -9,11 +9,7 @@ #include "importdl.h" -const struct filedescr _PyImport_DynLoadFiletab[] = { - {".pyd", "rb", C_EXTENSION}, - {".dll", "rb", C_EXTENSION}, - {0, 0} -}; +const char *_PyImport_DynLoadFiletab[] = {".pyd", ".dll", NULL}; dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) diff --git a/Python/dynload_shlib.c b/Python/dynload_shlib.c --- a/Python/dynload_shlib.c +++ b/Python/dynload_shlib.c @@ -36,25 +36,25 @@ live in the same directory. E.g. foomodule.cpython-32.so */ -const struct filedescr _PyImport_DynLoadFiletab[] = { +const char *_PyImport_DynLoadFiletab[] = { #ifdef __CYGWIN__ - {".dll", "rb", C_EXTENSION}, + ".dll", #else /* !__CYGWIN__ */ #if defined(PYOS_OS2) && defined(PYCC_GCC) - {".pyd", "rb", C_EXTENSION}, - {".dll", "rb", C_EXTENSION}, + ".pyd", + ".dll", #else /* !(defined(PYOS_OS2) && defined(PYCC_GCC)) */ #ifdef __VMS - {".exe", "rb", C_EXTENSION}, - {".EXE", "rb", C_EXTENSION}, + ".exe", + ".EXE", #else /* !__VMS */ - {"." SOABI ".so", "rb", C_EXTENSION}, - {".abi" PYTHON_ABI_STRING ".so", "rb", C_EXTENSION}, - {".so", "rb", C_EXTENSION}, + "." SOABI ".so", + ".abi" PYTHON_ABI_STRING ".so", + ".so", #endif /* __VMS */ #endif /* defined(PYOS_OS2) && defined(PYCC_GCC) */ #endif /* __CYGWIN__ */ - {0, 0} + NULL, }; static struct { diff --git a/Python/dynload_stub.c b/Python/dynload_stub.c --- a/Python/dynload_stub.c +++ b/Python/dynload_stub.c @@ -6,6 +6,4 @@ #include "importdl.h" -const struct filedescr _PyImport_DynLoadFiletab[] = { - {0, 0} -}; +const char *_PyImport_DynLoadFiletab[] = {NULL}; diff --git a/Python/dynload_win.c b/Python/dynload_win.c --- a/Python/dynload_win.c +++ b/Python/dynload_win.c @@ -15,13 +15,13 @@ extern ULONG_PTR _Py_ActivateActCtx(); void _Py_DeactivateActCtx(ULONG_PTR cookie); -const struct filedescr _PyImport_DynLoadFiletab[] = { +const char *_PyImport_DynLoadFiletab[] = { #ifdef _DEBUG - {"_d.pyd", "rb", C_EXTENSION}, + "_d.pyd", #else - {".pyd", "rb", C_EXTENSION}, + ".pyd", #endif - {0, 0} + NULL }; diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -140,18 +140,6 @@ struct _inittab *PyImport_Inittab = _PyImport_Inittab; -/* these tables define the module suffixes that Python recognizes */ -struct filedescr * _PyImport_Filetab = NULL; - -static const struct filedescr _PyImport_StandardFiletab[] = { - {".py", "U", PY_SOURCE}, -#ifdef MS_WINDOWS - {".pyw", "U", PY_SOURCE}, -#endif - {".pyc", "rb", PY_COMPILED}, - {0, 0} -}; - static PyObject *initstr = NULL; /* Initialize things */ @@ -159,44 +147,9 @@ void _PyImport_Init(void) { - const struct filedescr *scan; - struct filedescr *filetab; - int countD = 0; - int countS = 0; - initstr = PyUnicode_InternFromString("__init__"); if (initstr == NULL) Py_FatalError("Can't initialize import variables"); - - /* prepare _PyImport_Filetab: copy entries from - _PyImport_DynLoadFiletab and _PyImport_StandardFiletab. - */ -#ifdef HAVE_DYNAMIC_LOADING - for (scan = _PyImport_DynLoadFiletab; scan->suffix != NULL; ++scan) - ++countD; -#endif - for (scan = _PyImport_StandardFiletab; scan->suffix != NULL; ++scan) - ++countS; - filetab = PyMem_NEW(struct filedescr, countD + countS + 1); - if (filetab == NULL) - Py_FatalError("Can't initialize import file table."); -#ifdef HAVE_DYNAMIC_LOADING - memcpy(filetab, _PyImport_DynLoadFiletab, - countD * sizeof(struct filedescr)); -#endif - memcpy(filetab + countD, _PyImport_StandardFiletab, - countS * sizeof(struct filedescr)); - filetab[countD + countS].suffix = NULL; - - _PyImport_Filetab = filetab; - - if (Py_OptimizeFlag) { - /* Replace ".pyc" with ".pyo" in _PyImport_Filetab */ - for (; filetab->suffix != NULL; filetab++) { - if (strcmp(filetab->suffix, ".pyc") == 0) - filetab->suffix = ".pyo"; - } - } } void @@ -400,8 +353,6 @@ { Py_XDECREF(extensions); extensions = NULL; - PyMem_DEL(_PyImport_Filetab); - _PyImport_Filetab = NULL; #ifdef WITH_THREAD if (import_lock != NULL) { PyThread_free_lock(import_lock); @@ -1911,17 +1862,18 @@ } static PyObject * -imp_get_suffixes(PyObject *self, PyObject *noargs) +imp_extension_suffixes(PyObject *self, PyObject *noargs) { PyObject *list; - struct filedescr *fdp; + const char *suffix; + unsigned int index = 0; list = PyList_New(0); if (list == NULL) return NULL; - for (fdp = _PyImport_Filetab; fdp->suffix != NULL; fdp++) { - PyObject *item = Py_BuildValue("ssi", - fdp->suffix, fdp->mode, fdp->type); +#ifdef HAVE_DYNAMIC_LOADING + while ((suffix = _PyImport_DynLoadFiletab[index])) { + PyObject *item = PyUnicode_FromString(suffix); if (item == NULL) { Py_DECREF(list); return NULL; @@ -1932,7 +1884,9 @@ return NULL; } Py_DECREF(item); + index += 1; } +#endif return list; } @@ -2101,10 +2055,9 @@ "get_tag() -> string\n\ Return the magic tag for .pyc or .pyo files."); -PyDoc_STRVAR(doc_get_suffixes, -"get_suffixes() -> [(suffix, mode, type), ...]\n\ -Return a list of (suffix, mode, type) tuples describing the files\n\ -that find_module() looks for."); +PyDoc_STRVAR(doc_extension_suffixes, +"extension_suffixes() -> list of strings\n\ +Returns the list of file suffixes used to identify extension modules."); PyDoc_STRVAR(doc_lock_held, "lock_held() -> boolean\n\ @@ -2126,7 +2079,8 @@ static PyMethodDef imp_methods[] = { {"get_magic", imp_get_magic, METH_NOARGS, doc_get_magic}, {"get_tag", imp_get_tag, METH_NOARGS, doc_get_tag}, - {"get_suffixes", imp_get_suffixes, METH_NOARGS, doc_get_suffixes}, + {"extension_suffixes", imp_extension_suffixes, METH_NOARGS, + doc_extension_suffixes}, {"lock_held", imp_lock_held, METH_NOARGS, doc_lock_held}, {"acquire_lock", imp_acquire_lock, METH_NOARGS, doc_acquire_lock}, {"release_lock", imp_release_lock, METH_NOARGS, doc_release_lock}, diff --git a/Python/importdl.h b/Python/importdl.h --- a/Python/importdl.h +++ b/Python/importdl.h @@ -20,13 +20,8 @@ IMP_HOOK }; -struct filedescr { - char *suffix; - char *mode; - enum filetype type; -}; -extern struct filedescr * _PyImport_Filetab; -extern const struct filedescr _PyImport_DynLoadFiletab[]; + +extern const char *_PyImport_DynLoadFiletab[]; extern PyObject *_PyImport_LoadDynamicModule(PyObject *name, PyObject *pathname, FILE *); diff --git a/Python/importlib.h b/Python/importlib.h index e9f5faf2fa9b7e239563aff1db46d6e1d70fcb88..16816053fb32d26813d91a9a69202190e73d4828 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 21:40:15 2012 From: python-checkins at python.org (barry.warsaw) Date: Fri, 04 May 2012 21:40:15 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A__*_Relax_the_constraint_that_m?= =?utf8?q?=2E=5F=5Ffile=5F=5F_must_exist_=28except_for_built-ins=29?= Message-ID: http://hg.python.org/peps/rev/f0e1c485f8d8 changeset: 4352:f0e1c485f8d8 user: Barry Warsaw date: Fri May 04 15:40:10 2012 -0400 summary: * Relax the constraint that m.__file__ must exist (except for built-ins) * Describe the loader.module_repr() protocol. * Fix some misspellings. files: pep-0420.txt | 61 +++++++++++++++++++++++++++++++++------ 1 files changed, 51 insertions(+), 10 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -13,13 +13,13 @@ Abstract ======== -Namespace packages are a mechanism for splitting a single Python -package across multiple directories on disk. In current Python -versions, an algorithm to compute the packages ``__path__`` must be -formulated. With the enhancement proposed here, the import machinery -itself will construct the list of directories that make up the -package. This PEP builds upon the work started in rejected PEPs 382 -and 402. An implementation of this PEP is at [1]_. +Namespace packages are a mechanism for splitting a single Python package +across multiple directories on disk. In current Python versions, an algorithm +to compute the packages ``__path__`` must be formulated. With the enhancement +proposed here, the import machinery itself will construct the list of +directories that make up the package. This PEP builds upon previous work, +documented in PEP 382 and PEP 402. Those PEPs have since been rejected in +favor of this one. An implementation of this PEP is at [1]_. Terminology =========== @@ -111,7 +111,7 @@ consequence, ``pkgutil.extend_path`` and ``pkg_resources.declare_namespace`` become obsolete for purposes of namespace package creation. There will be no marker file or directory -for specifing a namespace package. +for specifying a namespace package. During import processing, the import machinery will continue to iterate over each directory in the parent path as it does in Python @@ -171,7 +171,9 @@ namespace module's ``__path__``, as described above. This string must not contain a trailing path separator. -There is no impact on PEP 302 "loaders". +The specification expands PEP 302 loaders to include an optional method called +``module_repr()`` which if present, is used to generate module object reprs. +See the section below for further details. If an existing finder is not updated to support returning a string from ``find_module``, the only impact is that such a finder will be @@ -200,7 +202,7 @@ "foo" directories would be in directories that are on ``sys.path``. "foo/bar" would be in one of these sys.path entries, and "foo/baz" would be in the other. Upon removal of "foo.bar", the "foo/bar" and -corresonding "foo" directories can be completely removed. But +corresponding "foo" directories can be completely removed. But "foo/baz" and its corresponding "foo" directory cannot be removed. It is also possible to have the "foo.bar" portion installed in a @@ -258,6 +260,45 @@ date to add such features. Several possible ways to do so were discussed in the referenced email thread. +Module reprs +============ + +Previously, module reprs were hard coded based on assumptions about a module's +``__file__`` attribute. If this attribute existed and was a string, it was +assumed to be a file system path, and the module object's repr would include +this in its value. The only exception was that PEP 302 reserved missing +``__file__`` attributes to built-in modules, and in CPython, this assumption +was baked into the module object's implementation. Because of this +restriction, some module contained contrived ``__file__`` values that did not +reflect file system paths, and which could cause unexpected problems later +(e.g. ``os.path.join()`` on a non-path ``__file__`` would return gibberish). + +This PEP relaxes this constraint, and leaves the setting of ``__file__`` to +the purview of the loader producing the module. Loaders may opt to leave +``__file__`` unset if no file system path is appropriate. This means that the +definitive way to determine the origin of a module is to check its +``__loader__`` attribute. + +For example, namespace packages as described in this PEP will have no +``__file__`` attribute because no corresponding file exists. In order to +provide flexibility and descriptiveness in the reprs of such modules, a new +optional protocol is added to PEP 302 loaders. Loaders can implement a +``module_repr()`` method which takes a single argument, the module object. +This method should return the string to be used verbatim as the repr of the +module. The rules for producing a module repr are now standardized as: + + * If the module has an ``__loader__`` and that loader has a ``module_repr()`` + method, call it with a single argument, which is the module object. The + value returned is used as the module's repr. + * Exceptions from ``module_repr()`` are ignored, and the following steps + are used instead. + * If the module has an ``__file__`` attribute, this is used as part of the + module's repr. + * If the module has no ``__file__`` but does have an ``__loader__``, then the + loader's repr is used as part of the module's repr. + * Otherwise, just use the module's ``__name__`` in the repr. + + References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 4 21:46:09 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 21:46:09 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Jython-friendly_tweak=2E?= Message-ID: http://hg.python.org/cpython/rev/c6a97506b2ee changeset: 76755:c6a97506b2ee user: Brett Cannon date: Fri May 04 15:46:04 2012 -0400 summary: Jython-friendly tweak. files: Lib/importlib/_bootstrap.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -156,7 +156,7 @@ The module is not entered into sys.modules. """ - return type(sys)(name) + return type(_io)(name) # Finder/loader utility code ################################################## -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 21:51:14 2012 From: python-checkins at python.org (barry.warsaw) Date: Fri, 04 May 2012 21:51:14 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Make_it_explicit_that_loaders_?= =?utf8?q?may_install_additional_=5F=5Fthings=5F=5F_on?= Message-ID: http://hg.python.org/peps/rev/7bc9b5f75033 changeset: 4353:7bc9b5f75033 user: Barry Warsaw date: Fri May 04 15:51:09 2012 -0400 summary: Make it explicit that loaders may install additional __things__ on modules. files: pep-0420.txt | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -275,8 +275,9 @@ This PEP relaxes this constraint, and leaves the setting of ``__file__`` to the purview of the loader producing the module. Loaders may opt to leave -``__file__`` unset if no file system path is appropriate. This means that the -definitive way to determine the origin of a module is to check its +``__file__`` unset if no file system path is appropriate. Loaders may also +set additional reserved attributes on the module if useful. This means that +the definitive way to determine the origin of a module is to check its ``__loader__`` attribute. For example, namespace packages as described in this PEP will have no -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 4 21:52:06 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 04 May 2012 21:52:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Closes_=2314093=3A_Added_Me?= =?utf8?q?rcurial_version_information_to_Windows_builds=2E?= Message-ID: http://hg.python.org/cpython/rev/4459d82ff127 changeset: 76756:4459d82ff127 user: Vinay Sajip date: Fri May 04 20:51:59 2012 +0100 summary: Closes #14093: Added Mercurial version information to Windows builds. files: PCbuild/make_buildinfo.c | 67 +++++++++++++++++++++++++++- 1 files changed, 66 insertions(+), 1 deletions(-) diff --git a/PCbuild/make_buildinfo.c b/PCbuild/make_buildinfo.c --- a/PCbuild/make_buildinfo.c +++ b/PCbuild/make_buildinfo.c @@ -2,6 +2,7 @@ #include #include #include +#include #define CMD_SIZE 500 @@ -61,6 +62,51 @@ return 1; } +const char DELIMS[] = { " \n" }; + +int get_mercurial_info(char * hgbranch, char * hgtag, char * hgrev, int size) +{ + int result = 0; + char filename[CMD_SIZE]; + char cmdline[CMD_SIZE]; + + strcpy_s(filename, CMD_SIZE, "tmpXXXXXX"); + if (_mktemp_s(filename, CMD_SIZE) == 0) { + int rc; + + strcpy_s(cmdline, CMD_SIZE, "hg id -bit > "); + strcat_s(cmdline, CMD_SIZE, filename); + rc = system(cmdline); + if (rc == 0) { + FILE * fp; + + if (fopen_s(&fp, filename, "r") == 0) { + char * cp = fgets(cmdline, CMD_SIZE, fp); + + if (cp) { + char * context = NULL; + char * tp = strtok_s(cp, DELIMS, &context); + if (tp) { + strcpy_s(hgrev, size, tp); + tp = strtok_s(NULL, DELIMS, &context); + if (tp) { + strcpy_s(hgbranch, size, tp); + tp = strtok_s(NULL, DELIMS, &context); + if (tp) { + strcpy_s(hgtag, size, tp); + result = 1; + } + } + } + } + fclose(fp); + } + } + _unlink(filename); + } + return result; +} + int main(int argc, char*argv[]) { char command[CMD_SIZE] = "cl.exe -c -D_WIN32 -DUSE_DL_EXPORT -D_WINDOWS -DWIN32 -D_WINDLL "; @@ -109,8 +155,27 @@ strcat_s(command, CMD_SIZE, "\""); strcat_s(command, CMD_SIZE, tmppath); strcat_s(command, CMD_SIZE, "getbuildinfo2.c\" -DSUBWCREV "); - } else + } + else { + char hgtag[CMD_SIZE]; + char hgbranch[CMD_SIZE]; + char hgrev[CMD_SIZE]; + + if (get_mercurial_info(hgbranch, hgtag, hgrev, CMD_SIZE)) { + strcat_s(command, CMD_SIZE, "-DHGBRANCH=\\\""); + strcat_s(command, CMD_SIZE, hgbranch); + strcat_s(command, CMD_SIZE, "\\\""); + + strcat_s(command, CMD_SIZE, " -DHGTAG=\\\""); + strcat_s(command, CMD_SIZE, hgtag); + strcat_s(command, CMD_SIZE, "\\\""); + + strcat_s(command, CMD_SIZE, " -DHGVERSION=\\\""); + strcat_s(command, CMD_SIZE, hgrev); + strcat_s(command, CMD_SIZE, "\\\" "); + } strcat_s(command, CMD_SIZE, "..\\Modules\\getbuildinfo.c"); + } strcat_s(command, CMD_SIZE, " -Fo\""); strcat_s(command, CMD_SIZE, tmppath); strcat_s(command, CMD_SIZE, "getbuildinfo.o\" -I..\\Include -I..\\PC"); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:06:04 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 22:06:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_update_importlib=2Eh?= Message-ID: http://hg.python.org/cpython/rev/0ee88c9bb46f changeset: 76757:0ee88c9bb46f parent: 76755:c6a97506b2ee user: Brett Cannon date: Fri May 04 16:03:20 2012 -0400 summary: update importlib.h files: Python/importlib.h | Bin 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/Python/importlib.h b/Python/importlib.h index 16816053fb32d26813d91a9a69202190e73d4828..9c531f013f45b202d3523994af2799d4270339a7 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:06:05 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 22:06:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_dead_Windows_code_wh?= =?utf8?q?ich_no_longer_will_compile=2E?= Message-ID: http://hg.python.org/cpython/rev/7fbb4443ffb7 changeset: 76758:7fbb4443ffb7 user: Brett Cannon date: Fri May 04 16:04:14 2012 -0400 summary: Remove dead Windows code which no longer will compile. files: PC/VS7.1/pythoncore.vcproj | 27 ----- PC/VS8.0/pythoncore.vcproj | 4 - PC/import_nt.c | 117 ------------------------- Python/import.c | 6 - 4 files changed, 0 insertions(+), 154 deletions(-) diff --git a/PC/VS7.1/pythoncore.vcproj b/PC/VS7.1/pythoncore.vcproj --- a/PC/VS7.1/pythoncore.vcproj +++ b/PC/VS7.1/pythoncore.vcproj @@ -616,33 +616,6 @@ RelativePath="..\..\Python\import.c"> - - - - - - - - - - - - - - - - diff --git a/PC/import_nt.c b/PC/import_nt.c deleted file mode 100644 --- a/PC/import_nt.c +++ /dev/null @@ -1,117 +0,0 @@ -/******************************************************************** - - import_nt.c - - Win32 specific import code. - -*/ - -#include "Python.h" -#include "osdefs.h" -#include -#include "importdl.h" -#include "malloc.h" /* for alloca */ - -/* a string loaded from the DLL at startup */ -extern const char *PyWin_DLLVersionString; - -/* Find a module on Windows. - - Read the registry Software\Python\PythonCore\\Modules\ (or - Software\Python\PythonCore\\Modules\\Debug in debug mode) - from HKEY_CURRENT_USER, or HKEY_LOCAL_MACHINE. Find the file descriptor using - the file extension. Open the file. - - On success, write the file descriptor into *ppFileDesc, the module path - (Unicode object) into *pPath, and return the opened file object. If the - module cannot be found (e.g. no registry key or the file doesn't exist), - return NULL. On error, raise a Python exception and return NULL. - */ -FILE * -_PyWin_FindRegisteredModule(PyObject *moduleName, - struct filedescr **ppFileDesc, - PyObject **pPath) -{ - wchar_t pathBuf[MAXPATHLEN+1]; - int pathLen = MAXPATHLEN+1; - PyObject *path, *moduleKey, *suffix; - wchar_t *wmoduleKey, *wsuffix; - struct filedescr *fdp; - HKEY keyBase; - int modNameSize; - long regStat; - Py_ssize_t extLen; - FILE *fp; - - moduleKey = PyUnicode_FromFormat( -#ifdef _DEBUG - /* In debugging builds, we _must_ have the debug version registered */ - "Software\\Python\\PythonCore\\%s\\Modules\\%U\\Debug", -#else - "Software\\Python\\PythonCore\\%s\\Modules\\%U", -#endif - PyWin_DLLVersionString, moduleName); - if (moduleKey == NULL) - return NULL; - wmoduleKey = PyUnicode_AsUnicode(moduleKey); - if (wmoduleKey == NULL) { - Py_DECREF(moduleKey); - return NULL; - } - - keyBase = HKEY_CURRENT_USER; - modNameSize = pathLen; - regStat = RegQueryValueW(keyBase, wmoduleKey, - pathBuf, &modNameSize); - if (regStat != ERROR_SUCCESS) { - /* No user setting - lookup in machine settings */ - keyBase = HKEY_LOCAL_MACHINE; - /* be anal - failure may have reset size param */ - modNameSize = pathLen; - regStat = RegQueryValueW(keyBase, wmoduleKey, - pathBuf, &modNameSize); - if (regStat != ERROR_SUCCESS) { - Py_DECREF(moduleKey); - return NULL; - } - } - Py_DECREF(moduleKey); - if (modNameSize < 3) { - /* path shorter than "a.o" or negative length (cast to - size_t is wrong) */ - return NULL; - } - /* use the file extension to locate the type entry. */ - for (fdp = _PyImport_Filetab; fdp->suffix != NULL; fdp++) { - suffix = PyUnicode_FromString(fdp->suffix); - if (suffix == NULL) - return NULL; - wsuffix = PyUnicode_AsUnicodeAndSize(suffix, &extLen); - if (wsuffix == NULL) { - Py_DECREF(suffix); - return NULL; - } - if ((Py_ssize_t)modNameSize > extLen && - _wcsnicmp(pathBuf + ((Py_ssize_t)modNameSize-extLen-1), - wsuffix, - extLen) == 0) - { - Py_DECREF(suffix); - break; - } - Py_DECREF(suffix); - } - if (fdp->suffix == NULL) - return NULL; - path = PyUnicode_FromWideChar(pathBuf, wcslen(pathBuf)); - if (path == NULL) - return NULL; - fp = _Py_fopen(path, fdp->mode); - if (fp == NULL) { - Py_DECREF(path); - return NULL; - } - *pPath = path; - *ppFileDesc = fdp; - return fp; -} diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1150,12 +1150,6 @@ } -#ifdef MS_COREDLL -extern FILE *_PyWin_FindRegisteredModule(PyObject *, struct filedescr **, - PyObject **p_path); -#endif - - static int init_builtin(PyObject *); /* Forward */ /* Initialize a built-in module. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:06:05 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 22:06:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge?= Message-ID: http://hg.python.org/cpython/rev/20f8b105525e changeset: 76759:20f8b105525e parent: 76758:7fbb4443ffb7 parent: 76756:4459d82ff127 user: Brett Cannon date: Fri May 04 16:04:59 2012 -0400 summary: merge files: PCbuild/make_buildinfo.c | 67 +++++++++++++++++++++++++++- 1 files changed, 66 insertions(+), 1 deletions(-) diff --git a/PCbuild/make_buildinfo.c b/PCbuild/make_buildinfo.c --- a/PCbuild/make_buildinfo.c +++ b/PCbuild/make_buildinfo.c @@ -2,6 +2,7 @@ #include #include #include +#include #define CMD_SIZE 500 @@ -61,6 +62,51 @@ return 1; } +const char DELIMS[] = { " \n" }; + +int get_mercurial_info(char * hgbranch, char * hgtag, char * hgrev, int size) +{ + int result = 0; + char filename[CMD_SIZE]; + char cmdline[CMD_SIZE]; + + strcpy_s(filename, CMD_SIZE, "tmpXXXXXX"); + if (_mktemp_s(filename, CMD_SIZE) == 0) { + int rc; + + strcpy_s(cmdline, CMD_SIZE, "hg id -bit > "); + strcat_s(cmdline, CMD_SIZE, filename); + rc = system(cmdline); + if (rc == 0) { + FILE * fp; + + if (fopen_s(&fp, filename, "r") == 0) { + char * cp = fgets(cmdline, CMD_SIZE, fp); + + if (cp) { + char * context = NULL; + char * tp = strtok_s(cp, DELIMS, &context); + if (tp) { + strcpy_s(hgrev, size, tp); + tp = strtok_s(NULL, DELIMS, &context); + if (tp) { + strcpy_s(hgbranch, size, tp); + tp = strtok_s(NULL, DELIMS, &context); + if (tp) { + strcpy_s(hgtag, size, tp); + result = 1; + } + } + } + } + fclose(fp); + } + } + _unlink(filename); + } + return result; +} + int main(int argc, char*argv[]) { char command[CMD_SIZE] = "cl.exe -c -D_WIN32 -DUSE_DL_EXPORT -D_WINDOWS -DWIN32 -D_WINDLL "; @@ -109,8 +155,27 @@ strcat_s(command, CMD_SIZE, "\""); strcat_s(command, CMD_SIZE, tmppath); strcat_s(command, CMD_SIZE, "getbuildinfo2.c\" -DSUBWCREV "); - } else + } + else { + char hgtag[CMD_SIZE]; + char hgbranch[CMD_SIZE]; + char hgrev[CMD_SIZE]; + + if (get_mercurial_info(hgbranch, hgtag, hgrev, CMD_SIZE)) { + strcat_s(command, CMD_SIZE, "-DHGBRANCH=\\\""); + strcat_s(command, CMD_SIZE, hgbranch); + strcat_s(command, CMD_SIZE, "\\\""); + + strcat_s(command, CMD_SIZE, " -DHGTAG=\\\""); + strcat_s(command, CMD_SIZE, hgtag); + strcat_s(command, CMD_SIZE, "\\\""); + + strcat_s(command, CMD_SIZE, " -DHGVERSION=\\\""); + strcat_s(command, CMD_SIZE, hgrev); + strcat_s(command, CMD_SIZE, "\\\" "); + } strcat_s(command, CMD_SIZE, "..\\Modules\\getbuildinfo.c"); + } strcat_s(command, CMD_SIZE, " -Fo\""); strcat_s(command, CMD_SIZE, tmppath); strcat_s(command, CMD_SIZE, "getbuildinfo.o\" -I..\\Include -I..\\PC"); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:13:35 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 22:13:35 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313959=3A_Move_modu?= =?utf8?q?le_type_constants_to_Lib/imp=2Epy=2E?= Message-ID: http://hg.python.org/cpython/rev/22b0689346f9 changeset: 76760:22b0689346f9 user: Brett Cannon date: Fri May 04 16:13:30 2012 -0400 summary: Issue #13959: Move module type constants to Lib/imp.py. files: Python/import.c | 22 ---------------------- Python/importdl.h | 15 --------------- 2 files changed, 0 insertions(+), 37 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -2091,17 +2091,6 @@ {NULL, NULL} /* sentinel */ }; -static int -setint(PyObject *d, char *name, int value) -{ - PyObject *v; - int err; - - v = PyLong_FromLong((long)value); - err = PyDict_SetItemString(d, name, v); - Py_XDECREF(v); - return err; -} static struct PyModuleDef impmodule = { PyModuleDef_HEAD_INIT, @@ -2127,17 +2116,6 @@ if (d == NULL) goto failure; - if (setint(d, "SEARCH_ERROR", SEARCH_ERROR) < 0) goto failure; - if (setint(d, "PY_SOURCE", PY_SOURCE) < 0) goto failure; - if (setint(d, "PY_COMPILED", PY_COMPILED) < 0) goto failure; - if (setint(d, "C_EXTENSION", C_EXTENSION) < 0) goto failure; - if (setint(d, "PY_RESOURCE", PY_RESOURCE) < 0) goto failure; - if (setint(d, "PKG_DIRECTORY", PKG_DIRECTORY) < 0) goto failure; - if (setint(d, "C_BUILTIN", C_BUILTIN) < 0) goto failure; - if (setint(d, "PY_FROZEN", PY_FROZEN) < 0) goto failure; - if (setint(d, "PY_CODERESOURCE", PY_CODERESOURCE) < 0) goto failure; - if (setint(d, "IMP_HOOK", IMP_HOOK) < 0) goto failure; - return m; failure: Py_XDECREF(m); diff --git a/Python/importdl.h b/Python/importdl.h --- a/Python/importdl.h +++ b/Python/importdl.h @@ -6,21 +6,6 @@ #endif -/* Definitions for dynamic loading of extension modules */ -enum filetype { - SEARCH_ERROR, - PY_SOURCE, - PY_COMPILED, - C_EXTENSION, - PY_RESOURCE, /* Mac only */ - PKG_DIRECTORY, - C_BUILTIN, - PY_FROZEN, - PY_CODERESOURCE, /* Mac only */ - IMP_HOOK -}; - - extern const char *_PyImport_DynLoadFiletab[]; extern PyObject *_PyImport_LoadDynamicModule(PyObject *name, PyObject *pathname, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:15:33 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 22:15:33 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Kill_off_another_entry_of_i?= =?utf8?q?mport=5Fnt=2Ec?= Message-ID: http://hg.python.org/cpython/rev/06944dad6098 changeset: 76761:06944dad6098 user: Brett Cannon date: Fri May 04 16:15:26 2012 -0400 summary: Kill off another entry of import_nt.c files: PCbuild/pythoncore.vcproj | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/PCbuild/pythoncore.vcproj b/PCbuild/pythoncore.vcproj --- a/PCbuild/pythoncore.vcproj +++ b/PCbuild/pythoncore.vcproj @@ -1703,10 +1703,6 @@ > - - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:18:01 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 22:18:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Kill_remaining_mentions_of_?= =?utf8?q?import=5Fnt=2Ec?= Message-ID: http://hg.python.org/cpython/rev/5bcce348571e changeset: 76762:5bcce348571e parent: 76759:20f8b105525e user: Antoine Pitrou date: Fri May 04 22:15:57 2012 +0200 summary: Kill remaining mentions of import_nt.c files: PC/VC6/pythoncore.dsp | 5 ----- PC/readme.txt | 2 +- PCbuild/pythoncore.vcproj | 4 ---- 3 files changed, 1 insertions(+), 10 deletions(-) diff --git a/PC/VC6/pythoncore.dsp b/PC/VC6/pythoncore.dsp --- a/PC/VC6/pythoncore.dsp +++ b/PC/VC6/pythoncore.dsp @@ -466,11 +466,6 @@ # End Source File # Begin Source File -SOURCE=..\import_nt.c -# ADD CPP /I "..\..\Python" -# End Source File -# Begin Source File - SOURCE=..\..\Python\importdl.c # End Source File # Begin Source File diff --git a/PC/readme.txt b/PC/readme.txt --- a/PC/readme.txt +++ b/PC/readme.txt @@ -63,7 +63,7 @@ python_nt.rc Resource compiler input for python15.dll. -dl_nt.c, import_nt.c +dl_nt.c Additional sources used for 32-bit Windows features. getpathp.c Default sys.path calculations (for all PC platforms). diff --git a/PCbuild/pythoncore.vcproj b/PCbuild/pythoncore.vcproj --- a/PCbuild/pythoncore.vcproj +++ b/PCbuild/pythoncore.vcproj @@ -1703,10 +1703,6 @@ > - - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:18:02 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 22:18:02 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/5b2f40ed32fd changeset: 76763:5b2f40ed32fd parent: 76762:5bcce348571e parent: 76761:06944dad6098 user: Antoine Pitrou date: Fri May 04 22:16:09 2012 +0200 summary: Merge files: Python/import.c | 22 ---------------------- Python/importdl.h | 15 --------------- 2 files changed, 0 insertions(+), 37 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -2091,17 +2091,6 @@ {NULL, NULL} /* sentinel */ }; -static int -setint(PyObject *d, char *name, int value) -{ - PyObject *v; - int err; - - v = PyLong_FromLong((long)value); - err = PyDict_SetItemString(d, name, v); - Py_XDECREF(v); - return err; -} static struct PyModuleDef impmodule = { PyModuleDef_HEAD_INIT, @@ -2127,17 +2116,6 @@ if (d == NULL) goto failure; - if (setint(d, "SEARCH_ERROR", SEARCH_ERROR) < 0) goto failure; - if (setint(d, "PY_SOURCE", PY_SOURCE) < 0) goto failure; - if (setint(d, "PY_COMPILED", PY_COMPILED) < 0) goto failure; - if (setint(d, "C_EXTENSION", C_EXTENSION) < 0) goto failure; - if (setint(d, "PY_RESOURCE", PY_RESOURCE) < 0) goto failure; - if (setint(d, "PKG_DIRECTORY", PKG_DIRECTORY) < 0) goto failure; - if (setint(d, "C_BUILTIN", C_BUILTIN) < 0) goto failure; - if (setint(d, "PY_FROZEN", PY_FROZEN) < 0) goto failure; - if (setint(d, "PY_CODERESOURCE", PY_CODERESOURCE) < 0) goto failure; - if (setint(d, "IMP_HOOK", IMP_HOOK) < 0) goto failure; - return m; failure: Py_XDECREF(m); diff --git a/Python/importdl.h b/Python/importdl.h --- a/Python/importdl.h +++ b/Python/importdl.h @@ -6,21 +6,6 @@ #endif -/* Definitions for dynamic loading of extension modules */ -enum filetype { - SEARCH_ERROR, - PY_SOURCE, - PY_COMPILED, - C_EXTENSION, - PY_RESOURCE, /* Mac only */ - PKG_DIRECTORY, - C_BUILTIN, - PY_FROZEN, - PY_CODERESOURCE, /* Mac only */ - IMP_HOOK -}; - - extern const char *_PyImport_DynLoadFiletab[]; extern PyObject *_PyImport_LoadDynamicModule(PyObject *name, PyObject *pathname, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:42:31 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 22:42:31 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Simplify_code_for_load=5Fdy?= =?utf8?q?namic=28=29?= Message-ID: http://hg.python.org/cpython/rev/cd519a923d41 changeset: 76764:cd519a923d41 user: Antoine Pitrou date: Fri May 04 22:40:25 2012 +0200 summary: Simplify code for load_dynamic() files: Python/import.c | 46 ++---------------------------------- 1 files changed, 3 insertions(+), 43 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1964,48 +1964,6 @@ return PyBool_FromLong((long) (p == NULL ? 0 : p->size)); } -static FILE * -get_file(PyObject *pathname, PyObject *fob, char *mode) -{ - FILE *fp; - if (mode[0] == 'U') - mode = "r" PY_STDIOTEXTMODE; - if (fob == NULL) { - fp = _Py_fopen(pathname, mode); - if (!fp) { - if (!PyErr_Occurred()) - PyErr_SetFromErrno(PyExc_IOError); - return NULL; - } - return fp; - } - else { - int fd = PyObject_AsFileDescriptor(fob); - if (fd == -1) - return NULL; - if (!_PyVerify_fd(fd)) { - PyErr_SetFromErrno(PyExc_IOError); - return NULL; - } - - /* the FILE struct gets a new fd, so that it can be closed - * independently of the file descriptor given - */ - fd = dup(fd); - if (fd == -1) { - PyErr_SetFromErrno(PyExc_IOError); - return NULL; - } - - fp = fdopen(fd, mode); - if (!fp) { - PyErr_SetFromErrno(PyExc_IOError); - return NULL; - } - return fp; - } -} - #ifdef HAVE_DYNAMIC_LOADING static PyObject * @@ -2018,9 +1976,11 @@ &name, PyUnicode_FSDecoder, &pathname, &fob)) return NULL; if (fob != NULL) { - fp = get_file(NULL, fob, "r"); + fp = _Py_fopen(pathname, "r"); if (fp == NULL) { Py_DECREF(pathname); + if (!PyErr_Occurred()) + PyErr_SetFromErrno(PyExc_IOError); return NULL; } } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 22:48:09 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 04 May 2012 22:48:09 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Clean_up_a_docstring=2E?= Message-ID: http://hg.python.org/cpython/rev/146001e3364d changeset: 76765:146001e3364d user: Brett Cannon date: Fri May 04 16:47:54 2012 -0400 summary: Clean up a docstring. files: Lib/importlib/_bootstrap.py | 7 +------ Python/importlib.h | Bin 2 files changed, 1 insertions(+), 6 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1151,12 +1151,7 @@ def _install(sys_module, _imp_module): - """Install importlib as the implementation of import. - - It is assumed that _imp and sys have been imported and injected into the - global namespace for the module prior to calling this function. - - """ + """Install importlib as the implementation of import.""" _setup(sys_module, _imp_module) extensions = ExtensionFileLoader, _imp_module.extension_suffixes(), False source = SourceFileLoader, _SOURCE_SUFFIXES, True diff --git a/Python/importlib.h b/Python/importlib.h index 9c531f013f45b202d3523994af2799d4270339a7..292308fc78123e9fa10c7131bb110c47559975dd GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 23:18:56 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 23:18:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_some_whatsnew_entries?= Message-ID: http://hg.python.org/cpython/rev/66ccd993dfdf changeset: 76766:66ccd993dfdf parent: 76764:cd519a923d41 user: Antoine Pitrou date: Fri May 04 23:15:47 2012 +0200 summary: Add some whatsnew entries files: Doc/whatsnew/3.3.rst | 55 ++++++++++++++++++++++++++++++++ 1 files changed, 55 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -569,8 +569,17 @@ (:issue:`10516`) +* Raw bytes literals can now be written ``rb"..."`` as well as ``br"..."``. + (Contributed by Antoine Pitrou in :issue:`13748`.) + +* :meth:`dict.setdefault` now does only one lookup for the given key, making + it atomic when used with built-in types. + (Contributed by Filip Gruszczy?ski in :issue:`13521`.) + + .. XXX mention new error messages for passing wrong number of arguments to functions + New and Improved Modules ======================== @@ -851,6 +860,18 @@ (Written by Mark Dickinson in :issue:`11888`). +multiprocessing +--------------- + +The new :func:`multiprocessing.connection.wait` function allows to poll +multiple objects (such as connections, sockets and pipes) with a timeout. +(Contributed by Richard Oudkerk in :issue:`12328`.) + +:class:`multiprocessing.Connection` objects can now be transferred over +multiprocessing connections. +(Contributed by Richard Oudkerk in :issue:`4892`.) + + nntplib ------- @@ -897,6 +918,16 @@ :func:`~os.walk` except that it also yields file descriptors referring to the directories visited. This is especially useful to avoid symlink races. +* The new :func:`os.replace` function allows cross-platform renaming of a + file with overwriting the destination. With :func:`os.rename`, an existing + destination file is overwritten under POSIX, but raises an error under + Windows. + (Contributed by Antoine Pitrou in :issue:`8828`.) + +* The new :func:`os.get_terminal_size` function queries the size of the + terminal attached to a file descriptor. + (Contributed by Zbigniew J?drzejewski-Szmek in :issue:`13609`.) + * "at" functions (:issue:`4761`): * :func:`~os.faccessat` @@ -988,6 +1019,15 @@ are completed. (Contributed by Georg Brandl in :issue:`14210`) +pickle +------ + +:class:`pickle.Pickler` objects now have an optional +:attr:`~pickle.Pickler.dispatch_table` attribute allowing to set per-pickler +reduction functions. +(Contributed by Richard Oudkerk in :issue:`14166`.) + + pydoc ----- @@ -1034,6 +1074,16 @@ path also specifying the user/group names and not only their numeric ids. (Contributed by Sandro Tosi in :issue:`12191`) +* The new :func:`shutil.get_terminal_size` function returns the size of the + terminal window the interpreter is attached to. + (Contributed by Zbigniew J?drzejewski-Szmek in :issue:`13609`.) + +* Several functions now take an optional ``symlinks`` argument: when that + parameter is true, symlinks aren't dereferenced and the operation instead + acts on the symlink itself (or creates one, if relevant). + (Contributed by Hynek Schlawack in :issue:`12715`.) + + signal ------ @@ -1129,6 +1179,11 @@ (Contributed by Antoine Pitrou in :issue:`13634`) +* Support has been added for the Next Procotol Negotiation extension using + the :meth:`ssl.SSLContext.set_npn_protocols` method. + + (Contributed by Colin Marc in :issue:`14204`) + sys --- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 23:18:57 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 04 May 2012 23:18:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/57cbffae4f74 changeset: 76767:57cbffae4f74 parent: 76766:66ccd993dfdf parent: 76765:146001e3364d user: Antoine Pitrou date: Fri May 04 23:17:03 2012 +0200 summary: Merge files: Lib/importlib/_bootstrap.py | 7 +------ Python/importlib.h | Bin 2 files changed, 1 insertions(+), 6 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1151,12 +1151,7 @@ def _install(sys_module, _imp_module): - """Install importlib as the implementation of import. - - It is assumed that _imp and sys have been imported and injected into the - global namespace for the module prior to calling this function. - - """ + """Install importlib as the implementation of import.""" _setup(sys_module, _imp_module) extensions = ExtensionFileLoader, _imp_module.extension_suffixes(), False source = SourceFileLoader, _SOURCE_SUFFIXES, True diff --git a/Python/importlib.h b/Python/importlib.h index 9c531f013f45b202d3523994af2799d4270339a7..292308fc78123e9fa10c7131bb110c47559975dd GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 4 23:44:36 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 04 May 2012 23:44:36 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Note_that_=5F=5Ffile=5F=5F_is_?= =?utf8?q?not_set_on_the_namespace_package=2E?= Message-ID: http://hg.python.org/peps/rev/aa5f4a4b0289 changeset: 4354:aa5f4a4b0289 user: Eric V. Smith date: Fri May 04 17:44:32 2012 -0400 summary: Note that __file__ is not set on the namespace package. files: pep-0420.txt | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -135,12 +135,11 @@ least one directory was recorded, then a namespace package is created. The new namespace package: - * Has a ``__file__`` attribute set to the first directory that was - found during the scan, including the trailing path separator. - * Has a ``__path__`` attribute set to the list of directories that were found and recorded during the scan. + * Does not have a ``__file__`` attribute. + There is no mechanism to automatically recompute the ``__path__`` if ``sys.path`` is altered after a namespace package has already been created. However, existing namespace utilities (like -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Sat May 5 05:44:37 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 05 May 2012 05:44:37 +0200 Subject: [Python-checkins] Daily reference leaks (57cbffae4f74): sum=9 Message-ID: results for 57cbffae4f74 on branch "default" -------------------------------------------- test_imp leaked [1, 1, 1] references, sum=3 test_importhooks leaked [2, 2, 2] references, sum=6 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog9H8qFH', '-x'] From python-checkins at python.org Sat May 5 12:27:37 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sat, 05 May 2012 12:27:37 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo_in_changeset_eb5c5?= =?utf8?q?c23ca9b=2E?= Message-ID: http://hg.python.org/cpython/rev/babe4dd3fe0d changeset: 76768:babe4dd3fe0d user: Nadeem Vawda date: Sat May 05 12:27:30 2012 +0200 summary: Fix typo in changeset eb5c5c23ca9b. files: Python/import.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -181,7 +181,7 @@ error: PyErr_Print(); Py_FatalError("initializing sys.meta_path, sys.path_hooks, " - "or path_importer_cache"); + "or path_importer_cache failed"); } Py_DECREF(path_hooks); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 5 12:35:53 2012 From: python-checkins at python.org (sandro.tosi) Date: Sat, 05 May 2012 12:35:53 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_fix_a_=27ERROR=3A_invalid_?= =?utf8?q?option_block=27_with_newer_sphinx?= Message-ID: http://hg.python.org/devguide/rev/8e7b834f92e3 changeset: 507:8e7b834f92e3 user: Sandro Tosi date: Sat May 05 12:33:58 2012 +0200 summary: fix a 'ERROR: invalid option block' with newer sphinx files: runtests.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/runtests.rst b/runtests.rst --- a/runtests.rst +++ b/runtests.rst @@ -4,6 +4,7 @@ ======================= .. note:: + This document assumes you are working from an :ref:`in-development ` checkout of Python. If you are not then some things presented here may not work as they may depend -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sat May 5 14:57:09 2012 From: python-checkins at python.org (nick.coghlan) Date: Sat, 05 May 2012 14:57:09 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Update_PEP_1_to_better_reflect?= =?utf8?q?_current_practice?= Message-ID: http://hg.python.org/peps/rev/bdbbd3ce97d9 changeset: 4355:bdbbd3ce97d9 user: Nick Coghlan date: Sat May 05 22:56:57 2012 +1000 summary: Update PEP 1 to better reflect current practice files: pep-0001.txt | 54 +++++++++++++++++++++++++++++++-------- 1 files changed, 43 insertions(+), 11 deletions(-) diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -2,12 +2,12 @@ Title: PEP Purpose and Guidelines Version: $Revision$ Last-Modified: $Date$ -Author: Barry Warsaw, Jeremy Hylton, David Goodger +Author: Barry Warsaw, Jeremy Hylton, David Goodger, Nick Coghlan Status: Active Type: Process Content-Type: text/x-rst Created: 13-Jun-2000 -Post-History: 21-Mar-2001, 29-Jul-2002, 03-May-2003 +Post-History: 21-Mar-2001, 29-Jul-2002, 03-May-2003, 05-May-2012 What is a PEP? @@ -60,6 +60,9 @@ PEP Work Flow ============= +Submitting a PEP +---------------- + The PEP editors assign PEP numbers and change their status. Please send all PEP-related email to (no cross-posting please). Also see `PEP Editor Responsibilities & Workflow`_ below. @@ -115,6 +118,11 @@ Dictator for Life, Guido van Rossum) can be consulted during the approval phase, and is the final arbiter of the draft's PEP-ability. +Developers with commit privileges for the `PEP repository`_ may claim +PEP numbers directly by creating and committing a new PEP. When doing so, +the developer must handle the tasks that would normally be taken care of by +the PEP editors (see `PEP Editor Responsibilities & Workflow`_). + As updates are necessary, the PEP author can check in new versions if they have hg push privileges, or can email new PEP versions to the PEP editor for committing. @@ -134,6 +142,9 @@ private comments in the early design phases, setting up a wiki page, etc. PEP authors should use their discretion here. +PEP Review & Resolution +----------------------- + Once the authors have completed a PEP, they must inform the PEP editor that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to @@ -143,6 +154,16 @@ first notifying the PEP author(s) and giving them a chance to make revisions. +The final authority for PEP approval is the BDFL. However, whenever a new +PEP is put forward, any core developer that believes they are suitably +experienced to make the final decision on that PEP may offer to serve as +the "PEP czar" for that PEP. If their self-nomination is accepted by the +other core developers and the BDFL, then they will have the authority to +approve (or reject) that PEP. This process happens most frequently with PEPs +where the BDFL has granted in principle approval for *something* to be done, +but there are details that need to be worked out before the PEP can be +accepted. + For a PEP to be accepted it must meet certain minimum criteria. It must be a clear and complete description of the proposed enhancement. The enhancement must represent a net improvement. The proposed @@ -154,8 +175,8 @@ standard library module acceptance criteria. Once a PEP has been accepted, the reference implementation must be -completed. When the reference implementation is complete and accepted -by the BDFL, the status will be changed to "Final". +completed. When the reference implementation is complete and incorporated +into the main source code repository, the status will be changed to "Final". A PEP can also be assigned status "Deferred". The PEP author or editor can assign the PEP this status when no progress is being made @@ -164,7 +185,14 @@ A PEP can also be "Rejected". Perhaps after all is said and done it was not a good idea. It is still important to have a record of this -fact. +fact. The "Withdrawn" status is similar - it means that the PEP author +themselves has decided that the PEP is actually a bad idea, or has +accepted that a competing proposal is a better alternative. + +When a PEP is Accepted, Rejected or Withdrawn, the PEP should be updated +accordingly. In addition to updating the status field, at the very least +the Resolution header should be added with a link to the relevant post +in the python-dev mailing list archives. PEPs can also be superseded by a different PEP, rendering the original obsolete. This is intended for Informational PEPs, where version 2 of @@ -420,17 +448,19 @@ Python 3 only, we're back to using numbers in the 100s again. Remember that numbers below 100 are meta-PEPs.) -* List the PEP in PEP 0 (in two places: the categorized list, and the - numeric list). - -* Add the PEP to Mercurial. For mercurial work flow instructions, follow - `The Python Developers Guide `_ +* Add the PEP to a local clone of the PEP repository. For mercurial work + flow instructions, follow `The Python Developers Guide `_ The mercurial repo for the peps is:: http://hg.python.org/peps/ +* Run ``./genpepindex.py`` and ``./pep2html.py `` to ensure they + are generated without errors. If either triggers errors, then the web site + will not be updated to reflect the PEP changes. +* Commit and push the new (or updated) PEP + * Monitor python.org to make sure the PEP gets added to the site properly. @@ -438,7 +468,7 @@ python-list & -dev). Updates to existing PEPs also come in to peps at python.org. Many PEP -authors are not Python committers yet, so we do the commits for them. +authors are not Python committers yet, so PEP editors do the commits for them. Many PEPs are written and maintained by developers with write access to the Python codebase. The PEP editors monitor the python-checkins @@ -492,6 +522,8 @@ .. _Docutils: http://docutils.sourceforge.net/ +.. _PEP repository: http://hg.python.org/peps + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat May 5 15:44:13 2012 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 05 May 2012 15:44:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_don=27t_append_the_bytecode?= =?utf8?q?_suffix_to_the_source_suffixes_global?= Message-ID: http://hg.python.org/cpython/rev/e1c2c575fc0e changeset: 76769:e1c2c575fc0e user: Benjamin Peterson date: Sat May 05 09:44:08 2012 -0400 summary: don't append the bytecode suffix to the source suffixes global files: Lib/imp.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/imp.py b/Lib/imp.py --- a/Lib/imp.py +++ b/Lib/imp.py @@ -126,8 +126,7 @@ # XXX deprecate def load_package(name, path): if os.path.isdir(path): - extensions = _bootstrap._SOURCE_SUFFIXES - extensions += [_bootstrap._BYTECODE_SUFFIX] + extensions = _bootstrap._SOURCE_SUFFIXES + [_bootstrap._BYTECODE_SUFFIX] for extension in extensions: path = os.path.join(path, '__init__'+extension) if os.path.exists(path): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 5 17:58:04 2012 From: python-checkins at python.org (georg.brandl) Date: Sat, 05 May 2012 17:58:04 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_419=3A_update_from_Paul=2E?= Message-ID: http://hg.python.org/peps/rev/13f1bc1df1ce changeset: 4356:13f1bc1df1ce user: Georg Brandl date: Sat May 05 17:58:14 2012 +0200 summary: PEP 419: update from Paul. files: pep-0419.txt | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/pep-0419.txt b/pep-0419.txt --- a/pep-0419.txt +++ b/pep-0419.txt @@ -310,6 +310,9 @@ explicitly. An example of such code might be a file-based lock implementation. +:func:`signal.pthread_sigmask` can be used to block signals inside +cleanup handlers which can be interrupted with ``EINTR``. + Setting Interruption Context Inside Finally Itself -------------------------------------------------- @@ -500,6 +503,9 @@ .. [4] Original discussion http://mail.python.org/pipermail/python-ideas/2012-April/014705.html +.. [5] Issue #14730: Implementation of the PEP 419 + http://bugs.python.org/issue14730 + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat May 5 18:16:18 2012 From: python-checkins at python.org (lars.gustaebel) Date: Sat, 05 May 2012 18:16:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313815=3A_TarFile?= =?utf8?q?=2Eextractfile=28=29_now_returns_io=2EBufferedReader_objects=2E?= Message-ID: http://hg.python.org/cpython/rev/254cb4f5d0ff changeset: 76770:254cb4f5d0ff user: Lars Gust?bel date: Sat May 05 18:15:03 2012 +0200 summary: Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. The ExFileObject class was removed, some of its code went into _FileInFile. files: Doc/library/tarfile.rst | 13 +- Lib/tarfile.py | 202 ++++++-------------------- Lib/test/test_tarfile.py | 69 ++++----- Misc/NEWS | 2 + 4 files changed, 83 insertions(+), 203 deletions(-) diff --git a/Doc/library/tarfile.rst b/Doc/library/tarfile.rst --- a/Doc/library/tarfile.rst +++ b/Doc/library/tarfile.rst @@ -376,15 +376,12 @@ .. method:: TarFile.extractfile(member) Extract a member from the archive as a file object. *member* may be a filename - or a :class:`TarInfo` object. If *member* is a regular file, a :term:`file-like - object` is returned. If *member* is a link, a file-like object is constructed from - the link's target. If *member* is none of the above, :const:`None` is returned. + or a :class:`TarInfo` object. If *member* is a regular file or a link, an + :class:`io.BufferedReader` object is returned. Otherwise, :const:`None` is + returned. - .. note:: - - The file-like object is read-only. It provides the methods - :meth:`read`, :meth:`readline`, :meth:`readlines`, :meth:`seek`, :meth:`tell`, - and :meth:`close`, and also supports iteration over its lines. + .. versionchanged:: 3.3 + Return an :class:`io.BufferedReader` object. .. method:: TarFile.add(name, arcname=None, recursive=True, exclude=None, *, filter=None) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -668,6 +668,8 @@ self.offset = offset self.size = size self.position = 0 + self.name = getattr(fileobj, "name", None) + self.closed = False if blockinfo is None: blockinfo = [(0, size)] @@ -686,10 +688,16 @@ if lastpos < self.size: self.map.append((False, lastpos, self.size, None)) + def flush(self): + pass + + def readable(self): + return True + + def writable(self): + return False + def seekable(self): - if not hasattr(self.fileobj, "seekable"): - # XXX gzip.GzipFile and bz2.BZ2File - return True return self.fileobj.seekable() def tell(self): @@ -697,10 +705,21 @@ """ return self.position - def seek(self, position): + def seek(self, position, whence=io.SEEK_SET): """Seek to a position in the file. """ - self.position = position + if whence == io.SEEK_SET: + self.position = min(max(position, 0), self.size) + elif whence == io.SEEK_CUR: + if position < 0: + self.position = max(self.position + position, 0) + else: + self.position = min(self.position + position, self.size) + elif whence == io.SEEK_END: + self.position = max(min(self.size + position, self.size), 0) + else: + raise ValueError("Invalid argument") + return self.position def read(self, size=None): """Read data from the file. @@ -729,147 +748,17 @@ size -= length self.position += length return buf + + def readinto(self, b): + buf = self.read(len(b)) + b[:len(buf)] = buf + return len(buf) + + def close(self): + self.closed = True #class _FileInFile -class ExFileObject(object): - """File-like object for reading an archive member. - Is returned by TarFile.extractfile(). - """ - blocksize = 1024 - - def __init__(self, tarfile, tarinfo): - self.fileobj = _FileInFile(tarfile.fileobj, - tarinfo.offset_data, - tarinfo.size, - tarinfo.sparse) - self.name = tarinfo.name - self.mode = "r" - self.closed = False - self.size = tarinfo.size - - self.position = 0 - self.buffer = b"" - - def readable(self): - return True - - def writable(self): - return False - - def seekable(self): - return self.fileobj.seekable() - - def read(self, size=None): - """Read at most size bytes from the file. If size is not - present or None, read all data until EOF is reached. - """ - if self.closed: - raise ValueError("I/O operation on closed file") - - buf = b"" - if self.buffer: - if size is None: - buf = self.buffer - self.buffer = b"" - else: - buf = self.buffer[:size] - self.buffer = self.buffer[size:] - - if size is None: - buf += self.fileobj.read() - else: - buf += self.fileobj.read(size - len(buf)) - - self.position += len(buf) - return buf - - # XXX TextIOWrapper uses the read1() method. - read1 = read - - def readline(self, size=-1): - """Read one entire line from the file. If size is present - and non-negative, return a string with at most that - size, which may be an incomplete line. - """ - if self.closed: - raise ValueError("I/O operation on closed file") - - pos = self.buffer.find(b"\n") + 1 - if pos == 0: - # no newline found. - while True: - buf = self.fileobj.read(self.blocksize) - self.buffer += buf - if not buf or b"\n" in buf: - pos = self.buffer.find(b"\n") + 1 - if pos == 0: - # no newline found. - pos = len(self.buffer) - break - - if size != -1: - pos = min(size, pos) - - buf = self.buffer[:pos] - self.buffer = self.buffer[pos:] - self.position += len(buf) - return buf - - def readlines(self): - """Return a list with all remaining lines. - """ - result = [] - while True: - line = self.readline() - if not line: break - result.append(line) - return result - - def tell(self): - """Return the current file position. - """ - if self.closed: - raise ValueError("I/O operation on closed file") - - return self.position - - def seek(self, pos, whence=io.SEEK_SET): - """Seek to a position in the file. - """ - if self.closed: - raise ValueError("I/O operation on closed file") - - if whence == io.SEEK_SET: - self.position = min(max(pos, 0), self.size) - elif whence == io.SEEK_CUR: - if pos < 0: - self.position = max(self.position + pos, 0) - else: - self.position = min(self.position + pos, self.size) - elif whence == io.SEEK_END: - self.position = max(min(self.size + pos, self.size), 0) - else: - raise ValueError("Invalid argument") - - self.buffer = b"" - self.fileobj.seek(self.position) - - def close(self): - """Close the file object. - """ - self.closed = True - - def __iter__(self): - """Get an iterator over the file's lines. - """ - while True: - line = self.readline() - if not line: - break - yield line -#class ExFileObject - #------------------ # Exported Classes #------------------ @@ -1554,7 +1443,8 @@ tarinfo = TarInfo # The default TarInfo class to use. - fileobject = ExFileObject # The default ExFileObject class to use. + fileobject = None # The file-object for extractfile() or + # io.BufferedReader if None. def __init__(self, name=None, mode="r", fileobj=None, format=None, tarinfo=None, dereference=None, ignore_zeros=None, encoding=None, @@ -2178,12 +2068,9 @@ def extractfile(self, member): """Extract a member from the archive as a file object. `member' may be - a filename or a TarInfo object. If `member' is a regular file, a - file-like object is returned. If `member' is a link, a file-like - object is constructed from the link's target. If `member' is none of - the above, None is returned. - The file-like object is read-only and provides the following - methods: read(), readline(), readlines(), seek() and tell() + a filename or a TarInfo object. If `member' is a regular file or a + link, an io.BufferedReader object is returned. Otherwise, None is + returned. """ self._check("r") @@ -2192,13 +2079,14 @@ else: tarinfo = member - if tarinfo.isreg(): - return self.fileobject(self, tarinfo) - - elif tarinfo.type not in SUPPORTED_TYPES: - # If a member's type is unknown, it is treated as a - # regular file. - return self.fileobject(self, tarinfo) + if tarinfo.isreg() or tarinfo.type not in SUPPORTED_TYPES: + # Members with unknown types are treated as regular files. + if self.fileobject is None: + fileobj = _FileInFile(self.fileobj, tarinfo.offset_data, tarinfo.size, tarinfo.sparse) + return io.BufferedReader(fileobj) + else: + # Keep the traditional pre-3.3 API intact. + return self.fileobject(self, tarinfo) elif tarinfo.islnk() or tarinfo.issym(): if isinstance(self.fileobj, _Stream): diff --git a/Lib/test/test_tarfile.py b/Lib/test/test_tarfile.py --- a/Lib/test/test_tarfile.py +++ b/Lib/test/test_tarfile.py @@ -56,13 +56,10 @@ def test_fileobj_regular_file(self): tarinfo = self.tar.getmember("ustar/regtype") - fobj = self.tar.extractfile(tarinfo) - try: + with self.tar.extractfile(tarinfo) as fobj: data = fobj.read() self.assertTrue((len(data), md5sum(data)) == (tarinfo.size, md5_regtype), "regular file extraction failed") - finally: - fobj.close() def test_fileobj_readlines(self): self.tar.extract("ustar/regtype", TEMPDIR) @@ -70,8 +67,7 @@ with open(os.path.join(TEMPDIR, "ustar/regtype"), "r") as fobj1: lines1 = fobj1.readlines() - fobj = self.tar.extractfile(tarinfo) - try: + with self.tar.extractfile(tarinfo) as fobj: fobj2 = io.TextIOWrapper(fobj) lines2 = fobj2.readlines() self.assertTrue(lines1 == lines2, @@ -81,21 +77,16 @@ self.assertTrue(lines2[83] == "I will gladly admit that Python is not the fastest running scripting language.\n", "fileobj.readlines() failed") - finally: - fobj.close() def test_fileobj_iter(self): self.tar.extract("ustar/regtype", TEMPDIR) tarinfo = self.tar.getmember("ustar/regtype") with open(os.path.join(TEMPDIR, "ustar/regtype"), "r") as fobj1: lines1 = fobj1.readlines() - fobj2 = self.tar.extractfile(tarinfo) - try: + with self.tar.extractfile(tarinfo) as fobj2: lines2 = list(io.TextIOWrapper(fobj2)) self.assertTrue(lines1 == lines2, "fileobj.__iter__() failed") - finally: - fobj2.close() def test_fileobj_seek(self): self.tar.extract("ustar/regtype", TEMPDIR) @@ -147,17 +138,24 @@ "read() after readline() failed") fobj.close() + def test_fileobj_text(self): + with self.tar.extractfile("ustar/regtype") as fobj: + fobj = io.TextIOWrapper(fobj) + data = fobj.read().encode("iso8859-1") + self.assertEqual(md5sum(data), md5_regtype) + try: + fobj.seek(100) + except AttributeError: + # Issue #13815: seek() complained about a missing + # flush() method. + self.fail("seeking failed in text mode") + # Test if symbolic and hard links are resolved by extractfile(). The # test link members each point to a regular member whose data is # supposed to be exported. def _test_fileobj_link(self, lnktype, regtype): - a = self.tar.extractfile(lnktype) - b = self.tar.extractfile(regtype) - try: + with self.tar.extractfile(lnktype) as a, self.tar.extractfile(regtype) as b: self.assertEqual(a.name, b.name) - finally: - a.close() - b.close() def test_fileobj_link1(self): self._test_fileobj_link("ustar/lnktype", "ustar/regtype") @@ -265,9 +263,8 @@ t = tar.next() name = t.name offset = t.offset - f = tar.extractfile(t) - data = f.read() - f.close() + with tar.extractfile(t) as f: + data = f.read() finally: tar.close() @@ -439,27 +436,26 @@ for tarinfo in self.tar: if not tarinfo.isreg(): continue - fobj = self.tar.extractfile(tarinfo) - while True: - try: - buf = fobj.read(512) - except tarfile.StreamError: - self.fail("simple read-through using TarFile.extractfile() failed") - if not buf: - break - fobj.close() + with self.tar.extractfile(tarinfo) as fobj: + while True: + try: + buf = fobj.read(512) + except tarfile.StreamError: + self.fail("simple read-through using TarFile.extractfile() failed") + if not buf: + break def test_fileobj_regular_file(self): tarinfo = self.tar.next() # get "regtype" (can't use getmember) - fobj = self.tar.extractfile(tarinfo) - data = fobj.read() + with self.tar.extractfile(tarinfo) as fobj: + data = fobj.read() self.assertTrue((len(data), md5sum(data)) == (tarinfo.size, md5_regtype), "regular file extraction failed") def test_provoke_stream_error(self): tarinfos = self.tar.getmembers() - f = self.tar.extractfile(tarinfos[0]) # read the first member - self.assertRaises(tarfile.StreamError, f.read) + with self.tar.extractfile(tarinfos[0]) as f: # read the first member + self.assertRaises(tarfile.StreamError, f.read) def test_compare_members(self): tar1 = tarfile.open(tarname, encoding="iso8859-1") @@ -1484,12 +1480,9 @@ with tarfile.open(tarname, encoding="iso8859-1") as src: t = src.getmember("ustar/regtype") t.name = "foo" - f = src.extractfile(t) - try: + with src.extractfile(t) as f: with tarfile.open(self.tarname, mode) as tar: tar.addfile(t, f) - finally: - f.close() def _test(self, names=["bar"], fileobj=None): with tarfile.open(self.tarname, fileobj=fileobj) as tar: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -89,6 +89,8 @@ Library ------- +- Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. + - Issue #14371: Support bzip2 in zipfile module. Patch by Serhiy Storchaka. -- Repository URL: http://hg.python.org/cpython From barry at python.org Sat May 5 18:56:54 2012 From: barry at python.org (Barry Warsaw) Date: Sat, 5 May 2012 12:56:54 -0400 Subject: [Python-checkins] peps: Update PEP 1 to better reflect current practice In-Reply-To: References: Message-ID: <20120505125654.77d3b9ab@resist.wooz.org> Thanks for doing this update Nick. I have just a few comments. On May 05, 2012, at 02:57 PM, nick.coghlan wrote: >+Developers with commit privileges for the `PEP repository`_ may claim >+PEP numbers directly by creating and committing a new PEP. When doing so, >+the developer must handle the tasks that would normally be taken care of by >+the PEP editors (see `PEP Editor Responsibilities & Workflow`_). While I certainly don't mind (in fact, prefer) those with commit privileges to just go ahead and commit their PEP to the repo, I'd like for there to be *some* communication with the PEP editors first. E.g. sanity checks on the basic format or idea (was this discussed on python-ideas first?), or reservation of PEP numbers. When you do contact the PEP editors, please also specify whether you have commit privileges or not. It's too hard to remember or know who has those rights, and too much hassle to look them up. ;) OTOH, I'm also happy to adopt an EAFP style rather than LBYL, so that the PEP editors can re-assign numbers or whatever after the fact. We've done this in a few cases, and it's never been that much of a problem. Still, core developers needn't block (for too long) on the PEP editors. >+The final authority for PEP approval is the BDFL. However, whenever a new >+PEP is put forward, any core developer that believes they are suitably >+experienced to make the final decision on that PEP may offer to serve as >+the "PEP czar" for that PEP. If their self-nomination is accepted by the >+other core developers and the BDFL, then they will have the authority to >+approve (or reject) that PEP. This process happens most frequently with PEPs >+where the BDFL has granted in principle approval for *something* to be done, >+but there are details that need to be worked out before the PEP can be >+accepted. I'd reword this to something like the following: The final authority for the PEP approval is the BDFL. However, the BDFL may delegate the final approval authority to a "PEP czar" for that PEP. This happens most frequently with PEPs where the BDFL has granted approval in principle for *something* to be done, and in agreement with the general proposals of the PEP, but there are details that need to be worked out before the final PEP can be approved. When an `PEP-Czar` header must be added to the PEP to record this delegation. The format of this header is the same as the `Author` header. This leave out the whole self-nomination text, which I think isn't very relevant to the official addition of the czar role (sadly, no clever bacronym has come to mind, and BDFOP hasn't really taken off ;). >+* Run ``./genpepindex.py`` and ``./pep2html.py `` to ensure they >+ are generated without errors. If either triggers errors, then the web site >+ will not be updated to reflect the PEP changes. Or just run "make" on systems that have that handy convenience. :) Cheers, -Barry (Nick, if you agree with these changes, please just go ahead and make them.) From barry at python.org Sat May 5 18:59:53 2012 From: barry at python.org (Barry Warsaw) Date: Sat, 5 May 2012 12:59:53 -0400 Subject: [Python-checkins] peps: Update PEP 1 to better reflect current practice In-Reply-To: <20120505125654.77d3b9ab@resist.wooz.org> References: <20120505125654.77d3b9ab@resist.wooz.org> Message-ID: <20120505125953.1ef90d2f@resist.wooz.org> On May 05, 2012, at 12:56 PM, Barry Warsaw wrote: > before the final PEP can be approved. When an `PEP-Czar` header must be > added to the PEP to record this delegation. The format of this header is > the same as the `Author` header. s/When an/A/ -Barry From g.brandl at gmx.net Sat May 5 19:04:49 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 05 May 2012 19:04:49 +0200 Subject: [Python-checkins] peps: Update PEP 1 to better reflect current practice In-Reply-To: <20120505125654.77d3b9ab@resist.wooz.org> References: <20120505125654.77d3b9ab@resist.wooz.org> Message-ID: On 05/05/2012 06:56 PM, Barry Warsaw wrote: >>+* Run ``./genpepindex.py`` and ``./pep2html.py `` to ensure they >>+ are generated without errors. If either triggers errors, then the web site >>+ will not be updated to reflect the PEP changes. > > Or just run "make" on systems that have that handy convenience. :) Though without target, it builds all PEPs, which takes quite a long time. Georg From python-checkins at python.org Sat May 5 20:48:15 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sat, 05 May 2012 20:48:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_for_Issue_14725_for_3?= =?utf8?q?=2E3_branch=2E?= Message-ID: http://hg.python.org/cpython/rev/44f078ea05f3 changeset: 76771:44f078ea05f3 user: Richard Oudkerk date: Sat May 05 19:45:37 2012 +0100 summary: Fix for Issue 14725 for 3.3 branch. files: Lib/multiprocessing/connection.py | 26 ++++++++++++------ Lib/test/test_multiprocessing.py | 16 +++++++++++ Modules/_winapi.c | 4 +-- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -621,16 +621,24 @@ def accept(self): self._handle_queue.append(self._new_handle()) handle = self._handle_queue.pop(0) - ov = _winapi.ConnectNamedPipe(handle, overlapped=True) try: - res = _winapi.WaitForMultipleObjects([ov.event], False, INFINITE) - except: - ov.cancel() - _winapi.CloseHandle(handle) - raise - finally: - _, err = ov.GetOverlappedResult(True) - assert err == 0 + ov = _winapi.ConnectNamedPipe(handle, overlapped=True) + except OSError as e: + if e.winerror != _winapi.ERROR_NO_DATA: + raise + # ERROR_NO_DATA can occur if a client has already connected, + # written data and then disconnected -- see Issue 14725. + else: + try: + res = _winapi.WaitForMultipleObjects( + [ov.event], False, INFINITE) + except: + ov.cancel() + _winapi.CloseHandle(handle) + raise + finally: + _, err = ov.GetOverlappedResult(True) + assert err == 0 return PipeConnection(handle) @staticmethod diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -1878,6 +1878,22 @@ p.join() l.close() + def test_issue14725(self): + l = self.connection.Listener() + p = self.Process(target=self._test, args=(l.address,)) + p.daemon = True + p.start() + time.sleep(1) + # On Windows the client process should by now have connected, + # written data and closed the pipe handle by now. This causes + # ConnectNamdedPipe() to fail with ERROR_NO_DATA. See Issue + # 14725. + conn = l.accept() + self.assertEqual(conn.recv(), 'hello') + conn.close() + p.join() + l.close() + class _TestPoll(unittest.TestCase): ALLOWED_TYPES = ('processes', 'threads') diff --git a/Modules/_winapi.c b/Modules/_winapi.c --- a/Modules/_winapi.c +++ b/Modules/_winapi.c @@ -1287,11 +1287,9 @@ WINAPI_CONSTANT(F_DWORD, ERROR_MORE_DATA); WINAPI_CONSTANT(F_DWORD, ERROR_NETNAME_DELETED); WINAPI_CONSTANT(F_DWORD, ERROR_NO_SYSTEM_RESOURCES); - WINAPI_CONSTANT(F_DWORD, ERROR_NETNAME_DELETED); - WINAPI_CONSTANT(F_DWORD, ERROR_NO_SYSTEM_RESOURCES); - WINAPI_CONSTANT(F_DWORD, ERROR_IO_PENDING); WINAPI_CONSTANT(F_DWORD, ERROR_MORE_DATA); WINAPI_CONSTANT(F_DWORD, ERROR_NETNAME_DELETED); + WINAPI_CONSTANT(F_DWORD, ERROR_NO_DATA); WINAPI_CONSTANT(F_DWORD, ERROR_NO_SYSTEM_RESOURCES); WINAPI_CONSTANT(F_DWORD, ERROR_OPERATION_ABORTED); WINAPI_CONSTANT(F_DWORD, ERROR_PIPE_BUSY); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 5 21:59:38 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sat, 05 May 2012 21:59:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_for_issue_1?= =?utf8?q?4725_for_2=2E7_branch?= Message-ID: http://hg.python.org/cpython/rev/35ef949e85d7 changeset: 76772:35ef949e85d7 branch: 2.7 parent: 76751:8215aaccc9dd user: Richard Oudkerk date: Sat May 05 20:41:08 2012 +0100 summary: Fix for issue 14725 for 2.7 branch files: Lib/multiprocessing/connection.py | 5 ++- Lib/test/test_multiprocessing.py | 17 ++++++++++ Modules/_multiprocessing/win32_functions.c | 1 + 3 files changed, 22 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -348,7 +348,10 @@ try: win32.ConnectNamedPipe(handle, win32.NULL) except WindowsError, e: - if e.args[0] != win32.ERROR_PIPE_CONNECTED: + # ERROR_NO_DATA can occur if a client has already connected, + # written data and then disconnected -- see Issue 14725. + if e.args[0] not in (win32.ERROR_PIPE_CONNECTED, + win32.ERROR_NO_DATA): raise return _multiprocessing.PipeConnection(handle) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -1669,6 +1669,23 @@ self.assertEqual(conn.recv(), 'hello') p.join() l.close() + + def test_issue14725(self): + l = self.connection.Listener() + p = self.Process(target=self._test, args=(l.address,)) + p.daemon = True + p.start() + time.sleep(1) + # On Windows the client process should by now have connected, + # written data and closed the pipe handle by now. This causes + # ConnectNamdedPipe() to fail with ERROR_NO_DATA. See Issue + # 14725. + conn = l.accept() + self.assertEqual(conn.recv(), 'hello') + conn.close() + p.join() + l.close() + # # Test of sending connection and socket objects between processes # diff --git a/Modules/_multiprocessing/win32_functions.c b/Modules/_multiprocessing/win32_functions.c --- a/Modules/_multiprocessing/win32_functions.c +++ b/Modules/_multiprocessing/win32_functions.c @@ -244,6 +244,7 @@ Py_INCREF(&Win32Type); WIN32_CONSTANT(F_DWORD, ERROR_ALREADY_EXISTS); + WIN32_CONSTANT(F_DWORD, ERROR_NO_DATA); WIN32_CONSTANT(F_DWORD, ERROR_PIPE_BUSY); WIN32_CONSTANT(F_DWORD, ERROR_PIPE_CONNECTED); WIN32_CONSTANT(F_DWORD, ERROR_SEM_TIMEOUT); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 5 21:59:38 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sat, 05 May 2012 21:59:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_for_issue_1?= =?utf8?q?4725_for_3=2E2_branch?= Message-ID: http://hg.python.org/cpython/rev/afab4d14d5e7 changeset: 76773:afab4d14d5e7 branch: 3.2 parent: 76749:683295cf9489 user: Richard Oudkerk date: Sat May 05 20:41:23 2012 +0100 summary: Fix for issue 14725 for 3.2 branch files: Lib/multiprocessing/connection.py | 5 ++- Lib/test/test_multiprocessing.py | 17 ++++++++++ Modules/_multiprocessing/win32_functions.c | 1 + 3 files changed, 22 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -360,7 +360,10 @@ try: win32.ConnectNamedPipe(handle, win32.NULL) except WindowsError as e: - if e.args[0] != win32.ERROR_PIPE_CONNECTED: + # ERROR_NO_DATA can occur if a client has already connected, + # written data and then disconnected -- see Issue 14725. + if e.args[0] not in (win32.ERROR_PIPE_CONNECTED, + win32.ERROR_NO_DATA): raise return _multiprocessing.PipeConnection(handle) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -1732,6 +1732,23 @@ self.assertEqual(conn.recv(), 'hello') p.join() l.close() + + def test_issue14725(self): + l = self.connection.Listener() + p = self.Process(target=self._test, args=(l.address,)) + p.daemon = True + p.start() + time.sleep(1) + # On Windows the client process should by now have connected, + # written data and closed the pipe handle by now. This causes + # ConnectNamdedPipe() to fail with ERROR_NO_DATA. See Issue + # 14725. + conn = l.accept() + self.assertEqual(conn.recv(), 'hello') + conn.close() + p.join() + l.close() + # # Test of sending connection and socket objects between processes # diff --git a/Modules/_multiprocessing/win32_functions.c b/Modules/_multiprocessing/win32_functions.c --- a/Modules/_multiprocessing/win32_functions.c +++ b/Modules/_multiprocessing/win32_functions.c @@ -244,6 +244,7 @@ Py_INCREF(&Win32Type); WIN32_CONSTANT(F_DWORD, ERROR_ALREADY_EXISTS); + WIN32_CONSTANT(F_DWORD, ERROR_NO_DATA); WIN32_CONSTANT(F_DWORD, ERROR_PIPE_BUSY); WIN32_CONSTANT(F_DWORD, ERROR_PIPE_CONNECTED); WIN32_CONSTANT(F_DWORD, ERROR_SEM_TIMEOUT); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 5 21:59:39 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sat, 05 May 2012 21:59:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Dummy_merge?= Message-ID: http://hg.python.org/cpython/rev/2e7dd44533f4 changeset: 76774:2e7dd44533f4 parent: 76771:44f078ea05f3 parent: 76773:afab4d14d5e7 user: Richard Oudkerk date: Sat May 05 20:55:46 2012 +0100 summary: Dummy merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 5 22:38:18 2012 From: python-checkins at python.org (stefan.krah) Date: Sat, 05 May 2012 22:38:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo_in_exception_messa?= =?utf8?b?Z2Uu?= Message-ID: http://hg.python.org/cpython/rev/ff8fc2819f04 changeset: 76775:ff8fc2819f04 user: Stefan Krah date: Sat May 05 22:37:05 2012 +0200 summary: Fix typo in exception message. files: Modules/posixmodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3711,7 +3711,7 @@ if (times && (times != Py_None)) { if (!PyTuple_CheckExact(times) || (PyTuple_Size(times) != 2)) { PyErr_Format(PyExc_TypeError, - "%s: 'time' must be either" + "%s: 'times' must be either" " a tuple of two ints or None", ua->function_name); return_value = utime_times_conversion_failure; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 01:55:38 2012 From: python-checkins at python.org (larry.hastings) Date: Sun, 06 May 2012 01:55:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314705=3A_Add_=27p?= =?utf8?q?=27_format_character_to_PyArg=5FParseTuple*_for_bool_support=2E?= Message-ID: http://hg.python.org/cpython/rev/bc6d28e726d8 changeset: 76776:bc6d28e726d8 user: Larry Hastings date: Sat May 05 16:54:29 2012 -0700 summary: Issue #14705: Add 'p' format character to PyArg_ParseTuple* for bool support. files: Doc/c-api/arg.rst | 9 +++++++ Lib/test/test_getargs2.py | 31 +++++++++++++++++++++++++++ Modules/_testcapimodule.c | 10 ++++++++ Python/getargs.c | 12 ++++++++++ 4 files changed, 62 insertions(+), 0 deletions(-) diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst --- a/Doc/c-api/arg.rst +++ b/Doc/c-api/arg.rst @@ -317,6 +317,15 @@ .. versionchanged:: 3.1 ``Py_CLEANUP_SUPPORTED`` was added. +``p`` (:class:`bool`) [int] + Tests the value passed in for truth (a boolean **p**\redicate) and converts + the result to its equivalent C true/false integer value. + Sets the int to 1 if the expression was true and 0 if it was false. + This accepts any valid Python value. See :ref:`truth` for more + information about how Python tests values for truth. + + .. versionchanged:: 3.3 + ``(items)`` (:class:`tuple`) [*matching-items*] The object must be a Python sequence whose length is the number of format units in *items*. The C arguments must correspond to the individual format units in diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py --- a/Lib/test/test_getargs2.py +++ b/Lib/test/test_getargs2.py @@ -214,6 +214,36 @@ self.assertEqual(VERY_LARGE & ULLONG_MAX, getargs_K(VERY_LARGE)) +class Paradox: + "This statement is false." + def __bool__(self): + raise NotImplementedError + +class Boolean_TestCase(unittest.TestCase): + def test_p(self): + from _testcapi import getargs_p + self.assertEqual(0, getargs_p(False)) + self.assertEqual(0, getargs_p(None)) + self.assertEqual(0, getargs_p(0)) + self.assertEqual(0, getargs_p(0.0)) + self.assertEqual(0, getargs_p(0j)) + self.assertEqual(0, getargs_p('')) + self.assertEqual(0, getargs_p(())) + self.assertEqual(0, getargs_p([])) + self.assertEqual(0, getargs_p({})) + + self.assertEqual(1, getargs_p(True)) + self.assertEqual(1, getargs_p(1)) + self.assertEqual(1, getargs_p(1.0)) + self.assertEqual(1, getargs_p(1j)) + self.assertEqual(1, getargs_p('x')) + self.assertEqual(1, getargs_p((1,))) + self.assertEqual(1, getargs_p([1])) + self.assertEqual(1, getargs_p({1:2})) + self.assertEqual(1, getargs_p(unittest.TestCase)) + + self.assertRaises(NotImplementedError, getargs_p, Paradox()) + class Tuple_TestCase(unittest.TestCase): def test_tuple(self): @@ -510,6 +540,7 @@ tests = [ Signed_TestCase, Unsigned_TestCase, + Boolean_TestCase, Tuple_TestCase, Keywords_TestCase, KeywordOnly_TestCase, diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -916,6 +916,15 @@ return PyLong_FromSsize_t(value); } +static PyObject * +getargs_p(PyObject *self, PyObject *args) +{ + int value; + if (!PyArg_ParseTuple(args, "p", &value)) + return NULL; + return PyLong_FromLong(value); +} + #ifdef HAVE_LONG_LONG static PyObject * getargs_L(PyObject *self, PyObject *args) @@ -2439,6 +2448,7 @@ {"getargs_i", getargs_i, METH_VARARGS}, {"getargs_l", getargs_l, METH_VARARGS}, {"getargs_n", getargs_n, METH_VARARGS}, + {"getargs_p", getargs_p, METH_VARARGS}, #ifdef HAVE_LONG_LONG {"getargs_L", getargs_L, METH_VARARGS}, {"getargs_K", getargs_K, METH_VARARGS}, diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -814,6 +814,18 @@ break; } + case 'p': {/* boolean *p*redicate */ + int *p = va_arg(*p_va, int *); + int val = PyObject_IsTrue(arg); + if (val > 0) + *p = 1; + else if (val == 0) + *p = 0; + else + RETURN_ERR_OCCURRED; + break; + } + /* XXX WAAAAH! 's', 'y', 'z', 'u', 'Z', 'e', 'w' codes all need to be cleaned up! */ -- Repository URL: http://hg.python.org/cpython From benjamin at python.org Sun May 6 02:04:29 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 5 May 2012 20:04:29 -0400 Subject: [Python-checkins] cpython: Issue #14705: Add 'p' format character to PyArg_ParseTuple* for bool support. In-Reply-To: References: Message-ID: 2012/5/5 larry.hastings : > http://hg.python.org/cpython/rev/bc6d28e726d8 > changeset: ? 76776:bc6d28e726d8 > user: ? ? ? ?Larry Hastings > date: ? ? ? ?Sat May 05 16:54:29 2012 -0700 > summary: > ?Issue #14705: Add 'p' format character to PyArg_ParseTuple* for bool support. > > files: > ?Doc/c-api/arg.rst ? ? ? ? | ? 9 +++++++ > ?Lib/test/test_getargs2.py | ?31 +++++++++++++++++++++++++++ > ?Modules/_testcapimodule.c | ?10 ++++++++ > ?Python/getargs.c ? ? ? ? ?| ?12 ++++++++++ > ?4 files changed, 62 insertions(+), 0 deletions(-) You forgot Misc/NEWS. -- Regards, Benjamin From python-checkins at python.org Sun May 6 02:40:16 2012 From: python-checkins at python.org (larry.hastings) Date: Sun, 06 May 2012 02:40:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_Misc/NEWS_for_issues?= =?utf8?b?ICMxNDEyNyBhbmQgIzE0NzA1LiAgKEFuZCwgdGVjaG5pY2FsbHksICMxMDE0OC4p?= Message-ID: http://hg.python.org/cpython/rev/709850f1ec67 changeset: 76777:709850f1ec67 user: Larry Hastings date: Sat May 05 17:39:09 2012 -0700 summary: Update Misc/NEWS for issues #14127 and #14705. (And, technically, #10148.) files: Modules/posixmodule.c | 372 +++++++++++++++++++++++++++-- 1 files changed, 337 insertions(+), 35 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -398,6 +398,268 @@ #endif #endif +/* + * De-vararg'd PyArg_ParseTupleAndKeywords() + */ +typedef struct argument_path_s { + struct argument_path_s *next; + PyObject *object; + char *format; + wchar_t **wide; + char **narrow; + Py_ssize_t *length; +} argument_path_t; + +typedef struct { + PyObject *args; + PyObject *kwargs; + char format[32]; + char *format_trace; + char *keywords[8]; + int keyword_count; + PyObject **varargs[16]; + int vararg_count; + const char *function_name; + argument_path_t *path_head; +} argument_parser_table_t; + +#define APT_DECLARE(apt, fname) \ + argument_parser_table_t apt; \ + memset(&apt, 0, sizeof(apt)); \ + apt.function_name = fname; \ + apt.args = args; \ + apt.kwargs = kwargs; \ + apt.format_trace = apt.format + +#define _APT_ARGUMENT_OBJECT(apt, name, variable, format) \ + *apt.format_trace++ = (format); \ + apt.keywords[apt.keyword_count++] = (name); \ + apt.varargs[apt.vararg_count++] = (PyObject **)(variable); \ + +#define APT_ARGUMENT_OBJECT(apt, name, variable) \ + _APT_ARGUMENT_OBJECT(apt, name, variable, 'O') + +#define APT_ARGUMENT_OBJECT_WITH_CONVERTER(apt, name, variable, converter) \ + apt.varargs[apt.vararg_count++] = (PyObject **)converter; \ + APT_ARGUMENT_OBJECT(apt, name, variable); \ + *apt.format_trace++ = '&'; \ + +#define APT_ARGUMENT_UNICODE(apt, name, variable) \ + _APT_ARGUMENT_OBJECT(apt, name, variable, 'U') + +#define APT_ARGUMENT_BYTES(apt, name, variable) \ + _APT_ARGUMENT_OBJECT(apt, name, variable, 'y') + +#define APT_ARGUMENT_INT(apt, name, variable) \ + _APT_ARGUMENT_OBJECT(apt, name, variable, 'i') + +static int +bool_converter(PyObject *object, void *address) { + int *output = (int *)address; + int value = PyObject_IsTrue(object); + if (value == -1) + return 0; + *output = value; + return 1; +} + +#define APT_ARGUMENT_BOOL(apt, name, variable) \ + _APT_ARGUMENT_OBJECT(apt, name, variable, 'p') + +#if !MS_WINDOWS + #define _APT_DEFAULT_PATH_TYPE 'U' + #define _APT_PATH_BEFORE ; + #define _APT_PATH_AFTER ; +#else + #define _APT_DEFAULT_PATH_TYPE 'O' + #define _APT_PATH_BEFORE apt.varargs[apt.vararg_count++] = (PyObject **)PyUnicode_FSConverter; + #define _APT_PATH_AFTER *apt.format_trace++ = '&'; +#endif + +#define APT_ARGUMENT_PATH(apt, name, w, n, l) \ + { \ + argument_path_t *_path_ = (argument_path_t *)alloca(sizeof(argument_path_t)); \ + _APT_PATH_BEFORE; \ + memset(_path_, 0, sizeof(*_path_)); \ + _path_->wide = w; \ + _path_->narrow = n; \ + _path_->length = l; \ + _path_->next = apt.path_head; \ + apt.path_head = _path_; \ + _APT_ARGUMENT_OBJECT(apt, name, &_path_->object, _APT_DEFAULT_PATH_TYPE); \ + _APT_PATH_AFTER; \ + } \ + +#define APT_OPTIONAL(apt) \ + *apt.format_trace++ = '|' \ + +#define APT_KEYWORD_ONLY(apt) \ + *apt.format_trace++ = '$' \ + + +static int apt_parse(argument_parser_table_t *apt) { + argument_path_t *trace; + + *apt->format_trace++ = ':'; + strcpy(apt->format_trace, apt->function_name); + apt->keywords[apt->keyword_count] = NULL; + + for (;;) { + int result; + + switch (apt->vararg_count) { + case 0: + PyErr_SetString(PyExc_RuntimeError, "suspiciously too few arguments for apt_parse"); + return 0; + + #define APT_PREAMBLE \ + result = PyArg_ParseTupleAndKeywords(apt->args, apt->kwargs, apt->format, apt->keywords, \ + + #define APT_POSTSCRIPT \ + ); \ + break; \ + + case 1: + APT_PREAMBLE + apt->varargs[0] + APT_POSTSCRIPT + case 2: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1] + APT_POSTSCRIPT + case 3: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1], + apt->varargs[2] + APT_POSTSCRIPT + case 4: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1], + apt->varargs[2], + apt->varargs[3] + APT_POSTSCRIPT + case 5: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1], + apt->varargs[2], + apt->varargs[3], + apt->varargs[4] + APT_POSTSCRIPT + case 6: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1], + apt->varargs[2], + apt->varargs[3], + apt->varargs[4], + apt->varargs[5] + APT_POSTSCRIPT + case 7: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1], + apt->varargs[2], + apt->varargs[3], + apt->varargs[4], + apt->varargs[5], + apt->varargs[6] + APT_POSTSCRIPT + case 8: + APT_PREAMBLE + apt->varargs[0], + apt->varargs[1], + apt->varargs[2], + apt->varargs[3], + apt->varargs[4], + apt->varargs[5], + apt->varargs[6], + apt->varargs[7] + APT_POSTSCRIPT + default: + PyErr_SetString(PyExc_RuntimeError, "too many arguments for apt_parse"); + return 0; + } + + if (result) { + for (trace = apt->path_head; trace != NULL; trace = trace->next) { +#if MS_WINDOWS + switch (*trace->format) { + case 'U': + *trace->wide = PyUnicode_AsUnicode(trace->object); + if (trace->length) + *trace->length = PyUnicode_GET_SIZE(trace->object); + break; + case 'y' + if (win32_warn_bytes_api()) + return 0; + *trace->narrow = trace->object; + if (trace->length) + *trace->length = strlen((char *)trace->object); + break; + default: + PyErr_SetString(PyExc_RuntimeError, "unexpected format character in apt_parse"); + return 0; + } +#else + assert(*trace->format == 'O'); + *trace->narrow = PyBytes_AsString(trace->object); + if (trace->length) + *trace->length = PyBytes_GET_SIZE(trace->object); +#endif + } + return result; + } + +#if !MS_WINDOWS + break; +#else + for (trace = apt->path_head; trace != NULL; trace = trace->next) { + /* + * try flipping paths between wide and narrow. + * + * each element always started with wide. + * so: + * * if we see a wide, flip it to narrow and stop. + * * if we see a narrow, flip it to wide and move on to the next field. + * * if we run out of fields, we have exhausted all possibilities. fail. + * + * (conceptually it helps to think of the fields as a binary number + * with as many digits as there are entries in the path list. + * wide is 0, narrow is 1. we keep incrementing the number + * until we wrap around to 0 again.) + */ + if (*trace->format == 'U') { + *trace->format = 'y'; + break; + } + if (*trace->format == 'y') { + *trace->format = 'U'; + continue; + } + PyErr_SetString(PyExc_RuntimeError, "unexpected format character in apt_parse"); + return 0; + } + if (!trace) + break; +#endif + } + + return 0; +} + +static void apt_cleanup(argument_parser_table_t *apt) { +#if !MS_WINDOWS + argument_path_t *trace; + for (trace = apt->path_head; trace != NULL; trace = trace->next) { + Py_XDECREF(trace->object); + } +#endif +} + /* A helper used by a number of POSIX-only functions */ #ifndef MS_WINDOWS static int @@ -1942,6 +2204,12 @@ /* POSIX methods */ +#ifdef AT_FDCWD +#define DEFAULT_DIR_FD AT_FDCWD +#else +#define DEFAULT_DIR_FD (-100) +#endif + PyDoc_STRVAR(posix_access__doc__, "access(path, mode) -> True if granted, False otherwise\n\n\ Use the real uid/gid to test for access to a path. Note that most\n\ @@ -1951,34 +2219,54 @@ existence, or the inclusive-OR of R_OK, W_OK, and X_OK."); static PyObject * -posix_access(PyObject *self, PyObject *args) -{ - const char *path; +posix_access(PyObject *self, PyObject *args, PyObject *kwargs) +{ int mode; + int dir_fd = DEFAULT_DIR_FD; + wchar_t *wide_path; + char *path; + int follow_symlinks = 1; + int effective_ids = 0; #ifdef MS_WINDOWS DWORD attr; - PyObject *po; - if (PyArg_ParseTuple(args, "Ui:access", &po, &mode)) { - wchar_t* wpath = PyUnicode_AsUnicode(po); - if (wpath == NULL) - return NULL; - Py_BEGIN_ALLOW_THREADS - attr = GetFileAttributesW(wpath); - Py_END_ALLOW_THREADS - goto finish; - } - /* Drop the argument parsing error as narrow strings - are also valid. */ - PyErr_Clear(); - if (!PyArg_ParseTuple(args, "yi:access", &path, &mode)) - return NULL; - if (win32_warn_bytes_api()) - return NULL; - Py_BEGIN_ALLOW_THREADS - attr = GetFileAttributesA(path); - Py_END_ALLOW_THREADS -finish: +#else + int res; +#endif + + APT_DECLARE(apt, "access"); + APT_ARGUMENT_PATH(apt, "path", &wide_path, &path, NULL); + APT_ARGUMENT_INT(apt, "mode", &mode); + APT_OPTIONAL(apt); + APT_KEYWORD_ONLY(apt); + APT_ARGUMENT_INT(apt, "dir_fd", &dir_fd); + APT_ARGUMENT_BOOL(apt, "follow_symlinks", &follow_symlinks); + APT_ARGUMENT_BOOL(apt, "effective_ids", &effective_ids); + +#define ACCESS_FAIL_IF_KEYWORD_USED \ + if (dir_fd != DEFAULT_DIR_FD) \ + PyErr_SetString(PyExc_NotImplementedError, "access: dir_fd unavailable on this platform"); \ + if (!follow_symlinks) \ + PyErr_SetString(PyExc_NotImplementedError, "access: follow_symlinks unavailable on this platform"); \ + if (effective_ids) \ + PyErr_SetString(PyExc_NotImplementedError, "access: effective_ids unavailable on this platform") + + if (!apt_parse(&apt)) + return NULL; + Py_RETURN_NONE; + +#ifdef MS_WINDOWS + ACCESS_FAIL_IF_KEYWORD_USED; + + Py_BEGIN_ALLOW_THREADS + if (wide != NULL) + attr = GetFileAttributesW(wide_path); + else + attr = GetFileAttributesA(path); + Py_END_ALLOW_THREADS + + apt_cleanup(&apt); + if (attr == 0xFFFFFFFF) /* File does not exist, or cannot read attributes */ return PyBool_FromLong(0); @@ -1989,16 +2277,28 @@ || !(attr & FILE_ATTRIBUTE_READONLY) || (attr & FILE_ATTRIBUTE_DIRECTORY)); #else - PyObject *opath; - int res; - if (!PyArg_ParseTuple(args, "O&i:access", - PyUnicode_FSConverter, &opath, &mode)) - return NULL; - path = PyBytes_AsString(opath); - Py_BEGIN_ALLOW_THREADS - res = access(path, mode); - Py_END_ALLOW_THREADS - Py_DECREF(opath); + + if ((dir_fd != DEFAULT_DIR_FD) + || effective_ids || !follow_symlinks) { +#ifdef HAVE_FACCESSAT + int flags = 0; + if (!follow_symlinks) + flags |= AT_SYMLINK_NOFOLLOW; + if (effective_ids) + flags |= AT_EACCESS; + Py_BEGIN_ALLOW_THREADS + res = faccessat(dir_fd, path, mode, flags); + Py_END_ALLOW_THREADS +#else + ACCESS_FAIL_IF_KEYWORD_USED; +#endif + } else { + Py_BEGIN_ALLOW_THREADS + res = access(path, mode); + Py_END_ALLOW_THREADS + } + + apt_cleanup(&apt); return PyBool_FromLong(res == 0); #endif } @@ -10596,7 +10896,9 @@ static PyMethodDef posix_methods[] = { - {"access", posix_access, METH_VARARGS, posix_access__doc__}, + {"access", (PyCFunction)posix_access, + METH_VARARGS | METH_KEYWORDS, + posix_access__doc__}, #ifdef HAVE_TTYNAME {"ttyname", posix_ttyname, METH_VARARGS, posix_ttyname__doc__}, #endif -- Repository URL: http://hg.python.org/cpython From benjamin at python.org Sun May 6 03:11:07 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 5 May 2012 21:11:07 -0400 Subject: [Python-checkins] cpython: Update Misc/NEWS for issues #14127 and #14705. (And, technically, #10148.) In-Reply-To: References: Message-ID: 2012/5/5 larry.hastings : > http://hg.python.org/cpython/rev/709850f1ec67 > changeset: ? 76777:709850f1ec67 > user: ? ? ? ?Larry Hastings > date: ? ? ? ?Sat May 05 17:39:09 2012 -0700 > summary: > ?Update Misc/NEWS for issues #14127 and #14705. ?(And, technically, #10148.) > > files: > ?Modules/posixmodule.c | ?372 +++++++++++++++++++++++++++-- Um? -- Regards, Benjamin From python-checkins at python.org Sun May 6 03:22:20 2012 From: python-checkins at python.org (benjamin.peterson) Date: Sun, 06 May 2012 03:22:20 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Backed_out_changeset_709850?= =?utf8?q?f1ec67?= Message-ID: http://hg.python.org/cpython/rev/2129e5e2db5c changeset: 76778:2129e5e2db5c user: Benjamin Peterson date: Sat May 05 21:22:14 2012 -0400 summary: Backed out changeset 709850f1ec67 files: Modules/posixmodule.c | 368 ++--------------------------- 1 files changed, 33 insertions(+), 335 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -398,268 +398,6 @@ #endif #endif -/* - * De-vararg'd PyArg_ParseTupleAndKeywords() - */ -typedef struct argument_path_s { - struct argument_path_s *next; - PyObject *object; - char *format; - wchar_t **wide; - char **narrow; - Py_ssize_t *length; -} argument_path_t; - -typedef struct { - PyObject *args; - PyObject *kwargs; - char format[32]; - char *format_trace; - char *keywords[8]; - int keyword_count; - PyObject **varargs[16]; - int vararg_count; - const char *function_name; - argument_path_t *path_head; -} argument_parser_table_t; - -#define APT_DECLARE(apt, fname) \ - argument_parser_table_t apt; \ - memset(&apt, 0, sizeof(apt)); \ - apt.function_name = fname; \ - apt.args = args; \ - apt.kwargs = kwargs; \ - apt.format_trace = apt.format - -#define _APT_ARGUMENT_OBJECT(apt, name, variable, format) \ - *apt.format_trace++ = (format); \ - apt.keywords[apt.keyword_count++] = (name); \ - apt.varargs[apt.vararg_count++] = (PyObject **)(variable); \ - -#define APT_ARGUMENT_OBJECT(apt, name, variable) \ - _APT_ARGUMENT_OBJECT(apt, name, variable, 'O') - -#define APT_ARGUMENT_OBJECT_WITH_CONVERTER(apt, name, variable, converter) \ - apt.varargs[apt.vararg_count++] = (PyObject **)converter; \ - APT_ARGUMENT_OBJECT(apt, name, variable); \ - *apt.format_trace++ = '&'; \ - -#define APT_ARGUMENT_UNICODE(apt, name, variable) \ - _APT_ARGUMENT_OBJECT(apt, name, variable, 'U') - -#define APT_ARGUMENT_BYTES(apt, name, variable) \ - _APT_ARGUMENT_OBJECT(apt, name, variable, 'y') - -#define APT_ARGUMENT_INT(apt, name, variable) \ - _APT_ARGUMENT_OBJECT(apt, name, variable, 'i') - -static int -bool_converter(PyObject *object, void *address) { - int *output = (int *)address; - int value = PyObject_IsTrue(object); - if (value == -1) - return 0; - *output = value; - return 1; -} - -#define APT_ARGUMENT_BOOL(apt, name, variable) \ - _APT_ARGUMENT_OBJECT(apt, name, variable, 'p') - -#if !MS_WINDOWS - #define _APT_DEFAULT_PATH_TYPE 'U' - #define _APT_PATH_BEFORE ; - #define _APT_PATH_AFTER ; -#else - #define _APT_DEFAULT_PATH_TYPE 'O' - #define _APT_PATH_BEFORE apt.varargs[apt.vararg_count++] = (PyObject **)PyUnicode_FSConverter; - #define _APT_PATH_AFTER *apt.format_trace++ = '&'; -#endif - -#define APT_ARGUMENT_PATH(apt, name, w, n, l) \ - { \ - argument_path_t *_path_ = (argument_path_t *)alloca(sizeof(argument_path_t)); \ - _APT_PATH_BEFORE; \ - memset(_path_, 0, sizeof(*_path_)); \ - _path_->wide = w; \ - _path_->narrow = n; \ - _path_->length = l; \ - _path_->next = apt.path_head; \ - apt.path_head = _path_; \ - _APT_ARGUMENT_OBJECT(apt, name, &_path_->object, _APT_DEFAULT_PATH_TYPE); \ - _APT_PATH_AFTER; \ - } \ - -#define APT_OPTIONAL(apt) \ - *apt.format_trace++ = '|' \ - -#define APT_KEYWORD_ONLY(apt) \ - *apt.format_trace++ = '$' \ - - -static int apt_parse(argument_parser_table_t *apt) { - argument_path_t *trace; - - *apt->format_trace++ = ':'; - strcpy(apt->format_trace, apt->function_name); - apt->keywords[apt->keyword_count] = NULL; - - for (;;) { - int result; - - switch (apt->vararg_count) { - case 0: - PyErr_SetString(PyExc_RuntimeError, "suspiciously too few arguments for apt_parse"); - return 0; - - #define APT_PREAMBLE \ - result = PyArg_ParseTupleAndKeywords(apt->args, apt->kwargs, apt->format, apt->keywords, \ - - #define APT_POSTSCRIPT \ - ); \ - break; \ - - case 1: - APT_PREAMBLE - apt->varargs[0] - APT_POSTSCRIPT - case 2: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1] - APT_POSTSCRIPT - case 3: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1], - apt->varargs[2] - APT_POSTSCRIPT - case 4: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1], - apt->varargs[2], - apt->varargs[3] - APT_POSTSCRIPT - case 5: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1], - apt->varargs[2], - apt->varargs[3], - apt->varargs[4] - APT_POSTSCRIPT - case 6: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1], - apt->varargs[2], - apt->varargs[3], - apt->varargs[4], - apt->varargs[5] - APT_POSTSCRIPT - case 7: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1], - apt->varargs[2], - apt->varargs[3], - apt->varargs[4], - apt->varargs[5], - apt->varargs[6] - APT_POSTSCRIPT - case 8: - APT_PREAMBLE - apt->varargs[0], - apt->varargs[1], - apt->varargs[2], - apt->varargs[3], - apt->varargs[4], - apt->varargs[5], - apt->varargs[6], - apt->varargs[7] - APT_POSTSCRIPT - default: - PyErr_SetString(PyExc_RuntimeError, "too many arguments for apt_parse"); - return 0; - } - - if (result) { - for (trace = apt->path_head; trace != NULL; trace = trace->next) { -#if MS_WINDOWS - switch (*trace->format) { - case 'U': - *trace->wide = PyUnicode_AsUnicode(trace->object); - if (trace->length) - *trace->length = PyUnicode_GET_SIZE(trace->object); - break; - case 'y' - if (win32_warn_bytes_api()) - return 0; - *trace->narrow = trace->object; - if (trace->length) - *trace->length = strlen((char *)trace->object); - break; - default: - PyErr_SetString(PyExc_RuntimeError, "unexpected format character in apt_parse"); - return 0; - } -#else - assert(*trace->format == 'O'); - *trace->narrow = PyBytes_AsString(trace->object); - if (trace->length) - *trace->length = PyBytes_GET_SIZE(trace->object); -#endif - } - return result; - } - -#if !MS_WINDOWS - break; -#else - for (trace = apt->path_head; trace != NULL; trace = trace->next) { - /* - * try flipping paths between wide and narrow. - * - * each element always started with wide. - * so: - * * if we see a wide, flip it to narrow and stop. - * * if we see a narrow, flip it to wide and move on to the next field. - * * if we run out of fields, we have exhausted all possibilities. fail. - * - * (conceptually it helps to think of the fields as a binary number - * with as many digits as there are entries in the path list. - * wide is 0, narrow is 1. we keep incrementing the number - * until we wrap around to 0 again.) - */ - if (*trace->format == 'U') { - *trace->format = 'y'; - break; - } - if (*trace->format == 'y') { - *trace->format = 'U'; - continue; - } - PyErr_SetString(PyExc_RuntimeError, "unexpected format character in apt_parse"); - return 0; - } - if (!trace) - break; -#endif - } - - return 0; -} - -static void apt_cleanup(argument_parser_table_t *apt) { -#if !MS_WINDOWS - argument_path_t *trace; - for (trace = apt->path_head; trace != NULL; trace = trace->next) { - Py_XDECREF(trace->object); - } -#endif -} - /* A helper used by a number of POSIX-only functions */ #ifndef MS_WINDOWS static int @@ -2204,12 +1942,6 @@ /* POSIX methods */ -#ifdef AT_FDCWD -#define DEFAULT_DIR_FD AT_FDCWD -#else -#define DEFAULT_DIR_FD (-100) -#endif - PyDoc_STRVAR(posix_access__doc__, "access(path, mode) -> True if granted, False otherwise\n\n\ Use the real uid/gid to test for access to a path. Note that most\n\ @@ -2219,54 +1951,34 @@ existence, or the inclusive-OR of R_OK, W_OK, and X_OK."); static PyObject * -posix_access(PyObject *self, PyObject *args, PyObject *kwargs) -{ +posix_access(PyObject *self, PyObject *args) +{ + const char *path; int mode; - int dir_fd = DEFAULT_DIR_FD; - wchar_t *wide_path; - char *path; - int follow_symlinks = 1; - int effective_ids = 0; #ifdef MS_WINDOWS DWORD attr; -#else - int res; -#endif - - APT_DECLARE(apt, "access"); - APT_ARGUMENT_PATH(apt, "path", &wide_path, &path, NULL); - APT_ARGUMENT_INT(apt, "mode", &mode); - APT_OPTIONAL(apt); - APT_KEYWORD_ONLY(apt); - APT_ARGUMENT_INT(apt, "dir_fd", &dir_fd); - APT_ARGUMENT_BOOL(apt, "follow_symlinks", &follow_symlinks); - APT_ARGUMENT_BOOL(apt, "effective_ids", &effective_ids); - -#define ACCESS_FAIL_IF_KEYWORD_USED \ - if (dir_fd != DEFAULT_DIR_FD) \ - PyErr_SetString(PyExc_NotImplementedError, "access: dir_fd unavailable on this platform"); \ - if (!follow_symlinks) \ - PyErr_SetString(PyExc_NotImplementedError, "access: follow_symlinks unavailable on this platform"); \ - if (effective_ids) \ - PyErr_SetString(PyExc_NotImplementedError, "access: effective_ids unavailable on this platform") - - if (!apt_parse(&apt)) - return NULL; - Py_RETURN_NONE; - -#ifdef MS_WINDOWS - ACCESS_FAIL_IF_KEYWORD_USED; - + PyObject *po; + if (PyArg_ParseTuple(args, "Ui:access", &po, &mode)) { + wchar_t* wpath = PyUnicode_AsUnicode(po); + if (wpath == NULL) + return NULL; + Py_BEGIN_ALLOW_THREADS + attr = GetFileAttributesW(wpath); + Py_END_ALLOW_THREADS + goto finish; + } + /* Drop the argument parsing error as narrow strings + are also valid. */ + PyErr_Clear(); + if (!PyArg_ParseTuple(args, "yi:access", &path, &mode)) + return NULL; + if (win32_warn_bytes_api()) + return NULL; Py_BEGIN_ALLOW_THREADS - if (wide != NULL) - attr = GetFileAttributesW(wide_path); - else - attr = GetFileAttributesA(path); + attr = GetFileAttributesA(path); Py_END_ALLOW_THREADS - - apt_cleanup(&apt); - +finish: if (attr == 0xFFFFFFFF) /* File does not exist, or cannot read attributes */ return PyBool_FromLong(0); @@ -2277,28 +1989,16 @@ || !(attr & FILE_ATTRIBUTE_READONLY) || (attr & FILE_ATTRIBUTE_DIRECTORY)); #else - - if ((dir_fd != DEFAULT_DIR_FD) - || effective_ids || !follow_symlinks) { -#ifdef HAVE_FACCESSAT - int flags = 0; - if (!follow_symlinks) - flags |= AT_SYMLINK_NOFOLLOW; - if (effective_ids) - flags |= AT_EACCESS; - Py_BEGIN_ALLOW_THREADS - res = faccessat(dir_fd, path, mode, flags); - Py_END_ALLOW_THREADS -#else - ACCESS_FAIL_IF_KEYWORD_USED; -#endif - } else { - Py_BEGIN_ALLOW_THREADS - res = access(path, mode); - Py_END_ALLOW_THREADS - } - - apt_cleanup(&apt); + PyObject *opath; + int res; + if (!PyArg_ParseTuple(args, "O&i:access", + PyUnicode_FSConverter, &opath, &mode)) + return NULL; + path = PyBytes_AsString(opath); + Py_BEGIN_ALLOW_THREADS + res = access(path, mode); + Py_END_ALLOW_THREADS + Py_DECREF(opath); return PyBool_FromLong(res == 0); #endif } @@ -10896,9 +10596,7 @@ static PyMethodDef posix_methods[] = { - {"access", (PyCFunction)posix_access, - METH_VARARGS | METH_KEYWORDS, - posix_access__doc__}, + {"access", posix_access, METH_VARARGS, posix_access__doc__}, #ifdef HAVE_TTYNAME {"ttyname", posix_ttyname, METH_VARARGS, posix_ttyname__doc__}, #endif -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun May 6 05:38:04 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 06 May 2012 05:38:04 +0200 Subject: [Python-checkins] Daily reference leaks (2129e5e2db5c): sum=2 Message-ID: results for 2129e5e2db5c on branch "default" -------------------------------------------- test_site leaked [2, 0, 0] references, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog6sB8Qm', '-x'] From python-checkins at python.org Sun May 6 06:58:38 2012 From: python-checkins at python.org (larry.hastings) Date: Sun, 06 May 2012 06:58:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_Misc/NEWS_for_issues?= =?utf8?b?ICMxNDEyNyBhbmQgIzE0NzA1LiAgKEFuZCwgdGVjaG5pY2FsbHksICMxMDE0OC4p?= Message-ID: http://hg.python.org/cpython/rev/05274ab06182 changeset: 76779:05274ab06182 user: Larry Hastings date: Sat May 05 21:57:17 2012 -0700 summary: Update Misc/NEWS for issues #14127 and #14705. (And, technically, #10148.) files: Misc/NEWS | 24 ++++++++++++++++++++++++ 1 files changed, 24 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,30 @@ Python News +++++++++++ +What's New in Python 3.3.0 Alpha 4? +=================================== + +*Release date: 26-May-2012* + +Core and Builtins +----------------- + +- Issue #14705: The PyArg_Parse() family of functions now support the 'p' + format unit, which accepts a "boolean predicate" argument. It converts + any Python value into an integer--0 if it is "false", and 1 otherwise. + +Library +------- + +- Issue #14127: The os.stat() result object now provides three additional + fields: st_ctime_ns, st_mtime_ns, and st_atime_ns, providing those times + as an integer with nanosecond resolution. The functions os.utime(), + os.lutimes(), and os.futimes() now accept a new parameter, ns, which + accepts mtime and atime as integers with nanosecond resolution. + +- Issue #14127 and #10148: shutil.copystat now preserves exact mtime + and atime on filesystems providing nanosecond resolution. + What's New in Python 3.3.0 Alpha 3? =================================== -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Sun May 6 07:08:52 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 6 May 2012 15:08:52 +1000 Subject: [Python-checkins] peps: Update PEP 1 to better reflect current practice In-Reply-To: <20120505125654.77d3b9ab@resist.wooz.org> References: <20120505125654.77d3b9ab@resist.wooz.org> Message-ID: On Sun, May 6, 2012 at 2:56 AM, Barry Warsaw wrote: > Thanks for doing this update Nick. ?I have just a few comments. > > On May 05, 2012, at 02:57 PM, nick.coghlan wrote: > >>+Developers with commit privileges for the `PEP repository`_ may claim >>+PEP numbers directly by creating and committing a new PEP. When doing so, >>+the developer must handle the tasks that would normally be taken care of by >>+the PEP editors (see `PEP Editor Responsibilities & Workflow`_). > > While I certainly don't mind (in fact, prefer) those with commit privileges to > just go ahead and commit their PEP to the repo, I'd like for there to be > *some* communication with the PEP editors first. ?E.g. sanity checks on the > basic format or idea (was this discussed on python-ideas first?), or > reservation of PEP numbers. > > When you do contact the PEP editors, please also specify whether you have > commit privileges or not. ?It's too hard to remember or know who has those > rights, and too much hassle to look them up. ;) Good point, especially for committers that haven't done much PEP editing in the past. > OTOH, I'm also happy to adopt an EAFP style rather than LBYL, so that the PEP > editors can re-assign numbers or whatever after the fact. ?We've done this in > a few cases, and it's never been that much of a problem. > > Still, core developers needn't block (for too long) on the PEP editors. I'll see if I can figure out something - I may just put in text like "if you're at all unsure about what needs to be done, email the PEP editors anyway". >>+The final authority for PEP approval is the BDFL. However, whenever a new >>+PEP is put forward, any core developer that believes they are suitably >>+experienced to make the final decision on that PEP may offer to serve as >>+the "PEP czar" for that PEP. If their self-nomination is accepted by the >>+other core developers and the BDFL, then they will have the authority to >>+approve (or reject) that PEP. This process happens most frequently with PEPs >>+where the BDFL has granted in principle approval for *something* to be done, >>+but there are details that need to be worked out before the PEP can be >>+accepted. > > I'd reword this to something like the following: > > ? ?The final authority for the PEP approval is the BDFL. ?However, the BDFL > ? ?may delegate the final approval authority to a "PEP czar" for that PEP. > ? ?This happens most frequently with PEPs where the BDFL has granted approval > ? ?in principle for *something* to be done, and in agreement with the general > ? ?proposals of the PEP, but there are details that need to be worked out > ? ?before the final PEP can be approved. ?When an `PEP-Czar` header must be > ? ?added to the PEP to record this delegation. ?The format of this header is > ? ?the same as the `Author` header. > > This leave out the whole self-nomination text, which I think isn't very > relevant to the official addition of the czar role (sadly, no clever bacronym > has come to mind, and BDFOP hasn't really taken off ;). Including the self-nomination wording was deliberate - it summarises the gist of an off-list conversation between Victor, Guido and myself a while back. At the time, I thought the delegation had to come directly from Guido, but it turned out Guido was happy for people to volunteer for the role (or for PEP authors to suggest someone, which pretty much amounts to the same thing), with the acceptance of nominations covered by the same "rough consensus" rules as checkins (i.e. silence is taken as assent). That way Guido only has to get involved if he is personally interested, or none of the rest of us feel entitled to make the call. Since the way the czar gets appointed is important, I figured it was worth including. (The conversation was a while ago though, so hopefully Guido will chime in if I'm mischaracterising what he wrote at the time) Agreed we should have a new header field to record the BDFL delegate, but I think I'll go with BDFL-Delegate rather than PEP-Czar. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python-checkins at python.org Sun May 6 07:36:48 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 06 May 2012 07:36:48 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Address_Barry=27s_comments_and?= =?utf8?q?_make_a_couple_of_other_cleanups?= Message-ID: http://hg.python.org/peps/rev/0f66c13456af changeset: 4357:0f66c13456af user: Nick Coghlan date: Sun May 06 15:36:37 2012 +1000 summary: Address Barry's comments and make a couple of other cleanups files: pep-0001.txt | 72 ++++++++++++++++++++++++++++++--------- 1 files changed, 55 insertions(+), 17 deletions(-) diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -19,7 +19,7 @@ should provide a concise technical specification of the feature and a rationale for the feature. -We intend PEPs to be the primary mechanisms for proposing new +We intend PEPs to be the primary mechanisms for proposing major new features, for collecting community input on an issue, and for documenting the design decisions that have gone into Python. The PEP author is responsible for building consensus within the community and @@ -60,6 +60,16 @@ PEP Work Flow ============= + +Python's BDFL +------------- + +There are several reference in this PEP to the "BDFL". This acronym stands +for "Benevolent Dictator for Life" and refers to Guido van Rossum, the +original creator of, and the final design authority for, the Python +programming language. + + Submitting a PEP ---------------- @@ -108,24 +118,29 @@ sent back without further regard until proper formatting rules are followed. -If the PEP editor approves, he will assign the PEP a number, label it +If the PEP editor approves, they will assign the PEP a number, label it as Standards Track, Informational, or Process, give it status "Draft", and create and check-in the initial draft of the PEP. The PEP editor will not unreasonably deny a PEP. Reasons for denying PEP status include duplication of effort, being technically unsound, not providing proper motivation or addressing backwards compatibility, or -not in keeping with the Python philosophy. The BDFL (Benevolent -Dictator for Life, Guido van Rossum) can be consulted during the -approval phase, and is the final arbiter of the draft's PEP-ability. +not in keeping with the Python philosophy. The BDFL can be consulted +during the approval phase, and is the final arbiter of the draft's +PEP-ability. -Developers with commit privileges for the `PEP repository`_ may claim +Developers with hg push privileges for the `PEP repository`_ may claim PEP numbers directly by creating and committing a new PEP. When doing so, the developer must handle the tasks that would normally be taken care of by -the PEP editors (see `PEP Editor Responsibilities & Workflow`_). +the PEP editors (see `PEP Editor Responsibilities & Workflow`_). This +includes ensuring the initial version meets the expected standards for +submitting a PEP. Alternately, even developers may choose to submit PEPs +through the PEP editors. When doing so, let the PEP editors know you have +hg push privileges and they can guide you through the process of updating +the PEP repository directly. As updates are necessary, the PEP author can check in new versions if they have hg push privileges, or can email new PEP versions to -the PEP editor for committing. +the PEP editors for publication. Standards Track PEPs consist of two parts, a design document and a reference implementation. The PEP should be reviewed and accepted @@ -142,10 +157,11 @@ private comments in the early design phases, setting up a wiki page, etc. PEP authors should use their discretion here. + PEP Review & Resolution ----------------------- -Once the authors have completed a PEP, they must inform the PEP editor +Once the authors have completed a PEP, they must inform the PEP editors that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to the author(s) for revision. For a PEP that is pre-determined to be @@ -157,12 +173,16 @@ The final authority for PEP approval is the BDFL. However, whenever a new PEP is put forward, any core developer that believes they are suitably experienced to make the final decision on that PEP may offer to serve as -the "PEP czar" for that PEP. If their self-nomination is accepted by the -other core developers and the BDFL, then they will have the authority to -approve (or reject) that PEP. This process happens most frequently with PEPs -where the BDFL has granted in principle approval for *something* to be done, -but there are details that need to be worked out before the PEP can be -accepted. +the BDFL's delegate (or "PEP czar") for that PEP. If their self-nomination +is accepted by the other core developers and the BDFL, then they will have +the authority to approve (or reject) that PEP. This process happens most +frequently with PEPs where the BDFL has granted in principle approval for +*something* to be done, but there are details that need to be worked out +before the PEP can be accepted. + +If the final decision on a PEP is to be made by a delegate rather than +directly by the BDFL, this will be recorded by including the +"BDFL-Delegate" header in the PEP. For a PEP to be accepted it must meet certain minimum criteria. It must be a clear and complete description of the proposed enhancement. @@ -206,6 +226,20 @@ if they are never meant to be completed. E.g. PEP 1 (this PEP). +PEP Maintenance +--------------- + +In general, Standards track PEPs are no longer modified after they have +reached the Final state. Once a PEP has been completed, the Language and +Standard Library References become the formal documentation of the expected +behaviour. + +Informational and Process PEPs may be updated over time to reflect changes +to development practices and other details. The precise process followed in +these cases will depend on the nature and purpose of the PEP being updated. + + + What belongs in a successful PEP? ================================= @@ -307,6 +341,7 @@ Post-History: * Replaces: * Superseded-By: + * BDFL-Delegate: * Resolution: The Author header lists the names, and optionally the email addresses @@ -329,6 +364,9 @@ email addresses in PEPs will be obscured as a defense against spam harvesters. +The BDFL-Delegate field is used to record cases where the final decision to +approve or reject a PEP rests with someone other than the BDFL. + *Note: The Resolution header is required for Standards Track PEPs only. It contains a URL that should point to an email message or other web resource where the pronouncement about the PEP is made.* @@ -337,8 +375,8 @@ Draft phase), a Discussions-To header will indicate the mailing list or URL where the PEP is being discussed. No Discussions-To header is necessary if the PEP is being discussed privately with the author, or -on the python-list or python-dev email mailing lists. Note that email -addresses in the Discussions-To header will not be obscured. +on the python-list, python-ideas or python-dev email mailing lists. Note +that email addresses in the Discussions-To header will not be obscured. The Type header specifies the type of PEP: Standards Track, Informational, or Process. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun May 6 07:55:58 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 06 May 2012 07:55:58 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Don=27t_use_a_role_that_doesn?= =?utf8?q?=27t_exist_for_PEPs?= Message-ID: http://hg.python.org/peps/rev/2b346457cea7 changeset: 4358:2b346457cea7 user: Nick Coghlan date: Sun May 06 15:55:36 2012 +1000 summary: Don't use a role that doesn't exist for PEPs files: pep-0419.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0419.txt b/pep-0419.txt --- a/pep-0419.txt +++ b/pep-0419.txt @@ -310,7 +310,7 @@ explicitly. An example of such code might be a file-based lock implementation. -:func:`signal.pthread_sigmask` can be used to block signals inside +``signal.pthread_sigmask`` can be used to block signals inside cleanup handlers which can be interrupted with ``EINTR``. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun May 6 08:34:01 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 06 May 2012 08:34:01 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_the_BDFL-Delegate_field=2C?= =?utf8?q?_using_myself_as_the_test_case=2E?= Message-ID: http://hg.python.org/peps/rev/39930299aee0 changeset: 4359:39930299aee0 user: Nick Coghlan date: Sun May 06 16:31:52 2012 +1000 summary: Add the BDFL-Delegate field, using myself as the test case. Including the delegate's email address would be nice, but the PEP writer lives in docutils upstream rather than our PEPs repo and I'm not updating docutils just to mask an additional field, nor am I inclined to figure out how to move the writer definition downstream where it belongs files: pep-0001.txt | 6 ++++-- pep-0405.txt | 1 + pep-0415.txt | 2 ++ pep-3144.txt | 1 + pep0/pep.py | 1 + 5 files changed, 9 insertions(+), 2 deletions(-) diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -330,6 +330,7 @@ Version: Last-Modified: Author: + * BDFL-Delegate: * Discussions-To: Status: @@ -341,7 +342,6 @@ Post-History: * Replaces: * Superseded-By: - * BDFL-Delegate: * Resolution: The Author header lists the names, and optionally the email addresses @@ -365,7 +365,9 @@ harvesters. The BDFL-Delegate field is used to record cases where the final decision to -approve or reject a PEP rests with someone other than the BDFL. +approve or reject a PEP rests with someone other than the BDFL. (The +delegate's email address is currently omitted due to a limitation in the +email address masking for reStructuredText PEPs) *Note: The Resolution header is required for Standards Track PEPs only. It contains a URL that should point to an email message or diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -3,6 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Carl Meyer +BDFL-Delegate: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst diff --git a/pep-0415.txt b/pep-0415.txt --- a/pep-0415.txt +++ b/pep-0415.txt @@ -3,10 +3,12 @@ Version: $Revision$ Last-Modified: $Date$ Author: Benjamin Peterson +BDFL-Delegate: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 26-Feb-2012 +Python-Version: 3.3 Post-History: 26-Feb-2012 diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -3,6 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Peter Moody +BDFL-Delegate: Nick Coghlan Discussions-To: Status: Draft Type: Standards Track diff --git a/pep0/pep.py b/pep0/pep.py --- a/pep0/pep.py +++ b/pep0/pep.py @@ -156,6 +156,7 @@ # required or not. headers = (('PEP', True), ('Title', True), ('Version', True), ('Last-Modified', True), ('Author', True), + ('BDFL-Delegate', False), ('Discussions-To', False), ('Status', True), ('Type', True), ('Content-Type', False), ('Requires', False), ('Created', True), ('Python-Version', False), -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun May 6 08:35:23 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 06 May 2012 08:35:23 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Record_BDFL_delegation_and_res?= =?utf8?q?olution_for_PEP_3151?= Message-ID: http://hg.python.org/peps/rev/7de560665b2c changeset: 4360:7de560665b2c user: Nick Coghlan date: Sun May 06 16:35:13 2012 +1000 summary: Record BDFL delegation and resolution for PEP 3151 files: pep-3151.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-3151.txt b/pep-3151.txt --- a/pep-3151.txt +++ b/pep-3151.txt @@ -3,13 +3,14 @@ Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou +BDFL-Delegate: Barry Warsaw Status: Final Type: Standards Track Content-Type: text/x-rst Created: 2010-07-21 Python-Version: 3.3 Post-History: - +Resolution: http://mail.python.org/pipermail/python-dev/2011-October/114033.html Abstract ======== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun May 6 11:17:44 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 11:17:44 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_421=3A_edits_from_Eric=2E?= Message-ID: http://hg.python.org/peps/rev/afe29321dca3 changeset: 4361:afe29321dca3 user: Georg Brandl date: Sun May 06 11:17:25 2012 +0200 summary: PEP 421: edits from Eric. files: pep-0421.txt | 336 ++++++++++++++++++++++++-------------- 1 files changed, 211 insertions(+), 125 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -50,82 +50,61 @@ Proposal ======== -We will add a new variable to the ``sys`` module, called -``sys.implementation``, as a mapping to contain +We will add a new attribute to the ``sys`` module, called +``sys.implementation``, as an instance of a new type to contain implementation-specific information. -The contents of this mapping will remain fixed during interpreter +The attributes of this object will remain fixed during interpreter execution and through the course of an implementation version. This ensures behaviors don't change between versions which depend on variables in ``sys.implementation``. -The mapping will contain at least the values described in the -`Required Variables`_ section below. However, implementations are -free to add other implementation information there. Some -*conceivable* extra variables are described in the `Other Possible -Variables`_ section. +The object will have each of the attributes described in the `Required +Variables`_ section below. Any other per-implementation values may be +stored in ``sys.implementation.metadata``. However, nothing in the +standard library will rely on ``sys.implementation.metadata``. +Examples of possible metadata values are described in the `Example +Metadata Values`_ section. -This proposal takes a conservative approach in requiring only two +This proposal takes a conservative approach in requiring only four variables. As more become appropriate, they may be added with discretion. Required Variables --------------------- +------------------ These are variables in ``sys.implementation`` on which the standard -library would rely, meaning implementers must define them: +library would rely, with the exception of ``metadata``, meaning +implementers must define them: **name** - This is the name of the implementation (case sensitive). Examples - include 'PyPy', 'Jython', 'IronPython', and 'CPython'. + This is the common name of the implementation (case sensitive). + Examples include 'PyPy', 'Jython', 'IronPython', and 'CPython'. **version** - This is the version of the implementation, as opposed to the - version of the language it implements. This value conforms to the - format described in `Version Format`_. - - -Other Possible Variables ------------------------- - -These variables could be useful, but don't necessarily have a clear -use case presently. They are listed here as values to consider in the -future, with appropriate discussion, relative to clear use-cases. -Their descriptions are therefore intentionally unhindered by details. + This is the version of the implementation, as opposed to the + version of the language it implements. This value conforms to the + format described in `Version Format`_. **cache_tag** - A string used for the PEP 3147 cache tag (e.g. 'cpython33' for - CPython 3.3). The name and version from above could be used to - compose this, though an implementation may want something else. - However, module caching is not a requirement of implementations, - nor is the use of cache tags. + A string used for the PEP 3147 cache tag [#cachetag]_. It would + normally be a composite of the name and version (e.g. 'cpython-33' + for CPython 3.3). However, an implementation may explicitly use a + different cache tag. If ``cache_tag`` is set to None, it indicates + that module caching should be disabled. -**vcs_url** - The URL pointing to the main VCS repository for the implementation - project. +**metadata** + Any other values that an implementation wishes to specify, + particularly informational ones. Neither the standard library nor + the language specification will rely on implementation metadata. + Also see the list of `Example Metadata Values`_. + -**vcs_revision_id** - A value that identifies the VCS revision of the implementation - that is currently running. +Adding New Required Attributes +------------------------------ -**build_toolchain** - Identifies the tools used to build the interpreter. - -**homepage** - The URL of the implementation's website. - -**site_prefix** - The preferred site prefix for this implementation. - -**runtime** - The run-time environment in which the interpreter is running, as - in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* - Executable". - -**gc_type** - The type of garbage collection used, like "reference counting" or - "mark and sweep". +XXX PEP? something lighter? Version Format @@ -136,82 +115,143 @@ facilitate the usefulness of a version variable, its value should be in a consistent format across implementations. -XXX Subject to feedback +As such, the format of ``sys.implementation.version`` must follow that +of ``sys.version_info``, which is effectively a named tuple. It is a +familiar format and generally consistent with normal version format +conventions. -As such, the format of ``sys.implementation['version']`` must follow -that of ``sys.version_info``, which is effectively a named tuple. It -is a familiar format and generally consistent with normal version -format conventions. +XXX The following is not exactly true: -Keep in mind, however, that ``sys.implementation['version']`` is the +Keep in mind, however, that ``sys.implementation.version`` is the version of the Python *implementation*, while ``sys.version_info`` (and friends) is the version of the Python language. +Example Metadata Values +----------------------- + +These are the sorts of values an implementation may put into +``sys.implementation.metadata``. However, these names and +descriptions are only examples and are not being proposed here. If +they later have meaningful uses cases, they can be added by following +the process described in `Adding New Required Attributes`_. + +**vcs_url** + The URL pointing to the main VCS repository for the implementation + project. + +**vcs_revision_id** + A value that identifies the VCS revision of the implementation that + is currently running. + +**build_toolchain** + Identifies the tools used to build the interpreter. + +**build_date** + The timestamp of when the interpreter was built. + +**homepage** + The URL of the implementation's website. + +**site_prefix** + The preferred site prefix for this implementation. + +**runtime** + The run-time environment in which the interpreter is running, as + in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* + Executable". + +**gc_type** + The type of garbage collection used, like "reference counting" or + "mark and sweep". + + Rationale ========= The status quo for implementation-specific information gives us that -information in a more fragile, harder to maintain way. It's spread +information in a more fragile, harder to maintain way. It is spread out over different modules or inferred from other information, as we -see with ``platform.python_implementation()``. +see with `platform.python_implementation()`_. This PEP is the main alternative to that approach. It consolidates the implementation-specific information into a single namespace and makes explicit that which was implicit. -Why a Dictionary? ------------------ +Why a Custom Type? +------------------ -A dictionary reflects a simple namespace. It maps names to values and -that's it. Really that's all we need. +A dedicated class, of which ``sys.implementation`` is an instance, would +facilitate the dotted access of a "named" tuple. At the same time, it +allows us to avoid the problems of the other approaches (see below), +like confusion about ordering and iteration. The alternatives to a dictionary are considered separately here: -**"Named" Tuple** +**Dictionary** -The first alternative is a namedtuple or a structseq or some other +A dictionary reflects a simple namespace with item access. It +maps names to values and that's all. It also reflects the more variable +nature of ``sys.implementation``. + +However, a simple dictionary does not set expectations very well about +the nature of ``sys.implementation``. The custom type approach, with +a fixed set of required attributes, does a better job of this. + +**Named Tuple** + +Another close alternative is a namedtuple or a structseq or some other tuple type with dotted access (a la ``sys.version_info``). This type is immutable and simple. It is a well established pattern for -implementation- specific variables in Python. Dotted access on a +implementation-specific variables in Python. Dotted access on a namespace is also very convenient. -However, sys.implementation does not have meaning as a sequence. -Also, unlike other such variables, it has the potential for more -variation over time. Finally, generic lookup may favor dicts:: +Fallback lookup may favor dicts:: - cache_tag = sys.implementation.get('cache_tag') + cache_tag = sys.implementation.get('cache_tag') - vs. + vs. - cache_tag = getattr(sys.implementation.get, 'cache_tag', None) + cache_tag = getattr(sys.implementation.get, 'cache_tag', None) + +However, this is mitigated by having ``sys.implementation.metadata``. + +One problem with using a named tuple is that ``sys.implementation`` does +not have meaning as a sequence. Also, unlike other similar ``sys`` +variables, it has a far greater potential to change over time. If a named tuple were used, we'd be very clear in the documentation that the length and order of the value are not reliable. Iterability would not be guaranteed. -**Concrete Class** - -Another option would be to have a dedicated class, of which -``sys.implementation`` is an instance. This would facilitate the -dotted access of a "named" tuple, without exposing any confusion about -ordering and iteration. - -One downside is that you lose the immutable aspect of a tuple, making -it less clear that ``sys.implementation`` should not be manipulated. -Another downside is that classes often imply the presence (or -possibility) of methods, which may be misleading in this case. - **Module** Using a module instead of a dict is another option. It has similar characteristics to an instance, but with a slight hint of immutability (at least by convention). Such a module could be a stand-alone sub- module of ``sys`` or added on, like ``os.path``. Unlike a concrete -class, no new type would be necessary. +class, no new type would be necessary. This is a pretty close fit to +what we need. -The downsides are similar to those of a concrete class. +The downside is that the module type is much less conducive to +extension, making it more difficult to address the weaknesses of using +an instance of a concrete class. + + +Why metadata? +------------- + +``sys.implementation.metadata`` will hold any optional, strictly- +informational, or per-implementation data. This allows us to restrict +``sys.implementation`` to the required attributes. In that way, its +type can reflect the more stable namespace and +``sys.implementation.metadata`` (as a dict) can reflect the less +certain namespace. + +``sys.implementation.metadata`` is the place an implementation can put +values that must be built-in, "without having to pollute the main sys +namespace" [#Nick]_. Why a Part of ``sys``? @@ -229,40 +269,41 @@ ``sys.implementation`` are intended for use by the standard library. Constraining those values, essentially specifying an API for them, allows them to be used consistently, regardless of how they are -implemented otherwise. +otherwise implemented. Discussion ========== The topic of ``sys.implementation`` came up on the python-ideas list -in 2009, where the reception was broadly positive [1]_. I revived the -discussion recently while working on a pure-python ``imp.get_tag()`` -[2]_. The messages in `issue #14673`_ are also relevant. +in 2009, where the reception was broadly positive [#original]_. I +revived the discussion recently while working on a pure-python +``imp.get_tag()`` [#revived]_. Discussion has been ongoing +[#feedback]_. The messages in `issue #14673`_ are also relevant. Use-cases ========= -``platform.python_implementation()`` ------------------------------------- +platform.python_implementation() +-------------------------------- "explicit is better than implicit" -The ``platform`` module guesses the python implementation by looking -for clues in a couple different ``sys`` variables [3]_. However, this -approach is fragile. Beyond that, it's limited to those -implementations that core developers have blessed by special-casing -them in the ``platform`` module. +The ``platform`` module determines the python implementation by +looking for clues in a couple different ``sys`` variables [#guess]_. +However, this approach is fragile, requiring changes to the standard +library each time an implementation changes. Beyond that, support in +``platform`` is limited to those implementations that core developers +have blessed by special-casing them in the ``platform`` module. With ``sys.implementation`` the various implementations would *explicitly* set the values in their own version of the ``sys`` module. -Aside from the guessing, another concern is that the ``platform`` -module is part of the stdlib, which ideally would minimize -implementation details such as would be moved to -``sys.implementation``. +Another concern is that the ``platform`` module is part of the stdlib, +which ideally would minimize implementation details such as would be +moved to ``sys.implementation``. Any overlap between ``sys.implementation`` and the ``platform`` module would simply defer to ``sys.implementation`` (with the same interface @@ -275,18 +316,18 @@ PEP 3147 defined the use of a module cache and cache tags for file names. The importlib bootstrap code, frozen into the Python binary as of 3.3, uses the cache tags during the import process. Part of the -project to bootstrap importlib has been to clean out of -`Python/import.c` any code that did not need to be there. +project to bootstrap importlib has been to clean code out of +`Python/import.c`_ that did not need to be there any longer. -The cache tag defined in `Python/import.c` was hard-coded to -``"cpython" MAJOR MINOR`` [4]_. For importlib the options are either -hard-coding it in the same way, or guessing the implementation in the -same way as does ``platform.python_implementation()``. +The cache tag defined in ``Python/import.c`` was hard-coded to +``"cpython" MAJOR MINOR`` [#cachetag]_. For importlib the options are +either hard-coding it in the same way, or guessing the implementation +in the same way as does ``platform.python_implementation()``. -As long as the hard-coded tag is limited to CPython-specific code, -it's livable. However, inasmuch as other Python implementations use -the importlib code to work with the module cache, a hard-coded tag -would become a problem.. +As long as the hard-coded tag is limited to CPython-specific code, it +is livable. However, inasmuch as other Python implementations use the +importlib code to work with the module cache, a hard-coded tag would +become a problem. Directly using the ``platform`` module in this case is a non-starter. Any module used in the importlib bootstrap must be built-in or frozen, @@ -314,9 +355,11 @@ Jython's ``os.name`` Hack ------------------------- -XXX - -http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512 +In Jython, ``os.name`` is set to 'java' to accommodate special +treatment of the java environment in the standard library [#os_name]_ +[#javatest]_. Unfortunately it masks the os name that would otherwise +go there. ``sys.implementation`` would help obviate the need for this +special case. Feedback From Other Python Implementers @@ -362,8 +405,8 @@ Alternatives ============ -With the single-namespace-under-sys so straightforward, no -alternatives have been considered for this PEP. +Since the single-namespace-under-sys approach is relatively +straightforward, no alternatives have been considered for this PEP. Open Issues @@ -374,6 +417,16 @@ - possibly pull in implementation details from the main ``sys`` namespace and elsewhere (PEP 3137 lite). +* What is the process for introducing new required variables? PEP? + +* Is the ``sys.version_info`` format the right one here? + +* Should ``sys.implementation.hexversion`` be part of the PEP? + +* Does ``sys.(version|version_info|hexversion)`` need to better + reflect the version of the language spec? Micro version, series, + and release seem like implementation-specific values. + * Alternatives to the approach dictated by this PEP? * Do we really want to commit to using a dict for @@ -387,10 +440,19 @@ oriented toward internal use then the data structure is not as critical. However, ``sys.implementation`` is intended to have a non-localized impact across the standard library and the - interpreter. It's better to *not* make hacking it become an + interpreter. It is better to *not* make hacking it become an attractive nuisance, regardless of our intentions for usage. -* use (immutable?) nameddict (analogous to namedtuple/structseq)? +* Should ``sys.implementation`` and its values be immutable? A benefit + of an immutable type is it communicates that the value is not + expected to change and should not be manipulated. + +* Should ``sys.implementation`` be strictly disallowed to have methods? + Classes often imply the presence (or possibility) of methods, which + may be misleading in this case. + +* Should ``sys.implementation`` implement the collections.abc.Mapping + interface? Implementation @@ -402,25 +464,45 @@ References ========== -.. [1] http://mail.python.org/pipermail/python-dev/2009-October/092893.html +.. [#original] The 2009 sys.implementation discussion: + http://mail.python.org/pipermail/python-dev/2009-October/092893.html -.. [2] http://mail.python.org/pipermail/python-ideas/2012-April/014878.html +.. [#revived] The initial 2012 discussion: + http://mail.python.org/pipermail/python-ideas/2012-April/014878.html -.. [3] http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247 +.. [#feedback] Feedback on the PEP: + http://mail.python.org/pipermail/python-ideas/2012-April/014954.html -.. [4] http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c#l121 +.. [#guess] The ``platform`` code which divines the implementation name: + http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247 -.. [5] Examples of implementation-specific handling in test.support: +.. [#cachetag] The definition for cache tags in PEP 3147: + http://www.python.org/dev/peps/pep-3147/#id53 -| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509 -| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246 -| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252 -| http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275 +.. [#tag_impl] The original implementation of the cache tag in CPython: + http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c#l121 + +.. [#tests] Examples of implementation-specific handling in test.support: + * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509 + * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246 + * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252 + * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275 + +.. [#os_name] The standard library entry for os.name: + http://docs.python.org/3.3/library/os.html#os.name + +.. [#javatest] The use of ``os.name`` as 'java' in the stdlib test suite. + http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512 + +.. [#Nick] Nick Coghlan's proposal for ``sys.implementation.metadata``: + http://mail.python.org/pipermail/python-ideas/2012-May/014984.html .. _issue #14673: http://bugs.python.org/issue14673 .. _Lib/test/support.py: http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py +.. _Python/import.c: http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sun May 6 11:19:59 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 11:19:59 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Post-3=2E3a3_bump=2E?= Message-ID: http://hg.python.org/cpython/rev/9cbe6227bf3c changeset: 76780:9cbe6227bf3c user: Georg Brandl date: Sun May 06 11:20:09 2012 +0200 summary: Post-3.3a3 bump. files: Include/patchlevel.h | 2 +- Misc/NEWS | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 3 /* Version as a string */ -#define PY_VERSION "3.3.0a3" +#define PY_VERSION "3.3.0a3+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -5,7 +5,7 @@ What's New in Python 3.3.0 Alpha 4? =================================== -*Release date: 26-May-2012* +*Release date: XX-XXX-2012* Core and Builtins ----------------- @@ -26,6 +26,7 @@ - Issue #14127 and #10148: shutil.copystat now preserves exact mtime and atime on filesystems providing nanosecond resolution. + What's New in Python 3.3.0 Alpha 3? =================================== @@ -38,7 +39,7 @@ - Issue #14433: Prevent msvcrt crash in interactive prompt when stdin is closed. - + - Issue #14521: Make result of float('nan') and float('-nan') more consistent across platforms. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 11:23:55 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 11:23:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_some_rst_errors_in_NEWS?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/a07a95a590b7 changeset: 76781:a07a95a590b7 user: Georg Brandl date: Sun May 06 11:24:06 2012 +0200 summary: Fix some rst errors in NEWS. files: Misc/NEWS | 74 ++++++++++++++++++++---------------------- 1 files changed, 36 insertions(+), 38 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,21 +10,21 @@ Core and Builtins ----------------- -- Issue #14705: The PyArg_Parse() family of functions now support the 'p' - format unit, which accepts a "boolean predicate" argument. It converts - any Python value into an integer--0 if it is "false", and 1 otherwise. +- Issue #14705: The PyArg_Parse() family of functions now support the 'p' format + unit, which accepts a "boolean predicate" argument. It converts any Python + value into an integer--0 if it is "false", and 1 otherwise. Library ------- - Issue #14127: The os.stat() result object now provides three additional - fields: st_ctime_ns, st_mtime_ns, and st_atime_ns, providing those times - as an integer with nanosecond resolution. The functions os.utime(), - os.lutimes(), and os.futimes() now accept a new parameter, ns, which - accepts mtime and atime as integers with nanosecond resolution. - -- Issue #14127 and #10148: shutil.copystat now preserves exact mtime - and atime on filesystems providing nanosecond resolution. + fields: st_ctime_ns, st_mtime_ns, and st_atime_ns, providing those times as an + integer with nanosecond resolution. The functions os.utime(), os.lutimes(), + and os.futimes() now accept a new parameter, ns, which accepts mtime and atime + as integers with nanosecond resolution. + +- Issue #14127 and #10148: shutil.copystat now preserves exact mtime and atime + on filesystems providing nanosecond resolution. What's New in Python 3.3.0 Alpha 3? @@ -37,11 +37,10 @@ - Issue #14699: Fix calling the classmethod descriptor directly. -- Issue #14433: Prevent msvcrt crash in interactive prompt when stdin - is closed. - -- Issue #14521: Make result of float('nan') and float('-nan') more - consistent across platforms. +- Issue #14433: Prevent msvcrt crash in interactive prompt when stdin is closed. + +- Issue #14521: Make result of float('nan') and float('-nan') more consistent + across platforms. - Issue #14646: __import__() sets __loader__ if the loader did not. @@ -116,8 +115,7 @@ - Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. -- Issue #14371: Support bzip2 in zipfile module. - Patch by Serhiy Storchaka. +- Issue #14371: Support bzip2 in zipfile module. Patch by Serhiy Storchaka. - Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye. @@ -673,8 +671,8 @@ - Issue #9896: Add start, stop, and step attributes to range objects. - Issue #13343: Fix a SystemError when a lambda expression uses a global - variable in the default value of a keyword-only argument: - (lambda *, arg=GLOBAL_NAME: None) + variable in the default value of a keyword-only argument: ``lambda *, + arg=GLOBAL_NAME: None`` - Issue #12797: Added custom opener parameter to builtin open() and FileIO.open(). @@ -696,8 +694,8 @@ class object. These cases now behave correctly. Patch by Daniel Urban. - Issue #12753: Add support for Unicode name aliases and named sequences. - Both :func:`unicodedata.lookup()` and '\N{...}' now resolve aliases, - and :func:`unicodedata.lookup()` resolves named sequences too. + Both ``unicodedata.lookup()`` and '\N{...}' now resolve aliases, + and ``unicodedata.lookup()`` resolves named sequences too. - Issue #12170: The count(), find(), rfind(), index() and rindex() methods of bytes and bytearray objects now accept an integer between 0 and 255 @@ -1311,9 +1309,9 @@ - Issue #13591: A bug in importlib has been fixed that caused import_module to load a module twice. -- Issue #4625: If IDLE cannot write to its recent file or breakpoint - files, display a message popup and continue rather than crash. - (original patch by Roger Serwy) +- Issue #4625: If IDLE cannot write to its recent file or breakpoint files, + display a message popup and continue rather than crash. Original patch by + Roger Serwy. - Issue #13449 sched.scheduler.run() method has a new "blocking" parameter which when set to False makes run() execute the scheduled events due to expire @@ -1921,8 +1919,8 @@ would reliably freeze/deadlock. - Issue #12040: Expose a new attribute ``sentinel`` on instances of - :class:`multiprocessing.Process`. Also, fix Process.join() to not use - polling anymore, when given a timeout. + ``multiprocessing.Process``. Also, fix Process.join() to not use polling + anymore, when given a timeout. - Issue #11893: Remove obsolete internal wrapper class ``SSLFakeFile`` in the smtplib module. Patch by Catalin Iacob. @@ -2424,8 +2422,8 @@ - Issue #10755: Add the posix.flistdir() function. Patch by Ross Lagerwall. -- Issue #4761: Add the *at() family of functions (openat(), etc.) to the posix - module. Patch by Ross Lagerwall. +- Issue #4761: Add the ``*at()`` family of functions (openat(), etc.) to the + posix module. Patch by Ross Lagerwall. - Issue #7322: Trying to read from a socket's file-like object after a timeout occurred now raises an error instead of silently losing data. @@ -2523,7 +2521,7 @@ - Issue #8746: Correct faulty configure checks so that os.chflags() and os.lchflags() are once again built on systems that support these - functions (*BSD and OS X). Also add new stat file flags for OS X + functions (BSD and OS X). Also add new stat file flags for OS X (UF_HIDDEN and UF_COMPRESSED). - Issue #10645: Installing Python does no longer create a @@ -5738,7 +5736,7 @@ handled. In many cases this order wasn't being followed, leading to the wrong Python exception being raised. -- Issue #7865: The close() method of :mod:`io` objects should not swallow +- Issue #7865: The close() method of ``io`` objects should not swallow exceptions raised by the implicit flush(). Also qensure that calling close() several times is supported. Patch by Pascal Chambon. @@ -5850,7 +5848,7 @@ - Issue #5277: Fix quote counting when parsing RFC 2231 encoded parameters. -- Issue #7316: The acquire() method of lock objects in the :mod:`threading` +- Issue #7316: The acquire() method of lock objects in the ``threading`` module now takes an optional timeout argument in seconds. Timeout support relies on the system threading library, so as to avoid a semi-busy wait loop. @@ -5869,7 +5867,7 @@ - Issue #8375: test_distutils now checks if the temporary directory are still present before it cleans them. -- Issue #8374: Update the internal alias table in the :mod:`locale` module to +- Issue #8374: Update the internal alias table in the ``locale`` module to cover recent locale changes and additions. - Issue #8321: Give access to OpenSSL version numbers from the `ssl` module, @@ -6054,9 +6052,9 @@ ftruncate() and other truncation APIs. Patch by Pascal Chambon. - Issue #7610: Reworked implementation of the internal - :class:`zipfile.ZipExtFile` class used to represent files stored inside an + ``zipfile.ZipExtFile`` class used to represent files stored inside an archive. The new implementation is significantly faster and can be wrapped in - a :class:`io.BufferedReader` object for more speedups. It also solves an + a ``io.BufferedReader`` object for more speedups. It also solves an issue where interleaved calls to `read()` and `readline()` give wrong results. Patch by Nir Aides. @@ -6814,10 +6812,10 @@ calling methods on the object. - Issue #7222: Make thread "reaping" more reliable so that reference - leak-chasing test runs give sensible results. The previous method of reaping + leak-chasing test runs give sensible results. The previous method of reaping threads could return successfully while some Thread objects were still - referenced. This also introduces a new private function: - :func:`_thread._count()`. + referenced. This also introduces a new private function: + ``_thread._count()``. - Issue #7151: Fixed regrtest -j so that output to stderr from a test no longer runs the risk of causing the worker thread to fail. @@ -7258,7 +7256,7 @@ - Issue #5923: Update the ``turtle`` module to version 1.1, add two new turtle demos in Demo/turtle. -- Issue #5692: In :class:`zipfile.Zipfile`, fix wrong path calculation when +- Issue #5692: In ``zipfile.Zipfile``, fix wrong path calculation when extracting a file to the root directory. - Issue #5913: os.listdir() should fail for empty path on windows. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 11:49:54 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 11:49:54 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEzMTgzOiBiYWNr?= =?utf8?q?port_fixes_to_test=5Fpdb_to_3=2E2_branch?= Message-ID: http://hg.python.org/cpython/rev/e275a9f7daa9 changeset: 76782:e275a9f7daa9 branch: 3.2 parent: 76773:afab4d14d5e7 user: Georg Brandl date: Sun May 06 11:50:00 2012 +0200 summary: #13183: backport fixes to test_pdb to 3.2 branch files: Lib/test/test_pdb.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -601,6 +601,7 @@ filename = 'main.py' with open(filename, 'w') as f: f.write(textwrap.dedent(script)) + self.addCleanup(support.unlink, filename) cmd = [sys.executable, '-m', 'pdb', filename] stdout = stderr = None with subprocess.Popen(cmd, stdout=subprocess.PIPE, @@ -657,9 +658,11 @@ """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) + self.addCleanup(support.unlink, 'bar.py') stdout, stderr = self.run_pdb(script, commands) - self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3], - 'Fail to step into the caller after a return') + self.assertTrue( + any('main.py(5)foo()->None' in l for l in stdout.splitlines()), + 'Fail to step into the caller after a return') def tearDown(self): support.unlink(support.TESTFN) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 11:53:40 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 11:53:40 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzEzMTgzOiBiYWNr?= =?utf8?q?port_fixes_to_test=5Fpdb_to_2=2E7_branch?= Message-ID: http://hg.python.org/cpython/rev/2644e4ea02d3 changeset: 76783:2644e4ea02d3 branch: 2.7 parent: 76772:35ef949e85d7 user: Georg Brandl date: Sun May 06 11:53:51 2012 +0200 summary: #13183: backport fixes to test_pdb to 2.7 branch files: Lib/test/test_pdb.py | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py --- a/Lib/test/test_pdb.py +++ b/Lib/test/test_pdb.py @@ -20,6 +20,7 @@ filename = 'main.py' with open(filename, 'w') as f: f.write(textwrap.dedent(script)) + self.addCleanup(test_support.unlink, filename) cmd = [sys.executable, '-m', 'pdb', filename] stdout = stderr = None proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, @@ -61,9 +62,11 @@ """ with open('bar.py', 'w') as f: f.write(textwrap.dedent(bar)) + self.addCleanup(test_support.unlink, 'bar.py') stdout, stderr = self.run_pdb(script, commands) - self.assertIn('main.py(5)foo()->None', stdout.split('\n')[-3], - 'Fail to step into the caller after a return') + self.assertTrue( + any('main.py(5)foo()->None' in l for l in stdout.splitlines()), + 'Fail to step into the caller after a return') class PdbTestInput(object): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 12:06:14 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 12:06:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_with_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/a262ad6536fc changeset: 76784:a262ad6536fc parent: 76781:a07a95a590b7 parent: 76782:e275a9f7daa9 user: Georg Brandl date: Sun May 06 12:06:24 2012 +0200 summary: merge with 3.2 files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 12:28:53 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 06 May 2012 12:28:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Closes_=2314729=3A_Allowed_?= =?utf8?q?test_to_pass_on_Windows_by_adjusting_the_test_condition?= Message-ID: http://hg.python.org/cpython/rev/2e71f25912d4 changeset: 76785:2e71f25912d4 user: Vinay Sajip date: Sun May 06 11:28:46 2012 +0100 summary: Closes #14729: Allowed test to pass on Windows by adjusting the test condition slightly to allow for a Windows-specific error message. files: Lib/test/test_faulthandler.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_faulthandler.py b/Lib/test/test_faulthandler.py --- a/Lib/test/test_faulthandler.py +++ b/Lib/test/test_faulthandler.py @@ -92,7 +92,7 @@ ^Fatal Python error: {name} {header}: - File "", line {lineno} in $ + File "", line {lineno} in """.strip() regex = regex.format( lineno=line_number, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 12:35:04 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 06 May 2012 12:35:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=239116=3A_Allowed_te?= =?utf8?q?st_to_pass_on_Windows_by_adjusting_the_test_condition?= Message-ID: http://hg.python.org/cpython/rev/4d74d275224d changeset: 76786:4d74d275224d user: Vinay Sajip date: Sun May 06 11:34:50 2012 +0100 summary: Issue #9116: Allowed test to pass on Windows by adjusting the test condition slightly to allow for a Windows-specific error message. files: Lib/test/test_capi.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -48,9 +48,9 @@ (out, err) = p.communicate() self.assertEqual(out, b'') # This used to cause an infinite loop. - self.assertEqual(err.rstrip(), + self.assertTrue(err.rstrip().startswith( b'Fatal Python error:' - b' PyThreadState_Get: no current thread') + b' PyThreadState_Get: no current thread')) def test_memoryview_from_NULL_pointer(self): self.assertRaises(ValueError, _testcapi.make_memoryview_from_NULL_pointer) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 13:03:16 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 06 May 2012 13:03:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2312660=3A_Skip_test=5Fgdb?= =?utf8?q?_when_run_from_an_installed_Python=2E?= Message-ID: http://hg.python.org/cpython/rev/84bbb8d2d237 changeset: 76787:84bbb8d2d237 user: Vinay Sajip date: Sun May 06 12:03:05 2012 +0100 summary: #12660: Skip test_gdb when run from an installed Python. files: Lib/test/test_gdb.py | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -7,6 +7,7 @@ import re import subprocess import sys +import sysconfig import unittest import locale @@ -24,6 +25,9 @@ raise unittest.SkipTest("gdb versions before 7.0 didn't support python embedding" " Saw:\n" + gdb_version.decode('ascii', 'replace')) +if not sysconfig.is_python_build(): + raise unittest.SkipTest("test_gdb only works on source builds at the moment.") + # Verify that "gdb" was built with the embedded python support enabled: cmd = "--eval-command=python import sys; print sys.version_info" p = subprocess.Popen(["gdb", "--batch", cmd], -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 13:42:41 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 06 May 2012 13:42:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_formatting_of_preset_va?= =?utf8?q?lues_for_exception_messages_in_=5Flzmamodule=2Ec=2E?= Message-ID: http://hg.python.org/cpython/rev/02374060397b changeset: 76788:02374060397b user: Nadeem Vawda date: Sun May 06 13:35:47 2012 +0200 summary: Fix formatting of preset values for exception messages in _lzmamodule.c. files: Modules/_lzmamodule.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -203,7 +203,7 @@ if (lzma_lzma_preset(options, preset)) { PyMem_Free(options); - PyErr_Format(Error, "lzma_lzma_preset() failed for preset %#x", preset); + PyErr_Format(Error, "Invalid compression preset: %d", preset); return NULL; } @@ -485,7 +485,7 @@ lzma_options_lzma options; if (lzma_lzma_preset(&options, preset)) { - PyErr_Format(Error, "Invalid compression preset: %#x", preset); + PyErr_Format(Error, "Invalid compression preset: %d", preset); return -1; } lzret = lzma_alone_encoder(lzs, &options); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 15:11:07 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 06 May 2012 15:11:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Closes_=2313989=3A_Add_supp?= =?utf8?q?ort_for_text_modes_to_gzip=2Eopen=28=29=2E?= Message-ID: http://hg.python.org/cpython/rev/55202ca694d7 changeset: 76789:55202ca694d7 user: Nadeem Vawda date: Sun May 06 15:04:01 2012 +0200 summary: Closes #13989: Add support for text modes to gzip.open(). Also, add tests for gzip.open(). files: Doc/library/gzip.rst | 46 +++++++++++---- Lib/gzip.py | 42 +++++++++++--- Lib/test/test_gzip.py | 88 +++++++++++++++++++++++++++++++ Misc/NEWS | 2 + 4 files changed, 157 insertions(+), 21 deletions(-) diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst --- a/Doc/library/gzip.rst +++ b/Doc/library/gzip.rst @@ -13,9 +13,11 @@ The data compression is provided by the :mod:`zlib` module. -The :mod:`gzip` module provides the :class:`GzipFile` class. The :class:`GzipFile` -class reads and writes :program:`gzip`\ -format files, automatically compressing -or decompressing the data so that it looks like an ordinary :term:`file object`. +The :mod:`gzip` module provides the :class:`GzipFile` class, as well as the +:func:`gzip.open`, :func:`compress` and :func:`decompress` convenience +functions. The :class:`GzipFile` class reads and writes :program:`gzip`\ -format +files, automatically compressing or decompressing the data so that it looks like +an ordinary :term:`file object`. Note that additional file formats which can be decompressed by the :program:`gzip` and :program:`gunzip` programs, such as those produced by @@ -24,6 +26,32 @@ The module defines the following items: +.. function:: open(filename, mode='rb', compresslevel=9, encoding=None, errors=None, newline=None) + + Open *filename* as a gzip-compressed file in binary or text mode. + + Returns a :term:`file object`. + + The *mode* argument can be any of ``'r'``, ``'rb'``, ``'a'``, ``'ab'``, + ``'w'``, or ``'wb'`` for binary mode, or ``'rt'``, ``'at'``, or ``'wt'`` for + text mode. The default is ``'rb'``. + + The *compresslevel* argument is an integer from 1 to 9, as for the + :class:`GzipFile` constructor. + + For binary mode, this function is equivalent to the :class:`GzipFile` + constructor: ``GzipFile(filename, mode, compresslevel)``. In this case, the + *encoding*, *errors* and *newline* arguments must not be provided. + + For text mode, a :class:`GzipFile` object is created, and wrapped in an + :class:`io.TextIOWrapper` instance with the specified encoding, error + handling behavior, and line ending(s). + + .. versionchanged:: 3.3 + Support for text mode was added, along with the *encoding*, *errors* and + *newline* arguments. + + .. class:: GzipFile(filename=None, mode=None, compresslevel=9, fileobj=None, mtime=None) Constructor for the :class:`GzipFile` class, which simulates most of the @@ -46,9 +74,9 @@ or ``'wb'``, depending on whether the file will be read or written. The default is the mode of *fileobj* if discernible; otherwise, the default is ``'rb'``. - Note that the file is always opened in binary mode; text mode is not - supported. If you need to read a compressed file in text mode, wrap your - :class:`GzipFile` with an :class:`io.TextIOWrapper`. + Note that the file is always opened in binary mode. To open a compressed file + in text mode, use :func:`gzip.open` (or wrap your :class:`GzipFile` with an + :class:`io.TextIOWrapper`). The *compresslevel* argument is an integer from ``1`` to ``9`` controlling the level of compression; ``1`` is fastest and produces the least compression, and @@ -97,12 +125,6 @@ The :meth:`io.BufferedIOBase.read1` method is now implemented. -.. function:: open(filename, mode='rb', compresslevel=9) - - This is a shorthand for ``GzipFile(filename,`` ``mode,`` ``compresslevel)``. - The *filename* argument is required; *mode* defaults to ``'rb'`` and - *compresslevel* defaults to ``9``. - .. function:: compress(data, compresslevel=9) Compress the *data*, returning a :class:`bytes` object containing diff --git a/Lib/gzip.py b/Lib/gzip.py --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -16,6 +16,39 @@ READ, WRITE = 1, 2 +def open(filename, mode="rb", compresslevel=9, + encoding=None, errors=None, newline=None): + """Open a gzip-compressed file in binary or text mode. + + The mode argument can be "r", "rb", "w", "wb", "a" or "ab" for binary mode, + or "rt", "wt" or "at" for text mode. The default mode is "rb", and the + default compresslevel is 9. + + For binary mode, this function is equivalent to the GzipFile constructor: + GzipFile(filename, mode, compresslevel). In this case, the encoding, errors + and newline arguments must not be provided. + + For text mode, a GzipFile object is created, and wrapped in an + io.TextIOWrapper instance with the specified encoding, error handling + behavior, and line ending(s). + + """ + if "t" in mode: + if "b" in mode: + raise ValueError("Invalid mode: %r" % (mode,)) + else: + if encoding is not None: + raise ValueError("Argument 'encoding' not supported in binary mode") + if errors is not None: + raise ValueError("Argument 'errors' not supported in binary mode") + if newline is not None: + raise ValueError("Argument 'newline' not supported in binary mode") + binary_file = GzipFile(filename, mode.replace("t", ""), compresslevel) + if "t" in mode: + return io.TextIOWrapper(binary_file, encoding, errors, newline) + else: + return binary_file + def write32u(output, value): # The L format writes the bit pattern correctly whether signed # or unsigned. @@ -24,15 +57,6 @@ def read32(input): return struct.unpack(" http://hg.python.org/cpython/rev/d75934e88234 changeset: 76790:d75934e88234 user: Nadeem Vawda date: Sun May 06 15:17:52 2012 +0200 summary: Separate tests for gzip.GzipFile and gzip.open. files: Lib/test/test_gzip.py | 29 ++++++++++++++--------------- 1 files changed, 14 insertions(+), 15 deletions(-) diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -33,7 +33,7 @@ raise io.UnsupportedOperation -class TestGzip(unittest.TestCase): +class BaseTest(unittest.TestCase): filename = support.TESTFN def setUp(self): @@ -43,6 +43,7 @@ support.unlink(self.filename) +class TestGzip(BaseTest): def test_write(self): with gzip.GzipFile(self.filename, 'wb') as f: f.write(data1 * 50) @@ -115,14 +116,14 @@ # Bug #1074261 was triggered when reading a file that contained # many, many members. Create such a file and verify that reading it # works. - with gzip.open(self.filename, 'wb', 9) as f: + with gzip.GzipFile(self.filename, 'wb', 9) as f: f.write(b'a') for i in range(0, 200): - with gzip.open(self.filename, "ab", 9) as f: # append + with gzip.GzipFile(self.filename, "ab", 9) as f: # append f.write(b'a') # Try reading the file - with gzip.open(self.filename, "rb") as zgfile: + with gzip.GzipFile(self.filename, "rb") as zgfile: contents = b"" while 1: ztxt = zgfile.read(8192) @@ -374,10 +375,9 @@ datac = gzip.compress(data) self.assertEqual(gzip.decompress(datac), data) - # Test the 'open' convenience function. - def test_open_binary(self): - # Test explicit binary modes. +class TestOpen(BaseTest): + def test_binary_modes(self): uncompressed = data1 * 50 with gzip.open(self.filename, "wb") as f: f.write(uncompressed) @@ -392,7 +392,7 @@ file_data = gzip.decompress(f.read()) self.assertEqual(file_data, uncompressed * 2) - def test_open_default_binary(self): + def test_implicit_binary_modes(self): # Test implicit binary modes (no "b" or "t" in mode string). uncompressed = data1 * 50 with gzip.open(self.filename, "w") as f: @@ -408,8 +408,7 @@ file_data = gzip.decompress(f.read()) self.assertEqual(file_data, uncompressed * 2) - def test_open_text(self): - # Test text modes. + def test_text_modes(self): uncompressed = data1.decode("ascii") * 50 with gzip.open(self.filename, "wt") as f: f.write(uncompressed) @@ -424,7 +423,7 @@ file_data = gzip.decompress(f.read()).decode("ascii") self.assertEqual(file_data, uncompressed * 2) - def test_open_bad_params(self): + def test_bad_params(self): # Test invalid parameter combinations. with self.assertRaises(ValueError): gzip.open(self.filename, "wbt") @@ -435,7 +434,7 @@ with self.assertRaises(ValueError): gzip.open(self.filename, "rb", newline="\n") - def test_open_with_encoding(self): + def test_encoding(self): # Test non-default encoding. uncompressed = data1.decode("ascii") * 50 with gzip.open(self.filename, "wt", encoding="utf-16") as f: @@ -446,7 +445,7 @@ with gzip.open(self.filename, "rt", encoding="utf-16") as f: self.assertEqual(f.read(), uncompressed) - def test_open_with_encoding_error_handler(self): + def test_encoding_error_handler(self): # Test with non-default encoding error handler. with gzip.open(self.filename, "wb") as f: f.write(b"foo\xffbar") @@ -454,7 +453,7 @@ as f: self.assertEqual(f.read(), "foobar") - def test_open_with_newline(self): + def test_newline(self): # Test with explicit newline (universal newline mode disabled). uncompressed = data1.decode("ascii") * 50 with gzip.open(self.filename, "wt") as f: @@ -463,7 +462,7 @@ self.assertEqual(f.readlines(), [uncompressed]) def test_main(verbose=None): - support.run_unittest(TestGzip) + support.run_unittest(TestGzip, TestOpen) if __name__ == "__main__": test_main(verbose=True) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 15:55:25 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 15:55:25 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0MDM0OiBhZGRl?= =?utf8?q?d_the_argparse_tutorial=2E__Patch_by_Tshepang_Lekhonkhobe=2E?= Message-ID: http://hg.python.org/cpython/rev/48385618525b changeset: 76791:48385618525b branch: 2.7 parent: 76783:2644e4ea02d3 user: Ezio Melotti date: Sun May 06 16:15:35 2012 +0300 summary: #14034: added the argparse tutorial. Patch by Tshepang Lekhonkhobe. files: Doc/howto/argparse.rst | 764 +++++++++++++++++++++++++++ Doc/howto/index.rst | 1 + Doc/library/argparse.rst | 6 + Misc/ACKS | 1 + Misc/NEWS | 5 + 5 files changed, 777 insertions(+), 0 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst new file mode 100644 --- /dev/null +++ b/Doc/howto/argparse.rst @@ -0,0 +1,764 @@ +************************ +:mod:`argparse` Tutorial +************************ + +:author: Tshepang Lekhonkhobe + +.. _argparse-tutorial: + +This tutorial is intended to be a gentle introduction to :mod:`argparse`, the +recommended command-line parsing module in the Python standard library. + +.. note:: + + There's two other modules that fulfill the same task, namely + :mod:`getopt` (an equivalent for :c:func:`getopt` from the C + language) and the deprecated :mod:`optparse`. + Note also that :mod:`argparse` is based on :mod:`optparse`, + and therefore very similar in terms of usage. + + +Concepts +======== + +Let's show the sort of functionality that we are going to explore in this +introductory tutorial by making use of the :command:`ls` command: + +.. code-block:: sh + + $ ls + cpython devguide prog.py pypy rm-unused-function.patch + $ ls pypy + ctypes_configure demo dotviewer include lib_pypy lib-python ... + $ ls -l + total 20 + drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython + drwxr-xr-x 4 wena wena 4096 Feb 8 12:04 devguide + -rwxr-xr-x 1 wena wena 535 Feb 19 00:05 prog.py + drwxr-xr-x 14 wena wena 4096 Feb 7 00:59 pypy + -rw-r--r-- 1 wena wena 741 Feb 18 01:01 rm-unused-function.patch + $ ls --help + Usage: ls [OPTION]... [FILE]... + List information about the FILEs (the current directory by default). + Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. + ... + +A few concepts we can learn from the four commands: + +* The :command:`ls` command is useful when run without any options at all. It defaults + to displaying the contents of the current directory. + +* If we want beyond what it provides by default, we tell it a bit more. In + this case, we want it to display a different directory, ``pypy``. + What we did is specify what is known as a positional argument. It's named so + because the program should know what to do with the value, solely based on + where it appears on the command line. This concept is more relevant + to a command like :command:`cp`, whose most basic usage is ``cp SRC DEST``. + The first position is *what you want copied,* and the second + position is *where you want it copied to*. + +* Now, say we want to change behaviour of the program. In our example, + we display more info for each file instead of just showing the file names. + The ``-l`` in that case is known as an optional argument. + +* That's a snippet of the help text. It's very useful in that you can + come across a program you have never used before, and can figure out + how it works simply by reading it's help text. + + +The basics +========== + +Let us start with a very simple example which does (almost) nothing:: + + import argparse + parser = argparse.ArgumentParser() + parser.parse_args() + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py --verbose + usage: prog.py [-h] + prog.py: error: unrecognized arguments: --verbose + $ python3 prog.py foo + usage: prog.py [-h] + prog.py: error: unrecognized arguments: foo + +Here is what is happening: + +* Running the script without any options results in nothing displayed to + stdout. Not so useful. + +* The second one starts to display the usefulness of the :mod:`argparse` + module. We have done almost nothing, but already we get a nice help message. + +* The ``--help`` option, which can also be shortened to ``-h``, is the only + option we get for free (i.e. no need to specify it). Specifying anything + else results in an error. But even then, we do get a useful usage message, + also for free. + + +Introducing Positional arguments +================================ + +An example:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo") + args = parser.parse_args() + print(args.echo) + +And running the code: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] echo + prog.py: error: the following arguments are required: echo + $ python3 prog.py --help + usage: prog.py [-h] echo + + positional arguments: + echo + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py foo + foo + +Here is what's happening: + +* We've added the :meth:`add_argument` method, which is what we use to specify + which command-line options the program is willing to accept. In this case, + I've named it ``echo`` so that it's in line with its function. + +* Calling our program now requires us to specify an option. + +* The :meth:`parse_args` method actually returns some data from the + options specified, in this case, ``echo``. + +* The variable is some form of 'magic' that :mod:`argparse` performs for free + (i.e. no need to specify which variable that value is stored in). + You will also notice that its name matches the string argument given + to the method, ``echo``. + +Note however that, although the help display looks nice and all, it currently +is not as helpful as it can be. For example we see that we got ``echo`` as a +positional argument, but we don't know what it does, other than by guessing or +by reading the source code. So, let's make it a bit more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo", help="echo the string you use here") + args = parser.parse_args() + print(args.echo) + +And we get: + +.. code-block:: sh + + $ python3 prog.py -h + usage: prog.py [-h] echo + + positional arguments: + echo echo the string you use here + + optional arguments: + -h, --help show this help message and exit + +Now, how about doing something even more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number") + args = parser.parse_args() + print(args.square**2)) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 5, in + print(args.square**2) + TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' + +That didn't go so well. That's because :mod:`argparse` treats the options we +give it as strings, unless we tell it otherwise. So, let's tell +:mod:`argparse` to treat that input as an integer:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number", + type=int) + args = parser.parse_args() + print(args.square**2) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py four + usage: prog.py [-h] square + prog.py: error: argument square: invalid int value: 'four' + +That went well. The program now even helpfully quits on bad illegal input +before proceeding. + + +Introducing Optional arguments +============================== + +So far we, have been playing with positional arguments. Let us +have a look on how to add optional ones:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbosity", help="increase output verbosity") + args = parser.parse_args() + if args.verbosity: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbosity 1 + verbosity turned on + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] [--verbosity VERBOSITY] + + optional arguments: + -h, --help show this help message and exit + --verbosity VERBOSITY + increase output verbosity + $ python3 prog.py --verbosity + usage: prog.py [-h] [--verbosity VERBOSITY] + prog.py: error: argument --verbosity: expected one argument + +Here is what is happening: + +* The program is written so as to display something when ``--verbosity`` is + specified and display nothing when not. + +* To show that the option is actually optional, there is no error when running + the program without it. Note that by default, if an optional argument isn't + used, the relevant variable, in this case :attr:`args.verbosity`, is + given ``None`` as a value, which is the reason it fails the truth + test of the :keyword:`if` statement. + +* The help message is a bit different. + +* When using the ``--verbosity`` option, one must also specify some value, + any value. + +The above example accepts arbitrary integer values for ``--verbosity``, but for +our simple program, only two values are actually useful, ``True`` or ``False``. +Let's modify the code accordingly:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbose + verbosity turned on + $ python3 prog.py --verbose 1 + usage: prog.py [-h] [--verbose] + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py --help + usage: prog.py [-h] [--verbose] + + optional arguments: + -h, --help show this help message and exit + --verbose increase output verbosity + +Here is what is happening: + +* The option is now more of a flag than something that requires a value. + We even changed the name of the option to match that idea. + Note that we now specify a new keyword, ``action``, and give it the value + ``"store_true"``. This means that, if the option is specified, + assign the value ``True`` to :data:`args.verbose`. + Not specifying it implies ``False``. + +* It complains when you specify a value, in true spirit of what flags + actually are. + +* Notice the different help text. + + +Short options +------------- + +If you are familiar with command line usage, +you will notice that I haven't yet touched on the topic of short +versions of the options. It's quite simple:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("-v", "--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And here goes: + +.. code-block:: sh + + $ python3 prog.py -v + verbosity turned on + $ python3 prog.py --help + usage: prog.py [-h] [-v] + + optional arguments: + -h, --help show this help message and exit + -v, --verbose increase output verbosity + +Note that the new ability is also reflected in the help text. + + +Combining Positional and Optional arguments +=========================================== + +Our program keeps growing in complexity:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbose", action="store_true", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbose: + print("the square of {} equals {}".format(args.square, answer)) + else: + print(answer) + +And now the output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] square + prog.py: error: the following arguments are required: square + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 --verbose + the square of 4 equals 16 + $ python3 prog.py --verbose 4 + the square of 4 equals 16 + +* We've brought back a positional argument, hence the complaint. + +* Note that the order does not matter. + +How about we give this program of ours back the ability to have +multiple verbosity values, and actually get to use them:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + usage: prog.py [-h] [-v VERBOSITY] square + prog.py: error: argument -v/--verbosity: expected one argument + $ python3 prog.py 4 -v 1 + 4^2 == 16 + $ python3 prog.py 4 -v 2 + the square of 4 equals 16 + $ python3 prog.py 4 -v 3 + 16 + +These all look good except the last one, which exposes a bug in our program. +Let's fix it by restricting the values the ``--verbosity`` option can accept:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 -v 3 + usage: prog.py [-h] [-v {0,1,2}] square + prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2) + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v {0,1,2}] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} + increase output verbosity + +Note that the change also reflects both in the error message as well as the +help string. + +Now, let's use a different approach of playing with verbosity, which is pretty +common. It also matches the way the CPython executable handles its own +verbosity argument (check the output of ``python --help``):: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display the square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We have introduced another action, "count", +to count the number of occurences of a specific optional arguments: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + 4^2 == 16 + $ python3 prog.py 4 -vv + the square of 4 equals 16 + $ python3 prog.py 4 --verbosity --verbosity + the square of 4 equals 16 + $ python3 prog.py 4 -v 1 + usage: prog.py [-h] [-v] square + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity increase output verbosity + $ python3 prog.py 4 -vvv + 16 + +* Yes, it's now more of a flag (similar to ``action="store_true"``) in the + previous version of our script. That should explain the complaint. + +* It also behaves similar to "store_true" action. + +* Now here's a demonstration of what the "count" action gives. You've probably + seen this sort of usage before. + +* And, just like the "store_true" action, if you don't specify the ``-v`` flag, + that flag is considered to have ``None`` value. + +* As should be expected, specifying the long form of the flag, we should get + the same output. + +* Sadly, our help output isn't very informative on the new ability our script + has acquired, but that can always be fixed by improving the documentation for + out script (e.g. via the ``help`` keyword argument). + +* That last output exposes a bug in our program. + + +Let's fix:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + + # bugfix: replace == with >= + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And this is what it gives: + +.. code-block:: sh + + $ python3 prog.py 4 -vvv + the square of 4 equals 16 + $ python3 prog.py 4 -vvvv + the square of 4 equals 16 + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 11, in + if args.verbosity >= 2: + TypeError: unorderable types: NoneType() >= int() + +* First output went well, and fixes the bug we had before. + That is, we want any value >= 2 to be as verbose as possible. + +* Third output not so good. + +Let's fix that bug:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", default=0, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We've just introduced yet another keyword, ``default``. +We've set it to ``0`` in order to make it comparable to the other int values. +Remember that by default, +if an optional argument isn't specified, +it gets the ``None`` value, and that cannot be compared to an int value +(hence the :exc:`TypeError` exception). + +And: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + +You can go quite far just with what we've learned so far, +and we have only scratched the surface. +The :mod:`argparse` module is very powerful, +and we'll explore a bit more of it before we end this tutorial. + + +Getting a little more advanced +============================== + +What if we wanted to expand our tiny program to perform other powers, +not just squares:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + elif args.verbosity >= 1: + print("{}^{} == {}".format(args.x, args.y, answer)) + else: + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] x y + prog.py: error: the following arguments are required: x, y + $ python3 prog.py -h + usage: prog.py [-h] [-v] x y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity + $ python3 prog.py 4 2 -v + 4^2 == 16 + + +Notice that so far we've been using verbosity level to *change* the text +that gets displayed. The following example instead uses verbosity level +to display *more* text instead:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("Running '{}'".format(__file__)) + if args.verbosity >= 1: + print("{}^{} == ".format(args.x, args.y), end="") + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 16 + $ python3 prog.py 4 2 -v + 4^2 == 16 + $ python3 prog.py 4 2 -vv + Running 'prog.py' + 4^2 == 16 + + +Conflicting options +------------------- + +So far, we have been working with two methods of an +:class:`argparse.ArgumentParser` instance. Let's introduce a third one, +:meth:`add_mutually_exclusive_group`. It allows for us to specify options that +conflict with each other. Let's also change the rest of the program make the +new functionality makes more sense: +we'll introduce the ``--quiet`` option, +which will be the opposite of the ``--verbose`` one:: + + import argparse + + parser = argparse.ArgumentParser() + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Our program is now simpler, and we've lost some functionality for the sake of +demonstration. Anyways, here's the output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 4^2 == 16 + $ python3 prog.py 4 2 -q + 16 + $ python3 prog.py 4 2 -v + 4 to the power 2 equals 16 + $ python3 prog.py 4 2 -vq + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + $ python3 prog.py 4 2 -v --quiet + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + +That should be easy to follow. I've added that last output so you can see the +sort of flexibility you get, i.e. mixing long form options with short form +ones. + +Before we conclude, you probably want to tell your users the main purpose of +your program, just in case they don't know:: + + import argparse + + parser = argparse.ArgumentParser(description="calculate X to the power of Y") + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Note that slight difference in the usage text. Note the ``[-v | -q]``, +which tells us that we can either use ``-v`` or ``-q``, +but not both at the same time: + +.. code-block:: sh + + $ python3 prog.py --help + usage: prog.py [-h] [-v | -q] x y + + calculate X to the power of Y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbose + -q, --quiet + + +Conclusion +========== + +The :mod:`argparse` module offers a lot more than shown here. +Its docs are quite detailed and thorough, and full of examples. +Having gone through this tutorial, you should easily digest them +without feeling overwhelmed. diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -28,4 +28,5 @@ unicode.rst urllib2.rst webservers.rst + argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -12,6 +12,12 @@ -------------- +.. sidebar:: Tutorial + + This page contains the API reference information. For a more gentle + introduction to Python command-line parsing, have a look at the + :ref:`argparse tutorial `. + The :mod:`argparse` module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and :mod:`argparse` will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -500,6 +500,7 @@ Robert Lehmann Petri Lehtinen Luke Kenneth Casson Leighton +Tshepang Lekhonkhobe Marc-Andre Lemburg John Lenton Christopher Tur Lesniewski-Laas diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -148,6 +148,11 @@ - Issue #14437: Fix building the _io module under Cygwin. +Documentation +------------- + +- Issue #14034: added the argparse tutorial. + What's New in Python 2.7.3 release candidate 2? =============================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 15:55:26 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 15:55:26 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0MDM0OiBhZGRl?= =?utf8?q?d_the_argparse_tutorial=2E__Patch_by_Tshepang_Lekhonkhobe=2E?= Message-ID: http://hg.python.org/cpython/rev/11703cb2a2f3 changeset: 76792:11703cb2a2f3 branch: 3.2 parent: 76782:e275a9f7daa9 user: Ezio Melotti date: Sun May 06 16:15:35 2012 +0300 summary: #14034: added the argparse tutorial. Patch by Tshepang Lekhonkhobe. files: Doc/howto/argparse.rst | 764 +++++++++++++++++++++++++++ Doc/howto/index.rst | 1 + Doc/library/argparse.rst | 6 + Misc/ACKS | 1 + Misc/NEWS | 5 + 5 files changed, 777 insertions(+), 0 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst new file mode 100644 --- /dev/null +++ b/Doc/howto/argparse.rst @@ -0,0 +1,764 @@ +************************ +:mod:`argparse` Tutorial +************************ + +:author: Tshepang Lekhonkhobe + +.. _argparse-tutorial: + +This tutorial is intended to be a gentle introduction to :mod:`argparse`, the +recommended command-line parsing module in the Python standard library. + +.. note:: + + There's two other modules that fulfill the same task, namely + :mod:`getopt` (an equivalent for :c:func:`getopt` from the C + language) and the deprecated :mod:`optparse`. + Note also that :mod:`argparse` is based on :mod:`optparse`, + and therefore very similar in terms of usage. + + +Concepts +======== + +Let's show the sort of functionality that we are going to explore in this +introductory tutorial by making use of the :command:`ls` command: + +.. code-block:: sh + + $ ls + cpython devguide prog.py pypy rm-unused-function.patch + $ ls pypy + ctypes_configure demo dotviewer include lib_pypy lib-python ... + $ ls -l + total 20 + drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython + drwxr-xr-x 4 wena wena 4096 Feb 8 12:04 devguide + -rwxr-xr-x 1 wena wena 535 Feb 19 00:05 prog.py + drwxr-xr-x 14 wena wena 4096 Feb 7 00:59 pypy + -rw-r--r-- 1 wena wena 741 Feb 18 01:01 rm-unused-function.patch + $ ls --help + Usage: ls [OPTION]... [FILE]... + List information about the FILEs (the current directory by default). + Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. + ... + +A few concepts we can learn from the four commands: + +* The :command:`ls` command is useful when run without any options at all. It defaults + to displaying the contents of the current directory. + +* If we want beyond what it provides by default, we tell it a bit more. In + this case, we want it to display a different directory, ``pypy``. + What we did is specify what is known as a positional argument. It's named so + because the program should know what to do with the value, solely based on + where it appears on the command line. This concept is more relevant + to a command like :command:`cp`, whose most basic usage is ``cp SRC DEST``. + The first position is *what you want copied,* and the second + position is *where you want it copied to*. + +* Now, say we want to change behaviour of the program. In our example, + we display more info for each file instead of just showing the file names. + The ``-l`` in that case is known as an optional argument. + +* That's a snippet of the help text. It's very useful in that you can + come across a program you have never used before, and can figure out + how it works simply by reading it's help text. + + +The basics +========== + +Let us start with a very simple example which does (almost) nothing:: + + import argparse + parser = argparse.ArgumentParser() + parser.parse_args() + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py --verbose + usage: prog.py [-h] + prog.py: error: unrecognized arguments: --verbose + $ python3 prog.py foo + usage: prog.py [-h] + prog.py: error: unrecognized arguments: foo + +Here is what is happening: + +* Running the script without any options results in nothing displayed to + stdout. Not so useful. + +* The second one starts to display the usefulness of the :mod:`argparse` + module. We have done almost nothing, but already we get a nice help message. + +* The ``--help`` option, which can also be shortened to ``-h``, is the only + option we get for free (i.e. no need to specify it). Specifying anything + else results in an error. But even then, we do get a useful usage message, + also for free. + + +Introducing Positional arguments +================================ + +An example:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo") + args = parser.parse_args() + print(args.echo) + +And running the code: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] echo + prog.py: error: the following arguments are required: echo + $ python3 prog.py --help + usage: prog.py [-h] echo + + positional arguments: + echo + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py foo + foo + +Here is what's happening: + +* We've added the :meth:`add_argument` method, which is what we use to specify + which command-line options the program is willing to accept. In this case, + I've named it ``echo`` so that it's in line with its function. + +* Calling our program now requires us to specify an option. + +* The :meth:`parse_args` method actually returns some data from the + options specified, in this case, ``echo``. + +* The variable is some form of 'magic' that :mod:`argparse` performs for free + (i.e. no need to specify which variable that value is stored in). + You will also notice that its name matches the string argument given + to the method, ``echo``. + +Note however that, although the help display looks nice and all, it currently +is not as helpful as it can be. For example we see that we got ``echo`` as a +positional argument, but we don't know what it does, other than by guessing or +by reading the source code. So, let's make it a bit more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo", help="echo the string you use here") + args = parser.parse_args() + print(args.echo) + +And we get: + +.. code-block:: sh + + $ python3 prog.py -h + usage: prog.py [-h] echo + + positional arguments: + echo echo the string you use here + + optional arguments: + -h, --help show this help message and exit + +Now, how about doing something even more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number") + args = parser.parse_args() + print(args.square**2)) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 5, in + print(args.square**2) + TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' + +That didn't go so well. That's because :mod:`argparse` treats the options we +give it as strings, unless we tell it otherwise. So, let's tell +:mod:`argparse` to treat that input as an integer:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number", + type=int) + args = parser.parse_args() + print(args.square**2) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py four + usage: prog.py [-h] square + prog.py: error: argument square: invalid int value: 'four' + +That went well. The program now even helpfully quits on bad illegal input +before proceeding. + + +Introducing Optional arguments +============================== + +So far we, have been playing with positional arguments. Let us +have a look on how to add optional ones:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbosity", help="increase output verbosity") + args = parser.parse_args() + if args.verbosity: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbosity 1 + verbosity turned on + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] [--verbosity VERBOSITY] + + optional arguments: + -h, --help show this help message and exit + --verbosity VERBOSITY + increase output verbosity + $ python3 prog.py --verbosity + usage: prog.py [-h] [--verbosity VERBOSITY] + prog.py: error: argument --verbosity: expected one argument + +Here is what is happening: + +* The program is written so as to display something when ``--verbosity`` is + specified and display nothing when not. + +* To show that the option is actually optional, there is no error when running + the program without it. Note that by default, if an optional argument isn't + used, the relevant variable, in this case :attr:`args.verbosity`, is + given ``None`` as a value, which is the reason it fails the truth + test of the :keyword:`if` statement. + +* The help message is a bit different. + +* When using the ``--verbosity`` option, one must also specify some value, + any value. + +The above example accepts arbitrary integer values for ``--verbosity``, but for +our simple program, only two values are actually useful, ``True`` or ``False``. +Let's modify the code accordingly:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbose + verbosity turned on + $ python3 prog.py --verbose 1 + usage: prog.py [-h] [--verbose] + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py --help + usage: prog.py [-h] [--verbose] + + optional arguments: + -h, --help show this help message and exit + --verbose increase output verbosity + +Here is what is happening: + +* The option is now more of a flag than something that requires a value. + We even changed the name of the option to match that idea. + Note that we now specify a new keyword, ``action``, and give it the value + ``"store_true"``. This means that, if the option is specified, + assign the value ``True`` to :data:`args.verbose`. + Not specifying it implies ``False``. + +* It complains when you specify a value, in true spirit of what flags + actually are. + +* Notice the different help text. + + +Short options +------------- + +If you are familiar with command line usage, +you will notice that I haven't yet touched on the topic of short +versions of the options. It's quite simple:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("-v", "--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And here goes: + +.. code-block:: sh + + $ python3 prog.py -v + verbosity turned on + $ python3 prog.py --help + usage: prog.py [-h] [-v] + + optional arguments: + -h, --help show this help message and exit + -v, --verbose increase output verbosity + +Note that the new ability is also reflected in the help text. + + +Combining Positional and Optional arguments +=========================================== + +Our program keeps growing in complexity:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbose", action="store_true", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbose: + print("the square of {} equals {}".format(args.square, answer)) + else: + print(answer) + +And now the output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] square + prog.py: error: the following arguments are required: square + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 --verbose + the square of 4 equals 16 + $ python3 prog.py --verbose 4 + the square of 4 equals 16 + +* We've brought back a positional argument, hence the complaint. + +* Note that the order does not matter. + +How about we give this program of ours back the ability to have +multiple verbosity values, and actually get to use them:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + usage: prog.py [-h] [-v VERBOSITY] square + prog.py: error: argument -v/--verbosity: expected one argument + $ python3 prog.py 4 -v 1 + 4^2 == 16 + $ python3 prog.py 4 -v 2 + the square of 4 equals 16 + $ python3 prog.py 4 -v 3 + 16 + +These all look good except the last one, which exposes a bug in our program. +Let's fix it by restricting the values the ``--verbosity`` option can accept:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 -v 3 + usage: prog.py [-h] [-v {0,1,2}] square + prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2) + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v {0,1,2}] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} + increase output verbosity + +Note that the change also reflects both in the error message as well as the +help string. + +Now, let's use a different approach of playing with verbosity, which is pretty +common. It also matches the way the CPython executable handles its own +verbosity argument (check the output of ``python --help``):: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display the square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We have introduced another action, "count", +to count the number of occurences of a specific optional arguments: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + 4^2 == 16 + $ python3 prog.py 4 -vv + the square of 4 equals 16 + $ python3 prog.py 4 --verbosity --verbosity + the square of 4 equals 16 + $ python3 prog.py 4 -v 1 + usage: prog.py [-h] [-v] square + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity increase output verbosity + $ python3 prog.py 4 -vvv + 16 + +* Yes, it's now more of a flag (similar to ``action="store_true"``) in the + previous version of our script. That should explain the complaint. + +* It also behaves similar to "store_true" action. + +* Now here's a demonstration of what the "count" action gives. You've probably + seen this sort of usage before. + +* And, just like the "store_true" action, if you don't specify the ``-v`` flag, + that flag is considered to have ``None`` value. + +* As should be expected, specifying the long form of the flag, we should get + the same output. + +* Sadly, our help output isn't very informative on the new ability our script + has acquired, but that can always be fixed by improving the documentation for + out script (e.g. via the ``help`` keyword argument). + +* That last output exposes a bug in our program. + + +Let's fix:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + + # bugfix: replace == with >= + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And this is what it gives: + +.. code-block:: sh + + $ python3 prog.py 4 -vvv + the square of 4 equals 16 + $ python3 prog.py 4 -vvvv + the square of 4 equals 16 + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 11, in + if args.verbosity >= 2: + TypeError: unorderable types: NoneType() >= int() + +* First output went well, and fixes the bug we had before. + That is, we want any value >= 2 to be as verbose as possible. + +* Third output not so good. + +Let's fix that bug:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", default=0, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We've just introduced yet another keyword, ``default``. +We've set it to ``0`` in order to make it comparable to the other int values. +Remember that by default, +if an optional argument isn't specified, +it gets the ``None`` value, and that cannot be compared to an int value +(hence the :exc:`TypeError` exception). + +And: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + +You can go quite far just with what we've learned so far, +and we have only scratched the surface. +The :mod:`argparse` module is very powerful, +and we'll explore a bit more of it before we end this tutorial. + + +Getting a little more advanced +============================== + +What if we wanted to expand our tiny program to perform other powers, +not just squares:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + elif args.verbosity >= 1: + print("{}^{} == {}".format(args.x, args.y, answer)) + else: + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] x y + prog.py: error: the following arguments are required: x, y + $ python3 prog.py -h + usage: prog.py [-h] [-v] x y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity + $ python3 prog.py 4 2 -v + 4^2 == 16 + + +Notice that so far we've been using verbosity level to *change* the text +that gets displayed. The following example instead uses verbosity level +to display *more* text instead:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("Running '{}'".format(__file__)) + if args.verbosity >= 1: + print("{}^{} == ".format(args.x, args.y), end="") + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 16 + $ python3 prog.py 4 2 -v + 4^2 == 16 + $ python3 prog.py 4 2 -vv + Running 'prog.py' + 4^2 == 16 + + +Conflicting options +------------------- + +So far, we have been working with two methods of an +:class:`argparse.ArgumentParser` instance. Let's introduce a third one, +:meth:`add_mutually_exclusive_group`. It allows for us to specify options that +conflict with each other. Let's also change the rest of the program make the +new functionality makes more sense: +we'll introduce the ``--quiet`` option, +which will be the opposite of the ``--verbose`` one:: + + import argparse + + parser = argparse.ArgumentParser() + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Our program is now simpler, and we've lost some functionality for the sake of +demonstration. Anyways, here's the output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 4^2 == 16 + $ python3 prog.py 4 2 -q + 16 + $ python3 prog.py 4 2 -v + 4 to the power 2 equals 16 + $ python3 prog.py 4 2 -vq + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + $ python3 prog.py 4 2 -v --quiet + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + +That should be easy to follow. I've added that last output so you can see the +sort of flexibility you get, i.e. mixing long form options with short form +ones. + +Before we conclude, you probably want to tell your users the main purpose of +your program, just in case they don't know:: + + import argparse + + parser = argparse.ArgumentParser(description="calculate X to the power of Y") + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Note that slight difference in the usage text. Note the ``[-v | -q]``, +which tells us that we can either use ``-v`` or ``-q``, +but not both at the same time: + +.. code-block:: sh + + $ python3 prog.py --help + usage: prog.py [-h] [-v | -q] x y + + calculate X to the power of Y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbose + -q, --quiet + + +Conclusion +========== + +The :mod:`argparse` module offers a lot more than shown here. +Its docs are quite detailed and thorough, and full of examples. +Having gone through this tutorial, you should easily digest them +without feeling overwhelmed. diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -27,4 +27,5 @@ unicode.rst urllib2.rst webservers.rst + argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -12,6 +12,12 @@ -------------- +.. sidebar:: Tutorial + + This page contains the API reference information. For a more gentle + introduction to Python command-line parsing, have a look at the + :ref:`argparse tutorial `. + The :mod:`argparse` module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and :mod:`argparse` will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -550,6 +550,7 @@ Robert Lehmann Petri Lehtinen Luke Kenneth Casson Leighton +Tshepang Lekhonkhobe Marc-Andre Lemburg John Lenton Christopher Tur Lesniewski-Laas diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -220,6 +220,11 @@ - Issue #8799: Fix and improve the threading.Condition documentation. +Documentation +------------- + +- Issue #14034: added the argparse tutorial. + What's New in Python 3.2.3 release candidate 2? =============================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 15:55:27 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 15:55:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314034=3A_merge_argparse_tutorial_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/645969f4193b changeset: 76793:645969f4193b parent: 76789:55202ca694d7 parent: 76792:11703cb2a2f3 user: Ezio Melotti date: Sun May 06 16:34:43 2012 +0300 summary: #14034: merge argparse tutorial from 3.2. files: Doc/howto/argparse.rst | 764 +++++++++++++++++++++++++++ Doc/howto/index.rst | 1 + Doc/library/argparse.rst | 6 + Misc/NEWS | 5 + 4 files changed, 776 insertions(+), 0 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst new file mode 100644 --- /dev/null +++ b/Doc/howto/argparse.rst @@ -0,0 +1,764 @@ +************************ +:mod:`argparse` Tutorial +************************ + +:author: Tshepang Lekhonkhobe + +.. _argparse-tutorial: + +This tutorial is intended to be a gentle introduction to :mod:`argparse`, the +recommended command-line parsing module in the Python standard library. + +.. note:: + + There's two other modules that fulfill the same task, namely + :mod:`getopt` (an equivalent for :c:func:`getopt` from the C + language) and the deprecated :mod:`optparse`. + Note also that :mod:`argparse` is based on :mod:`optparse`, + and therefore very similar in terms of usage. + + +Concepts +======== + +Let's show the sort of functionality that we are going to explore in this +introductory tutorial by making use of the :command:`ls` command: + +.. code-block:: sh + + $ ls + cpython devguide prog.py pypy rm-unused-function.patch + $ ls pypy + ctypes_configure demo dotviewer include lib_pypy lib-python ... + $ ls -l + total 20 + drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython + drwxr-xr-x 4 wena wena 4096 Feb 8 12:04 devguide + -rwxr-xr-x 1 wena wena 535 Feb 19 00:05 prog.py + drwxr-xr-x 14 wena wena 4096 Feb 7 00:59 pypy + -rw-r--r-- 1 wena wena 741 Feb 18 01:01 rm-unused-function.patch + $ ls --help + Usage: ls [OPTION]... [FILE]... + List information about the FILEs (the current directory by default). + Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. + ... + +A few concepts we can learn from the four commands: + +* The :command:`ls` command is useful when run without any options at all. It defaults + to displaying the contents of the current directory. + +* If we want beyond what it provides by default, we tell it a bit more. In + this case, we want it to display a different directory, ``pypy``. + What we did is specify what is known as a positional argument. It's named so + because the program should know what to do with the value, solely based on + where it appears on the command line. This concept is more relevant + to a command like :command:`cp`, whose most basic usage is ``cp SRC DEST``. + The first position is *what you want copied,* and the second + position is *where you want it copied to*. + +* Now, say we want to change behaviour of the program. In our example, + we display more info for each file instead of just showing the file names. + The ``-l`` in that case is known as an optional argument. + +* That's a snippet of the help text. It's very useful in that you can + come across a program you have never used before, and can figure out + how it works simply by reading it's help text. + + +The basics +========== + +Let us start with a very simple example which does (almost) nothing:: + + import argparse + parser = argparse.ArgumentParser() + parser.parse_args() + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py --verbose + usage: prog.py [-h] + prog.py: error: unrecognized arguments: --verbose + $ python3 prog.py foo + usage: prog.py [-h] + prog.py: error: unrecognized arguments: foo + +Here is what is happening: + +* Running the script without any options results in nothing displayed to + stdout. Not so useful. + +* The second one starts to display the usefulness of the :mod:`argparse` + module. We have done almost nothing, but already we get a nice help message. + +* The ``--help`` option, which can also be shortened to ``-h``, is the only + option we get for free (i.e. no need to specify it). Specifying anything + else results in an error. But even then, we do get a useful usage message, + also for free. + + +Introducing Positional arguments +================================ + +An example:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo") + args = parser.parse_args() + print(args.echo) + +And running the code: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] echo + prog.py: error: the following arguments are required: echo + $ python3 prog.py --help + usage: prog.py [-h] echo + + positional arguments: + echo + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py foo + foo + +Here is what's happening: + +* We've added the :meth:`add_argument` method, which is what we use to specify + which command-line options the program is willing to accept. In this case, + I've named it ``echo`` so that it's in line with its function. + +* Calling our program now requires us to specify an option. + +* The :meth:`parse_args` method actually returns some data from the + options specified, in this case, ``echo``. + +* The variable is some form of 'magic' that :mod:`argparse` performs for free + (i.e. no need to specify which variable that value is stored in). + You will also notice that its name matches the string argument given + to the method, ``echo``. + +Note however that, although the help display looks nice and all, it currently +is not as helpful as it can be. For example we see that we got ``echo`` as a +positional argument, but we don't know what it does, other than by guessing or +by reading the source code. So, let's make it a bit more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo", help="echo the string you use here") + args = parser.parse_args() + print(args.echo) + +And we get: + +.. code-block:: sh + + $ python3 prog.py -h + usage: prog.py [-h] echo + + positional arguments: + echo echo the string you use here + + optional arguments: + -h, --help show this help message and exit + +Now, how about doing something even more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number") + args = parser.parse_args() + print(args.square**2)) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 5, in + print(args.square**2) + TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' + +That didn't go so well. That's because :mod:`argparse` treats the options we +give it as strings, unless we tell it otherwise. So, let's tell +:mod:`argparse` to treat that input as an integer:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number", + type=int) + args = parser.parse_args() + print(args.square**2) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py four + usage: prog.py [-h] square + prog.py: error: argument square: invalid int value: 'four' + +That went well. The program now even helpfully quits on bad illegal input +before proceeding. + + +Introducing Optional arguments +============================== + +So far we, have been playing with positional arguments. Let us +have a look on how to add optional ones:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbosity", help="increase output verbosity") + args = parser.parse_args() + if args.verbosity: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbosity 1 + verbosity turned on + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] [--verbosity VERBOSITY] + + optional arguments: + -h, --help show this help message and exit + --verbosity VERBOSITY + increase output verbosity + $ python3 prog.py --verbosity + usage: prog.py [-h] [--verbosity VERBOSITY] + prog.py: error: argument --verbosity: expected one argument + +Here is what is happening: + +* The program is written so as to display something when ``--verbosity`` is + specified and display nothing when not. + +* To show that the option is actually optional, there is no error when running + the program without it. Note that by default, if an optional argument isn't + used, the relevant variable, in this case :attr:`args.verbosity`, is + given ``None`` as a value, which is the reason it fails the truth + test of the :keyword:`if` statement. + +* The help message is a bit different. + +* When using the ``--verbosity`` option, one must also specify some value, + any value. + +The above example accepts arbitrary integer values for ``--verbosity``, but for +our simple program, only two values are actually useful, ``True`` or ``False``. +Let's modify the code accordingly:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbose + verbosity turned on + $ python3 prog.py --verbose 1 + usage: prog.py [-h] [--verbose] + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py --help + usage: prog.py [-h] [--verbose] + + optional arguments: + -h, --help show this help message and exit + --verbose increase output verbosity + +Here is what is happening: + +* The option is now more of a flag than something that requires a value. + We even changed the name of the option to match that idea. + Note that we now specify a new keyword, ``action``, and give it the value + ``"store_true"``. This means that, if the option is specified, + assign the value ``True`` to :data:`args.verbose`. + Not specifying it implies ``False``. + +* It complains when you specify a value, in true spirit of what flags + actually are. + +* Notice the different help text. + + +Short options +------------- + +If you are familiar with command line usage, +you will notice that I haven't yet touched on the topic of short +versions of the options. It's quite simple:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("-v", "--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And here goes: + +.. code-block:: sh + + $ python3 prog.py -v + verbosity turned on + $ python3 prog.py --help + usage: prog.py [-h] [-v] + + optional arguments: + -h, --help show this help message and exit + -v, --verbose increase output verbosity + +Note that the new ability is also reflected in the help text. + + +Combining Positional and Optional arguments +=========================================== + +Our program keeps growing in complexity:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbose", action="store_true", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbose: + print("the square of {} equals {}".format(args.square, answer)) + else: + print(answer) + +And now the output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] square + prog.py: error: the following arguments are required: square + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 --verbose + the square of 4 equals 16 + $ python3 prog.py --verbose 4 + the square of 4 equals 16 + +* We've brought back a positional argument, hence the complaint. + +* Note that the order does not matter. + +How about we give this program of ours back the ability to have +multiple verbosity values, and actually get to use them:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + usage: prog.py [-h] [-v VERBOSITY] square + prog.py: error: argument -v/--verbosity: expected one argument + $ python3 prog.py 4 -v 1 + 4^2 == 16 + $ python3 prog.py 4 -v 2 + the square of 4 equals 16 + $ python3 prog.py 4 -v 3 + 16 + +These all look good except the last one, which exposes a bug in our program. +Let's fix it by restricting the values the ``--verbosity`` option can accept:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 -v 3 + usage: prog.py [-h] [-v {0,1,2}] square + prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2) + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v {0,1,2}] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} + increase output verbosity + +Note that the change also reflects both in the error message as well as the +help string. + +Now, let's use a different approach of playing with verbosity, which is pretty +common. It also matches the way the CPython executable handles its own +verbosity argument (check the output of ``python --help``):: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display the square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We have introduced another action, "count", +to count the number of occurences of a specific optional arguments: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + 4^2 == 16 + $ python3 prog.py 4 -vv + the square of 4 equals 16 + $ python3 prog.py 4 --verbosity --verbosity + the square of 4 equals 16 + $ python3 prog.py 4 -v 1 + usage: prog.py [-h] [-v] square + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity increase output verbosity + $ python3 prog.py 4 -vvv + 16 + +* Yes, it's now more of a flag (similar to ``action="store_true"``) in the + previous version of our script. That should explain the complaint. + +* It also behaves similar to "store_true" action. + +* Now here's a demonstration of what the "count" action gives. You've probably + seen this sort of usage before. + +* And, just like the "store_true" action, if you don't specify the ``-v`` flag, + that flag is considered to have ``None`` value. + +* As should be expected, specifying the long form of the flag, we should get + the same output. + +* Sadly, our help output isn't very informative on the new ability our script + has acquired, but that can always be fixed by improving the documentation for + out script (e.g. via the ``help`` keyword argument). + +* That last output exposes a bug in our program. + + +Let's fix:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + + # bugfix: replace == with >= + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And this is what it gives: + +.. code-block:: sh + + $ python3 prog.py 4 -vvv + the square of 4 equals 16 + $ python3 prog.py 4 -vvvv + the square of 4 equals 16 + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 11, in + if args.verbosity >= 2: + TypeError: unorderable types: NoneType() >= int() + +* First output went well, and fixes the bug we had before. + That is, we want any value >= 2 to be as verbose as possible. + +* Third output not so good. + +Let's fix that bug:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", default=0, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We've just introduced yet another keyword, ``default``. +We've set it to ``0`` in order to make it comparable to the other int values. +Remember that by default, +if an optional argument isn't specified, +it gets the ``None`` value, and that cannot be compared to an int value +(hence the :exc:`TypeError` exception). + +And: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + +You can go quite far just with what we've learned so far, +and we have only scratched the surface. +The :mod:`argparse` module is very powerful, +and we'll explore a bit more of it before we end this tutorial. + + +Getting a little more advanced +============================== + +What if we wanted to expand our tiny program to perform other powers, +not just squares:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + elif args.verbosity >= 1: + print("{}^{} == {}".format(args.x, args.y, answer)) + else: + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] x y + prog.py: error: the following arguments are required: x, y + $ python3 prog.py -h + usage: prog.py [-h] [-v] x y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity + $ python3 prog.py 4 2 -v + 4^2 == 16 + + +Notice that so far we've been using verbosity level to *change* the text +that gets displayed. The following example instead uses verbosity level +to display *more* text instead:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("Running '{}'".format(__file__)) + if args.verbosity >= 1: + print("{}^{} == ".format(args.x, args.y), end="") + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 16 + $ python3 prog.py 4 2 -v + 4^2 == 16 + $ python3 prog.py 4 2 -vv + Running 'prog.py' + 4^2 == 16 + + +Conflicting options +------------------- + +So far, we have been working with two methods of an +:class:`argparse.ArgumentParser` instance. Let's introduce a third one, +:meth:`add_mutually_exclusive_group`. It allows for us to specify options that +conflict with each other. Let's also change the rest of the program make the +new functionality makes more sense: +we'll introduce the ``--quiet`` option, +which will be the opposite of the ``--verbose`` one:: + + import argparse + + parser = argparse.ArgumentParser() + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Our program is now simpler, and we've lost some functionality for the sake of +demonstration. Anyways, here's the output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 4^2 == 16 + $ python3 prog.py 4 2 -q + 16 + $ python3 prog.py 4 2 -v + 4 to the power 2 equals 16 + $ python3 prog.py 4 2 -vq + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + $ python3 prog.py 4 2 -v --quiet + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + +That should be easy to follow. I've added that last output so you can see the +sort of flexibility you get, i.e. mixing long form options with short form +ones. + +Before we conclude, you probably want to tell your users the main purpose of +your program, just in case they don't know:: + + import argparse + + parser = argparse.ArgumentParser(description="calculate X to the power of Y") + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Note that slight difference in the usage text. Note the ``[-v | -q]``, +which tells us that we can either use ``-v`` or ``-q``, +but not both at the same time: + +.. code-block:: sh + + $ python3 prog.py --help + usage: prog.py [-h] [-v | -q] x y + + calculate X to the power of Y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbose + -q, --quiet + + +Conclusion +========== + +The :mod:`argparse` module offers a lot more than shown here. +Its docs are quite detailed and thorough, and full of examples. +Having gone through this tutorial, you should easily digest them +without feeling overwhelmed. diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -27,4 +27,5 @@ unicode.rst urllib2.rst webservers.rst + argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -12,6 +12,12 @@ -------------- +.. sidebar:: Tutorial + + This page contains the API reference information. For a more gentle + introduction to Python command-line parsing, have a look at the + :ref:`argparse tutorial `. + The :mod:`argparse` module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and :mod:`argparse` will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -480,6 +480,11 @@ Based on patch from Herv? Coatanhay. - Issue #14321: Do not run pgen during the build if files are up to date. +Documentation +------------- + +- Issue #14034: added the argparse tutorial. + - Issue #14324: Fix configure tests for cross builds. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 15:55:28 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 15:55:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?b?KTogTWVyZ2UgaGVhZHMu?= Message-ID: http://hg.python.org/cpython/rev/50a83d63cb40 changeset: 76794:50a83d63cb40 parent: 76790:d75934e88234 parent: 76793:645969f4193b user: Ezio Melotti date: Sun May 06 16:55:17 2012 +0300 summary: Merge heads. files: Doc/howto/argparse.rst | 764 +++++++++++++++++++++++++++ Doc/howto/index.rst | 1 + Doc/library/argparse.rst | 6 + Misc/NEWS | 5 + 4 files changed, 776 insertions(+), 0 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst new file mode 100644 --- /dev/null +++ b/Doc/howto/argparse.rst @@ -0,0 +1,764 @@ +************************ +:mod:`argparse` Tutorial +************************ + +:author: Tshepang Lekhonkhobe + +.. _argparse-tutorial: + +This tutorial is intended to be a gentle introduction to :mod:`argparse`, the +recommended command-line parsing module in the Python standard library. + +.. note:: + + There's two other modules that fulfill the same task, namely + :mod:`getopt` (an equivalent for :c:func:`getopt` from the C + language) and the deprecated :mod:`optparse`. + Note also that :mod:`argparse` is based on :mod:`optparse`, + and therefore very similar in terms of usage. + + +Concepts +======== + +Let's show the sort of functionality that we are going to explore in this +introductory tutorial by making use of the :command:`ls` command: + +.. code-block:: sh + + $ ls + cpython devguide prog.py pypy rm-unused-function.patch + $ ls pypy + ctypes_configure demo dotviewer include lib_pypy lib-python ... + $ ls -l + total 20 + drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython + drwxr-xr-x 4 wena wena 4096 Feb 8 12:04 devguide + -rwxr-xr-x 1 wena wena 535 Feb 19 00:05 prog.py + drwxr-xr-x 14 wena wena 4096 Feb 7 00:59 pypy + -rw-r--r-- 1 wena wena 741 Feb 18 01:01 rm-unused-function.patch + $ ls --help + Usage: ls [OPTION]... [FILE]... + List information about the FILEs (the current directory by default). + Sort entries alphabetically if none of -cftuvSUX nor --sort is specified. + ... + +A few concepts we can learn from the four commands: + +* The :command:`ls` command is useful when run without any options at all. It defaults + to displaying the contents of the current directory. + +* If we want beyond what it provides by default, we tell it a bit more. In + this case, we want it to display a different directory, ``pypy``. + What we did is specify what is known as a positional argument. It's named so + because the program should know what to do with the value, solely based on + where it appears on the command line. This concept is more relevant + to a command like :command:`cp`, whose most basic usage is ``cp SRC DEST``. + The first position is *what you want copied,* and the second + position is *where you want it copied to*. + +* Now, say we want to change behaviour of the program. In our example, + we display more info for each file instead of just showing the file names. + The ``-l`` in that case is known as an optional argument. + +* That's a snippet of the help text. It's very useful in that you can + come across a program you have never used before, and can figure out + how it works simply by reading it's help text. + + +The basics +========== + +Let us start with a very simple example which does (almost) nothing:: + + import argparse + parser = argparse.ArgumentParser() + parser.parse_args() + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py --verbose + usage: prog.py [-h] + prog.py: error: unrecognized arguments: --verbose + $ python3 prog.py foo + usage: prog.py [-h] + prog.py: error: unrecognized arguments: foo + +Here is what is happening: + +* Running the script without any options results in nothing displayed to + stdout. Not so useful. + +* The second one starts to display the usefulness of the :mod:`argparse` + module. We have done almost nothing, but already we get a nice help message. + +* The ``--help`` option, which can also be shortened to ``-h``, is the only + option we get for free (i.e. no need to specify it). Specifying anything + else results in an error. But even then, we do get a useful usage message, + also for free. + + +Introducing Positional arguments +================================ + +An example:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo") + args = parser.parse_args() + print(args.echo) + +And running the code: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] echo + prog.py: error: the following arguments are required: echo + $ python3 prog.py --help + usage: prog.py [-h] echo + + positional arguments: + echo + + optional arguments: + -h, --help show this help message and exit + $ python3 prog.py foo + foo + +Here is what's happening: + +* We've added the :meth:`add_argument` method, which is what we use to specify + which command-line options the program is willing to accept. In this case, + I've named it ``echo`` so that it's in line with its function. + +* Calling our program now requires us to specify an option. + +* The :meth:`parse_args` method actually returns some data from the + options specified, in this case, ``echo``. + +* The variable is some form of 'magic' that :mod:`argparse` performs for free + (i.e. no need to specify which variable that value is stored in). + You will also notice that its name matches the string argument given + to the method, ``echo``. + +Note however that, although the help display looks nice and all, it currently +is not as helpful as it can be. For example we see that we got ``echo`` as a +positional argument, but we don't know what it does, other than by guessing or +by reading the source code. So, let's make it a bit more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("echo", help="echo the string you use here") + args = parser.parse_args() + print(args.echo) + +And we get: + +.. code-block:: sh + + $ python3 prog.py -h + usage: prog.py [-h] echo + + positional arguments: + echo echo the string you use here + + optional arguments: + -h, --help show this help message and exit + +Now, how about doing something even more useful:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number") + args = parser.parse_args() + print(args.square**2)) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 5, in + print(args.square**2) + TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' + +That didn't go so well. That's because :mod:`argparse` treats the options we +give it as strings, unless we tell it otherwise. So, let's tell +:mod:`argparse` to treat that input as an integer:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", help="display a square of a given number", + type=int) + args = parser.parse_args() + print(args.square**2) + +Following is a result of running the code: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py four + usage: prog.py [-h] square + prog.py: error: argument square: invalid int value: 'four' + +That went well. The program now even helpfully quits on bad illegal input +before proceeding. + + +Introducing Optional arguments +============================== + +So far we, have been playing with positional arguments. Let us +have a look on how to add optional ones:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbosity", help="increase output verbosity") + args = parser.parse_args() + if args.verbosity: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbosity 1 + verbosity turned on + $ python3 prog.py + $ python3 prog.py --help + usage: prog.py [-h] [--verbosity VERBOSITY] + + optional arguments: + -h, --help show this help message and exit + --verbosity VERBOSITY + increase output verbosity + $ python3 prog.py --verbosity + usage: prog.py [-h] [--verbosity VERBOSITY] + prog.py: error: argument --verbosity: expected one argument + +Here is what is happening: + +* The program is written so as to display something when ``--verbosity`` is + specified and display nothing when not. + +* To show that the option is actually optional, there is no error when running + the program without it. Note that by default, if an optional argument isn't + used, the relevant variable, in this case :attr:`args.verbosity`, is + given ``None`` as a value, which is the reason it fails the truth + test of the :keyword:`if` statement. + +* The help message is a bit different. + +* When using the ``--verbosity`` option, one must also specify some value, + any value. + +The above example accepts arbitrary integer values for ``--verbosity``, but for +our simple program, only two values are actually useful, ``True`` or ``False``. +Let's modify the code accordingly:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And the output: + +.. code-block:: sh + + $ python3 prog.py --verbose + verbosity turned on + $ python3 prog.py --verbose 1 + usage: prog.py [-h] [--verbose] + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py --help + usage: prog.py [-h] [--verbose] + + optional arguments: + -h, --help show this help message and exit + --verbose increase output verbosity + +Here is what is happening: + +* The option is now more of a flag than something that requires a value. + We even changed the name of the option to match that idea. + Note that we now specify a new keyword, ``action``, and give it the value + ``"store_true"``. This means that, if the option is specified, + assign the value ``True`` to :data:`args.verbose`. + Not specifying it implies ``False``. + +* It complains when you specify a value, in true spirit of what flags + actually are. + +* Notice the different help text. + + +Short options +------------- + +If you are familiar with command line usage, +you will notice that I haven't yet touched on the topic of short +versions of the options. It's quite simple:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("-v", "--verbose", help="increase output verbosity", + action="store_true") + args = parser.parse_args() + if args.verbose: + print("verbosity turned on") + +And here goes: + +.. code-block:: sh + + $ python3 prog.py -v + verbosity turned on + $ python3 prog.py --help + usage: prog.py [-h] [-v] + + optional arguments: + -h, --help show this help message and exit + -v, --verbose increase output verbosity + +Note that the new ability is also reflected in the help text. + + +Combining Positional and Optional arguments +=========================================== + +Our program keeps growing in complexity:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbose", action="store_true", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbose: + print("the square of {} equals {}".format(args.square, answer)) + else: + print(answer) + +And now the output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] square + prog.py: error: the following arguments are required: square + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 --verbose + the square of 4 equals 16 + $ python3 prog.py --verbose 4 + the square of 4 equals 16 + +* We've brought back a positional argument, hence the complaint. + +* Note that the order does not matter. + +How about we give this program of ours back the ability to have +multiple verbosity values, and actually get to use them:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + usage: prog.py [-h] [-v VERBOSITY] square + prog.py: error: argument -v/--verbosity: expected one argument + $ python3 prog.py 4 -v 1 + 4^2 == 16 + $ python3 prog.py 4 -v 2 + the square of 4 equals 16 + $ python3 prog.py 4 -v 3 + 16 + +These all look good except the last one, which exposes a bug in our program. +Let's fix it by restricting the values the ``--verbosity`` option can accept:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And the output: + +.. code-block:: sh + + $ python3 prog.py 4 -v 3 + usage: prog.py [-h] [-v {0,1,2}] square + prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2) + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v {0,1,2}] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} + increase output verbosity + +Note that the change also reflects both in the error message as well as the +help string. + +Now, let's use a different approach of playing with verbosity, which is pretty +common. It also matches the way the CPython executable handles its own +verbosity argument (check the output of ``python --help``):: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display the square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity == 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity == 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We have introduced another action, "count", +to count the number of occurences of a specific optional arguments: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + $ python3 prog.py 4 -v + 4^2 == 16 + $ python3 prog.py 4 -vv + the square of 4 equals 16 + $ python3 prog.py 4 --verbosity --verbosity + the square of 4 equals 16 + $ python3 prog.py 4 -v 1 + usage: prog.py [-h] [-v] square + prog.py: error: unrecognized arguments: 1 + $ python3 prog.py 4 -h + usage: prog.py [-h] [-v] square + + positional arguments: + square display a square of a given number + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity increase output verbosity + $ python3 prog.py 4 -vvv + 16 + +* Yes, it's now more of a flag (similar to ``action="store_true"``) in the + previous version of our script. That should explain the complaint. + +* It also behaves similar to "store_true" action. + +* Now here's a demonstration of what the "count" action gives. You've probably + seen this sort of usage before. + +* And, just like the "store_true" action, if you don't specify the ``-v`` flag, + that flag is considered to have ``None`` value. + +* As should be expected, specifying the long form of the flag, we should get + the same output. + +* Sadly, our help output isn't very informative on the new ability our script + has acquired, but that can always be fixed by improving the documentation for + out script (e.g. via the ``help`` keyword argument). + +* That last output exposes a bug in our program. + + +Let's fix:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + + # bugfix: replace == with >= + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +And this is what it gives: + +.. code-block:: sh + + $ python3 prog.py 4 -vvv + the square of 4 equals 16 + $ python3 prog.py 4 -vvvv + the square of 4 equals 16 + $ python3 prog.py 4 + Traceback (most recent call last): + File "prog.py", line 11, in + if args.verbosity >= 2: + TypeError: unorderable types: NoneType() >= int() + +* First output went well, and fixes the bug we had before. + That is, we want any value >= 2 to be as verbose as possible. + +* Third output not so good. + +Let's fix that bug:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("square", type=int, + help="display a square of a given number") + parser.add_argument("-v", "--verbosity", action="count", default=0, + help="increase output verbosity") + args = parser.parse_args() + answer = args.square**2 + if args.verbosity >= 2: + print("the square of {} equals {}".format(args.square, answer)) + elif args.verbosity >= 1: + print("{}^2 == {}".format(args.square, answer)) + else: + print(answer) + +We've just introduced yet another keyword, ``default``. +We've set it to ``0`` in order to make it comparable to the other int values. +Remember that by default, +if an optional argument isn't specified, +it gets the ``None`` value, and that cannot be compared to an int value +(hence the :exc:`TypeError` exception). + +And: + +.. code-block:: sh + + $ python3 prog.py 4 + 16 + +You can go quite far just with what we've learned so far, +and we have only scratched the surface. +The :mod:`argparse` module is very powerful, +and we'll explore a bit more of it before we end this tutorial. + + +Getting a little more advanced +============================== + +What if we wanted to expand our tiny program to perform other powers, +not just squares:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + elif args.verbosity >= 1: + print("{}^{} == {}".format(args.x, args.y, answer)) + else: + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py + usage: prog.py [-h] [-v] x y + prog.py: error: the following arguments are required: x, y + $ python3 prog.py -h + usage: prog.py [-h] [-v] x y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbosity + $ python3 prog.py 4 2 -v + 4^2 == 16 + + +Notice that so far we've been using verbosity level to *change* the text +that gets displayed. The following example instead uses verbosity level +to display *more* text instead:: + + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + parser.add_argument("-v", "--verbosity", action="count", default=0) + args = parser.parse_args() + answer = args.x**args.y + if args.verbosity >= 2: + print("Running '{}'".format(__file__)) + if args.verbosity >= 1: + print("{}^{} == ".format(args.x, args.y), end="") + print(answer) + +Output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 16 + $ python3 prog.py 4 2 -v + 4^2 == 16 + $ python3 prog.py 4 2 -vv + Running 'prog.py' + 4^2 == 16 + + +Conflicting options +------------------- + +So far, we have been working with two methods of an +:class:`argparse.ArgumentParser` instance. Let's introduce a third one, +:meth:`add_mutually_exclusive_group`. It allows for us to specify options that +conflict with each other. Let's also change the rest of the program make the +new functionality makes more sense: +we'll introduce the ``--quiet`` option, +which will be the opposite of the ``--verbose`` one:: + + import argparse + + parser = argparse.ArgumentParser() + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Our program is now simpler, and we've lost some functionality for the sake of +demonstration. Anyways, here's the output: + +.. code-block:: sh + + $ python3 prog.py 4 2 + 4^2 == 16 + $ python3 prog.py 4 2 -q + 16 + $ python3 prog.py 4 2 -v + 4 to the power 2 equals 16 + $ python3 prog.py 4 2 -vq + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + $ python3 prog.py 4 2 -v --quiet + usage: prog.py [-h] [-v | -q] x y + prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose + +That should be easy to follow. I've added that last output so you can see the +sort of flexibility you get, i.e. mixing long form options with short form +ones. + +Before we conclude, you probably want to tell your users the main purpose of +your program, just in case they don't know:: + + import argparse + + parser = argparse.ArgumentParser(description="calculate X to the power of Y") + group = parser.add_mutually_exclusive_group() + group.add_argument("-v", "--verbose", action="store_true") + group.add_argument("-q", "--quiet", action="store_true") + parser.add_argument("x", type=int, help="the base") + parser.add_argument("y", type=int, help="the exponent") + args = parser.parse_args() + answer = args.x**args.y + + if args.quiet: + print(answer) + elif args.verbose: + print("{} to the power {} equals {}".format(args.x, args.y, answer)) + else: + print("{}^{} == {}".format(args.x, args.y, answer)) + +Note that slight difference in the usage text. Note the ``[-v | -q]``, +which tells us that we can either use ``-v`` or ``-q``, +but not both at the same time: + +.. code-block:: sh + + $ python3 prog.py --help + usage: prog.py [-h] [-v | -q] x y + + calculate X to the power of Y + + positional arguments: + x the base + y the exponent + + optional arguments: + -h, --help show this help message and exit + -v, --verbose + -q, --quiet + + +Conclusion +========== + +The :mod:`argparse` module offers a lot more than shown here. +Its docs are quite detailed and thorough, and full of examples. +Having gone through this tutorial, you should easily digest them +without feeling overwhelmed. diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -27,4 +27,5 @@ unicode.rst urllib2.rst webservers.rst + argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -12,6 +12,12 @@ -------------- +.. sidebar:: Tutorial + + This page contains the API reference information. For a more gentle + introduction to Python command-line parsing, have a look at the + :ref:`argparse tutorial `. + The :mod:`argparse` module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and :mod:`argparse` will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -480,6 +480,11 @@ Based on patch from Herv? Coatanhay. - Issue #14321: Do not run pgen during the build if files are up to date. +Documentation +------------- + +- Issue #14034: added the argparse tutorial. + - Issue #14324: Fix configure tests for cross builds. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 16:06:00 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 16:06:00 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0MDM0OiBhZGFw?= =?utf8?q?t_to_Python_2_and_fix_indentation=2E?= Message-ID: http://hg.python.org/cpython/rev/549aa1460811 changeset: 76795:549aa1460811 branch: 2.7 parent: 76791:48385618525b user: Ezio Melotti date: Sun May 06 17:03:19 2012 +0300 summary: #14034: adapt to Python 2 and fix indentation. files: Doc/howto/argparse.rst | 270 ++++++++++++++-------------- 1 files changed, 135 insertions(+), 135 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -79,16 +79,16 @@ .. code-block:: sh - $ python3 prog.py - $ python3 prog.py --help + $ python prog.py + $ python prog.py --help usage: prog.py [-h] optional arguments: - -h, --help show this help message and exit - $ python3 prog.py --verbose + -h, --help show this help message and exit + $ python prog.py --verbose usage: prog.py [-h] prog.py: error: unrecognized arguments: --verbose - $ python3 prog.py foo + $ python prog.py foo usage: prog.py [-h] prog.py: error: unrecognized arguments: foo @@ -115,24 +115,24 @@ parser = argparse.ArgumentParser() parser.add_argument("echo") args = parser.parse_args() - print(args.echo) + print args.echo And running the code: .. code-block:: sh - $ python3 prog.py + $ python prog.py usage: prog.py [-h] echo prog.py: error: the following arguments are required: echo - $ python3 prog.py --help + $ python prog.py --help usage: prog.py [-h] echo positional arguments: - echo + echo optional arguments: - -h, --help show this help message and exit - $ python3 prog.py foo + -h, --help show this help message and exit + $ python prog.py foo foo Here is what's happening: @@ -160,20 +160,20 @@ parser = argparse.ArgumentParser() parser.add_argument("echo", help="echo the string you use here") args = parser.parse_args() - print(args.echo) + print args.echo And we get: .. code-block:: sh - $ python3 prog.py -h + $ python prog.py -h usage: prog.py [-h] echo positional arguments: - echo echo the string you use here + echo echo the string you use here optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit Now, how about doing something even more useful:: @@ -181,16 +181,16 @@ parser = argparse.ArgumentParser() parser.add_argument("square", help="display a square of a given number") args = parser.parse_args() - print(args.square**2)) + print args.square**2 Following is a result of running the code: .. code-block:: sh - $ python3 prog.py 4 + $ python prog.py 4 Traceback (most recent call last): - File "prog.py", line 5, in - print(args.square**2) + File "prog.py", line 5, in + print args.square**2 TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' That didn't go so well. That's because :mod:`argparse` treats the options we @@ -200,17 +200,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", help="display a square of a given number", - type=int) + type=int) args = parser.parse_args() - print(args.square**2) + print args.square**2 Following is a result of running the code: .. code-block:: sh - $ python3 prog.py 4 + $ python prog.py 4 16 - $ python3 prog.py four + $ python prog.py four usage: prog.py [-h] square prog.py: error: argument square: invalid int value: 'four' @@ -229,23 +229,23 @@ parser.add_argument("--verbosity", help="increase output verbosity") args = parser.parse_args() if args.verbosity: - print("verbosity turned on") + print "verbosity turned on" And the output: .. code-block:: sh - $ python3 prog.py --verbosity 1 + $ python prog.py --verbosity 1 verbosity turned on - $ python3 prog.py - $ python3 prog.py --help + $ python prog.py + $ python prog.py --help usage: prog.py [-h] [--verbosity VERBOSITY] optional arguments: - -h, --help show this help message and exit - --verbosity VERBOSITY + -h, --help show this help message and exit + --verbosity VERBOSITY increase output verbosity - $ python3 prog.py --verbosity + $ python prog.py --verbosity usage: prog.py [-h] [--verbosity VERBOSITY] prog.py: error: argument --verbosity: expected one argument @@ -272,26 +272,26 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("--verbose", help="increase output verbosity", - action="store_true") + action="store_true") args = parser.parse_args() if args.verbose: - print("verbosity turned on") + print "verbosity turned on" And the output: .. code-block:: sh - $ python3 prog.py --verbose + $ python prog.py --verbose verbosity turned on - $ python3 prog.py --verbose 1 + $ python prog.py --verbose 1 usage: prog.py [-h] [--verbose] prog.py: error: unrecognized arguments: 1 - $ python3 prog.py --help + $ python prog.py --help usage: prog.py [-h] [--verbose] optional arguments: - -h, --help show this help message and exit - --verbose increase output verbosity + -h, --help show this help message and exit + --verbose increase output verbosity Here is what is happening: @@ -318,23 +318,23 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("-v", "--verbose", help="increase output verbosity", - action="store_true") + action="store_true") args = parser.parse_args() if args.verbose: - print("verbosity turned on") + print "verbosity turned on" And here goes: .. code-block:: sh - $ python3 prog.py -v + $ python prog.py -v verbosity turned on - $ python3 prog.py --help + $ python prog.py --help usage: prog.py [-h] [-v] optional arguments: - -h, --help show this help message and exit - -v, --verbose increase output verbosity + -h, --help show this help message and exit + -v, --verbose increase output verbosity Note that the new ability is also reflected in the help text. @@ -347,28 +347,28 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbose", action="store_true", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbose: - print("the square of {} equals {}".format(args.square, answer)) + print "the square of {} equals {}".format(args.square, answer) else: - print(answer) + print answer And now the output: .. code-block:: sh - $ python3 prog.py + $ python prog.py usage: prog.py [-h] [-v] square prog.py: error: the following arguments are required: square - $ python3 prog.py 4 + $ python prog.py 4 16 - $ python3 prog.py 4 --verbose + $ python prog.py 4 --verbose the square of 4 equals 16 - $ python3 prog.py --verbose 4 + $ python prog.py --verbose 4 the square of 4 equals 16 * We've brought back a positional argument, hence the complaint. @@ -381,32 +381,32 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", type=int, - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print "the square of {} equals {}".format(args.square, answer) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print "{}^2 == {}".format(args.square, answer) else: - print(answer) + print answer And the output: .. code-block:: sh - $ python3 prog.py 4 + $ python prog.py 4 16 - $ python3 prog.py 4 -v + $ python prog.py 4 -v usage: prog.py [-h] [-v VERBOSITY] square prog.py: error: argument -v/--verbosity: expected one argument - $ python3 prog.py 4 -v 1 + $ python prog.py 4 -v 1 4^2 == 16 - $ python3 prog.py 4 -v 2 + $ python prog.py 4 -v 2 the square of 4 equals 16 - $ python3 prog.py 4 -v 3 + $ python prog.py 4 -v 3 16 These all look good except the last one, which exposes a bug in our program. @@ -415,34 +415,34 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print "the square of {} equals {}".format(args.square, answer) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print "{}^2 == {}".format(args.square, answer) else: - print(answer) + print answer And the output: .. code-block:: sh - $ python3 prog.py 4 -v 3 + $ python prog.py 4 -v 3 usage: prog.py [-h] [-v {0,1,2}] square prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2) - $ python3 prog.py 4 -h + $ python prog.py 4 -h usage: prog.py [-h] [-v {0,1,2}] square positional arguments: - square display a square of a given number + square display a square of a given number optional arguments: - -h, --help show this help message and exit - -v {0,1,2}, --verbosity {0,1,2} + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} increase output verbosity Note that the change also reflects both in the error message as well as the @@ -455,44 +455,44 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display the square of a given number") + help="display the square of a given number") parser.add_argument("-v", "--verbosity", action="count", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print "the square of {} equals {}".format(args.square, answer) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print "{}^2 == {}".format(args.square, answer) else: - print(answer) + print answer We have introduced another action, "count", to count the number of occurences of a specific optional arguments: .. code-block:: sh - $ python3 prog.py 4 + $ python prog.py 4 16 - $ python3 prog.py 4 -v + $ python prog.py 4 -v 4^2 == 16 - $ python3 prog.py 4 -vv + $ python prog.py 4 -vv the square of 4 equals 16 - $ python3 prog.py 4 --verbosity --verbosity + $ python prog.py 4 --verbosity --verbosity the square of 4 equals 16 - $ python3 prog.py 4 -v 1 + $ python prog.py 4 -v 1 usage: prog.py [-h] [-v] square prog.py: error: unrecognized arguments: 1 - $ python3 prog.py 4 -h + $ python prog.py 4 -h usage: prog.py [-h] [-v] square positional arguments: - square display a square of a given number + square display a square of a given number optional arguments: - -h, --help show this help message and exit - -v, --verbosity increase output verbosity - $ python3 prog.py 4 -vvv + -h, --help show this help message and exit + -v, --verbosity increase output verbosity + $ python prog.py 4 -vvv 16 * Yes, it's now more of a flag (similar to ``action="store_true"``) in the @@ -521,32 +521,32 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", action="count", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 # bugfix: replace == with >= if args.verbosity >= 2: - print("the square of {} equals {}".format(args.square, answer)) + print "the square of {} equals {}".format(args.square, answer) elif args.verbosity >= 1: - print("{}^2 == {}".format(args.square, answer)) + print "{}^2 == {}".format(args.square, answer) else: - print(answer) + print answer And this is what it gives: .. code-block:: sh - $ python3 prog.py 4 -vvv + $ python prog.py 4 -vvv the square of 4 equals 16 - $ python3 prog.py 4 -vvvv + $ python prog.py 4 -vvvv the square of 4 equals 16 - $ python3 prog.py 4 + $ python prog.py 4 Traceback (most recent call last): - File "prog.py", line 11, in - if args.verbosity >= 2: + File "prog.py", line 11, in + if args.verbosity >= 2: TypeError: unorderable types: NoneType() >= int() * First output went well, and fixes the bug we had before. @@ -559,17 +559,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", action="count", default=0, - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity >= 2: - print("the square of {} equals {}".format(args.square, answer)) + print "the square of {} equals {}".format(args.square, answer) elif args.verbosity >= 1: - print("{}^2 == {}".format(args.square, answer)) + print "{}^2 == {}".format(args.square, answer) else: - print(answer) + print answer We've just introduced yet another keyword, ``default``. We've set it to ``0`` in order to make it comparable to the other int values. @@ -582,7 +582,7 @@ .. code-block:: sh - $ python3 prog.py 4 + $ python prog.py 4 16 You can go quite far just with what we've learned so far, @@ -605,30 +605,30 @@ args = parser.parse_args() answer = args.x**args.y if args.verbosity >= 2: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print "{} to the power {} equals {}".format(args.x, args.y, answer) elif args.verbosity >= 1: - print("{}^{} == {}".format(args.x, args.y, answer)) + print "{}^{} == {}".format(args.x, args.y, answer) else: - print(answer) + print answer Output: .. code-block:: sh - $ python3 prog.py + $ python prog.py usage: prog.py [-h] [-v] x y prog.py: error: the following arguments are required: x, y - $ python3 prog.py -h + $ python prog.py -h usage: prog.py [-h] [-v] x y positional arguments: - x the base - y the exponent + x the base + y the exponent optional arguments: - -h, --help show this help message and exit - -v, --verbosity - $ python3 prog.py 4 2 -v + -h, --help show this help message and exit + -v, --verbosity + $ python prog.py 4 2 -v 4^2 == 16 @@ -644,20 +644,20 @@ args = parser.parse_args() answer = args.x**args.y if args.verbosity >= 2: - print("Running '{}'".format(__file__)) + print "Running '{}'".format(__file__) if args.verbosity >= 1: - print("{}^{} == ".format(args.x, args.y), end="") - print(answer) + print "{}^{} == ".format(args.x, args.y), end="" + print answer Output: .. code-block:: sh - $ python3 prog.py 4 2 + $ python prog.py 4 2 16 - $ python3 prog.py 4 2 -v + $ python prog.py 4 2 -v 4^2 == 16 - $ python3 prog.py 4 2 -vv + $ python prog.py 4 2 -vv Running 'prog.py' 4^2 == 16 @@ -685,27 +685,27 @@ answer = args.x**args.y if args.quiet: - print(answer) + print answer elif args.verbose: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print "{} to the power {} equals {}".format(args.x, args.y, answer) else: - print("{}^{} == {}".format(args.x, args.y, answer)) + print "{}^{} == {}".format(args.x, args.y, answer) Our program is now simpler, and we've lost some functionality for the sake of demonstration. Anyways, here's the output: .. code-block:: sh - $ python3 prog.py 4 2 + $ python prog.py 4 2 4^2 == 16 - $ python3 prog.py 4 2 -q + $ python prog.py 4 2 -q 16 - $ python3 prog.py 4 2 -v + $ python prog.py 4 2 -v 4 to the power 2 equals 16 - $ python3 prog.py 4 2 -vq + $ python prog.py 4 2 -vq usage: prog.py [-h] [-v | -q] x y prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose - $ python3 prog.py 4 2 -v --quiet + $ python prog.py 4 2 -v --quiet usage: prog.py [-h] [-v | -q] x y prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose @@ -728,11 +728,11 @@ answer = args.x**args.y if args.quiet: - print(answer) + print answer elif args.verbose: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print "{} to the power {} equals {}".format(args.x, args.y, answer) else: - print("{}^{} == {}".format(args.x, args.y, answer)) + print "{}^{} == {}".format(args.x, args.y, answer) Note that slight difference in the usage text. Note the ``[-v | -q]``, which tells us that we can either use ``-v`` or ``-q``, @@ -740,19 +740,19 @@ .. code-block:: sh - $ python3 prog.py --help + $ python prog.py --help usage: prog.py [-h] [-v | -q] x y calculate X to the power of Y positional arguments: - x the base - y the exponent + x the base + y the exponent optional arguments: - -h, --help show this help message and exit - -v, --verbose - -q, --quiet + -h, --help show this help message and exit + -v, --verbose + -q, --quiet Conclusion -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 16:06:00 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 16:06:00 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0MDM0OiBmaXgg?= =?utf8?q?indentation=2E?= Message-ID: http://hg.python.org/cpython/rev/d5b7be0629c0 changeset: 76796:d5b7be0629c0 branch: 3.2 parent: 76792:11703cb2a2f3 user: Ezio Melotti date: Sun May 06 17:05:16 2012 +0300 summary: #14034: fix indentation. files: Doc/howto/argparse.rst | 156 ++++++++++++++-------------- 1 files changed, 78 insertions(+), 78 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -84,7 +84,7 @@ usage: prog.py [-h] optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit $ python3 prog.py --verbose usage: prog.py [-h] prog.py: error: unrecognized arguments: --verbose @@ -128,10 +128,10 @@ usage: prog.py [-h] echo positional arguments: - echo + echo optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit $ python3 prog.py foo foo @@ -170,10 +170,10 @@ usage: prog.py [-h] echo positional arguments: - echo echo the string you use here + echo echo the string you use here optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit Now, how about doing something even more useful:: @@ -181,7 +181,7 @@ parser = argparse.ArgumentParser() parser.add_argument("square", help="display a square of a given number") args = parser.parse_args() - print(args.square**2)) + print(args.square**2) Following is a result of running the code: @@ -189,8 +189,8 @@ $ python3 prog.py 4 Traceback (most recent call last): - File "prog.py", line 5, in - print(args.square**2) + File "prog.py", line 5, in + print(args.square**2) TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' That didn't go so well. That's because :mod:`argparse` treats the options we @@ -200,7 +200,7 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", help="display a square of a given number", - type=int) + type=int) args = parser.parse_args() print(args.square**2) @@ -229,7 +229,7 @@ parser.add_argument("--verbosity", help="increase output verbosity") args = parser.parse_args() if args.verbosity: - print("verbosity turned on") + print("verbosity turned on") And the output: @@ -242,8 +242,8 @@ usage: prog.py [-h] [--verbosity VERBOSITY] optional arguments: - -h, --help show this help message and exit - --verbosity VERBOSITY + -h, --help show this help message and exit + --verbosity VERBOSITY increase output verbosity $ python3 prog.py --verbosity usage: prog.py [-h] [--verbosity VERBOSITY] @@ -272,10 +272,10 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("--verbose", help="increase output verbosity", - action="store_true") + action="store_true") args = parser.parse_args() if args.verbose: - print("verbosity turned on") + print("verbosity turned on") And the output: @@ -283,15 +283,15 @@ $ python3 prog.py --verbose verbosity turned on - $ python3 prog.py --verbose 1 + $ python3 prog.py --verbose 1 usage: prog.py [-h] [--verbose] prog.py: error: unrecognized arguments: 1 $ python3 prog.py --help usage: prog.py [-h] [--verbose] optional arguments: - -h, --help show this help message and exit - --verbose increase output verbosity + -h, --help show this help message and exit + --verbose increase output verbosity Here is what is happening: @@ -318,10 +318,10 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("-v", "--verbose", help="increase output verbosity", - action="store_true") + action="store_true") args = parser.parse_args() if args.verbose: - print("verbosity turned on") + print("verbosity turned on") And here goes: @@ -333,8 +333,8 @@ usage: prog.py [-h] [-v] optional arguments: - -h, --help show this help message and exit - -v, --verbose increase output verbosity + -h, --help show this help message and exit + -v, --verbose increase output verbosity Note that the new ability is also reflected in the help text. @@ -347,15 +347,15 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbose", action="store_true", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbose: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) else: - print(answer) + print(answer) And now the output: @@ -381,17 +381,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", type=int, - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) And the output: @@ -415,17 +415,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) And the output: @@ -438,11 +438,11 @@ usage: prog.py [-h] [-v {0,1,2}] square positional arguments: - square display a square of a given number + square display a square of a given number optional arguments: - -h, --help show this help message and exit - -v {0,1,2}, --verbosity {0,1,2} + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} increase output verbosity Note that the change also reflects both in the error message as well as the @@ -455,17 +455,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display the square of a given number") + help="display the square of a given number") parser.add_argument("-v", "--verbosity", action="count", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) We have introduced another action, "count", to count the number of occurences of a specific optional arguments: @@ -487,11 +487,11 @@ usage: prog.py [-h] [-v] square positional arguments: - square display a square of a given number + square display a square of a given number optional arguments: - -h, --help show this help message and exit - -v, --verbosity increase output verbosity + -h, --help show this help message and exit + -v, --verbosity increase output verbosity $ python3 prog.py 4 -vvv 16 @@ -521,19 +521,19 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", action="count", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 # bugfix: replace == with >= if args.verbosity >= 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity >= 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) And this is what it gives: @@ -545,8 +545,8 @@ the square of 4 equals 16 $ python3 prog.py 4 Traceback (most recent call last): - File "prog.py", line 11, in - if args.verbosity >= 2: + File "prog.py", line 11, in + if args.verbosity >= 2: TypeError: unorderable types: NoneType() >= int() * First output went well, and fixes the bug we had before. @@ -559,17 +559,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", action="count", default=0, - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity >= 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity >= 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) We've just introduced yet another keyword, ``default``. We've set it to ``0`` in order to make it comparable to the other int values. @@ -605,11 +605,11 @@ args = parser.parse_args() answer = args.x**args.y if args.verbosity >= 2: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print("{} to the power {} equals {}".format(args.x, args.y, answer)) elif args.verbosity >= 1: - print("{}^{} == {}".format(args.x, args.y, answer)) + print("{}^{} == {}".format(args.x, args.y, answer)) else: - print(answer) + print(answer) Output: @@ -622,12 +622,12 @@ usage: prog.py [-h] [-v] x y positional arguments: - x the base - y the exponent + x the base + y the exponent optional arguments: - -h, --help show this help message and exit - -v, --verbosity + -h, --help show this help message and exit + -v, --verbosity $ python3 prog.py 4 2 -v 4^2 == 16 @@ -644,9 +644,9 @@ args = parser.parse_args() answer = args.x**args.y if args.verbosity >= 2: - print("Running '{}'".format(__file__)) + print("Running '{}'".format(__file__)) if args.verbosity >= 1: - print("{}^{} == ".format(args.x, args.y), end="") + print("{}^{} == ".format(args.x, args.y), end="") print(answer) Output: @@ -685,11 +685,11 @@ answer = args.x**args.y if args.quiet: - print(answer) + print(answer) elif args.verbose: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print("{} to the power {} equals {}".format(args.x, args.y, answer)) else: - print("{}^{} == {}".format(args.x, args.y, answer)) + print("{}^{} == {}".format(args.x, args.y, answer)) Our program is now simpler, and we've lost some functionality for the sake of demonstration. Anyways, here's the output: @@ -728,11 +728,11 @@ answer = args.x**args.y if args.quiet: - print(answer) + print(answer) elif args.verbose: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print("{} to the power {} equals {}".format(args.x, args.y, answer)) else: - print("{}^{} == {}".format(args.x, args.y, answer)) + print("{}^{} == {}".format(args.x, args.y, answer)) Note that slight difference in the usage text. Note the ``[-v | -q]``, which tells us that we can either use ``-v`` or ``-q``, @@ -746,13 +746,13 @@ calculate X to the power of Y positional arguments: - x the base - y the exponent + x the base + y the exponent optional arguments: - -h, --help show this help message and exit - -v, --verbose - -q, --quiet + -h, --help show this help message and exit + -v, --verbose + -q, --quiet Conclusion -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 16:06:01 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 06 May 2012 16:06:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314034=3A_merge_indentation_fixes_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/e14c860f6eee changeset: 76797:e14c860f6eee parent: 76794:50a83d63cb40 parent: 76796:d5b7be0629c0 user: Ezio Melotti date: Sun May 06 17:05:54 2012 +0300 summary: #14034: merge indentation fixes from 3.2. files: Doc/howto/argparse.rst | 156 ++++++++++++++-------------- 1 files changed, 78 insertions(+), 78 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -84,7 +84,7 @@ usage: prog.py [-h] optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit $ python3 prog.py --verbose usage: prog.py [-h] prog.py: error: unrecognized arguments: --verbose @@ -128,10 +128,10 @@ usage: prog.py [-h] echo positional arguments: - echo + echo optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit $ python3 prog.py foo foo @@ -170,10 +170,10 @@ usage: prog.py [-h] echo positional arguments: - echo echo the string you use here + echo echo the string you use here optional arguments: - -h, --help show this help message and exit + -h, --help show this help message and exit Now, how about doing something even more useful:: @@ -181,7 +181,7 @@ parser = argparse.ArgumentParser() parser.add_argument("square", help="display a square of a given number") args = parser.parse_args() - print(args.square**2)) + print(args.square**2) Following is a result of running the code: @@ -189,8 +189,8 @@ $ python3 prog.py 4 Traceback (most recent call last): - File "prog.py", line 5, in - print(args.square**2) + File "prog.py", line 5, in + print(args.square**2) TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int' That didn't go so well. That's because :mod:`argparse` treats the options we @@ -200,7 +200,7 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", help="display a square of a given number", - type=int) + type=int) args = parser.parse_args() print(args.square**2) @@ -229,7 +229,7 @@ parser.add_argument("--verbosity", help="increase output verbosity") args = parser.parse_args() if args.verbosity: - print("verbosity turned on") + print("verbosity turned on") And the output: @@ -242,8 +242,8 @@ usage: prog.py [-h] [--verbosity VERBOSITY] optional arguments: - -h, --help show this help message and exit - --verbosity VERBOSITY + -h, --help show this help message and exit + --verbosity VERBOSITY increase output verbosity $ python3 prog.py --verbosity usage: prog.py [-h] [--verbosity VERBOSITY] @@ -272,10 +272,10 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("--verbose", help="increase output verbosity", - action="store_true") + action="store_true") args = parser.parse_args() if args.verbose: - print("verbosity turned on") + print("verbosity turned on") And the output: @@ -283,15 +283,15 @@ $ python3 prog.py --verbose verbosity turned on - $ python3 prog.py --verbose 1 + $ python3 prog.py --verbose 1 usage: prog.py [-h] [--verbose] prog.py: error: unrecognized arguments: 1 $ python3 prog.py --help usage: prog.py [-h] [--verbose] optional arguments: - -h, --help show this help message and exit - --verbose increase output verbosity + -h, --help show this help message and exit + --verbose increase output verbosity Here is what is happening: @@ -318,10 +318,10 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("-v", "--verbose", help="increase output verbosity", - action="store_true") + action="store_true") args = parser.parse_args() if args.verbose: - print("verbosity turned on") + print("verbosity turned on") And here goes: @@ -333,8 +333,8 @@ usage: prog.py [-h] [-v] optional arguments: - -h, --help show this help message and exit - -v, --verbose increase output verbosity + -h, --help show this help message and exit + -v, --verbose increase output verbosity Note that the new ability is also reflected in the help text. @@ -347,15 +347,15 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbose", action="store_true", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbose: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) else: - print(answer) + print(answer) And now the output: @@ -381,17 +381,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", type=int, - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) And the output: @@ -415,17 +415,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2], - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) And the output: @@ -438,11 +438,11 @@ usage: prog.py [-h] [-v {0,1,2}] square positional arguments: - square display a square of a given number + square display a square of a given number optional arguments: - -h, --help show this help message and exit - -v {0,1,2}, --verbosity {0,1,2} + -h, --help show this help message and exit + -v {0,1,2}, --verbosity {0,1,2} increase output verbosity Note that the change also reflects both in the error message as well as the @@ -455,17 +455,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display the square of a given number") + help="display the square of a given number") parser.add_argument("-v", "--verbosity", action="count", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity == 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity == 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) We have introduced another action, "count", to count the number of occurences of a specific optional arguments: @@ -487,11 +487,11 @@ usage: prog.py [-h] [-v] square positional arguments: - square display a square of a given number + square display a square of a given number optional arguments: - -h, --help show this help message and exit - -v, --verbosity increase output verbosity + -h, --help show this help message and exit + -v, --verbosity increase output verbosity $ python3 prog.py 4 -vvv 16 @@ -521,19 +521,19 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", action="count", - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 # bugfix: replace == with >= if args.verbosity >= 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity >= 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) And this is what it gives: @@ -545,8 +545,8 @@ the square of 4 equals 16 $ python3 prog.py 4 Traceback (most recent call last): - File "prog.py", line 11, in - if args.verbosity >= 2: + File "prog.py", line 11, in + if args.verbosity >= 2: TypeError: unorderable types: NoneType() >= int() * First output went well, and fixes the bug we had before. @@ -559,17 +559,17 @@ import argparse parser = argparse.ArgumentParser() parser.add_argument("square", type=int, - help="display a square of a given number") + help="display a square of a given number") parser.add_argument("-v", "--verbosity", action="count", default=0, - help="increase output verbosity") + help="increase output verbosity") args = parser.parse_args() answer = args.square**2 if args.verbosity >= 2: - print("the square of {} equals {}".format(args.square, answer)) + print("the square of {} equals {}".format(args.square, answer)) elif args.verbosity >= 1: - print("{}^2 == {}".format(args.square, answer)) + print("{}^2 == {}".format(args.square, answer)) else: - print(answer) + print(answer) We've just introduced yet another keyword, ``default``. We've set it to ``0`` in order to make it comparable to the other int values. @@ -605,11 +605,11 @@ args = parser.parse_args() answer = args.x**args.y if args.verbosity >= 2: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print("{} to the power {} equals {}".format(args.x, args.y, answer)) elif args.verbosity >= 1: - print("{}^{} == {}".format(args.x, args.y, answer)) + print("{}^{} == {}".format(args.x, args.y, answer)) else: - print(answer) + print(answer) Output: @@ -622,12 +622,12 @@ usage: prog.py [-h] [-v] x y positional arguments: - x the base - y the exponent + x the base + y the exponent optional arguments: - -h, --help show this help message and exit - -v, --verbosity + -h, --help show this help message and exit + -v, --verbosity $ python3 prog.py 4 2 -v 4^2 == 16 @@ -644,9 +644,9 @@ args = parser.parse_args() answer = args.x**args.y if args.verbosity >= 2: - print("Running '{}'".format(__file__)) + print("Running '{}'".format(__file__)) if args.verbosity >= 1: - print("{}^{} == ".format(args.x, args.y), end="") + print("{}^{} == ".format(args.x, args.y), end="") print(answer) Output: @@ -685,11 +685,11 @@ answer = args.x**args.y if args.quiet: - print(answer) + print(answer) elif args.verbose: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print("{} to the power {} equals {}".format(args.x, args.y, answer)) else: - print("{}^{} == {}".format(args.x, args.y, answer)) + print("{}^{} == {}".format(args.x, args.y, answer)) Our program is now simpler, and we've lost some functionality for the sake of demonstration. Anyways, here's the output: @@ -728,11 +728,11 @@ answer = args.x**args.y if args.quiet: - print(answer) + print(answer) elif args.verbose: - print("{} to the power {} equals {}".format(args.x, args.y, answer)) + print("{} to the power {} equals {}".format(args.x, args.y, answer)) else: - print("{}^{} == {}".format(args.x, args.y, answer)) + print("{}^{} == {}".format(args.x, args.y, answer)) Note that slight difference in the usage text. Note the ``[-v | -q]``, which tells us that we can either use ``-v`` or ``-q``, @@ -746,13 +746,13 @@ calculate X to the power of Y positional arguments: - x the base - y the exponent + x the base + y the exponent optional arguments: - -h, --help show this help message and exit - -v, --verbose - -q, --quiet + -h, --help show this help message and exit + -v, --verbose + -q, --quiet Conclusion -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 16:26:01 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 06 May 2012 16:26:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_test=5Fgzip_failures_on?= =?utf8?q?_Windows=2E?= Message-ID: http://hg.python.org/cpython/rev/f52df8fd3235 changeset: 76798:f52df8fd3235 user: Nadeem Vawda date: Sun May 06 16:25:35 2012 +0200 summary: Fix test_gzip failures on Windows. files: Lib/test/test_gzip.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -409,7 +409,7 @@ self.assertEqual(file_data, uncompressed * 2) def test_text_modes(self): - uncompressed = data1.decode("ascii") * 50 + uncompressed = data1.decode("ascii").replace("\n", os.linesep) * 50 with gzip.open(self.filename, "wt") as f: f.write(uncompressed) with open(self.filename, "rb") as f: @@ -436,7 +436,7 @@ def test_encoding(self): # Test non-default encoding. - uncompressed = data1.decode("ascii") * 50 + uncompressed = data1.decode("ascii").replace("\n", os.linesep) * 50 with gzip.open(self.filename, "wt", encoding="utf-16") as f: f.write(uncompressed) with open(self.filename, "rb") as f: @@ -456,7 +456,7 @@ def test_newline(self): # Test with explicit newline (universal newline mode disabled). uncompressed = data1.decode("ascii") * 50 - with gzip.open(self.filename, "wt") as f: + with gzip.open(self.filename, "wt", newline="\n") as f: f.write(uncompressed) with gzip.open(self.filename, "rt", newline="\r") as f: self.assertEqual(f.readlines(), [uncompressed]) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 17:17:19 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 06 May 2012 17:17:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_too_early_decrefs=2E?= Message-ID: http://hg.python.org/cpython/rev/49306ceaf91f changeset: 76799:49306ceaf91f user: Antoine Pitrou date: Sun May 06 17:15:23 2012 +0200 summary: Fix too early decrefs. files: Python/import.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1639,7 +1639,6 @@ if (level == 0) { final_mod = PyDict_GetItem(interp->modules, front); - Py_DECREF(front); if (final_mod == NULL) { PyErr_Format(PyExc_KeyError, "%R not in sys.modules as expected", front); @@ -1647,6 +1646,7 @@ else { Py_INCREF(final_mod); } + Py_DECREF(front); } else { Py_ssize_t cut_off = PyUnicode_GET_LENGTH(name) - @@ -1660,7 +1660,6 @@ } final_mod = PyDict_GetItem(interp->modules, to_return); - Py_DECREF(to_return); if (final_mod == NULL) { PyErr_Format(PyExc_KeyError, "%R not in sys.modules as expected", @@ -1669,6 +1668,7 @@ else { Py_INCREF(final_mod); } + Py_DECREF(to_return); } } else { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 17:56:55 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 06 May 2012 17:56:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_AcquirerProxy=2Eacquir?= =?utf8?q?e=28=29_support_timeout_argument?= Message-ID: http://hg.python.org/cpython/rev/b4a1d9287780 changeset: 76800:b4a1d9287780 user: Richard Oudkerk date: Sun May 06 16:45:02 2012 +0100 summary: Make AcquirerProxy.acquire() support timeout argument files: Lib/multiprocessing/managers.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -957,8 +957,9 @@ class AcquirerProxy(BaseProxy): _exposed_ = ('acquire', 'release') - def acquire(self, blocking=True): - return self._callmethod('acquire', (blocking,)) + def acquire(self, blocking=True, timeout=None): + args = (blocking,) if timeout is None else (blocking, timeout) + return self._callmethod('acquire', args) def release(self): return self._callmethod('release') def __enter__(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 17:56:56 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sun, 06 May 2012 17:56:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_test=5Fmultiprocessing?= =?utf8?q?_more_lenient_about_another_timeout_check?= Message-ID: http://hg.python.org/cpython/rev/5c60c142f099 changeset: 76801:5c60c142f099 user: Richard Oudkerk date: Sun May 06 16:46:36 2012 +0100 summary: Make test_multiprocessing more lenient about another timeout check files: Lib/test/test_multiprocessing.py | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -922,7 +922,8 @@ self.assertEqual(p.exitcode, 0) @classmethod - def _test_waitfor_timeout_f(cls, cond, state, success): + def _test_waitfor_timeout_f(cls, cond, state, success, sem): + sem.release() with cond: expected = 0.1 dt = time.time() @@ -938,11 +939,13 @@ cond = self.Condition() state = self.Value('i', 0) success = self.Value('i', False) + sem = self.Semaphore(0) p = self.Process(target=self._test_waitfor_timeout_f, - args=(cond, state, success)) + args=(cond, state, success, sem)) p.daemon = True p.start() + self.assertTrue(sem.acquire(timeout=10)) # Only increment 3 times, so state == 4 is never reached. for i in range(3): @@ -2723,8 +2726,8 @@ delta = time.time() - start self.assertEqual(res, []) - self.assertLess(delta, expected + 1) - self.assertGreater(delta, expected - 1) + self.assertLess(delta, expected * 2) + self.assertGreater(delta, expected * 0.5) b.send(None) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 18:35:44 2012 From: python-checkins at python.org (mark.dickinson) Date: Sun, 06 May 2012 18:35:44 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0OTY1?= =?utf8?q?=3A_Fix_missing_support_for_starred_assignments_in?= Message-ID: http://hg.python.org/cpython/rev/c80576303892 changeset: 76802:c80576303892 branch: 3.2 parent: 76796:d5b7be0629c0 user: Mark Dickinson date: Sun May 06 17:27:39 2012 +0100 summary: Issue #14965: Fix missing support for starred assignments in Tools/parser/unparse.py. files: Misc/NEWS | 6 ++++++ Tools/parser/test_unparse.py | 7 +++++++ Tools/parser/unparse.py | 4 ++++ 3 files changed, 17 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -225,6 +225,12 @@ - Issue #14034: added the argparse tutorial. +Tools/Demos +----------- + +- Issue #14965: Fix missing support for starred assignments in + Tools/parser/unparse.py. + What's New in Python 3.2.3 release candidate 2? =============================================== diff --git a/Tools/parser/test_unparse.py b/Tools/parser/test_unparse.py --- a/Tools/parser/test_unparse.py +++ b/Tools/parser/test_unparse.py @@ -209,6 +209,13 @@ def test_try_except_finally(self): self.check_roundtrip(try_except_finally) + def test_starred_assignment(self): + self.check_roundtrip("a, *b, c = seq") + self.check_roundtrip("a, (*b, c) = seq") + self.check_roundtrip("a, *b[0], c = seq") + self.check_roundtrip("a, *(b, c) = seq") + + class DirectoryTestCase(ASTTestCase): """Test roundtrip behaviour on all files in Lib and Lib/test.""" diff --git a/Tools/parser/unparse.py b/Tools/parser/unparse.py --- a/Tools/parser/unparse.py +++ b/Tools/parser/unparse.py @@ -472,6 +472,10 @@ self.dispatch(t.slice) self.write("]") + def _Starred(self, t): + self.write("*") + self.dispatch(t.value) + # slice def _Ellipsis(self, t): self.write("...") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 18:35:44 2012 From: python-checkins at python.org (mark.dickinson) Date: Sun, 06 May 2012 18:35:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314965=3A__Bring_Tools/parser/unparse=2Epy_up_to_dat?= =?utf8?q?e_with_the_Python_3=2E3=2E?= Message-ID: http://hg.python.org/cpython/rev/89e928048903 changeset: 76803:89e928048903 parent: 76801:5c60c142f099 parent: 76802:c80576303892 user: Mark Dickinson date: Sun May 06 17:35:19 2012 +0100 summary: Issue #14965: Bring Tools/parser/unparse.py up to date with the Python 3.3. Grammar. files: Misc/NEWS | 6 +++ Tools/parser/test_unparse.py | 31 +++++++++++++++++ Tools/parser/unparse.py | 42 +++++++++++++---------- 3 files changed, 60 insertions(+), 19 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -28,6 +28,12 @@ - Issue #14127 and #10148: shutil.copystat now preserves exact mtime and atime on filesystems providing nanosecond resolution. +Tools/Demos +----------- + +- Issue #14965: Bring Tools/parser/unparse.py support up to date with + the Python 3.3 Grammar. + What's New in Python 3.3.0 Alpha 3? =================================== diff --git a/Tools/parser/test_unparse.py b/Tools/parser/test_unparse.py --- a/Tools/parser/test_unparse.py +++ b/Tools/parser/test_unparse.py @@ -93,6 +93,21 @@ suite5 """ +with_simple = """\ +with f(): + suite1 +""" + +with_as = """\ +with f() as x: + suite1 +""" + +with_two_items = """\ +with f() as x, g() as y: + suite1 +""" + class ASTTestCase(unittest.TestCase): def assertASTEqual(self, ast1, ast2): self.assertEqual(ast.dump(ast1), ast.dump(ast2)) @@ -209,6 +224,22 @@ def test_try_except_finally(self): self.check_roundtrip(try_except_finally) + def test_starred_assignment(self): + self.check_roundtrip("a, *b, c = seq") + self.check_roundtrip("a, (*b, c) = seq") + self.check_roundtrip("a, *b[0], c = seq") + self.check_roundtrip("a, *(b, c) = seq") + + def test_with_simple(self): + self.check_roundtrip(with_simple) + + def test_with_as(self): + self.check_roundtrip(with_as) + + def test_with_two_items(self): + self.check_roundtrip(with_two_items) + + class DirectoryTestCase(ASTTestCase): """Test roundtrip behaviour on all files in Lib and Lib/test.""" diff --git a/Tools/parser/unparse.py b/Tools/parser/unparse.py --- a/Tools/parser/unparse.py +++ b/Tools/parser/unparse.py @@ -147,6 +147,14 @@ self.dispatch(t.value) self.write(")") + def _YieldFrom(self, t): + self.write("(") + self.write("yield from") + if t.value: + self.write(" ") + self.dispatch(t.value) + self.write(")") + def _Raise(self, t): self.fill("raise") if not t.exc: @@ -158,12 +166,11 @@ self.write(" from ") self.dispatch(t.cause) - def _TryExcept(self, t): + def _Try(self, t): self.fill("try") self.enter() self.dispatch(t.body) self.leave() - for ex in t.handlers: self.dispatch(ex) if t.orelse: @@ -171,22 +178,12 @@ self.enter() self.dispatch(t.orelse) self.leave() - - def _TryFinally(self, t): - if len(t.body) == 1 and isinstance(t.body[0], ast.TryExcept): - # try-except-finally - self.dispatch(t.body) - else: - self.fill("try") + if t.finalbody: + self.fill("finally") self.enter() - self.dispatch(t.body) + self.dispatch(t.finalbody) self.leave() - self.fill("finally") - self.enter() - self.dispatch(t.finalbody) - self.leave() - def _ExceptHandler(self, t): self.fill("except") if t.type: @@ -296,10 +293,7 @@ def _With(self, t): self.fill("with ") - self.dispatch(t.context_expr) - if t.optional_vars: - self.write(" as ") - self.dispatch(t.optional_vars) + interleave(lambda: self.write(", "), self.dispatch, t.items) self.enter() self.dispatch(t.body) self.leave() @@ -472,6 +466,10 @@ self.dispatch(t.slice) self.write("]") + def _Starred(self, t): + self.write("*") + self.dispatch(t.value) + # slice def _Ellipsis(self, t): self.write("...") @@ -560,6 +558,12 @@ if t.asname: self.write(" as "+t.asname) + def _withitem(self, t): + self.dispatch(t.context_expr) + if t.optional_vars: + self.write(" as ") + self.dispatch(t.optional_vars) + def roundtrip(filename, output=sys.stdout): with open(filename, "rb") as pyfile: encoding = tokenize.detect_encoding(pyfile.readline)[0] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 20:13:36 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 06 May 2012 20:13:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Really_fix_test=5Fgzip_fail?= =?utf8?q?ures_on_Windows=2E?= Message-ID: http://hg.python.org/cpython/rev/875ba78bee10 changeset: 76804:875ba78bee10 user: Nadeem Vawda date: Sun May 06 19:24:18 2012 +0200 summary: Really fix test_gzip failures on Windows. files: Lib/test/test_gzip.py | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_gzip.py b/Lib/test/test_gzip.py --- a/Lib/test/test_gzip.py +++ b/Lib/test/test_gzip.py @@ -409,19 +409,20 @@ self.assertEqual(file_data, uncompressed * 2) def test_text_modes(self): - uncompressed = data1.decode("ascii").replace("\n", os.linesep) * 50 + uncompressed = data1.decode("ascii") * 50 + uncompressed_raw = uncompressed.replace("\n", os.linesep) with gzip.open(self.filename, "wt") as f: f.write(uncompressed) with open(self.filename, "rb") as f: file_data = gzip.decompress(f.read()).decode("ascii") - self.assertEqual(file_data, uncompressed) + self.assertEqual(file_data, uncompressed_raw) with gzip.open(self.filename, "rt") as f: self.assertEqual(f.read(), uncompressed) with gzip.open(self.filename, "at") as f: f.write(uncompressed) with open(self.filename, "rb") as f: file_data = gzip.decompress(f.read()).decode("ascii") - self.assertEqual(file_data, uncompressed * 2) + self.assertEqual(file_data, uncompressed_raw * 2) def test_bad_params(self): # Test invalid parameter combinations. @@ -436,12 +437,13 @@ def test_encoding(self): # Test non-default encoding. - uncompressed = data1.decode("ascii").replace("\n", os.linesep) * 50 + uncompressed = data1.decode("ascii") * 50 + uncompressed_raw = uncompressed.replace("\n", os.linesep) with gzip.open(self.filename, "wt", encoding="utf-16") as f: f.write(uncompressed) with open(self.filename, "rb") as f: file_data = gzip.decompress(f.read()).decode("utf-16") - self.assertEqual(file_data, uncompressed) + self.assertEqual(file_data, uncompressed_raw) with gzip.open(self.filename, "rt", encoding="utf-16") as f: self.assertEqual(f.read(), uncompressed) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 21:40:14 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 21:40:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_document_ti?= =?utf8?q?tle_for_Sphinx=2E?= Message-ID: http://hg.python.org/cpython/rev/8b59c83eed58 changeset: 76805:8b59c83eed58 branch: 3.2 parent: 76802:c80576303892 user: Georg Brandl date: Sun May 06 21:39:35 2012 +0200 summary: Fix document title for Sphinx. files: Doc/howto/argparse.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -1,6 +1,6 @@ -************************ -:mod:`argparse` Tutorial -************************ +***************** +Argparse Tutorial +***************** :author: Tshepang Lekhonkhobe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 21:40:21 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 21:40:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_with_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/6e799c0091db changeset: 76806:6e799c0091db parent: 76804:875ba78bee10 parent: 76805:8b59c83eed58 user: Georg Brandl date: Sun May 06 21:39:59 2012 +0200 summary: Merge with 3.2. files: Doc/howto/argparse.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -1,6 +1,6 @@ -************************ -:mod:`argparse` Tutorial -************************ +***************** +Argparse Tutorial +***************** :author: Tshepang Lekhonkhobe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 21:40:22 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 06 May 2012 21:40:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_document_ti?= =?utf8?q?tle_for_Sphinx=2E?= Message-ID: http://hg.python.org/cpython/rev/9bb9bd9237c1 changeset: 76807:9bb9bd9237c1 branch: 2.7 parent: 76795:549aa1460811 user: Georg Brandl date: Sun May 06 21:39:35 2012 +0200 summary: Fix document title for Sphinx. files: Doc/howto/argparse.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/argparse.rst b/Doc/howto/argparse.rst --- a/Doc/howto/argparse.rst +++ b/Doc/howto/argparse.rst @@ -1,6 +1,6 @@ -************************ -:mod:`argparse` Tutorial -************************ +***************** +Argparse Tutorial +***************** :author: Tshepang Lekhonkhobe -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 23:07:51 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 06 May 2012 23:07:51 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_lzma=2E=7Bencode=2Cdeco?= =?utf8?b?ZGV9X2ZpbHRlcl9wcm9wZXJ0aWVzKCku?= Message-ID: http://hg.python.org/cpython/rev/9118ef2b651a changeset: 76808:9118ef2b651a parent: 76806:6e799c0091db user: Nadeem Vawda date: Sun May 06 23:01:27 2012 +0200 summary: Add lzma.{encode,decode}_filter_properties(). files: Doc/library/lzma.rst | 26 ++++ Lib/lzma.py | 1 + Lib/test/test_lzma.py | 43 ++++++ Modules/_lzmamodule.c | 186 +++++++++++++++++++++++++++++- 4 files changed, 252 insertions(+), 4 deletions(-) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -235,6 +235,32 @@ feature set. +.. function:: encode_filter_properties(filter) + + Return a :class:`bytes` object encoding the options (properties) of the + filter specified by *filter* (a dictionary). + + *filter* is interpreted as a filter specifier, as described in + :ref:`filter-chain-specs`. + + The returned data does not include the filter ID itself, only the options. + + This function is primarily of interest to users implementing custom file + formats. + + +.. function:: decode_filter_properties(filter_id, encoded_props) + + Return a dictionary describing a filter with ID *filter_id*, and options + (properties) decoded from the :class:`bytes` object *encoded_props*. + + The returned dictionary is a filter specifier, as described in + :ref:`filter-chain-specs`. + + This function is primarily of interest to users implementing custom file + formats. + + .. _filter-chain-specs: Specifying custom filter chains diff --git a/Lib/lzma.py b/Lib/lzma.py --- a/Lib/lzma.py +++ b/Lib/lzma.py @@ -19,6 +19,7 @@ "LZMACompressor", "LZMADecompressor", "LZMAFile", "LZMAError", "compress", "decompress", "check_is_supported", + "encode_filter_properties", "decode_filter_properties", ] import io diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -944,6 +944,49 @@ # This value should not be a valid check ID. self.assertFalse(lzma.check_is_supported(lzma.CHECK_UNKNOWN)) + def test_encode_filter_properties(self): + with self.assertRaises(TypeError): + lzma.encode_filter_properties(b"not a dict") + with self.assertRaises(ValueError): + lzma.encode_filter_properties({"id": 0x100}) + with self.assertRaises(ValueError): + lzma.encode_filter_properties({"id": lzma.FILTER_LZMA2, "junk": 12}) + with self.assertRaises(lzma.LZMAError): + lzma.encode_filter_properties({"id": lzma.FILTER_DELTA, + "dist": 9001}) + + # Test with parameters used by zipfile module. + props = lzma.encode_filter_properties({ + "id": lzma.FILTER_LZMA1, + "pb": 2, + "lp": 0, + "lc": 3, + "dict_size": 8 << 20, + }) + self.assertEqual(props, b"]\x00\x00\x80\x00") + + def test_decode_filter_properties(self): + with self.assertRaises(TypeError): + lzma.decode_filter_properties(lzma.FILTER_X86, {"should be": bytes}) + with self.assertRaises(lzma.LZMAError): + lzma.decode_filter_properties(lzma.FILTER_DELTA, b"too long") + + # Test with parameters used by zipfile module. + filterspec = lzma.decode_filter_properties( + lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00") + self.assertEqual(filterspec["id"], lzma.FILTER_LZMA1) + self.assertEqual(filterspec["pb"], 2) + self.assertEqual(filterspec["lp"], 0) + self.assertEqual(filterspec["lc"], 3) + self.assertEqual(filterspec["dict_size"], 8 << 20) + + def test_filter_properties_roundtrip(self): + spec1 = lzma.decode_filter_properties( + lzma.FILTER_LZMA1, b"]\x00\x00\x80\x00") + reencoded = lzma.encode_filter_properties(spec1) + spec2 = lzma.decode_filter_properties(lzma.FILTER_LZMA1, reencoded) + self.assertEqual(spec1, spec2) + # Test data: diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -137,6 +137,9 @@ uint32_t - the "I" (unsigned int) specifier is the right size, but silently ignores overflows on conversion. + lzma_vli - the "K" (unsigned PY_LONG_LONG) specifier is the right + size, but like "I" it silently ignores overflows on conversion. + lzma_mode and lzma_match_finder - these are enumeration types, and so the size of each is implementation-defined. Worse, different enum types can be of different sizes within the same program, so @@ -147,12 +150,12 @@ static int \ FUNCNAME(PyObject *obj, void *ptr) \ { \ - unsigned long val; \ + unsigned PY_LONG_LONG val; \ \ - val = PyLong_AsUnsignedLong(obj); \ + val = PyLong_AsUnsignedLongLong(obj); \ if (PyErr_Occurred()) \ return 0; \ - if ((unsigned long)(TYPE)val != val) { \ + if ((unsigned PY_LONG_LONG)(TYPE)val != val) { \ PyErr_SetString(PyExc_OverflowError, \ "Value too large for " #TYPE " type"); \ return 0; \ @@ -162,13 +165,17 @@ } INT_TYPE_CONVERTER_FUNC(uint32_t, uint32_converter) +INT_TYPE_CONVERTER_FUNC(lzma_vli, lzma_vli_converter) INT_TYPE_CONVERTER_FUNC(lzma_mode, lzma_mode_converter) INT_TYPE_CONVERTER_FUNC(lzma_match_finder, lzma_mf_converter) #undef INT_TYPE_CONVERTER_FUNC -/* Filter specifier parsing functions. */ +/* Filter specifier parsing. + + This code handles converting filter specifiers (Python dicts) into + the C lzma_filter structs expected by liblzma. */ static void * parse_filter_spec_lzma(PyObject *spec) @@ -358,6 +365,88 @@ } +/* Filter specifier construction. + + This code handles converting C lzma_filter structs into + Python-level filter specifiers (represented as dicts). */ + +static int +spec_add_field(PyObject *spec, _Py_Identifier *key, unsigned PY_LONG_LONG value) +{ + int status; + PyObject *value_object; + + value_object = PyLong_FromUnsignedLongLong(value); + if (value_object == NULL) + return -1; + + status = _PyDict_SetItemId(spec, key, value_object); + Py_DECREF(value_object); + return status; +} + +static PyObject * +build_filter_spec(const lzma_filter *f) +{ + PyObject *spec; + + spec = PyDict_New(); + if (spec == NULL) + return NULL; + +#define ADD_FIELD(SOURCE, FIELD) \ + do { \ + _Py_IDENTIFIER(FIELD); \ + if (spec_add_field(spec, &PyId_##FIELD, SOURCE->FIELD) == -1) \ + goto error;\ + } while (0) + + ADD_FIELD(f, id); + + switch (f->id) { + case LZMA_FILTER_LZMA1: + case LZMA_FILTER_LZMA2: { + lzma_options_lzma *options = f->options; + ADD_FIELD(options, dict_size); + ADD_FIELD(options, lc); + ADD_FIELD(options, lp); + ADD_FIELD(options, pb); + ADD_FIELD(options, mode); + ADD_FIELD(options, nice_len); + ADD_FIELD(options, mf); + ADD_FIELD(options, depth); + break; + } + case LZMA_FILTER_DELTA: { + lzma_options_delta *options = f->options; + ADD_FIELD(options, dist); + break; + } + case LZMA_FILTER_X86: + case LZMA_FILTER_POWERPC: + case LZMA_FILTER_IA64: + case LZMA_FILTER_ARM: + case LZMA_FILTER_ARMTHUMB: + case LZMA_FILTER_SPARC: { + lzma_options_bcj *options = f->options; + ADD_FIELD(options, start_offset); + break; + } + default: + PyErr_Format(PyExc_ValueError, "Invalid filter ID: %llu", f->id); + goto error; + } + +#undef ADD_FIELD + + return spec; + +error: + Py_DECREF(spec); + return NULL; +} + + /* LZMACompressor class. */ static PyObject * @@ -1005,11 +1094,100 @@ } +PyDoc_STRVAR(encode_filter_properties_doc, +"encode_filter_properties(filter) -> bytes\n" +"\n" +"Return a bytes object encoding the options (properties) of the filter\n" +"specified by *filter* (a dict).\n" +"\n" +"The result does not include the filter ID itself, only the options.\n" +"\n" +"This function is primarily of interest to users implementing custom\n" +"file formats.\n"); + +static PyObject * +encode_filter_properties(PyObject *self, PyObject *args) +{ + PyObject *filterspec; + lzma_filter filter; + lzma_ret lzret; + uint32_t encoded_size; + PyObject *result = NULL; + + if (!PyArg_ParseTuple(args, "O:encode_filter_properties", &filterspec)) + return NULL; + + if (parse_filter_spec(&filter, filterspec) == NULL) + return NULL; + + lzret = lzma_properties_size(&encoded_size, &filter); + if (catch_lzma_error(lzret)) + goto error; + + result = PyBytes_FromStringAndSize(NULL, encoded_size); + if (result == NULL) + goto error; + + lzret = lzma_properties_encode( + &filter, (uint8_t *)PyBytes_AS_STRING(result)); + if (catch_lzma_error(lzret)) + goto error; + + PyMem_Free(filter.options); + return result; + +error: + Py_XDECREF(result); + PyMem_Free(filter.options); + return NULL; +} + + +PyDoc_STRVAR(decode_filter_properties_doc, +"decode_filter_properties(filter_id, encoded_props) -> dict\n" +"\n" +"Return a dict describing a filter with ID *filter_id*, and options\n" +"(properties) decoded from the bytes object *encoded_props*.\n" +"\n" +"This function is primarily of interest to users implementing custom\n" +"file formats.\n"); + +static PyObject * +decode_filter_properties(PyObject *self, PyObject *args) +{ + Py_buffer encoded_props; + lzma_filter filter; + lzma_ret lzret; + PyObject *result = NULL; + + if (!PyArg_ParseTuple(args, "O&y*:decode_filter_properties", + lzma_vli_converter, &filter.id, &encoded_props)) + return NULL; + + lzret = lzma_properties_decode( + &filter, NULL, encoded_props.buf, encoded_props.len); + PyBuffer_Release(&encoded_props); + if (catch_lzma_error(lzret)) + return NULL; + + result = build_filter_spec(&filter); + + /* We use vanilla free() here instead of PyMem_Free() - filter.options was + allocated by lzma_properties_decode() using the default allocator. */ + free(filter.options); + return result; +} + + /* Module initialization. */ static PyMethodDef module_methods[] = { {"check_is_supported", (PyCFunction)check_is_supported, METH_VARARGS, check_is_supported_doc}, + {"encode_filter_properties", (PyCFunction)encode_filter_properties, + METH_VARARGS, encode_filter_properties_doc}, + {"decode_filter_properties", (PyCFunction)decode_filter_properties, + METH_VARARGS, decode_filter_properties_doc}, {NULL} }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 6 23:07:52 2012 From: python-checkins at python.org (nadeem.vawda) Date: Sun, 06 May 2012 23:07:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Rename_lzma=2Echeck=5Fis=5F?= =?utf8?q?supported=28=29_to_is=5Fcheck=5Fsupported=28=29_to_avoid_grammat?= =?utf8?q?ical?= Message-ID: http://hg.python.org/cpython/rev/ab57e29157bb changeset: 76809:ab57e29157bb user: Nadeem Vawda date: Sun May 06 23:01:51 2012 +0200 summary: Rename lzma.check_is_supported() to is_check_supported() to avoid grammatical confusion. files: Doc/library/lzma.rst | 2 +- Lib/lzma.py | 2 +- Lib/test/test_lzma.py | 8 ++++---- Modules/_lzmamodule.c | 12 ++++++------ 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst --- a/Doc/library/lzma.rst +++ b/Doc/library/lzma.rst @@ -225,7 +225,7 @@ Miscellaneous ------------- -.. function:: check_is_supported(check) +.. function:: is_check_supported(check) Returns true if the given integrity check is supported on this system. diff --git a/Lib/lzma.py b/Lib/lzma.py --- a/Lib/lzma.py +++ b/Lib/lzma.py @@ -18,7 +18,7 @@ "MODE_FAST", "MODE_NORMAL", "PRESET_DEFAULT", "PRESET_EXTREME", "LZMACompressor", "LZMADecompressor", "LZMAFile", "LZMAError", - "compress", "decompress", "check_is_supported", + "compress", "decompress", "is_check_supported", "encode_filter_properties", "decode_filter_properties", ] diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py --- a/Lib/test/test_lzma.py +++ b/Lib/test/test_lzma.py @@ -935,14 +935,14 @@ def test_is_check_supported(self): # CHECK_NONE and CHECK_CRC32 should always be supported, # regardless of the options liblzma was compiled with. - self.assertTrue(lzma.check_is_supported(lzma.CHECK_NONE)) - self.assertTrue(lzma.check_is_supported(lzma.CHECK_CRC32)) + self.assertTrue(lzma.is_check_supported(lzma.CHECK_NONE)) + self.assertTrue(lzma.is_check_supported(lzma.CHECK_CRC32)) # The .xz format spec cannot store check IDs above this value. - self.assertFalse(lzma.check_is_supported(lzma.CHECK_ID_MAX + 1)) + self.assertFalse(lzma.is_check_supported(lzma.CHECK_ID_MAX + 1)) # This value should not be a valid check ID. - self.assertFalse(lzma.check_is_supported(lzma.CHECK_UNKNOWN)) + self.assertFalse(lzma.is_check_supported(lzma.CHECK_UNKNOWN)) def test_encode_filter_properties(self): with self.assertRaises(TypeError): diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -1075,19 +1075,19 @@ /* Module-level functions. */ -PyDoc_STRVAR(check_is_supported_doc, -"check_is_supported(check_id) -> bool\n" +PyDoc_STRVAR(is_check_supported_doc, +"is_check_supported(check_id) -> bool\n" "\n" "Test whether the given integrity check is supported.\n" "\n" "Always returns True for CHECK_NONE and CHECK_CRC32.\n"); static PyObject * -check_is_supported(PyObject *self, PyObject *args) +is_check_supported(PyObject *self, PyObject *args) { int check_id; - if (!PyArg_ParseTuple(args, "i:check_is_supported", &check_id)) + if (!PyArg_ParseTuple(args, "i:is_check_supported", &check_id)) return NULL; return PyBool_FromLong(lzma_check_is_supported(check_id)); @@ -1182,8 +1182,8 @@ /* Module initialization. */ static PyMethodDef module_methods[] = { - {"check_is_supported", (PyCFunction)check_is_supported, - METH_VARARGS, check_is_supported_doc}, + {"is_check_supported", (PyCFunction)is_check_supported, + METH_VARARGS, is_check_supported_doc}, {"encode_filter_properties", (PyCFunction)encode_filter_properties, METH_VARARGS, encode_filter_properties_doc}, {"decode_filter_properties", (PyCFunction)decode_filter_properties, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 00:41:02 2012 From: python-checkins at python.org (nadeem.vawda) Date: Mon, 07 May 2012 00:41:02 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_buildbot_failures_in_te?= =?utf8?q?st=5Flzma=2E?= Message-ID: http://hg.python.org/cpython/rev/10ccbb90a8e9 changeset: 76810:10ccbb90a8e9 user: Nadeem Vawda date: Mon May 07 00:40:57 2012 +0200 summary: Fix buildbot failures in test_lzma. files: Modules/_lzmamodule.c | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-) diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -404,17 +404,16 @@ ADD_FIELD(f, id); switch (f->id) { - case LZMA_FILTER_LZMA1: - case LZMA_FILTER_LZMA2: { + /* For LZMA1 filters, lzma_properties_{encode,decode}() only look at the + lc, lp, pb, and dict_size fields. For LZMA2 filters, only the + dict_size field is used. */ + case LZMA_FILTER_LZMA1: { lzma_options_lzma *options = f->options; - ADD_FIELD(options, dict_size); ADD_FIELD(options, lc); ADD_FIELD(options, lp); ADD_FIELD(options, pb); - ADD_FIELD(options, mode); - ADD_FIELD(options, nice_len); - ADD_FIELD(options, mf); - ADD_FIELD(options, depth); + case LZMA_FILTER_LZMA2: + ADD_FIELD(options, dict_size); break; } case LZMA_FILTER_DELTA: { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 01:13:26 2012 From: python-checkins at python.org (richard.oudkerk) Date: Mon, 07 May 2012 01:13:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_Misc/NEWS_entry_for_rev?= =?utf8?q?_b4a1d9287780?= Message-ID: http://hg.python.org/cpython/rev/165bd3483d0d changeset: 76811:165bd3483d0d user: Richard Oudkerk date: Mon May 07 00:12:02 2012 +0100 summary: Add Misc/NEWS entry for rev b4a1d9287780 files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -17,6 +17,9 @@ Library ------- +- Add support for timeouts to the acquire() methods of + multiprocessing's lock/semaphore/condition proxies. + - Issue #13989: Add support for text mode to gzip.open(). - Issue #14127: The os.stat() result object now provides three additional -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon May 7 05:44:49 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 07 May 2012 05:44:49 +0200 Subject: [Python-checkins] Daily reference leaks (165bd3483d0d): sum=0 Message-ID: results for 165bd3483d0d on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogM91qJz', '-x'] From python-checkins at python.org Mon May 7 11:26:10 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 11:26:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_issue_numbe?= =?utf8?q?r_in_Misc/NEWS=2E?= Message-ID: http://hg.python.org/cpython/rev/84a4bbcbed1d changeset: 76812:84a4bbcbed1d branch: 3.2 parent: 76805:8b59c83eed58 user: Mark Dickinson date: Mon May 07 10:24:02 2012 +0100 summary: Fix issue number in Misc/NEWS. files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -228,7 +228,7 @@ Tools/Demos ----------- -- Issue #14965: Fix missing support for starred assignments in +- Issue #14695: Fix missing support for starred assignments in Tools/parser/unparse.py. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 11:26:11 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 11:26:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_Misc/NEWS_issue_number_fix_from_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/3c3a0d4ccbd3 changeset: 76813:3c3a0d4ccbd3 parent: 76811:165bd3483d0d parent: 76812:84a4bbcbed1d user: Mark Dickinson date: Mon May 07 10:25:56 2012 +0100 summary: Merge Misc/NEWS issue number fix from 3.2 files: Misc/NEWS | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -34,7 +34,7 @@ Tools/Demos ----------- -- Issue #14965: Bring Tools/parser/unparse.py support up to date with +- Issue #14695: Bring Tools/parser/unparse.py support up to date with the Python 3.3 Grammar. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 11:27:30 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 11:27:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314695=3A_Run_Tools?= =?utf8?q?/parser/test=5Funparse=2Epy_as_part_of_test=5Ftools=2E?= Message-ID: http://hg.python.org/cpython/rev/f9344a3eaaa6 changeset: 76814:f9344a3eaaa6 user: Mark Dickinson date: Mon May 07 10:27:23 2012 +0100 summary: Issue #14695: Run Tools/parser/test_unparse.py as part of test_tools. files: Lib/test/test_tools.py | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -122,6 +122,11 @@ self.assertTrue(wmock.open.called) +# Run the tests in Tools/parser/test_unparse.py +with support.DirsOnSysPath(os.path.join(basepath, 'parser')): + from test_unparse import UnparseTestCase, DirectoryTestCase + + def test_main(): support.run_unittest(*[obj for obj in globals().values() if isinstance(obj, type)]) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 11:37:44 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 11:37:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_John_Regehr_to_Misc/ACK?= =?utf8?q?S_for_his_help_with_finding_integer_overflows_=28issue?= Message-ID: http://hg.python.org/cpython/rev/c9c2031cf16d changeset: 76815:c9c2031cf16d user: Mark Dickinson date: Mon May 07 10:37:37 2012 +0100 summary: Add John Regehr to Misc/ACKS for his help with finding integer overflows (issue #9530). files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -836,6 +836,7 @@ Gareth Rees Steve Reeves Lennart Regebro +John Regehr Federico Reghenzani Ofir Reichenberg Sean Reifschneider -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 11:45:43 2012 From: python-checkins at python.org (larry.hastings) Date: Mon, 07 May 2012 11:45:43 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314705=3A_Added_sup?= =?utf8?q?port_for_the_new_=27p=27_format_unit_to_skipitem=28=29=2E?= Message-ID: http://hg.python.org/cpython/rev/e4617650f006 changeset: 76816:e4617650f006 user: Larry Hastings date: Mon May 07 02:44:50 2012 -0700 summary: Issue #14705: Added support for the new 'p' format unit to skipitem(). files: Python/getargs.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1629,6 +1629,7 @@ case 'D': /* complex double */ case 'c': /* char */ case 'C': /* unicode char */ + case 'p': /* boolean predicate */ { (void) va_arg(*p_va, void *); break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 12:21:09 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 12:21:09 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314700=3A_Fix_two_b?= =?utf8?q?roken_and_undefined-behaviour-inducing_overflow_checks?= Message-ID: http://hg.python.org/cpython/rev/064c2d0483f8 changeset: 76817:064c2d0483f8 user: Mark Dickinson date: Mon May 07 11:20:50 2012 +0100 summary: Issue #14700: Fix two broken and undefined-behaviour-inducing overflow checks in old-style string formatting. Thanks Serhiy Storchaka for report and original patch. files: Lib/test/string_tests.py | 4 ++++ Misc/NEWS | 3 +++ Objects/unicodeobject.c | 4 ++-- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py --- a/Lib/test/string_tests.py +++ b/Lib/test/string_tests.py @@ -1197,6 +1197,10 @@ self.checkraises(TypeError, '%10.*f', '__mod__', ('foo', 42.)) self.checkraises(ValueError, '%10', '__mod__', (42,)) + # Outrageously large width or precision should raise ValueError. + self.checkraises(ValueError, '%%%df' % (2**64), '__mod__', (3.2)) + self.checkraises(ValueError, '%%.%df' % (2**64), '__mod__', (3.2)) + def test_floatformatting(self): # float formatting for prec in range(100): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14700: Fix two broken and undefined-behaviour-inducing overflow checks + in old-style string formatting. + - Issue #14705: The PyArg_Parse() family of functions now support the 'p' format unit, which accepts a "boolean predicate" argument. It converts any Python value into an integer--0 if it is "false", and 1 otherwise. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13933,7 +13933,7 @@ c = PyUnicode_READ(fmtkind, fmt, fmtpos++); if (c < '0' || c > '9') break; - if ((width*10) / 10 != width) { + if (width > (PY_SSIZE_T_MAX - (c - '0')) / 10) { PyErr_SetString(PyExc_ValueError, "width too big"); goto onError; @@ -13968,7 +13968,7 @@ c = PyUnicode_READ(fmtkind, fmt, fmtpos++); if (c < '0' || c > '9') break; - if ((prec*10) / 10 != prec) { + if (prec > (INT_MAX - (c - '0')) / 10) { PyErr_SetString(PyExc_ValueError, "prec too big"); goto onError; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 13:03:23 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 13:03:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0NzAx?= =?utf8?q?=3A__Add_missing_support_for_=27raise_=2E=2E=2E_from=27_in_parse?= =?utf8?q?r_module=2E?= Message-ID: http://hg.python.org/cpython/rev/fc17f70292f6 changeset: 76818:fc17f70292f6 branch: 3.2 parent: 76812:84a4bbcbed1d user: Mark Dickinson date: Mon May 07 12:01:27 2012 +0100 summary: Issue #14701: Add missing support for 'raise ... from' in parser module. files: Lib/test/test_parser.py | 8 ++++++++ Misc/NEWS | 2 ++ Modules/parsermodule.c | 23 +++++++++++------------ 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -297,6 +297,14 @@ self.check_suite("[*a, *b] = y") self.check_suite("for [*x, b] in x: pass") + def test_raise_statement(self): + self.check_suite("raise\n") + self.check_suite("raise e\n") + self.check_suite("try:\n" + " suite\n" + "except Exception as e:\n" + " raise ValueError from e\n") + # # Second, we take *invalid* trees and make sure we get ParserError diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -61,6 +61,8 @@ Library ------- +- Issue #14701: Fix missing support for 'raise ... from' in parser module. + - Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye. diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -1608,31 +1608,30 @@ } +/* + * raise_stmt: + * + * 'raise' [test ['from' test]] + */ static int validate_raise_stmt(node *tree) { int nch = NCH(tree); int res = (validate_ntype(tree, raise_stmt) - && ((nch == 1) || (nch == 2) || (nch == 4) || (nch == 6))); + && ((nch == 1) || (nch == 2) || (nch == 4))); + + if (!res && !PyErr_Occurred()) + (void) validate_numnodes(tree, 2, "raise"); if (res) { res = validate_name(CHILD(tree, 0), "raise"); if (res && (nch >= 2)) res = validate_test(CHILD(tree, 1)); - if (res && nch > 2) { - res = (validate_comma(CHILD(tree, 2)) + if (res && (nch == 4)) { + res = (validate_name(CHILD(tree, 2), "from") && validate_test(CHILD(tree, 3))); - if (res && (nch > 4)) - res = (validate_comma(CHILD(tree, 4)) - && validate_test(CHILD(tree, 5))); } } - else - (void) validate_numnodes(tree, 2, "raise"); - if (res && (nch == 4)) - res = (validate_comma(CHILD(tree, 2)) - && validate_test(CHILD(tree, 3))); - return (res); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 13:03:24 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 13:03:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314701=3A_Merge_fix_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/0dd0d56bdcc1 changeset: 76819:0dd0d56bdcc1 parent: 76817:064c2d0483f8 parent: 76818:fc17f70292f6 user: Mark Dickinson date: Mon May 07 12:03:11 2012 +0100 summary: Issue #14701: Merge fix from 3.2. files: Lib/test/test_parser.py | 8 ++++++++ Misc/NEWS | 2 ++ Modules/parsermodule.c | 23 +++++++++++------------ 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -301,6 +301,14 @@ self.check_suite("[*a, *b] = y") self.check_suite("for [*x, b] in x: pass") + def test_raise_statement(self): + self.check_suite("raise\n") + self.check_suite("raise e\n") + self.check_suite("try:\n" + " suite\n" + "except Exception as e:\n" + " raise ValueError from e\n") + # # Second, we take *invalid* trees and make sure we get ParserError diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,8 @@ Library ------- +- Issue #14701: Fix missing support for 'raise ... from' in parser module. + - Add support for timeouts to the acquire() methods of multiprocessing's lock/semaphore/condition proxies. diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -1611,31 +1611,30 @@ } +/* + * raise_stmt: + * + * 'raise' [test ['from' test]] + */ static int validate_raise_stmt(node *tree) { int nch = NCH(tree); int res = (validate_ntype(tree, raise_stmt) - && ((nch == 1) || (nch == 2) || (nch == 4) || (nch == 6))); + && ((nch == 1) || (nch == 2) || (nch == 4))); + + if (!res && !PyErr_Occurred()) + (void) validate_numnodes(tree, 2, "raise"); if (res) { res = validate_name(CHILD(tree, 0), "raise"); if (res && (nch >= 2)) res = validate_test(CHILD(tree, 1)); - if (res && nch > 2) { - res = (validate_comma(CHILD(tree, 2)) + if (res && (nch == 4)) { + res = (validate_name(CHILD(tree, 2), "from") && validate_test(CHILD(tree, 3))); - if (res && (nch > 4)) - res = (validate_comma(CHILD(tree, 4)) - && validate_test(CHILD(tree, 5))); } } - else - (void) validate_numnodes(tree, 2, "raise"); - if (res && (nch == 4)) - res = (validate_comma(CHILD(tree, 2)) - && validate_test(CHILD(tree, 3))); - return (res); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 13:08:10 2012 From: python-checkins at python.org (victor.stinner) Date: Mon, 07 May 2012 13:08:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314716=3A_str=2Efor?= =?utf8?q?mat=28=29_now_uses_the_new_=22unicode_writer=22_API_instead_of_t?= =?utf8?q?he?= Message-ID: http://hg.python.org/cpython/rev/7be716a47e9d changeset: 76820:7be716a47e9d user: Victor Stinner date: Mon May 07 12:47:02 2012 +0200 summary: Close #14716: str.format() now uses the new "unicode writer" API instead of the PyAccu API. For example, it makes str.format() from 25% to 30% faster on Linux. files: Objects/stringlib/unicode_format.h | 60 +-- Objects/unicodeobject.c | 258 ++++++++-------- 2 files changed, 148 insertions(+), 170 deletions(-) diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h --- a/Objects/stringlib/unicode_format.h +++ b/Objects/stringlib/unicode_format.h @@ -2,8 +2,6 @@ unicode_format.h -- implementation of str.format(). */ -#include "accu.h" - /* Defines for more efficiently reallocating the string buffer */ #define INITIAL_SIZE_INCREMENT 100 #define SIZE_MULTIPLIER 2 @@ -112,33 +110,6 @@ /************************************************************************/ -/*********** Output string management functions ****************/ -/************************************************************************/ - -/* - output_data dumps characters into our output string - buffer. - - In some cases, it has to reallocate the string. - - It returns a status: 0 for a failed reallocation, - 1 for success. -*/ -static int -output_data(_PyAccu *acc, PyObject *s, Py_ssize_t start, Py_ssize_t end) -{ - PyObject *substring; - int r; - - substring = PyUnicode_Substring(s, start, end); - if (substring == NULL) - return 0; - r = _PyAccu_Accumulate(acc, substring); - Py_DECREF(substring); - return r == 0; -} - -/************************************************************************/ /*********** Format string parsing -- integers and identifiers *********/ /************************************************************************/ @@ -523,7 +494,7 @@ appends to the output. */ static int -render_field(PyObject *fieldobj, SubString *format_spec, _PyAccu *acc) +render_field(PyObject *fieldobj, SubString *format_spec, unicode_writer_t *writer) { int ok = 0; PyObject *result = NULL; @@ -566,7 +537,8 @@ goto done; assert(PyUnicode_Check(result)); - ok = output_data(acc, result, 0, PyUnicode_GET_LENGTH(result)); + + ok = (unicode_writer_write_str(writer, result, 0, PyUnicode_GET_LENGTH(result)) == 0); done: Py_XDECREF(format_spec_object); Py_XDECREF(result); @@ -831,7 +803,7 @@ static int output_markup(SubString *field_name, SubString *format_spec, int format_spec_needs_expanding, Py_UCS4 conversion, - _PyAccu *acc, PyObject *args, PyObject *kwargs, + unicode_writer_t *writer, PyObject *args, PyObject *kwargs, int recursion_depth, AutoNumber *auto_number) { PyObject *tmp = NULL; @@ -872,7 +844,7 @@ else actual_format_spec = format_spec; - if (render_field(fieldobj, actual_format_spec, acc) == 0) + if (render_field(fieldobj, actual_format_spec, writer) == 0) goto done; result = 1; @@ -892,7 +864,7 @@ */ static int do_markup(SubString *input, PyObject *args, PyObject *kwargs, - _PyAccu *acc, int recursion_depth, AutoNumber *auto_number) + unicode_writer_t *writer, int recursion_depth, AutoNumber *auto_number) { MarkupIterator iter; int format_spec_needs_expanding; @@ -902,17 +874,21 @@ SubString field_name; SubString format_spec; Py_UCS4 conversion; + int err; MarkupIterator_init(&iter, input->str, input->start, input->end); while ((result = MarkupIterator_next(&iter, &literal, &field_present, &field_name, &format_spec, &conversion, &format_spec_needs_expanding)) == 2) { - if (!output_data(acc, literal.str, literal.start, literal.end)) + err = unicode_writer_write_str(writer, + literal.str, literal.start, + literal.end - literal.start); + if (err == -1) return 0; if (field_present) if (!output_markup(&field_name, &format_spec, - format_spec_needs_expanding, conversion, acc, + format_spec_needs_expanding, conversion, writer, args, kwargs, recursion_depth, auto_number)) return 0; } @@ -928,7 +904,8 @@ build_string(SubString *input, PyObject *args, PyObject *kwargs, int recursion_depth, AutoNumber *auto_number) { - _PyAccu acc; + unicode_writer_t writer; + Py_ssize_t initlen; /* check the recursion level */ if (recursion_depth <= 0) { @@ -937,16 +914,17 @@ return NULL; } - if (_PyAccu_Init(&acc)) + initlen = PyUnicode_GET_LENGTH(input->str) + 100; + if (unicode_writer_init(&writer, initlen, 127) == -1) return NULL; - if (!do_markup(input, args, kwargs, &acc, recursion_depth, + if (!do_markup(input, args, kwargs, &writer, recursion_depth, auto_number)) { - _PyAccu_Destroy(&acc); + unicode_writer_dealloc(&writer); return NULL; } - return _PyAccu_Finish(&acc); + return unicode_writer_finish(&writer); } /************************************************************************/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13200,6 +13200,135 @@ return PyBool_FromLong(result); } +typedef struct { + PyObject *buffer; + void *data; + enum PyUnicode_Kind kind; + Py_UCS4 maxchar; + Py_ssize_t pos; +} unicode_writer_t; + +Py_LOCAL_INLINE(void) +unicode_writer_update(unicode_writer_t *writer) +{ + writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); + writer->data = PyUnicode_DATA(writer->buffer); + writer->kind = PyUnicode_KIND(writer->buffer); +} + +Py_LOCAL(int) +unicode_writer_init(unicode_writer_t *writer, + Py_ssize_t length, Py_UCS4 maxchar) +{ + writer->pos = 0; + writer->buffer = PyUnicode_New(length, maxchar); + if (writer->buffer == NULL) + return -1; + unicode_writer_update(writer); + return 0; +} + +Py_LOCAL_INLINE(int) +unicode_writer_prepare(unicode_writer_t *writer, + Py_ssize_t length, Py_UCS4 maxchar) +{ + Py_ssize_t newlen; + PyObject *newbuffer; + + if (length > PY_SSIZE_T_MAX - writer->pos) { + PyErr_NoMemory(); + return -1; + } + newlen = writer->pos + length; + + if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { + /* overallocate 25% to limit the number of resize */ + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + newlen += newlen / 4; + + if (maxchar > writer->maxchar) { + /* resize + widen */ + newbuffer = PyUnicode_New(newlen, maxchar); + if (newbuffer == NULL) + return -1; + PyUnicode_CopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); + Py_DECREF(writer->buffer); + } + else { + newbuffer = resize_compact(writer->buffer, newlen); + if (newbuffer == NULL) + return -1; + } + writer->buffer = newbuffer; + unicode_writer_update(writer); + } + else if (maxchar > writer->maxchar) { + if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) + return -1; + unicode_writer_update(writer); + } + return 0; +} + +Py_LOCAL_INLINE(int) +unicode_writer_write_str( + unicode_writer_t *writer, + PyObject *str, Py_ssize_t start, Py_ssize_t length) +{ + Py_UCS4 maxchar; + + assert(str != NULL); + assert(PyUnicode_Check(str)); + if (PyUnicode_READY(str) == -1) + return -1; + + assert(0 <= start); + assert(0 <= length); + assert(start + length <= PyUnicode_GET_LENGTH(str)); + if (length == 0) + return 0; + + maxchar = _PyUnicode_FindMaxChar(str, start, start + length); + if (unicode_writer_prepare(writer, length, maxchar) == -1) + return -1; + + assert((writer->pos + length) <= PyUnicode_GET_LENGTH(writer->buffer)); + copy_characters(writer->buffer, writer->pos, + str, start, length); + writer->pos += length; + return 0; +} + +Py_LOCAL_INLINE(int) +unicode_writer_write_char( + unicode_writer_t *writer, + Py_UCS4 ch) +{ + if (unicode_writer_prepare(writer, 1, ch) == -1) + return -1; + assert((writer->pos + 1) <= PyUnicode_GET_LENGTH(writer->buffer)); + PyUnicode_WRITE(writer->kind, writer->data, writer->pos, ch); + writer->pos += 1; + return 0; +} + +Py_LOCAL(PyObject *) +unicode_writer_finish(unicode_writer_t *writer) +{ + if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { + Py_DECREF(writer->buffer); + return NULL; + } + return writer->buffer; +} + +Py_LOCAL(void) +unicode_writer_dealloc(unicode_writer_t *writer) +{ + Py_CLEAR(writer->buffer); +} + #include "stringlib/unicode_format.h" PyDoc_STRVAR(format__doc__, @@ -13649,135 +13778,6 @@ return (Py_UCS4) -1; } -typedef struct { - PyObject *buffer; - void *data; - enum PyUnicode_Kind kind; - Py_UCS4 maxchar; - Py_ssize_t pos; -} unicode_writer_t; - -Py_LOCAL_INLINE(void) -unicode_writer_update(unicode_writer_t *writer) -{ - writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); - writer->data = PyUnicode_DATA(writer->buffer); - writer->kind = PyUnicode_KIND(writer->buffer); -} - -Py_LOCAL(int) -unicode_writer_init(unicode_writer_t *writer, - Py_ssize_t length, Py_UCS4 maxchar) -{ - writer->pos = 0; - writer->buffer = PyUnicode_New(length, maxchar); - if (writer->buffer == NULL) - return -1; - unicode_writer_update(writer); - return 0; -} - -Py_LOCAL_INLINE(int) -unicode_writer_prepare(unicode_writer_t *writer, - Py_ssize_t length, Py_UCS4 maxchar) -{ - Py_ssize_t newlen; - PyObject *newbuffer; - - if (length > PY_SSIZE_T_MAX - writer->pos) { - PyErr_NoMemory(); - return -1; - } - newlen = writer->pos + length; - - if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { - /* overallocate 25% to limit the number of resize */ - if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) - newlen += newlen / 4; - - if (maxchar > writer->maxchar) { - /* resize + widen */ - newbuffer = PyUnicode_New(newlen, maxchar); - if (newbuffer == NULL) - return -1; - PyUnicode_CopyCharacters(newbuffer, 0, - writer->buffer, 0, writer->pos); - Py_DECREF(writer->buffer); - } - else { - newbuffer = resize_compact(writer->buffer, newlen); - if (newbuffer == NULL) - return -1; - } - writer->buffer = newbuffer; - unicode_writer_update(writer); - } - else if (maxchar > writer->maxchar) { - if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) - return -1; - unicode_writer_update(writer); - } - return 0; -} - -Py_LOCAL_INLINE(int) -unicode_writer_write_str( - unicode_writer_t *writer, - PyObject *str, Py_ssize_t start, Py_ssize_t length) -{ - Py_UCS4 maxchar; - - assert(str != NULL); - assert(PyUnicode_Check(str)); - if (PyUnicode_READY(str) == -1) - return -1; - - assert(0 <= start); - assert(0 <= length); - assert(start + length <= PyUnicode_GET_LENGTH(str)); - if (length == 0) - return 0; - - maxchar = _PyUnicode_FindMaxChar(str, start, start + length); - if (unicode_writer_prepare(writer, length, maxchar) == -1) - return -1; - - assert((writer->pos + length) <= PyUnicode_GET_LENGTH(writer->buffer)); - copy_characters(writer->buffer, writer->pos, - str, start, length); - writer->pos += length; - return 0; -} - -Py_LOCAL_INLINE(int) -unicode_writer_write_char( - unicode_writer_t *writer, - Py_UCS4 ch) -{ - if (unicode_writer_prepare(writer, 1, ch) == -1) - return -1; - assert((writer->pos + 1) <= PyUnicode_GET_LENGTH(writer->buffer)); - PyUnicode_WRITE(writer->kind, writer->data, writer->pos, ch); - writer->pos += 1; - return 0; -} - -Py_LOCAL(PyObject *) -unicode_writer_finish(unicode_writer_t *writer) -{ - if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { - Py_DECREF(writer->buffer); - return NULL; - } - return writer->buffer; -} - -Py_LOCAL(void) -unicode_writer_dealloc(unicode_writer_t *writer) -{ - Py_CLEAR(writer->buffer); -} - PyObject * PyUnicode_Format(PyObject *format, PyObject *args) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 13:08:11 2012 From: python-checkins at python.org (victor.stinner) Date: Mon, 07 May 2012 13:08:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314716=3A_Change_in?= =?utf8?q?teger_overflow_check_in_unicode=5Fwriter=5Fprepare=28=29?= Message-ID: http://hg.python.org/cpython/rev/ab500b297900 changeset: 76821:ab500b297900 user: Victor Stinner date: Mon May 07 13:02:44 2012 +0200 summary: Issue #14716: Change integer overflow check in unicode_writer_prepare() to compute the limit at compile time instead of runtime. Patch writen by Serhiy Storchaka. files: Objects/unicodeobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13242,8 +13242,10 @@ newlen = writer->pos + length; if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { - /* overallocate 25% to limit the number of resize */ - if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + /* Overallocate 25% to limit the number of resize. + Check for integer overflow: + (newlen + newlen / 4) <= PY_SSIZE_T_MAX */ + if (newlen <= (PY_SSIZE_T_MAX - PY_SSIZE_T_MAX / 5)) newlen += newlen / 4; if (maxchar > writer->maxchar) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 17:36:45 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 17:36:45 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0Njk3?= =?utf8?q?=3A_Fix_missing_parser_module_support_for_set_displays_and_set?= Message-ID: http://hg.python.org/cpython/rev/129289144dfb changeset: 76822:129289144dfb branch: 3.2 parent: 76818:fc17f70292f6 user: Mark Dickinson date: Mon May 07 16:34:34 2012 +0100 summary: Issue #14697: Fix missing parser module support for set displays and set comprehensions. files: Lib/test/test_parser.py | 23 ++++++ Misc/NEWS | 3 + Modules/parsermodule.c | 102 +++++++++++++++++++++------ 3 files changed, 106 insertions(+), 22 deletions(-) diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -305,6 +305,29 @@ "except Exception as e:\n" " raise ValueError from e\n") + def test_set_displays(self): + self.check_expr('{2}') + self.check_expr('{2,}') + self.check_expr('{2, 3}') + self.check_expr('{2, 3,}') + + def test_dict_displays(self): + self.check_expr('{}') + self.check_expr('{a:b}') + self.check_expr('{a:b,}') + self.check_expr('{a:b, c:d}') + self.check_expr('{a:b, c:d,}') + + def test_set_comprehensions(self): + self.check_expr('{x for x in seq}') + self.check_expr('{f(x) for x in seq}') + self.check_expr('{f(x) for x in seq if condition(x)}') + + def test_dict_comprehensions(self): + self.check_expr('{x:x for x in seq}') + self.check_expr('{x**2:x[3] for x in seq if condition(x)}') + self.check_expr('{x:x for x in seq1 for y in seq2 if condition(x, y)}') + # # Second, we take *invalid* trees and make sure we get ParserError diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -61,6 +61,9 @@ Library ------- +- Issue #14697: Fix missing support for set displays and set comprehensions in + parser module. + - Issue #14701: Fix missing support for 'raise ... from' in parser module. - Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -2865,34 +2865,92 @@ validate_expr_or_star_expr, "exprlist")); } - +/* + * dictorsetmaker: + * + * (test ':' test (comp_for | (',' test ':' test)* [','])) | + * (test (comp_for | (',' test)* [','])) + */ static int validate_dictorsetmaker(node *tree) { int nch = NCH(tree); - int res = (validate_ntype(tree, dictorsetmaker) - && (nch >= 3) - && validate_test(CHILD(tree, 0)) - && validate_colon(CHILD(tree, 1)) - && validate_test(CHILD(tree, 2))); - - if (res && ((nch % 4) == 0)) - res = validate_comma(CHILD(tree, --nch)); - else if (res) - res = ((nch % 4) == 3); - - if (res && (nch > 3)) { - int pos = 3; - /* ( ',' test ':' test )* */ - while (res && (pos < nch)) { - res = (validate_comma(CHILD(tree, pos)) - && validate_test(CHILD(tree, pos + 1)) - && validate_colon(CHILD(tree, pos + 2)) - && validate_test(CHILD(tree, pos + 3))); - pos += 4; + int res; + int i = 0; + + res = validate_ntype(tree, dictorsetmaker); + if (!res) + return 0; + + if (nch - i < 1) { + (void) validate_numnodes(tree, 1, "dictorsetmaker"); + return 0; + } + + res = validate_test(CHILD(tree, i++)); + if (!res) + return 0; + + if (nch - i >= 2 && TYPE(CHILD(tree, i)) == COLON) { + /* Dictionary display or dictionary comprehension. */ + res = (validate_colon(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++))); + if (!res) + return 0; + + if (nch - i >= 1 && TYPE(CHILD(tree, i)) == comp_for) { + /* Dictionary comprehension. */ + res = validate_comp_for(CHILD(tree, i++)); + if (!res) + return 0; + } + else { + /* Dictionary display. */ + while (nch - i >= 4) { + res = (validate_comma(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++)) + && validate_colon(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++))); + if (!res) + return 0; + } + if (nch - i == 1) { + res = validate_comma(CHILD(tree, i++)); + if (!res) + return 0; + } } } - return (res); + else { + /* Set display or set comprehension. */ + if (nch - i >= 1 && TYPE(CHILD(tree, i)) == comp_for) { + /* Set comprehension. */ + res = validate_comp_for(CHILD(tree, i++)); + if (!res) + return 0; + } + else { + /* Set display. */ + while (nch - i >= 2) { + res = (validate_comma(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++))); + if (!res) + return 0; + } + if (nch - i == 1) { + res = validate_comma(CHILD(tree, i++)); + if (!res) + return 0; + } + } + } + + if (nch - i > 0) { + err_string("Illegal trailing nodes for dictorsetmaker."); + return 0; + } + + return 1; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 17:36:47 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 17:36:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314697=3A__Merge_fix_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/4815a4a4a852 changeset: 76823:4815a4a4a852 parent: 76821:ab500b297900 parent: 76822:129289144dfb user: Mark Dickinson date: Mon May 07 16:36:33 2012 +0100 summary: Issue #14697: Merge fix from 3.2. files: Lib/test/test_parser.py | 23 ++++++ Misc/NEWS | 3 + Modules/parsermodule.c | 102 +++++++++++++++++++++------ 3 files changed, 106 insertions(+), 22 deletions(-) diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -309,6 +309,29 @@ "except Exception as e:\n" " raise ValueError from e\n") + def test_set_displays(self): + self.check_expr('{2}') + self.check_expr('{2,}') + self.check_expr('{2, 3}') + self.check_expr('{2, 3,}') + + def test_dict_displays(self): + self.check_expr('{}') + self.check_expr('{a:b}') + self.check_expr('{a:b,}') + self.check_expr('{a:b, c:d}') + self.check_expr('{a:b, c:d,}') + + def test_set_comprehensions(self): + self.check_expr('{x for x in seq}') + self.check_expr('{f(x) for x in seq}') + self.check_expr('{f(x) for x in seq if condition(x)}') + + def test_dict_comprehensions(self): + self.check_expr('{x:x for x in seq}') + self.check_expr('{x**2:x[3] for x in seq if condition(x)}') + self.check_expr('{x:x for x in seq1 for y in seq2 if condition(x, y)}') + # # Second, we take *invalid* trees and make sure we get ParserError diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,9 @@ Library ------- +- Issue #14697: Fix missing support for set displays and set comprehensions in + parser module. + - Issue #14701: Fix missing support for 'raise ... from' in parser module. - Add support for timeouts to the acquire() methods of diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -2895,34 +2895,92 @@ validate_expr_or_star_expr, "exprlist")); } - +/* + * dictorsetmaker: + * + * (test ':' test (comp_for | (',' test ':' test)* [','])) | + * (test (comp_for | (',' test)* [','])) + */ static int validate_dictorsetmaker(node *tree) { int nch = NCH(tree); - int res = (validate_ntype(tree, dictorsetmaker) - && (nch >= 3) - && validate_test(CHILD(tree, 0)) - && validate_colon(CHILD(tree, 1)) - && validate_test(CHILD(tree, 2))); - - if (res && ((nch % 4) == 0)) - res = validate_comma(CHILD(tree, --nch)); - else if (res) - res = ((nch % 4) == 3); - - if (res && (nch > 3)) { - int pos = 3; - /* ( ',' test ':' test )* */ - while (res && (pos < nch)) { - res = (validate_comma(CHILD(tree, pos)) - && validate_test(CHILD(tree, pos + 1)) - && validate_colon(CHILD(tree, pos + 2)) - && validate_test(CHILD(tree, pos + 3))); - pos += 4; + int res; + int i = 0; + + res = validate_ntype(tree, dictorsetmaker); + if (!res) + return 0; + + if (nch - i < 1) { + (void) validate_numnodes(tree, 1, "dictorsetmaker"); + return 0; + } + + res = validate_test(CHILD(tree, i++)); + if (!res) + return 0; + + if (nch - i >= 2 && TYPE(CHILD(tree, i)) == COLON) { + /* Dictionary display or dictionary comprehension. */ + res = (validate_colon(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++))); + if (!res) + return 0; + + if (nch - i >= 1 && TYPE(CHILD(tree, i)) == comp_for) { + /* Dictionary comprehension. */ + res = validate_comp_for(CHILD(tree, i++)); + if (!res) + return 0; + } + else { + /* Dictionary display. */ + while (nch - i >= 4) { + res = (validate_comma(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++)) + && validate_colon(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++))); + if (!res) + return 0; + } + if (nch - i == 1) { + res = validate_comma(CHILD(tree, i++)); + if (!res) + return 0; + } } } - return (res); + else { + /* Set display or set comprehension. */ + if (nch - i >= 1 && TYPE(CHILD(tree, i)) == comp_for) { + /* Set comprehension. */ + res = validate_comp_for(CHILD(tree, i++)); + if (!res) + return 0; + } + else { + /* Set display. */ + while (nch - i >= 2) { + res = (validate_comma(CHILD(tree, i++)) + && validate_test(CHILD(tree, i++))); + if (!res) + return 0; + } + if (nch - i == 1) { + res = validate_comma(CHILD(tree, i++)); + if (!res) + return 0; + } + } + } + + if (nch - i > 0) { + err_string("Illegal trailing nodes for dictorsetmaker."); + return 0; + } + + return 1; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 18:25:23 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 18:25:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0NzQx?= =?utf8?q?=3A_Fix_missing_support_for_ellipsis_in_parser_module=2E?= Message-ID: http://hg.python.org/cpython/rev/2b1cc84bf1d9 changeset: 76824:2b1cc84bf1d9 branch: 3.2 parent: 76822:129289144dfb user: Mark Dickinson date: Mon May 07 17:24:04 2012 +0100 summary: Issue #14741: Fix missing support for ellipsis in parser module. files: Lib/test/test_parser.py | 2 ++ Misc/NEWS | 2 ++ Modules/parsermodule.c | 6 +----- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -106,6 +106,8 @@ self.check_expr("lambda x, *y, **z: 0") self.check_expr("(x for x in range(10))") self.check_expr("foo(x for x in range(10))") + self.check_expr("...") + self.check_expr("a[...]") def test_simple_expression(self): # expr_stmt diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -61,6 +61,8 @@ Library ------- +- Issue #14741: Fix missing support for Ellipsis ('...') in parser module. + - Issue #14697: Fix missing support for set displays and set comprehensions in parser module. diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -2389,17 +2389,13 @@ break; case NAME: case NUMBER: + case ELLIPSIS: res = (nch == 1); break; case STRING: for (pos = 1; res && (pos < nch); ++pos) res = validate_ntype(CHILD(tree, pos), STRING); break; - case DOT: - res = (nch == 3 && - validate_ntype(CHILD(tree, 1), DOT) && - validate_ntype(CHILD(tree, 2), DOT)); - break; default: res = 0; break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 18:25:24 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 18:25:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314741=3A_Merge_fix_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/d50577c5711b changeset: 76825:d50577c5711b parent: 76823:4815a4a4a852 parent: 76824:2b1cc84bf1d9 user: Mark Dickinson date: Mon May 07 17:25:14 2012 +0100 summary: Issue #14741: Merge fix from 3.2. files: Lib/test/test_parser.py | 2 ++ Misc/NEWS | 2 ++ Modules/parsermodule.c | 6 +----- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_parser.py b/Lib/test/test_parser.py --- a/Lib/test/test_parser.py +++ b/Lib/test/test_parser.py @@ -110,6 +110,8 @@ self.check_expr("lambda x, *y, **z: 0") self.check_expr("(x for x in range(10))") self.check_expr("foo(x for x in range(10))") + self.check_expr("...") + self.check_expr("a[...]") def test_simple_expression(self): # expr_stmt diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,8 @@ Library ------- +- Issue #14741: Fix missing support for Ellipsis ('...') in parser module. + - Issue #14697: Fix missing support for set displays and set comprehensions in parser module. diff --git a/Modules/parsermodule.c b/Modules/parsermodule.c --- a/Modules/parsermodule.c +++ b/Modules/parsermodule.c @@ -2419,17 +2419,13 @@ break; case NAME: case NUMBER: + case ELLIPSIS: res = (nch == 1); break; case STRING: for (pos = 1; res && (pos < nch); ++pos) res = validate_ntype(CHILD(tree, pos), STRING); break; - case DOT: - res = (nch == 3 && - validate_ntype(CHILD(tree, 1), DOT) && - validate_ntype(CHILD(tree, 2), DOT)); - break; default: res = 0; break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 21:44:03 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 07 May 2012 21:44:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314583=3A_Fix_impor?= =?utf8?q?tlib_bug_when_a_package=27s_=5F=5Finit=5F=5F=2Epy_would_first_im?= =?utf8?q?port?= Message-ID: http://hg.python.org/cpython/rev/d6324941b739 changeset: 76826:d6324941b739 user: Antoine Pitrou date: Mon May 07 21:41:59 2012 +0200 summary: Issue #14583: Fix importlib bug when a package's __init__.py would first import one of its modules then raise an error. files: Lib/importlib/_bootstrap.py | 2 +- Lib/importlib/test/import_/test_packages.py | 56 ++++++++++ Lib/importlib/test/util.py | 6 +- Misc/NEWS | 3 + Python/import.c | 19 +- Python/importlib.h | Bin 6 files changed, 76 insertions(+), 10 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -1082,7 +1082,7 @@ # Return up to the first dot in 'name'. This is complicated by the fact # that 'name' may be relative. if level == 0: - return sys.modules[name.partition('.')[0]] + return _gcd_import(name.partition('.')[0]) elif not name: return module else: diff --git a/Lib/importlib/test/import_/test_packages.py b/Lib/importlib/test/import_/test_packages.py --- a/Lib/importlib/test/import_/test_packages.py +++ b/Lib/importlib/test/import_/test_packages.py @@ -23,6 +23,62 @@ import_util.import_('pkg.module') self.assertEqual(cm.exception.name, 'pkg') + def test_raising_parent_after_importing_child(self): + def __init__(): + import pkg.module + 1/0 + mock = util.mock_modules('pkg.__init__', 'pkg.module', + module_code={'pkg': __init__}) + with mock: + with util.import_state(meta_path=[mock]): + with self.assertRaises(ZeroDivisionError): + import_util.import_('pkg') + self.assertFalse('pkg' in sys.modules) + self.assertTrue('pkg.module' in sys.modules) + with self.assertRaises(ZeroDivisionError): + import_util.import_('pkg.module') + self.assertFalse('pkg' in sys.modules) + self.assertTrue('pkg.module' in sys.modules) + + def test_raising_parent_after_relative_importing_child(self): + def __init__(): + from . import module + 1/0 + mock = util.mock_modules('pkg.__init__', 'pkg.module', + module_code={'pkg': __init__}) + with mock: + with util.import_state(meta_path=[mock]): + with self.assertRaises((ZeroDivisionError, ImportError)): + # This raises ImportError on the "from . import module" + # line, not sure why. + import_util.import_('pkg') + self.assertFalse('pkg' in sys.modules) + with self.assertRaises((ZeroDivisionError, ImportError)): + import_util.import_('pkg.module') + self.assertFalse('pkg' in sys.modules) + # XXX False + #self.assertTrue('pkg.module' in sys.modules) + + def test_raising_parent_after_double_relative_importing_child(self): + def __init__(): + from ..subpkg import module + 1/0 + mock = util.mock_modules('pkg.__init__', 'pkg.subpkg.__init__', + 'pkg.subpkg.module', + module_code={'pkg.subpkg': __init__}) + with mock: + with util.import_state(meta_path=[mock]): + with self.assertRaises((ZeroDivisionError, ImportError)): + # This raises ImportError on the "from ..subpkg import module" + # line, not sure why. + import_util.import_('pkg.subpkg') + self.assertFalse('pkg.subpkg' in sys.modules) + with self.assertRaises((ZeroDivisionError, ImportError)): + import_util.import_('pkg.subpkg.module') + self.assertFalse('pkg.subpkg' in sys.modules) + # XXX False + #self.assertTrue('pkg.subpkg.module' in sys.modules) + def test_module_not_package(self): # Try to import a submodule from a non-package should raise ImportError. assert not hasattr(sys, '__path__') diff --git a/Lib/importlib/test/util.py b/Lib/importlib/test/util.py --- a/Lib/importlib/test/util.py +++ b/Lib/importlib/test/util.py @@ -124,7 +124,11 @@ else: sys.modules[fullname] = self.modules[fullname] if fullname in self.module_code: - self.module_code[fullname]() + try: + self.module_code[fullname]() + except Exception: + del sys.modules[fullname] + raise return self.modules[fullname] def __enter__(self): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -20,6 +20,9 @@ Library ------- +- Issue #14583: Fix importlib bug when a package's __init__.py would first + import one of its modules then raise an error. + - Issue #14741: Fix missing support for Ellipsis ('...') in parser module. - Issue #14697: Fix missing support for set displays and set comprehensions in diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1633,19 +1633,20 @@ goto error_with_unlock; } + if (PyUnicode_GET_LENGTH(PyTuple_GET_ITEM(partition, 1)) == 0) { + /* No dot in module name, simple exit */ + Py_DECREF(partition); + final_mod = mod; + Py_INCREF(mod); + goto exit_with_unlock; + } + front = PyTuple_GET_ITEM(partition, 0); Py_INCREF(front); Py_DECREF(partition); if (level == 0) { - final_mod = PyDict_GetItem(interp->modules, front); - if (final_mod == NULL) { - PyErr_Format(PyExc_KeyError, - "%R not in sys.modules as expected", front); - } - else { - Py_INCREF(final_mod); - } + final_mod = PyObject_CallFunctionObjArgs(builtins_import, front, NULL); Py_DECREF(front); } else { @@ -1682,6 +1683,8 @@ fromlist, builtins_import, NULL); } + + exit_with_unlock: error_with_unlock: #ifdef WITH_THREAD if (_PyImport_ReleaseLock() < 0) { diff --git a/Python/importlib.h b/Python/importlib.h index 292308fc78123e9fa10c7131bb110c47559975dd..c88fc8018149bd01cc54b8759f937df81e7016cc GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 21:58:46 2012 From: python-checkins at python.org (sandro.tosi) Date: Mon, 07 May 2012 21:58:46 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_backport_7a05cb?= =?utf8?q?3beddf_to_2=2E7?= Message-ID: http://hg.python.org/cpython/rev/07b3fc67bf45 changeset: 76827:07b3fc67bf45 branch: 2.7 parent: 76807:9bb9bd9237c1 user: Sandro Tosi date: Mon May 07 21:56:24 2012 +0200 summary: backport 7a05cb3beddf to 2.7 files: Doc/library/functions.rst | 25 +++++++++++++++++++------ 1 files changed, 19 insertions(+), 6 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -18,16 +18,25 @@ :func:`bool` :func:`filter` :func:`len` :func:`range` :func:`type` :func:`bytearray` :func:`float` :func:`list` :func:`raw_input` :func:`unichr` :func:`callable` :func:`format` :func:`locals` :func:`reduce` :func:`unicode` -:func:`chr` :func:`frozenset` :func:`long` :func:`reload` :func:`vars` -:func:`classmethod` :func:`getattr` :func:`map` :func:`.repr` :func:`xrange` +:func:`chr` |func-frozenset|_ :func:`long` :func:`reload` :func:`vars` +:func:`classmethod` :func:`getattr` :func:`map` :func:`repr` :func:`xrange` :func:`cmp` :func:`globals` :func:`max` :func:`reversed` :func:`zip` -:func:`compile` :func:`hasattr` :func:`memoryview` :func:`round` :func:`__import__` -:func:`complex` :func:`hash` :func:`min` :func:`set` :func:`apply` +:func:`compile` :func:`hasattr` |func-memoryview|_ :func:`round` :func:`__import__` +:func:`complex` :func:`hash` :func:`min` |func-set|_ :func:`apply` :func:`delattr` :func:`help` :func:`next` :func:`setattr` :func:`buffer` -:func:`dict` :func:`hex` :func:`object` :func:`slice` :func:`coerce` +|func-dict|_ :func:`hex` :func:`object` :func:`slice` :func:`coerce` :func:`dir` :func:`id` :func:`oct` :func:`sorted` :func:`intern` =================== ================= ================== ================= ==================== +.. using :func:`dict` would create a link to another page, so local targets are + used, with replacement texts to make the output in the table consistent + +.. |func-dict| replace:: ``dict()`` +.. |func-frozenset| replace:: ``frozenset()`` +.. |func-memoryview| replace:: ``memoryview()`` +.. |func-set| replace:: ``set()`` + + .. function:: abs(x) Return the absolute value of a number. The argument may be a plain or long @@ -265,6 +274,7 @@ example, ``delattr(x, 'foobar')`` is equivalent to ``del x.foobar``. +.. _func-dict: .. function:: dict([arg]) :noindex: @@ -513,6 +523,7 @@ .. versionadded:: 2.6 +.. _func-frozenset: .. function:: frozenset([iterable]) :noindex: @@ -752,7 +763,7 @@ .. versionchanged:: 2.5 Added support for the optional *key* argument. - +.. _func-memoryview: .. function:: memoryview(obj) :noindex: @@ -1181,6 +1192,8 @@ can't be represented exactly as a float. See :ref:`tut-fp-issues` for more information. + +.. _func-set: .. function:: set([iterable]) :noindex: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 23:37:02 2012 From: python-checkins at python.org (mark.dickinson) Date: Mon, 07 May 2012 23:37:02 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314742=3A__Don=27t_?= =?utf8?q?include_DirectoryTestCase_from_test=5Funparse_in_test=5Ftools?= Message-ID: http://hg.python.org/cpython/rev/dbfacec7e368 changeset: 76828:dbfacec7e368 parent: 76826:d6324941b739 user: Mark Dickinson date: Mon May 07 22:36:43 2012 +0100 summary: Issue #14742: Don't include DirectoryTestCase from test_unparse in test_tools until we can speed it up. files: Lib/test/test_tools.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -124,7 +124,7 @@ # Run the tests in Tools/parser/test_unparse.py with support.DirsOnSysPath(os.path.join(basepath, 'parser')): - from test_unparse import UnparseTestCase, DirectoryTestCase + from test_unparse import UnparseTestCase def test_main(): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 7 23:50:12 2012 From: python-checkins at python.org (victor.stinner) Date: Mon, 07 May 2012 23:50:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Backout_ab500b297900=3A_the?= =?utf8?q?_check_for_integer_overflow_is_wrong?= Message-ID: http://hg.python.org/cpython/rev/01581e8b50f2 changeset: 76829:01581e8b50f2 user: Victor Stinner date: Mon May 07 23:50:05 2012 +0200 summary: Backout ab500b297900: the check for integer overflow is wrong Issue #14716: Change integer overflow check in unicode_writer_prepare() to compute the limit at compile time instead of runtime. Patch writen by Serhiy Storchaka. files: Objects/unicodeobject.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13242,10 +13242,8 @@ newlen = writer->pos + length; if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { - /* Overallocate 25% to limit the number of resize. - Check for integer overflow: - (newlen + newlen / 4) <= PY_SSIZE_T_MAX */ - if (newlen <= (PY_SSIZE_T_MAX - PY_SSIZE_T_MAX / 5)) + /* overallocate 25% to limit the number of resize */ + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) newlen += newlen / 4; if (maxchar > writer->maxchar) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 04:19:49 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 04:19:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_explicitly_set_UnsupportedO?= =?utf8?q?peration=27s_module_rather_than_relying_on_incorrect?= Message-ID: http://hg.python.org/cpython/rev/2cd9dadd5c6c changeset: 76830:2cd9dadd5c6c user: Benjamin Peterson date: Mon May 07 22:19:42 2012 -0400 summary: explicitly set UnsupportedOperation's module rather than relying on incorrect globals on startup (closes #14745) files: Lib/io.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/io.py b/Lib/io.py --- a/Lib/io.py +++ b/Lib/io.py @@ -67,6 +67,9 @@ OpenWrapper = _io.open # for compatibility with _pyio +# Pretend this exception was created here. +UnsupportedOperation.__module__ = "io" + # for seek() SEEK_SET = 0 SEEK_CUR = 1 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 04:24:12 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 04:24:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_remove_basicall?= =?utf8?q?y_bitrotted_XXXs?= Message-ID: http://hg.python.org/cpython/rev/197f47238753 changeset: 76831:197f47238753 branch: 3.2 parent: 76824:2b1cc84bf1d9 user: Benjamin Peterson date: Mon May 07 22:23:48 2012 -0400 summary: remove basically bitrotted XXXs files: Lib/io.py | 9 --------- 1 files changed, 0 insertions(+), 9 deletions(-) diff --git a/Lib/io.py b/Lib/io.py --- a/Lib/io.py +++ b/Lib/io.py @@ -34,15 +34,6 @@ """ # New I/O library conforming to PEP 3116. -# XXX edge cases when switching between reading/writing -# XXX need to support 1 meaning line-buffered -# XXX whenever an argument is None, use the default value -# XXX read/write ops should check readable/writable -# XXX buffered readinto should work with arbitrary buffer objects -# XXX use incremental encoder for text output, at least for UTF-16 and UTF-8-SIG -# XXX check writable, readable and seekable in appropriate places - - __author__ = ("Guido van Rossum , " "Mike Verdone , " "Mark Russell , " -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 04:24:13 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 04:24:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_remove_basicall?= =?utf8?q?y_bitrotted_XXXs?= Message-ID: http://hg.python.org/cpython/rev/72df255a7716 changeset: 76832:72df255a7716 branch: 2.7 parent: 76827:07b3fc67bf45 user: Benjamin Peterson date: Mon May 07 22:23:48 2012 -0400 summary: remove basically bitrotted XXXs files: Lib/io.py | 9 --------- 1 files changed, 0 insertions(+), 9 deletions(-) diff --git a/Lib/io.py b/Lib/io.py --- a/Lib/io.py +++ b/Lib/io.py @@ -34,15 +34,6 @@ """ # New I/O library conforming to PEP 3116. -# XXX edge cases when switching between reading/writing -# XXX need to support 1 meaning line-buffered -# XXX whenever an argument is None, use the default value -# XXX read/write ops should check readable/writable -# XXX buffered readinto should work with arbitrary buffer objects -# XXX use incremental encoder for text output, at least for UTF-16 and UTF-8-SIG -# XXX check writable, readable and seekable in appropriate places - - __author__ = ("Guido van Rossum , " "Mike Verdone , " "Mark Russell , " -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 04:24:13 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 04:24:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/d533663caee8 changeset: 76833:d533663caee8 parent: 76830:2cd9dadd5c6c parent: 76831:197f47238753 user: Benjamin Peterson date: Mon May 07 22:24:05 2012 -0400 summary: merge 3.2 files: Lib/io.py | 9 --------- 1 files changed, 0 insertions(+), 9 deletions(-) diff --git a/Lib/io.py b/Lib/io.py --- a/Lib/io.py +++ b/Lib/io.py @@ -34,15 +34,6 @@ """ # New I/O library conforming to PEP 3116. -# XXX edge cases when switching between reading/writing -# XXX need to support 1 meaning line-buffered -# XXX whenever an argument is None, use the default value -# XXX read/write ops should check readable/writable -# XXX buffered readinto should work with arbitrary buffer objects -# XXX use incremental encoder for text output, at least for UTF-16 and UTF-8-SIG -# XXX check writable, readable and seekable in appropriate places - - __author__ = ("Guido van Rossum , " "Mike Verdone , " "Mark Russell , " -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue May 8 05:42:38 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 08 May 2012 05:42:38 +0200 Subject: [Python-checkins] Daily reference leaks (01581e8b50f2): sum=2 Message-ID: results for 01581e8b50f2 on branch "default" -------------------------------------------- test_dbm leaked [0, 2, 0] references, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog3sunH5', '-x'] From python-checkins at python.org Tue May 8 12:51:41 2012 From: python-checkins at python.org (larry.hastings) Date: Tue, 08 May 2012 12:51:41 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0NzQ5?= =?utf8?q?=3A_Add_support_for_=27Z=27_to_skipitem=28=29_in_Python/getargs?= =?utf8?b?LmMu?= Message-ID: http://hg.python.org/cpython/rev/91612618985b changeset: 76834:91612618985b branch: 3.2 parent: 76831:197f47238753 user: Larry Hastings date: Tue May 08 03:51:18 2012 -0700 summary: Issue #14749: Add support for 'Z' to skipitem() in Python/getargs.c. files: Python/getargs.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1633,6 +1633,7 @@ case 'z': /* string or None */ case 'y': /* bytes */ case 'u': /* unicode string */ + case 'Z': /* unicode string or None */ case 'w': /* buffer, read-write */ { (void) va_arg(*p_va, char **); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 12:54:20 2012 From: python-checkins at python.org (larry.hastings) Date: Tue, 08 May 2012 12:54:20 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_from_3=2E2=2E__Issue_=2314749=3A_Add_support_for_=27Z?= =?utf8?q?=27_to_skipitem=28=29=2E?= Message-ID: http://hg.python.org/cpython/rev/b32baa5b7626 changeset: 76835:b32baa5b7626 parent: 76833:d533663caee8 parent: 76834:91612618985b user: Larry Hastings date: Tue May 08 03:54:05 2012 -0700 summary: Merge from 3.2. Issue #14749: Add support for 'Z' to skipitem(). files: Python/getargs.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1657,6 +1657,7 @@ case 'z': /* string or None */ case 'y': /* bytes */ case 'u': /* unicode string */ + case 'Z': /* unicode string or None */ case 'w': /* buffer, read-write */ { (void) va_arg(*p_va, char **); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 15:23:14 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 15:23:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fix_possible_re?= =?utf8?q?fleak_=28closes_=2314752=29?= Message-ID: http://hg.python.org/cpython/rev/d937b527b76e changeset: 76836:d937b527b76e branch: 3.2 parent: 76834:91612618985b user: Benjamin Peterson date: Tue May 08 09:22:24 2012 -0400 summary: fix possible refleak (closes #14752) files: Objects/typeobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3527,6 +3527,7 @@ for (; meth->ml_name != NULL; meth++) { PyObject *descr; + int err; if (PyDict_GetItemString(dict, meth->ml_name) && !(meth->ml_flags & METH_COEXIST)) continue; @@ -3550,9 +3551,10 @@ } if (descr == NULL) return -1; - if (PyDict_SetItemString(dict, meth->ml_name, descr) < 0) + err = PyDict_SetItemString(dict, meth->ml_name, descr); + Py_DECREF(descr); + if (err < 0) return -1; - Py_DECREF(descr); } return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 15:23:15 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 15:23:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_fix_possible_re?= =?utf8?q?fleak_=28closes_=2314752=29?= Message-ID: http://hg.python.org/cpython/rev/5319a4bf72e7 changeset: 76837:5319a4bf72e7 branch: 2.7 parent: 76832:72df255a7716 user: Benjamin Peterson date: Tue May 08 09:22:24 2012 -0400 summary: fix possible refleak (closes #14752) files: Objects/typeobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3565,6 +3565,7 @@ for (; meth->ml_name != NULL; meth++) { PyObject *descr; + int err; if (PyDict_GetItemString(dict, meth->ml_name) && !(meth->ml_flags & METH_COEXIST)) continue; @@ -3588,9 +3589,10 @@ } if (descr == NULL) return -1; - if (PyDict_SetItemString(dict, meth->ml_name, descr) < 0) + err = PyDict_SetItemString(dict, meth->ml_name, descr); + Py_DECREF(descr); + if (err < 0) return -1; - Py_DECREF(descr); } return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 15:23:16 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 08 May 2012 15:23:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?b?OiBtZXJnZSAzLjIgKCMxNDc1Mik=?= Message-ID: http://hg.python.org/cpython/rev/07b04373aef8 changeset: 76838:07b04373aef8 parent: 76835:b32baa5b7626 parent: 76836:d937b527b76e user: Benjamin Peterson date: Tue May 08 09:22:45 2012 -0400 summary: merge 3.2 (#14752) files: Objects/typeobject.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3709,6 +3709,7 @@ for (; meth->ml_name != NULL; meth++) { PyObject *descr; + int err; if (PyDict_GetItemString(dict, meth->ml_name) && !(meth->ml_flags & METH_COEXIST)) continue; @@ -3732,9 +3733,10 @@ } if (descr == NULL) return -1; - if (PyDict_SetItemString(dict, meth->ml_name, descr) < 0) + err = PyDict_SetItemString(dict, meth->ml_name, descr); + Py_DECREF(descr); + if (err < 0) return -1; - Py_DECREF(descr); } return 0; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 17:10:34 2012 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 08 May 2012 17:10:34 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_fix_for_test=5Fmultip?= =?utf8?q?rocessing?= Message-ID: http://hg.python.org/cpython/rev/bb30116024ac changeset: 76839:bb30116024ac user: Richard Oudkerk date: Tue May 08 16:08:07 2012 +0100 summary: Minor fix for test_multiprocessing files: Lib/test/test_multiprocessing.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2713,7 +2713,7 @@ self.test_wait(True) def test_wait_socket_slow(self): - self.test_wait(True) + self.test_wait_socket(True) def test_wait_timeout(self): from multiprocessing.connection import wait -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 8 18:37:21 2012 From: python-checkins at python.org (barry.warsaw) Date: Tue, 08 May 2012 18:37:21 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Eric_Snow=27s_latest_update=2E?= Message-ID: http://hg.python.org/peps/rev/38c03e160957 changeset: 4362:38c03e160957 user: Barry Warsaw date: Tue May 08 09:37:14 2012 -0700 summary: Eric Snow's latest update. files: pep-0421.txt | 113 ++++++++++++++++++++++++++++---------- 1 files changed, 84 insertions(+), 29 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -98,7 +98,9 @@ Any other values that an implementation wishes to specify, particularly informational ones. Neither the standard library nor the language specification will rely on implementation metadata. - Also see the list of `Example Metadata Values`_. + Also see the list of `Example Metadata Values`_. ``metadata`` is a + dictionary. While a mutable type, neither it nor its items will + change (as already noted). Likewise they should not be modified. Adding New Required Attributes @@ -141,8 +143,8 @@ project. **vcs_revision_id** - A value that identifies the VCS revision of the implementation that - is currently running. + A value that identifies the VCS revision of the implementation + that is currently running. **build_toolchain** Identifies the tools used to build the interpreter. @@ -182,18 +184,18 @@ Why a Custom Type? ------------------ -A dedicated class, of which ``sys.implementation`` is an instance, would -facilitate the dotted access of a "named" tuple. At the same time, it -allows us to avoid the problems of the other approaches (see below), -like confusion about ordering and iteration. +A dedicated class, of which ``sys.implementation`` is an instance, +would facilitate the dotted access of a "named" tuple. At the same +time, it allows us to avoid the problems of the other approaches (see +below), like confusion about ordering and iteration. The alternatives to a dictionary are considered separately here: **Dictionary** A dictionary reflects a simple namespace with item access. It -maps names to values and that's all. It also reflects the more variable -nature of ``sys.implementation``. +maps names to values and that's all. It also reflects the more +variable nature of ``sys.implementation``. However, a simple dictionary does not set expectations very well about the nature of ``sys.implementation``. The custom type approach, with @@ -217,9 +219,9 @@ However, this is mitigated by having ``sys.implementation.metadata``. -One problem with using a named tuple is that ``sys.implementation`` does -not have meaning as a sequence. Also, unlike other similar ``sys`` -variables, it has a far greater potential to change over time. +One problem with using a named tuple is that ``sys.implementation`` +does not have meaning as a sequence. Also, unlike other similar +``sys`` variables, it has a far greater potential to change over time. If a named tuple were used, we'd be very clear in the documentation that the length and order of the value are not reliable. Iterability @@ -290,10 +292,10 @@ "explicit is better than implicit" -The ``platform`` module determines the python implementation by -looking for clues in a couple different ``sys`` variables [#guess]_. -However, this approach is fragile, requiring changes to the standard -library each time an implementation changes. Beyond that, support in +The ``platform`` module determines the python implementation by looking +for clues in a couple different ``sys`` variables [#guess]_. However, +this approach is fragile, requiring changes to the standard library +each time an implementation changes. Beyond that, support in ``platform`` is limited to those implementations that core developers have blessed by special-casing them in the ``platform`` module. @@ -359,26 +361,43 @@ treatment of the java environment in the standard library [#os_name]_ [#javatest]_. Unfortunately it masks the os name that would otherwise go there. ``sys.implementation`` would help obviate the need for this -special case. +special case. Currently Jython sets os._name for the normal os.name +value. -Feedback From Other Python Implementers +Feedback From Other Python Implementors ======================================= IronPython ---------- -XXX +Jeff Hardy responded to a request for feedback [#ironpython]_. He +said, "I'll probably add it the day after it's approved" +[#jeff_hardy_2012]_. He also gave useful feedback on both the type of +``sys.implementation`` and on the ``metadata`` attribute. Jython ------ -XXX +In 2009 Frank Wierzbicki said this (relative to Jython implementing the +required attributes) [#frank_wierzbicki_2009]_:: + + Speaking for Jython, so far it looks like something we would adopt + soonish after it was accepted (it looks pretty useful to me). PyPy ---- -XXX +Some of the PyPy developers have responded to a request for feedback +[#pypy]_. Armin Rigo said the following [#armin_rigo_2012]_:: + + For myself, I can only say that it looks like a good idea, which we + will happily adhere to when we migrate to Python 3.3. + +He also expressed support for keeping the required list small. Both +Armin and Laura indicated that an effort to better catalog Python's +implementation would be welcome. Such an effort, for which this PEP is +a small start, will be considered separately. Past Efforts @@ -402,6 +421,27 @@ is proposed in that same spirit. +The Bigger Picture +================== + +It's worth noting again that this PEP is a small part of a larger +on-going effort to identify the implementation-specific parts of Python +and mitigate their impact on other implementations than CPython. + +``sys.implementation`` is a focal point for implementation-specific +data, acting as a nexus for cooperation between the language, the +standard library, and the different implementations. As time goes by +it is feasible that ``sys.implementation`` will assume current +attributes of ``sys`` and other builtin/stdlib modules, where +appropriate. In this way, it is a PEP 3137-lite, but starting as +small as possible. + +However, as already noted, many other efforts predate +``sys.implementation``. Neither is it necessarily a major part of the +effort. Rather, consider it as part of the infrastructure of the +effort to make Python friendler to alternate implementations. + + Alternatives ============ @@ -412,11 +452,6 @@ Open Issues =========== -* What are the long-term objectives for ``sys.implementation``? - - - possibly pull in implementation details from the main ``sys`` - namespace and elsewhere (PEP 3137 lite). - * What is the process for introducing new required variables? PEP? * Is the ``sys.version_info`` format the right one here? @@ -427,8 +462,6 @@ reflect the version of the language spec? Micro version, series, and release seem like implementation-specific values. -* Alternatives to the approach dictated by this PEP? - * Do we really want to commit to using a dict for ``sys.implementation``? @@ -468,11 +501,33 @@ http://mail.python.org/pipermail/python-dev/2009-October/092893.html .. [#revived] The initial 2012 discussion: - http://mail.python.org/pipermail/python-ideas/2012-April/014878.html + http://mail.python.org/pipermail/python-ideas/2012-March/014555.html + (and http://mail.python.org/pipermail/python-ideas/2012-April/014878.html) .. [#feedback] Feedback on the PEP: http://mail.python.org/pipermail/python-ideas/2012-April/014954.html +.. [#ironpython] Feedback from the IronPython developers: + http://mail.python.org/pipermail/ironpython-users/2012-May/015980.html + +.. [#dino_viehland_2009] (2009) Dino Viehland offers his opinion: + http://mail.python.org/pipermail/python-dev/2009-October/092894.html + +.. [#jeff_hardy_2012] (2012) Jeff Hardy offers his opinion: + http://mail.python.org/pipermail/ironpython-users/2012-May/015981.html + +.. [#jython] Feedback from the Jython developers: + ??? + +.. [#frank_wierzbicki_2009] (2009) Frank Wierzbicki offers his opinion: + http://mail.python.org/pipermail/python-dev/2009-October/092974.html + +.. [#pypy] Feedback from the PyPy developers: + http://mail.python.org/pipermail/pypy-dev/2012-May/009883.html + +.. [#armin_rigo_2012] (2012) Armin Rigo offers his opinion: + http://mail.python.org/pipermail/pypy-dev/2012-May/009884.html + .. [#guess] The ``platform`` code which divines the implementation name: http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 8 23:28:16 2012 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 08 May 2012 23:28:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314727=3A_Fix_race_?= =?utf8?q?in_test=5Fmultiprocessing?= Message-ID: http://hg.python.org/cpython/rev/3d900c9641c9 changeset: 76840:3d900c9641c9 user: Richard Oudkerk date: Tue May 08 22:24:47 2012 +0100 summary: Issue #14727: Fix race in test_multiprocessing files: Lib/test/test_multiprocessing.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2003,8 +2003,8 @@ l = socket.socket() l.bind(('localhost', 0)) + l.listen(1) conn.send(l.getsockname()) - l.listen(1) new_conn, addr = l.accept() conn.send(new_conn) new_conn.close() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 03:25:11 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 09 May 2012 03:25:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_unicode=5Fwriter=5Ffinish?= =?utf8?q?=28=29_checks_string_consistency?= Message-ID: http://hg.python.org/cpython/rev/6d39b5ea4542 changeset: 76841:6d39b5ea4542 user: Victor Stinner date: Wed May 09 03:24:14 2012 +0200 summary: unicode_writer_finish() checks string consistency files: Lib/test/test_unicode.py | 4 ++++ Objects/unicodeobject.c | 1 + 2 files changed, 5 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1064,6 +1064,10 @@ self.assertEqual('%f' % INF, 'inf') self.assertEqual('%F' % INF, 'INF') + # PEP 393 + self.assertEqual('%.1s' % "a\xe9\u20ac", 'a') + self.assertEqual('%.2s' % "a\xe9\u20ac", 'a\xe9') + def test_startswith_endswith_errors(self): for meth in ('foo'.startswith, 'foo'.endswith): with self.assertRaises(TypeError) as cm: diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13320,6 +13320,7 @@ Py_DECREF(writer->buffer); return NULL; } + assert(_PyUnicode_CheckConsistency(writer->buffer, 1)); return writer->buffer; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 03:29:53 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 09 May 2012 03:29:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Improve_the_gra?= =?utf8?q?mmar_of_a_non-sentence=2E?= Message-ID: http://hg.python.org/cpython/rev/597826f7f6f5 changeset: 76842:597826f7f6f5 branch: 3.2 parent: 76836:d937b527b76e user: R David Murray date: Tue May 08 21:28:24 2012 -0400 summary: Improve the grammar of a non-sentence. files: Doc/library/queue.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -15,8 +15,8 @@ availability of thread support in Python; see the :mod:`threading` module. -Implements three types of queue whose only difference is the order that -the entries are retrieved. In a FIFO queue, the first tasks added are +The module implements three types of queue, which differ only in the order in +which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 03:29:54 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 09 May 2012 03:29:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge=3A_Improve_the_grammar_of_a_non-sentence=2E?= Message-ID: http://hg.python.org/cpython/rev/8bfe89e9d851 changeset: 76843:8bfe89e9d851 parent: 76841:6d39b5ea4542 parent: 76842:597826f7f6f5 user: R David Murray date: Tue May 08 21:29:06 2012 -0400 summary: Merge: Improve the grammar of a non-sentence. files: Doc/library/queue.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -15,8 +15,8 @@ availability of thread support in Python; see the :mod:`threading` module. -Implements three types of queue whose only difference is the order that -the entries are retrieved. In a FIFO queue, the first tasks added are +The module implements three types of queue, which differ only in the order in +which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 03:29:55 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 09 May 2012 03:29:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Improve_the_gra?= =?utf8?q?mmar_of_a_non-sentence=2E?= Message-ID: http://hg.python.org/cpython/rev/5683ca62a954 changeset: 76844:5683ca62a954 branch: 2.7 parent: 76837:5319a4bf72e7 user: R David Murray date: Tue May 08 21:29:36 2012 -0400 summary: Improve the grammar of a non-sentence. files: Doc/library/queue.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -20,8 +20,8 @@ availability of thread support in Python; see the :mod:`threading` module. -Implements three types of queue whose only difference is the order that -the entries are retrieved. In a FIFO queue, the first tasks added are +The module implements three types of queue, which differ only in the order in +which the entries are retrieved. In a FIFO queue, the first tasks added are the first retrieved. In a LIFO queue, the most recently added entry is the first retrieved (operating like a stack). With a priority queue, the entries are kept sorted (using the :mod:`heapq` module) and the -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed May 9 05:42:44 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 09 May 2012 05:42:44 +0200 Subject: [Python-checkins] Daily reference leaks (8bfe89e9d851): sum=0 Message-ID: results for 8bfe89e9d851 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog_3yUyX', '-x'] From python-checkins at python.org Wed May 9 08:52:25 2012 From: python-checkins at python.org (larry.hastings) Date: Wed, 09 May 2012 08:52:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314746=3A_Remove_re?= =?utf8?q?dundant_paragraphs_from_skipitem=28=29_in_Python/getargs=2Ec=2E?= Message-ID: http://hg.python.org/cpython/rev/8ab37fa24e58 changeset: 76845:8ab37fa24e58 parent: 76843:8bfe89e9d851 user: Larry Hastings date: Tue May 08 23:52:03 2012 -0700 summary: Issue #14746: Remove redundant paragraphs from skipitem() in Python/getargs.c. files: Python/getargs.c | 26 ++++++++------------------ 1 files changed, 8 insertions(+), 18 deletions(-) diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1609,8 +1609,10 @@ switch (c) { - /* simple codes - * The individual types (second arg of va_arg) are irrelevant */ + /* + * codes that take a single data pointer as an argument + * (the type of the pointer is irrelevant) + */ case 'b': /* byte -- very short int */ case 'B': /* byte as bitfield */ @@ -1624,23 +1626,21 @@ case 'L': /* PY_LONG_LONG */ case 'K': /* PY_LONG_LONG sized bitfield */ #endif + case 'n': /* Py_ssize_t */ case 'f': /* float */ case 'd': /* double */ case 'D': /* complex double */ case 'c': /* char */ case 'C': /* unicode char */ case 'p': /* boolean predicate */ + case 'S': /* string object */ + case 'Y': /* string object */ + case 'U': /* unicode string object */ { (void) va_arg(*p_va, void *); break; } - case 'n': /* Py_ssize_t */ - { - (void) va_arg(*p_va, Py_ssize_t *); - break; - } - /* string codes */ case 'e': /* string with encoding */ @@ -1673,16 +1673,6 @@ break; } - /* object codes */ - - case 'S': /* string object */ - case 'Y': /* string object */ - case 'U': /* unicode string object */ - { - (void) va_arg(*p_va, PyObject **); - break; - } - case 'O': /* object */ { if (*format == '!') { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 13:27:18 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 09 May 2012 13:27:18 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0NzYx?= =?utf8?q?=3A_Fix_potential_leak_on_an_error_case_in_the_import_machinery?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/a775fc27f469 changeset: 76846:a775fc27f469 branch: 2.7 parent: 76844:5683ca62a954 user: Antoine Pitrou date: Wed May 09 13:24:31 2012 +0200 summary: Issue #14761: Fix potential leak on an error case in the import machinery. files: Misc/ACKS | 1 + Misc/NEWS | 2 ++ Python/import.c | 3 ++- 3 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -127,6 +127,7 @@ Brett Cannon Mike Carlton Terry Carroll +Damien Cassou Lorenzo M. Catucci Donn Cave Charles Cazabon diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,8 @@ Core and Builtins ----------------- +- Issue #14761: Fix potential leak on an error case in the import machinery. + - Issue #14699: Fix calling the classmethod descriptor directly. - Issue #11603 (again): Setting __repr__ to __str__ now raises a RuntimeError diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -998,7 +998,7 @@ FILE *fpc; char *buf; char *cpathname; - PyCodeObject *co; + PyCodeObject *co = NULL; PyObject *m; if (fstat(fileno(fp), &st) != 0) { @@ -1054,6 +1054,7 @@ return m; error_exit: + Py_XDECREF(co); PyMem_FREE(buf); return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 13:30:24 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 09 May 2012 13:30:24 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0NzYx?= =?utf8?q?=3A_Fix_potential_leak_on_an_error_case_in_the_import_machinery?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/9de4d85e4197 changeset: 76847:9de4d85e4197 branch: 3.2 parent: 76842:597826f7f6f5 user: Antoine Pitrou date: Wed May 09 13:24:31 2012 +0200 summary: Issue #14761: Fix potential leak on an error case in the import machinery. files: Misc/ACKS | 1 + Misc/NEWS | 2 ++ Python/import.c | 3 ++- 3 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -145,6 +145,7 @@ Brett Cannon Mike Carlton Terry Carroll +Damien Cassou Lorenzo M. Catucci Donn Cave Charles Cazabon diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #14761: Fix potential leak on an error case in the import machinery. + - Issue #14699: Fix calling the classmethod descriptor directly. - Issue #14433: Prevent msvcrt crash in interactive prompt when stdin diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1293,7 +1293,7 @@ FILE *fpc; char *buf; char *cpathname; - PyCodeObject *co; + PyCodeObject *co = NULL; PyObject *m; if (fstat(fileno(fp), &st) != 0) { @@ -1350,6 +1350,7 @@ return m; error_exit: + Py_XDECREF(co); PyMem_FREE(buf); return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 13:30:24 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 09 May 2012 13:30:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Null_merge_for_issue_=2314761=2E?= Message-ID: http://hg.python.org/cpython/rev/840cb46d0395 changeset: 76848:840cb46d0395 parent: 76845:8ab37fa24e58 parent: 76847:9de4d85e4197 user: Antoine Pitrou date: Wed May 09 13:28:05 2012 +0200 summary: Null merge for issue #14761. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 13:59:33 2012 From: python-checkins at python.org (ezio.melotti) Date: Wed, 09 May 2012 13:59:33 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NzYzOiBmaXgg?= =?utf8?q?documentation_for_string=2Esplit/rsplit=2E?= Message-ID: http://hg.python.org/cpython/rev/d3ddbad31b3e changeset: 76849:d3ddbad31b3e branch: 2.7 parent: 76846:a775fc27f469 user: Ezio Melotti date: Wed May 09 14:59:24 2012 +0300 summary: #14763: fix documentation for string.split/rsplit. files: Doc/library/string.rst | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-) diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -905,14 +905,15 @@ Return a list of the words of the string *s*. If the optional second argument *sep* is absent or ``None``, the words are separated by arbitrary strings of - whitespace characters (space, tab, newline, return, formfeed). If the second + whitespace characters (space, tab, newline, return, formfeed). If the second argument *sep* is present and not ``None``, it specifies a string to be used as the word separator. The returned list will then have one more item than the - number of non-overlapping occurrences of the separator in the string. The - optional third argument *maxsplit* defaults to 0. If it is nonzero, at most - *maxsplit* number of splits occur, and the remainder of the string is returned - as the final element of the list (thus, the list will have at most - ``maxsplit+1`` elements). + number of non-overlapping occurrences of the separator in the string. + If *maxsplit* is given, at most *maxsplit* number of splits occur, and the + remainder of the string is returned as the final element of the list (thus, + the list will have at most ``maxsplit+1`` elements). If *maxsplit* is not + specified or ``-1``, then there is no limit on the number of splits (all + possible splits are made). The behavior of split on an empty string depends on the value of *sep*. If *sep* is not specified, or specified as ``None``, the result will be an empty list. @@ -925,7 +926,7 @@ Return a list of the words of the string *s*, scanning *s* from the end. To all intents and purposes, the resulting list of words is the same as returned by :func:`split`, except when the optional third argument *maxsplit* is explicitly - specified and nonzero. When *maxsplit* is nonzero, at most *maxsplit* number of + specified and nonzero. If *maxsplit* is given, at most *maxsplit* number of splits -- the *rightmost* ones -- occur, and the remainder of the string is returned as the first element of the list (thus, the list will have at most ``maxsplit+1`` elements). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 22:32:52 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 09 May 2012 22:32:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314744=3A_Inline_un?= =?utf8?q?icode=5Fwriter=5Fwrite=5Fchar=28=29_and_unicode=5Fwrite=5Fstr=28?= =?utf8?q?=29?= Message-ID: http://hg.python.org/cpython/rev/6c8a117f8966 changeset: 76850:6c8a117f8966 parent: 76848:840cb46d0395 user: Victor Stinner date: Wed May 09 22:24:08 2012 +0200 summary: Issue #14744: Inline unicode_writer_write_char() and unicode_write_str() Optimize also PyUnicode_Format(): call unicode_writer_prepare() only once per argument. files: Objects/stringlib/unicode_format.h | 36 +++- Objects/unicodeobject.c | 119 +++++++--------- 2 files changed, 79 insertions(+), 76 deletions(-) diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h --- a/Objects/stringlib/unicode_format.h +++ b/Objects/stringlib/unicode_format.h @@ -500,6 +500,7 @@ PyObject *result = NULL; PyObject *format_spec_object = NULL; PyObject *(*formatter)(PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL; + Py_ssize_t len; /* If we know the type exactly, skip the lookup of __format__ and just call the formatter directly. */ @@ -533,12 +534,19 @@ result = PyObject_Format(fieldobj, format_spec_object); } - if (result == NULL || PyUnicode_READY(result) == -1) + if (result == NULL) + goto done; + if (PyUnicode_READY(result) == -1) goto done; - assert(PyUnicode_Check(result)); - - ok = (unicode_writer_write_str(writer, result, 0, PyUnicode_GET_LENGTH(result)) == 0); + len = PyUnicode_GET_LENGTH(result); + if (unicode_writer_prepare(writer, + len, PyUnicode_MAX_CHAR_VALUE(result)) == -1) + goto done; + copy_characters(writer->buffer, writer->pos, + result, 0, len); + writer->pos += len; + ok = 1; done: Py_XDECREF(format_spec_object); Py_XDECREF(result); @@ -873,7 +881,8 @@ SubString literal; SubString field_name; SubString format_spec; - Py_UCS4 conversion; + Py_UCS4 conversion, maxchar; + Py_ssize_t sublen; int err; MarkupIterator_init(&iter, input->str, input->start, input->end); @@ -881,11 +890,18 @@ &field_name, &format_spec, &conversion, &format_spec_needs_expanding)) == 2) { - err = unicode_writer_write_str(writer, - literal.str, literal.start, - literal.end - literal.start); - if (err == -1) - return 0; + sublen = literal.end - literal.start; + if (sublen) { + maxchar = _PyUnicode_FindMaxChar(literal.str, + literal.start, literal.end); + err = unicode_writer_prepare(writer, sublen, maxchar); + if (err == -1) + return 0; + copy_characters(writer->buffer, writer->pos, + literal.str, literal.start, sublen); + writer->pos += sublen; + } + if (field_present) if (!output_markup(&field_name, &format_spec, format_spec_needs_expanding, conversion, writer, diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -1150,14 +1150,15 @@ void *from_data, *to_data; int fast; + assert(0 <= how_many); + assert(0 <= from_start); + assert(0 <= to_start); assert(PyUnicode_Check(from)); assert(PyUnicode_Check(to)); assert(PyUnicode_IS_READY(from)); assert(PyUnicode_IS_READY(to)); - - assert(PyUnicode_GET_LENGTH(from) >= how_many); + assert(from_start + how_many <= PyUnicode_GET_LENGTH(from)); assert(to_start + how_many <= PyUnicode_GET_LENGTH(to)); - assert(0 <= how_many); if (how_many == 0) return 0; @@ -13271,48 +13272,6 @@ return 0; } -Py_LOCAL_INLINE(int) -unicode_writer_write_str( - unicode_writer_t *writer, - PyObject *str, Py_ssize_t start, Py_ssize_t length) -{ - Py_UCS4 maxchar; - - assert(str != NULL); - assert(PyUnicode_Check(str)); - if (PyUnicode_READY(str) == -1) - return -1; - - assert(0 <= start); - assert(0 <= length); - assert(start + length <= PyUnicode_GET_LENGTH(str)); - if (length == 0) - return 0; - - maxchar = _PyUnicode_FindMaxChar(str, start, start + length); - if (unicode_writer_prepare(writer, length, maxchar) == -1) - return -1; - - assert((writer->pos + length) <= PyUnicode_GET_LENGTH(writer->buffer)); - copy_characters(writer->buffer, writer->pos, - str, start, length); - writer->pos += length; - return 0; -} - -Py_LOCAL_INLINE(int) -unicode_writer_write_char( - unicode_writer_t *writer, - Py_UCS4 ch) -{ - if (unicode_writer_prepare(writer, 1, ch) == -1) - return -1; - assert((writer->pos + 1) <= PyUnicode_GET_LENGTH(writer->buffer)); - PyUnicode_WRITE(writer->kind, writer->data, writer->pos, ch); - writer->pos += 1; - return 0; -} - Py_LOCAL(PyObject *) unicode_writer_finish(unicode_writer_t *writer) { @@ -13791,6 +13750,8 @@ void *fmt; enum PyUnicode_Kind kind, fmtkind; unicode_writer_t writer; + Py_ssize_t sublen; + Py_UCS4 maxchar; if (format == NULL || args == NULL) { PyErr_BadInternalCall(); @@ -13833,8 +13794,15 @@ } if (fmtcnt < 0) fmtpos--; - if (unicode_writer_write_str(&writer, uformat, nonfmtpos, fmtpos - nonfmtpos) < 0) + sublen = fmtpos - nonfmtpos; + maxchar = _PyUnicode_FindMaxChar(uformat, + nonfmtpos, nonfmtpos + sublen); + if (unicode_writer_prepare(&writer, sublen, maxchar) == -1) goto onError; + + copy_characters(writer.buffer, writer.pos, + uformat, nonfmtpos, sublen); + writer.pos += sublen; } else { /* Got a format specifier */ @@ -13849,6 +13817,8 @@ PyObject *v = NULL; void *pbuf = NULL; Py_ssize_t pindex, len; + Py_UCS4 bufmaxchar; + Py_ssize_t buflen; fmtpos++; c = PyUnicode_READ(fmtkind, fmt, fmtpos); @@ -13991,8 +13961,10 @@ } if (c == '%') { - if (unicode_writer_write_char(&writer, '%') < 0) + if (unicode_writer_prepare(&writer, 1, '%') == -1) goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, '%'); + writer.pos += 1; continue; } @@ -14126,10 +14098,35 @@ } if (width < len) width = len; + + /* Compute the length and maximum character of the + written characters */ + bufmaxchar = 127; + if (!(flags & F_LJUST)) { + if (sign) { + if ((width-1) > len) + bufmaxchar = Py_MAX(bufmaxchar, fill); + } + else { + if (width > len) + bufmaxchar = Py_MAX(bufmaxchar, fill); + } + } + maxchar = _PyUnicode_FindMaxChar(temp, 0, pindex+len); + bufmaxchar = Py_MAX(bufmaxchar, maxchar); + + buflen = width; + if (sign && len == width) + buflen++; + + if (unicode_writer_prepare(&writer, buflen, bufmaxchar) == -1) + goto onError; + + /* Write characters */ if (sign) { if (fill != ' ') { - if (unicode_writer_write_char(&writer, signchar) < 0) - goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, signchar); + writer.pos += 1; } if (width > len) width--; @@ -14138,8 +14135,6 @@ assert(PyUnicode_READ(kind, pbuf, pindex) == '0'); assert(PyUnicode_READ(kind, pbuf, pindex + 1) == c); if (fill != ' ') { - if (unicode_writer_prepare(&writer, 2, 127) < 0) - goto onError; PyUnicode_WRITE(writer.kind, writer.data, writer.pos, '0'); PyUnicode_WRITE(writer.kind, writer.data, writer.pos+1, c); writer.pos += 2; @@ -14151,43 +14146,35 @@ len -= 2; } if (width > len && !(flags & F_LJUST)) { - Py_ssize_t sublen; sublen = width - len; - if (unicode_writer_prepare(&writer, sublen, fill) < 0) - goto onError; FILL(writer.kind, writer.data, fill, writer.pos, sublen); writer.pos += sublen; width = len; } if (fill == ' ') { if (sign) { - if (unicode_writer_write_char(&writer, signchar) < 0) - goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, signchar); + writer.pos += 1; } if ((flags & F_ALT) && (c == 'x' || c == 'X' || c == 'o')) { assert(PyUnicode_READ(kind, pbuf, pindex) == '0'); assert(PyUnicode_READ(kind, pbuf, pindex+1) == c); - - if (unicode_writer_prepare(&writer, 2, 127) < 0) - goto onError; PyUnicode_WRITE(writer.kind, writer.data, writer.pos, '0'); PyUnicode_WRITE(writer.kind, writer.data, writer.pos+1, c); writer.pos += 2; - pindex += 2; } } - /* Copy all characters, preserving len */ - if (unicode_writer_write_str(&writer, temp, pindex, len) < 0) - goto onError; + copy_characters(writer.buffer, writer.pos, + temp, pindex, len); + writer.pos += len; if (width > len) { - Py_ssize_t sublen = width - len; - if (unicode_writer_prepare(&writer, sublen, ' ') < 0) - goto onError; + sublen = width - len; FILL(writer.kind, writer.data, ' ', writer.pos, sublen); writer.pos += sublen; } + if (dict && (argidx < arglen) && c != '%') { PyErr_SetString(PyExc_TypeError, "not all arguments converted during string formatting"); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 9 22:32:53 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 09 May 2012 22:32:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Rename_unicode=5Fwrite=5Ft_?= =?utf8?q?structure_and_its_methods_to_=22=5FPyUnicodeWriter=22?= Message-ID: http://hg.python.org/cpython/rev/0d9067cde462 changeset: 76851:0d9067cde462 user: Victor Stinner date: Wed May 09 22:25:00 2012 +0200 summary: Rename unicode_write_t structure and its methods to "_PyUnicodeWriter" files: Objects/stringlib/unicode_format.h | 18 +++++----- Objects/unicodeobject.c | 32 +++++++++--------- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h --- a/Objects/stringlib/unicode_format.h +++ b/Objects/stringlib/unicode_format.h @@ -494,7 +494,7 @@ appends to the output. */ static int -render_field(PyObject *fieldobj, SubString *format_spec, unicode_writer_t *writer) +render_field(PyObject *fieldobj, SubString *format_spec, _PyUnicodeWriter *writer) { int ok = 0; PyObject *result = NULL; @@ -540,7 +540,7 @@ goto done; len = PyUnicode_GET_LENGTH(result); - if (unicode_writer_prepare(writer, + if (_PyUnicodeWriter_Prepare(writer, len, PyUnicode_MAX_CHAR_VALUE(result)) == -1) goto done; copy_characters(writer->buffer, writer->pos, @@ -811,7 +811,7 @@ static int output_markup(SubString *field_name, SubString *format_spec, int format_spec_needs_expanding, Py_UCS4 conversion, - unicode_writer_t *writer, PyObject *args, PyObject *kwargs, + _PyUnicodeWriter *writer, PyObject *args, PyObject *kwargs, int recursion_depth, AutoNumber *auto_number) { PyObject *tmp = NULL; @@ -872,7 +872,7 @@ */ static int do_markup(SubString *input, PyObject *args, PyObject *kwargs, - unicode_writer_t *writer, int recursion_depth, AutoNumber *auto_number) + _PyUnicodeWriter *writer, int recursion_depth, AutoNumber *auto_number) { MarkupIterator iter; int format_spec_needs_expanding; @@ -894,7 +894,7 @@ if (sublen) { maxchar = _PyUnicode_FindMaxChar(literal.str, literal.start, literal.end); - err = unicode_writer_prepare(writer, sublen, maxchar); + err = _PyUnicodeWriter_Prepare(writer, sublen, maxchar); if (err == -1) return 0; copy_characters(writer->buffer, writer->pos, @@ -920,7 +920,7 @@ build_string(SubString *input, PyObject *args, PyObject *kwargs, int recursion_depth, AutoNumber *auto_number) { - unicode_writer_t writer; + _PyUnicodeWriter writer; Py_ssize_t initlen; /* check the recursion level */ @@ -931,16 +931,16 @@ } initlen = PyUnicode_GET_LENGTH(input->str) + 100; - if (unicode_writer_init(&writer, initlen, 127) == -1) + if (_PyUnicodeWriter_Init(&writer, initlen, 127) == -1) return NULL; if (!do_markup(input, args, kwargs, &writer, recursion_depth, auto_number)) { - unicode_writer_dealloc(&writer); + _PyUnicodeWriter_Dealloc(&writer); return NULL; } - return unicode_writer_finish(&writer); + return _PyUnicodeWriter_Finish(&writer); } /************************************************************************/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13207,10 +13207,10 @@ enum PyUnicode_Kind kind; Py_UCS4 maxchar; Py_ssize_t pos; -} unicode_writer_t; +} _PyUnicodeWriter ; Py_LOCAL_INLINE(void) -unicode_writer_update(unicode_writer_t *writer) +_PyUnicodeWriter_Update(_PyUnicodeWriter *writer) { writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); writer->data = PyUnicode_DATA(writer->buffer); @@ -13218,19 +13218,19 @@ } Py_LOCAL(int) -unicode_writer_init(unicode_writer_t *writer, +_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, Py_ssize_t length, Py_UCS4 maxchar) { writer->pos = 0; writer->buffer = PyUnicode_New(length, maxchar); if (writer->buffer == NULL) return -1; - unicode_writer_update(writer); + _PyUnicodeWriter_Update(writer); return 0; } Py_LOCAL_INLINE(int) -unicode_writer_prepare(unicode_writer_t *writer, +_PyUnicodeWriter_Prepare(_PyUnicodeWriter *writer, Py_ssize_t length, Py_UCS4 maxchar) { Py_ssize_t newlen; @@ -13262,18 +13262,18 @@ return -1; } writer->buffer = newbuffer; - unicode_writer_update(writer); + _PyUnicodeWriter_Update(writer); } else if (maxchar > writer->maxchar) { if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) return -1; - unicode_writer_update(writer); + _PyUnicodeWriter_Update(writer); } return 0; } Py_LOCAL(PyObject *) -unicode_writer_finish(unicode_writer_t *writer) +_PyUnicodeWriter_Finish(_PyUnicodeWriter *writer) { if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { Py_DECREF(writer->buffer); @@ -13284,7 +13284,7 @@ } Py_LOCAL(void) -unicode_writer_dealloc(unicode_writer_t *writer) +_PyUnicodeWriter_Dealloc(_PyUnicodeWriter *writer) { Py_CLEAR(writer->buffer); } @@ -13749,7 +13749,7 @@ PyObject *uformat; void *fmt; enum PyUnicode_Kind kind, fmtkind; - unicode_writer_t writer; + _PyUnicodeWriter writer; Py_ssize_t sublen; Py_UCS4 maxchar; @@ -13768,7 +13768,7 @@ fmtcnt = PyUnicode_GET_LENGTH(uformat); fmtpos = 0; - if (unicode_writer_init(&writer, fmtcnt + 100, 127) < 0) + if (_PyUnicodeWriter_Init(&writer, fmtcnt + 100, 127) < 0) goto onError; if (PyTuple_Check(args)) { @@ -13797,7 +13797,7 @@ sublen = fmtpos - nonfmtpos; maxchar = _PyUnicode_FindMaxChar(uformat, nonfmtpos, nonfmtpos + sublen); - if (unicode_writer_prepare(&writer, sublen, maxchar) == -1) + if (_PyUnicodeWriter_Prepare(&writer, sublen, maxchar) == -1) goto onError; copy_characters(writer.buffer, writer.pos, @@ -13961,7 +13961,7 @@ } if (c == '%') { - if (unicode_writer_prepare(&writer, 1, '%') == -1) + if (_PyUnicodeWriter_Prepare(&writer, 1, '%') == -1) goto onError; PyUnicode_WRITE(writer.kind, writer.data, writer.pos, '%'); writer.pos += 1; @@ -14119,7 +14119,7 @@ if (sign && len == width) buflen++; - if (unicode_writer_prepare(&writer, buflen, bufmaxchar) == -1) + if (_PyUnicodeWriter_Prepare(&writer, buflen, bufmaxchar) == -1) goto onError; /* Write characters */ @@ -14195,13 +14195,13 @@ Py_DECREF(uformat); Py_XDECREF(temp); Py_XDECREF(second); - return unicode_writer_finish(&writer); + return _PyUnicodeWriter_Finish(&writer); onError: Py_DECREF(uformat); Py_XDECREF(temp); Py_XDECREF(second); - unicode_writer_dealloc(&writer); + _PyUnicodeWriter_Dealloc(&writer); if (args_owned) { Py_DECREF(args); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 02:16:15 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 10 May 2012 02:16:15 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Updated_to_discuss_find=5Fload?= =?utf8?q?er_method_on_finders=2E?= Message-ID: http://hg.python.org/peps/rev/4aceb79a3859 changeset: 4363:4aceb79a3859 user: Eric V. Smith date: Wed May 09 20:16:09 2012 -0400 summary: Updated to discuss find_loader method on finders. files: pep-0420.txt | 61 +++++++++++++++++++++++++++++++++------ 1 files changed, 51 insertions(+), 10 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -163,21 +163,39 @@ ------------------------------------ PEP 302 defines "finders" that are called to search path elements. -These finders' ``find_module`` methods currently return either a -"loader" object or None. For a finder to contribute to namespace -packages, ``find_module`` will return a third type: a string. This is -the string that will be recorded and later used as a component of the -namespace module's ``__path__``, as described above. This string must -not contain a trailing path separator. +These finders' ``find_module`` methods return either a "loader" object +or None. + +For a finder to contribute to namespace packages, it must implement a +new ``find_loader(fullname)`` method. ``fullname`` has the same +meaning as for ``find_module``. ``find_loader`` has 3 possible return +values, depending on whether it found a normal package, a namespace +package portion, or neither. In all cases the return value is a +2-tuple. + +===== ======================= ============ +Case Meaning Return value +----- ----------------------- ------------ +1 normal package found ``(loader, None)`` +2 portion(s) found ``(None, )`` +3 neither found ``(None, None)`` +===== ======================= ============ + +Note that in case 2 a sequence of strings is returned. This is to +support the case where a finder discovers multiple namespace portions +for a given ``fullname``. Many finders will support only a single +namespace package portion per ``find_loader`` call, in which case this +will normally be a list containing only a single string. + +Legacy finders which implement ``find_module`` but not ``find_loader`` +will be unable to contribute portions to a namespace package. The +import machinery will call ``find_loader`` if it exists, else fall +back to ``find_module``. The specification expands PEP 302 loaders to include an optional method called ``module_repr()`` which if present, is used to generate module object reprs. See the section below for further details. -If an existing finder is not updated to support returning a string -from ``find_module``, the only impact is that such a finder will be -unable to provide portions of a namespace package. - Packaging Implications ====================== @@ -259,6 +277,26 @@ date to add such features. Several possible ways to do so were discussed in the referenced email thread. +``find_module`` versus ``find_loader`` +-------------------------------------- + +An early draft of this PEP specified a change to the ``find_module`` +method in order to support namespace packages. It would be modified +to return a string in the case where a namespace package portion was +discovered. + +However, this caused a problem with existing code outside of the +standard library which calls ``find_module``. Because this code would +not be upgraded in concert with changes required by this PEP, it would +fail when it would receive unexpected return values from +``find_module``. Because of this incompatibility, this PEP now +specifies that finders that want to provide namespace portions must +implement the ``find_loader`` method, described above. + +The use case for supporting multiple portions per ``find_loader`` call +is given in [6]_. + + Module reprs ============ @@ -316,6 +354,9 @@ .. [5] Phillip Eby's question about auto-updating __path__ (http://mail.python.org/pipermail/import-sig/2012-April/000468.html) +.. [6] Use case for multiple portions per ``find_loader`` call + (http://mail.python.org/pipermail/import-sig/2012-May/000585.html) + Copyright ========= -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 10 02:22:35 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 10 May 2012 02:22:35 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Two_spaces_after_periods=2E?= Message-ID: http://hg.python.org/peps/rev/342624df7c93 changeset: 4364:342624df7c93 user: Eric V. Smith date: Wed May 09 20:22:29 2012 -0400 summary: Two spaces after periods. files: pep-0420.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -152,9 +152,9 @@ deferred until a sub-level import occurs. A namespace package is not fundamentally different from a regular -package. It is just a different way of creating packages. Once a +package. It is just a different way of creating packages. Once a namespace package is created, there is no functional difference -between it and a regular package. The only observable difference is +between it and a regular package. The only observable difference is that the namespace package's ``__file__`` attribute will end with a path separator (typically a slash or backslash, depending on the platform). @@ -219,7 +219,7 @@ "foo" directories would be in directories that are on ``sys.path``. "foo/bar" would be in one of these sys.path entries, and "foo/baz" would be in the other. Upon removal of "foo.bar", the "foo/bar" and -corresponding "foo" directories can be completely removed. But +corresponding "foo" directories can be completely removed. But "foo/baz" and its corresponding "foo" directory cannot be removed. It is also possible to have the "foo.bar" portion installed in a -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 10 05:16:58 2012 From: python-checkins at python.org (jesus.cea) Date: Thu, 10 May 2012 05:16:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_=2314768?= =?utf8?b?OiBvcy5wYXRoLmV4cGFuZHVzZXIoJ34vYScpIGRvZXNuJ3Qgd29ya3MgY29ycmVj?= =?utf8?q?tly_when_HOME_is?= Message-ID: http://hg.python.org/cpython/rev/e472481b6d73 changeset: 76852:e472481b6d73 branch: 3.2 parent: 76847:9de4d85e4197 user: Jesus Cea date: Thu May 10 05:10:50 2012 +0200 summary: Closes #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/' files: Lib/posixpath.py | 4 ++-- Lib/test/test_posixpath.py | 1 + Misc/ACKS | 1 + Misc/NEWS | 2 ++ 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/posixpath.py b/Lib/posixpath.py --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -266,8 +266,8 @@ root = b'/' else: root = '/' - userhome = userhome.rstrip(root) or userhome - return userhome + path[i:] + userhome = userhome.rstrip(root) + return (userhome + path[i:]) or root # Expand paths containing shell variable substitutions. diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -298,6 +298,7 @@ with support.EnvironmentVarGuard() as env: env['HOME'] = '/' self.assertEqual(posixpath.expanduser("~"), "/") + self.assertEqual(posixpath.expanduser("~/foo"), "/foo") # expanduser should fall back to using the password database del env['HOME'] home = pwd.getpwuid(os.getuid()).pw_dir diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -308,6 +308,7 @@ John Fouhy Martin Franklin Robin Friedrich +Bradley Froehle Ivan Frohne Jim Fulton Tadayoshi Funaba diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,8 @@ Library ------- +- Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. + - Issue #14741: Fix missing support for Ellipsis ('...') in parser module. - Issue #14697: Fix missing support for set displays and set comprehensions in -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 05:16:59 2012 From: python-checkins at python.org (jesus.cea) Date: Thu, 10 May 2012 05:16:59 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_=2314768?= =?utf8?b?OiBvcy5wYXRoLmV4cGFuZHVzZXIoJ34vYScpIGRvZXNuJ3Qgd29ya3MgY29ycmVj?= =?utf8?q?tly_when_HOME_is?= Message-ID: http://hg.python.org/cpython/rev/299dc54ad014 changeset: 76853:299dc54ad014 branch: 2.7 parent: 76849:d3ddbad31b3e user: Jesus Cea date: Thu May 10 05:01:11 2012 +0200 summary: Closes #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/' files: Lib/posixpath.py | 4 ++-- Lib/test/test_posixpath.py | 1 + Misc/ACKS | 1 + Misc/NEWS | 2 ++ 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/posixpath.py b/Lib/posixpath.py --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -267,8 +267,8 @@ except KeyError: return path userhome = pwent.pw_dir - userhome = userhome.rstrip('/') or userhome - return userhome + path[i:] + userhome = userhome.rstrip('/') + return (userhome + path[i:]) or '/' # Expand paths containing shell variable substitutions. diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -201,6 +201,7 @@ with test_support.EnvironmentVarGuard() as env: env['HOME'] = '/' self.assertEqual(posixpath.expanduser("~"), "/") + self.assertEqual(posixpath.expanduser("~/foo"), "/foo") def test_normpath(self): self.assertEqual(posixpath.normpath(""), ".") diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -274,6 +274,7 @@ John Fouhy Martin Franklin Robin Friedrich +Bradley Froehle Ivan Frohne Jim Fulton Tadayoshi Funaba diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,8 @@ Library ------- +- Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. + - Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running step. Patch by Xavier de Gaye. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 05:17:00 2012 From: python-checkins at python.org (jesus.cea) Date: Thu, 10 May 2012 05:17:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?b?OiBNRVJHRTogQ2xvc2VzICMxNDc2ODogb3MucGF0aC5leHBhbmR1c2VyKCd+L2En?= =?utf8?q?=29_doesn=27t_works_correctly_when?= Message-ID: http://hg.python.org/cpython/rev/ce39a4ec2906 changeset: 76854:ce39a4ec2906 parent: 76851:0d9067cde462 parent: 76852:e472481b6d73 user: Jesus Cea date: Thu May 10 05:16:41 2012 +0200 summary: MERGE: Closes #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/' files: Lib/posixpath.py | 4 ++-- Lib/test/test_posixpath.py | 1 + Misc/ACKS | 1 + Misc/NEWS | 2 ++ 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/posixpath.py b/Lib/posixpath.py --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -266,8 +266,8 @@ root = b'/' else: root = '/' - userhome = userhome.rstrip(root) or userhome - return userhome + path[i:] + userhome = userhome.rstrip(root) + return (userhome + path[i:]) or root # Expand paths containing shell variable substitutions. diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -300,6 +300,7 @@ with support.EnvironmentVarGuard() as env: env['HOME'] = '/' self.assertEqual(posixpath.expanduser("~"), "/") + self.assertEqual(posixpath.expanduser("~/foo"), "/foo") # expanduser should fall back to using the password database del env['HOME'] home = pwd.getpwuid(os.getuid()).pw_dir diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -335,6 +335,7 @@ Martin Franklin Bruce Frederiksen Robin Friedrich +Bradley Froehle Ivan Frohne Matthias Fuchs Jim Fulton diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -139,6 +139,8 @@ - Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. +- Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. + - Issue #14371: Support bzip2 in zipfile module. Patch by Serhiy Storchaka. - Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu May 10 05:41:12 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 10 May 2012 05:41:12 +0200 Subject: [Python-checkins] Daily reference leaks (0d9067cde462): sum=0 Message-ID: results for 0d9067cde462 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog5L9O44', '-x'] From python-checkins at python.org Thu May 10 14:33:21 2012 From: python-checkins at python.org (ezio.melotti) Date: Thu, 10 May 2012 14:33:21 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NzYzOiBkb2N1?= =?utf8?q?ment_default_maxsplit_value_for_str=2Esplit=2E?= Message-ID: http://hg.python.org/cpython/rev/0415ecd7b0e3 changeset: 76855:0415ecd7b0e3 branch: 2.7 parent: 76853:299dc54ad014 user: Ezio Melotti date: Thu May 10 15:30:42 2012 +0300 summary: #14763: document default maxsplit value for str.split. files: Doc/library/stdtypes.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1161,8 +1161,8 @@ Return a list of the words in the string, using *sep* as the delimiter string. If *maxsplit* is given, at most *maxsplit* splits are done (thus, the list will have at most ``maxsplit+1`` elements). If *maxsplit* is not - specified, then there is no limit on the number of splits (all possible - splits are made). + specified or ``-1``, then there is no limit on the number of splits + (all possible splits are made). If *sep* is given, consecutive delimiters are not grouped together and are deemed to delimit empty strings (for example, ``'1,,2'.split(',')`` returns -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 14:33:23 2012 From: python-checkins at python.org (ezio.melotti) Date: Thu, 10 May 2012 14:33:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NzYzOiBkb2N1?= =?utf8?q?ment_default_maxsplit_value_for_str=2Esplit=2E?= Message-ID: http://hg.python.org/cpython/rev/62659067f5b6 changeset: 76856:62659067f5b6 branch: 3.2 parent: 76852:e472481b6d73 user: Ezio Melotti date: Thu May 10 15:30:42 2012 +0300 summary: #14763: document default maxsplit value for str.split. files: Doc/library/stdtypes.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1304,8 +1304,8 @@ Return a list of the words in the string, using *sep* as the delimiter string. If *maxsplit* is given, at most *maxsplit* splits are done (thus, the list will have at most ``maxsplit+1`` elements). If *maxsplit* is not - specified, then there is no limit on the number of splits (all possible - splits are made). + specified or ``-1``, then there is no limit on the number of splits + (all possible splits are made). If *sep* is given, consecutive delimiters are not grouped together and are deemed to delimit empty strings (for example, ``'1,,2'.split(',')`` returns -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 14:33:28 2012 From: python-checkins at python.org (ezio.melotti) Date: Thu, 10 May 2012 14:33:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314763=3A_merge_with_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/bcc964092437 changeset: 76857:bcc964092437 parent: 76854:ce39a4ec2906 parent: 76856:62659067f5b6 user: Ezio Melotti date: Thu May 10 15:33:13 2012 +0300 summary: #14763: merge with 3.2. files: Doc/library/stdtypes.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1328,8 +1328,8 @@ Return a list of the words in the string, using *sep* as the delimiter string. If *maxsplit* is given, at most *maxsplit* splits are done (thus, the list will have at most ``maxsplit+1`` elements). If *maxsplit* is not - specified, then there is no limit on the number of splits (all possible - splits are made). + specified or ``-1``, then there is no limit on the number of splits + (all possible splits are made). If *sep* is given, consecutive delimiters are not grouped together and are deemed to delimit empty strings (for example, ``'1,,2'.split(',')`` returns -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 15:21:13 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 15:21:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Removed_outdate?= =?utf8?q?d_statement_about_pickle=27s_and_marshal=27s_relative_performanc?= =?utf8?q?e=2E?= Message-ID: http://hg.python.org/cpython/rev/b058f22b5723 changeset: 76858:b058f22b5723 branch: 3.2 parent: 76856:62659067f5b6 user: Antoine Pitrou date: Thu May 10 15:18:20 2012 +0200 summary: Removed outdated statement about pickle's and marshal's relative performance. files: Doc/faq/library.rst | 8 -------- 1 files changed, 0 insertions(+), 8 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -805,14 +805,6 @@ :mod:`shelve` library module uses pickle and (g)dbm to create persistent mappings containing arbitrary Python objects. -A more awkward way of doing things is to use pickle's little sister, marshal. -The :mod:`marshal` module provides very fast ways to store noncircular basic -Python types to files and strings, and back again. Although marshal does not do -fancy things like store instances or handle shared references properly, it does -run extremely fast. For example loading a half megabyte of data may take less -than a third of a second. This often beats doing something more complex and -general such as using gdbm with pickle/shelve. - Mathematics and Numerics ======================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 15:21:14 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 15:21:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Removed_outdated_statement_about_pickle=27s_and_marshal=27s_?= =?utf8?q?relative_performance=2E?= Message-ID: http://hg.python.org/cpython/rev/016ba8918d2e changeset: 76859:016ba8918d2e parent: 76857:bcc964092437 parent: 76858:b058f22b5723 user: Antoine Pitrou date: Thu May 10 15:18:40 2012 +0200 summary: Removed outdated statement about pickle's and marshal's relative performance. files: Doc/faq/library.rst | 8 -------- 1 files changed, 0 insertions(+), 8 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -805,14 +805,6 @@ :mod:`shelve` library module uses pickle and (g)dbm to create persistent mappings containing arbitrary Python objects. -A more awkward way of doing things is to use pickle's little sister, marshal. -The :mod:`marshal` module provides very fast ways to store noncircular basic -Python types to files and strings, and back again. Although marshal does not do -fancy things like store instances or handle shared references properly, it does -run extremely fast. For example loading a half megabyte of data may take less -than a third of a second. This often beats doing something more complex and -general such as using gdbm with pickle/shelve. - Mathematics and Numerics ======================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 15:42:17 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 15:42:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Some_nits_in_th?= =?utf8?q?e_pickle_docs=2E?= Message-ID: http://hg.python.org/cpython/rev/1b5429e835e0 changeset: 76860:1b5429e835e0 branch: 3.2 parent: 76858:b058f22b5723 user: Antoine Pitrou date: Thu May 10 15:38:30 2012 +0200 summary: Some nits in the pickle docs. files: Doc/library/pickle.rst | 44 +++++++++++++++-------------- 1 files changed, 23 insertions(+), 21 deletions(-) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -85,45 +85,48 @@ ------------------ .. index:: - single: XDR single: External Data Representation The data format used by :mod:`pickle` is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as -XDR (which can't represent pointer sharing); however it means that non-Python -programs may not be able to reconstruct pickled Python objects. +JSON or XDR (which can't represent pointer sharing); however it means that +non-Python programs may not be able to reconstruct pickled Python objects. -By default, the :mod:`pickle` data format uses a compact binary representation. +By default, the :mod:`pickle` data format uses a relatively compact binary +representation. If you need optimal size characteristics, you can efficiently +:doc:`compress ` pickled data. + The module :mod:`pickletools` contains tools for analyzing data streams -generated by :mod:`pickle`. +generated by :mod:`pickle`. :mod:`pickletools` source code has extensive +comments about opcodes used by pickle protocols. There are currently 4 different protocols which can be used for pickling. -* Protocol version 0 is the original human-readable protocol and is +* Protocol version 0 is the original "human-readable" protocol and is backwards compatible with earlier versions of Python. -* Protocol version 1 is the old binary format which is also compatible with +* Protocol version 1 is an old binary format which is also compatible with earlier versions of Python. * Protocol version 2 was introduced in Python 2.3. It provides much more - efficient pickling of :term:`new-style class`\es. + efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for + information about improvements brought by protocol 2. -* Protocol version 3 was added in Python 3.0. It has explicit support for - bytes and cannot be unpickled by Python 2.x pickle modules. This is - the current recommended protocol, use it whenever it is possible. - -Refer to :pep:`307` for information about improvements brought by -protocol 2. See :mod:`pickletools`'s source code for extensive -comments about opcodes used by pickle protocols. +* Protocol version 3 was added in Python 3. It has explicit support for + :class:`bytes` objects and cannot be unpickled by Python 2.x. This is + the default as well as the current recommended protocol; use it whenever + possible. Module Interface ---------------- -To serialize an object hierarchy, you first create a pickler, then you call the -pickler's :meth:`dump` method. To de-serialize a data stream, you first create -an unpickler, then you call the unpickler's :meth:`load` method. The -:mod:`pickle` module provides the following constant: +To serialize an object hierarchy, you simply call the :func:`dumps` function. +Similarly, to de-serialize a data stream, you call the :func:`loads` function. +However, if you want more control over serialization and de-serialization, +you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively. + +The :mod:`pickle` module provides the following constants: .. data:: HIGHEST_PROTOCOL @@ -134,8 +137,7 @@ .. data:: DEFAULT_PROTOCOL The default protocol used for pickling. May be less than HIGHEST_PROTOCOL. - Currently the default protocol is 3; a backward-incompatible protocol - designed for Python 3.0. + Currently the default protocol is 3, a new protocol designed for Python 3.0. The :mod:`pickle` module provides the following functions to make the pickling -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 15:42:18 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 15:42:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Some_nits_in_the_pickle_docs=2E?= Message-ID: http://hg.python.org/cpython/rev/b88e424fd993 changeset: 76861:b88e424fd993 parent: 76859:016ba8918d2e parent: 76860:1b5429e835e0 user: Antoine Pitrou date: Thu May 10 15:38:47 2012 +0200 summary: Some nits in the pickle docs. files: Doc/library/pickle.rst | 44 +++++++++++++++-------------- 1 files changed, 23 insertions(+), 21 deletions(-) diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -85,45 +85,48 @@ ------------------ .. index:: - single: XDR single: External Data Representation The data format used by :mod:`pickle` is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as -XDR (which can't represent pointer sharing); however it means that non-Python -programs may not be able to reconstruct pickled Python objects. +JSON or XDR (which can't represent pointer sharing); however it means that +non-Python programs may not be able to reconstruct pickled Python objects. -By default, the :mod:`pickle` data format uses a compact binary representation. +By default, the :mod:`pickle` data format uses a relatively compact binary +representation. If you need optimal size characteristics, you can efficiently +:doc:`compress ` pickled data. + The module :mod:`pickletools` contains tools for analyzing data streams -generated by :mod:`pickle`. +generated by :mod:`pickle`. :mod:`pickletools` source code has extensive +comments about opcodes used by pickle protocols. There are currently 4 different protocols which can be used for pickling. -* Protocol version 0 is the original human-readable protocol and is +* Protocol version 0 is the original "human-readable" protocol and is backwards compatible with earlier versions of Python. -* Protocol version 1 is the old binary format which is also compatible with +* Protocol version 1 is an old binary format which is also compatible with earlier versions of Python. * Protocol version 2 was introduced in Python 2.3. It provides much more - efficient pickling of :term:`new-style class`\es. + efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for + information about improvements brought by protocol 2. -* Protocol version 3 was added in Python 3.0. It has explicit support for - bytes and cannot be unpickled by Python 2.x pickle modules. This is - the current recommended protocol, use it whenever it is possible. - -Refer to :pep:`307` for information about improvements brought by -protocol 2. See :mod:`pickletools`'s source code for extensive -comments about opcodes used by pickle protocols. +* Protocol version 3 was added in Python 3. It has explicit support for + :class:`bytes` objects and cannot be unpickled by Python 2.x. This is + the default as well as the current recommended protocol; use it whenever + possible. Module Interface ---------------- -To serialize an object hierarchy, you first create a pickler, then you call the -pickler's :meth:`dump` method. To de-serialize a data stream, you first create -an unpickler, then you call the unpickler's :meth:`load` method. The -:mod:`pickle` module provides the following constant: +To serialize an object hierarchy, you simply call the :func:`dumps` function. +Similarly, to de-serialize a data stream, you call the :func:`loads` function. +However, if you want more control over serialization and de-serialization, +you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively. + +The :mod:`pickle` module provides the following constants: .. data:: HIGHEST_PROTOCOL @@ -134,8 +137,7 @@ .. data:: DEFAULT_PROTOCOL The default protocol used for pickling. May be less than HIGHEST_PROTOCOL. - Currently the default protocol is 3; a backward-incompatible protocol - designed for Python 3.0. + Currently the default protocol is 3, a new protocol designed for Python 3.0. The :mod:`pickle` module provides the following functions to make the pickling -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 16:38:10 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 16:38:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314738=3A_Speed-up_?= =?utf8?q?UTF-8_decoding_on_non-ASCII_data=2E__Patch_by_Serhiy?= Message-ID: http://hg.python.org/cpython/rev/e08c3791f035 changeset: 76862:e08c3791f035 user: Antoine Pitrou date: Thu May 10 16:36:02 2012 +0200 summary: Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. files: Misc/NEWS | 3 + Objects/stringlib/asciilib.h | 1 + Objects/stringlib/codecs.h | 239 +++++-- Objects/stringlib/ucs1lib.h | 1 + Objects/stringlib/ucs2lib.h | 1 + Objects/stringlib/ucs4lib.h | 1 + Objects/stringlib/undef.h | 1 + Objects/unicodeobject.c | 687 ++++++---------------- 8 files changed, 349 insertions(+), 585 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy + Storchaka. + - Issue #14700: Fix two broken and undefined-behaviour-inducing overflow checks in old-style string formatting. diff --git a/Objects/stringlib/asciilib.h b/Objects/stringlib/asciilib.h --- a/Objects/stringlib/asciilib.h +++ b/Objects/stringlib/asciilib.h @@ -7,6 +7,7 @@ #define STRINGLIB(F) asciilib_##F #define STRINGLIB_OBJECT PyUnicodeObject #define STRINGLIB_SIZEOF_CHAR 1 +#define STRINGLIB_MAX_CHAR 0x7Fu #define STRINGLIB_CHAR Py_UCS1 #define STRINGLIB_TYPE_NAME "unicode" #define STRINGLIB_PARSE_CODE "U" diff --git a/Objects/stringlib/codecs.h b/Objects/stringlib/codecs.h --- a/Objects/stringlib/codecs.h +++ b/Objects/stringlib/codecs.h @@ -15,19 +15,18 @@ # error C 'long' size should be either 4 or 8! #endif -Py_LOCAL_INLINE(int) -STRINGLIB(utf8_try_decode)(const char *start, const char *end, - STRINGLIB_CHAR *dest, - const char **src_pos, Py_ssize_t *dest_index) +Py_LOCAL_INLINE(Py_UCS4) +STRINGLIB(utf8_decode)(const char **inptr, const char *end, + STRINGLIB_CHAR *dest, + Py_ssize_t *outpos) { - int ret; - Py_ssize_t n; - const char *s = start; + Py_UCS4 ch; + const char *s = *inptr; const char *aligned_end = (const char *) ((size_t) end & ~LONG_PTR_MASK); - STRINGLIB_CHAR *p = dest; + STRINGLIB_CHAR *p = dest + *outpos; while (s < end) { - Py_UCS4 ch = (unsigned char)*s; + ch = (unsigned char)*s; if (ch < 0x80) { /* Fast path for runs of ASCII characters. Given that common UTF-8 @@ -48,15 +47,33 @@ unsigned long value = *(unsigned long *) _s; if (value & ASCII_CHAR_MASK) break; - _p[0] = _s[0]; - _p[1] = _s[1]; - _p[2] = _s[2]; - _p[3] = _s[3]; -#if (SIZEOF_LONG == 8) - _p[4] = _s[4]; - _p[5] = _s[5]; - _p[6] = _s[6]; - _p[7] = _s[7]; +#ifdef BYTEORDER_IS_LITTLE_ENDIAN + _p[0] = (STRINGLIB_CHAR)(value & 0xFFu); + _p[1] = (STRINGLIB_CHAR)((value >> 8) & 0xFFu); + _p[2] = (STRINGLIB_CHAR)((value >> 16) & 0xFFu); + _p[3] = (STRINGLIB_CHAR)((value >> 24) & 0xFFu); +# if SIZEOF_LONG == 8 + _p[4] = (STRINGLIB_CHAR)((value >> 32) & 0xFFu); + _p[5] = (STRINGLIB_CHAR)((value >> 40) & 0xFFu); + _p[6] = (STRINGLIB_CHAR)((value >> 48) & 0xFFu); + _p[7] = (STRINGLIB_CHAR)((value >> 56) & 0xFFu); +# endif +#else +# if SIZEOF_LONG == 8 + _p[0] = (STRINGLIB_CHAR)((value >> 56) & 0xFFu); + _p[1] = (STRINGLIB_CHAR)((value >> 48) & 0xFFu); + _p[2] = (STRINGLIB_CHAR)((value >> 40) & 0xFFu); + _p[3] = (STRINGLIB_CHAR)((value >> 32) & 0xFFu); + _p[4] = (STRINGLIB_CHAR)((value >> 24) & 0xFFu); + _p[5] = (STRINGLIB_CHAR)((value >> 16) & 0xFFu); + _p[6] = (STRINGLIB_CHAR)((value >> 8) & 0xFFu); + _p[7] = (STRINGLIB_CHAR)(value & 0xFFu); +# else + _p[0] = (STRINGLIB_CHAR)((value >> 24) & 0xFFu); + _p[1] = (STRINGLIB_CHAR)((value >> 16) & 0xFFu); + _p[2] = (STRINGLIB_CHAR)((value >> 8) & 0xFFu); + _p[3] = (STRINGLIB_CHAR)(value & 0xFFu); +# endif #endif _s += SIZEOF_LONG; _p += SIZEOF_LONG; @@ -67,87 +84,135 @@ break; ch = (unsigned char)*s; } + if (ch < 0x80) { + s++; + *p++ = ch; + continue; + } } - if (ch < 0x80) { - s++; + if (ch < 0xC2) { + /* invalid sequence + \x80-\xBF -- continuation byte + \xC0-\xC1 -- fake 0000-007F */ + goto InvalidStart; + } + + if (ch < 0xE0) { + /* \xC2\x80-\xDF\xBF -- 0080-07FF */ + Py_UCS4 ch2; + if (end - s < 2) { + /* unexpected end of data: the caller will decide whether + it's an error or not */ + break; + } + ch2 = (unsigned char)s[1]; + if ((ch2 & 0xC0) != 0x80) + /* invalid continuation byte */ + goto InvalidContinuation; + ch = (ch << 6) + ch2 - + ((0xC0 << 6) + 0x80); + assert ((ch > 0x007F) && (ch <= 0x07FF)); + s += 2; + if (STRINGLIB_MAX_CHAR <= 0x007F || + (STRINGLIB_MAX_CHAR < 0x07FF && ch > STRINGLIB_MAX_CHAR)) + goto Overflow; *p++ = ch; continue; } - n = utf8_code_length[ch]; - - if (s + n > end) { - /* unexpected end of data: the caller will decide whether - it's an error or not */ - goto _error; + if (ch < 0xF0) { + /* \xE0\xA0\x80-\xEF\xBF\xBF -- 0800-FFFF */ + Py_UCS4 ch2, ch3; + if (end - s < 3) { + /* unexpected end of data: the caller will decide whether + it's an error or not */ + break; + } + ch2 = (unsigned char)s[1]; + ch3 = (unsigned char)s[2]; + if ((ch2 & 0xC0) != 0x80 || + (ch3 & 0xC0) != 0x80) { + /* invalid continuation byte */ + goto InvalidContinuation; + } + if (ch == 0xE0) { + if (ch2 < 0xA0) + /* invalid sequence + \xE0\x80\x80-\xE0\x9F\xBF -- fake 0000-0800 */ + goto InvalidContinuation; + } + else if (ch == 0xED && ch2 > 0x9F) { + /* Decoding UTF-8 sequences in range \xED\xA0\x80-\xED\xBF\xBF + will result in surrogates in range D800-DFFF. Surrogates are + not valid UTF-8 so they are rejected. + See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf + (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt */ + goto InvalidContinuation; + } + ch = (ch << 12) + (ch2 << 6) + ch3 - + ((0xE0 << 12) + (0x80 << 6) + 0x80); + assert ((ch > 0x07FF) && (ch <= 0xFFFF)); + s += 3; + if (STRINGLIB_MAX_CHAR <= 0x07FF || + (STRINGLIB_MAX_CHAR < 0xFFFF && ch > STRINGLIB_MAX_CHAR)) + goto Overflow; + *p++ = ch; + continue; } - switch (n) { - case 0: - /* invalid start byte */ - goto _error; - case 1: - /* internal error */ - goto _error; - case 2: - if ((s[1] & 0xc0) != 0x80) + if (ch < 0xF5) { + /* \xF0\x90\x80\x80-\xF4\x8F\xBF\xBF -- 10000-10FFFF */ + Py_UCS4 ch2, ch3, ch4; + if (end - s < 4) { + /* unexpected end of data: the caller will decide whether + it's an error or not */ + break; + } + ch2 = (unsigned char)s[1]; + ch3 = (unsigned char)s[2]; + ch4 = (unsigned char)s[3]; + if ((ch2 & 0xC0) != 0x80 || + (ch3 & 0xC0) != 0x80 || + (ch4 & 0xC0) != 0x80) { /* invalid continuation byte */ - goto _error; - ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f); - assert ((ch > 0x007F) && (ch <= 0x07FF)); - s += 2; + goto InvalidContinuation; + } + if (ch == 0xF0) { + if (ch2 < 0x90) + /* invalid sequence + \xF0\x80\x80\x80-\xF0\x80\xBF\xBF -- fake 0000-FFFF */ + goto InvalidContinuation; + } + else if (ch == 0xF4 && ch2 > 0x8F) { + /* invalid sequence + \xF4\x90\x80\80- -- 110000- overflow */ + goto InvalidContinuation; + } + ch = (ch << 18) + (ch2 << 12) + (ch3 << 6) + ch4 - + ((0xF0 << 18) + (0x80 << 12) + (0x80 << 6) + 0x80); + assert ((ch > 0xFFFF) && (ch <= 0x10FFFF)); + s += 4; + if (STRINGLIB_MAX_CHAR <= 0xFFFF || + (STRINGLIB_MAX_CHAR < 0x10FFFF && ch > STRINGLIB_MAX_CHAR)) + goto Overflow; *p++ = ch; - break; - - case 3: - /* Decoding UTF-8 sequences in range \xed\xa0\x80-\xed\xbf\xbf - will result in surrogates in range d800-dfff. Surrogates are - not valid UTF-8 so they are rejected. - See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf - (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt */ - if ((s[1] & 0xc0) != 0x80 || - (s[2] & 0xc0) != 0x80 || - ((unsigned char)s[0] == 0xE0 && - (unsigned char)s[1] < 0xA0) || - ((unsigned char)s[0] == 0xED && - (unsigned char)s[1] > 0x9F)) { - /* invalid continuation byte */ - goto _error; - } - ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f); - assert ((ch > 0x07FF) && (ch <= 0xFFFF)); - s += 3; - *p++ = ch; - break; - - case 4: - if ((s[1] & 0xc0) != 0x80 || - (s[2] & 0xc0) != 0x80 || - (s[3] & 0xc0) != 0x80 || - ((unsigned char)s[0] == 0xF0 && - (unsigned char)s[1] < 0x90) || - ((unsigned char)s[0] == 0xF4 && - (unsigned char)s[1] > 0x8F)) { - /* invalid continuation byte */ - goto _error; - } - ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) + - ((s[2] & 0x3f) << 6) + (s[3] & 0x3f); - assert ((ch > 0xFFFF) && (ch <= 0x10ffff)); - s += 4; - *p++ = ch; - break; + continue; } + goto InvalidStart; } - ret = 0; - goto _ok; -_error: - ret = -1; -_ok: - *src_pos = s; - *dest_index = p - dest; - return ret; + ch = 0; +Overflow: +Return: + *inptr = s; + *outpos = p - dest; + return ch; +InvalidStart: + ch = 1; + goto Return; +InvalidContinuation: + ch = 2; + goto Return; } #undef LONG_PTR_MASK diff --git a/Objects/stringlib/ucs1lib.h b/Objects/stringlib/ucs1lib.h --- a/Objects/stringlib/ucs1lib.h +++ b/Objects/stringlib/ucs1lib.h @@ -7,6 +7,7 @@ #define STRINGLIB(F) ucs1lib_##F #define STRINGLIB_OBJECT PyUnicodeObject #define STRINGLIB_SIZEOF_CHAR 1 +#define STRINGLIB_MAX_CHAR 0xFFu #define STRINGLIB_CHAR Py_UCS1 #define STRINGLIB_TYPE_NAME "unicode" #define STRINGLIB_PARSE_CODE "U" diff --git a/Objects/stringlib/ucs2lib.h b/Objects/stringlib/ucs2lib.h --- a/Objects/stringlib/ucs2lib.h +++ b/Objects/stringlib/ucs2lib.h @@ -7,6 +7,7 @@ #define STRINGLIB(F) ucs2lib_##F #define STRINGLIB_OBJECT PyUnicodeObject #define STRINGLIB_SIZEOF_CHAR 2 +#define STRINGLIB_MAX_CHAR 0xFFFFu #define STRINGLIB_CHAR Py_UCS2 #define STRINGLIB_TYPE_NAME "unicode" #define STRINGLIB_PARSE_CODE "U" diff --git a/Objects/stringlib/ucs4lib.h b/Objects/stringlib/ucs4lib.h --- a/Objects/stringlib/ucs4lib.h +++ b/Objects/stringlib/ucs4lib.h @@ -7,6 +7,7 @@ #define STRINGLIB(F) ucs4lib_##F #define STRINGLIB_OBJECT PyUnicodeObject #define STRINGLIB_SIZEOF_CHAR 4 +#define STRINGLIB_MAX_CHAR 0x10FFFFu #define STRINGLIB_CHAR Py_UCS4 #define STRINGLIB_TYPE_NAME "unicode" #define STRINGLIB_PARSE_CODE "U" diff --git a/Objects/stringlib/undef.h b/Objects/stringlib/undef.h --- a/Objects/stringlib/undef.h +++ b/Objects/stringlib/undef.h @@ -1,6 +1,7 @@ #undef FASTSEARCH #undef STRINGLIB #undef STRINGLIB_SIZEOF_CHAR +#undef STRINGLIB_MAX_CHAR #undef STRINGLIB_CHAR #undef STRINGLIB_STR #undef STRINGLIB_LEN diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -4615,28 +4615,6 @@ /* --- UTF-8 Codec -------------------------------------------------------- */ -static -char utf8_code_length[256] = { - /* Map UTF-8 encoded prefix byte to sequence length. Zero means - illegal prefix. See RFC 3629 for details */ - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 00-0F */ - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 70-7F */ - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 80-8F */ - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* B0-BF */ - 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, /* C0-C1 + C2-CF */ - 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, /* D0-DF */ - 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, /* E0-EF */ - 4, 4, 4, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 /* F0-F4 + F5-FF */ -}; - PyObject * PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, @@ -4645,6 +4623,10 @@ return PyUnicode_DecodeUTF8Stateful(s, size, errors, NULL); } +#include "stringlib/asciilib.h" +#include "stringlib/codecs.h" +#include "stringlib/undef.h" + #include "stringlib/ucs1lib.h" #include "stringlib/codecs.h" #include "stringlib/undef.h" @@ -4670,310 +4652,60 @@ # error C 'long' size should be either 4 or 8! #endif -/* Scans a UTF-8 string and returns the maximum character to be expected - and the size of the decoded unicode string. - - This function doesn't check for errors, these checks are performed in - PyUnicode_DecodeUTF8Stateful. - */ -static Py_UCS4 -utf8_scanner(const unsigned char *p, Py_ssize_t string_size, Py_ssize_t *unicode_size) -{ - Py_ssize_t char_count = 0; - const unsigned char *end = p + string_size; - const unsigned char *aligned_end = (const unsigned char *) ((size_t) end & ~LONG_PTR_MASK); - - assert(unicode_size != NULL); - - /* By having a cascade of independent loops which fallback onto each - other, we minimize the amount of work done in the average loop - iteration, and we also maximize the CPU's ability to predict - branches correctly (because a given condition will have always the - same boolean outcome except perhaps in the last iteration of the - corresponding loop). - In the general case this brings us rather close to decoding - performance pre-PEP 393, despite the two-pass decoding. - - Note that the pure ASCII loop is not duplicated once a non-ASCII - character has been encountered. It is actually a pessimization (by - a significant factor) to use this loop on text with many non-ASCII - characters, and it is important to avoid bad performance on valid - utf-8 data (invalid utf-8 being a different can of worms). - */ - - /* ASCII */ - for (; p < end; ++p) { - /* Only check value if it's not a ASCII char... */ - if (*p < 0x80) { - /* Fast path, see below in PyUnicode_DecodeUTF8Stateful for - an explanation. */ - if (!((size_t) p & LONG_PTR_MASK)) { - /* Help register allocation */ - register const unsigned char *_p = p; - while (_p < aligned_end) { - unsigned long value = *(unsigned long *) _p; - if (value & ASCII_CHAR_MASK) - break; - _p += SIZEOF_LONG; - char_count += SIZEOF_LONG; - } - p = _p; - if (p == end) +static Py_ssize_t +ascii_decode(const char *start, const char *end, Py_UCS1 *dest) +{ + const char *p = start; + const char *aligned_end = (const char *) ((size_t) end & ~LONG_PTR_MASK); + +#if SIZEOF_LONG <= SIZEOF_VOID_P + assert(!((size_t) dest & LONG_PTR_MASK)); + if (!((size_t) p & LONG_PTR_MASK)) { + /* Fast path, see in STRINGLIB(utf8_decode) for + an explanation. */ + /* Help register allocation */ + register const char *_p = p; + register Py_UCS1 * q = dest; + while (_p < aligned_end) { + unsigned long value = *(const unsigned long *) _p; + if (value & ASCII_CHAR_MASK) + break; + *((unsigned long *)q) = value; + _p += SIZEOF_LONG; + q += SIZEOF_LONG; + } + p = _p; + while (p < end) { + if ((unsigned char)*p & 0x80) + break; + *q++ = *p++; + } + return p - start; + } +#endif + while (p < end) { + /* Fast path, see in STRINGLIB(utf8_decode) in stringlib/codecs.h + for an explanation. */ + if (!((size_t) p & LONG_PTR_MASK)) { + /* Help register allocation */ + register const char *_p = p; + while (_p < aligned_end) { + unsigned long value = *(unsigned long *) _p; + if (value & ASCII_CHAR_MASK) break; - } - } - if (*p < 0x80) - ++char_count; - else - goto _ucs1loop; - } - *unicode_size = char_count; - return 127; - -_ucs1loop: - for (; p < end; ++p) { - if (*p < 0xc4) - char_count += ((*p & 0xc0) != 0x80); - else - goto _ucs2loop; - } - *unicode_size = char_count; - return 255; - -_ucs2loop: - for (; p < end; ++p) { - if (*p < 0xf0) - char_count += ((*p & 0xc0) != 0x80); - else - goto _ucs4loop; - } - *unicode_size = char_count; - return 65535; - -_ucs4loop: - for (; p < end; ++p) { - char_count += ((*p & 0xc0) != 0x80); - } - *unicode_size = char_count; - return 65537; -} - -/* Similar to PyUnicode_WRITE but may attempt to widen and resize the string - in case of errors. Implicit parameters: unicode, kind, data, onError. - Potential resizing overallocates, so the result needs to shrink at the end. -*/ -#define WRITE_MAYBE_FAIL(index, value) \ - do { \ - Py_ssize_t pos = index; \ - if (pos > PyUnicode_GET_LENGTH(unicode) && \ - unicode_resize(&unicode, pos + pos/8) < 0) \ - goto onError; \ - if (unicode_putchar(&unicode, &pos, value) < 0) \ - goto onError; \ - } while (0) - -static PyObject * -decode_utf8_errors(const char *starts, - Py_ssize_t size, - const char *errors, - Py_ssize_t *consumed, - const char *s, - PyObject *unicode, - Py_ssize_t i) -{ - int n; - int k; - Py_ssize_t startinpos; - Py_ssize_t endinpos; - const char *e = starts + size; - const char *aligned_end; - const char *errmsg = ""; - PyObject *errorHandler = NULL; - PyObject *exc = NULL; - - aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK); - - while (s < e) { - Py_UCS4 ch = (unsigned char)*s; - - if (ch < 0x80) { - /* Fast path for runs of ASCII characters. Given that common UTF-8 - input will consist of an overwhelming majority of ASCII - characters, we try to optimize for this case by checking - as many characters as a C 'long' can contain. - First, check if we can do an aligned read, as most CPUs have - a penalty for unaligned reads. - */ - if (!((size_t) s & LONG_PTR_MASK)) { - /* Help register allocation */ - register const char *_s = s; - register Py_ssize_t _i = i; - while (_s < aligned_end) { - /* Read a whole long at a time (either 4 or 8 bytes), - and do a fast unrolled copy if it only contains ASCII - characters. */ - unsigned long value = *(unsigned long *) _s; - if (value & ASCII_CHAR_MASK) - break; - WRITE_MAYBE_FAIL(_i+0, _s[0]); - WRITE_MAYBE_FAIL(_i+1, _s[1]); - WRITE_MAYBE_FAIL(_i+2, _s[2]); - WRITE_MAYBE_FAIL(_i+3, _s[3]); -#if (SIZEOF_LONG == 8) - WRITE_MAYBE_FAIL(_i+4, _s[4]); - WRITE_MAYBE_FAIL(_i+5, _s[5]); - WRITE_MAYBE_FAIL(_i+6, _s[6]); - WRITE_MAYBE_FAIL(_i+7, _s[7]); -#endif - _s += SIZEOF_LONG; - _i += SIZEOF_LONG; - } - s = _s; - i = _i; - if (s == e) - break; - ch = (unsigned char)*s; - } - } - - if (ch < 0x80) { - WRITE_MAYBE_FAIL(i++, ch); - s++; - continue; - } - - n = utf8_code_length[ch]; - - if (s + n > e) { - if (consumed) + _p += SIZEOF_LONG; + } + p = _p; + if (_p == end) break; - else { - errmsg = "unexpected end of data"; - startinpos = s-starts; - endinpos = startinpos+1; - for (k=1; (k < size-startinpos) && ((s[k]&0xC0) == 0x80); k++) - endinpos++; - goto utf8Error; - } - } - - switch (n) { - - case 0: - errmsg = "invalid start byte"; - startinpos = s-starts; - endinpos = startinpos+1; - goto utf8Error; - - case 1: - errmsg = "internal error"; - startinpos = s-starts; - endinpos = startinpos+1; - goto utf8Error; - - case 2: - if ((s[1] & 0xc0) != 0x80) { - errmsg = "invalid continuation byte"; - startinpos = s-starts; - endinpos = startinpos + 1; - goto utf8Error; - } - ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f); - assert ((ch > 0x007F) && (ch <= 0x07FF)); - WRITE_MAYBE_FAIL(i++, ch); + } + if ((unsigned char)*p & 0x80) break; - - case 3: - /* Decoding UTF-8 sequences in range \xed\xa0\x80-\xed\xbf\xbf - will result in surrogates in range d800-dfff. Surrogates are - not valid UTF-8 so they are rejected. - See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf - (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt */ - if ((s[1] & 0xc0) != 0x80 || - (s[2] & 0xc0) != 0x80 || - ((unsigned char)s[0] == 0xE0 && - (unsigned char)s[1] < 0xA0) || - ((unsigned char)s[0] == 0xED && - (unsigned char)s[1] > 0x9F)) { - errmsg = "invalid continuation byte"; - startinpos = s-starts; - endinpos = startinpos + 1; - - /* if s[1] first two bits are 1 and 0, then the invalid - continuation byte is s[2], so increment endinpos by 1, - if not, s[1] is invalid and endinpos doesn't need to - be incremented. */ - if ((s[1] & 0xC0) == 0x80) - endinpos++; - goto utf8Error; - } - ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f); - assert ((ch > 0x07FF) && (ch <= 0xFFFF)); - WRITE_MAYBE_FAIL(i++, ch); - break; - - case 4: - if ((s[1] & 0xc0) != 0x80 || - (s[2] & 0xc0) != 0x80 || - (s[3] & 0xc0) != 0x80 || - ((unsigned char)s[0] == 0xF0 && - (unsigned char)s[1] < 0x90) || - ((unsigned char)s[0] == 0xF4 && - (unsigned char)s[1] > 0x8F)) { - errmsg = "invalid continuation byte"; - startinpos = s-starts; - endinpos = startinpos + 1; - if ((s[1] & 0xC0) == 0x80) { - endinpos++; - if ((s[2] & 0xC0) == 0x80) - endinpos++; - } - goto utf8Error; - } - ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) + - ((s[2] & 0x3f) << 6) + (s[3] & 0x3f); - assert ((ch > 0xFFFF) && (ch <= MAX_UNICODE)); - - WRITE_MAYBE_FAIL(i++, ch); - break; - } - s += n; - continue; - - utf8Error: - if (unicode_decode_call_errorhandler( - errors, &errorHandler, - "utf-8", errmsg, - &starts, &e, &startinpos, &endinpos, &exc, &s, - &unicode, &i)) - goto onError; - /* Update data because unicode_decode_call_errorhandler might have - re-created or resized the unicode object. */ - aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK); - } - if (consumed) - *consumed = s-starts; - - /* Adjust length and ready string when it contained errors and - is of the old resizable kind. */ - if (unicode_resize(&unicode, i) < 0) - goto onError; - unicode_adjust_maxchar(&unicode); - if (unicode == NULL) - goto onError; - - Py_XDECREF(errorHandler); - Py_XDECREF(exc); - assert(_PyUnicode_CheckConsistency(unicode, 1)); - return unicode; - - onError: - Py_XDECREF(errorHandler); - Py_XDECREF(exc); - Py_XDECREF(unicode); - return NULL; -} -#undef WRITE_MAYBE_FAIL + ++p; + } + memcpy(dest, start, p - start); + return p - start; +} PyObject * PyUnicode_DecodeUTF8Stateful(const char *s, @@ -4981,15 +4713,16 @@ const char *errors, Py_ssize_t *consumed) { - Py_UCS4 maxchar = 0; - Py_ssize_t unicode_size; - int has_errors = 0; PyObject *unicode; - int kind; - void *data; const char *starts = s; - const char *e; - Py_ssize_t i; + const char *end = s + size; + Py_ssize_t outpos; + + Py_ssize_t startinpos; + Py_ssize_t endinpos; + const char *errmsg = ""; + PyObject *errorHandler = NULL; + PyObject *exc = NULL; if (size == 0) { if (consumed) @@ -4998,49 +4731,91 @@ return unicode_empty; } - maxchar = utf8_scanner((const unsigned char *)s, size, &unicode_size); - - /* When the string is ASCII only, just use memcpy and return. - unicode_size may be != size if there is an incomplete UTF-8 - sequence at the end of the ASCII block. */ - if (maxchar < 128 && size == unicode_size) { + /* ASCII is equivalent to the first 128 ordinals in Unicode. */ + if (size == 1 && (unsigned char)s[0] < 128) { if (consumed) - *consumed = size; - return unicode_fromascii((const unsigned char *)s, size); - } - - unicode = PyUnicode_New(unicode_size, maxchar); + *consumed = 1; + return get_latin1_char((unsigned char)s[0]); + } + + unicode = PyUnicode_New(size, 127); if (!unicode) return NULL; - kind = PyUnicode_KIND(unicode); - data = PyUnicode_DATA(unicode); - - /* Unpack UTF-8 encoded data */ - i = 0; - e = starts + size; - switch (kind) { - case PyUnicode_1BYTE_KIND: - has_errors = ucs1lib_utf8_try_decode(s, e, (Py_UCS1 *) data, &s, &i); - break; - case PyUnicode_2BYTE_KIND: - has_errors = ucs2lib_utf8_try_decode(s, e, (Py_UCS2 *) data, &s, &i); - break; - case PyUnicode_4BYTE_KIND: - has_errors = ucs4lib_utf8_try_decode(s, e, (Py_UCS4 *) data, &s, &i); - break; - } - if (!has_errors) { - /* Ensure the unicode size calculation was correct */ - assert(i == unicode_size); - assert(s == e); - if (consumed) - *consumed = size; - return unicode; - } - - /* In case of errors, maxchar and size computation might be incorrect; - code below refits and resizes as necessary. */ - return decode_utf8_errors(starts, size, errors, consumed, s, unicode, i); + + outpos = ascii_decode(s, end, PyUnicode_1BYTE_DATA(unicode)); + s += outpos; + while (s < end) { + Py_UCS4 ch; + int kind = PyUnicode_KIND(unicode); + if (kind == PyUnicode_1BYTE_KIND) { + if (PyUnicode_IS_ASCII(unicode)) + ch = asciilib_utf8_decode(&s, end, + PyUnicode_1BYTE_DATA(unicode), &outpos); + else + ch = ucs1lib_utf8_decode(&s, end, + PyUnicode_1BYTE_DATA(unicode), &outpos); + } else if (kind == PyUnicode_2BYTE_KIND) { + ch = ucs2lib_utf8_decode(&s, end, + PyUnicode_2BYTE_DATA(unicode), &outpos); + } else { + assert(kind == PyUnicode_4BYTE_KIND); + ch = ucs4lib_utf8_decode(&s, end, + PyUnicode_4BYTE_DATA(unicode), &outpos); + } + + switch (ch) { + case 0: + if (s == end || consumed) + goto End; + errmsg = "unexpected end of data"; + startinpos = s - starts; + endinpos = startinpos + 1; + while (endinpos < size && (starts[endinpos] & 0xC0) == 0x80) + endinpos++; + break; + case 1: + errmsg = "invalid start byte"; + startinpos = s - starts; + endinpos = startinpos + 1; + break; + case 2: + errmsg = "invalid continuation byte"; + startinpos = s - starts; + endinpos = startinpos + 1; + while (endinpos < size && (starts[endinpos] & 0xC0) == 0x80) + endinpos++; + break; + default: + if (unicode_putchar(&unicode, &outpos, ch) < 0) + goto onError; + continue; + } + + if (unicode_decode_call_errorhandler( + errors, &errorHandler, + "utf-8", errmsg, + &starts, &end, &startinpos, &endinpos, &exc, &s, + &unicode, &outpos)) + goto onError; + } + +End: + if (unicode_resize(&unicode, outpos) < 0) + goto onError; + + if (consumed) + *consumed = s - starts; + + Py_XDECREF(errorHandler); + Py_XDECREF(exc); + assert(_PyUnicode_CheckConsistency(unicode, 1)); + return unicode; + +onError: + Py_XDECREF(errorHandler); + Py_XDECREF(exc); + Py_XDECREF(unicode); + return NULL; } #ifdef __APPLE__ @@ -5051,9 +4826,9 @@ wchar_t* _Py_DecodeUTF8_surrogateescape(const char *s, Py_ssize_t size) { - int n; const char *e; - wchar_t *unicode, *p; + wchar_t *unicode; + Py_ssize_t outpos; /* Note: size will always be longer than the resulting Unicode character count */ @@ -5066,86 +4841,33 @@ return NULL; /* Unpack UTF-8 encoded data */ - p = unicode; e = s + size; + outpos = 0; while (s < e) { - Py_UCS4 ch = (unsigned char)*s; - - if (ch < 0x80) { - *p++ = (wchar_t)ch; - s++; - continue; - } - - n = utf8_code_length[ch]; - if (s + n > e) { - goto surrogateescape; - } - - switch (n) { - case 0: - case 1: - goto surrogateescape; - - case 2: - if ((s[1] & 0xc0) != 0x80) - goto surrogateescape; - ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f); - assert ((ch > 0x007F) && (ch <= 0x07FF)); - *p++ = (wchar_t)ch; - break; - - case 3: - /* Decoding UTF-8 sequences in range \xed\xa0\x80-\xed\xbf\xbf - will result in surrogates in range d800-dfff. Surrogates are - not valid UTF-8 so they are rejected. - See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf - (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt */ - if ((s[1] & 0xc0) != 0x80 || - (s[2] & 0xc0) != 0x80 || - ((unsigned char)s[0] == 0xE0 && - (unsigned char)s[1] < 0xA0) || - ((unsigned char)s[0] == 0xED && - (unsigned char)s[1] > 0x9F)) { - - goto surrogateescape; - } - ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f); - assert ((ch > 0x07FF) && (ch <= 0xFFFF)); - *p++ = (wchar_t)ch; - break; - - case 4: - if ((s[1] & 0xc0) != 0x80 || - (s[2] & 0xc0) != 0x80 || - (s[3] & 0xc0) != 0x80 || - ((unsigned char)s[0] == 0xF0 && - (unsigned char)s[1] < 0x90) || - ((unsigned char)s[0] == 0xF4 && - (unsigned char)s[1] > 0x8F)) { - goto surrogateescape; - } - ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) + - ((s[2] & 0x3f) << 6) + (s[3] & 0x3f); - assert ((ch > 0xFFFF) && (ch <= MAX_UNICODE)); - + Py_UCS4 ch; #if SIZEOF_WCHAR_T == 4 - *p++ = (wchar_t)ch; + ch = ucs4lib_utf8_decode(&s, e, (Py_UCS4 *)unicode, &outpos); #else + ch = ucs2lib_utf8_decode(&s, e, (Py_UCS2 *)unicode, &outpos); +#endif + if (ch > 0xFF) { +#if SIZEOF_WCHAR_T == 4 + assert(0); +#else + assert(Py_UNICODE_IS_SURROGATE(ch)); /* compute and append the two surrogates: */ - *p++ = (wchar_t)Py_UNICODE_HIGH_SURROGATE(ch); - *p++ = (wchar_t)Py_UNICODE_LOW_SURROGATE(ch); -#endif - break; - } - s += n; - continue; - - surrogateescape: - *p++ = 0xDC00 + ch; - s++; - } - *p = L'\0'; + unicode[outpos++] = (wchar_t)Py_UNICODE_HIGH_SURROGATE(ch); + unicode[outpos++] = (wchar_t)Py_UNICODE_LOW_SURROGATE(ch); +#endif + } + else { + if (!ch && s == e) + break; + /* surrogateescape */ + unicode[outpos++] = 0xDC00 + (unsigned char)*s++; + } + } + unicode[outpos] = L'\0'; return unicode; } @@ -6970,17 +6692,13 @@ const char *errors) { const char *starts = s; - PyObject *v; + PyObject *unicode; int kind; void *data; Py_ssize_t startinpos; Py_ssize_t endinpos; Py_ssize_t outpos; const char *e; - int has_error; - const unsigned char *p = (const unsigned char *)s; - const unsigned char *end = p + size; - const unsigned char *aligned_end = (const unsigned char *) ((size_t) end & ~LONG_PTR_MASK); PyObject *errorHandler = NULL; PyObject *exc = NULL; @@ -6993,45 +6711,18 @@ if (size == 1 && (unsigned char)s[0] < 128) return get_latin1_char((unsigned char)s[0]); - has_error = 0; - while (p < end && !has_error) { - /* Fast path, see below in PyUnicode_DecodeUTF8Stateful for - an explanation. */ - if (!((size_t) p & LONG_PTR_MASK)) { - /* Help register allocation */ - register const unsigned char *_p = p; - while (_p < aligned_end) { - unsigned long value = *(unsigned long *) _p; - if (value & ASCII_CHAR_MASK) { - has_error = 1; - break; - } - _p += SIZEOF_LONG; - } - if (_p == end) - break; - if (has_error) - break; - p = _p; - } - if (*p & 0x80) { - has_error = 1; - break; - } - else { - ++p; - } - } - if (!has_error) - return unicode_fromascii((const unsigned char *)s, size); - - v = PyUnicode_New(size, 127); - if (v == NULL) + unicode = PyUnicode_New(size, 127); + if (unicode == NULL) goto onError; - kind = PyUnicode_KIND(v); - data = PyUnicode_DATA(v); - outpos = 0; + e = s + size; + data = PyUnicode_1BYTE_DATA(unicode); + outpos = ascii_decode(s, e, (Py_UCS1 *)data); + if (outpos == size) + return unicode; + + s += outpos; + kind = PyUnicode_1BYTE_KIND; while (s < e) { register unsigned char c = (unsigned char)*s; if (c < 128) { @@ -7045,21 +6736,21 @@ errors, &errorHandler, "ascii", "ordinal not in range(128)", &starts, &e, &startinpos, &endinpos, &exc, &s, - &v, &outpos)) + &unicode, &outpos)) goto onError; - kind = PyUnicode_KIND(v); - data = PyUnicode_DATA(v); - } - } - if (unicode_resize(&v, outpos) < 0) + kind = PyUnicode_KIND(unicode); + data = PyUnicode_DATA(unicode); + } + } + if (unicode_resize(&unicode, outpos) < 0) goto onError; Py_XDECREF(errorHandler); Py_XDECREF(exc); - assert(_PyUnicode_CheckConsistency(v, 1)); - return v; + assert(_PyUnicode_CheckConsistency(unicode, 1)); + return unicode; onError: - Py_XDECREF(v); + Py_XDECREF(unicode); Py_XDECREF(errorHandler); Py_XDECREF(exc); return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 17:13:31 2012 From: python-checkins at python.org (richard.oudkerk) Date: Thu, 10 May 2012 17:13:31 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314753=3A_Make_mult?= =?utf8?q?iprocessing_treat_negative_timeouts_as_it_did_in_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/99ef4501205b changeset: 76863:99ef4501205b user: Richard Oudkerk date: Thu May 10 16:11:12 2012 +0100 summary: Issue #14753: Make multiprocessing treat negative timeouts as it did in 3.2 In Python 3.2 and earlier, Process.join() and Connection.poll() treated negative timeouts as zero timeouts. Earlier versions from the 3.3 line of development treat them as infinite timeouts. The patch reverts to the old behaviour. files: Doc/library/multiprocessing.rst | 7 ++ Lib/multiprocessing/connection.py | 7 +-- Lib/multiprocessing/forking.py | 9 +-- Lib/multiprocessing/util.py | 15 ----- Lib/test/test_multiprocessing.py | 48 +++++++++++------- Misc/NEWS | 3 + 6 files changed, 44 insertions(+), 45 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -928,6 +928,12 @@ .. note:: + The :meth:`acquire` and :meth:`wait` methods of each of these types + treat negative timeouts as zero timeouts. This differs from + :mod:`threading` where, since version 3.2, the equivalent + :meth:`acquire` methods treat negative timeouts as infinite + timeouts. + On Mac OS X, ``sem_timedwait`` is unsupported, so calling ``acquire()`` with a timeout will emulate that function's behavior using a sleeping loop. @@ -1899,6 +1905,7 @@ those objects in *object_list* which are ready. If *timeout* is a float then the call blocks for at most that many seconds. If *timeout* is ``None`` then it will block for an unlimited period. + A negative timeout is equivalent to a zero timeout. For both Unix and Windows, an object can appear in *object_list* if it is diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py --- a/Lib/multiprocessing/connection.py +++ b/Lib/multiprocessing/connection.py @@ -23,8 +23,7 @@ import _multiprocessing from multiprocessing import current_process, AuthenticationError, BufferTooShort -from multiprocessing.util import ( - get_temp_dir, Finalize, sub_debug, debug, _eintr_retry) +from multiprocessing.util import get_temp_dir, Finalize, sub_debug, debug from multiprocessing.forking import ForkingPickler try: import _winapi @@ -323,8 +322,6 @@ if (self._got_empty_message or _winapi.PeekNamedPipe(self._handle)[0] != 0): return True - if timeout < 0: - timeout = None return bool(wait([self], timeout)) def _get_more_data(self, ov, maxsize): @@ -402,8 +399,6 @@ return self._recv(size) def _poll(self, timeout): - if timeout < 0.0: - timeout = None r = wait([self._handle], timeout) return bool(r) diff --git a/Lib/multiprocessing/forking.py b/Lib/multiprocessing/forking.py --- a/Lib/multiprocessing/forking.py +++ b/Lib/multiprocessing/forking.py @@ -75,12 +75,9 @@ # if sys.platform != 'win32': - import select - exit = os._exit duplicate = os.dup close = os.close - _select = util._eintr_retry(select.select) # # We define a Popen class similar to the one from subprocess, but @@ -130,10 +127,10 @@ def wait(self, timeout=None): if self.returncode is None: if timeout is not None: - r = _select([self.sentinel], [], [], timeout)[0] - if not r: + from .connection import wait + if not wait([self.sentinel], timeout): return None - # This shouldn't block if select() returned successfully. + # This shouldn't block if wait() returned successfully. return self.poll(os.WNOHANG if timeout == 0.0 else 0) return self.returncode diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -295,18 +295,3 @@ register_after_fork(self, lambda obj : obj.__dict__.clear()) def __reduce__(self): return type(self), () - - -# -# Automatic retry after EINTR -# - -def _eintr_retry(func): - @functools.wraps(func) - def wrapped(*args, **kwargs): - while True: - try: - return func(*args, **kwargs) - except InterruptedError: - continue - return wrapped diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -83,23 +83,13 @@ 'HAVE_BROKEN_SEM_GETVALUE', False) WIN32 = (sys.platform == "win32") -if WIN32: - from _winapi import WaitForSingleObject, INFINITE, WAIT_OBJECT_0 - - def wait_for_handle(handle, timeout): - if timeout is None or timeout < 0.0: - timeout = INFINITE - else: - timeout = int(1000 * timeout) - return WaitForSingleObject(handle, timeout) == WAIT_OBJECT_0 -else: - from select import select - _select = util._eintr_retry(select) - - def wait_for_handle(handle, timeout): - if timeout is not None and timeout < 0.0: - timeout = None - return handle in _select([handle], [], [], timeout)[0] + +from multiprocessing.connection import wait + +def wait_for_handle(handle, timeout): + if timeout is not None and timeout < 0.0: + timeout = None + return wait([handle], timeout) try: MAXFD = os.sysconf("SC_OPEN_MAX") @@ -291,9 +281,18 @@ self.assertIn(p, self.active_children()) self.assertEqual(p.exitcode, None) + join = TimingWrapper(p.join) + + self.assertEqual(join(0), None) + self.assertTimingAlmostEqual(join.elapsed, 0.0) + self.assertEqual(p.is_alive(), True) + + self.assertEqual(join(-1), None) + self.assertTimingAlmostEqual(join.elapsed, 0.0) + self.assertEqual(p.is_alive(), True) + p.terminate() - join = TimingWrapper(p.join) self.assertEqual(join(), None) self.assertTimingAlmostEqual(join.elapsed, 0.0) @@ -1664,6 +1663,9 @@ self.assertEqual(poll(), False) self.assertTimingAlmostEqual(poll.elapsed, 0) + self.assertEqual(poll(-1), False) + self.assertTimingAlmostEqual(poll.elapsed, 0) + self.assertEqual(poll(TIMEOUT1), False) self.assertTimingAlmostEqual(poll.elapsed, TIMEOUT1) @@ -2785,6 +2787,16 @@ p.terminate() p.join() + def test_neg_timeout(self): + from multiprocessing.connection import wait + a, b = multiprocessing.Pipe() + t = time.time() + res = wait([a], timeout=-1) + t = time.time() - t + self.assertEqual(res, []) + self.assertLess(t, 1) + a.close() + b.close() # # Issue 14151: Test invalid family on invalid environment diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #14753: Make multiprocessing's handling of negative timeouts + the same as it was in Python 3.2. + - Issue #14583: Fix importlib bug when a package's __init__.py would first import one of its modules then raise an error. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 17:21:57 2012 From: python-checkins at python.org (georg.brandl) Date: Thu, 10 May 2012 17:21:57 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_3144=3A_update_from_Peter_?= =?utf8?q?Moody=2E?= Message-ID: http://hg.python.org/peps/rev/a1dc8b665b06 changeset: 4365:a1dc8b665b06 user: Georg Brandl date: Thu May 10 17:22:13 2012 +0200 summary: PEP 3144: update from Peter Moody. files: pep-3144.txt | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -55,6 +55,23 @@ * A few attributes were renamed to disambiguate their purpose as well. (eg. network, network_address) + * A number of methods and functions which returned containers in ipaddr now + return iterators. This includes, subnets, address_exclude, + summarize_address_range and collapse_address_list. + + + Due to the backwards incompatible API changes between ipaddress and ipaddr, + the proposal is to add the module using the new provisional API status: + + * http://docs.python.org/dev/glossary.html#term-provisional-package + + + Relevant messages on python-dev: + + * http://mail.python.org/pipermail/python-dev/2012-January/116016.html + * http://mail.python.org/pipermail/python-dev/2012-February/116656.html + * http://mail.python.org/pipermail/python-dev/2012-February/116688.html + Specification: @@ -115,6 +132,9 @@ Reference Implementation: The current reference implementation can be found at: + https://code.google.com/p/ipaddr-py/source/browse/branches/3144/ipaddress.py + + Or see the tarball to include the README and unittests. http://code.google.com/p/ipaddr-py/downloads/detail?name=3144.tar.gz More information about using the reference implementation can be -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 10 20:20:56 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 20:20:56 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0MTU3?= =?utf8?q?=3A_Fix_time=2Estrptime_failing_without_a_year_on_February_29th?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/f2ea7505c0d7 changeset: 76864:f2ea7505c0d7 branch: 3.2 parent: 76860:1b5429e835e0 user: Antoine Pitrou date: Thu May 10 20:17:46 2012 +0200 summary: Issue #14157: Fix time.strptime failing without a year on February 29th. Patch by Hynek Schlawack. files: Lib/_strptime.py | 6 +++++- Lib/test/test_strptime.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -339,7 +339,7 @@ raise ValueError("unconverted data remains: %s" % data_string[found.end():]) - year = 1900 + year = None month = day = 1 hour = minute = second = fraction = 0 tz = -1 @@ -444,6 +444,10 @@ else: tz = value break + if year is None and month == 2 and day == 29: + year = 1904 # 1904 is first leap year of 20th century + elif year is None: + year = 1900 # If we know the week of the year and what day of that week, we can figure # out the Julian day of the year. if julian == -1 and week_of_year != -1 and weekday != -1: diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -378,6 +378,9 @@ need_escaping = ".^$*+?{}\[]|)(" self.assertTrue(_strptime._strptime_time(need_escaping, need_escaping)) + def test_feb29_on_leap_year_without_year(self): + time.strptime("Feb 29", "%b %d") + class Strptime12AMPMTests(unittest.TestCase): """Test a _strptime regression in '%I %p' at 12 noon (12 PM)""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ Library ------- +- Issue #14157: Fix time.strptime failing without a year on February 29th. + Patch by Hynek Schlawack. + - Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. - Issue #14741: Fix missing support for Ellipsis ('...') in parser module. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 20:20:57 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 20:20:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314157=3A_Fix_time=2Estrptime_failing_without_a_year?= =?utf8?q?_on_February_29th=2E?= Message-ID: http://hg.python.org/cpython/rev/a5a254e8a291 changeset: 76865:a5a254e8a291 parent: 76863:99ef4501205b parent: 76864:f2ea7505c0d7 user: Antoine Pitrou date: Thu May 10 20:18:46 2012 +0200 summary: Issue #14157: Fix time.strptime failing without a year on February 29th. Patch by Hynek Schlawack. files: Lib/_strptime.py | 6 +++++- Lib/test/test_strptime.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -339,7 +339,7 @@ raise ValueError("unconverted data remains: %s" % data_string[found.end():]) - year = 1900 + year = None month = day = 1 hour = minute = second = fraction = 0 tz = -1 @@ -444,6 +444,10 @@ else: tz = value break + if year is None and month == 2 and day == 29: + year = 1904 # 1904 is first leap year of 20th century + elif year is None: + year = 1900 # If we know the week of the year and what day of that week, we can figure # out the Julian day of the year. if julian == -1 and week_of_year != -1 and weekday != -1: diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -378,6 +378,9 @@ need_escaping = ".^$*+?{}\[]|)(" self.assertTrue(_strptime._strptime_time(need_escaping, need_escaping)) + def test_feb29_on_leap_year_without_year(self): + time.strptime("Feb 29", "%b %d") + class Strptime12AMPMTests(unittest.TestCase): """Test a _strptime regression in '%I %p' at 12 noon (12 PM)""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #14157: Fix time.strptime failing without a year on February 29th. + Patch by Hynek Schlawack. + - Issue #14753: Make multiprocessing's handling of negative timeouts the same as it was in Python 3.2. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 20:23:03 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 20:23:03 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0MTU3?= =?utf8?q?=3A_Fix_time=2Estrptime_failing_without_a_year_on_February_29th?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/69d407b016c1 changeset: 76866:69d407b016c1 branch: 2.7 parent: 76855:0415ecd7b0e3 user: Antoine Pitrou date: Thu May 10 20:17:46 2012 +0200 summary: Issue #14157: Fix time.strptime failing without a year on February 29th. Patch by Hynek Schlawack. files: Lib/_strptime.py | 7 ++++++- Lib/test/test_strptime.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -326,7 +326,8 @@ if len(data_string) != found.end(): raise ValueError("unconverted data remains: %s" % data_string[found.end():]) - year = 1900 + + year = None month = day = 1 hour = minute = second = fraction = 0 tz = -1 @@ -425,6 +426,10 @@ else: tz = value break + if year is None and month == 2 and day == 29: + year = 1904 # 1904 is first leap year of 20th century + elif year is None: + year = 1900 # If we know the week of the year and what day of that week, we can figure # out the Julian day of the year. if julian == -1 and week_of_year != -1 and weekday != -1: diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -378,6 +378,9 @@ need_escaping = ".^$*+?{}\[]|)(" self.assertTrue(_strptime._strptime_time(need_escaping, need_escaping)) + def test_feb29_on_leap_year_without_year(self): + time.strptime("Feb 29", "%b %d") + class Strptime12AMPMTests(unittest.TestCase): """Test a _strptime regression in '%I %p' at 12 noon (12 PM)""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #14157: Fix time.strptime failing without a year on February 29th. + Patch by Hynek Schlawack. + - Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. - Issue #13183: Fix pdb skipping frames after hitting a breakpoint and running -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 20:30:39 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 10 May 2012 20:30:39 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogVXBkYXRlIEFDS1Mu?= Message-ID: http://hg.python.org/cpython/rev/63bde882e311 changeset: 76867:63bde882e311 branch: 2.7 user: Antoine Pitrou date: Thu May 10 20:28:01 2012 +0200 summary: Update ACKS. files: Misc/ACKS | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -756,6 +756,7 @@ Michael Scharf Neil Schemenauer David Scherer +Hynek Schlawack Gregor Schmid Ralf Schmitt Michael Schneider -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 10 23:17:47 2012 From: python-checkins at python.org (benjamin.peterson) Date: Thu, 10 May 2012 23:17:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_use_yield_from?= Message-ID: http://hg.python.org/cpython/rev/7c2df4af83dd changeset: 76868:7c2df4af83dd parent: 76865:a5a254e8a291 user: Benjamin Peterson date: Thu May 10 16:17:35 2012 -0500 summary: use yield from files: Lib/os.py | 9 +++------ 1 files changed, 3 insertions(+), 6 deletions(-) diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -294,8 +294,7 @@ for name in dirs: new_path = join(top, name) if followlinks or not islink(new_path): - for x in walk(new_path, topdown, onerror, followlinks): - yield x + yield from walk(new_path, topdown, onerror, followlinks) if not topdown: yield top, dirs, nondirs @@ -339,8 +338,7 @@ try: if (followlinks or (st.S_ISDIR(orig_st.st_mode) and path.samestat(orig_st, fstat(topfd)))): - for x in _fwalk(topfd, top, topdown, onerror, followlinks): - yield x + yield from _fwalk(topfd, top, topdown, onerror, followlinks) finally: close(topfd) @@ -377,8 +375,7 @@ try: if followlinks or path.samestat(orig_st, fstat(dirfd)): dirpath = path.join(toppath, name) - for x in _fwalk(dirfd, dirpath, topdown, onerror, followlinks): - yield x + yield from _fwalk(dirfd, dirpath, topdown, onerror, followlinks) finally: close(dirfd) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 03:04:44 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 11 May 2012 03:04:44 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Simplified_find=5Floader_retur?= =?utf8?q?n_values=2E?= Message-ID: http://hg.python.org/peps/rev/29a064eab994 changeset: 4366:29a064eab994 parent: 4364:342624df7c93 user: Eric V. Smith date: Thu May 10 21:01:01 2012 -0400 summary: Simplified find_loader return values. files: pep-0420.txt | 29 ++++++++++++----------------- 1 files changed, 12 insertions(+), 17 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -168,29 +168,23 @@ For a finder to contribute to namespace packages, it must implement a new ``find_loader(fullname)`` method. ``fullname`` has the same -meaning as for ``find_module``. ``find_loader`` has 3 possible return -values, depending on whether it found a normal package, a namespace -package portion, or neither. In all cases the return value is a -2-tuple. +meaning as for ``find_module``. ``find_loader`` always returns a +2-tuple of ``(loader, )``. ``loader`` may +be None, in which case ```` (which may be +empty) is added to the list of recorded path entries and path +searching continues. If ``loader`` is not None, it is immediately +used to load the regular package. -===== ======================= ============ -Case Meaning Return value ------ ----------------------- ------------ -1 normal package found ``(loader, None)`` -2 portion(s) found ``(None, )`` -3 neither found ``(None, None)`` -===== ======================= ============ - -Note that in case 2 a sequence of strings is returned. This is to +Note that multiple path entries per finder are allowed. This is to support the case where a finder discovers multiple namespace portions for a given ``fullname``. Many finders will support only a single namespace package portion per ``find_loader`` call, in which case this -will normally be a list containing only a single string. +iterable will be a list containing only a single string. -Legacy finders which implement ``find_module`` but not ``find_loader`` -will be unable to contribute portions to a namespace package. The -import machinery will call ``find_loader`` if it exists, else fall -back to ``find_module``. +The import machinery will call ``find_loader`` if it exists, else fall +back to ``find_module``. Legacy finders which implement +``find_module`` but not ``find_loader`` will be unable to contribute +portions to a namespace package. The specification expands PEP 302 loaders to include an optional method called ``module_repr()`` which if present, is used to generate module object reprs. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 11 03:04:45 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 11 May 2012 03:04:45 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merge=2E?= Message-ID: http://hg.python.org/peps/rev/a4abaabbc801 changeset: 4367:a4abaabbc801 parent: 4366:29a064eab994 parent: 4365:a1dc8b665b06 user: Eric V. Smith date: Thu May 10 21:04:39 2012 -0400 summary: Merge. files: pep-3144.txt | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -55,6 +55,23 @@ * A few attributes were renamed to disambiguate their purpose as well. (eg. network, network_address) + * A number of methods and functions which returned containers in ipaddr now + return iterators. This includes, subnets, address_exclude, + summarize_address_range and collapse_address_list. + + + Due to the backwards incompatible API changes between ipaddress and ipaddr, + the proposal is to add the module using the new provisional API status: + + * http://docs.python.org/dev/glossary.html#term-provisional-package + + + Relevant messages on python-dev: + + * http://mail.python.org/pipermail/python-dev/2012-January/116016.html + * http://mail.python.org/pipermail/python-dev/2012-February/116656.html + * http://mail.python.org/pipermail/python-dev/2012-February/116688.html + Specification: @@ -115,6 +132,9 @@ Reference Implementation: The current reference implementation can be found at: + https://code.google.com/p/ipaddr-py/source/browse/branches/3144/ipaddress.py + + Or see the tarball to include the README and unittests. http://code.google.com/p/ipaddr-py/downloads/detail?name=3144.tar.gz More information about using the reference implementation can be -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 11 03:14:32 2012 From: python-checkins at python.org (ned.deily) Date: Fri, 11 May 2012 03:14:32 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0NjYy?= =?utf8?q?=3A_Prevent_shutil_failures_on_OS_X_when_destination_does_not?= Message-ID: http://hg.python.org/cpython/rev/e12efebc3ba6 changeset: 76869:e12efebc3ba6 branch: 2.7 parent: 76867:63bde882e311 user: Ned Deily date: Thu May 10 17:45:49 2012 -0700 summary: Issue #14662: Prevent shutil failures on OS X when destination does not support chflag operations. (Patch by Hynek Schlawack) files: Lib/shutil.py | 6 +++- Lib/test/test_shutil.py | 30 +++++++++++++++++++++++++++++ Misc/NEWS | 3 ++ Modules/errnomodule.c | 3 ++ 4 files changed, 40 insertions(+), 2 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -102,8 +102,10 @@ try: os.chflags(dst, st.st_flags) except OSError, why: - if (not hasattr(errno, 'EOPNOTSUPP') or - why.errno != errno.EOPNOTSUPP): + for err in 'EOPNOTSUPP', 'ENOTSUP': + if hasattr(errno, err) and why.errno == getattr(errno, err): + break + else: raise def copy(src, dst): diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -7,6 +7,7 @@ import stat import os import os.path +import errno from os.path import splitdrive from distutils.spawn import find_executable, spawn from shutil import (_make_tarball, _make_zipfile, make_archive, @@ -339,6 +340,35 @@ shutil.rmtree(TESTFN, ignore_errors=True) shutil.rmtree(TESTFN2, ignore_errors=True) + @unittest.skipUnless(hasattr(os, 'chflags') and + hasattr(errno, 'EOPNOTSUPP') and + hasattr(errno, 'ENOTSUP'), + "requires os.chflags, EOPNOTSUPP & ENOTSUP") + def test_copystat_handles_harmless_chflags_errors(self): + tmpdir = self.mkdtemp() + file1 = os.path.join(tmpdir, 'file1') + file2 = os.path.join(tmpdir, 'file2') + self.write_file(file1, 'xxx') + self.write_file(file2, 'xxx') + + def make_chflags_raiser(err): + ex = OSError() + + def _chflags_raiser(path, flags): + ex.errno = err + raise ex + return _chflags_raiser + old_chflags = os.chflags + try: + for err in errno.EOPNOTSUPP, errno.ENOTSUP: + os.chflags = make_chflags_raiser(err) + shutil.copystat(file1, file2) + # assert others errors break it + os.chflags = make_chflags_raiser(errno.EOPNOTSUPP + errno.ENOTSUP) + self.assertRaises(OSError, shutil.copystat, file1, file2) + finally: + os.chflags = old_chflags + @unittest.skipUnless(zlib, "requires zlib") def test_make_tarball(self): # creating something to tar diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #14662: Prevent shutil failures on OS X when destination does not + support chflag operations. Patch by Hynek Schlawack. + - Issue #14157: Fix time.strptime failing without a year on February 29th. Patch by Hynek Schlawack. diff --git a/Modules/errnomodule.c b/Modules/errnomodule.c --- a/Modules/errnomodule.c +++ b/Modules/errnomodule.c @@ -783,6 +783,9 @@ #ifdef WSAN inscode(d, ds, de, "WSAN", WSAN, "Error WSAN"); #endif +#ifdef ENOTSUP + inscode(d, ds, de, "ENOTSUP", ENOTSUP, "Operation not supported"); +#endif Py_DECREF(de); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 03:14:33 2012 From: python-checkins at python.org (ned.deily) Date: Fri, 11 May 2012 03:14:33 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0NjYy?= =?utf8?q?=3A_Prevent_shutil_failures_on_OS_X_when_destination_does_not?= Message-ID: http://hg.python.org/cpython/rev/ae141eebcf96 changeset: 76870:ae141eebcf96 branch: 3.2 parent: 76864:f2ea7505c0d7 user: Ned Deily date: Thu May 10 17:21:23 2012 -0700 summary: Issue #14662: Prevent shutil failures on OS X when destination does not support chflag operations. (Patch by Hynek Schlawack) files: Lib/shutil.py | 6 +++- Lib/test/test_shutil.py | 30 +++++++++++++++++++++++++++++ Misc/NEWS | 3 ++ 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -118,8 +118,10 @@ try: os.chflags(dst, st.st_flags) except OSError as why: - if (not hasattr(errno, 'EOPNOTSUPP') or - why.errno != errno.EOPNOTSUPP): + for err in 'EOPNOTSUPP', 'ENOTSUP': + if hasattr(errno, err) and why.errno == getattr(errno, err): + break + else: raise def copy(src, dst): diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -8,6 +8,7 @@ import os import os.path import functools +import errno from test import support from test.support import TESTFN from os.path import splitdrive @@ -307,6 +308,35 @@ finally: shutil.rmtree(TESTFN, ignore_errors=True) + @unittest.skipUnless(hasattr(os, 'chflags') and + hasattr(errno, 'EOPNOTSUPP') and + hasattr(errno, 'ENOTSUP'), + "requires os.chflags, EOPNOTSUPP & ENOTSUP") + def test_copystat_handles_harmless_chflags_errors(self): + tmpdir = self.mkdtemp() + file1 = os.path.join(tmpdir, 'file1') + file2 = os.path.join(tmpdir, 'file2') + self.write_file(file1, 'xxx') + self.write_file(file2, 'xxx') + + def make_chflags_raiser(err): + ex = OSError() + + def _chflags_raiser(path, flags): + ex.errno = err + raise ex + return _chflags_raiser + old_chflags = os.chflags + try: + for err in errno.EOPNOTSUPP, errno.ENOTSUP: + os.chflags = make_chflags_raiser(err) + shutil.copystat(file1, file2) + # assert others errors break it + os.chflags = make_chflags_raiser(errno.EOPNOTSUPP + errno.ENOTSUP) + self.assertRaises(OSError, shutil.copystat, file1, file2) + finally: + os.chflags = old_chflags + @support.skip_unless_symlink def test_dont_copy_file_onto_symlink_to_itself(self): # bug 851123. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ Library ------- +- Issue #14662: Prevent shutil failures on OS X when destination does not + support chflag operations. Patch by Hynek Schlawack. + - Issue #14157: Fix time.strptime failing without a year on February 29th. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 03:14:34 2012 From: python-checkins at python.org (ned.deily) Date: Fri, 11 May 2012 03:14:34 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314662=3A_Prevent_s?= =?utf8?q?hutil_failures_on_OS_X_when_destination_does_not?= Message-ID: http://hg.python.org/cpython/rev/93599d5e0a23 changeset: 76871:93599d5e0a23 parent: 76868:7c2df4af83dd user: Ned Deily date: Thu May 10 17:05:19 2012 -0700 summary: Issue #14662: Prevent shutil failures on OS X when destination does not support chflag operations. (Patch by Hynek Schlawack) files: Lib/shutil.py | 6 ++++-- Lib/test/test_shutil.py | 29 +++++++++++++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 36 insertions(+), 2 deletions(-) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -160,8 +160,10 @@ try: chflags_func(dst, st.st_flags) except OSError as why: - if (not hasattr(errno, 'EOPNOTSUPP') or - why.errno != errno.EOPNOTSUPP): + for err in 'EOPNOTSUPP', 'ENOTSUP': + if hasattr(errno, err) and why.errno == getattr(errno, err): + break + else: raise def copy(src, dst, symlinks=False): diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -282,6 +282,35 @@ self.assertTrue(abs(os.stat(src).st_mtime - os.stat(dst).st_mtime) < 00000.1) + @unittest.skipUnless(hasattr(os, 'chflags') and + hasattr(errno, 'EOPNOTSUPP') and + hasattr(errno, 'ENOTSUP'), + "requires os.chflags, EOPNOTSUPP & ENOTSUP") + def test_copystat_handles_harmless_chflags_errors(self): + tmpdir = self.mkdtemp() + file1 = os.path.join(tmpdir, 'file1') + file2 = os.path.join(tmpdir, 'file2') + write_file(file1, 'xxx') + write_file(file2, 'xxx') + + def make_chflags_raiser(err): + ex = OSError() + + def _chflags_raiser(path, flags): + ex.errno = err + raise ex + return _chflags_raiser + old_chflags = os.chflags + try: + for err in errno.EOPNOTSUPP, errno.ENOTSUP: + os.chflags = make_chflags_raiser(err) + shutil.copystat(file1, file2) + # assert others errors break it + os.chflags = make_chflags_raiser(errno.EOPNOTSUPP + errno.ENOTSUP) + self.assertRaises(OSError, shutil.copystat, file1, file2) + finally: + os.chflags = old_chflags + @support.skip_unless_symlink def test_copy_symlinks(self): tmp_dir = self.mkdtemp() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #14662: Prevent shutil failures on OS X when destination does not + support chflag operations. Patch by Hynek Schlawack. + - Issue #14157: Fix time.strptime failing without a year on February 29th. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 03:14:38 2012 From: python-checkins at python.org (ned.deily) Date: Fri, 11 May 2012 03:14:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge?= Message-ID: http://hg.python.org/cpython/rev/4e9680570be8 changeset: 76872:4e9680570be8 parent: 76871:93599d5e0a23 parent: 76870:ae141eebcf96 user: Ned Deily date: Thu May 10 18:11:30 2012 -0700 summary: merge files: -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri May 11 05:50:28 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 11 May 2012 05:50:28 +0200 Subject: [Python-checkins] Daily reference leaks (4e9680570be8): sum=0 Message-ID: results for 4e9680570be8 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog4ANHjc', '-x'] From python-checkins at python.org Fri May 11 17:12:16 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 May 2012 17:12:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314764=3A_Update_im?= =?utf8?q?portlib=2Etest=2Ebenchmark_to_work_in_a_world_where?= Message-ID: http://hg.python.org/cpython/rev/e1d0535372d0 changeset: 76873:e1d0535372d0 user: Brett Cannon date: Fri May 11 11:12:00 2012 -0400 summary: Issue #14764: Update importlib.test.benchmark to work in a world where import machinery is no longer implicit. files: Lib/importlib/test/benchmark.py | 14 ++++++++++++++ 1 files changed, 14 insertions(+), 0 deletions(-) diff --git a/Lib/importlib/test/benchmark.py b/Lib/importlib/test/benchmark.py --- a/Lib/importlib/test/benchmark.py +++ b/Lib/importlib/test/benchmark.py @@ -9,6 +9,8 @@ import decimal import imp import importlib +import importlib._bootstrap +import importlib.machinery import json import os import py_compile @@ -68,6 +70,10 @@ # Clears out sys.modules and puts an entry at the front of sys.path. with source_util.create_modules(name) as mapping: assert not os.path.exists(imp.cache_from_source(mapping[name])) + sys.meta_path.append(importlib.machinery.PathFinder) + loader = (importlib.machinery.SourceFileLoader, + importlib._bootstrap._SOURCE_SUFFIXES, True) + sys.path_hooks.append(importlib.machinery.FileFinder.path_hook(loader)) for result in bench(name, lambda: sys.modules.pop(name), repeat=repeat, seconds=seconds): yield result @@ -102,6 +108,10 @@ assert not sys.dont_write_bytecode name = '__importlib_test_benchmark__' with source_util.create_modules(name) as mapping: + sys.meta_path.append(importlib.machinery.PathFinder) + loader = (importlib.machinery.SourceFileLoader, + importlib._bootstrap._SOURCE_SUFFIXES, True) + sys.path_hooks.append(importlib.machinery.FileFinder.path_hook(loader)) def cleanup(): sys.modules.pop(name) os.unlink(imp.cache_from_source(mapping[name])) @@ -133,6 +143,10 @@ """Source w/ bytecode: small""" name = '__importlib_test_benchmark__' with source_util.create_modules(name) as mapping: + sys.meta_path.append(importlib.machinery.PathFinder) + loader = (importlib.machinery.SourceFileLoader, + importlib._bootstrap._SOURCE_SUFFIXES, True) + sys.path_hooks.append(importlib.machinery.FileFinder.path_hook(loader)) py_compile.compile(mapping[name]) assert os.path.exists(imp.cache_from_source(mapping[name])) for result in bench(name, lambda: sys.modules.pop(name), repeat=repeat, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 18:46:53 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 11 May 2012 18:46:53 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Suggestions_by_Nick_on_import-?= =?utf8?q?sig=2E?= Message-ID: http://hg.python.org/peps/rev/60474bd2a1fe changeset: 4368:60474bd2a1fe user: Eric V. Smith date: Fri May 11 12:46:44 2012 -0400 summary: Suggestions by Nick on import-sig. files: pep-0420.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -173,13 +173,13 @@ be None, in which case ```` (which may be empty) is added to the list of recorded path entries and path searching continues. If ``loader`` is not None, it is immediately -used to load the regular package. +used to load a module or regular package. Note that multiple path entries per finder are allowed. This is to support the case where a finder discovers multiple namespace portions for a given ``fullname``. Many finders will support only a single namespace package portion per ``find_loader`` call, in which case this -iterable will be a list containing only a single string. +iterable will contain only a single string. The import machinery will call ``find_loader`` if it exists, else fall back to ``find_module``. Legacy finders which implement -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 11 18:57:04 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 11 May 2012 18:57:04 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Added_dynamic_path_calculation?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/peps/rev/3775ecd04d4d changeset: 4369:3775ecd04d4d user: Eric V. Smith date: Fri May 11 12:56:56 2012 -0400 summary: Added dynamic path calculation. files: pep-0420.txt | 33 ++++++++++++++++----------------- 1 files changed, 16 insertions(+), 17 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -140,12 +140,6 @@ * Does not have a ``__file__`` attribute. -There is no mechanism to automatically recompute the ``__path__`` if -``sys.path`` is altered after a namespace package has already been -created. However, existing namespace utilities (like -``pkgutil.extend_path``) can be used to update them explicitly if -desired. - Note that if "import foo" is executed and "foo" is found as a namespace package (using the above rules), then "foo" is immediately created as a package. The creation of the namespace package is not @@ -159,6 +153,20 @@ path separator (typically a slash or backslash, depending on the platform). +Dynamic path computation +------------------------ + +A namespace package's ``__path__`` will be recomputed if the value of +the parent path changes. In order for this feature to work, the parent +path must be modified in-place, not replaced with a new object. For +example, for top-level namespace packages, this will work:: + + sys.path.append('new-dir') + +But this will not:: + + sys.path = sys.path + ['new-dir'] + Impact on Import Finders and Loaders ------------------------------------ @@ -265,12 +273,6 @@ 4. This will also be addressed in PEP 395. -Phillip Eby asked about auto-updating of ``__path__``, instead of it -being a simple list [5]_. It is the intent of this PEP to get the -simplest possible solution working. It will be possible at a later -date to add such features. Several possible ways to do so were -discussed in the referenced email thread. - ``find_module`` versus ``find_loader`` -------------------------------------- @@ -288,7 +290,7 @@ implement the ``find_loader`` method, described above. The use case for supporting multiple portions per ``find_loader`` call -is given in [6]_. +is given in [5]_. Module reprs @@ -345,10 +347,7 @@ .. [4] Nick Coghlan's response to his initial objections (http://mail.python.org/pipermail/import-sig/2012-April/000464.html) -.. [5] Phillip Eby's question about auto-updating __path__ - (http://mail.python.org/pipermail/import-sig/2012-April/000468.html) - -.. [6] Use case for multiple portions per ``find_loader`` call +.. [5] Use case for multiple portions per ``find_loader`` call (http://mail.python.org/pipermail/import-sig/2012-May/000585.html) Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 11 18:59:01 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 May 2012 18:59:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313959=3A_Deprecate?= =?utf8?q?_imp=2Eget=5Fsuffixes=28=29_for_new_attributes_on?= Message-ID: http://hg.python.org/cpython/rev/b81ddaf0db47 changeset: 76874:b81ddaf0db47 user: Brett Cannon date: Fri May 11 12:58:42 2012 -0400 summary: Issue #13959: Deprecate imp.get_suffixes() for new attributes on importlib.machinery that provide the suffix details for import. The attributes were not put on imp so as to compartmentalize everything importlib needs for setting up imports in importlib.machinery. This also led to an indirect deprecation of inspect.getmoduleinfo() as it directly returned imp.get_suffix's returned tuple which no longer makes sense. files: Doc/library/imp.rst | 3 + Doc/library/importlib.rst | 37 ++++++++++ Doc/library/inspect.rst | 4 + Lib/ctypes/util.py | 4 +- Lib/idlelib/PathBrowser.py | 5 +- Lib/imp.py | 33 +++++--- Lib/importlib/_bootstrap.py | 24 ++++-- Lib/importlib/machinery.py | 6 + Lib/importlib/test/benchmark.py | 7 +- Lib/importlib/test/extension/util.py | 5 +- Lib/importlib/test/source/test_case_sensitivity.py | 11 +- Lib/importlib/test/source/test_finder.py | 20 ++-- Lib/importlib/test/source/test_path_hook.py | 6 +- Lib/inspect.py | 29 ++++--- Lib/modulefinder.py | 5 +- Lib/pydoc.py | 25 +++++- Misc/NEWS | 5 + 17 files changed, 160 insertions(+), 69 deletions(-) diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -30,6 +30,9 @@ :const:`PY_SOURCE`, :const:`PY_COMPILED`, or :const:`C_EXTENSION`, described below. + .. deprecated:: 3.3 + Use the constants defined on :mod:`importlib.machinery` instead. + .. function:: find_module(name[, path]) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -477,6 +477,43 @@ This module contains the various objects that help :keyword:`import` find and load modules. +.. attribute:: SOURCE_SUFFIXES + + A list of strings representing the recognized file suffixes for source + modules. + + .. versionadded:: 3.3 + +.. attribute:: DEBUG_BYTECODE_SUFFIXES + + A list of strings representing the file suffixes for non-optimized bytecode + modules. + + .. versionadded:: 3.3 + +.. attribute:: OPTIMIZED_BYTECODE_SUFFIXES + + A list of strings representing the file suffixes for optimized bytecode + modules. + + .. versionadded:: 3.3 + +.. attribute:: BYTECODE_SUFFIXES + + A list of strings representing the recognized file suffixes for bytecode + modules. Set to either :attr:`DEBUG_BYTECODE_SUFFIXES` or + :attr:`OPTIMIZED_BYTECODE_SUFFIXES` based on whether ``__debug__`` is true. + + .. versionadded:: 3.3 + +.. attribute:: EXTENSION_SUFFIXES + + A list of strings representing the the recognized file suffixes for + extension modules. + + .. versionadded:: 3.3 + + .. class:: BuiltinImporter An :term:`importer` for built-in modules. All known built-in modules are diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -190,6 +190,10 @@ compared to the constants defined in the :mod:`imp` module; see the documentation for that module for more information on module types. + .. deprecated:: 3.3 + You may check the file path's suffix against the supported suffixes + listed in :mod:`importlib.machinery` to infer the same information. + .. function:: getmodulename(path) diff --git a/Lib/ctypes/util.py b/Lib/ctypes/util.py --- a/Lib/ctypes/util.py +++ b/Lib/ctypes/util.py @@ -40,8 +40,8 @@ clibname = 'msvcr%d' % (version * 10) # If python was built with in debug mode - import imp - if imp.get_suffixes()[0][0] == '_d.pyd': + import importlib.machinery + if '_d.pyd' in importlib.machinery.EXTENSION_SUFFIXES: clibname += 'd' return clibname+'.dll' diff --git a/Lib/idlelib/PathBrowser.py b/Lib/idlelib/PathBrowser.py --- a/Lib/idlelib/PathBrowser.py +++ b/Lib/idlelib/PathBrowser.py @@ -1,6 +1,7 @@ import os import sys import imp +import importlib.machinery from idlelib.TreeWidget import TreeItem from idlelib.ClassBrowser import ClassBrowser, ModuleBrowserTreeItem @@ -70,7 +71,9 @@ def listmodules(self, allnames): modules = {} - suffixes = imp.get_suffixes() + suffixes = importlib.machinery.EXTENSION_SUFFIXES[:] + suffixes += importlib.machinery.SOURCE_SUFFIXES[:] + suffixes += importlib.machinery.BYTECODE_SUFFIXES[:] sorted = [] for suff, mode, flag in suffixes: i = -len(suff) diff --git a/Lib/imp.py b/Lib/imp.py --- a/Lib/imp.py +++ b/Lib/imp.py @@ -17,9 +17,11 @@ from importlib._bootstrap import cache_from_source from importlib import _bootstrap +from importlib import machinery import os import sys import tokenize +import warnings # XXX "deprecate" once find_module(), load_module(), and get_suffixes() are @@ -37,9 +39,12 @@ def get_suffixes(): + warnings.warn('imp.get_suffixes() is deprecated; use the constants ' + 'defined on importlib.machinery instead', + DeprecationWarning, 2) extensions = [(s, 'rb', C_EXTENSION) for s in extension_suffixes()] - source = [(s, 'U', PY_SOURCE) for s in _bootstrap._SOURCE_SUFFIXES] - bytecode = [(_bootstrap._BYTECODE_SUFFIX, 'rb', PY_COMPILED)] + source = [(s, 'U', PY_SOURCE) for s in machinery.SOURCE_SUFFIXES] + bytecode = [(s, 'rb', PY_COMPILED) for s in machinery.BYTECODE_SUFFIXES] return extensions + source + bytecode @@ -61,7 +66,7 @@ raise ValueError('expected only 2 dots in ' '{!r}'.format(pycache_filename)) base_filename = pycache_filename.partition('.')[0] - return os.path.join(head, base_filename + _bootstrap._SOURCE_SUFFIXES[0]) + return os.path.join(head, base_filename + machinery.SOURCE_SUFFIXES[0]) class NullImporter: @@ -126,7 +131,7 @@ # XXX deprecate def load_package(name, path): if os.path.isdir(path): - extensions = _bootstrap._SOURCE_SUFFIXES + [_bootstrap._BYTECODE_SUFFIX] + extensions = machinery.SOURCE_SUFFIXES[:] + [machinery.BYTECODE_SUFFIXES] for extension in extensions: path = os.path.join(path, '__init__'+extension) if os.path.exists(path): @@ -190,19 +195,21 @@ for entry in path: package_directory = os.path.join(entry, name) - for suffix in ['.py', _bootstrap._BYTECODE_SUFFIX]: + for suffix in ['.py', machinery.BYTECODE_SUFFIXES[0]]: package_file_name = '__init__' + suffix file_path = os.path.join(package_directory, package_file_name) if os.path.isfile(file_path): return None, package_directory, ('', '', PKG_DIRECTORY) - for suffix, mode, type_ in get_suffixes(): - file_name = name + suffix - file_path = os.path.join(entry, file_name) - if os.path.isfile(file_path): - break - else: - continue - break # Break out of outer loop when breaking out of inner loop. + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + for suffix, mode, type_ in get_suffixes(): + file_name = name + suffix + file_path = os.path.join(entry, file_name) + if os.path.isfile(file_path): + break + else: + continue + break # Break out of outer loop when breaking out of inner loop. else: raise ImportError('No module name {!r}'.format(name), name=name) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -163,11 +163,14 @@ _PYCACHE = '__pycache__' -_SOURCE_SUFFIXES = ['.py'] # _setup() adds .pyw as needed. +SOURCE_SUFFIXES = ['.py'] # _setup() adds .pyw as needed. -_DEBUG_BYTECODE_SUFFIX = '.pyc' -_OPT_BYTECODE_SUFFIX = '.pyo' -_BYTECODE_SUFFIX = _DEBUG_BYTECODE_SUFFIX if __debug__ else _OPT_BYTECODE_SUFFIX +DEBUG_BYTECODE_SUFFIXES = ['.pyc'] +OPTIMIZED_BYTECODE_SUFFIXES = ['.pyo'] +if __debug__: + BYTECODE_SUFFIXES = DEBUG_BYTECODE_SUFFIXES +else: + BYTECODE_SUFFIXES = OPTIMIZED_BYTECODE_SUFFIXES def cache_from_source(path, debug_override=None): """Given the path to a .py file, return the path to its .pyc/.pyo file. @@ -181,10 +184,13 @@ """ debug = __debug__ if debug_override is None else debug_override - suffix = _DEBUG_BYTECODE_SUFFIX if debug else _OPT_BYTECODE_SUFFIX + if debug: + suffixes = DEBUG_BYTECODE_SUFFIXES + else: + suffixes = OPTIMIZED_BYTECODE_SUFFIXES head, tail = _path_split(path) base_filename, sep, _ = tail.partition('.') - filename = ''.join([base_filename, sep, _TAG, suffix]) + filename = ''.join([base_filename, sep, _TAG, suffixes[0]]) return _path_join(head, _PYCACHE, filename) @@ -1147,15 +1153,15 @@ setattr(self_module, '_MAGIC_NUMBER', _imp_module.get_magic()) setattr(self_module, '_TAG', _imp.get_tag()) if builtin_os == 'nt': - _SOURCE_SUFFIXES.append('.pyw') + SOURCE_SUFFIXES.append('.pyw') def _install(sys_module, _imp_module): """Install importlib as the implementation of import.""" _setup(sys_module, _imp_module) extensions = ExtensionFileLoader, _imp_module.extension_suffixes(), False - source = SourceFileLoader, _SOURCE_SUFFIXES, True - bytecode = SourcelessFileLoader, [_BYTECODE_SUFFIX], True + source = SourceFileLoader, SOURCE_SUFFIXES, True + bytecode = SourcelessFileLoader, BYTECODE_SUFFIXES, True supported_loaders = [extensions, source, bytecode] sys.path_hooks.extend([FileFinder.path_hook(*supported_loaders)]) sys.meta_path.extend([BuiltinImporter, FrozenImporter, PathFinder]) diff --git a/Lib/importlib/machinery.py b/Lib/importlib/machinery.py --- a/Lib/importlib/machinery.py +++ b/Lib/importlib/machinery.py @@ -1,5 +1,9 @@ """The machinery of importlib: finders, loaders, hooks, etc.""" +import _imp + +from ._bootstrap import (SOURCE_SUFFIXES, DEBUG_BYTECODE_SUFFIXES, + OPTIMIZED_BYTECODE_SUFFIXES, BYTECODE_SUFFIXES) from ._bootstrap import BuiltinImporter from ._bootstrap import FrozenImporter from ._bootstrap import PathFinder @@ -7,3 +11,5 @@ from ._bootstrap import SourceFileLoader from ._bootstrap import SourcelessFileLoader from ._bootstrap import ExtensionFileLoader + +EXTENSION_SUFFIXES = _imp.extension_suffixes() diff --git a/Lib/importlib/test/benchmark.py b/Lib/importlib/test/benchmark.py --- a/Lib/importlib/test/benchmark.py +++ b/Lib/importlib/test/benchmark.py @@ -9,7 +9,6 @@ import decimal import imp import importlib -import importlib._bootstrap import importlib.machinery import json import os @@ -72,7 +71,7 @@ assert not os.path.exists(imp.cache_from_source(mapping[name])) sys.meta_path.append(importlib.machinery.PathFinder) loader = (importlib.machinery.SourceFileLoader, - importlib._bootstrap._SOURCE_SUFFIXES, True) + importlib.machinery.SOURCE_SUFFIXES, True) sys.path_hooks.append(importlib.machinery.FileFinder.path_hook(loader)) for result in bench(name, lambda: sys.modules.pop(name), repeat=repeat, seconds=seconds): @@ -110,7 +109,7 @@ with source_util.create_modules(name) as mapping: sys.meta_path.append(importlib.machinery.PathFinder) loader = (importlib.machinery.SourceFileLoader, - importlib._bootstrap._SOURCE_SUFFIXES, True) + importlib.machinery.SOURCE_SUFFIXES, True) sys.path_hooks.append(importlib.machinery.FileFinder.path_hook(loader)) def cleanup(): sys.modules.pop(name) @@ -145,7 +144,7 @@ with source_util.create_modules(name) as mapping: sys.meta_path.append(importlib.machinery.PathFinder) loader = (importlib.machinery.SourceFileLoader, - importlib._bootstrap._SOURCE_SUFFIXES, True) + importlib.machinery.SOURCE_SUFFIXES, True) sys.path_hooks.append(importlib.machinery.FileFinder.path_hook(loader)) py_compile.compile(mapping[name]) assert os.path.exists(imp.cache_from_source(mapping[name])) diff --git a/Lib/importlib/test/extension/util.py b/Lib/importlib/test/extension/util.py --- a/Lib/importlib/test/extension/util.py +++ b/Lib/importlib/test/extension/util.py @@ -1,4 +1,5 @@ import imp +from importlib import machinery import os import sys @@ -6,10 +7,9 @@ EXT = None FILENAME = None NAME = '_testcapi' -_file_exts = [x[0] for x in imp.get_suffixes() if x[2] == imp.C_EXTENSION] try: for PATH in sys.path: - for EXT in _file_exts: + for EXT in machinery.EXTENSION_SUFFIXES: FILENAME = NAME + EXT FILEPATH = os.path.join(PATH, FILENAME) if os.path.exists(os.path.join(PATH, FILENAME)): @@ -18,4 +18,3 @@ PATH = EXT = FILENAME = FILEPATH = None except StopIteration: pass -del _file_exts diff --git a/Lib/importlib/test/source/test_case_sensitivity.py b/Lib/importlib/test/source/test_case_sensitivity.py --- a/Lib/importlib/test/source/test_case_sensitivity.py +++ b/Lib/importlib/test/source/test_case_sensitivity.py @@ -1,5 +1,6 @@ """Test case-sensitivity (PEP 235).""" from importlib import _bootstrap +from importlib import machinery from .. import util from . import util as source_util import imp @@ -20,12 +21,12 @@ assert name != name.lower() def find(self, path): - finder = _bootstrap.FileFinder(path, - (_bootstrap.SourceFileLoader, - _bootstrap._SOURCE_SUFFIXES, + finder = machinery.FileFinder(path, + (machinery.SourceFileLoader, + machinery.SOURCE_SUFFIXES, True), - (_bootstrap.SourcelessFileLoader, - [_bootstrap._BYTECODE_SUFFIX], + (machinery.SourcelessFileLoader, + machinery.BYTECODE_SUFFIXES, True)) return finder.find_module(self.name) diff --git a/Lib/importlib/test/source/test_finder.py b/Lib/importlib/test/source/test_finder.py --- a/Lib/importlib/test/source/test_finder.py +++ b/Lib/importlib/test/source/test_finder.py @@ -1,7 +1,7 @@ from .. import abc from . import util as source_util -from importlib import _bootstrap +from importlib import machinery import errno import imp import os @@ -36,11 +36,11 @@ """ def import_(self, root, module): - loader_details = [(_bootstrap.SourceFileLoader, - _bootstrap._SOURCE_SUFFIXES, True), - (_bootstrap.SourcelessFileLoader, - [_bootstrap._BYTECODE_SUFFIX], True)] - finder = _bootstrap.FileFinder(root, *loader_details) + loader_details = [(machinery.SourceFileLoader, + machinery.SOURCE_SUFFIXES, True), + (machinery.SourcelessFileLoader, + machinery.BYTECODE_SUFFIXES, True)] + finder = machinery.FileFinder(root, *loader_details) return finder.find_module(module) def run_test(self, test, create=None, *, compile_=None, unlink=None): @@ -138,8 +138,8 @@ def test_empty_string_for_dir(self): # The empty string from sys.path means to search in the cwd. - finder = _bootstrap.FileFinder('', (_bootstrap.SourceFileLoader, - _bootstrap._SOURCE_SUFFIXES, True)) + finder = machinery.FileFinder('', (machinery.SourceFileLoader, + machinery.SOURCE_SUFFIXES, True)) with open('mod.py', 'w') as file: file.write("# test file for importlib") try: @@ -150,8 +150,8 @@ def test_invalidate_caches(self): # invalidate_caches() should reset the mtime. - finder = _bootstrap.FileFinder('', (_bootstrap.SourceFileLoader, - _bootstrap._SOURCE_SUFFIXES, True)) + finder = machinery.FileFinder('', (machinery.SourceFileLoader, + machinery.SOURCE_SUFFIXES, True)) finder._path_mtime = 42 finder.invalidate_caches() self.assertEqual(finder._path_mtime, -1) diff --git a/Lib/importlib/test/source/test_path_hook.py b/Lib/importlib/test/source/test_path_hook.py --- a/Lib/importlib/test/source/test_path_hook.py +++ b/Lib/importlib/test/source/test_path_hook.py @@ -1,6 +1,6 @@ from . import util as source_util -from importlib import _bootstrap +from importlib import machinery import imp import unittest @@ -10,8 +10,8 @@ """Test the path hook for source.""" def path_hook(self): - return _bootstrap.FileFinder.path_hook((_bootstrap.SourceFileLoader, - _bootstrap._SOURCE_SUFFIXES, True)) + return machinery.FileFinder.path_hook((machinery.SourceFileLoader, + machinery.SOURCE_SUFFIXES, True)) def test_success(self): with source_util.create_modules('dummy') as mapping: diff --git a/Lib/inspect.py b/Lib/inspect.py --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -29,14 +29,15 @@ __author__ = 'Ka-Ping Yee ' __date__ = '1 Jan 2001' +import imp +import importlib.machinery +import itertools +import linecache +import os +import re import sys -import os +import tokenize import types -import itertools -import re -import imp -import tokenize -import linecache from operator import attrgetter from collections import namedtuple @@ -432,6 +433,8 @@ def getmoduleinfo(path): """Get the module name, suffix, mode, and module type for a given file.""" + warnings.warn('inspect.getmoduleinfo() is deprecated', DeprecationWarning, + 2) filename = os.path.basename(path) suffixes = [(-len(suffix), suffix, mode, mtype) for suffix, mode, mtype in imp.get_suffixes()] @@ -450,12 +453,14 @@ Return None if no way can be identified to get the source. """ filename = getfile(object) - if filename[-4:].lower() in ('.pyc', '.pyo'): - filename = filename[:-4] + '.py' - for suffix, mode, kind in imp.get_suffixes(): - if 'b' in mode and filename[-len(suffix):].lower() == suffix: - # Looks like a binary file. We want to only return a text file. - return None + all_bytecode_suffixes = importlib.machinery.DEBUG_BYTECODE_SUFFIXES[:] + all_bytecode_suffixes += importlib.machinery.OPTIMIZED_BYTECODE_SUFFIXES[:] + if any(filename.endswith(s) for s in all_bytecode_suffixes): + filename = (os.path.splitext(filename)[0] + + importlib.machinery.SOURCE_SUFFIXES[0]) + elif any(filename.endswith(s) for s in + importlib.machinery.EXTENSION_SUFFIXES): + return None if os.path.exists(filename): return filename # only return a non-existent filename if the module has a PEP 302 loader diff --git a/Lib/modulefinder.py b/Lib/modulefinder.py --- a/Lib/modulefinder.py +++ b/Lib/modulefinder.py @@ -222,8 +222,9 @@ # But we must also collect Python extension modules - although # we cannot separate normal dlls from Python extensions. suffixes = [] - for triple in imp.get_suffixes(): - suffixes.append(triple[0]) + suffixes += importlib.machinery.EXTENSION_SUFFIXES[:] + suffixes += importlib.machinery.SOURCE_SUFFIXES[:] + suffixes += importlib.machinery.BYTECODE_SUFFIXES[:] for dir in m.__path__: try: names = os.listdir(dir) diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -53,6 +53,7 @@ import builtins import imp +import importlib.machinery import inspect import io import os @@ -220,20 +221,34 @@ mtime = os.stat(filename).st_mtime lastupdate, result = cache.get(filename, (None, None)) if lastupdate is None or lastupdate < mtime: - info = inspect.getmoduleinfo(filename) try: file = tokenize.open(filename) except IOError: # module can't be opened, so skip it return None - if info and 'b' in info[2]: # binary modules have to be imported - try: module = imp.load_module('__temp__', file, filename, info[1:]) - except: return None + binary_suffixes = importlib.machinery.BYTECODE_SUFFIXES[:] + binary_suffixes += importlib.machinery.EXTENSION_SUFFIXES[:] + if any(filename.endswith(x) for x in binary_suffixes): + # binary modules have to be imported + file.close() + if any(filename.endswith(x) for x in + importlib.machinery.BYTECODE_SUFFIXES): + loader = importlib.machinery.SourcelessFileLoader('__temp__', + filename) + else: + loader = importlib.machinery.ExtensionFileLoader('__temp__', + filename) + try: + module = loader.load_module('__temp__') + except: + return None result = (module.__doc__ or '').splitlines()[0] del sys.modules['__temp__'] - else: # text modules can be directly examined + else: + # text modules can be directly examined result = source_synopsis(file) file.close() + cache[filename] = (mtime, result) return result diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,11 @@ Library ------- +- Issue #13959: imp.get_suffixes() has been deprecated in favour of the new + attributes on importlib.machinery: SOURCE_SUFFIXES, DEBUG_BYTECODE_SUFFIXES, + OPTIMIZED_BYTECODE_SUFFIXES, BYTECODE_SUFFIXES, and EXTENSION_SUFFIXES. This + led to an indirect deprecation of inspect.getmoduleinfo(). + - Issue #14662: Prevent shutil failures on OS X when destination does not support chflag operations. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 19:11:07 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 May 2012 19:11:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_importlib=2Eh?= Message-ID: http://hg.python.org/cpython/rev/51913a3898dc changeset: 76875:51913a3898dc user: Brett Cannon date: Fri May 11 13:11:02 2012 -0400 summary: Update importlib.h files: Python/importlib.h | Bin 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/Python/importlib.h b/Python/importlib.h index c88fc8018149bd01cc54b8759f937df81e7016cc..598eba50044a197361df15d0c0aeb6f755d16b5b GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 20:48:48 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 May 2012 20:48:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Deprecate_the_imp_constants?= =?utf8?q?_related_to_imp=2Eget=5Fsuffixes=28=29=2E?= Message-ID: http://hg.python.org/cpython/rev/f11015632f6d changeset: 76876:f11015632f6d user: Brett Cannon date: Fri May 11 14:27:29 2012 -0400 summary: Deprecate the imp constants related to imp.get_suffixes(). files: Doc/library/imp.rst | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -236,31 +236,43 @@ The module was found as a source file. + .. deprecated:: 3.3 + .. data:: PY_COMPILED The module was found as a compiled code object file. + .. deprecated:: 3.3 + .. data:: C_EXTENSION The module was found as dynamically loadable shared library. + .. deprecated:: 3.3 + .. data:: PKG_DIRECTORY The module was found as a package directory. + .. deprecated:: 3.3 + .. data:: C_BUILTIN The module was found as a built-in module. + .. deprecated:: 3.3 + .. data:: PY_FROZEN The module was found as a frozen module. + .. deprecated:: 3.3 + .. class:: NullImporter(path_string) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 20:48:49 2012 From: python-checkins at python.org (brett.cannon) Date: Fri, 11 May 2012 20:48:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313959=3A_Have?= Message-ID: http://hg.python.org/cpython/rev/626d5c6fbd95 changeset: 76877:626d5c6fbd95 user: Brett Cannon date: Fri May 11 14:48:41 2012 -0400 summary: Issue #13959: Have importlib.abc.FileLoader.load_module()/get_filename() and importlib.machinery.ExtensionFileLoader.load_module() have their single argument be optional as the loader's constructor has all the ncessary information. This allows for the deprecation of imp.load_source()/load_compile()/load_package(). files: Doc/library/importlib.rst | 21 +-- Lib/imp.py | 60 ++++++---- Lib/importlib/_bootstrap.py | 14 +- Lib/importlib/test/extension/test_loader.py | 20 ++- Lib/importlib/test/source/test_file_loader.py | 35 +++++ Lib/test/test_imp.py | 19 ++- Lib/test/test_tools.py | 8 +- Misc/NEWS | 5 + Python/importlib.h | Bin 9 files changed, 127 insertions(+), 55 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -256,9 +256,14 @@ Path to the file of the module. + .. method:: load_module(fullname=None) + + Calls + ``super().load_module(fullname if fullname is not None else self.name)``. + .. method:: get_filename(fullname) - Returns :attr:`path`. + Returns :attr:`path` when ``fullname`` equals :attr:`name` or ``None``. .. method:: get_data(path) @@ -638,10 +643,6 @@ Concrete implementation of :meth:`importlib.abc.SourceLoader.set_data`. - .. method:: load_module(fullname) - - Load the specified module if it is the same as :attr:`name`. - .. class:: SourcelessFileLoader(fullname, path) @@ -676,10 +677,6 @@ Returns ``None`` as bytecode files have no source when this loader is used. - .. method:: load_module(fullname) - - Loads the specified module if it is the same as :attr:`name`. - .. class:: ExtensionFileLoader(fullname, path) @@ -699,10 +696,10 @@ Path to the extension module. - .. method:: load_module(fullname) + .. method:: load_module(fullname=None) - Loads the extension module if and only if *fullname** is the same as - :attr:`name`. + Loads the extension module if and only if *fullname* is the same as + :attr:`name` or is ``None``. .. method:: is_package(fullname) diff --git a/Lib/imp.py b/Lib/imp.py --- a/Lib/imp.py +++ b/Lib/imp.py @@ -24,8 +24,7 @@ import warnings -# XXX "deprecate" once find_module(), load_module(), and get_suffixes() are -# deprecated. +# DEPRECATED SEARCH_ERROR = 0 PY_SOURCE = 1 PY_COMPILED = 2 @@ -112,8 +111,11 @@ """Compatibility support for implementing load_source().""" -# XXX deprecate after better API exposed in importlib def load_source(name, pathname, file=None): + msg = ('imp.load_source() is deprecated; use ' + 'importlib.machinery.SourceFileLoader(name, pathname).load_module()' + ' instead') + warnings.warn(msg, DeprecationWarning, 2) return _LoadSourceCompatibility(name, pathname, file).load_module(name) @@ -123,15 +125,22 @@ """Compatibility support for implementing load_compiled().""" -# XXX deprecate def load_compiled(name, pathname, file=None): + msg = ('imp.load_compiled() is deprecated; use ' + 'importlib.machinery.SourcelessFileLoader(name, pathname).' + 'load_module() instead ') + warnings.warn(msg, DeprecationWarning, 2) return _LoadCompiledCompatibility(name, pathname, file).load_module(name) -# XXX deprecate def load_package(name, path): + msg = ('imp.load_package() is deprecated; use either ' + 'importlib.machinery.SourceFileLoader() or ' + 'importlib.machinery.SourcelessFileLoader() instead') + warnings.warn(msg, DeprecationWarning, 2) if os.path.isdir(path): - extensions = machinery.SOURCE_SUFFIXES[:] + [machinery.BYTECODE_SUFFIXES] + extensions = (machinery.SOURCE_SUFFIXES[:] + + machinery.BYTECODE_SUFFIXES[:]) for extension in extensions: path = os.path.join(path, '__init__'+extension) if os.path.exists(path): @@ -149,26 +158,29 @@ """ suffix, mode, type_ = details - if mode and (not mode.startswith(('r', 'U')) or '+' in mode): - raise ValueError('invalid file open mode {!r}'.format(mode)) - elif file is None and type_ in {PY_SOURCE, PY_COMPILED}: - msg = 'file object required for import (type code {})'.format(type_) - raise ValueError(msg) - elif type_ == PY_SOURCE: - return load_source(name, filename, file) - elif type_ == PY_COMPILED: - return load_compiled(name, filename, file) - elif type_ == PKG_DIRECTORY: - return load_package(name, filename) - elif type_ == C_BUILTIN: - return init_builtin(name) - elif type_ == PY_FROZEN: - return init_frozen(name) - else: - msg = "Don't know how to import {} (type code {}".format(name, type_) - raise ImportError(msg, name=name) + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + if mode and (not mode.startswith(('r', 'U')) or '+' in mode): + raise ValueError('invalid file open mode {!r}'.format(mode)) + elif file is None and type_ in {PY_SOURCE, PY_COMPILED}: + msg = 'file object required for import (type code {})'.format(type_) + raise ValueError(msg) + elif type_ == PY_SOURCE: + return load_source(name, filename, file) + elif type_ == PY_COMPILED: + return load_compiled(name, filename, file) + elif type_ == PKG_DIRECTORY: + return load_package(name, filename) + elif type_ == C_BUILTIN: + return init_builtin(name) + elif type_ == PY_FROZEN: + return init_frozen(name) + else: + msg = "Don't know how to import {} (type code {}".format(name, type_) + raise ImportError(msg, name=name) +# XXX deprecate def find_module(name, path=None): """Search for a module. diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -282,8 +282,10 @@ compared against. If the comparison fails then ImportError is raised. """ - def _check_name_wrapper(self, name, *args, **kwargs): - if self.name != name: + def _check_name_wrapper(self, name=None, *args, **kwargs): + if name is None: + name = self.name + elif self.name != name: raise ImportError("loader cannot handle %s" % name, name=name) return method(self, name, *args, **kwargs) _wrap(_check_name_wrapper, method) @@ -614,6 +616,11 @@ self.path = path @_check_name + def load_module(self, fullname): + """Load a module from a file.""" + return super().load_module(fullname) + + @_check_name def get_filename(self, fullname): """Return the path to the source file as found by the finder.""" return self.path @@ -713,17 +720,14 @@ del sys.modules[fullname] raise - @_check_name def is_package(self, fullname): """Return False as an extension module can never be a package.""" return False - @_check_name def get_code(self, fullname): """Return None as an extension module cannot create a code object.""" return None - @_check_name def get_source(self, fullname): """Return None as extension modules have no source code.""" return None diff --git a/Lib/importlib/test/extension/test_loader.py b/Lib/importlib/test/extension/test_loader.py --- a/Lib/importlib/test/extension/test_loader.py +++ b/Lib/importlib/test/extension/test_loader.py @@ -1,4 +1,4 @@ -from importlib import _bootstrap +from importlib import machinery from . import util as ext_util from .. import abc from .. import util @@ -11,10 +11,20 @@ """Test load_module() for extension modules.""" + def setUp(self): + self.loader = machinery.ExtensionFileLoader(ext_util.NAME, + ext_util.FILEPATH) + def load_module(self, fullname): - loader = _bootstrap.ExtensionFileLoader(ext_util.NAME, - ext_util.FILEPATH) - return loader.load_module(fullname) + return self.loader.load_module(fullname) + + def test_load_module_API(self): + # Test the default argument for load_module(). + self.loader.load_module() + self.loader.load_module(None) + with self.assertRaises(ImportError): + self.load_module('XXX') + def test_module(self): with util.uncache(ext_util.NAME): @@ -25,7 +35,7 @@ self.assertEqual(getattr(module, attr), value) self.assertTrue(ext_util.NAME in sys.modules) self.assertTrue(isinstance(module.__loader__, - _bootstrap.ExtensionFileLoader)) + machinery.ExtensionFileLoader)) def test_package(self): # Extensions are not found in packages. diff --git a/Lib/importlib/test/source/test_file_loader.py b/Lib/importlib/test/source/test_file_loader.py --- a/Lib/importlib/test/source/test_file_loader.py +++ b/Lib/importlib/test/source/test_file_loader.py @@ -1,5 +1,6 @@ from ... import _bootstrap import importlib +import importlib.abc from .. import abc from .. import util from . import util as source_util @@ -24,6 +25,40 @@ """ + def test_load_module_API(self): + # If fullname is not specified that assume self.name is desired. + class TesterMixin(importlib.abc.Loader): + def load_module(self, fullname): return fullname + + class Tester(importlib.abc.FileLoader, TesterMixin): + def get_code(self, _): pass + def get_source(self, _): pass + def is_package(self, _): pass + + name = 'mod_name' + loader = Tester(name, 'some_path') + self.assertEqual(name, loader.load_module()) + self.assertEqual(name, loader.load_module(None)) + self.assertEqual(name, loader.load_module(name)) + with self.assertRaises(ImportError): + loader.load_module(loader.name + 'XXX') + + def test_get_filename_API(self): + # If fullname is not set then assume self.path is desired. + class Tester(importlib.abc.FileLoader): + def get_code(self, _): pass + def get_source(self, _): pass + def is_package(self, _): pass + + path = 'some_path' + name = 'some_name' + loader = Tester(name, path) + self.assertEqual(path, loader.get_filename(name)) + self.assertEqual(path, loader.get_filename()) + self.assertEqual(path, loader.get_filename(None)) + with self.assertRaises(ImportError): + loader.get_filename(name + 'XXX') + # [basic] def test_module(self): with source_util.create_modules('_temp') as mapping: diff --git a/Lib/test/test_imp.py b/Lib/test/test_imp.py --- a/Lib/test/test_imp.py +++ b/Lib/test/test_imp.py @@ -1,11 +1,12 @@ import imp +import importlib import os import os.path import shutil import sys +from test import support import unittest -from test import support -import importlib +import warnings class LockTests(unittest.TestCase): @@ -154,18 +155,24 @@ mod = imp.load_module(temp_mod_name, file, filename, info) self.assertEqual(mod.a, 1) - mod = imp.load_source(temp_mod_name, temp_mod_name + '.py') + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + mod = imp.load_source(temp_mod_name, temp_mod_name + '.py') self.assertEqual(mod.a, 1) - mod = imp.load_compiled( - temp_mod_name, imp.cache_from_source(temp_mod_name + '.py')) + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + mod = imp.load_compiled( + temp_mod_name, imp.cache_from_source(temp_mod_name + '.py')) self.assertEqual(mod.a, 1) if not os.path.exists(test_package_name): os.mkdir(test_package_name) with open(init_file_name, 'w') as file: file.write('b = 2\n') - package = imp.load_package(test_package_name, test_package_name) + with warnings.catch_warnings(): + warnings.simplefilter('ignore') + package = imp.load_package(test_package_name, test_package_name) self.assertEqual(package.b, 2) finally: del sys.path[0] diff --git a/Lib/test/test_tools.py b/Lib/test/test_tools.py --- a/Lib/test/test_tools.py +++ b/Lib/test/test_tools.py @@ -6,7 +6,7 @@ import os import sys -import imp +import importlib.machinery import unittest from unittest import mock import sysconfig @@ -80,7 +80,8 @@ @classmethod def setUpClass(self): path = os.path.join(scriptsdir, 'pdeps.py') - self.pdeps = imp.load_source('pdeps', path) + loader = importlib.machinery.SourceFileLoader('pdeps', path) + self.pdeps = loader.load_module() @classmethod def tearDownClass(self): @@ -104,7 +105,8 @@ def setUp(self): path = os.path.join(scriptsdir, 'gprof2html.py') - self.gprof = imp.load_source('gprof2html', path) + loader = importlib.machinery.SourceFileLoader('gprof2html', path) + self.gprof = loader.load_module() oldargv = sys.argv def fixup(): sys.argv = oldargv diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,11 @@ Library ------- +- Issue #13959: Make importlib.abc.FileLoader.load_module()/get_filename() and + importlib.machinery.ExtensionFileLoader.load_module() have their single + argument be optional. Allows for the replacement (and thus deprecation) of + imp.load_source()/load_package()/load_compiled(). + - Issue #13959: imp.get_suffixes() has been deprecated in favour of the new attributes on importlib.machinery: SOURCE_SUFFIXES, DEBUG_BYTECODE_SUFFIXES, OPTIMIZED_BYTECODE_SUFFIXES, BYTECODE_SUFFIXES, and EXTENSION_SUFFIXES. This diff --git a/Python/importlib.h b/Python/importlib.h index 598eba50044a197361df15d0c0aeb6f755d16b5b..38f2c8a53c91bc094a3ef23023d4a06dd9ffa30e GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 11 21:50:23 2012 From: python-checkins at python.org (raymond.hettinger) Date: Fri, 11 May 2012 21:50:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Additional_exam?= =?utf8?q?ple_of_using_decimal=2Elocalcontext=28=29=2E?= Message-ID: http://hg.python.org/cpython/rev/a8af63c83f53 changeset: 76878:a8af63c83f53 branch: 2.7 parent: 76869:e12efebc3ba6 user: Raymond Hettinger date: Fri May 11 12:50:11 2012 -0700 summary: Additional example of using decimal.localcontext(). files: Doc/library/decimal.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -944,6 +944,10 @@ s = calculate_something() s = +s # Round the final result back to the default precision + with localcontext(BasicContext): # temporarily use the BasicContext + print Decimal(1) / Decimal(7) + print Decimal(355) / Decimal(113) + New contexts can also be created using the :class:`Context` constructor described below. In addition, the module provides three pre-made contexts: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 00:41:31 2012 From: python-checkins at python.org (benjamin.peterson) Date: Sat, 12 May 2012 00:41:31 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_simplify_by_shortcutting_wh?= =?utf8?q?en_the_kind_of_the_needle_is_larger_than_the_haystack?= Message-ID: http://hg.python.org/cpython/rev/83da67651687 changeset: 76879:83da67651687 parent: 76877:626d5c6fbd95 user: Benjamin Peterson date: Fri May 11 17:41:20 2012 -0500 summary: simplify by shortcutting when the kind of the needle is larger than the haystack files: Objects/unicodeobject.c | 32 +++++++++------------------- 1 files changed, 11 insertions(+), 21 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9075,15 +9075,14 @@ kind1 = PyUnicode_KIND(str_obj); kind2 = PyUnicode_KIND(sub_obj); - kind = kind1 > kind2 ? kind1 : kind2; + kind = kind2; buf1 = PyUnicode_DATA(str_obj); - if (kind1 != kind) - buf1 = _PyUnicode_AsKind(str_obj, kind); - if (!buf1) - goto onError; buf2 = PyUnicode_DATA(sub_obj); - if (kind2 != kind) + if (kind2 != kind) { + if (kind2 > kind) + return 0; buf2 = _PyUnicode_AsKind(sub_obj, kind); + } if (!buf2) goto onError; len1 = PyUnicode_GET_LENGTH(str_obj); @@ -9122,8 +9121,6 @@ Py_DECREF(sub_obj); Py_DECREF(str_obj); - if (kind1 != kind) - PyMem_Free(buf1); if (kind2 != kind) PyMem_Free(buf2); @@ -9131,8 +9128,6 @@ onError: Py_DECREF(sub_obj); Py_DECREF(str_obj); - if (kind1 != kind && buf1) - PyMem_Free(buf1); if (kind2 != kind && buf2) PyMem_Free(buf2); return -1; @@ -10660,20 +10655,17 @@ kind1 = PyUnicode_KIND(str); kind2 = PyUnicode_KIND(sub); - kind = kind1 > kind2 ? kind1 : kind2; + kind = kind1; buf1 = PyUnicode_DATA(str); buf2 = PyUnicode_DATA(sub); - if (kind1 != kind) - buf1 = _PyUnicode_AsKind(str, kind); - if (!buf1) { - Py_DECREF(sub); - return -1; - } - if (kind2 != kind) + if (kind2 != kind) { + if (kind2 > kind) + return 0; buf2 = _PyUnicode_AsKind(sub, kind); + } if (!buf2) { Py_DECREF(sub); - if (kind1 != kind) PyMem_Free(buf1); + Py_DECREF(str); return -1; } len1 = PyUnicode_GET_LENGTH(str); @@ -10697,8 +10689,6 @@ Py_DECREF(str); Py_DECREF(sub); - if (kind1 != kind) - PyMem_Free(buf1); if (kind2 != kind) PyMem_Free(buf2); -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat May 12 05:49:14 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 12 May 2012 05:49:14 +0200 Subject: [Python-checkins] Daily reference leaks (83da67651687): sum=60 Message-ID: results for 83da67651687 on branch "default" -------------------------------------------- test_getargs2 leaked [2, 2, 2] references, sum=6 test_ntpath leaked [2, 2, 2] references, sum=6 test_re leaked [12, 12, 12] references, sum=36 test_sqlite leaked [4, 4, 4] references, sum=12 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog1P6gdN', '-x'] From python-checkins at python.org Sat May 12 08:32:45 2012 From: python-checkins at python.org (ross.lagerwall) Date: Sat, 12 May 2012 08:32:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_uninitialized_compil?= =?utf8?q?er_warning=2E?= Message-ID: http://hg.python.org/cpython/rev/d4b559faefc4 changeset: 76880:d4b559faefc4 user: Ross Lagerwall date: Sat May 12 08:30:33 2012 +0200 summary: Remove uninitialized compiler warning. files: Modules/_lzmamodule.c | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -412,7 +412,11 @@ ADD_FIELD(options, lc); ADD_FIELD(options, lp); ADD_FIELD(options, pb); - case LZMA_FILTER_LZMA2: + ADD_FIELD(options, dict_size); + break; + } + case LZMA_FILTER_LZMA2: { + lzma_options_lzma *options = f->options; ADD_FIELD(options, dict_size); break; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 16:00:29 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 16:00:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_logic_error_introduced_?= =?utf8?q?by_83da67651687=2E?= Message-ID: http://hg.python.org/cpython/rev/e27e1ac8806e changeset: 76881:e27e1ac8806e user: Antoine Pitrou date: Sat May 12 15:49:07 2012 +0200 summary: Fix logic error introduced by 83da67651687. files: Objects/unicodeobject.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9075,12 +9075,12 @@ kind1 = PyUnicode_KIND(str_obj); kind2 = PyUnicode_KIND(sub_obj); - kind = kind2; + kind = kind1; buf1 = PyUnicode_DATA(str_obj); buf2 = PyUnicode_DATA(sub_obj); if (kind2 != kind) { if (kind2 > kind) - return 0; + return 0; buf2 = _PyUnicode_AsKind(sub_obj, kind); } if (!buf2) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 16:00:31 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 16:00:31 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_refleaks_introduced_by_?= =?utf8?q?83da67651687=2E?= Message-ID: http://hg.python.org/cpython/rev/9d9495fabeb9 changeset: 76882:9d9495fabeb9 user: Antoine Pitrou date: Sat May 12 15:51:51 2012 +0200 summary: Fix refleaks introduced by 83da67651687. files: Objects/unicodeobject.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9079,8 +9079,11 @@ buf1 = PyUnicode_DATA(str_obj); buf2 = PyUnicode_DATA(sub_obj); if (kind2 != kind) { - if (kind2 > kind) + if (kind2 > kind) { + Py_DECREF(sub_obj); + Py_DECREF(str_obj); return 0; + } buf2 = _PyUnicode_AsKind(sub_obj, kind); } if (!buf2) @@ -10659,8 +10662,11 @@ buf1 = PyUnicode_DATA(str); buf2 = PyUnicode_DATA(sub); if (kind2 != kind) { - if (kind2 > kind) + if (kind2 > kind) { + Py_DECREF(sub); + Py_DECREF(str); return 0; + } buf2 = _PyUnicode_AsKind(sub, kind); } if (!buf2) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 16:27:24 2012 From: python-checkins at python.org (barry.warsaw) Date: Sat, 12 May 2012 16:27:24 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Eric=27s_latest_update?= Message-ID: http://hg.python.org/peps/rev/e3f8cbef3f4b changeset: 4370:e3f8cbef3f4b user: Barry Warsaw date: Sat May 12 07:27:18 2012 -0700 summary: Eric's latest update files: pep-0421.txt | 323 ++++++++++++++++---------------------- 1 files changed, 140 insertions(+), 183 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -51,41 +51,50 @@ ======== We will add a new attribute to the ``sys`` module, called -``sys.implementation``, as an instance of a new type to contain -implementation-specific information. +``sys.implementation``, as an object with attribute-access (as opposed +to a mapping). It will contain contain implementation-specific +information. The attributes of this object will remain fixed during interpreter execution and through the course of an implementation version. This ensures behaviors don't change between versions which depend on -variables in ``sys.implementation``. +attributes of ``sys.implementation``. The object will have each of the attributes described in the `Required -Variables`_ section below. Any other per-implementation values may be -stored in ``sys.implementation.metadata``. However, nothing in the -standard library will rely on ``sys.implementation.metadata``. -Examples of possible metadata values are described in the `Example -Metadata Values`_ section. +Attributes`_ section below. Those attribute names will never start +with an underscore. The standard library and the language definition +will rely only on those required attributes. -This proposal takes a conservative approach in requiring only four -variables. As more become appropriate, they may be added with -discretion. +This proposal takes a conservative approach in requiring only a small +number of attributes. As more become appropriate, they may be added +with discretion, as described in `Adding New Required Attributes`_. +While this PEP places no other constraints on ``sys.implementation``, +it also recommends that no one rely on capabilities outside those +described here. The only exception to that recommendation is for +attributes starting with an underscore. Implementors may use those +as appropriate to store per-implementation data. -Required Variables ------------------- -These are variables in ``sys.implementation`` on which the standard -library would rely, with the exception of ``metadata``, meaning -implementers must define them: +Required Attributes +------------------- + +These are attributes in ``sys.implementation`` on which the standard +library and language definition will rely, meaning implementers must +define them: **name** - This is the common name of the implementation (case sensitive). - Examples include 'PyPy', 'Jython', 'IronPython', and 'CPython'. + A lower-case identifer representing the implementation. Examples + include 'pypy', 'jython', 'ironpython', and 'cpython'. **version** - This is the version of the implementation, as opposed to the - version of the language it implements. This value conforms to the - format described in `Version Format`_. + The version of the implementation, as opposed to the version of the + language it implements. This value conforms to the format described + in `Version Format`_. + +**hexversion** + The version of the implementation in the same hexadecimal format as + ``sys.hexversion``. **cache_tag** A string used for the PEP 3147 cache tag [#cachetag]_. It would @@ -94,19 +103,21 @@ different cache tag. If ``cache_tag`` is set to None, it indicates that module caching should be disabled. -**metadata** - Any other values that an implementation wishes to specify, - particularly informational ones. Neither the standard library nor - the language specification will rely on implementation metadata. - Also see the list of `Example Metadata Values`_. ``metadata`` is a - dictionary. While a mutable type, neither it nor its items will - change (as already noted). Likewise they should not be modified. - Adding New Required Attributes ------------------------------ -XXX PEP? something lighter? +In time more required attributes will be added to +``sys.implementation``. However, each must have a meaningful use case +across all Python implementations in order to be considered. This is +made most clear by a use case in the standard library or language +specification. + +All proposals for new required attributes will go through the normal +PEP process. Such a PEP need not be all that long, but will need to +sufficiently spell out the rationale for the new attribute, its use +cases, and the impact it will have on the various Python +implemenations. Version Format @@ -114,59 +125,14 @@ A main point of ``sys.implementation`` is to contain information that will be used internally in the standard library. In order to -facilitate the usefulness of a version variable, its value should be -in a consistent format across implementations. +facilitate the usefulness of the version attribute, its value should +be in a consistent format across implementations. -As such, the format of ``sys.implementation.version`` must follow that +As such, the format of ``sys.implementation.version`` will follow that of ``sys.version_info``, which is effectively a named tuple. It is a familiar format and generally consistent with normal version format conventions. -XXX The following is not exactly true: - -Keep in mind, however, that ``sys.implementation.version`` is the -version of the Python *implementation*, while ``sys.version_info`` -(and friends) is the version of the Python language. - - -Example Metadata Values ------------------------ - -These are the sorts of values an implementation may put into -``sys.implementation.metadata``. However, these names and -descriptions are only examples and are not being proposed here. If -they later have meaningful uses cases, they can be added by following -the process described in `Adding New Required Attributes`_. - -**vcs_url** - The URL pointing to the main VCS repository for the implementation - project. - -**vcs_revision_id** - A value that identifies the VCS revision of the implementation - that is currently running. - -**build_toolchain** - Identifies the tools used to build the interpreter. - -**build_date** - The timestamp of when the interpreter was built. - -**homepage** - The URL of the implementation's website. - -**site_prefix** - The preferred site prefix for this implementation. - -**runtime** - The run-time environment in which the interpreter is running, as - in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* - Executable". - -**gc_type** - The type of garbage collection used, like "reference counting" or - "mark and sweep". - Rationale ========= @@ -181,79 +147,33 @@ makes explicit that which was implicit. -Why a Custom Type? ------------------- +Type Considerations +------------------- -A dedicated class, of which ``sys.implementation`` is an instance, -would facilitate the dotted access of a "named" tuple. At the same -time, it allows us to avoid the problems of the other approaches (see -below), like confusion about ordering and iteration. +It's very easy to get bogged down in discussions about the type of +``sys.implementation``. However, its purpose is to support the +standard library and language definition. As such, there isn't much +that really matters in this regard, as opposed to a feature that would +be more generally used. Thus characteristics like immutability and +sequence-ness have been disregarded. -The alternatives to a dictionary are considered separately here: +The only real choice was between an object with attribute access and a +mapping with item access. This PEP espouses dotted access to reflect +the relatively fixed nature of the namespace. -**Dictionary** -A dictionary reflects a simple namespace with item access. It -maps names to values and that's all. It also reflects the more -variable nature of ``sys.implementation``. +Non-Required Attributes +----------------------- -However, a simple dictionary does not set expectations very well about -the nature of ``sys.implementation``. The custom type approach, with -a fixed set of required attributes, does a better job of this. +Earlier versions of this PEP included a required attribute called +``metadata`` that held any non-required, per-implementation data +[#Nick]_. However, this proved to be an unnecessary addition +considering the purpose of ``sys.implementation``. -**Named Tuple** - -Another close alternative is a namedtuple or a structseq or some other -tuple type with dotted access (a la ``sys.version_info``). This type -is immutable and simple. It is a well established pattern for -implementation-specific variables in Python. Dotted access on a -namespace is also very convenient. - -Fallback lookup may favor dicts:: - - cache_tag = sys.implementation.get('cache_tag') - - vs. - - cache_tag = getattr(sys.implementation.get, 'cache_tag', None) - -However, this is mitigated by having ``sys.implementation.metadata``. - -One problem with using a named tuple is that ``sys.implementation`` -does not have meaning as a sequence. Also, unlike other similar -``sys`` variables, it has a far greater potential to change over time. - -If a named tuple were used, we'd be very clear in the documentation -that the length and order of the value are not reliable. Iterability -would not be guaranteed. - -**Module** - -Using a module instead of a dict is another option. It has similar -characteristics to an instance, but with a slight hint of immutability -(at least by convention). Such a module could be a stand-alone sub- -module of ``sys`` or added on, like ``os.path``. Unlike a concrete -class, no new type would be necessary. This is a pretty close fit to -what we need. - -The downside is that the module type is much less conducive to -extension, making it more difficult to address the weaknesses of using -an instance of a concrete class. - - -Why metadata? -------------- - -``sys.implementation.metadata`` will hold any optional, strictly- -informational, or per-implementation data. This allows us to restrict -``sys.implementation`` to the required attributes. In that way, its -type can reflect the more stable namespace and -``sys.implementation.metadata`` (as a dict) can reflect the less -certain namespace. - -``sys.implementation.metadata`` is the place an implementation can put -values that must be built-in, "without having to pollute the main sys -namespace" [#Nick]_. +Ultimately, non-required attributes are virtually ignored in this PEP. +They have no impact other than that careless use may collide with +future required attributes. That, however, is but a marginal concern +for ``sys.implementation``. Why a Part of ``sys``? @@ -261,7 +181,7 @@ The ``sys`` module should hold the new namespace because ``sys`` is the depot for interpreter-centric variables and functions. Many -implementation-specific variables are already found in ``sys``. +implementation-specific attributes are already found in ``sys``. Why Strict Constraints on Any of the Values? @@ -271,7 +191,8 @@ ``sys.implementation`` are intended for use by the standard library. Constraining those values, essentially specifying an API for them, allows them to be used consistently, regardless of how they are -otherwise implemented. +otherwise implemented. However, care should be take to not over- +specify the constraints. Discussion @@ -283,6 +204,9 @@ ``imp.get_tag()`` [#revived]_. Discussion has been ongoing [#feedback]_. The messages in `issue #14673`_ are also relevant. +A good part of the discussion centered on the type to use for +``sys.implementation``. + Use-cases ========= @@ -361,8 +285,30 @@ treatment of the java environment in the standard library [#os_name]_ [#javatest]_. Unfortunately it masks the os name that would otherwise go there. ``sys.implementation`` would help obviate the need for this -special case. Currently Jython sets os._name for the normal os.name -value. +special case. Currently Jython sets ``os._name`` for the normal +``os.name`` value. + + +The Problem With ``sys.(version|version_info|hexversion)`` +---------------------------------------------------------- + +Earlier versions of this PEP made the mistake of calling +``sys.version_info`` (and friends) the version of the Python language, +in contrast to the implementation. However, this is not the case. +Instead, it is the version of the CPython implementation. Incidently, +the first two components of ``sys.version_info`` (major and minor) also +reflect the version of the language definition. + +As Barry Warsaw noted, the "semantics of sys.version_info have been +sufficiently squishy in the past" [#Barry]_. With +``sys.implementation`` we have the opportunity to improving the +situation by first establishing an explicit location for the version of +the implementation. + +This PEP makes no other effort to directly clarify the semantics of +``sys.version_info``. Regardless, having an explicit version for the +implementation will definitely help to clarify the distinction from the +language version. Feedback From Other Python Implementors @@ -395,9 +341,9 @@ will happily adhere to when we migrate to Python 3.3. He also expressed support for keeping the required list small. Both -Armin and Laura indicated that an effort to better catalog Python's -implementation would be welcome. Such an effort, for which this PEP is -a small start, will be considered separately. +Armin and Laura Creighton indicated that an effort to better catalog +Python's implementation would be welcome. Such an effort, for which +this PEP is a small start, will be considered separately. Past Efforts @@ -408,7 +354,7 @@ This PEP from 2008 recommended a clean-up of the ``sys`` module in part by extracting implementation-specific variables and functions -into a separate module. PEP 421 is a much lighter version of that +into a separate module. PEP 421 is less ambitious version of that idea. While PEP 3139 was rejected, its goals are reflected in PEP 421 to a large extent, though with a much lighter approach. @@ -449,43 +395,47 @@ straightforward, no alternatives have been considered for this PEP. +Examples of Other Attributes +============================ + +These are examples only and not part of the proposal. (See `Adding +New Required Attributes`_ if they get you excited.) + +**common_name** + The case-sensitive name by which the implementation is known. + +**vcs_url** + A URL for the main VCS repository for the implementation project. + +**vcs_revision_id** + A value that identifies the VCS revision of the implementation. + +**build_toolchain** + The tools used to build the interpreter. + +**build_date** + The timestamp of when the interpreter was built. + +**homepage** + The URL of the implementation's website. + +**site_prefix** + The preferred site prefix for the implementation. + +**runtime** + The run-time environment in which the interpreter is running, as + in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* + Executable". + +**gc_type** + The type of garbage collection used, like "reference counting" or + "mark and sweep". + + Open Issues =========== -* What is the process for introducing new required variables? PEP? - -* Is the ``sys.version_info`` format the right one here? - -* Should ``sys.implementation.hexversion`` be part of the PEP? - -* Does ``sys.(version|version_info|hexversion)`` need to better - reflect the version of the language spec? Micro version, series, - and release seem like implementation-specific values. - -* Do we really want to commit to using a dict for - ``sys.implementation``? - - Backward compatibility issues will make it difficult to change our - minds later. - - The type we use ultimately depends on how general we expect the - consumption of ``sys.implementation`` to be. If its practicality is - oriented toward internal use then the data structure is not as - critical. However, ``sys.implementation`` is intended to have a - non-localized impact across the standard library and the - interpreter. It is better to *not* make hacking it become an - attractive nuisance, regardless of our intentions for usage. - -* Should ``sys.implementation`` and its values be immutable? A benefit - of an immutable type is it communicates that the value is not - expected to change and should not be manipulated. - -* Should ``sys.implementation`` be strictly disallowed to have methods? - Classes often imply the presence (or possibility) of methods, which - may be misleading in this case. - -* Should ``sys.implementation`` implement the collections.abc.Mapping - interface? +Currently none. Implementation @@ -552,6 +502,9 @@ .. [#Nick] Nick Coghlan's proposal for ``sys.implementation.metadata``: http://mail.python.org/pipermail/python-ideas/2012-May/014984.html +.. [#Barry] Feedback from Barry Warsaw: + http://mail.python.org/pipermail/python-dev/2012-May/119374.html + .. _issue #14673: http://bugs.python.org/issue14673 .. _Lib/test/support.py: http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat May 12 19:05:03 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 19:05:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314082=3A_shutil=2E?= =?utf8?q?copy2=28=29_now_copies_extended_attributes=2C_if_possible=2E?= Message-ID: http://hg.python.org/cpython/rev/85824b819bcb changeset: 76883:85824b819bcb parent: 76880:d4b559faefc4 user: Antoine Pitrou date: Sat May 12 19:02:01 2012 +0200 summary: Issue #14082: shutil.copy2() now copies extended attributes, if possible. Patch by Hynek Schlawack. files: Doc/library/shutil.rst | 6 +- Lib/shutil.py | 31 ++++++++++++ Lib/test/support.py | 29 +++++++++++ Lib/test/test_os.py | 20 +------- Lib/test/test_shutil.py | 74 +++++++++++++++++++++++++++++ Misc/NEWS | 3 + 6 files changed, 141 insertions(+), 22 deletions(-) diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -102,14 +102,14 @@ .. function:: copy2(src, dst[, symlinks=False]) - Similar to :func:`shutil.copy`, but metadata is copied as well -- in fact, - this is just :func:`shutil.copy` followed by :func:`copystat`. This is + Similar to :func:`shutil.copy`, but metadata is copied as well. This is similar to the Unix command :program:`cp -p`. If *symlinks* is true, symbolic links won't be followed but recreated instead -- this resembles GNU's :program:`cp -P`. .. versionchanged:: 3.3 - Added *symlinks* argument. + Added *symlinks* argument, try to copy extended file system attributes + too (currently Linux only). .. function:: ignore_patterns(\*patterns) diff --git a/Lib/shutil.py b/Lib/shutil.py --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -166,6 +166,36 @@ else: raise +if hasattr(os, 'listxattr'): + def _copyxattr(src, dst, symlinks=False): + """Copy extended filesystem attributes from `src` to `dst`. + + Overwrite existing attributes. + + If the optional flag `symlinks` is set, symlinks won't be followed. + + """ + if symlinks: + listxattr = os.llistxattr + removexattr = os.lremovexattr + setxattr = os.lsetxattr + getxattr = os.lgetxattr + else: + listxattr = os.listxattr + removexattr = os.removexattr + setxattr = os.setxattr + getxattr = os.getxattr + + for attr in listxattr(src): + try: + setxattr(dst, attr, getxattr(src, attr)) + except OSError as e: + if e.errno not in (errno.EPERM, errno.ENOTSUP, errno.ENODATA): + raise +else: + def _copyxattr(*args, **kwargs): + pass + def copy(src, dst, symlinks=False): """Copy data and mode bits ("cp src dst"). @@ -193,6 +223,7 @@ dst = os.path.join(dst, os.path.basename(src)) copyfile(src, dst, symlinks=symlinks) copystat(src, dst, symlinks=symlinks) + _copyxattr(src, dst, symlinks=symlinks) def ignore_patterns(*patterns): """Function that can be used as copytree() ignore parameter. diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -1696,6 +1696,35 @@ msg = "Requires functional symlink implementation" return test if ok else unittest.skip(msg)(test) +_can_xattr = None +def can_xattr(): + global _can_xattr + if _can_xattr is not None: + return _can_xattr + if not hasattr(os, "setxattr"): + can = False + else: + try: + with open(TESTFN, "wb") as fp: + try: + os.fsetxattr(fp.fileno(), b"user.test", b"") + # Kernels < 2.6.39 don't respect setxattr flags. + kernel_version = platform.release() + m = re.match("2.6.(\d{1,2})", kernel_version) + can = m is None or int(m.group(1)) >= 39 + except OSError: + can = False + finally: + unlink(TESTFN) + _can_xattr = can + return can + +def skip_unless_xattr(test): + """Skip decorator for tests that require functional extended attributes""" + ok = can_xattr() + msg = "no non-broken extended attribute support" + return test if ok else unittest.skip(msg)(test) + def patch(test_instance, object_to_patch, attr_name, new_value): """Override 'object_to_patch'.'attr_name' with 'new_value'. diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -1810,25 +1810,7 @@ raise -def supports_extended_attributes(): - if not hasattr(os, "setxattr"): - return False - try: - with open(support.TESTFN, "wb") as fp: - try: - os.fsetxattr(fp.fileno(), b"user.test", b"") - except OSError: - return False - finally: - support.unlink(support.TESTFN) - # Kernels < 2.6.39 don't respect setxattr flags. - kernel_version = platform.release() - m = re.match("2.6.(\d{1,2})", kernel_version) - return m is None or int(m.group(1)) >= 39 - - - at unittest.skipUnless(supports_extended_attributes(), - "no non-broken extended attribute support") + at support.skip_unless_xattr class ExtendedAttributeTests(unittest.TestCase): def tearDown(self): diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -311,6 +311,67 @@ finally: os.chflags = old_chflags + @support.skip_unless_xattr + def test_copyxattr(self): + tmp_dir = self.mkdtemp() + src = os.path.join(tmp_dir, 'foo') + write_file(src, 'foo') + dst = os.path.join(tmp_dir, 'bar') + write_file(dst, 'bar') + + # no xattr == no problem + shutil._copyxattr(src, dst) + # common case + os.setxattr(src, 'user.foo', b'42') + os.setxattr(src, 'user.bar', b'43') + shutil._copyxattr(src, dst) + self.assertEqual(os.listxattr(src), os.listxattr(dst)) + self.assertEqual( + os.getxattr(src, 'user.foo'), + os.getxattr(dst, 'user.foo')) + # check errors don't affect other attrs + os.remove(dst) + write_file(dst, 'bar') + os_error = OSError(errno.EPERM, 'EPERM') + + def _raise_on_user_foo(fname, attr, val): + if attr == 'user.foo': + raise os_error + else: + orig_setxattr(fname, attr, val) + try: + orig_setxattr = os.setxattr + os.setxattr = _raise_on_user_foo + shutil._copyxattr(src, dst) + self.assertEqual(['user.bar'], os.listxattr(dst)) + finally: + os.setxattr = orig_setxattr + + @support.skip_unless_symlink + @support.skip_unless_xattr + @unittest.skipUnless(hasattr(os, 'geteuid') and os.geteuid() == 0, + 'root privileges required') + def test_copyxattr_symlinks(self): + # On Linux, it's only possible to access non-user xattr for symlinks; + # which in turn require root privileges. This test should be expanded + # as soon as other platforms gain support for extended attributes. + tmp_dir = self.mkdtemp() + src = os.path.join(tmp_dir, 'foo') + src_link = os.path.join(tmp_dir, 'baz') + write_file(src, 'foo') + os.symlink(src, src_link) + os.setxattr(src, 'trusted.foo', b'42') + os.lsetxattr(src_link, 'trusted.foo', b'43') + dst = os.path.join(tmp_dir, 'bar') + dst_link = os.path.join(tmp_dir, 'qux') + write_file(dst, 'bar') + os.symlink(dst, dst_link) + shutil._copyxattr(src_link, dst_link, symlinks=True) + self.assertEqual(os.lgetxattr(dst_link, 'trusted.foo'), b'43') + self.assertRaises(OSError, os.getxattr, dst, 'trusted.foo') + shutil._copyxattr(src_link, dst, symlinks=True) + self.assertEqual(os.getxattr(dst, 'trusted.foo'), b'43') + @support.skip_unless_symlink def test_copy_symlinks(self): tmp_dir = self.mkdtemp() @@ -369,6 +430,19 @@ if hasattr(os, 'lchflags') and hasattr(src_link_stat, 'st_flags'): self.assertEqual(src_link_stat.st_flags, dst_stat.st_flags) + @support.skip_unless_xattr + def test_copy2_xattr(self): + tmp_dir = self.mkdtemp() + src = os.path.join(tmp_dir, 'foo') + dst = os.path.join(tmp_dir, 'bar') + write_file(src, 'foo') + os.setxattr(src, 'user.foo', b'42') + shutil.copy2(src, dst) + self.assertEqual( + os.getxattr(src, 'user.foo'), + os.getxattr(dst, 'user.foo')) + os.remove(dst) + @support.skip_unless_symlink def test_copyfile_symlinks(self): tmp_dir = self.mkdtemp() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #14082: shutil.copy2() now copies extended attributes, if possible. + Patch by Hynek Schlawack. + - Issue #13959: Make importlib.abc.FileLoader.load_module()/get_filename() and importlib.machinery.ExtensionFileLoader.load_module() have their single argument be optional. Allows for the replacement (and thus deprecation) of -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 19:05:04 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 19:05:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/3b368c4ad7c5 changeset: 76884:3b368c4ad7c5 parent: 76883:85824b819bcb parent: 76882:9d9495fabeb9 user: Antoine Pitrou date: Sat May 12 19:02:47 2012 +0200 summary: Merge files: Objects/unicodeobject.c | 14 ++++++++++---- 1 files changed, 10 insertions(+), 4 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -9075,12 +9075,15 @@ kind1 = PyUnicode_KIND(str_obj); kind2 = PyUnicode_KIND(sub_obj); - kind = kind2; + kind = kind1; buf1 = PyUnicode_DATA(str_obj); buf2 = PyUnicode_DATA(sub_obj); if (kind2 != kind) { - if (kind2 > kind) - return 0; + if (kind2 > kind) { + Py_DECREF(sub_obj); + Py_DECREF(str_obj); + return 0; + } buf2 = _PyUnicode_AsKind(sub_obj, kind); } if (!buf2) @@ -10659,8 +10662,11 @@ buf1 = PyUnicode_DATA(str); buf2 = PyUnicode_DATA(sub); if (kind2 != kind) { - if (kind2 > kind) + if (kind2 > kind) { + Py_DECREF(sub); + Py_DECREF(str); return 0; + } buf2 = _PyUnicode_AsKind(sub, kind); } if (!buf2) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:13:57 2012 From: python-checkins at python.org (stefan.krah) Date: Sat, 12 May 2012 23:13:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314779=3A_Do_not_us?= =?utf8?q?e_get=5Fconfig=5Fvar=28=27SIZEOF=5FVOID=5FP=27=29_on_OS_X_64-/32?= =?utf8?q?-bit?= Message-ID: http://hg.python.org/cpython/rev/8f22e5be18c8 changeset: 76885:8f22e5be18c8 user: Stefan Krah date: Sat May 12 23:11:51 2012 +0200 summary: Issue #14779: Do not use get_config_var('SIZEOF_VOID_P') on OS X 64-/32-bit universal: it returns a meaningless result. Use sys.maxsize instead of platform.architecture as a fallback. Patch by Ned Deily. files: Lib/test/test_buffer.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_buffer.py b/Lib/test/test_buffer.py --- a/Lib/test/test_buffer.py +++ b/Lib/test/test_buffer.py @@ -16,7 +16,6 @@ from itertools import permutations, product from random import randrange, sample, choice from sysconfig import get_config_var -from platform import architecture import warnings import sys, array, io from decimal import Decimal @@ -748,9 +747,10 @@ class TestBufferProtocol(unittest.TestCase): def setUp(self): - self.sizeof_void_p = get_config_var('SIZEOF_VOID_P') + self.sizeof_void_p = get_config_var('SIZEOF_VOID_P') \ + if sys.platform != 'darwin' else None if not self.sizeof_void_p: - self.sizeof_void_p = 8 if architecture()[0] == '64bit' else 4 + self.sizeof_void_p = 8 if sys.maxsize > 2**32 else 4 def verify(self, result, obj=-1, itemsize={1}, fmt=-1, readonly={1}, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:32:29 2012 From: python-checkins at python.org (sandro.tosi) Date: Sat, 12 May 2012 23:32:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_correctly_defin?= =?utf8?q?e_what_=27fill=27_could_be=3B_thanks_to_Leland_Hulbert_from_docs?= =?utf8?q?=40?= Message-ID: http://hg.python.org/cpython/rev/94fb31fb3a9b changeset: 76886:94fb31fb3a9b branch: 2.7 parent: 76878:a8af63c83f53 user: Sandro Tosi date: Sat May 12 23:29:06 2012 +0200 summary: correctly define what 'fill' could be; thanks to Leland Hulbert from docs@ files: Doc/library/string.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -322,7 +322,7 @@ .. productionlist:: sf format_spec: [[`fill`]`align`][`sign`][#][0][`width`][,][.`precision`][`type`] - fill: + fill: align: "<" | ">" | "=" | "^" sign: "+" | "-" | " " width: `integer` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:32:30 2012 From: python-checkins at python.org (sandro.tosi) Date: Sat, 12 May 2012 23:32:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_correctly_defin?= =?utf8?q?e_what_=27fill=27_could_be=3B_thanks_to_Leland_Hulbert_from_docs?= =?utf8?q?=40?= Message-ID: http://hg.python.org/cpython/rev/04e7264e8fcf changeset: 76887:04e7264e8fcf branch: 3.2 parent: 76870:ae141eebcf96 user: Sandro Tosi date: Sat May 12 23:29:32 2012 +0200 summary: correctly define what 'fill' could be; thanks to Leland Hulbert from docs@ files: Doc/library/string.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -292,7 +292,7 @@ .. productionlist:: sf format_spec: [[`fill`]`align`][`sign`][#][0][`width`][,][.`precision`][`type`] - fill: + fill: align: "<" | ">" | "=" | "^" sign: "+" | "-" | " " width: `integer` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:32:32 2012 From: python-checkins at python.org (sandro.tosi) Date: Sat, 12 May 2012 23:32:32 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_with_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/54b8337c306a changeset: 76888:54b8337c306a parent: 76885:8f22e5be18c8 parent: 76887:04e7264e8fcf user: Sandro Tosi date: Sat May 12 23:30:05 2012 +0200 summary: merge with 3.2 files: Doc/library/string.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/string.rst b/Doc/library/string.rst --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -292,7 +292,7 @@ .. productionlist:: sf format_spec: [[`fill`]`align`][`sign`][#][0][`width`][,][.`precision`][`type`] - fill: + fill: align: "<" | ">" | "=" | "^" sign: "+" | "-" | " " width: `integer` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:39:50 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 23:39:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Try_to_fix_test=5Fshutil_fa?= =?utf8?q?ilure_under_Fedora_-_patch_by_Hynek=2E?= Message-ID: http://hg.python.org/cpython/rev/8d85f9920878 changeset: 76889:8d85f9920878 user: Antoine Pitrou date: Sat May 12 23:37:35 2012 +0200 summary: Try to fix test_shutil failure under Fedora - patch by Hynek. files: Lib/test/test_shutil.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -343,7 +343,7 @@ orig_setxattr = os.setxattr os.setxattr = _raise_on_user_foo shutil._copyxattr(src, dst) - self.assertEqual(['user.bar'], os.listxattr(dst)) + self.assertIn('user.bar', os.listxattr(dst)) finally: os.setxattr = orig_setxattr -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:43:32 2012 From: python-checkins at python.org (brett.cannon) Date: Sat, 12 May 2012 23:43:32 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_a_now_worthless_test?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/3ab5aa8bc60e changeset: 76890:3ab5aa8bc60e user: Brett Cannon date: Sat May 12 17:40:28 2012 -0400 summary: Remove a now worthless test. files: Lib/test/test_unicode.py | 8 -------- 1 files changed, 0 insertions(+), 8 deletions(-) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1422,14 +1422,6 @@ self.assertRaises(TypeError, str, b"hello", "test.unicode2") self.assertRaises(TypeError, "hello".encode, "test.unicode1") self.assertRaises(TypeError, "hello".encode, "test.unicode2") - # executes PyUnicode_Encode() - import imp - self.assertRaises( - ImportError, - imp.find_module, - "non-existing module", - ["non-existing dir"] - ) # Error handling (wrong arguments) self.assertRaises(TypeError, "hello".encode, 42, 42, 42) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:43:33 2012 From: python-checkins at python.org (brett.cannon) Date: Sat, 12 May 2012 23:43:33 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313959=3A_Introduce?= =?utf8?q?_importlib=2Efind=5Floader=28=29=2E?= Message-ID: http://hg.python.org/cpython/rev/7bf8ac742d2f changeset: 76891:7bf8ac742d2f user: Brett Cannon date: Sat May 12 17:43:17 2012 -0400 summary: Issue #13959: Introduce importlib.find_loader(). The long-term goal is to deprecate imp.find_module() in favour of this API, but it will take some time as some APIs explicitly return/use what imp.find_module() returns. files: Doc/library/importlib.rst | 14 ++++++ Lib/importlib/__init__.py | 24 ++++++++++ Lib/importlib/test/test_api.py | 50 +++++++++++++++++++++- Lib/pyclbr.py | 25 +++++++--- Misc/NEWS | 2 + 5 files changed, 105 insertions(+), 10 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -86,6 +86,20 @@ that was imported (e.g. ``pkg.mod``), while :func:`__import__` returns the top-level package or module (e.g. ``pkg``). +.. function:: find_loader(name, path=None) + + Find the loader for a module, optionally within the specified *path*. If the + module is in :attr:`sys.modules`, then ``sys.modules[name].__loader__`` is + returned (unless the loader would be ``None``, in which case + :exc:`ValueError` is raised). Otherwise a search using :attr:`sys.meta_path` + is done. ``None`` is returned if no loader is found. + + A dotted name does not have its parent's implicitly imported. If that is + desired (although not nessarily required to find the loader, it will most + likely be needed if the loader actually is used to load the module), then + you will have to import the packages containing the module prior to calling + this function. + .. function:: invalidate_caches() Invalidate the internal caches of the finders stored at diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py --- a/Lib/importlib/__init__.py +++ b/Lib/importlib/__init__.py @@ -29,6 +29,30 @@ finder.invalidate_caches() +def find_loader(name, path=None): + """Find the loader for the specified module. + + First, sys.modules is checked to see if the module was already imported. If + so, then sys.modules[name].__loader__ is returned. If that happens to be + set to None, then ValueError is raised. If the module is not in + sys.modules, then sys.meta_path is searched for a suitable loader with the + value of 'path' given to the finders. None is returned if no loader could + be found. + + Dotted names do not have their parent packages implicitly imported. + + """ + try: + loader = sys.modules[name].__loader__ + if loader is None: + raise ValueError('{}.__loader__ is None'.format(name)) + else: + return loader + except KeyError: + pass + return _bootstrap._find_module(name, path) + + def import_module(name, package=None): """Import a module. diff --git a/Lib/importlib/test/test_api.py b/Lib/importlib/test/test_api.py --- a/Lib/importlib/test/test_api.py +++ b/Lib/importlib/test/test_api.py @@ -85,6 +85,54 @@ self.assertEqual(b_load_count, 1) +class FindLoaderTests(unittest.TestCase): + + class FakeMetaFinder: + @staticmethod + def find_module(name, path=None): return name, path + + def test_sys_modules(self): + # If a module with __loader__ is in sys.modules, then return it. + name = 'some_mod' + with util.uncache(name): + module = imp.new_module(name) + loader = 'a loader!' + module.__loader__ = loader + sys.modules[name] = module + found = importlib.find_loader(name) + self.assertEqual(loader, found) + + def test_sys_modules_loader_is_None(self): + # If sys.modules[name].__loader__ is None, raise ValueError. + name = 'some_mod' + with util.uncache(name): + module = imp.new_module(name) + module.__loader__ = None + sys.modules[name] = module + with self.assertRaises(ValueError): + importlib.find_loader(name) + + def test_success(self): + # Return the loader found on sys.meta_path. + name = 'some_mod' + with util.uncache(name): + with util.import_state(meta_path=[self.FakeMetaFinder]): + self.assertEqual((name, None), importlib.find_loader(name)) + + def test_success_path(self): + # Searching on a path should work. + name = 'some_mod' + path = 'path to some place' + with util.uncache(name): + with util.import_state(meta_path=[self.FakeMetaFinder]): + self.assertEqual((name, path), + importlib.find_loader(name, path)) + + def test_nothing(self): + # None is returned upon failure to find a loader. + self.assertIsNone(importlib.find_loader('nevergoingtofindthismodule')) + + class InvalidateCacheTests(unittest.TestCase): def test_method_called(self): @@ -114,7 +162,7 @@ def test_main(): from test.support import run_unittest - run_unittest(ImportModuleTests) + run_unittest(ImportModuleTests, FindLoaderTests, InvalidateCacheTests) if __name__ == '__main__': diff --git a/Lib/pyclbr.py b/Lib/pyclbr.py --- a/Lib/pyclbr.py +++ b/Lib/pyclbr.py @@ -39,8 +39,10 @@ lineno -- the line in the file on which the class statement occurred """ +import io +import os import sys -import imp +import importlib import tokenize from token import NAME, DEDENT, OP from operator import itemgetter @@ -133,19 +135,24 @@ # Search the path for the module f = None if inpackage is not None: - f, fname, (_s, _m, ty) = imp.find_module(module, path) + search_path = path else: - f, fname, (_s, _m, ty) = imp.find_module(module, path + sys.path) - if ty == imp.PKG_DIRECTORY: - dict['__path__'] = [fname] - path = [fname] + path - f, fname, (_s, _m, ty) = imp.find_module('__init__', [fname]) + search_path = path + sys.path + loader = importlib.find_loader(fullmodule, search_path) + fname = loader.get_filename(fullmodule) _modules[fullmodule] = dict - if ty != imp.PY_SOURCE: + if loader.is_package(fullmodule): + dict['__path__'] = [os.path.dirname(fname)] + try: + source = loader.get_source(fullmodule) + if source is None: + return dict + except (AttributeError, ImportError): # not Python source, can't do anything with this module - f.close() return dict + f = io.StringIO(source) + stack = [] # stack of (class, indent) pairs g = tokenize.generate_tokens(f.readline) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #13959: Introduce importlib.find_loader(). + - Issue #14082: shutil.copy2() now copies extended attributes, if possible. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:52:57 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 23:52:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_the_reference_counting?= =?utf8?q?_of_dictkeys_objects_participate_in_refleak_hunting?= Message-ID: http://hg.python.org/cpython/rev/10e8b97d0fd7 changeset: 76892:10e8b97d0fd7 parent: 76888:54b8337c306a user: Antoine Pitrou date: Sat May 12 23:43:44 2012 +0200 summary: Make the reference counting of dictkeys objects participate in refleak hunting (issue #13903). files: Objects/dictobject.c | 16 ++++++++++------ 1 files changed, 10 insertions(+), 6 deletions(-) diff --git a/Objects/dictobject.c b/Objects/dictobject.c --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -261,8 +261,11 @@ PyDict_ClearFreeList(); } -#define DK_INCREF(dk) (++(dk)->dk_refcnt) -#define DK_DECREF(dk) if ((--(dk)->dk_refcnt) == 0) free_keys_object(dk) +#define DK_DEBUG_INCREF _Py_INC_REFTOTAL _Py_REF_DEBUG_COMMA +#define DK_DEBUG_DECREF _Py_DEC_REFTOTAL _Py_REF_DEBUG_COMMA + +#define DK_INCREF(dk) (DK_DEBUG_INCREF ++(dk)->dk_refcnt) +#define DK_DECREF(dk) if (DK_DEBUG_DECREF (--(dk)->dk_refcnt) == 0) free_keys_object(dk) #define DK_SIZE(dk) ((dk)->dk_size) #define DK_MASK(dk) (((dk)->dk_size)-1) #define IS_POWER_OF_2(x) (((x) & (x-1)) == 0) @@ -324,7 +327,7 @@ PyErr_NoMemory(); return NULL; } - dk->dk_refcnt = 1; + DK_DEBUG_INCREF dk->dk_refcnt = 1; dk->dk_size = size; dk->dk_usable = USABLE_FRACTION(size); ep0 = &dk->dk_entries[0]; @@ -959,7 +962,7 @@ } } assert(oldkeys->dk_refcnt == 1); - PyMem_FREE(oldkeys); + DK_DEBUG_DECREF PyMem_FREE(oldkeys); } return 0; } @@ -1259,7 +1262,7 @@ } else { assert(oldkeys->dk_refcnt == 1); - free_keys_object(oldkeys); + DK_DECREF(oldkeys); } } @@ -1367,7 +1370,8 @@ DK_DECREF(keys); } else { - free_keys_object(keys); + assert(keys->dk_refcnt == 1); + DK_DECREF(keys); } if (numfree < PyDict_MAXFREELIST && Py_TYPE(mp) == &PyDict_Type) free_list[numfree++] = mp; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:52:58 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 23:52:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/964cfda9ccc5 changeset: 76893:964cfda9ccc5 parent: 76892:10e8b97d0fd7 parent: 76889:8d85f9920878 user: Antoine Pitrou date: Sat May 12 23:43:55 2012 +0200 summary: Merge files: Lib/test/test_shutil.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_shutil.py b/Lib/test/test_shutil.py --- a/Lib/test/test_shutil.py +++ b/Lib/test/test_shutil.py @@ -343,7 +343,7 @@ orig_setxattr = os.setxattr os.setxattr = _raise_on_user_foo shutil._copyxattr(src, dst) - self.assertEqual(['user.bar'], os.listxattr(dst)) + self.assertIn('user.bar', os.listxattr(dst)) finally: os.setxattr = orig_setxattr -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 12 23:52:58 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sat, 12 May 2012 23:52:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/5bd55fb3e091 changeset: 76894:5bd55fb3e091 parent: 76893:964cfda9ccc5 parent: 76891:7bf8ac742d2f user: Antoine Pitrou date: Sat May 12 23:44:59 2012 +0200 summary: Merge files: Doc/library/importlib.rst | 14 ++++++ Lib/importlib/__init__.py | 24 ++++++++++ Lib/importlib/test/test_api.py | 50 +++++++++++++++++++++- Lib/pyclbr.py | 25 +++++++--- Lib/test/test_unicode.py | 8 --- Misc/NEWS | 2 + 6 files changed, 105 insertions(+), 18 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -86,6 +86,20 @@ that was imported (e.g. ``pkg.mod``), while :func:`__import__` returns the top-level package or module (e.g. ``pkg``). +.. function:: find_loader(name, path=None) + + Find the loader for a module, optionally within the specified *path*. If the + module is in :attr:`sys.modules`, then ``sys.modules[name].__loader__`` is + returned (unless the loader would be ``None``, in which case + :exc:`ValueError` is raised). Otherwise a search using :attr:`sys.meta_path` + is done. ``None`` is returned if no loader is found. + + A dotted name does not have its parent's implicitly imported. If that is + desired (although not nessarily required to find the loader, it will most + likely be needed if the loader actually is used to load the module), then + you will have to import the packages containing the module prior to calling + this function. + .. function:: invalidate_caches() Invalidate the internal caches of the finders stored at diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py --- a/Lib/importlib/__init__.py +++ b/Lib/importlib/__init__.py @@ -29,6 +29,30 @@ finder.invalidate_caches() +def find_loader(name, path=None): + """Find the loader for the specified module. + + First, sys.modules is checked to see if the module was already imported. If + so, then sys.modules[name].__loader__ is returned. If that happens to be + set to None, then ValueError is raised. If the module is not in + sys.modules, then sys.meta_path is searched for a suitable loader with the + value of 'path' given to the finders. None is returned if no loader could + be found. + + Dotted names do not have their parent packages implicitly imported. + + """ + try: + loader = sys.modules[name].__loader__ + if loader is None: + raise ValueError('{}.__loader__ is None'.format(name)) + else: + return loader + except KeyError: + pass + return _bootstrap._find_module(name, path) + + def import_module(name, package=None): """Import a module. diff --git a/Lib/importlib/test/test_api.py b/Lib/importlib/test/test_api.py --- a/Lib/importlib/test/test_api.py +++ b/Lib/importlib/test/test_api.py @@ -85,6 +85,54 @@ self.assertEqual(b_load_count, 1) +class FindLoaderTests(unittest.TestCase): + + class FakeMetaFinder: + @staticmethod + def find_module(name, path=None): return name, path + + def test_sys_modules(self): + # If a module with __loader__ is in sys.modules, then return it. + name = 'some_mod' + with util.uncache(name): + module = imp.new_module(name) + loader = 'a loader!' + module.__loader__ = loader + sys.modules[name] = module + found = importlib.find_loader(name) + self.assertEqual(loader, found) + + def test_sys_modules_loader_is_None(self): + # If sys.modules[name].__loader__ is None, raise ValueError. + name = 'some_mod' + with util.uncache(name): + module = imp.new_module(name) + module.__loader__ = None + sys.modules[name] = module + with self.assertRaises(ValueError): + importlib.find_loader(name) + + def test_success(self): + # Return the loader found on sys.meta_path. + name = 'some_mod' + with util.uncache(name): + with util.import_state(meta_path=[self.FakeMetaFinder]): + self.assertEqual((name, None), importlib.find_loader(name)) + + def test_success_path(self): + # Searching on a path should work. + name = 'some_mod' + path = 'path to some place' + with util.uncache(name): + with util.import_state(meta_path=[self.FakeMetaFinder]): + self.assertEqual((name, path), + importlib.find_loader(name, path)) + + def test_nothing(self): + # None is returned upon failure to find a loader. + self.assertIsNone(importlib.find_loader('nevergoingtofindthismodule')) + + class InvalidateCacheTests(unittest.TestCase): def test_method_called(self): @@ -114,7 +162,7 @@ def test_main(): from test.support import run_unittest - run_unittest(ImportModuleTests) + run_unittest(ImportModuleTests, FindLoaderTests, InvalidateCacheTests) if __name__ == '__main__': diff --git a/Lib/pyclbr.py b/Lib/pyclbr.py --- a/Lib/pyclbr.py +++ b/Lib/pyclbr.py @@ -39,8 +39,10 @@ lineno -- the line in the file on which the class statement occurred """ +import io +import os import sys -import imp +import importlib import tokenize from token import NAME, DEDENT, OP from operator import itemgetter @@ -133,19 +135,24 @@ # Search the path for the module f = None if inpackage is not None: - f, fname, (_s, _m, ty) = imp.find_module(module, path) + search_path = path else: - f, fname, (_s, _m, ty) = imp.find_module(module, path + sys.path) - if ty == imp.PKG_DIRECTORY: - dict['__path__'] = [fname] - path = [fname] + path - f, fname, (_s, _m, ty) = imp.find_module('__init__', [fname]) + search_path = path + sys.path + loader = importlib.find_loader(fullmodule, search_path) + fname = loader.get_filename(fullmodule) _modules[fullmodule] = dict - if ty != imp.PY_SOURCE: + if loader.is_package(fullmodule): + dict['__path__'] = [os.path.dirname(fname)] + try: + source = loader.get_source(fullmodule) + if source is None: + return dict + except (AttributeError, ImportError): # not Python source, can't do anything with this module - f.close() return dict + f = io.StringIO(source) + stack = [] # stack of (class, indent) pairs g = tokenize.generate_tokens(f.readline) diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -1422,14 +1422,6 @@ self.assertRaises(TypeError, str, b"hello", "test.unicode2") self.assertRaises(TypeError, "hello".encode, "test.unicode1") self.assertRaises(TypeError, "hello".encode, "test.unicode2") - # executes PyUnicode_Encode() - import imp - self.assertRaises( - ImportError, - imp.find_module, - "non-existing module", - ["non-existing dir"] - ) # Error handling (wrong arguments) self.assertRaises(TypeError, "hello".encode, 42, 42, 42) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Issue #13959: Introduce importlib.find_loader(). + - Issue #14082: shutil.copy2() now copies extended attributes, if possible. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun May 13 05:49:44 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 13 May 2012 05:49:44 +0200 Subject: [Python-checkins] Daily reference leaks (5bd55fb3e091): sum=0 Message-ID: results for 5bd55fb3e091 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogspMuiB', '-x'] From python-checkins at python.org Sun May 13 10:06:44 2012 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 13 May 2012 10:06:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314366=3A_Support_l?= =?utf8?q?zma_compression_in_zip_files=2E?= Message-ID: http://hg.python.org/cpython/rev/fccdcd83708a changeset: 76895:fccdcd83708a user: Martin v. L?wis date: Sun May 13 10:06:36 2012 +0200 summary: Issue #14366: Support lzma compression in zip files. Patch by Serhiy Storchaka. files: Doc/library/zipfile.rst | 26 +++- Lib/test/support.py | 9 +- Lib/test/test_zipfile.py | 125 ++++++++++++++++++++++++++- Lib/zipfile.py | 121 ++++++++++++++++++++++--- Misc/NEWS | 3 + 5 files changed, 257 insertions(+), 27 deletions(-) diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -97,12 +97,20 @@ .. versionadded:: 3.3 +.. data:: ZIP_LZMA + + The numeric constant for the LZMA compression method. This requires the + lzma module. + + .. versionadded:: 3.3 + .. note:: The ZIP file format specification has included support for bzip2 compression - since 2001. However, some tools (including older Python releases) do not - support it, and may either refuse to process the ZIP file altogether, or - fail to extract individual files. + since 2001, and for LZMA compression since 2006. However, some tools + (including older Python releases) do not support these compression + methods, and may either refuse to process the ZIP file altogether, + or fail to extract individual files. .. seealso:: @@ -133,11 +141,11 @@ adding a ZIP archive to another file (such as :file:`python.exe`). If *mode* is ``a`` and the file does not exist at all, it is created. *compression* is the ZIP compression method to use when writing the archive, - and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`; or - :const:`ZIP_DEFLATED`; unrecognized - values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED` or - :const:`ZIP_BZIP2` is specified but the corresponded module - (:mod:`zlib` or :mod:`bz2`) is not available, :exc:`RuntimeError` + and should be :const:`ZIP_STORED`, :const:`ZIP_DEFLATED`, + :const:`ZIP_BZIP2` or :const:`ZIP_LZMA`; unrecognized + values will cause :exc:`RuntimeError` to be raised. If :const:`ZIP_DEFLATED`, + :const:`ZIP_BZIP2` or :const:`ZIP_LZMA` is specified but the corresponded module + (:mod:`zlib`, :mod:`bz2` or :mod:`lzma`) is not available, :exc:`RuntimeError` is also raised. The default is :const:`ZIP_STORED`. If *allowZip64* is ``True`` zipfile will create ZIP files that use the ZIP64 extensions when the zipfile is larger than 2 GB. If it is false (the default) :mod:`zipfile` @@ -161,7 +169,7 @@ Added the ability to use :class:`ZipFile` as a context manager. .. versionchanged:: 3.3 - Added support for :mod:`bzip2` compression. + Added support for :mod:`bzip2` and :mod:`lzma` compression. .. method:: ZipFile.close() diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -45,6 +45,11 @@ except ImportError: bz2 = None +try: + import lzma +except ImportError: + lzma = None + __all__ = [ "Error", "TestFailed", "ResourceDenied", "import_module", "verbose", "use_resources", "max_memuse", "record_original_stdout", @@ -62,7 +67,7 @@ "get_attribute", "swap_item", "swap_attr", "requires_IEEE_754", "TestHandler", "Matcher", "can_symlink", "skip_unless_symlink", "import_fresh_module", "requires_zlib", "PIPE_MAX_SIZE", "failfast", - "anticipate_failure", "run_with_tz", "requires_bz2" + "anticipate_failure", "run_with_tz", "requires_bz2", "requires_lzma" ] class Error(Exception): @@ -513,6 +518,8 @@ requires_bz2 = unittest.skipUnless(bz2, 'requires bz2') +requires_lzma = unittest.skipUnless(lzma, 'requires lzma') + is_jython = sys.platform.startswith('java') # Filename used for testing diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -13,7 +13,7 @@ from random import randint, random from unittest import skipUnless -from test.support import TESTFN, run_unittest, findfile, unlink, requires_zlib, requires_bz2 +from test.support import TESTFN, run_unittest, findfile, unlink, requires_zlib, requires_bz2, requires_lzma TESTFN2 = TESTFN + "2" TESTFNDIR = TESTFN + "d" @@ -361,6 +361,55 @@ self.assertEqual(openobj.read(1), b'1') self.assertEqual(openobj.read(1), b'2') + @requires_lzma + def test_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_open_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_open_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_random_open_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_random_open_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_readline_read_lzma(self): + # Issue #7610: calls to readline() interleaved with calls to read(). + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_readline_read_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_readline_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_readline_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_readlines_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_readlines_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_iterlines_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_iterlines_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_low_compression_lzma(self): + """Check for cases where compressed data is larger than original.""" + # Create the ZIP archive + with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_LZMA) as zipfp: + zipfp.writestr("strfile", '12') + + # Get an open object for strfile + with zipfile.ZipFile(TESTFN2, "r", zipfile.ZIP_LZMA) as zipfp: + with zipfp.open("strfile") as openobj: + self.assertEqual(openobj.read(1), b'1') + self.assertEqual(openobj.read(1), b'2') + def test_absolute_arcnames(self): with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_STORED) as zipfp: zipfp.write(TESTFN, "/absolute") @@ -508,6 +557,13 @@ info = zipfp.getinfo('b.txt') self.assertEqual(info.compress_type, zipfile.ZIP_BZIP2) + @requires_lzma + def test_writestr_compression_lzma(self): + zipfp = zipfile.ZipFile(TESTFN2, "w") + zipfp.writestr("b.txt", "hello world", compress_type=zipfile.ZIP_LZMA) + info = zipfp.getinfo('b.txt') + self.assertEqual(info.compress_type, zipfile.ZIP_LZMA) + def zip_test_writestr_permissions(self, f, compression): # Make sure that writestr creates files with mode 0600, # when it is passed a name rather than a ZipInfo instance. @@ -686,6 +742,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_test(f, zipfile.ZIP_BZIP2) + @requires_lzma + def test_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_test(f, zipfile.ZIP_LZMA) + def test_absolute_arcnames(self): with zipfile.ZipFile(TESTFN2, "w", zipfile.ZIP_STORED, allowZip64=True) as zipfp: @@ -826,6 +887,16 @@ b'\x00 \x80\x80\x81\x00\x00\x00\x00afilePK' b'\x05\x06\x00\x00\x00\x00\x01\x00\x01\x003\x00\x00\x00[\x00' b'\x00\x00\x00\x00'), + zipfile.ZIP_LZMA: ( + b'PK\x03\x04\x14\x03\x00\x00\x0e\x00nu\x0c=FA' + b'KE\x1b\x00\x00\x00n\x00\x00\x00\x05\x00\x00\x00af' + b'ile\t\x04\x05\x00]\x00\x00\x00\x04\x004\x19I' + b'\xee\x8d\xe9\x17\x89:3`\tq!.8\x00PK' + b'\x01\x02\x14\x03\x14\x03\x00\x00\x0e\x00nu\x0c=FA' + b'KE\x1b\x00\x00\x00n\x00\x00\x00\x05\x00\x00\x00\x00\x00' + b'\x00\x00\x00\x00 \x80\x80\x81\x00\x00\x00\x00afil' + b'ePK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x003\x00\x00' + b'\x00>\x00\x00\x00\x00\x00'), } def test_unsupported_version(self): @@ -1104,6 +1175,10 @@ def test_testzip_with_bad_crc_bzip2(self): self.check_testzip_with_bad_crc(zipfile.ZIP_BZIP2) + @requires_lzma + def test_testzip_with_bad_crc_lzma(self): + self.check_testzip_with_bad_crc(zipfile.ZIP_LZMA) + def check_read_with_bad_crc(self, compression): """Tests that files with bad CRCs raise a BadZipFile exception when read.""" zipdata = self.zips_with_bad_crc[compression] @@ -1136,6 +1211,10 @@ def test_read_with_bad_crc_bzip2(self): self.check_read_with_bad_crc(zipfile.ZIP_BZIP2) + @requires_lzma + def test_read_with_bad_crc_lzma(self): + self.check_read_with_bad_crc(zipfile.ZIP_LZMA) + def check_read_return_size(self, compression): # Issue #9837: ZipExtFile.read() shouldn't return more bytes # than requested. @@ -1160,6 +1239,10 @@ def test_read_return_size_bzip2(self): self.check_read_return_size(zipfile.ZIP_BZIP2) + @requires_lzma + def test_read_return_size_lzma(self): + self.check_read_return_size(zipfile.ZIP_LZMA) + def test_empty_zipfile(self): # Check that creating a file in 'w' or 'a' mode and closing without # adding any files to the archives creates a valid empty ZIP file @@ -1306,6 +1389,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_test(f, zipfile.ZIP_BZIP2) + @requires_lzma + def test_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_test(f, zipfile.ZIP_LZMA) + def zip_open_test(self, f, compression): self.make_test_archive(f, compression) @@ -1351,6 +1439,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_open_test(f, zipfile.ZIP_BZIP2) + @requires_lzma + def test_open_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_open_test(f, zipfile.ZIP_LZMA) + def zip_random_open_test(self, f, compression): self.make_test_archive(f, compression) @@ -1384,6 +1477,11 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.zip_random_open_test(f, zipfile.ZIP_BZIP2) + @requires_lzma + def test_random_open_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.zip_random_open_test(f, zipfile.ZIP_LZMA) + @requires_zlib class TestsWithMultipleOpens(unittest.TestCase): @@ -1628,6 +1726,31 @@ for f in (TESTFN2, TemporaryFile(), io.BytesIO()): self.iterlines_test(f, zipfile.ZIP_BZIP2) + @requires_lzma + def test_read_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.read_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_readline_read_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.readline_read_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_readline_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.readline_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_readlines_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.readlines_test(f, zipfile.ZIP_LZMA) + + @requires_lzma + def test_iterlines_lzma(self): + for f in (TESTFN2, TemporaryFile(), io.BytesIO()): + self.iterlines_test(f, zipfile.ZIP_LZMA) + def tearDown(self): for sep, fn in self.arcfiles.items(): os.remove(fn) diff --git a/Lib/zipfile.py b/Lib/zipfile.py --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -27,8 +27,13 @@ except ImportError: bz2 = None +try: + import lzma # We may need its compression method +except ImportError: + lzma = None + __all__ = ["BadZipFile", "BadZipfile", "error", - "ZIP_STORED", "ZIP_DEFLATED", "ZIP_BZIP2", + "ZIP_STORED", "ZIP_DEFLATED", "ZIP_BZIP2", "ZIP_LZMA", "is_zipfile", "ZipInfo", "ZipFile", "PyZipFile", "LargeZipFile"] class BadZipFile(Exception): @@ -52,13 +57,15 @@ ZIP_STORED = 0 ZIP_DEFLATED = 8 ZIP_BZIP2 = 12 +ZIP_LZMA = 14 # Other ZIP compression methods not supported DEFAULT_VERSION = 20 ZIP64_VERSION = 45 BZIP2_VERSION = 46 +LZMA_VERSION = 63 # we recognize (but not necessarily support) all features up to that version -MAX_EXTRACT_VERSION = 46 +MAX_EXTRACT_VERSION = 63 # Below are some formats and associated data for reading/writing headers using # the struct module. The names and structures of headers/records are those used @@ -367,6 +374,8 @@ if self.compress_type == ZIP_BZIP2: min_version = max(BZIP2_VERSION, min_version) + elif self.compress_type == ZIP_LZMA: + min_version = max(LZMA_VERSION, min_version) self.extract_version = max(min_version, self.extract_version) self.create_version = max(min_version, self.create_version) @@ -480,6 +489,77 @@ return c +class LZMACompressor: + + def __init__(self): + self._comp = None + + def _init(self): + props = lzma.encode_filter_properties({'id': lzma.FILTER_LZMA1}) + self._comp = lzma.LZMACompressor(lzma.FORMAT_RAW, filters=[ + lzma.decode_filter_properties(lzma.FILTER_LZMA1, props) + ]) + return struct.pack(' http://hg.python.org/cpython/rev/e0dcd732055f changeset: 76896:e0dcd732055f branch: 3.2 parent: 76887:04e7264e8fcf user: Sandro Tosi date: Sun May 13 11:01:20 2012 +0200 summary: Issue #14793: fix grammar in bytes object paragraph; patch by Tshepang Lekhonkhobe files: Doc/library/stdtypes.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -819,7 +819,8 @@ also string-specific methods described in the :ref:`string-methods` section. Bytes and bytearray objects contain single bytes -- the former is immutable -while the latter is a mutable sequence. Bytes objects can be constructed the +while the latter is a mutable sequence. +Bytes objects can be constructed by using the constructor, :func:`bytes`, and from literals; use a ``b`` prefix with normal string syntax: ``b'xyzzy'``. To construct byte arrays, use the :func:`bytearray` function. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 11:04:00 2012 From: python-checkins at python.org (sandro.tosi) Date: Sun, 13 May 2012 11:04:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_with_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/9b1f618b648b changeset: 76897:9b1f618b648b parent: 76895:fccdcd83708a parent: 76896:e0dcd732055f user: Sandro Tosi date: Sun May 13 11:01:36 2012 +0200 summary: merge with 3.2 files: Doc/library/stdtypes.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -823,7 +823,8 @@ also string-specific methods described in the :ref:`string-methods` section. Bytes and bytearray objects contain single bytes -- the former is immutable -while the latter is a mutable sequence. Bytes objects can be constructed the +while the latter is a mutable sequence. +Bytes objects can be constructed by using the constructor, :func:`bytes`, and from literals; use a ``b`` prefix with normal string syntax: ``b'xyzzy'``. To construct byte arrays, use the :func:`bytearray` function. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 14:29:42 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 13 May 2012 14:29:42 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Remove_Skip_from_the_csv_e?= =?utf8?q?xperts_=28see_issue_=2314732=29=2E?= Message-ID: http://hg.python.org/devguide/rev/90cf321615e5 changeset: 508:90cf321615e5 user: Antoine Pitrou date: Sun May 13 14:27:16 2012 +0200 summary: Remove Skip from the csv experts (see issue #14732). files: experts.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -86,7 +86,7 @@ copyreg alexandre.vassalotti cProfile crypt jafo* -csv skip.montanaro +csv skip.montanaro (inactive) ctypes theller (inactive), belopolsky, amaury.forgeotdarc, meador.inge curses -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sun May 13 18:19:44 2012 From: python-checkins at python.org (brian.curtin) Date: Sun, 13 May 2012 18:19:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_=2313210=2E_Port_the_Wi?= =?utf8?q?ndows_build_from_VS2008_to_VS2010=2E?= Message-ID: http://hg.python.org/cpython/rev/38d7d944370e changeset: 76898:38d7d944370e user: Brian Curtin date: Sun May 13 11:19:23 2012 -0500 summary: Fix #13210. Port the Windows build from VS2008 to VS2010. files: .hgignore | 1 + Lib/distutils/command/build_ext.py | 2 +- Lib/distutils/command/wininst-10.0-amd64.exe | Bin Lib/distutils/command/wininst-10.0.exe | Bin Lib/packaging/command/build_ext.py | 2 +- Lib/packaging/compiler/msvc9compiler.py | 11 +- Misc/NEWS | 5 + Modules/errnomodule.c | 27 + Objects/exceptions.c | 28 + PCbuild/_bz2.vcproj | 0 PCbuild/_ctypes.vcproj | 0 PCbuild/_ctypes_test.vcproj | 0 PCbuild/_elementtree.vcproj | 0 PCbuild/_hashlib.vcproj | 0 PCbuild/_msi.vcproj | 0 PCbuild/_multiprocessing.vcproj | 0 PCbuild/_socket.vcproj | 0 PCbuild/_sqlite3.vcproj | 0 PCbuild/_ssl.vcproj | 0 PCbuild/_testcapi.vcproj | 0 PCbuild/_tkinter.vcproj | 0 PCbuild/bdist_wininst.vcproj | 0 PCbuild/debug.vsprops | 0 PCbuild/kill_python.vcproj | 0 PCbuild/make_buildinfo.vcproj | 0 PCbuild/make_versioninfo.vcproj | 0 PCbuild/pcbuild.sln | 0 PCbuild/pginstrument.vsprops | 0 PCbuild/pgupdate.vsprops | 0 PCbuild/pyd.vsprops | 0 PCbuild/pyd_d.vsprops | 0 PCbuild/pyexpat.vcproj | 0 PCbuild/pyproject.vsprops | 0 PCbuild/python.vcproj | 0 PCbuild/python3dll.vcproj | 0 PCbuild/pythoncore.vcproj | 0 PCbuild/pythonw.vcproj | 0 PCbuild/release.vsprops | 0 PCbuild/select.vcproj | 0 PCbuild/sqlite3.vcproj | 0 PCbuild/sqlite3.vsprops | 0 PCbuild/ssl.vcproj | 0 PCbuild/unicodedata.vcproj | 0 PCbuild/w9xpopen.vcproj | 0 PCbuild/winsound.vcproj | 0 PCbuild/x64.vsprops | 0 PCbuild/xxlimited.vcproj | 0 PC/dl_nt.c | 10 +- PC/msvcrtmodule.c | 15 +- PC/pyconfig.h | 9 +- PCbuild/_bz2.vcxproj | 263 ++ PCbuild/_bz2.vcxproj.filters | 48 + PCbuild/_ctypes.vcxproj | 283 +++ PCbuild/_ctypes.vcxproj.filters | 65 + PCbuild/_ctypes_test.vcxproj | 195 ++ PCbuild/_ctypes_test.vcxproj.filters | 21 + PCbuild/_decimal.vcxproj | 297 +++ PCbuild/_decimal.vcxproj.filters | 116 + PCbuild/_elementtree.vcxproj | 272 ++ PCbuild/_elementtree.vcxproj.filters | 72 + PCbuild/_hashlib.vcxproj | 283 +++ PCbuild/_hashlib.vcxproj.filters | 13 + PCbuild/_lzma.vcxproj | 252 ++ PCbuild/_lzma.vcxproj.filters | 13 + PCbuild/_msi.vcxproj | 228 ++ PCbuild/_msi.vcxproj.filters | 13 + PCbuild/_multiprocessing.vcxproj | 232 ++ PCbuild/_multiprocessing.vcxproj.filters | 24 + PCbuild/_socket.vcxproj | 231 ++ PCbuild/_socket.vcxproj.filters | 21 + PCbuild/_sqlite3.vcxproj | 276 ++ PCbuild/_sqlite3.vcxproj.filters | 72 + PCbuild/_ssl.vcxproj | 287 +++ PCbuild/_ssl.vcxproj.filters | 13 + PCbuild/_testbuffer.vcxproj | 206 ++ PCbuild/_testbuffer.vcxproj.filters | 13 + PCbuild/_testcapi.vcxproj | 220 ++ PCbuild/_testcapi.vcxproj.filters | 13 + PCbuild/_tkinter.vcxproj | 252 ++ PCbuild/_tkinter.vcxproj.filters | 16 + PCbuild/bdist_wininst.vcxproj | 158 + PCbuild/bdist_wininst.vcxproj.filters | 61 + PCbuild/debug.props | 23 + PCbuild/kill_python.vcxproj | 124 + PCbuild/kill_python.vcxproj.filters | 13 + PCbuild/make_buildinfo.vcxproj | 54 + PCbuild/make_buildinfo.vcxproj.filters | 14 + PCbuild/make_versioninfo.vcxproj | 197 ++ PCbuild/make_versioninfo.vcxproj.filters | 13 + PCbuild/pcbuild.sln | 224 +- PCbuild/pginstrument.props | 38 + PCbuild/pgupdate.props | 16 + PCbuild/pyd.props | 27 + PCbuild/pyd_d.props | 41 + PCbuild/pyexpat.vcxproj | 239 ++ PCbuild/pyexpat.vcxproj.filters | 33 + PCbuild/pyproject.props | 96 + PCbuild/python.vcxproj | 372 +++ PCbuild/python.vcxproj.filters | 26 + PCbuild/python3dll.vcxproj | 184 + PCbuild/python3dll.vcxproj.filters | 32 + PCbuild/pythoncore.vcxproj | 703 +++++++ PCbuild/pythoncore.vcxproj.filters | 932 ++++++++++ PCbuild/pythonw.vcxproj | 350 +++ PCbuild/pythonw.vcxproj.filters | 21 + PCbuild/release.props | 19 + PCbuild/select.vcxproj | 236 ++ PCbuild/select.vcxproj.filters | 13 + PCbuild/sqlite3.props | 16 + PCbuild/sqlite3.vcxproj | 241 ++ PCbuild/sqlite3.vcxproj.filters | 24 + PCbuild/ssl.vcxproj | 221 ++ PCbuild/unicodedata.vcxproj | 224 ++ PCbuild/unicodedata.vcxproj.filters | 24 + PCbuild/w9xpopen.vcxproj | 263 ++ PCbuild/w9xpopen.vcxproj.filters | 13 + PCbuild/winsound.vcxproj | 220 ++ PCbuild/winsound.vcxproj.filters | 14 + PCbuild/x64.props | 26 + PCbuild/xxlimited.vcxproj | 191 ++ PCbuild/xxlimited.vcxproj.filters | 13 + Python/dynload_win.c | 8 + Tools/buildbot/build-amd64.bat | 2 +- Tools/buildbot/build.bat | 2 +- Tools/buildbot/buildmsi.bat | 2 +- Tools/buildbot/clean-amd64.bat | 2 +- Tools/buildbot/clean.bat | 2 +- Tools/buildbot/external-amd64.bat | 2 +- Tools/buildbot/external-common.bat | 6 +- Tools/buildbot/external.bat | 12 +- Tools/msi/msi.py | 32 +- 131 files changed, 10047 insertions(+), 192 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -66,6 +66,7 @@ PCbuild/*.bsc PCbuild/*.user PCbuild/*.suo +PCbuild/*.*sdf PCbuild/Win32-temp-* PCbuild/x64-temp-* __pycache__ diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py --- a/Lib/distutils/command/build_ext.py +++ b/Lib/distutils/command/build_ext.py @@ -197,7 +197,7 @@ # Append the source distribution include and library directories, # this allows distutils on windows to work in the source tree self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC')) - if MSVC_VERSION == 9: + if MSVC_VERSION >= 9: # Use the .lib files for the correct architecture if self.plat_name == 'win32': suffix = '' diff --git a/Lib/distutils/command/wininst-10.0-amd64.exe b/Lib/distutils/command/wininst-10.0-amd64.exe new file mode 100644 index 0000000000000000000000000000000000000000..6fa0dce16315854223a9fefc3ac9f3cf71730a6a GIT binary patch [stripped] diff --git a/Lib/distutils/command/wininst-10.0.exe b/Lib/distutils/command/wininst-10.0.exe new file mode 100644 index 0000000000000000000000000000000000000000..afc3bc6c14847283f32f53327184c6c2806efe0a GIT binary patch [stripped] diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py --- a/Lib/packaging/command/build_ext.py +++ b/Lib/packaging/command/build_ext.py @@ -191,7 +191,7 @@ # Append the source distribution include and library directories, # this allows distutils on windows to work in the source tree self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC')) - if MSVC_VERSION == 9: + if MSVC_VERSION >= 9: # Use the .lib files for the correct architecture if self.plat_name == 'win32': suffix = '' diff --git a/Lib/packaging/compiler/msvc9compiler.py b/Lib/packaging/compiler/msvc9compiler.py --- a/Lib/packaging/compiler/msvc9compiler.py +++ b/Lib/packaging/compiler/msvc9compiler.py @@ -634,11 +634,12 @@ mfid = 2 self._remove_visual_c_ref(temp_manifest) out_arg = '-outputresource:%s;%s' % (output_filename, mfid) - try: - self.spawn(['mt.exe', '-nologo', '-manifest', - temp_manifest, out_arg]) - except PackagingExecError as msg: - raise LinkError(msg) + if self.__version < 10: + try: + self.spawn(['mt.exe', '-nologo', '-manifest', + temp_manifest, out_arg]) + except PackagingExecError as msg: + raise LinkError(msg) else: logger.debug("skipping %s (up-to-date)", output_filename) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -80,6 +80,11 @@ - Issue #14695: Bring Tools/parser/unparse.py support up to date with the Python 3.3 Grammar. +Build +----- + +- Issue #13210: Windows build now uses VS2010, ported from VS2008. + What's New in Python 3.3.0 Alpha 3? =================================== diff --git a/Modules/errnomodule.c b/Modules/errnomodule.c --- a/Modules/errnomodule.c +++ b/Modules/errnomodule.c @@ -7,6 +7,33 @@ #ifdef MS_WINDOWS #define WIN32_LEAN_AND_MEAN #include +/* The following constants were added to errno.h in VS2010 but have + preferred WSA equivalents. */ +#undef EADDRINUSE +#undef EADDRNOTAVAIL +#undef EAFNOSUPPORT +#undef EALREADY +#undef ECONNABORTED +#undef ECONNREFUSED +#undef ECONNRESET +#undef EDESTADDRREQ +#undef EHOSTUNREACH +#undef EINPROGRESS +#undef EISCONN +#undef ELOOP +#undef EMSGSIZE +#undef ENETDOWN +#undef ENETRESET +#undef ENETUNREACH +#undef ENOBUFS +#undef ENOPROTOOPT +#undef ENOTCONN +#undef ENOTSOCK +#undef EOPNOTSUPP +#undef EPROTONOSUPPORT +#undef EPROTOTYPE +#undef ETIMEDOUT +#undef EWOULDBLOCK #endif /* diff --git a/Objects/exceptions.c b/Objects/exceptions.c --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -2317,6 +2317,34 @@ #ifdef MS_WINDOWS #include +/* The following constants were added to errno.h in VS2010 but have + preferred WSA equivalents. */ +#undef EADDRINUSE +#undef EADDRNOTAVAIL +#undef EAFNOSUPPORT +#undef EALREADY +#undef ECONNABORTED +#undef ECONNREFUSED +#undef ECONNRESET +#undef EDESTADDRREQ +#undef EHOSTUNREACH +#undef EINPROGRESS +#undef EISCONN +#undef ELOOP +#undef EMSGSIZE +#undef ENETDOWN +#undef ENETRESET +#undef ENETUNREACH +#undef ENOBUFS +#undef ENOPROTOOPT +#undef ENOTCONN +#undef ENOTSOCK +#undef EOPNOTSUPP +#undef EPROTONOSUPPORT +#undef EPROTOTYPE +#undef ETIMEDOUT +#undef EWOULDBLOCK + #if defined(WSAEALREADY) && !defined(EALREADY) #define EALREADY WSAEALREADY #endif diff --git a/PCbuild/_bz2.vcproj b/PC/VS9.0/_bz2.vcproj copy from PCbuild/_bz2.vcproj copy to PC/VS9.0/_bz2.vcproj diff --git a/PCbuild/_ctypes.vcproj b/PC/VS9.0/_ctypes.vcproj copy from PCbuild/_ctypes.vcproj copy to PC/VS9.0/_ctypes.vcproj diff --git a/PCbuild/_ctypes_test.vcproj b/PC/VS9.0/_ctypes_test.vcproj copy from PCbuild/_ctypes_test.vcproj copy to PC/VS9.0/_ctypes_test.vcproj diff --git a/PCbuild/_elementtree.vcproj b/PC/VS9.0/_elementtree.vcproj copy from PCbuild/_elementtree.vcproj copy to PC/VS9.0/_elementtree.vcproj diff --git a/PCbuild/_hashlib.vcproj b/PC/VS9.0/_hashlib.vcproj copy from PCbuild/_hashlib.vcproj copy to PC/VS9.0/_hashlib.vcproj diff --git a/PCbuild/_msi.vcproj b/PC/VS9.0/_msi.vcproj copy from PCbuild/_msi.vcproj copy to PC/VS9.0/_msi.vcproj diff --git a/PCbuild/_multiprocessing.vcproj b/PC/VS9.0/_multiprocessing.vcproj copy from PCbuild/_multiprocessing.vcproj copy to PC/VS9.0/_multiprocessing.vcproj diff --git a/PCbuild/_socket.vcproj b/PC/VS9.0/_socket.vcproj copy from PCbuild/_socket.vcproj copy to PC/VS9.0/_socket.vcproj diff --git a/PCbuild/_sqlite3.vcproj b/PC/VS9.0/_sqlite3.vcproj copy from PCbuild/_sqlite3.vcproj copy to PC/VS9.0/_sqlite3.vcproj diff --git a/PCbuild/_ssl.vcproj b/PC/VS9.0/_ssl.vcproj copy from PCbuild/_ssl.vcproj copy to PC/VS9.0/_ssl.vcproj diff --git a/PCbuild/_testcapi.vcproj b/PC/VS9.0/_testcapi.vcproj copy from PCbuild/_testcapi.vcproj copy to PC/VS9.0/_testcapi.vcproj diff --git a/PCbuild/_tkinter.vcproj b/PC/VS9.0/_tkinter.vcproj copy from PCbuild/_tkinter.vcproj copy to PC/VS9.0/_tkinter.vcproj diff --git a/PCbuild/bdist_wininst.vcproj b/PC/VS9.0/bdist_wininst.vcproj copy from PCbuild/bdist_wininst.vcproj copy to PC/VS9.0/bdist_wininst.vcproj diff --git a/PCbuild/debug.vsprops b/PC/VS9.0/debug.vsprops rename from PCbuild/debug.vsprops rename to PC/VS9.0/debug.vsprops diff --git a/PCbuild/kill_python.vcproj b/PC/VS9.0/kill_python.vcproj copy from PCbuild/kill_python.vcproj copy to PC/VS9.0/kill_python.vcproj diff --git a/PCbuild/make_buildinfo.vcproj b/PC/VS9.0/make_buildinfo.vcproj copy from PCbuild/make_buildinfo.vcproj copy to PC/VS9.0/make_buildinfo.vcproj diff --git a/PCbuild/make_versioninfo.vcproj b/PC/VS9.0/make_versioninfo.vcproj copy from PCbuild/make_versioninfo.vcproj copy to PC/VS9.0/make_versioninfo.vcproj diff --git a/PCbuild/pcbuild.sln b/PC/VS9.0/pcbuild.sln copy from PCbuild/pcbuild.sln copy to PC/VS9.0/pcbuild.sln diff --git a/PCbuild/pginstrument.vsprops b/PC/VS9.0/pginstrument.vsprops rename from PCbuild/pginstrument.vsprops rename to PC/VS9.0/pginstrument.vsprops diff --git a/PCbuild/pgupdate.vsprops b/PC/VS9.0/pgupdate.vsprops rename from PCbuild/pgupdate.vsprops rename to PC/VS9.0/pgupdate.vsprops diff --git a/PCbuild/pyd.vsprops b/PC/VS9.0/pyd.vsprops rename from PCbuild/pyd.vsprops rename to PC/VS9.0/pyd.vsprops diff --git a/PCbuild/pyd_d.vsprops b/PC/VS9.0/pyd_d.vsprops rename from PCbuild/pyd_d.vsprops rename to PC/VS9.0/pyd_d.vsprops diff --git a/PCbuild/pyexpat.vcproj b/PC/VS9.0/pyexpat.vcproj copy from PCbuild/pyexpat.vcproj copy to PC/VS9.0/pyexpat.vcproj diff --git a/PCbuild/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops rename from PCbuild/pyproject.vsprops rename to PC/VS9.0/pyproject.vsprops diff --git a/PCbuild/python.vcproj b/PC/VS9.0/python.vcproj copy from PCbuild/python.vcproj copy to PC/VS9.0/python.vcproj diff --git a/PCbuild/python3dll.vcproj b/PC/VS9.0/python3dll.vcproj copy from PCbuild/python3dll.vcproj copy to PC/VS9.0/python3dll.vcproj diff --git a/PCbuild/pythoncore.vcproj b/PC/VS9.0/pythoncore.vcproj copy from PCbuild/pythoncore.vcproj copy to PC/VS9.0/pythoncore.vcproj diff --git a/PCbuild/pythonw.vcproj b/PC/VS9.0/pythonw.vcproj copy from PCbuild/pythonw.vcproj copy to PC/VS9.0/pythonw.vcproj diff --git a/PCbuild/release.vsprops b/PC/VS9.0/release.vsprops rename from PCbuild/release.vsprops rename to PC/VS9.0/release.vsprops diff --git a/PCbuild/select.vcproj b/PC/VS9.0/select.vcproj copy from PCbuild/select.vcproj copy to PC/VS9.0/select.vcproj diff --git a/PCbuild/sqlite3.vcproj b/PC/VS9.0/sqlite3.vcproj copy from PCbuild/sqlite3.vcproj copy to PC/VS9.0/sqlite3.vcproj diff --git a/PCbuild/sqlite3.vsprops b/PC/VS9.0/sqlite3.vsprops rename from PCbuild/sqlite3.vsprops rename to PC/VS9.0/sqlite3.vsprops diff --git a/PCbuild/ssl.vcproj b/PC/VS9.0/ssl.vcproj copy from PCbuild/ssl.vcproj copy to PC/VS9.0/ssl.vcproj diff --git a/PCbuild/unicodedata.vcproj b/PC/VS9.0/unicodedata.vcproj copy from PCbuild/unicodedata.vcproj copy to PC/VS9.0/unicodedata.vcproj diff --git a/PCbuild/w9xpopen.vcproj b/PC/VS9.0/w9xpopen.vcproj copy from PCbuild/w9xpopen.vcproj copy to PC/VS9.0/w9xpopen.vcproj diff --git a/PCbuild/winsound.vcproj b/PC/VS9.0/winsound.vcproj copy from PCbuild/winsound.vcproj copy to PC/VS9.0/winsound.vcproj diff --git a/PCbuild/x64.vsprops b/PC/VS9.0/x64.vsprops rename from PCbuild/x64.vsprops rename to PC/VS9.0/x64.vsprops diff --git a/PCbuild/xxlimited.vcproj b/PC/VS9.0/xxlimited.vcproj copy from PCbuild/xxlimited.vcproj copy to PC/VS9.0/xxlimited.vcproj diff --git a/PC/dl_nt.c b/PC/dl_nt.c --- a/PC/dl_nt.c +++ b/PC/dl_nt.c @@ -18,7 +18,8 @@ HMODULE PyWin_DLLhModule = NULL; const char *PyWin_DLLVersionString = dllVersionBuffer; -// Windows "Activation Context" work: +#if HAVE_SXS +// Windows "Activation Context" work. // Our .pyd extension modules are generally built without a manifest (ie, // those included with Python and those built with a default distutils. // This requires we perform some "activation context" magic when loading our @@ -29,6 +30,8 @@ // As an added complication, this magic only works on XP or later - we simply // use the existence (or not) of the relevant function pointers from kernel32. // See bug 4566 (http://python.org/sf/4566) for more details. +// In Visual Studio 2010, side by side assemblies are no longer used by +// default. typedef BOOL (WINAPI * PFN_GETCURRENTACTCTX)(HANDLE *); typedef BOOL (WINAPI * PFN_ACTIVATEACTCTX)(HANDLE, ULONG_PTR *); @@ -75,6 +78,7 @@ if (!(*pfnDeactivateActCtx)(0, cookie)) OutputDebugString("Python failed to de-activate the activation context\n"); } +#endif /* HAVE_SXS */ BOOL WINAPI DllMain (HANDLE hInst, ULONG ul_reason_for_call, @@ -87,17 +91,21 @@ // 1000 is a magic number I picked out of the air. Could do with a #define, I spose... LoadString(hInst, 1000, dllVersionBuffer, sizeof(dllVersionBuffer)); +#if HAVE_SXS // and capture our activation context for use when loading extensions. _LoadActCtxPointers(); if (pfnGetCurrentActCtx && pfnAddRefActCtx) if ((*pfnGetCurrentActCtx)(&PyWin_DLLhActivationContext)) if (!(*pfnAddRefActCtx)(PyWin_DLLhActivationContext)) OutputDebugString("Python failed to load the default activation context\n"); +#endif break; case DLL_PROCESS_DETACH: +#if HAVE_SXS if (pfnReleaseActCtx) (*pfnReleaseActCtx)(PyWin_DLLhActivationContext); +#endif break; } return TRUE; diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c --- a/PC/msvcrtmodule.c +++ b/PC/msvcrtmodule.c @@ -27,6 +27,8 @@ #ifdef _MSC_VER #if _MSC_VER >= 1500 && _MSC_VER < 1600 #include +#elif _MSC_VER >= 1600 +#include #endif #endif @@ -464,7 +466,7 @@ PyInit_msvcrt(void) { int st; - PyObject *d; + PyObject *d, *version; PyObject *m = PyModule_Create(&msvcrtmodule); if (m == NULL) return NULL; @@ -494,6 +496,7 @@ #endif /* constants for the crt versions */ + (void)st; #ifdef _VC_ASSEMBLY_PUBLICKEYTOKEN st = PyModule_AddStringConstant(m, "VC_ASSEMBLY_PUBLICKEYTOKEN", _VC_ASSEMBLY_PUBLICKEYTOKEN); @@ -510,5 +513,15 @@ if (st < 0) return NULL; #endif + /* constants for the 2010 crt versions */ +#if defined(_VC_CRT_MAJOR_VERSION) && defined (_VC_CRT_MINOR_VERSION) && defined(_VC_CRT_BUILD_VERSION) && defined(_VC_CRT_RBUILD_VERSION) + version = PyUnicode_FromFormat("%d.%d.%d.%d", _VC_CRT_MAJOR_VERSION, + _VC_CRT_MINOR_VERSION, + _VC_CRT_BUILD_VERSION, + _VC_CRT_RBUILD_VERSION); + st = PyModule_AddObject(m, "CRT_ASSEMBLY_VERSION", version); + if (st < 0) return NULL; +#endif + return m; } diff --git a/PC/pyconfig.h b/PC/pyconfig.h --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -215,14 +215,19 @@ #define copysign _copysign #define hypot _hypot -#endif /* _MSC_VER */ +/* Side by Side assemblies supported in VS 2005 and VS 2008 but not 2010*/ +#if _MSC_VER >= 1400 && _MSC_VER < 1600 +#define HAVE_SXS 1 +#endif /* define some ANSI types that are not defined in earlier Win headers */ -#if defined(_MSC_VER) && _MSC_VER >= 1200 +#if _MSC_VER >= 1200 /* This file only exists in VC 6.0 or higher */ #include #endif +#endif /* _MSC_VER */ + /* ------------------------------------------------------------------------*/ /* The Borland compiler defines __BORLANDC__ */ /* XXX These defines are likely incomplete, but should be easy to fix. */ diff --git a/PCbuild/_bz2.vcxproj b/PCbuild/_bz2.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_bz2.vcxproj @@ -0,0 +1,263 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {73FCD2BD-F133-46B7-8EC1-144CD82A59D5} + bz2 + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + + + + + X64 + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + + + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + + + + + X64 + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + + + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + + + + + X64 + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + MachineX64 + + + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + + + + + X64 + + + $(bz2Dir);%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + + + 0x1D170000 + MachineX64 + + + + + + + + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_bz2.vcxproj.filters b/PCbuild/_bz2.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_bz2.vcxproj.filters @@ -0,0 +1,48 @@ +? + + + + {f53a859d-dad2-4d5b-ae41-f28d8b571f5a} + + + {7e0bed05-ae33-43b7-8797-656455bbb7f3} + + + {ed574b89-6983-4cdf-9f98-fe7048d9e89c} + + + + + Source Files + + + bzip2 1.0.5 Source Files + + + bzip2 1.0.5 Source Files + + + bzip2 1.0.5 Source Files + + + bzip2 1.0.5 Source Files + + + bzip2 1.0.5 Source Files + + + bzip2 1.0.5 Source Files + + + bzip2 1.0.5 Source Files + + + + + bzip2 1.0.5 Header Files + + + bzip2 1.0.5 Header Files + + + \ No newline at end of file diff --git a/PCbuild/_ctypes.vcxproj b/PCbuild/_ctypes.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_ctypes.vcxproj @@ -0,0 +1,283 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {0E9791DB-593A-465F-98BC-681011311618} + _ctypes + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.40219.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + 0x1D1A0000 + + + + + X64 + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + 0x1D1A0000 + $(OutDir)python33_d.lib;%(AdditionalDependencies) + + + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) + NotSet + 0x1D1A0000 + + + + + X64 + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) + NotSet + 0x1D1A0000 + $(OutDir)python33.lib;%(AdditionalDependencies) + + + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) + NotSet + 0x1D1A0000 + + + + + X64 + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) + NotSet + 0x1D1A0000 + MachineX64 + + + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) + NotSet + 0x1D1A0000 + + + + + X64 + + + ..\Modules\_ctypes\libffi_msvc;%(AdditionalIncludeDirectories) + + + /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) + NotSet + 0x1D1A0000 + MachineX64 + + + + + + + + + + + + + + + + + + + + + true + true + true + true + + + + + true + ml64 /nologo /c /Zi /Fo "$(IntDir)win64.obj" "%(FullPath)" + + $(IntDir)win64.obj;%(Outputs) + true + ml64 /nologo /c /Fo "$(IntDir)win64.obj" "%(FullPath)" + + $(IntDir)win64.obj;%(Outputs) + true + ml64 /nologo /c /Fo "$(IntDir)win64.obj" "%(FullPath)" + + $(IntDir)win64.obj;%(Outputs) + true + ml64 /nologo /c /Fo "$(IntDir)win64.obj" "%(FullPath)" + + $(IntDir)win64.obj;%(Outputs) + + + + + + \ No newline at end of file diff --git a/PCbuild/_ctypes.vcxproj.filters b/PCbuild/_ctypes.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_ctypes.vcxproj.filters @@ -0,0 +1,65 @@ +? + + + + {806081ee-2af0-48d0-a83e-ee02a74baa0f} + + + {dbdea1f2-ad8b-44ca-b782-fcf65d91559b} + + + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_ctypes_test.vcxproj b/PCbuild/_ctypes_test.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_ctypes_test.vcxproj @@ -0,0 +1,195 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {9EC7190A-249F-4180-A900-548FDCF3055F} + _ctypes_test + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + X64 + + + + + X64 + + + + + X64 + + + MachineX64 + + + + + X64 + + + MachineX64 + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_ctypes_test.vcxproj.filters b/PCbuild/_ctypes_test.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_ctypes_test.vcxproj.filters @@ -0,0 +1,21 @@ +? + + + + {8fd70119-5481-4e5d-b187-d0b14eb27e38} + + + {38abc486-e143-49dc-8cf0-8aefab0e0d3d} + + + + + Header Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_decimal.vcxproj b/PCbuild/_decimal.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_decimal.vcxproj @@ -0,0 +1,297 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {0E9791DB-593A-465F-98BC-681011311617} + _decimal + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.40219.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + $(SolutionDir)\amd64\ + $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ + + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_32 /DPPRO /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + 0x1D1A0000 + + + + + X64 + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_64 /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + 0x1D1A0000 + $(OutDir)python33_d.lib;%(AdditionalDependencies) + + + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_32 /DPPRO /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + NotSet + 0x1D1A0000 + + + + + X64 + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_64 /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;..\Include;..\PC;%(AdditionalIncludeDirectories) + + + NotSet + 0x1D1A0000 + $(OutDir)python33.lib;%(AdditionalDependencies) + + + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_32 /DPPRO /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + NotSet + 0x1D1A0000 + + + + + X64 + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_64 /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + NotSet + 0x1D1A0000 + MachineX64 + + + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_32 /DPPRO /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + NotSet + 0x1D1A0000 + + + + + X64 + + + /D_CRT_SECURE_NO_WARNINGS /DCONFIG_64 /DMASM %(AdditionalOptions) + ..\Modules\_decimal;..\Modules\_decimal\libmpdec;%(AdditionalIncludeDirectories) + + + NotSet + 0x1D1A0000 + MachineX64 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + true + ml64 /nologo /c /Zi /Fo "$(IntDir)vcdiv64.obj" "%(FullPath)" + + $(IntDir)vcdiv64.obj;%(Outputs) + true + ml64 /nologo /c /Fo "$(IntDir)vcdiv64.obj" "%(FullPath)" + + $(IntDir)vcdiv64.obj;%(Outputs) + true + ml64 /nologo /c /Fo "$(IntDir)vcdiv64.obj" "%(FullPath)" + + $(IntDir)vcdiv64.obj;%(Outputs) + true + ml64 /nologo /c /Zi /Fo "$(IntDir)vcdiv64.obj" "%(FullPath)" + + $(IntDir)vcdiv64.obj;%(Outputs) + + + + + + \ No newline at end of file diff --git a/PCbuild/_decimal.vcxproj.filters b/PCbuild/_decimal.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_decimal.vcxproj.filters @@ -0,0 +1,116 @@ +? + + + + {f35a78a6-3ef0-4e36-bd8b-afaef22fbb7c} + + + {138089f8-faba-494f-b6ed-051f31fbaf2d} + + + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_elementtree.vcxproj @@ -0,0 +1,272 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {17E1E049-C309-4D79-843F-AE483C264AEA} + _elementtree + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + + + + + X64 + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + + + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + + + + + X64 + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + + + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + + + + + X64 + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + MachineX64 + + + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + + + + + X64 + + + ..\Modules\expat;%(AdditionalIncludeDirectories) + XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;USE_PYEXPAT_CAPI;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + 0x1D100000 + MachineX64 + + + + + + + + + + + + + + + + + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_elementtree.vcxproj.filters b/PCbuild/_elementtree.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_elementtree.vcxproj.filters @@ -0,0 +1,72 @@ +? + + + + {643d8607-d024-40fe-8583-1823c96430f0} + + + {7b5335ad-059f-486f-85e4-f4757e26a9bf} + + + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_hashlib.vcxproj b/PCbuild/_hashlib.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_hashlib.vcxproj @@ -0,0 +1,283 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {447F05A8-F581-4CAC-A466-5AC7936E207E} + _hashlib + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + MachineX64 + + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + MachineX64 + + + + + + + + {b11d750f-cd1f-4a96-85ce-e69a5c5259f9} + false + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + {e5b04cc0-eb4c-42ab-b4dc-18ef95f864b0} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_hashlib.vcxproj.filters b/PCbuild/_hashlib.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_hashlib.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {cc45963d-bd25-4eb8-bdba-a5507090bca4} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_lzma.vcxproj b/PCbuild/_lzma.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_lzma.vcxproj @@ -0,0 +1,252 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {F9D71780-F393-11E0-BE50-0800200C9A66} + lzma + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_i486\liblzma.a;%(AdditionalDependencies) + + + + + X64 + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_x86-64\liblzma.a;%(AdditionalDependencies) + + + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_i486\liblzma.a;%(AdditionalDependencies) + + + + + X64 + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_x86-64\liblzma.a;%(AdditionalDependencies) + + + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_i486\liblzma.a;%(AdditionalDependencies) + + + + + X64 + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_x86-64\liblzma.a;%(AdditionalDependencies) + MachineX64 + + + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_i486\liblzma.a;%(AdditionalDependencies) + + + + + X64 + + + $(lzmaDir)\include;%(AdditionalIncludeDirectories) + WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC;%(PreprocessorDefinitions) + + + $(lzmaDir)\bin_x86-64\liblzma.a;%(AdditionalDependencies) + MachineX64 + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_lzma.vcxproj.filters b/PCbuild/_lzma.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_lzma.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {53e68eda-39fc-4336-a658-dc5f5d598760} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_msi.vcxproj b/PCbuild/_msi.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_msi.vcxproj @@ -0,0 +1,228 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {31FFC478-7B4A-43E8-9954-8D03E2187E9C} + _msi + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + + + + + X64 + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + + + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + + + + + X64 + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + + + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + + + + + X64 + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + MachineX64 + + + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + + + + + X64 + + + fci.lib;msi.lib;rpcrt4.lib;%(AdditionalDependencies) + 0x1D160000 + MachineX64 + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_msi.vcxproj.filters b/PCbuild/_msi.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_msi.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {bdef7710-e433-4ac0-84e0-14f34454bd3e} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_multiprocessing.vcxproj b/PCbuild/_multiprocessing.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_multiprocessing.vcxproj @@ -0,0 +1,232 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {9E48B300-37D1-11DD-8C41-005056C00008} + _multiprocessing + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + MachineX64 + + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + MachineX64 + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_multiprocessing.vcxproj.filters b/PCbuild/_multiprocessing.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_multiprocessing.vcxproj.filters @@ -0,0 +1,24 @@ +? + + + + {623c956c-1893-43d9-a7dc-96e4fef20f93} + + + {34615a62-f999-4659-83f5-19d17a644530} + + + + + Header Files + + + + + Source Files + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_socket.vcxproj b/PCbuild/_socket.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_socket.vcxproj @@ -0,0 +1,231 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {86937F53-C189-40EF-8CE8-8759D8E7D480} + _socket + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + MachineX64 + + + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + + + + + X64 + + + ws2_32.lib;%(AdditionalDependencies) + 0x1e1D0000 + MachineX64 + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_socket.vcxproj.filters b/PCbuild/_socket.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_socket.vcxproj.filters @@ -0,0 +1,21 @@ +? + + + + {1452207f-707c-4e84-b532-307193a0fd85} + + + {1edfe0d0-7b9d-4dc8-a335-b21fef7cc77a} + + + + + Header Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_sqlite3.vcxproj @@ -0,0 +1,276 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {13CECB97-4119-4316-9D42-8534019A5A44} + _sqlite3 + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + + + + + X64 + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + + + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + + + + + X64 + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + + + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + + + + + X64 + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + MachineX64 + + + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + + + + + X64 + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + MODULE_NAME="sqlite3";%(PreprocessorDefinitions) + + + 0x1e180000 + MachineX64 + + + + + + + + + + + + + + + + + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + {a1a295e5-463c-437f-81ca-1f32367685da} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_sqlite3.vcxproj.filters b/PCbuild/_sqlite3.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_sqlite3.vcxproj.filters @@ -0,0 +1,72 @@ +? + + + + {dac8ab3b-ce16-4bef-bef9-76463a01f5c4} + + + {814b187d-44ad-4f2b-baa7-18ca8a8a6a77} + + + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + Header Files + + + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_ssl.vcxproj b/PCbuild/_ssl.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_ssl.vcxproj @@ -0,0 +1,287 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {C6E20F84-3247-4AD6-B051-B073268F73BA} + _ssl + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + MachineX64 + + + + + $(opensslDir)\inc32;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out32\libeay32.lib;$(opensslDir)\out32\ssleay32.lib;%(AdditionalDependencies) + + + + + X64 + + + $(opensslDir)\inc64;%(AdditionalIncludeDirectories) + + + + + + + ws2_32.lib;$(opensslDir)\out64\libeay32.lib;$(opensslDir)\out64\ssleay32.lib;%(AdditionalDependencies) + MachineX64 + + + + + + + + {b11d750f-cd1f-4a96-85ce-e69a5c5259f9} + false + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + {e5b04cc0-eb4c-42ab-b4dc-18ef95f864b0} + false + + + {86937f53-c189-40ef-8ce8-8759d8e7d480} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_ssl.vcxproj.filters b/PCbuild/_ssl.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_ssl.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {695348f7-e9f6-4fe1-bc03-5f08ffc8095b} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_testbuffer.vcxproj b/PCbuild/_testbuffer.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_testbuffer.vcxproj @@ -0,0 +1,206 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {A2697BD3-28C1-4AEC-9106-8B748639FD16} + _testbuffer + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.40219.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + $(OutDir)python33_d.lib;%(AdditionalDependencies) + + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + $(OutDir)python33.lib;%(AdditionalDependencies) + + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + MachineX64 + + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + MachineX64 + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/_testbuffer.vcxproj.filters b/PCbuild/_testbuffer.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_testbuffer.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {8d232240-921a-4bc2-87c3-93ffd3462f0a} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_testcapi.vcxproj @@ -0,0 +1,220 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {6901D91C-6E48-4BB7-9FEC-700C8131DF1D} + _testcapi + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + MachineX64 + + + + + 0x1e1F0000 + + + + + X64 + + + 0x1e1F0000 + MachineX64 + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_testcapi.vcxproj.filters b/PCbuild/_testcapi.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_testcapi.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {a76a90d8-8e8b-4c36-8f58-8bd46abe9f5e} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/_tkinter.vcxproj b/PCbuild/_tkinter.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/_tkinter.vcxproj @@ -0,0 +1,252 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {4946ECAC-2E69-4BF8-A90A-F5136F5094DF} + _tkinter + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + + + + $(tcltkDir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltkLibDebug);%(AdditionalDependencies) + + + + + X64 + + + $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltk64LibDebug);%(AdditionalDependencies) + + + + + $(tcltkDir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltkLib);%(AdditionalDependencies) + + + + + X64 + + + $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltk64Lib);%(AdditionalDependencies) + + + + + $(tcltkDir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltkLib);%(AdditionalDependencies) + + + + + X64 + + + $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltk64Lib);%(AdditionalDependencies) + MachineX64 + + + + + $(tcltkDir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltkLib);%(AdditionalDependencies) + + + + + X64 + + + $(tcltk64Dir)\include;%(AdditionalIncludeDirectories) + WITH_APPINIT;%(PreprocessorDefinitions) + + + $(tcltk64Lib);%(AdditionalDependencies) + MachineX64 + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/_tkinter.vcxproj.filters b/PCbuild/_tkinter.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/_tkinter.vcxproj.filters @@ -0,0 +1,16 @@ +? + + + + {b9ce64dd-cb95-472d-bbe8-5583b2cd375b} + + + + + Source Files + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/bdist_wininst.vcxproj b/PCbuild/bdist_wininst.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/bdist_wininst.vcxproj @@ -0,0 +1,158 @@ +? + + + + Release + Win32 + + + Release + x64 + + + + {EB1C19C1-1F18-421E-9735-CAEE69DC6A3C} + wininst + + + + Application + false + NotSet + + + Application + false + NotSet + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + ..\lib\distutils\command\ + false + ..\lib\distutils\command\ + $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ + false + AllRules.ruleset + + + AllRules.ruleset + + + .exe + .exe + + + + NDEBUG;%(PreprocessorDefinitions) + true + true + Win32 + .\..\lib\distutils\command\wininst.tlb + + + + + MinSpace + OnlyExplicitInline + ..\PC\bdist_wininst;..\Include;..\Modules\zlib;%(AdditionalIncludeDirectories) + _CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + true + MultiThreaded + true + Level3 + true + + + NDEBUG;%(PreprocessorDefinitions) + 0x0000 + ..\PC;..\PC\bdist_wininst;..\Include;%(AdditionalIncludeDirectories) + + + comctl32.lib;imagehlp.lib;%(AdditionalDependencies) + ..\lib\distutils\command\wininst-10.0.exe + true + LIBC;%(IgnoreSpecificDefaultLibraries) + ..\lib\distutils\command\wininst-10.0.pdb + Windows + false + + + MachineX86 + + + + + NDEBUG;%(PreprocessorDefinitions) + true + true + X64 + .\..\lib\distutils\command\wininst.tlb + + + + + MinSpace + OnlyExplicitInline + ..\PC\bdist_wininst;..\Include;..\Modules\zlib;%(AdditionalIncludeDirectories) + _CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;%(PreprocessorDefinitions) + true + MultiThreaded + true + Level3 + true + + + NDEBUG;%(PreprocessorDefinitions) + 0x0000 + ..\PC;..\PC\bdist_wininst;..\Include;%(AdditionalIncludeDirectories) + + + comctl32.lib;imagehlp.lib;%(AdditionalDependencies) + ..\lib\distutils\command\wininst-10.0-amd64.exe + true + LIBC;%(IgnoreSpecificDefaultLibraries) + ..\lib\distutils\command\wininst-9.0-amd64.pdb + Windows + false + + + MachineX64 + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/bdist_wininst.vcxproj.filters b/PCbuild/bdist_wininst.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/bdist_wininst.vcxproj.filters @@ -0,0 +1,61 @@ +? + + + + {293b1092-03ad-4b7c-acb9-c4ab62e52f55} + cpp;c;cxx;rc;def;r;odl;idl;hpj;bat + + + {0edc0406-282f-4dbc-b60e-a867c34a2a31} + + + {ea0c0f0e-3b73-474e-a999-e9689d032ccc} + h;hpp;hxx;hm;inl + + + {0c77c1cf-3f87-4f87-bd86-b425211c2181} + ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe + + + + + Source Files + + + Source Files + + + Source Files\zlib + + + Source Files\zlib + + + Source Files\zlib + + + Source Files\zlib + + + Source Files\zlib + + + Source Files\zlib + + + + + Header Files + + + + + Resource Files + + + + + Resource Files + + + \ No newline at end of file diff --git a/PCbuild/debug.props b/PCbuild/debug.props new file mode 100644 --- /dev/null +++ b/PCbuild/debug.props @@ -0,0 +1,23 @@ +? + + + $(OutDir)kill_python_d.exe + + + <_ProjectFileVersion>10.0.30319.1 + $(ProjectName)_d + + + + _DEBUG;%(PreprocessorDefinitions) + + + $(OutDir)$(TargetName)$(TargetExt) + + + + + $(KillPythonExe) + + + \ No newline at end of file diff --git a/PCbuild/kill_python.vcxproj b/PCbuild/kill_python.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/kill_python.vcxproj @@ -0,0 +1,124 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} + kill_python + Win32Proj + + + + Application + NotSet + true + + + Application + NotSet + + + Application + NotSet + true + + + Application + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .exe + .exe + .exe + .exe + + + + $(OutDir)$(ProjectName)_d.exe + Console + + + + + X64 + + + $(OutDir)$(ProjectName)_d.exe + Console + + + + + Console + + + + + X64 + + + Console + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/kill_python.vcxproj.filters b/PCbuild/kill_python.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/kill_python.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {48606591-c8b6-41a5-95c5-9a0890c5504f} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/make_buildinfo.vcxproj b/PCbuild/make_buildinfo.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/make_buildinfo.vcxproj @@ -0,0 +1,54 @@ +? + + + + Release + Win32 + + + + {C73F0EC1-358B-4177-940F-0846AC8B04CD} + make_buildinfo + Win32Proj + + + + Application + NotSet + + + + + + + + + + + + <_ProjectFileVersion>10.0.40219.1 + AllRules.ruleset + + + .exe + + + + Disabled + OnlyExplicitInline + _CONSOLE;%(PreprocessorDefinitions) + MultiThreaded + + + $(OutDir)make_buildinfo.exe + $(TargetDir)$(TargetName).pdb + Console + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/make_buildinfo.vcxproj.filters b/PCbuild/make_buildinfo.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/make_buildinfo.vcxproj.filters @@ -0,0 +1,14 @@ +? + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cxx;def;odl;idl;hpj;bat;asm;asmx + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/make_versioninfo.vcxproj b/PCbuild/make_versioninfo.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/make_versioninfo.vcxproj @@ -0,0 +1,197 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + Release + Win32 + + + Release + x64 + + + + {F0E0541E-F17D-430B-97C4-93ADF0DD284E} + make_versioninfo + + + + Application + false + NotSet + + + Application + false + MultiByte + + + Application + + + Application + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.40219.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .exe + .exe + + + + Build PC/pythonnt_rc(_d).h + cd $(SolutionDir) +make_versioninfo.exe > ..\PC\pythonnt_rc.h + + $(SolutionDir)..\PC\pythonnt_rc.h;%(Outputs) + + + MaxSpeed + OnlyExplicitInline + true + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + $(SolutionDir)make_versioninfo.exe + $(TargetDir)$(TargetName).pdb + Console + 0x1d000000 + + + cd $(SolutionDir) +make_versioninfo.exe > ..\PC\python_nt.h + + + + + + Build PC/pythonnt_rc(_d).h + cd $(SolutionDir) +make_versioninfo.exe > ..\PC\pythonnt_rc.h + + $(SolutionDir)..\PC\pythonnt_rc.h;%(Outputs) + + + MaxSpeed + OnlyExplicitInline + true + _CONSOLE;%(PreprocessorDefinitions) + + + $(SolutionDir)make_versioninfo.exe + + + cd $(SolutionDir) +make_versioninfo.exe > ..\PC\python_nt.h + + + + + + Build PC/pythonnt_rc(_d).h + cd $(SolutionDir) +make_versioninfo_d.exe > ..\PC\pythonnt_rc_d.h + + $(SolutionDir)..\PC\pythonnt_rc_d.h;%(Outputs) + + + Disabled + OnlyExplicitInline + false + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + $(SolutionDir)make_versioninfo_d.exe + $(TargetDir)$(TargetName).pdb + Console + 0x1d000000 + + + cd $(SolutionDir) +make_versioninfo_d.exe > ..\PC\python_nt_d.h + + + + + + Build PC/pythonnt_rc(_d).h + cd $(SolutionDir) +make_versioninfo_d.exe > ..\PC\pythonnt_rc_d.h + + $(SolutionDir)..\PC\pythonnt_rc_d.h;%(Outputs) + + + X64 + + + Disabled + OnlyExplicitInline + false + _CONSOLE;%(PreprocessorDefinitions) + + + $(SolutionDir)make_versioninfo_d.exe + MachineX64 + + + cd $(SolutionDir) +make_versioninfo_d.exe > ..\PC\python_nt_d.h + + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/make_versioninfo.vcxproj.filters b/PCbuild/make_versioninfo.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/make_versioninfo.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {e4180954-c3a5-4749-b9a4-31b804ee4fa8} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -1,153 +1,81 @@ -Microsoft Visual Studio Solution File, Format Version 10.00 -# Visual Studio 2008 -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "python", "python.vcproj", "{B11D750F-CD1F-4A96-85CE-E69A5C5259F9}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - {E9E0A1F6-0009-4E8C-B8F8-1B8F5D49A058} = {E9E0A1F6-0009-4E8C-B8F8-1B8F5D49A058} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "make_versioninfo", "make_versioninfo.vcproj", "{F0E0541E-F17D-430B-97C4-93ADF0DD284E}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pythoncore", "pythoncore.vcproj", "{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}" - ProjectSection(ProjectDependencies) = postProject - {F0E0541E-F17D-430B-97C4-93ADF0DD284E} = {F0E0541E-F17D-430B-97C4-93ADF0DD284E} - {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} = {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} - {C73F0EC1-358B-4177-940F-0846AC8B04CD} = {C73F0EC1-358B-4177-940F-0846AC8B04CD} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pythonw", "pythonw.vcproj", "{F4229CC3-873C-49AE-9729-DD308ED4CD4A}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "w9xpopen", "w9xpopen.vcproj", "{E9E0A1F6-0009-4E8C-B8F8-1B8F5D49A058}" - ProjectSection(ProjectDependencies) = postProject - {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} = {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "make_buildinfo", "make_buildinfo.vcproj", "{C73F0EC1-358B-4177-940F-0846AC8B04CD}" -EndProject +Microsoft Visual Studio Solution File, Format Version 11.00 +# Visual Studio 2010 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{553EC33E-9816-4996-A660-5D6186A0B0B3}" ProjectSection(SolutionItems) = preProject ..\Modules\getbuildinfo.c = ..\Modules\getbuildinfo.c readme.txt = readme.txt EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "winsound", "winsound.vcproj", "{28B5D777-DDF2-4B6B-B34F-31D938813856}" +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "python", "python.vcxproj", "{B11D750F-CD1F-4A96-85CE-E69A5C5259F9}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "make_versioninfo", "make_versioninfo.vcxproj", "{F0E0541E-F17D-430B-97C4-93ADF0DD284E}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pythoncore", "pythoncore.vcxproj", "{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pythonw", "pythonw.vcxproj", "{F4229CC3-873C-49AE-9729-DD308ED4CD4A}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "w9xpopen", "w9xpopen.vcxproj", "{E9E0A1F6-0009-4E8C-B8F8-1B8F5D49A058}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "make_buildinfo", "make_buildinfo.vcxproj", "{C73F0EC1-358B-4177-940F-0846AC8B04CD}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "winsound", "winsound.vcxproj", "{28B5D777-DDF2-4B6B-B34F-31D938813856}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_decimal", "_decimal.vcxproj", "{0E9791DB-593A-465F-98BC-681011311617}" ProjectSection(ProjectDependencies) = postProject {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_decimal", "_decimal.vcproj", "{0E9791DB-593A-465F-98BC-681011311617}" +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes", "_ctypes.vcxproj", "{0E9791DB-593A-465F-98BC-681011311618}" ProjectSection(ProjectDependencies) = postProject {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes", "_ctypes.vcproj", "{0E9791DB-593A-465F-98BC-681011311618}" +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes_test", "_ctypes_test.vcxproj", "{9EC7190A-249F-4180-A900-548FDCF3055F}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_elementtree", "_elementtree.vcxproj", "{17E1E049-C309-4D79-843F-AE483C264AEA}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_msi", "_msi.vcxproj", "{31FFC478-7B4A-43E8-9954-8D03E2187E9C}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_socket", "_socket.vcxproj", "{86937F53-C189-40EF-8CE8-8759D8E7D480}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_sqlite3", "_sqlite3.vcxproj", "{13CECB97-4119-4316-9D42-8534019A5A44}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ssl", "_ssl.vcxproj", "{C6E20F84-3247-4AD6-B051-B073268F73BA}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testcapi", "_testcapi.vcxproj", "{6901D91C-6E48-4BB7-9FEC-700C8131DF1D}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_tkinter", "_tkinter.vcxproj", "{4946ECAC-2E69-4BF8-A90A-F5136F5094DF}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_bz2", "_bz2.vcxproj", "{73FCD2BD-F133-46B7-8EC1-144CD82A59D5}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "select", "select.vcxproj", "{18CAE28C-B454-46C1-87A0-493D91D97F03}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_lzma", "_lzma.vcxproj", "{F9D71780-F393-11E0-BE50-0800200C9A66}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "unicodedata", "unicodedata.vcxproj", "{ECC7CEAC-A5E5-458E-BB9E-2413CC847881}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pyexpat", "pyexpat.vcxproj", "{D06B6426-4762-44CC-8BAD-D79052507F2F}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bdist_wininst", "bdist_wininst.vcxproj", "{EB1C19C1-1F18-421E-9735-CAEE69DC6A3C}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_hashlib", "_hashlib.vcxproj", "{447F05A8-F581-4CAC-A466-5AC7936E207E}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "sqlite3", "sqlite3.vcxproj", "{A1A295E5-463C-437F-81CA-1F32367685DA}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_multiprocessing", "_multiprocessing.vcxproj", "{9E48B300-37D1-11DD-8C41-005056C00008}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "ssl", "ssl.vcxproj", "{E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "kill_python", "kill_python.vcxproj", "{6DE10744-E396-40A5-B4E2-1B69AA7C8D31}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "python3dll", "python3dll.vcxproj", "{885D4898-D08D-4091-9C40-C700CFE3FC5A}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "xxlimited", "xxlimited.vcxproj", "{F749B822-B489-4CA5-A3AD-CE078F5F338A}" ProjectSection(ProjectDependencies) = postProject {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} EndProjectSection EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes_test", "_ctypes_test.vcproj", "{9EC7190A-249F-4180-A900-548FDCF3055F}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_elementtree", "_elementtree.vcproj", "{17E1E049-C309-4D79-843F-AE483C264AEA}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_msi", "_msi.vcproj", "{31FFC478-7B4A-43E8-9954-8D03E2187E9C}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_socket", "_socket.vcproj", "{86937F53-C189-40EF-8CE8-8759D8E7D480}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_sqlite3", "_sqlite3.vcproj", "{13CECB97-4119-4316-9D42-8534019A5A44}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - {A1A295E5-463C-437F-81CA-1F32367685DA} = {A1A295E5-463C-437F-81CA-1F32367685DA} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ssl", "_ssl.vcproj", "{C6E20F84-3247-4AD6-B051-B073268F73BA}" - ProjectSection(ProjectDependencies) = postProject - {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} = {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} - {86937F53-C189-40EF-8CE8-8759D8E7D480} = {86937F53-C189-40EF-8CE8-8759D8E7D480} - {E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0} = {E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0} - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testcapi", "_testcapi.vcproj", "{6901D91C-6E48-4BB7-9FEC-700C8131DF1D}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_tkinter", "_tkinter.vcproj", "{4946ECAC-2E69-4BF8-A90A-F5136F5094DF}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_bz2", "_bz2.vcproj", "{73FCD2BD-F133-46B7-8EC1-144CD82A59D5}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_lzma", "_lzma.vcproj", "{F9D71780-F393-11E0-BE50-0800200C9A66}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "select", "select.vcproj", "{18CAE28C-B454-46C1-87A0-493D91D97F03}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "unicodedata", "unicodedata.vcproj", "{ECC7CEAC-A5E5-458E-BB9E-2413CC847881}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pyexpat", "pyexpat.vcproj", "{D06B6426-4762-44CC-8BAD-D79052507F2F}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "bdist_wininst", "bdist_wininst.vcproj", "{EB1C19C1-1F18-421E-9735-CAEE69DC6A3C}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_hashlib", "_hashlib.vcproj", "{447F05A8-F581-4CAC-A466-5AC7936E207E}" - ProjectSection(ProjectDependencies) = postProject - {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} = {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} - {E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0} = {E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0} - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "sqlite3", "sqlite3.vcproj", "{A1A295E5-463C-437F-81CA-1F32367685DA}" - ProjectSection(ProjectDependencies) = postProject - {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} = {6DE10744-E396-40A5-B4E2-1B69AA7C8D31} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_multiprocessing", "_multiprocessing.vcproj", "{9E48B300-37D1-11DD-8C41-005056C00008}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "ssl", "ssl.vcproj", "{E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0}" - ProjectSection(ProjectDependencies) = postProject - {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} = {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "kill_python", "kill_python.vcproj", "{6DE10744-E396-40A5-B4E2-1B69AA7C8D31}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "python3dll", "python3dll.vcproj", "{885D4898-D08D-4091-9C40-C700CFE3FC5A}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "xxlimited", "xxlimited.vcproj", "{F749B822-B489-4CA5-A3AD-CE078F5F338A}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testbuffer", "_testbuffer.vcproj", "{A2697BD3-28C1-4AEC-9106-8B748639FD16}" +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testbuffer", "_testbuffer.vcxproj", "{A2697BD3-28C1-4AEC-9106-8B748639FD16}" ProjectSection(ProjectDependencies) = postProject {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} EndProjectSection @@ -452,6 +380,22 @@ {73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|Win32.Build.0 = Release|Win32 {73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|x64.ActiveCfg = Release|x64 {73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|x64.Build.0 = Release|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.ActiveCfg = Debug|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.Build.0 = Debug|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|x64.ActiveCfg = Debug|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|x64.Build.0 = Debug|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|x64.Build.0 = PGUpdate|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|Win32.ActiveCfg = Release|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|Win32.Build.0 = Release|Win32 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|x64.ActiveCfg = Release|x64 + {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|x64.Build.0 = Release|x64 {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|Win32.ActiveCfg = Debug|Win32 {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|Win32.Build.0 = Debug|Win32 {F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|x64.ActiveCfg = Debug|x64 @@ -468,22 +412,6 @@ {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|Win32.Build.0 = Release|Win32 {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|x64.ActiveCfg = Release|x64 {F9D71780-F393-11E0-BE50-0800200C9A66}.Release|x64.Build.0 = Release|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.ActiveCfg = Debug|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.Build.0 = Debug|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|x64.ActiveCfg = Debug|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|x64.Build.0 = Debug|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGInstrument|x64.Build.0 = PGInstrument|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|x64.ActiveCfg = PGUpdate|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.PGUpdate|x64.Build.0 = PGUpdate|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|Win32.ActiveCfg = Release|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|Win32.Build.0 = Release|Win32 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|x64.ActiveCfg = Release|x64 - {18CAE28C-B454-46C1-87A0-493D91D97F03}.Release|x64.Build.0 = Release|x64 {ECC7CEAC-A5E5-458E-BB9E-2413CC847881}.Debug|Win32.ActiveCfg = Debug|Win32 {ECC7CEAC-A5E5-458E-BB9E-2413CC847881}.Debug|Win32.Build.0 = Debug|Win32 {ECC7CEAC-A5E5-458E-BB9E-2413CC847881}.Debug|x64.ActiveCfg = Debug|x64 diff --git a/PCbuild/pginstrument.props b/PCbuild/pginstrument.props new file mode 100644 --- /dev/null +++ b/PCbuild/pginstrument.props @@ -0,0 +1,38 @@ +? + + + $(SolutionDir)$(Platform)-pgi\ + + + <_ProjectFileVersion>10.0.30319.1 + $(OutDirPGI)\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + + + + MaxSpeed + OnlyExplicitInline + false + Size + true + false + true + true + + + false + + + true + true + PGInstrument + $(SolutionDir)$(Platform)-pgi\$(TargetName).pgd + $(OutDirPGI)\$(TargetName).lib + + + + + $(OutDirPGI) + + + \ No newline at end of file diff --git a/PCbuild/pgupdate.props b/PCbuild/pgupdate.props new file mode 100644 --- /dev/null +++ b/PCbuild/pgupdate.props @@ -0,0 +1,16 @@ +? + + + + + + <_ProjectFileVersion>10.0.30319.1 + $(SolutionDir)$(PlatformName)-pgo\ + + + + %(AdditionalManifestDependencies) + PGUpdate + + + \ No newline at end of file diff --git a/PCbuild/pyd.props b/PCbuild/pyd.props new file mode 100644 --- /dev/null +++ b/PCbuild/pyd.props @@ -0,0 +1,27 @@ +? + + + + + + + <_ProjectFileVersion>10.0.30319.1 + false + false + + + + Py_BUILD_CORE_MODULE;%(PreprocessorDefinitions) + MultiThreadedDLL + + + $(OutDir)$(ProjectName).pyd + $(OutDir)$(ProjectName).pdb + $(OutDir)$(TargetName).lib + + + + + + + \ No newline at end of file diff --git a/PCbuild/pyd_d.props b/PCbuild/pyd_d.props new file mode 100644 --- /dev/null +++ b/PCbuild/pyd_d.props @@ -0,0 +1,41 @@ +? + + + + + + + $(SolutionDir)python_d.exe + + + <_ProjectFileVersion>10.0.30319.1 + false + false + false + .pyd + $(ProjectName)_d + + + + Disabled + Default + false + Py_BUILD_CORE_MODULE;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + + + $(OutDir)$(ProjectName)_d.pyd + $(OutDir)$(ProjectName)_d.pdb + $(OutDir)$(TargetName).lib + + + + + + + + + $(PythonExe) + + + \ No newline at end of file diff --git a/PCbuild/pyexpat.vcxproj b/PCbuild/pyexpat.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/pyexpat.vcxproj @@ -0,0 +1,239 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {D06B6426-4762-44CC-8BAD-D79052507F2F} + pyexpat + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + + + X64 + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + + + X64 + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + + + X64 + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + MachineX64 + + + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + + + X64 + + + .\..\Modules\expat;%(AdditionalIncludeDirectories) + PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_NS;XML_DTD;BYTEORDER=1234;XML_CONTEXT_BYTES=1024;XML_STATIC;HAVE_MEMMOVE;%(PreprocessorDefinitions) + + + MachineX64 + + + + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/pyexpat.vcxproj.filters b/PCbuild/pyexpat.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/pyexpat.vcxproj.filters @@ -0,0 +1,33 @@ +? + + + + {ddae77a6-7ca0-4a1b-b71c-deea5f4025de} + + + {5af9d40c-fc46-4640-ad84-3d1dd34a71d7} + + + + + Header Files + + + Header Files + + + + + Source Files + + + Source Files + + + Source Files + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props new file mode 100644 --- /dev/null +++ b/PCbuild/pyproject.props @@ -0,0 +1,96 @@ +? + + + python33 + $(SolutionDir)\python.exe + ..\.. + $(externalsDir)\sqlite-3.7.4 + $(externalsDir)\bzip2-1.0.5 + $(externalsDir)\xz-5.0.3 + $(externalsDir)\openssl-1.0.0a + $(externalsDir)\tcltk + $(externalsDir)\tcltk64 + $(tcltkDir)\lib\tcl85.lib;$(tcltkDir)\lib\tk85.lib + $(tcltkDir)\lib\tcl85g.lib;$(tcltkDir)\lib\tk85g.lib + $(tcltk64Dir)\lib\tcl85.lib;$(tcltk64Dir)\lib\tk85.lib + $(tcltk64Dir)\lib\tcl85g.lib;$(tcltk64Dir)\lib\tk85g.lib + + + <_ProjectFileVersion>10.0.30319.1 + $(SolutionDir)\ + $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ + false + .dll + + + + MaxSpeed + OnlyExplicitInline + true + ..\Include; ..\PC;%(AdditionalIncludeDirectories) + _WIN32;%(PreprocessorDefinitions) + true + + + MultiThreaded + true + Level3 + ProgramDatabase + Default + + + $(OutDir);%(AdditionalLibraryDirectories) + true + $(OutDir)$(TargetName).pdb + Windows + false + + + MachineX86 + + + ..\PC;..\Include;%(AdditionalIncludeDirectories) + + + + + $(PyDllName) + + + $(PythonExe) + + + $(externalsDir) + + + $(sqlite3Dir) + + + $(bz2Dir) + + + $(lzmaDir) + + + $(opensslDir) + + + $(tcltkDir) + + + $(tcltk64Dir) + + + $(tcltkLib) + + + $(tcltkLibDebug) + + + $(tcltk64Lib) + + + $(tcltk64LibDebug) + + + diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/python.vcxproj @@ -0,0 +1,372 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {B11D750F-CD1F-4A96-85CE-E69A5C5259F9} + + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + NotSet + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + MultiByte + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .exe + .exe + .exe + .exe + + + + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)python.exe + Console + 2000000 + 0x1d000000 + + + + + X64 + + + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)python.exe + Console + 2000000 + 0x1d000000 + + + + + Disabled + false + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + true + Default + + + _DEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + $(OutDir)python_d.exe + Console + 2000000 + 0x1d000000 + + + + + X64 + + + Disabled + false + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + true + Default + + + _DEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + $(OutDir)python_d.exe + Console + 2100000 + 0x1d000000 + + + + + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)python.exe + Console + 2000000 + 0x1d000000 + + + + + + + X64 + + + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)python.exe + Console + 2000000 + 0x1d000000 + + + MachineX64 + + + + + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)python.exe + Console + 2000000 + 0x1d000000 + + + + + + + X64 + + + %(AdditionalIncludeDirectories) + _CONSOLE;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)python.exe + Console + 2000000 + 0x1d000000 + + + MachineX64 + + + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + {e9e0a1f6-0009-4e8c-b8f8-1b8f5d49a058} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/python.vcxproj.filters b/PCbuild/python.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/python.vcxproj.filters @@ -0,0 +1,26 @@ +? + + + + {2d690795-de83-4a33-8235-3c5dafe45efa} + + + {8b010a19-5b29-45f1-a8a0-f672e2bbb11a} + + + + + Resource Files + + + + + Resource Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/python3dll.vcxproj b/PCbuild/python3dll.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/python3dll.vcxproj @@ -0,0 +1,184 @@ +? + + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {885D4898-D08D-4091-9C40-C700CFE3FC5A} + python3dll + Win32Proj + + + + Makefile + + + Makefile + + + Makefile + true + + + Makefile + + + Makefile + + + Makefile + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) rebuild + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) clean + $(OutDir)python3.dll + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) rebuild + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) clean + $(OutDir)python3.dll + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) rebuild + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) clean + $(OutDir)python3.dll + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) rebuild + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) clean + $(OutDir)python3.dll + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) rebuild + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x86 OutDir=$(OutDir) clean + $(OutDir)python3.dll + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) rebuild + cd $(ProjectDir)\..\PC +nmake /f python3.mak MACHINE=x64 OutDir=$(OutDir) clean + $(OutDir)python3.dll + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/python3dll.vcxproj.filters b/PCbuild/python3dll.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/python3dll.vcxproj.filters @@ -0,0 +1,32 @@ +? + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hpp;hxx;hm;inl;inc;xsd + + + {67DA6AB6-F800-4c08-8B7A-83BB121AAD01} + rc;ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe;resx;tiff;tif;png;wav + + + + + Source Files + + + + + Source Files + + + + + Resource Files + + + \ No newline at end of file diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/pythoncore.vcxproj @@ -0,0 +1,703 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} + pythoncore + + + + DynamicLibrary + false + + + DynamicLibrary + false + + + DynamicLibrary + false + NotSet + + + DynamicLibrary + false + + + DynamicLibrary + false + + + DynamicLibrary + false + + + DynamicLibrary + false + + + DynamicLibrary + false + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + $(PyDllName)_d + $(PyDllName) + $(PyDllName)_d + $(PyDllName) + + + + /Zm200 %(AdditionalOptions) + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDLL + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName).dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName).pdb + 0x1e000000 + $(OutDir)$(PyDllName).lib + + + + + X64 + + + /Zm200 %(AdditionalOptions) + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDLL + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName).dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName).pdb + 0x1e000000 + $(OutDir)$(PyDllName).lib + + + + + /Zm200 %(AdditionalOptions) + Disabled + Default + false + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + + + _DEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Debug "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName)_d.dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName)_d.pdb + 0x1e000000 + $(OutDir)$(PyDllName)_d.lib + + + + + X64 + + + /Zm200 %(AdditionalOptions) + Disabled + Default + false + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + + + _DEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Debug "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName)_d.dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName)_d.pdb + 0x1e000000 + $(OutDir)$(PyDllName)_d.lib + + + + + /Zm200 %(AdditionalOptions) + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDLL + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName).dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName).pdb + 0x1e000000 + $(OutDirPGI)$(PyDllName).lib + + + + + X64 + + + /Zm200 %(AdditionalOptions) + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDLL + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName).dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName).pdb + 0x1e000000 + $(OutDirPGI)$(PyDllName).lib + MachineX64 + + + + + /Zm200 %(AdditionalOptions) + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDLL + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName).dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName).pdb + 0x1e000000 + $(OutDirPGI)$(PyDllName).lib + + + + + X64 + + + /Zm200 %(AdditionalOptions) + ..\Python;..\Modules\zlib;%(AdditionalIncludeDirectories) + _USRDLL;Py_BUILD_CORE;Py_ENABLE_SHARED;WIN32;%(PreprocessorDefinitions) + MultiThreadedDLL + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + ..\Include;%(AdditionalIncludeDirectories) + + + Generate build information... + "$(SolutionDir)make_buildinfo.exe" Release "$(IntDir)" + + + $(IntDir)getbuildinfo.o;%(AdditionalDependencies) + $(OutDir)$(PyDllName).dll + libc;%(IgnoreSpecificDefaultLibraries) + $(OutDir)$(PyDllName).pdb + 0x1e000000 + $(OutDirPGI)$(PyDllName).lib + MachineX64 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) + _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) + _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) + _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + {6de10744-e396-40a5-b4e2-1b69aa7c8d31} + false + + + {c73f0ec1-358b-4177-940f-0846ac8b04cd} + false + + + {f0e0541e-f17d-430b-97c4-93adf0dd284e} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/pythoncore.vcxproj.filters @@ -0,0 +1,932 @@ +? + + + + {086b0afb-270c-4603-a02a-63d46f0b2b92} + + + {8e81609f-13ca-4eae-9fdb-f8af20c710c7} + + + {8787c5bb-bab6-4586-a42e-4a27c7b3ffb6} + + + {5d6d2d6c-9e61-4a1d-b0b2-5cc2f446d69e} + + + {9f12c4b1-322e-431e-abf1-e02550f50032} + + + {ab29a558-143d-4fe7-a039-b431fb429856} + + + {97349fee-0abf-48b0-a8f5-771bf39b8aee} + + + {ea21fc98-de89-4746-a979-c5616964329a} + + + {f2696406-14bc-48bd-90c5-e93ab82a21ac} + + + {c3e03a5c-56c7-45fd-8543-e5d2326b907d} + + + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Include + + + Modules + + + Modules + + + Modules + + + Modules\_io + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Parser + + + Parser + + + PC + + + PC + + + Python + + + Python + + + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules + + + Modules\_io + + + Modules\_io + + + Modules\_io + + + Modules\_io + + + Modules\_io + + + Modules\_io + + + Modules\_io + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\zlib + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Modules\cjkcodecs + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Objects + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + Parser + + + PC + + + PC + + + PC + + + PC + + + PC + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + Python + + + PC + + + + + Resource Files + + + \ No newline at end of file diff --git a/PCbuild/pythonw.vcxproj b/PCbuild/pythonw.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/pythonw.vcxproj @@ -0,0 +1,350 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {F4229CC3-873C-49AE-9729-DD308ED4CD4A} + + + + Application + false + + + Application + false + + + Application + false + + + Application + false + NotSet + + + Application + false + + + Application + false + + + Application + false + + + Application + false + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .exe + .exe + .exe + .pyd + + + + Disabled + false + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + Default + + + _DEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw_d.exe + 2000000 + 0x1d000000 + MachineX86 + + + + + X64 + + + Disabled + false + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + Default + + + _DEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw_d.exe + 2000000 + 0x1d000000 + + + + + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw.exe + 2000000 + 0x1d000000 + MachineX86 + + + + + X64 + + + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw.exe + 2000000 + 0x1d000000 + + + + + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw.exe + 2000000 + 0x1d000000 + + + MachineX86 + + + + + X64 + + + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw.exe + 2000000 + 0x1d000000 + + + MachineX64 + + + + + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw.exe + 2000000 + 0x1d000000 + + + MachineX86 + + + + + X64 + + + %(AdditionalIncludeDirectories) + _WINDOWS;%(PreprocessorDefinitions) + true + MultiThreadedDLL + true + Default + + + NDEBUG;%(PreprocessorDefinitions) + 0x0409 + + + $(OutDir)pythonw.exe + 2000000 + 0x1d000000 + + + MachineX64 + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/pythonw.vcxproj.filters b/PCbuild/pythonw.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/pythonw.vcxproj.filters @@ -0,0 +1,21 @@ +? + + + + {0434cf11-a311-4a92-8a6c-4164aa79a7f2} + + + {e1d8ea6b-c65d-42f4-9eed-6010846ed378} + + + + + Resource Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/release.props b/PCbuild/release.props new file mode 100644 --- /dev/null +++ b/PCbuild/release.props @@ -0,0 +1,19 @@ +? + + + $(OutDir)kill_python.exe + + + <_ProjectFileVersion>10.0.30319.1 + + + + NDEBUG;%(PreprocessorDefinitions) + + + + + $(KillPythonExe) + + + \ No newline at end of file diff --git a/PCbuild/select.vcxproj b/PCbuild/select.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/select.vcxproj @@ -0,0 +1,236 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {18CAE28C-B454-46C1-87A0-493D91D97F03} + select + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + MachineX64 + + + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + MachineX64 + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/select.vcxproj.filters b/PCbuild/select.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/select.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {98346077-900c-4c7a-852f-a23470e37b40} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/sqlite3.props b/PCbuild/sqlite3.props new file mode 100644 --- /dev/null +++ b/PCbuild/sqlite3.props @@ -0,0 +1,16 @@ +? + + + + + + <_ProjectFileVersion>10.0.30319.1 + + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + SQLITE_API=__declspec(dllexport);%(PreprocessorDefinitions) + Level1 + + + \ No newline at end of file diff --git a/PCbuild/sqlite3.vcxproj b/PCbuild/sqlite3.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/sqlite3.vcxproj @@ -0,0 +1,241 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {A1A295E5-463C-437F-81CA-1F32367685DA} + sqlite3 + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .dll + + + + %(AdditionalIncludeDirectories) + + + $(OutDir)$(ProjectName)_d.dll + + + + + X64 + + + %(AdditionalIncludeDirectories) + + + $(OutDir)$(ProjectName)_d.dll + + + + + %(AdditionalIncludeDirectories) + + + $(OutDir)$(ProjectName).dll + + + + + X64 + + + %(AdditionalIncludeDirectories) + + + $(OutDir)$(ProjectName).dll + + + + + $(sqlite3Dir);%(AdditionalIncludeDirectories) + + + $(OutDir)$(ProjectName).dll + + + + + X64 + + + %(AdditionalIncludeDirectories) + + + + + %(AdditionalIncludeDirectories) + + + $(OutDir)$(ProjectName).dll + + + + + X64 + + + %(AdditionalIncludeDirectories) + + + + + + + + + + + + {6de10744-e396-40a5-b4e2-1b69aa7c8d31} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/sqlite3.vcxproj.filters b/PCbuild/sqlite3.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/sqlite3.vcxproj.filters @@ -0,0 +1,24 @@ +? + + + + {ce5b649d-a6f7-4459-9425-c883795d79df} + + + {0e842fe2-176b-4e83-9d1f-0ad13a859efd} + + + + + Header Files + + + Header Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/ssl.vcxproj b/PCbuild/ssl.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/ssl.vcxproj @@ -0,0 +1,221 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {E5B04CC0-EB4C-42AB-B4DC-18EF95F864B0} + ssl + MakeFileProj + + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + Makefile + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + cd "$(SolutionDir)" +"$(PythonExe)" build_ssl.py Release $(Platform) -a + + + + + $(NMakePreprocessorDefinitions) + $(NMakeIncludeSearchPath) + $(NMakeForcedIncludes) + $(NMakeAssemblySearchPath) + $(NMakeForcedUsingAssemblies) + + + + + + {b11d750f-cd1f-4a96-85ce-e69a5c5259f9} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/unicodedata.vcxproj b/PCbuild/unicodedata.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/unicodedata.vcxproj @@ -0,0 +1,224 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {ECC7CEAC-A5E5-458E-BB9E-2413CC847881} + unicodedata + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + 0x1D120000 + + + + + X64 + + + 0x1D120000 + + + + + 0x1D120000 + + + + + X64 + + + 0x1D120000 + + + + + 0x1D120000 + + + + + X64 + + + 0x1D120000 + MachineX64 + + + + + 0x1D120000 + + + + + X64 + + + 0x1D120000 + MachineX64 + + + + + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/unicodedata.vcxproj.filters b/PCbuild/unicodedata.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/unicodedata.vcxproj.filters @@ -0,0 +1,24 @@ +? + + + + {b939a8f1-ccd7-420a-974a-243606dccd74} + + + {e2c055bb-ec62-4bbc-aa1c-d88da4d4ad1c} + + + + + Header Files + + + Header Files + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/w9xpopen.vcxproj b/PCbuild/w9xpopen.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/w9xpopen.vcxproj @@ -0,0 +1,263 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {E9E0A1F6-0009-4E8C-B8F8-1B8F5D49A058} + w9xpopen + + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + NotSet + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + MultiByte + + + Application + false + MultiByte + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.40219.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + + + + Disabled + EnableFastChecks + MultiThreadedDebug + + + Console + + + + + X64 + + + Disabled + EnableFastChecks + MultiThreadedDebug + + + Console + + + + + MaxSpeed + OnlyExplicitInline + true + MultiThreaded + true + + + false + Console + + + + + X64 + + + MaxSpeed + OnlyExplicitInline + true + MultiThreaded + true + + + false + Console + + + + + MaxSpeed + OnlyExplicitInline + true + MultiThreaded + true + + + false + Console + + + + + + + X64 + + + MaxSpeed + OnlyExplicitInline + true + MultiThreaded + true + + + false + Console + + + MachineX64 + + + + + MaxSpeed + OnlyExplicitInline + true + MultiThreaded + true + + + false + Console + + + + + + + X64 + + + MaxSpeed + OnlyExplicitInline + true + MultiThreaded + true + + + false + Console + + + MachineX64 + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/w9xpopen.vcxproj.filters b/PCbuild/w9xpopen.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/w9xpopen.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {abc2dffd-3f2a-47bd-b89b-0314c99ef21e} + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/winsound.vcxproj b/PCbuild/winsound.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/winsound.vcxproj @@ -0,0 +1,220 @@ +? + + + + Debug + Win32 + + + Debug + x64 + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {28B5D777-DDF2-4B6B-B34F-31D938813856} + winsound + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + winmm.lib;%(AdditionalDependencies) + + + + + X64 + + + winmm.lib;%(AdditionalDependencies) + + + + + winmm.lib;%(AdditionalDependencies) + + + + + X64 + + + winmm.lib;%(AdditionalDependencies) + + + + + winmm.lib;%(AdditionalDependencies) + + + + + X64 + + + winmm.lib;%(AdditionalDependencies) + MachineX64 + + + + + winmm.lib;%(AdditionalDependencies) + + + + + X64 + + + winmm.lib;%(AdditionalDependencies) + MachineX64 + + + + + + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + false + + + + + + \ No newline at end of file diff --git a/PCbuild/winsound.vcxproj.filters b/PCbuild/winsound.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/winsound.vcxproj.filters @@ -0,0 +1,14 @@ +? + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + + + Source Files + + + \ No newline at end of file diff --git a/PCbuild/x64.props b/PCbuild/x64.props new file mode 100644 --- /dev/null +++ b/PCbuild/x64.props @@ -0,0 +1,26 @@ +? + + + $(HOST_PYTHON) + + + <_ProjectFileVersion>10.0.30319.1 + <_PropertySheetDisplayName>amd64 + $(SolutionDir)\amd64\ + $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ + + + + /USECL:MS_OPTERON /GS- %(AdditionalOptions) + _WIN64;_M_X64;%(PreprocessorDefinitions) + + + MachineX64 + + + + + $(PythonExe) + + + \ No newline at end of file diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj new file mode 100644 --- /dev/null +++ b/PCbuild/xxlimited.vcxproj @@ -0,0 +1,191 @@ +? + + + + PGInstrument + Win32 + + + PGInstrument + x64 + + + PGUpdate + Win32 + + + PGUpdate + x64 + + + Release + Win32 + + + Release + x64 + + + + {F749B822-B489-4CA5-A3AD-CE078F5F338A} + xxlimited + Win32Proj + + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + DynamicLibrary + NotSet + true + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <_ProjectFileVersion>10.0.30319.1 + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + AllRules.ruleset + + + .pyd + .pyd + + + + NDEBUG;_WIN32;_WINDLL;Py_LIMITED_API + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;$(OutDir)python33.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + Py_LIMITED_API;%(PreprocessorDefinitions) + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + MachineX64 + + + + + Py_LIMITED_API;%(PreprocessorDefinitions) + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + + + + + X64 + + + wsock32.lib;%(AdditionalDependencies) + libc;%(IgnoreSpecificDefaultLibraries) + 0x1D110000 + MachineX64 + + + + + + + + + \ No newline at end of file diff --git a/PCbuild/xxlimited.vcxproj.filters b/PCbuild/xxlimited.vcxproj.filters new file mode 100644 --- /dev/null +++ b/PCbuild/xxlimited.vcxproj.filters @@ -0,0 +1,13 @@ +? + + + + {eb5eebc7-3787-4f8f-b282-204db2962cc3} + + + + + Source Files + + + \ No newline at end of file diff --git a/Python/dynload_win.c b/Python/dynload_win.c --- a/Python/dynload_win.c +++ b/Python/dynload_win.c @@ -12,8 +12,10 @@ #include // "activation context" magic - see dl_nt.c... +#if HAVE_SXS extern ULONG_PTR _Py_ActivateActCtx(); void _Py_DeactivateActCtx(ULONG_PTR cookie); +#endif const char *_PyImport_DynLoadFiletab[] = { #ifdef _DEBUG @@ -191,18 +193,24 @@ { HINSTANCE hDLL = NULL; unsigned int old_mode; +#if HAVE_SXS ULONG_PTR cookie = 0; +#endif /* Don't display a message box when Python can't load a DLL */ old_mode = SetErrorMode(SEM_FAILCRITICALERRORS); +#if HAVE_SXS cookie = _Py_ActivateActCtx(); +#endif /* We use LoadLibraryEx so Windows looks for dependent DLLs in directory of pathname first. */ /* XXX This call doesn't exist in Windows CE */ hDLL = LoadLibraryExW(wpathname, NULL, LOAD_WITH_ALTERED_SEARCH_PATH); +#if HAVE_SXS _Py_DeactivateActCtx(cookie); +#endif /* restore old error mode settings */ SetErrorMode(old_mode); diff --git a/Tools/buildbot/build-amd64.bat b/Tools/buildbot/build-amd64.bat --- a/Tools/buildbot/build-amd64.bat +++ b/Tools/buildbot/build-amd64.bat @@ -1,6 +1,6 @@ @rem Used by the buildbot "compile" step. cmd /c Tools\buildbot\external-amd64.bat -call "%VS90COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 +call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 cmd /c Tools\buildbot\clean-amd64.bat vcbuild /useenv PCbuild\kill_python.vcproj "Debug|x64" && PCbuild\amd64\kill_python_d.exe vcbuild PCbuild\pcbuild.sln "Debug|x64" diff --git a/Tools/buildbot/build.bat b/Tools/buildbot/build.bat --- a/Tools/buildbot/build.bat +++ b/Tools/buildbot/build.bat @@ -1,6 +1,6 @@ @rem Used by the buildbot "compile" step. cmd /c Tools\buildbot\external.bat -call "%VS90COMNTOOLS%vsvars32.bat" +call "%VS100COMNTOOLS%vsvars32.bat" cmd /c Tools\buildbot\clean.bat vcbuild /useenv PCbuild\kill_python.vcproj "Debug|Win32" && PCbuild\kill_python_d.exe vcbuild /useenv PCbuild\pcbuild.sln "Debug|Win32" diff --git a/Tools/buildbot/buildmsi.bat b/Tools/buildbot/buildmsi.bat --- a/Tools/buildbot/buildmsi.bat +++ b/Tools/buildbot/buildmsi.bat @@ -2,7 +2,7 @@ cmd /c Tools\buildbot\external.bat @rem build release versions of things -call "%VS90COMNTOOLS%vsvars32.bat" +call "%VS100COMNTOOLS%vsvars32.bat" @rem build Python vcbuild /useenv PCbuild\pcbuild.sln "Release|Win32" diff --git a/Tools/buildbot/clean-amd64.bat b/Tools/buildbot/clean-amd64.bat --- a/Tools/buildbot/clean-amd64.bat +++ b/Tools/buildbot/clean-amd64.bat @@ -1,5 +1,5 @@ @rem Used by the buildbot "clean" step. -call "%VS90COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 +call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 @echo Deleting .pyc/.pyo files ... del /s Lib\*.pyc Lib\*.pyo @echo Deleting test leftovers ... diff --git a/Tools/buildbot/clean.bat b/Tools/buildbot/clean.bat --- a/Tools/buildbot/clean.bat +++ b/Tools/buildbot/clean.bat @@ -1,5 +1,5 @@ @rem Used by the buildbot "clean" step. -call "%VS90COMNTOOLS%vsvars32.bat" +call "%VS100COMNTOOLS%vsvars32.bat" @echo Deleting .pyc/.pyo files ... del /s Lib\*.pyc Lib\*.pyo @echo Deleting test leftovers ... diff --git a/Tools/buildbot/external-amd64.bat b/Tools/buildbot/external-amd64.bat --- a/Tools/buildbot/external-amd64.bat +++ b/Tools/buildbot/external-amd64.bat @@ -2,7 +2,7 @@ @rem Assume we start inside the Python source directory call "Tools\buildbot\external-common.bat" -call "%VS90COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 +call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 if not exist tcltk64\bin\tcl85g.dll ( cd tcl-8.5.9.0\win diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -30,11 +30,11 @@ if not exist openssl-1.0.0a svn export http://svn.python.org/projects/external/openssl-1.0.0a @rem tcl/tk -if not exist tcl-8.5.9.0 ( +if not exist tcl-8.5.11.0 ( rd /s/q tcltk tcltk64 - svn export http://svn.python.org/projects/external/tcl-8.5.9.0 + svn export http://svn.python.org/projects/external/tcl-8.5.11.0 ) -if not exist tk-8.5.9.0 svn export http://svn.python.org/projects/external/tk-8.5.9.0 +if not exist tk-8.5.11.0 svn export http://svn.python.org/projects/external/tk-8.5.11.0 @rem sqlite3 if not exist sqlite-3.7.4 ( diff --git a/Tools/buildbot/external.bat b/Tools/buildbot/external.bat --- a/Tools/buildbot/external.bat +++ b/Tools/buildbot/external.bat @@ -2,20 +2,20 @@ @rem Assume we start inside the Python source directory call "Tools\buildbot\external-common.bat" -call "%VS90COMNTOOLS%\vsvars32.bat" +call "%VS100COMNTOOLS%\vsvars32.bat" if not exist tcltk\bin\tcl85g.dll ( @rem all and install need to be separate invocations, otherwise nmakehlp is not found on install - cd tcl-8.5.9.0\win + cd tcl-8.5.11.0\win nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 DEBUG=1 INSTALLDIR=..\..\tcltk clean all nmake -f makefile.vc DEBUG=1 INSTALLDIR=..\..\tcltk install cd ..\.. ) if not exist tcltk\bin\tk85g.dll ( - cd tk-8.5.9.0\win - nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.9.0 clean - nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.9.0 all - nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.9.0 install + cd tk-8.5.11.0\win + nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.11.0 clean + nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.11.0 all + nmake -f makefile.vc COMPILERFLAGS=-DWINVER=0x0500 OPTS=noxp DEBUG=1 INSTALLDIR=..\..\tcltk TCLDIR=..\..\tcl-8.5.11.0 install cd ..\.. ) diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -27,7 +27,7 @@ # path to PCbuild directory PCBUILD="PCbuild" # msvcrt version -MSVCR = "90" +MSVCR = "100" # Name of certificate in default store to sign MSI with certname = None # Make a zip file containing the PDB files for this build? @@ -872,26 +872,23 @@ parent = default_feature, attributes=2|8, level=2) -def extract_msvcr90(): +def extract_msvcr100(): # Find the redistributable files if msilib.Win64: arch = "amd64" else: arch = "x86" - dir = os.path.join(os.environ['VS90COMNTOOLS'], r"..\..\VC\redist\%s\Microsoft.VC90.CRT" % arch) + dir = os.path.join(os.environ['VS100COMNTOOLS'], r"..\..\VC\redist\%s\Microsoft.VC100.CRT" % arch) result = [] installer = msilib.MakeInstaller() - # omit msvcm90 and msvcp90, as they aren't really needed - files = ["Microsoft.VC90.CRT.manifest", "msvcr90.dll"] - for f in files: - path = os.path.join(dir, f) - kw = {'src':path} - if f.endswith('.dll'): - kw['version'] = installer.FileVersion(path, 0) - kw['language'] = installer.FileVersion(path, 1) - result.append((f, kw)) - return result + # At least for VS2010, manifests are no longer provided + name = "msvcr100.dll" + path = os.path.join(dir, name) + kw = {'src':path} + kw['version'] = installer.FileVersion(path, 0) + kw['language'] = installer.FileVersion(path, 1) + return name, kw def generate_license(): import shutil, glob @@ -981,9 +978,8 @@ # pointing to the root directory root.start_component("msvcr90", feature=private_crt) # Results are ID,keyword pairs - manifest, crtdll = extract_msvcr90() - root.add_file(manifest[0], **manifest[1]) - root.add_file(crtdll[0], **crtdll[1]) + crtdll, kwds = extract_msvcr100() + root.add_file(crtdll, **kwds) # Copy the manifest # Actually, don't do that anymore - no DLL in DLLs should have a manifest # dependency on msvcr90.dll anymore, so this should not be necessary @@ -1317,9 +1313,9 @@ # Merge CRT into MSI file. This requires the database to be closed. mod_dir = os.path.join(os.environ["ProgramFiles"], "Common Files", "Merge Modules") if msilib.Win64: - modules = ["Microsoft_VC90_CRT_x86_x64.msm", "policy_9_0_Microsoft_VC90_CRT_x86_x64.msm"] + modules = ["Microsoft_VC100_CRT_x64.msm"] else: - modules = ["Microsoft_VC90_CRT_x86.msm","policy_9_0_Microsoft_VC90_CRT_x86.msm"] + modules = ["Microsoft_VC100_CRT_x86.msm"] for i, n in enumerate(modules): modules[i] = os.path.join(mod_dir, n) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:04:27 2012 From: python-checkins at python.org (brett.cannon) Date: Sun, 13 May 2012 19:04:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313959=3A_Document_?= =?utf8?q?imp=2Efind=5Fmodule/load=5Fmodule_as_deprecated=2E?= Message-ID: http://hg.python.org/cpython/rev/59870239813c changeset: 76899:59870239813c user: Brett Cannon date: Sun May 13 13:04:21 2012 -0400 summary: Issue #13959: Document imp.find_module/load_module as deprecated. The code itself does not raise a DeprecationWarning as the functions are technically fine, it's just a bad API. Unfortunately experience has shown that the terrible API has been exposed in various places, necessitating that it stick around probably until py4k comes around since it is such a shift to move over to importlib.find_loader(). files: Doc/library/imp.rst | 7 +++++++ Misc/NEWS | 3 ++- 2 files changed, 9 insertions(+), 1 deletions(-) diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -72,6 +72,9 @@ then use :func:`find_module` with the *path* argument set to ``P.__path__``. When *P* itself has a dotted name, apply this recipe recursively. + .. deprecated:: 3.3 + Use :func:`importlib.find_loader` instead. + .. function:: load_module(name, file, pathname, description) @@ -93,6 +96,10 @@ it was not ``None``, even when an exception is raised. This is best done using a :keyword:`try` ... :keyword:`finally` statement. + .. deprecated:: 3.3 + Unneeded as loaders should be used to load modules and + :func:`find_module` is deprecated. + .. function:: new_module(name) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,7 +26,8 @@ - Issue #14366: Support lzma compression in zip files. Patch by Serhiy Storchaka. -- Issue #13959: Introduce importlib.find_loader(). +- Issue #13959: Introduce importlib.find_loader() and document + imp.find_module/load_module as deprecated. - Issue #14082: shutil.copy2() now copies extended attributes, if possible. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:18:16 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 13 May 2012 19:18:16 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NzcwOiBpbXBy?= =?utf8?q?ove_the_library_FAQ=2E?= Message-ID: http://hg.python.org/cpython/rev/9d2a1f35421d changeset: 76900:9d2a1f35421d branch: 2.7 parent: 76886:94fb31fb3a9b user: Ezio Melotti date: Sun May 13 19:49:00 2012 +0300 summary: #14770: improve the library FAQ. files: Doc/faq/library.rst | 104 ++++++++++++++----------------- 1 files changed, 48 insertions(+), 56 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -14,7 +14,7 @@ Check :ref:`the Library Reference ` to see if there's a relevant standard library module. (Eventually you'll learn what's in the standard -library and will able to skip this step.) +library and will be able to skip this step.) For third-party packages, search the `Python Package Index `_ or try `Google `_ or @@ -28,7 +28,7 @@ If you can't find a source file for a module it may be a built-in or dynamically loaded module implemented in C, C++ or other compiled language. In this case you may not have the source file or it may be something like -mathmodule.c, somewhere in a C source directory (not on the Python Path). +:file:`mathmodule.c`, somewhere in a C source directory (not on the Python Path). There are (at least) three kinds of modules in Python: @@ -60,18 +60,18 @@ interpreter is installed on your platform. If you would like the script to be independent of where the Python interpreter -lives, you can use the "env" program. Almost all Unix variants support the -following, assuming the Python interpreter is in a directory on the user's -$PATH:: +lives, you can use the :program:`env` program. Almost all Unix variants support +the following, assuming the Python interpreter is in a directory on the user's +:envvar:`PATH`:: #!/usr/bin/env python -*Don't* do this for CGI scripts. The $PATH variable for CGI scripts is often -very minimal, so you need to use the actual absolute pathname of the +*Don't* do this for CGI scripts. The :envvar:`PATH` variable for CGI scripts is +often very minimal, so you need to use the actual absolute pathname of the interpreter. -Occasionally, a user's environment is so full that the /usr/bin/env program -fails; or there's no env program at all. In that case, you can try the +Occasionally, a user's environment is so full that the :program:`/usr/bin/env` +program fails; or there's no env program at all. In that case, you can try the following hack (due to Alex Rezinsky):: #! /bin/sh @@ -92,11 +92,11 @@ .. XXX curses *is* built by default, isn't it? For Unix variants: The standard Python source distribution comes with a curses -module in the ``Modules/`` subdirectory, though it's not compiled by default -(note that this is not available in the Windows distribution -- there is no -curses module for Windows). +module in the :source:`Modules` subdirectory, though it's not compiled by default. +(Note that this is not available in the Windows distribution -- there is no +curses module for Windows.) -The curses module supports basic curses features as well as many additional +The :mod:`curses` module supports basic curses features as well as many additional functions from ncurses and SYSV curses such as colour, alternative character set support, pads, and mouse support. This means the module isn't compatible with operating systems that only have BSD curses, but there don't seem to be any @@ -110,7 +110,7 @@ ------------------------------------------------- The :mod:`atexit` module provides a register function that is similar to C's -onexit. +:c:func:`onexit`. Why don't my signal handlers work? @@ -140,8 +140,8 @@ The :mod:`unittest` module is a fancier testing framework modelled on Java and Smalltalk testing frameworks. -For testing, it helps to write the program so that it may be easily tested by -using good modular design. Your program should have almost all functionality +To make testing easier, you should use good modular design in your program. +Your program should have almost all functionality encapsulated in either functions or class methods -- and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program @@ -157,7 +157,7 @@ Once your program is organized as a tractable collection of functions and class behaviours you should write test functions that exercise the behaviours. A test -suite can be associated with each module which automates a sequence of tests. +suite that automates a sequence of tests can be associated with each module. This sounds like a lot of work, but since Python is so terse and flexible it's surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the "production code", since this makes it @@ -186,7 +186,7 @@ How do I get a single keypress at a time? ----------------------------------------- -For Unix variants: There are several solutions. It's straightforward to do this +For Unix variants there are several solutions. It's straightforward to do this using curses, but curses is a fairly large module to learn. Here's a solution without curses:: @@ -273,7 +273,7 @@ time.sleep(10) -Instead of trying to guess how long a :func:`time.sleep` delay will be enough, +Instead of trying to guess a good delay value for :func:`time.sleep`, it's better to use some kind of semaphore mechanism. One idea is to use the :mod:`Queue` module to create a queue object, let each thread append a token to the queue when it finishes, and let the main thread read as many tokens from the @@ -284,10 +284,10 @@ --------------------------------------------------------- Use the :mod:`Queue` module to create a queue containing a list of jobs. The -:class:`~Queue.Queue` class maintains a list of objects with ``.put(obj)`` to -add an item to the queue and ``.get()`` to return an item. The class will take -care of the locking necessary to ensure that each job is handed out exactly -once. +:class:`~Queue.Queue` class maintains a list of objects and has a ``.put(obj)`` +method that adds items to the queue and a ``.get()`` method to return them. +The class will take care of the locking necessary to ensure that each job is +handed out exactly once. Here's a trivial example:: @@ -296,7 +296,7 @@ # The worker thread gets jobs off the queue. When the queue is empty, it # assumes there will be no more work and exits. # (Realistically workers will run until terminated.) - def worker (): + def worker(): print 'Running worker' time.sleep(0.1) while True: @@ -329,6 +329,8 @@ When run, this will produce the following output: +.. code-block:: none + Running worker Running worker Running worker @@ -343,15 +345,15 @@ Worker running with argument 5 ... -Consult the module's documentation for more details; the ``Queue`` class -provides a featureful interface. +Consult the module's documentation for more details; the :class:`~Queue.Queue` +class provides a featureful interface. What kinds of global value mutation are thread-safe? ---------------------------------------------------- -A global interpreter lock (GIL) is used internally to ensure that only one -thread runs in the Python VM at a time. In general, Python offers to switch +A global interpreter lock (:term:`GIL`) is used internally to ensure that only +one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via :func:`sys.setcheckinterval`. Each bytecode instruction and therefore all the C implementation code reached from each instruction is @@ -396,7 +398,7 @@ .. XXX mention multiprocessing .. XXX link to dbeazley's talk about GIL? -The Global Interpreter Lock (GIL) is often seen as a hindrance to Python's +The Global Interpreter Lock (:term:`GIL`) is often seen as a hindrance to Python's deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held. @@ -459,7 +461,7 @@ To truncate a file, open it using ``f = open(filename, "r+")``, and use ``f.truncate(offset)``; offset defaults to the current seek position. There's also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where -``fd`` is the file descriptor (a small integer). +*fd* is the file descriptor (a small integer). The :mod:`shutil` module also contains a number of functions to work on files including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and @@ -493,7 +495,7 @@ "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the string. -For data that is more regular (e.g. a homogeneous list of ints or thefloats), +For data that is more regular (e.g. a homogeneous list of ints or floats), you can also use the :mod:`array` module. @@ -503,7 +505,7 @@ :func:`os.read` is a low-level function which takes a file descriptor, a small integer representing the opened file. :func:`os.popen` creates a high-level file object, the same type returned by the built-in :func:`open` function. -Thus, to read n bytes from a pipe p created with :func:`os.popen`, you need to +Thus, to read *n* bytes from a pipe *p* created with :func:`os.popen`, you need to use ``p.read(n)``. @@ -522,9 +524,9 @@ Warning: in general it is unwise to do this because you can easily cause a deadlock where your process is blocked waiting for output from the child while -the child is blocked waiting for input from you. This can be caused because the -parent expects the child to output more text than it does, or it can be caused -by data being stuck in stdio buffers due to lack of flushing. The Python parent +the child is blocked waiting for input from you. This can be caused by the +parent expecting the child to output more text than it does or by data being +stuck in stdio buffers due to lack of flushing. The Python parent can of course explicitly flush the data it sends to the child before it reads any output, but if the child is a naive C program it may have been written to never explicitly flush its output, even if it is interactive, since flushing is @@ -544,8 +546,8 @@ In many cases, all you really need is to run some data through a command and get the result back. Unless the amount of data is very large, the easiest way to do this is to write it to a temporary file and run the command with that temporary -file as input. The standard module :mod:`tempfile` exports a ``mktemp()`` -function to generate unique temporary file names. :: +file as input. The standard module :mod:`tempfile` exports a +:func:`~tempfile.mktemp` function to generate unique temporary file names. :: import tempfile import os @@ -673,7 +675,8 @@ sys.stdout.write(httpobj.getfile().read()) Note that in general for percent-encoded POST operations, query strings must be -quoted using :func:`urllib.quote`. For example to send name="Guy Steele, Jr.":: +quoted using :func:`urllib.quote`. For example, to send +``name="Guy Steele, Jr."``:: >>> from urllib import quote >>> x = quote("Guy Steele, Jr.") @@ -689,19 +692,8 @@ .. XXX add modern template languages -There are many different modules available: - -* HTMLgen is a class library of objects corresponding to all the HTML 3.2 markup - tags. It's used when you are writing in Python and wish to synthesize HTML - pages for generating a web or for CGI forms, etc. - -* DocumentTemplate and Zope Page Templates are two different systems that are - part of Zope. - -* Quixote's PTL uses Python syntax to assemble strings of text. - -Consult the `Web Programming wiki pages -`_ for more links. +You can find a collection of useful links on the `Web Programming wiki page +`_. How do I send mail from a Python script? @@ -730,7 +722,7 @@ server.quit() A Unix-only alternative uses sendmail. The location of the sendmail program -varies between systems; sometimes it is ``/usr/lib/sendmail``, sometime +varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's some sample code:: @@ -797,7 +789,7 @@ The :mod:`marshal` module provides very fast ways to store noncircular basic Python types to files and strings, and back again. Although marshal does not do fancy things like store instances or handle shared references properly, it does -run extremely fast. For example loading a half megabyte of data may take less +run extremely fast. For example, loading a half megabyte of data may take less than a third of a second. This often beats doing something more complex and general such as using gdbm with pickle/shelve. @@ -807,9 +799,9 @@ .. XXX update this, default protocol is 2/3 -The default format used by the pickle module is a slow one that results in -readable pickles. Making it the default, but it would break backward -compatibility:: +By default :mod:`pickle` uses a relatively old and slow format for backward +compatibility. You can however specify other protocol versions that are +faster:: largeString = 'z' * (100 * 1024) myPickle = cPickle.dumps(largeString, protocol=1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:18:17 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 13 May 2012 19:18:17 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NzcwOiBpbXBy?= =?utf8?q?ove_the_library_FAQ=2E?= Message-ID: http://hg.python.org/cpython/rev/7a046f1ddd07 changeset: 76901:7a046f1ddd07 branch: 3.2 parent: 76896:e0dcd732055f user: Ezio Melotti date: Sun May 13 20:14:04 2012 +0300 summary: #14770: improve the library FAQ. files: Doc/faq/library.rst | 99 +++++++++++++++------------------ 1 files changed, 45 insertions(+), 54 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -14,7 +14,7 @@ Check :ref:`the Library Reference ` to see if there's a relevant standard library module. (Eventually you'll learn what's in the standard -library and will able to skip this step.) +library and will be able to skip this step.) For third-party packages, search the `Python Package Index `_ or try `Google `_ or @@ -28,7 +28,7 @@ If you can't find a source file for a module it may be a built-in or dynamically loaded module implemented in C, C++ or other compiled language. In this case you may not have the source file or it may be something like -mathmodule.c, somewhere in a C source directory (not on the Python Path). +:file:`mathmodule.c`, somewhere in a C source directory (not on the Python Path). There are (at least) three kinds of modules in Python: @@ -60,18 +60,18 @@ interpreter is installed on your platform. If you would like the script to be independent of where the Python interpreter -lives, you can use the "env" program. Almost all Unix variants support the -following, assuming the Python interpreter is in a directory on the user's -$PATH:: +lives, you can use the :program:`env` program. Almost all Unix variants support +the following, assuming the Python interpreter is in a directory on the user's +:envvar:`PATH`:: #!/usr/bin/env python -*Don't* do this for CGI scripts. The $PATH variable for CGI scripts is often -very minimal, so you need to use the actual absolute pathname of the +*Don't* do this for CGI scripts. The :envvar:`PATH` variable for CGI scripts is +often very minimal, so you need to use the actual absolute pathname of the interpreter. -Occasionally, a user's environment is so full that the /usr/bin/env program -fails; or there's no env program at all. In that case, you can try the +Occasionally, a user's environment is so full that the :program:`/usr/bin/env` +program fails; or there's no env program at all. In that case, you can try the following hack (due to Alex Rezinsky):: #! /bin/sh @@ -92,11 +92,11 @@ .. XXX curses *is* built by default, isn't it? For Unix variants: The standard Python source distribution comes with a curses -module in the ``Modules/`` subdirectory, though it's not compiled by default -(note that this is not available in the Windows distribution -- there is no -curses module for Windows). +module in the :source:`Modules` subdirectory, though it's not compiled by default. +(Note that this is not available in the Windows distribution -- there is no +curses module for Windows.) -The curses module supports basic curses features as well as many additional +The :mod:`curses` module supports basic curses features as well as many additional functions from ncurses and SYSV curses such as colour, alternative character set support, pads, and mouse support. This means the module isn't compatible with operating systems that only have BSD curses, but there don't seem to be any @@ -110,7 +110,7 @@ ------------------------------------------------- The :mod:`atexit` module provides a register function that is similar to C's -onexit. +:c:func:`onexit`. Why don't my signal handlers work? @@ -140,8 +140,8 @@ The :mod:`unittest` module is a fancier testing framework modelled on Java and Smalltalk testing frameworks. -For testing, it helps to write the program so that it may be easily tested by -using good modular design. Your program should have almost all functionality +To make testing easier, you should use good modular design in your program. +Your program should have almost all functionality encapsulated in either functions or class methods -- and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program @@ -157,7 +157,7 @@ Once your program is organized as a tractable collection of functions and class behaviours you should write test functions that exercise the behaviours. A test -suite can be associated with each module which automates a sequence of tests. +suite that automates a sequence of tests can be associated with each module. This sounds like a lot of work, but since Python is so terse and flexible it's surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the "production code", since this makes it @@ -186,7 +186,7 @@ How do I get a single keypress at a time? ----------------------------------------- -For Unix variants: There are several solutions. It's straightforward to do this +For Unix variants there are several solutions. It's straightforward to do this using curses, but curses is a fairly large module to learn. .. XXX this doesn't work out of the box, some IO expert needs to check why @@ -275,7 +275,7 @@ time.sleep(10) -Instead of trying to guess how long a :func:`time.sleep` delay will be enough, +Instead of trying to guess a good delay value for :func:`time.sleep`, it's better to use some kind of semaphore mechanism. One idea is to use the :mod:`queue` module to create a queue object, let each thread append a token to the queue when it finishes, and let the main thread read as many tokens from the @@ -291,9 +291,9 @@ Or, if you want fine control over the dispatching algorithm, you can write your own logic manually. Use the :mod:`queue` module to create a queue containing a list of jobs. The :class:`~queue.Queue` class maintains a -list of objects with ``.put(obj)`` to add an item to the queue and ``.get()`` -to return an item. The class will take care of the locking necessary to -ensure that each job is handed out exactly once. +list of objects and has a ``.put(obj)`` method that adds items to the queue and +a ``.get()`` method to return them. The class will take care of the locking +necessary to ensure that each job is handed out exactly once. Here's a trivial example:: @@ -302,7 +302,7 @@ # The worker thread gets jobs off the queue. When the queue is empty, it # assumes there will be no more work and exits. # (Realistically workers will run until terminated.) - def worker (): + def worker(): print('Running worker') time.sleep(0.1) while True: @@ -333,7 +333,9 @@ print('Main thread sleeping') time.sleep(5) -When run, this will produce the following output:: +When run, this will produce the following output: + +.. code-block:: none Running worker Running worker @@ -349,8 +351,8 @@ Worker running with argument 5 ... -Consult the module's documentation for more details; the ``Queue`` class -provides a featureful interface. +Consult the module's documentation for more details; the :class:`~queue.Queue`` +class provides a featureful interface. What kinds of global value mutation are thread-safe? @@ -467,7 +469,7 @@ To truncate a file, open it using ``f = open(filename, "rb+")``, and use ``f.truncate(offset)``; offset defaults to the current seek position. There's also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where -``fd`` is the file descriptor (a small integer). +*fd* is the file descriptor (a small integer). The :mod:`shutil` module also contains a number of functions to work on files including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and @@ -501,15 +503,15 @@ "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the string. -For data that is more regular (e.g. a homogeneous list of ints or thefloats), +For data that is more regular (e.g. a homogeneous list of ints or floats), you can also use the :mod:`array` module. - .. note:: - To read and write binary data, it is mandatory to open the file in - binary mode (here, passing ``"rb"`` to :func:`open`). If you use - ``"r"`` instead (the default), the file will be open in text mode - and ``f.read()`` will return :class:`str` objects rather than - :class:`bytes` objects. +.. note:: + To read and write binary data, it is mandatory to open the file in + binary mode (here, passing ``"rb"`` to :func:`open`). If you use + ``"r"`` instead (the default), the file will be open in text mode + and ``f.read()`` will return :class:`str` objects rather than + :class:`bytes` objects. I can't seem to use os.read() on a pipe created with os.popen(); why? @@ -518,7 +520,7 @@ :func:`os.read` is a low-level function which takes a file descriptor, a small integer representing the opened file. :func:`os.popen` creates a high-level file object, the same type returned by the built-in :func:`open` function. -Thus, to read n bytes from a pipe p created with :func:`os.popen`, you need to +Thus, to read *n* bytes from a pipe *p* created with :func:`os.popen`, you need to use ``p.read(n)``. @@ -538,8 +540,8 @@ Warning: in general it is unwise to do this because you can easily cause a deadlock where your process is blocked waiting for output from the child while the child is blocked waiting for input from you. This can be caused - because the parent expects the child to output more text than it does, or it - can be caused by data being stuck in stdio buffers due to lack of flushing. + by the parent expecting the child to output more text than it does or + by data being stuck in stdio buffers due to lack of flushing. The Python parent can of course explicitly flush the data it sends to the child before it reads any output, but if the child is a naive C program it may have been written to never explicitly flush its output, even if it is @@ -561,7 +563,7 @@ get the result back. Unless the amount of data is very large, the easiest way to do this is to write it to a temporary file and run the command with that temporary file as input. The standard module :mod:`tempfile` exports a - ``mktemp()`` function to generate unique temporary file names. :: + :func:`~tempfile.mktemp` function to generate unique temporary file names. :: import tempfile import os @@ -681,8 +683,8 @@ msg, hdrs = req.read(), req.info() Note that in general for percent-encoded POST operations, query strings must be -quoted using :func:`urllib.parse.urlencode`. For example to send name="Guy Steele, -Jr.":: +quoted using :func:`urllib.parse.urlencode`. For example, to send +``name=Guy Steele, Jr.``:: >>> import urllib.parse >>> urllib.parse.urlencode({'name': 'Guy Steele, Jr.'}) @@ -696,19 +698,8 @@ .. XXX add modern template languages -There are many different modules available: - -* HTMLgen is a class library of objects corresponding to all the HTML 3.2 markup - tags. It's used when you are writing in Python and wish to synthesize HTML - pages for generating a web or for CGI forms, etc. - -* DocumentTemplate and Zope Page Templates are two different systems that are - part of Zope. - -* Quixote's PTL uses Python syntax to assemble strings of text. - -Consult the `Web Programming wiki pages -`_ for more links. +You can find a collection of useful links on the `Web Programming wiki page +`_. How do I send mail from a Python script? @@ -737,7 +728,7 @@ server.quit() A Unix-only alternative uses sendmail. The location of the sendmail program -varies between systems; sometimes it is ``/usr/lib/sendmail``, sometime +varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's some sample code:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:18:18 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 13 May 2012 19:18:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314770=3A_merge_with_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/59fd56c1be0d changeset: 76902:59fd56c1be0d parent: 76899:59870239813c parent: 76901:7a046f1ddd07 user: Ezio Melotti date: Sun May 13 20:17:40 2012 +0300 summary: #14770: merge with 3.2. files: Doc/faq/library.rst | 99 +++++++++++++++------------------ 1 files changed, 45 insertions(+), 54 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -14,7 +14,7 @@ Check :ref:`the Library Reference ` to see if there's a relevant standard library module. (Eventually you'll learn what's in the standard -library and will able to skip this step.) +library and will be able to skip this step.) For third-party packages, search the `Python Package Index `_ or try `Google `_ or @@ -28,7 +28,7 @@ If you can't find a source file for a module it may be a built-in or dynamically loaded module implemented in C, C++ or other compiled language. In this case you may not have the source file or it may be something like -mathmodule.c, somewhere in a C source directory (not on the Python Path). +:file:`mathmodule.c`, somewhere in a C source directory (not on the Python Path). There are (at least) three kinds of modules in Python: @@ -60,18 +60,18 @@ interpreter is installed on your platform. If you would like the script to be independent of where the Python interpreter -lives, you can use the "env" program. Almost all Unix variants support the -following, assuming the Python interpreter is in a directory on the user's -$PATH:: +lives, you can use the :program:`env` program. Almost all Unix variants support +the following, assuming the Python interpreter is in a directory on the user's +:envvar:`PATH`:: #!/usr/bin/env python -*Don't* do this for CGI scripts. The $PATH variable for CGI scripts is often -very minimal, so you need to use the actual absolute pathname of the +*Don't* do this for CGI scripts. The :envvar:`PATH` variable for CGI scripts is +often very minimal, so you need to use the actual absolute pathname of the interpreter. -Occasionally, a user's environment is so full that the /usr/bin/env program -fails; or there's no env program at all. In that case, you can try the +Occasionally, a user's environment is so full that the :program:`/usr/bin/env` +program fails; or there's no env program at all. In that case, you can try the following hack (due to Alex Rezinsky):: #! /bin/sh @@ -92,11 +92,11 @@ .. XXX curses *is* built by default, isn't it? For Unix variants: The standard Python source distribution comes with a curses -module in the ``Modules/`` subdirectory, though it's not compiled by default -(note that this is not available in the Windows distribution -- there is no -curses module for Windows). +module in the :source:`Modules` subdirectory, though it's not compiled by default. +(Note that this is not available in the Windows distribution -- there is no +curses module for Windows.) -The curses module supports basic curses features as well as many additional +The :mod:`curses` module supports basic curses features as well as many additional functions from ncurses and SYSV curses such as colour, alternative character set support, pads, and mouse support. This means the module isn't compatible with operating systems that only have BSD curses, but there don't seem to be any @@ -110,7 +110,7 @@ ------------------------------------------------- The :mod:`atexit` module provides a register function that is similar to C's -onexit. +:c:func:`onexit`. Why don't my signal handlers work? @@ -140,8 +140,8 @@ The :mod:`unittest` module is a fancier testing framework modelled on Java and Smalltalk testing frameworks. -For testing, it helps to write the program so that it may be easily tested by -using good modular design. Your program should have almost all functionality +To make testing easier, you should use good modular design in your program. +Your program should have almost all functionality encapsulated in either functions or class methods -- and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program @@ -157,7 +157,7 @@ Once your program is organized as a tractable collection of functions and class behaviours you should write test functions that exercise the behaviours. A test -suite can be associated with each module which automates a sequence of tests. +suite that automates a sequence of tests can be associated with each module. This sounds like a lot of work, but since Python is so terse and flexible it's surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the "production code", since this makes it @@ -186,7 +186,7 @@ How do I get a single keypress at a time? ----------------------------------------- -For Unix variants: There are several solutions. It's straightforward to do this +For Unix variants there are several solutions. It's straightforward to do this using curses, but curses is a fairly large module to learn. .. XXX this doesn't work out of the box, some IO expert needs to check why @@ -275,7 +275,7 @@ time.sleep(10) -Instead of trying to guess how long a :func:`time.sleep` delay will be enough, +Instead of trying to guess a good delay value for :func:`time.sleep`, it's better to use some kind of semaphore mechanism. One idea is to use the :mod:`queue` module to create a queue object, let each thread append a token to the queue when it finishes, and let the main thread read as many tokens from the @@ -291,9 +291,9 @@ Or, if you want fine control over the dispatching algorithm, you can write your own logic manually. Use the :mod:`queue` module to create a queue containing a list of jobs. The :class:`~queue.Queue` class maintains a -list of objects with ``.put(obj)`` to add an item to the queue and ``.get()`` -to return an item. The class will take care of the locking necessary to -ensure that each job is handed out exactly once. +list of objects and has a ``.put(obj)`` method that adds items to the queue and +a ``.get()`` method to return them. The class will take care of the locking +necessary to ensure that each job is handed out exactly once. Here's a trivial example:: @@ -302,7 +302,7 @@ # The worker thread gets jobs off the queue. When the queue is empty, it # assumes there will be no more work and exits. # (Realistically workers will run until terminated.) - def worker (): + def worker(): print('Running worker') time.sleep(0.1) while True: @@ -333,7 +333,9 @@ print('Main thread sleeping') time.sleep(5) -When run, this will produce the following output:: +When run, this will produce the following output: + +.. code-block:: none Running worker Running worker @@ -349,8 +351,8 @@ Worker running with argument 5 ... -Consult the module's documentation for more details; the ``Queue`` class -provides a featureful interface. +Consult the module's documentation for more details; the :class:`~queue.Queue`` +class provides a featureful interface. What kinds of global value mutation are thread-safe? @@ -467,7 +469,7 @@ To truncate a file, open it using ``f = open(filename, "rb+")``, and use ``f.truncate(offset)``; offset defaults to the current seek position. There's also ``os.ftruncate(fd, offset)`` for files opened with :func:`os.open`, where -``fd`` is the file descriptor (a small integer). +*fd* is the file descriptor (a small integer). The :mod:`shutil` module also contains a number of functions to work on files including :func:`~shutil.copyfile`, :func:`~shutil.copytree`, and @@ -501,15 +503,15 @@ "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the string. -For data that is more regular (e.g. a homogeneous list of ints or thefloats), +For data that is more regular (e.g. a homogeneous list of ints or floats), you can also use the :mod:`array` module. - .. note:: - To read and write binary data, it is mandatory to open the file in - binary mode (here, passing ``"rb"`` to :func:`open`). If you use - ``"r"`` instead (the default), the file will be open in text mode - and ``f.read()`` will return :class:`str` objects rather than - :class:`bytes` objects. +.. note:: + To read and write binary data, it is mandatory to open the file in + binary mode (here, passing ``"rb"`` to :func:`open`). If you use + ``"r"`` instead (the default), the file will be open in text mode + and ``f.read()`` will return :class:`str` objects rather than + :class:`bytes` objects. I can't seem to use os.read() on a pipe created with os.popen(); why? @@ -518,7 +520,7 @@ :func:`os.read` is a low-level function which takes a file descriptor, a small integer representing the opened file. :func:`os.popen` creates a high-level file object, the same type returned by the built-in :func:`open` function. -Thus, to read n bytes from a pipe p created with :func:`os.popen`, you need to +Thus, to read *n* bytes from a pipe *p* created with :func:`os.popen`, you need to use ``p.read(n)``. @@ -538,8 +540,8 @@ Warning: in general it is unwise to do this because you can easily cause a deadlock where your process is blocked waiting for output from the child while the child is blocked waiting for input from you. This can be caused - because the parent expects the child to output more text than it does, or it - can be caused by data being stuck in stdio buffers due to lack of flushing. + by the parent expecting the child to output more text than it does or + by data being stuck in stdio buffers due to lack of flushing. The Python parent can of course explicitly flush the data it sends to the child before it reads any output, but if the child is a naive C program it may have been written to never explicitly flush its output, even if it is @@ -561,7 +563,7 @@ get the result back. Unless the amount of data is very large, the easiest way to do this is to write it to a temporary file and run the command with that temporary file as input. The standard module :mod:`tempfile` exports a - ``mktemp()`` function to generate unique temporary file names. :: + :func:`~tempfile.mktemp` function to generate unique temporary file names. :: import tempfile import os @@ -681,8 +683,8 @@ msg, hdrs = req.read(), req.info() Note that in general for percent-encoded POST operations, query strings must be -quoted using :func:`urllib.parse.urlencode`. For example to send name="Guy Steele, -Jr.":: +quoted using :func:`urllib.parse.urlencode`. For example, to send +``name=Guy Steele, Jr.``:: >>> import urllib.parse >>> urllib.parse.urlencode({'name': 'Guy Steele, Jr.'}) @@ -696,19 +698,8 @@ .. XXX add modern template languages -There are many different modules available: - -* HTMLgen is a class library of objects corresponding to all the HTML 3.2 markup - tags. It's used when you are writing in Python and wish to synthesize HTML - pages for generating a web or for CGI forms, etc. - -* DocumentTemplate and Zope Page Templates are two different systems that are - part of Zope. - -* Quixote's PTL uses Python syntax to assemble strings of text. - -Consult the `Web Programming wiki pages -`_ for more links. +You can find a collection of useful links on the `Web Programming wiki page +`_. How do I send mail from a Python script? @@ -737,7 +728,7 @@ server.quit() A Unix-only alternative uses sendmail. The location of the sendmail program -varies between systems; sometimes it is ``/usr/lib/sendmail``, sometime +varies between systems; sometimes it is ``/usr/lib/sendmail``, sometimes ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's some sample code:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:19:48 2012 From: python-checkins at python.org (ezio.melotti) Date: Sun, 13 May 2012 19:19:48 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NzcwOiBiYWNr?= =?utf8?q?port_a_couple_of_changes_from_3=2Ex=2E?= Message-ID: http://hg.python.org/cpython/rev/bf3cb58dcfe7 changeset: 76903:bf3cb58dcfe7 branch: 2.7 parent: 76900:9d2a1f35421d user: Ezio Melotti date: Sun May 13 20:19:41 2012 +0300 summary: #14770: backport a couple of changes from 3.x. files: Doc/faq/library.rst | 20 ++++++++------------ 1 files changed, 8 insertions(+), 12 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -91,7 +91,7 @@ .. XXX curses *is* built by default, isn't it? -For Unix variants: The standard Python source distribution comes with a curses +For Unix variants the standard Python source distribution comes with a curses module in the :source:`Modules` subdirectory, though it's not compiled by default. (Note that this is not available in the Windows distribution -- there is no curses module for Windows.) @@ -352,7 +352,7 @@ What kinds of global value mutation are thread-safe? ---------------------------------------------------- -A global interpreter lock (:term:`GIL`) is used internally to ensure that only +A :term:`global interpreter lock` (GIL) is used internally to ensure that only one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via :func:`sys.setcheckinterval`. Each bytecode instruction and @@ -398,7 +398,7 @@ .. XXX mention multiprocessing .. XXX link to dbeazley's talk about GIL? -The Global Interpreter Lock (:term:`GIL`) is often seen as a hindrance to Python's +The :term:`global interpreter lock` (GIL) is often seen as a hindrance to Python's deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held. @@ -675,16 +675,12 @@ sys.stdout.write(httpobj.getfile().read()) Note that in general for percent-encoded POST operations, query strings must be -quoted using :func:`urllib.quote`. For example, to send -``name="Guy Steele, Jr."``:: +quoted using :func:`urllib.urlencode`. For example, to send +``name=Guy Steele, Jr.``:: - >>> from urllib import quote - >>> x = quote("Guy Steele, Jr.") - >>> x - 'Guy%20Steele,%20Jr.' - >>> query_string = "name="+x - >>> query_string - 'name=Guy%20Steele,%20Jr.' + >>> import urllib + >>> urllib.urlencode({'name': 'Guy Steele, Jr.'}) + 'name=Guy+Steele%2C+Jr.' What module should I use to help with generating HTML? -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:40:29 2012 From: python-checkins at python.org (brian.curtin) Date: Sun, 13 May 2012 19:40:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_out_VS9_project_files_?= =?utf8?q?to_PC=5CVS9=2E0_folder=2E_Fixes_=2313210?= Message-ID: http://hg.python.org/cpython/rev/924c178c0d1d changeset: 76904:924c178c0d1d parent: 76902:59fd56c1be0d user: Brian Curtin date: Sun May 13 12:40:15 2012 -0500 summary: Move out VS9 project files to PC\VS9.0 folder. Fixes #13210 files: PCbuild/_decimal.vcproj | 0 PCbuild/_lzma.vcproj | 0 PCbuild/_testbuffer.vcproj | 0 PCbuild/_bz2.vcproj | 581 ----- PCbuild/_ctypes.vcproj | 705 ------ PCbuild/_ctypes_test.vcproj | 521 ----- PCbuild/_elementtree.vcproj | 613 ------ PCbuild/_hashlib.vcproj | 537 ----- PCbuild/_msi.vcproj | 529 ----- PCbuild/_multiprocessing.vcproj | 541 ----- PCbuild/_socket.vcproj | 537 ----- PCbuild/_sqlite3.vcproj | 613 ------ PCbuild/_ssl.vcproj | 537 ----- PCbuild/_testcapi.vcproj | 521 ----- PCbuild/_tkinter.vcproj | 541 ----- PCbuild/bdist_wininst.vcproj | 270 -- PCbuild/kill_python.vcproj | 279 -- PCbuild/make_buildinfo.vcproj | 101 - PCbuild/make_versioninfo.vcproj | 324 --- PCbuild/pyexpat.vcproj | 553 ----- PCbuild/python.vcproj | 637 ------ PCbuild/python3dll.vcproj | 246 -- PCbuild/pythoncore.vcproj | 1917 ------------------- PCbuild/pythonw.vcproj | 618 ------ PCbuild/select.vcproj | 537 ----- PCbuild/sqlite3.vcproj | 537 ----- PCbuild/ssl.vcproj | 189 - PCbuild/unicodedata.vcproj | 533 ----- PCbuild/w9xpopen.vcproj | 576 ----- PCbuild/winsound.vcproj | 523 ----- PCbuild/xxlimited.vcproj | 417 ---- 31 files changed, 0 insertions(+), 15033 deletions(-) diff --git a/PCbuild/_decimal.vcproj b/PC/VS9.0/_decimal.vcproj rename from PCbuild/_decimal.vcproj rename to PC/VS9.0/_decimal.vcproj diff --git a/PCbuild/_lzma.vcproj b/PC/VS9.0/_lzma.vcproj rename from PCbuild/_lzma.vcproj rename to PC/VS9.0/_lzma.vcproj diff --git a/PCbuild/_testbuffer.vcproj b/PC/VS9.0/_testbuffer.vcproj rename from PCbuild/_testbuffer.vcproj rename to PC/VS9.0/_testbuffer.vcproj diff --git a/PCbuild/_bz2.vcproj b/PCbuild/_bz2.vcproj deleted file mode 100644 --- a/PCbuild/_bz2.vcproj +++ /dev/null @@ -1,581 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_ctypes.vcproj b/PCbuild/_ctypes.vcproj deleted file mode 100644 --- a/PCbuild/_ctypes.vcproj +++ /dev/null @@ -1,705 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_ctypes_test.vcproj b/PCbuild/_ctypes_test.vcproj deleted file mode 100644 --- a/PCbuild/_ctypes_test.vcproj +++ /dev/null @@ -1,521 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_elementtree.vcproj b/PCbuild/_elementtree.vcproj deleted file mode 100644 --- a/PCbuild/_elementtree.vcproj +++ /dev/null @@ -1,613 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_hashlib.vcproj b/PCbuild/_hashlib.vcproj deleted file mode 100644 --- a/PCbuild/_hashlib.vcproj +++ /dev/null @@ -1,537 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_msi.vcproj b/PCbuild/_msi.vcproj deleted file mode 100644 --- a/PCbuild/_msi.vcproj +++ /dev/null @@ -1,529 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_multiprocessing.vcproj b/PCbuild/_multiprocessing.vcproj deleted file mode 100644 --- a/PCbuild/_multiprocessing.vcproj +++ /dev/null @@ -1,541 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_socket.vcproj b/PCbuild/_socket.vcproj deleted file mode 100644 --- a/PCbuild/_socket.vcproj +++ /dev/null @@ -1,537 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_sqlite3.vcproj b/PCbuild/_sqlite3.vcproj deleted file mode 100644 --- a/PCbuild/_sqlite3.vcproj +++ /dev/null @@ -1,613 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_ssl.vcproj b/PCbuild/_ssl.vcproj deleted file mode 100644 --- a/PCbuild/_ssl.vcproj +++ /dev/null @@ -1,537 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_testcapi.vcproj b/PCbuild/_testcapi.vcproj deleted file mode 100644 --- a/PCbuild/_testcapi.vcproj +++ /dev/null @@ -1,521 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/_tkinter.vcproj b/PCbuild/_tkinter.vcproj deleted file mode 100644 --- a/PCbuild/_tkinter.vcproj +++ /dev/null @@ -1,541 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/bdist_wininst.vcproj b/PCbuild/bdist_wininst.vcproj deleted file mode 100644 --- a/PCbuild/bdist_wininst.vcproj +++ /dev/null @@ -1,270 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/kill_python.vcproj b/PCbuild/kill_python.vcproj deleted file mode 100644 --- a/PCbuild/kill_python.vcproj +++ /dev/null @@ -1,279 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/make_buildinfo.vcproj b/PCbuild/make_buildinfo.vcproj deleted file mode 100644 --- a/PCbuild/make_buildinfo.vcproj +++ /dev/null @@ -1,101 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/make_versioninfo.vcproj b/PCbuild/make_versioninfo.vcproj deleted file mode 100644 --- a/PCbuild/make_versioninfo.vcproj +++ /dev/null @@ -1,324 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/pyexpat.vcproj b/PCbuild/pyexpat.vcproj deleted file mode 100644 --- a/PCbuild/pyexpat.vcproj +++ /dev/null @@ -1,553 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/python.vcproj b/PCbuild/python.vcproj deleted file mode 100644 --- a/PCbuild/python.vcproj +++ /dev/null @@ -1,637 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/python3dll.vcproj b/PCbuild/python3dll.vcproj deleted file mode 100644 --- a/PCbuild/python3dll.vcproj +++ /dev/null @@ -1,246 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/pythoncore.vcproj b/PCbuild/pythoncore.vcproj deleted file mode 100644 --- a/PCbuild/pythoncore.vcproj +++ /dev/null @@ -1,1917 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/pythonw.vcproj b/PCbuild/pythonw.vcproj deleted file mode 100644 --- a/PCbuild/pythonw.vcproj +++ /dev/null @@ -1,618 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/select.vcproj b/PCbuild/select.vcproj deleted file mode 100644 --- a/PCbuild/select.vcproj +++ /dev/null @@ -1,537 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/sqlite3.vcproj b/PCbuild/sqlite3.vcproj deleted file mode 100644 --- a/PCbuild/sqlite3.vcproj +++ /dev/null @@ -1,537 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/ssl.vcproj b/PCbuild/ssl.vcproj deleted file mode 100644 --- a/PCbuild/ssl.vcproj +++ /dev/null @@ -1,189 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/unicodedata.vcproj b/PCbuild/unicodedata.vcproj deleted file mode 100644 --- a/PCbuild/unicodedata.vcproj +++ /dev/null @@ -1,533 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/w9xpopen.vcproj b/PCbuild/w9xpopen.vcproj deleted file mode 100644 --- a/PCbuild/w9xpopen.vcproj +++ /dev/null @@ -1,576 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/winsound.vcproj b/PCbuild/winsound.vcproj deleted file mode 100644 --- a/PCbuild/winsound.vcproj +++ /dev/null @@ -1,523 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/PCbuild/xxlimited.vcproj b/PCbuild/xxlimited.vcproj deleted file mode 100644 --- a/PCbuild/xxlimited.vcproj +++ /dev/null @@ -1,417 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:45:14 2012 From: python-checkins at python.org (brett.cannon) Date: Sun, 13 May 2012 19:45:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_importlib=2Eutil=2Ereso?= =?utf8?b?bHZlX25hbWUoKS4=?= Message-ID: http://hg.python.org/cpython/rev/767503ba72e7 changeset: 76905:767503ba72e7 user: Brett Cannon date: Sun May 13 13:45:09 2012 -0400 summary: Add importlib.util.resolve_name(). files: Doc/library/importlib.rst | 16 ++++++++ Lib/importlib/test/test_util.py | 40 ++++++++++++++++++++- Lib/importlib/util.py | 16 ++++++++ Misc/NEWS | 2 + 4 files changed, 73 insertions(+), 1 deletions(-) diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -737,6 +737,22 @@ This module contains the various objects that help in the construction of an :term:`importer`. +.. function:: resolve_name(name, package) + + Resolve a relative module name to an absolute one. + + If **name** has no leading dots, then **name** is simply returned. This + allows for usage such as + ``importlib.util.resolve_name('sys', __package__)`` without doing a + check to see if the **package** argument is needed. + + :exc:`ValueError` is raised if **name** is a relative module name but + package is a false value (e.g. ``None`` or the empty string). + :exc:`ValueError` is also raised a relative name would escape its containing + package (e.g. requesting ``..bacon`` from within the ``spam`` package). + + .. versionadded:: 3.3 + .. decorator:: module_for_loader A :term:`decorator` for a :term:`loader` method, diff --git a/Lib/importlib/test/test_util.py b/Lib/importlib/test/test_util.py --- a/Lib/importlib/test/test_util.py +++ b/Lib/importlib/test/test_util.py @@ -161,9 +161,47 @@ self.assertEqual(wrapped.__name__, fxn.__name__) self.assertEqual(wrapped.__qualname__, fxn.__qualname__) + +class ResolveNameTests(unittest.TestCase): + + """Tests importlib.util.resolve_name().""" + + def test_absolute(self): + # bacon + self.assertEqual('bacon', util.resolve_name('bacon', None)) + + def test_aboslute_within_package(self): + # bacon in spam + self.assertEqual('bacon', util.resolve_name('bacon', 'spam')) + + def test_no_package(self): + # .bacon in '' + with self.assertRaises(ValueError): + util.resolve_name('.bacon', '') + + def test_in_package(self): + # .bacon in spam + self.assertEqual('spam.eggs.bacon', + util.resolve_name('.bacon', 'spam.eggs')) + + def test_other_package(self): + # ..bacon in spam.bacon + self.assertEqual('spam.bacon', + util.resolve_name('..bacon', 'spam.eggs')) + + def test_escape(self): + # ..bacon in spam + with self.assertRaises(ValueError): + util.resolve_name('..bacon', 'spam') + + def test_main(): from test import support - support.run_unittest(ModuleForLoaderTests, SetPackageTests) + support.run_unittest( + ModuleForLoaderTests, + SetPackageTests, + ResolveNameTests + ) if __name__ == '__main__': diff --git a/Lib/importlib/util.py b/Lib/importlib/util.py --- a/Lib/importlib/util.py +++ b/Lib/importlib/util.py @@ -3,3 +3,19 @@ from ._bootstrap import module_for_loader from ._bootstrap import set_loader from ._bootstrap import set_package +from ._bootstrap import _resolve_name + + +def resolve_name(name, package): + """Resolve a relative module name to an absolute one.""" + if not name.startswith('.'): + return name + elif not package: + raise ValueError('{!r} is not a relative name ' + '(no leading dot)'.format(name)) + level = 0 + for character in name: + if character != '.': + break + level += 1 + return _resolve_name(name[level:], package, level) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,8 @@ Library ------- +- Add importlib.util.resolve_name(). + - Issue #14366: Support lzma compression in zip files. Patch by Serhiy Storchaka. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 19:53:27 2012 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 13 May 2012 19:53:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314532=3A_Add_a_sec?= =?utf8?q?ure=5Fcompare=28=29_helper_to_the_hmac_module=2C_to_mitigate?= Message-ID: http://hg.python.org/cpython/rev/ddcc8ee680d7 changeset: 76906:ddcc8ee680d7 user: Charles-Fran?ois Natali date: Sun May 13 19:53:07 2012 +0200 summary: Issue #14532: Add a secure_compare() helper to the hmac module, to mitigate timing attacks. Patch by Jon Oberheide. files: Doc/library/hmac.rst | 32 ++++++++++++++++++++++++++ Lib/hmac.py | 21 +++++++++++++++++ Lib/test/test_hmac.py | 38 ++++++++++++++++++++++++++++++- Misc/ACKS | 1 + Misc/NEWS | 3 ++ 5 files changed, 94 insertions(+), 1 deletions(-) diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -38,6 +38,13 @@ given to the constructor. It may contain non-ASCII bytes, including NUL bytes. + .. warning:: + + When comparing the output of :meth:`digest` to an externally-supplied + digest during a verification routine, it is recommended to use the + :func:`hmac.secure_compare` function instead of the ``==`` operator + to avoid potential timing attacks. + .. method:: HMAC.hexdigest() @@ -45,6 +52,13 @@ length containing only hexadecimal digits. This may be used to exchange the value safely in email or other non-binary environments. + .. warning:: + + When comparing the output of :meth:`hexdigest` to an externally-supplied + digest during a verification routine, it is recommended to use the + :func:`hmac.secure_compare` function instead of the ``==`` operator + to avoid potential timing attacks. + .. method:: HMAC.copy() @@ -52,6 +66,24 @@ compute the digests of strings that share a common initial substring. +This module also provides the following helper function: + +.. function:: secure_compare(a, b) + + Returns the equivalent of ``a == b``, but using a time-independent + comparison method. Comparing the full lengths of the inputs *a* and *b*, + instead of short-circuiting the comparison upon the first unequal byte, + prevents leaking information about the inputs being compared and mitigates + potential timing attacks. The inputs must be either :class:`str` or + :class:`bytes` instances. + + .. note:: + + While the :func:`hmac.secure_compare` function prevents leaking the + contents of the inputs via a timing attack, it does leak the length + of the inputs. However, this generally is not a security risk. + + .. seealso:: Module :mod:`hashlib` diff --git a/Lib/hmac.py b/Lib/hmac.py --- a/Lib/hmac.py +++ b/Lib/hmac.py @@ -13,6 +13,27 @@ digest_size = None +def secure_compare(a, b): + """Returns the equivalent of 'a == b', but using a time-independent + comparison method to prevent timing attacks.""" + if not ((isinstance(a, str) and isinstance(b, str)) or + (isinstance(a, bytes) and isinstance(b, bytes))): + raise TypeError("inputs must be strings or bytes") + + if len(a) != len(b): + return False + + result = 0 + if isinstance(a, bytes): + for x, y in zip(a, b): + result |= x ^ y + else: + for x, y in zip(a, b): + result |= ord(x) ^ ord(y) + + return result == 0 + + class HMAC: """RFC 2104 HMAC class. Also complies with RFC 4231. diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py --- a/Lib/test/test_hmac.py +++ b/Lib/test/test_hmac.py @@ -302,12 +302,48 @@ self.assertEqual(h1.hexdigest(), h2.hexdigest(), "Hexdigest of copy doesn't match original hexdigest.") +class SecureCompareTestCase(unittest.TestCase): + + def test_compare(self): + # Testing input type exception handling + a, b = 100, 200 + self.assertRaises(TypeError, hmac.secure_compare, a, b) + a, b = 100, "foobar" + self.assertRaises(TypeError, hmac.secure_compare, a, b) + a, b = "foobar", b"foobar" + self.assertRaises(TypeError, hmac.secure_compare, a, b) + + # Testing str/bytes of different lengths + a, b = "foobar", "foo" + self.assertFalse(hmac.secure_compare(a, b)) + a, b = b"foobar", b"foo" + self.assertFalse(hmac.secure_compare(a, b)) + a, b = b"\xde\xad\xbe\xef", b"\xde\xad" + self.assertFalse(hmac.secure_compare(a, b)) + + # Testing str/bytes of same lengths, different values + a, b = "foobar", "foobaz" + self.assertFalse(hmac.secure_compare(a, b)) + a, b = b"foobar", b"foobaz" + self.assertFalse(hmac.secure_compare(a, b)) + a, b = b"\xde\xad\xbe\xef", b"\xab\xad\x1d\xea" + self.assertFalse(hmac.secure_compare(a, b)) + + # Testing str/bytes of same lengths, same values + a, b = "foobar", "foobar" + self.assertTrue(hmac.secure_compare(a, b)) + a, b = b"foobar", b"foobar" + self.assertTrue(hmac.secure_compare(a, b)) + a, b = b"\xde\xad\xbe\xef", b"\xde\xad\xbe\xef" + self.assertTrue(hmac.secure_compare(a, b)) + def test_main(): support.run_unittest( TestVectorsTestCase, ConstructorTestCase, SanityTestCase, - CopyTestCase + CopyTestCase, + SecureCompareTestCase ) if __name__ == "__main__": diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -746,6 +746,7 @@ John O'Connor Kevin O'Connor Tim O'Malley +Jon Oberheide Pascal Oberndoerfer Jeffrey Ollie Adam Olsen diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -23,6 +23,9 @@ Library ------- +- Issue #14532: Add a secure_compare() helper to the hmac module, to mitigate + timing attacks. Patch by Jon Oberheide. + - Add importlib.util.resolve_name(). - Issue #14366: Support lzma compression in zip files. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 20:23:42 2012 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 13 May 2012 20:23:42 +0200 (CEST) Subject: [Python-checkins] r88951 - tracker/instances/python-dev/scripts/addpatchsets Message-ID: <3VrD266V4PzM66@mail.python.org> Author: martin.v.loewis Date: Sun May 13 20:23:42 2012 New Revision: 88951 Log: Ignore unparsable diff blocks - they might be binary diffs. Modified: tracker/instances/python-dev/scripts/addpatchsets Modified: tracker/instances/python-dev/scripts/addpatchsets ============================================================================== --- tracker/instances/python-dev/scripts/addpatchsets (original) +++ tracker/instances/python-dev/scripts/addpatchsets Sun May 13 20:23:42 2012 @@ -60,10 +60,8 @@ if line.startswith('diff '): if filename: chunks = patching.ParsePatchToChunks(diff) - if not chunks: - # diff cannot be parsed - return None - patches.append((filename, ''.join(diff), chunks)) + if chunks: + patches.append((filename, ''.join(diff), chunks)) diff = [] filename = line.split()[-1] if filename.startswith("b/"): @@ -75,9 +73,8 @@ # add last patch if filename: chunks = patching.ParsePatchToChunks(diff) - if not chunks: - return None - patches.append((filename, ''.join(diff), chunks)) + if chunks: + patches.append((filename, ''.join(diff), chunks)) return patches def find_bases(data): @@ -92,6 +89,8 @@ c = connection.cursor() pieces = hg_splitpatch(data) if not pieces: + if verbose: + print "Splitting failed" return None, None bases = [] for filename, data, chunks in pieces: From python-checkins at python.org Sun May 13 20:50:17 2012 From: python-checkins at python.org (antoine.pitrou) Date: Sun, 13 May 2012 20:50:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314417=3A_Mutating_?= =?utf8?q?a_dict_during_lookup_now_restarts_the_lookup_instead_of?= Message-ID: http://hg.python.org/cpython/rev/93748e2d64e3 changeset: 76907:93748e2d64e3 user: Antoine Pitrou date: Sun May 13 20:48:01 2012 +0200 summary: Issue #14417: Mutating a dict during lookup now restarts the lookup instead of raising a RuntimeError (undoes issue #14205). files: Doc/library/stdtypes.rst | 4 ---- Doc/whatsnew/3.3.rst | 7 ------- Lib/test/test_dict.py | 7 +++---- Misc/NEWS | 3 +++ Objects/dictobject.c | 17 +++++++++-------- 5 files changed, 15 insertions(+), 23 deletions(-) diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2210,10 +2210,6 @@ See :class:`collections.Counter` for a complete implementation including other methods helpful for accumulating and managing tallies. - .. versionchanged:: 3.3 - If the dict is modified during the lookup, a :exc:`RuntimeError` - exception is now raised. - .. describe:: d[key] = value Set ``d[key]`` to *value*. diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -557,13 +557,6 @@ (:issue:`12170`) -* A dict lookup now raises a :exc:`RuntimeError` if the dict is modified during - the lookup. If you implement your own comparison function for objects used - as dict keys and the dict is shared by multiple threads, access to the dict - should be protected by a lock. - - (:issue:`14205`) - * New methods have been added to :class:`list` and :class:`bytearray`: ``copy()`` and ``clear()``. diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -411,7 +411,7 @@ d[i+1] = 1 def test_mutating_lookup(self): - # changing dict during a lookup + # changing dict during a lookup (issue #14417) class NastyKey: mutate_dict = None @@ -433,9 +433,8 @@ key2 = NastyKey(2) d = {key1: 1} NastyKey.mutate_dict = (d, key1) - with self.assertRaisesRegex(RuntimeError, - 'dictionary changed size during lookup'): - d[key2] = 2 + d[key2] = 2 + self.assertEqual(d, {key2: 2}) def test_repr(self): d = {} diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14417: Mutating a dict during lookup now restarts the lookup instead + of raising a RuntimeError (undoes issue #14205). + - Issue #14738: Speed-up UTF-8 decoding on non-ASCII data. Patch by Serhiy Storchaka. diff --git a/Objects/dictobject.c b/Objects/dictobject.c --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -439,12 +439,15 @@ register size_t i; register size_t perturb; register PyDictKeyEntry *freeslot; - register size_t mask = DK_MASK(mp->ma_keys); - PyDictKeyEntry *ep0 = &mp->ma_keys->dk_entries[0]; + register size_t mask; + PyDictKeyEntry *ep0; register PyDictKeyEntry *ep; register int cmp; PyObject *startkey; +top: + mask = DK_MASK(mp->ma_keys); + ep0 = &mp->ma_keys->dk_entries[0]; i = (size_t)hash & mask; ep = &ep0[i]; if (ep->me_key == NULL || ep->me_key == key) { @@ -468,9 +471,8 @@ } } else { - PyErr_SetString(PyExc_RuntimeError, - "dictionary changed size during lookup"); - return NULL; + /* The dict was mutated, restart */ + goto top; } } freeslot = NULL; @@ -510,9 +512,8 @@ } } else { - PyErr_SetString(PyExc_RuntimeError, - "dictionary changed size during lookup"); - return NULL; + /* The dict was mutated, restart */ + goto top; } } else if (ep->me_key == dummy && freeslot == NULL) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 22:04:25 2012 From: python-checkins at python.org (mark.dickinson) Date: Sun, 13 May 2012 22:04:25 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0MjQ1?= =?utf8?q?=3A_Improve_floating-point_entry_in_FAQ=2E__Thanks_Zbyszek?= Message-ID: http://hg.python.org/cpython/rev/2b2a7861255d changeset: 76908:2b2a7861255d branch: 3.2 parent: 76901:7a046f1ddd07 user: Mark Dickinson date: Sun May 13 21:00:35 2012 +0100 summary: Issue #14245: Improve floating-point entry in FAQ. Thanks Zbyszek J?drzejewski-Szmek for some of the wording. files: Doc/faq/design.rst | 63 ++++++++++++++------------------- Misc/ACKS | 1 + 2 files changed, 27 insertions(+), 37 deletions(-) diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -43,56 +43,45 @@ See the next question. -Why are floating point calculations so inaccurate? +Why are floating-point calculations so inaccurate? -------------------------------------------------- -People are often very surprised by results like this:: +Users are often surprised by results like this:: - >>> 1.2 - 1.0 - 0.199999999999999996 + >>> 1.2 - 1.0 + 0.199999999999999996 -and think it is a bug in Python. It's not. This has nothing to do with Python, -but with how the underlying C platform handles floating point numbers, and -ultimately with the inaccuracies introduced when writing down numbers as a -string of a fixed number of digits. +and think it is a bug in Python. It's not. This has little to do with Python, +and much more to do with how the underlying platform handles floating-point +numbers. -The internal representation of floating point numbers uses a fixed number of -binary digits to represent a decimal number. Some decimal numbers can't be -represented exactly in binary, resulting in small roundoff errors. +The :class:`float` type in CPython uses a C ``double`` for storage. A +:class:`float` object's value is stored in binary floating-point with a fixed +precision (typically 53 bits) and Python uses C operations, which in turn rely +on the hardware implementation in the processor, to perform floating-point +operations. This means that as far as floating-point operations are concerned, +Python behaves like many popular languages including C and Java. -In decimal math, there are many numbers that can't be represented with a fixed -number of decimal digits, e.g. 1/3 = 0.3333333333....... +Many numbers that can be written easily in decimal notation cannot be expressed +exactly in binary floating-point. For example, after:: -In base 2, 1/2 = 0.1, 1/4 = 0.01, 1/8 = 0.001, etc. .2 equals 2/10 equals 1/5, -resulting in the binary fractional number 0.001100110011001... + >>> x = 1.2 -Floating point numbers only have 32 or 64 bits of precision, so the digits are -cut off at some point, and the resulting number is 0.199999999999999996 in -decimal, not 0.2. +the value stored for ``x`` is a (very good) approximation to the decimal value +``1.2``, but is not exactly equal to it. On a typical machine, the actual +stored value is:: -A floating point number's ``repr()`` function prints as many digits are -necessary to make ``eval(repr(f)) == f`` true for any float f. The ``str()`` -function prints fewer digits and this often results in the more sensible number -that was probably intended:: + 1.0011001100110011001100110011001100110011001100110011 (binary) - >>> 1.1 - 0.9 - 0.20000000000000007 - >>> print(1.1 - 0.9) - 0.2 +which is exactly:: -One of the consequences of this is that it is error-prone to compare the result -of some computation to a float with ``==``. Tiny inaccuracies may mean that -``==`` fails. Instead, you have to check that the difference between the two -numbers is less than a certain threshold:: + 1.1999999999999999555910790149937383830547332763671875 (decimal) - epsilon = 0.0000000000001 # Tiny allowed error - expected_result = 0.4 +The typical precision of 53 bits provides Python floats with 15-16 +decimal digits of accuracy. - if expected_result-epsilon <= computation() <= expected_result+epsilon: - ... - -Please see the chapter on :ref:`floating point arithmetic ` in -the Python tutorial for more information. +For a fuller explanation, please see the :ref:`floating point arithmetic +` chapter in the Python tutorial. Why are Python strings immutable? diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -452,6 +452,7 @@ Jack Jansen Bill Janssen Thomas Jarosch +Zbyszek J?drzejewski-Szmek Drew Jenkins Flemming Kj?r Jensen MunSic Jeong -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 22:04:26 2012 From: python-checkins at python.org (mark.dickinson) Date: Sun, 13 May 2012 22:04:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314245=3A_Merge_changes_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/a79b07e05d0d changeset: 76909:a79b07e05d0d parent: 76907:93748e2d64e3 parent: 76908:2b2a7861255d user: Mark Dickinson date: Sun May 13 21:02:22 2012 +0100 summary: Issue #14245: Merge changes from 3.2. files: Doc/faq/design.rst | 63 ++++++++++++++------------------- 1 files changed, 26 insertions(+), 37 deletions(-) diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -43,56 +43,45 @@ See the next question. -Why are floating point calculations so inaccurate? +Why are floating-point calculations so inaccurate? -------------------------------------------------- -People are often very surprised by results like this:: +Users are often surprised by results like this:: - >>> 1.2 - 1.0 - 0.199999999999999996 + >>> 1.2 - 1.0 + 0.199999999999999996 -and think it is a bug in Python. It's not. This has nothing to do with Python, -but with how the underlying C platform handles floating point numbers, and -ultimately with the inaccuracies introduced when writing down numbers as a -string of a fixed number of digits. +and think it is a bug in Python. It's not. This has little to do with Python, +and much more to do with how the underlying platform handles floating-point +numbers. -The internal representation of floating point numbers uses a fixed number of -binary digits to represent a decimal number. Some decimal numbers can't be -represented exactly in binary, resulting in small roundoff errors. +The :class:`float` type in CPython uses a C ``double`` for storage. A +:class:`float` object's value is stored in binary floating-point with a fixed +precision (typically 53 bits) and Python uses C operations, which in turn rely +on the hardware implementation in the processor, to perform floating-point +operations. This means that as far as floating-point operations are concerned, +Python behaves like many popular languages including C and Java. -In decimal math, there are many numbers that can't be represented with a fixed -number of decimal digits, e.g. 1/3 = 0.3333333333....... +Many numbers that can be written easily in decimal notation cannot be expressed +exactly in binary floating-point. For example, after:: -In base 2, 1/2 = 0.1, 1/4 = 0.01, 1/8 = 0.001, etc. .2 equals 2/10 equals 1/5, -resulting in the binary fractional number 0.001100110011001... + >>> x = 1.2 -Floating point numbers only have 32 or 64 bits of precision, so the digits are -cut off at some point, and the resulting number is 0.199999999999999996 in -decimal, not 0.2. +the value stored for ``x`` is a (very good) approximation to the decimal value +``1.2``, but is not exactly equal to it. On a typical machine, the actual +stored value is:: -A floating point number's ``repr()`` function prints as many digits are -necessary to make ``eval(repr(f)) == f`` true for any float f. The ``str()`` -function prints fewer digits and this often results in the more sensible number -that was probably intended:: + 1.0011001100110011001100110011001100110011001100110011 (binary) - >>> 1.1 - 0.9 - 0.20000000000000007 - >>> print(1.1 - 0.9) - 0.2 +which is exactly:: -One of the consequences of this is that it is error-prone to compare the result -of some computation to a float with ``==``. Tiny inaccuracies may mean that -``==`` fails. Instead, you have to check that the difference between the two -numbers is less than a certain threshold:: + 1.1999999999999999555910790149937383830547332763671875 (decimal) - epsilon = 0.0000000000001 # Tiny allowed error - expected_result = 0.4 +The typical precision of 53 bits provides Python floats with 15-16 +decimal digits of accuracy. - if expected_result-epsilon <= computation() <= expected_result+epsilon: - ... - -Please see the chapter on :ref:`floating point arithmetic ` in -the Python tutorial for more information. +For a fuller explanation, please see the :ref:`floating point arithmetic +` chapter in the Python tutorial. Why are Python strings immutable? -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 13 23:17:28 2012 From: python-checkins at python.org (brian.curtin) Date: Sun, 13 May 2012 23:17:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_changeset=3A_76969=3A0cbe10?= =?utf8?q?99226d?= Message-ID: http://hg.python.org/cpython/rev/3d509c4a72bc changeset: 76910:3d509c4a72bc user: Brian Curtin date: Sun May 13 16:16:09 2012 -0500 summary: changeset: 76969:0cbe1099226d branch: vs2010 tag: tip user: Brian Curtin date: Sun May 13 16:15:11 2012 -0500 summary: Changes to allow Profile Guided Optimization builds to succeed on VS2010 files: PCbuild/_bz2.vcxproj | 4 +++ PCbuild/_ctypes.vcxproj | 22 +++++++++++++++++++ PCbuild/_ctypes_test.vcxproj | 4 +++ PCbuild/_decimal.vcxproj | 22 +++++++++++++++++++ PCbuild/_elementtree.vcxproj | 4 +++ PCbuild/_hashlib.vcxproj | 4 +++ PCbuild/_lzma.vcxproj | 4 +++ PCbuild/_msi.vcxproj | 4 +++ PCbuild/_multiprocessing.vcxproj | 4 +++ PCbuild/_socket.vcxproj | 4 +++ PCbuild/_sqlite3.vcxproj | 4 +++ PCbuild/_ssl.vcxproj | 4 +++ PCbuild/_testbuffer.vcxproj | 23 ++++++++++++++++++++ PCbuild/_testcapi.vcxproj | 4 +++ PCbuild/_tkinter.vcxproj | 4 +++ PCbuild/kill_python.vcxproj | 2 + PCbuild/pcbuild.sln | 12 +++++----- PCbuild/pyexpat.vcxproj | 4 +++ PCbuild/python.vcxproj | 4 +++ PCbuild/pythoncore.vcxproj | 5 ++++ PCbuild/pythonw.vcxproj | 4 +++ PCbuild/select.vcxproj | 4 +++ PCbuild/sqlite3.vcxproj | 2 + PCbuild/unicodedata.vcxproj | 4 +++ PCbuild/winsound.vcxproj | 4 +++ PCbuild/xxlimited.vcxproj | 13 +++++++++- 26 files changed, 165 insertions(+), 8 deletions(-) diff --git a/PCbuild/_bz2.vcxproj b/PCbuild/_bz2.vcxproj --- a/PCbuild/_bz2.vcxproj +++ b/PCbuild/_bz2.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_ctypes.vcxproj b/PCbuild/_ctypes.vcxproj --- a/PCbuild/_ctypes.vcxproj +++ b/PCbuild/_ctypes.vcxproj @@ -83,9 +83,13 @@ + + + + @@ -97,9 +101,15 @@ + + + + + + @@ -140,6 +150,16 @@ .pyd .pyd + .pyd + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + $(OutDirPGI)\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + .pyd @@ -207,6 +227,7 @@ NotSet 0x1D1A0000 MachineX64 + $(OutDir)python33.lib;%(AdditionalDependencies) @@ -231,6 +252,7 @@ NotSet 0x1D1A0000 MachineX64 + $(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) diff --git a/PCbuild/_ctypes_test.vcxproj b/PCbuild/_ctypes_test.vcxproj --- a/PCbuild/_ctypes_test.vcxproj +++ b/PCbuild/_ctypes_test.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_decimal.vcxproj b/PCbuild/_decimal.vcxproj --- a/PCbuild/_decimal.vcxproj +++ b/PCbuild/_decimal.vcxproj @@ -83,9 +83,13 @@ + + + + @@ -97,9 +101,15 @@ + + + + + + @@ -140,6 +150,16 @@ .pyd $(SolutionDir)\amd64\ $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ + .pyd + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + $(OutDirPGI)\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + .pyd @@ -209,6 +229,7 @@ NotSet 0x1D1A0000 MachineX64 + $(OutDir)python33.lib;%(AdditionalDependencies) @@ -233,6 +254,7 @@ NotSet 0x1D1A0000 MachineX64 + $(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj --- a/PCbuild/_elementtree.vcxproj +++ b/PCbuild/_elementtree.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_hashlib.vcxproj b/PCbuild/_hashlib.vcxproj --- a/PCbuild/_hashlib.vcxproj +++ b/PCbuild/_hashlib.vcxproj @@ -149,6 +149,10 @@ .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_lzma.vcxproj b/PCbuild/_lzma.vcxproj --- a/PCbuild/_lzma.vcxproj +++ b/PCbuild/_lzma.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_msi.vcxproj b/PCbuild/_msi.vcxproj --- a/PCbuild/_msi.vcxproj +++ b/PCbuild/_msi.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_multiprocessing.vcxproj b/PCbuild/_multiprocessing.vcxproj --- a/PCbuild/_multiprocessing.vcxproj +++ b/PCbuild/_multiprocessing.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_socket.vcxproj b/PCbuild/_socket.vcxproj --- a/PCbuild/_socket.vcxproj +++ b/PCbuild/_socket.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj --- a/PCbuild/_sqlite3.vcxproj +++ b/PCbuild/_sqlite3.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_ssl.vcxproj b/PCbuild/_ssl.vcxproj --- a/PCbuild/_ssl.vcxproj +++ b/PCbuild/_ssl.vcxproj @@ -149,6 +149,10 @@ .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_testbuffer.vcxproj b/PCbuild/_testbuffer.vcxproj --- a/PCbuild/_testbuffer.vcxproj +++ b/PCbuild/_testbuffer.vcxproj @@ -83,9 +83,13 @@ + + + + @@ -97,9 +101,15 @@ + + + + + + @@ -140,6 +150,17 @@ .pyd .pyd + .pyd + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + $(OutDirPGI)\ + $(ProjectName) + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + .pyd @@ -181,6 +202,7 @@ 0x1e1F0000 MachineX64 + $(OutDir)python33.lib;%(AdditionalDependencies) @@ -195,6 +217,7 @@ 0x1e1F0000 MachineX64 + $(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj --- a/PCbuild/_testcapi.vcxproj +++ b/PCbuild/_testcapi.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/_tkinter.vcxproj b/PCbuild/_tkinter.vcxproj --- a/PCbuild/_tkinter.vcxproj +++ b/PCbuild/_tkinter.vcxproj @@ -149,6 +149,10 @@ .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/kill_python.vcxproj b/PCbuild/kill_python.vcxproj --- a/PCbuild/kill_python.vcxproj +++ b/PCbuild/kill_python.vcxproj @@ -86,6 +86,8 @@ .exe .exe .exe + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -551,12 +551,12 @@ {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|Win32.Build.0 = Debug|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|x64.ActiveCfg = Debug|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|x64.Build.0 = Debug|x64 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.ActiveCfg = Release|Win32 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.Build.0 = Release|Win32 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|x64.ActiveCfg = Release|x64 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|x64.Build.0 = Release|x64 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGUpdate|Win32.ActiveCfg = Release|Win32 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGUpdate|Win32.Build.0 = Release|Win32 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|x64.Build.0 = PGInstrument|x64 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGUpdate|Win32.Build.0 = PGUpdate|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGUpdate|x64.ActiveCfg = Release|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGUpdate|x64.Build.0 = Release|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Release|Win32.ActiveCfg = Release|Win32 diff --git a/PCbuild/pyexpat.vcxproj b/PCbuild/pyexpat.vcxproj --- a/PCbuild/pyexpat.vcxproj +++ b/PCbuild/pyexpat.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj --- a/PCbuild/python.vcxproj +++ b/PCbuild/python.vcxproj @@ -160,6 +160,10 @@ .exe .exe .exe + .exe + .exe + .exe + .exe diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -154,6 +154,11 @@ $(PyDllName) $(PyDllName)_d $(PyDllName) + .dll + $(PyDllName) + $(PyDllName) + $(PyDllName) + $(PyDllName) diff --git a/PCbuild/pythonw.vcxproj b/PCbuild/pythonw.vcxproj --- a/PCbuild/pythonw.vcxproj +++ b/PCbuild/pythonw.vcxproj @@ -153,6 +153,10 @@ .exe .exe .pyd + .exe + .exe + .exe + .exe diff --git a/PCbuild/select.vcxproj b/PCbuild/select.vcxproj --- a/PCbuild/select.vcxproj +++ b/PCbuild/select.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/sqlite3.vcxproj b/PCbuild/sqlite3.vcxproj --- a/PCbuild/sqlite3.vcxproj +++ b/PCbuild/sqlite3.vcxproj @@ -151,6 +151,8 @@ .dll + .dll + .dll diff --git a/PCbuild/unicodedata.vcxproj b/PCbuild/unicodedata.vcxproj --- a/PCbuild/unicodedata.vcxproj +++ b/PCbuild/unicodedata.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/winsound.vcxproj b/PCbuild/winsound.vcxproj --- a/PCbuild/winsound.vcxproj +++ b/PCbuild/winsound.vcxproj @@ -150,6 +150,10 @@ .pyd .pyd + .pyd + .pyd + .pyd + .pyd diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj --- a/PCbuild/xxlimited.vcxproj +++ b/PCbuild/xxlimited.vcxproj @@ -119,6 +119,12 @@ .pyd .pyd + $(SolutionDir)$(PlatformName)-pgo\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + $(OutDirPGI)\ + $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ + .pyd + .pyd @@ -128,6 +134,7 @@ wsock32.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 + $(OutDir)$(ProjectName).pyd @@ -135,7 +142,7 @@ X64 - wsock32.lib;$(OutDir)python33.lib;%(AdditionalDependencies) + wsock32.lib;$(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 @@ -148,6 +155,7 @@ wsock32.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 + $(OutDirPGI)$(ProjectName).pyd @@ -155,10 +163,11 @@ X64 - wsock32.lib;%(AdditionalDependencies) + wsock32.lib;$(OutDir)python33.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 MachineX64 + $(OutDirPGI)$(ProjectName).pyd -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 01:52:38 2012 From: python-checkins at python.org (brian.curtin) Date: Mon, 14 May 2012 01:52:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Convert_build_slave_scripts?= =?utf8?q?_to_work_with_VS2010?= Message-ID: http://hg.python.org/cpython/rev/faecaa6dd5bc changeset: 76911:faecaa6dd5bc user: Brian Curtin date: Sun May 13 18:51:43 2012 -0500 summary: Convert build slave scripts to work with VS2010 files: Tools/buildbot/build-amd64.bat | 5 +++-- Tools/buildbot/build.bat | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Tools/buildbot/build-amd64.bat b/Tools/buildbot/build-amd64.bat --- a/Tools/buildbot/build-amd64.bat +++ b/Tools/buildbot/build-amd64.bat @@ -2,5 +2,6 @@ cmd /c Tools\buildbot\external-amd64.bat call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat" x86_amd64 cmd /c Tools\buildbot\clean-amd64.bat -vcbuild /useenv PCbuild\kill_python.vcproj "Debug|x64" && PCbuild\amd64\kill_python_d.exe -vcbuild PCbuild\pcbuild.sln "Debug|x64" +msbuild /p:useenv=true PCbuild\kill_python.vcxproj /p:Configuration=Debug /p:PlatformTarget=x64 +PCbuild\amd64\kill_python_d.exe +msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x64 diff --git a/Tools/buildbot/build.bat b/Tools/buildbot/build.bat --- a/Tools/buildbot/build.bat +++ b/Tools/buildbot/build.bat @@ -2,6 +2,7 @@ cmd /c Tools\buildbot\external.bat call "%VS100COMNTOOLS%vsvars32.bat" cmd /c Tools\buildbot\clean.bat -vcbuild /useenv PCbuild\kill_python.vcproj "Debug|Win32" && PCbuild\kill_python_d.exe -vcbuild /useenv PCbuild\pcbuild.sln "Debug|Win32" +msbuild /p:useenv=true PCbuild\kill_python.vcxproj /p:Configuration=Debug /p:PlatformTarget=x86 +PCbuild\kill_python_d.exe +msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x86 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 02:00:00 2012 From: python-checkins at python.org (brian.curtin) Date: Mon, 14 May 2012 02:00:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Correct_PlatformTarget-=3EP?= =?utf8?q?latform_usage_for_building_the_solution?= Message-ID: http://hg.python.org/cpython/rev/6b8f34a1cb22 changeset: 76912:6b8f34a1cb22 user: Brian Curtin date: Sun May 13 18:59:26 2012 -0500 summary: Correct PlatformTarget->Platform usage for building the solution files: Tools/buildbot/build-amd64.bat | 2 +- Tools/buildbot/build.bat | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Tools/buildbot/build-amd64.bat b/Tools/buildbot/build-amd64.bat --- a/Tools/buildbot/build-amd64.bat +++ b/Tools/buildbot/build-amd64.bat @@ -4,4 +4,4 @@ cmd /c Tools\buildbot\clean-amd64.bat msbuild /p:useenv=true PCbuild\kill_python.vcxproj /p:Configuration=Debug /p:PlatformTarget=x64 PCbuild\amd64\kill_python_d.exe -msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x64 +msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=x64 diff --git a/Tools/buildbot/build.bat b/Tools/buildbot/build.bat --- a/Tools/buildbot/build.bat +++ b/Tools/buildbot/build.bat @@ -4,5 +4,5 @@ cmd /c Tools\buildbot\clean.bat msbuild /p:useenv=true PCbuild\kill_python.vcxproj /p:Configuration=Debug /p:PlatformTarget=x86 PCbuild\kill_python_d.exe -msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x86 +msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Debug /p:Platform=Win32 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 05:46:21 2012 From: python-checkins at python.org (brian.curtin) Date: Mon, 14 May 2012 05:46:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_the_clean_and_MSI_sc?= =?utf8?q?ripts_for_VS2010?= Message-ID: http://hg.python.org/cpython/rev/e0540079940c changeset: 76913:e0540079940c user: Brian Curtin date: Sun May 13 22:45:57 2012 -0500 summary: Update the clean and MSI scripts for VS2010 files: Tools/buildbot/buildmsi.bat | 2 +- Tools/buildbot/clean-amd64.bat | 4 ++-- Tools/buildbot/clean.bat | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Tools/buildbot/buildmsi.bat b/Tools/buildbot/buildmsi.bat --- a/Tools/buildbot/buildmsi.bat +++ b/Tools/buildbot/buildmsi.bat @@ -5,7 +5,7 @@ call "%VS100COMNTOOLS%vsvars32.bat" @rem build Python -vcbuild /useenv PCbuild\pcbuild.sln "Release|Win32" +msbuild /p:useenv=true PCbuild\pcbuild.sln /p:Configuration=Release /p:Platform=Win32 @rem build the documentation bash.exe -c 'cd Doc;make PYTHON=python2.5 update htmlhelp' diff --git a/Tools/buildbot/clean-amd64.bat b/Tools/buildbot/clean-amd64.bat --- a/Tools/buildbot/clean-amd64.bat +++ b/Tools/buildbot/clean-amd64.bat @@ -5,6 +5,6 @@ @echo Deleting test leftovers ... rmdir /s /q build cd PCbuild -vcbuild /clean pcbuild.sln "Release|x64" -vcbuild /clean pcbuild.sln "Debug|x64" +msbuild /target:clean pcbuild.sln /p:Configuration=Release /p:PlatformTarget=x64 +msbuild /target:clean pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x64 cd .. diff --git a/Tools/buildbot/clean.bat b/Tools/buildbot/clean.bat --- a/Tools/buildbot/clean.bat +++ b/Tools/buildbot/clean.bat @@ -5,6 +5,6 @@ @echo Deleting test leftovers ... rmdir /s /q build cd PCbuild -vcbuild /clean pcbuild.sln "Release|Win32" -vcbuild /clean pcbuild.sln "Debug|Win32" +msbuild /target:clean pcbuild.sln /p:Configuration=Release /p:PlatformTarget=x86 +msbuild /target:clean pcbuild.sln /p:Configuration=Debug /p:PlatformTarget=x86 cd .. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon May 14 05:50:39 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 14 May 2012 05:50:39 +0200 Subject: [Python-checkins] Daily reference leaks (6b8f34a1cb22): sum=0 Message-ID: results for 6b8f34a1cb22 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogBHtDz1', '-x'] From python-checkins at python.org Mon May 14 12:22:51 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 12:22:51 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_release_build_settings?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/6fb8151c0579 changeset: 76914:6fb8151c0579 user: Martin v. L?wis date: Mon May 14 12:22:08 2012 +0200 summary: Fix release build settings. files: PCbuild/xxlimited.vcxproj | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj --- a/PCbuild/xxlimited.vcxproj +++ b/PCbuild/xxlimited.vcxproj @@ -121,8 +121,6 @@ .pyd $(SolutionDir)$(PlatformName)-pgo\ $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - $(OutDirPGI)\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ .pyd .pyd -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 12:32:38 2012 From: python-checkins at python.org (ezio.melotti) Date: Mon, 14 May 2012 12:32:38 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NDA1OiByZW1v?= =?utf8?q?ve_outdated/broken/duplicate_links=2E?= Message-ID: http://hg.python.org/cpython/rev/855a6796312b changeset: 76915:855a6796312b branch: 2.7 parent: 76903:bf3cb58dcfe7 user: Ezio Melotti date: Mon May 14 13:26:45 2012 +0300 summary: #14405: remove outdated/broken/duplicate links. files: Doc/tools/sphinxext/indexsidebar.html | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/Doc/tools/sphinxext/indexsidebar.html b/Doc/tools/sphinxext/indexsidebar.html --- a/Doc/tools/sphinxext/indexsidebar.html +++ b/Doc/tools/sphinxext/indexsidebar.html @@ -11,13 +11,8 @@

Other resources

 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 12:32:39 2012 From: python-checkins at python.org (ezio.melotti) Date: Mon, 14 May 2012 12:32:39 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NDA1OiByZW1v?= =?utf8?q?ve_outdated/broken/duplicate_links=2E?= Message-ID: http://hg.python.org/cpython/rev/b50d4ae6aaa3 changeset: 76916:b50d4ae6aaa3 branch: 3.2 parent: 76908:2b2a7861255d user: Ezio Melotti date: Mon May 14 13:26:45 2012 +0300 summary: #14405: remove outdated/broken/duplicate links. files: Doc/tools/sphinxext/indexsidebar.html | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/Doc/tools/sphinxext/indexsidebar.html b/Doc/tools/sphinxext/indexsidebar.html --- a/Doc/tools/sphinxext/indexsidebar.html +++ b/Doc/tools/sphinxext/indexsidebar.html @@ -10,13 +10,8 @@

Other resources

-- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 12:32:41 2012 From: python-checkins at python.org (ezio.melotti) Date: Mon, 14 May 2012 12:32:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314405=3A_merge_with_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/8a8120ee1202 changeset: 76917:8a8120ee1202 parent: 76914:6fb8151c0579 parent: 76916:b50d4ae6aaa3 user: Ezio Melotti date: Mon May 14 13:32:26 2012 +0300 summary: #14405: merge with 3.2. files: Doc/tools/sphinxext/indexsidebar.html | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/Doc/tools/sphinxext/indexsidebar.html b/Doc/tools/sphinxext/indexsidebar.html --- a/Doc/tools/sphinxext/indexsidebar.html +++ b/Doc/tools/sphinxext/indexsidebar.html @@ -10,13 +10,8 @@

Other resources

-- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 13:20:15 2012 From: python-checkins at python.org (lars.gustaebel) Date: Mon, 14 May 2012 13:20:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313815=3A_Resurrect?= =?utf8?q?_the_ExFileObject_class=2E?= Message-ID: http://hg.python.org/cpython/rev/ab6496b98ac4 changeset: 76918:ab6496b98ac4 user: Lars Gust?bel date: Mon May 14 13:18:16 2012 +0200 summary: Issue #13815: Resurrect the ExFileObject class. After a discussion in the tracker, the decision was made to keep the ExFileObject class after all as a subclass of io.BufferedReader instead of removing it completely. files: Lib/tarfile.py | 17 +++++++++-------- Misc/NEWS | 4 ++-- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -758,6 +758,13 @@ self.closed = True #class _FileInFile +class ExFileObject(io.BufferedReader): + + def __init__(self, tarfile, tarinfo): + fileobj = _FileInFile(tarfile.fileobj, tarinfo.offset_data, + tarinfo.size, tarinfo.sparse) + super().__init__(fileobj) +#class ExFileObject #------------------ # Exported Classes @@ -1443,8 +1450,7 @@ tarinfo = TarInfo # The default TarInfo class to use. - fileobject = None # The file-object for extractfile() or - # io.BufferedReader if None. + fileobject = ExFileObject # The file-object for extractfile(). def __init__(self, name=None, mode="r", fileobj=None, format=None, tarinfo=None, dereference=None, ignore_zeros=None, encoding=None, @@ -2081,12 +2087,7 @@ if tarinfo.isreg() or tarinfo.type not in SUPPORTED_TYPES: # Members with unknown types are treated as regular files. - if self.fileobject is None: - fileobj = _FileInFile(self.fileobj, tarinfo.offset_data, tarinfo.size, tarinfo.sparse) - return io.BufferedReader(fileobj) - else: - # Keep the traditional pre-3.3 API intact. - return self.fileobject(self, tarinfo) + return self.fileobject(self, tarinfo) elif tarinfo.islnk() or tarinfo.issym(): if isinstance(self.fileobj, _Stream): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -26,6 +26,8 @@ Library ------- +- Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. + - Issue #14532: Add a secure_compare() helper to the hmac module, to mitigate timing attacks. Patch by Jon Oberheide. @@ -181,8 +183,6 @@ Library ------- -- Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. - - Issue #14768: os.path.expanduser('~/a') doesn't works correctly when HOME is '/'. - Issue #14371: Support bzip2 in zipfile module. Patch by Serhiy Storchaka. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 13:52:12 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 13:52:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Upgrade_bzip2_to_1=2E0=2E6?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/1338b6fece2d changeset: 76919:1338b6fece2d user: Martin v. L?wis date: Mon May 14 13:52:03 2012 +0200 summary: Upgrade bzip2 to 1.0.6. files: PC/VC6/bz2.dsp | 8 ++++---- PC/VC6/readme.txt | 6 +++--- PC/VS8.0/pyproject.vsprops | 2 +- PC/VS9.0/pyproject.vsprops | 2 +- PCbuild/pyproject.props | 2 +- PCbuild/readme.txt | 10 +++++----- README | 2 ++ Tools/buildbot/external-common.bat | 8 ++++---- 8 files changed, 21 insertions(+), 19 deletions(-) diff --git a/PC/VC6/bz2.dsp b/PC/VC6/bz2.dsp --- a/PC/VC6/bz2.dsp +++ b/PC/VC6/bz2.dsp @@ -44,7 +44,7 @@ # PROP Target_Dir "" F90=df.exe # ADD BASE CPP /nologo /MT /W3 /GX /O2 /D "Py_BUILD_CORE_MODULE" /D "WIN32" /D "NDEBUG" /D "_WINDOWS" /YX /FD /c -# ADD CPP /nologo /MD /W3 /GX /Zi /O2 /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.5" /D "Py_BUILD_CORE_MODULE" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c +# ADD CPP /nologo /MD /W3 /GX /Zi /O2 /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.6" /D "Py_BUILD_CORE_MODULE" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c # ADD BASE MTL /nologo /D "NDEBUG" /mktyplib203 /o "NUL" /win32 # ADD MTL /nologo /D "NDEBUG" /mktyplib203 /o "NUL" /win32 # ADD BASE RSC /l 0x409 /d "NDEBUG" @@ -54,7 +54,7 @@ # ADD BSC32 /nologo LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /machine:I386 -# ADD LINK32 ..\..\..\bzip2-1.0.5\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"libc" /out:"./bz2.pyd" +# ADD LINK32 ..\..\..\bzip2-1.0.6\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"libc" /out:"./bz2.pyd" # SUBTRACT LINK32 /pdb:none /nodefaultlib !ELSEIF "$(CFG)" == "bz2 - Win32 Debug" @@ -72,7 +72,7 @@ # PROP Target_Dir "" F90=df.exe # ADD BASE CPP /nologo /MTd /W3 /Gm /GX /Zi /Od /D "Py_BUILD_CORE_MODULE" /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /YX /FD /c -# ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.5" /D "Py_BUILD_CORE_MODULE" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c +# ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.6" /D "Py_BUILD_CORE_MODULE" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c # ADD BASE MTL /nologo /D "_DEBUG" /mktyplib203 /o "NUL" /win32 # ADD MTL /nologo /D "_DEBUG" /mktyplib203 /o "NUL" /win32 # ADD BASE RSC /l 0x409 /d "_DEBUG" @@ -82,7 +82,7 @@ # ADD BSC32 /nologo LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /debug /machine:I386 /pdbtype:sept -# ADD LINK32 ..\..\..\bzip2-1.0.5\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"msvcrt" /nodefaultlib:"libc" /out:"./bz2_d.pyd" /pdbtype:sept +# ADD LINK32 ..\..\..\bzip2-1.0.6\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"msvcrt" /nodefaultlib:"libc" /out:"./bz2_d.pyd" /pdbtype:sept # SUBTRACT LINK32 /pdb:none !ENDIF diff --git a/PC/VC6/readme.txt b/PC/VC6/readme.txt --- a/PC/VC6/readme.txt +++ b/PC/VC6/readme.txt @@ -120,14 +120,14 @@ Download the source from the python.org copy into the dist directory: - svn export http://svn.python.org/projects/external/bzip2-1.0.5 + svn export http://svn.python.org/projects/external/bzip2-1.0.6 And requires building bz2 first. - cd dist\bzip2-1.0.5 + cd dist\bzip2-1.0.6 nmake -f makefile.msc - All of this managed to build bzip2-1.0.5\libbz2.lib, which the Python + All of this managed to build bzip2-1.0.6\libbz2.lib, which the Python project links in. diff --git a/PC/VS8.0/pyproject.vsprops b/PC/VS8.0/pyproject.vsprops --- a/PC/VS8.0/pyproject.vsprops +++ b/PC/VS8.0/pyproject.vsprops @@ -54,7 +54,7 @@ /> $(SolutionDir)\python.exe ..\.. $(externalsDir)\sqlite-3.7.4 - $(externalsDir)\bzip2-1.0.5 + $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.3 $(externalsDir)\openssl-1.0.0a $(externalsDir)\tcltk diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -121,21 +121,21 @@ Download the source from the python.org copy into the dist directory: - svn export http://svn.python.org/projects/external/bzip2-1.0.5 + svn export http://svn.python.org/projects/external/bzip2-1.0.6 ** NOTE: if you use the Tools\buildbot\external(-amd64).bat approach for obtaining external sources then you don't need to manually get the source above via subversion. ** A custom pre-link step in the bz2 project settings should manage to - build bzip2-1.0.5\libbz2.lib by magic before bz2.pyd (or bz2_d.pyd) is + build bzip2-1.0.6\libbz2.lib by magic before bz2.pyd (or bz2_d.pyd) is linked in PCbuild\. However, the bz2 project is not smart enough to remove anything under - bzip2-1.0.5\ when you do a clean, so if you want to rebuild bzip2.lib - you need to clean up bzip2-1.0.5\ by hand. + bzip2-1.0.6\ when you do a clean, so if you want to rebuild bzip2.lib + you need to clean up bzip2-1.0.6\ by hand. All of this managed to build libbz2.lib in - bzip2-1.0.5\$platform-$configuration\, which the Python project links in. + bzip2-1.0.6\$platform-$configuration\, which the Python project links in. _lzma Python wrapper for the liblzma compression library. diff --git a/README b/README --- a/README +++ b/README @@ -15,6 +15,8 @@ On Unix, Linux, BSD, OSX, and Cygwin: +New text + ./configure make make test diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -4,7 +4,7 @@ cd .. @rem XXX: If you need to force the buildbots to start from a fresh environment, uncomment @rem the following, check it in, then check it out, comment it out, then check it back in. - at rem if exist bzip2-1.0.5 rd /s/q bzip2-1.0.5 + at rem if exist bzip2-1.0.6 rd /s/q bzip2-1.0.6 @rem if exist tcltk rd /s/q tcltk @rem if exist tcltk64 rd /s/q tcltk64 @rem if exist tcl8.4.12 rd /s/q tcl8.4.12 @@ -18,9 +18,9 @@ @rem if exist sqlite-3.7.4 rd /s/q sqlite-3.7.4 @rem bzip -if not exist bzip2-1.0.5 ( - rd /s/q bzip2-1.0.3 - svn export http://svn.python.org/projects/external/bzip2-1.0.5 +if not exist bzip2-1.0.6 ( + rd /s/q bzip2-1.0.5 + svn export http://svn.python.org/projects/external/bzip2-1.0.6 ) @rem Sleepycat db -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 14:08:53 2012 From: python-checkins at python.org (nick.coghlan) Date: Mon, 14 May 2012 14:08:53 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Accept_PEP_415?= Message-ID: http://hg.python.org/peps/rev/f8e5111b941d changeset: 4371:f8e5111b941d user: Nick Coghlan date: Mon May 14 22:08:40 2012 +1000 summary: Accept PEP 415 files: pep-0415.txt | 13 ++++++++++++- pep-3144.txt | 14 ++++++++++---- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/pep-0415.txt b/pep-0415.txt --- a/pep-0415.txt +++ b/pep-0415.txt @@ -4,12 +4,13 @@ Last-Modified: $Date$ Author: Benjamin Peterson BDFL-Delegate: Nick Coghlan -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 26-Feb-2012 Python-Version: 3.3 Post-History: 26-Feb-2012 +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119467.html Abstract @@ -20,6 +21,13 @@ syntax. This PEP proposes to implement context and cause suppression differently. + +PEP Acceptance +============== + +This PEP was accepted by Nick Coghlan on the 14th of May, 2012. + + Rationale ========= @@ -40,6 +48,7 @@ ``__cause__`` should be set to ``Ellipsis``. Using ``Ellipsis`` by default for ``__cause__`` makes it asymmetrical with ``__context__``. + Proposal ======== @@ -62,6 +71,7 @@ where ``exc.__cause__ = cause`` implicitly sets ``exc.__suppress_context__``. + Patches ======= @@ -74,6 +84,7 @@ .. _issue 14133: http://bugs.python.org/issue14133 + Copyright ========= diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -5,11 +5,12 @@ Author: Peter Moody BDFL-Delegate: Nick Coghlan Discussions-To: -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/plain Created: 6-Feb-2012 Python-Version: 3.3 +Resolution: Refer to PEP text Abstract: @@ -17,6 +18,11 @@ This PEP proposes a design and for an IP address manipulation module for python. +PEP Acceptance: + + This PEP was accepted by Nick Coghlan (on the BDFL's behalf) on May + 13th, 2012, based on the PEP text and the reference implementation at + Motivation: Several very good IP address modules for python already exist. @@ -55,9 +61,9 @@ * A few attributes were renamed to disambiguate their purpose as well. (eg. network, network_address) - * A number of methods and functions which returned containers in ipaddr now - return iterators. This includes, subnets, address_exclude, - summarize_address_range and collapse_address_list. + * A number of methods and functions which returned containers in ipaddr + now return iterators. This includes, subnets, address_exclude, + summarize_address_range and collapse_address_list. Due to the backwards incompatible API changes between ipaddress and ipaddr, -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 14 14:12:08 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 14:12:08 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Relabel_bzip2_filters=2E?= Message-ID: http://hg.python.org/cpython/rev/a5d56e2273f1 changeset: 76920:a5d56e2273f1 user: Martin v. L?wis date: Mon May 14 14:12:00 2012 +0200 summary: Relabel bzip2 filters. files: PC/VS8.0/bz2.vcproj | 4 ++-- PC/VS9.0/_bz2.vcproj | 4 ++-- PCbuild/_bz2.vcxproj.filters | 24 ++++++++++++------------ 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/PC/VS8.0/bz2.vcproj b/PC/VS8.0/bz2.vcproj --- a/PC/VS8.0/bz2.vcproj +++ b/PC/VS8.0/bz2.vcproj @@ -532,7 +532,7 @@ {f53a859d-dad2-4d5b-ae41-f28d8b571f5a} - + {7e0bed05-ae33-43b7-8797-656455bbb7f3} - + {ed574b89-6983-4cdf-9f98-fe7048d9e89c} @@ -16,33 +16,33 @@ Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Source Files + bzip2 1.0.6 Source Files - bzip2 1.0.5 Header Files + bzip2 1.0.6 Header Files - bzip2 1.0.5 Header Files + bzip2 1.0.6 Header Files - \ No newline at end of file + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 14:17:42 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 14:17:42 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Upgrade_bzip2_t?= =?utf8?b?byAxLjAuNi4=?= Message-ID: http://hg.python.org/cpython/rev/db97a8248863 changeset: 76921:db97a8248863 branch: 3.2 parent: 76916:b50d4ae6aaa3 user: Martin v. L?wis date: Mon May 14 14:13:48 2012 +0200 summary: Upgrade bzip2 to 1.0.6. files: PC/VC6/bz2.dsp | 8 ++++---- PC/VC6/readme.txt | 6 +++--- PC/VS8.0/bz2.vcproj | 4 ++-- PC/VS8.0/pyproject.vsprops | 2 +- PCbuild/bz2.vcproj | 4 ++-- PCbuild/pyproject.vsprops | 2 +- PCbuild/readme.txt | 10 +++++----- Tools/buildbot/external-common.bat | 8 ++++---- 8 files changed, 22 insertions(+), 22 deletions(-) diff --git a/PC/VC6/bz2.dsp b/PC/VC6/bz2.dsp --- a/PC/VC6/bz2.dsp +++ b/PC/VC6/bz2.dsp @@ -44,7 +44,7 @@ # PROP Target_Dir "" F90=df.exe # ADD BASE CPP /nologo /MT /W3 /GX /O2 /D "Py_BUILD_CORE_MODULE" /D "WIN32" /D "NDEBUG" /D "_WINDOWS" /YX /FD /c -# ADD CPP /nologo /MD /W3 /GX /Zi /O2 /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.5" /D "Py_BUILD_CORE_MODULE" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c +# ADD CPP /nologo /MD /W3 /GX /Zi /O2 /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.6" /D "Py_BUILD_CORE_MODULE" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c # ADD BASE MTL /nologo /D "NDEBUG" /mktyplib203 /o "NUL" /win32 # ADD MTL /nologo /D "NDEBUG" /mktyplib203 /o "NUL" /win32 # ADD BASE RSC /l 0x409 /d "NDEBUG" @@ -54,7 +54,7 @@ # ADD BSC32 /nologo LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /machine:I386 -# ADD LINK32 ..\..\..\bzip2-1.0.5\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"libc" /out:"./bz2.pyd" +# ADD LINK32 ..\..\..\bzip2-1.0.6\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"libc" /out:"./bz2.pyd" # SUBTRACT LINK32 /pdb:none /nodefaultlib !ELSEIF "$(CFG)" == "bz2 - Win32 Debug" @@ -72,7 +72,7 @@ # PROP Target_Dir "" F90=df.exe # ADD BASE CPP /nologo /MTd /W3 /Gm /GX /Zi /Od /D "Py_BUILD_CORE_MODULE" /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /YX /FD /c -# ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.5" /D "Py_BUILD_CORE_MODULE" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c +# ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.6" /D "Py_BUILD_CORE_MODULE" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c # ADD BASE MTL /nologo /D "_DEBUG" /mktyplib203 /o "NUL" /win32 # ADD MTL /nologo /D "_DEBUG" /mktyplib203 /o "NUL" /win32 # ADD BASE RSC /l 0x409 /d "_DEBUG" @@ -82,7 +82,7 @@ # ADD BSC32 /nologo LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /debug /machine:I386 /pdbtype:sept -# ADD LINK32 ..\..\..\bzip2-1.0.5\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"msvcrt" /nodefaultlib:"libc" /out:"./bz2_d.pyd" /pdbtype:sept +# ADD LINK32 ..\..\..\bzip2-1.0.6\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"msvcrt" /nodefaultlib:"libc" /out:"./bz2_d.pyd" /pdbtype:sept # SUBTRACT LINK32 /pdb:none !ENDIF diff --git a/PC/VC6/readme.txt b/PC/VC6/readme.txt --- a/PC/VC6/readme.txt +++ b/PC/VC6/readme.txt @@ -120,14 +120,14 @@ Download the source from the python.org copy into the dist directory: - svn export http://svn.python.org/projects/external/bzip2-1.0.5 + svn export http://svn.python.org/projects/external/bzip2-1.0.6 And requires building bz2 first. - cd dist\bzip2-1.0.5 + cd dist\bzip2-1.0.6 nmake -f makefile.msc - All of this managed to build bzip2-1.0.5\libbz2.lib, which the Python + All of this managed to build bzip2-1.0.6\libbz2.lib, which the Python project links in. diff --git a/PC/VS8.0/bz2.vcproj b/PC/VS8.0/bz2.vcproj --- a/PC/VS8.0/bz2.vcproj +++ b/PC/VS8.0/bz2.vcproj @@ -532,7 +532,7 @@ http://hg.python.org/cpython/rev/e33181ff15d0 changeset: 76922:e33181ff15d0 parent: 76920:a5d56e2273f1 parent: 76921:db97a8248863 user: Martin v. L?wis date: Mon May 14 14:17:20 2012 +0200 summary: null merge files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 14:18:58 2012 From: python-checkins at python.org (nick.coghlan) Date: Mon, 14 May 2012 14:18:58 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Not_quite_ready_to_accept_3144?= =?utf8?q?_just_yet?= Message-ID: http://hg.python.org/peps/rev/53352c322036 changeset: 4372:53352c322036 user: Nick Coghlan date: Mon May 14 22:18:46 2012 +1000 summary: Not quite ready to accept 3144 just yet files: pep-3144.txt | 14 ++++---------- 1 files changed, 4 insertions(+), 10 deletions(-) diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -5,12 +5,11 @@ Author: Peter Moody BDFL-Delegate: Nick Coghlan Discussions-To: -Status: Accepted +Status: Draft Type: Standards Track Content-Type: text/plain Created: 6-Feb-2012 Python-Version: 3.3 -Resolution: Refer to PEP text Abstract: @@ -18,11 +17,6 @@ This PEP proposes a design and for an IP address manipulation module for python. -PEP Acceptance: - - This PEP was accepted by Nick Coghlan (on the BDFL's behalf) on May - 13th, 2012, based on the PEP text and the reference implementation at - Motivation: Several very good IP address modules for python already exist. @@ -61,9 +55,9 @@ * A few attributes were renamed to disambiguate their purpose as well. (eg. network, network_address) - * A number of methods and functions which returned containers in ipaddr - now return iterators. This includes, subnets, address_exclude, - summarize_address_range and collapse_address_list. + * A number of methods and functions which returned containers in ipaddr now + return iterators. This includes, subnets, address_exclude, + summarize_address_range and collapse_address_list. Due to the backwards incompatible API changes between ipaddress and ipaddr, -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 14 14:19:22 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 14:19:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Upgrade_to_bzip?= =?utf8?b?MiAxLjAuNi4=?= Message-ID: http://hg.python.org/cpython/rev/2a440ff10f37 changeset: 76923:2a440ff10f37 branch: 2.7 parent: 76915:855a6796312b user: Martin v. L?wis date: Mon May 14 14:19:09 2012 +0200 summary: Upgrade to bzip2 1.0.6. files: PC/VC6/bz2.dsp | 8 ++++---- PC/VC6/readme.txt | 6 +++--- PC/VS8.0/bz2.vcproj | 4 ++-- PC/VS8.0/pyproject.vsprops | 2 +- PCbuild/bz2.vcproj | 4 ++-- PCbuild/pyproject.vsprops | 2 +- PCbuild/readme.txt | 10 +++++----- Tools/buildbot/external-common.bat | 8 ++++---- 8 files changed, 22 insertions(+), 22 deletions(-) diff --git a/PC/VC6/bz2.dsp b/PC/VC6/bz2.dsp --- a/PC/VC6/bz2.dsp +++ b/PC/VC6/bz2.dsp @@ -44,7 +44,7 @@ # PROP Target_Dir "" F90=df.exe # ADD BASE CPP /nologo /MT /W3 /GX /O2 /D "Py_BUILD_CORE_MODULE" /D "WIN32" /D "NDEBUG" /D "_WINDOWS" /YX /FD /c -# ADD CPP /nologo /MD /W3 /GX /Zi /O2 /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.5" /D "Py_BUILD_CORE_MODULE" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c +# ADD CPP /nologo /MD /W3 /GX /Zi /O2 /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.6" /D "Py_BUILD_CORE_MODULE" /D "NDEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c # ADD BASE MTL /nologo /D "NDEBUG" /mktyplib203 /o "NUL" /win32 # ADD MTL /nologo /D "NDEBUG" /mktyplib203 /o "NUL" /win32 # ADD BASE RSC /l 0x409 /d "NDEBUG" @@ -54,7 +54,7 @@ # ADD BSC32 /nologo LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /machine:I386 -# ADD LINK32 ..\..\..\bzip2-1.0.5\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"libc" /out:"./bz2.pyd" +# ADD LINK32 ..\..\..\bzip2-1.0.6\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"libc" /out:"./bz2.pyd" # SUBTRACT LINK32 /pdb:none /nodefaultlib !ELSEIF "$(CFG)" == "bz2 - Win32 Debug" @@ -72,7 +72,7 @@ # PROP Target_Dir "" F90=df.exe # ADD BASE CPP /nologo /MTd /W3 /Gm /GX /Zi /Od /D "Py_BUILD_CORE_MODULE" /D "WIN32" /D "_DEBUG" /D "_WINDOWS" /YX /FD /c -# ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.5" /D "Py_BUILD_CORE_MODULE" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c +# ADD CPP /nologo /MDd /W3 /Gm /GX /Zi /Od /I "..\..\Include" /I ".." /I "..\..\..\bzip2-1.0.6" /D "Py_BUILD_CORE_MODULE" /D "_DEBUG" /D "WIN32" /D "_WINDOWS" /YX /FD /c # ADD BASE MTL /nologo /D "_DEBUG" /mktyplib203 /o "NUL" /win32 # ADD MTL /nologo /D "_DEBUG" /mktyplib203 /o "NUL" /win32 # ADD BASE RSC /l 0x409 /d "_DEBUG" @@ -82,7 +82,7 @@ # ADD BSC32 /nologo LINK32=link.exe # ADD BASE LINK32 kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /nologo /subsystem:windows /dll /debug /machine:I386 /pdbtype:sept -# ADD LINK32 ..\..\..\bzip2-1.0.5\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"msvcrt" /nodefaultlib:"libc" /out:"./bz2_d.pyd" /pdbtype:sept +# ADD LINK32 ..\..\..\bzip2-1.0.6\libbz2.lib /nologo /base:"0x1D170000" /subsystem:windows /dll /debug /machine:I386 /nodefaultlib:"msvcrt" /nodefaultlib:"libc" /out:"./bz2_d.pyd" /pdbtype:sept # SUBTRACT LINK32 /pdb:none !ENDIF diff --git a/PC/VC6/readme.txt b/PC/VC6/readme.txt --- a/PC/VC6/readme.txt +++ b/PC/VC6/readme.txt @@ -120,14 +120,14 @@ Download the source from the python.org copy into the dist directory: - svn export http://svn.python.org/projects/external/bzip2-1.0.5 + svn export http://svn.python.org/projects/external/bzip2-1.0.6 And requires building bz2 first. - cd dist\bzip2-1.0.5 + cd dist\bzip2-1.0.6 nmake -f makefile.msc - All of this managed to build bzip2-1.0.5\libbz2.lib, which the Python + All of this managed to build bzip2-1.0.6\libbz2.lib, which the Python project links in. diff --git a/PC/VS8.0/bz2.vcproj b/PC/VS8.0/bz2.vcproj --- a/PC/VS8.0/bz2.vcproj +++ b/PC/VS8.0/bz2.vcproj @@ -532,7 +532,7 @@ http://hg.python.org/cpython/rev/2bbf3ba30435 changeset: 76924:2bbf3ba30435 branch: 3.2 parent: 76921:db97a8248863 user: Antoine Pitrou date: Mon May 14 14:43:03 2012 +0200 summary: Use size_t, not ssize_t (issue #14801). files: Objects/typeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2379,7 +2379,7 @@ /* need to make a copy of the docstring slot, which usually points to a static string literal */ if (slot->slot == Py_tp_doc) { - ssize_t len = strlen(slot->pfunc)+1; + size_t len = strlen(slot->pfunc)+1; char *tp_doc = PyObject_MALLOC(len); if (tp_doc == NULL) goto fail; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 14:51:35 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 14 May 2012 14:51:35 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_tab_char?= =?utf8?q?acters?= Message-ID: http://hg.python.org/cpython/rev/6b037a9eaba7 changeset: 76925:6b037a9eaba7 branch: 3.2 user: Antoine Pitrou date: Mon May 14 14:43:25 2012 +0200 summary: Remove tab characters files: Objects/typeobject.c | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2360,21 +2360,21 @@ return NULL; res->ht_name = PyUnicode_FromString(spec->name); if (!res->ht_name) - goto fail; + goto fail; res->ht_type.tp_name = _PyUnicode_AsString(res->ht_name); if (!res->ht_type.tp_name) - goto fail; + goto fail; res->ht_type.tp_basicsize = spec->basicsize; res->ht_type.tp_itemsize = spec->itemsize; res->ht_type.tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE; for (slot = spec->slots; slot->slot; slot++) { - if (slot->slot >= sizeof(slotoffsets)/sizeof(slotoffsets[0])) { - PyErr_SetString(PyExc_RuntimeError, "invalid slot offset"); - goto fail; - } - *(void**)(res_start + slotoffsets[slot->slot]) = slot->pfunc; + if (slot->slot >= sizeof(slotoffsets)/sizeof(slotoffsets[0])) { + PyErr_SetString(PyExc_RuntimeError, "invalid slot offset"); + goto fail; + } + *(void**)(res_start + slotoffsets[slot->slot]) = slot->pfunc; /* need to make a copy of the docstring slot, which usually points to a static string literal */ @@ -2382,7 +2382,7 @@ size_t len = strlen(slot->pfunc)+1; char *tp_doc = PyObject_MALLOC(len); if (tp_doc == NULL) - goto fail; + goto fail; memcpy(tp_doc, slot->pfunc, len); res->ht_type.tp_doc = tp_doc; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 14:51:36 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 14 May 2012 14:51:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?b?OiBVc2Ugc2l6ZV90LCBub3Qgc3NpemVfdCAoaXNzdWUgIzE0ODAxKS4=?= Message-ID: http://hg.python.org/cpython/rev/64b695a6cc3d changeset: 76926:64b695a6cc3d parent: 76922:e33181ff15d0 parent: 76925:6b037a9eaba7 user: Antoine Pitrou date: Mon May 14 14:44:37 2012 +0200 summary: Use size_t, not ssize_t (issue #14801). files: Objects/typeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2406,7 +2406,7 @@ /* need to make a copy of the docstring slot, which usually points to a static string literal */ if (slot->slot == Py_tp_doc) { - ssize_t len = strlen(slot->pfunc)+1; + size_t len = strlen(slot->pfunc)+1; char *tp_doc = PyObject_MALLOC(len); if (tp_doc == NULL) goto fail; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 14:55:18 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Mon, 14 May 2012 14:55:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_14800=3A_add_comments?= =?utf8?q?_explaining_stat=2Epy_constants_+_docstring_for_S=5F*?= Message-ID: http://hg.python.org/cpython/rev/2a695cbdf090 changeset: 76927:2a695cbdf090 user: Giampaolo Rodola' date: Mon May 14 14:53:33 2012 +0200 summary: Issue 14800: add comments explaining stat.py constants + docstring for S_* functions. files: Lib/stat.py | 89 +++++++++++++++++++++++----------------- 1 files changed, 51 insertions(+), 38 deletions(-) diff --git a/Lib/stat.py b/Lib/stat.py --- a/Lib/stat.py +++ b/Lib/stat.py @@ -19,78 +19,91 @@ # Extract bits from the mode def S_IMODE(mode): + """Return the portion of the file's mode that can be set by + os.chmod(). + """ return mode & 0o7777 def S_IFMT(mode): + """Return the portion of the file's mode that describes the + file type. + """ return mode & 0o170000 # Constants used as S_IFMT() for various file types # (not all are implemented on all systems) -S_IFDIR = 0o040000 -S_IFCHR = 0o020000 -S_IFBLK = 0o060000 -S_IFREG = 0o100000 -S_IFIFO = 0o010000 -S_IFLNK = 0o120000 -S_IFSOCK = 0o140000 +S_IFDIR = 0o040000 # directory +S_IFCHR = 0o020000 # character device +S_IFBLK = 0o060000 # block device +S_IFREG = 0o100000 # regular file +S_IFIFO = 0o010000 # fifo (named pipe) +S_IFLNK = 0o120000 # symbolic link +S_IFSOCK = 0o140000 # socket file # Functions to test for each file type def S_ISDIR(mode): + """Return True if mode is from a directory.""" return S_IFMT(mode) == S_IFDIR def S_ISCHR(mode): + """Return True if mode is from a character special device file.""" return S_IFMT(mode) == S_IFCHR def S_ISBLK(mode): + """Return True if mode is from a block special device file.""" return S_IFMT(mode) == S_IFBLK def S_ISREG(mode): + """Return True if mode is from a regular file.""" return S_IFMT(mode) == S_IFREG def S_ISFIFO(mode): + """Return True if mode is from a FIFO (named pipe).""" return S_IFMT(mode) == S_IFIFO def S_ISLNK(mode): + """Return True if mode is from a symbolic link.""" return S_IFMT(mode) == S_IFLNK def S_ISSOCK(mode): + """Return True if mode is from a socket.""" return S_IFMT(mode) == S_IFSOCK # Names for permission bits -S_ISUID = 0o4000 -S_ISGID = 0o2000 -S_ENFMT = S_ISGID -S_ISVTX = 0o1000 -S_IREAD = 0o0400 -S_IWRITE = 0o0200 -S_IEXEC = 0o0100 -S_IRWXU = 0o0700 -S_IRUSR = 0o0400 -S_IWUSR = 0o0200 -S_IXUSR = 0o0100 -S_IRWXG = 0o0070 -S_IRGRP = 0o0040 -S_IWGRP = 0o0020 -S_IXGRP = 0o0010 -S_IRWXO = 0o0007 -S_IROTH = 0o0004 -S_IWOTH = 0o0002 -S_IXOTH = 0o0001 +S_ISUID = 0o4000 # set UID bit +S_ISGID = 0o2000 # set GID bit +S_ENFMT = S_ISGID # file locking enforcement +S_ISVTX = 0o1000 # sticky bit +S_IREAD = 0o0400 # Unix V7 synonym for S_IRUSR +S_IWRITE = 0o0200 # Unix V7 synonym for S_IWUSR +S_IEXEC = 0o0100 # Unix V7 synonym for S_IXUSR +S_IRWXU = 0o0700 # mask for owner permissions +S_IRUSR = 0o0400 # read by owner +S_IWUSR = 0o0200 # write by owner +S_IXUSR = 0o0100 # execute by owner +S_IRWXG = 0o0070 # mask for group permissions +S_IRGRP = 0o0040 # read by group +S_IWGRP = 0o0020 # write by group +S_IXGRP = 0o0010 # execute by group +S_IRWXO = 0o0007 # mask for others (not in group) permissions +S_IROTH = 0o0004 # read by others +S_IWOTH = 0o0002 # write by others +S_IXOTH = 0o0001 # execute by others # Names for file flags -UF_NODUMP = 0x00000001 -UF_IMMUTABLE = 0x00000002 -UF_APPEND = 0x00000004 -UF_OPAQUE = 0x00000008 -UF_NOUNLINK = 0x00000010 -UF_COMPRESSED = 0x00000020 # OS X: file is hfs-compressed -UF_HIDDEN = 0x00008000 # OS X: file should not be displayed -SF_ARCHIVED = 0x00010000 -SF_IMMUTABLE = 0x00020000 -SF_APPEND = 0x00040000 -SF_NOUNLINK = 0x00100000 -SF_SNAPSHOT = 0x00200000 +UF_NODUMP = 0x00000001 # do not dump file +UF_IMMUTABLE = 0x00000002 # file may not be changed +UF_APPEND = 0x00000004 # file may only be appended to +UF_OPAQUE = 0x00000008 # directory is opaque when viewed through a union stack +UF_NOUNLINK = 0x00000010 # file may not be renamed or deleted +UF_COMPRESSED = 0x00000020 # OS X: file is hfs-compressed +UF_HIDDEN = 0x00008000 # OS X: file should not be displayed +SF_ARCHIVED = 0x00010000 # file may be archived +SF_IMMUTABLE = 0x00020000 # file may not be changed +SF_APPEND = 0x00040000 # file may only be appended to +SF_NOUNLINK = 0x00100000 # file may not be renamed or deleted +SF_SNAPSHOT = 0x00200000 # file is a snapshot file -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 16:25:16 2012 From: python-checkins at python.org (barry.warsaw) Date: Mon, 14 May 2012 16:25:16 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Update_from_Peter_Moody?= Message-ID: http://hg.python.org/peps/rev/d5b90dfe1c92 changeset: 4373:d5b90dfe1c92 user: Barry Warsaw date: Mon May 14 10:25:14 2012 -0400 summary: Update from Peter Moody files: pep-3144.txt | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-) diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -59,6 +59,23 @@ return iterators. This includes, subnets, address_exclude, summarize_address_range and collapse_address_list. + * A number of methods and functions which returned containers in ipaddr now + return iterators. This includes, subnets, address_exclude, + summarize_address_range and collapse_address_list. + + + Due to the backwards incompatible API changes between ipaddress and ipaddr, + the proposal is to add the module using the new provisional API status: + + * http://docs.python.org/dev/glossary.html#term-provisional-package + + + Relevant messages on python-dev: + + * http://mail.python.org/pipermail/python-dev/2012-January/116016.html + * http://mail.python.org/pipermail/python-dev/2012-February/116656.html + * http://mail.python.org/pipermail/python-dev/2012-February/116688.html + Due to the backwards incompatible API changes between ipaddress and ipaddr, the proposal is to add the module using the new provisional API status: @@ -118,8 +135,7 @@ _BaseV4 - Provides methods and variables (eg, _max_prefixlen) common to all IPv4 classes. - _BaseV6 - Provides methods and variables common to all IPv6 - classes. + _BaseV6 - Provides methods and variables common to all IPv6 classes. Comparisons between objects of differing IP versions results in a TypeError [1]. Additionally, comparisons of objects with @@ -132,10 +148,10 @@ Reference Implementation: The current reference implementation can be found at: - https://code.google.com/p/ipaddr-py/source/browse/branches/3144/ipaddress.py + http://code.google.com/p/ipaddress-py/source/browse/ipaddress.py Or see the tarball to include the README and unittests. - http://code.google.com/p/ipaddr-py/downloads/detail?name=3144.tar.gz + http://code.google.com/p/ipaddress-py/downloads/detail?name=ipaddress-1.0.tar.gz More information about using the reference implementation can be found at: http://code.google.com/p/ipaddr-py/wiki/Using3144 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 14 16:45:34 2012 From: python-checkins at python.org (eric.smith) Date: Mon, 14 May 2012 16:45:34 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Changes_suggested_by_PJE=2E?= Message-ID: http://hg.python.org/peps/rev/c417da431f8e changeset: 4374:c417da431f8e user: Eric V. Smith date: Mon May 14 10:45:28 2012 -0400 summary: Changes suggested by PJE. files: pep-0420.txt | 54 +++++++++++++++++++++++++++++---------- 1 files changed, 40 insertions(+), 14 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -76,6 +76,10 @@ ``setup.py``, so that distribution developers don't need to put the magic ``__path__`` modification into ``__init__.py`` themselves. +See PEP 402's "The Problem" section [2]_ for more details on the +motivation for namespace packages. Note that PEP 402 has been +rejected, but the motivating use cases are still valid. + Rationale ========= @@ -135,8 +139,8 @@ least one directory was recorded, then a namespace package is created. The new namespace package: - * Has a ``__path__`` attribute set to the list of directories that - were found and recorded during the scan. + * Has a ``__path__`` attribute set to an iterable of the path strings + that were found and recorded during the scan. * Does not have a ``__file__`` attribute. @@ -148,10 +152,7 @@ A namespace package is not fundamentally different from a regular package. It is just a different way of creating packages. Once a namespace package is created, there is no functional difference -between it and a regular package. The only observable difference is -that the namespace package's ``__file__`` attribute will end with a -path separator (typically a slash or backslash, depending on the -platform). +between it and a regular package. Dynamic path computation ------------------------ @@ -198,6 +199,28 @@ ``module_repr()`` which if present, is used to generate module object reprs. See the section below for further details. +Differences between namespace packages and regular packages +----------------------------------------------------------- + +Namespace packages and regular packages are very similar. The +differences are: + + * Portions of namespace packages need not all come from the same + directory structure, or even from the same loader. Regular packages + are self-contained: all parts live in the same directory hierarchy. + + * Namespace packages have no ``__file__`` attribute. + + * Namespace packages' ``__path__`` attribute is a read-only iterable + of strings, which is automatically updated when the parent path is + modified. + + * Namespace packages have no ``__init__.py`` module. + + * Namespace packages have a different type of object for their + ``__loader__`` attribute. + + Packaging Implications ====================== @@ -232,7 +255,7 @@ ========== At PyCon 2012, we had a discussion about namespace packages at which -PEP 382 and PEP 402 were rejected, to be replaced by this PEP [2]_. +PEP 382 and PEP 402 were rejected, to be replaced by this PEP [3]_. There is no intention to remove support of regular packages. If a developer knows that her package will never be a portion of a @@ -247,7 +270,7 @@ imported as a namespace package, whereas in prior Python versions an ImportWarning would be raised. -Nick Coghlan presented a list of his objections to this proposal [3]_. +Nick Coghlan presented a list of his objections to this proposal [4]_. They are: 1. Implicit package directories go against the Zen of Python. @@ -261,7 +284,7 @@ 4. Implicit package directories will permanently entrench current newbie-hostile behaviour in ``__main__``. -Nick gave a detailed response [4]_, which is summarized here: +Nick gave a detailed response [5]_, which is summarized here: 1. The practicality of this PEP wins over other proposals and the status quo. @@ -290,7 +313,7 @@ implement the ``find_loader`` method, described above. The use case for supporting multiple portions per ``find_loader`` call -is given in [5]_. +is given in [6]_. Module reprs @@ -338,16 +361,19 @@ .. [1] PEP 420 branch (http://hg.python.org/features/pep-420) -.. [2] PyCon 2012 Namespace Package discussion outcome +.. [2] PEP 402's description of use cases for namespace packages + (http://www.python.org/dev/peps/pep-0402/#the-problem) + +.. [3] PyCon 2012 Namespace Package discussion outcome (http://mail.python.org/pipermail/import-sig/2012-March/000421.html) -.. [3] Nick Coghlan's objection to the lack of marker files or directories +.. [4] Nick Coghlan's objection to the lack of marker files or directories (http://mail.python.org/pipermail/import-sig/2012-March/000423.html) -.. [4] Nick Coghlan's response to his initial objections +.. [5] Nick Coghlan's response to his initial objections (http://mail.python.org/pipermail/import-sig/2012-April/000464.html) -.. [5] Use case for multiple portions per ``find_loader`` call +.. [6] Use case for multiple portions per ``find_loader`` call (http://mail.python.org/pipermail/import-sig/2012-May/000585.html) Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 14 16:51:48 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 16:51:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Upgrade_sqlite_to_3=2E7=2E1?= =?utf8?q?2=2E?= Message-ID: http://hg.python.org/cpython/rev/c5171d50e752 changeset: 76928:c5171d50e752 user: Martin v. L?wis date: Mon May 14 16:51:35 2012 +0200 summary: Upgrade sqlite to 3.7.12. files: PC/VS9.0/pyproject.vsprops | 2 +- PCbuild/pyproject.props | 2 +- Tools/buildbot/external-common.bat | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops --- a/PC/VS9.0/pyproject.vsprops +++ b/PC/VS9.0/pyproject.vsprops @@ -50,7 +50,7 @@ /> python33 $(SolutionDir)\python.exe ..\.. - $(externalsDir)\sqlite-3.7.4 + $(externalsDir)\sqlite-3.7.12 $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.3 $(externalsDir)\openssl-1.0.0a diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -15,7 +15,7 @@ @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1 @rem if exist db-4.4.20 rd /s/q db-4.4.20 @rem if exist openssl-1.0.0a rd /s/q openssl-1.0.0a - at rem if exist sqlite-3.7.4 rd /s/q sqlite-3.7.4 + at rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12 @rem bzip if not exist bzip2-1.0.6 ( @@ -37,9 +37,9 @@ if not exist tk-8.5.11.0 svn export http://svn.python.org/projects/external/tk-8.5.11.0 @rem sqlite3 -if not exist sqlite-3.7.4 ( - rd /s/q sqlite-source-3.6.21 - svn export http://svn.python.org/projects/external/sqlite-3.7.4 +if not exist sqlite-3.7.12 ( + rd /s/q sqlite-source-3.7.4 + svn export http://svn.python.org/projects/external/sqlite-3.7.12 ) @rem lzma -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 18:19:32 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 18:19:32 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Stop_deleting_n?= =?utf8?q?on-existing_bytecode_files=2E?= Message-ID: http://hg.python.org/cpython/rev/7b64365d414f changeset: 76929:7b64365d414f branch: 3.2 parent: 76925:6b037a9eaba7 user: Martin v. L?wis date: Mon May 14 18:18:07 2012 +0200 summary: Stop deleting non-existing bytecode files. files: Tools/buildbot/clean.bat | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Tools/buildbot/clean.bat b/Tools/buildbot/clean.bat --- a/Tools/buildbot/clean.bat +++ b/Tools/buildbot/clean.bat @@ -1,7 +1,5 @@ @rem Used by the buildbot "clean" step. call "%VS90COMNTOOLS%vsvars32.bat" - at echo Deleting .pyc/.pyo files ... -del /s Lib\*.pyc Lib\*.pyo @echo Deleting test leftovers ... rmdir /s /q build cd PCbuild -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 18:19:33 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 18:19:33 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/5298b39fa6c0 changeset: 76930:5298b39fa6c0 parent: 76928:c5171d50e752 parent: 76929:7b64365d414f user: Martin v. L?wis date: Mon May 14 18:19:16 2012 +0200 summary: merge 3.2 files: Tools/buildbot/clean.bat | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Tools/buildbot/clean.bat b/Tools/buildbot/clean.bat --- a/Tools/buildbot/clean.bat +++ b/Tools/buildbot/clean.bat @@ -1,7 +1,5 @@ @rem Used by the buildbot "clean" step. call "%VS100COMNTOOLS%vsvars32.bat" - at echo Deleting .pyc/.pyo files ... -del /s Lib\*.pyc Lib\*.pyo @echo Deleting test leftovers ... rmdir /s /q build cd PCbuild -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 18:24:10 2012 From: python-checkins at python.org (martin.v.loewis) Date: Mon, 14 May 2012 18:24:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Don=27t_build_xxlimited_in_?= =?utf8?q?debug_mode=2E?= Message-ID: http://hg.python.org/cpython/rev/c99c527f1186 changeset: 76931:c99c527f1186 user: Martin v. L?wis date: Mon May 14 18:23:44 2012 +0200 summary: Don't build xxlimited in debug mode. files: PCbuild/pcbuild.sln | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -548,9 +548,7 @@ {885D4898-D08D-4091-9C40-C700CFE3FC5A}.Release|x64.ActiveCfg = Release|x64 {885D4898-D08D-4091-9C40-C700CFE3FC5A}.Release|x64.Build.0 = Release|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|Win32.ActiveCfg = Debug|Win32 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|Win32.Build.0 = Debug|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|x64.ActiveCfg = Debug|x64 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|x64.Build.0 = Debug|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 19:50:49 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 14 May 2012 19:50:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Followup_to_iss?= =?utf8?q?ue_=2314157=3A_respect_the_relative_ordering_of_values_produced_?= =?utf8?q?by?= Message-ID: http://hg.python.org/cpython/rev/83598eb0d761 changeset: 76932:83598eb0d761 branch: 3.2 parent: 76929:7b64365d414f user: Antoine Pitrou date: Mon May 14 19:44:59 2012 +0200 summary: Followup to issue #14157: respect the relative ordering of values produced by time.strptime(). Patch by Hynek. files: Lib/_strptime.py | 8 ++++++++ Lib/test/test_strptime.py | 5 +++++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -444,8 +444,10 @@ else: tz = value break + leap_year_fix = False if year is None and month == 2 and day == 29: year = 1904 # 1904 is first leap year of 20th century + leap_year_fix = True elif year is None: year = 1900 # If we know the week of the year and what day of that week, we can figure @@ -476,6 +478,12 @@ else: gmtoff = None + if leap_year_fix: + # the caller didn't supply a year but asked for Feb 29th. We couldn't + # use the default of 1900 for computations. We set it back to ensure + # that February 29th is smaller than March 1st. + year = 1900 + return (year, month, day, hour, minute, second, weekday, julian, tz, gmtoff, tzname), fraction diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -381,6 +381,11 @@ def test_feb29_on_leap_year_without_year(self): time.strptime("Feb 29", "%b %d") + def test_mar1_comes_after_feb29_even_when_omitting_the_year(self): + self.assertLess( + time.strptime("Feb 29", "%b %d"), + time.strptime("Mar 1", "%b %d")) + class Strptime12AMPMTests(unittest.TestCase): """Test a _strptime regression in '%I %p' at 12 noon (12 PM)""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 19:50:50 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 14 May 2012 19:50:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Followup_to_issue_=2314157=3A_respect_the_relative_ordering_?= =?utf8?q?of_values_produced_by?= Message-ID: http://hg.python.org/cpython/rev/d1c0b57aeb1b changeset: 76933:d1c0b57aeb1b parent: 76931:c99c527f1186 parent: 76932:83598eb0d761 user: Antoine Pitrou date: Mon May 14 19:45:27 2012 +0200 summary: Followup to issue #14157: respect the relative ordering of values produced by time.strptime(). Patch by Hynek. files: Lib/_strptime.py | 8 ++++++++ Lib/test/test_strptime.py | 5 +++++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -444,8 +444,10 @@ else: tz = value break + leap_year_fix = False if year is None and month == 2 and day == 29: year = 1904 # 1904 is first leap year of 20th century + leap_year_fix = True elif year is None: year = 1900 # If we know the week of the year and what day of that week, we can figure @@ -476,6 +478,12 @@ else: gmtoff = None + if leap_year_fix: + # the caller didn't supply a year but asked for Feb 29th. We couldn't + # use the default of 1900 for computations. We set it back to ensure + # that February 29th is smaller than March 1st. + year = 1900 + return (year, month, day, hour, minute, second, weekday, julian, tz, gmtoff, tzname), fraction diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -381,6 +381,11 @@ def test_feb29_on_leap_year_without_year(self): time.strptime("Feb 29", "%b %d") + def test_mar1_comes_after_feb29_even_when_omitting_the_year(self): + self.assertLess( + time.strptime("Feb 29", "%b %d"), + time.strptime("Mar 1", "%b %d")) + class Strptime12AMPMTests(unittest.TestCase): """Test a _strptime regression in '%I %p' at 12 noon (12 PM)""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 19:50:51 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 14 May 2012 19:50:51 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Followup_to_iss?= =?utf8?q?ue_=2314157=3A_respect_the_relative_ordering_of_values_produced_?= =?utf8?q?by?= Message-ID: http://hg.python.org/cpython/rev/cbc9dc1c977e changeset: 76934:cbc9dc1c977e branch: 2.7 parent: 76923:2a440ff10f37 user: Antoine Pitrou date: Mon May 14 19:44:59 2012 +0200 summary: Followup to issue #14157: respect the relative ordering of values produced by time.strptime(). Patch by Hynek. files: Lib/_strptime.py | 8 ++++++++ Lib/test/test_strptime.py | 5 +++++ 2 files changed, 13 insertions(+), 0 deletions(-) diff --git a/Lib/_strptime.py b/Lib/_strptime.py --- a/Lib/_strptime.py +++ b/Lib/_strptime.py @@ -426,8 +426,10 @@ else: tz = value break + leap_year_fix = False if year is None and month == 2 and day == 29: year = 1904 # 1904 is first leap year of 20th century + leap_year_fix = True elif year is None: year = 1900 # If we know the week of the year and what day of that week, we can figure @@ -451,6 +453,12 @@ day = datetime_result.day if weekday == -1: weekday = datetime_date(year, month, day).weekday() + if leap_year_fix: + # the caller didn't supply a year but asked for Feb 29th. We couldn't + # use the default of 1900 for computations. We set it back to ensure + # that February 29th is smaller than March 1st. + year = 1900 + return (time.struct_time((year, month, day, hour, minute, second, weekday, julian, tz)), fraction) diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -381,6 +381,11 @@ def test_feb29_on_leap_year_without_year(self): time.strptime("Feb 29", "%b %d") + def test_mar1_comes_after_feb29_even_when_omitting_the_year(self): + self.assertLess( + time.strptime("Feb 29", "%b %d"), + time.strptime("Mar 1", "%b %d")) + class Strptime12AMPMTests(unittest.TestCase): """Test a _strptime regression in '%I %p' at 12 noon (12 PM)""" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 14 22:21:25 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 14 May 2012 22:21:25 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_Hynek_to_developers?= Message-ID: http://hg.python.org/devguide/rev/f175f5fe0bcd changeset: 509:f175f5fe0bcd user: Antoine Pitrou date: Mon May 14 22:19:02 2012 +0200 summary: Add Hynek to developers files: developers.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/developers.rst b/developers.rst --- a/developers.rst +++ b/developers.rst @@ -24,6 +24,9 @@ Permissions History ------------------- +- Hynek Schlawack was given push privileges on May 14 2012 by Antoine Pitrou + for general contributions. + - Richard Oudkerk was given push privileges on Apr 29 2012 by Antoine Pitrou on recommendation by Charles-Fran?ois Natali and Jesse Noller, for various contributions to multiprocessing (and original authorship of -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Tue May 15 02:43:30 2012 From: python-checkins at python.org (brian.curtin) Date: Tue, 15 May 2012 02:43:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_bsddb_never_exi?= =?utf8?q?sted_in_3=2Ex=2C_no_need_to_keep_downloading_sleepycat?= Message-ID: http://hg.python.org/cpython/rev/ffe39fba03ae changeset: 76935:ffe39fba03ae branch: 3.2 parent: 76932:83598eb0d761 user: Brian Curtin date: Mon May 14 19:42:36 2012 -0500 summary: bsddb never existed in 3.x, no need to keep downloading sleepycat files: Tools/buildbot/external-common.bat | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -23,9 +23,6 @@ svn export http://svn.python.org/projects/external/bzip2-1.0.6 ) - at rem Sleepycat db -if not exist db-4.4.20 svn export http://svn.python.org/projects/external/db-4.4.20-vs9 db-4.4.20 - @rem OpenSSL if not exist openssl-1.0.0a svn export http://svn.python.org/projects/external/openssl-1.0.0a -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 02:43:31 2012 From: python-checkins at python.org (brian.curtin) Date: Tue, 15 May 2012 02:43:31 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/07afe1ec76a7 changeset: 76936:07afe1ec76a7 parent: 76933:d1c0b57aeb1b parent: 76935:ffe39fba03ae user: Brian Curtin date: Mon May 14 19:43:16 2012 -0500 summary: Merge 3.2 files: Tools/buildbot/external-common.bat | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -23,9 +23,6 @@ svn export http://svn.python.org/projects/external/bzip2-1.0.6 ) - at rem Sleepycat db -if not exist db-4.4.20 svn export http://svn.python.org/projects/external/db-4.4.20-vs9 db-4.4.20 - @rem OpenSSL if not exist openssl-1.0.0a svn export http://svn.python.org/projects/external/openssl-1.0.0a -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 02:52:02 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 15 May 2012 02:52:02 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Allow_standard_library_package?= =?utf8?q?s_to_be_namespace_packages=2E?= Message-ID: http://hg.python.org/peps/rev/d52109a2cd33 changeset: 4375:d52109a2cd33 user: Eric V. Smith date: Mon May 14 20:51:54 2012 -0400 summary: Allow standard library packages to be namespace packages. files: pep-0420.txt | 11 ++++++++++- 1 files changed, 10 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -77,7 +77,7 @@ magic ``__path__`` modification into ``__init__.py`` themselves. See PEP 402's "The Problem" section [2]_ for more details on the -motivation for namespace packages. Note that PEP 402 has been +motivation for namespace packages. Note that PEP 402 has been rejected, but the motivating use cases are still valid. Rationale @@ -221,6 +221,15 @@ ``__loader__`` attribute. +Namespace packages in the standard library +------------------------------------------ + +It is possible, and this PEP explicitely allows, that parts of the +standard library be implemented as namespace packages. When and if +any standard library packages become namespace packages is outside the +scope of this PEP. + + Packaging Implications ====================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 04:33:47 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 15 May 2012 04:33:47 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NzY2OiBBZGQg?= =?utf8?q?correct_algorithm_for_when_a_=27time=27_object_is_naive=2E?= Message-ID: http://hg.python.org/cpython/rev/28ed782949f9 changeset: 76937:28ed782949f9 branch: 3.2 parent: 76935:ffe39fba03ae user: R David Murray date: Mon May 14 22:14:46 2012 -0400 summary: #14766: Add correct algorithm for when a 'time' object is naive. This patch also clarifies the definition of Naive and Aware. Original patch by Greg Weller, I modified the first hunk somewhat to make the exposition even clearer (I hope). files: Doc/library/datetime.rst | 38 +++++++++++++++++++-------- 1 files changed, 26 insertions(+), 12 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -15,16 +15,23 @@ formatting and manipulation. For related functionality, see also the :mod:`time` and :mod:`calendar` modules. -There are two kinds of date and time objects: "naive" and "aware". This -distinction refers to whether the object has any notion of time zone, daylight -saving time, or other kind of algorithmic or political time adjustment. Whether -a naive :class:`.datetime` object represents Coordinated Universal Time (UTC), +There are two kinds of date and time objects: "naive" and "aware". + +An aware object has sufficient knowledge of applicable algorithmic and +political time adjustments, such as time zone and daylight saving time +information, to locate itself relative to other aware objects. An aware object +is used to represent a specific moment in time that is not open to +interpretation [#]_. + +A naive object does not contain enough information to unambiguously locate +itself relative to other date/time objects. Whether a naive object represents +Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program, just like it's up to the program whether a particular number represents metres, -miles, or mass. Naive :class:`.datetime` objects are easy to understand and to +miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality. -For applications requiring more, :class:`.datetime` and :class:`.time` objects +For applications requiring aware objects, :class:`.datetime` and :class:`.time` objects have an optional time zone information attribute, :attr:`tzinfo`, that can be set to an instance of a subclass of the abstract :class:`tzinfo` class. These :class:`tzinfo` objects capture information about the offset from UTC time, the @@ -32,8 +39,8 @@ one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the :mod:`datetime` module. The :class:`timezone` class can represent simple timezones with fixed offset from UTC such as UTC itself or North American EST and -EDT timezones. Supporting timezones at whatever level of detail is -required is up to the application. The rules for time adjustment across the +EDT timezones. Supporting timezones at deeper levels of detail is +up to the application. The rules for time adjustment across the world are more political than rational, change frequently, and there is no standard suitable for every application aside from UTC. @@ -114,10 +121,13 @@ Objects of the :class:`date` type are always naive. -An object *d* of type :class:`.time` or :class:`.datetime` may be naive or aware. -*d* is aware if ``d.tzinfo`` is not ``None`` and ``d.tzinfo.utcoffset(d)`` does -not return ``None``. If ``d.tzinfo`` is ``None``, or if ``d.tzinfo`` is not -``None`` but ``d.tzinfo.utcoffset(d)`` returns ``None``, *d* is naive. +An object of type :class:`.time` or :class:`.datetime` may be naive or aware. +A :class:`.datetime` object *d* is aware if ``d.tzinfo`` is not ``None`` and +``d.tzinfo.utcoffset(d)`` does not return ``None``. If ``d.tzinfo`` is +``None``, or if ``d.tzinfo`` is not ``None`` but ``d.tzinfo.utcoffset(d)`` +returns ``None``, *d* is naive. A :class:`.time` object *t* is aware +if ``t.tzinfo`` is not ``None`` and ``t.tzinfo.utcoffset(None)`` does not return +``None``. Otherwise, *t* is naive. The distinction between naive and aware doesn't apply to :class:`timedelta` objects. @@ -1806,3 +1816,7 @@ When the ``%z`` directive is provided to the :meth:`strptime` method, an aware :class:`.datetime` object will be produced. The ``tzinfo`` of the result will be set to a :class:`timezone` instance. + +.. rubric:: Footnotes + +.. [#] If, that is, we ignore the effects of Relativity -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 04:33:48 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 15 May 2012 04:33:48 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NzY2OiBSZWZs?= =?utf8?q?ow_the_altered_paragraphs=2E?= Message-ID: http://hg.python.org/cpython/rev/b1e03e863386 changeset: 76938:b1e03e863386 branch: 3.2 user: R David Murray date: Mon May 14 22:17:23 2012 -0400 summary: #14766: Reflow the altered paragraphs. files: Doc/library/datetime.rst | 33 +++++++++++++-------------- 1 files changed, 16 insertions(+), 17 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -12,8 +12,8 @@ The :mod:`datetime` module supplies classes for manipulating dates and times in both simple and complex ways. While date and time arithmetic is supported, the focus of the implementation is on efficient attribute extraction for output -formatting and manipulation. For related -functionality, see also the :mod:`time` and :mod:`calendar` modules. +formatting and manipulation. For related functionality, see also the +:mod:`time` and :mod:`calendar` modules. There are two kinds of date and time objects: "naive" and "aware". @@ -25,22 +25,21 @@ A naive object does not contain enough information to unambiguously locate itself relative to other date/time objects. Whether a naive object represents -Coordinated Universal Time (UTC), -local time, or time in some other timezone is purely up to the program, just -like it's up to the program whether a particular number represents metres, -miles, or mass. Naive objects are easy to understand and to -work with, at the cost of ignoring some aspects of reality. +Coordinated Universal Time (UTC), local time, or time in some other timezone is +purely up to the program, just like it is up to the program whether a +particular number represents metres, miles, or mass. Naive objects are easy to +understand and to work with, at the cost of ignoring some aspects of reality. -For applications requiring aware objects, :class:`.datetime` and :class:`.time` objects -have an optional time zone information attribute, :attr:`tzinfo`, that can be -set to an instance of a subclass of the abstract :class:`tzinfo` class. These -:class:`tzinfo` objects capture information about the offset from UTC time, the -time zone name, and whether Daylight Saving Time is in effect. Note that only -one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the -:mod:`datetime` module. The :class:`timezone` class can represent simple -timezones with fixed offset from UTC such as UTC itself or North American EST and -EDT timezones. Supporting timezones at deeper levels of detail is -up to the application. The rules for time adjustment across the +For applications requiring aware objects, :class:`.datetime` and :class:`.time` +objects have an optional time zone information attribute, :attr:`tzinfo`, that +can be set to an instance of a subclass of the abstract :class:`tzinfo` class. +These :class:`tzinfo` objects capture information about the offset from UTC +time, the time zone name, and whether Daylight Saving Time is in effect. Note +that only one concrete :class:`tzinfo` class, the :class:`timezone` class, is +supplied by the :mod:`datetime` module. The :class:`timezone` class can +represent simple timezones with fixed offset from UTC, such as UTC itself or +North American EST and EDT timezones. Supporting timezones at deeper levels of +detail is up to the application. The rules for time adjustment across the world are more political than rational, change frequently, and there is no standard suitable for every application aside from UTC. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 04:33:49 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 15 May 2012 04:33:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_=2314766=3A_Add_correct_algorithm_for_when_a_=27time?= =?utf8?q?=27_object_is_naive=2E?= Message-ID: http://hg.python.org/cpython/rev/6ff172db8114 changeset: 76939:6ff172db8114 parent: 76936:07afe1ec76a7 parent: 76938:b1e03e863386 user: R David Murray date: Mon May 14 22:19:10 2012 -0400 summary: Merge #14766: Add correct algorithm for when a 'time' object is naive. This patch also clarifies the definition of Naive and Aware. Original patch by Greg Weller, I modified the first hunk somewhat to make the exposition even clearer (I hope). files: Doc/library/datetime.rst | 61 ++++++++++++++++----------- 1 files changed, 37 insertions(+), 24 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -12,28 +12,34 @@ The :mod:`datetime` module supplies classes for manipulating dates and times in both simple and complex ways. While date and time arithmetic is supported, the focus of the implementation is on efficient attribute extraction for output -formatting and manipulation. For related -functionality, see also the :mod:`time` and :mod:`calendar` modules. +formatting and manipulation. For related functionality, see also the +:mod:`time` and :mod:`calendar` modules. -There are two kinds of date and time objects: "naive" and "aware". This -distinction refers to whether the object has any notion of time zone, daylight -saving time, or other kind of algorithmic or political time adjustment. Whether -a naive :class:`.datetime` object represents Coordinated Universal Time (UTC), -local time, or time in some other timezone is purely up to the program, just -like it's up to the program whether a particular number represents metres, -miles, or mass. Naive :class:`.datetime` objects are easy to understand and to -work with, at the cost of ignoring some aspects of reality. +There are two kinds of date and time objects: "naive" and "aware". -For applications requiring more, :class:`.datetime` and :class:`.time` objects -have an optional time zone information attribute, :attr:`tzinfo`, that can be -set to an instance of a subclass of the abstract :class:`tzinfo` class. These -:class:`tzinfo` objects capture information about the offset from UTC time, the -time zone name, and whether Daylight Saving Time is in effect. Note that only -one concrete :class:`tzinfo` class, the :class:`timezone` class, is supplied by the -:mod:`datetime` module. The :class:`timezone` class can represent simple -timezones with fixed offset from UTC such as UTC itself or North American EST and -EDT timezones. Supporting timezones at whatever level of detail is -required is up to the application. The rules for time adjustment across the +An aware object has sufficient knowledge of applicable algorithmic and +political time adjustments, such as time zone and daylight saving time +information, to locate itself relative to other aware objects. An aware object +is used to represent a specific moment in time that is not open to +interpretation [#]_. + +A naive object does not contain enough information to unambiguously locate +itself relative to other date/time objects. Whether a naive object represents +Coordinated Universal Time (UTC), local time, or time in some other timezone is +purely up to the program, just like it is up to the program whether a +particular number represents metres, miles, or mass. Naive objects are easy to +understand and to work with, at the cost of ignoring some aspects of reality. + +For applications requiring aware objects, :class:`.datetime` and :class:`.time` +objects have an optional time zone information attribute, :attr:`tzinfo`, that +can be set to an instance of a subclass of the abstract :class:`tzinfo` class. +These :class:`tzinfo` objects capture information about the offset from UTC +time, the time zone name, and whether Daylight Saving Time is in effect. Note +that only one concrete :class:`tzinfo` class, the :class:`timezone` class, is +supplied by the :mod:`datetime` module. The :class:`timezone` class can +represent simple timezones with fixed offset from UTC, such as UTC itself or +North American EST and EDT timezones. Supporting timezones at deeper levels of +detail is up to the application. The rules for time adjustment across the world are more political than rational, change frequently, and there is no standard suitable for every application aside from UTC. @@ -114,10 +120,13 @@ Objects of the :class:`date` type are always naive. -An object *d* of type :class:`.time` or :class:`.datetime` may be naive or aware. -*d* is aware if ``d.tzinfo`` is not ``None`` and ``d.tzinfo.utcoffset(d)`` does -not return ``None``. If ``d.tzinfo`` is ``None``, or if ``d.tzinfo`` is not -``None`` but ``d.tzinfo.utcoffset(d)`` returns ``None``, *d* is naive. +An object of type :class:`.time` or :class:`.datetime` may be naive or aware. +A :class:`.datetime` object *d* is aware if ``d.tzinfo`` is not ``None`` and +``d.tzinfo.utcoffset(d)`` does not return ``None``. If ``d.tzinfo`` is +``None``, or if ``d.tzinfo`` is not ``None`` but ``d.tzinfo.utcoffset(d)`` +returns ``None``, *d* is naive. A :class:`.time` object *t* is aware +if ``t.tzinfo`` is not ``None`` and ``t.tzinfo.utcoffset(None)`` does not return +``None``. Otherwise, *t* is naive. The distinction between naive and aware doesn't apply to :class:`timedelta` objects. @@ -1846,3 +1855,7 @@ When the ``%z`` directive is provided to the :meth:`strptime` method, an aware :class:`.datetime` object will be produced. The ``tzinfo`` of the result will be set to a :class:`timezone` instance. + +.. rubric:: Footnotes + +.. [#] If, that is, we ignore the effects of Relativity -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 04:33:50 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 15 May 2012 04:33:50 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NzY2OiBBZGQg?= =?utf8?q?correct_algorithm_for_when_a_=27time=27_object_is_naive=2E?= Message-ID: http://hg.python.org/cpython/rev/5d1b32e33e67 changeset: 76940:5d1b32e33e67 branch: 2.7 parent: 76934:cbc9dc1c977e user: R David Murray date: Mon May 14 22:32:44 2012 -0400 summary: #14766: Add correct algorithm for when a 'time' object is naive. This patch also clarifies the definition of Naive and Aware. Original patch by Greg Weller, I modified the first hunk somewhat to make the exposition even clearer (I hope). files: Doc/library/datetime.rst | 35 ++++++++++++++++++++-------- 1 files changed, 25 insertions(+), 10 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -17,16 +17,23 @@ formatting and manipulation. For related functionality, see also the :mod:`time` and :mod:`calendar` modules. -There are two kinds of date and time objects: "naive" and "aware". This -distinction refers to whether the object has any notion of time zone, daylight -saving time, or other kind of algorithmic or political time adjustment. Whether -a naive :class:`.datetime` object represents Coordinated Universal Time (UTC), +There are two kinds of date and time objects: "naive" and "aware". + +An aware object has sufficient knowledge of applicable algorithmic and +political time adjustments, such as time zone and daylight saving time +information, to locate itself relative to other aware objects. An aware object +is used to represent a specific moment in time that is not open to +interpretation [#]_. + ++A naive object does not contain enough information to unambiguously locate ++itself relative to other date/time objects. Whether +a naive object represents Coordinated Universal Time (UTC), local time, or time in some other timezone is purely up to the program, just like it's up to the program whether a particular number represents metres, -miles, or mass. Naive :class:`.datetime` objects are easy to understand and to +miles, or mass. Naive objects are easy to understand and to work with, at the cost of ignoring some aspects of reality. -For applications requiring more, :class:`.datetime` and :class:`.time` objects +For applications requiring aware objects, :class:`.datetime` and :class:`.time` objects have an optional time zone information attribute, :attr:`tzinfo`, that can be set to an instance of a subclass of the abstract :class:`tzinfo` class. These :class:`tzinfo` objects capture information about the offset from UTC time, the @@ -105,10 +112,13 @@ Objects of the :class:`date` type are always naive. -An object *d* of type :class:`.time` or :class:`.datetime` may be naive or aware. -*d* is aware if ``d.tzinfo`` is not ``None`` and ``d.tzinfo.utcoffset(d)`` does -not return ``None``. If ``d.tzinfo`` is ``None``, or if ``d.tzinfo`` is not -``None`` but ``d.tzinfo.utcoffset(d)`` returns ``None``, *d* is naive. +An object of type :class:`.time` or :class:`.datetime` may be naive or aware. +A :class:`.datetime` object *d* is aware if ``d.tzinfo`` is not ``None`` and +``d.tzinfo.utcoffset(d)`` does not return ``None``. If ``d.tzinfo`` is +``None``, or if ``d.tzinfo`` is not ``None`` but ``d.tzinfo.utcoffset(d)`` +returns ``None``, *d* is naive. A :class:`.time` object *t* is aware +if ``t.tzinfo`` is not ``None`` and ``t.tzinfo.utcoffset(None)`` does not return +``None``. Otherwise, *t* is naive. The distinction between naive and aware doesn't apply to :class:`timedelta` objects. @@ -1707,3 +1717,8 @@ (5) For example, if :meth:`utcoffset` returns ``timedelta(hours=-3, minutes=-30)``, ``%z`` is replaced with the string ``'-0330'``. + + +.. rubric:: Footnotes + +.. [#] If, that is, we ignore the effects of Relativity -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 04:33:51 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 15 May 2012 04:33:51 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NzY2OiBSZWZs?= =?utf8?q?ow_the_altered_paragraphs=2E?= Message-ID: http://hg.python.org/cpython/rev/a002638919d8 changeset: 76941:a002638919d8 branch: 2.7 user: R David Murray date: Mon May 14 22:33:36 2012 -0400 summary: #14766: Reflow the altered paragraphs. files: Doc/library/datetime.rst | 34 ++++++++++++++-------------- 1 files changed, 17 insertions(+), 17 deletions(-) diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -14,8 +14,8 @@ The :mod:`datetime` module supplies classes for manipulating dates and times in both simple and complex ways. While date and time arithmetic is supported, the focus of the implementation is on efficient attribute extraction for output -formatting and manipulation. For related -functionality, see also the :mod:`time` and :mod:`calendar` modules. +formatting and manipulation. For related functionality, see also the +:mod:`time` and :mod:`calendar` modules. There are two kinds of date and time objects: "naive" and "aware". @@ -26,22 +26,22 @@ interpretation [#]_. +A naive object does not contain enough information to unambiguously locate -+itself relative to other date/time objects. Whether -a naive object represents Coordinated Universal Time (UTC), -local time, or time in some other timezone is purely up to the program, just -like it's up to the program whether a particular number represents metres, -miles, or mass. Naive objects are easy to understand and to -work with, at the cost of ignoring some aspects of reality. ++itself relative to other date/time objects. Whether a naive object represents +Coordinated Universal Time (UTC), local time, or time in some other timezone is +purely up to the program, just like it's up to the program whether a particular +number represents metres, miles, or mass. Naive objects are easy to understand +and to work with, at the cost of ignoring some aspects of reality. -For applications requiring aware objects, :class:`.datetime` and :class:`.time` objects -have an optional time zone information attribute, :attr:`tzinfo`, that can be -set to an instance of a subclass of the abstract :class:`tzinfo` class. These -:class:`tzinfo` objects capture information about the offset from UTC time, the -time zone name, and whether Daylight Saving Time is in effect. Note that no -concrete :class:`tzinfo` classes are supplied by the :mod:`datetime` module. -Supporting timezones at whatever level of detail is required is up to the -application. The rules for time adjustment across the world are more political -than rational, and there is no standard suitable for every application. +For applications requiring aware objects, :class:`.datetime` and :class:`.time` +objects have an optional time zone information attribute, :attr:`tzinfo`, that +can be set to an instance of a subclass of the abstract :class:`tzinfo` class. +These :class:`tzinfo` objects capture information about the offset from UTC +time, the time zone name, and whether Daylight Saving Time is in effect. Note +that no concrete :class:`tzinfo` classes are supplied by the :mod:`datetime` +module. Supporting timezones at whatever level of detail is required is up to +the application. The rules for time adjustment across the world are more +political than rational, and there is no standard suitable for every +application. The :mod:`datetime` module exports the following constants: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 04:41:32 2012 From: python-checkins at python.org (brian.curtin) Date: Tue, 15 May 2012 04:41:32 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_VS2010_link_and_text?= =?utf8?q?=2C_then_restructure_a_few_things?= Message-ID: http://hg.python.org/devguide/rev/a5de0958d056 changeset: 510:a5de0958d056 user: Brian Curtin date: Mon May 14 21:41:20 2012 -0500 summary: Add VS2010 link and text, then restructure a few things files: setup.rst | 26 +++++++++++++++++--------- 1 files changed, 17 insertions(+), 9 deletions(-) diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -157,24 +157,32 @@ Windows ''''''' -For Microsoft Visual Studio 2008 (also named Visual C++ 9), the ``PCbuild`` -directory contains the build files (for older versions of Visual Studio, see -the ``PC`` directory). The full version of Visual Studio is not necessary -for common tasks with 32-bit builds; the gratis C++ version from -https://www.microsoft.com/visualstudio/en-us/products/2008-editions/express -is sufficient. Its limitations are given at +**Python 3.3** uses Microsoft Visual Studio 2010, which is available at +http://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-cpp-express. + +All versions previous to 3.3 use Microsoft Visual Studio 2008, available at +https://www.microsoft.com/visualstudio/en-us/products/2008-editions/express. + +Regardless of Visual Studio version, the ``PCbuild`` directory of a source +checkout contains the build files for the Python version you are building. +The full version of Visual Studio is not necessary for common tasks with +32-bit builds; the gratis C++ Express versions linked above are sufficient. +The limitations of the Express versions are given at http://msdn.microsoft.com/en-us/library/hs24szh9%28v=VS.90%29.aspx . To build from the Visual Studio GUI, open pcbuild.sln to load the project -files and press F7. Make sure you have chosen the "Debug" build first. +files and choose the Build Solution option from the Build menu, often +associated with the F7 key. Make sure you have chosen the "Debug" option from +the build configuration drop-down first. Once built you might want to set Python as a startup project. Pressing F5 in -Visual Studio will launch the interpreter. +Visual Studio, or choosing Start Debugging from the Debug menu, will launch +the interpreter. If you want to launch the compiled interpreter from the command-line, the path varies according to the build. For a 32-bit build in debug mode, you have to invoke ``PCBuild\python_d.exe``, for a 64-bit build in debug mode, -``PCBuild\AMD64\python_d.exe``. If you are compiling in release mode (which +``PCBuild\amd64\python_d.exe``. If you are compiling in release mode (which you shouldn't, in general), replace ``python_d.exe`` with ``python.exe``. -- Repository URL: http://hg.python.org/devguide From solipsis at pitrou.net Tue May 15 05:50:51 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 15 May 2012 05:50:51 +0200 Subject: [Python-checkins] Daily reference leaks (07afe1ec76a7): sum=1 Message-ID: results for 07afe1ec76a7 on branch "default" -------------------------------------------- test_support leaked [1, 0, 0] references, sum=1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogJEvLSO', '-x'] From python-checkins at python.org Tue May 15 06:03:02 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 May 2012 06:03:02 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_more_descriptive_title?= Message-ID: http://hg.python.org/peps/rev/18ffe33229f1 changeset: 4376:18ffe33229f1 user: Benjamin Peterson date: Mon May 14 21:02:57 2012 -0700 summary: more descriptive title files: pep-0415.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0415.txt b/pep-0415.txt --- a/pep-0415.txt +++ b/pep-0415.txt @@ -1,5 +1,5 @@ PEP: 415 -Title: Implementing PEP 409 differently +Title: Implement context suppression with exception attributes Version: $Revision$ Last-Modified: $Date$ Author: Benjamin Peterson -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 07:09:55 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 May 2012 07:09:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_PEP_415=3A_Implement_suppre?= =?utf8?q?ssion_of_=5F=5Fcontext=5F=5F_display_with_an_exception?= Message-ID: http://hg.python.org/cpython/rev/b0eb7d2a9542 changeset: 76942:b0eb7d2a9542 parent: 76939:6ff172db8114 user: Benjamin Peterson date: Mon May 14 22:09:31 2012 -0700 summary: PEP 415: Implement suppression of __context__ display with an exception attribute This replaces the original PEP 409 implementation. See #14133. files: Doc/c-api/exceptions.rst | 8 +---- Doc/library/exceptions.rst | 21 ++++++------ Doc/library/stdtypes.rst | 9 ++--- Include/pyerrors.h | 4 +- Lib/test/test_exceptions.py | 15 ++++---- Lib/test/test_raise.py | 7 +++- Lib/test/test_sys.py | 6 +- Lib/traceback.py | 13 ++++--- Misc/NEWS | 3 + Objects/exceptions.c | 41 ++++++++++++------------ Python/ceval.c | 21 +++++++----- Python/pythonrun.c | 9 +--- 12 files changed, 79 insertions(+), 78 deletions(-) diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -471,10 +471,6 @@ set by ``raise ... from ...``) associated with the exception as a new reference, as accessible from Python through :attr:`__cause__`. - If there is no cause associated, this returns *NULL* (from Python - ``__cause__ is Ellipsis``). If the cause is :const:`None`, the default - exception display routines stop showing the context chain. - .. c:function:: void PyException_SetCause(PyObject *ex, PyObject *ctx) @@ -482,9 +478,7 @@ it. There is no type check to make sure that *ctx* is either an exception instance or :const:`None`. This steals a reference to *ctx*. - If the cause is set to :const:`None` the default exception display - routines will not display this exception's context, and will not follow the - chain any further. + :attr:`__suppress_context__` is implicitly set to ``True`` by this function. .. _unicodeexceptions: diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -39,18 +39,17 @@ new exception is not handled the traceback that is eventually displayed will include the originating exception(s) and the final exception. -This implicit exception chain can be made explicit by using :keyword:`from` -with :keyword:`raise`. The single argument to :keyword:`from` must be an -exception or :const:`None`, and it will be set as :attr:`__cause__` on the -raised exception. If :attr:`__cause__` is an exception it will be displayed -instead of :attr:`__context__`; if :attr:`__cause__` is None, -:attr:`__context__` will not be displayed by the default exception handling -code. (Note: the default value for :attr:`__context__` is :const:`None`, -while the default value for :attr:`__cause__` is :const:`Ellipsis`.) +This implicit exception chain can be made explicit by using :keyword:`from` with +:keyword:`raise`. The single argument to :keyword:`from` must be an exception +or ``None``. It will be set as :attr:`__cause__` on the raised exception. +Setting :attr:`__cause__` implicitly sets the :attr:`__suppress_context__` to +``True``. If :attr:`__cause__` is an exception, it will be displayed. If +:attr:`__cause__` is present or :attr:`__suppress_context__` has a true value, +:attr:`__context__` will not be displayed. -In either case, the default exception handling code will not display -any of the remaining links in the :attr:`__context__` chain if -:attr:`__cause__` has been set. +In either case, the default exception handling code will not display any of the +remaining links in the :attr:`__context__` chain if :attr:`__cause__` has been +set. Base classes diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2996,11 +2996,10 @@ The Ellipsis Object ------------------- -This object is commonly used by slicing (see :ref:`slicings`), but may also -be used in other situations where a sentinel value other than :const:`None` -is needed. It supports no special operations. There is exactly one ellipsis -object, named :const:`Ellipsis` (a built-in name). ``type(Ellipsis)()`` -produces the :const:`Ellipsis` singleton. +This object is commonly used by slicing (see :ref:`slicings`). It supports no +special operations. There is exactly one ellipsis object, named +:const:`Ellipsis` (a built-in name). ``type(Ellipsis)()`` produces the +:const:`Ellipsis` singleton. It is written as ``Ellipsis`` or ``...``. diff --git a/Include/pyerrors.h b/Include/pyerrors.h --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -10,7 +10,8 @@ /* PyException_HEAD defines the initial segment of every exception class. */ #define PyException_HEAD PyObject_HEAD PyObject *dict;\ PyObject *args; PyObject *traceback;\ - PyObject *context; PyObject *cause; + PyObject *context; PyObject *cause;\ + int suppress_context; typedef struct { PyException_HEAD @@ -114,7 +115,6 @@ /* Cause manipulation (PEP 3134) */ PyAPI_FUNC(PyObject *) PyException_GetCause(PyObject *); PyAPI_FUNC(void) PyException_SetCause(PyObject *, PyObject *); -PyAPI_FUNC(int) _PyException_SetCauseChecked(PyObject *, PyObject *); /* Context manipulation (PEP 3134) */ PyAPI_FUNC(PyObject *) PyException_GetContext(PyObject *); diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -388,18 +388,18 @@ def testChainingAttrs(self): e = Exception() self.assertIsNone(e.__context__) - self.assertIs(e.__cause__, Ellipsis) + self.assertIsNone(e.__cause__) e = TypeError() self.assertIsNone(e.__context__) - self.assertIs(e.__cause__, Ellipsis) + self.assertIsNone(e.__cause__) class MyException(EnvironmentError): pass e = MyException() self.assertIsNone(e.__context__) - self.assertIs(e.__cause__, Ellipsis) + self.assertIsNone(e.__cause__) def testChainingDescriptors(self): try: @@ -408,15 +408,16 @@ e = exc self.assertIsNone(e.__context__) - self.assertIs(e.__cause__, Ellipsis) + self.assertIsNone(e.__cause__) + self.assertFalse(e.__suppress_context__) e.__context__ = NameError() e.__cause__ = None self.assertIsInstance(e.__context__, NameError) self.assertIsNone(e.__cause__) - - e.__cause__ = Ellipsis - self.assertIs(e.__cause__, Ellipsis) + self.assertTrue(e.__suppress_context__) + e.__suppress_context__ = False + self.assertFalse(e.__suppress_context__) def testKeywordArgs(self): # test that builtin exception don't take keyword args, diff --git a/Lib/test/test_raise.py b/Lib/test/test_raise.py --- a/Lib/test/test_raise.py +++ b/Lib/test/test_raise.py @@ -174,11 +174,14 @@ raise ValueError from None except ValueError as exc: self.assertIsNone(exc.__cause__) - raise exc from Ellipsis + self.assertTrue(exc.__suppress_context__) + exc.__suppress_context__ = False + raise exc except ValueError as exc: e = exc - self.assertIs(e.__cause__, Ellipsis) + self.assertIsNone(e.__cause__) + self.assertFalse(e.__suppress_context__) self.assertIsInstance(e.__context__, TypeError) def test_invalid_cause(self): diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -700,14 +700,14 @@ class C(object): pass check(C.__dict__, size(h + 'P')) # BaseException - check(BaseException(), size(h + '5P')) + check(BaseException(), size(h + '5Pi')) # UnicodeEncodeError - check(UnicodeEncodeError("", "", 0, 0, ""), size(h + '5P 2P2PP')) + check(UnicodeEncodeError("", "", 0, 0, ""), size(h + '5Pi 2P2PP')) # UnicodeDecodeError # XXX # check(UnicodeDecodeError("", "", 0, 0, ""), size(h + '5P2PP')) # UnicodeTranslateError - check(UnicodeTranslateError("", 0, 1, ""), size(h + '5P 2P2PP')) + check(UnicodeTranslateError("", 0, 1, ""), size(h + '5Pi 2P2PP')) # ellipses check(Ellipsis, size(h + '')) # EncodingMap diff --git a/Lib/traceback.py b/Lib/traceback.py --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -119,15 +119,16 @@ seen = set() seen.add(exc) its = [] + context = exc.__context__ cause = exc.__cause__ - if cause is Ellipsis: - context = exc.__context__ - if context is not None and context not in seen: - its.append(_iter_chain(context, None, seen)) - its.append([(_context_message, None)]) - elif cause is not None and cause not in seen: + if cause is not None and cause not in seen: its.append(_iter_chain(cause, False, seen)) its.append([(_cause_message, None)]) + elif (context is not None and + not exc.__suppress_context__ and + context not in seen): + its.append(_iter_chain(context, None, seen)) + its.append([(_context_message, None)]) its.append([(exc, custom_tb or exc.__traceback__)]) # itertools.chain is in an extension module and may be unavailable for it in its: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14133 (PEP 415): Implement suppression of __context__ display with an + attribute on BaseException. This replaces the original mechanism of PEP 409. + - Issue #14417: Mutating a dict during lookup now restarts the lookup instead of raising a RuntimeError (undoes issue #14205). diff --git a/Objects/exceptions.c b/Objects/exceptions.c --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -42,6 +42,7 @@ /* the dict is created on the fly in PyObject_GenericSetAttr */ self->dict = NULL; self->traceback = self->cause = self->context = NULL; + self->suppress_context = 0; self->args = PyTuple_New(0); if (!self->args) { @@ -266,24 +267,7 @@ PyObject *res = PyException_GetCause(self); if (res) return res; /* new reference already returned above */ - Py_INCREF(Py_Ellipsis); - return Py_Ellipsis; -} - -int -_PyException_SetCauseChecked(PyObject *self, PyObject *arg) { - if (arg == Py_Ellipsis) { - arg = NULL; - } else if (arg != Py_None && !PyExceptionInstance_Check(arg)) { - PyErr_SetString(PyExc_TypeError, "exception cause must be None, " - "Ellipsis or derive from BaseException"); - return -1; - } else { - /* PyException_SetCause steals a reference */ - Py_INCREF(arg); - } - PyException_SetCause(self, arg); - return 0; + Py_RETURN_NONE; } static int @@ -291,8 +275,18 @@ if (arg == NULL) { PyErr_SetString(PyExc_TypeError, "__cause__ may not be deleted"); return -1; + } else if (arg == Py_None) { + arg = NULL; + } else if (!PyExceptionInstance_Check(arg)) { + PyErr_SetString(PyExc_TypeError, "exception cause must be None " + "or derive from BaseException"); + return -1; + } else { + /* PyException_SetCause steals this reference */ + Py_INCREF(arg); } - return _PyException_SetCauseChecked(self, arg); + PyException_SetCause(self, arg); + return 0; } @@ -333,6 +327,7 @@ PyException_SetCause(PyObject *self, PyObject *cause) { PyObject *old_cause = ((PyBaseExceptionObject *)self)->cause; ((PyBaseExceptionObject *)self)->cause = cause; + ((PyBaseExceptionObject *)self)->suppress_context = 1; Py_XDECREF(old_cause); } @@ -352,6 +347,12 @@ } +static struct PyMemberDef BaseException_members[] = { + {"__suppress_context__", T_BOOL, + offsetof(PyBaseExceptionObject, suppress_context)} +}; + + static PyTypeObject _PyExc_BaseException = { PyVarObject_HEAD_INIT(NULL, 0) "BaseException", /*tp_name*/ @@ -382,7 +383,7 @@ 0, /* tp_iter */ 0, /* tp_iternext */ BaseException_methods, /* tp_methods */ - 0, /* tp_members */ + BaseException_members, /* tp_members */ BaseException_getset, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ diff --git a/Python/ceval.c b/Python/ceval.c --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3572,23 +3572,26 @@ if (cause) { PyObject *fixed_cause; - int result; if (PyExceptionClass_Check(cause)) { fixed_cause = PyObject_CallObject(cause, NULL); if (fixed_cause == NULL) goto raise_error; - Py_CLEAR(cause); - } else { - /* Let "exc.__cause__ = cause" handle all further checks */ + Py_DECREF(cause); + } + else if (PyExceptionInstance_Check(cause)) { fixed_cause = cause; - cause = NULL; /* Steal the reference */ } - /* We retain ownership of the reference to fixed_cause */ - result = _PyException_SetCauseChecked(value, fixed_cause); - Py_DECREF(fixed_cause); - if (result < 0) { + else if (cause == Py_None) { + Py_DECREF(cause); + fixed_cause = NULL; + } + else { + PyErr_SetString(PyExc_TypeError, + "exception causes must derive from " + "BaseException"); goto raise_error; } + PyException_SetCause(value, fixed_cause); } PyErr_SetObject(type, value); diff --git a/Python/pythonrun.c b/Python/pythonrun.c --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1761,11 +1761,7 @@ else if (PyExceptionInstance_Check(value)) { cause = PyException_GetCause(value); context = PyException_GetContext(value); - if (cause && cause == Py_None) { - /* print neither cause nor context */ - ; - } - else if (cause) { + if (cause) { res = PySet_Contains(seen, cause); if (res == -1) PyErr_Clear(); @@ -1776,7 +1772,8 @@ cause_message, f); } } - else if (context) { + else if (context && + !((PyBaseExceptionObject *)value)->suppress_context) { res = PySet_Contains(seen, context); if (res == -1) PyErr_Clear(); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 07:10:31 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 May 2012 07:10:31 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_415_now_implemented?= Message-ID: http://hg.python.org/peps/rev/afff3c93b5aa changeset: 4377:afff3c93b5aa user: Benjamin Peterson date: Mon May 14 22:10:26 2012 -0700 summary: 415 now implemented files: pep-0415.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0415.txt b/pep-0415.txt --- a/pep-0415.txt +++ b/pep-0415.txt @@ -4,7 +4,7 @@ Last-Modified: $Date$ Author: Benjamin Peterson BDFL-Delegate: Nick Coghlan -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 26-Feb-2012 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 07:17:39 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 May 2012 07:17:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_apparently_importlib=2Eh_wa?= =?utf8?q?nts_to_be_updated?= Message-ID: http://hg.python.org/cpython/rev/26b8ec8a7800 changeset: 76943:26b8ec8a7800 user: Benjamin Peterson date: Mon May 14 22:17:34 2012 -0700 summary: apparently importlib.h wants to be updated files: Python/importlib.h | Bin 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/Python/importlib.h b/Python/importlib.h index 38f2c8a53c91bc094a3ef23023d4a06dd9ffa30e..0beeb595dbc38d821fb4f6de0981347f3983420a GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 13:46:19 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 13:46:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Silence_VS_2010_signed/unsi?= =?utf8?q?gned_warnings=2E?= Message-ID: http://hg.python.org/cpython/rev/cda2fbbf2a31 changeset: 76944:cda2fbbf2a31 user: Martin v. L?wis date: Tue May 15 13:45:49 2012 +0200 summary: Silence VS 2010 signed/unsigned warnings. files: Objects/unicodeobject.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13591,7 +13591,10 @@ c = PyUnicode_READ(fmtkind, fmt, fmtpos++); if (c < '0' || c > '9') break; - if (width > (PY_SSIZE_T_MAX - (c - '0')) / 10) { + /* Since c is unsigned, the RHS would end up as unsigned, + mixing signed and unsigned comparison. Since c is between + '0' and '9', casting to int is safe. */ + if (width > (PY_SSIZE_T_MAX - ((int)c - '0')) / 10) { PyErr_SetString(PyExc_ValueError, "width too big"); goto onError; @@ -13626,7 +13629,7 @@ c = PyUnicode_READ(fmtkind, fmt, fmtpos++); if (c < '0' || c > '9') break; - if (prec > (INT_MAX - (c - '0')) / 10) { + if (prec > (INT_MAX - ((int)c - '0')) / 10) { PyErr_SetString(PyExc_ValueError, "prec too big"); goto onError; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 13:53:36 2012 From: python-checkins at python.org (nick.coghlan) Date: Tue, 15 May 2012 13:53:36 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Make_the_relationship_between_?= =?utf8?q?409_and_415_clearer?= Message-ID: http://hg.python.org/peps/rev/630a00e4f19b changeset: 4378:630a00e4f19b user: Nick Coghlan date: Tue May 15 21:53:25 2012 +1000 summary: Make the relationship between 409 and 415 clearer files: pep-0409.txt | 5 +++++ pep-0415.txt | 11 +++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/pep-0409.txt b/pep-0409.txt --- a/pep-0409.txt +++ b/pep-0409.txt @@ -8,6 +8,7 @@ Content-Type: text/x-rst Created: 26-Jan-2012 Post-History: 30-Aug-2002, 01-Feb-2012, 03-Feb-2012 +Superseded-By: 415 Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116136.html @@ -87,6 +88,10 @@ Implementation Discussion ========================= +Note: after acceptance of this PEP, a cleaner implementation mechanism +was proposed and accepted in PEP 415. Refer to that PEP for more +details on the implementation actually used in Python 3.3. + Currently, ``None`` is the default for both ``__context__`` and ``__cause__``. In order to support ``raise ... from None`` (which would set ``__cause__`` to ``None``) we need a different default value for ``__cause__``. Several ideas diff --git a/pep-0415.txt b/pep-0415.txt --- a/pep-0415.txt +++ b/pep-0415.txt @@ -10,16 +10,19 @@ Created: 26-Feb-2012 Python-Version: 3.3 Post-History: 26-Feb-2012 +Replaces: 409 Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119467.html Abstract ======== -PEP 409 allows PEP 3134 exception contexts and causes to be suppressed when the -exception is printed. This is done using the ``raise exc from None`` -syntax. This PEP proposes to implement context and cause suppression -differently. +PEP 409 introduced support for the ``raise exc from None`` construct to +allow the display of the exception context to be explicitly suppressed. +This PEP retains the language level changes already implemented in PEP 409, +but replaces the underlying implementation mechanism with a simpler approach +based on a new ``__suppress_context__`` attribute on all ``BaseException`` +instances. PEP Acceptance -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 14:06:43 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 14:06:43 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Silence_VS_2010_warning_on_?= =?utf8?q?loss_of_precision_=28=5Fint64_-=3E_=5Fint32=29=2E?= Message-ID: http://hg.python.org/cpython/rev/d341898ee4e3 changeset: 76945:d341898ee4e3 user: Martin v. L?wis date: Tue May 15 14:06:21 2012 +0200 summary: Silence VS 2010 warning on loss of precision (_int64 -> _int32). This is safe because the actual value is already range-checked. files: Modules/_lzmamodule.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -160,7 +160,7 @@ "Value too large for " #TYPE " type"); \ return 0; \ } \ - *(TYPE *)ptr = val; \ + *(TYPE *)ptr = (TYPE)val; \ return 1; \ } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 14:19:28 2012 From: python-checkins at python.org (nick.coghlan) Date: Tue, 15 May 2012 14:19:28 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Accept_PEP_3144=2C_and_clean_u?= =?utf8?q?p_some_typos_and_text_duplication?= Message-ID: http://hg.python.org/peps/rev/7c0fd1cb08c8 changeset: 4379:7c0fd1cb08c8 user: Nick Coghlan date: Tue May 15 22:19:17 2012 +1000 summary: Accept PEP 3144, and clean up some typos and text duplication files: pep-3144.txt | 35 ++++++++++++----------------------- 1 files changed, 12 insertions(+), 23 deletions(-) diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -5,30 +5,36 @@ Author: Peter Moody BDFL-Delegate: Nick Coghlan Discussions-To: -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/plain Created: 6-Feb-2012 Python-Version: 3.3 - +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119474.html Abstract: This PEP proposes a design and for an IP address manipulation module for python. + +PEP Acceptance: + + This PEP was accepted by Nick Coghlan on the 15th of May, 2012. + + Motivation: Several very good IP address modules for python already exist. - The truth is that all of the struggle with the balance between + The truth is that all of them struggle with the balance between adherence to Pythonic principals and the shorthand upon which - network engineers and administrators rely. I believe ipaddr - strikes the right balance. + network engineers and administrators rely. ipaddress aims to + strike the right balance. Rationale: - The existance of several Python IP address manipulation moduels is + The existence of several Python IP address manipulation modules is evidence of an outstanding need for the functionality this module seeks to provide. @@ -56,10 +62,6 @@ well. (eg. network, network_address) * A number of methods and functions which returned containers in ipaddr now - return iterators. This includes, subnets, address_exclude, - summarize_address_range and collapse_address_list. - - * A number of methods and functions which returned containers in ipaddr now return iterators. This includes, subnets, address_exclude, summarize_address_range and collapse_address_list. @@ -68,19 +70,6 @@ the proposal is to add the module using the new provisional API status: * http://docs.python.org/dev/glossary.html#term-provisional-package - - - Relevant messages on python-dev: - - * http://mail.python.org/pipermail/python-dev/2012-January/116016.html - * http://mail.python.org/pipermail/python-dev/2012-February/116656.html - * http://mail.python.org/pipermail/python-dev/2012-February/116688.html - - - Due to the backwards incompatible API changes between ipaddress and ipaddr, - the proposal is to add the module using the new provisional API status: - - * http://docs.python.org/dev/glossary.html#term-provisional-package Relevant messages on python-dev: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 14:35:24 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 14:35:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_safecase_to_silence_Win?= =?utf8?q?64_warning=2E?= Message-ID: http://hg.python.org/cpython/rev/d274fa027cf5 changeset: 76946:d274fa027cf5 user: Martin v. L?wis date: Tue May 15 14:34:58 2012 +0200 summary: Add safecase to silence Win64 warning. files: Modules/itertoolsmodule.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -533,7 +533,8 @@ tdo->values[i] = PyList_GET_ITEM(values, i); Py_INCREF(tdo->values[i]); } - tdo->numread = len; + /* len <= LINKCELLS < INT_MAX */ + tdo->numread = Py_SAFE_DOWNCAST(len, Py_ssize_t, int); if (len == LINKCELLS) { if (next != Py_None) { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 14:45:23 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 14:45:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Widen_ASDL_sequences_to_Py?= =?utf8?q?=5Fssize=5Ft_lengths_to_better_match_PEP_353=2E?= Message-ID: http://hg.python.org/cpython/rev/f4d7ad6c9d6e changeset: 76947:f4d7ad6c9d6e user: Martin v. L?wis date: Tue May 15 14:45:03 2012 +0200 summary: Widen ASDL sequences to Py_ssize_t lengths to better match PEP 353. files: Include/asdl.h | 8 ++++---- Python/Python-ast.c | 4 ++-- Python/asdl.c | 4 ++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/Include/asdl.h b/Include/asdl.h --- a/Include/asdl.h +++ b/Include/asdl.h @@ -15,17 +15,17 @@ /* XXX A sequence should be typed so that its use can be typechecked. */ typedef struct { - int size; + Py_ssize_t size; void *elements[1]; } asdl_seq; typedef struct { - int size; + Py_ssize_t size; int elements[1]; } asdl_int_seq; -asdl_seq *asdl_seq_new(int size, PyArena *arena); -asdl_int_seq *asdl_int_seq_new(int size, PyArena *arena); +asdl_seq *asdl_seq_new(Py_ssize_t size, PyArena *arena); +asdl_int_seq *asdl_int_seq_new(Py_ssize_t size, PyArena *arena); #define asdl_seq_GET(S, I) (S)->elements[(I)] #define asdl_seq_LEN(S) ((S) == NULL ? 0 : (S)->size) diff --git a/Python/Python-ast.c b/Python/Python-ast.c --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -636,7 +636,7 @@ static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) { - int i, n = asdl_seq_LEN(seq); + Py_ssize_t i, n = asdl_seq_LEN(seq); PyObject *result = PyList_New(n); PyObject *value; if (!result) @@ -2857,7 +2857,7 @@ goto failed; Py_DECREF(value); { - int i, n = asdl_seq_LEN(o->v.Compare.ops); + Py_ssize_t i, n = asdl_seq_LEN(o->v.Compare.ops); value = PyList_New(n); if (!value) goto failed; for(i = 0; i < n; i++) diff --git a/Python/asdl.c b/Python/asdl.c --- a/Python/asdl.c +++ b/Python/asdl.c @@ -2,7 +2,7 @@ #include "asdl.h" asdl_seq * -asdl_seq_new(int size, PyArena *arena) +asdl_seq_new(Py_ssize_t size, PyArena *arena) { asdl_seq *seq = NULL; size_t n = (size ? (sizeof(void *) * (size - 1)) : 0); @@ -33,7 +33,7 @@ } asdl_int_seq * -asdl_int_seq_new(int size, PyArena *arena) +asdl_int_seq_new(Py_ssize_t size, PyArena *arena) { asdl_int_seq *seq = NULL; size_t n = (size ? (sizeof(void *) * (size - 1)) : 0); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 14:52:45 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 14:52:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Document_f4d7ad6c9d6e=2E?= Message-ID: http://hg.python.org/cpython/rev/99ed87c8e24c changeset: 76948:99ed87c8e24c user: Martin v. L?wis date: Tue May 15 14:52:36 2012 +0200 summary: Document f4d7ad6c9d6e. files: Misc/NEWS | 2 ++ Python/Python-ast.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- asdl_seq and asdl_int_seq are now Py_ssize_t sized. + - Issue #14133 (PEP 415): Implement suppression of __context__ display with an attribute on BaseException. This replaces the original mechanism of PEP 409. diff --git a/Python/Python-ast.c b/Python/Python-ast.c --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -636,7 +636,7 @@ static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) { - Py_ssize_t i, n = asdl_seq_LEN(seq); + int i, n = asdl_seq_LEN(seq); PyObject *result = PyList_New(n); PyObject *value; if (!result) @@ -2857,7 +2857,7 @@ goto failed; Py_DECREF(value); { - Py_ssize_t i, n = asdl_seq_LEN(o->v.Compare.ops); + int i, n = asdl_seq_LEN(o->v.Compare.ops); value = PyList_New(n); if (!value) goto failed; for(i = 0; i < n; i++) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 15:30:29 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Tue, 15 May 2012 15:30:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314807=3A_move_undocument?= =?utf8?q?ed_tarfile=2Efilemode=28=29_to_stat=2Efilemode=28=29=2E_Add?= Message-ID: http://hg.python.org/cpython/rev/492e6c6a01bb changeset: 76949:492e6c6a01bb user: Giampaolo Rodola' date: Tue May 15 15:30:25 2012 +0200 summary: #14807: move undocumented tarfile.filemode() to stat.filemode(). Add tarfile.filemode alias with deprecation warning. files: Doc/library/stat.rst | 11 +++++- Doc/whatsnew/3.3.rst | 8 ++++ Lib/stat.py | 40 ++++++++++++++++++++++ Lib/tarfile.py | 48 +++----------------------- Lib/test/test_stat.py | 55 +++++++++++++++++++++++++++++++ Misc/NEWS | 5 ++- 6 files changed, 124 insertions(+), 43 deletions(-) diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -104,6 +104,16 @@ if __name__ == '__main__': walktree(sys.argv[1], visitfile) +An additional utility function is provided to covert a file's mode in a human +readable string: + +.. function:: filemode(mode) + + Convert a file's mode to a string of the form '-rwxrwxrwx'. + + .. versionadded:: 3.3 + + All the variables below are simply symbolic indexes into the 10-tuple returned by :func:`os.stat`, :func:`os.fstat` or :func:`os.lstat`. @@ -344,4 +354,3 @@ The file is a snapshot file. See the \*BSD or Mac OS systems man page :manpage:`chflags(2)` for more information. - diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1177,6 +1177,14 @@ (Contributed by Colin Marc in :issue:`14204`) +stat +---- + +- The undocumented tarfile.filemode function has been moved to + :func:`stat.filemode`. It can be used to convert a file's mode to a string of + the form '-rwxrwxrwx'. + + (Contributed by Giampaolo Rodol? in :issue:`14807`) sys --- diff --git a/Lib/stat.py b/Lib/stat.py --- a/Lib/stat.py +++ b/Lib/stat.py @@ -107,3 +107,43 @@ SF_APPEND = 0x00040000 # file may only be appended to SF_NOUNLINK = 0x00100000 # file may not be renamed or deleted SF_SNAPSHOT = 0x00200000 # file is a snapshot file + + +_filemode_table = ( + ((S_IFLNK, "l"), + (S_IFREG, "-"), + (S_IFBLK, "b"), + (S_IFDIR, "d"), + (S_IFCHR, "c"), + (S_IFIFO, "p")), + + ((S_IRUSR, "r"),), + ((S_IWUSR, "w"),), + ((S_IXUSR|S_ISUID, "s"), + (S_ISUID, "S"), + (S_IXUSR, "x")), + + ((S_IRGRP, "r"),), + ((S_IWGRP, "w"),), + ((S_IXGRP|S_ISGID, "s"), + (S_ISGID, "S"), + (S_IXGRP, "x")), + + ((S_IROTH, "r"),), + ((S_IWOTH, "w"),), + ((S_IXOTH|S_ISVTX, "t"), + (S_ISVTX, "T"), + (S_IXOTH, "x")) +) + +def filemode(mode): + """Convert a file's mode to a string of the form '-rwxrwxrwx'.""" + perm = [] + for table in _filemode_table: + for bit, char in table: + if mode & bit == bit: + perm.append(char) + break + else: + perm.append("-") + return "".join(perm) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -274,47 +274,13 @@ dst.write(buf) return -filemode_table = ( - ((S_IFLNK, "l"), - (S_IFREG, "-"), - (S_IFBLK, "b"), - (S_IFDIR, "d"), - (S_IFCHR, "c"), - (S_IFIFO, "p")), +def filemode(mode): + """Deprecated in this location; use stat.filemode.""" + import warnings + warnings.warn("deprecated in favor of stat.filemode", + DeprecationWarning, 2) + return stat.filemode(mode) - ((TUREAD, "r"),), - ((TUWRITE, "w"),), - ((TUEXEC|TSUID, "s"), - (TSUID, "S"), - (TUEXEC, "x")), - - ((TGREAD, "r"),), - ((TGWRITE, "w"),), - ((TGEXEC|TSGID, "s"), - (TSGID, "S"), - (TGEXEC, "x")), - - ((TOREAD, "r"),), - ((TOWRITE, "w"),), - ((TOEXEC|TSVTX, "t"), - (TSVTX, "T"), - (TOEXEC, "x")) -) - -def filemode(mode): - """Convert a file's mode to a string of the form - -rwxrwxrwx. - Used by TarFile.list() - """ - perm = [] - for table in filemode_table: - for bit, char in table: - if mode & bit == bit: - perm.append(char) - break - else: - perm.append("-") - return "".join(perm) class TarError(Exception): """Base exception.""" @@ -1891,7 +1857,7 @@ for tarinfo in self: if verbose: - print(filemode(tarinfo.mode), end=' ') + print(stat.filemode(tarinfo.mode), end=' ') print("%s/%s" % (tarinfo.uname or tarinfo.uid, tarinfo.gname or tarinfo.gid), end=' ') if tarinfo.ischr() or tarinfo.isblk(): diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_stat.py @@ -0,0 +1,55 @@ +import unittest +import os +import stat +from test.support import TESTFN, run_unittest + + +def get_mode(fname=TESTFN): + return stat.filemode(os.lstat(fname).st_mode) + + +class TestFilemode(unittest.TestCase): + + def setUp(self): + try: + os.remove(TESTFN) + except OSError: + try: + os.rmdir(TESTFN) + except OSError: + pass + tearDown = setUp + + def test_mode(self): + with open(TESTFN, 'w'): + pass + os.chmod(TESTFN, 0o700) + self.assertEqual(get_mode(), '-rwx------') + os.chmod(TESTFN, 0o070) + self.assertEqual(get_mode(), '----rwx---') + os.chmod(TESTFN, 0o007) + self.assertEqual(get_mode(), '-------rwx') + os.chmod(TESTFN, 0o444) + self.assertEqual(get_mode(), '-r--r--r--') + + def test_directory(self): + os.mkdir(TESTFN) + os.chmod(TESTFN, 0o700) + self.assertEqual(get_mode(), 'drwx------') + + @unittest.skipUnless(hasattr(os, 'symlink'), 'os.symlink not available') + def test_link(self): + os.symlink(os.getcwd(), TESTFN) + self.assertEqual(get_mode(), 'lrwxrwxrwx') + + @unittest.skipUnless(hasattr(os, 'mkfifo'), 'os.mkfifo not available') + def test_fifo(self): + os.mkfifo(TESTFN, 0o700) + self.assertEqual(get_mode(), 'prwx------') + + +def test_main(): + run_unittest(TestFilemode) + +if __name__ == '__main__': + test_main() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,10 +31,13 @@ Library ------- +- Issue 14807: move undocumented tarfile.filemode() to stat.filemode() and add + doc entry. Add tarfile.filemode alias with deprecation warning. + - Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. - Issue #14532: Add a secure_compare() helper to the hmac module, to mitigate - timing attacks. Patch by Jon Oberheide. + timing attacks. Patch by Jon Oberheide. - Add importlib.util.resolve_name(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 15:51:53 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Tue, 15 May 2012 15:51:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_remove_unused_import_in_asy?= =?utf8?q?ncore_doc?= Message-ID: http://hg.python.org/cpython/rev/d4c590cee68b changeset: 76950:d4c590cee68b user: Giampaolo Rodola' date: Tue May 15 15:46:00 2012 +0200 summary: remove unused import in asyncore doc files: Doc/library/asynchat.rst | 3 +++ Doc/library/asyncore.rst | 4 +--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Doc/library/asynchat.rst b/Doc/library/asynchat.rst --- a/Doc/library/asynchat.rst +++ b/Doc/library/asynchat.rst @@ -197,6 +197,9 @@ marshalled, after setting the channel terminator to ``None`` to ensure that any extraneous data sent by the web client are ignored. :: + + import asynchat + class http_request_handler(asynchat.async_chat): def __init__(self, sock, addr, sessions, log): diff --git a/Doc/library/asyncore.rst b/Doc/library/asyncore.rst --- a/Doc/library/asyncore.rst +++ b/Doc/library/asyncore.rst @@ -277,7 +277,7 @@ Here is a very basic HTTP client that uses the :class:`dispatcher` class to implement its socket handling:: - import asyncore, socket + import asyncore class HTTPClient(asyncore.dispatcher): @@ -317,7 +317,6 @@ connections and dispatches the incoming connections to a handler:: import asyncore - import socket class EchoHandler(asyncore.dispatcher_with_send): @@ -341,4 +340,3 @@ server = EchoServer('localhost', 8080) asyncore.loop() - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 16:34:47 2012 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 15 May 2012 16:34:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314773=3A_Fix_os=2Efwalk?= =?utf8?q?=28=29_failing_on_dangling_symlinks?= Message-ID: http://hg.python.org/cpython/rev/cbe7560d4443 changeset: 76951:cbe7560d4443 user: Hynek Schlawack date: Tue May 15 16:32:21 2012 +0200 summary: #14773: Fix os.fwalk() failing on dangling symlinks files: Lib/os.py | 24 +++++++++++++++++------- Lib/test/test_os.py | 6 +++++- Misc/NEWS | 2 ++ 3 files changed, 24 insertions(+), 8 deletions(-) diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -353,13 +353,23 @@ names = flistdir(topfd) dirs, nondirs = [], [] for name in names: - # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with - # walk() which reports symlinks to directories as directories. We do - # however check for symlinks before recursing into a subdirectory. - if st.S_ISDIR(fstatat(topfd, name).st_mode): - dirs.append(name) - else: - nondirs.append(name) + try: + # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with + # walk() which reports symlinks to directories as directories. + # We do however check for symlinks before recursing into + # a subdirectory. + if st.S_ISDIR(fstatat(topfd, name).st_mode): + dirs.append(name) + else: + nondirs.append(name) + except FileNotFoundError: + try: + # Add dangling symlinks, ignore disappeared files + if st.S_ISLNK(fstatat(topfd, name, AT_SYMLINK_NOFOLLOW) + .st_mode): + nondirs.append(name) + except FileNotFoundError: + continue if topdown: yield toppath, dirs, nondirs, topfd diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -651,6 +651,7 @@ # SUB2/ a file kid and a dirsymlink kid # tmp3 # link/ a symlink to TESTFN.2 + # broken_link # TEST2/ # tmp4 a lone file walk_path = join(support.TESTFN, "TEST1") @@ -663,6 +664,8 @@ link_path = join(sub2_path, "link") t2_path = join(support.TESTFN, "TEST2") tmp4_path = join(support.TESTFN, "TEST2", "tmp4") + link_path = join(sub2_path, "link") + broken_link_path = join(sub2_path, "broken_link") # Create stuff. os.makedirs(sub11_path) @@ -679,7 +682,8 @@ else: symlink_to_dir = os.symlink symlink_to_dir(os.path.abspath(t2_path), link_path) - sub2_tree = (sub2_path, ["link"], ["tmp3"]) + symlink_to_dir('broken', broken_link_path) + sub2_tree = (sub2_path, ["link"], ["broken_link", "tmp3"]) else: sub2_tree = (sub2_path, [], ["tmp3"]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,6 +31,8 @@ Library ------- +- Issue 14773: Fix os.fwalk() failing on dangling symlinks. + - Issue 14807: move undocumented tarfile.filemode() to stat.filemode() and add doc entry. Add tarfile.filemode alias with deprecation warning. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 16:42:27 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 16:42:27 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyNTQx?= =?utf8?q?=3A_Be_lenient_with_quotes_around_Realm_field_with_HTTP_Basic?= Message-ID: http://hg.python.org/cpython/rev/3e10d0148f79 changeset: 76952:3e10d0148f79 branch: 2.7 parent: 76941:a002638919d8 user: Senthil Kumaran date: Tue May 15 22:24:10 2012 +0800 summary: Issue #12541: Be lenient with quotes around Realm field with HTTP Basic Authentation in urllib2. files: Lib/test/test_urllib2.py | 16 ++++++++++++++++ Lib/urllib2.py | 2 +- Misc/NEWS | 3 +++ 3 files changed, 20 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1112,6 +1112,22 @@ def test_basic_auth_with_single_quoted_realm(self): self.test_basic_auth(quote_char="'") + def test_basic_auth_with_unquoted_realm(self): + opener = OpenerDirector() + password_manager = MockPasswordManager() + auth_handler = urllib2.HTTPBasicAuthHandler(password_manager) + realm = "ACME Widget Store" + http_handler = MockHTTPHandler( + 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) + opener.add_handler(auth_handler) + opener.add_handler(http_handler) + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) + + def test_proxy_basic_auth(self): opener = OpenerDirector() ph = urllib2.ProxyHandler(dict(http="proxy.example.com:3128")) diff --git a/Lib/urllib2.py b/Lib/urllib2.py --- a/Lib/urllib2.py +++ b/Lib/urllib2.py @@ -828,7 +828,7 @@ # allow for double- and single-quoted realm values # (single quotes are a violation of the RFC, but appear in the wild) rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+' - 'realm=(["\'])(.*?)\\2', re.I) + 'realm=(["\']?)([^"\']*)\\2', re.I) # XXX could pre-emptively send auth info already accepted (RFC 2617, # end of section 2, and section 1.2 immediately after "credentials" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #12541: Be lenient with quotes around Realm field with HTTP Basic + Authentation in urllib2. + - Issue #14662: Prevent shutil failures on OS X when destination does not support chflag operations. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 16:42:27 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 16:42:27 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNTQx?= =?utf8?q?=3A_Be_lenient_with_quotes_around_Realm_field_of_HTTP_Basic?= Message-ID: http://hg.python.org/cpython/rev/bb94fec5c5ab changeset: 76953:bb94fec5c5ab branch: 3.2 parent: 76938:b1e03e863386 user: Senthil Kumaran date: Tue May 15 22:30:25 2012 +0800 summary: Issue #12541: Be lenient with quotes around Realm field of HTTP Basic Authentation in urllib2. G: changed Misc/NEWS files: Lib/test/test_urllib2.py | 15 +++++++++++++++ Lib/urllib/request.py | 2 +- Misc/NEWS | 3 +++ 3 files changed, 19 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1218,6 +1218,21 @@ def test_basic_auth_with_single_quoted_realm(self): self.test_basic_auth(quote_char="'") + def test_basic_auth_with_unquoted_realm(self): + opener = OpenerDirector() + password_manager = MockPasswordManager() + auth_handler = urllib.request.HTTPBasicAuthHandler(password_manager) + realm = "ACME Widget Store" + http_handler = MockHTTPHandler( + 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) + opener.add_handler(auth_handler) + opener.add_handler(http_handler) + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) + def test_proxy_basic_auth(self): opener = OpenerDirector() ph = urllib.request.ProxyHandler(dict(http="proxy.example.com:3128")) diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -794,7 +794,7 @@ # allow for double- and single-quoted realm values # (single quotes are a violation of the RFC, but appear in the wild) rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+' - 'realm=(["\'])(.*?)\\2', re.I) + 'realm=(["\']?)([^"\']*)\\2', re.I) # XXX could pre-emptively send auth info already accepted (RFC 2617, # end of section 2, and section 1.2 immediately after "credentials" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ Library ------- +- Issue #12541: Be lenient with quotes around Realm field of HTTP Basic + Authentation in urllib2. + - Issue #14662: Prevent shutil failures on OS X when destination does not support chflag operations. Patch by Hynek Schlawack. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 16:42:29 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 16:42:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_from_3=2E2_-_Issue_=2312541=3A_Be_lenient_with_quotes_?= =?utf8?q?around_Realm_field_of?= Message-ID: http://hg.python.org/cpython/rev/bf20564296aa changeset: 76954:bf20564296aa parent: 76950:d4c590cee68b parent: 76953:bb94fec5c5ab user: Senthil Kumaran date: Tue May 15 22:39:17 2012 +0800 summary: merge from 3.2 - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic Authentation in urllib2. files: Lib/test/test_urllib2.py | 15 +++++++++++++++ Lib/urllib/request.py | 2 +- Misc/NEWS | 3 +++ 3 files changed, 19 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1252,6 +1252,21 @@ def test_basic_auth_with_single_quoted_realm(self): self.test_basic_auth(quote_char="'") + def test_basic_auth_with_unquoted_realm(self): + opener = OpenerDirector() + password_manager = MockPasswordManager() + auth_handler = urllib.request.HTTPBasicAuthHandler(password_manager) + realm = "ACME Widget Store" + http_handler = MockHTTPHandler( + 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) + opener.add_handler(auth_handler) + opener.add_handler(http_handler) + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) + def test_proxy_basic_auth(self): opener = OpenerDirector() ph = urllib.request.ProxyHandler(dict(http="proxy.example.com:3128")) diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -895,7 +895,7 @@ # allow for double- and single-quoted realm values # (single quotes are a violation of the RFC, but appear in the wild) rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+' - 'realm=(["\'])(.*?)\\2', re.I) + 'realm=(["\']?)([^"\']*)\\2', re.I) # XXX could pre-emptively send auth info already accepted (RFC 2617, # end of section 2, and section 1.2 immediately after "credentials" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,6 +31,9 @@ Library ------- +- Issue #12541: Be lenient with quotes around Realm field of HTTP Basic + Authentation in urllib2. + - Issue 14807: move undocumented tarfile.filemode() to stat.filemode() and add doc entry. Add tarfile.filemode alias with deprecation warning. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 16:42:30 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 16:42:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/cdea40514623 changeset: 76955:cdea40514623 parent: 76954:bf20564296aa parent: 76951:cbe7560d4443 user: Senthil Kumaran date: Tue May 15 22:42:11 2012 +0800 summary: merge heads files: Lib/os.py | 24 +++++++++++++++++------- Lib/test/test_os.py | 6 +++++- Misc/NEWS | 2 ++ 3 files changed, 24 insertions(+), 8 deletions(-) diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -353,13 +353,23 @@ names = flistdir(topfd) dirs, nondirs = [], [] for name in names: - # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with - # walk() which reports symlinks to directories as directories. We do - # however check for symlinks before recursing into a subdirectory. - if st.S_ISDIR(fstatat(topfd, name).st_mode): - dirs.append(name) - else: - nondirs.append(name) + try: + # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with + # walk() which reports symlinks to directories as directories. + # We do however check for symlinks before recursing into + # a subdirectory. + if st.S_ISDIR(fstatat(topfd, name).st_mode): + dirs.append(name) + else: + nondirs.append(name) + except FileNotFoundError: + try: + # Add dangling symlinks, ignore disappeared files + if st.S_ISLNK(fstatat(topfd, name, AT_SYMLINK_NOFOLLOW) + .st_mode): + nondirs.append(name) + except FileNotFoundError: + continue if topdown: yield toppath, dirs, nondirs, topfd diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -651,6 +651,7 @@ # SUB2/ a file kid and a dirsymlink kid # tmp3 # link/ a symlink to TESTFN.2 + # broken_link # TEST2/ # tmp4 a lone file walk_path = join(support.TESTFN, "TEST1") @@ -663,6 +664,8 @@ link_path = join(sub2_path, "link") t2_path = join(support.TESTFN, "TEST2") tmp4_path = join(support.TESTFN, "TEST2", "tmp4") + link_path = join(sub2_path, "link") + broken_link_path = join(sub2_path, "broken_link") # Create stuff. os.makedirs(sub11_path) @@ -679,7 +682,8 @@ else: symlink_to_dir = os.symlink symlink_to_dir(os.path.abspath(t2_path), link_path) - sub2_tree = (sub2_path, ["link"], ["tmp3"]) + symlink_to_dir('broken', broken_link_path) + sub2_tree = (sub2_path, ["link"], ["broken_link", "tmp3"]) else: sub2_tree = (sub2_path, [], ["tmp3"]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,6 +31,8 @@ Library ------- +- Issue 14773: Fix os.fwalk() failing on dangling symlinks. + - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic Authentation in urllib2. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 17:04:31 2012 From: python-checkins at python.org (barry.warsaw) Date: Tue, 15 May 2012 17:04:31 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Latest_update_from_Eric?= Message-ID: http://hg.python.org/peps/rev/b9919c6057ca changeset: 4380:b9919c6057ca user: Barry Warsaw date: Tue May 15 11:04:25 2012 -0400 summary: Latest update from Eric files: pep-0421.txt | 61 +++++++++++++++++++++------------------ 1 files changed, 33 insertions(+), 28 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -34,7 +34,7 @@ viable alternate implementations of Python. Consider, however, the nearly two decades of CPython-centric Python -(i.e. most of its existence). That focus had understandably +(i.e. most of its existence). That focus has understandably contributed to quite a few CPython-specific artifacts both in the standard library and exposed in the interpreter. Though the core developers have made an effort in recent years to address this, quite @@ -60,7 +60,7 @@ ensures behaviors don't change between versions which depend on attributes of ``sys.implementation``. -The object will have each of the attributes described in the `Required +The object has each of the attributes described in the `Required Attributes`_ section below. Those attribute names will never start with an underscore. The standard library and the language definition will rely only on those required attributes. @@ -114,9 +114,9 @@ specification. All proposals for new required attributes will go through the normal -PEP process. Such a PEP need not be all that long, but will need to -sufficiently spell out the rationale for the new attribute, its use -cases, and the impact it will have on the various Python +PEP process. Such a PEP need not be long, just long enough. It will +need to sufficiently spell out the rationale for the new attribute, +its use cases, and the impact it will have on the various Python implemenations. @@ -153,13 +153,13 @@ It's very easy to get bogged down in discussions about the type of ``sys.implementation``. However, its purpose is to support the standard library and language definition. As such, there isn't much -that really matters in this regard, as opposed to a feature that would -be more generally used. Thus characteristics like immutability and -sequence-ness have been disregarded. +that really matters regarding its type, as opposed to a feature that +would be more generally used. Thus characteristics like immutability +and sequence-ness have been disregarded. -The only real choice was between an object with attribute access and a -mapping with item access. This PEP espouses dotted access to reflect -the relatively fixed nature of the namespace. +The only real choice has been between an object with attribute access +and a mapping with item access. This PEP espouses dotted access to +reflect the relatively fixed nature of the namespace. Non-Required Attributes @@ -179,8 +179,8 @@ Why a Part of ``sys``? ---------------------- -The ``sys`` module should hold the new namespace because ``sys`` is -the depot for interpreter-centric variables and functions. Many +The ``sys`` module holds the new namespace because ``sys`` is the depot +for interpreter-centric variables and functions. Many implementation-specific attributes are already found in ``sys``. @@ -191,8 +191,8 @@ ``sys.implementation`` are intended for use by the standard library. Constraining those values, essentially specifying an API for them, allows them to be used consistently, regardless of how they are -otherwise implemented. However, care should be take to not over- -specify the constraints. +otherwise implemented. However, care should be take to not +over-specify the constraints. Discussion @@ -204,7 +204,7 @@ ``imp.get_tag()`` [#revived]_. Discussion has been ongoing [#feedback]_. The messages in `issue #14673`_ are also relevant. -A good part of the discussion centered on the type to use for +A good part of the recent discussion centered on the type to use for ``sys.implementation``. @@ -228,7 +228,7 @@ module. Another concern is that the ``platform`` module is part of the stdlib, -which ideally would minimize implementation details such as would be +which ideally should minimize implementation details such as would be moved to ``sys.implementation``. Any overlap between ``sys.implementation`` and the ``platform`` module @@ -301,7 +301,7 @@ As Barry Warsaw noted, the "semantics of sys.version_info have been sufficiently squishy in the past" [#Barry]_. With -``sys.implementation`` we have the opportunity to improving the +``sys.implementation`` we have the opportunity to improve this situation by first establishing an explicit location for the version of the implementation. @@ -320,7 +320,8 @@ Jeff Hardy responded to a request for feedback [#ironpython]_. He said, "I'll probably add it the day after it's approved" [#jeff_hardy_2012]_. He also gave useful feedback on both the type of -``sys.implementation`` and on the ``metadata`` attribute. +``sys.implementation`` and on the ``metadata`` attribute (which has +since been removed from the PEP). Jython ------ @@ -349,22 +350,22 @@ Past Efforts ============ -PEP 3139 --------- +``PEP 3139`` +------------ -This PEP from 2008 recommended a clean-up of the ``sys`` module in +PEP 3139, from 2008, recommended a clean-up of the ``sys`` module in part by extracting implementation-specific variables and functions into a separate module. PEP 421 is less ambitious version of that idea. While PEP 3139 was rejected, its goals are reflected in PEP 421 to a large extent, though with a much lighter approach. -PEP 399 -------- +``PEP 399`` +----------- -This informational PEP dictates policy regarding the standard library, -helping to make it friendlier to alternate implementations. PEP 421 -is proposed in that same spirit. +PEP 399 dictates policy regarding the standard library, helping to make +it friendlier to alternate implementations. PEP 421 is proposed in +that same spirit. The Bigger Picture @@ -372,7 +373,7 @@ It's worth noting again that this PEP is a small part of a larger on-going effort to identify the implementation-specific parts of Python -and mitigate their impact on other implementations than CPython. +and mitigate their impact on alternate implementations. ``sys.implementation`` is a focal point for implementation-specific data, acting as a nexus for cooperation between the language, the @@ -398,8 +399,10 @@ Examples of Other Attributes ============================ -These are examples only and not part of the proposal. (See `Adding -New Required Attributes`_ if they get you excited.) +These are examples only and not part of the proposal. Most of them +were suggested during previous discussions, but did not fit into the +goals of this PEP. (See `Adding New Required Attributes`_ if they get +you excited.) **common_name** The case-sensitive name by which the implementation is known. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 17:26:38 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 17:26:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Upgrade_openssl?= =?utf8?b?IHRvIDAuOS44eC4=?= Message-ID: http://hg.python.org/cpython/rev/458986418841 changeset: 76956:458986418841 branch: 2.7 parent: 76952:3e10d0148f79 user: Martin v. L?wis date: Tue May 15 17:26:31 2012 +0200 summary: Upgrade openssl to 0.9.8x. files: Misc/NEWS | 2 ++ PC/VS8.0/pyproject.vsprops | 2 +- PCbuild/pyproject.vsprops | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 4 ++-- 5 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -157,6 +157,8 @@ Build ----- +- Build against bzip2 1.0.6 and openssl 0.9.8x on Windows. + - Issue #14557: Fix extensions build on HP-UX. Patch by Adi Roiban. - Issue #14437: Fix building the _io module under Cygwin. diff --git a/PC/VS8.0/pyproject.vsprops b/PC/VS8.0/pyproject.vsprops --- a/PC/VS8.0/pyproject.vsprops +++ b/PC/VS8.0/pyproject.vsprops @@ -82,7 +82,7 @@ /> http://hg.python.org/cpython/rev/cfb870c6bb8e changeset: 76957:cfb870c6bb8e parent: 76955:cdea40514623 user: Hynek Schlawack date: Tue May 15 17:55:38 2012 +0200 summary: Sort file list in test_os.WalkTests Adding new files into the tree lead to buildbot fails as the order wasn't deterministic. files: Lib/test/test_os.py | 1 + Misc/NEWS | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -695,6 +695,7 @@ # flipped: TESTFN, SUB2, SUB1, SUB11 flipped = all[0][1][0] != "SUB1" all[0][1].sort() + all[3 - 2 * flipped][-1].sort() self.assertEqual(all[0], (walk_path, ["SUB1", "SUB2"], ["tmp1"])) self.assertEqual(all[1 + flipped], (sub1_path, ["SUB11"], ["tmp2"])) self.assertEqual(all[2 + flipped], (sub11_path, [], [])) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,12 +31,12 @@ Library ------- -- Issue 14773: Fix os.fwalk() failing on dangling symlinks. +- Issue #14773: Fix os.fwalk() failing on dangling symlinks. - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic Authentation in urllib2. -- Issue 14807: move undocumented tarfile.filemode() to stat.filemode() and add +- Issue #14807: move undocumented tarfile.filemode() to stat.filemode() and add doc entry. Add tarfile.filemode alias with deprecation warning. - Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 18:08:24 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 18:08:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Issue12541_-_Ad?= =?utf8?q?d_UserWarning_for_unquoted_realms?= Message-ID: http://hg.python.org/cpython/rev/b82178b07e0f changeset: 76958:b82178b07e0f branch: 2.7 parent: 76952:3e10d0148f79 user: Senthil Kumaran date: Tue May 15 23:59:19 2012 +0800 summary: Issue12541 - Add UserWarning for unquoted realms files: Lib/test/test_urllib2.py | 16 +++++++++------- Lib/urllib2.py | 4 ++++ 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1106,8 +1106,8 @@ self._test_basic_auth(opener, auth_handler, "Authorization", realm, http_handler, password_manager, "http://acme.example.com/protected", - "http://acme.example.com/protected", - ) + "http://acme.example.com/protected" + ) def test_basic_auth_with_single_quoted_realm(self): self.test_basic_auth(quote_char="'") @@ -1121,11 +1121,13 @@ 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) opener.add_handler(auth_handler) opener.add_handler(http_handler) - self._test_basic_auth(opener, auth_handler, "Authorization", - realm, http_handler, password_manager, - "http://acme.example.com/protected", - "http://acme.example.com/protected", - ) + msg = "Basic Auth Realm was unquoted" + with test_support.check_warnings((msg, UserWarning)): + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected" + ) def test_proxy_basic_auth(self): diff --git a/Lib/urllib2.py b/Lib/urllib2.py --- a/Lib/urllib2.py +++ b/Lib/urllib2.py @@ -102,6 +102,7 @@ import time import urlparse import bisect +import warnings try: from cStringIO import StringIO @@ -861,6 +862,9 @@ mo = AbstractBasicAuthHandler.rx.search(authreq) if mo: scheme, quote, realm = mo.groups() + if quote not in ['"', "'"]: + warnings.warn("Basic Auth Realm was unquoted", + UserWarning, 2) if scheme.lower() == 'basic': response = self.retry_http_basic_auth(host, req, realm) if response and response.code != 401: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 18:08:25 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 18:08:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Issue12541_-_Ad?= =?utf8?q?d_UserWarning_for_unquoted_realms?= Message-ID: http://hg.python.org/cpython/rev/b5b38bda9fc4 changeset: 76959:b5b38bda9fc4 branch: 3.2 parent: 76953:bb94fec5c5ab user: Senthil Kumaran date: Tue May 15 23:59:42 2012 +0800 summary: Issue12541 - Add UserWarning for unquoted realms files: Lib/test/test_urllib2.py | 11 ++++++----- Lib/urllib/request.py | 4 ++++ 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1227,11 +1227,12 @@ 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) opener.add_handler(auth_handler) opener.add_handler(http_handler) - self._test_basic_auth(opener, auth_handler, "Authorization", - realm, http_handler, password_manager, - "http://acme.example.com/protected", - "http://acme.example.com/protected", - ) + with self.assertWarns(UserWarning): + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) def test_proxy_basic_auth(self): opener = OpenerDirector() diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -95,6 +95,7 @@ import sys import time import collections +import warnings from urllib.error import URLError, HTTPError, ContentTooShortError from urllib.parse import ( @@ -827,6 +828,9 @@ mo = AbstractBasicAuthHandler.rx.search(authreq) if mo: scheme, quote, realm = mo.groups() + if quote not in ["'", '"']: + warnings.warn("Basic Auth Realm was unquoted", + UserWarning, 2) if scheme.lower() == 'basic': response = self.retry_http_basic_auth(host, req, realm) if response and response.code != 401: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 18:08:26 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 18:08:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue12541_-_Add_UserWarning_for_unquoted_realms?= Message-ID: http://hg.python.org/cpython/rev/08fa1a47fa97 changeset: 76960:08fa1a47fa97 parent: 76955:cdea40514623 parent: 76959:b5b38bda9fc4 user: Senthil Kumaran date: Wed May 16 00:03:29 2012 +0800 summary: Issue12541 - Add UserWarning for unquoted realms files: Lib/test/test_urllib2.py | 11 ++++++----- Lib/urllib/request.py | 3 +++ 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1261,11 +1261,12 @@ 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) opener.add_handler(auth_handler) opener.add_handler(http_handler) - self._test_basic_auth(opener, auth_handler, "Authorization", - realm, http_handler, password_manager, - "http://acme.example.com/protected", - "http://acme.example.com/protected", - ) + with self.assertWarns(UserWarning): + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) def test_proxy_basic_auth(self): opener = OpenerDirector() diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -934,6 +934,9 @@ mo = AbstractBasicAuthHandler.rx.search(authreq) if mo: scheme, quote, realm = mo.groups() + if quote not in ['"',"'"]: + warnings.warn("Basic Auth Realm was unquoted", + UserWarning, 2) if scheme.lower() == 'basic': response = self.retry_http_basic_auth(host, req, realm) if response and response.code != 401: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 18:08:27 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 18:08:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/54d63a86a876 changeset: 76961:54d63a86a876 parent: 76960:08fa1a47fa97 parent: 76957:cfb870c6bb8e user: Senthil Kumaran date: Wed May 16 00:07:24 2012 +0800 summary: merge heads files: Lib/test/test_os.py | 1 + Misc/NEWS | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -695,6 +695,7 @@ # flipped: TESTFN, SUB2, SUB1, SUB11 flipped = all[0][1][0] != "SUB1" all[0][1].sort() + all[3 - 2 * flipped][-1].sort() self.assertEqual(all[0], (walk_path, ["SUB1", "SUB2"], ["tmp1"])) self.assertEqual(all[1 + flipped], (sub1_path, ["SUB11"], ["tmp2"])) self.assertEqual(all[2 + flipped], (sub11_path, [], [])) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,12 +31,12 @@ Library ------- -- Issue 14773: Fix os.fwalk() failing on dangling symlinks. +- Issue #14773: Fix os.fwalk() failing on dangling symlinks. - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic Authentation in urllib2. -- Issue 14807: move undocumented tarfile.filemode() to stat.filemode() and add +- Issue #14807: move undocumented tarfile.filemode() to stat.filemode() and add doc entry. Add tarfile.filemode alias with deprecation warning. - Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 18:08:28 2012 From: python-checkins at python.org (senthil.kumaran) Date: Tue, 15 May 2012 18:08:28 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf8?q?_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/658478c37fa5 changeset: 76962:658478c37fa5 branch: 2.7 parent: 76958:b82178b07e0f parent: 76956:458986418841 user: Senthil Kumaran date: Wed May 16 00:07:55 2012 +0800 summary: merge heads files: Misc/NEWS | 2 ++ PC/VS8.0/pyproject.vsprops | 2 +- PCbuild/pyproject.vsprops | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 4 ++-- 5 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -157,6 +157,8 @@ Build ----- +- Build against bzip2 1.0.6 and openssl 0.9.8x on Windows. + - Issue #14557: Fix extensions build on HP-UX. Patch by Adi Roiban. - Issue #14437: Fix building the _io module under Cygwin. diff --git a/PC/VS8.0/pyproject.vsprops b/PC/VS8.0/pyproject.vsprops --- a/PC/VS8.0/pyproject.vsprops +++ b/PC/VS8.0/pyproject.vsprops @@ -82,7 +82,7 @@ /> http://hg.python.org/cpython/rev/f8b8babffd4d changeset: 76963:f8b8babffd4d parent: 76961:54d63a86a876 user: Hynek Schlawack date: Tue May 15 18:40:17 2012 +0200 summary: Add two more sorts to test_os.WalkTests I've missed before files: Lib/test/test_os.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -711,6 +711,7 @@ dirs.remove('SUB1') self.assertEqual(len(all), 2) self.assertEqual(all[0], (walk_path, ["SUB2"], ["tmp1"])) + all[1][-1].sort() self.assertEqual(all[1], sub2_tree) # Walk bottom-up. @@ -721,6 +722,7 @@ # flipped: SUB2, SUB11, SUB1, TESTFN flipped = all[3][1][0] != "SUB1" all[3][1].sort() + all[2 - 2 * flipped][-1].sort() self.assertEqual(all[3], (walk_path, ["SUB1", "SUB2"], ["tmp1"])) self.assertEqual(all[flipped], (sub11_path, [], [])) self.assertEqual(all[flipped + 1], (sub1_path, ["SUB11"], ["tmp2"])) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 19:12:53 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 May 2012 19:12:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_use_Py=5Fssize=5Ft_for_ast_?= =?utf8?q?sequence_lengths?= Message-ID: http://hg.python.org/cpython/rev/c18a1a82bb07 changeset: 76964:c18a1a82bb07 parent: 76961:54d63a86a876 user: Benjamin Peterson date: Tue May 15 10:10:27 2012 -0700 summary: use Py_ssize_t for ast sequence lengths files: Parser/asdl_c.py | 4 ++-- Python/Python-ast.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py --- a/Parser/asdl_c.py +++ b/Parser/asdl_c.py @@ -784,7 +784,7 @@ static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) { - int i, n = asdl_seq_LEN(seq); + Py_ssize_t i, n = asdl_seq_LEN(seq); PyObject *result = PyList_New(n); PyObject *value; if (!result) @@ -1106,7 +1106,7 @@ # While the sequence elements are stored as void*, # ast2obj_cmpop expects an enum self.emit("{", depth) - self.emit("int i, n = asdl_seq_LEN(%s);" % value, depth+1) + self.emit("Py_ssize_t i, n = asdl_seq_LEN(%s);" % value, depth+1) self.emit("value = PyList_New(n);", depth+1) self.emit("if (!value) goto failed;", depth+1) self.emit("for(i = 0; i < n; i++)", depth+1) diff --git a/Python/Python-ast.c b/Python/Python-ast.c --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -636,7 +636,7 @@ static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) { - int i, n = asdl_seq_LEN(seq); + Py_ssize_t i, n = asdl_seq_LEN(seq); PyObject *result = PyList_New(n); PyObject *value; if (!result) @@ -2857,7 +2857,7 @@ goto failed; Py_DECREF(value); { - int i, n = asdl_seq_LEN(o->v.Compare.ops); + Py_ssize_t i, n = asdl_seq_LEN(o->v.Compare.ops); value = PyList_New(n); if (!value) goto failed; for(i = 0; i < n; i++) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 19:12:54 2012 From: python-checkins at python.org (benjamin.peterson) Date: Tue, 15 May 2012 19:12:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/ac39dc507f21 changeset: 76965:ac39dc507f21 parent: 76964:c18a1a82bb07 parent: 76963:f8b8babffd4d user: Benjamin Peterson date: Tue May 15 10:11:33 2012 -0700 summary: merge heads files: Lib/test/test_os.py | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -711,6 +711,7 @@ dirs.remove('SUB1') self.assertEqual(len(all), 2) self.assertEqual(all[0], (walk_path, ["SUB2"], ["tmp1"])) + all[1][-1].sort() self.assertEqual(all[1], sub2_tree) # Walk bottom-up. @@ -721,6 +722,7 @@ # flipped: SUB2, SUB11, SUB1, TESTFN flipped = all[3][1][0] != "SUB1" all[3][1].sort() + all[2 - 2 * flipped][-1].sort() self.assertEqual(all[3], (walk_path, ["SUB1", "SUB2"], ["tmp1"])) self.assertEqual(all[flipped], (sub11_path, [], [])) self.assertEqual(all[flipped + 1], (sub1_path, ["SUB11"], ["tmp2"])) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 19:55:40 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 19:55:40 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Force_recheckou?= =?utf8?q?t_of_openssl=2E?= Message-ID: http://hg.python.org/cpython/rev/59b381832fc5 changeset: 76966:59b381832fc5 branch: 2.7 parent: 76962:658478c37fa5 user: Martin v. L?wis date: Tue May 15 19:55:33 2012 +0200 summary: Force recheckout of openssl. files: Tools/buildbot/external-common.bat | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -15,7 +15,7 @@ @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1 @rem if exist db-4.4.20 rd /s/q db-4.4.20 @rem if exist db-4.7.25.0 rd /s/q db-4.7.25.0 - at rem if exist openssl-0.9.8x rd /s/q openssl-0.9.8x +if exist openssl-0.9.8x rd /s/q openssl-0.9.8x @rem if exist sqlite-3.6.21 rd /s/q sqlite-3.6.21 @rem bzip -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 20:05:00 2012 From: python-checkins at python.org (martin.v.loewis) Date: Tue, 15 May 2012 20:05:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Stop_including_gzio_in_the_?= =?utf8?q?build=3B_it=27s_not_used=2E?= Message-ID: http://hg.python.org/cpython/rev/4e9eabe3e24f changeset: 76967:4e9eabe3e24f parent: 76965:ac39dc507f21 user: Martin v. L?wis date: Tue May 15 20:04:25 2012 +0200 summary: Stop including gzio in the build; it's not used. files: PCbuild/pythoncore.vcxproj | 6 ------ PCbuild/pythoncore.vcxproj.filters | 3 --- 2 files changed, 0 insertions(+), 9 deletions(-) diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -564,12 +564,6 @@ - - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -578,9 +578,6 @@ Modules\zlib - - Modules\zlib - Modules\zlib -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 20:08:52 2012 From: python-checkins at python.org (eric.smith) Date: Tue, 15 May 2012 20:08:52 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Change_case_to_match_the_rest_?= =?utf8?q?of_the_PEP=2E?= Message-ID: http://hg.python.org/peps/rev/5e5d04d791af changeset: 4381:5e5d04d791af user: Eric V. Smith date: Tue May 15 14:08:43 2012 -0400 summary: Change case to match the rest of the PEP. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -168,7 +168,7 @@ sys.path = sys.path + ['new-dir'] -Impact on Import Finders and Loaders +Impact on import finders and loaders ------------------------------------ PEP 302 defines "finders" that are called to search path elements. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Tue May 15 21:02:50 2012 From: python-checkins at python.org (charles-francois.natali) Date: Tue, 15 May 2012 21:02:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_versionadded_for_hmac?= =?utf8?b?LnNlY3VyZV9jb21wYXJlKCku?= Message-ID: http://hg.python.org/cpython/rev/59e93e1b8116 changeset: 76968:59e93e1b8116 user: Charles-Fran?ois Natali date: Tue May 15 21:00:32 2012 +0200 summary: Add versionadded for hmac.secure_compare(). files: Doc/library/hmac.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -83,6 +83,7 @@ contents of the inputs via a timing attack, it does leak the length of the inputs. However, this generally is not a security risk. + .. versionadded:: 3.3 .. seealso:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 22:21:07 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Tue, 15 May 2012 22:21:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314807=3A_fix_bb_failure_?= =?utf8?q?due_to_symlink_test_relying_on_hard-coded_permissions?= Message-ID: http://hg.python.org/cpython/rev/539fbd6e58f6 changeset: 76969:539fbd6e58f6 parent: 76950:d4c590cee68b user: Giampaolo Rodola' date: Tue May 15 22:20:10 2012 +0200 summary: #14807: fix bb failure due to symlink test relying on hard-coded permissions files: Lib/test/test_stat.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py --- a/Lib/test/test_stat.py +++ b/Lib/test/test_stat.py @@ -40,7 +40,7 @@ @unittest.skipUnless(hasattr(os, 'symlink'), 'os.symlink not available') def test_link(self): os.symlink(os.getcwd(), TESTFN) - self.assertEqual(get_mode(), 'lrwxrwxrwx') + self.assertEqual(get_mode()[0], 'l') @unittest.skipUnless(hasattr(os, 'mkfifo'), 'os.mkfifo not available') def test_fifo(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 22:21:08 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Tue, 15 May 2012 22:21:08 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/003204f34cc3 changeset: 76970:003204f34cc3 parent: 76969:539fbd6e58f6 parent: 76968:59e93e1b8116 user: Giampaolo Rodola' date: Tue May 15 22:21:01 2012 +0200 summary: merge heads files: Doc/library/hmac.rst | 1 + Lib/os.py | 24 ++++++++++++----- Lib/test/test_os.py | 9 ++++++- Lib/test/test_urllib2.py | 16 ++++++++++++ Lib/urllib/request.py | 5 +++- Misc/NEWS | 7 ++++- PCbuild/pythoncore.vcxproj | 6 ---- PCbuild/pythoncore.vcxproj.filters | 3 -- Parser/asdl_c.py | 4 +- Python/Python-ast.c | 4 +- 10 files changed, 56 insertions(+), 23 deletions(-) diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst --- a/Doc/library/hmac.rst +++ b/Doc/library/hmac.rst @@ -83,6 +83,7 @@ contents of the inputs via a timing attack, it does leak the length of the inputs. However, this generally is not a security risk. + .. versionadded:: 3.3 .. seealso:: diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -353,13 +353,23 @@ names = flistdir(topfd) dirs, nondirs = [], [] for name in names: - # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with - # walk() which reports symlinks to directories as directories. We do - # however check for symlinks before recursing into a subdirectory. - if st.S_ISDIR(fstatat(topfd, name).st_mode): - dirs.append(name) - else: - nondirs.append(name) + try: + # Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with + # walk() which reports symlinks to directories as directories. + # We do however check for symlinks before recursing into + # a subdirectory. + if st.S_ISDIR(fstatat(topfd, name).st_mode): + dirs.append(name) + else: + nondirs.append(name) + except FileNotFoundError: + try: + # Add dangling symlinks, ignore disappeared files + if st.S_ISLNK(fstatat(topfd, name, AT_SYMLINK_NOFOLLOW) + .st_mode): + nondirs.append(name) + except FileNotFoundError: + continue if topdown: yield toppath, dirs, nondirs, topfd diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -651,6 +651,7 @@ # SUB2/ a file kid and a dirsymlink kid # tmp3 # link/ a symlink to TESTFN.2 + # broken_link # TEST2/ # tmp4 a lone file walk_path = join(support.TESTFN, "TEST1") @@ -663,6 +664,8 @@ link_path = join(sub2_path, "link") t2_path = join(support.TESTFN, "TEST2") tmp4_path = join(support.TESTFN, "TEST2", "tmp4") + link_path = join(sub2_path, "link") + broken_link_path = join(sub2_path, "broken_link") # Create stuff. os.makedirs(sub11_path) @@ -679,7 +682,8 @@ else: symlink_to_dir = os.symlink symlink_to_dir(os.path.abspath(t2_path), link_path) - sub2_tree = (sub2_path, ["link"], ["tmp3"]) + symlink_to_dir('broken', broken_link_path) + sub2_tree = (sub2_path, ["link"], ["broken_link", "tmp3"]) else: sub2_tree = (sub2_path, [], ["tmp3"]) @@ -691,6 +695,7 @@ # flipped: TESTFN, SUB2, SUB1, SUB11 flipped = all[0][1][0] != "SUB1" all[0][1].sort() + all[3 - 2 * flipped][-1].sort() self.assertEqual(all[0], (walk_path, ["SUB1", "SUB2"], ["tmp1"])) self.assertEqual(all[1 + flipped], (sub1_path, ["SUB11"], ["tmp2"])) self.assertEqual(all[2 + flipped], (sub11_path, [], [])) @@ -706,6 +711,7 @@ dirs.remove('SUB1') self.assertEqual(len(all), 2) self.assertEqual(all[0], (walk_path, ["SUB2"], ["tmp1"])) + all[1][-1].sort() self.assertEqual(all[1], sub2_tree) # Walk bottom-up. @@ -716,6 +722,7 @@ # flipped: SUB2, SUB11, SUB1, TESTFN flipped = all[3][1][0] != "SUB1" all[3][1].sort() + all[2 - 2 * flipped][-1].sort() self.assertEqual(all[3], (walk_path, ["SUB1", "SUB2"], ["tmp1"])) self.assertEqual(all[flipped], (sub11_path, [], [])) self.assertEqual(all[flipped + 1], (sub1_path, ["SUB11"], ["tmp2"])) diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1252,6 +1252,22 @@ def test_basic_auth_with_single_quoted_realm(self): self.test_basic_auth(quote_char="'") + def test_basic_auth_with_unquoted_realm(self): + opener = OpenerDirector() + password_manager = MockPasswordManager() + auth_handler = urllib.request.HTTPBasicAuthHandler(password_manager) + realm = "ACME Widget Store" + http_handler = MockHTTPHandler( + 401, 'WWW-Authenticate: Basic realm=%s\r\n\r\n' % realm) + opener.add_handler(auth_handler) + opener.add_handler(http_handler) + with self.assertWarns(UserWarning): + self._test_basic_auth(opener, auth_handler, "Authorization", + realm, http_handler, password_manager, + "http://acme.example.com/protected", + "http://acme.example.com/protected", + ) + def test_proxy_basic_auth(self): opener = OpenerDirector() ph = urllib.request.ProxyHandler(dict(http="proxy.example.com:3128")) diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -895,7 +895,7 @@ # allow for double- and single-quoted realm values # (single quotes are a violation of the RFC, but appear in the wild) rx = re.compile('(?:.*,)*[ \t]*([^ \t]+)[ \t]+' - 'realm=(["\'])(.*?)\\2', re.I) + 'realm=(["\']?)([^"\']*)\\2', re.I) # XXX could pre-emptively send auth info already accepted (RFC 2617, # end of section 2, and section 1.2 immediately after "credentials" @@ -934,6 +934,9 @@ mo = AbstractBasicAuthHandler.rx.search(authreq) if mo: scheme, quote, realm = mo.groups() + if quote not in ['"',"'"]: + warnings.warn("Basic Auth Realm was unquoted", + UserWarning, 2) if scheme.lower() == 'basic': response = self.retry_http_basic_auth(host, req, realm) if response and response.code != 401: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -31,7 +31,12 @@ Library ------- -- Issue 14807: move undocumented tarfile.filemode() to stat.filemode() and add +- Issue #14773: Fix os.fwalk() failing on dangling symlinks. + +- Issue #12541: Be lenient with quotes around Realm field of HTTP Basic + Authentation in urllib2. + +- Issue #14807: move undocumented tarfile.filemode() to stat.filemode() and add doc entry. Add tarfile.filemode alias with deprecation warning. - Issue #13815: TarFile.extractfile() now returns io.BufferedReader objects. diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -564,12 +564,6 @@ - - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - _CRT_SECURE_NO_DEPRECATE;%(PreprocessorDefinitions) - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -578,9 +578,6 @@ Modules\zlib - - Modules\zlib - Modules\zlib diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py --- a/Parser/asdl_c.py +++ b/Parser/asdl_c.py @@ -784,7 +784,7 @@ static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) { - int i, n = asdl_seq_LEN(seq); + Py_ssize_t i, n = asdl_seq_LEN(seq); PyObject *result = PyList_New(n); PyObject *value; if (!result) @@ -1106,7 +1106,7 @@ # While the sequence elements are stored as void*, # ast2obj_cmpop expects an enum self.emit("{", depth) - self.emit("int i, n = asdl_seq_LEN(%s);" % value, depth+1) + self.emit("Py_ssize_t i, n = asdl_seq_LEN(%s);" % value, depth+1) self.emit("value = PyList_New(n);", depth+1) self.emit("if (!value) goto failed;", depth+1) self.emit("for(i = 0; i < n; i++)", depth+1) diff --git a/Python/Python-ast.c b/Python/Python-ast.c --- a/Python/Python-ast.c +++ b/Python/Python-ast.c @@ -636,7 +636,7 @@ static PyObject* ast2obj_list(asdl_seq *seq, PyObject* (*func)(void*)) { - int i, n = asdl_seq_LEN(seq); + Py_ssize_t i, n = asdl_seq_LEN(seq); PyObject *result = PyList_New(n); PyObject *value; if (!result) @@ -2857,7 +2857,7 @@ goto failed; Py_DECREF(value); { - int i, n = asdl_seq_LEN(o->v.Compare.ops); + Py_ssize_t i, n = asdl_seq_LEN(o->v.Compare.ops); value = PyList_New(n); if (!value) goto failed; for(i = 0; i < n; i++) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 15 23:50:52 2012 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 15 May 2012 23:50:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314624=3A_UTF-16_de?= =?utf8?q?coding_is_now_3x_to_4x_faster_on_various_inputs=2E?= Message-ID: http://hg.python.org/cpython/rev/cdcc816dea85 changeset: 76971:cdcc816dea85 user: Antoine Pitrou date: Tue May 15 23:48:04 2012 +0200 summary: Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Patch by Serhiy Storchaka. files: Misc/NEWS | 3 + Objects/stringlib/codecs.h | 149 ++++++++++++- Objects/unicodeobject.c | 297 +++++++----------------- 3 files changed, 240 insertions(+), 209 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. + Patch by Serhiy Storchaka. + - asdl_seq and asdl_int_seq are now Py_ssize_t sized. - Issue #14133 (PEP 415): Implement suppression of __context__ display with an diff --git a/Objects/stringlib/codecs.h b/Objects/stringlib/codecs.h --- a/Objects/stringlib/codecs.h +++ b/Objects/stringlib/codecs.h @@ -215,7 +215,6 @@ goto Return; } -#undef LONG_PTR_MASK #undef ASCII_CHAR_MASK @@ -415,4 +414,152 @@ #undef MAX_SHORT_UNICHARS } +/* The pattern for constructing UCS2-repeated masks. */ +#if SIZEOF_LONG == 8 +# define UCS2_REPEAT_MASK 0x0001000100010001ul +#elif SIZEOF_LONG == 4 +# define UCS2_REPEAT_MASK 0x00010001ul +#else +# error C 'long' size should be either 4 or 8! +#endif + +/* The mask for fast checking. */ +#if STRINGLIB_SIZEOF_CHAR == 1 +/* The mask for fast checking of whether a C 'long' contains a + non-ASCII or non-Latin1 UTF16-encoded characters. */ +# define FAST_CHAR_MASK (UCS2_REPEAT_MASK * (0xFFFFu & ~STRINGLIB_MAX_CHAR)) +#else +/* The mask for fast checking of whether a C 'long' may contain + UTF16-encoded surrogate characters. This is an efficient heuristic, + assuming that non-surrogate characters with a code point >= 0x8000 are + rare in most input. +*/ +# define FAST_CHAR_MASK (UCS2_REPEAT_MASK * 0x8000u) +#endif +/* The mask for fast byte-swapping. */ +#define STRIPPED_MASK (UCS2_REPEAT_MASK * 0x00FFu) +/* Swap bytes. */ +#define SWAB(value) ((((value) >> 8) & STRIPPED_MASK) | \ + (((value) & STRIPPED_MASK) << 8)) + +Py_LOCAL_INLINE(Py_UCS4) +STRINGLIB(utf16_decode)(const unsigned char **inptr, const unsigned char *e, + STRINGLIB_CHAR *dest, Py_ssize_t *outpos, + int native_ordering) +{ + Py_UCS4 ch; + const unsigned char *aligned_end = + (const unsigned char *) ((size_t) e & ~LONG_PTR_MASK); + const unsigned char *q = *inptr; + STRINGLIB_CHAR *p = dest + *outpos; + /* Offsets from q for retrieving byte pairs in the right order. */ +#ifdef BYTEORDER_IS_LITTLE_ENDIAN + int ihi = !!native_ordering, ilo = !native_ordering; +#else + int ihi = !native_ordering, ilo = !!native_ordering; +#endif + --e; + + while (q < e) { + Py_UCS4 ch2; + /* First check for possible aligned read of a C 'long'. Unaligned + reads are more expensive, better to defer to another iteration. */ + if (!((size_t) q & LONG_PTR_MASK)) { + /* Fast path for runs of in-range non-surrogate chars. */ + register const unsigned char *_q = q; + while (_q < aligned_end) { + unsigned long block = * (unsigned long *) _q; + if (native_ordering) { + /* Can use buffer directly */ + if (block & FAST_CHAR_MASK) + break; + } + else { + /* Need to byte-swap */ + if (block & SWAB(FAST_CHAR_MASK)) + break; +#if STRINGLIB_SIZEOF_CHAR == 1 + block >>= 8; +#else + block = SWAB(block); +#endif + } +#ifdef BYTEORDER_IS_LITTLE_ENDIAN +# if SIZEOF_LONG == 4 + p[0] = (STRINGLIB_CHAR)(block & 0xFFFFu); + p[1] = (STRINGLIB_CHAR)(block >> 16); +# elif SIZEOF_LONG == 8 + p[0] = (STRINGLIB_CHAR)(block & 0xFFFFu); + p[1] = (STRINGLIB_CHAR)((block >> 16) & 0xFFFFu); + p[2] = (STRINGLIB_CHAR)((block >> 32) & 0xFFFFu); + p[3] = (STRINGLIB_CHAR)(block >> 48); +# endif +#else +# if SIZEOF_LONG == 4 + p[0] = (STRINGLIB_CHAR)(block >> 16); + p[1] = (STRINGLIB_CHAR)(block & 0xFFFFu); +# elif SIZEOF_LONG == 8 + p[0] = (STRINGLIB_CHAR)(block >> 48); + p[1] = (STRINGLIB_CHAR)((block >> 32) & 0xFFFFu); + p[2] = (STRINGLIB_CHAR)((block >> 16) & 0xFFFFu); + p[3] = (STRINGLIB_CHAR)(block & 0xFFFFu); +# endif +#endif + _q += SIZEOF_LONG; + p += SIZEOF_LONG / 2; + } + q = _q; + if (q >= e) + break; + } + + ch = (q[ihi] << 8) | q[ilo]; + q += 2; + if (!Py_UNICODE_IS_SURROGATE(ch)) { +#if STRINGLIB_SIZEOF_CHAR < 2 + if (ch > STRINGLIB_MAX_CHAR) + /* Out-of-range */ + goto Return; +#endif + *p++ = (STRINGLIB_CHAR)ch; + continue; + } + + /* UTF-16 code pair: */ + if (q >= e) + goto UnexpectedEnd; + if (!Py_UNICODE_IS_HIGH_SURROGATE(ch)) + goto IllegalEncoding; + ch2 = (q[ihi] << 8) | q[ilo]; + q += 2; + if (!Py_UNICODE_IS_LOW_SURROGATE(ch2)) + goto IllegalSurrogate; + ch = Py_UNICODE_JOIN_SURROGATES(ch, ch2); +#if STRINGLIB_SIZEOF_CHAR < 4 + /* Out-of-range */ + goto Return; +#else + *p++ = (STRINGLIB_CHAR)ch; +#endif + } + ch = 0; +Return: + *inptr = q; + *outpos = p - dest; + return ch; +UnexpectedEnd: + ch = 1; + goto Return; +IllegalEncoding: + ch = 2; + goto Return; +IllegalSurrogate: + ch = 3; + goto Return; +} +#undef UCS2_REPEAT_MASK +#undef FAST_CHAR_MASK +#undef STRIPPED_MASK +#undef SWAB +#undef LONG_PTR_MASK #endif /* STRINGLIB_IS_UNICODE */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5195,25 +5195,6 @@ return PyUnicode_DecodeUTF16Stateful(s, size, errors, byteorder, NULL); } -/* Two masks for fast checking of whether a C 'long' may contain - UTF16-encoded surrogate characters. This is an efficient heuristic, - assuming that non-surrogate characters with a code point >= 0x8000 are - rare in most input. - FAST_CHAR_MASK is used when the input is in native byte ordering, - SWAPPED_FAST_CHAR_MASK when the input is in byteswapped ordering. -*/ -#if (SIZEOF_LONG == 8) -# define FAST_CHAR_MASK 0x8000800080008000L -# define SWAPPED_FAST_CHAR_MASK 0x0080008000800080L -# define STRIPPED_MASK 0x00FF00FF00FF00FFL -#elif (SIZEOF_LONG == 4) -# define FAST_CHAR_MASK 0x80008000L -# define SWAPPED_FAST_CHAR_MASK 0x00800080L -# define STRIPPED_MASK 0x00FF00FFL -#else -# error C 'long' size should be either 4 or 8! -#endif - PyObject * PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, @@ -5226,30 +5207,15 @@ Py_ssize_t endinpos; Py_ssize_t outpos; PyObject *unicode; - const unsigned char *q, *e, *aligned_end; + const unsigned char *q, *e; int bo = 0; /* assume native ordering by default */ - int native_ordering = 0; + int native_ordering; const char *errmsg = ""; - /* Offsets from q for retrieving byte pairs in the right order. */ -#ifdef BYTEORDER_IS_LITTLE_ENDIAN - int ihi = 1, ilo = 0; -#else - int ihi = 0, ilo = 1; -#endif PyObject *errorHandler = NULL; PyObject *exc = NULL; - /* Note: size will always be longer than the resulting Unicode - character count */ - unicode = PyUnicode_New(size, 127); - if (!unicode) - return NULL; - if (size == 0) - return unicode; - outpos = 0; - q = (unsigned char *)s; - e = q + size - 1; + e = q + size; if (byteorder) bo = *byteorder; @@ -5258,155 +5224,98 @@ byte order setting accordingly. In native mode, the leading BOM mark is skipped, in all other modes, it is copied to the output stream as-is (giving a ZWNBSP character). */ - if (bo == 0) { - if (size >= 2) { - const Py_UCS4 bom = (q[ihi] << 8) | q[ilo]; + if (bo == 0 && size >= 2) { + const Py_UCS4 bom = (q[1] << 8) | q[0]; + if (bom == 0xFEFF) { + q += 2; + bo = -1; + } + else if (bom == 0xFFFE) { + q += 2; + bo = 1; + } + if (byteorder) + *byteorder = bo; + } + + if (q == e) { + if (consumed) + *consumed = size; + Py_INCREF(unicode_empty); + return unicode_empty; + } + #ifdef BYTEORDER_IS_LITTLE_ENDIAN - if (bom == 0xFEFF) { - q += 2; - bo = -1; - } - else if (bom == 0xFFFE) { - q += 2; - bo = 1; - } + native_ordering = bo <= 0; #else - if (bom == 0xFEFF) { - q += 2; - bo = 1; - } - else if (bom == 0xFFFE) { - q += 2; - bo = -1; - } -#endif - } - } - - if (bo == -1) { - /* force LE */ - ihi = 1; - ilo = 0; - } - else if (bo == 1) { - /* force BE */ - ihi = 0; - ilo = 1; - } -#ifdef BYTEORDER_IS_LITTLE_ENDIAN - native_ordering = ilo < ihi; -#else - native_ordering = ilo > ihi; -#endif - - aligned_end = (const unsigned char *) ((size_t) e & ~LONG_PTR_MASK); - while (q < e) { - Py_UCS4 ch; - /* First check for possible aligned read of a C 'long'. Unaligned - reads are more expensive, better to defer to another iteration. */ - if (!((size_t) q & LONG_PTR_MASK)) { - /* Fast path for runs of non-surrogate chars. */ - register const unsigned char *_q = q; + native_ordering = bo >= 0; +#endif + + /* Note: size will always be longer than the resulting Unicode + character count */ + unicode = PyUnicode_New((e - q + 1) / 2, 127); + if (!unicode) + return NULL; + + outpos = 0; + while (1) { + Py_UCS4 ch = 0; + if (e - q >= 2) { int kind = PyUnicode_KIND(unicode); - void *data = PyUnicode_DATA(unicode); - while (_q < aligned_end) { - unsigned long block = * (unsigned long *) _q; - Py_UCS4 maxch; - if (native_ordering) { - /* Can use buffer directly */ - if (block & FAST_CHAR_MASK) - break; - } - else { - /* Need to byte-swap */ - if (block & SWAPPED_FAST_CHAR_MASK) - break; - block = ((block >> 8) & STRIPPED_MASK) | - ((block & STRIPPED_MASK) << 8); - } - maxch = (Py_UCS2)(block & 0xFFFF); -#if SIZEOF_LONG == 8 - ch = (Py_UCS2)((block >> 16) & 0xFFFF); - maxch = MAX_MAXCHAR(maxch, ch); - ch = (Py_UCS2)((block >> 32) & 0xFFFF); - maxch = MAX_MAXCHAR(maxch, ch); - ch = (Py_UCS2)(block >> 48); - maxch = MAX_MAXCHAR(maxch, ch); -#else - ch = (Py_UCS2)(block >> 16); - maxch = MAX_MAXCHAR(maxch, ch); -#endif - if (maxch > PyUnicode_MAX_CHAR_VALUE(unicode)) { - if (unicode_widen(&unicode, outpos, maxch) < 0) - goto onError; - kind = PyUnicode_KIND(unicode); - data = PyUnicode_DATA(unicode); - } -#ifdef BYTEORDER_IS_LITTLE_ENDIAN - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block & 0xFFFF)); -#if SIZEOF_LONG == 8 - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 16) & 0xFFFF)); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 32) & 0xFFFF)); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 48))); -#else - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block >> 16)); -#endif -#else -#if SIZEOF_LONG == 8 - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 48))); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 32) & 0xFFFF)); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 16) & 0xFFFF)); -#else - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block >> 16)); -#endif - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block & 0xFFFF)); -#endif - _q += SIZEOF_LONG; - } - q = _q; - if (q >= e) - break; - } - ch = (q[ihi] << 8) | q[ilo]; - - q += 2; - - if (!Py_UNICODE_IS_SURROGATE(ch)) { + if (kind == PyUnicode_1BYTE_KIND) { + if (PyUnicode_IS_ASCII(unicode)) + ch = asciilib_utf16_decode(&q, e, + PyUnicode_1BYTE_DATA(unicode), &outpos, + native_ordering); + else + ch = ucs1lib_utf16_decode(&q, e, + PyUnicode_1BYTE_DATA(unicode), &outpos, + native_ordering); + } else if (kind == PyUnicode_2BYTE_KIND) { + ch = ucs2lib_utf16_decode(&q, e, + PyUnicode_2BYTE_DATA(unicode), &outpos, + native_ordering); + } else { + assert(kind == PyUnicode_4BYTE_KIND); + ch = ucs4lib_utf16_decode(&q, e, + PyUnicode_4BYTE_DATA(unicode), &outpos, + native_ordering); + } + } + + switch (ch) + { + case 0: + /* remaining byte at the end? (size should be even) */ + if (q == e || consumed) + goto End; + errmsg = "truncated data"; + startinpos = ((const char *)q) - starts; + endinpos = ((const char *)e) - starts; + break; + /* The remaining input chars are ignored if the callback + chooses to skip the input */ + case 1: + errmsg = "unexpected end of data"; + startinpos = ((const char *)q) - 2 - starts; + endinpos = ((const char *)e) - starts; + break; + case 2: + errmsg = "illegal encoding"; + startinpos = ((const char *)q) - 2 - starts; + endinpos = startinpos + 2; + break; + case 3: + errmsg = "illegal UTF-16 surrogate"; + startinpos = ((const char *)q) - 4 - starts; + endinpos = startinpos + 2; + break; + default: if (unicode_putchar(&unicode, &outpos, ch) < 0) goto onError; continue; } - /* UTF-16 code pair: */ - if (q > e) { - errmsg = "unexpected end of data"; - startinpos = (((const char *)q) - 2) - starts; - endinpos = ((const char *)e) + 1 - starts; - goto utf16Error; - } - if (Py_UNICODE_IS_HIGH_SURROGATE(ch)) { - Py_UCS4 ch2 = (q[ihi] << 8) | q[ilo]; - q += 2; - if (Py_UNICODE_IS_LOW_SURROGATE(ch2)) { - if (unicode_putchar(&unicode, &outpos, - Py_UNICODE_JOIN_SURROGATES(ch, ch2)) < 0) - goto onError; - continue; - } - else { - errmsg = "illegal UTF-16 surrogate"; - startinpos = (((const char *)q)-4)-starts; - endinpos = startinpos+2; - goto utf16Error; - } - - } - errmsg = "illegal encoding"; - startinpos = (((const char *)q)-2)-starts; - endinpos = startinpos+2; - /* Fall through to report the error */ - - utf16Error: if (unicode_decode_call_errorhandler( errors, &errorHandler, @@ -5421,33 +5330,8 @@ &outpos)) goto onError; } - /* remaining byte at the end? (size should be even) */ - if (e == q) { - if (!consumed) { - errmsg = "truncated data"; - startinpos = ((const char *)q) - starts; - endinpos = ((const char *)e) + 1 - starts; - if (unicode_decode_call_errorhandler( - errors, - &errorHandler, - "utf16", errmsg, - &starts, - (const char **)&e, - &startinpos, - &endinpos, - &exc, - (const char **)&q, - &unicode, - &outpos)) - goto onError; - /* The remaining input chars are ignored if the callback - chooses to skip the input */ - } - } - - if (byteorder) - *byteorder = bo; - + +End: if (consumed) *consumed = (const char *)q-starts; @@ -5466,9 +5350,6 @@ return NULL; } -#undef FAST_CHAR_MASK -#undef SWAPPED_FAST_CHAR_MASK - PyObject * _PyUnicode_EncodeUTF16(PyObject *str, const char *errors, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 02:46:17 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 02:46:17 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0ODE3?= =?utf8?q?=3A_Add_rudimentary_tests_for_pkgutil=2Eextend=5Fpath=2E?= Message-ID: http://hg.python.org/cpython/rev/48cb6b67d306 changeset: 76972:48cb6b67d306 branch: 3.2 parent: 76959:b5b38bda9fc4 user: Eric V. Smith date: Tue May 15 20:44:06 2012 -0400 summary: Issue #14817: Add rudimentary tests for pkgutil.extend_path. files: Lib/test/test_pkgutil.py | 48 +++++++++++++++++++++++++++- 1 files changed, 47 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -137,8 +137,54 @@ self.assertEqual(foo.loads, 1) del sys.modules['foo'] + +class ExtendPathTests(unittest.TestCase): + def create_init(self, pkgname): + dirname = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, dirname) + sys.path.insert(0, dirname) + + pkgdir = os.path.join(dirname, pkgname) + os.mkdir(pkgdir) + with open(os.path.join(pkgdir, '__init__.py'), 'w') as fl: + fl.write('from pkgutil import extend_path\n__path__ = extend_path(__path__, __name__)\n') + + return dirname + + def create_submodule(self, dirname, pkgname, submodule_name, value): + module_name = os.path.join(dirname, pkgname, submodule_name + '.py') + with open(module_name, 'w') as fl: + print('value={}'.format(value), file=fl) + + def setUp(self): + # Create 2 directories on sys.path + self.pkgname = 'foo' + self.dirname_0 = self.create_init(self.pkgname) + self.dirname_1 = self.create_init(self.pkgname) + + def tearDown(self): + del sys.path[0] + del sys.path[0] + + def test_simple(self): + self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) + self.create_submodule(self.dirname_1, self.pkgname, 'baz', 1) + import foo.bar + import foo.baz + # Ensure we read the expected values + self.assertEqual(foo.bar.value, 0) + self.assertEqual(foo.baz.value, 1) + + # Ensure the path is set up correctly + self.assertEqual(sorted(foo.__path__), + sorted([os.path.join(self.dirname_0, self.pkgname), + os.path.join(self.dirname_1, self.pkgname)])) + + # XXX: test .pkg files + + def test_main(): - run_unittest(PkgutilTests, PkgutilPEP302Tests) + run_unittest(PkgutilTests, PkgutilPEP302Tests, ExtendPathTests) # this is necessary if test is run repeated (like when finding leaks) import zipimport zipimport._zip_directory_cache.clear() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 02:46:18 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 02:46:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/d9bf364d75e2 changeset: 76973:d9bf364d75e2 parent: 76971:cdcc816dea85 parent: 76972:48cb6b67d306 user: Eric V. Smith date: Tue May 15 20:46:06 2012 -0400 summary: Merge from 3.2. files: Lib/test/test_pkgutil.py | 48 +++++++++++++++++++++++++++- 1 files changed, 47 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -137,8 +137,54 @@ self.assertEqual(foo.loads, 1) del sys.modules['foo'] + +class ExtendPathTests(unittest.TestCase): + def create_init(self, pkgname): + dirname = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, dirname) + sys.path.insert(0, dirname) + + pkgdir = os.path.join(dirname, pkgname) + os.mkdir(pkgdir) + with open(os.path.join(pkgdir, '__init__.py'), 'w') as fl: + fl.write('from pkgutil import extend_path\n__path__ = extend_path(__path__, __name__)\n') + + return dirname + + def create_submodule(self, dirname, pkgname, submodule_name, value): + module_name = os.path.join(dirname, pkgname, submodule_name + '.py') + with open(module_name, 'w') as fl: + print('value={}'.format(value), file=fl) + + def setUp(self): + # Create 2 directories on sys.path + self.pkgname = 'foo' + self.dirname_0 = self.create_init(self.pkgname) + self.dirname_1 = self.create_init(self.pkgname) + + def tearDown(self): + del sys.path[0] + del sys.path[0] + + def test_simple(self): + self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) + self.create_submodule(self.dirname_1, self.pkgname, 'baz', 1) + import foo.bar + import foo.baz + # Ensure we read the expected values + self.assertEqual(foo.bar.value, 0) + self.assertEqual(foo.baz.value, 1) + + # Ensure the path is set up correctly + self.assertEqual(sorted(foo.__path__), + sorted([os.path.join(self.dirname_0, self.pkgname), + os.path.join(self.dirname_1, self.pkgname)])) + + # XXX: test .pkg files + + def test_main(): - run_unittest(PkgutilTests, PkgutilPEP302Tests) + run_unittest(PkgutilTests, PkgutilPEP302Tests, ExtendPathTests) # this is necessary if test is run repeated (like when finding leaks) import zipimport zipimport._zip_directory_cache.clear() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 03:14:05 2012 From: python-checkins at python.org (ned.deily) Date: Wed, 16 May 2012 03:14:05 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0Nzc3?= =?utf8?q?=3A_In_an_X11_windowing_environment=2C_tkinter_may_return?= Message-ID: http://hg.python.org/cpython/rev/f70fa654f70e changeset: 76974:f70fa654f70e branch: 2.7 parent: 76966:59b381832fc5 user: Ned Deily date: Tue May 15 18:05:57 2012 -0700 summary: Issue #14777: In an X11 windowing environment, tkinter may return undecoded UTF-8 bytes as a string when accessing the Tk clipboard. Modify clipboad_get() to first request type UTF8_STRING when no specific type is requested in an X11 windowing environment, falling back to the current default type STRING if that fails. Original patch by Thomas Kluyver. files: Lib/lib-tk/Tkinter.py | 28 ++++++++++++++++++++++++++-- Misc/NEWS | 6 ++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/Lib/lib-tk/Tkinter.py b/Lib/lib-tk/Tkinter.py --- a/Lib/lib-tk/Tkinter.py +++ b/Lib/lib-tk/Tkinter.py @@ -534,12 +534,19 @@ The type keyword specifies the form in which the data is to be returned and should be an atom name such as STRING - or FILE_NAME. Type defaults to STRING. + or FILE_NAME. Type defaults to STRING, except on X11, where the default + is to try UTF8_STRING and fall back to STRING. This command is equivalent to: selection_get(CLIPBOARD) """ + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('clipboard', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('clipboard', 'get') + self._options(kw)) def clipboard_clear(self, **kw): @@ -621,8 +628,16 @@ A keyword parameter selection specifies the name of the selection and defaults to PRIMARY. A keyword parameter displayof specifies a widget on the display - to use.""" + to use. A keyword parameter type specifies the form of data to be + fetched, defaulting to STRING except on X11, where UTF8_STRING is tried + before STRING.""" if 'displayof' not in kw: kw['displayof'] = self._w + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('selection', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('selection', 'get') + self._options(kw)) def selection_handle(self, command, **kw): """Specify a function COMMAND to call if the X @@ -1037,6 +1052,15 @@ if displayof is None: return ('-displayof', self._w) return () + @property + def _windowingsystem(self): + """Internal function.""" + try: + return self._root()._windowingsystem_cached + except AttributeError: + ws = self._root()._windowingsystem_cached = \ + self.tk.call('tk', 'windowingsystem') + return ws def _options(self, cnf, kw = None): """Internal function.""" if kw: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,12 @@ Library ------- +- Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when + accessing the Tk clipboard. Modify clipboad_get() to first request type + UTF8_STRING when no specific type is requested in an X11 windowing + environment, falling back to the current default type STRING if that fails. + Original patch by Thomas Kluyver. + - Issue #12541: Be lenient with quotes around Realm field with HTTP Basic Authentation in urllib2. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 03:14:06 2012 From: python-checkins at python.org (ned.deily) Date: Wed, 16 May 2012 03:14:06 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0Nzc3?= =?utf8?q?=3A_In_an_X11_windowing_environment=2C_tkinter_may_return?= Message-ID: http://hg.python.org/cpython/rev/41382250e5e1 changeset: 76975:41382250e5e1 branch: 3.2 parent: 76972:48cb6b67d306 user: Ned Deily date: Tue May 15 18:08:11 2012 -0700 summary: Issue #14777: In an X11 windowing environment, tkinter may return undecoded UTF-8 bytes as a string when accessing the Tk clipboard. Modify clipboad_get() to first request type UTF8_STRING when no specific type is requested in an X11 windowing environment, falling back to the current default type STRING if that fails. Original patch by Thomas Kluyver. files: Lib/tkinter/__init__.py | 28 ++++++++++++++++++++++++++-- Misc/NEWS | 6 ++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -526,12 +526,19 @@ The type keyword specifies the form in which the data is to be returned and should be an atom name such as STRING - or FILE_NAME. Type defaults to STRING. + or FILE_NAME. Type defaults to STRING, except on X11, where the default + is to try UTF8_STRING and fall back to STRING. This command is equivalent to: selection_get(CLIPBOARD) """ + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('clipboard', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('clipboard', 'get') + self._options(kw)) def clipboard_clear(self, **kw): @@ -613,8 +620,16 @@ A keyword parameter selection specifies the name of the selection and defaults to PRIMARY. A keyword parameter displayof specifies a widget on the display - to use.""" + to use. A keyword parameter type specifies the form of data to be + fetched, defaulting to STRING except on X11, where UTF8_STRING is tried + before STRING.""" if 'displayof' not in kw: kw['displayof'] = self._w + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('selection', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('selection', 'get') + self._options(kw)) def selection_handle(self, command, **kw): """Specify a function COMMAND to call if the X @@ -1029,6 +1044,15 @@ if displayof is None: return ('-displayof', self._w) return () + @property + def _windowingsystem(self): + """Internal function.""" + try: + return self._root()._windowingsystem_cached + except AttributeError: + ws = self._root()._windowingsystem_cached = \ + self.tk.call('tk', 'windowingsystem') + return ws def _options(self, cnf, kw = None): """Internal function.""" if kw: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,12 @@ Library ------- +- Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when + accessing the Tk clipboard. Modify clipboad_get() to first request type + UTF8_STRING when no specific type is requested in an X11 windowing + environment, falling back to the current default type STRING if that fails. + Original patch by Thomas Kluyver. + - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic Authentation in urllib2. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 03:14:07 2012 From: python-checkins at python.org (ned.deily) Date: Wed, 16 May 2012 03:14:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314777=3A_merge?= Message-ID: http://hg.python.org/cpython/rev/97601cbf169f changeset: 76976:97601cbf169f parent: 76973:d9bf364d75e2 parent: 76975:41382250e5e1 user: Ned Deily date: Tue May 15 18:13:02 2012 -0700 summary: Issue #14777: merge files: Lib/tkinter/__init__.py | 28 ++++++++++++++++++++++++++-- Misc/NEWS | 6 ++++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -540,12 +540,19 @@ The type keyword specifies the form in which the data is to be returned and should be an atom name such as STRING - or FILE_NAME. Type defaults to STRING. + or FILE_NAME. Type defaults to STRING, except on X11, where the default + is to try UTF8_STRING and fall back to STRING. This command is equivalent to: selection_get(CLIPBOARD) """ + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('clipboard', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('clipboard', 'get') + self._options(kw)) def clipboard_clear(self, **kw): @@ -627,8 +634,16 @@ A keyword parameter selection specifies the name of the selection and defaults to PRIMARY. A keyword parameter displayof specifies a widget on the display - to use.""" + to use. A keyword parameter type specifies the form of data to be + fetched, defaulting to STRING except on X11, where UTF8_STRING is tried + before STRING.""" if 'displayof' not in kw: kw['displayof'] = self._w + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('selection', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('selection', 'get') + self._options(kw)) def selection_handle(self, command, **kw): """Specify a function COMMAND to call if the X @@ -1043,6 +1058,15 @@ if displayof is None: return ('-displayof', self._w) return () + @property + def _windowingsystem(self): + """Internal function.""" + try: + return self._root()._windowingsystem_cached + except AttributeError: + ws = self._root()._windowingsystem_cached = \ + self.tk.call('tk', 'windowingsystem') + return ws def _options(self, cnf, kw = None): """Internal function.""" if kw: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -34,6 +34,12 @@ Library ------- +- Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when + accessing the Tk clipboard. Modify clipboad_get() to first request type + UTF8_STRING when no specific type is requested in an X11 windowing + environment, falling back to the current default type STRING if that fails. + Original patch by Thomas Kluyver. + - Issue #14773: Fix os.fwalk() failing on dangling symlinks. - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 03:43:44 2012 From: python-checkins at python.org (barry.warsaw) Date: Wed, 16 May 2012 03:43:44 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_spelling?= Message-ID: http://hg.python.org/peps/rev/07962f9c78e8 changeset: 4382:07962f9c78e8 user: Barry Warsaw date: Tue May 15 21:43:40 2012 -0400 summary: spelling files: pep-0420.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -224,7 +224,7 @@ Namespace packages in the standard library ------------------------------------------ -It is possible, and this PEP explicitely allows, that parts of the +It is possible, and this PEP explicitly allows, that parts of the standard library be implemented as namespace packages. When and if any standard library packages become namespace packages is outside the scope of this PEP. @@ -291,7 +291,7 @@ layouts. 4. Implicit package directories will permanently entrench current - newbie-hostile behaviour in ``__main__``. + newbie-hostile behavior in ``__main__``. Nick gave a detailed response [5]_, which is summarized here: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 03:53:05 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 03:53:05 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_None_references=2E?= Message-ID: http://hg.python.org/peps/rev/9966458fc062 changeset: 4383:9966458fc062 user: Eric V. Smith date: Tue May 15 21:53:01 2012 -0400 summary: Fix None references. files: pep-0420.txt | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -173,15 +173,15 @@ PEP 302 defines "finders" that are called to search path elements. These finders' ``find_module`` methods return either a "loader" object -or None. +or ``None``. For a finder to contribute to namespace packages, it must implement a new ``find_loader(fullname)`` method. ``fullname`` has the same meaning as for ``find_module``. ``find_loader`` always returns a 2-tuple of ``(loader, )``. ``loader`` may -be None, in which case ```` (which may be -empty) is added to the list of recorded path entries and path -searching continues. If ``loader`` is not None, it is immediately +be ``None``, in which case ```` (which may +be empty) is added to the list of recorded path entries and path +searching continues. If ``loader`` is not ``None``, it is immediately used to load a module or regular package. Note that multiple path entries per finder are allowed. This is to -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 04:13:38 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 04:13:38 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Support_legacy_portions_along_?= =?utf8?q?with_PEP_420_portions=2E?= Message-ID: http://hg.python.org/peps/rev/10b1bffbb69a changeset: 4384:10b1bffbb69a user: Eric V. Smith date: Tue May 15 22:13:32 2012 -0400 summary: Support legacy portions along with PEP 420 portions. files: pep-0420.txt | 19 +++++++++++++++++++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -38,6 +38,8 @@ stored in a zip file) that contribute to a namespace package. * "regular package" refers to packages as they are implemented in Python 3.2 and earlier. + * "legacy portion" refers to a portion that uses ``__path__`` + manipulation in order to implement namespace packages. This PEP defines a new type of package, the "namespace package". @@ -184,6 +186,11 @@ searching continues. If ``loader`` is not ``None``, it is immediately used to load a module or regular package. +Even if ``loader`` is returned and is not ``None``, +```` must still contain the path entries for +the package. This allows code such as ``pkgutil.extend_path()`` to +compute path entries for packages that it does not load. + Note that multiple path entries per finder are allowed. This is to support the case where a finder discovers multiple namespace portions for a given ``fullname``. Many finders will support only a single @@ -230,6 +237,18 @@ scope of this PEP. +Migrating from legacy namespace packages +---------------------------------------- + +As described above, prior to this PEP ``pkgutil.extend_path()`` was +used by legacy portions to create namespace packages. Because it is +likely not practical for all existing portions of a namespace package +to be migrated to this PEP at once, ``extend_path()`` will be modified +to also recognize PEP 420 namespace packages. This will allow some +portions of a namespace to be legacty portions while others are +migrated to PEP 420. + + Packaging Implications ====================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 04:14:09 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 04:14:09 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NDA0NzI6IEV4?= =?utf8?q?plain_that_email_parser/generator_isn=27t_*quite*_=22idempotent?= =?utf8?q?=22?= Message-ID: http://hg.python.org/cpython/rev/8bd30967bc4b changeset: 76977:8bd30967bc4b branch: 3.2 parent: 76975:41382250e5e1 user: R David Murray date: Tue May 15 22:07:52 2012 -0400 summary: #1440472: Explain that email parser/generator isn't *quite* "idempotent" files: Doc/library/email.generator.rst | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -17,7 +17,8 @@ standards-compliant way, should handle MIME and non-MIME email messages just fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, -is idempotent (the input is identical to the output). On the other hand, using +is idempotent (the input is identical to the output) [#]_. On the other hand, +using the Generator on a :class:`~email.message.Message` constructed by program may result in changes to the :class:`~email.message.Message` object as defaults are filled in. @@ -204,3 +205,12 @@ The default value for *fmt* is ``None``, meaning :: [Non-text (%(type)s) part of message omitted, filename %(filename)s] + + +.. rubric:: Footnotes + +.. [#] This statement assumes that you use the appropriate setting for the + ``unixfrom`` argument, and that you set maxheaderlen=0 (which will + preserve whatever the input line lengths were). It is also not strictly + true, since in many cases runs of whitespace in headers are collapsed + into single blanks. The latter is a bug that will eventually be fixed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 04:14:10 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 04:14:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_=231440472=3A_Explain_that_email_parser/generator_isn?= =?utf8?q?=27t_*quite*_=22idempotent=22?= Message-ID: http://hg.python.org/cpython/rev/f534e6363bfb changeset: 76978:f534e6363bfb parent: 76976:97601cbf169f parent: 76977:8bd30967bc4b user: R David Murray date: Tue May 15 22:09:14 2012 -0400 summary: merge #1440472: Explain that email parser/generator isn't *quite* "idempotent" files: Doc/library/email.generator.rst | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -17,7 +17,8 @@ standards-compliant way, should handle MIME and non-MIME email messages just fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, -is idempotent (the input is identical to the output). On the other hand, using +is idempotent (the input is identical to the output) [#]_. On the other hand, +using the Generator on a :class:`~email.message.Message` constructed by program may result in changes to the :class:`~email.message.Message` object as defaults are filled in. @@ -223,3 +224,12 @@ The default value for *fmt* is ``None``, meaning :: [Non-text (%(type)s) part of message omitted, filename %(filename)s] + + +.. rubric:: Footnotes + +.. [#] This statement assumes that you use the appropriate setting for the + ``unixfrom`` argument, and that you set maxheaderlen=0 (which will + preserve whatever the input line lengths were). It is also not strictly + true, since in many cases runs of whitespace in headers are collapsed + into single blanks. The latter is a bug that will eventually be fixed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 04:14:11 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 04:14:11 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NDA0NzI6IEV4?= =?utf8?q?plain_that_email_parser/generator_isn=27t_*quite*_=22idempotent?= =?utf8?q?=22?= Message-ID: http://hg.python.org/cpython/rev/9d99273c2f74 changeset: 76979:9d99273c2f74 branch: 2.7 parent: 76974:f70fa654f70e user: R David Murray date: Tue May 15 22:12:09 2012 -0400 summary: #1440472: Explain that email parser/generator isn't *quite* "idempotent" files: Doc/library/email.generator.rst | 11 ++++++++++- 1 files changed, 10 insertions(+), 1 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -17,7 +17,8 @@ standards-compliant way, should handle MIME and non-MIME email messages just fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, -is idempotent (the input is identical to the output). On the other hand, using +is idempotent (the input is identical to the output) [#]_. On the other hand, +using the Generator on a :class:`~email.message.Message` constructed by program may result in changes to the :class:`~email.message.Message` object as defaults are filled in. @@ -125,3 +126,11 @@ .. versionchanged:: 2.5 The previously deprecated method :meth:`__call__` was removed. + +.. rubric:: Footnotes + +.. [#] This statement assumes that you use the appropriate setting for the + ``unixfrom`` argument, and that you set maxheaderlen=0 (which will + preserve whatever the input line lengths were). It is also not strictly + true, since in many cases runs of whitespace in headers are collapsed + into single blanks. The latter is a bug that will eventually be fixed. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 04:14:12 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 04:14:12 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NDA0NzI6IHJl?= =?utf8?q?flow?= Message-ID: http://hg.python.org/cpython/rev/180d16af22e9 changeset: 76980:180d16af22e9 branch: 2.7 user: R David Murray date: Tue May 15 22:12:56 2012 -0400 summary: #1440472: reflow files: Doc/library/email.generator.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -18,10 +18,9 @@ fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, is idempotent (the input is identical to the output) [#]_. On the other hand, -using -the Generator on a :class:`~email.message.Message` constructed by program may -result in changes to the :class:`~email.message.Message` object as defaults are -filled in. +using the Generator on a :class:`~email.message.Message` constructed by program +may result in changes to the :class:`~email.message.Message` object as defaults +are filled in. Here are the public methods of the :class:`Generator` class, imported from the :mod:`email.generator` module: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 04:14:13 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 04:14:13 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NDA0NzI6IHJl?= =?utf8?q?flow?= Message-ID: http://hg.python.org/cpython/rev/d1fbfd9af5c5 changeset: 76981:d1fbfd9af5c5 branch: 3.2 parent: 76977:8bd30967bc4b user: R David Murray date: Tue May 15 22:13:29 2012 -0400 summary: #1440472: reflow files: Doc/library/email.generator.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -18,10 +18,9 @@ fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, is idempotent (the input is identical to the output) [#]_. On the other hand, -using -the Generator on a :class:`~email.message.Message` constructed by program may -result in changes to the :class:`~email.message.Message` object as defaults are -filled in. +using the Generator on a :class:`~email.message.Message` constructed by program +may result in changes to the :class:`~email.message.Message` object as defaults +are filled in. :class:`bytes` output can be generated using the :class:`BytesGenerator` class. If the message object structure contains non-ASCII bytes, this generator's -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 04:14:13 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 04:14:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_=231440472=3A_reflow?= Message-ID: http://hg.python.org/cpython/rev/4ec95207281c changeset: 76982:4ec95207281c parent: 76978:f534e6363bfb parent: 76981:d1fbfd9af5c5 user: R David Murray date: Tue May 15 22:13:55 2012 -0400 summary: merge #1440472: reflow files: Doc/library/email.generator.rst | 7 +++---- 1 files changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -18,10 +18,9 @@ fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, is idempotent (the input is identical to the output) [#]_. On the other hand, -using -the Generator on a :class:`~email.message.Message` constructed by program may -result in changes to the :class:`~email.message.Message` object as defaults are -filled in. +using the Generator on a :class:`~email.message.Message` constructed by program +may result in changes to the :class:`~email.message.Message` object as defaults +are filled in. :class:`bytes` output can be generated using the :class:`BytesGenerator` class. If the message object structure contains non-ASCII bytes, this generator's -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed May 16 05:49:15 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 16 May 2012 05:49:15 +0200 Subject: [Python-checkins] Daily reference leaks (97601cbf169f): sum=0 Message-ID: results for 97601cbf169f on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogiAtKjA', '-x'] From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Ignore_virtualenv?= =?utf8?q?=E2=80=99s_man_directory_=28installing_nose_creates_it=29?= Message-ID: http://hg.python.org/distutils2/rev/ac1d3c21e155 changeset: 1315:ac1d3c21e155 parent: 1312:e0d1ba9899d5 user: ?ric Araujo date: Sun Mar 18 11:54:08 2012 -0400 summary: Ignore virtualenv?s man directory (installing nose creates it) files: .hgignore | 18 +++++++----------- 1 files changed, 7 insertions(+), 11 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -1,18 +1,14 @@ syntax: glob *.py[co] -__pycache__/ -*.so -configure.cache +*.swp +bin/ +include/ +lib/ +man/ +Distutils2.egg-info MANIFEST build/ dist/ -_static/ -_build/ -*.swp +.tox/ .coverage -.tox -lib -include -bin nosetests.xml -Distutils2.egg-info -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_=2314270=3A_Fixes_to_add?= =?utf8?q?_dest=5Fdir_params_working_when_using_install=5Ffrom=5Finfos?= Message-ID: http://hg.python.org/distutils2/rev/fd5d379c36f4 changeset: 1316:fd5d379c36f4 parent: 1314:6e5312167b4a user: Mathieu Leduc-Hamel date: Sat Apr 21 17:42:52 2012 -0400 summary: #14270: Fixes to add dest_dir params working when using install_from_infos files: distutils2/install.py | 18 ++++++++-------- distutils2/tests/test_install.py | 22 ++++++++++++++++++++ 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -58,7 +58,7 @@ yield old, new -def _run_distutils_install(path): +def _run_distutils_install(path, dest): # backward compat: using setuptools or plain-distutils cmd = '%s setup.py install --record=%s' record_file = os.path.join(path, 'RECORD') @@ -69,7 +69,7 @@ egginfo_to_distinfo(record_file, remove_egginfo=True) -def _run_setuptools_install(path): +def _run_setuptools_install(path, dest): cmd = '%s setup.py install --record=%s --single-version-externally-managed' record_file = os.path.join(path, 'RECORD') @@ -80,12 +80,12 @@ egginfo_to_distinfo(record_file, remove_egginfo=True) -def _run_packaging_install(path): +def _run_packaging_install(path, dest): # XXX check for a valid setup.cfg? dist = Distribution() dist.parse_config_files() try: - dist.run_command('install_dist') + dist.run_command('install_dist', dict(prefix=(None,dest))) name = dist.metadata['Name'] return database.get_distribution(name) is not None except (IOError, os.error, PackagingError, CCompilerError), msg: @@ -106,7 +106,7 @@ if where is None: raise ValueError('Cannot locate the unpacked archive') - return _run_install_from_archive(where) + return _run_install_from_archive(where, path) def install_local_project(path): @@ -134,14 +134,14 @@ return False -def _run_install_from_archive(source_dir): +def _run_install_from_archive(source_dir, dest_dir): # XXX need a better way for item in os.listdir(source_dir): fullpath = os.path.join(source_dir, item) if os.path.isdir(fullpath): source_dir = fullpath break - return _run_install_from_dir(source_dir) + return _run_install_from_dir(source_dir, dest_dir) install_methods = { @@ -150,7 +150,7 @@ 'distutils': _run_distutils_install} -def _run_install_from_dir(source_dir): +def _run_install_from_dir(source_dir, destination_dir=None): old_dir = os.getcwd() os.chdir(source_dir) install_method = get_install_method(source_dir) @@ -158,7 +158,7 @@ try: func = install_methods[install_method] try: - func(source_dir) + func(source_dir, destination_dir) return True except ValueError, err: # failed to install diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py --- a/distutils2/tests/test_install.py +++ b/distutils2/tests/test_install.py @@ -1,6 +1,8 @@ """Tests for the distutils2.install module.""" import os import logging +import sys + from tempfile import mkstemp from distutils2 import install @@ -258,6 +260,26 @@ for key in expect: self.assertEqual(expect[key], dict1[key]) + def test_install_custom_dir(self): + dest = self.mkdtemp() + src = self.mkdtemp() + + project_dir, dist = self.create_dist( + name='Spamlib', version='0.1', + data_files={'spamd': '{scripts}/spamd'}) + + dist.name = MagicMock(return_value='Spamlib') + dist.version = MagicMock(return_value='0.1') + dist.unpack = MagicMock(return_value=project_dir) + + self.write_file([project_dir, 'setup.cfg'], + ("[metadata]\n" + "name = mypackage\n" + "version = 0.1.0\n")) + + install.install_from_infos(dest, install=[dist]) + self.assertEqual(len(os.listdir(dest)), 1) + def test_install_dists_rollback(self): # if one of the distribution installation fails, call uninstall on all # installed distributions. -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_=2313166=3A_Implement_?= =?utf8?q?=5F=5Fstr=5F=5F_on_Distribution_and_EggInfoDistribution=2C_use_f?= =?utf8?q?or?= Message-ID: http://hg.python.org/distutils2/rev/ff44594dd3c0 changeset: 1317:ff44594dd3c0 user: Guillaume Pratte date: Sat Apr 21 17:43:54 2012 -0400 summary: #13166: Implement __str__ on Distribution and EggInfoDistribution, use for pysetup list, pysetup graph and installation log files: CONTRIBUTORS.txt | 1 + distutils2/database.py | 6 +++++ distutils2/depgraph.py | 10 +++----- distutils2/install.py | 2 +- distutils2/run.py | 2 +- distutils2/tests/test_database.py | 21 ++++++++++++++++-- 6 files changed, 31 insertions(+), 11 deletions(-) diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -59,6 +59,7 @@ - Ga?l Pasgrimaud - George Peristerakis - Mathieu Perreault +- Guillaume Pratte - Sean Reifschneider - Antoine Reversat - Arc Riley diff --git a/distutils2/database.py b/distutils2/database.py --- a/distutils2/database.py +++ b/distutils2/database.py @@ -161,6 +161,9 @@ return '' % ( self.name, self.version, self.path) + def __str__(self): + return "%s %s" % (self.name, self.version) + def _get_records(self, local=False): results = [] record = self.get_distinfo_file('RECORD') @@ -365,6 +368,9 @@ return '' % ( self.name, self.version, self.path) + def __str__(self): + return "%s %s" % (self.name, self.version) + def list_installed_files(self, local=False): def _md5(path): diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py --- a/distutils2/depgraph.py +++ b/distutils2/depgraph.py @@ -71,18 +71,16 @@ """ self.missing[distribution].append(requirement) - def _repr_dist(self, dist): - return '%r %s' % (dist.name, dist.version) - def repr_node(self, dist, level=1): """Prints only a subgraph""" output = [] - output.append(self._repr_dist(dist)) + output.append(str(dist)) + # XXX: this code needs cleanup for other, label in self.adjacency_list[dist]: - dist = self._repr_dist(other) + dist = str(other) if label is not None: dist = '%s [%s]' % (dist, label) - output.append(' ' * level + str(dist)) + output.append(' ' * level + dist) suboutput = self.repr_node(other, level + 1) subs = suboutput.split('\n') output.extend(subs[1:]) diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -183,7 +183,7 @@ installed_dists = [] for dist in dists: - logger.info('Installing %r %s...', dist.name, dist.version) + logger.info('Installing %s...', dist) try: _install_dist(dist, path) installed_dists.append(dist) diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -308,7 +308,7 @@ number = 0 for dist in results: - print '%r %s (from %r)' % (dist.name, dist.version, dist.path) + print "%s (from %r)" % (dist, dist.path) number += 1 if number == 0: diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py --- a/distutils2/tests/test_database.py +++ b/distutils2/tests/test_database.py @@ -80,12 +80,14 @@ attributes are used in test methods. See source code for details. """ + def _get_dist_path(self, distdir): + here = os.path.abspath(os.path.dirname(__file__)) + return os.path.join(here, 'fake_dists', distdir) + def test_instantiation(self): # check that useful attributes are here name, version, distdir = self.sample_dist - here = os.path.abspath(os.path.dirname(__file__)) - dist_path = os.path.join(here, 'fake_dists', distdir) - + dist_path = self._get_dist_path(distdir) dist = self.dist = self.cls(dist_path) self.assertEqual(dist.path, dist_path) self.assertEqual(dist.name, name) @@ -101,6 +103,17 @@ self.assertIn(self.cls.__name__, repr(dist)) @requires_zlib + def test_str(self): + name, version, distdir = self.sample_dist + dist = self.cls(self._get_dist_path(distdir)) + self.assertEqual(name, dist.name) + # Sanity test: dist.name is unicode, + # but str output contains no u prefix. + self.assertIsInstance(dist.name, unicode) + self.assertEqual(version, dist.version) + self.assertEqual(str(dist), self.expected_str_output) + + @requires_zlib def test_comparison(self): # tests for __eq__ and __hash__ dist = self.cls(self.dirs[0]) @@ -128,6 +141,7 @@ cls = Distribution sample_dist = 'choxie', '2.0.0.9', 'choxie-2.0.0.9.dist-info' + expected_str_output = 'choxie 2.0.0.9' def setUp(self): super(TestDistribution, self).setUp() @@ -265,6 +279,7 @@ cls = EggInfoDistribution sample_dist = 'bacon', '0.1', 'bacon-0.1.egg-info' + expected_str_output = 'bacon 0.1' def setUp(self): super(TestEggInfoDistribution, self).setUp() -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_remove_changes_that_modi?= =?utf8?q?fied_output_of_pysetup_graph=3A_tests_are_needed_first?= Message-ID: http://hg.python.org/distutils2/rev/64b0bf5e0596 changeset: 1318:64b0bf5e0596 user: Guillaume Pratte date: Sat Apr 21 18:25:22 2012 -0400 summary: remove changes that modified output of pysetup graph: tests are needed first files: distutils2/depgraph.py | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py --- a/distutils2/depgraph.py +++ b/distutils2/depgraph.py @@ -71,16 +71,18 @@ """ self.missing[distribution].append(requirement) + def _repr_dist(self, dist): + return '%r %s' % (dist.name, dist.version) + def repr_node(self, dist, level=1): """Prints only a subgraph""" output = [] - output.append(str(dist)) - # XXX: this code needs cleanup + output.append(self._repr_dist(dist)) for other, label in self.adjacency_list[dist]: - dist = str(other) + dist = self._repr_dist(other) if label is not None: dist = '%s [%s]' % (dist, label) - output.append(' ' * level + dist) + output.append(' ' * level + str(dist)) suboutput = self.repr_node(other, level + 1) subs = suboutput.split('\n') output.extend(subs[1:]) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_incorrect_docstring?= Message-ID: http://hg.python.org/distutils2/rev/852995484bfb changeset: 1319:852995484bfb user: ?ric Araujo date: Sat May 12 14:49:09 2012 -0400 summary: Fix incorrect docstring files: distutils2/install.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -110,7 +110,9 @@ def install_local_project(path): - """Install a distribution from a source directory. + """Install a distribution from a source directory or archive. + + If *path* is an archive, it will be unarchived first. If the source directory contains a setup.py install using distutils1. If a setup.cfg is found, install using the install_dist command. -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_typo?= Message-ID: http://hg.python.org/distutils2/rev/0d41cc838194 changeset: 1320:0d41cc838194 parent: 1314:6e5312167b4a user: Julien Courteau date: Sat Apr 21 17:51:00 2012 -0400 summary: Fix typo files: distutils2/metadata.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -480,7 +480,7 @@ return value def check(self, strict=False, restructuredtext=False): - """Check if the metadata is compliant. If strict is False then raise if + """Check if the metadata is compliant. If strict is True then raise if no Name or Version are provided""" self.set_metadata_version() -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_fix=3A_setup=2Epy_regist?= =?utf8?q?er_fails_if_description_has_ReST_uncompatible_with_Sphinx?= Message-ID: http://hg.python.org/distutils2/rev/70c683338f9b changeset: 1321:70c683338f9b user: Julien Courteau date: Sat Apr 21 20:59:06 2012 -0400 summary: fix: setup.py register fails if description has ReST uncompatible with Sphinx (#13614) files: distutils2/metadata.py | 3 +++ distutils2/tests/test_metadata.py | 9 +++++++++ 2 files changed, 12 insertions(+), 0 deletions(-) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -35,6 +35,9 @@ def system_message(self, level, message, *children, **kwargs): self.messages.append((level, message, children, kwargs)) + return nodes.system_message(message, level=level, type=self. + levels[level], *children, **kwargs) + _HAS_DOCUTILS = True except ImportError: diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py --- a/distutils2/tests/test_metadata.py +++ b/distutils2/tests/test_metadata.py @@ -379,6 +379,15 @@ folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|') self.assertIn(folded_desc, out.getvalue()) + def test_description_invalid_rst(self): + # make sure bad rst is well handled in the description attribute + metadata = Metadata() + description_with_bad_rst = ':funkie:`str`' # Sphinx-specific markup + metadata['description'] = description_with_bad_rst + missing, warnings = metadata.check(restructuredtext=True) + warning = warnings[0][1] + self.assertIn('funkie', warning) + def test_project_url(self): metadata = Metadata() metadata['Project-URL'] = [('one', 'http://ok')] -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_Branch_merge?= Message-ID: http://hg.python.org/distutils2/rev/52394a308caf changeset: 1323:52394a308caf parent: 1322:6480b23cdf48 parent: 1319:852995484bfb user: ?ric Araujo date: Sat May 12 14:50:08 2012 -0400 summary: Branch merge files: distutils2/install.py | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -110,7 +110,9 @@ def install_local_project(path): - """Install a distribution from a source directory. + """Install a distribution from a source directory or archive. + + If *path* is an archive, it will be unarchived first. If the source directory contains a setup.py install using distutils1. If a setup.cfg is found, install using the install_dist command. -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_merge_=231364?= Message-ID: http://hg.python.org/distutils2/rev/6480b23cdf48 changeset: 1322:6480b23cdf48 parent: 1318:64b0bf5e0596 parent: 1321:70c683338f9b user: Julien Courteau date: Sat May 12 14:23:20 2012 -0400 summary: merge #1364 files: distutils2/metadata.py | 5 ++++- distutils2/tests/test_metadata.py | 9 +++++++++ 2 files changed, 13 insertions(+), 1 deletions(-) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -35,6 +35,9 @@ def system_message(self, level, message, *children, **kwargs): self.messages.append((level, message, children, kwargs)) + return nodes.system_message(message, level=level, type=self. + levels[level], *children, **kwargs) + _HAS_DOCUTILS = True except ImportError: @@ -480,7 +483,7 @@ return value def check(self, strict=False, restructuredtext=False): - """Check if the metadata is compliant. If strict is False then raise if + """Check if the metadata is compliant. If strict is True then raise if no Name or Version are provided""" self.set_metadata_version() diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py --- a/distutils2/tests/test_metadata.py +++ b/distutils2/tests/test_metadata.py @@ -379,6 +379,15 @@ folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|') self.assertIn(folded_desc, out.getvalue()) + def test_description_invalid_rst(self): + # make sure bad rst is well handled in the description attribute + metadata = Metadata() + description_with_bad_rst = ':funkie:`str`' # Sphinx-specific markup + metadata['description'] = description_with_bad_rst + missing, warnings = metadata.check(restructuredtext=True) + warning = warnings[0][1] + self.assertIn('funkie', warning) + def test_project_url(self): metadata = Metadata() metadata['Project-URL'] = [('one', 'http://ok')] -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_=2313614=3A_Raises_a_Typ?= =?utf8?q?eError_now_if_the_RST_description_is_invalid?= Message-ID: http://hg.python.org/distutils2/rev/7daa144a7af0 changeset: 1324:7daa144a7af0 parent: 1314:6e5312167b4a user: Pierre Paul date: Sat May 12 15:02:15 2012 -0400 summary: #13614: Raises a TypeError now if the RST description is invalid files: distutils2/metadata.py | 2 +- distutils2/tests/fake_dists/python-pager-readme.rst | 65 ++++++++++ distutils2/tests/test_command_register.py | 18 ++ 3 files changed, 84 insertions(+), 1 deletions(-) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -276,7 +276,7 @@ document.note_source(source_path, -1) try: parser.parse(data, document) - except AttributeError: + except (AttributeError, TypeError): reporter.messages.append((-1, 'Could not finish the parsing.', '', {})) diff --git a/distutils2/tests/fake_dists/python-pager-readme.rst b/distutils2/tests/fake_dists/python-pager-readme.rst new file mode 100644 --- /dev/null +++ b/distutils2/tests/fake_dists/python-pager-readme.rst @@ -0,0 +1,65 @@ + +Python module to page screen output and get dimensions +of available console space. + +It is meant to be finally included into standard library +http://bugs.python.org/issue8408 + +| Author: anatoly techtonik +| License: Public Domain (or MIT if a license is required) + + +Status +------ + +0.1 (stable) + - shows content page by page + - allows to get console/terminal dimensions + - works on Windows + - works on Linux + + +API +--- + +..function:: getwidth() + + Return width of available window in characters. If detection fails, + return value of standard width 80. Coordinate of the last character + on a line is -1 from returned value. + + +..function:: getheight() + + Return available window height in characters or 25 if detection fails. + Coordinate of the last line is -1 from returned value. + + +..function:: getch() + + Wait for keypress and return character in a cross-platform way. + Credits: Danny Yoo, Python Cookbook + + +..function:: page(content, [pagecallback=prompt]) + + Output `content` iterable, calling `pagecallback` function after each + page. Default :func:`prompt` callback shows 'Press any key . . . ' prompt + and waits for keypress. + + +References +---------- + +Excellent tutorials for Win32 Console by Adrian Worley +http://www.adrianxw.dk/SoftwareSite/index.html +Console Reference on MSDN +http://msdn.microsoft.com/en-us/library/ms682087%28VS.85%29.aspx + +Public Domain Curses library maintained by William McBrine +http://pdcurses.sourceforge.net/ + +Ioctl (input/output control) introduction from Wikipedia +http://en.wikipedia.org/wiki/Ioctl +Linux Programmer's Manual - ioctls for terminals and serial lines +http://www.kernel.org/doc/man-pages/online/pages/man4/tty_ioctl.4.html diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py --- a/distutils2/tests/test_command_register.py +++ b/distutils2/tests/test_command_register.py @@ -253,6 +253,24 @@ self.assertEqual(data['metadata_version'], '1.2') self.assertEqual(data['requires_dist'], ['lxml']) + def test_register_invalid_long_description(self): + readme_file = os.path.join(os.path.dirname(__file__), + 'fake_dists', 'python-pager-readme.rst') + + # Contains :func: which break the rst format + data = "".join(open(readme_file).readlines()) + + metadata = {'Home-page': 'xxx', 'Author': 'xxx', + 'Author-email': 'xxx', + 'Name': 'xxx', 'Version': 'xxx'} + + metadata['Description'] = data + cmd = self._get_cmd(metadata) + cmd.ensure_finalized() + cmd.strict = True + inputs = Inputs('2', 'tarek', 'tarek at ziade.org') + register_module.raw_input = inputs + self.assertRaises(PackagingSetupError, cmd.run) def test_suite(): return unittest.makeSuite(RegisterTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_Merge?= Message-ID: http://hg.python.org/distutils2/rev/089d57251a00 changeset: 1325:089d57251a00 parent: 1324:7daa144a7af0 parent: 1323:52394a308caf user: Pierre Paul date: Sat May 12 15:03:53 2012 -0400 summary: Merge files: CONTRIBUTORS.txt | 1 + distutils2/database.py | 6 ++++ distutils2/install.py | 24 ++++++++++-------- distutils2/metadata.py | 5 +++- distutils2/run.py | 2 +- distutils2/tests/test_database.py | 21 ++++++++++++++-- distutils2/tests/test_install.py | 22 +++++++++++++++++ distutils2/tests/test_metadata.py | 9 +++++++ 8 files changed, 74 insertions(+), 16 deletions(-) diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -59,6 +59,7 @@ - Ga?l Pasgrimaud - George Peristerakis - Mathieu Perreault +- Guillaume Pratte - Sean Reifschneider - Antoine Reversat - Arc Riley diff --git a/distutils2/database.py b/distutils2/database.py --- a/distutils2/database.py +++ b/distutils2/database.py @@ -161,6 +161,9 @@ return '' % ( self.name, self.version, self.path) + def __str__(self): + return "%s %s" % (self.name, self.version) + def _get_records(self, local=False): results = [] record = self.get_distinfo_file('RECORD') @@ -365,6 +368,9 @@ return '' % ( self.name, self.version, self.path) + def __str__(self): + return "%s %s" % (self.name, self.version) + def list_installed_files(self, local=False): def _md5(path): diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -58,7 +58,7 @@ yield old, new -def _run_distutils_install(path): +def _run_distutils_install(path, dest): # backward compat: using setuptools or plain-distutils cmd = '%s setup.py install --record=%s' record_file = os.path.join(path, 'RECORD') @@ -69,7 +69,7 @@ egginfo_to_distinfo(record_file, remove_egginfo=True) -def _run_setuptools_install(path): +def _run_setuptools_install(path, dest): cmd = '%s setup.py install --record=%s --single-version-externally-managed' record_file = os.path.join(path, 'RECORD') @@ -80,12 +80,12 @@ egginfo_to_distinfo(record_file, remove_egginfo=True) -def _run_packaging_install(path): +def _run_packaging_install(path, dest): # XXX check for a valid setup.cfg? dist = Distribution() dist.parse_config_files() try: - dist.run_command('install_dist') + dist.run_command('install_dist', dict(prefix=(None,dest))) name = dist.metadata['Name'] return database.get_distribution(name) is not None except (IOError, os.error, PackagingError, CCompilerError), msg: @@ -106,11 +106,13 @@ if where is None: raise ValueError('Cannot locate the unpacked archive') - return _run_install_from_archive(where) + return _run_install_from_archive(where, path) def install_local_project(path): - """Install a distribution from a source directory. + """Install a distribution from a source directory or archive. + + If *path* is an archive, it will be unarchived first. If the source directory contains a setup.py install using distutils1. If a setup.cfg is found, install using the install_dist command. @@ -134,14 +136,14 @@ return False -def _run_install_from_archive(source_dir): +def _run_install_from_archive(source_dir, dest_dir): # XXX need a better way for item in os.listdir(source_dir): fullpath = os.path.join(source_dir, item) if os.path.isdir(fullpath): source_dir = fullpath break - return _run_install_from_dir(source_dir) + return _run_install_from_dir(source_dir, dest_dir) install_methods = { @@ -150,7 +152,7 @@ 'distutils': _run_distutils_install} -def _run_install_from_dir(source_dir): +def _run_install_from_dir(source_dir, destination_dir=None): old_dir = os.getcwd() os.chdir(source_dir) install_method = get_install_method(source_dir) @@ -158,7 +160,7 @@ try: func = install_methods[install_method] try: - func(source_dir) + func(source_dir, destination_dir) return True except ValueError, err: # failed to install @@ -183,7 +185,7 @@ installed_dists = [] for dist in dists: - logger.info('Installing %r %s...', dist.name, dist.version) + logger.info('Installing %s...', dist) try: _install_dist(dist, path) installed_dists.append(dist) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -35,6 +35,9 @@ def system_message(self, level, message, *children, **kwargs): self.messages.append((level, message, children, kwargs)) + return nodes.system_message(message, level=level, type=self. + levels[level], *children, **kwargs) + _HAS_DOCUTILS = True except ImportError: @@ -480,7 +483,7 @@ return value def check(self, strict=False, restructuredtext=False): - """Check if the metadata is compliant. If strict is False then raise if + """Check if the metadata is compliant. If strict is True then raise if no Name or Version are provided""" self.set_metadata_version() diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -308,7 +308,7 @@ number = 0 for dist in results: - print '%r %s (from %r)' % (dist.name, dist.version, dist.path) + print "%s (from %r)" % (dist, dist.path) number += 1 if number == 0: diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py --- a/distutils2/tests/test_database.py +++ b/distutils2/tests/test_database.py @@ -80,12 +80,14 @@ attributes are used in test methods. See source code for details. """ + def _get_dist_path(self, distdir): + here = os.path.abspath(os.path.dirname(__file__)) + return os.path.join(here, 'fake_dists', distdir) + def test_instantiation(self): # check that useful attributes are here name, version, distdir = self.sample_dist - here = os.path.abspath(os.path.dirname(__file__)) - dist_path = os.path.join(here, 'fake_dists', distdir) - + dist_path = self._get_dist_path(distdir) dist = self.dist = self.cls(dist_path) self.assertEqual(dist.path, dist_path) self.assertEqual(dist.name, name) @@ -101,6 +103,17 @@ self.assertIn(self.cls.__name__, repr(dist)) @requires_zlib + def test_str(self): + name, version, distdir = self.sample_dist + dist = self.cls(self._get_dist_path(distdir)) + self.assertEqual(name, dist.name) + # Sanity test: dist.name is unicode, + # but str output contains no u prefix. + self.assertIsInstance(dist.name, unicode) + self.assertEqual(version, dist.version) + self.assertEqual(str(dist), self.expected_str_output) + + @requires_zlib def test_comparison(self): # tests for __eq__ and __hash__ dist = self.cls(self.dirs[0]) @@ -128,6 +141,7 @@ cls = Distribution sample_dist = 'choxie', '2.0.0.9', 'choxie-2.0.0.9.dist-info' + expected_str_output = 'choxie 2.0.0.9' def setUp(self): super(TestDistribution, self).setUp() @@ -265,6 +279,7 @@ cls = EggInfoDistribution sample_dist = 'bacon', '0.1', 'bacon-0.1.egg-info' + expected_str_output = 'bacon 0.1' def setUp(self): super(TestEggInfoDistribution, self).setUp() diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py --- a/distutils2/tests/test_install.py +++ b/distutils2/tests/test_install.py @@ -1,6 +1,8 @@ """Tests for the distutils2.install module.""" import os import logging +import sys + from tempfile import mkstemp from distutils2 import install @@ -258,6 +260,26 @@ for key in expect: self.assertEqual(expect[key], dict1[key]) + def test_install_custom_dir(self): + dest = self.mkdtemp() + src = self.mkdtemp() + + project_dir, dist = self.create_dist( + name='Spamlib', version='0.1', + data_files={'spamd': '{scripts}/spamd'}) + + dist.name = MagicMock(return_value='Spamlib') + dist.version = MagicMock(return_value='0.1') + dist.unpack = MagicMock(return_value=project_dir) + + self.write_file([project_dir, 'setup.cfg'], + ("[metadata]\n" + "name = mypackage\n" + "version = 0.1.0\n")) + + install.install_from_infos(dest, install=[dist]) + self.assertEqual(len(os.listdir(dest)), 1) + def test_install_dists_rollback(self): # if one of the distribution installation fails, call uninstall on all # installed distributions. diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py --- a/distutils2/tests/test_metadata.py +++ b/distutils2/tests/test_metadata.py @@ -379,6 +379,15 @@ folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|') self.assertIn(folded_desc, out.getvalue()) + def test_description_invalid_rst(self): + # make sure bad rst is well handled in the description attribute + metadata = Metadata() + description_with_bad_rst = ':funkie:`str`' # Sphinx-specific markup + metadata['description'] = description_with_bad_rst + missing, warnings = metadata.check(restructuredtext=True) + warning = warnings[0][1] + self.assertIn('funkie', warning) + def test_project_url(self): metadata = Metadata() metadata['Project-URL'] = [('one', 'http://ok')] -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Only_run_specific_test_i?= =?utf8?q?f_docutils_is_installed?= Message-ID: http://hg.python.org/distutils2/rev/bb9ca80afc84 changeset: 1326:bb9ca80afc84 user: Mathieu Leduc-Hamel date: Sat May 12 15:36:01 2012 -0400 summary: Only run specific test if docutils is installed files: distutils2/tests/support.py | 8 +++++++- distutils2/tests/test_metadata.py | 3 ++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py --- a/distutils2/tests/support.py +++ b/distutils2/tests/support.py @@ -46,6 +46,10 @@ import zlib except ImportError: zlib = None +try: + import docutils +except ImportError: + docutils = None from distutils2.dist import Distribution from distutils2.util import resolve_name @@ -63,7 +67,7 @@ # misc. functions and decorators 'fake_dec', 'create_distribution', 'use_command', 'copy_xxmodule_c', 'fixup_build_ext', - 'requires_py26_min', 'skip_2to3_optimize', + 'requires_py26_min', 'skip_2to3_optimize', 'requires_docutils', # imported from this module for backport purposes 'unittest', 'requires_zlib', 'skip_unless_symlink', ] @@ -411,6 +415,8 @@ requires_zlib = unittest.skipUnless(zlib, 'requires zlib') +requires_docutils = unittest.skipUnless(docutils, 'requires docutils') + def unlink(filename): try: diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py --- a/distutils2/tests/test_metadata.py +++ b/distutils2/tests/test_metadata.py @@ -12,7 +12,7 @@ from distutils2.tests import unittest from distutils2.tests.support import (LoggingCatcher, TempdirManager, - EnvironRestorer) + EnvironRestorer, requires_docutils) class MetadataTestCase(LoggingCatcher, @@ -379,6 +379,7 @@ folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|') self.assertIn(folded_desc, out.getvalue()) + @requires_docutils def test_description_invalid_rst(self): # make sure bad rst is well handled in the description attribute metadata = Metadata() -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_=2313614=3A_Merge_change?= =?utf8?q?s_from_julien=5Fcourteau_and_PierrePaul_regarding_bad_rst?= Message-ID: http://hg.python.org/distutils2/rev/0aa2406697aa changeset: 1327:0aa2406697aa user: Mathieu Leduc-Hamel date: Sat May 12 16:02:50 2012 -0400 summary: #13614: Merge changes from julien_courteau and PierrePaul regarding bad rst files: distutils2/metadata.py | 4 +- distutils2/tests/fake_dists/python-pager-readme.rst | 62 ---------- distutils2/tests/test_command_register.py | 15 +- 3 files changed, 10 insertions(+), 71 deletions(-) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -36,7 +36,7 @@ def system_message(self, level, message, *children, **kwargs): self.messages.append((level, message, children, kwargs)) return nodes.system_message(message, level=level, type=self. - levels[level], *children, **kwargs) + levels[level], *children, **kwargs) _HAS_DOCUTILS = True @@ -279,7 +279,7 @@ document.note_source(source_path, -1) try: parser.parse(data, document) - except (AttributeError, TypeError): + except AttributeError: reporter.messages.append((-1, 'Could not finish the parsing.', '', {})) diff --git a/distutils2/tests/fake_dists/python-pager-readme.rst b/distutils2/tests/fake_dists/python-pager-readme.rst deleted file mode 100644 --- a/distutils2/tests/fake_dists/python-pager-readme.rst +++ /dev/null @@ -1,65 +0,0 @@ - -Python module to page screen output and get dimensions -of available console space. - -It is meant to be finally included into standard library -http://bugs.python.org/issue8408 - -| Author: anatoly techtonik -| License: Public Domain (or MIT if a license is required) - - -Status ------- - -0.1 (stable) - - shows content page by page - - allows to get console/terminal dimensions - - works on Windows - - works on Linux - - -API ---- - -..function:: getwidth() - - Return width of available window in characters. If detection fails, - return value of standard width 80. Coordinate of the last character - on a line is -1 from returned value. - - -..function:: getheight() - - Return available window height in characters or 25 if detection fails. - Coordinate of the last line is -1 from returned value. - - -..function:: getch() - - Wait for keypress and return character in a cross-platform way. - Credits: Danny Yoo, Python Cookbook - - -..function:: page(content, [pagecallback=prompt]) - - Output `content` iterable, calling `pagecallback` function after each - page. Default :func:`prompt` callback shows 'Press any key . . . ' prompt - and waits for keypress. - - -References ----------- - -Excellent tutorials for Win32 Console by Adrian Worley -http://www.adrianxw.dk/SoftwareSite/index.html -Console Reference on MSDN -http://msdn.microsoft.com/en-us/library/ms682087%28VS.85%29.aspx - -Public Domain Curses library maintained by William McBrine -http://pdcurses.sourceforge.net/ - -Ioctl (input/output control) introduction from Wikipedia -http://en.wikipedia.org/wiki/Ioctl -Linux Programmer's Manual - ioctls for terminals and serial lines -http://www.kernel.org/doc/man-pages/online/pages/man4/tty_ioctl.4.html diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py --- a/distutils2/tests/test_command_register.py +++ b/distutils2/tests/test_command_register.py @@ -10,7 +10,7 @@ DOCUTILS_SUPPORT = False from distutils2.tests import unittest, support -from distutils2.tests.support import Inputs +from distutils2.tests.support import (Inputs, requires_docutils) from distutils2.command import register as register_module from distutils2.command.register import register from distutils2.errors import PackagingSetupError @@ -253,12 +253,10 @@ self.assertEqual(data['metadata_version'], '1.2') self.assertEqual(data['requires_dist'], ['lxml']) + @requires_docutils def test_register_invalid_long_description(self): - readme_file = os.path.join(os.path.dirname(__file__), - 'fake_dists', 'python-pager-readme.rst') - - # Contains :func: which break the rst format - data = "".join(open(readme_file).readlines()) + # Contains :func: which break the rst format + data = "Default :func:`prompt` callback shows" metadata = {'Home-page': 'xxx', 'Author': 'xxx', 'Author-email': 'xxx', @@ -270,7 +268,10 @@ cmd.strict = True inputs = Inputs('2', 'tarek', 'tarek at ziade.org') register_module.raw_input = inputs - self.assertRaises(PackagingSetupError, cmd.run) + with self.assertRaises(PackagingSetupError) as e: + cmd.run() + self.assertIsNotNone(e) + self.assertIn('func', repr(e.exception)) def test_suite(): return unittest.makeSuite(RegisterTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_=2313399=3A_Fixes_the_in?= =?utf8?q?valid_arguments_handling?= Message-ID: http://hg.python.org/distutils2/rev/478700dd4f6c changeset: 1328:478700dd4f6c user: Patrice Gauthier date: Sat May 12 17:39:00 2012 -0400 summary: #13399: Fixes the invalid arguments handling files: distutils2/run.py | 17 +++++++++--- distutils2/tests/test_run.py | 32 +++++++++++++++++++++++- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -397,8 +397,9 @@ allowed = [action[0] for action in actions] + [None] if self.action not in allowed: - msg = 'Unrecognized action "%s"' % self.action - raise PackagingArgError(msg) + msg = 'Unrecognized action %s' % self.action + self.show_help() + self.exit_with_error_msg(msg) self._set_logger() self.args = args @@ -444,7 +445,8 @@ try: cmd_class = get_command_class(command) except PackagingModuleError, msg: - raise PackagingArgError(msg) + self.show_help() + self.exit_with_error_msg(msg) # XXX We want to push this in distutils2.command # @@ -485,7 +487,11 @@ cmd_class.user_options + help_options) parser.set_negative_aliases(_negative_opt) - args, opts = parser.getopt(args[1:]) + try: + args, opts = parser.getopt(args[1:]) + except PackagingArgError, msg: + self.show_help() + self.exit_with_error_msg(msg) if hasattr(opts, 'help') and opts.help: self._show_command_help(cmd_class) @@ -530,6 +536,9 @@ def show_help(self): self._show_help(self.parser) + def exit_with_error_msg(self, msg): + sys.exit('error: ' + msg.__str__()) + def print_usage(self, parser): parser.set_option_table(global_options) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -8,7 +8,7 @@ from distutils2.tests import unittest, support from distutils2.run import main -from distutils2.tests.support import assert_python_ok +from distutils2.tests.support import assert_python_ok, assert_python_failure # setup script that uses __file__ setup_using___file__ = """\ @@ -94,6 +94,36 @@ self.assertTrue(build_position, out) self.assertLess(check_position, build_position, out) + def test_unknown_run_option(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'run', 'build', + '--unknown', PYTHONPATH=self.get_pythonpath() + ) + self.assertEqual(status, 1) + self.assertGreater(out, '') + self.assertEqual(err.splitlines()[-1], + 'error: option --unknown not recognized') + + def test_unknown_command(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'run', + 'invalid_command', PYTHONPATH=self.get_pythonpath() + ) + self.assertEqual(status, 1) + self.assertGreater(out, 1) + self.assertEqual(err.splitlines()[-1], + 'error: Invalid command invalid_command') + + def test_unknown_action(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'invalid_action', + PYTHONPATH=self.get_pythonpath() + ) + self.assertEqual(status, 1) + self.assertGreater(out, 1) + self.assertEqual(err.splitlines()[-1], + 'error: Unrecognized action invalid_action') + # TODO test that custom commands don't break --list-commands -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Now_creating_scripts_eve?= =?utf8?q?rytime_when_build=5Fscripts_is_called=2C_as_a_side_effect=2C?= Message-ID: http://hg.python.org/distutils2/rev/99382aafa4c5 changeset: 1329:99382aafa4c5 parent: 1326:bb9ca80afc84 user: Pierre Paul date: Sat May 12 18:14:31 2012 -0400 summary: Now creating scripts everytime when build_scripts is called, as a side effect, --force option has been removed files: CHANGES.txt | 5 + CONTRIBUTORS.txt | 1 + distutils2/command/build_scripts.py | 11 +--- distutils2/tests/test_command_build_scripts.py | 27 ++++++++- 4 files changed, 30 insertions(+), 14 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt --- a/CHANGES.txt +++ b/CHANGES.txt @@ -8,6 +8,11 @@ (and last name initial when needed) are given for each item; see CONTRIBUTORS.txt for full names. Bug numbers refer to http://bugs.python.org/. +1.0a5 - 2012-xx-xx +------------------ + +- #10374 Now creating scripts everytime when build_scripts is called, + as a side effect, --force option has been removed [PierrePaul] 1.0a4 - 2012-03-13 ------------------ diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -42,6 +42,7 @@ - Jeremy Kloth - Amos Latteier - Mathieu Leduc-Hamel +- Pierre Paul Lefebvre - Tshepang Lekhonkhobe - Alain Leufroy - Martin von L?wis diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py --- a/distutils2/command/build_scripts.py +++ b/distutils2/command/build_scripts.py @@ -20,17 +20,12 @@ user_options = [ ('build-dir=', 'd', "directory to build (copy) to"), - ('force', 'f', "forcibly build everything (ignore file timestamps"), ('executable=', 'e', "specify final destination interpreter path"), ] - boolean_options = ['force'] - - def initialize_options(self): self.build_dir = None self.scripts = None - self.force = None self.executable = None self.outfiles = None self.use_2to3 = False @@ -41,7 +36,7 @@ self.set_undefined_options('build', ('build_scripts', 'build_dir'), 'use_2to3', 'use_2to3_fixers', - 'convert_2to3_doctests', 'force', + 'convert_2to3_doctests', 'executable') self.scripts = self.distribution.scripts @@ -69,10 +64,6 @@ outfile = os.path.join(self.build_dir, os.path.basename(script)) outfiles.append(outfile) - if not self.force and not newer(script, outfile): - logger.debug("not copying %s (up-to-date)", script) - continue - # Always open the file, but ignore failures in dry-run mode -- # that way, we'll get accurate feedback if we can read the # script. diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py --- a/distutils2/tests/test_command_build_scripts.py +++ b/distutils2/tests/test_command_build_scripts.py @@ -20,7 +20,7 @@ cmd.finalize_options() - self.assertTrue(cmd.force) + self.assertFalse(cmd.force) self.assertEqual(cmd.build_dir, "/foo/bar") def test_build(self): @@ -38,13 +38,13 @@ for name in expected: self.assertIn(name, built) - def get_build_scripts_cmd(self, target, scripts): + def get_build_scripts_cmd(self, target, scripts, executable=sys.executable): dist = Distribution() dist.scripts = scripts dist.command_obj["build"] = support.DummyCommand( build_scripts=target, - force=True, - executable=sys.executable, + force=False, + executable=executable, use_2to3=False, use_2to3_fixers=None, convert_2to3_doctests=None @@ -105,6 +105,25 @@ for name in expected: self.assertIn(name, built) + def test_build_dir_recreated(self): + source = self.mkdtemp() + target = self.mkdtemp() + self.write_script(source, 'taunt', '#! /usr/bin/python') + + built = os.path.join(target, 'taunt') + + cmd = self.get_build_scripts_cmd(target, [os.path.join(source, 'taunt')], 'pythona') + cmd.finalize_options() + cmd.run() + + self.assertEqual(open(built).readline(), '#!pythona\n') + + cmd = self.get_build_scripts_cmd(target, [os.path.join(source, 'taunt')], 'pythonx') + cmd.finalize_options() + cmd.run() + + self.assertEqual(open(built).readline(), '#!pythonx\n') + def test_suite(): return unittest.makeSuite(BuildScriptsTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:26 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:26 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_Merge_commit?= Message-ID: http://hg.python.org/distutils2/rev/c74e6b9f1b5e changeset: 1330:c74e6b9f1b5e parent: 1329:99382aafa4c5 parent: 1328:478700dd4f6c user: Pierre Paul date: Sat May 12 18:15:19 2012 -0400 summary: Merge commit files: distutils2/metadata.py | 4 +- distutils2/run.py | 17 ++- distutils2/tests/fake_dists/python-pager-readme.rst | 62 ---------- distutils2/tests/test_command_register.py | 15 +- distutils2/tests/test_run.py | 32 +++++- 5 files changed, 54 insertions(+), 76 deletions(-) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -36,7 +36,7 @@ def system_message(self, level, message, *children, **kwargs): self.messages.append((level, message, children, kwargs)) return nodes.system_message(message, level=level, type=self. - levels[level], *children, **kwargs) + levels[level], *children, **kwargs) _HAS_DOCUTILS = True @@ -279,7 +279,7 @@ document.note_source(source_path, -1) try: parser.parse(data, document) - except (AttributeError, TypeError): + except AttributeError: reporter.messages.append((-1, 'Could not finish the parsing.', '', {})) diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -397,8 +397,9 @@ allowed = [action[0] for action in actions] + [None] if self.action not in allowed: - msg = 'Unrecognized action "%s"' % self.action - raise PackagingArgError(msg) + msg = 'Unrecognized action %s' % self.action + self.show_help() + self.exit_with_error_msg(msg) self._set_logger() self.args = args @@ -444,7 +445,8 @@ try: cmd_class = get_command_class(command) except PackagingModuleError, msg: - raise PackagingArgError(msg) + self.show_help() + self.exit_with_error_msg(msg) # XXX We want to push this in distutils2.command # @@ -485,7 +487,11 @@ cmd_class.user_options + help_options) parser.set_negative_aliases(_negative_opt) - args, opts = parser.getopt(args[1:]) + try: + args, opts = parser.getopt(args[1:]) + except PackagingArgError, msg: + self.show_help() + self.exit_with_error_msg(msg) if hasattr(opts, 'help') and opts.help: self._show_command_help(cmd_class) @@ -530,6 +536,9 @@ def show_help(self): self._show_help(self.parser) + def exit_with_error_msg(self, msg): + sys.exit('error: ' + msg.__str__()) + def print_usage(self, parser): parser.set_option_table(global_options) diff --git a/distutils2/tests/fake_dists/python-pager-readme.rst b/distutils2/tests/fake_dists/python-pager-readme.rst deleted file mode 100644 --- a/distutils2/tests/fake_dists/python-pager-readme.rst +++ /dev/null @@ -1,65 +0,0 @@ - -Python module to page screen output and get dimensions -of available console space. - -It is meant to be finally included into standard library -http://bugs.python.org/issue8408 - -| Author: anatoly techtonik -| License: Public Domain (or MIT if a license is required) - - -Status ------- - -0.1 (stable) - - shows content page by page - - allows to get console/terminal dimensions - - works on Windows - - works on Linux - - -API ---- - -..function:: getwidth() - - Return width of available window in characters. If detection fails, - return value of standard width 80. Coordinate of the last character - on a line is -1 from returned value. - - -..function:: getheight() - - Return available window height in characters or 25 if detection fails. - Coordinate of the last line is -1 from returned value. - - -..function:: getch() - - Wait for keypress and return character in a cross-platform way. - Credits: Danny Yoo, Python Cookbook - - -..function:: page(content, [pagecallback=prompt]) - - Output `content` iterable, calling `pagecallback` function after each - page. Default :func:`prompt` callback shows 'Press any key . . . ' prompt - and waits for keypress. - - -References ----------- - -Excellent tutorials for Win32 Console by Adrian Worley -http://www.adrianxw.dk/SoftwareSite/index.html -Console Reference on MSDN -http://msdn.microsoft.com/en-us/library/ms682087%28VS.85%29.aspx - -Public Domain Curses library maintained by William McBrine -http://pdcurses.sourceforge.net/ - -Ioctl (input/output control) introduction from Wikipedia -http://en.wikipedia.org/wiki/Ioctl -Linux Programmer's Manual - ioctls for terminals and serial lines -http://www.kernel.org/doc/man-pages/online/pages/man4/tty_ioctl.4.html diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py --- a/distutils2/tests/test_command_register.py +++ b/distutils2/tests/test_command_register.py @@ -10,7 +10,7 @@ DOCUTILS_SUPPORT = False from distutils2.tests import unittest, support -from distutils2.tests.support import Inputs +from distutils2.tests.support import (Inputs, requires_docutils) from distutils2.command import register as register_module from distutils2.command.register import register from distutils2.errors import PackagingSetupError @@ -253,12 +253,10 @@ self.assertEqual(data['metadata_version'], '1.2') self.assertEqual(data['requires_dist'], ['lxml']) + @requires_docutils def test_register_invalid_long_description(self): - readme_file = os.path.join(os.path.dirname(__file__), - 'fake_dists', 'python-pager-readme.rst') - - # Contains :func: which break the rst format - data = "".join(open(readme_file).readlines()) + # Contains :func: which break the rst format + data = "Default :func:`prompt` callback shows" metadata = {'Home-page': 'xxx', 'Author': 'xxx', 'Author-email': 'xxx', @@ -270,7 +268,10 @@ cmd.strict = True inputs = Inputs('2', 'tarek', 'tarek at ziade.org') register_module.raw_input = inputs - self.assertRaises(PackagingSetupError, cmd.run) + with self.assertRaises(PackagingSetupError) as e: + cmd.run() + self.assertIsNotNone(e) + self.assertIn('func', repr(e.exception)) def test_suite(): return unittest.makeSuite(RegisterTestCase) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -8,7 +8,7 @@ from distutils2.tests import unittest, support from distutils2.run import main -from distutils2.tests.support import assert_python_ok +from distutils2.tests.support import assert_python_ok, assert_python_failure # setup script that uses __file__ setup_using___file__ = """\ @@ -94,6 +94,36 @@ self.assertTrue(build_position, out) self.assertLess(check_position, build_position, out) + def test_unknown_run_option(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'run', 'build', + '--unknown', PYTHONPATH=self.get_pythonpath() + ) + self.assertEqual(status, 1) + self.assertGreater(out, '') + self.assertEqual(err.splitlines()[-1], + 'error: option --unknown not recognized') + + def test_unknown_command(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'run', + 'invalid_command', PYTHONPATH=self.get_pythonpath() + ) + self.assertEqual(status, 1) + self.assertGreater(out, 1) + self.assertEqual(err.splitlines()[-1], + 'error: Invalid command invalid_command') + + def test_unknown_action(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'invalid_action', + PYTHONPATH=self.get_pythonpath() + ) + self.assertEqual(status, 1) + self.assertGreater(out, 1) + self.assertEqual(err.splitlines()[-1], + 'error: Unrecognized action invalid_action') + # TODO test that custom commands don't break --list-commands -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_Branch_merge?= Message-ID: http://hg.python.org/distutils2/rev/5ec9d893725a changeset: 1331:5ec9d893725a parent: 1330:c74e6b9f1b5e parent: 1315:ac1d3c21e155 user: ?ric Araujo date: Tue May 15 18:51:10 2012 -0400 summary: Branch merge files: .hgignore | 18 +++++++----------- 1 files changed, 7 insertions(+), 11 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -1,18 +1,14 @@ syntax: glob *.py[co] -__pycache__/ -*.so -configure.cache +*.swp +bin/ +include/ +lib/ +man/ +Distutils2.egg-info MANIFEST build/ dist/ -_static/ -_build/ -*.swp +.tox/ .coverage -.tox -lib -include -bin nosetests.xml -Distutils2.egg-info -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_missing_changelog_en?= =?utf8?q?tries_for_recent_contributions?= Message-ID: http://hg.python.org/distutils2/rev/10b3c8959421 changeset: 1332:10b3c8959421 user: ?ric Araujo date: Tue May 15 19:06:28 2012 -0400 summary: Add missing changelog entries for recent contributions files: CHANGES.txt | 10 ++++++++-- CONTRIBUTORS.txt | 3 +++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt --- a/CHANGES.txt +++ b/CHANGES.txt @@ -11,8 +11,14 @@ 1.0a5 - 2012-xx-xx ------------------ -- #10374 Now creating scripts everytime when build_scripts is called, - as a side effect, --force option has been removed [PierrePaul] +- #14294: Let Metadata convert setuptools-style requires.txt [preston] +- #14270: Add dest_dir parameter to install functions [mathieu] +- #13166: Add __str__ to database.*Distribution for nicer output [guillaume] +- #13614: Fix register failure with invalid rst in description [julien c, + mathieu, pierre paul, ?ric] +- #13399: Display error message instead of unhandled traceback for missing + actions, commands and options [patrice] +- #10374: Recreate scripts everytime build_scripts is called [pierre paul] 1.0a4 - 2012-03-13 ------------------ diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -20,6 +20,7 @@ - Francisco Mart?n Brugu? - Nicolas Cadou - Godefroid Chapelle +- Julien Courteau - Christophe Combelles - Jason R. Coombs - Pierre-Yves David @@ -31,12 +32,14 @@ - Boris Feld - Andrew Francis - Hallvard B Furuseth +- Patrice Gauthier - Yannick Gingras - Filip Gruszczy?ski - Walker Hale IV - Alexandre Hamelin - Kelsey Hightower - Thomas Holmes +- Preston Holmes - Christian Hudon - Julien Jehannet - Jeremy Kloth -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_unused_names?= Message-ID: http://hg.python.org/distutils2/rev/14c8348f16ba changeset: 1333:14c8348f16ba user: ?ric Araujo date: Tue May 15 19:24:09 2012 -0400 summary: Remove unused names files: distutils2/command/build_scripts.py | 2 +- distutils2/database.py | 2 -- 2 files changed, 1 insertions(+), 3 deletions(-) diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py --- a/distutils2/command/build_scripts.py +++ b/distutils2/command/build_scripts.py @@ -4,7 +4,7 @@ import re from distutils2.command.cmd import Command -from distutils2.util import convert_path, newer +from distutils2.util import convert_path from distutils2 import logger from distutils2.compat import Mixin2to3 from distutils2._backport import sysconfig diff --git a/distutils2/database.py b/distutils2/database.py --- a/distutils2/database.py +++ b/distutils2/database.py @@ -1,7 +1,6 @@ """PEP 376 implementation.""" import os -import re import csv import sys import zipimport @@ -11,7 +10,6 @@ except ImportError: from distutils2._backport.hashlib import md5 -from distutils2 import logger from distutils2.errors import PackagingError from distutils2.version import suggest_normalized_version, VersionPredicate from distutils2.metadata import Metadata -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_docutils_to_tox_deps?= Message-ID: http://hg.python.org/distutils2/rev/3f4051823314 changeset: 1334:3f4051823314 user: ?ric Araujo date: Tue May 15 20:28:51 2012 -0400 summary: Add docutils to tox deps files: tox.ini | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/tox.ini b/tox.ini --- a/tox.ini +++ b/tox.ini @@ -8,8 +8,11 @@ [testenv] commands= + # explicit argument to exclude _backport/tests; + # their coverage is not interesting nosetests --with-xunit distutils2/tests deps= + docutils unittest2 nose @@ -18,5 +21,3 @@ [testenv:py27] basepython=python2.7 -commands= - nosetests --with-xunit distutils2/tests -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_minor_whitespace=2C_?= =?utf8?q?syntax_and_idiom_nits?= Message-ID: http://hg.python.org/distutils2/rev/25b264b1c612 changeset: 1335:25b264b1c612 user: ?ric Araujo date: Wed May 16 00:45:26 2012 -0400 summary: Fix minor whitespace, syntax and idiom nits files: distutils2/database.py | 9 +- distutils2/install.py | 11 +- distutils2/metadata.py | 1 - distutils2/run.py | 21 +- distutils2/tests/test_command_build_scripts.py | 26 ++- distutils2/tests/test_command_register.py | 22 +-- distutils2/tests/test_install.py | 2 - distutils2/tests/test_metadata.py | 4 +- distutils2/tests/test_support.py | 2 +- distutils2/util.py | 63 ++++----- 10 files changed, 74 insertions(+), 87 deletions(-) diff --git a/distutils2/database.py b/distutils2/database.py --- a/distutils2/database.py +++ b/distutils2/database.py @@ -302,7 +302,6 @@ """A :class:`distutils2.metadata.Metadata` instance loaded with the distribution's ``METADATA`` file.""" - def __init__(self, path): self.path = path if _cache_enabled and path in _cache_path_egg: @@ -311,8 +310,6 @@ self.version = self.metadata['Version'] return - - requires = None if path.endswith('.egg'): @@ -347,16 +344,12 @@ raise ValueError('path must end with .egg-info or .egg, got %r' % path) - if requires is not None: + if requires: if self.metadata['Metadata-Version'] == '1.1': # we can't have 1.1 metadata *and* Setuptools requires for field in ('Obsoletes', 'Requires', 'Provides'): if field in self.metadata: del self.metadata[field] - - - - if requires is not None and len(requires)>0: self.metadata['Requires-Dist'] += requires if _cache_enabled: diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -60,8 +60,10 @@ def _run_distutils_install(path, dest): # backward compat: using setuptools or plain-distutils + # FIXME pass dest argument to the command cmd = '%s setup.py install --record=%s' record_file = os.path.join(path, 'RECORD') + # FIXME use subprocess os.system(cmd % (sys.executable, record_file)) if not os.path.exists(record_file): raise ValueError('failed to install') @@ -85,7 +87,7 @@ dist = Distribution() dist.parse_config_files() try: - dist.run_command('install_dist', dict(prefix=(None,dest))) + dist.run_command('install_dist', {'prefix': (None, dest)}) name = dist.metadata['Name'] return database.get_distribution(name) is not None except (IOError, os.error, PackagingError, CCompilerError), msg: @@ -152,15 +154,14 @@ 'distutils': _run_distutils_install} -def _run_install_from_dir(source_dir, destination_dir=None): +def _run_install_from_dir(source_dir, dest_dir=None): old_dir = os.getcwd() os.chdir(source_dir) - install_method = get_install_method(source_dir) - func = install_methods[install_method] try: + install_method = get_install_method(source_dir) func = install_methods[install_method] try: - func(source_dir, destination_dir) + func(source_dir, dest_dir) return True except ValueError, err: # failed to install diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -38,7 +38,6 @@ return nodes.system_message(message, level=level, type=self. levels[level], *children, **kwargs) - _HAS_DOCUTILS = True except ImportError: # docutils is not installed diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -653,17 +653,16 @@ old_level = logger.level old_handlers = list(logger.handlers) try: - try: - dispatcher = Dispatcher(args) - if dispatcher.action is None: - return - return dispatcher() - except KeyboardInterrupt: - logger.info('interrupted') - return 1 - except (IOError, os.error, PackagingError, CCompilerError), exc: - logger.exception(exc) - return 1 + dispatcher = Dispatcher(args) + if dispatcher.action is None: + return + return dispatcher() + except KeyboardInterrupt: + logger.info('interrupted') + return 1 + except (IOError, os.error, PackagingError, CCompilerError), exc: + logger.exception(exc) + return 1 finally: logger.setLevel(old_level) logger.handlers[:] = old_handlers diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py --- a/distutils2/tests/test_command_build_scripts.py +++ b/distutils2/tests/test_command_build_scripts.py @@ -38,7 +38,8 @@ for name in expected: self.assertIn(name, built) - def get_build_scripts_cmd(self, target, scripts, executable=sys.executable): + def get_build_scripts_cmd(self, target, scripts, + executable=sys.executable): dist = Distribution() dist.scripts = scripts dist.command_obj["build"] = support.DummyCommand( @@ -82,10 +83,8 @@ target = self.mkdtemp() expected = self.write_sample_scripts(source) - - cmd = self.get_build_scripts_cmd(target, - [os.path.join(source, fn) - for fn in expected]) + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, fn) for fn in expected]) cmd.finalize_options() # http://bugs.python.org/issue4524 @@ -112,18 +111,25 @@ built = os.path.join(target, 'taunt') - cmd = self.get_build_scripts_cmd(target, [os.path.join(source, 'taunt')], 'pythona') + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, 'taunt')], 'pythona') cmd.finalize_options() cmd.run() - self.assertEqual(open(built).readline(), '#!pythona\n') + with open(built) as fp: + firstline = fp.readline().strip() + self.assertEqual(firstline, '#!pythona') - cmd = self.get_build_scripts_cmd(target, [os.path.join(source, 'taunt')], 'pythonx') + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, 'taunt')], 'pythonx') cmd.finalize_options() cmd.run() - self.assertEqual(open(built).readline(), '#!pythonx\n') - + with open(built) as fp: + firstline = fp.readline().strip() + self.assertEqual(firstline, '#!pythonx') + + def test_suite(): return unittest.makeSuite(BuildScriptsTestCase) diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py --- a/distutils2/tests/test_command_register.py +++ b/distutils2/tests/test_command_register.py @@ -3,14 +3,9 @@ import os import getpass import urllib2 -try: - import docutils - DOCUTILS_SUPPORT = True -except ImportError: - DOCUTILS_SUPPORT = False from distutils2.tests import unittest, support -from distutils2.tests.support import (Inputs, requires_docutils) +from distutils2.tests.support import Inputs, requires_docutils from distutils2.command import register as register_module from distutils2.command.register import register from distutils2.errors import PackagingSetupError @@ -188,7 +183,7 @@ self.assertEqual(headers['Content-length'], '298') self.assertIn('tarek', req.data) - @unittest.skipUnless(DOCUTILS_SUPPORT, 'needs docutils') + @requires_docutils def test_strict(self): # testing the strict option: when on, the register command stops if the # metadata is incomplete or if description contains bad reST @@ -255,14 +250,11 @@ @requires_docutils def test_register_invalid_long_description(self): - # Contains :func: which break the rst format - data = "Default :func:`prompt` callback shows" - + description = ':funkie:`str`' # mimic Sphinx-specific markup metadata = {'Home-page': 'xxx', 'Author': 'xxx', 'Author-email': 'xxx', - 'Name': 'xxx', 'Version': 'xxx'} - - metadata['Description'] = data + 'Name': 'xxx', 'Version': 'xxx', + 'Description': description} cmd = self._get_cmd(metadata) cmd.ensure_finalized() cmd.strict = True @@ -270,8 +262,8 @@ register_module.raw_input = inputs with self.assertRaises(PackagingSetupError) as e: cmd.run() - self.assertIsNotNone(e) - self.assertIn('func', repr(e.exception)) + self.assertIn('funkie', str(e.exception)) + def test_suite(): return unittest.makeSuite(RegisterTestCase) diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py --- a/distutils2/tests/test_install.py +++ b/distutils2/tests/test_install.py @@ -1,7 +1,6 @@ """Tests for the distutils2.install module.""" import os import logging -import sys from tempfile import mkstemp @@ -262,7 +261,6 @@ def test_install_custom_dir(self): dest = self.mkdtemp() - src = self.mkdtemp() project_dir, dist = self.create_dist( name='Spamlib', version='0.1', diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py --- a/distutils2/tests/test_metadata.py +++ b/distutils2/tests/test_metadata.py @@ -383,8 +383,8 @@ def test_description_invalid_rst(self): # make sure bad rst is well handled in the description attribute metadata = Metadata() - description_with_bad_rst = ':funkie:`str`' # Sphinx-specific markup - metadata['description'] = description_with_bad_rst + description = ':funkie:`str`' # mimic Sphinx-specific markup + metadata['description'] = description missing, warnings = metadata.check(restructuredtext=True) warning = warnings[0][1] self.assertIn('funkie', warning) diff --git a/distutils2/tests/test_support.py b/distutils2/tests/test_support.py --- a/distutils2/tests/test_support.py +++ b/distutils2/tests/test_support.py @@ -21,7 +21,7 @@ def runTest(self): # empty method required for the backport pass - + def test_mktempfile(self2): tmpfile = self2.mktempfile() files['test_mktempfile'] = tmpfile.name diff --git a/distutils2/util.py b/distutils2/util.py --- a/distutils2/util.py +++ b/distutils2/util.py @@ -1173,13 +1173,16 @@ f.close() return record_path + def parse_requires(req_path): - """Takes the raw content of a requires.txt file and returns a list of requirements""" + """Create a list of dependencies from a requires.txt file. + + *req_path* must be the path to a setuptools-produced requires.txt file. + """ # reused from Distribute's pkg_resources def yield_lines(strs): - """Yield non-empty/non-comment lines of a ``basestring`` - or sequence""" + """Yield non-empty/non-comment lines of a string or sequence""" if isinstance(strs, basestring): for s in strs.splitlines(): s = s.strip() @@ -1208,37 +1211,35 @@ return None for line in yield_lines(requires): - if line.startswith('['): - logger.warning('extensions in requires.txt are not supported') - break + if line.startswith('['): + logger.warning('extensions in requires.txt are not supported') + break + else: + match = _REQUIREMENT.match(line.strip()) + if not match: + # this happens when we encounter extras; since they + # are written at the end of the file we just exit + break + else: + if match.group('extras'): + # msg = ('extra requirements are not supported ' + # '(used by %r %s)', self.name, self.version) + msg = 'extra requirements are not supported' + logger.warning(msg) + name = match.group('name') + version = None + if match.group('first'): + version = match.group('first') + if match.group('rest'): + version += match.group('rest') + version = version.replace(' ', '') # trim spaces + if version is None: + reqs.append(name) else: - match = _REQUIREMENT.match(line.strip()) - if not match: - # this happens when we encounter extras; since they - # are written at the end of the file we just exit - break - else: - if match.group('extras'): - # msg = ('extra requirements are not supported ' - # '(used by %r %s)', self.name, self.version) - msg = 'extra requirements are not supported' - logger.warning(msg) - name = match.group('name') - version = None - if match.group('first'): - version = match.group('first') - if match.group('rest'): - version += match.group('rest') - version = version.replace(' ', '') # trim spaces - if version is None: - reqs.append(name) - else: - reqs.append('%s (%s)' % (name, version)) + reqs.append('%s (%s)' % (name, version)) return reqs - - def egginfo_to_distinfo(record_file, installer=_DEFAULT_INSTALLER, requested=False, remove_egginfo=False): """Create files and directories required for PEP 376 @@ -1277,8 +1278,6 @@ metadata['Requires-Dist'] = requires metadata.write(metadata_path) - - installer_path = distinfo['installer_path'] logger.info('creating %s', installer_path) f = open(installer_path, 'w') -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Harmonize_error_messages?= =?utf8?q?_and_add_one_missing_test_for_=2313339?= Message-ID: http://hg.python.org/distutils2/rev/91ac9c36f09e changeset: 1336:91ac9c36f09e user: ?ric Araujo date: Wed May 16 00:51:01 2012 -0400 summary: Harmonize error messages and add one missing test for #13339 files: distutils2/run.py | 13 +++----- distutils2/tests/test_run.py | 33 +++++++++++++++-------- 2 files changed, 26 insertions(+), 20 deletions(-) diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -387,6 +387,7 @@ self.parser.set_negative_aliases(negative_opt) # FIXME this parses everything, including command options (e.g. "run # build -i" errors with "option -i not recognized") + # FIXME unknown options are not detected args = self.parser.getopt(args=args, object=self) # if first arg is "run", we have some commands @@ -397,9 +398,8 @@ allowed = [action[0] for action in actions] + [None] if self.action not in allowed: - msg = 'Unrecognized action %s' % self.action self.show_help() - self.exit_with_error_msg(msg) + sys.exit('error: action %r not recognized' % self.action) self._set_logger() self.args = args @@ -436,7 +436,7 @@ # Pull the current command from the head of the command line command = args[0] if not command_re.match(command): - raise SystemExit("invalid command name %r" % (command,)) + sys.exit('error: invalid command name %r' % command) self.commands.append(command) # Dig up the command class that implements this command, so we @@ -446,7 +446,7 @@ cmd_class = get_command_class(command) except PackagingModuleError, msg: self.show_help() - self.exit_with_error_msg(msg) + sys.exit('error: command %r not recognized' % command) # XXX We want to push this in distutils2.command # @@ -491,7 +491,7 @@ args, opts = parser.getopt(args[1:]) except PackagingArgError, msg: self.show_help() - self.exit_with_error_msg(msg) + sys.exit('error: %s' % msg) if hasattr(opts, 'help') and opts.help: self._show_command_help(cmd_class) @@ -536,9 +536,6 @@ def show_help(self): self._show_help(self.parser) - def exit_with_error_msg(self, msg): - sys.exit('error: ' + msg.__str__()) - def print_usage(self, parser): parser.set_option_table(global_options) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -94,37 +94,46 @@ self.assertTrue(build_position, out) self.assertLess(check_position, build_position, out) - def test_unknown_run_option(self): + # TODO test that custom commands don't break --list-commands + + def test_unknown_command_option(self): status, out, err = assert_python_failure( '-c', 'from distutils2.run import main; main()', 'run', 'build', - '--unknown', PYTHONPATH=self.get_pythonpath() - ) + '--unknown', PYTHONPATH=self.get_pythonpath()) self.assertEqual(status, 1) self.assertGreater(out, '') + # sadly this message comes straight from the getopt module and can't be + # modified to use repr instead of str for the unknown option; to be + # changed when the command line parsers are replaced by something clean self.assertEqual(err.splitlines()[-1], - 'error: option --unknown not recognized') + 'error: option --unknown not recognized') + + def test_invalid_command(self): + status, out, err = assert_python_failure( + '-c', 'from distutils2.run import main; main()', 'run', + 'com#mand', PYTHONPATH=self.get_pythonpath()) + self.assertEqual(status, 1) + self.assertGreater(out, 1) + self.assertEqual(err.splitlines()[-1], + "error: invalid command name 'com#mand'") def test_unknown_command(self): status, out, err = assert_python_failure( '-c', 'from distutils2.run import main; main()', 'run', - 'invalid_command', PYTHONPATH=self.get_pythonpath() - ) + 'invalid_command', PYTHONPATH=self.get_pythonpath()) self.assertEqual(status, 1) self.assertGreater(out, 1) self.assertEqual(err.splitlines()[-1], - 'error: Invalid command invalid_command') + "error: command 'invalid_command' not recognized") def test_unknown_action(self): status, out, err = assert_python_failure( '-c', 'from distutils2.run import main; main()', 'invalid_action', - PYTHONPATH=self.get_pythonpath() - ) + PYTHONPATH=self.get_pythonpath()) self.assertEqual(status, 1) self.assertGreater(out, 1) self.assertEqual(err.splitlines()[-1], - 'error: Unrecognized action invalid_action') - - # TODO test that custom commands don't break --list-commands + "error: action 'invalid_action' not recognized") def test_suite(): -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Now_deleting_build-scrip?= =?utf8?q?ts_directory_before_creating_it=2E_No_more_leftovers=2E?= Message-ID: http://hg.python.org/distutils2/rev/463d3014ee4a changeset: 1337:463d3014ee4a parent: 1330:c74e6b9f1b5e user: Pierre Paul date: Mon May 14 08:36:06 2012 -0400 summary: Now deleting build-scripts directory before creating it. No more leftovers. files: distutils2/command/build_scripts.py | 1 + distutils2/command/cmd.py | 16 ++++- distutils2/tests/test_command_build_scripts.py | 35 ++++++++++ 3 files changed, 51 insertions(+), 1 deletions(-) diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py --- a/distutils2/command/build_scripts.py +++ b/distutils2/command/build_scripts.py @@ -56,6 +56,7 @@ ie. starts with "\#!" and contains "python"), then adjust the first line to refer to the current Python interpreter as we copy. """ + self.rmpath(self.build_dir) self.mkpath(self.build_dir) outfiles = [] for script in self.scripts: diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py --- a/distutils2/command/cmd.py +++ b/distutils2/command/cmd.py @@ -5,7 +5,7 @@ from distutils2 import util from distutils2 import logger from distutils2.errors import PackagingOptionError -from distutils2._backport.shutil import copyfile, move, make_archive +from distutils2._backport.shutil import copyfile, move, make_archive, rmtree class Command(object): @@ -365,6 +365,20 @@ return os.makedirs(name, mode) + def rmpath(self, name, dry_run=None): + if dry_run is None: + dry_run = self.dry_run + name = os.path.normpath(name) + if not os.path.isdir(name) or name == '': + return + if dry_run: + head = '' + for part in name.split(os.sep): + logger.info("removing directory %s%s", head, part) + head += part + os.sep + return + rmtree(name) + def copy_file(self, infile, outfile, preserve_mode=True, preserve_times=True, link=None, level=1): """Copy a file respecting dry-run and force flags. diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py --- a/distutils2/tests/test_command_build_scripts.py +++ b/distutils2/tests/test_command_build_scripts.py @@ -123,6 +123,41 @@ cmd.run() self.assertEqual(open(built).readline(), '#!pythonx\n') + + def test_build_old_scripts_deleted(self): + source = self.mkdtemp() + + expected = [] + expected.append("script1.py") + self.write_script(source, "script1.py", + ("#! /usr/bin/env python2.3\n" + "# bogus script w/ Python sh-bang\n" + "pass\n")) + expected.append("script2.py") + self.write_script(source, "script2.py", + ("#!/usr/bin/python\n" + "# bogus script w/ Python sh-bang\n" + "pass\n")) + + target = self.mkdtemp() + cmd = self.get_build_scripts_cmd(target, + [os.path.join(source, fn) + for fn in expected]) + cmd.finalize_options() + cmd.run() + + built = os.listdir(target) + for name in expected: + self.assertIn(name, built) + + cmd = self.get_build_scripts_cmd(target, + [os.path.join(source, 'script1.py')]) + cmd.finalize_options() + cmd.run() + + built = os.listdir(target) + self.assertIn('script1.py', built) + self.assertNotIn('script2.py', built) def test_suite(): return unittest.makeSuite(BuildScriptsTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_=2313399_-_some_tweaking?= Message-ID: http://hg.python.org/distutils2/rev/fcfa635db609 changeset: 1338:fcfa635db609 parent: 1328:478700dd4f6c user: Patrice Gauthier date: Mon May 14 15:43:42 2012 -0400 summary: #13399 - some tweaking files: distutils2/run.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -436,6 +436,7 @@ # Pull the current command from the head of the command line command = args[0] if not command_re.match(command): + self.show_help() raise SystemExit("invalid command name %r" % (command,)) self.commands.append(command) @@ -537,7 +538,7 @@ self._show_help(self.parser) def exit_with_error_msg(self, msg): - sys.exit('error: ' + msg.__str__()) + sys.exit('error: %s ' % msg) def print_usage(self, parser): parser.set_option_table(global_options) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_merge_commit?= Message-ID: http://hg.python.org/distutils2/rev/4087d11330a1 changeset: 1339:4087d11330a1 parent: 1338:fcfa635db609 parent: 1337:463d3014ee4a user: Patrice Gauthier date: Mon May 14 16:01:47 2012 -0400 summary: merge commit files: CHANGES.txt | 5 + CONTRIBUTORS.txt | 1 + distutils2/command/build_scripts.py | 12 +- distutils2/command/cmd.py | 16 ++- distutils2/tests/test_command_build_scripts.py | 62 +++++++++- 5 files changed, 81 insertions(+), 15 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt --- a/CHANGES.txt +++ b/CHANGES.txt @@ -8,6 +8,11 @@ (and last name initial when needed) are given for each item; see CONTRIBUTORS.txt for full names. Bug numbers refer to http://bugs.python.org/. +1.0a5 - 2012-xx-xx +------------------ + +- #10374 Now creating scripts everytime when build_scripts is called, + as a side effect, --force option has been removed [PierrePaul] 1.0a4 - 2012-03-13 ------------------ diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -42,6 +42,7 @@ - Jeremy Kloth - Amos Latteier - Mathieu Leduc-Hamel +- Pierre Paul Lefebvre - Tshepang Lekhonkhobe - Alain Leufroy - Martin von L?wis diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py --- a/distutils2/command/build_scripts.py +++ b/distutils2/command/build_scripts.py @@ -20,17 +20,12 @@ user_options = [ ('build-dir=', 'd', "directory to build (copy) to"), - ('force', 'f', "forcibly build everything (ignore file timestamps"), ('executable=', 'e', "specify final destination interpreter path"), ] - boolean_options = ['force'] - - def initialize_options(self): self.build_dir = None self.scripts = None - self.force = None self.executable = None self.outfiles = None self.use_2to3 = False @@ -41,7 +36,7 @@ self.set_undefined_options('build', ('build_scripts', 'build_dir'), 'use_2to3', 'use_2to3_fixers', - 'convert_2to3_doctests', 'force', + 'convert_2to3_doctests', 'executable') self.scripts = self.distribution.scripts @@ -61,6 +56,7 @@ ie. starts with "\#!" and contains "python"), then adjust the first line to refer to the current Python interpreter as we copy. """ + self.rmpath(self.build_dir) self.mkpath(self.build_dir) outfiles = [] for script in self.scripts: @@ -69,10 +65,6 @@ outfile = os.path.join(self.build_dir, os.path.basename(script)) outfiles.append(outfile) - if not self.force and not newer(script, outfile): - logger.debug("not copying %s (up-to-date)", script) - continue - # Always open the file, but ignore failures in dry-run mode -- # that way, we'll get accurate feedback if we can read the # script. diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py --- a/distutils2/command/cmd.py +++ b/distutils2/command/cmd.py @@ -5,7 +5,7 @@ from distutils2 import util from distutils2 import logger from distutils2.errors import PackagingOptionError -from distutils2._backport.shutil import copyfile, move, make_archive +from distutils2._backport.shutil import copyfile, move, make_archive, rmtree class Command(object): @@ -365,6 +365,20 @@ return os.makedirs(name, mode) + def rmpath(self, name, dry_run=None): + if dry_run is None: + dry_run = self.dry_run + name = os.path.normpath(name) + if not os.path.isdir(name) or name == '': + return + if dry_run: + head = '' + for part in name.split(os.sep): + logger.info("removing directory %s%s", head, part) + head += part + os.sep + return + rmtree(name) + def copy_file(self, infile, outfile, preserve_mode=True, preserve_times=True, link=None, level=1): """Copy a file respecting dry-run and force flags. diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py --- a/distutils2/tests/test_command_build_scripts.py +++ b/distutils2/tests/test_command_build_scripts.py @@ -20,7 +20,7 @@ cmd.finalize_options() - self.assertTrue(cmd.force) + self.assertFalse(cmd.force) self.assertEqual(cmd.build_dir, "/foo/bar") def test_build(self): @@ -38,13 +38,13 @@ for name in expected: self.assertIn(name, built) - def get_build_scripts_cmd(self, target, scripts): + def get_build_scripts_cmd(self, target, scripts, executable=sys.executable): dist = Distribution() dist.scripts = scripts dist.command_obj["build"] = support.DummyCommand( build_scripts=target, - force=True, - executable=sys.executable, + force=False, + executable=executable, use_2to3=False, use_2to3_fixers=None, convert_2to3_doctests=None @@ -105,6 +105,60 @@ for name in expected: self.assertIn(name, built) + def test_build_dir_recreated(self): + source = self.mkdtemp() + target = self.mkdtemp() + self.write_script(source, 'taunt', '#! /usr/bin/python') + + built = os.path.join(target, 'taunt') + + cmd = self.get_build_scripts_cmd(target, [os.path.join(source, 'taunt')], 'pythona') + cmd.finalize_options() + cmd.run() + + self.assertEqual(open(built).readline(), '#!pythona\n') + + cmd = self.get_build_scripts_cmd(target, [os.path.join(source, 'taunt')], 'pythonx') + cmd.finalize_options() + cmd.run() + + self.assertEqual(open(built).readline(), '#!pythonx\n') + + def test_build_old_scripts_deleted(self): + source = self.mkdtemp() + + expected = [] + expected.append("script1.py") + self.write_script(source, "script1.py", + ("#! /usr/bin/env python2.3\n" + "# bogus script w/ Python sh-bang\n" + "pass\n")) + expected.append("script2.py") + self.write_script(source, "script2.py", + ("#!/usr/bin/python\n" + "# bogus script w/ Python sh-bang\n" + "pass\n")) + + target = self.mkdtemp() + cmd = self.get_build_scripts_cmd(target, + [os.path.join(source, fn) + for fn in expected]) + cmd.finalize_options() + cmd.run() + + built = os.listdir(target) + for name in expected: + self.assertIn(name, built) + + cmd = self.get_build_scripts_cmd(target, + [os.path.join(source, 'script1.py')]) + cmd.finalize_options() + cmd.run() + + built = os.listdir(target) + self.assertIn('script1.py', built) + self.assertNotIn('script2.py', built) + def test_suite(): return unittest.makeSuite(BuildScriptsTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 07:07:27 2012 From: python-checkins at python.org (eric.araujo) Date: Wed, 16 May 2012 07:07:27 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_defau?= =?utf8?q?lt=29=3A_Merge_further_changes_by_Montreal_sprinters?= Message-ID: http://hg.python.org/distutils2/rev/8d9fbdbd65b0 changeset: 1340:8d9fbdbd65b0 parent: 1336:91ac9c36f09e parent: 1339:4087d11330a1 user: ?ric Araujo date: Wed May 16 01:06:53 2012 -0400 summary: Merge further changes by Montreal sprinters files: distutils2/command/build_scripts.py | 2 + distutils2/command/cmd.py | 16 +++++- distutils2/run.py | 3 +- distutils2/tests/test_command_build_scripts.py | 30 ++++++++++ 4 files changed, 49 insertions(+), 2 deletions(-) diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py --- a/distutils2/command/build_scripts.py +++ b/distutils2/command/build_scripts.py @@ -56,6 +56,8 @@ ie. starts with "\#!" and contains "python"), then adjust the first line to refer to the current Python interpreter as we copy. """ + # XXX use self.execute(shutil.rmtree, ...) + self.rmpath(self.build_dir) self.mkpath(self.build_dir) outfiles = [] for script in self.scripts: diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py --- a/distutils2/command/cmd.py +++ b/distutils2/command/cmd.py @@ -5,7 +5,7 @@ from distutils2 import util from distutils2 import logger from distutils2.errors import PackagingOptionError -from distutils2._backport.shutil import copyfile, move, make_archive +from distutils2._backport.shutil import copyfile, move, make_archive, rmtree class Command(object): @@ -365,6 +365,20 @@ return os.makedirs(name, mode) + def rmpath(self, name, dry_run=None): + if dry_run is None: + dry_run = self.dry_run + name = os.path.normpath(name) + if not os.path.isdir(name) or name == '': + return + if dry_run: + head = '' + for part in name.split(os.sep): + logger.info("removing directory %s%s", head, part) + head += part + os.sep + return + rmtree(name) + def copy_file(self, infile, outfile, preserve_mode=True, preserve_times=True, link=None, level=1): """Copy a file respecting dry-run and force flags. diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -436,6 +436,7 @@ # Pull the current command from the head of the command line command = args[0] if not command_re.match(command): + self.show_help() # TODO list only commands, not actions sys.exit('error: invalid command name %r' % command) self.commands.append(command) @@ -445,7 +446,7 @@ try: cmd_class = get_command_class(command) except PackagingModuleError, msg: - self.show_help() + self.show_help() # TODO list only commands, not actions sys.exit('error: command %r not recognized' % command) # XXX We want to push this in distutils2.command diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py --- a/distutils2/tests/test_command_build_scripts.py +++ b/distutils2/tests/test_command_build_scripts.py @@ -129,6 +129,36 @@ firstline = fp.readline().strip() self.assertEqual(firstline, '#!pythonx') + def test_build_old_scripts_deleted(self): + source = self.mkdtemp() + target = self.mkdtemp() + + expected = ['script1.py', 'script2.py'] + self.write_script(source, "script1.py", + ("#! /usr/bin/env python2.3\n" + "pass\n")) + self.write_script(source, "script2.py", + ("#!/usr/bin/python\n" + "pass\n")) + + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, fn) for fn in expected]) + cmd.finalize_options() + cmd.run() + + built = sorted(os.listdir(target)) + self.assertEqual(built, expected) + + # if we run build_scripts with a different list of scripts, the old + # ones used to be left over in the build directory and installed anyway + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, 'script1.py')]) + cmd.finalize_options() + cmd.run() + + built = os.listdir(target) + self.assertEqual(built, ['script1.py']) + def test_suite(): return unittest.makeSuite(BuildScriptsTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 10:49:34 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 10:49:34 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Cleanup_so_subs?= =?utf8?q?equent_tests_won=27t_fail=2E_Needs_to_be_moved_into_a_support?= Message-ID: http://hg.python.org/cpython/rev/57936bd91bcf changeset: 76983:57936bd91bcf branch: 3.2 parent: 76981:d1fbfd9af5c5 user: Eric V. Smith date: Wed May 16 04:48:04 2012 -0400 summary: Cleanup so subsequent tests won't fail. Needs to be moved into a support routine (see 14715). files: Lib/test/test_pkgutil.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -165,6 +165,9 @@ def tearDown(self): del sys.path[0] del sys.path[0] + del sys.modules['foo'] + del sys.modules['foo.bar'] + del sys.modules['foo.baz'] def test_simple(self): self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 10:49:36 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 10:49:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/9b66caf0e217 changeset: 76984:9b66caf0e217 parent: 76982:4ec95207281c parent: 76983:57936bd91bcf user: Eric V. Smith date: Wed May 16 04:49:22 2012 -0400 summary: Merge from 3.2. files: Lib/test/test_pkgutil.py | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -165,6 +165,9 @@ def tearDown(self): del sys.path[0] del sys.path[0] + del sys.modules['foo'] + del sys.modules['foo.bar'] + del sys.modules['foo.baz'] def test_simple(self): self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 11:24:28 2012 From: python-checkins at python.org (hynek.schlawack) Date: Wed, 16 May 2012 11:24:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314809=3A_Add_HTTP_status?= =?utf8?q?_codes_from_RFC_6585_to_http=2Eserver_and_http=2Eclient?= Message-ID: http://hg.python.org/cpython/rev/981aabe6ea2f changeset: 76985:981aabe6ea2f user: Hynek Schlawack date: Wed May 16 09:51:07 2012 +0200 summary: #14809: Add HTTP status codes from RFC 6585 to http.server and http.client Patch by EungJun Yi. files: Doc/library/http.client.rst | 15 +++++++++++++++ Lib/http/client.py | 8 ++++++++ Lib/http/server.py | 10 +++++++++- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 5 files changed, 36 insertions(+), 1 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -339,6 +339,15 @@ | :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | | | | :rfc:`2817`, Section 6 | +------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 3 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 4 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 5 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ | :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | | | | 10.5.1 | | | | `_ | @@ -369,6 +378,12 @@ | :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | | | | :rfc:`2774`, Section 7 | +------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 6 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ + + .. versionchanged:: 3.3 + Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. .. data:: responses diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -141,6 +141,9 @@ LOCKED = 423 FAILED_DEPENDENCY = 424 UPGRADE_REQUIRED = 426 +PRECONDITION_REQUIRED = 428 +TOO_MANY_REQUESTS = 429 +REQUEST_HEADER_FIELDS_TOO_LARGE = 431 # server error INTERNAL_SERVER_ERROR = 500 @@ -151,6 +154,7 @@ HTTP_VERSION_NOT_SUPPORTED = 505 INSUFFICIENT_STORAGE = 507 NOT_EXTENDED = 510 +NETWORK_AUTHENTICATION_REQUIRED = 511 # Mapping status codes to official W3C names responses = { @@ -192,6 +196,9 @@ 415: 'Unsupported Media Type', 416: 'Requested Range Not Satisfiable', 417: 'Expectation Failed', + 428: 'Precondition Required', + 429: 'Too Many Requests', + 431: 'Request Header Fields Too Large', 500: 'Internal Server Error', 501: 'Not Implemented', @@ -199,6 +206,7 @@ 503: 'Service Unavailable', 504: 'Gateway Timeout', 505: 'HTTP Version Not Supported', + 511: 'Network Authentication Required', } # maximal amount of data to read at one time in _safe_read diff --git a/Lib/http/server.py b/Lib/http/server.py --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -573,7 +573,7 @@ # Table mapping response codes to messages; entries have the # form {code: (shortmessage, longmessage)}. - # See RFC 2616. + # See RFC 2616 and 6585. responses = { 100: ('Continue', 'Request received, please continue'), 101: ('Switching Protocols', @@ -628,6 +628,12 @@ 'Cannot satisfy request range.'), 417: ('Expectation Failed', 'Expect condition could not be satisfied.'), + 428: ('Precondition Required', + 'The origin server requires the request to be conditional.'), + 429: ('Too Many Requests', 'The user has sent too many requests ' + 'in a given amount of time ("rate limiting").'), + 431: ('Request Header Fields Too Large', 'The server is unwilling to ' + 'process the request because its header fields are too large.'), 500: ('Internal Server Error', 'Server got itself in trouble'), 501: ('Not Implemented', @@ -638,6 +644,8 @@ 504: ('Gateway Timeout', 'The gateway server did not receive a timely response'), 505: ('HTTP Version Not Supported', 'Cannot fulfill request.'), + 511: ('Network Authentication Required', + 'The client needs to authenticate to gain network access.'), } diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1129,6 +1129,7 @@ Hirokazu Yamamoto Ka-Ping Yee Jason Yeo +EungJun Yi Bob Yodlowski Danny Yoo George Yoshida diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -34,6 +34,9 @@ Library ------- +- Issue #14809: Add HTTP status codes introduced by RFC 6585 to http.server + and http.client. Patch by EungJun Yi. + - Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when accessing the Tk clipboard. Modify clipboad_get() to first request type UTF8_STRING when no specific type is requested in an X11 windowing -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 11:33:39 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 11:33:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_=2Ehgignore_for_new_?= =?utf8?q?MSVC_files?= Message-ID: http://hg.python.org/cpython/rev/086b4eabc23e changeset: 76986:086b4eabc23e user: Antoine Pitrou date: Wed May 16 11:31:13 2012 +0200 summary: Update .hgignore for new MSVC files files: .hgignore | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -56,6 +56,9 @@ PC/pythonnt_rc*.h PC/*.obj PC/*.exe +PC/*/*.user +PC/*/*.ncb +PC/*/*.suo PCbuild/*.exe PCbuild/*.dll PCbuild/*.pdb @@ -69,6 +72,7 @@ PCbuild/*.*sdf PCbuild/Win32-temp-* PCbuild/x64-temp-* +BuildLog.htm __pycache__ Modules/_testembed .coverage -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 11:35:30 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 11:35:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314732=3A_The_=5Fcs?= =?utf8?q?v_module_now_uses_PEP_3121_module_initialization=2E?= Message-ID: http://hg.python.org/cpython/rev/2496602a56e5 changeset: 76987:2496602a56e5 user: Antoine Pitrou date: Wed May 16 11:33:08 2012 +0200 summary: Issue #14732: The _csv module now uses PEP 3121 module initialization. Patch by Robin Schreiber. files: Misc/ACKS | 1 + Misc/NEWS | 3 + Modules/_csv.c | 105 ++++++++++++++++++++++++------------ 3 files changed, 74 insertions(+), 35 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -919,6 +919,7 @@ Michael Schneider Peter Schneider-Kamp Arvin Schnell +Robin Schreiber Chad J. Schroeder Sam Schulenburg Stefan Schwarzer diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -34,6 +34,9 @@ Library ------- +- Issue #14732: The _csv module now uses PEP 3121 module initialization. + Patch by Robin Schreiber. + - Issue #14809: Add HTTP status codes introduced by RFC 6585 to http.server and http.client. Patch by EungJun Yi. diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -16,9 +16,39 @@ #define IS_BASESTRING(o) \ PyUnicode_Check(o) -static PyObject *error_obj; /* CSV exception */ -static PyObject *dialects; /* Dialect registry */ -static long field_limit = 128 * 1024; /* max parsed field size */ +typedef struct { + PyObject *error_obj; /* CSV exception */ + PyObject *dialects; /* Dialect registry */ + long field_limit; /* max parsed field size */ +} _csvstate; + +#define _csvstate(o) ((_csvstate *)PyModule_GetState(o)) + +static int +_csv_clear(PyObject *m) +{ + Py_CLEAR(_csvstate(m)->error_obj); + Py_CLEAR(_csvstate(m)->dialects); + return 0; +} + +static int +_csv_traverse(PyObject *m, visitproc visit, void *arg) +{ + Py_VISIT(_csvstate(m)->error_obj); + Py_VISIT(_csvstate(m)->dialects); + return 0; +} + +static void +_csv_free(void *m) +{ + _csv_clear((PyObject *)m); +} + +static struct PyModuleDef _csvmodule; + +#define _csvstate_global ((_csvstate *)PyModule_GetState(PyState_FindModule(&_csvmodule))) typedef enum { START_RECORD, START_FIELD, ESCAPED_CHAR, IN_FIELD, @@ -103,10 +133,10 @@ { PyObject *dialect_obj; - dialect_obj = PyDict_GetItem(dialects, name_obj); + dialect_obj = PyDict_GetItem(_csvstate_global->dialects, name_obj); if (dialect_obj == NULL) { if (!PyErr_Occurred()) - PyErr_Format(error_obj, "unknown dialect"); + PyErr_Format(_csvstate_global->error_obj, "unknown dialect"); } else Py_INCREF(dialect_obj); @@ -544,9 +574,9 @@ static int parse_add_char(ReaderObj *self, Py_UCS4 c) { - if (self->field_len >= field_limit) { - PyErr_Format(error_obj, "field larger than field limit (%ld)", - field_limit); + if (self->field_len >= _csvstate_global->field_limit) { + PyErr_Format(_csvstate_global->error_obj, "field larger than field limit (%ld)", + _csvstate_global->field_limit); return -1; } if (self->field_len == self->field_size && !parse_grow_buff(self)) @@ -703,7 +733,7 @@ } else { /* illegal */ - PyErr_Format(error_obj, "'%c' expected after '%c'", + PyErr_Format(_csvstate_global->error_obj, "'%c' expected after '%c'", dialect->delimiter, dialect->quotechar); return -1; @@ -716,7 +746,7 @@ else if (c == '\0') self->state = START_RECORD; else { - PyErr_Format(error_obj, "new-line character seen in unquoted field - do you need to open the file in universal-newline mode?"); + PyErr_Format(_csvstate_global->error_obj, "new-line character seen in unquoted field - do you need to open the file in universal-newline mode?"); return -1; } break; @@ -755,12 +785,12 @@ if (lineobj == NULL) { /* End of input OR exception */ if (!PyErr_Occurred() && self->field_len != 0) - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "newline inside string"); return NULL; } if (!PyUnicode_Check(lineobj)) { - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "iterator should return strings, " "not %.200s " "(did you open the file in text mode?)", @@ -778,7 +808,7 @@ c = PyUnicode_READ(kind, data, pos); if (c == '\0') { Py_DECREF(lineobj); - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "line contains NULL byte"); goto err; } @@ -994,7 +1024,7 @@ } if (want_escape) { if (!dialect->escapechar) { - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "need to escape, but no escapechar set"); return -1; } @@ -1010,7 +1040,7 @@ */ if (i == 0 && quote_empty) { if (dialect->quoting == QUOTE_NONE) { - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "single empty field record must be quoted"); return -1; } @@ -1127,7 +1157,7 @@ PyObject *line, *result; if (!PySequence_Check(seq)) - return PyErr_Format(error_obj, "sequence expected"); + return PyErr_Format(_csvstate_global->error_obj, "sequence expected"); len = PySequence_Length(seq); if (len < 0) @@ -1353,7 +1383,7 @@ static PyObject * csv_list_dialects(PyObject *module, PyObject *args) { - return PyDict_Keys(dialects); + return PyDict_Keys(_csvstate_global->dialects); } static PyObject * @@ -1372,7 +1402,7 @@ dialect = _call_dialect(dialect_obj, kwargs); if (dialect == NULL) return NULL; - if (PyDict_SetItem(dialects, name_obj, dialect) < 0) { + if (PyDict_SetItem(_csvstate_global->dialects, name_obj, dialect) < 0) { Py_DECREF(dialect); return NULL; } @@ -1384,8 +1414,8 @@ static PyObject * csv_unregister_dialect(PyObject *module, PyObject *name_obj) { - if (PyDict_DelItem(dialects, name_obj) < 0) - return PyErr_Format(error_obj, "unknown dialect"); + if (PyDict_DelItem(_csvstate_global->dialects, name_obj) < 0) + return PyErr_Format(_csvstate_global->error_obj, "unknown dialect"); Py_INCREF(Py_None); return Py_None; } @@ -1400,7 +1430,7 @@ csv_field_size_limit(PyObject *module, PyObject *args) { PyObject *new_limit = NULL; - long old_limit = field_limit; + long old_limit = _csvstate_global->field_limit; if (!PyArg_UnpackTuple(args, "field_size_limit", 0, 1, &new_limit)) return NULL; @@ -1410,9 +1440,9 @@ "limit must be an integer"); return NULL; } - field_limit = PyLong_AsLong(new_limit); - if (field_limit == -1 && PyErr_Occurred()) { - field_limit = old_limit; + _csvstate_global->field_limit = PyLong_AsLong(new_limit); + if (_csvstate_global->field_limit == -1 && PyErr_Occurred()) { + _csvstate_global->field_limit = old_limit; return NULL; } } @@ -1551,17 +1581,16 @@ { NULL, NULL } }; - static struct PyModuleDef _csvmodule = { PyModuleDef_HEAD_INIT, "_csv", csv_module_doc, - -1, + sizeof(_csvstate), csv_methods, NULL, - NULL, - NULL, - NULL + _csv_traverse, + _csv_clear, + _csv_free }; PyMODINIT_FUNC @@ -1589,11 +1618,16 @@ MODULE_VERSION) == -1) return NULL; + /* Set the field limit */ + _csvstate(module)->field_limit = 128 * 1024; + /* Do I still need to add this var to the Module Dict? */ + /* Add _dialects dictionary */ - dialects = PyDict_New(); - if (dialects == NULL) + _csvstate(module)->dialects = PyDict_New(); + if (_csvstate(module)->dialects == NULL) return NULL; - if (PyModule_AddObject(module, "_dialects", dialects)) + Py_INCREF(_csvstate(module)->dialects); + if (PyModule_AddObject(module, "_dialects", _csvstate(module)->dialects)) return NULL; /* Add quote styles into dictionary */ @@ -1609,9 +1643,10 @@ return NULL; /* Add the CSV exception object to the module. */ - error_obj = PyErr_NewException("_csv.Error", NULL, NULL); - if (error_obj == NULL) + _csvstate(module)->error_obj = PyErr_NewException("_csv.Error", NULL, NULL); + if (_csvstate(module)->error_obj == NULL) return NULL; - PyModule_AddObject(module, "Error", error_obj); + Py_INCREF(_csvstate(module)->error_obj); + PyModule_AddObject(module, "Error", _csvstate(module)->error_obj); return module; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 12:54:18 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 12:54:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_build_failure=2E?= Message-ID: http://hg.python.org/cpython/rev/1ecd10260649 changeset: 76988:1ecd10260649 user: Antoine Pitrou date: Wed May 16 12:51:55 2012 +0200 summary: Fix build failure. files: Objects/exceptions.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Objects/exceptions.c b/Objects/exceptions.c --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -349,7 +349,8 @@ static struct PyMemberDef BaseException_members[] = { {"__suppress_context__", T_BOOL, - offsetof(PyBaseExceptionObject, suppress_context)} + offsetof(PyBaseExceptionObject, suppress_context)}, + {NULL} }; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 13:00:28 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 13:00:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_=22warning=3A_no_newl?= =?utf8?q?ine_at_end_of_file=22_in_importlib=2Eh=2E?= Message-ID: http://hg.python.org/cpython/rev/6e5c517da087 changeset: 76989:6e5c517da087 user: Antoine Pitrou date: Wed May 16 12:58:04 2012 +0200 summary: Avoid "warning: no newline at end of file" in importlib.h. files: Python/freeze_importlib.py | 2 ++ Python/importlib.h | Bin 2 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Python/freeze_importlib.py b/Python/freeze_importlib.py --- a/Python/freeze_importlib.py +++ b/Python/freeze_importlib.py @@ -25,6 +25,8 @@ with open(output_path, 'w', encoding='utf-8') as output_file: output_file.write('\n'.join(lines)) output_file.write('/* Mercurial binary marker: \x00 */') + # Avoid a compiler warning for lack of EOL + output_file.write('\n') if __name__ == '__main__': diff --git a/Python/importlib.h b/Python/importlib.h index 0beeb595dbc38d821fb4f6de0981347f3983420a..cf5619a6c4b0587815b87145eae5bf212ec7e5f1 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 14:22:49 2012 From: python-checkins at python.org (tarek.ziade) Date: Wed, 16 May 2012 14:22:49 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_make_sure_the_existing_m?= =?utf8?q?etadata_Provides_field_is_translated_when_translating?= Message-ID: http://hg.python.org/distutils2/rev/d015f9edccb8 changeset: 1341:d015f9edccb8 user: Tarek Ziade date: Wed May 16 14:22:44 2012 +0200 summary: make sure the existing metadata Provides field is translated when translating an old PKG-INFO files: distutils2/tests/test_util.py | 45 +++++++++++++++++++--- distutils2/util.py | 11 ++++- 2 files changed, 48 insertions(+), 8 deletions(-) diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py --- a/distutils2/tests/test_util.py +++ b/distutils2/tests/test_util.py @@ -756,6 +756,31 @@ self.assertRaises(ValueError, iglob, pattern) +PKG_INFO = '''\ +Metadata-Version: 1.1 +Name: hello +Version: 0.1.1 +Summary: Hello World +Home-page: https://example.com +Author: John Doe +Author-email: j.doe at example.com +License: UNKNOWN +Download-URL: https://example.com/tarball/master +Description: UNKNOWN +Platform: Any +Classifier: Development Status :: 3 - Alpha +Classifier: License :: OSI Approved :: Apache Software License +Classifier: Programming Language :: Python +Classifier: Programming Language :: Python :: 2 +Classifier: Programming Language :: Python :: 2.7 +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.2 +Classifier: Intended Audience :: Developers +Classifier: Environment :: Console +Provides: hello +''' + + class EggInfoToDistInfoTestCase(support.TempdirManager, support.LoggingCatcher, unittest.TestCase): @@ -774,10 +799,14 @@ dirs = [egginfo] files = ['hello.py', 'hello.pyc'] extra_metadata = ['dependency_links.txt', 'entry_points.txt', - 'not-zip-safe', 'PKG-INFO', 'top_level.txt', - 'SOURCES.txt'] + 'not-zip-safe', ('PKG-INFO', PKG_INFO), + 'top_level.txt', 'SOURCES.txt'] for f in extra_metadata: - files.append(os.path.join(egginfo, f)) + if isinstance(f, tuple): + f, content = f + else: + content = 'XXX' + files.append((os.path.join(egginfo, f), content)) tempdir, record_file = self.build_dist_tree(files, dirs) distinfo_path = os.path.join(tempdir, distinfo) @@ -796,13 +825,12 @@ distinfo = 'hello-0.1.1-py3.3.dist-info' egginfo = 'hello-0.1.1-py3.3.egg-info' # egginfo is a file in distutils which contains the metadata - files = ['hello.py', 'hello.pyc', egginfo] + files = ['hello.py', 'hello.pyc', (egginfo, PKG_INFO)] tempdir, record_file = self.build_dist_tree(files, dirs=[]) distinfo_path = os.path.join(tempdir, distinfo) egginfo_path = os.path.join(tempdir, egginfo) metadata_file_paths = self.get_metadata_file_paths(distinfo_path) - egginfo_to_distinfo(record_file) # test that directories and files get created self.assertTrue(os.path.isdir(distinfo_path)) @@ -820,10 +848,15 @@ os.makedirs(path) dir_paths.append(path) for f in files: + if isinstance(f, (list, tuple)): + f, content = f + else: + content = '' + path = os.path.join(tempdir, f) _f = open(path, 'w') try: - _f.write(f) + _f.write(content) finally: _f.close() file_paths.append(path) diff --git a/distutils2/util.py b/distutils2/util.py --- a/distutils2/util.py +++ b/distutils2/util.py @@ -1272,11 +1272,18 @@ requires = None req_path = os.path.join(distinfo_dir, 'requires.txt') requires = parse_requires(req_path) + + # adapting the metadata + metadata = Metadata(path=metadata_path) + if metadata['Provides'] != []: + metadata['Provides-Dist'] = metadata['Provides'] + metadata['Provides'] = [] + if requires is not None: # create a metadata instance to handle the reqs injection - metadata = Metadata(path=metadata_path) metadata['Requires-Dist'] = requires - metadata.write(metadata_path) + + metadata.write(metadata_path) installer_path = distinfo['installer_path'] logger.info('creating %s', installer_path) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Wed May 16 14:42:01 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 14:42:01 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0ODI5?= =?utf8?q?=3A_Fix_bisect_and_range=28=29_indexing_with_large_indices_=28?= =?utf8?b?Pj0gMiAqKiAzMik=?= Message-ID: http://hg.python.org/cpython/rev/888f5f3bfcb6 changeset: 76990:888f5f3bfcb6 branch: 3.2 parent: 76983:57936bd91bcf user: Antoine Pitrou date: Wed May 16 14:37:54 2012 +0200 summary: Issue #14829: Fix bisect and range() indexing with large indices (>= 2 ** 32) under 64-bit Windows. files: Misc/NEWS | 3 +++ Modules/_bisectmodule.c | 3 ++- Objects/rangeobject.c | 2 +- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ Library ------- +- Issue #14829: Fix bisect and range() indexing with large indices + (>= 2 ** 32) under 64-bit Windows. + - Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when accessing the Tk clipboard. Modify clipboad_get() to first request type UTF8_STRING when no specific type is requested in an X11 windowing diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -3,6 +3,7 @@ Converted to C by Dmitry Vasiliev (dima at hlabs.spb.ru). */ +#define PY_SSIZE_T_CLEAN #include "Python.h" static Py_ssize_t @@ -192,7 +193,7 @@ if (PyList_Insert(list, index, item) < 0) return NULL; } else { - result = PyObject_CallMethod(list, "insert", "iO", index, item); + result = PyObject_CallMethod(list, "insert", "nO", index, item); if (result == NULL) return NULL; Py_DECREF(result); diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -307,7 +307,7 @@ static PyObject * range_item(rangeobject *r, Py_ssize_t i) { - PyObject *res, *arg = PyLong_FromLong(i); + PyObject *res, *arg = PyLong_FromSsize_t(i); if (!arg) { return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 14:42:01 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 14:42:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314829=3A_Fix_bisect_and_range=28=29_indexing_with_l?= =?utf8?q?arge_indices_=28=3E=3D_2_**_32=29?= Message-ID: http://hg.python.org/cpython/rev/a3784c8f165e changeset: 76991:a3784c8f165e parent: 76989:6e5c517da087 parent: 76990:888f5f3bfcb6 user: Antoine Pitrou date: Wed May 16 14:39:36 2012 +0200 summary: Issue #14829: Fix bisect and range() indexing with large indices (>= 2 ** 32) under 64-bit Windows. (untested, because of Windows build issues under 3.x) files: Misc/NEWS | 3 +++ Modules/_bisectmodule.c | 4 ++-- Objects/rangeobject.c | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -34,6 +34,9 @@ Library ------- +- Issue #14829: Fix bisect and range() indexing with large indices + (>= 2 ** 32) under 64-bit Windows. + - Issue #14732: The _csv module now uses PEP 3121 module initialization. Patch by Robin Schreiber. diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -3,6 +3,7 @@ Converted to C by Dmitry Vasiliev (dima at hlabs.spb.ru). */ +#define PY_SSIZE_T_CLEAN #include "Python.h" static Py_ssize_t @@ -195,8 +196,7 @@ return NULL; } else { _Py_IDENTIFIER(insert); - - result = _PyObject_CallMethodId(list, &PyId_insert, "iO", index, item); + result = _PyObject_CallMethodId(list, &PyId_insert, "nO", index, item); if (result == NULL) return NULL; Py_DECREF(result); diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -308,7 +308,7 @@ static PyObject * range_item(rangeobject *r, Py_ssize_t i) { - PyObject *res, *arg = PyLong_FromLong(i); + PyObject *res, *arg = PyLong_FromSsize_t(i); if (!arg) { return NULL; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 14:45:01 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 14:45:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_More_=2Ehgignore_additions_?= =?utf8?q?for_new_VS_build_files?= Message-ID: http://hg.python.org/cpython/rev/c858878e358d changeset: 76992:c858878e358d user: Antoine Pitrou date: Wed May 16 14:42:38 2012 +0200 summary: More .hgignore additions for new VS build files files: .hgignore | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -32,7 +32,6 @@ Modules/config.c Modules/ld_so_aix$ Parser/pgen$ -PCbuild/amd64/ ^core ^python-gdb.py ^python.exe-gdb.py @@ -59,6 +58,9 @@ PC/*/*.user PC/*/*.ncb PC/*/*.suo +PC/*/Win32-temp-* +PC/*/x64-temp-* +PC/*/amd64 PCbuild/*.exe PCbuild/*.dll PCbuild/*.pdb @@ -72,6 +74,7 @@ PCbuild/*.*sdf PCbuild/Win32-temp-* PCbuild/x64-temp-* +PCbuild/amd64 BuildLog.htm __pycache__ Modules/_testembed -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 15:04:06 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 15:04:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Skip_test_under?= =?utf8?q?_64-bit_Windows?= Message-ID: http://hg.python.org/cpython/rev/6c164d788140 changeset: 76993:6c164d788140 branch: 2.7 parent: 76980:180d16af22e9 user: Antoine Pitrou date: Wed May 16 14:50:25 2012 +0200 summary: Skip test under 64-bit Windows files: Lib/test/test_bisect.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py --- a/Lib/test/test_bisect.py +++ b/Lib/test/test_bisect.py @@ -125,7 +125,10 @@ def test_large_range(self): # Issue 13496 mod = self.module - data = xrange(sys.maxsize-1) + try: + data = xrange(sys.maxsize-1) + except OverflowError: + self.skipTest("can't create a xrange() object of size `sys.maxsize`") self.assertEqual(mod.bisect_left(data, sys.maxsize-3), sys.maxsize-3) self.assertEqual(mod.bisect_right(data, sys.maxsize-3), sys.maxsize-2) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 15:04:07 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 15:04:07 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0ODI5?= =?utf8?q?=3A_Fix_bisect_issues_under_64-bit_Windows=2E?= Message-ID: http://hg.python.org/cpython/rev/e957b93571a8 changeset: 76994:e957b93571a8 branch: 2.7 user: Antoine Pitrou date: Wed May 16 15:01:40 2012 +0200 summary: Issue #14829: Fix bisect issues under 64-bit Windows. files: Lib/test/test_bisect.py | 47 +++++++++++++++++++++++++++- Misc/NEWS | 2 + Modules/_bisectmodule.c | 2 +- 3 files changed, 47 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py --- a/Lib/test/test_bisect.py +++ b/Lib/test/test_bisect.py @@ -23,6 +23,28 @@ import bisect as c_bisect +class Range(object): + """A trivial xrange()-like object without any integer width limitations.""" + def __init__(self, start, stop): + self.start = start + self.stop = stop + self.last_insert = None + + def __len__(self): + return self.stop - self.start + + def __getitem__(self, idx): + n = self.stop - self.start + if idx < 0: + idx += n + if idx >= n: + raise IndexError(idx) + return self.start + idx + + def insert(self, idx, item): + self.last_insert = idx, item + + class TestBisect(unittest.TestCase): module = None @@ -125,12 +147,31 @@ def test_large_range(self): # Issue 13496 mod = self.module + n = sys.maxsize try: - data = xrange(sys.maxsize-1) + data = xrange(n-1) except OverflowError: self.skipTest("can't create a xrange() object of size `sys.maxsize`") - self.assertEqual(mod.bisect_left(data, sys.maxsize-3), sys.maxsize-3) - self.assertEqual(mod.bisect_right(data, sys.maxsize-3), sys.maxsize-2) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + + def test_large_pyrange(self): + # Same as above, but without C-imposed limits on range() parameters + mod = self.module + n = sys.maxsize + data = Range(0, n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + x = n - 100 + mod.insort_left(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x, x)) + x = n - 200 + mod.insort_right(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x + 1, x)) def test_random(self, n=25): from random import randrange diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,8 @@ Library ------- +- Issue #14829: Fix bisect issues under 64-bit Windows. + - Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when accessing the Tk clipboard. Modify clipboad_get() to first request type UTF8_STRING when no specific type is requested in an X11 windowing diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -195,7 +195,7 @@ if (PyList_Insert(list, index, item) < 0) return NULL; } else { - result = PyObject_CallMethod(list, "insert", "iO", + result = PyObject_CallMethod(list, "insert", "nO", index, item); if (result == NULL) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 15:09:39 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 15:09:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Forward_port_ad?= =?utf8?q?ditional_tests_from_2=2E7_=28issue_=2314829=29=2E?= Message-ID: http://hg.python.org/cpython/rev/13900edf13be changeset: 76995:13900edf13be branch: 3.2 parent: 76990:888f5f3bfcb6 user: Antoine Pitrou date: Wed May 16 15:01:40 2012 +0200 summary: Forward port additional tests from 2.7 (issue #14829). files: Lib/test/test_bisect.py | 47 +++++++++++++++++++++++++++- 1 files changed, 44 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py --- a/Lib/test/test_bisect.py +++ b/Lib/test/test_bisect.py @@ -23,6 +23,28 @@ import bisect as c_bisect +class Range(object): + """A trivial range()-like object without any integer width limitations.""" + def __init__(self, start, stop): + self.start = start + self.stop = stop + self.last_insert = None + + def __len__(self): + return self.stop - self.start + + def __getitem__(self, idx): + n = self.stop - self.start + if idx < 0: + idx += n + if idx >= n: + raise IndexError(idx) + return self.start + idx + + def insert(self, idx, item): + self.last_insert = idx, item + + class TestBisect(unittest.TestCase): module = None @@ -125,9 +147,28 @@ def test_large_range(self): # Issue 13496 mod = self.module - data = range(sys.maxsize-1) - self.assertEqual(mod.bisect_left(data, sys.maxsize-3), sys.maxsize-3) - self.assertEqual(mod.bisect_right(data, sys.maxsize-3), sys.maxsize-2) + n = sys.maxsize + data = range(n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + + def test_large_pyrange(self): + # Same as above, but without C-imposed limits on range() parameters + mod = self.module + n = sys.maxsize + data = Range(0, n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + x = n - 100 + mod.insort_left(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x, x)) + x = n - 200 + mod.insort_right(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x + 1, x)) def test_random(self, n=25): from random import randrange -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 15:09:41 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 15:09:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Forward_port_additional_tests_from_2=2E7_=28issue_=2314829?= =?utf8?b?KS4=?= Message-ID: http://hg.python.org/cpython/rev/8c8709b98762 changeset: 76996:8c8709b98762 parent: 76992:c858878e358d parent: 76995:13900edf13be user: Antoine Pitrou date: Wed May 16 15:06:00 2012 +0200 summary: Forward port additional tests from 2.7 (issue #14829). files: Lib/test/test_bisect.py | 47 +++++++++++++++++++++++++++- 1 files changed, 44 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py --- a/Lib/test/test_bisect.py +++ b/Lib/test/test_bisect.py @@ -23,6 +23,28 @@ import bisect as c_bisect +class Range(object): + """A trivial range()-like object without any integer width limitations.""" + def __init__(self, start, stop): + self.start = start + self.stop = stop + self.last_insert = None + + def __len__(self): + return self.stop - self.start + + def __getitem__(self, idx): + n = self.stop - self.start + if idx < 0: + idx += n + if idx >= n: + raise IndexError(idx) + return self.start + idx + + def insert(self, idx, item): + self.last_insert = idx, item + + class TestBisect(unittest.TestCase): module = None @@ -125,9 +147,28 @@ def test_large_range(self): # Issue 13496 mod = self.module - data = range(sys.maxsize-1) - self.assertEqual(mod.bisect_left(data, sys.maxsize-3), sys.maxsize-3) - self.assertEqual(mod.bisect_right(data, sys.maxsize-3), sys.maxsize-2) + n = sys.maxsize + data = range(n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + + def test_large_pyrange(self): + # Same as above, but without C-imposed limits on range() parameters + mod = self.module + n = sys.maxsize + data = Range(0, n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + x = n - 100 + mod.insort_left(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x, x)) + x = n - 200 + mod.insort_right(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x + 1, x)) def test_random(self, n=25): from random import randrange -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 15:42:38 2012 From: python-checkins at python.org (brian.curtin) Date: Wed, 16 May 2012 15:42:38 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Soften_wording_about_VS200?= =?utf8?q?8?= Message-ID: http://hg.python.org/devguide/rev/0ac1d3863208 changeset: 511:0ac1d3863208 user: Brian Curtin date: Wed May 16 08:42:24 2012 -0500 summary: Soften wording about VS2008 files: setup.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -160,7 +160,7 @@ **Python 3.3** uses Microsoft Visual Studio 2010, which is available at http://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-cpp-express. -All versions previous to 3.3 use Microsoft Visual Studio 2008, available at +Most versions previous to 3.3 use Microsoft Visual Studio 2008, available at https://www.microsoft.com/visualstudio/en-us/products/2008-editions/express. Regardless of Visual Studio version, the ``PCbuild`` directory of a source -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Wed May 16 15:50:58 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:50:58 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_404=3A_Switch_the_sys=2Epr?= =?utf8?q?efix_choice=2C_add_activation_scripts=2C_mention_include?= Message-ID: http://hg.python.org/peps/rev/0936d8e00e5b changeset: 4385:0936d8e00e5b parent: 3978:8cfc3835e115 user: Carl Meyer date: Mon Oct 31 11:01:42 2011 -0600 summary: PEP 404: Switch the sys.prefix choice, add activation scripts, mention include directories in layout. files: pep-0404.txt | 247 +++++++++++--------------------------- 1 files changed, 72 insertions(+), 175 deletions(-) diff --git a/pep-0404.txt b/pep-0404.txt --- a/pep-0404.txt +++ b/pep-0404.txt @@ -42,9 +42,9 @@ Python's ``site`` module and manually symlink/copy an ever-changing set of standard-library modules into the virtual environment in order to perform a delicate boot-strapping dance at every startup. -(Virtualenv copies the binary because symlinking it does not provide -isolation, as Python dereferences a symlinked executable before -searching for ``sys.prefix``.) +(Virtualenv must copy the binary in order to provide isolation, as +Python dereferences a symlinked executable before searching for +``sys.prefix``.) The ``PYTHONHOME`` environment variable, Python's only existing built-in solution for virtual environments, requires @@ -85,22 +85,22 @@ create this virtual environment. In this case, prefix-finding continues as normal using the value of -the ``home`` key as the effective Python binary location, which -results in ``sys.prefix`` being set to the system installation prefix, -while ``sys.site_prefix`` is set to the directory containing +the ``home`` key as the effective Python binary location, which finds +the prefix of the base installation. ``sys.base_prefix`` is set to +this value, while ``sys.prefix`` is set to the directory containing ``pyvenv.cfg``. (If ``pyvenv.cfg`` is not found or does not contain the ``home`` key, -prefix-finding continues normally, and ``sys.site_prefix`` will be -equal to ``sys.prefix``.) +prefix-finding continues normally, and ``sys.prefix`` will be equal to +``sys.base_prefix``.) The ``site`` and ``sysconfig`` standard-library modules are modified -such that site-package directories ("purelib" and "platlib", in -``sysconfig`` terms) are found relative to ``sys.site_prefix``, while -other directories (the standard library, include files) are still -found relative to ``sys.prefix``. +such that the standard library and header files are are found relative +to ``sys.base_prefix``, while site-package directories ("purelib" and +"platlib", in ``sysconfig`` terms) are still found relative to +``sys.prefix``. -(Also, ``sys.site_exec_prefix`` is added, and handled similarly with +(Also, ``sys.base_exec_prefix`` is added, and handled similarly with regard to ``sys.exec_prefix``.) Thus, a Python virtual environment in its simplest form would consist @@ -166,6 +166,13 @@ virtualenv will be created, according to the given options, at each provided path. +The ``venv`` module also provides "shell activation scripts" for POSIX +and Windows systems which simply add the virtual environment's ``bin`` +(or ``Scripts``) directory to the front of the user's shell PATH. +This is not strictly necessary for use of a virtual environment (as an +explicit path to the venv's python binary or scripts can just as well +be used), but it is convenient. + The ``venv`` module also adds a ``pysetup3`` script into each venv. In order to allow ``pysetup`` and other Python package managers to install packages into the virtual environment the same way they would @@ -179,6 +186,7 @@ bin/python3 bin/python bin/pysetup3 + include/ lib/python3.3/site-packages/ While on a Windows system:: @@ -188,7 +196,8 @@ Scripts/python3.dll Scripts/pysetup3.exe Scripts/pysetup3-script.py - ... other DLLs and pyds... + ... other DLLs and pyds... + Include/ Lib/site-packages/ Third-party packages installed into the virtual environment will have @@ -266,22 +275,18 @@ target directory instead of raising an exception. Defaults to ``False``. -* ``use_symlinks`` - A Boolean value indicating whether to attempt to +* ``symlinks`` - A Boolean value indicating whether to attempt to symlink the Python binary (and any necessary DLLs or other binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to ``False``. -The returned env-builder is an object with a ``create`` method, which -takes as required argument the path (absolute or relative to the -current directory) of the target directory which is to contain the -virtual environment. The ``create`` method either creates the -environment in the specified directory, or raises an appropriate -exception. +The instantiated env-builder has a ``create`` method, which takes as +required argument the path (absolute or relative to the current +directory) of the target directory which is to contain the virtual +environment. The ``create`` method either creates the environment in +the specified directory, or raises an appropriate exception. -Creators of third-party virtual environment tools are free to use the -provided ``EnvBuilder`` class as a base class. - -The ``venv`` module also provides a module-level function as a -convenience:: +The ``venv`` module also provides a module-level ``create`` function +as a convenience:: def create(env_dir, system_site_packages=False, clear=False, use_symlinks=False): @@ -291,6 +296,9 @@ use_symlinks=use_symlinks) builder.create(env_dir) +Creators of third-party virtual environment tools are free to use the +provided ``EnvBuilder`` class as a base class. + The ``create`` method of the ``EnvBuilder`` class illustrates the hooks available for customization:: @@ -322,18 +330,18 @@ under Windows, DLLs) in the environment. * ``post_setup`` - A (no-op by default) hook method which can be - overridden in third party implementations to pre-install packages or + overridden in third party subclasses to pre-install packages or install scripts in the virtual environment. In addition, ``EnvBuilder`` provides a utility method that can be called from ``post_setup`` in subclasses to assist in installing -scripts into the virtual environment. The method ``install_scripts`` -accepts as arguments the ``context`` object (see above) and a -bytestring. The bytestring should be a base64-encoded zip file -containing directories "common", "posix", "nt", each containing -scripts destined for the bin directory in the environment. The -contents of "common" and the directory corresponding to ``os.name`` -are copied after doing some text replacement of placeholders: +custom scripts into the virtual environment. The method +``install_scripts`` accepts as arguments the ``context`` object (see +above) and a path to a directory. The directory should contain +subdirectories "common", "posix", "nt", each containing scripts +destined for the bin directory in the environment. The contents of +"common" and the directory corresponding to ``os.name`` are copied +after doing some text replacement of placeholders: * ``__VENV_DIR__`` is replaced with absolute path of the environment directory. @@ -349,27 +357,10 @@ The ``DistributeEnvBuilder`` subclass in the reference implementation illustrates how the customization hook can be used in practice to -pre-install Distribute and shell activation scripts into the virtual -environment. It's not envisaged that ``DistributeEnvBuilder`` will be -actually added to Python core, but it makes the reference -implementation more immediately useful for testing and exploratory -purposes. - -The "shell activation scripts" provided by ``DistributeEnvBuilder`` -simply add the virtual environment's ``bin/`` (or ``Scripts\``) -directory to the front of the user's shell PATH. This is not strictly -necessary for use of a virtual environment (as an explicit path to the -venv's python binary or scripts can just as well be used), but it is -convenient. - -This PEP does not propose that the ``venv`` module in core Python will -add such activation scripts by default, as they are shell-specific. -Adding activation scripts for the wide variety of possible shells is -an added maintenance burden, and is left to third-party extension -tools. - -No doubt the process of PEP review will show up any customization -requirements which have not yet been considered. +pre-install Distribute into the virtual environment. It's not +envisaged that ``DistributeEnvBuilder`` will be actually added to +Python core, but it makes the reference implementation more +immediately useful for testing and exploratory purposes. Backwards Compatibility @@ -405,33 +396,39 @@ __ http://docs.python.org/dev/library/sys.html#sys.prefix -This PEP currently proposes to leave ``sys.prefix`` pointing to the -base system installation (which is where the standard library and -header files are found), and introduce a new value in ``sys`` -(``sys.site_prefix``) to point to the prefix for ``site-packages``. -This maintains the documented semantics of ``sys.prefix``, but risks -breaking isolation if third-party code uses ``sys.prefix`` rather than -``sys.site_prefix`` or the appropriate ``site`` API to find +Maintaining this documented definition would mean leaving +``sys.prefix`` pointing to the base system installation (which is +where the standard library and header files are found), and +introducing a new value in ``sys`` (something like +``sys.site_prefix``) to point to the prefix for ``site-packages``. +This would maintain the documented semantics of ``sys.prefix``, but +risk breaking isolation if third-party code uses ``sys.prefix`` rather +than ``sys.site_prefix`` or the appropriate ``site`` API to find site-packages directories. The most notable case is probably `setuptools`_ and its fork `distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but do use ``sys.prefix`` directly to build up a list of site directories for pre-flight checking where ``pth`` files can usefully be placed. -It would be trivial to modify these tools (currently only -`distribute`_ is Python 3 compatible) to check ``sys.site_prefix`` and -fall back to ``sys.prefix`` if it doesn't exist (for earlier versions -of Python). If Distribute is modified in this way and released before -Python 3.3 is released with the ``venv`` module, there would be no -likely reason for an older version of Distribute to ever be installed -in a virtual environment. -In terms of other third-party usage, a `Google Code Search`_ turns up -what appears to be a roughly even mix of usage between packages using -``sys.prefix`` to build up a site-packages path and packages using it -to e.g. eliminate the standard-library from code-execution tracing. -Either choice that's made here will require one or the other of these -uses to be updated. +Otherwise, a `Google Code Search`_ turns up what appears to be a +roughly even mix of usage between packages using ``sys.prefix`` to +build up a site-packages path and packages using it to e.g. eliminate +the standard-library from code-execution tracing. + +Although it requires modifying the documented definition of +``sys.prefix``, this PEP prefers to have ``sys.prefix`` point to the +virtual environment (where ``site-packages`` is found), and introduce +``sys.base_prefix`` to point to the standard library and Python header +files. Rationale for this choice: + +* It is preferable to err on the side of greater isolation of the + virtual environment. + +* Virtualenv already modifies ``sys.prefix`` to point at the virtual + environment, and in practice this has not been a problem. + +* No modification is required to setuptools/distribute. .. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools .. _distribute: http://packages.python.org/distribute/ @@ -441,72 +438,6 @@ Open Questions ============== -Naming of the new ``sys`` prefix attributes -------------------------------------------- - -The name ``sys.site_prefix`` was chosen with the following -considerations in mind: - -* Inasmuch as "site" has a meaning in Python, it means a combination - of Python version, standard library, and specific set of - site-packages. This is, fundamentally, what a venv is (although it - shares the standard library with its "base" site). - -* It is the Python ``site`` module which implements adding - site-packages directories to ``sys.path``, so ``sys.site_prefix`` is - a prefix used (and set) primarily by the ``site`` module. - -A concern has been raised that the term ``site`` in Python is already -overloaded and of unclear meaning, and this usage will increase the -overload. - -One proposed alternative is ``sys.venv_prefix``, which has the -advantage of being clearly related to the venv implementation. The -downside of this proposal is that it implies the attribute is only -useful/relevant when in a venv and should be absent or ``None`` when -not in a venv. This imposes an unnecessary extra burden on code using -the attribute: ``sys.venv_prefix if sys.venv_prefix else sys.prefix``. -The prefix attributes are more usable and general if they are always -present and set, and split by meaning (stdlib vs site-packages, -roughly), rather than specifically tied to venv. Also, third-party -code should be encouraged to not know or care whether it is running in -a virtual environment or not; this option seems to work against that -goal. - -Another option would be ``sys.local_prefix``, which has both the -advantage and disadvantage, depending on perspective, that it -introduces the new term "local" rather than drawing on existing -associations with the term "site". - - -Why not modify sys.prefix? --------------------------- - -As discussed above under `Backwards Compatibility`_, this PEP proposes -to add ``sys.site_prefix`` as "the prefix relative to which -site-package directories are found". This maintains compatibility -with the documented meaning of ``sys.prefix`` (as the location -relative to which the standard library can be found), but means that -code assuming that site-packages directories are found relative to -``sys.prefix`` will not respect the virtual environment correctly. - -Since it is unable to modify ``distutils``/``sysconfig``, -`virtualenv`_ is forced to instead re-point ``sys.prefix`` at the -virtual environment. - -An argument could be made that this PEP should follow virtualenv's -lead here (and introduce something like ``sys.base_prefix`` to point -to the standard library and header files), since virtualenv already -does this and it doesn't appear to have caused major problems with -existing code. - -Another argument in favor of this is that it would be preferable to -err on the side of greater, rather than lesser, isolation. Changing -``sys.prefix`` to point to the virtual environment and introducing a -new ``sys.base_prefix`` attribute would err on the side of greater -isolation in the face of existing code's use of ``sys.prefix``. - - What about include files? ------------------------- @@ -537,21 +468,6 @@ really because there's no supporting concept in ``Python/sysconfig``. -Interface with packaging tools ------------------------------- - -Some work will be needed in packaging tools (Python 3.3 packaging, -Distribute) to support implementation of this PEP. For example: - -* How Distribute and packaging use ``sys.prefix`` and/or - ``sys.site_prefix``. Clearly, in practice we'll need to use - Distribute for a while, until packages have migrated over to usage - of setup.cfg. - -* How packaging and Distribute set up shebang lines in scripts which they - install in virtual environments. - - Testability and Source Build Issues ----------------------------------- @@ -587,29 +503,6 @@ .. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 -Activation and Utility Scripts ------------------------------- - -Virtualenv provides shell "activation" scripts as a user convenience, -to put the virtual environment's Python binary first on the shell -PATH. This is a maintenance burden, as separate activation scripts -need to be provided and maintained for every supported shell. For -this reason, this PEP proposes to leave such scripts to be provided by -third-party extensions; virtual environments created by the core -functionality would be used by directly invoking the environment's -Python binary or scripts. - -If we are going to rely on external code to provide these -conveniences, we need to check with existing third-party projects in -this space (virtualenv, zc.buildout) and ensure that the proposed API -meets their needs. - -(Virtualenv would be fine with the proposed API; it would become a -relatively thin wrapper with a subclass of the env builder that adds -shell activation and automatic installation of ``pip`` inside the -virtual environment). - - Provide a mode that is isolated only from user site packages? ------------------------------------------------------------- -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:50:59 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:50:59 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merge_from_upstream=2E?= Message-ID: http://hg.python.org/peps/rev/85ed05373838 changeset: 4386:85ed05373838 parent: 4385:0936d8e00e5b parent: 4118:0a7019500350 user: Carl Meyer date: Mon Mar 05 17:08:36 2012 -0700 summary: Merge from upstream. files: Makefile | 2 +- pep-0001.txt | 3 +- pep-0007.txt | 3 +- pep-0008.txt | 6 - pep-0042.txt | 4 +- pep-0101.txt | 25 +- pep-0360.txt | 28 +- pep-0361.txt | 12 +- pep-0362.txt | 2 +- pep-0373.txt | 8 +- pep-0374.txt | 2 +- pep-0375.txt | 2 +- pep-0380.txt | 2 +- pep-0382.txt | 58 +- pep-0383.txt | 2 +- pep-0385.txt | 22 +- pep-0392.txt | 9 + pep-0393.txt | 23 +- pep-0394.txt | 131 +++- pep-0395.txt | 696 +++++++++++++++++++++++---- pep-0398.txt | 25 +- pep-0400.txt | 2 +- pep-0403.txt | 256 ++++++---- pep-0404.txt | 615 +++++------------------- pep-0404.txt | 2 +- pep-0406.txt | 274 +++++++++++ pep-0407.txt | 179 +++++++ pep-0408.txt | 317 ++++++++++++ pep-0409.txt | 201 ++++++++ pep-0410.txt | 547 ++++++++++++++++++++++ pep-0411.txt | 208 ++++++++ pep-0412.txt | 181 +++++++ pep-0413.txt | 919 +++++++++++++++++++++++++++++++++++++ pep-0414.txt | 415 ++++++++++++++++ pep-0415.txt | 88 +++ pep-0416.txt | 134 +++++ pep-3000.txt | 2 +- pep-3002.txt | 2 +- pep-3003.txt | 2 +- pep-3099.txt | 2 +- pep-3100.txt | 4 +- pep-3144.txt | 430 ++++------------- pep-3150.txt | 50 +- pep-3154.txt | 37 +- pep-3155.txt | 67 ++- pep0/output.py | 10 +- 46 files changed, 4734 insertions(+), 1275 deletions(-) diff --git a/Makefile b/Makefile --- a/Makefile +++ b/Makefile @@ -18,7 +18,7 @@ $(TARGETS): pep2html.py -pep-0000.txt: $(wildcard pep-????.txt) +pep-0000.txt: $(wildcard pep-????.txt) $(wildcard pep0/*.py) $(PYTHON) genpepindex.py . install: diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -461,8 +461,7 @@ ======================== .. [1] This historical record is available by the normal hg commands - for retrieving older revisions. For those without direct access to - the hg repo, you can browse the current and past PEP revisions here: + for retrieving older revisions, and can also be browsed via HTTP here: http://hg.python.org/peps/ .. [2] PEP 2, Procedure for Adding New Modules, Faassen diff --git a/pep-0007.txt b/pep-0007.txt --- a/pep-0007.txt +++ b/pep-0007.txt @@ -72,7 +72,8 @@ } - Code structure: one space between keywords like 'if', 'for' and - the following left paren; no spaces inside the paren; braces as + the following left paren; no spaces inside the paren; braces may be + omitted where C permits but when present, they should be formatted as shown: if (mro != NULL) { diff --git a/pep-0008.txt b/pep-0008.txt --- a/pep-0008.txt +++ b/pep-0008.txt @@ -840,12 +840,6 @@ Worse: if greeting is True: -Rules that apply only to the standard library - - - Do not use function type annotations in the standard library. - These are reserved for users and third-party modules. See - PEP 3107 and the bug 10899 for details. - References diff --git a/pep-0042.txt b/pep-0042.txt --- a/pep-0042.txt +++ b/pep-0042.txt @@ -3,8 +3,8 @@ Version: $Revision$ Last-Modified: $Date$ Author: Jeremy Hylton -Status: Active -Type: Informational +Status: Final +Type: Process Created: 12-Sep-2000 Post-History: diff --git a/pep-0101.txt b/pep-0101.txt --- a/pep-0101.txt +++ b/pep-0101.txt @@ -62,11 +62,10 @@ http://hg.python.org/release/ We use the following conventions in the examples below. Where a release - number is given, it is of the form X.YaZ, e.g. 2.6a3 for Python 2.6 alpha - 3, where "a" == alpha, "b" == beta, "rc" == release candidate. If a micro - release number is used, then we'll say X.Y.MaZ. + number is given, it is of the form X.Y.ZaN, e.g. 3.3.0a3 for Python 3.3.0 + alpha 3, where "a" == alpha, "b" == beta, "rc" == release candidate. - Release tags are named "vX.YaZ". The branch name for minor release + Release tags are named "vX.Y.ZaN". The branch name for minor release maintenance branches is "X.Y". This helps by performing several automatic editing steps, and guides you @@ -156,7 +155,7 @@ ___ Bump version numbers via the release script. - $ .../release/release.py --bump X.YaZ + $ .../release/release.py --bump X.Y.ZaN This automates updating various release numbers, but you will have to modify a few files manually. If your $EDITOR environment variable is @@ -197,9 +196,9 @@ alpha or beta releases. Note that Andrew Kuchling often takes care of this. - ___ Tag the release for X.YaZ. + ___ Tag the release for X.Y.ZaN. - $ .../release/release.py --tag X.YaZ + $ .../release/release.py --tag X.Y.ZaN ___ If this is a final major release, branch the tree for X.Y. @@ -309,10 +308,10 @@ ___ Use the release script to create the source gzip and bz2 tarballs, md5 checksums, documentation tar and zip files, and gpg signature files. - $ .../release/release.py --export X.YaZ + $ .../release/release.py --export X.Y.ZaN This will leave all the relevant files in a subdirectory called - 'X.YaZ/src', and the built docs in 'X.YaZ/docs' (for final releases). + 'X.Y.ZaN/src', and the built docs in 'X.Y.ZaN/docs' (for final releases). ___ scp or rsync all the files to your home directory on dinsdale.python.org. @@ -361,7 +360,7 @@ Python-3.2.tgz, along with a "prev" subdirectory containing Python-3.2a1.msi, Python-3.2a1.tgz, Python-3.2a1.tar.bz2, etc. - ___ On dinsdale, cd /data/ftp.python.org/pub/python/X.Y[.Z] + ___ On dinsdale, cd /data/ftp.python.org/pub/python/X.Y.Z creating it if necessary. Make sure it is owned by group 'webmaster' and group-writable. @@ -383,14 +382,14 @@ ___ md5sum the files and make sure they got uploaded intact. ___ If this is a final release: Move the doc zips and tarballs to - /data/ftp.python.org/pub/python/doc/X.Y[.Z] creating the directory + /data/ftp.python.org/pub/python/doc/X.Y.Z creating the directory if necessary, and adapt the "current" symlink in .../doc to point to that directory. Note though that if you're releasing a maintenance release for an older version, don't change the current link. ___ If this is a final release (even a maintenance release), also unpack the HTML docs to - /data/ftp.python.org/pub/docs.python.org/release/X.Y[.Z]. + /data/ftp.python.org/pub/docs.python.org/release/X.Y.Z. ___ Let the DE check if the docs are built and work all right. @@ -513,7 +512,7 @@ ___ Do the guided post-release steps with the release script. - $ .../release/release.py --done X.YaZ + $ .../release/release.py --done X.Y.ZaN Review and commit these changes. diff --git a/pep-0360.txt b/pep-0360.txt --- a/pep-0360.txt +++ b/pep-0360.txt @@ -3,8 +3,8 @@ Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon -Status: Active -Type: Informational +Status: Final +Type: Process Content-Type: text/x-rst Created: 30-May-2006 Post-History: @@ -13,7 +13,7 @@ .. warning:: No new modules are to be added to this PEP. It has been deemed dangerous to codify external maintenance of any code checked into Python's code repository. Code - contributers should expect Python's development + contributors should expect Python's development methodology to be used for any and all code checked into Python's code repository. @@ -66,11 +66,8 @@ :Contact person: Fredrik Lundh -Patches should not be directly applied to Python HEAD, but instead -reported to the Python tracker [#python-tracker]_ (critical bug fixes -are the exception). Bugs should also be reported to the Python -tracker. Both bugs and patches should be assigned to Fredrik Lundh. - +Fredrik has ceded ElementTree maintenance to the core Python development +team [#element-tree]_. Expat XML parser ---------------- @@ -83,6 +80,7 @@ :Contact person: None + Optik ----- @@ -93,7 +91,9 @@ :Contact person: Greg Ward -External development seems to have ceased. +External development seems to have ceased. For new applications, optparse +itself has been largely superseded by argparse. + wsgiref ------- @@ -104,16 +104,16 @@ :Contact Person: Phillip J. Eby -Bugs and patches should pass through the Web-SIG mailing list [#web-sig]_ -before being applied to HEAD. External maintenance seems to have -ceased. +This module is maintained in the standard library, but significant bug +reports and patches should pass through the Web-SIG mailing list +[#web-sig]_ for discussion. References ========== -.. [#python-tracker] Python tracker - (http://sourceforge.net/tracker/?group_id=5470) +.. [#element-tree] Fredrik's handing over of ElementTree + (http://mail.python.org/pipermail/python-dev/2012-February/116389.html) .. [#web-sig] Web-SIG mailing list (http://mail.python.org/mailman/listinfo/web-sig) diff --git a/pep-0361.txt b/pep-0361.txt --- a/pep-0361.txt +++ b/pep-0361.txt @@ -63,9 +63,15 @@ Nov 06 2008: Python 3.0rc2 released Nov 21 2008: Python 3.0rc3 released Dec 03 2008: Python 3.0 final released - Dec 04 2008: Python 2.6.1 final release - Apr 14 2009: Python 2.6.2 final release - Oct 02 2009: Python 2.6.3 final release + Dec 04 2008: Python 2.6.1 final released + Apr 14 2009: Python 2.6.2 final released + Oct 02 2009: Python 2.6.3 final released + Oct 25 2009: Python 2.6.4 final released + Mar 19 2010: Python 2.6.5 final released + Aug 24 2010: Python 2.6.6 final released + Jun 03 2011: Python 2.6.7 final released (security-only) + + Python 2.6.8 (security-only) planned for Feb 10-17 2012 See the public `Google calendar`_ diff --git a/pep-0362.txt b/pep-0362.txt --- a/pep-0362.txt +++ b/pep-0362.txt @@ -44,7 +44,7 @@ representation affecting the function it represents (but this is an `Open Issues`_). -Indirecation of signature introspection can also occur. If a +Indirection of signature introspection can also occur. If a decorator took a decorated function's signature object and set it on the decorating function then introspection could be redirected to what is actually expected instead of the typical ``*args, **kwargs`` diff --git a/pep-0373.txt b/pep-0373.txt --- a/pep-0373.txt +++ b/pep-0373.txt @@ -13,10 +13,10 @@ Abstract ======== -This document describes the development and release schedule for Python 2.7. -The schedule primarily concerns itself with PEP-sized items. Small features may -be added up to and including the first beta release. Bugs may be fixed until -the final release. +This document describes the development and release schedule for +Python 2.7. The schedule primarily concerns itself with PEP-sized +items. Small features may be added up to and including the first beta +release. Bugs may be fixed until the final release. Release Manager and Crew diff --git a/pep-0374.txt b/pep-0374.txt --- a/pep-0374.txt +++ b/pep-0374.txt @@ -7,7 +7,7 @@ Alexandre Vassalotti , Barry Warsaw , Dirkjan Ochtman -Status: Active +Status: Final Type: Process Content-Type: text/x-rst Created: 07-Nov-2008 diff --git a/pep-0375.txt b/pep-0375.txt --- a/pep-0375.txt +++ b/pep-0375.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Benjamin Peterson -Status: Active +Status: Final Type: Informational Content-Type: text/x-rst Created: 8-Feb-2009 diff --git a/pep-0380.txt b/pep-0380.txt --- a/pep-0380.txt +++ b/pep-0380.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing -Status: Accepted +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 diff --git a/pep-0382.txt b/pep-0382.txt --- a/pep-0382.txt +++ b/pep-0382.txt @@ -18,7 +18,7 @@ versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the -package. +package. An implementation of this PEP is available at [1]_. Terminology =========== @@ -94,11 +94,15 @@ with ``.pyp`` (for Python package) contains a portion of a package. The import statement is extended so that computes the package's -``__path__`` attribute for a package named ``P`` as consisting of all -directories named ``P.pyp``, plus optionally a single directory named -``P`` containing a file ``__init__.py``. If either of these are -found on the path of the parent package (or sys.path for a top-level -package), search for additional portions of the package continues. +``__path__`` attribute for a package named ``P`` as consisting of +optionally a single directory name ``P`` containing a file +``__init__.py``, plus all directories named ``P.pyp``, in the order in +which they are found in the parent's package ``__path__`` (or +``sys.path``). If either of these are found, search for additional +portions of the package continues. + +A directory may contain both a package in the ``P/__init__.py`` and +the ``P.pyp`` form. No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module @@ -129,27 +133,22 @@ algorithm. To do so, a finder used as a path hook must support a method: - finder.is_package_portion(fullname) + finder.find_package_portion(fullname) This method will be called in the same manner as find_module, and it -must return True if there is a package portion with that name; False -if fullname indicates a subpackage/submodule that is not a package -portion; otherwise, it shall raise an ImportError. +must return an string to be added to the package's ``__path__``. +If the finder doesn't find a portion of the package, it shall return +``None``. Raising ``AttributeError`` from above call will be treated +as non-conformance with this PEP, and the exception will be ignored. +All other exceptions are reported. -If any \*.pyp directories are found, but no loader was returned from -find_module, a package is created and initialized with the path. +A finder may report both success from ``find_module`` and from +``find_package_portion``, allowing for both a package containing +an ``__init__.py`` and a portion of the same package. -If a loader was return, but no \*.pyp directories, load_module is -called as defined in PEP 302. - -If both \*.pyp directories where found, and a loader was returned, a -new method is called on the loader: - - loader.load_module_with_path(load_module, path) - -where the path parameter is the list that will become the __path__ -attribute of the new package. - +All strings returned from ``find_package_portion``, along with all +path names of ``.pyp`` directories are added to the new package's +``__path__``. Discussion ========== @@ -181,7 +180,7 @@ package as being a Python package, if there is no need to declare Python packages. -packages can stop filling out the namespace package's __init__.py. As +Packages can stop filling out the namespace package's __init__.py. As a consequence, extend_path and declare_namespace become obsolete. Namespace packages can start providing non-trivial __init__.py @@ -195,14 +194,11 @@ any other mechanism might cause portions to get added twice to __path__. -It has been proposed to also add this feature to Python 2.7. Given -that 2.x reaches its end-of-life, it is questionable whether the -addition of the feature would really do more good than harm (in having -users and tools starting to special-case 2.7). Prospective users of -this feature are encouraged to comment on this particular question. In -addition, Python 2.7 is in bug-fix mode now, so adding new features to -it would be a violation of the established policies. +References +========== +.. [1] PEP 382 branch + (http://hg.python.org/features/pep-382-2#pep-382) Copyright ========= diff --git a/pep-0383.txt b/pep-0383.txt --- a/pep-0383.txt +++ b/pep-0383.txt @@ -89,7 +89,7 @@ deprecated. External libraries that operate on file names (such as GUI file -chosers) should also encode them according to the PEP. +choosers) should also encode them according to the PEP. Discussion ========== diff --git a/pep-0385.txt b/pep-0385.txt --- a/pep-0385.txt +++ b/pep-0385.txt @@ -5,7 +5,7 @@ Author: Dirkjan Ochtman , Antoine Pitrou , Georg Brandl -Status: Active +Status: Final Type: Process Content-Type: text/x-rst Created: 25-May-2009 @@ -62,15 +62,6 @@ switch over to the new repository. -Todo list -========= - -The current list of issues to resolve at various steps in the -conversion is kept `in the pymigr repo`_. - -.. _in the pymigr repo: http://hg.python.org/pymigr/file/tip/todo.txt - - Transition plan =============== @@ -158,14 +149,13 @@ In order to provide user names the way they are common in hg (in the 'First Last ' format), we need an author map to map cvs and svn user names to real names and their email addresses. We -have a complete version of such a map in my `migration tools -repository`_. The email addresses in it might be out of date; that's +have a complete version of such a map in the migration tools +repository (not publicly accessible to avoid leaking addresses to +harvesters). The email addresses in it might be out of date; that's bound to happen, although it would be nice to try and have as many people as possible review it for addresses that are out of date. The current version also still seems to contain some encoding problems. -.. _migration tools repository: http://hg.python.org/pymigr/ - Generating .hgignore -------------------- @@ -313,15 +303,13 @@ A more or less stock hgwebdir installation should be set up. We might want to come up with a style to match the Python website. -A `small WSGI application`_ has been written that can look up +A small WSGI application has been written that can look up Subversion revisions and redirect to the appropriate hgweb page for the given changeset, regardless in which repository the converted revision ended up (since one big Subversion repository is converted into several Mercurial repositories). It can also look up Mercurial changesets by their hexadecimal ID. -.. _small WSGI application: http://hg.python.org/pymigr/file/tip/hglookup.py - roundup ------- diff --git a/pep-0392.txt b/pep-0392.txt --- a/pep-0392.txt +++ b/pep-0392.txt @@ -64,6 +64,15 @@ No large-scale changes have been recorded yet. +Bugfix Releases +=============== + +- 3.2.1: released July 10, 2011 +- 3.2.2: released September 4, 2011 + +- 3.2.3: planned February 10-17, 2012 + + References ========== diff --git a/pep-0393.txt b/pep-0393.txt --- a/pep-0393.txt +++ b/pep-0393.txt @@ -189,7 +189,7 @@ gives the void pointer to the data. Access to individual characters should use PyUnicode_{READ|WRITE}[_CHAR]: - - PyUnciode_READ(kind, data, index) + - PyUnicode_READ(kind, data, index) - PyUnicode_WRITE(kind, data, index, value) - PyUnicode_READ_CHAR(unicode, index) @@ -372,6 +372,25 @@ slowdowns of 1% to 30%; for specific benchmarks, speedups may happen as may happen significantly larger slowdowns. +In actual measurements of a Django application ([2]_), significant +reductions of memory usage could be found. For example, the storage +for Unicode objects reduced to 2216807 bytes, down from 6378540 bytes +for a wide Unicode build, and down from 3694694 bytes for a narrow +Unicode build (all on a 32-bit system). This reduction came from the +prevalence of ASCII strings in this application; out of 36,000 strings +(with 1,310,000 chars), 35713 where ASCII strings (with 1,300,000 +chars). The sources for these strings where not further analysed; +many of them likely originate from identifiers in the library, and +string constants in Django's source code. + +In comparison to Python 2, both Unicode and byte strings need to be +accounted. In the test application, Unicode and byte strings combined +had a length of 2,046,000 units (bytes/chars) in 2.x, and 2,200,000 +units in 3.x. On a 32-bit system, where the 2.x build used 32-bit +wchar_t/Py_UNICODE, the 2.x test used 3,620,000 bytes, and the 3.x +build 3,340,000 bytes. This reduction in 3.x using the PEP compared +to 2.x only occurs when comparing with a wide unicode build. + Porting Guidelines ================== @@ -435,6 +454,8 @@ .. [1] PEP 393 branch https://bitbucket.org/t0rsten/pep-393 +.. [2] Django measurement results + http://www.dcl.hpi.uni-potsdam.de/home/loewis/djmemprof/ Copyright ========= diff --git a/pep-0394.txt b/pep-0394.txt --- a/pep-0394.txt +++ b/pep-0394.txt @@ -4,11 +4,12 @@ Last-Modified: $Date$ Author: Kerrick Staley , Nick Coghlan -Status: Draft +Status: Active Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 -Post-History: 04-Mar-2011, 20-Jul-2011 +Post-History: 04-Mar-2011, 20-Jul-2011, 16-Feb-2012 +Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116594.html Abstract @@ -39,7 +40,7 @@ Python as either ``python2`` or ``python3``. * For the time being, it is recommended that ``python`` should refer to ``python2`` (however, some distributions have already chosen otherwise; see - Notes below). + the `Rationale`_ and `Migration Notes`_ below). * The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``, with the original commands invoking these versions by default, but possibly @@ -48,7 +49,7 @@ * In order to tolerate differences across platforms, all new code that needs to invoke the Python interpreter should not specify ``python``, but rather should specify either ``python2`` or ``python3`` (or the more specific - ``python2.x`` and ``python3.x`` versions; see the Notes). + ``python2.x`` and ``python3.x`` versions; see the `Migration Notes`_). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context. @@ -59,29 +60,48 @@ ``sys.executable`` to avoid hardcoded assumptions regarding the interpreter location remains the preferred approach. -These recommendations are the outcome of the relevant python-dev discussion in -March and July 2011 [1][2] (NOTE: More accurately, they will be such once the -"Draft" status disappears from the PEP header, it has been moved into the -"Other Informational PEP" section in PEP 0 and this note has been deleted) +These recommendations are the outcome of the relevant python-dev discussions +in March and July 2011 ([1]_, [2]_) and February 2012 ([4]_). Rationale ========= -This is needed as, even though the majority of distributions still alias the -``python`` command to Python 2, some now alias it to Python 3. Some of -the former also do not provide a ``python2`` command; hence, there is -currently no way for Python 2 code (or any code that invokes the Python 2 -interpreter directly rather than via ``sys.executable``) to reliably run on -all Unix-like systems without modification, as the ``python`` command will -invoke the wrong interpreter version on some systems, and the ``python2`` -command will fail completely on others. The recommendations in this PEP -provide a very simple mechanism to restore cross-platform support, with -minimal additional work required on the part of distribution maintainers. +This recommendation is needed as, even though the majority of distributions +still alias the ``python`` command to Python 2, some now alias it to +Python 3 ([5]_). As some of the former distributions do not yet provide a +``python2`` command by default, there is currently no way for Python 2 code +(or any code that invokes the Python 2 interpreter directly rather than via +``sys.executable``) to reliably run on all Unix-like systems without +modification, as the ``python`` command will invoke the wrong interpreter +version on some systems, and the ``python2`` command will fail completely +on others. The recommendations in this PEP provide a very simple mechanism +to restore cross-platform support, with minimal additional work required +on the part of distribution maintainers. -Notes -===== +Future Changes to this Recommendation +===================================== + +It is anticipated that there will eventually come a time where the third +party ecosystem surrounding Python 3 is sufficiently mature for this +recommendation to be updated to suggest that the ``python`` symlink +refer to ``python3`` rather than ``python2``. + +This recommendation will be periodically reviewed over the next few years, +and updated when the core development team judges it appropriate. As a +point of reference, regular maintenance releases for the Python 2.7 series +will continue until at least 2015. + + +Migration Notes +=============== + +This section does not contain any official recommendations from the core +CPython developers. It's merely a collection of notes regarding various +aspects of migrating to Python 3 as the default version of Python for a +system. They will hopefully be helpful to any distributions considering +making such a change. * Distributions that only include ``python3`` in their base install (i.e. they do not provide ``python2`` by default) along with those that are @@ -107,7 +127,7 @@ rather being provided as a separate binary file. * It is suggested that even distribution-specific packages follow the ``python2``/``python3`` convention, even in code that is not intended to - operate on other distributions. This will prevent problems if the + operate on other distributions. This will reduce problems if the distribution later decides to change the version of the Python interpreter that the ``python`` command invokes, or if a sysadmin installs a custom ``python`` command with a different major version than the distribution @@ -120,16 +140,14 @@ versa. That way, if a sysadmin does decide to replace the installed ``python`` file, they can do so without inadvertently deleting the previously installed binary. -* As an alternative to the recommendation presented above, some distributions - may choose to leave the ``python`` command itself undefined, leaving - sysadmins and users with the responsibility to choose their own preferred - version to be made available as the ``python`` command. * If the Python 2 interpreter becomes uncommon, scripts should nevertheless continue to use the ``python3`` convention rather that just ``python``. This will ease transition in the event that yet another major version of Python is released. -* If these conventions are adhered to, it will be the case that the ``python`` - command is only executed in an interactive manner. +* If these conventions are adhered to, it will become the case that the + ``python`` command is only executed in an interactive manner as a user + convenience, or to run scripts that are source compatible with both Python + 2 and Python 3. Backwards Compatibility @@ -147,25 +165,38 @@ Application to the CPython Reference Interpreter ================================================ -While technically a new feature, the ``make install`` command in the 2.7 -version of CPython will be adjusted to create the ``python2.7``, ``idle2.7``, -``pydoc2.7``, and ``python2.7-config`` binaries, with ``python2``, ``idle2``, -``pydoc2``, and ``python2-config`` as hard links to the respective binaries, -and ``python``, ``idle``, ``pydoc``, and ``python-config`` as symbolic links -to the respective hard links. This feature will first appear in CPython -2.7.3. +While technically a new feature, the ``make install`` and ``make bininstall`` +command in the 2.7 version of CPython will be adjusted to create the +following chains of symbolic links in the relevant ``bin`` directory (the +final item listed in the chain is the actual installed binary, preceding +items are relative symbolic links):: -The ``make install`` command in the CPython 3.x series will similarly install -the ``python3.x``, ``idle3.x``, ``pydoc3.x``, and ``python3.x-config`` -binaries (with appropriate ``x``), and ``python3``, ``idle3``, ``pydoc3``, -and ``python3-config`` as hard links. This feature will first appear in -CPython 3.3. + python -> python2 -> python2.7 + python-config -> python2-config -> python2.7-config Similar adjustments will be made to the Mac OS X binary installer. -As implementation of these features in the default installers does not alter -the recommendations in this PEP, the implementation progress is managed on the -tracker as issue . +This feature will first appear in the default installation process in +CPython 2.7.3. + +The installation commands in the CPython 3.x series already create the +appropriate symlinks. For example, CPython 3.2 creates:: + + python3 -> python3.2 + idle3 -> idle3.2 + pydoc3 -> pydoc3.2 + python3-config -> python3.2-config + +And CPython 3.3 will create:: + + python3 -> python3.3 + idle3 -> idle3.3 + pydoc3 -> pydoc3.3 + python3-config -> python3.3-config + pysetup3 -> pysetup3.3 + +The implementation progress of these features in the default installers is +managed on the tracker as issue #12627 ([3]_). Impact on PYTHON* Environment Variables @@ -192,12 +223,20 @@ References ========== -[1] Support the /usr/bin/python2 symlink upstream (with bonus grammar class!) - (http://mail.python.org/pipermail/python-dev/2011-March/108491.html) +.. [1] Support the /usr/bin/python2 symlink upstream (with bonus grammar class!) + (http://mail.python.org/pipermail/python-dev/2011-March/108491.html) -[2] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream) - (http://mail.python.org/pipermail/python-dev/2011-July/112322.html) +.. [2] Rebooting \PEP 394 (aka Support the /usr/bin/python2 symlink upstream) + (http://mail.python.org/pipermail/python-dev/2011-July/112322.html) +.. [3] Implement \PEP 394 in the CPython Makefile + (http://bugs.python.org/issue12627) + +.. [4] \PEP 394 request for pronouncement (python2 symlink in \*nix systems) + (http://mail.python.org/pipermail/python-dev/2012-February/116435.html) + +.. [5] Arch Linux announcement that their "python" link now refers Python 3 + (https://www.archlinux.org/news/python-is-now-python-3/) Copyright =========== diff --git a/pep-0395.txt b/pep-0395.txt --- a/pep-0395.txt +++ b/pep-0395.txt @@ -1,5 +1,5 @@ PEP: 395 -Title: Module Aliasing +Title: Qualifed Names for Modules Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan @@ -8,19 +8,41 @@ Content-Type: text/x-rst Created: 4-Mar-2011 Python-Version: 3.3 -Post-History: 5-Mar-2011 +Post-History: 5-Mar-2011, 19-Nov-2011 Abstract ======== This PEP proposes new mechanisms that eliminate some longstanding traps for -the unwary when dealing with Python's import system, the pickle module and -introspection interfaces. +the unwary when dealing with Python's import system, as well as serialisation +and introspection of functions and classes. It builds on the "Qualified Name" concept defined in PEP 3155. +Relationship with Other PEPs +---------------------------- + +This PEP builds on the "qualified name" concept introduced by PEP 3155, and +also shares in that PEP's aim of fixing some ugly corner cases when dealing +with serialisation of arbitrary functions and classes. + +It also builds on PEP 366, which took initial tentative steps towards making +explicit relative imports from the main module work correctly in at least +*some* circumstances. + +This PEP is also affected by the two competing "namespace package" PEPs +(PEP 382 and PEP 402). This PEP would require some minor adjustments to +accommodate PEP 382, but has some critical incompatibilities with respect to +the implicit namespace package mechanism proposed in PEP 402. + +Finally, PEP 328 eliminated implicit relative imports from imported modules. +This PEP proposes that the de facto implicit relative imports from main +modules that are provided by the current initialisation behaviour for +``sys.path[0]`` also be eliminated. + + What's in a ``__name__``? ========================= @@ -41,71 +63,173 @@ Traps for the Unwary ==================== -The overloading of the semantics of ``__name__`` have resulted in several -traps for the unwary. These traps can be quite annoying in practice, as -they are highly unobvious and can cause quite confusing behaviour. A lot of -the time, you won't even notice them, which just makes them all the more -surprising when they do come up. +The overloading of the semantics of ``__name__``, along with some historically +associated behaviour in the initialisation of ``sys.path[0]``, has resulted in +several traps for the unwary. These traps can be quite annoying in practice, +as they are highly unobvious (especially to beginners) and can cause quite +confusing behaviour. + + +Why are my imports broken? +-------------------------- + +There's a general principle that applies when modifying ``sys.path``: *never* +put a package directory directly on ``sys.path``. The reason this is +problematic is that every module in that directory is now potentially +accessible under two different names: as a top level module (since the +package directory is on ``sys.path``) and as a submodule of the package (if +the higher level directory containing the package itself is also on +``sys.path``). + +As an example, Django (up to and including version 1.3) is guilty of setting +up exactly this situation for site-specific applications - the application +ends up being accessible as both ``app`` and ``site.app`` in the module +namespace, and these are actually two *different* copies of the module. This +is a recipe for confusion if there is any meaningful mutable module level +state, so this behaviour is being eliminated from the default site set up in +version 1.4 (site-specific apps will always be fully qualified with the site +name). + +However, it's hard to blame Django for this, when the same part of Python +responsible for setting ``__name__ = "__main__"`` in the main module commits +the exact same error when determining the value for ``sys.path[0]``. + +The impact of this can be seen relatively frequently if you follow the +"python" and "import" tags on Stack Overflow. When I had the time to follow +it myself, I regularly encountered people struggling to understand the +behaviour of straightforward package layouts like the following (I actually +use package layouts along these lines in my own projects):: + + project/ + setup.py + example/ + __init__.py + foo.py + tests/ + __init__.py + test_foo.py + +While I would often see it without the ``__init__.py`` files first, that's a +trivial fix to explain. What's hard to explain is that all of the following +ways to invoke ``test_foo.py`` *probably won't work* due to broken imports +(either failing to find ``example`` for absolute imports, complaining +about relative imports in a non-package or beyond the toplevel package for +explicit relative imports, or issuing even more obscure errors if some other +submodule happens to shadow the name of a top-level module, such as an +``example.json`` module that handled serialisation or an +``example.tests.unittest`` test runner):: + + # These commands will most likely *FAIL*, even if the code is correct + + # working directory: project/example/tests + ./test_foo.py + python test_foo.py + python -m package.tests.test_foo + python -c "from package.tests.test_foo import main; main()" + + # working directory: project/package + tests/test_foo.py + python tests/test_foo.py + python -m package.tests.test_foo + python -c "from package.tests.test_foo import main; main()" + + # working directory: project + example/tests/test_foo.py + python example/tests/test_foo.py + + # working directory: project/.. + project/example/tests/test_foo.py + python project/example/tests/test_foo.py + # The -m and -c approaches don't work from here either, but the failure + # to find 'package' correctly is easier to explain in this case + +That's right, that long list is of all the methods of invocation that will +almost certainly *break* if you try them, and the error messages won't make +any sense if you're not already intimately familiar not only with the way +Python's import system works, but also with how it gets initialised. + +For a long time, the only way to get ``sys.path`` right with that kind of +setup was to either set it manually in ``test_foo.py`` itself (hardly +something a novice, or even many veteran, Python programmers are going to +know how to do) or else to make sure to import the module instead of +executing it directly:: + + # working directory: project + python -c "from package.tests.test_foo import main; main()" + +Since the implementation of PEP 366 (which defined a mechanism that allows +relative imports to work correctly when a module inside a package is executed +via the ``-m`` switch), the following also works properly:: + + # working directory: project + python -m package.tests.test_foo + +The fact that most methods of invoking Python code from the command line +break when that code is inside a package, and the two that do work are highly +sensitive to the current working directory is all thoroughly confusing for a +beginner. I personally believe it is one of the key factors leading +to the perception that Python packages are complicated and hard to get right. + +This problem isn't even limited to the command line - if ``test_foo.py`` is +open in Idle and you attempt to run it by pressing F5, or if you try to run +it by clicking on it in a graphical filebrowser, then it will fail in just +the same way it would if run directly from the command line. + +There's a reason the general "no package directories on ``sys.path``" +guideline exists, and the fact that the interpreter itself doesn't follow +it when determining ``sys.path[0]`` is the root cause of all sorts of grief. + +In the past, this couldn't be fixed due to backwards compatibility concerns. +However, scripts potentially affected by this problem will *already* require +fixes when porting to the Python 3.x (due to the elimination of implicit +relative imports when importing modules normally). This provides a convenient +opportunity to implement a corresponding change in the initialisation +semantics for ``sys.path[0]``. Importing the main module twice ------------------------------- -The most venerable of these traps is the issue of (effectively) importing -``__main__`` twice. This occurs when the main module is also imported under -its real name, effectively creating two instances of the same module under -different names. +Another venerable trap is the issue of importing ``__main__`` twice. This +occurs when the main module is also imported under its real name, effectively +creating two instances of the same module under different names. -This problem used to be significantly worse due to implicit relative imports -from the main module, but the switch to allowing only absolute imports and -explicit relative imports means this issue is now restricted to affecting the -main module itself. - - -Why are my relative imports broken? ------------------------------------ - -PEP 366 defines a mechanism that allows relative imports to work correctly -when a module inside a package is executed via the ``-m`` switch. - -Unfortunately, many users still attempt to directly execute scripts inside -packages. While this no longer silently does the wrong thing by -creating duplicate copies of peer modules due to implicit relative imports, it -now fails noisily at the first explicit relative import, even though the -interpreter actually has sufficient information available on the filesystem to -make it work properly. - - +If the state stored in ``__main__`` is significant to the correct operation +of the program, or if there is top-level code in the main module that has +non-idempotent side effects, then this duplication can cause obscure and +surprising errors. In a bit of a pickle -------------------- -Something many users may not realise is that the ``pickle`` module serialises -objects based on the ``__name__`` of the containing module. So objects -defined in ``__main__`` are pickled that way, and won't be unpickled -correctly by another python instance that only imported that module instead -of running it directly. This behaviour is the underlying reason for the -advice from many Python veterans to do as little as possible in the -``__main__`` module in any application that involves any form of object -serialisation and persistence. +Something many users may not realise is that the ``pickle`` module sometimes +relies on the ``__module__`` attribute when serialising instances of arbitrary +classes. So instances of classes defined in ``__main__`` are pickled that way, +and won't be unpickled correctly by another python instance that only imported +that module instead of running it directly. This behaviour is the underlying +reason for the advice from many Python veterans to do as little as possible +in the ``__main__`` module in any application that involves any form of +object serialisation and persistence. -Similarly, when creating a pseudo-module\*, pickles rely on the name of the -module where a class is actually defined, rather than the officially -documented location for that class in the module hierarchy. +Similarly, when creating a pseudo-module (see next paragraph), pickles rely +on the name of the module where a class is actually defined, rather than the +officially documented location for that class in the module hierarchy. + +For the purposes of this PEP, a "pseudo-module" is a package designed like +the Python 3.2 ``unittest`` and ``concurrent.futures`` packages. These +packages are documented as if they were single modules, but are in fact +internally implemented as a package. This is *supposed* to be an +implementation detail that users and other implementations don't need to +worry about, but, thanks to ``pickle`` (and serialisation in general), +the details are often exposed and can effectively become part of the public +API. While this PEP focuses specifically on ``pickle`` as the principal serialisation scheme in the standard library, this issue may also affect -other mechanisms that support serialisation of arbitrary class instances. - -\*For the purposes of this PEP, a "pseudo-module" is a package designed like -the Python 3.2 ``unittest`` and ``concurrent.futures`` packages. These -packages are documented as if they were single modules, but are in fact -internally implemented as a package. This is *supposed* to be an -implementation detail that users and other implementations don't need to worry -about, but, thanks to ``pickle`` (and serialisation in general), the details -are exposed and effectively become part of the public API. +other mechanisms that support serialisation of arbitrary class instances +and rely on ``__module__`` attributes to determine how to handle +deserialisation. Where's the source? @@ -134,15 +258,73 @@ executed script or top-level module. Packages and non-top-level modules executed via the ``-m`` switch, as well as directly executed zipfiles or directories, are likely to make multiprocessing on Windows do the wrong thing -(either quietly or noisily) when spawning a new process. +(either quietly or noisily, depending on application details) when spawning a +new process. While this issue currently only affects Windows directly, it also impacts any proposals to provide Windows-style "clean process" invocation via the multiprocessing module on other platforms. -Proposed Changes -================ +Qualified Names for Modules +=========================== + +To make it feasible to fix these problems once and for all, it is proposed +to add a new module level attribute: ``__qualname__``. This abbreviation of +"qualified name" is taken from PEP 3155, where it is used to store the naming +path to a nested class or function definition relative to the top level +module. + +For modules, ``__qualname__`` will normally be the same as ``__name__``, just +as it is for top-level functions and classes in PEP 3155. However, it will +differ in some situations so that the above problems can be addressed. + +Specifically, whenever ``__name__`` is modified for some other purpose (such +as to denote the main module), then ``__qualname__`` will remain unchanged, +allowing code that needs it to access the original unmodified value. + +If a module loader does not initialise ``__qualname__`` itself, then the +import system will add it automatically (setting it to the same value as +``__name__``). + + +Alternative Names +----------------- + +Two alternative names were also considered for the new attribute: "full name" +(``__fullname__``) and "implementation name" (``__implname__``). + +Either of those would actually be valid for the use case in this PEP. +However, as a meta-issue, PEP 3155 is *also* adding a new attribute (for +functions and classes) that is "like ``__name__``, but different in some cases +where ``__name__`` is missing necessary information" and those terms aren't +accurate for the PEP 3155 function and class use case. + +PEP 3155 deliberately omits the module information, so the term "full name" +is simply untrue, and "implementation name" implies that it may specify an +object other than that specified by ``__name__``, and that is never the +case for PEP 3155 (in that PEP, ``__name__`` and ``__qualname__`` always +refer to the same function or class, it's just that ``__name__`` is +insufficient to accurately identify nested functions and classes). + +Since it seems needlessly inconsistent to add *two* new terms for attributes +that only exist because backwards compatibility concerns keep us from +changing the behaviour of ``__name__`` itself, this PEP instead chose to +adopt the PEP 3155 terminology. + +If the relative inscrutability of "qualified name" and ``__qualname__`` +encourages interested developers to look them up at least once rather than +assuming they know what they mean just from the name and guessing wrong, +that's not necessarily a bad outcome. + +Besides, 99% of Python developers should never need to even care these extra +attributes exist - they're really an implementation detail to let us fix a +few problematic behaviours exhibited by imports, pickling and introspection, +not something people are going to be dealing with on a regular basis. + + +Eliminating the Traps +===================== The following changes are interrelated and make the most sense when considered together. They collectively either completely eliminate the traps @@ -150,104 +332,362 @@ dealing with them. A rough draft of some of the concepts presented here was first posted on the -python-ideas list [1], but they have evolved considerably since first being -discussed in that thread. +python-ideas list ([1]_), but they have evolved considerably since first being +discussed in that thread. Further discussion has subsequently taken place on +the import-sig mailing list ([2]_. [3]_). + + +Fixing main module imports inside packages +------------------------------------------ + +To eliminate this trap, it is proposed that an additional filesystem check be +performed when determining a suitable value for ``sys.path[0]``. This check +will look for Python's explicit package directory markers and use them to find +the appropriate directory to add to ``sys.path``. + +The current algorithm for setting ``sys.path[0]`` in relevant cases is roughly +as follows:: + + # Interactive prompt, -m switch, -c switch + sys.path.insert(0, '') + +:: + + # Valid sys.path entry execution (i.e. directory and zip execution) + sys.path.insert(0, sys.argv[0]) + +:: + + # Direct script execution + sys.path.insert(0, os.path.dirname(sys.argv[0])) + +It is proposed that this initialisation process be modified to take +package details stored on the filesystem into account:: + + # Interactive prompt, -m switch, -c switch + in_package, path_entry, _ignored = split_path_module(os.getcwd(), '') + if in_package: + sys.path.insert(0, path_entry) + else: + sys.path.insert(0, '') + + # Start interactive prompt or run -c command as usual + # __main__.__qualname__ is set to "__main__" + + # The -m switches uses the same sys.path[0] calculation, but: + # modname is the argument to the -m switch + # modname is passed to ``runpy._run_module_as_main()`` as usual + # __main__.__qualname__ is set to modname + +:: + + # Valid sys.path entry execution (i.e. directory and zip execution) + modname = "__main__" + path_entry, modname = split_path_module(sys.argv[0], modname) + sys.path.insert(0, path_entry) + + # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()`` + # __main__.__qualname__ is set to modname + +:: + + # Direct script execution + in_package, path_entry, modname = split_path_module(sys.argv[0]) + sys.path.insert(0, path_entry) + if in_package: + # Pass modname to ``runpy._run_module_as_main()`` + else: + # Run script directly + # __main__.__qualname__ is set to modname + +The ``split_path_module()`` supporting function used in the above pseudo-code +would have the following semantics:: + + def _splitmodname(fspath): + path_entry, fname = os.path.split(fspath) + modname = os.path.splitext(fname)[0] + return path_entry, modname + + def _is_package_dir(fspath): + return any(os.exists("__init__" + info[0]) for info + in imp.get_suffixes()) + + def split_path_module(fspath, modname=None): + """Given a filesystem path and a relative module name, determine an + appropriate sys.path entry and a fully qualified module name. + + Returns a 3-tuple of (package_depth, fspath, modname). A reported + package depth of 0 indicates that this would be a top level import. + + If no relative module name is given, it is derived from the final + component in the supplied path with the extension stripped. + """ + if modname is None: + fspath, modname = _splitmodname(fspath) + package_depth = 0 + while _is_package_dir(fspath): + fspath, pkg = _splitmodname(fspath) + modname = pkg + '.' + modname + return package_depth, fspath, modname + +This PEP also proposes that the ``split_path_module()`` functionality be +exposed directly to Python users via the ``runpy`` module. + +With this fix in place, and the same simple package layout described earlier, +*all* of the following commands would invoke the test suite correctly:: + + # working directory: project/example/tests + ./test_foo.py + python test_foo.py + python -m package.tests.test_foo + python -c "from .test_foo import main; main()" + python -c "from ..tests.test_foo import main; main()" + python -c "from package.tests.test_foo import main; main()" + + # working directory: project/package + tests/test_foo.py + python tests/test_foo.py + python -m package.tests.test_foo + python -c "from .tests.test_foo import main; main()" + python -c "from package.tests.test_foo import main; main()" + + # working directory: project + example/tests/test_foo.py + python example/tests/test_foo.py + python -m package.tests.test_foo + python -c "from package.tests.test_foo import main; main()" + + # working directory: project/.. + project/example/tests/test_foo.py + python project/example/tests/test_foo.py + # The -m and -c approaches still don't work from here, but the failure + # to find 'package' correctly is pretty easy to explain in this case + +With these changes, clicking Python modules in a graphical file browser +should always execute them correctly, even if they live inside a package. +Depending on the details of how it invokes the script, Idle would likely also +be able to run ``test_foo.py`` correctly with F5, without needing any Idle +specific fixes. + +Optional addition: command line relative imports +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +With the above changes in place, it would be a fairly minor addition to allow +explicit relative imports as arguments to the ``-m`` switch:: + + # working directory: project/example/tests + python -m .test_foo + python -m ..tests.test_foo + + # working directory: project/example/ + python -m .tests.test_foo + +With this addition, system initialisation for the ``-m`` switch would change +as follows:: + + # -m switch (permitting explicit relative imports) + in_package, path_entry, pkg_name = split_path_module(os.getcwd(), '') + qualname= <> + if qualname.startswith('.'): + modname = qualname + while modname.startswith('.'): + modname = modname[1:] + pkg_name, sep, _ignored = pkg_name.rpartition('.') + if not sep: + raise ImportError("Attempted relative import beyond top level package") + qualname = pkg_name + '.' modname + if in_package: + sys.path.insert(0, path_entry) + else: + sys.path.insert(0, '') + + # qualname is passed to ``runpy._run_module_as_main()`` + # _main__.__qualname__ is set to qualname + + + +Compatibility with PEP 382 +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Making this proposal compatible with the PEP 382 namespace packaging PEP is +trivial. The semantics of ``_is_package_dir()`` are merely changed to be:: + + def _is_package_dir(fspath): + return (fspath.endswith(".pyp") or + any(os.exists("__init__" + info[0]) for info + in imp.get_suffixes())) + + +Incompatibility with PEP 402 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +PEP 402 proposes the elimination of explicit markers in the file system for +Python packages. This fundamentally breaks the proposed concept of being able +to take a filesystem path and a Python module name and work out an unambiguous +mapping to the Python module namespace. Instead, the appropriate mapping +would depend on the current values in ``sys.path``, rendering it impossible +to ever fix the problems described above with the calculation of +``sys.path[0]`` when the interpreter is initialised. + +While some aspects of this PEP could probably be salvaged if PEP 402 were +adopted, the core concept of making import semantics from main and other +modules more consistent would no longer be feasible. + +This incompatibility is discussed in more detail in the relevant import-sig +threads ([2]_, [3]_). + + +Potential incompatibilities with scripts stored in packages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The proposed change to ``sys.path[0]`` initialisation *may* break some +existing code. Specifically, it will break scripts stored in package +directories that rely on the implicit relative imports from ``__main__`` in +order to run correctly under Python 3. + +While such scripts could be imported in Python 2 (due to implicit relative +imports) it is already the case that they cannot be imported in Python 3, +as implicit relative imports are no longer permitted when a module is +imported. + +By disallowing implicit relatives imports from the main module as well, +such modules won't even work as scripts with this PEP. Switching them +over to explicit relative imports will then get them working again as +both executable scripts *and* as importable modules. + +To support earlier versions of Python, a script could be written to use +different forms of import based on the Python version:: + + if __name__ == "__main__" and sys.version_info < (3, 3): + import peer # Implicit relative import + else: + from . import peer # explicit relative import Fixing dual imports of the main module -------------------------------------- -Two simple changes are proposed to fix this problem: +Given the above proposal to get ``__qualname__`` consistently set correctly +in the main module, one simple change is proposed to eliminate the problem +of dual imports of the main module: the addition of a ``sys.metapath`` hook +that detects attempts to import ``__main__`` under its real name and returns +the original main module instead:: -1. In ``runpy``, modify the implementation of the ``-m`` switch handling to - install the specified module in ``sys.modules`` under both its real name - and the name ``__main__``. (Currently it is only installed as the latter) -2. When directly executing a module, install it in ``sys.modules`` under - ``os.path.splitext(os.path.basename(__file__))[0]`` as well as under - ``__main__``. + class AliasImporter: + def __init__(self, module, alias): + self.module = module + self.alias = alias -With the main module also stored under its "real" name, attempts to import it -will pick it up from the ``sys.modules`` cache rather than reimporting it -under the new name. + def __repr__(self): + fmt = "{0.__class__.__name__}({0.module.__name__}, {0.alias})" + return fmt.format(self) + def find_module(self, fullname, path=None): + if path is None and fullname == self.alias: + return self + return None -Fixing direct execution inside packages ---------------------------------------- + def load_module(self, fullname): + if fullname != self.alias: + raise ImportError("{!r} cannot load {!r}".format(self, fullname)) + return self.main_module -To fix this problem, it is proposed that an additional filesystem check be -performed before proceeding with direct execution of a ``PY_SOURCE`` or -``PY_COMPILED`` file that has been named on the command line. +This metapath hook would be added automatically during import system +initialisation based on the following logic:: -This additional check would look for an ``__init__`` file that is a peer to -the specified file with a matching extension (either ``.py``, ``.pyc`` or -``.pyo``, depending what was passed on the command line). + main = sys.modules["__main__"] + if main.__name__ != main.__qualname__: + sys.metapath.append(AliasImporter(main, main.__qualname__)) -If this check fails to find anything, direct execution proceeds as usual. - -If, however, it finds something, execution is handed over to a -helper function in the ``runpy`` module that ``runpy.run_path`` also invokes -in the same circumstances. That function will walk back up the -directory hierarchy from the supplied path, looking for the first directory -that doesn't contain an ``__init__`` file. Once that directory is found, it -will be set to ``sys.path[0]``, ``sys.argv[0]`` will be set to ``-m`` and -``runpy._run_module_as_main`` will be invoked with the appropriate module -name (as calculated based on the original filename and the directories -traversed while looking for a directory without an ``__init__`` file). - -The two current PEPs for namespace packages (PEP 382 and PEP 402) would both -affect this part of the proposal. For PEP 382 (with its current suggestion of -"*.pyp" package directories, this check would instead just walk up the -supplied path, looking for the first non-package directory (this would not -require any filesystem stat calls). Since PEP 402 deliberately omits explicit -directory markers, it would need an alternative approach, based on checking -the supplied path against the contents of ``sys.path``. In both cases, the -direct execution behaviour can still be corrected. +This is probably the least important proposal in the PEP - it just +closes off the last mechanism that is likely to lead to module duplication +after the configuration of ``sys.path[0]`` at interpreter startup is +addressed. Fixing pickling without breaking introspection ---------------------------------------------- -To fix this problem, it is proposed to add a new optional module level -attribute: ``__qname__``. This abbreviation of "qualified name" is taken -from PEP 3155, where it is used to store the naming path to a nested class -or function definition relative to the top level module. By default, -``__qname__`` will be the same as ``__name__``, which covers the typical -case where there is a one-to-one correspondence between the documented API -and the actual module implementation. +To fix this problem, it is proposed to make use of the new module level +``__qualname__`` attributes to determine the real module location when +``__name__`` has been modified for any reason. -Functions and classes will gain a corresponding ``__qmodule__`` attribute -that refers to their module's ``__qname__``. +In the main module, ``__qualname__`` will automatically be set to the main +module's "real" name (as described above) by the interpreter. Pseudo-modules that adjust ``__name__`` to point to the public namespace will -leave ``__qname__`` untouched, so the implementation location remains readily +leave ``__qualname__`` untouched, so the implementation location remains readily accessible for introspection. -In the main module, ``__qname__`` will automatically be set to the main -module's "real" name (as described above under the fix to prevent duplicate -imports of the main module) by the interpreter. +If ``__name__`` is adjusted at the top of a module, then this will +automatically adjust the ``__module__`` attribute for all functions and +classes subsequently defined in that module. -At the interactive prompt, both ``__name__`` and ``__qname__`` will be set -to ``"__main__"``. +Since multiple submodules may be set to use the same "public" namespace, +functions and classes will be given a new ``__qualmodule__`` attribute +that refers to the ``__qualname__`` of their module. -These changes on their own will fix most pickling and serialisation problems, -but one additional change is needed to fix the problem with serialisation of -items in ``__main__``: as a slight adjustment to the definition process for -functions and classes, in the ``__name__ == "__main__"`` case, the module -``__qname__`` attribute will be used to set ``__module__``. +This isn't strictly necessary for functions (you could find out their +module's qualified name by looking in their globals dictionary), but it is +needed for classes, since they don't hold a reference to the globals of +their defining module. Once a new attribute is added to classes, it is +more convenient to keep the API consistent and add a new attribute to +functions as well. -``pydoc`` and ``inspect`` would also be updated appropriately to: -- use ``__qname__`` instead of ``__name__`` and ``__qmodule__`` instead of - ``__module__``where appropriate (e.g. ``inspect.getsource()`` would prefer - the qualified variants) -- report both the public names and the qualified names for affected objects +These changes mean that adjusting ``__name__`` (and, either directly or +indirectly, the corresponding function and class ``__module__`` attributes) +becomes the officially sanctioned way to implement a namespace as a package, +while exposing the API as if it were still a single module. + +All serialisation code that currently uses ``__name__`` and ``__module__`` +attributes will then avoid exposing implementation details by default. + +To correctly handle serialisation of items from the main module, the class +and function definition logic will be updated to also use ``__qualname__`` +for the ``__module__`` attribute in the case where ``__name__ == "__main__"``. + +With ``__name__`` and ``__module__`` being officially blessed as being used +for the *public* names of things, the introspection tools in the standard +library will be updated to use ``__qualname__`` and ``__qualmodule__`` +where appropriate. For example: + +- ``pydoc`` will report both public and qualified names for modules +- ``inspect.getsource()`` (and similar tools) will use the qualified names + that point to the implementation of the code +- additional ``pydoc`` and/or ``inspect`` APIs may be provided that report + all modules with a given public ``__name__``. + Fixing multiprocessing on Windows --------------------------------- -With ``__qname__`` now available to tell ``multiprocessing`` the real -name of the main module, it should be able to simply include it in the +With ``__qualname__`` now available to tell ``multiprocessing`` the real +name of the main module, it will be able to simply include it in the serialised information passed to the child process, eliminating the -need for dubious reverse engineering of the ``__file__`` attribute. +need for the current dubious introspection of the ``__file__`` attribute. + +For older Python versions, ``multiprocessing`` could be improved by applying +the ``split_path_module()`` algorithm described above when attempting to +work out how to execute the main module based on its ``__file__`` attribute. + + +Explicit relative imports +========================= + +This PEP proposes that ``__package__`` be unconditionally defined in the +main module as ``__qualname__.rpartition('.')[0]``. Aside from that, it +proposes that the behaviour of explicit relative imports be left alone. + +In particular, if ``__package__`` is not set in a module when an explicit +relative import occurs, the automatically cached value will continue to be +derived from ``__name__`` rather than ``__qualname__``. This minimises any +backwards incompatibilities with existing code that deliberately manipulates +relative imports by adjusting ``__name__`` rather than setting ``__package__`` +directly. + +This PEP does *not* propose that ``__package__`` be deprecated. While it is +technically redundant following the introduction of ``__qualname__``, it just +isn't worth the hassle of deprecating it within the lifetime of Python 3.x. Reference Implementation @@ -262,6 +702,14 @@ .. [1] Module aliases and/or "real names" (http://mail.python.org/pipermail/python-ideas/2011-January/008983.html) +.. [2] PEP 395 (Module aliasing) and the namespace PEPs + (http://mail.python.org/pipermail/import-sig/2011-November/000382.html) + +.. [3] Updated PEP 395 (aka "Implicit Relative Imports Must Die!") + (http://mail.python.org/pipermail/import-sig/2011-November/000397.html) + +.. [4] Elaboration of compatibility problems between this PEP and PEP 402 + (http://mail.python.org/pipermail/import-sig/2011-November/000403.html) Copyright ========= diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -57,29 +57,40 @@ Features for 3.3 ================ +Implemented / Final PEPs: + +* PEP 380: Syntax for Delegating to a Subgenerator +* PEP 393: Flexible String Representation +* PEP 399: Pure Python/C Accelerator Module Compatibility Requirements +* PEP 409: Suppressing exception context +* PEP 414: Explicit Unicode Literal for Python 3.3 +* PEP 3151: Reworking the OS and IO exception hierarchy +* PEP 3155: Qualified name for classes and functions + +Other final large-scale changes: + +* Addition of the "packaging" module, deprecating "distutils" +* Addition of the "faulthandler" module +* Addition of the "lzma" module, and lzma/xz support in tarfile + Candidate PEPs: * PEP 362: Function Signature Object -* PEP 380: Syntax for Delegating to a Subgenerator * PEP 382: Namespace Packages -* PEP 393: Flexible String Representation * PEP 395: Module Aliasing * PEP 397: Python launcher for Windows * PEP 3143: Standard daemon process library -* PEP 3151: Reworking the OS and IO exception hierarchy (Note that these are not accepted yet and even if they are, they might not be finished in time for Python 3.3.) Other planned large-scale changes: -* Addition of the "packaging" module, replacing "distutils" +* Addition of the "regex" module +* Email version 6 * Implementing ``__import__`` using importlib -* Email version 6 * A standard event-loop interface (PEP by Jim Fulton pending) -* Adding the faulthandler module. * Breaking out standard library and docs in separate repos? -* A PEP on supplementing C modules with equivalent Python modules? Copyright diff --git a/pep-0400.txt b/pep-0400.txt --- a/pep-0400.txt +++ b/pep-0400.txt @@ -2,7 +2,7 @@ Title: Deprecate codecs.StreamReader and codecs.StreamWriter Version: $Revision$ Last-Modified: $Date$ -Author: Victor Stinner +Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst diff --git a/pep-0403.txt b/pep-0403.txt --- a/pep-0403.txt +++ b/pep-0403.txt @@ -1,13 +1,13 @@ PEP: 403 -Title: Prefix syntax for post function definition operations +Title: Statement local functions and classes Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan -Status: Withdrawn +Status: Deferred Type: Standards Track Content-Type: text/x-rst Created: 2011-10-13 -Python-Version: 3.x +Python-Version: 3.4 Post-History: 2011-10-13 Resolution: TBD @@ -15,33 +15,23 @@ Abstract ======== -This PEP proposes the addition of ``postdef`` as a new function prefix -syntax (analogous to decorators) that permits the execution of a single simple -statement (potentially including substatements separated by semi-colons) after +This PEP proposes the addition of a new ``in`` statement that accepts a +statement local function or class definition. -In addition, the new syntax would allow the 'def' keyword to be used to refer -to the function being defined without needing to repeat the name. +The statement accepts a single simple statement that can make a forward +reference to a trailing function or class definition. -When the 'postdef' prefix syntax is used, the associated statement would be -executed *in addition to* the normal local name binding implicit in function -definitions. Any name collision are expected to be minor, analagous to those -encountered with ``for`` loop iteration variables. +This new statement is designed to be used whenever a "one-shot" function or +class is needed, and placing the function or class definition before the +statement that uses it actually makes the code harder to read. It also +avoids any name shadowing concerns by making sure the new name is visible +only to the statement in the ``in`` clause. This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local Namespaces) so some elements of the rationale will be familiar to readers of that PEP. That PEP has now been withdrawn in favour of this one. -PEP Withdrawal -============== - -The python-ideas thread discussing this PEP [1]_ persuaded me that it was -essentially am unnecessarily cryptic, wholly inferior version of PEP 3150's -statement local namespaces. The discussion also resolved some of my concerns -with PEP 3150, so I am withdrawing this more limited version of the idea in -favour of resurrecting the original concept. - - Basic Examples ============== @@ -49,14 +39,14 @@ rationale for this specific proposed solution, here are a few simple examples of the kind of code it is designed to simplify. -As a trivial example, weakref callbacks could be defined as follows:: +As a trivial example, a weakref callback could be defined as follows:: - postdef x = weakref.ref(target, def) + in x = weakref.ref(target, report_destruction) def report_destruction(obj): print("{} is being destroyed".format(obj)) -This contrasts with the current repetitive "out of order" syntax for this -operation:: +This contrasts with the current (conceptually) "out of order" syntax for +this operation:: def report_destruction(obj): print("{} is being destroyed".format(obj)) @@ -66,11 +56,19 @@ That structure is OK when you're using the callable multiple times, but it's irritating to be forced into it for one-off operations. +If the repetition of the name seems especially annoying, then a throwaway +name like ``f`` can be used instead:: + + in x = weakref.ref(target, f) + def f(obj): + print("{} is being destroyed".format(obj)) + + Similarly, a sorted operation on a particularly poorly defined type could now be defined as:: - postdef sorted_list = sorted(original, key=def) - def force_sort(item): + in sorted_list = sorted(original, key=f) + def f(item): try: return item.calc_sort_order() except NotSortableError: @@ -88,32 +86,41 @@ And early binding semantics in a list comprehension could be attained via:: - postdef funcs = [def(i) for i in range(10)] - def make_incrementor(i): - postdef return def - def incrementor(x): - return x + i + in funcs = [adder(i) for i in range(10)] + def adder(i): + return lambda x: x + i Proposal ======== -This PEP proposes the addition of an optional block prefix clause to the -syntax for function and class definitions. +This PEP proposes the addition of a new ``in`` statement that is a variant +of the existing class and function definition syntax. -This block prefix would be introduced by a leading ``postdef`` and would be -allowed to contain any simple statement (including those that don't -make any sense in that context - while such code would be legal, -there wouldn't be any point in writing it). This permissive structure is -easier to define and easier to explain, but a more restrictive approach that -only permits operations that "make sense" would also be possible (see PEP -3150 for a list of possible candidates) +The new ``in`` clause replaces the decorator lines, and allows forward +references to the trailing function or class definition. -The function definition keyword ``def`` would be repurposed inside the block prefix -to refer to the function being defined. +The trailing function or class definition is always named - the name of +the trailing definition is then used to make the forward reference from the +preceding statement. -When a block prefix is provided, the standard local name binding implicit -in the function definition still takes place. +The ``in`` clause is allowed to contain any simple statement (including those +that don't make any sense in that context, such as ``pass`` - while such code +would be legal, there wouldn't be any point in writing it). This permissive +structure is easier to define and easier to explain, but a more restrictive +approach that only permits operations that "make sense" would also be +possible (see PEP 3150 for a list of possible candidates). + +The ``in`` statement will not create a new scope - all name binding +operations aside from the trailing function or class definition will affect +the containing scope. + +The name used in the trailing function or class definition is only visible +from the associated ``in`` clause, and behaves as if it was an ordinary +variable defined in that scope. If any nested scopes are created in either +the ``in`` clause or the trailing function or class definition, those scopes +will see the trailing function or class definition rather than any other +bindings for that name in the containing scope. Background @@ -125,10 +132,11 @@ so much: Python's demand that the function be named and introduced before the operation that needs it breaks the developer's flow of thought. They get to a point where they go "I need a one-shot operation that does -", and instead of being able to just *say* that, they instead have to back -up, name a function to do , then call that function from the operation -they actually wanted to do in the first place. Lambda expressions can help -sometimes, but they're no substitute for being able to use a full suite. +", and instead of being able to just *say* that directly, they instead +have to back up, name a function to do , then call that function from +the operation they actually wanted to do in the first place. Lambda +expressions can help sometimes, but they're no substitute for being able to +use a full suite. Ruby's block syntax also heavily inspired the style of the solution in this PEP, by making it clear that even when limited to *one* anonymous function per @@ -144,13 +152,19 @@ However, adopting Ruby's block syntax directly won't work for Python, since the effectiveness of Ruby's blocks relies heavily on various conventions in -the way functions are *defined* (specifically, Ruby's ``yield`` syntax to -call blocks directly and the ``&arg`` mechanism to accept a block as a +the way functions are *defined* (specifically, using Ruby's ``yield`` syntax +to call blocks directly and the ``&arg`` mechanism to accept a block as a function's final argument). Since Python has relied on named functions for so long, the signatures of APIs that accept callbacks are far more diverse, thus requiring a solution -that allows anonymous functions to be slotted in at the appropriate location. +that allows one-shot functions to be slotted in at the appropriate location. + +The approach taken in this PEP is to retain the requirement to name the +function explicitly, but allow the relative order of the definition and the +statement that references it to be changed to match the developer's flow of +thought. The rationale is essentially the same as that used when introducing +decorators, but covering a broader set of applications. Relation to PEP 3150 @@ -166,8 +180,9 @@ This PEP also achieves most of the other effects described in PEP 3150 without introducing a new brainbending kind of scope. All of the complex -scoping rules in PEP 3150 are replaced in this PEP with the simple ``def`` -reference to the associated function definition. +scoping rules in PEP 3150 are replaced in this PEP with allowing a forward +reference to the associated function or class definition without creating an +actual name binding in the current scope. Keyword Choice @@ -176,53 +191,49 @@ The proposal definitely requires *some* kind of prefix to avoid parsing ambiguity and backwards compatibility problems with existing constructs. It also needs to be clearly highlighted to readers, since it declares that -the following piece of code is going to be executed out of order. +the following piece of code is going to be executed only after the trailing +function or class definition has been executed. -The 'postdef' keyword was chosen as a literal explanation of exactly what -the new clause does: execute the specified statement *after* the associated -function definition, even though it is physically written *before* the -definition in the source code. +The ``in`` keyword was chosen as an existing keyword that can be used to +denote the concept of a forward reference. +For functions, the construct is intended to be read as "in define NAME as a function that does ". -Requirement to Name Functions -============================= +The mapping to English prose isn't as obvious for the class definition case, +but the concept remains the same. + + +Better Debugging Support for Functions and Classes with Short Names +=================================================================== One of the objections to widespread use of lambda expressions is that they -have an atrocious effect on traceback intelligibility and other aspects of -introspection. Accordingly, this PEP requires that even throwaway functions -be given some kind of name. +have a negative effect on traceback intelligibility and other aspects of +introspection. Similarly objections are raised regarding constructs that +promote short, cryptic function names (including this one, which requires +that the name of the trailing definition be supplied at least twice) -To help encourage the use of meaningful names without users having to repeat -themselves, the PEP suggests the provision of the ``def`` shorthand reference -to the current function from the ``postdef`` clause. +However, the introduction of qualified names in PEP 3155 means that even +anonymous classes and functions will now have different representations if +they occur in different scopes. For example:: + >>> def f(): + ... return lambda: y + ... + >>> f() + . at 0x7f6f46faeae0> + +Anonymous functions (or functions that share a name) within the *same* scope +will still share representations (aside from the object ID), but this is +still a major improvement over the historical situation where everything +*except* the object ID was identical. Syntax Change ============= -Current:: - - atom: ('(' [yield_expr|testlist_comp] ')' | - '[' [testlist_comp] ']' | - '{' [dictorsetmaker] '}' | - NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False') - -Changed:: - - atom: ('(' [yield_expr|testlist_comp] ')' | - '[' [testlist_comp] ']' | - '{' [dictorsetmaker] '}' | - NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' | 'def') - New:: - blockprefix: 'postdef' simple_stmt - block: blockprefix funcdef - -The above is the general idea, but I suspect that the change to the 'atom' -definition may cause an ambiguity problem in the parser when it comes to -detecting function definitions. So the actual implementation may need to be -more complex than that. + in_stmt: 'in' simple_stmt (classdef|funcdef) Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar @@ -230,14 +241,18 @@ Possible Implementation Strategy ================================ -This proposal has one titanic advantage over PEP 3150: implementation -should be relatively straightforward. +This proposal has at least one titanic advantage over PEP 3150: +implementation should be relatively straightforward. -The post definition statement can be incorporated into the AST for the -function node and simply visited out of sequence. +The AST for the ``in`` statement will include both the function or class +definition and the statement that references it, so it should just be a +matter of emitting the two operations out of order and using a hidden +variable to link up any references. -The one potentially tricky part is working out how to allow the dual -use of 'def' without rewriting half the grammar definition. +The one potentially tricky part is changing the meaning of the references to +the statement local function or namespace while within the scope of the +``in`` statement, but that shouldn't be too hard to address by maintaining +some additional state within the compiler. More Examples @@ -253,7 +268,7 @@ del _createenviron # Becomes: - postdef environ = def() + in environ = _createenviron() def _createenviron(): ... # 27 line function @@ -263,17 +278,25 @@ funcs = [(lambda x, i=i: x + i) for i in range(10)] # Becomes: - postdef funcs = [def(i) for i in range(10)] - def make_incrementor(i): + in funcs = [adder(i) for i in range(10)] + def adder(i): return lambda x: x + i # Or even: - postdef funcs = [def(i) for i in range(10)] - def make_incrementor(i): - postdef return def - def incrementor(x): + in funcs = [adder(i) for i in range(10)] + def adder(i): + in return incr + def incr(x): return x + i +A trailing class can be used as a statement local namespace:: + + # Evaluate subexpressions only once + in c = math.sqrt(x.a*x.a + x.b*x.b) + class x: + a = calculate_a() + b = calculate_b() + Reference Implementation ======================== @@ -288,15 +311,36 @@ idea what I was talking about in criticising Ruby's blocks, kicking off a rather enlightening process of investigation. -Even though this PEP has been withdrawn, the process of writing and arguing -in its favour has been quite influential on the future direction of PEP 3150. + +Rejected Concepts +================= + +A previous incarnation of this PEP (see [1]) proposed a much uglier syntax +that (quite rightly) was not well received. The current proposal is +significantly easier both to read and write. + +A more recent variant always used ``...`` for forward references, along +with genuinely anonymous function and class definitions. However, this +degenerated quickly into a mass of unintelligible dots in more complex +cases:: + + in funcs = [...(i) for i in range(10)] + def ...(i): + in return ... + def ...(x): + return x + i + + in c = math.sqrt(....a*....a + ....b*....b) + class ...: + a = calculate_a() + b = calculate_b() References ========== -[1] Start of python-ideas thread: - http://mail.python.org/pipermail/python-ideas/2011-October/012276.html +.. [1] Start of python-ideas thread: + http://mail.python.org/pipermail/python-ideas/2011-October/012276.html Copyright diff --git a/pep-0404.txt b/pep-0404.txt --- a/pep-0404.txt +++ b/pep-0404.txt @@ -1,536 +1,175 @@ PEP: 404 -Title: Python Virtual Environments +Title: Python 2.8 Un-release Schedule Version: $Revision$ Last-Modified: $Date$ -Author: Carl Meyer -Status: Draft -Type: Standards Track +Author: Barry Warsaw +Status: Final +Type: Informational Content-Type: text/x-rst -Created: 13-Jun-2011 -Python-Version: 3.3 -Post-History: 24-Oct-2011, 28-Oct-2011 +Created: 2011-11-09 +Python-Version: 2.8 Abstract ======== -This PEP proposes to add to Python a mechanism for lightweight -"virtual environments" with their own site directories, optionally -isolated from system site directories. Each virtual environment has -its own Python binary (allowing creation of environments with various -Python versions) and can have its own independent set of installed -Python packages in its site directories, but shares the standard -library with the base installed Python. +This document describes the un-development and un-release schedule for Python +2.8. -Motivation -========== +Un-release Manager and Crew +=========================== -The utility of Python virtual environments has already been well -established by the popularity of existing third-party -virtual-environment tools, primarily Ian Bicking's `virtualenv`_. -Virtual environments are already widely used for dependency management -and isolation, ease of installing and using Python packages without -system-administrator access, and automated testing of Python software -across multiple Python versions, among other uses. +============================ ================== +Position Name +============================ ================== +2.8 Un-release Manager Cardinal Biggles +============================ ================== -Existing virtual environment tools suffer from lack of support from -the behavior of Python itself. Tools such as `rvirtualenv`_, which do -not copy the Python binary into the virtual environment, cannot -provide reliable isolation from system site directories. Virtualenv, -which does copy the Python binary, is forced to duplicate much of -Python's ``site`` module and manually symlink/copy an ever-changing -set of standard-library modules into the virtual environment in order -to perform a delicate boot-strapping dance at every startup. -(Virtualenv must copy the binary in order to provide isolation, as -Python dereferences a symlinked executable before searching for -``sys.prefix``.) -The ``PYTHONHOME`` environment variable, Python's only existing -built-in solution for virtual environments, requires -copying/symlinking the entire standard library into every environment. -Copying the whole standard library is not a lightweight solution, and -cross-platform support for symlinks remains inconsistent (even on -Windows platforms that do support them, creating them often requires -administrator privileges). +Un-release Schedule +=================== -A virtual environment mechanism integrated with Python and drawing on -years of experience with existing third-party tools can be lower -maintenance, more reliable, and more easily available to all Python -users. +The current un-schedule is: -.. _virtualenv: http://www.virtualenv.org + - 2.8 final Never -.. _rvirtualenv: https://github.com/kvbik/rvirtualenv +Official pronouncement +====================== -Specification -============= +Rule number six: there is *no* official Python 2.8 release. There never will +be an official Python 2.8 release. It is an ex-release. Python 2.7 +is the end of the Python 2 line of development. -When the Python binary is executed, it attempts to determine its -prefix (which it stores in ``sys.prefix``), which is then used to find -the standard library and other key files, and by the ``site`` module -to determine the location of the site-package directories. Currently -the prefix is found (assuming ``PYTHONHOME`` is not set) by first -walking up the filesystem tree looking for a marker file (``os.py``) -that signifies the presence of the standard library, and if none is -found, falling back to the build-time prefix hardcoded in the binary. -This PEP proposes to add a new first step to this search. If a -``pyvenv.cfg`` file is found either adjacent to the Python executable, -or one directory above it, this file is scanned for lines of the form -``key = value``. If a ``home`` key is found, this signifies that the -Python binary belongs to a virtual environment, and the value of the -``home`` key is the directory containing the Python executable used to -create this virtual environment. +Upgrade path +============ -In this case, prefix-finding continues as normal using the value of -the ``home`` key as the effective Python binary location, which finds -the prefix of the base installation. ``sys.base_prefix`` is set to -this value, while ``sys.prefix`` is set to the directory containing -``pyvenv.cfg``. +The official upgrade path from Python 2.7 is to Python 3. -(If ``pyvenv.cfg`` is not found or does not contain the ``home`` key, -prefix-finding continues normally, and ``sys.prefix`` will be equal to -``sys.base_prefix``.) -The ``site`` and ``sysconfig`` standard-library modules are modified -such that the standard library and header files are are found relative -to ``sys.base_prefix``, while site-package directories ("purelib" and -"platlib", in ``sysconfig`` terms) are still found relative to -``sys.prefix``. +And Now For Something Completely Different +========================================== -(Also, ``sys.base_exec_prefix`` is added, and handled similarly with -regard to ``sys.exec_prefix``.) +In all seriousness, there are important reasons why there won't be an +official Python 2.8 release, and why you should plan to migrate +instead to Python 3. -Thus, a Python virtual environment in its simplest form would consist -of nothing more than a copy or symlink of the Python binary -accompanied by a ``pyvenv.cfg`` file and a site-packages directory. +Python is (as of this writing) more than 20 years old, and Guido and the +community have learned a lot in those intervening years. Guido's +original concept for Python 3 was to make changes to the language +primarily to remove the warts that had grown in the preceding +versions. Python 3 was not to be a complete redesign, but instead an +evolution of the language, and while maintaining full backward +compatibility with Python 2 was explicitly off-the-table, neither were +gratuitous changes in syntax or semantics acceptable. In most cases, +Python 2 code can be translated fairly easily to Python 3, sometimes +entirely mechanically by such tools as `2to3`_ (there's also a non-trivial +subset of the language that will run without modification on both 2.7 and +3.x). +Because maintaining multiple versions of Python is a significant drag +on the resources of the Python developers, and because the +improvements to the language and libraries embodied in Python 3 are so +important, it was decided to end the Python 2 lineage with Python +2.7. Thus, all new development occurs in the Python 3 line of +development, and there will never be an official Python 2.8 release. +Python 2.7 will however be maintained for longer than the usual period +of time. -Isolation from system site-packages ------------------------------------ +Here are some highlights of the significant improvements in Python 3. +You can read in more detail on the differences_ between Python 2 and +Python 3. There are also many good guides on porting_ from Python 2 +to Python 3. -By default, a virtual environment is entirely isolated from the -system-level site-packages directories. -If the ``pyvenv.cfg`` file also contains a key -``include-system-site-packages`` with a value of ``true`` (not case -sensitive), the ``site`` module will also add the system site -directories to ``sys.path`` after the virtual environment site -directories. Thus system-installed packages will still be importable, -but a package of the same name installed in the virtual environment -will take precedence. +Strings and bytes +----------------- -:pep:`370` user-level site-packages are considered part of the system -site-packages for venv purposes: they are not available from an -isolated venv, but are available from an -``include-system-site-packages = true`` venv. +Python 2's basic original strings are called 8-bit strings, and +they play a dual role in Python 2 as both ASCII text and as byte +sequences. While Python 2 also has a unicode string type, the +fundamental ambiguity of the core string type, coupled with Python 2's +default behavior of supporting automatic coercion from 8-bit strings +to unicode objects when the two are combined, often leads to +``UnicodeError``\ s. Python 3's standard string type is Unicode based, and +Python 3 adds a dedicated bytes type, but critically, no automatic coercion +between bytes and unicode strings is provided. The closest the language gets +to implicit coercion are a few text-based APIs that assume a default +encoding (usually UTF-8) if no encoding is explicitly stated. Thus, the core +interpreter, its I/O libraries, module names, etc. are clear in their +distinction between unicode strings and bytes. Python 3's unicode +support even extends to the filesystem, so that non-ASCII file names are +natively supported. +This string/bytes clarity is often a source of difficulty in +transitioning existing code to Python 3, because many third party +libraries and applications are themselves ambiguous in this +distinction. Once migrated though, most ``UnicodeError``\ s can be +eliminated. -Creating virtual environments ------------------------------ -This PEP also proposes adding a new ``venv`` module to the standard -library which implements the creation of virtual environments. This -module can be executed using the ``-m`` flag:: +Numbers +------- - python3 -m venv /path/to/new/virtual/environment +Python 2 has two basic integer types, a native machine-sized ``int`` +type, and an arbitrary length ``long`` type. These have been merged in +Python 3 into a single ``int`` type analogous to Python 2's ``long`` +type. -A ``pyvenv`` installed script is also provided to make this more -convenient:: +In addition, integer division now produces floating point numbers for +non-integer results. - pyvenv /path/to/new/virtual/environment -Running this command creates the target directory (creating any parent -directories that don't exist already) and places a ``pyvenv.cfg`` file -in it with a ``home`` key pointing to the Python installation the -command was run from. It also creates a ``bin/`` (or ``Scripts`` on -Windows) subdirectory containing a copy (or symlink) of the -``python3`` executable, and the ``pysetup3`` script from the -``packaging`` standard library module (to facilitate easy installation -of packages from PyPI into the new virtualenv). And it creates an -(initially empty) ``lib/pythonX.Y/site-packages`` (or -``Lib\site-packages`` on Windows) subdirectory. +Classes +------- -If the target directory already exists an error will be raised, unless -the ``--clear`` option was provided, in which case the target -directory will be deleted and virtual environment creation will -proceed as usual. +Python 2 has two core class hierarchies, often called *classic +classes* and *new-style classes*. The latter allow for such things as +inheriting from the builtin basic types, support descriptor based tools +like the ``property`` builtin and provide a generally more sane and coherent +system for dealing with multiple inheritance. Python 3 provided the +opportunity to completely drop support for classic classes, so all classes +in Python 3 automatically use the new-style semantics (although that's a +misnomer now). There is no need to explicitly inherit from ``object`` or set +the default metatype to enable them (in fact, setting a default metatype at +the module level is no longer supported - the default metatype is always +``object``). -The created ``pyvenv.cfg`` file also includes the -``include-system-site-packages`` key, set to ``true`` if ``venv`` is -run with the ``--system-site-packages`` option, ``false`` by default. +The mechanism for explicitly specifying a metaclass has also changed to use +a ``metaclass`` keyword argument in the class header line rather than a +``__metaclass__`` magic attribute in the class body. -Multiple paths can be given to ``pyvenv``, in which case an identical -virtualenv will be created, according to the given options, at each -provided path. -The ``venv`` module also provides "shell activation scripts" for POSIX -and Windows systems which simply add the virtual environment's ``bin`` -(or ``Scripts``) directory to the front of the user's shell PATH. -This is not strictly necessary for use of a virtual environment (as an -explicit path to the venv's python binary or scripts can just as well -be used), but it is convenient. +Multiple spellings +------------------ -The ``venv`` module also adds a ``pysetup3`` script into each venv. -In order to allow ``pysetup`` and other Python package managers to -install packages into the virtual environment the same way they would -install into a normal Python installation, and avoid special-casing -virtual environments in ``sysconfig`` beyond using ``sys.site_prefix`` -in place of ``sys.prefix``, the internal virtual environment layout -mimics the layout of the Python installation itself on each platform. -So a typical virtual environment layout on a POSIX system would be:: +There are many cases in Python 2 where multiple spellings of some +constructs exist, such as ``repr()`` and *backticks*, or the two +inequality operators ``!=`` and ``<>``. In all cases, Python 3 has chosen +exactly one spelling and removed the other (e.g. ``repr()`` and ``!=`` +were kept). - pyvenv.cfg - bin/python3 - bin/python - bin/pysetup3 - include/ - lib/python3.3/site-packages/ -While on a Windows system:: +Imports +------- - pyvenv.cfg - Scripts/python.exe - Scripts/python3.dll - Scripts/pysetup3.exe - Scripts/pysetup3-script.py - ... other DLLs and pyds... - Include/ - Lib/site-packages/ +In Python 3, implicit relative imports within packages are no longer +available - only absolute imports and explicit relative imports are +supported. In addition, star imports (e.g. ``from x import *``) are only +permitted in module level code. -Third-party packages installed into the virtual environment will have -their Python modules placed in the ``site-packages`` directory, and -their executables placed in ``bin/`` or ``Scripts\``. +Also, some areas of the standard library have been reorganized to make +the naming scheme more intuitive. Some rarely used builtins have been +relocated to standard library modules. -.. note:: - On a normal Windows system-level installation, the Python binary - itself wouldn't go inside the "Scripts/" subdirectory, as it does - in the default venv layout. This is useful in a virtual - environment so that a user only has to add a single directory to - their shell PATH in order to effectively "activate" the virtual - environment. +Iterators and views +------------------- -.. note:: - - On Windows, it is necessary to also copy or symlink DLLs and pyd - files from compiled stdlib modules into the env, because if the - venv is created from a non-system-wide Python installation, - Windows won't be able to find the Python installation's copies of - those files when Python is run from the venv. - - -Copies versus symlinks ----------------------- - -The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs on Windows). Some -users prefer a copied binary (for greater isolation from system -changes) and some prefer a symlinked one (so that e.g. security -updates automatically propagate to virtual environments). - -There are some cross-platform difficulties with symlinks: - -* Not all Windows versions support symlinks, and even on those that - do, creating them often requires administrator privileges. - -* On OSX framework builds of Python, sys.executable is just a stub - that executes the real Python binary. Symlinking this stub does not - work with the implementation in this PEP; it must be copied. - (Fortunately the stub is also small, so copying it is not an issue). - -Because of these issues, this PEP proposes to copy the Python binary -by default, to maintain cross-platform consistency in the default -behavior. - -The ``pyvenv`` script accepts a ``--symlink`` option. If this option -is provided, the script will attempt to symlink instead of copy. If a -symlink fails (e.g. because they are not supported by the platform, or -additional privileges are needed), the script will warn the user and -fall back to a copy. - -On OSX framework builds, where a symlink of the executable would -succeed but create a non-functional virtual environment, the script -will fail with an error message that symlinking is not supported on -OSX framework builds. - - -API ---- - -The high-level method described above makes use of a simple API which -provides mechanisms for third-party virtual environment creators to -customize environment creation according to their needs. - -The ``venv`` module contains an ``EnvBuilder`` class which accepts the -following keyword arguments on instantiation: - -* ``system_site_packages`` - A Boolean value indicating that the - system Python site-packages should be available to the environment. - Defaults to ``False``. - -* ``clear`` - A Boolean value which, if true, will delete any existing - target directory instead of raising an exception. Defaults to - ``False``. - -* ``symlinks`` - A Boolean value indicating whether to attempt to - symlink the Python binary (and any necessary DLLs or other binaries, - e.g. ``pythonw.exe``), rather than copying. Defaults to ``False``. - -The instantiated env-builder has a ``create`` method, which takes as -required argument the path (absolute or relative to the current -directory) of the target directory which is to contain the virtual -environment. The ``create`` method either creates the environment in -the specified directory, or raises an appropriate exception. - -The ``venv`` module also provides a module-level ``create`` function -as a convenience:: - - def create(env_dir, - system_site_packages=False, clear=False, use_symlinks=False): - builder = EnvBuilder( - system_site_packages=system_site_packages, - clear=clear, - use_symlinks=use_symlinks) - builder.create(env_dir) - -Creators of third-party virtual environment tools are free to use the -provided ``EnvBuilder`` class as a base class. - -The ``create`` method of the ``EnvBuilder`` class illustrates the -hooks available for customization:: - - def create(self, env_dir): - """ - Create a virtualized Python environment in a directory. - - :param env_dir: The target directory to create an environment in. - - """ - env_dir = os.path.abspath(env_dir) - context = self.create_directories(env_dir) - self.create_configuration(context) - self.setup_python(context) - self.post_setup(context) - -Each of the methods ``create_directories``, ``create_configuration``, -``setup_python``, and ``post_setup`` can be overridden. The functions -of these methods are: - -* ``create_directories`` - creates the environment directory and all - necessary directories, and returns a context object. This is just a - holder for attributes (such as paths), for use by the other methods. - -* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration - file in the environment. - -* ``setup_python`` - creates a copy of the Python executable (and, - under Windows, DLLs) in the environment. - -* ``post_setup`` - A (no-op by default) hook method which can be - overridden in third party subclasses to pre-install packages or - install scripts in the virtual environment. - -In addition, ``EnvBuilder`` provides a utility method that can be -called from ``post_setup`` in subclasses to assist in installing -custom scripts into the virtual environment. The method -``install_scripts`` accepts as arguments the ``context`` object (see -above) and a path to a directory. The directory should contain -subdirectories "common", "posix", "nt", each containing scripts -destined for the bin directory in the environment. The contents of -"common" and the directory corresponding to ``os.name`` are copied -after doing some text replacement of placeholders: - -* ``__VENV_DIR__`` is replaced with absolute path of the environment - directory. - -* ``__VENV_NAME__`` is replaced with the environment name (final path - segment of environment directory). - -* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory - (either ``bin`` or ``Scripts``). - -* ``__VENV_PYTHON__`` is replaced with the absolute path of the - environment's executable. - -The ``DistributeEnvBuilder`` subclass in the reference implementation -illustrates how the customization hook can be used in practice to -pre-install Distribute into the virtual environment. It's not -envisaged that ``DistributeEnvBuilder`` will be actually added to -Python core, but it makes the reference implementation more -immediately useful for testing and exploratory purposes. - - -Backwards Compatibility -======================= - -Splitting the meanings of ``sys.prefix`` ----------------------------------------- - -Any virtual environment tool along these lines (which attempts to -isolate site-packages, while still making use of the base Python's -standard library with no need for it to be symlinked into the virtual -environment) is proposing a split between two different meanings -(among others) that are currently both wrapped up in ``sys.prefix``: -the answers to the questions "Where is the standard library?" and -"Where is the site-packages location where third-party modules should -be installed?" - -This split could be handled by introducing a new ``sys`` attribute for -either the former prefix or the latter prefix. Either option -potentially introduces some backwards-incompatibility with software -written to assume the other meaning for ``sys.prefix``. (Such -software should preferably be using the APIs in the ``site`` and -``sysconfig`` modules to answer these questions rather than using -``sys.prefix`` directly, in which case there is no -backwards-compatibility issue, but in practice ``sys.prefix`` is -sometimes used.) - -The `documentation`__ for ``sys.prefix`` describes it as "A string -giving the site-specific directory prefix where the platform -independent Python files are installed," and specifically mentions the -standard library and header files as found under ``sys.prefix``. It -does not mention ``site-packages``. - -__ http://docs.python.org/dev/library/sys.html#sys.prefix - -Maintaining this documented definition would mean leaving -``sys.prefix`` pointing to the base system installation (which is -where the standard library and header files are found), and -introducing a new value in ``sys`` (something like -``sys.site_prefix``) to point to the prefix for ``site-packages``. -This would maintain the documented semantics of ``sys.prefix``, but -risk breaking isolation if third-party code uses ``sys.prefix`` rather -than ``sys.site_prefix`` or the appropriate ``site`` API to find -site-packages directories. - -The most notable case is probably `setuptools`_ and its fork -`distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but -do use ``sys.prefix`` directly to build up a list of site directories -for pre-flight checking where ``pth`` files can usefully be placed. - -Otherwise, a `Google Code Search`_ turns up what appears to be a -roughly even mix of usage between packages using ``sys.prefix`` to -build up a site-packages path and packages using it to e.g. eliminate -the standard-library from code-execution tracing. - -Although it requires modifying the documented definition of -``sys.prefix``, this PEP prefers to have ``sys.prefix`` point to the -virtual environment (where ``site-packages`` is found), and introduce -``sys.base_prefix`` to point to the standard library and Python header -files. Rationale for this choice: - -* It is preferable to err on the side of greater isolation of the - virtual environment. - -* Virtualenv already modifies ``sys.prefix`` to point at the virtual - environment, and in practice this has not been a problem. - -* No modification is required to setuptools/distribute. - -.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools -.. _distribute: http://packages.python.org/distribute/ -.. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs - - -Open Questions -============== - -What about include files? -------------------------- - -For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in -``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default -in ``$VE/include/pythonX.Y``. With virtualenv, everything works -because the PythonX.Y include is symlinked, so everything that's -needed is in ``$VE/include``. At the moment the reference -implementation doesn't do anything with include files, besides -creating the include directory; this might need to change, to -copy/symlink ``$VE/include/pythonX.Y``. - -As in Python there's no abstraction for a site-specific include -directory, other than for platform-specific stuff, then the user -expectation would seem to be that all include files anyone could ever -want should be found in one of just two locations, with sysconfig -labels "include" & "platinclude". - -There's another issue: what if includes are Python-version-specific? -For example, SIP installs by default into ``$VE/include/pythonX.Y`` -rather than ``$VE/include``, presumably because there's -version-specific stuff in there - but even if that's not the case with -SIP, it could be the case with some other package. And the problem -that gives is that you can't just symlink the ``include/pythonX.Y`` -directory, but actually have to provide a writable directory and -symlink/copy the contents from the system ``include/pythonX.Y``. Of -course this is not hard to do, but it does seem inelegant. OTOH it's -really because there's no supporting concept in ``Python/sysconfig``. - - -Testability and Source Build Issues ------------------------------------ - -Currently in the reference implementation, virtual environments must -be created with an installed Python, rather than a source build, as -the base installation. In order to be able to fully test the ``venv`` -module in the Python regression test suite, some anomalies in how -sysconfig data is configured in source builds will need to be removed. -For example, ``sysconfig.get_paths()`` in a source build gives -(partial output):: - - { - 'include': '/home/vinay/tools/pythonv/Include', - 'libdir': '/usr/lib ; or /usr/lib64 on a multilib system', - 'platinclude': '/home/vinay/tools/pythonv', - 'platlib': '/usr/local/lib/python3.3/site-packages', - 'platstdlib': '/usr/local/lib/python3.3', - 'purelib': '/usr/local/lib/python3.3/site-packages', - 'stdlib': '/usr/local/lib/python3.3' - } - - -Need for ``install_name_tool`` on OSX? --------------------------------------- - -`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode -developer tools, to modify the copied executable on OSX. We need -input from OSX developers on whether this is actually necessary in -this PEP's implementation of virtual environments, and if so, if there -is an alternative to ``install_name_tool`` that would allow ``venv`` -to not require that Xcode is installed. - -.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 - - -Provide a mode that is isolated only from user site packages? -------------------------------------------------------------- - -Is there sufficient rationale for providing a mode that isolates the -venv from :pep:`370` user site packages, but not from the system-level -site-packages? - - -Other Python implementations? ------------------------------ - -We should get feedback from Jython, IronPython, and PyPy about whether -there's anything in this PEP that they foresee as a difficulty for -their implementation. - - -Reference Implementation -======================== - -The in-progress reference implementation is found in `a clone of the -CPython Mercurial repository`_. To test it, build and install it (the -virtual environment tool currently does not run from a source tree). -From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` -to create a virtual environment. - -The reference implementation (like this PEP!) is a work in progress. - -.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv +Many APIs, which in Python 2 returned concrete lists, in Python 3 now +return iterators or lightweight *views*. Copyright @@ -539,13 +178,17 @@ This document has been placed in the public domain. +.. _`2to3`: http://docs.python.org/library/2to3.html +.. _differences: http://docs.python.org/release/3.0.1/whatsnew/3.0.html +.. _porting: http://python3porting.com/ + + .. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: - + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0404.txt b/pep-0405.txt copy from pep-0404.txt copy to pep-0405.txt --- a/pep-0404.txt +++ b/pep-0405.txt @@ -1,4 +1,4 @@ -PEP: 404 +PEP: 405 Title: Python Virtual Environments Version: $Revision$ Last-Modified: $Date$ diff --git a/pep-0406.txt b/pep-0406.txt new file mode 100644 --- /dev/null +++ b/pep-0406.txt @@ -0,0 +1,274 @@ +PEP: 406 +Title: Improved Encapsulation of Import State +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan , Greg Slodkowicz +Status: Deferred +Type: Standards Track +Content-Type: text/x-rst +Created: 4-Jul-2011 +Python-Version: 3.4 +Post-History: 31-Jul-2011, 13-Nov-2011, 4-Dec-2011 + +Abstract +======== + +This PEP proposes the introduction of a new 'ImportEngine' class as part of +``importlib`` which would encapsulate all state related to importing modules +into a single object. Creating new instances of this object would then provide +an alternative to completely replacing the built-in implementation of the +import statement, by overriding the ``__import__()`` function. To work with +the builtin import functionality and importing via import engine objects, +this PEP proposes a context management based approach to temporarily replacing +the global import state. + +The PEP also proposes inclusion of a ``GlobalImportEngine`` subclass and a +globally accessible instance of that class, which "writes through" to the +process global state. This provides a backwards compatible bridge between the +proposed encapsulated API and the legacy process global state, and allows +straightforward support for related state updates (e.g. selectively +invalidating path cache entries when ``sys.path`` is modified). + + +PEP Deferral +============ + +The import system is already seeing substantial changes in Python 3.3, to +natively handle packages split across multiple directories (PEP 382) and +(potentially) to make the import semantics in the main module better match +those in other modules (PEP 395). + +Accordingly, the proposal in this PEP will not be seriously considered until +Python 3.4 at the earliest. + + +Rationale +========= + +Currently, most state related to the import system is stored as module level +attributes in the ``sys`` module. The one exception is the import lock, which +is not accessible directly, but only via the related functions in the ``imp`` +module. The current process global import state comprises: + +* sys.modules +* sys.path +* sys.path_hooks +* sys.meta_path +* sys.path_importer_cache +* the import lock (imp.lock_held()/acquire_lock()/release_lock()) + +Isolating this state would allow multiple import states to be +conveniently stored within a process. Placing the import functionality +in a self-contained object would also allow subclassing to add additional +features (e.g. module import notifications or fine-grained control +over which modules can be imported). The engine would also be +subclassed to make it possible to use the import engine API to +interact with the existing process-global state. + +The namespace PEPs (especially PEP 402) raise a potential need for +*additional* process global state, in order to correctly update package paths +as ``sys.path`` is modified. + +Finally, providing a coherent object for all this state makes it feasible to +also provide context management features that allow the import state to be +temporarily substituted. + + +Proposal +======== + +We propose introducing an ImportEngine class to encapsulate import +functionality. This includes an ``__import__()`` method which can +be used as an alternative to the built-in ``__import__()`` when +desired and also an ``import_module()`` method, equivalent to +``importlib.import_module()`` [3]_. + +Since there are global import state invariants that are assumed and should be +maintained, we introduce a ``GlobalImportState`` class with an interface +identical to ``ImportEngine`` but directly accessing the current global import +state. This can be easily implemented using class properties. + + +Specification +============= + +ImportEngine API +~~~~~~~~~~~~~~~~ + +The proposed extension consists of the following objects: + +``importlib.engine.ImportEngine`` + + ``from_engine(self, other)`` + + Create a new import object from another ImportEngine instance. The + new object is initialised with a copy of the state in ``other``. When + called on ``importlib engine.sysengine``, ``from_engine()`` can be + used to create an ``ImportEngine`` object with a **copy** of the + global import state. + + ``__import__(self, name, globals={}, locals={}, fromlist=[], level=0)`` + + Reimplementation of the builtin ``__import__()`` function. The + import of a module will proceed using the state stored in the + ImportEngine instance rather than the global import state. For full + documentation of ``__import__`` funtionality, see [2]_ . + ``__import__()`` from ``ImportEngine`` and its subclasses can be used + to customise the behaviour of the ``import`` statement by replacing + ``__builtin__.__import__`` with ``ImportEngine().__import__``. + + ``import_module(name, package=None)`` + + A reimplementation of ``importlib.import_module()`` which uses the + import state stored in the ImportEngine instance. See [3]_ for a full + reference. + + ``modules, path, path_hooks, meta_path, path_importer_cache`` + + Instance-specific versions of their process global ``sys`` equivalents + + +``importlib.engine.GlobalImportEngine(ImportEngine)`` + + Convenience class to provide engine-like access to the global state. + Provides ``__import__()``, ``import_module()`` and ``from_engine()`` + methods like ``ImportEngine`` but writes through to the global state + in ``sys``. + +To support various namespace package mechanisms, when ``sys.path`` is altered, +tools like ``pkgutil.extend_path`` should be used to also modify other parts +of the import state (in this case, package ``__path__`` attributes). The path +importer cache should also be invalidated when a variety of changes are made. + +The ``ImportEngine`` API will provide convenience methods that automatically +make related import state updates as part of a single operation. + + +Global variables +~~~~~~~~~~~~~~~~ + +``importlib.engine.sysengine`` + + A precreated instance of ``GlobalImportEngine``. Intended for use by + importers and loaders that have been updated to accept optional ``engine`` + parameters and with ``ImportEngine.from_engine(sysengine)`` to start with + a copy of the process global import state. + + +No changes to finder/loader interfaces +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Rather than attempting to update the PEP 302 APIs to accept additional state, +this PEP proposes that ``ImportEngine`` support the content management +protocol (similar to the context substitution mechanisms in the ``decimal`` +module). + +The context management mechanism for ``ImportEngine`` would: + +* On entry: + * Acquire the import lock + * Substitute the global import state with the import engine's own state +* On exit: + * Restore the previous global import state + * Release the import lock + +The precise API for this is TBD (but will probably use a distinct context +management object, along the lines of that created by +``decimal.localcontext``). + + +Open Issues +=========== + + +API design for falling back to global import state +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The current proposal relies on the ``from_engine()`` API to fall back to the +global import state. It may be desirable to offer a variant that instead falls +back to the global import state dynamically. + +However, one big advantage of starting with an "as isolated as possible" +design is that it becomes possible to experiment with subclasses that blur +the boundaries between the engine instance state and the process global state +in various ways. + + +Builtin and extension modules must be process global +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Due to platform limitations, only one copy of each builtin and extension +module can readily exist in each process. Accordingly, it is impossible for +each ``ImportEngine`` instance to load such modules independently. + +The simplest solution is for ``ImportEngine`` to refuse to load such modules, +raising ``ImportError``. ``GlobalImportEngine`` would be able to load them +normally. + +``ImportEngine`` will still return such modules from a prepopulated module +cache - it's only loading them directly which causes problems. + + +Scope of substitution +~~~~~~~~~~~~~~~~~~~~~ + +Related to the previous open issue is the question of what state to substitute +when using the context management API. It is currently the case that replacing +``sys.modules`` can be unreliable due to cached references and there's the +underlying fact that having independent copies of some modules is simply +impossible due to platform limitations. + +As part of this PEP, it will be necessary to document explicitly: + +* Which parts of the global import state can be substituted (and declare code + which caches references to that state without dealing with the substitution + case buggy) +* Which parts must be modified in-place (and hence are not substituted by the + ``ImportEngine`` context management API, or otherwise scoped to + ``ImportEngine`` instances) + + +Reference Implementation +======================== + +A reference implementation [4]_ for an earlier draft of this PEP, based on +Brett Cannon's importlib has been developed by Greg Slodkowicz as part of the +2011 Google Summer of Code. Note that the current implementation avoids +modifying existing code, and hence duplicates a lot of things unnecessarily. +An actual implementation would just modify any such affected code in place. + +That earlier draft of the PEP proposed change the PEP 302 APIs to support passing +in an optional engine instance. This had the (serious) downside of not correctly +affecting further imports from the imported module, hence the change to the +context management based proposal for substituting the global state. + + +References +========== + +.. [1] PEP 302, New Import Hooks, J van Rossum, Moore + (http://www.python.org/dev/peps/pep-0302) + +.. [2] __import__() builtin function, The Python Standard Library documentation + (http://docs.python.org/library/functions.html#__import__) + +.. [3] Importlib documentation, Cannon + (http://docs.python.org/dev/library/importlib) + +.. [4] Reference implentation + (https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py) + + +Copyright +========= + +This document has been placed in the public domain. + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0407.txt b/pep-0407.txt new file mode 100644 --- /dev/null +++ b/pep-0407.txt @@ -0,0 +1,179 @@ +PEP: 407 +Title: New release cycle and introducing long-term support versions +Version: $Revision$ +Last-Modified: $Date$ +Author: Antoine Pitrou , + Georg Brandl , + Barry Warsaw +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 2012-01-12 +Post-History: http://mail.python.org/pipermail/python-dev/2012-January/115838.html +Resolution: TBD + + +Abstract +======== + +Finding a release cycle for an open-source project is a delicate +exercise in managing mutually contradicting constraints: developer +manpower, availability of release management volunteers, ease of +maintenance for users and third-party packagers, quick availability of +new features (and behavioural changes), availability of bug fixes +without pulling in new features or behavioural changes. + +The current release cycle errs on the conservative side. It is +adequate for people who value stability over reactivity. This PEP is +an attempt to keep the stability that has become a Python trademark, +while offering a more fluid release of features, by introducing the +notion of long-term support versions. + + +Scope +===== + +This PEP doesn't try to change the maintenance period or release +scheme for the 2.7 branch. Only 3.x versions are considered. + + +Proposal +======== + +Under the proposed scheme, there would be two kinds of feature +versions (sometimes dubbed "minor versions", for example 3.2 or 3.3): +normal feature versions and long-term support (LTS) versions. + +Normal feature versions would get either zero or at most one bugfix +release; the latter only if needed to fix critical issues. Security +fix handling for these branches needs to be decided. + +LTS versions would get regular bugfix releases until the next LTS +version is out. They then would go into security fixes mode, up to a +termination date at the release manager's discretion. + +Periodicity +----------- + +A new feature version would be released every X months. We +tentatively propose X = 6 months. + +LTS versions would be one out of N feature versions. We tentatively +propose N = 4. + +With these figures, a new LTS version would be out every 24 months, +and remain supported until the next LTS version 24 months later. This +is mildly similar to today's 18 months bugfix cycle for every feature +version. + +Pre-release versions +-------------------- + +More frequent feature releases imply a smaller number of disruptive +changes per release. Therefore, the number of pre-release builds +(alphas and betas) can be brought down considerably. Two alpha builds +and a single beta build would probably be enough in the regular case. +The number of release candidates depends, as usual, on the number of +last-minute fixes before final release. + + +Effects +======= + +Effect on development cycle +--------------------------- + +More feature releases might mean more stress on the development and +release management teams. This is quantitatively alleviated by the +smaller number of pre-release versions; and qualitatively by the +lesser amount of disruptive changes (meaning less potential for +breakage). The shorter feature freeze period (after the first beta +build until the final release) is easier to accept. The rush for +adding features just before feature freeze should also be much +smaller. + +Effect on bugfix cycle +---------------------- + +The effect on fixing bugs should be minimal with the proposed figures. +The same number of branches would be simultaneously open for bugfix +maintenance (two until 2.x is terminated, then one). + +Effect on workflow +------------------ + +The workflow for new features would be the same: developers would only +commit them on the ``default`` branch. + +The workflow for bug fixes would be slightly updated: developers would +commit bug fixes to the current LTS branch (for example ``3.3``) and +then merge them into ``default``. + +If some critical fixes are needed to a non-LTS version, they can be +grafted from the current LTS branch to the non-LTS branch, just like +fixes are ported from 3.x to 2.7 today. + +Effect on the community +----------------------- + +People who value stability can just synchronize on the LTS releases +which, with the proposed figures, would give a similar support cycle +(both in duration and in stability). + +People who value reactivity and access to new features (without taking +the risk to install alpha versions or Mercurial snapshots) would get +much more value from the new release cycle than currently. + +People who want to contribute new features or improvements would be +more motivated to do so, knowing that their contributions will be more +quickly available to normal users. Also, a smaller feature freeze +period makes it less cumbersome to interact with contributors of +features. + + +Discussion +========== + +These are open issues that should be worked out during discussion: + +* Decide on X (months between feature releases) and N (feature releases + per LTS release) as defined above. + +* For given values of X and N, is the no-bugfix-releases policy for + non-LTS versions feasible? + +* What is the policy for security fixes? + +* Restrict new syntax and similar changes (i.e. everything that was + prohibited by PEP 3003) to LTS versions? + +* What is the effect on packagers such as Linux distributions? + +* How will release version numbers or other identifying and marketing + material make it clear to users which versions are normal feature + releases and which are LTS releases? How do we manage user + expectations? + +* Does the faster release cycle mean we could some day reach 3.10 and + above? Some people expressed a tacit expectation that version numbers + always fit in one decimal digit. + +A community poll or survey to collect opinions from the greater Python +community would be valuable before making a final decision. + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0408.txt b/pep-0408.txt new file mode 100644 --- /dev/null +++ b/pep-0408.txt @@ -0,0 +1,317 @@ +PEP: 408 +Title: Standard library __preview__ package +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan , + Eli Bendersky +Status: Rejected +Type: Standards Track +Content-Type: text/x-rst +Created: 2012-01-07 +Python-Version: 3.3 +Post-History: 2012-01-27 +Resolution: http://mail.python.org/pipermail/python-dev/2012-January/115962.html + + +Abstract +======== + +The process of including a new module into the Python standard library is +hindered by the API lock-in and promise of backward compatibility implied by +a module being formally part of Python. This PEP proposes a transitional +state for modules - inclusion in a special ``__preview__`` package for the +duration of a minor release (roughly 18 months) prior to full acceptance into +the standard library. On one hand, this state provides the module with the +benefits of being formally part of the Python distribution. On the other hand, +the core development team explicitly states that no promises are made with +regards to the module's eventual full inclusion into the standard library, +or to the stability of its API, which may change for the next release. + + +PEP Rejection +============= + +Based on his experience with a similar "labs" namespace in Google App Engine, +Guido has rejected this PEP [3] in favour of the simpler alternative of +explicitly marking provisional modules as such in their documentation. + +If a module is otherwise considered suitable for standard library inclusion, +but some concerns remain regarding maintainability or certain API details, +then the module can be accepted on a provisional basis. While it is considered +an unlikely outcome, such modules *may* be removed from the standard library +without a deprecation period if the lingering concerns prove well-founded. + +As part of the same announcement, Guido explicitly accepted Matthew +Barnett's 'regex' module [4] as a provisional addition to the standard +library for Python 3.3 (using the 'regex' name, rather than as a drop-in +replacement for the existing 're' module). + + +Proposal - the __preview__ package +================================== + +Whenever the Python core development team decides that a new module should be +included into the standard library, but isn't entirely sure about whether the +module's API is optimal, the module can be placed in a special package named +``__preview__`` for a single minor release. + +In the next minor release, the module may either be "graduated" into the +standard library (and occupy its natural place within its namespace, leaving the +``__preview__`` package), or be rejected and removed entirely from the Python +source tree. If the module ends up graduating into the standard library after +spending a minor release in ``__preview__``, its API may be changed according +to accumulated feedback. The core development team explicitly makes no +guarantees about API stability and backward compatibility of modules in +``__preview__``. + +Entry into the ``__preview__`` package marks the start of a transition of the +module into the standard library. It means that the core development team +assumes responsibility of the module, similarly to any other module in the +standard library. + + +Which modules should go through ``__preview__`` +----------------------------------------------- + +We expect most modules proposed for addition into the Python standard library +to go through a minor release in ``__preview__``. There may, however, be some +exceptions, such as modules that use a pre-defined API (for example ``lzma``, +which generally follows the API of the existing ``bz2`` module), or modules +with an API that has wide acceptance in the Python development community. + +In any case, modules that are proposed to be added to the standard library, +whether via ``__preview__`` or directly, must fulfill the acceptance conditions +set by PEP 2. + +It is important to stress that the aim of of this proposal is not to make the +process of adding new modules to the standard library more difficult. On the +contrary, it tries to provide a means to add *more* useful libraries. Modules +which are obvious candidates for entry can be added as before. Modules which +due to uncertainties about the API could be stalled for a long time now have +a means to still be distributed with Python, via an incubation period in the +``__preview__`` package. + + +Criteria for "graduation" +------------------------- + +In principle, most modules in the ``__preview__`` package should eventually +graduate to the stable standard library. Some reasons for not graduating are: + +* The module may prove to be unstable or fragile, without sufficient developer + support to maintain it. +* A much better alternative module may be found during the preview release + +Essentially, the decision will be made by the core developers on a per-case +basis. The point to emphasize here is that a module's appearance in the +``__preview__`` package in some release does not guarantee it will continue +being part of Python in the next release. + + +Example +------- + +Suppose the ``example`` module is a candidate for inclusion in the standard +library, but some Python developers aren't convinced that it presents the best +API for the problem it intends to solve. The module can then be added to the +``__preview__`` package in release ``3.X``, importable via:: + + from __preview__ import example + +Assuming the module is then promoted to the the standard library proper in +release ``3.X+1``, it will be moved to a permanent location in the library:: + + import example + +And importing it from ``__preview__`` will no longer work. + + +Rationale +========= + +Benefits for the core development team +-------------------------------------- + +Currently, the core developers are really reluctant to add new interfaces to +the standard library. This is because as soon as they're published in a +release, API design mistakes get locked in due to backward compatibility +concerns. + +By gating all major API additions through some kind of a preview mechanism +for a full release, we get one full release cycle of community feedback +before we lock in the APIs with our standard backward compatibility guarantee. + +We can also start integrating preview modules with the rest of the standard +library early, so long as we make it clear to packagers that the preview +modules should not be considered optional. The only difference between preview +APIs and the rest of the standard library is that preview APIs are explicitly +exempted from the usual backward compatibility guarantees. + +Essentially, the ``__preview__`` package is intended to lower the risk of +locking in minor API design mistakes for extended periods of time. Currently, +this concern can block new additions, even when the core development team +consensus is that a particular addition is a good idea in principle. + + +Benefits for end users +---------------------- + +For future end users, the broadest benefit lies in a better "out-of-the-box" +experience - rather than being told "oh, the standard library tools for task X +are horrible, download this 3rd party library instead", those superior tools +are more likely to be just be an import away. + +For environments where developers are required to conduct due diligence on +their upstream dependencies (severely harming the cost-effectiveness of, or +even ruling out entirely, much of the material on PyPI), the key benefit lies +in ensuring that anything in the ``__preview__`` package is clearly under +python-dev's aegis from at least the following perspectives: + +* Licensing: Redistributed by the PSF under a Contributor Licensing Agreement. +* Documentation: The documentation of the module is published and organized via + the standard Python documentation tools (i.e. ReST source, output generated + with Sphinx and published on http://docs.python.org). +* Testing: The module test suites are run on the python.org buildbot fleet + and results published via http://www.python.org/dev/buildbot. +* Issue management: Bugs and feature requests are handled on + http://bugs.python.org +* Source control: The master repository for the software is published + on http://hg.python.org. + + +Candidates for inclusion into __preview__ +========================================= + +For Python 3.3, there are a number of clear current candidates: + +* ``regex`` (http://pypi.python.org/pypi/regex) +* ``daemon`` (PEP 3143) +* ``ipaddr`` (PEP 3144) + +Other possible future use cases include: + +* Improved HTTP modules (e.g. ``requests``) +* HTML 5 parsing support (e.g. ``html5lib``) +* Improved URL/URI/IRI parsing +* A standard image API (PEP 368) +* Encapsulation of the import state (PEP 368) +* Standard event loop API (PEP 3153) +* A binary version of WSGI for Python 3 (e.g. PEP 444) +* Generic function support (e.g. ``simplegeneric``) + + +Relationship with PEP 407 +========================= + +PEP 407 proposes a change to the core Python release cycle to permit interim +releases every 6 months (perhaps limited to standard library updates). If +such a change to the release cycle is made, the following policy for the +``__preview__`` namespace is suggested: + +* For long term support releases, the ``__preview__`` namespace would always + be empty. +* New modules would be accepted into the ``__preview__`` namespace only in + interim releases that immediately follow a long term support release. +* All modules added will either be migrated to their final location in the + standard library or dropped entirely prior to the next long term support + release. + + +Rejected alternatives and variations +==================================== + + +Using ``__future__`` +-------------------- + +Python already has a "forward-looking" namespace in the form of the +``__future__`` module, so it's reasonable to ask why that can't be re-used for +this new purpose. + +There are two reasons why doing so not appropriate: + +1. The ``__future__`` module is actually linked to a separate compiler +directives feature that can actually change the way the Python interpreter +compiles a module. We don't want that for the preview package - we just want +an ordinary Python package. + +2. The ``__future__`` module comes with an express promise that names will be +maintained in perpetuity, long after the associated features have become the +compiler's default behaviour. Again, this is precisely the opposite of what is +intended for the preview package - it is almost certain that all names added to +the preview will be removed at some point, most likely due to their being moved +to a permanent home in the standard library, but also potentially due to their +being reverted to third party package status (if community feedback suggests the +proposed addition is irredeemably broken). + + +Versioning the package +---------------------- + +One proposed alternative [1]_ was to add explicit versioning to the +``__preview__`` package, i.e. ``__preview34__``. We think that it's better to +simply define that a module being in ``__preview__`` in Python 3.X will either +graduate to the normal standard library namespace in Python 3.X+1 or will +disappear from the Python source tree altogether. Versioning the ``_preview__`` +package complicates the process and does not align well with the main intent of +this proposal. + + +Using a package name without leading and trailing underscores +------------------------------------------------------------- + +It was proposed [1]_ to use a package name like ``preview`` or ``exp``, instead +of ``__preview__``. This was rejected in the discussion due to the special +meaning a "dunder" package name (that is, a name *with* leading and +trailing double-underscores) conveys in Python. Besides, a non-dunder name +would suggest normal standard library API stability guarantees, which is not +the intention of the ``__preview__`` package. + + +Preserving pickle compatibility +------------------------------- + +A pickled class instance based on a module in ``__preview__`` in release 3.X +won't be unpickle-able in release 3.X+1, where the module won't be in +``__preview__``. Special code may be added to make this work, but this goes +against the intent of this proposal, since it implies backward compatibility. +Therefore, this PEP does not propose to preserve pickle compatibility. + + +Credits +======= + +Dj Gilcrease initially proposed the idea of having a ``__preview__`` package +in Python [2]_. Although his original proposal uses the name +``__experimental__``, we feel that ``__preview__`` conveys the meaning of this +package in a better way. + + +References +========== + +.. [#] Discussed in this thread: + http://mail.python.org/pipermail/python-ideas/2012-January/013246.html + +.. [#] http://mail.python.org/pipermail/python-ideas/2011-August/011278.html + +.. [#] Guido's decision: + http://mail.python.org/pipermail/python-dev/2012-January/115962.html + +.. [#] Proposal for inclusion of regex: http://bugs.python.org/issue2636 + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0409.txt b/pep-0409.txt new file mode 100644 --- /dev/null +++ b/pep-0409.txt @@ -0,0 +1,201 @@ +PEP: 409 +Title: Suppressing exception context +Version: $Revision$ +Last-Modified: $Date$ +Author: Ethan Furman +Status: Accepted +Type: Standards Track +Content-Type: text/x-rst +Created: 26-Jan-2012 +Post-History: 30-Aug-2002, 01-Feb-2012, 03-Feb-2012 +Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116136.html + + +Abstract +======== + +One of the open issues from PEP 3134 is suppressing context: currently +there is no way to do it. This PEP proposes one. + + +Rationale +========= + +There are two basic ways to generate exceptions: + +1) Python does it (buggy code, missing resources, ending loops, etc.) + +2) manually (with a raise statement) + +When writing libraries, or even just custom classes, it can become +necessary to raise exceptions; moreover it can be useful, even +necessary, to change from one exception to another. To take an example +from my dbf module:: + + try: + value = int(value) + except Exception: + raise DbfError(...) + +Whatever the original exception was (``ValueError``, ``TypeError``, or +something else) is irrelevant. The exception from this point on is a +``DbfError``, and the original exception is of no value. However, if +this exception is printed, we would currently see both. + + +Alternatives +============ +Several possibilities have been put forth: + +* ``raise as NewException()`` + + Reuses the ``as`` keyword; can be confusing since we are not really + reraising the originating exception + +* ``raise NewException() from None`` + + Follows existing syntax of explicitly declaring the originating + exception + +* ``exc = NewException(); exc.__context__ = None; raise exc`` + + Very verbose way of the previous method + +* ``raise NewException.no_context(...)`` + + Make context suppression a class method. + +All of the above options will require changes to the core. + + +Proposal +======== + +I proprose going with the second option:: + + raise NewException from None + +It has the advantage of using the existing pattern of explicitly setting +the cause:: + + raise KeyError() from NameError() + +but because the cause is ``None`` the previous context is not displayed +by the default exception printing routines. + + +Implementation Discussion +========================= + +Currently, ``None`` is the default for both ``__context__`` and ``__cause__``. +In order to support ``raise ... from None`` (which would set ``__cause__`` to +``None``) we need a different default value for ``__cause__``. Several ideas +were put forth on how to implement this at the language level: + +* Overwrite the previous exception information (side-stepping the issue and + leaving ``__cause__`` at ``None``). + + Rejected as this can seriously hinder debugging due to + `poor error messages`_. + +* Use one of the boolean values in ``__cause__``: ``False`` would be the + default value, and would be replaced when ``from ...`` was used with the + explicity chained exception or ``None``. + + Rejected as this encourages the use of two different objects types for + ``__cause__`` with one of them (boolean) not allowed to have the full range + of possible values (``True`` would never be used). + +* Create a special exception class, ``__NoException__``. + + Rejected as possibly confusing, possibly being mistakenly raised by users, + and not being a truly unique value as ``None``, ``True``, and ``False`` are. + +* Use ``Ellipsis`` as the default value (the ``...`` singleton). + + Accepted. + + Ellipses are commonly used in English as place holders when words are + omitted. This works in our favor here as a signal that ``__cause__`` is + omitted, so look in ``__context__`` for more details. + + Ellipsis is not an exception, so cannot be raised. + + There is only one Ellipsis, so no unused values. + + Error information is not thrown away, so custom code can trace the entire + exception chain even if the default code does not. + + +Language Details +================ + +To support ``raise Exception from None``, ``__context__`` will stay as it is, +but ``__cause__`` will start out as ``Ellipsis`` and will change to ``None`` +when the ``raise Exception from None`` method is used. + +============================================ ================== ======================================= +form __context__ __cause__ +============================================ ================== ======================================= +raise ``None`` ``Ellipsis`` +reraise previous exception ``Ellipsis`` +reraise from ``None`` | ``ChainedException`` previous exception ``None`` | explicitly chained exception +============================================ ================== ======================================= + +The default exception printing routine will then: + +* If ``__cause__`` is ``Ellipsis`` the ``__context__`` (if any) will be + printed. + +* If ``__cause__`` is ``None`` the ``__context__`` will not be printed. + +* if ``__cause__`` is anything else, ``__cause__`` will be printed. + +In both of the latter cases the exception chain will stop being followed. + +Because the default value for ``__cause__`` is now ``Ellipsis`` and ``raise +Exception from Cause`` is simply syntactic sugar for:: + + _exc = NewException() + _exc.__cause__ = Cause() + raise _exc + +``Ellipsis``, as well as ``None``, is now allowed as a cause:: + + raise Exception from Ellipsis + + +Patches +======= + +There is a patch for CPython implementing this attached to `Issue 6210`_. + + +References +========== + +Discussion and refinements in this `thread on python-dev`_. + +.. _poor error messages: + http://bugs.python.org/msg152294 +.. _issue 6210: + http://bugs.python.org/issue6210 +.. _Thread on python-dev: + http://mail.python.org/pipermail/python-dev/2012-January/115838.html + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: + diff --git a/pep-0410.txt b/pep-0410.txt new file mode 100644 --- /dev/null +++ b/pep-0410.txt @@ -0,0 +1,547 @@ +PEP: 410 +Title: Use decimal.Decimal type for timestamps +Version: $Revision$ +Last-Modified: $Date$ +Author: Victor Stinner +Status: Rejected +Type: Standards Track +Content-Type: text/x-rst +Created: 01-February-2012 +Python-Version: 3.3 +Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116837.html + + +Rejection Notice +================ + +This PEP is rejected. +See http://mail.python.org/pipermail/python-dev/2012-February/116837.html. + + +Abstract +======== + +Decimal becomes the official type for high-resolution timestamps to make Python +support new functions using a nanosecond resolution without loss of precision. + + +Rationale +========= + +Python 2.3 introduced float timestamps to support sub-second resolutions. +os.stat() uses float timestamps by default since Python 2.5. Python 3.3 +introduced functions supporting nanosecond resolutions: + + * os module: futimens(), utimensat() + * time module: clock_gettime(), clock_getres(), monotonic(), wallclock() + +os.stat() reads nanosecond timestamps but returns timestamps as float. + +The Python float type uses binary64 format of the IEEE 754 standard. With a +resolution of one nanosecond (10\ :sup:`-9`), float timestamps lose precision +for values bigger than 2\ :sup:`24` seconds (194 days: 1970-07-14 for an Epoch +timestamp). + +Nanosecond resolution is required to set the exact modification time on +filesystems supporting nanosecond timestamps (e.g. ext4, btrfs, NTFS, ...). It +helps also to compare the modification time to check if a file is newer than +another file. Use cases: copy the modification time of a file using +shutil.copystat(), create a TAR archive with the tarfile module, manage a +mailbox with the mailbox module, etc. + +An arbitrary resolution is preferred over a fixed resolution (like nanosecond) +to not have to change the API when a better resolution is required. For +example, the NTP protocol uses fractions of 2\ :sup:`32` seconds +(approximatively 2.3 ? 10\ :sup:`-10` second), whereas the NTP protocol version +4 uses fractions of 2\ :sup:`64` seconds (5.4 ? 10\ :sup:`-20` second). + +.. note:: + With a resolution of 1 microsecond (10\ :sup:`-6`), float timestamps lose + precision for values bigger than 2\ :sup:`33` seconds (272 years: 2242-03-16 + for an Epoch timestamp). With a resolution of 100 nanoseconds + (10\ :sup:`-7`, resolution used on Windows), float timestamps lose precision + for values bigger than 2\ :sup:`29` seconds (17 years: 1987-01-05 for an + Epoch timestamp). + + +Specification +============= + +Add decimal.Decimal as a new type for timestamps. Decimal supports any +timestamp resolution, support arithmetic operations and is comparable. It is +possible to coerce a Decimal to float, even if the conversion may lose +precision. The clock resolution can also be stored in a Decimal object. + +Add an optional *timestamp* argument to: + + * os module: fstat(), fstatat(), lstat(), stat() (st_atime, + st_ctime and st_mtime fields of the stat structure), + sched_rr_get_interval(), times(), wait3() and wait4() + * resource module: ru_utime and ru_stime fields of getrusage() + * signal module: getitimer(), setitimer() + * time module: clock(), clock_gettime(), clock_getres(), + monotonic(), time() and wallclock() + +The *timestamp* argument value can be float or Decimal, float is still the +default for backward compatibility. The following functions support Decimal as +input: + + * datetime module: date.fromtimestamp(), datetime.fromtimestamp() and + datetime.utcfromtimestamp() + * os module: futimes(), futimesat(), lutimes(), utime() + * select module: epoll.poll(), kqueue.control(), select() + * signal module: setitimer(), sigtimedwait() + * time module: ctime(), gmtime(), localtime(), sleep() + +The os.stat_float_times() function is deprecated: use an explicit cast using +int() instead. + +.. note:: + The decimal module is implemented in Python and is slower than float, but + there is a new C implementation which is almost ready for inclusion in + CPython. + + +Backwards Compatibility +======================= + +The default timestamp type (float) is unchanged, so there is no impact on +backward compatibility nor on performances. The new timestamp type, +decimal.Decimal, is only returned when requested explicitly. + + +Objection: clocks accuracy +========================== + +Computer clocks and operating systems are inaccurate and fail to provide +nanosecond accuracy in practice. A nanosecond is what it takes to execute a +couple of CPU instructions. Even on a real-time operating system, a +nanosecond-precise measurement is already obsolete when it starts being +processed by the higher-level application. A single cache miss in the CPU will +make the precision worthless. + +.. note:: + Linux *actually* is able to measure time in nanosecond precision, even + though it is not able to keep its clock synchronized to UTC with a + nanosecond accuracy. + + +Alternatives: Timestamp types +============================= + +To support timestamps with an arbitrary or nanosecond resolution, the following +types have been considered: + + * decimal.Decimal + * number of nanoseconds + * 128-bits float + * datetime.datetime + * datetime.timedelta + * tuple of integers + * timespec structure + +Criteria: + + * Doing arithmetic on timestamps must be possible + * Timestamps must be comparable + * An arbitrary resolution, or at least a resolution of one nanosecond without + losing precision + * It should be possible to coerce the new timestamp to float for backward + compatibility + + +A resolution of one nanosecond is enough to support all current C functions. + +The best resolution used by operating systems is one nanosecond. In practice, +most clock accuracy is closer to microseconds than nanoseconds. So it sounds +reasonable to use a fixed resolution of one nanosecond. + + +Number of nanoseconds (int) +--------------------------- + +A nanosecond resolution is enough for all current C functions and so a +timestamp can simply be a number of nanoseconds, an integer, not a float. + +The number of nanoseconds format has been rejected because it would require to +add new specialized functions for this format because it not possible to +differentiate a number of nanoseconds and a number of seconds just by checking +the object type. + + +128-bits float +-------------- + +Add a new IEEE 754-2008 quad-precision binary float type. The IEEE 754-2008 +quad precision float has 1 sign bit, 15 bits of exponent and 112 bits of +mantissa. 128-bits float is supported by GCC (4.3), Clang and ICC compilers. + +Python must be portable and so cannot rely on a type only available on some +platforms. For example, Visual C++ 2008 doesn't support 128-bits float, whereas +it is used to build the official Windows executables. Another example: GCC 4.3 +does not support __float128 in 32-bit mode on x86 (but GCC 4.4 does). + +There is also a license issue: GCC uses the MPFR library for 128-bits float, +library distributed under the GNU LGPL license. This license is not compatible +with the Python license. + +.. note:: + The x87 floating point unit of Intel CPU supports 80-bit floats. This format + is not supported by the SSE instruction set, which is now preferred over + float, especially on x86_64. Other CPU vendors don't support 80-bit float. + + + +datetime.datetime +----------------- + +The datetime.datetime type is the natural choice for a timestamp because it is +clear that this type contains a timestamp, whereas int, float and Decimal are +raw numbers. It is an absolute timestamp and so is well defined. It gives +direct access to the year, month, day, hours, minutes and seconds. It has +methods related to time like methods to format the timestamp as string (e.g. +datetime.datetime.strftime). + +The major issue is that except os.stat(), time.time() and +time.clock_gettime(time.CLOCK_GETTIME), all time functions have an unspecified +starting point and no timezone information, and so cannot be converted to +datetime.datetime. + +datetime.datetime has also issues with timezone. For example, a datetime object +without timezone (unaware) and a datetime with a timezone (aware) cannot be +compared. There is also an ordering issues with daylight saving time (DST) in +the duplicate hour of switching from DST to normal time. + +datetime.datetime has been rejected because it cannot be used for functions +using an unspecified starting point like os.times() or time.clock(). + +For time.time() and time.clock_gettime(time.CLOCK_GETTIME): it is already +possible to get the current time as a datetime.datetime object using:: + + datetime.datetime.now(datetime.timezone.utc) + +For os.stat(), it is simple to create a datetime.datetime object from a +decimal.Decimal timestamp in the UTC timezone:: + + datetime.datetime.fromtimestamp(value, datetime.timezone.utc) + +.. note:: + datetime.datetime only supports microsecond resolution, but can be enhanced + to support nanosecond. + +datetime.timedelta +------------------ + +datetime.timedelta is the natural choice for a relative timestamp because it is +clear that this type contains a timestamp, whereas int, float and Decimal are +raw numbers. It can be used with datetime.datetime to get an absolute timestamp +when the starting point is known. + +datetime.timedelta has been rejected because it cannot be coerced to float and +has a fixed resolution. One new standard timestamp type is enough, Decimal is +preferred over datetime.timedelta. Converting a datetime.timedelta to float +requires an explicit call to the datetime.timedelta.total_seconds() method. + +.. note:: + datetime.timedelta only supports microsecond resolution, but can be enhanced + to support nanosecond. + + +.. _tuple: + +Tuple of integers +----------------- + +To expose C functions in Python, a tuple of integers is the natural choice to +store a timestamp because the C language uses structures with integers fields +(e.g. timeval and timespec structures). Using only integers avoids the loss of +precision (Python supports integers of arbitrary length). Creating and parsing +a tuple of integers is simple and fast. + +Depending of the exact format of the tuple, the precision can be arbitrary or +fixed. The precision can be choose as the loss of precision is smaller than +an arbitrary limit like one nanosecond. + +Different formats have been proposed: + + * A: (numerator, denominator) + + * value = numerator / denominator + * resolution = 1 / denominator + * denominator > 0 + + * B: (seconds, numerator, denominator) + + * value = seconds + numerator / denominator + * resolution = 1 / denominator + * 0 <= numerator < denominator + * denominator > 0 + + * C: (intpart, floatpart, base, exponent) + + * value = intpart + floatpart / base\ :sup:`exponent` + * resolution = 1 / base \ :sup:`exponent` + * 0 <= floatpart < base \ :sup:`exponent` + * base > 0 + * exponent >= 0 + + * D: (intpart, floatpart, exponent) + + * value = intpart + floatpart / 10\ :sup:`exponent` + * resolution = 1 / 10 \ :sup:`exponent` + * 0 <= floatpart < 10 \ :sup:`exponent` + * exponent >= 0 + + * E: (sec, nsec) + + * value = sec + nsec ? 10\ :sup:`-9` + * resolution = 10 \ :sup:`-9` (nanosecond) + * 0 <= nsec < 10 \ :sup:`9` + +All formats support an arbitrary resolution, except of the format (E). + +The format (D) may not be able to store the exact value (may loss of precision) +if the clock frequency is arbitrary and cannot be expressed as a power of 10. +The format (C) has a similar issue, but in such case, it is possible to use +base=frequency and exponent=1. + +The formats (C), (D) and (E) allow optimization for conversion to float if the +base is 2 and to decimal.Decimal if the base is 10. + +The format (A) is a simple fraction. It supports arbitrary precision, is simple +(only two fields), only requires a simple division to get the floating point +value, and is already used by float.as_integer_ratio(). + +To simplify the implementation (especially the C implementation to avoid +integer overflow), a numerator bigger than the denominator can be accepted. +The tuple may be normalized later. + +Tuple of integers have been rejected because they don't support arithmetic +operations. + +.. note:: + On Windows, the ``QueryPerformanceCounter()`` clock uses the frequency of + the processor which is an arbitrary number and so may not be a power or 2 or + 10. The frequency can be read using ``QueryPerformanceFrequency()``. + + +timespec structure +------------------ + +timespec is the C structure used to store timestamp with a nanosecond +resolution. Python can use a type with the same structure: (seconds, +nanoseconds). For convenience, arithmetic operations on timespec are supported. + +Example of an incomplete timespec type supporting addition, subtraction and +coercion to float:: + + class timespec(tuple): + def __new__(cls, sec, nsec): + if not isinstance(sec, int): + raise TypeError + if not isinstance(nsec, int): + raise TypeError + asec, nsec = divmod(nsec, 10 ** 9) + sec += asec + obj = tuple.__new__(cls, (sec, nsec)) + obj.sec = sec + obj.nsec = nsec + return obj + + def __float__(self): + return self.sec + self.nsec * 1e-9 + + def total_nanoseconds(self): + return self.sec * 10 ** 9 + self.nsec + + def __add__(self, other): + if not isinstance(other, timespec): + raise TypeError + ns_sum = self.total_nanoseconds() + other.total_nanoseconds() + return timespec(*divmod(ns_sum, 10 ** 9)) + + def __sub__(self, other): + if not isinstance(other, timespec): + raise TypeError + ns_diff = self.total_nanoseconds() - other.total_nanoseconds() + return timespec(*divmod(ns_diff, 10 ** 9)) + + def __str__(self): + if self.sec < 0 and self.nsec: + sec = abs(1 + self.sec) + nsec = 10**9 - self.nsec + return '-%i.%09u' % (sec, nsec) + else: + return '%i.%09u' % (self.sec, self.nsec) + + def __repr__(self): + return '' % (self.sec, self.nsec) + +The timespec type is similar to the format (E) of tuples of integer, except +that it supports arithmetic and coercion to float. + +The timespec type was rejected because it only supports nanosecond resolution +and requires to implement each arithmetic operation, whereas the Decimal type +is already implemented and well tested. + + +Alternatives: API design +======================== + +Add a string argument to specify the return type +------------------------------------------------ + +Add an string argument to function returning timestamps, example: +time.time(format="datetime"). A string is more extensible than a type: it is +possible to request a format that has no type, like a tuple of integers. + +This API was rejected because it was necessary to import implicitly modules to +instantiate objects (e.g. import datetime to create datetime.datetime). +Importing a module may raise an exception and may be slow, such behaviour is +unexpected and surprising. + + +Add a global flag to change the timestamp type +---------------------------------------------- + +A global flag like os.stat_decimal_times(), similar to os.stat_float_times(), +can be added to set globally the timestamp type. + +A global flag may cause issues with libraries and applications expecting float +instead of Decimal. Decimal is not fully compatible with float. float+Decimal +raises a TypeError for example. The os.stat_float_times() case is different +because an int can be coerced to float and int+float gives float. + + +Add a protocol to create a timestamp +------------------------------------ + +Instead of hard coding how timestamps are created, a new protocol can be added +to create a timestamp from a fraction. + +For example, time.time(timestamp=type) would call the class method +type.__fromfraction__(numerator, denominator) to create a timestamp object of +the specified type. If the type doesn't support the protocol, a fallback is +used: type(numerator) / type(denominator). + +A variant is to use a "converter" callback to create a timestamp. Example +creating a float timestamp: + + def timestamp_to_float(numerator, denominator): + return float(numerator) / float(denominator) + +Common converters can be provided by time, datetime and other modules, or maybe +a specific "hires" module. Users can define their own converters. + +Such protocol has a limitation: the timestamp structure has to be decided once +and cannot be changed later. For example, adding a timezone or the absolute +start of the timestamp would break the API. + +The protocol proposition was as being excessive given the requirements, but +that the specific syntax proposed (time.time(timestamp=type)) allows this to be +introduced later if compelling use cases are discovered. + +.. note:: + Other formats may be used instead of a fraction: see the tuple of integers + section for example. + + +Add new fields to os.stat +------------------------- + +To get the creation, modification and access time of a file with a nanosecond +resolution, three fields can be added to os.stat() structure. + +The new fields can be timestamps with nanosecond resolution (e.g. Decimal) or +the nanosecond part of each timestamp (int). + +If the new fields are timestamps with nanosecond resolution, populating the +extra fields would be time consuming. Any call to os.stat() would be slower, +even if os.stat() is only called to check if a file exists. A parameter can be +added to os.stat() to make these fields optional, the structure would have a +variable number of fields. + +If the new fields only contain the fractional part (nanoseconds), os.stat() +would be efficient. These fields would always be present and so set to zero if +the operating system does not support sub-second resolution. Splitting a +timestamp in two parts, seconds and nanoseconds, is similar to the timespec +type and tuple of integers, and so have the same drawbacks. + +Adding new fields to the os.stat() structure does not solve the nanosecond +issue in other modules (e.g. the time module). + + +Add a boolean argument +---------------------- + +Because we only need one new type (Decimal), a simple boolean flag can be +added. Example: time.time(decimal=True) or time.time(hires=True). + +Such flag would require to do an hidden import which is considered as a bad +practice. + +The boolean argument API was rejected because it is not "pythonic". Changing +the return type with a parameter value is preferred over a boolean parameter (a +flag). + + +Add new functions +----------------- + +Add new functions for each type, examples: + + * time.clock_decimal() + * time.time_decimal() + * os.stat_decimal() + * os.stat_timespec() + * etc. + +Adding a new function for each function creating timestamps duplicate a lot of +code and would be a pain to maintain. + + +Add a new hires module +---------------------- + +Add a new module called "hires" with the same API than the time module, except +that it would return timestamp with high resolution, e.g. decimal.Decimal. +Adding a new module avoids to link low-level modules like time or os to the +decimal module. + +This idea was rejected because it requires to duplicate most of the code of the +time module, would be a pain to maintain, and timestamps are used modules other +than the time module. Examples: signal.sigtimedwait(), select.select(), +resource.getrusage(), os.stat(), etc. Duplicate the code of each module is not +acceptable. + + +Links +===== + +Python: + + * `Issue #7652: Merge C version of decimal into py3k `_ (cdecimal) + * `Issue #11457: os.stat(): add new fields to get timestamps as Decimal objects with nanosecond resolution `_ + * `Issue #13882: PEP 410: Use decimal.Decimal type for timestamps `_ + * `[Python-Dev] Store timestamps as decimal.Decimal objects `_ + +Other languages: + + * Ruby (1.9.3), the `Time class `_ + supports picosecond (10\ :sup:`-12`) + * .NET framework, `DateTime type `_: + number of 100-nanosecond intervals that have elapsed since 12:00:00 + midnight, January 1, 0001. DateTime.Ticks uses a signed 64-bit integer. + * Java (1.5), `System.nanoTime() `_: + wallclock with an unspecified starting point as a number of nanoseconds, use + a signed 64 bits integer (long). + * Perl, `Time::Hiref module `_: + use float so has the same loss of precision issue with nanosecond resolution + than Python float timestamps + + +Copyright +========= + +This document has been placed in the public domain. + diff --git a/pep-0411.txt b/pep-0411.txt new file mode 100644 --- /dev/null +++ b/pep-0411.txt @@ -0,0 +1,208 @@ +PEP: 411 +Title: Provisional packages in the Python standard library +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan , + Eli Bendersky +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 2012-02-10 +Python-Version: 3.3 +Post-History: 2012-02-10 + + +Abstract +======== + +The process of including a new package into the Python standard library is +hindered by the API lock-in and promise of backward compatibility implied by +a package being formally part of Python. This PEP describes a methodology +for marking a standard library package "provisional" for the period of a single +minor release. A provisional package may have its API modified prior to +"graduating" into a "stable" state. On one hand, this state provides the +package with the benefits of being formally part of the Python distribution. +On the other hand, the core development team explicitly states that no promises +are made with regards to the the stability of the package's API, which may +change for the next release. While it is considered an unlikely outcome, +such packages may even be removed from the standard library without a +deprecation period if the concerns regarding their API or maintenance prove +well-founded. + + +Proposal - a documented provisional state +========================================= + +Whenever the Python core development team decides that a new package should be +included into the standard library, but isn't entirely sure about whether the +package's API is optimal, the package can be included and marked as +"provisional". + +In the next minor release, the package may either be "graduated" into a normal +"stable" state in the standard library, remain in provisional state, or be +rejected and removed entirely from the Python source tree. If the package ends +up graduating into the stable state after being provisional, its API may +be changed according to accumulated feedback. The core development team +explicitly makes no guarantees about API stability and backward compatibility +of provisional packages. + + +Marking a package provisional +----------------------------- + +A package will be marked provisional by a notice in its documentation page and +its docstring. The following paragraph will be added as a note at the top of +the documentation page: + + The package has been included in the standard library on a + provisional basis. Backwards incompatible changes (up to and including + removal of the package) may occur if deemed necessary by the core + developers. + +The phrase "provisional basis" will then be a link to the glossary term +"provisional package", defined as: + + A provisional package is one which has been deliberately excluded from the + standard library's normal backwards compatibility guarantees. While major + changes to such packages are not expected, as long as they are marked + provisional, backwards incompatible changes (up to and including removal of + the package) may occur if deemed necessary by core developers. Such changes + will not be made gratuitously - they will occur only if serious flaws are + uncovered that were missed prior to the inclusion of the package. + + This process allows the standard library to continue to evolve over time, + without locking in problematic design errors for extended periods of time. + See PEP 411 for more details. + +The following will be added to the start of the packages's docstring: + + The API of this package is currently provisional. Refer to the + documentation for details. + +Moving a package from the provisional to the stable state simply implies +removing these notes from its documentation page and docstring. + + +Which packages should go through the provisional state +------------------------------------------------------ + +We expect most packages proposed for addition into the Python standard library +to go through a minor release in the provisional state. There may, however, +be some exceptions, such as packages that use a pre-defined API (for example +``lzma``, which generally follows the API of the existing ``bz2`` package), +or packages with an API that has wide acceptance in the Python development +community. + +In any case, packages that are proposed to be added to the standard library, +whether via the provisional state or directly, must fulfill the acceptance +conditions set by PEP 2. + +Criteria for "graduation" +------------------------- + +In principle, most provisional packages should eventually graduate to the +stable standard library. Some reasons for not graduating are: + +* The package may prove to be unstable or fragile, without sufficient developer + support to maintain it. +* A much better alternative package may be found during the preview release. + +Essentially, the decision will be made by the core developers on a per-case +basis. The point to emphasize here is that a package's inclusion in the +standard library as "provisional" in some release does not guarantee it will +continue being part of Python in the next release. + + +Rationale +========= + +Benefits for the core development team +-------------------------------------- + +Currently, the core developers are really reluctant to add new interfaces to +the standard library. This is because as soon as they're published in a +release, API design mistakes get locked in due to backward compatibility +concerns. + +By gating all major API additions through some kind of a provisional mechanism +for a full release, we get one full release cycle of community feedback +before we lock in the APIs with our standard backward compatibility guarantee. + +We can also start integrating provisional packages with the rest of the standard +library early, so long as we make it clear to packagers that the provisional +packages should not be considered optional. The only difference between +provisional APIs and the rest of the standard library is that provisional APIs +are explicitly exempted from the usual backward compatibility guarantees. + +Benefits for end users +---------------------- + +For future end users, the broadest benefit lies in a better "out-of-the-box" +experience - rather than being told "oh, the standard library tools for task X +are horrible, download this 3rd party library instead", those superior tools +are more likely to be just be an import away. + +For environments where developers are required to conduct due diligence on +their upstream dependencies (severely harming the cost-effectiveness of, or +even ruling out entirely, much of the material on PyPI), the key benefit lies +in ensuring that all packages in the provisional state are clearly under +python-dev's aegis from at least the following perspectives: + +* Licensing: Redistributed by the PSF under a Contributor Licensing Agreement. +* Documentation: The documentation of the package is published and organized via + the standard Python documentation tools (i.e. ReST source, output generated + with Sphinx and published on http://docs.python.org). +* Testing: The package test suites are run on the python.org buildbot fleet + and results published via http://www.python.org/dev/buildbot. +* Issue management: Bugs and feature requests are handled on + http://bugs.python.org +* Source control: The master repository for the software is published + on http://hg.python.org. + + +Candidates for provisional inclusion into the standard library +============================================================== + +For Python 3.3, there are a number of clear current candidates: + +* ``regex`` (http://pypi.python.org/pypi/regex) - approved by Guido [#]_. +* ``daemon`` (PEP 3143) +* ``ipaddr`` (PEP 3144) + +Other possible future use cases include: + +* Improved HTTP modules (e.g. ``requests``) +* HTML 5 parsing support (e.g. ``html5lib``) +* Improved URL/URI/IRI parsing +* A standard image API (PEP 368) +* Improved encapsulation of import state (PEP 406) +* Standard event loop API (PEP 3153) +* A binary version of WSGI for Python 3 (e.g. PEP 444) +* Generic function support (e.g. ``simplegeneric``) + + +Rejected alternatives and variations +==================================== + +See PEP 408. + + +References +========== + +.. [#] http://mail.python.org/pipermail/python-dev/2012-January/115962.html + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0412.txt b/pep-0412.txt new file mode 100644 --- /dev/null +++ b/pep-0412.txt @@ -0,0 +1,181 @@ +PEP: 412 +Title: Key-Sharing Dictionary +Version: $Revision$ +Last-Modified: $Date$ +Author: Mark Shannon +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 08-Feb-2012 +Python-Version: 3.3 or 3.4 +Post-History: 08-Feb-2012 + + +Abstract +======== + +This PEP proposes a change in the implementation of the builtin dictionary +type ``dict``. The new implementation allows dictionaries which are used as +attribute dictionaries (the ``__dict__`` attribute of an object) to share +keys with other attribute dictionaries of instances of the same class. + +Motivation +========== + +The current dictionary implementation uses more memory than is necessary +when used as a container for object attributes as the keys are +replicated for each instance rather than being shared across many instances +of the same class. +Despite this, the current dictionary implementation is finely tuned and +performs very well as a general-purpose mapping object. + +By separating the keys (and hashes) from the values it is possible to share +the keys between multiple dictionaries and improve memory use. +By ensuring that keys are separated from the values only when beneficial, +it is possible to retain the high-performance of the current dictionary +implementation when used as a general-purpose mapping object. + +Behaviour +========= + +The new dictionary behaves in the same way as the old implementation. +It fully conforms to the Python API, the C API and the ABI. + +Performance +=========== + +Memory Usage +------------ + +Reduction in memory use is directly related to the number of dictionaries +with shared keys in existence at any time. These dictionaries are typically +half the size of the current dictionary implementation. + +Benchmarking shows that memory use is reduced by 10% to 20% for +object-oriented programs with no significant change in memory use +for other programs. + +Speed +----- + +The performance of the new implementation is dominated by memory locality +effects. When keys are not shared (for example in module dictionaries +and dictionary explicitly created by dict() or {} ) then performance is +unchanged (within a percent or two) from the current implementation. + +For the shared keys case, the new implementation tends to separate keys +from values, but reduces total memory usage. This will improve performance +in many cases as the effects of reduced memory usage outweigh the loss of +locality, but some programs may show a small slow down. + +Benchmarking shows no significant change of speed for most benchmarks. +Object-oriented benchmarks show small speed ups when they create large +numbers of objects of the same class (the gcbench benchmark shows a 10% +speed up; this is likely to be an upper limit). + +Implementation +============== + +Both the old and new dictionaries consist of a fixed-sized dict struct and +a re-sizeable table. +In the new dictionary the table can be further split into a keys table and +values array. +The keys table holds the keys and hashes and (for non-split tables) the +values as well. It differs only from the original implementation in that it +contains a number of fields that were previously in the dict struct. +If a table is split the values in the keys table are ignored, instead the +values are held in a separate array. + +Split-Table dictionaries +------------------------ + +When dictionaries are created to fill the __dict__ slot of an object, they are +created in split form. The keys table is cached in the type, potentially +allowing all attribute dictionaries of instances of one class to share keys. +In the event of the keys of these dictionaries starting to diverge, +individual dictionaries will lazily convert to the combined-table form. +This ensures good memory use in the common case, and correctness in all cases. + +When resizing a split dictionary it is converted to a combined table. +If resizing is as a result of storing an instance attribute, and there is +only instance of a class, then the dictionary will be re-split immediately. +Since most OO code will set attributes in the __init__ method, all attributes +will be set before a second instance is created and no more resizing will be +necessary as all further instance dictionaries will have the correct size. +For more complex use patterns, it is impossible to know what is the best +approach, so the implementation allows extra insertions up to the point +of a resize when it reverts to the combined table (non-shared keys). + +A deletion from a split dictionary does not change the keys table, it simply +removes the value from the values array. + +Combined-Table dictionaries +--------------------------- + +Explicit dictionaries (dict() or {}), module dictionaries and most other +dictionaries are created as combined-table dictionaries. +A combined-table dictionary never becomes a split-table dictionary. +Combined tables are laid out in much the same way as the tables in the old +dictionary, resulting in very similar performance. + +Implementation +============== + +The new dictionary implementation is available at [1]_. + +Pros and Cons +============= + +Pros +---- + +Significant memory savings for object-oriented applications. +Small improvement to speed for programs which create lots of similar objects. + +Cons +---- + +Change to data structures: +Third party modules which meddle with the internals of the dictionary +implementation will break. +Changes to repr() output and iteration order: +For most cases, this will be unchanged. +However for some split-table dictionaries the iteration order will +change. + +Neither of these cons should be a problem. +Modules which meddle with the internals of the dictionary +implementation are already broken and should be fixed to use the API. +The iteration order of dictionaries was never defined and has always been +arbitrary; it is different for Jython and PyPy. + +Alternative Implementation +-------------------------- + +An alternative implementation for split tables, which could save even more +memory, is to store an index in the value field of the keys table (instead +of ignoring the value field). This index would explicitly state where in the +value array to look. The value array would then only require 1 field for each +usable slot in the key table, rather than each slot in the key table. + +This "indexed" version would reduce the size of value array by about +one third. The keys table would need an extra "values_size" field, increasing +the size of combined dicts by one word. +The extra indirection adds more complexity to the code, potentially reducing +performance a little. + +The "indexed" version will not be included in this implementation, +but should be considered deferred rather than rejected, +pending further experimentation. + +References +========== + +.. [1] Reference Implementation: + https://bitbucket.org/markshannon/cpython_new_dict + +Copyright +========= + +This document has been placed in the public domain. + diff --git a/pep-0413.txt b/pep-0413.txt new file mode 100644 --- /dev/null +++ b/pep-0413.txt @@ -0,0 +1,919 @@ +PEP: 413 +Title: Faster evolution of the Python Standard Library +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan +Status: Draft +Type: Process +Content-Type: text/x-rst +Created: 2012-02-24 +Post-History: 2012-02-24, 2012-02-25 +Resolution: TBD + + +Abstract +======== + +This PEP proposes the adoption of a separate versioning scheme for the +standard library (distinct from, but coupled to, the existing language +versioning scheme) that allows accelerated releases of the Python standard +library, while maintaining (or even slowing down) the current rate of +change in the core language definition. + +Like PEP 407, it aims to adjust the current balance between measured +change that allows the broader community time to adapt and being able to +keep pace with external influences that evolve more rapidly than the current +release cycle can handle (this problem is particularly notable for +standard library elements that relate to web technologies). + +However, it's more conservative in its aims than PEP 407, seeking to +restrict the increased pace of development to builtin and standard library +interfaces, without affecting the rate of change for other elements such +as the language syntax and version numbering as well as the CPython +binary API and bytecode format. + + +Rationale +========= + +To quote the PEP 407 abstract: + + Finding a release cycle for an open-source project is a delicate exercise + in managing mutually contradicting constraints: developer manpower, + availability of release management volunteers, ease of maintenance for + users and third-party packagers, quick availability of new features (and + behavioural changes), availability of bug fixes without pulling in new + features or behavioural changes. + + The current release cycle errs on the conservative side. It is adequate + for people who value stability over reactivity. This PEP is an attempt to + keep the stability that has become a Python trademark, while offering a + more fluid release of features, by introducing the notion of long-term + support versions. + +I agree with the PEP 407 authors that the current release cycle of the +*standard library* is too slow to effectively cope with the pace of change +in some key programming areas (specifically, web protocols and related +technologies, including databases, templating and serialisation formats). + +However, I have written this competing PEP because I believe that the +approach proposed in PEP 407 of offering full, potentially binary +incompatible releases of CPython every 6 months places too great a burden +on the wider Python ecosystem. + +Under the current CPython release cycle, distributors of key binary +extensions will often support Python releases even after the CPython branches +enter "security fix only" mode (for example, Twisted currently ships binaries +for 2.5, 2.6 and 2.7, NumPy and SciPy suport those 3 along with 3.1 and 3.2, +PyGame adds a 2.4 binary release, wxPython provides both 32-bit and 64-bit +binaries for 2.6 and 2.7, etc). + +If CPython were to triple (or more) its rate of releases, the developers of +those libraries (many of which are even more resource starved than CPython) +would face an unpalatable choice: either adopt the faster release cycle +themselves (up to 18 simultaneous binary releases for PyGame!), drop +older Python versions more quickly, or else tell their users to stick to the +CPython LTS releases (thus defeating the entire point of speeding up the +CPython release cycle in the first place). + +Similarly, many support tools for Python (e.g. syntax highlighters) can take +quite some time to catch up with language level changes. + +At a cultural level, the Python community is also accustomed to a certain +meaning for Python version numbers - they're linked to deprecation periods, +support periods, all sorts of things. PEP 407 proposes that collective +knowledge all be swept aside, without offering a compelling rationale for why +such a course of action is actually *necessary* (aside from, perhaps, making +the lives of the CPython core developers a little easier at the expense of +everyone else). + +However, if we go back to the primary rationale for increasing the pace of +change (i.e. more timely support for web protocols and related technologies), +we can note that those only require *standard library* changes. That means +many (perhaps even most) of the negative effects on the wider community can +be avoided by explicitly limiting which parts of CPython are affected by the +new release cycle, and allowing other parts to evolve at their current, more +sedate, pace. + + +Proposal +======== + +This PEP proposes the introduction of a new kind of CPython release: +"standard library releases". As with PEP 407, this will give CPython 3 kinds +of release: + +* Language release: "x.y.0" +* Maintenance release: "x.y.z" (where z > 0) +* Standard library release: "x.y (xy.z)" (where z > 0) + +Under this scheme, an unqualified version reference (such as "3.3") would +always refer to the most recent corresponding language or maintenance +release. It will never be used without qualification to refer to a standard +library release (at least, not by python-dev - obviously, we can only set an +example, not force the rest of the Python ecosystem to go along with it). + +Language releases will continue as they are now, as new versions of the +Python language definition, along with a new version of the CPython +interpreter and the Python standard library. Accordingly, a language +release may contain any and all of the following changes: + +* new language syntax +* new standard library changes (see below) +* new deprecation warnings +* removal of previously deprecated features +* changes to the emitted bytecode +* changes to the AST +* any other significant changes to the compilation toolchain +* changes to the core interpreter eval loop +* binary incompatible changes to the C ABI (although the PEP 384 stable ABI + must still be preserved) +* bug fixes + +Maintenance releases will also continue as they do today, being strictly +limited to bug fixes for the corresponding language release. No new features +or radical internal changes are permitted. + +The new standard library releases will occur in parallel with each +maintenance release and will be qualified with a new version identifier +documenting the standard library version. Standard library releases may +include the following changes: + +* new features in pure Python modules +* new features in C extension modules (subject to PEP 399 compatibility + requirements) +* new features in language builtins (provided the C ABI remains unaffected) +* bug fixes from the corresponding maintenance release + +Standard library version identifiers are constructed by combining the major +and minor version numbers for the Python language release into a single two +digit number and then appending a sequential standard library version +identifier. + + +Release Cycle +------------- + +When maintenance releases are created, *two* new versions of Python would +actually be published on python.org (using the first 3.3 maintenance release, +planned for February 2013 as an example):: + + 3.3.1 # Maintenance release + 3.3 (33.1) # Standard library release + +A further 6 months later, the next 3.3 maintenance release would again be +accompanied by a new standard library release:: + + 3.3.2 # Maintenance release + 3.3 (33.2) # Standard library release + +Again, the standard library release would be binary compatible with the +previous language release, merely offering additional features at the +Python level. + +Finally, 18 months after the release of 3.3, a new language release would +be made around the same time as the final 3.3 maintenance and standard +library releases:: + + 3.3.3 # Maintenance release + 3.3 (33.3) # Standard library release + 3.4.0 # Language release + +The 3.4 release cycle would then follow a similar pattern to that for 3.3:: + + 3.4.1 # Maintenance release + 3.4 (34.1) # Standard library release + + 3.4.2 # Maintenance release + 3.4 (34.2) # Standard library release + + 3.4.3 # Maintenance release + 3.4 (34.3) # Standard library release + 3.5.0 # Language release + + +Programmatic Version Identification +----------------------------------- + +To expose the new version details programmatically, this PEP proposes the +addition of a new ``sys.stdlib_info`` attribute that records the new +standard library version above and beyond the underlying interpreter +version. Using the initial Python 3.3 release as an example:: + + sys.stdlib_info(python=33, version=0, releaselevel='final', serial=0) + +This information would also be included in the ``sys.version`` string:: + + Python 3.3.0 (33.0, default, Feb 17 2012, 23:03:41) + [GCC 4.6.1] + + +Security Fixes and Other "Out of Cycle" Releases +------------------------------------------------ + +For maintenance releases the process of handling out-of-cycle releases (for +example, to fix a security issue or resolve a critical bug in a new release), +remains the same as it is now: the minor version number is incremented and a +new release is made incorporating the required bug fixes, as well as any +other bug fixes that have been committed since the previous release. + +For standard library releases, the process is essentially the same, but the +corresponding "What's New?" document may require some tidying up for the +release (as the standard library release may incorporate new features, +not just bug fixes). + + +User Scenarios +============== + +The versioning scheme proposed above is based on a number of user scenarios +that are likely to be encountered if this scheme is adopted. In each case, +the scenario is described for both the status quo (i.e. slow release cycle) +the versioning scheme in this PEP and the free wheeling minor version number +scheme proposed in PEP 407. + +To give away the ending, the point of using a separate version number is that +for almost all scenarios, the important number is the *language* version, not +the standard library version. Most users won't even need to care that the +standard library version number exists. In the two identified cases where +it matters, providing it as a separate number is actually clearer and more +explicit than embedding the two different kinds of number into a single +sequence and then tagging some of the numbers in the unified sequence as +special. + + +Novice user, downloading Python from python.org in March 2013 +------------------------------------------------------------- + +**Status quo:** must choose between 3.3 and 2.7 + +**This PEP:** must choose between 3.3 (33.1), 3.3 and 2.7. + +**PEP 407:** must choose between 3.4, 3.3 (LTS) and 2.7. + +**Verdict:** explaining the meaning of a Long Term Support release is about as +complicated as explaining the meaning of the proposed standard library release +version numbers. I call this a tie. + + +Novice user, attempting to judge currency of third party documentation +---------------------------------------------------------------------- + +**Status quo:** minor version differences indicate 18-24 months of +language evolution + +**This PEP:** same as status quo for language core, standard library version +numbers indicate 6 months of standard library evolution. + +**PEP 407:** minor version differences indicate 18-24 months of language +evolution up to 3.3, then 6 months of language evolution thereafter. + +**Verdict:** Since language changes and deprecations can have a much bigger +effect on the accuracy of third party documentation than the addition of new +features to the standard library, I'm calling this a win for the scheme +in this PEP. + + +Novice user, looking for an extension module binary release +----------------------------------------------------------- + +**Status quo:** look for the binary corresponding to the Python version you are +running. + +**This PEP:** same as status quo. + +**PEP 407 (full releases):** same as status quo, but corresponding binary version +is more likely to be missing (or, if it does exist, has to be found amongst +a much larger list of alternatives). + +**PEP 407 (ABI updates limited to LTS releases):** all binary release pages will +need to tell users that Python 3.3, 3.4 and 3.5 all need the 3.3 binary. + +**Verdict:** I call this a clear win for the scheme in this PEP. Absolutely +nothing changes from the current situation, since the standard library +version is actually irrelevant in this case (only binary extension +compatibility is important). + + +Extension module author, deciding whether or not to make a binary release +------------------------------------------------------------------------- + +**Status quo:** unless using the PEP 384 stable ABI, a new binary release is +needed every time the minor version number changes. + +**This PEP:** same as status quo. + +**PEP 407 (full releases):** same as status quo, but becomes a far more +frequent occurrence. + +**PEP 407 (ABI updates limited to LTS releases):** before deciding, must first +look up whether the new release is an LTS release or an interim release. If +it is an LTS release, then a new build is necessary. + +**Verdict:** I call this another clear win for the scheme in this PEP. As with +the end user facing side of this problem, the standard library version is +actually irrelevant in this case. Moving that information out to a +separate number avoids creating unnecessary confusion. + + +Python developer, deciding priority of eliminating a Deprecation Warning +------------------------------------------------------------------------ + +**Status quo:** code that triggers deprecation warnings is not guaranteed to +run on a version of Python with a higher minor version number. + +**This PEP:** same as status quo + +**PEP 407:** unclear, as the PEP doesn't currently spell this out. Assuming the +deprecation cycle is linked to LTS releases, then upgrading to a non-LTS +release is safe but upgrading to the next LTS release may require avoiding +the deprecated construct. + +**Verdict:** another clear win for the scheme in this PEP since, once again, the +standard library version is irrelevant in this scenario. + + +Alternative interpreter implementor, updating with new features +--------------------------------------------------------------- + +**Status quo:** new Python versions arrive infrequently, but are a mish-mash of +standard library updates and core language definition and interpreter +changes. + +**This PEP:** standard library updates, which are easier to integrate, are +made available more frequently in a form that is clearly and explicitly +compatible with the previous version of the language definition. This means +that, once an alternative implementation catches up to Python 3.3, they +should have a much easier time incorporating standard library features as +they happen (especially pure Python changes), leaving minor version number +updates as the only task that requires updates to their core compilation and +execution components. + +**PEP 407 (full releases):** same as status quo, but becomes a far more +frequent occurrence. + +**PEP 407 (language updates limited to LTS releases):** unclear, as the PEP +doesn't currently spell out a specific development strategy. Assuming a +3.3 compatibility branch is adopted (as proposed in this PEP), then the +outcome would be much the same, but the version number signalling would be +slightly less clear (since you would have to check to see if a particular +release was an LTS release or not). + +**Verdict:** while not as clear cut as some previous scenarios, I'm still +calling this one in favour of the scheme in this PEP. Explicit is better than +implicit, and the scheme in this PEP makes a clear split between the two +different kinds of update rather than adding a separate "LTS" tag to an +otherwise ordinary release number. Tagging a particular version as being +special is great for communicating with version control systems and associated +automated tools, but it's a lousy way to communicate information to other +humans. + +Python developer, deciding their minimum version dependency +----------------------------------------------------------- + +**Status quo:** look for "version added" or "version changed" markers in the +documentation, check against ``sys.version_info`` + +**This PEP:** look for "version added" or "version changed" markers in the +documentation. If written as a bare Python version, such as "3.3", check +against ``sys.version_info``. If qualified with a standard library version, +such as "3.3 (33.1)", check against ``sys.stdlib_info``. + +**PEP 407:** same as status quo + +**Verdict:** the scheme in this PEP actually allows third party libraries to be +more explicit about their rate of adoption of standard library features. More +conservative projects will likely pin their dependency to the language +version and avoid features added in the standard library releases. Faster +moving projects could instead declare their dependency on a particular +standard library version. However, since PEP 407 does have the advantage of +preserving the status quo, I'm calling this one for PEP 407 (albeit with a +slim margin). + + +Python developers, attempting to reproduce a tracker issue +---------------------------------------------------------- + +**Status quo:** if not already provided, ask the reporter which version of +Python they're using. This is often done by asking for the first two lines +displayed by the interactive prompt or the value of ``sys.version``. + +**This PEP:** same as the status quo (as ``sys.version`` will be updated to +also include the standard library version), but may be needed on additional +occasions (where the user knew enough to state their Python version, but that +proved to be insufficient to reproduce the fault). + +**PEP 407:** same as the status quo + +**Verdict:** another marginal win for PEP 407. The new standard library version +*is* an extra piece of information that users may need to pass back to +developers when reporting issues with Python libraries (or Python itself, +on our own tracker). However, by including it in ``sys.version``, many +fault reports will already include it, and it is easy to request if needed. + + +CPython release managers, handling a security fix +------------------------------------------------- + +**Status quo:** create a new maintenance release incorporating the security +fix and any other bug fixes under source control. Also create source releases +for any branches open solely for security fixes. + +**This PEP:** same as the status quo for maintenance branches. Also create a +new standard library release (potentially incorporating new features along +with the security fix). For security branches, create source releases for +both the former maintenance branch and the standard library update branch. + +**PEP 407:** same as the status quo for maintenance and security branches, +but handling security fixes for non-LTS releases is currently an open +question. + +**Verdict:** until PEP 407 is updated to actually address this scenario, a +clear win for this PEP. + + +Effects +======= + +Effect on development cycle +--------------------------- + +Similar to PEP 407, this PEP will break up the delivery of new features into +more discrete chunks. Instead of a whole raft of changes landing all at once +in a language release, each language release will be limited to 6 months +worth of standard library changes, as well as any changes associated with +new syntax. + + +Effect on workflow +------------------ + +This PEP proposes the creation of a single additional branch for use in the +normal workflow. After the release of 3.3, the following branches would be +in use:: + + 2.7 # Maintenance branch, no change + 3.3 # Maintenance branch, as for 3.2 + 3.3-compat # New branch, backwards compatible changes + default # Language changes, standard library updates that depend on them + +When working on a new feature, developers will need to decide whether or not +it is an acceptable change for a standard library release. If so, then it +should be checked in on ``3.3-compat`` and then merged to ``default``. +Otherwise it should be checked in directly to ``default``. + +The "version added" and "version changed" markers for any changes made on +the ``3.3-compat`` branch would need to be flagged with both the language +version and the standard library version. For example: "3.3 (33.1)". + +Any changes made directly on the ``default`` branch would just be flagged +with "3.4" as usual. + +The ``3.3-compat`` branch would be closed to normal development at the +same time as the ``3.3`` maintenance branch. The ``3.3-compat`` branch would +remain open for security fixes for the same period of time as the ``3.3`` +maintenance branch. + + +Effect on bugfix cycle +---------------------- + +The effect on the bug fix workflow is essentially the same as that on the +workflow for new features - there is one additional branch to pass through +before the change reaches the ``default`` branch. + +If critical bugs are found in a maintenance release, then new maintenance and +standard library releases will be created to resolve the problem. The final +part of the version number will be incremented for both the language version +and the standard library version. + +If critical bugs are found in a standard library release that do not affect +the associated maintenance release, then only a new standard library release +will be created and only the standard library's version number will be +incremented. + +Note that in these circumstances, the standard library release *may* include +additional features, rather than just containing the bug fix. It is +assumed that anyone that cares about receiving *only* bug fixes without any +new features mixed in will already be relying strictly on the maintenance +releases rather than using the new standard library releases. + + +Effect on the community +----------------------- + +PEP 407 has this to say about the effects on the community: + + People who value stability can just synchronize on the LTS releases which, + with the proposed figures, would give a similar support cycle (both in + duration and in stability). + +I believe this statement is just plain wrong. Life isn't that simple. Instead, +developers of third party modules and frameworks will come under pressure to +support the full pace of the new release cycle with binary updates, teachers +and book authors will receive complaints that they're only covering an "old" +version of Python ("You're only using 3.3, the latest is 3.5!"), etc. + +As the minor version number starts climbing 3 times faster than it has in the +past, I believe perceptions of language stability would also fall (whether +such opinions were justified or not). + +I believe isolating the increased pace of change to the standard library, +and clearly delineating it with a separate version number will greatly +reassure the rest of the community that no, we're not suddenly +asking them to triple their own rate of development. Instead, we're merely +going to ship standard library updates for the next language release in +6-monthly installments rather than delaying them all until the next language +definition update, even those changes that are backwards compatible with the +previously released version of Python. + +The community benefits listed in PEP 407 are equally applicable to this PEP, +at least as far as the standard library is concerned: + + People who value reactivity and access to new features (without taking the + risk to install alpha versions or Mercurial snapshots) would get much more + value from the new release cycle than currently. + + People who want to contribute new features or improvements would be more + motivated to do so, knowing that their contributions will be more quickly + available to normal users. + +If the faster release cycle encourages more people to focus on contributing +to the standard library rather than proposing changes to the language +definition, I don't see that as a bad thing. + + +Handling News Updates +===================== + + +What's New? +----------- + +The "What's New" documents would be split out into separate documents for +standard library releases and language releases. So, during the 3.3 release +cycle, we would see: + +* What's New in Python 3.3? +* What's New in the Python Standard Library 33.1? +* What's New in the Python Standard Library 33.2? +* What's New in the Python Standard Library 33.3? + +And then finally, we would see the next language release: + +* What's New in Python 3.4? + +For the benefit of users that ignore standard library releases, the 3.4 +What's New would link back to the What's New documents for each of the +standard library releases in the 3.3 series. + + +NEWS +---- + +Merge conflicts on the NEWS file are already a hassle. Since this PEP +proposes introduction of an additional branch into the normal workflow, +resolving this becomes even more critical. While Mercurial phases may +help to some degree, it would be good to eliminate the problem entirely. + +One suggestion from Barry Warsaw is to adopt a non-conflicting +separate-files-per-change approach, similar to that used by Twisted [2_]. + +Given that the current manually updated NEWS file will be used for the 3.3.0 +release, one possible layout for such an approach might look like:: + + Misc/ + NEWS # Now autogenerated from news_entries + news_entries/ + 3.3/ + NEWS # Original 3.3 NEWS file + maint.1/ # Maintenance branch changes + core/ + + builtins/ + + extensions/ + + library/ + + documentation/ + + tests/ + + compat.1/ # Compatibility branch changes + builtins/ + + extensions/ + + library/ + + documentation/ + + tests/ + + # Add maint.2, compat.2 etc as releases are made + 3.4/ + core/ + + builtins/ + + extensions/ + + library/ + + documentation/ + + tests/ + + # Add maint.1, compat.1 etc as releases are made + +Putting the version information in the directory heirarchy isn't strictly +necessary (since the NEWS file generator could figure out from the version +history), but does make it easier for *humans* to keep the different versions +in order. + + +Other benefits of reduced version coupling +========================================== + +Slowing down the language release cycle +--------------------------------------- + +The current release cycle is a compromise between the desire for stability +in the core language definition and C extension ABI, and the desire to get +new features (most notably standard library updates) into user's hands more +quickly. + +With the standard library release cycle decoupled (to some degree) from that +of the core language definition, it provides an opportunity to actually +*slow down* the rate of change in the language definition. The language +moratorium for Python 3.2 effectively slowed that cycle down to *more than 3 +years* (3.1: June 2009, 3.3: August 2012) without causing any major +problems or complaints. + +The NEWS file management scheme described above is actually designed to +allow us the flexibility to slow down language releases at the same time +as standard library releases become more frequent. + +As a simple example, if a full two years was allowed between 3.3 and 3.4, +the 3.3 release cycle would end up looking like:: + + 3.2.4 # Maintenance release + 3.3.0 # Language release + + 3.3.1 # Maintenance release + 3.3 (33.1) # Standard library release + + 3.3.2 # Maintenance release + 3.3 (33.2) # Standard library release + + 3.3.3 # Maintenance release + 3.3 (33.3) # Standard library release + + 3.3.4 # Maintenance release + 3.3 (33.4) # Standard library release + 3.4.0 # Language release + +The elegance of the proposed branch structure and NEWS entry layout is that +this decision wouldn't really need to be made until shortly before the planned +3.4 release date. At that point, the decision could be made to postpone the +3.4 release and keep the ``3.3`` and ``3.3-compat`` branches open after the +3.3.3 maintenance release and the 3.3 (33.3) standard library release, thus +adding another standard library release to the cycle. The choice between +another standard library release or a full language release would then be +available every 6 months after that. + + +Further increasing the pace of standard library development +----------------------------------------------------------- + +As noted in the previous section, one benefit of the scheme proposed in this +PEP is that it largely decouples the language release cycle from the +standard library release cycle. The standard library could be updated every +3 months, or even once a month, without having any flow on effects on the +language version numbering or the perceived stability of the core language. + +While that pace of development isn't practical as long as the binary +installer creation for Windows and Mac OS X involves several manual steps +(including manual testing) and for as long as we don't have separate +"-release" trees that only receive versions that have been marked as +good by the stable buildbots, it's still a useful criterion to keep in mind +when considering proposed new versioning schemes: what if we eventually want +to make standard library releases even *faster* than every 6 months? + +If the practical issues were ever resolved, then the separate standard +library versioning scheme in this PEP could handle it. The tagged version +number approach proposed in PEP 407 could not (at least, not without a lot +of user confusion and uncertainty). + + +Other Questions +=============== + +Why not use the major version number? +------------------------------------- + +The simplest and most logical solution would actually be to map the +major.minor.micro version numbers to the language version, stdlib version +and maintenance release version respectively. + +Instead of releasing Python 3.3.0, we would instead release Python 4.0.0 +and the release cycle would look like:: + + 4.0.0 # Language release + + 4.0.1 # Maintenance release + 4.1.0 # Standard library release + + 4.0.2 # Maintenance release + 4.2.0 # Standard library release + + 4.0.3 # Maintenance release + 4.3.0 # Standard library release + 5.0.0 # Language release + +However, the ongoing pain of the Python 2 -> Python 3 transition (and +associated workarounds like the ``python3`` and ``python2`` symlinks to +refer directly to the desired release series) means that this simple option +isn't viable for historical reasons. + +One way that this simple approach *could* be made to work is to merge the +current major and minor version numbers directly into a 2-digit major +version number:: + + 33.0.0 # Language release + + 33.0.1 # Maintenance release + 33.1.0 # Standard library release + + 33.0.2 # Maintenance release + 33.2.0 # Standard library release + + 33.0.3 # Maintenance release + 33.3.0 # Standard library release + 34.0.0 # Language release + + +Why not use a four part version number? +--------------------------------------- + +Another simple versioning scheme would just add a "standard library" version +into the existing versioning scheme:: + + 3.3.0.0 # Language release + + 3.3.0.1 # Maintenance release + 3.3.1.0 # Standard library release + + 3.3.0.2 # Maintenance release + 3.3.2.0 # Standard library release + + 3.3.0.3 # Maintenance release + 3.3.3.0 # Standard library release + 3.4.0.0 # Language release + +However, this scheme isn't viable due to backwards compatibility constraints +on the ``sys.version_info`` structure. + + +Why not use a date-based versioning scheme? +------------------------------------------- + +Earlier versions of this PEP proposed a date-based versioning scheme for +the standard library. However, such a scheme made it very difficult to +handle out-of-cycle releases to fix security issues and other critical +bugs in standard library releases, as it required the following steps: + +1. Change the release version number to the date of the current month. +2. Update the What's New, NEWS and documentation to refer to the new release + number. +3. Make the new release. + +With the sequential scheme now proposed, such releases should at most require +a little tidying up of the What's New document before making the release. + + +Why isn't PEP 384 enough? +------------------------- + +PEP 384 introduced the notion of a "Stable ABI" for CPython, a limited +subset of the full C ABI that is guaranteed to remain stable. Extensions +built against the stable ABI should be able to support all subsequent +Python versions with the same binary. + +This will help new projects to avoid coupling their C extension modules too +closely to a specific version of CPython. For existing modules, however, +migrating to the stable ABI can involve quite a lot of work (especially for +extension modules that define a lot of classes). With limited development +resources available, any time spent on such a change is time that could +otherwise have been spent working on features that offer more direct benefits +to end users. + +There are also other benefits to separate versioning (as described above) +that are not directly related to the question of binary compatibility with +third party C extensions. + + +Why no binary compatible additions to the C ABI in standard library releases? +----------------------------------------------------------------------------- + +There's a case to be made that *additions* to the CPython C ABI could +reasonably be permitted in standard library releases. This would give C +extension authors the same freedom as any other package or module author +to depend either on a particular language version or on a standard library +version. + +The PEP currently associates the interpreter version with the language +version, and therefore limits major interpreter changes (including C ABI +additions) to the language releases. + +An alternative, internally consistent, approach would be to link the +interpreter version with the standard library version, with only changes that +may affect backwards compatibility limited to language releases. + +Under such a scheme, the following changes would be acceptable in standard +library releases: + +* Standard library updates + + * new features in pure Python modules + * new features in C extension modules (subject to PEP 399 compatibility + requirements) + * new features in language builtins + +* Interpreter implementation updates + + * binary compatible additions to the C ABI + * changes to the compilation toolchain that do not affect the AST or alter + the bytecode magic number + * changes to the core interpreter eval loop + +* bug fixes from the corresponding maintenance release + +And the following changes would be acceptable in language releases: + +* new language syntax +* any updates acceptable in a standard library release +* new deprecation warnings +* removal of previously deprecated features +* changes to the AST +* changes to the emitted bytecode that require altering the magic number +* binary incompatible changes to the C ABI (although the PEP 384 stable ABI + must still be preserved) + +While such an approach could probably be made to work, there does not appear +to be a compelling justification for it, and the approach currently described +in the PEP is simpler and easier to explain. + + +Why not separate out the standard library entirely? +--------------------------------------------------- + +A concept that is occasionally discussed is the idea of making the standard +library truly independent from the CPython reference implementation. + +My personal opinion is that actually making such a change would involve a +lot of work for next to no pay-off. CPython without the standard library is +useless (the build chain won't even run, let alone the test suite). You also +can't create a standalone pure Python standard library either, because too +many "standard library modules" are actually tightly linked in to the +internal details of their respective interpreters (for example, the builtins, +``weakref``, ``gc``, ``sys``, ``inspect``, ``ast``). + +Creating a separate CPython development branch that is kept compatible with +the previous language release, and making releases from that branch that are +identified with a separate standard library version number should provide +most of the benefits of a separate standard library repository with only a +fraction of the pain. + + +Acknowledgements +================ + +Thanks go to the PEP 407 authors for starting this discussion, as well as +to those authors and Larry Hastings for initial discussions of the proposal +made in this PEP. + +References +========== + +.. [1] PEP 407: New release cycle and introducing long-term support versions + http://www.python.org/dev/peps/pep-0407/ + +.. [2] Twisted's "topfiles" approach to NEWS generation + http://twistedmatrix.com/trac/wiki/ReviewProcess#Newsfiles + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0414.txt b/pep-0414.txt new file mode 100644 --- /dev/null +++ b/pep-0414.txt @@ -0,0 +1,415 @@ +PEP: 414 +Title: Explicit Unicode Literal for Python 3.3 +Version: $Revision$ +Last-Modified: $Date$ +Author: Armin Ronacher , + Nick Coghlan +Status: Final +Type: Standards Track +Content-Type: text/x-rst +Created: 15-Feb-2012 +Post-History: 28-Feb-2012, 04-Mar-2012 +Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116995.html + + +Abstract +======== + +This document proposes the reintegration of an explicit unicode literal +from Python 2.x to the Python 3.x language specification, in order to +reduce the volume of changes needed when porting Unicode-aware +Python 2 applications to Python 3. + + +BDFL Pronouncement +================== + +This PEP has been formally accepted for Python 3.3: + + I'm accepting the PEP. It's about as harmless as they come. Make it so. + + +Proposal +======== + +This PEP proposes that Python 3.3 restore support for Python 2's Unicode +literal syntax, substantially increasing the number of lines of existing +Python 2 code in Unicode aware applications that will run without modification +on Python 3. + +Specifically, the Python 3 definition for string literal prefixes will be +expanded to allow:: + + "u" | "U" | "ur" | "UR" | "Ur" | "uR" + +in addition to the currently supported:: + + "r" | "R" + +The following will all denote ordinary Python 3 strings:: + + 'text' + "text" + '''text''' + """text""" + u'text' + u"text" + u'''text''' + u"""text""" + U'text' + U"text" + U'''text''' + U"""text""" + +Combination of the unicode prefix with the raw string prefix will also be +supported, just as it was in Python 2. + +No changes are proposed to Python 3's actual Unicode handling, only to the +acceptable forms for string literals. + + +Author's Note +============= + +This PEP was originally written by Armin Ronacher, and Guido's approval was +given based on that version. + +The currently published version has been rewritten by Nick Coghlan to +include additional historical details and rationale that were taken into +account when Guido made his decision, but were not explicitly documented in +Armin's version of the PEP. + +Readers should be aware that many of the arguments in this PEP are *not* +technical ones. Instead, they relate heavily to the *social* and *personal* +aspects of software development. + + +Rationale +========= + +With the release of a Python 3 compatible version of the Web Services Gateway +Interface (WSGI) specification (PEP 3333) for Python 3.2, many parts of the +Python web ecosystem have been making a concerted effort to support Python 3 +without adversely affecting their existing developer and user communities. + +One major item of feedback from key developers in those communities, including +Chris McDonough (WebOb, Pyramid), Armin Ronacher (Flask, Werkzeug), Jacob +Kaplan-Moss (Django) and Kenneth Reitz (``requests``) is that the requirement +to change the spelling of *every* Unicode literal in an application +(regardless of how that is accomplished) is a key stumbling block for porting +efforts. + +In particular, unlike many of the other Python 3 changes, it isn't one that +framework and library authors can easily handle on behalf of their users. Most +of those users couldn't care less about the "purity" of the Python language +specification, they just want their websites and applications to work as well +as possible. + +While it is the Python web community that has been most vocal in highlighting +this concern, it is expected that other highly Unicode aware domains (such as +GUI development) may run into similar issues as they (and their communities) +start making concerted efforts to support Python 3. + + +Common Objections +================= + + +Complaint: This PEP may harm adoption of Python 3.2 +--------------------------------------------------- + +This complaint is interesting, as it carries within it a tacit admission that +this PEP *will* make it easier to port Unicode aware Python 2 applications to +Python 3. + +There are many existing Python communities that are prepared to put up with +the constraints imposed by the existing suite of porting tools, or to update +their Python 2 code bases sufficiently that the problems are minimised. + +This PEP is not for those communities. Instead, it is designed specifically to +help people that *don't* want to put up with those difficulties. + +However, since the proposal is for a comparatively small tweak to the language +syntax with no semantic changes, it is feasible to support it as a third +party import hook. While such an import hook imposes some import time +overhead, and requires additional steps from each application that needs it +to get the hook in place, it allows applications that target Python 3.2 +to use libraries and frameworks that would otherwise only run on Python 3.3+ +due to their use of unicode literal prefixes. + +One such import hook project is Vinay Sajip's ``uprefix`` [4]_. + +For those that prefer to translate their code in advance rather than +converting on the fly at import time, Armin Ronacher is working on a hook +that runs at install time rather than during import [5]_. + +Combining the two approaches is of course also possible. For example, the +import hook could be used for rapid edit-test cycles during local +development, but the install hook for continuous integration tasks and +deployment on Python 3.2. + +The approaches described in this section may prove useful, for example, for +applications that wish to target Python 3 on the Ubuntu 12.04 LTS release, +which will ship with Python 2.7 and 3.2 as officially supported Python +versions. + +Complaint: Python 3 shouldn't be made worse just to support porting from Python 2 +--------------------------------------------------------------------------------- + +This is indeed one of the key design principles of Python 3. However, one of +the key design principles of Python as a whole is that "practicality beats +purity". If we're going to impose a significant burden on third party +developers, we should have a solid rationale for doing so. + +In most cases, the rationale for backwards incompatible Python 3 changes are +either to improve code correctness (for example, stricter default separation +of binary and text data and integer division upgrading to floats when +necessary), reduce typical memory usage (for example, increased usage of +iterators and views over concrete lists), or to remove distracting nuisances +that make Python code harder to read without increasing its expressiveness +(for example, the comma based syntax for naming caught exceptions). Changes +backed by such reasoning are *not* going to be reverted, regardless of +objections from Python 2 developers attempting to make the transition to +Python 3. + +In many cases, Python 2 offered two ways of doing things for historical reasons. +For example, inequality could be tested with both ``!=`` and ``<>`` and integer +literals could be specified with an optional ``L`` suffix. Such redundancies +have been eliminated in Python 3, which reduces the overall size of the +language and improves consistency across developers. + +In the original Python 3 design (up to and including Python 3.2), the explicit +prefix syntax for unicode literals was deemed to fall into this category, as it +is completely unnecessary in Python 3. However, the difference between those +other cases and unicode literals is that the unicode literal prefix is *not* +redundant in Python 2 code: it is a programmatically significant distinction +that needs to be preserved in some fashion to avoid losing information. + +While porting tools were created to help with the transition (see next section) +it still creates an additional burden on heavy users of unicode strings in +Python 2, solely so that future developers learning Python 3 don't need to be +told "For historical reasons, string literals may have an optional ``u`` or +``U`` prefix. Never use this yourselves, it's just there to help with porting +from an earlier version of the language." + +Plenty of students learning Python 2 received similar warnings regarding string +exceptions without being confused or irreparably stunted in their growth as +Python developers. It will be the same with this feature. + +This point is further reinforced by the fact that Python 3 *still* allows the +uppercase variants of the ``B`` and ``R`` prefixes for bytes literals and raw +bytes and string literals. If the potential for confusion due to string prefix +variants is that significant, where was the outcry asking that these +redundant prefixes be removed along with all the other redundancies that were +eliminated in Python 3? + +Just as support for string exceptions was eliminated from Python 2 using the +normal deprecation process, support for redundant string prefix characters +(specifically, ``B``, ``R``, ``u``, ``U``) may eventually be eliminated +from Python 3, regardless of the current acceptance of this PEP. However, +such a change will likely only occur once third party libraries supporting +Python 2.7 is about as common as libraries supporting Python 2.2 or 2.3 is +today. + + +Complaint: The WSGI "native strings" concept is an ugly hack +------------------------------------------------------------ + +One reason the removal of unicode literals has provoked such concern amongst +the web development community is that the updated WSGI specification had to +make a few compromises to minimise the disruption for existing web servers +that provide a WSGI-compatible interface (this was deemed necessary in order +to make the updated standard a viable target for web application authors and +web framework developers). + +One of those compromises is the concept of a "native string". WSGI defines +three different kinds of string: + +* text strings: handled as ``unicode`` in Python 2 and ``str`` in Python 3 +* native strings: handled as ``str`` in both Python 2 and Python 3 +* binary data: handled as ``str`` in Python 2 and ``bytes`` in Python 3 + +Some developers consider WSGI's "native strings" to be an ugly hack, as they +are *explicitly* documented as being used solely for ``latin-1`` decoded +"text", regardless of the actual encoding of the underlying data. Using this +approach bypasses many of the updates to Python 3's data model that are +designed to encourage correct handling of text encodings. However, it +generally works due to the specific details of the problem domain - web server +and web framework developers are some of the individuals *most* aware of how +blurry the line can get between binary data and text when working with HTTP +and related protocols, and how important it is to understand the implications +of the encodings in use when manipulating encoded text data. At the +*application* level most of these details are hidden from the developer by +the web frameworks and support libraries (both in Python 2 *and* in Python 3). + +In practice, native strings are a useful concept because there are some APIs +(both in the standard library and in third party frameworks and packages) and +some internal interpreter details that are designed primarily to work with +``str``. These components often don't support ``unicode`` in Python 2 +or ``bytes`` in Python 3, or, if they do, require additional encoding details +and/or impose constraints that don't apply to the ``str`` variants. + +Some example of interfaces that are best handled by using actual ``str`` +instances are: + +* Python identifiers (as attributes, dict keys, class names, module names, + import references, etc) +* URLs for the most part as well as HTTP headers in urllib/http servers +* WSGI environment keys and CGI-inherited values +* Python source code for dynamic compilation and AST hacks +* Exception messages +* ``__repr__`` return value +* preferred filesystem paths +* preferred OS environment + +In Python 2.6 and 2.7, these distinctions are most naturally expressed as +follows: + +* ``u""``: text string (``unicode``) +* ``""``: native string (``str``) +* ``b""``: binary data (``str``, also aliased as ``bytes``) + +In Python 3, the ``latin-1`` decoded native strings are not distinguished +from any other text strings: + +* ``""``: text string (``str``) +* ``""``: native string (``str``) +* ``b""``: binary data (``bytes``) + +If ``from __future__ import unicode_literals`` is used to modify the behaviour +of Python 2, then, along with an appropriate definition of ``n()``, the +distinction can be expressed as: + +* ``""``: text string +* ``n("")``: native string +* ``b""``: binary data + +(While ``n=str`` works for simple cases, it can sometimes have problems +due to non-ASCII source encodings) + +In the common subset of Python 2 and Python 3 (with appropriate +specification of a source encoding and definitions of the ``u()`` and ``b()`` +helper functions), they can be expressed as: + +* ``u("")``: text string +* ``""``: native string +* ``b("")``: binary data + +That last approach is the only variant that supports Python 2.5 and earlier. + +Of all the alternatives, the format currently supported in Python 2.6 and 2.7 +is by far the cleanest approach that clearly distinguishes the three desired +kinds of behaviour. With this PEP, that format will also be supported in +Python 3.3+. It will also be supported in Python 3.1 and 3.2 through the use +of import and install hooks. While it is significantly less likely, it is +also conceivable that the hooks could be adapted to allow the use of the +``b`` prefix on Python 2.5. + + +Complaint: The existing tools should be good enough for everyone +---------------------------------------------------------------- + +A commonly expressed sentiment from developers that have already sucessfully +ported applications to Python 3 is along the lines of "if you think it's hard, +you're doing it wrong" or "it's not that hard, just try it!". While it is no +doubt unintentional, these responses all have the effect of telling the +people that are pointing out inadequacies in the current porting toolset +"there's nothing wrong with the porting tools, you just suck and don't know +how to use them properly". + +These responses are a case of completely missing the point of what people are +complaining about. The feedback that resulted in this PEP isn't due to people complaining that ports aren't possible. Instead, the feedback is coming from +people that have succesfully *completed* ports and are objecting that they +found the experience thoroughly *unpleasant* for the class of application that +they needed to port (specifically, Unicode aware web frameworks and support +libraries). + +This is a subjective appraisal, and it's the reason why the Python 3 +porting tools ecosystem is a case where the "one obvious way to do it" +philosophy emphatically does *not* apply. While it was originally intended that +"develop in Python 2, convert with ``2to3``, test both" would be the standard +way to develop for both versions in parallel, in practice, the needs of +different projects and developer communities have proven to be sufficiently +diverse that a variety of approaches have been devised, allowing each group +to select an approach that best fits their needs. + +Lennart Regebro has produced an excellent overview of the available migration +strategies [2]_, and a similar review is provided in the official porting +guide [3]_. (Note that the official guidance has softened to "it depends on +your specific situation" since Lennart wrote his overview). + +However, both of those guides are written from the founding assumption that +all of the developers involved are *already* committed to the idea of +supporting Python 3. They make no allowance for the *social* aspects of such a +change when you're interacting with a user base that may not be especially +tolerant of disruptions without a clear benefit, or are trying to persuade +Python 2 focused upstream developers to accept patches that are solely about +improving Python 3 forward compatibility. + +With the current porting toolset, *every* migration strategy will result in +changes to *every* Unicode literal in a project. No exceptions. They will +be converted to either an unprefixed string literal (if the project decides to +adopt the ``unicode_literals`` import) or else to a converter call like +``u("text")``. + +If the ``unicode_literals`` import approach is employed, but is not adopted +across the entire project at the same time, then the meaning of a bare string +literal may become annoyingly ambiguous. This problem can be particularly +pernicious for *aggregated* software, like a Django site - in such a situation, +some files may end up using the ``unicode_literals`` import and others may not, +creating definite potential for confusion. + +While these problems are clearly solvable at a technical level, they're a +completely unnecessary distraction at the social level. Developer energy should +be reserved for addressing *real* technical difficulties associated with the +Python 3 transition (like distinguishing their 8-bit text strings from their +binary data). They shouldn't be punished with additional code changes (even +automated ones) solely due to the fact that they have *already* explicitly +identified their Unicode strings in Python 2. + +Armin Ronacher has created an experimental extension to 2to3 which only +modernizes Python code to the extent that it runs on Python 2.7 or later with +support from the cross-version compatibility ``six`` library. This tool is +available as ``python-modernize`` [1]_. Currently, the deltas generated by +this tool will affect every Unicode literal in the converted source. This +will create legitimate concerns amongst upstream developers asked to accept +such changes, and amongst framework *users* being asked to change their +applications. + +However, by eliminating the noise from changes to the Unicode literal syntax, +many projects could be cleanly and (comparatively) non-controversially made +forward compatible with Python 3.3+ just by running ``python-modernize`` and +applying the recommended changes. + + +References +========== + +.. [1] Python-Modernize + (http://github.com/mitsuhiko/python-modernize) + +.. [2] Porting to Python 3: Migration Strategies + (http://python3porting.com/strategies.html) + +.. [3] Porting Python 2 Code to Python 3 + (http://docs.python.org/howto/pyporting.html) + +.. [4] uprefix import hook project + (https://bitbucket.org/vinay.sajip/uprefix) + +.. [5] install hook to remove unicode string prefix characters + (https://github.com/mitsuhiko/unicode-literals-pep/tree/master/install-hook) + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + End: diff --git a/pep-0415.txt b/pep-0415.txt new file mode 100644 --- /dev/null +++ b/pep-0415.txt @@ -0,0 +1,88 @@ +PEP: 415 +Title: Implementing PEP 409 differently +Version: $Revision$ +Last-Modified: $Date$ +Author: Benjamin Peterson +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 26-Feb-2012 +Post-History: 26-Feb-2012 + + +Abstract +======== + +PEP 409 allows PEP 3134 exception contexts and causes to be suppressed when the +exception is printed. This is done using the ``raise exc from None`` +syntax. This PEP proposes to implement context and cause suppression +differently. + +Rationale +========= + +PEP 409 changes ``__cause__`` to be ``Ellipsis`` by default. Then if +``__cause__`` is set to ``None`` by ``raise exc from None``, no context or cause +will be printed should the exception be uncaught. + +The main problem with this scheme is it complicates the role of +``__cause__``. ``__cause__`` should indicate the cause of the exception not +whether ``__context__`` should be printed or not. This use of ``__cause__`` is +also not easily extended in the future. For example, we may someday want to +allow the programmer to select which of ``__context__`` and ``__cause__`` will +be printed. The PEP 409 implementation is not amenable to this. + +The use of ``Ellipsis`` is a hack. Before PEP 409, ``Ellipsis`` was used +exclusively in extended slicing. Extended slicing has nothing to do with +exceptions, so it's not clear to someone inspecting an exception object why +``__cause__`` should be set to ``Ellipsis``. Using ``Ellipsis`` by default for +``__cause__`` makes it asymmetrical with ``__context__``. + +Proposal +======== + +A new attribute on ``BaseException``, ``__suppress_context__``, will +be introduced. Whenever ``__cause__`` is set, ``__suppress_context__`` +will be set to ``True``. In particular, ``raise exc from cause`` +syntax will set ``exc.__suppress_context__`` to ``True``. Exception +printing code will check for that attribute to determine whether +context and cause will be printed. ``__cause__`` will return to its +original purpose and values. + +There is precedence for ``__suppress_context__`` with the +``print_line_and_file`` exception attribute. + +To summarize, ``raise exc from cause`` will be equivalent to:: + + exc.__cause__ = cause + raise exc + +where ``exc.__cause__ = cause`` implicitly sets +``exc.__suppress_context__``. + +Patches +======= + +There is a patch on `Issue 14133`_. + + +References +========== + +.. _issue 14133: + http://bugs.python.org/issue6210 + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: diff --git a/pep-0416.txt b/pep-0416.txt new file mode 100644 --- /dev/null +++ b/pep-0416.txt @@ -0,0 +1,134 @@ +PEP: 416 +Title: Add a frozendict builtin type +Version: $Revision$ +Last-Modified: $Date$ +Author: Victor Stinner +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 29-February-2012 +Python-Version: 3.3 + + +Abstract +======== + +Add a new frozendict builtin type. + + +Rationale +========= + +A frozendict is a read-only mapping: a key cannot be added nor removed, and a +key is always mapped to the same value. However, frozendict values can be +mutable (not hashable). A frozendict is hashable and so immutable if and only +if all values are hashable (immutable). + +Use cases: + + * frozendict lookup can be done at compile time instead of runtime because the + mapping is read-only. frozendict can be used instead of a preprocessor to + remove conditional code at compilation, like code specific to a debug build. + * hashable frozendict can be used as a key of a mapping or as a member of set. + frozendict can be used to implement a cache. + * frozendict avoids the need of a lock when the frozendict is shared + by multiple threads or processes, especially hashable frozendict. It would + also help to prohibe coroutines (generators + greenlets) to modify the + global state. + * frozendict helps to implement read-only object proxies for security modules. + For example, it would be possible to use frozendict type for __builtins__ + mapping or type.__dict__. This is possible because frozendict is compatible + with the PyDict C API. + * frozendict avoids the need of a read-only proxy in some cases. frozendict is + faster than a proxy because getting an item in a frozendict is a fast lookup + whereas a proxy requires a function call. + * use a frozendict as the default value of function argument: avoid the + problem of mutable default argument. + + +Constraints +=========== + + * frozendict has to implement the Mapping abstract base class + * frozendict keys and values can be unorderable + * a frozendict is hashable if all keys and values are hashable + * frozendict hash does not depend on the items creation order + + +Implementation +============== + + * Add a PyFrozenDictObject structure based on PyDictObject with an extra + "Py_hash_t hash;" field + * frozendict.__hash__() is implemented using hash(frozenset(self.items())) and + caches the result in its private hash attribute + * Register frozendict as a collections.abc.Mapping + * frozendict can be used with PyDict_GetItem(), but PyDict_SetItem() and + PyDict_DelItem() raise a TypeError + + +Recipe: hashable dict +====================== + +To ensure that a a frozendict is hashable, values can be checked +before creating the frozendict:: + + import itertools + + def hashabledict(*args, **kw): + # ensure that all values are hashable + for key, value in itertools.chain(args, kw.items()): + if isinstance(value, (int, str, bytes, float, frozenset, complex)): + # avoid the compute the hash (which may be slow) for builtin + # types known to be hashable for any value + continue + hash(value) + # don't check the key: frozendict already checks the key + return frozendict.__new__(cls, *args, **kw) + + +Objections +========== + +*namedtuple may fit the requiements of a frozendict.* + +A namedtuple is not a mapping, it does not implement the Mapping abstract base +class. + +*frozendict can be implemented in Python using descriptors" and "frozendict +just need to be practically constant.* + +If frozendict is used to harden Python (security purpose), it must be +implemented in C. A type implemented in C is also faster. + +*The PEP 351 was rejected.* + +The PEP 351 tries to freeze an object and so may convert a mutable object to an +immutable object (using a different type). frozendict doesn't convert anything: +hash(frozendict) raises a TypeError if a value is not hashable. Freezing an +object is not the purpose of this PEP. + + +Links +===== + + * PEP 412: Key-Sharing Dictionary + (`issue #13903 `_) + * PEP 351: The freeze protocol + * `The case for immutable dictionaries; and the central misunderstanding of PEP 351 `_ + * `Frozen dictionaries (Python recipe 414283) `_ + by Oren Tirosh + * Python security modules implementing read-only object proxies using a C + extension: + + * `pysandbox `_ + * `mxProxy `_ + * `zope.proxy `_ + * `zope.security `_ + + +Copyright +========= + +This document has been placed in the public domain. + diff --git a/pep-3000.txt b/pep-3000.txt --- a/pep-3000.txt +++ b/pep-3000.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum -Status: Active +Status: Final Type: Process Content-Type: text/x-rst Created: 05-Apr-2006 diff --git a/pep-3002.txt b/pep-3002.txt --- a/pep-3002.txt +++ b/pep-3002.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Steven Bethard -Status: Draft +Status: Final Type: Process Content-Type: text/x-rst Created: 27-Mar-2006 diff --git a/pep-3003.txt b/pep-3003.txt --- a/pep-3003.txt +++ b/pep-3003.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon, Jesse Noller, Guido van Rossum -Status: Active +Status: Final Type: Process Content-Type: text/x-rst Created: 21-Oct-2009 diff --git a/pep-3099.txt b/pep-3099.txt --- a/pep-3099.txt +++ b/pep-3099.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Georg Brandl -Status: Active +Status: Final Type: Process Content-Type: text/x-rst Created: 04-Apr-2006 diff --git a/pep-3100.txt b/pep-3100.txt --- a/pep-3100.txt +++ b/pep-3100.txt @@ -3,8 +3,8 @@ Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon -Status: Active -Type: Informational +Status: Final +Type: Process Content-Type: text/x-rst Created: 20-Aug-2004 Post-History: diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -1,382 +1,160 @@ -PEP: 3144 -Title: IP Address Manipulation Library for the Python Standard Library -Version: $Revision$ -Last-Modified: $Date$ -Author: Peter Moody -Discussions-To: ipaddr-py-dev at googlegroups.com -Status: Draft -Type: Standards Track -Content-Type: text/plain -Created: 13-Aug-2009 -Python-Version: 3.2 +PEP: 3144 +Title: IP Address Manipulation Library for the Python Standard Library +Version: $Revision$ +Last-Modified: $Date$ +Author: Peter Moody +Discussions-To: +Status: Draft +Type: Standards Track +Content-Type: text/plain +Created: 6-Feb-2012 +Python-Version: 3.3 Abstract: - This PEP proposes a design for a lightweight ip address manipulation module - for python. - + This PEP proposes a design and for an IP address manipulation module for + python. Motivation: - Many network administrators use python in their day to day jobs. Finding a - library to assist with the common ip address manipulation tasks is easy. - Finding a good library for performing those tasks can be somewhat more - difficult. For this reason, I (like many before me) scratched an itch and - wrote my own with an emphasis on being easy to understand and fast for the - most common operations. - - For context, a previous version of this library was up for inclusion in - python 3.1, see issue 3959 [1] for more information. + Several very good IP address modules for python already exist. + The truth is that all of the struggle with the balance between + adherence to Pythonic principals and the shorthand upon which + network engineers and administrators rely. I believe ipaddr + strikes the right balance. Rationale: - ipaddr was designed with a few basic principals in mind: + The existance of several Python IP address manipulation moduels is + evidence of an outstanding need for the functionality this module + seeks to provide. - - IPv4 and IPv6 objects are distinct. - - IP addresses and IP networks are distinct. - - the library should be useful and the assumptions obvious to the network - programmer. - - IP networks should be treated as lists (as opposed to some other - python intrinsic) in so far as it makes sense. - - the library should be lightweight and fast without sacrificing - expected functionality. - - - Distinct IPV4 and IPV6 objects. - While there are many similarities, IPV4 and IPV6 objects are fundamentally - different. The similarities allow for easy abstraction of certain - operations which affect the bits from both in the same manner, but their - differences mean attempts to combine them into one object yield unexpected - results. According to Vint Cerf, "I have seen a substantial amount of - traffic about IPv4 and IPv6 comparisons and the general consensus is that - these are not comparable." (Vint Cerf [2]). For python versions >= 3.0, - this means that (<, >, <=, >=) comparison operations between IPv4 and IPv6 - objects raise a TypeError per the Ordering Comparisons [3]. +Background: - - Distinct network and address objects. + PEP 3144 and ipaddr have been up for inclusion before. The + version of the library specified here is backwards incompatible + with the version on PyPI and the one which was discussed before. + In order to avoid confusing users of the current ipaddr, I've + renamed this version of the library "ipaddress". - An IPV4 address is a single 32 bit number while the IPV4 address assigned - to a networked computer is a 32 bit address and associated network. - Similarly, an IPV6 address is a 128 bit number while an IPV6 address - assigned to a networked computer is a 128 bit number and associated network - information. The similarities leads to easy abstraction of some methods - and properties, but there are obviously a number of address/network - specific properties which require they be distinct. For instance, IP - networks contain a network address (the base address of the network), - broadcast address (the upper end of the network, also the address to - which every machine on a given network is supposed listen, hence the name - broadcast), supernetworks and subnetworks, etc. The individual property - addresses in an IP network obviously don't have the same properties, - they're simply 32 or 128 bit numbers. + The main differences between ipaddr and ipaddress are: - - Principal of least confusion for network programmers. + * ipaddress *Network classes are equivalent to the ipaddr *Network + class counterparts with the strict flag set to True. - It should be understood that, above all, this module is designed with the - network administrator in mind. In practice, this means that a number of - assumptions are made with regards to common usage and the library prefers - the usefulness of accepted practice over strict adherence to RFCs. For - example, ipaddr accepts '192.168.1.1/24' as a network definition because - this is a very common way of describing an address + netmask despite the - fact that 192.168.1.1 is actually an IP address on the network - 192.168.1.0/24. Strict adherence would require that networks have all of - the host bits masked to zero, which would require two objects to describe - that IP + network. In practice, a looser interpretation of a network is - a very useful if common abstraction, so ipaddr prefers to make this - available. For the developer who is concerned with strict adherence, - ipaddr provides an optional 'strict' boolean argument to the - IPv(4|6)Network constructors which guarantees that all host bits are masked - down. - - - Treat network elements as lists (in so far as it's possible). + * ipaddress *Interface classes are equivalent to the ipaddr + *Network class counterparts with the strict flag set to False. - Treating IP networks as lists is a natural extension from viewing the - network as a series of individual ip addresses. Most of the standard list - methods should be implemented and should behave in a manner that would be - consistent if the IP network object were actually a list of strings or - integers. The methods which actually modify a lists contents don't extend - as well to this model (__add__, __iadd__, __sub__, __isub__, etc) but - others (__contains__, __iter__, etc) work quite nicely. It should be noted - that __len__ doesn't work as expected since python internals has this - limited to a 32 bit integer and it would need to be at least 128 bits to - work with IPV6. + * The factory functions in ipaddress were renamed to disambiguate + them from classes. - - Lightweight. - - While some network programmers will undoubtedly want more than this library - provides, keeping the functionality to strictly what's required from a IP - address manipulation module is critical to keeping the code fast, easily - comprehensible and extensible. It is a goal to provide enough options in - terms of functionality to allow the developer to easily do their work - without needlessly cluttering the library. Finally, It's important to note - that this design doesn't prevent subclassing or otherwise extending to meet - the unforeseen needs. + * A few attributes were renamed to disambiguate their purpose as + well. (eg. network, network_address) Specification: - A slightly more detailed look at the library follows. + The ipaddr module defines a total of 6 new public classes, 3 for + manipulating IPv4 objects and 3 for manipulating IPv6 objects. + The classes are as follows: - - Design + IPv4Address/IPv6Address - These define individual addresses, for + example the IPv4 address returned by an A record query for + www.google.com (74.125.224.84) or the IPv6 address returned by a + AAAA record query for ipv6.google.com (2001:4860:4001:801::1011). - ipaddr has four main classes most people will use: + IPv4Network/IPv6Network - These define networks or groups of + addresses, for example the IPv4 network reserved for multicast use + (224.0.0.0/4) or the IPv6 network reserved for multicast + (ff00::/8, wow, that's big). - 1. IPv4Address. (eg, '192.168.1.1') - 2. IPv4Network (eg, '192.168.0.0/16') - 3. IPv6Address (eg, '::1') - 4. IPv6Network (eg, '2001::/32') + IPv4Interface/IPv6Interface - These hybrid classes refer to an + individual address on a given network. For example, the IPV4 + address 192.0.2.1 on the network 192.0.2.0/24 could be referred to + as 192.0.2.1/24. Likewise, the IPv6 address 2001:DB8::1 on the + network 2001:DB8::/96 could be referred to as 2001:DB8::1/96. + It's very common to refer to addresses assigned to computer + network interfaces like this, hence the Interface name. - Most of the operations a network administrator performs on networks are - similar for both IPv4 and IPv6 networks. Ie. finding subnets, supernets, - determining if an address is contained in a given network, etc. Similarly, - both addresses and networks (of the same ip version!) have much in common; - the process for turning a given 32 or 128 bit number into a human readable - string notation, determining if the ip is within the valid specified range, - etc. Finally, there are some pythonic abstractions which are valid for all - addresses and networks, both IPv4 and IPv6. In short, there is common - functionality shared between (ipaddr class names in parentheses): + All IPv4 classes share certain characteristics and methods; the + number of bits needed to represent them, whether or not they + belong to certain special IPv4 network ranges, etc. Similarly, + all IPv6 classes share characteristics and methods. - 1. all IP addresses and networks, both IPv4 and IPv6. (_IPAddrBase) + ipaddr makes extensive use of inheritance to avoid code + duplication as much as possible. The parent classes are private, + but they are outlined here: - 2. all IP addresses of both versions. (_BaseIP) + _IPAddrBase - Provides methods common to all ipaddr objects. - 3. all IP networks of both version. (_BaseNet) + _BaseAddress - Provides methods common to IPv4Address and + IPv6Address. - 4. all IPv4 objects, both addresses and networks. (_BaseV4) + _BaseInterface - Provides methods common to IPv4Interface and + IPv6Interface, as well as IPv4Network and IPv6Network (ipaddr + treats the Network classes as a special case of Interface). - 5. all IPv6 objects, both addresses and networks. (_BaseV6) + _BaseV4 - Provides methods and variables (eg, _max_prefixlen) + common to all IPv4 classes. - Seeing this as a clear hierarchy is important for recognizing how much - code is common between the four main classes. For this reason, ipaddr uses - class inheritance to abstract out as much common code is possible and - appropriate. This lack of duplication and very clean layout also makes - the job of the developer much easier should they need to debug code (either - theirs or mine). + _BaseV6 - Provides methods and variables common to all IPv6 + classes. - Knowing that there might be cases where the developer doesn't so much care - as to the types of IP they might be receiving, ipaddr comes with two - important helper functions, IPAddress() and IPNetwork(). These, as you - might guess, return the appropriately typed address or network objects for - the given argument. + Comparisons between objects of differing IP versions results in a + TypeError [1]. Additionally, comparisons of objects with + different _Base parent classes results in a TypeError. The effect + of the _Base parent class limitation is that IPv4Interface's can + be compared to IPv4Network's and IPv6Interface's can be compared + to IPv6Network's. - Finally, as mentioned earlier, there is no meaningful natural ordering - between IPv4 and IPv6 addresses and networks [2]. Rather than invent a - standard, ipaddr follows Ordering Comparisons and returns a TypeError - when asked to compare objects of differing IP versions. In practice, there - are many ways a programmer may wish to order the addresses, so this this - shouldn't pose a problem for the developer who can easily write: - - v4 = [x for x in mixed_list if x._version == 4] - v6 = [x for x in mixed_list if x._version == 6] - - # perform operations on v4 and v6 here. - - return v4_return + v6_return - - - Multiple ways of displaying an IP Address. - - Not everyone will want to display the same information in the same format; - IP addresses in cisco syntax are represented by network/hostmask, junipers - are (network/IP)/prefixlength and IPTables are (network/IP)/(prefixlength/ - netmask). The ipaddr library provides multiple ways to display an address. - - In [1]: IPNetwork('1.1.1.1').with_prefixlen - Out[1]: '1.1.1.1/32' - - In [1]: IPNetwork('1.1.1.1').with_netmask - Out[1]: '1.1.1.1/255.255.255.255' - - In [1]: IPNetwork('1.1.1.1').with_hostmask - Out[1]: '1.1.1.1/0.0.0.0' - - the same applies to IPv6. It should be noted that netmasks and hostmasks - are not commonly used in IPv6, the methods exist for compatibility with - IPv4. - - - Lazy evaluation combined with aggressive caching of network elements. - - (the following example is for IPv6Network objects but the exact same - properties apply to IPv6Network objects). - - As mentioned, an IP network object is defined by a number of properties. - The object - - In [1]: IPv4Network('1.1.1.0/24') - - has a number of IPv4Address properties - - In [1]: o = IPv4Network('1.1.1.0/24') - - In [2]: o.network - Out[2]: IPv4Address('1.1.1.0') - - In [3]: o.broadcast - Out[3]: IPv4Address('1.1.1.255') - - In [4]: o.hostmask - Out[4]: IPv4Address('0.0.0.255') - - If we were to compute them all at object creation time, we would incur a - non-negligible performance hit. Since these properties are required to - define the object completely but their values aren't always of interest to - the programmer, their computation should be done only when requested. - However, in order to avoid the performance hit in the case where one - attribute for a particular object is requested repeatedly (and continuously - recomputed), the results of the computation should be cached. - - - Address list summarization. - - ipaddr supports easy summarization of lists of possibly contiguous - addresses, as this is something network administrators constantly find - themselves doing. This currently works in a number of ways. - - 1. collapse_address_list([list]): - - Given a list of networks, ipaddr will collapse the list into the smallest - possible list of networks that wholey contain the addresses supplied. - - In [1]: collapse_address_list([IPNetwork('1.1.0.0/24'), - ...: IPNetwork('1.1.1.0/24')]) - Out[1]: [IPv4Network('1.1.0.0/23')] - - more elaborately: - - In [1]: collapse_address_list([IPNetwork(x) for x in - ...: IPNetwork('1.1.0.0/23')]) - Out[1]: [IPv4Network('1.1.0.0/23')] - - 2. summarize_address_range(first, last). - - Given a start and end address, ipaddr will provide the smallest number of - networks to cover the given range. - - - In [1]: summarize_address_range(IPv4Address('1.1.1.0'), - ...: IPv4Address('2.2.2.0')) - Out[1]: - [IPv4Network('1.1.1.0/24'), - IPv4Network('1.1.2.0/23'), - IPv4Network('1.1.4.0/22'), - IPv4Network('1.1.8.0/21'), - IPv4Network('1.1.16.0/20'), - IPv4Network('1.1.32.0/19'), - IPv4Network('1.1.64.0/18'), - IPv4Network('1.1.128.0/17'), - IPv4Network('1.2.0.0/15'), - IPv4Network('1.4.0.0/14'), - IPv4Network('1.8.0.0/13'), - IPv4Network('1.16.0.0/12'), - IPv4Network('1.32.0.0/11'), - IPv4Network('1.64.0.0/10'), - IPv4Network('1.128.0.0/9'), - IPv4Network('2.0.0.0/15'), - IPv4Network('2.2.0.0/23'), - IPv4Network('2.2.2.0/32')] - - - Address Exclusion. - - Used somewhat less often, but all the more annoying, is the case where an - programmer would want "all of the addresses in a newtork *except* these". - ipaddr performs this exclusion equally well for IPv4 and IPv6 networks - and collapses the resulting address list. - - In [1]: IPNetwork('1.1.0.0/15').address_exclude(IPNetwork('1.1.1.0/24')) - Out[1]: - [IPv4Network('1.0.0.0/16'), - IPv4Network('1.1.0.0/24'), - IPv4Network('1.1.2.0/23'), - IPv4Network('1.1.4.0/22'), - IPv4Network('1.1.8.0/21'), - IPv4Network('1.1.16.0/20'), - IPv4Network('1.1.32.0/19'), - IPv4Network('1.1.64.0/18'), - IPv4Network('1.1.128.0/17')] - - In [1]: IPNewtork('::1/96').address_exclude(IPNetwork('::1/112')) - Out[1]: - [IPv6Network('::1:0/112'), - IPv6Network('::2:0/111'), - IPv6Network('::4:0/110'), - IPv6Network('::8:0/109'), - IPv6Network('::10:0/108'), - IPv6Network('::20:0/107'), - IPv6Network('::40:0/106'), - IPv6Network('::80:0/105'), - IPv6Network('::100:0/104'), - IPv6Network('::200:0/103'), - IPv6Network('::400:0/102'), - IPv6Network('::800:0/101'), - IPv6Network('::1000:0/100'), - IPv6Network('::2000:0/99'), - IPv6Network('::4000:0/98'), - IPv6Network('::8000:0/97')] - - - IPv6 address compression. - - By default, IPv6 addresses are compressed internally (see the method - BaseV6._compress_hextets), but ipaddr makes both the compressed and the - exploded representations available. - - In [1]: IPNetwork('::1').compressed - Out[1]: '::1/128' - - In [2]: IPNetwork('::1').exploded - Out[2]: '0000:0000:0000:0000:0000:0000:0000:1/128' - - In [3]: IPv6Address('::1').exploded - Out[3]: '0000:0000:0000:0000:0000:0000:0000:0001' - - In [4]: IPv6Address('::1').compressed - Out[4]: '::1' - - (the same methods exist for IPv4 networks and addresses, but they're - just stubs for returning the normal __str__ representation). - - - Most other common operations. - - It is a design goal to support all of the common operation expected from - an IP address manipulation module. As such, finding supernets, subnets, - address and network containment etc are all supported. Reference Implementation: - A reference implementation is available at: - http://ipaddr-py.googlecode.com/svn/trunk + The current reference implementation can be found at: + http://code.google.com/p/ipaddr-py/downloads/detail?name=3144.tar.gz + + More information about using the reference implementation can be + found at: http://code.google.com/p/ipaddr-py/wiki/Using3144 References: - [1] http://bugs.python.org/issue3959 - [2] Appealing to authority is a logical fallacy, but Vint Cerf is an - an authority who can't be ignored. Full text of the email follows: + + [1] Appealing to authority is a logical fallacy, but Vint Cerf is an + an authority who can't be ignored. Full text of the email + follows: """ - I have seen a substantial amount of traffic about IPv4 and IPv6 - comparisons and the general consensus is that these are not comparable. + I have seen a substantial amount of traffic about IPv4 and + IPv6 comparisons and the general consensus is that these are + not comparable. - If we were to take a very simple minded view, we might treat these as - pure integers in which case there is an ordering but not a useful one. + If we were to take a very simple minded view, we might treat + these as pure integers in which case there is an ordering but + not a useful one. - In the IPv4 world, "length" is important because we take longest (most - specific) address first for routing. Length is determine by the mask, - as you know. + In the IPv4 world, "length" is important because we take + longest (most specific) address first for routing. Length is + determine by the mask, as you know. - Assuming that the same style of argument works in IPv6, we would have - to conclude that treating an IPv6 value purely as an integer for - comparison with IPv4 would lead to some really strange results. + Assuming that the same style of argument works in IPv6, we + would have to conclude that treating an IPv6 value purely as + an integer for comparison with IPv4 would lead to some really + strange results. - All of IPv4 space would lie in the host space of 0::0/96 prefix of - IPv6. For any useful interpretation of IPv4, this is a non-starter. + All of IPv4 space would lie in the host space of 0::0/96 + prefix of IPv6. For any useful interpretation of IPv4, this is + a non-starter. - I think the only sensible conclusion is that IPv4 values and IPv6 values - should be treated as non-comparable. + I think the only sensible conclusion is that IPv4 values and + IPv6 values should be treated as non-comparable. Vint """ - [3] http://docs.python.org/dev/3.0/whatsnew/3.0.html#ordering-comparisons - Copyright: diff --git a/pep-3150.txt b/pep-3150.txt --- a/pep-3150.txt +++ b/pep-3150.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan -Status: Deferred +Status: Withdrawn Type: Standards Track Content-Type: text/x-rst Created: 2010-07-09 @@ -47,45 +47,21 @@ but has not yet itself been subject to the test of implementation. -PEP Deferral -============ +PEP Withdrawal +============== -Despite the lifting of the language moratorium (PEP 3003) for Python 3.3, -this PEP currently remains in a Deferred state. This idea, if implemented, -will potentially have a deep and pervasive effect on the way people write -Python code. +I've had a complicated history with this PEP. For a long time I left it in +the Deferred state because I wasn't convinced the additional complexity was +worth the payoff. Then, briefly, I became more enamoured of the idea and +only left it at Deferred because I didn't really have time to pursue it. -When this PEP was first put forward, even I, as the PEP author, was not -convinced it was a good idea. Instead, I was simply writing it as a way to -avoid endlessly rehashing similar topics on python-ideas. When someone -broached the subject, they could be pointed at this PEP and told "Come back -when you've read and understood the arguments presented there". Subsequent -discussions (most notably, those surrounding PEP 403's attempt at a more -restricted version of the idea) have convinced me that the idea is valuable -and will help address a number of situations where developers feel that -Python "gets in the way" instead of "matching the way they think". For me, -it is this aspect of "let people express what they're thinking, rather than -forcing them to think differently due to Python's limitations" that finally -allowed the idea to clear the "status quo wins a stalemate" bar ([5]_). +I'm now withdrawing it, as, the longer I reflect on the topic, the more I +feel this approach is simply far too intrusive and complicated to ever be +a good idea for Python as a language. -However, while I now think the idea is worthwhile, I don't think there is -sufficient time left in the 3.3 release cycle for the idea to mature. A -reference implementation is needed, and people need time to experiment with -that implementation and offer feedback on whether or not it helps with -programming paradigms that are currently somewhat clumsy in Python (like -callback programming). Even if a PEP co-author volunteered immediately to -work on the implementation and incorporate feedback into the PEP text, I feel -targetting 3.3 would be unnecessarily rushing things. So, I've marked this -PEP as a candidate for 3.4 rather than 3.3. - -Once that process is complete, Guido van Rossum (or his delegate) will need -to be sufficiently convinced of the idea's merit and accept the PEP. Such -acceptance will require not only a fully functional reference implementation -for CPython (as already mentioned), but also indications from the other three -major Python implementations (PyPy, Jython, IronPython) that they consider -it feasible to implement the proposed semantics once they reach the point of -targetting 3.4 compatibility. Input from related projects with a vested -interest in Python's syntax (e.g. Cython) will also be valuable. +I've also finally found a couple of syntax proposals for PEP 403 that +read quite nicely and address the same set of use cases as this PEP +while remaining significantly simpler. Proposal diff --git a/pep-3154.txt b/pep-3154.txt --- a/pep-3154.txt +++ b/pep-3154.txt @@ -71,27 +71,20 @@ special method (``__getnewargs_ex__`` ?) and a new opcode (NEWOBJEX ?) are needed. -Serializing more callable objects ---------------------------------- +Serializing more "lookupable" objects +------------------------------------- -Currently, only module-global functions are serializable. -Multiprocessing has custom support for pickling other callables such -as bound methods [4]_. This support could be folded in the protocol, -and made more efficient through a new GETATTR opcode. +For some kinds of objects, it only makes sense to serialize them by name +(for example classes and functions). By default, pickle is only able to +serialize module-global functions and classes by name. Supporting other +kinds of objects, such as unbound methods [4]_, is a common request. +Actually, third-party support for some of them, such as bound methods, +is implemented in the multiprocessing module [5]_. -Serializing "pseudo-global" objects ------------------------------------ - -Objects which are not module-global, but should be treated in a -similar fashion -- such as unbound methods [5]_ or nested classes -- -cannot currently be pickled (or, rather, unpickled) because the pickle -protocol does not correctly specify how to retrieve them. One -solution would be through the adjunction of a ``__namespace__`` (or -``__qualname__``) to all class and function objects, specifying the -full "path" by which they can be retrieved. For globals, this would -generally be ``"{}.{}".format(obj.__module__, obj.__name__)``. Then a -new opcode can resolve that path and push the object on the stack, -similarly to the GLOBAL opcode. +:pep:`3155` now makes it possible to lookup many more objects by name. +Generalizing the GLOBAL opcode to accept dot-separated names, or adding +a special GETATTR opcode, would allow the standard pickle implementation +to support, in an efficient way, all those kinds of objects. Binary encoding for all opcodes ------------------------------- @@ -131,12 +124,12 @@ .. [3] "pickle/copyreg doesn't support keyword only arguments in __new__": http://bugs.python.org/issue4727 -.. [4] Lib/multiprocessing/forking.py: +.. [4] "pickle should support methods": + http://bugs.python.org/issue9276 + +.. [5] Lib/multiprocessing/forking.py: http://hg.python.org/cpython/file/baea9f5f973c/Lib/multiprocessing/forking.py#l54 -.. [5] "pickle should support methods": - http://bugs.python.org/issue9276 - Copyright ========= diff --git a/pep-3155.txt b/pep-3155.txt --- a/pep-3155.txt +++ b/pep-3155.txt @@ -3,13 +3,13 @@ Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou -Status: Draft +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 2011-10-29 Python-Version: 3.3 Post-History: -Resolution: TBD +Resolution: http://mail.python.org/pipermail/python-dev/2011-November/114545.html Rationale @@ -59,15 +59,16 @@ Proposal ======== -This PEP proposes the addition of a ``__qname__`` attribute to +This PEP proposes the addition of a ``__qualname__`` attribute to functions and classes. For top-level functions and classes, the -``__qname__`` attribute is equal to the ``__name__`` attribute. For -nested classed, methods, and nested functions, the ``__qname__`` +``__qualname__`` attribute is equal to the ``__name__`` attribute. For +nested classed, methods, and nested functions, the ``__qualname__`` attribute contains a dotted path leading to the object from the module -top-level. +top-level. A function's local namespace is represented in that dotted +path by a component named ````. The repr() and str() of functions and classes is modified to use -``__qname__`` rather than ``__name__``. +``__qualname__`` rather than ``__name__``. Example with nested classes --------------------------- @@ -77,13 +78,13 @@ ... class D: ... def g(): pass ... ->>> C.__qname__ +>>> C.__qualname__ 'C' ->>> C.f.__qname__ +>>> C.f.__qualname__ 'C.f' ->>> C.D.__qname__ +>>> C.D.__qualname__ 'C.D' ->>> C.D.g.__qname__ +>>> C.D.g.__qualname__ 'C.D.g' Example with nested functions @@ -93,10 +94,10 @@ ... def g(): pass ... return g ... ->>> f.__qname__ +>>> f.__qualname__ 'f' ->>> f().__qname__ -'f.g' +>>> f().__qualname__ +'f..g' Limitations @@ -107,16 +108,52 @@ namespace is not available from the outside. It will still be more helpful to the human reader than the bare ``__name__``. -As the ``__name__`` attribute, the ``__qname__`` attribute is computed +As the ``__name__`` attribute, the ``__qualname__`` attribute is computed statically and it will not automatically follow rebinding. +Discussion +========== + +Excluding the module name +------------------------- + +As ``__name__``, ``__qualname__`` doesn't include the module name. This +makes it independent of module aliasing and rebinding, and also allows to +compute it at compile time. + +Reviving unbound methods +------------------------ + +Reviving unbound methods would only solve a fraction of the problems this +PEP solves, at a higher price (an additional object type and an additional +indirection, rather than an additional attribute). + + +Naming choice +============= + +"Qualified name" is the best approximation, as a short phrase, of what the +additional attribute is about. It is not a "full name" or "fully qualified +name" since it (deliberately) does not include the module name. Calling +it a "path" would risk confusion with filesystem paths and the ``__file__`` +attribute. + +The first proposal for the attribute name was to call it ``__qname__`` but +many people (who are not aware of previous use of such jargon in e.g. the +XML specification [2]_) found it obscure and non-obvious, which is why the +slightly less short and more explicit ``__qualname__`` was finally chosen. + + References ========== .. [1] "pickle should support methods": http://bugs.python.org/issue9276 +.. [2] "QName" entry in Wikipedia: + http://en.wikipedia.org/wiki/QName + Copyright ========= diff --git a/pep0/output.py b/pep0/output.py --- a/pep0/output.py +++ b/pep0/output.py @@ -36,15 +36,15 @@ for pep in peps: # Order of 'if' statement important. Key Status values take precedence # over Type value, and vice-versa. - if pep.type_ == 'Process': - if pep.status in ("Active", "Draft"): + if pep.status == 'Draft': + open_.append(pep) + elif pep.type_ == 'Process': + if pep.status == "Active": meta.append(pep) elif pep.status in ("Withdrawn", "Rejected"): dead.append(pep) else: historical.append(pep) - elif pep.status == 'Draft': - open_.append(pep) elif pep.status == 'Deferred': deferred.append(pep) elif pep.status in ('Rejected', 'Withdrawn', @@ -169,7 +169,7 @@ print>>output, unicode(pep) print>>output print>>output - print>>output, u" Numerical Index" + print>>output, u"Numerical Index" print>>output write_column_headers(output) prev_pep = 0 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:00 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:00 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Minor_tweaks_to_PEP_405=2E?= Message-ID: http://hg.python.org/peps/rev/64170c6045e6 changeset: 4387:64170c6045e6 user: Carl Meyer date: Mon Mar 05 17:26:27 2012 -0700 summary: Minor tweaks to PEP 405. files: pep-0405.txt | 17 +++++------------ 1 files changed, 5 insertions(+), 12 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -173,7 +173,6 @@ explicit path to the venv's python binary or scripts can just as well be used), but it is convenient. -The ``venv`` module also adds a ``pysetup3`` script into each venv. In order to allow ``pysetup`` and other Python package managers to install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing @@ -202,7 +201,7 @@ Third-party packages installed into the virtual environment will have their Python modules placed in the ``site-packages`` directory, and -their executables placed in ``bin/`` or ``Scripts\``. +their executables placed in ``bin/`` or ``Scripts``. .. note:: @@ -407,9 +406,10 @@ site-packages directories. The most notable case is probably `setuptools`_ and its fork -`distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but -do use ``sys.prefix`` directly to build up a list of site directories -for pre-flight checking where ``pth`` files can usefully be placed. +`distribute`_, which mostly use ``distutils``and ``sysconfig`` APIs, +but do use ``sys.prefix`` directly to build up a list of site +directories for pre-flight checking where ``pth`` files can usefully +be placed. Otherwise, a `Google Code Search`_ turns up what appears to be a roughly even mix of usage between packages using ``sys.prefix`` to @@ -503,14 +503,6 @@ .. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 -Provide a mode that is isolated only from user site packages? -------------------------------------------------------------- - -Is there sufficient rationale for providing a mode that isolates the -venv from :pep:`370` user site packages, but not from the system-level -site-packages? - - Other Python implementations? ----------------------------- -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:01 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:01 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Update_section_on_alternative_?= =?utf8?q?Python_implementations=2E?= Message-ID: http://hg.python.org/peps/rev/a250ca2a20cb changeset: 4388:a250ca2a20cb user: Carl Meyer date: Mon Mar 12 15:04:04 2012 -0700 summary: Update section on alternative Python implementations. files: pep-0405.txt | 19 ++++++++++++------- 1 files changed, 12 insertions(+), 7 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -435,6 +435,18 @@ .. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs +Impact on other Python implementations +-------------------------------------- + +The majority of this PEP's changes occur in the standard library, which is +shared by other Python implementations and should not present any +problem. + +Other Python implementations will need to replicate the new +``sys.prefix``-finding behavior of the interpreter bootstrap, including +locating and parsing the ``pyvenv.cfg`` file, if it is present. + + Open Questions ============== @@ -503,14 +515,6 @@ .. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 -Other Python implementations? ------------------------------ - -We should get feedback from Jython, IronPython, and PyPy about whether -there's anything in this PEP that they foresee as a difficulty for -their implementation. - - Reference Implementation ======================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:02 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:02 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merge_from_upstream=2E?= Message-ID: http://hg.python.org/peps/rev/3c065d361e6f changeset: 4389:3c065d361e6f parent: 4388:a250ca2a20cb parent: 4126:ef8a076004d6 user: Carl Meyer date: Mon Mar 12 15:21:02 2012 -0700 summary: Merge from upstream. files: pep-0373.txt | 7 +++ pep-0398.txt | 2 + pep-0405.txt | 2 +- pep-0416.txt | 5 +- pep-0417.txt | 71 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 83 insertions(+), 4 deletions(-) diff --git a/pep-0373.txt b/pep-0373.txt --- a/pep-0373.txt +++ b/pep-0373.txt @@ -46,6 +46,13 @@ - 2.7 rc2 2010-06-19 - 2.7 final 2010-07-03 +Maintenance releases +==================== + +- 2.7.1 2010-11-27 +- 2.7.2 2011-07-21 +- 2.7.3rc1 2012-02-23 + Possible features for 2.7 ========================= diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -79,7 +79,9 @@ * PEP 382: Namespace Packages * PEP 395: Module Aliasing * PEP 397: Python launcher for Windows +* PEP 412: Key-Sharing Dictionary * PEP 3143: Standard daemon process library +* PEP 3144: IP Address manipulation library (Note that these are not accepted yet and even if they are, they might not be finished in time for Python 3.3.) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -8,7 +8,7 @@ Content-Type: text/x-rst Created: 13-Jun-2011 Python-Version: 3.3 -Post-History: 24-Oct-2011, 28-Oct-2011 +Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012 Abstract diff --git a/pep-0416.txt b/pep-0416.txt --- a/pep-0416.txt +++ b/pep-0416.txt @@ -20,9 +20,8 @@ ========= A frozendict is a read-only mapping: a key cannot be added nor removed, and a -key is always mapped to the same value. However, frozendict values can be -mutable (not hashable). A frozendict is hashable and so immutable if and only -if all values are hashable (immutable). +key is always mapped to the same value. However, frozendict values can be not +hashable. A frozendict is hashable if and only if all values are hashable. Use cases: diff --git a/pep-0417.txt b/pep-0417.txt new file mode 100644 --- /dev/null +++ b/pep-0417.txt @@ -0,0 +1,71 @@ +PEP: 417 +Title: Including mock in the Standard Library +Version: $Revision$ +Last-Modified: $Date$ +Author: Michael Foord +Status: Draft +Type: Standards Track +Content-Type: text/plain +Created: 12-Mar-2012 +Python-Version: 3.3 +Post-History: 12-Mar-2012 + + +Abstract + + This PEP proposes adding the mock [1]_ testing library + to the Python standard library as ``unittest.mock``. + + +Rationale + + Creating mock objects for testing is a common need in Python. + Many developers create ad-hoc mocks, as needed, in their test + suites. This is currently what we do in the Python test suite, + where a standardised mock object library would be helpful. + + There are many mock object libraries available for Python [2]_. + Of these, mock is overwhelmingly the most popular, with as many + downloads on PyPI as the other mocking libraries combined. + + An advantage of mock is that it is a mocking library and not a + framework. It provides a configurable and flexible mock object, + without being opinionated about how you write your tests. The + mock api is now well battle-tested and stable. + + mock also handles safely monkeypatching and unmonkeypatching + objects during the scope of a test. This is hard to do safely + and many developers / projects mimic this functionality + (often incorrectly). A standardised way to do this, handling + the complexity of patching in the presence of the descriptor + protocol (etc) is useful. People are asking for a "patch" [3]_ + feature to unittest. Doing this via mock.patch is preferable + to re-implementing part of this functionality in unittest. + + +Background + Addition of mock to the Python standard library was discussed + and agreed to at the Python Language Summit 2012. + + +References + + [1] `mock library on PyPI`_ + [2] http://pypi.python.org/pypi?%3Aaction=search&term=mock&submit=search + [3] http://bugs.python.org/issue11664 + + + +Copyright + + This document has been placed in the public domain. + + + +Local Variables: +mode: indented-text +indent-tabs-mode: nil +sentence-end-double-space: t +fill-column: 70 +coding: utf-8 +End: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:02 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:02 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_According_to_Ronald_Oussoren_a?= =?utf8?q?nd_Ned_Deily=2C_this_approach_should_not_require?= Message-ID: http://hg.python.org/peps/rev/aa77b5b24012 changeset: 4390:aa77b5b24012 user: Carl Meyer date: Mon Mar 12 15:25:01 2012 -0700 summary: According to Ronald Oussoren and Ned Deily, this approach should not require install_name_tool. files: pep-0405.txt | 12 ------------ 1 files changed, 0 insertions(+), 12 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -502,19 +502,6 @@ } -Need for ``install_name_tool`` on OSX? --------------------------------------- - -`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode -developer tools, to modify the copied executable on OSX. We need -input from OSX developers on whether this is actually necessary in -this PEP's implementation of virtual environments, and if so, if there -is an alternative to ``install_name_tool`` that would allow ``venv`` -to not require that Xcode is installed. - -.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 - - Reference Implementation ======================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:03 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:03 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Reference_implementation_is_ge?= =?utf8?q?tting_pretty_close=2E?= Message-ID: http://hg.python.org/peps/rev/078a621d025f changeset: 4391:078a621d025f user: Carl Meyer date: Mon Mar 12 15:27:33 2012 -0700 summary: Reference implementation is getting pretty close. files: pep-0405.txt | 12 +++++------- 1 files changed, 5 insertions(+), 7 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -505,13 +505,11 @@ Reference Implementation ======================== -The in-progress reference implementation is found in `a clone of the -CPython Mercurial repository`_. To test it, build and install it (the -virtual environment tool currently does not run from a source tree). -From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` -to create a virtual environment. - -The reference implementation (like this PEP!) is a work in progress. +The reference implementation is found in `a clone of the CPython +Mercurial repository`_. To test it, build and install it (the virtual +environment tool currently does not run from a source tree). From the +installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create +a virtual environment. .. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:04 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:04 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_open_issues_from_PyCon=2E?= Message-ID: http://hg.python.org/peps/rev/864c31a20153 changeset: 4392:864c31a20153 user: Carl Meyer date: Thu Mar 15 15:09:46 2012 -0700 summary: Add open issues from PyCon. files: pep-0405.txt | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -502,6 +502,19 @@ } +OS X Framework builds +--------------------- + +The reference implementation currently does not work on an OS X +framework build of Python. + + +tkinter +------- + +Tkinter apps currently do not work within a virtual environment. + + Reference Implementation ======================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:05 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:05 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Various_updates_to_PEP_405=2E?= Message-ID: http://hg.python.org/peps/rev/1d647da3dd16 changeset: 4393:1d647da3dd16 user: Carl Meyer date: Mon May 07 10:28:11 2012 -0600 summary: Various updates to PEP 405. files: pep-0405.txt | 67 ++++++++++++++------------------------- 1 files changed, 24 insertions(+), 43 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -225,35 +225,36 @@ ---------------------- The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs on Windows). Some -users prefer a copied binary (for greater isolation from system -changes) and some prefer a symlinked one (so that e.g. security -updates automatically propagate to virtual environments). +or symlinked Python binary (and other needed DLLs on Windows). +Symlinking is preferable where possible, because in the case of an +upgrade to the underlying Python installation, a Python executable +copied in a venv might become out-of-sync with the installed standard +library and require manual upgrade. There are some cross-platform difficulties with symlinks: * Not all Windows versions support symlinks, and even on those that do, creating them often requires administrator privileges. -* On OSX framework builds of Python, sys.executable is just a stub +* On OS X framework builds of Python, sys.executable is just a stub that executes the real Python binary. Symlinking this stub does not - work with the implementation in this PEP; it must be copied. - (Fortunately the stub is also small, so copying it is not an issue). + work; it must be copied. (Fortunately the stub is also small, and + not changed by bugfix upgrades to Python, so copying it is not an + issue). -Because of these issues, this PEP proposes to copy the Python binary -by default, to maintain cross-platform consistency in the default -behavior. +Thus, this PEP proposes to symlink the binary on all platforms except +for Windows, and OS X framework builds. A ``--symlink`` option is +available to force the use of symlinks on Windows versions that +support them, if the appropriate permissions are available. (This +option has no effect on OS X framework builds, since symlinking can +never work there, and has no advantages). -The ``pyvenv`` script accepts a ``--symlink`` option. If this option -is provided, the script will attempt to symlink instead of copy. If a -symlink fails (e.g. because they are not supported by the platform, or -additional privileges are needed), the script will warn the user and -fall back to a copy. - -On OSX framework builds, where a symlink of the executable would -succeed but create a non-functional virtual environment, the script -will fail with an error message that symlinking is not supported on -OSX framework builds. +On Windows, if ``--symlink`` is not used, this means that if the +underlying Python installation is upgraded, the Python binary and DLLs +in the venv should be updated, or there could be issues of mismatch +with the upgraded standard library. The pyvenv script accepts a +``--upgrade`` option for easily performing this upgrade on an existing +venv. API @@ -480,33 +481,12 @@ really because there's no supporting concept in ``Python/sysconfig``. -Testability and Source Build Issues ------------------------------------ - -Currently in the reference implementation, virtual environments must -be created with an installed Python, rather than a source build, as -the base installation. In order to be able to fully test the ``venv`` -module in the Python regression test suite, some anomalies in how -sysconfig data is configured in source builds will need to be removed. -For example, ``sysconfig.get_paths()`` in a source build gives -(partial output):: - - { - 'include': '/home/vinay/tools/pythonv/Include', - 'libdir': '/usr/lib ; or /usr/lib64 on a multilib system', - 'platinclude': '/home/vinay/tools/pythonv', - 'platlib': '/usr/local/lib/python3.3/site-packages', - 'platstdlib': '/usr/local/lib/python3.3', - 'purelib': '/usr/local/lib/python3.3/site-packages', - 'stdlib': '/usr/local/lib/python3.3' - } - - OS X Framework builds --------------------- -The reference implementation currently does not work on an OS X -framework build of Python. +There have been some reports that the reference implementation does +not work on an OS X framework build of Python, but it seems to work +for us. This needs further investigation. tkinter -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:06 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:06 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Automated_merge_with_http=3A//hg=2Epython=2Eorg/peps?= Message-ID: http://hg.python.org/peps/rev/51369a92dd8a changeset: 4394:51369a92dd8a parent: 4361:afe29321dca3 parent: 4393:1d647da3dd16 user: Carl Meyer date: Mon May 07 10:40:57 2012 -0600 summary: Automated merge with http://hg.python.org/peps files: pep-0405.txt | 107 ++++++++++++++++---------------------- 1 files changed, 46 insertions(+), 61 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -226,35 +226,36 @@ ---------------------- The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs on Windows). Some -users prefer a copied binary (for greater isolation from system -changes) and some prefer a symlinked one (so that e.g. security -updates automatically propagate to virtual environments). +or symlinked Python binary (and other needed DLLs on Windows). +Symlinking is preferable where possible, because in the case of an +upgrade to the underlying Python installation, a Python executable +copied in a venv might become out-of-sync with the installed standard +library and require manual upgrade. There are some cross-platform difficulties with symlinks: * Not all Windows versions support symlinks, and even on those that do, creating them often requires administrator privileges. -* On OSX framework builds of Python, sys.executable is just a stub +* On OS X framework builds of Python, sys.executable is just a stub that executes the real Python binary. Symlinking this stub does not - work with the implementation in this PEP; it must be copied. - (Fortunately the stub is also small, so copying it is not an issue). + work; it must be copied. (Fortunately the stub is also small, and + not changed by bugfix upgrades to Python, so copying it is not an + issue). -Because of these issues, this PEP proposes to copy the Python binary -by default, to maintain cross-platform consistency in the default -behavior. +Thus, this PEP proposes to symlink the binary on all platforms except +for Windows, and OS X framework builds. A ``--symlink`` option is +available to force the use of symlinks on Windows versions that +support them, if the appropriate permissions are available. (This +option has no effect on OS X framework builds, since symlinking can +never work there, and has no advantages). -The ``pyvenv`` script accepts a ``--symlink`` option. If this option -is provided, the script will attempt to symlink instead of copy. If a -symlink fails (e.g. because they are not supported by the platform, or -additional privileges are needed), the script will warn the user and -fall back to a copy. - -On OSX framework builds, where a symlink of the executable would -succeed but create a non-functional virtual environment, the script -will fail with an error message that symlinking is not supported on -OSX framework builds. +On Windows, if ``--symlink`` is not used, this means that if the +underlying Python installation is upgraded, the Python binary and DLLs +in the venv should be updated, or there could be issues of mismatch +with the upgraded standard library. The pyvenv script accepts a +``--upgrade`` option for easily performing this upgrade on an existing +venv. API @@ -436,6 +437,18 @@ .. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs +Impact on other Python implementations +-------------------------------------- + +The majority of this PEP's changes occur in the standard library, which is +shared by other Python implementations and should not present any +problem. + +Other Python implementations will need to replicate the new +``sys.prefix``-finding behavior of the interpreter bootstrap, including +locating and parsing the ``pyvenv.cfg`` file, if it is present. + + Open Questions ============== @@ -469,59 +482,28 @@ really because there's no supporting concept in ``Python/sysconfig``. -Testability and Source Build Issues ------------------------------------ +OS X Framework builds +--------------------- -Currently in the reference implementation, virtual environments must -be created with an installed Python, rather than a source build, as -the base installation. In order to be able to fully test the ``venv`` -module in the Python regression test suite, some anomalies in how -sysconfig data is configured in source builds will need to be removed. -For example, ``sysconfig.get_paths()`` in a source build gives -(partial output):: +There have been some reports that the reference implementation does +not work on an OS X framework build of Python, but it seems to work +for us. This needs further investigation. - { - 'include': '/home/vinay/tools/pythonv/Include', - 'libdir': '/usr/lib ; or /usr/lib64 on a multilib system', - 'platinclude': '/home/vinay/tools/pythonv', - 'platlib': '/usr/local/lib/python3.3/site-packages', - 'platstdlib': '/usr/local/lib/python3.3', - 'purelib': '/usr/local/lib/python3.3/site-packages', - 'stdlib': '/usr/local/lib/python3.3' - } +tkinter +------- -Need for ``install_name_tool`` on OSX? --------------------------------------- - -`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode -developer tools, to modify the copied executable on OSX. We need -input from OSX developers on whether this is actually necessary in -this PEP's implementation of virtual environments, and if so, if there -is an alternative to ``install_name_tool`` that would allow ``venv`` -to not require that Xcode is installed. - -.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 - - -Other Python implementations? ------------------------------ - -We should get feedback from Jython, IronPython, and PyPy about whether -there's anything in this PEP that they foresee as a difficulty for -their implementation. +Tkinter apps currently do not work within a virtual environment. Reference Implementation ======================== -The in-progress reference implementation is found in `a clone of the -CPython Mercurial repository`_. To test it, build and install it (the -virtual environment tool currently does not run from a source tree). -From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` -to create a virtual environment. - -The reference implementation (like this PEP!) is a work in progress. +The reference implementation is found in `a clone of the CPython +Mercurial repository`_. To test it, build and install it (the virtual +environment tool currently does not run from a source tree). From the +installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create +a virtual environment. .. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:51:09 2012 From: python-checkins at python.org (vinay.sajip) Date: Wed, 16 May 2012 15:51:09 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merged_Carl_Meyer=27s_updates=2E?= Message-ID: http://hg.python.org/peps/rev/ec08182ee57b changeset: 4395:ec08182ee57b parent: 4384:10b1bffbb69a parent: 4394:51369a92dd8a user: Vinay Sajip date: Wed May 16 14:50:52 2012 +0100 summary: Merged Carl Meyer's updates. files: pep-0405.txt | 107 ++++++++++++++++---------------------- 1 files changed, 46 insertions(+), 61 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -226,35 +226,36 @@ ---------------------- The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs on Windows). Some -users prefer a copied binary (for greater isolation from system -changes) and some prefer a symlinked one (so that e.g. security -updates automatically propagate to virtual environments). +or symlinked Python binary (and other needed DLLs on Windows). +Symlinking is preferable where possible, because in the case of an +upgrade to the underlying Python installation, a Python executable +copied in a venv might become out-of-sync with the installed standard +library and require manual upgrade. There are some cross-platform difficulties with symlinks: * Not all Windows versions support symlinks, and even on those that do, creating them often requires administrator privileges. -* On OSX framework builds of Python, sys.executable is just a stub +* On OS X framework builds of Python, sys.executable is just a stub that executes the real Python binary. Symlinking this stub does not - work with the implementation in this PEP; it must be copied. - (Fortunately the stub is also small, so copying it is not an issue). + work; it must be copied. (Fortunately the stub is also small, and + not changed by bugfix upgrades to Python, so copying it is not an + issue). -Because of these issues, this PEP proposes to copy the Python binary -by default, to maintain cross-platform consistency in the default -behavior. +Thus, this PEP proposes to symlink the binary on all platforms except +for Windows, and OS X framework builds. A ``--symlink`` option is +available to force the use of symlinks on Windows versions that +support them, if the appropriate permissions are available. (This +option has no effect on OS X framework builds, since symlinking can +never work there, and has no advantages). -The ``pyvenv`` script accepts a ``--symlink`` option. If this option -is provided, the script will attempt to symlink instead of copy. If a -symlink fails (e.g. because they are not supported by the platform, or -additional privileges are needed), the script will warn the user and -fall back to a copy. - -On OSX framework builds, where a symlink of the executable would -succeed but create a non-functional virtual environment, the script -will fail with an error message that symlinking is not supported on -OSX framework builds. +On Windows, if ``--symlink`` is not used, this means that if the +underlying Python installation is upgraded, the Python binary and DLLs +in the venv should be updated, or there could be issues of mismatch +with the upgraded standard library. The pyvenv script accepts a +``--upgrade`` option for easily performing this upgrade on an existing +venv. API @@ -436,6 +437,18 @@ .. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs +Impact on other Python implementations +-------------------------------------- + +The majority of this PEP's changes occur in the standard library, which is +shared by other Python implementations and should not present any +problem. + +Other Python implementations will need to replicate the new +``sys.prefix``-finding behavior of the interpreter bootstrap, including +locating and parsing the ``pyvenv.cfg`` file, if it is present. + + Open Questions ============== @@ -469,59 +482,28 @@ really because there's no supporting concept in ``Python/sysconfig``. -Testability and Source Build Issues ------------------------------------ +OS X Framework builds +--------------------- -Currently in the reference implementation, virtual environments must -be created with an installed Python, rather than a source build, as -the base installation. In order to be able to fully test the ``venv`` -module in the Python regression test suite, some anomalies in how -sysconfig data is configured in source builds will need to be removed. -For example, ``sysconfig.get_paths()`` in a source build gives -(partial output):: +There have been some reports that the reference implementation does +not work on an OS X framework build of Python, but it seems to work +for us. This needs further investigation. - { - 'include': '/home/vinay/tools/pythonv/Include', - 'libdir': '/usr/lib ; or /usr/lib64 on a multilib system', - 'platinclude': '/home/vinay/tools/pythonv', - 'platlib': '/usr/local/lib/python3.3/site-packages', - 'platstdlib': '/usr/local/lib/python3.3', - 'purelib': '/usr/local/lib/python3.3/site-packages', - 'stdlib': '/usr/local/lib/python3.3' - } +tkinter +------- -Need for ``install_name_tool`` on OSX? --------------------------------------- - -`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode -developer tools, to modify the copied executable on OSX. We need -input from OSX developers on whether this is actually necessary in -this PEP's implementation of virtual environments, and if so, if there -is an alternative to ``install_name_tool`` that would allow ``venv`` -to not require that Xcode is installed. - -.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168 - - -Other Python implementations? ------------------------------ - -We should get feedback from Jython, IronPython, and PyPy about whether -there's anything in this PEP that they foresee as a difficulty for -their implementation. +Tkinter apps currently do not work within a virtual environment. Reference Implementation ======================== -The in-progress reference implementation is found in `a clone of the -CPython Mercurial repository`_. To test it, build and install it (the -virtual environment tool currently does not run from a source tree). -From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` -to create a virtual environment. - -The reference implementation (like this PEP!) is a work in progress. +The reference implementation is found in `a clone of the CPython +Mercurial repository`_. To test it, build and install it (the virtual +environment tool currently does not run from a source tree). From the +installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create +a virtual environment. .. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 15:54:04 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 15:54:04 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Clarify_that_hybrid_namespace_?= =?utf8?q?packages_don=27t_have_dynamic_path_computation=2E?= Message-ID: http://hg.python.org/peps/rev/db7c0f413645 changeset: 4396:db7c0f413645 user: Eric V. Smith date: Wed May 16 09:53:58 2012 -0400 summary: Clarify that hybrid namespace packages don't have dynamic path computation. files: pep-0420.txt | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -245,8 +245,10 @@ likely not practical for all existing portions of a namespace package to be migrated to this PEP at once, ``extend_path()`` will be modified to also recognize PEP 420 namespace packages. This will allow some -portions of a namespace to be legacty portions while others are -migrated to PEP 420. +portions of a namespace to be legacy portions while others are +migrated to PEP 420. These hybrid namespace packages will not have +the dynamic path computation that normal namespace packages have, +since ``extend_path()`` never provided this functionality in the past. Packaging Implications -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 16:03:12 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Wed, 16 May 2012 16:03:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314807=3A_fix_BB_failures?= =?utf8?q?_on_Windows_-_avoid_to_to_rely_too_many_details_of_the?= Message-ID: http://hg.python.org/cpython/rev/39d24533c6b7 changeset: 76997:39d24533c6b7 parent: 76970:003204f34cc3 user: Giampaolo Rodola' date: Wed May 16 16:01:59 2012 +0200 summary: #14807: fix BB failures on Windows - avoid to to rely too many details of the mode string. files: Lib/test/test_stat.py | 33 ++++++++++++++++++++---------- 1 files changed, 22 insertions(+), 11 deletions(-) diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py --- a/Lib/test/test_stat.py +++ b/Lib/test/test_stat.py @@ -23,24 +23,35 @@ def test_mode(self): with open(TESTFN, 'w'): pass - os.chmod(TESTFN, 0o700) - self.assertEqual(get_mode(), '-rwx------') - os.chmod(TESTFN, 0o070) - self.assertEqual(get_mode(), '----rwx---') - os.chmod(TESTFN, 0o007) - self.assertEqual(get_mode(), '-------rwx') - os.chmod(TESTFN, 0o444) - self.assertEqual(get_mode(), '-r--r--r--') + if os.name == 'posix': + os.chmod(TESTFN, 0o700) + self.assertEqual(get_mode(), '-rwx------') + os.chmod(TESTFN, 0o070) + self.assertEqual(get_mode(), '----rwx---') + os.chmod(TESTFN, 0o007) + self.assertEqual(get_mode(), '-------rwx') + os.chmod(TESTFN, 0o444) + self.assertEqual(get_mode(), '-r--r--r--') + else: + os.chmod(TESTFN, 0o700) + self.assertEqual(get_mode()[:3], '-rw') def test_directory(self): os.mkdir(TESTFN) os.chmod(TESTFN, 0o700) - self.assertEqual(get_mode(), 'drwx------') + if os.name == 'posix': + self.assertEqual(get_mode(), 'drwx------') + else: + self.assertEqual(get_mode()[0], 'd') @unittest.skipUnless(hasattr(os, 'symlink'), 'os.symlink not available') def test_link(self): - os.symlink(os.getcwd(), TESTFN) - self.assertEqual(get_mode()[0], 'l') + try: + os.symlink(os.getcwd(), TESTFN) + except (OSError, NotImplementedError) as err: + raise unittest.SkipTest(str(err)) + else: + self.assertEqual(get_mode()[0], 'l') @unittest.skipUnless(hasattr(os, 'mkfifo'), 'os.mkfifo not available') def test_fifo(self): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 16:03:13 2012 From: python-checkins at python.org (giampaolo.rodola) Date: Wed, 16 May 2012 16:03:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/828be43434e8 changeset: 76998:828be43434e8 parent: 76997:39d24533c6b7 parent: 76996:8c8709b98762 user: Giampaolo Rodola' date: Wed May 16 16:03:07 2012 +0200 summary: merge heads files: .hgignore | 9 +- Doc/library/email.generator.rst | 17 +- Doc/library/http.client.rst | 15 + Lib/http/client.py | 8 + Lib/http/server.py | 10 +- Lib/test/test_bisect.py | 47 ++- Lib/test/test_pkgutil.py | 51 +++- Lib/tkinter/__init__.py | 28 +- Misc/ACKS | 2 + Misc/NEWS | 18 + Modules/_bisectmodule.c | 4 +- Modules/_csv.c | 105 ++++-- Objects/exceptions.c | 3 +- Objects/rangeobject.c | 2 +- Objects/stringlib/codecs.h | 149 +++++++++- Objects/unicodeobject.c | 297 +++++-------------- Python/freeze_importlib.py | 2 + Python/importlib.h | Bin 18 files changed, 507 insertions(+), 260 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -32,7 +32,6 @@ Modules/config.c Modules/ld_so_aix$ Parser/pgen$ -PCbuild/amd64/ ^core ^python-gdb.py ^python.exe-gdb.py @@ -56,6 +55,12 @@ PC/pythonnt_rc*.h PC/*.obj PC/*.exe +PC/*/*.user +PC/*/*.ncb +PC/*/*.suo +PC/*/Win32-temp-* +PC/*/x64-temp-* +PC/*/amd64 PCbuild/*.exe PCbuild/*.dll PCbuild/*.pdb @@ -69,6 +74,8 @@ PCbuild/*.*sdf PCbuild/Win32-temp-* PCbuild/x64-temp-* +PCbuild/amd64 +BuildLog.htm __pycache__ Modules/_testembed .coverage diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -17,10 +17,10 @@ standards-compliant way, should handle MIME and non-MIME email messages just fine, and is designed so that the transformation from flat text, to a message structure via the :class:`~email.parser.Parser` class, and back to flat text, -is idempotent (the input is identical to the output). On the other hand, using -the Generator on a :class:`~email.message.Message` constructed by program may -result in changes to the :class:`~email.message.Message` object as defaults are -filled in. +is idempotent (the input is identical to the output) [#]_. On the other hand, +using the Generator on a :class:`~email.message.Message` constructed by program +may result in changes to the :class:`~email.message.Message` object as defaults +are filled in. :class:`bytes` output can be generated using the :class:`BytesGenerator` class. If the message object structure contains non-ASCII bytes, this generator's @@ -223,3 +223,12 @@ The default value for *fmt* is ``None``, meaning :: [Non-text (%(type)s) part of message omitted, filename %(filename)s] + + +.. rubric:: Footnotes + +.. [#] This statement assumes that you use the appropriate setting for the + ``unixfrom`` argument, and that you set maxheaderlen=0 (which will + preserve whatever the input line lengths were). It is also not strictly + true, since in many cases runs of whitespace in headers are collapsed + into single blanks. The latter is a bug that will eventually be fixed. diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -339,6 +339,15 @@ | :const:`UPGRADE_REQUIRED` | ``426`` | HTTP Upgrade to TLS, | | | | :rfc:`2817`, Section 6 | +------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`PRECONDITION_REQUIRED` | ``428`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 3 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`TOO_MANY_REQUESTS` | ``429`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 4 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`REQUEST_HEADER_FIELDS_TOO_LARGE` | ``431`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 5 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ | :const:`INTERNAL_SERVER_ERROR` | ``500`` | HTTP/1.1, `RFC 2616, Section | | | | 10.5.1 | | | | `_ | @@ -369,6 +378,12 @@ | :const:`NOT_EXTENDED` | ``510`` | An HTTP Extension Framework, | | | | :rfc:`2774`, Section 7 | +------------------------------------------+---------+-----------------------------------------------------------------------+ +| :const:`NETWORK_AUTHENTICATION_REQUIRED` | ``511`` | Additional HTTP Status Codes, | +| | | :rfc:`6585`, Section 6 | ++------------------------------------------+---------+-----------------------------------------------------------------------+ + + .. versionchanged:: 3.3 + Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. .. data:: responses diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -141,6 +141,9 @@ LOCKED = 423 FAILED_DEPENDENCY = 424 UPGRADE_REQUIRED = 426 +PRECONDITION_REQUIRED = 428 +TOO_MANY_REQUESTS = 429 +REQUEST_HEADER_FIELDS_TOO_LARGE = 431 # server error INTERNAL_SERVER_ERROR = 500 @@ -151,6 +154,7 @@ HTTP_VERSION_NOT_SUPPORTED = 505 INSUFFICIENT_STORAGE = 507 NOT_EXTENDED = 510 +NETWORK_AUTHENTICATION_REQUIRED = 511 # Mapping status codes to official W3C names responses = { @@ -192,6 +196,9 @@ 415: 'Unsupported Media Type', 416: 'Requested Range Not Satisfiable', 417: 'Expectation Failed', + 428: 'Precondition Required', + 429: 'Too Many Requests', + 431: 'Request Header Fields Too Large', 500: 'Internal Server Error', 501: 'Not Implemented', @@ -199,6 +206,7 @@ 503: 'Service Unavailable', 504: 'Gateway Timeout', 505: 'HTTP Version Not Supported', + 511: 'Network Authentication Required', } # maximal amount of data to read at one time in _safe_read diff --git a/Lib/http/server.py b/Lib/http/server.py --- a/Lib/http/server.py +++ b/Lib/http/server.py @@ -573,7 +573,7 @@ # Table mapping response codes to messages; entries have the # form {code: (shortmessage, longmessage)}. - # See RFC 2616. + # See RFC 2616 and 6585. responses = { 100: ('Continue', 'Request received, please continue'), 101: ('Switching Protocols', @@ -628,6 +628,12 @@ 'Cannot satisfy request range.'), 417: ('Expectation Failed', 'Expect condition could not be satisfied.'), + 428: ('Precondition Required', + 'The origin server requires the request to be conditional.'), + 429: ('Too Many Requests', 'The user has sent too many requests ' + 'in a given amount of time ("rate limiting").'), + 431: ('Request Header Fields Too Large', 'The server is unwilling to ' + 'process the request because its header fields are too large.'), 500: ('Internal Server Error', 'Server got itself in trouble'), 501: ('Not Implemented', @@ -638,6 +644,8 @@ 504: ('Gateway Timeout', 'The gateway server did not receive a timely response'), 505: ('HTTP Version Not Supported', 'Cannot fulfill request.'), + 511: ('Network Authentication Required', + 'The client needs to authenticate to gain network access.'), } diff --git a/Lib/test/test_bisect.py b/Lib/test/test_bisect.py --- a/Lib/test/test_bisect.py +++ b/Lib/test/test_bisect.py @@ -23,6 +23,28 @@ import bisect as c_bisect +class Range(object): + """A trivial range()-like object without any integer width limitations.""" + def __init__(self, start, stop): + self.start = start + self.stop = stop + self.last_insert = None + + def __len__(self): + return self.stop - self.start + + def __getitem__(self, idx): + n = self.stop - self.start + if idx < 0: + idx += n + if idx >= n: + raise IndexError(idx) + return self.start + idx + + def insert(self, idx, item): + self.last_insert = idx, item + + class TestBisect(unittest.TestCase): module = None @@ -125,9 +147,28 @@ def test_large_range(self): # Issue 13496 mod = self.module - data = range(sys.maxsize-1) - self.assertEqual(mod.bisect_left(data, sys.maxsize-3), sys.maxsize-3) - self.assertEqual(mod.bisect_right(data, sys.maxsize-3), sys.maxsize-2) + n = sys.maxsize + data = range(n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + + def test_large_pyrange(self): + # Same as above, but without C-imposed limits on range() parameters + mod = self.module + n = sys.maxsize + data = Range(0, n-1) + self.assertEqual(mod.bisect_left(data, n-3), n-3) + self.assertEqual(mod.bisect_right(data, n-3), n-2) + self.assertEqual(mod.bisect_left(data, n-3, n-10, n), n-3) + self.assertEqual(mod.bisect_right(data, n-3, n-10, n), n-2) + x = n - 100 + mod.insort_left(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x, x)) + x = n - 200 + mod.insort_right(data, x, x - 50, x + 50) + self.assertEqual(data.last_insert, (x + 1, x)) def test_random(self, n=25): from random import randrange diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -137,8 +137,57 @@ self.assertEqual(foo.loads, 1) del sys.modules['foo'] + +class ExtendPathTests(unittest.TestCase): + def create_init(self, pkgname): + dirname = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, dirname) + sys.path.insert(0, dirname) + + pkgdir = os.path.join(dirname, pkgname) + os.mkdir(pkgdir) + with open(os.path.join(pkgdir, '__init__.py'), 'w') as fl: + fl.write('from pkgutil import extend_path\n__path__ = extend_path(__path__, __name__)\n') + + return dirname + + def create_submodule(self, dirname, pkgname, submodule_name, value): + module_name = os.path.join(dirname, pkgname, submodule_name + '.py') + with open(module_name, 'w') as fl: + print('value={}'.format(value), file=fl) + + def setUp(self): + # Create 2 directories on sys.path + self.pkgname = 'foo' + self.dirname_0 = self.create_init(self.pkgname) + self.dirname_1 = self.create_init(self.pkgname) + + def tearDown(self): + del sys.path[0] + del sys.path[0] + del sys.modules['foo'] + del sys.modules['foo.bar'] + del sys.modules['foo.baz'] + + def test_simple(self): + self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) + self.create_submodule(self.dirname_1, self.pkgname, 'baz', 1) + import foo.bar + import foo.baz + # Ensure we read the expected values + self.assertEqual(foo.bar.value, 0) + self.assertEqual(foo.baz.value, 1) + + # Ensure the path is set up correctly + self.assertEqual(sorted(foo.__path__), + sorted([os.path.join(self.dirname_0, self.pkgname), + os.path.join(self.dirname_1, self.pkgname)])) + + # XXX: test .pkg files + + def test_main(): - run_unittest(PkgutilTests, PkgutilPEP302Tests) + run_unittest(PkgutilTests, PkgutilPEP302Tests, ExtendPathTests) # this is necessary if test is run repeated (like when finding leaks) import zipimport zipimport._zip_directory_cache.clear() diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -540,12 +540,19 @@ The type keyword specifies the form in which the data is to be returned and should be an atom name such as STRING - or FILE_NAME. Type defaults to STRING. + or FILE_NAME. Type defaults to STRING, except on X11, where the default + is to try UTF8_STRING and fall back to STRING. This command is equivalent to: selection_get(CLIPBOARD) """ + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('clipboard', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('clipboard', 'get') + self._options(kw)) def clipboard_clear(self, **kw): @@ -627,8 +634,16 @@ A keyword parameter selection specifies the name of the selection and defaults to PRIMARY. A keyword parameter displayof specifies a widget on the display - to use.""" + to use. A keyword parameter type specifies the form of data to be + fetched, defaulting to STRING except on X11, where UTF8_STRING is tried + before STRING.""" if 'displayof' not in kw: kw['displayof'] = self._w + if 'type' not in kw and self._windowingsystem == 'x11': + try: + kw['type'] = 'UTF8_STRING' + return self.tk.call(('selection', 'get') + self._options(kw)) + except TclError: + del kw['type'] return self.tk.call(('selection', 'get') + self._options(kw)) def selection_handle(self, command, **kw): """Specify a function COMMAND to call if the X @@ -1043,6 +1058,15 @@ if displayof is None: return ('-displayof', self._w) return () + @property + def _windowingsystem(self): + """Internal function.""" + try: + return self._root()._windowingsystem_cached + except AttributeError: + ws = self._root()._windowingsystem_cached = \ + self.tk.call('tk', 'windowingsystem') + return ws def _options(self, cnf, kw = None): """Internal function.""" if kw: diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -919,6 +919,7 @@ Michael Schneider Peter Schneider-Kamp Arvin Schnell +Robin Schreiber Chad J. Schroeder Sam Schulenburg Stefan Schwarzer @@ -1129,6 +1130,7 @@ Hirokazu Yamamoto Ka-Ping Yee Jason Yeo +EungJun Yi Bob Yodlowski Danny Yoo George Yoshida diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. + Patch by Serhiy Storchaka. + - asdl_seq and asdl_int_seq are now Py_ssize_t sized. - Issue #14133 (PEP 415): Implement suppression of __context__ display with an @@ -31,6 +34,21 @@ Library ------- +- Issue #14829: Fix bisect and range() indexing with large indices + (>= 2 ** 32) under 64-bit Windows. + +- Issue #14732: The _csv module now uses PEP 3121 module initialization. + Patch by Robin Schreiber. + +- Issue #14809: Add HTTP status codes introduced by RFC 6585 to http.server + and http.client. Patch by EungJun Yi. + +- Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when + accessing the Tk clipboard. Modify clipboad_get() to first request type + UTF8_STRING when no specific type is requested in an X11 windowing + environment, falling back to the current default type STRING if that fails. + Original patch by Thomas Kluyver. + - Issue #14773: Fix os.fwalk() failing on dangling symlinks. - Issue #12541: Be lenient with quotes around Realm field of HTTP Basic diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -3,6 +3,7 @@ Converted to C by Dmitry Vasiliev (dima at hlabs.spb.ru). */ +#define PY_SSIZE_T_CLEAN #include "Python.h" static Py_ssize_t @@ -195,8 +196,7 @@ return NULL; } else { _Py_IDENTIFIER(insert); - - result = _PyObject_CallMethodId(list, &PyId_insert, "iO", index, item); + result = _PyObject_CallMethodId(list, &PyId_insert, "nO", index, item); if (result == NULL) return NULL; Py_DECREF(result); diff --git a/Modules/_csv.c b/Modules/_csv.c --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -16,9 +16,39 @@ #define IS_BASESTRING(o) \ PyUnicode_Check(o) -static PyObject *error_obj; /* CSV exception */ -static PyObject *dialects; /* Dialect registry */ -static long field_limit = 128 * 1024; /* max parsed field size */ +typedef struct { + PyObject *error_obj; /* CSV exception */ + PyObject *dialects; /* Dialect registry */ + long field_limit; /* max parsed field size */ +} _csvstate; + +#define _csvstate(o) ((_csvstate *)PyModule_GetState(o)) + +static int +_csv_clear(PyObject *m) +{ + Py_CLEAR(_csvstate(m)->error_obj); + Py_CLEAR(_csvstate(m)->dialects); + return 0; +} + +static int +_csv_traverse(PyObject *m, visitproc visit, void *arg) +{ + Py_VISIT(_csvstate(m)->error_obj); + Py_VISIT(_csvstate(m)->dialects); + return 0; +} + +static void +_csv_free(void *m) +{ + _csv_clear((PyObject *)m); +} + +static struct PyModuleDef _csvmodule; + +#define _csvstate_global ((_csvstate *)PyModule_GetState(PyState_FindModule(&_csvmodule))) typedef enum { START_RECORD, START_FIELD, ESCAPED_CHAR, IN_FIELD, @@ -103,10 +133,10 @@ { PyObject *dialect_obj; - dialect_obj = PyDict_GetItem(dialects, name_obj); + dialect_obj = PyDict_GetItem(_csvstate_global->dialects, name_obj); if (dialect_obj == NULL) { if (!PyErr_Occurred()) - PyErr_Format(error_obj, "unknown dialect"); + PyErr_Format(_csvstate_global->error_obj, "unknown dialect"); } else Py_INCREF(dialect_obj); @@ -544,9 +574,9 @@ static int parse_add_char(ReaderObj *self, Py_UCS4 c) { - if (self->field_len >= field_limit) { - PyErr_Format(error_obj, "field larger than field limit (%ld)", - field_limit); + if (self->field_len >= _csvstate_global->field_limit) { + PyErr_Format(_csvstate_global->error_obj, "field larger than field limit (%ld)", + _csvstate_global->field_limit); return -1; } if (self->field_len == self->field_size && !parse_grow_buff(self)) @@ -703,7 +733,7 @@ } else { /* illegal */ - PyErr_Format(error_obj, "'%c' expected after '%c'", + PyErr_Format(_csvstate_global->error_obj, "'%c' expected after '%c'", dialect->delimiter, dialect->quotechar); return -1; @@ -716,7 +746,7 @@ else if (c == '\0') self->state = START_RECORD; else { - PyErr_Format(error_obj, "new-line character seen in unquoted field - do you need to open the file in universal-newline mode?"); + PyErr_Format(_csvstate_global->error_obj, "new-line character seen in unquoted field - do you need to open the file in universal-newline mode?"); return -1; } break; @@ -755,12 +785,12 @@ if (lineobj == NULL) { /* End of input OR exception */ if (!PyErr_Occurred() && self->field_len != 0) - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "newline inside string"); return NULL; } if (!PyUnicode_Check(lineobj)) { - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "iterator should return strings, " "not %.200s " "(did you open the file in text mode?)", @@ -778,7 +808,7 @@ c = PyUnicode_READ(kind, data, pos); if (c == '\0') { Py_DECREF(lineobj); - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "line contains NULL byte"); goto err; } @@ -994,7 +1024,7 @@ } if (want_escape) { if (!dialect->escapechar) { - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "need to escape, but no escapechar set"); return -1; } @@ -1010,7 +1040,7 @@ */ if (i == 0 && quote_empty) { if (dialect->quoting == QUOTE_NONE) { - PyErr_Format(error_obj, + PyErr_Format(_csvstate_global->error_obj, "single empty field record must be quoted"); return -1; } @@ -1127,7 +1157,7 @@ PyObject *line, *result; if (!PySequence_Check(seq)) - return PyErr_Format(error_obj, "sequence expected"); + return PyErr_Format(_csvstate_global->error_obj, "sequence expected"); len = PySequence_Length(seq); if (len < 0) @@ -1353,7 +1383,7 @@ static PyObject * csv_list_dialects(PyObject *module, PyObject *args) { - return PyDict_Keys(dialects); + return PyDict_Keys(_csvstate_global->dialects); } static PyObject * @@ -1372,7 +1402,7 @@ dialect = _call_dialect(dialect_obj, kwargs); if (dialect == NULL) return NULL; - if (PyDict_SetItem(dialects, name_obj, dialect) < 0) { + if (PyDict_SetItem(_csvstate_global->dialects, name_obj, dialect) < 0) { Py_DECREF(dialect); return NULL; } @@ -1384,8 +1414,8 @@ static PyObject * csv_unregister_dialect(PyObject *module, PyObject *name_obj) { - if (PyDict_DelItem(dialects, name_obj) < 0) - return PyErr_Format(error_obj, "unknown dialect"); + if (PyDict_DelItem(_csvstate_global->dialects, name_obj) < 0) + return PyErr_Format(_csvstate_global->error_obj, "unknown dialect"); Py_INCREF(Py_None); return Py_None; } @@ -1400,7 +1430,7 @@ csv_field_size_limit(PyObject *module, PyObject *args) { PyObject *new_limit = NULL; - long old_limit = field_limit; + long old_limit = _csvstate_global->field_limit; if (!PyArg_UnpackTuple(args, "field_size_limit", 0, 1, &new_limit)) return NULL; @@ -1410,9 +1440,9 @@ "limit must be an integer"); return NULL; } - field_limit = PyLong_AsLong(new_limit); - if (field_limit == -1 && PyErr_Occurred()) { - field_limit = old_limit; + _csvstate_global->field_limit = PyLong_AsLong(new_limit); + if (_csvstate_global->field_limit == -1 && PyErr_Occurred()) { + _csvstate_global->field_limit = old_limit; return NULL; } } @@ -1551,17 +1581,16 @@ { NULL, NULL } }; - static struct PyModuleDef _csvmodule = { PyModuleDef_HEAD_INIT, "_csv", csv_module_doc, - -1, + sizeof(_csvstate), csv_methods, NULL, - NULL, - NULL, - NULL + _csv_traverse, + _csv_clear, + _csv_free }; PyMODINIT_FUNC @@ -1589,11 +1618,16 @@ MODULE_VERSION) == -1) return NULL; + /* Set the field limit */ + _csvstate(module)->field_limit = 128 * 1024; + /* Do I still need to add this var to the Module Dict? */ + /* Add _dialects dictionary */ - dialects = PyDict_New(); - if (dialects == NULL) + _csvstate(module)->dialects = PyDict_New(); + if (_csvstate(module)->dialects == NULL) return NULL; - if (PyModule_AddObject(module, "_dialects", dialects)) + Py_INCREF(_csvstate(module)->dialects); + if (PyModule_AddObject(module, "_dialects", _csvstate(module)->dialects)) return NULL; /* Add quote styles into dictionary */ @@ -1609,9 +1643,10 @@ return NULL; /* Add the CSV exception object to the module. */ - error_obj = PyErr_NewException("_csv.Error", NULL, NULL); - if (error_obj == NULL) + _csvstate(module)->error_obj = PyErr_NewException("_csv.Error", NULL, NULL); + if (_csvstate(module)->error_obj == NULL) return NULL; - PyModule_AddObject(module, "Error", error_obj); + Py_INCREF(_csvstate(module)->error_obj); + PyModule_AddObject(module, "Error", _csvstate(module)->error_obj); return module; } diff --git a/Objects/exceptions.c b/Objects/exceptions.c --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -349,7 +349,8 @@ static struct PyMemberDef BaseException_members[] = { {"__suppress_context__", T_BOOL, - offsetof(PyBaseExceptionObject, suppress_context)} + offsetof(PyBaseExceptionObject, suppress_context)}, + {NULL} }; diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c --- a/Objects/rangeobject.c +++ b/Objects/rangeobject.c @@ -308,7 +308,7 @@ static PyObject * range_item(rangeobject *r, Py_ssize_t i) { - PyObject *res, *arg = PyLong_FromLong(i); + PyObject *res, *arg = PyLong_FromSsize_t(i); if (!arg) { return NULL; } diff --git a/Objects/stringlib/codecs.h b/Objects/stringlib/codecs.h --- a/Objects/stringlib/codecs.h +++ b/Objects/stringlib/codecs.h @@ -215,7 +215,6 @@ goto Return; } -#undef LONG_PTR_MASK #undef ASCII_CHAR_MASK @@ -415,4 +414,152 @@ #undef MAX_SHORT_UNICHARS } +/* The pattern for constructing UCS2-repeated masks. */ +#if SIZEOF_LONG == 8 +# define UCS2_REPEAT_MASK 0x0001000100010001ul +#elif SIZEOF_LONG == 4 +# define UCS2_REPEAT_MASK 0x00010001ul +#else +# error C 'long' size should be either 4 or 8! +#endif + +/* The mask for fast checking. */ +#if STRINGLIB_SIZEOF_CHAR == 1 +/* The mask for fast checking of whether a C 'long' contains a + non-ASCII or non-Latin1 UTF16-encoded characters. */ +# define FAST_CHAR_MASK (UCS2_REPEAT_MASK * (0xFFFFu & ~STRINGLIB_MAX_CHAR)) +#else +/* The mask for fast checking of whether a C 'long' may contain + UTF16-encoded surrogate characters. This is an efficient heuristic, + assuming that non-surrogate characters with a code point >= 0x8000 are + rare in most input. +*/ +# define FAST_CHAR_MASK (UCS2_REPEAT_MASK * 0x8000u) +#endif +/* The mask for fast byte-swapping. */ +#define STRIPPED_MASK (UCS2_REPEAT_MASK * 0x00FFu) +/* Swap bytes. */ +#define SWAB(value) ((((value) >> 8) & STRIPPED_MASK) | \ + (((value) & STRIPPED_MASK) << 8)) + +Py_LOCAL_INLINE(Py_UCS4) +STRINGLIB(utf16_decode)(const unsigned char **inptr, const unsigned char *e, + STRINGLIB_CHAR *dest, Py_ssize_t *outpos, + int native_ordering) +{ + Py_UCS4 ch; + const unsigned char *aligned_end = + (const unsigned char *) ((size_t) e & ~LONG_PTR_MASK); + const unsigned char *q = *inptr; + STRINGLIB_CHAR *p = dest + *outpos; + /* Offsets from q for retrieving byte pairs in the right order. */ +#ifdef BYTEORDER_IS_LITTLE_ENDIAN + int ihi = !!native_ordering, ilo = !native_ordering; +#else + int ihi = !native_ordering, ilo = !!native_ordering; +#endif + --e; + + while (q < e) { + Py_UCS4 ch2; + /* First check for possible aligned read of a C 'long'. Unaligned + reads are more expensive, better to defer to another iteration. */ + if (!((size_t) q & LONG_PTR_MASK)) { + /* Fast path for runs of in-range non-surrogate chars. */ + register const unsigned char *_q = q; + while (_q < aligned_end) { + unsigned long block = * (unsigned long *) _q; + if (native_ordering) { + /* Can use buffer directly */ + if (block & FAST_CHAR_MASK) + break; + } + else { + /* Need to byte-swap */ + if (block & SWAB(FAST_CHAR_MASK)) + break; +#if STRINGLIB_SIZEOF_CHAR == 1 + block >>= 8; +#else + block = SWAB(block); +#endif + } +#ifdef BYTEORDER_IS_LITTLE_ENDIAN +# if SIZEOF_LONG == 4 + p[0] = (STRINGLIB_CHAR)(block & 0xFFFFu); + p[1] = (STRINGLIB_CHAR)(block >> 16); +# elif SIZEOF_LONG == 8 + p[0] = (STRINGLIB_CHAR)(block & 0xFFFFu); + p[1] = (STRINGLIB_CHAR)((block >> 16) & 0xFFFFu); + p[2] = (STRINGLIB_CHAR)((block >> 32) & 0xFFFFu); + p[3] = (STRINGLIB_CHAR)(block >> 48); +# endif +#else +# if SIZEOF_LONG == 4 + p[0] = (STRINGLIB_CHAR)(block >> 16); + p[1] = (STRINGLIB_CHAR)(block & 0xFFFFu); +# elif SIZEOF_LONG == 8 + p[0] = (STRINGLIB_CHAR)(block >> 48); + p[1] = (STRINGLIB_CHAR)((block >> 32) & 0xFFFFu); + p[2] = (STRINGLIB_CHAR)((block >> 16) & 0xFFFFu); + p[3] = (STRINGLIB_CHAR)(block & 0xFFFFu); +# endif +#endif + _q += SIZEOF_LONG; + p += SIZEOF_LONG / 2; + } + q = _q; + if (q >= e) + break; + } + + ch = (q[ihi] << 8) | q[ilo]; + q += 2; + if (!Py_UNICODE_IS_SURROGATE(ch)) { +#if STRINGLIB_SIZEOF_CHAR < 2 + if (ch > STRINGLIB_MAX_CHAR) + /* Out-of-range */ + goto Return; +#endif + *p++ = (STRINGLIB_CHAR)ch; + continue; + } + + /* UTF-16 code pair: */ + if (q >= e) + goto UnexpectedEnd; + if (!Py_UNICODE_IS_HIGH_SURROGATE(ch)) + goto IllegalEncoding; + ch2 = (q[ihi] << 8) | q[ilo]; + q += 2; + if (!Py_UNICODE_IS_LOW_SURROGATE(ch2)) + goto IllegalSurrogate; + ch = Py_UNICODE_JOIN_SURROGATES(ch, ch2); +#if STRINGLIB_SIZEOF_CHAR < 4 + /* Out-of-range */ + goto Return; +#else + *p++ = (STRINGLIB_CHAR)ch; +#endif + } + ch = 0; +Return: + *inptr = q; + *outpos = p - dest; + return ch; +UnexpectedEnd: + ch = 1; + goto Return; +IllegalEncoding: + ch = 2; + goto Return; +IllegalSurrogate: + ch = 3; + goto Return; +} +#undef UCS2_REPEAT_MASK +#undef FAST_CHAR_MASK +#undef STRIPPED_MASK +#undef SWAB +#undef LONG_PTR_MASK #endif /* STRINGLIB_IS_UNICODE */ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5195,25 +5195,6 @@ return PyUnicode_DecodeUTF16Stateful(s, size, errors, byteorder, NULL); } -/* Two masks for fast checking of whether a C 'long' may contain - UTF16-encoded surrogate characters. This is an efficient heuristic, - assuming that non-surrogate characters with a code point >= 0x8000 are - rare in most input. - FAST_CHAR_MASK is used when the input is in native byte ordering, - SWAPPED_FAST_CHAR_MASK when the input is in byteswapped ordering. -*/ -#if (SIZEOF_LONG == 8) -# define FAST_CHAR_MASK 0x8000800080008000L -# define SWAPPED_FAST_CHAR_MASK 0x0080008000800080L -# define STRIPPED_MASK 0x00FF00FF00FF00FFL -#elif (SIZEOF_LONG == 4) -# define FAST_CHAR_MASK 0x80008000L -# define SWAPPED_FAST_CHAR_MASK 0x00800080L -# define STRIPPED_MASK 0x00FF00FFL -#else -# error C 'long' size should be either 4 or 8! -#endif - PyObject * PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, @@ -5226,30 +5207,15 @@ Py_ssize_t endinpos; Py_ssize_t outpos; PyObject *unicode; - const unsigned char *q, *e, *aligned_end; + const unsigned char *q, *e; int bo = 0; /* assume native ordering by default */ - int native_ordering = 0; + int native_ordering; const char *errmsg = ""; - /* Offsets from q for retrieving byte pairs in the right order. */ -#ifdef BYTEORDER_IS_LITTLE_ENDIAN - int ihi = 1, ilo = 0; -#else - int ihi = 0, ilo = 1; -#endif PyObject *errorHandler = NULL; PyObject *exc = NULL; - /* Note: size will always be longer than the resulting Unicode - character count */ - unicode = PyUnicode_New(size, 127); - if (!unicode) - return NULL; - if (size == 0) - return unicode; - outpos = 0; - q = (unsigned char *)s; - e = q + size - 1; + e = q + size; if (byteorder) bo = *byteorder; @@ -5258,155 +5224,98 @@ byte order setting accordingly. In native mode, the leading BOM mark is skipped, in all other modes, it is copied to the output stream as-is (giving a ZWNBSP character). */ - if (bo == 0) { - if (size >= 2) { - const Py_UCS4 bom = (q[ihi] << 8) | q[ilo]; + if (bo == 0 && size >= 2) { + const Py_UCS4 bom = (q[1] << 8) | q[0]; + if (bom == 0xFEFF) { + q += 2; + bo = -1; + } + else if (bom == 0xFFFE) { + q += 2; + bo = 1; + } + if (byteorder) + *byteorder = bo; + } + + if (q == e) { + if (consumed) + *consumed = size; + Py_INCREF(unicode_empty); + return unicode_empty; + } + #ifdef BYTEORDER_IS_LITTLE_ENDIAN - if (bom == 0xFEFF) { - q += 2; - bo = -1; - } - else if (bom == 0xFFFE) { - q += 2; - bo = 1; - } + native_ordering = bo <= 0; #else - if (bom == 0xFEFF) { - q += 2; - bo = 1; - } - else if (bom == 0xFFFE) { - q += 2; - bo = -1; - } -#endif - } - } - - if (bo == -1) { - /* force LE */ - ihi = 1; - ilo = 0; - } - else if (bo == 1) { - /* force BE */ - ihi = 0; - ilo = 1; - } -#ifdef BYTEORDER_IS_LITTLE_ENDIAN - native_ordering = ilo < ihi; -#else - native_ordering = ilo > ihi; -#endif - - aligned_end = (const unsigned char *) ((size_t) e & ~LONG_PTR_MASK); - while (q < e) { - Py_UCS4 ch; - /* First check for possible aligned read of a C 'long'. Unaligned - reads are more expensive, better to defer to another iteration. */ - if (!((size_t) q & LONG_PTR_MASK)) { - /* Fast path for runs of non-surrogate chars. */ - register const unsigned char *_q = q; + native_ordering = bo >= 0; +#endif + + /* Note: size will always be longer than the resulting Unicode + character count */ + unicode = PyUnicode_New((e - q + 1) / 2, 127); + if (!unicode) + return NULL; + + outpos = 0; + while (1) { + Py_UCS4 ch = 0; + if (e - q >= 2) { int kind = PyUnicode_KIND(unicode); - void *data = PyUnicode_DATA(unicode); - while (_q < aligned_end) { - unsigned long block = * (unsigned long *) _q; - Py_UCS4 maxch; - if (native_ordering) { - /* Can use buffer directly */ - if (block & FAST_CHAR_MASK) - break; - } - else { - /* Need to byte-swap */ - if (block & SWAPPED_FAST_CHAR_MASK) - break; - block = ((block >> 8) & STRIPPED_MASK) | - ((block & STRIPPED_MASK) << 8); - } - maxch = (Py_UCS2)(block & 0xFFFF); -#if SIZEOF_LONG == 8 - ch = (Py_UCS2)((block >> 16) & 0xFFFF); - maxch = MAX_MAXCHAR(maxch, ch); - ch = (Py_UCS2)((block >> 32) & 0xFFFF); - maxch = MAX_MAXCHAR(maxch, ch); - ch = (Py_UCS2)(block >> 48); - maxch = MAX_MAXCHAR(maxch, ch); -#else - ch = (Py_UCS2)(block >> 16); - maxch = MAX_MAXCHAR(maxch, ch); -#endif - if (maxch > PyUnicode_MAX_CHAR_VALUE(unicode)) { - if (unicode_widen(&unicode, outpos, maxch) < 0) - goto onError; - kind = PyUnicode_KIND(unicode); - data = PyUnicode_DATA(unicode); - } -#ifdef BYTEORDER_IS_LITTLE_ENDIAN - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block & 0xFFFF)); -#if SIZEOF_LONG == 8 - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 16) & 0xFFFF)); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 32) & 0xFFFF)); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 48))); -#else - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block >> 16)); -#endif -#else -#if SIZEOF_LONG == 8 - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 48))); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 32) & 0xFFFF)); - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)((block >> 16) & 0xFFFF)); -#else - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block >> 16)); -#endif - PyUnicode_WRITE(kind, data, outpos++, (Py_UCS2)(block & 0xFFFF)); -#endif - _q += SIZEOF_LONG; - } - q = _q; - if (q >= e) - break; - } - ch = (q[ihi] << 8) | q[ilo]; - - q += 2; - - if (!Py_UNICODE_IS_SURROGATE(ch)) { + if (kind == PyUnicode_1BYTE_KIND) { + if (PyUnicode_IS_ASCII(unicode)) + ch = asciilib_utf16_decode(&q, e, + PyUnicode_1BYTE_DATA(unicode), &outpos, + native_ordering); + else + ch = ucs1lib_utf16_decode(&q, e, + PyUnicode_1BYTE_DATA(unicode), &outpos, + native_ordering); + } else if (kind == PyUnicode_2BYTE_KIND) { + ch = ucs2lib_utf16_decode(&q, e, + PyUnicode_2BYTE_DATA(unicode), &outpos, + native_ordering); + } else { + assert(kind == PyUnicode_4BYTE_KIND); + ch = ucs4lib_utf16_decode(&q, e, + PyUnicode_4BYTE_DATA(unicode), &outpos, + native_ordering); + } + } + + switch (ch) + { + case 0: + /* remaining byte at the end? (size should be even) */ + if (q == e || consumed) + goto End; + errmsg = "truncated data"; + startinpos = ((const char *)q) - starts; + endinpos = ((const char *)e) - starts; + break; + /* The remaining input chars are ignored if the callback + chooses to skip the input */ + case 1: + errmsg = "unexpected end of data"; + startinpos = ((const char *)q) - 2 - starts; + endinpos = ((const char *)e) - starts; + break; + case 2: + errmsg = "illegal encoding"; + startinpos = ((const char *)q) - 2 - starts; + endinpos = startinpos + 2; + break; + case 3: + errmsg = "illegal UTF-16 surrogate"; + startinpos = ((const char *)q) - 4 - starts; + endinpos = startinpos + 2; + break; + default: if (unicode_putchar(&unicode, &outpos, ch) < 0) goto onError; continue; } - /* UTF-16 code pair: */ - if (q > e) { - errmsg = "unexpected end of data"; - startinpos = (((const char *)q) - 2) - starts; - endinpos = ((const char *)e) + 1 - starts; - goto utf16Error; - } - if (Py_UNICODE_IS_HIGH_SURROGATE(ch)) { - Py_UCS4 ch2 = (q[ihi] << 8) | q[ilo]; - q += 2; - if (Py_UNICODE_IS_LOW_SURROGATE(ch2)) { - if (unicode_putchar(&unicode, &outpos, - Py_UNICODE_JOIN_SURROGATES(ch, ch2)) < 0) - goto onError; - continue; - } - else { - errmsg = "illegal UTF-16 surrogate"; - startinpos = (((const char *)q)-4)-starts; - endinpos = startinpos+2; - goto utf16Error; - } - - } - errmsg = "illegal encoding"; - startinpos = (((const char *)q)-2)-starts; - endinpos = startinpos+2; - /* Fall through to report the error */ - - utf16Error: if (unicode_decode_call_errorhandler( errors, &errorHandler, @@ -5421,33 +5330,8 @@ &outpos)) goto onError; } - /* remaining byte at the end? (size should be even) */ - if (e == q) { - if (!consumed) { - errmsg = "truncated data"; - startinpos = ((const char *)q) - starts; - endinpos = ((const char *)e) + 1 - starts; - if (unicode_decode_call_errorhandler( - errors, - &errorHandler, - "utf16", errmsg, - &starts, - (const char **)&e, - &startinpos, - &endinpos, - &exc, - (const char **)&q, - &unicode, - &outpos)) - goto onError; - /* The remaining input chars are ignored if the callback - chooses to skip the input */ - } - } - - if (byteorder) - *byteorder = bo; - + +End: if (consumed) *consumed = (const char *)q-starts; @@ -5466,9 +5350,6 @@ return NULL; } -#undef FAST_CHAR_MASK -#undef SWAPPED_FAST_CHAR_MASK - PyObject * _PyUnicode_EncodeUTF16(PyObject *str, const char *errors, diff --git a/Python/freeze_importlib.py b/Python/freeze_importlib.py --- a/Python/freeze_importlib.py +++ b/Python/freeze_importlib.py @@ -25,6 +25,8 @@ with open(output_path, 'w', encoding='utf-8') as output_file: output_file.write('\n'.join(lines)) output_file.write('/* Mercurial binary marker: \x00 */') + # Avoid a compiler warning for lack of EOL + output_file.write('\n') if __name__ == '__main__': diff --git a/Python/importlib.h b/Python/importlib.h index 0beeb595dbc38d821fb4f6de0981347f3983420a..cf5619a6c4b0587815b87145eae5bf212ec7e5f1 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 16:17:47 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 16 May 2012 16:17:47 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Moved_=27portion=27_next_to_?= =?utf8?q?=27legacy_portion=27_in_terminology_section=2E?= Message-ID: http://hg.python.org/peps/rev/3cb12850bd6d changeset: 4397:3cb12850bd6d user: Eric V. Smith date: Wed May 16 10:17:39 2012 -0400 summary: Moved 'portion' next to 'legacy portion' in terminology section. files: pep-0420.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -34,10 +34,10 @@ * "vendor package" refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). + * "regular package" refers to packages as they are implemented in + Python 3.2 and earlier. * "portion" refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. - * "regular package" refers to packages as they are implemented in - Python 3.2 and earlier. * "legacy portion" refers to a portion that uses ``__path__`` manipulation in order to implement namespace packages. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 16 16:44:14 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 16:44:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314693=3A_Under_non?= =?utf8?q?-Windows_platforms=2C_hashlib=27s_fallback_modules_are?= Message-ID: http://hg.python.org/cpython/rev/0a1d7be10946 changeset: 76999:0a1d7be10946 user: Antoine Pitrou date: Wed May 16 16:41:26 2012 +0200 summary: Issue #14693: Under non-Windows platforms, hashlib's fallback modules are always compiled, even if OpenSSL is present at build time. files: Lib/test/test_hashlib.py | 4 +++- Misc/NEWS | 3 +++ setup.py | 25 +++++++++++-------------- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py --- a/Lib/test/test_hashlib.py +++ b/Lib/test/test_hashlib.py @@ -9,6 +9,7 @@ import array import hashlib import itertools +import os import sys try: import threading @@ -37,7 +38,8 @@ 'sha224', 'SHA224', 'sha256', 'SHA256', 'sha384', 'SHA384', 'sha512', 'SHA512' ) - _warn_on_extension_import = COMPILED_WITH_PYDEBUG + # Issue #14693: fallback modules are always compiled under POSIX + _warn_on_extension_import = os.name == 'posix' or COMPILED_WITH_PYDEBUG def _conditional_import_module(self, module_name): """Import a module and return a reference to it or None on failure.""" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -125,6 +125,9 @@ Build ----- +- Issue #14693: Under non-Windows platforms, hashlib's fallback modules are + always compiled, even if OpenSSL is present at build time. + - Issue #13210: Windows build now uses VS2010, ported from VS2008. diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -749,20 +749,17 @@ openssl_ver) missing.append('_hashlib') - min_sha2_openssl_ver = 0x00908000 - if COMPILED_WITH_PYDEBUG or openssl_ver < min_sha2_openssl_ver: - # OpenSSL doesn't do these until 0.9.8 so we'll bring our own hash - exts.append( Extension('_sha256', ['sha256module.c'], - depends=['hashlib.h']) ) - exts.append( Extension('_sha512', ['sha512module.c'], - depends=['hashlib.h']) ) - - if COMPILED_WITH_PYDEBUG or not have_usable_openssl: - # no openssl at all, use our own md5 and sha1 - exts.append( Extension('_md5', ['md5module.c'], - depends=['hashlib.h']) ) - exts.append( Extension('_sha1', ['sha1module.c'], - depends=['hashlib.h']) ) + # We always compile these even when OpenSSL is available (issue #14693). + # It's harmless and the object code is tiny (40-50 KB per module, + # only loaded when actually used). + exts.append( Extension('_sha256', ['sha256module.c'], + depends=['hashlib.h']) ) + exts.append( Extension('_sha512', ['sha512module.c'], + depends=['hashlib.h']) ) + exts.append( Extension('_md5', ['md5module.c'], + depends=['hashlib.h']) ) + exts.append( Extension('_sha1', ['sha1module.c'], + depends=['hashlib.h']) ) # Modules that provide persistent dictionary-like semantics. You will # probably want to arrange for at least one of them to be available on -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 19:10:35 2012 From: python-checkins at python.org (hynek.schlawack) Date: Wed, 16 May 2012 19:10:35 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_=2314692_Fix_js?= =?utf8?q?on_docs_to_reflect_changes_in_json=2Eload?= Message-ID: http://hg.python.org/cpython/rev/30d16d1e5175 changeset: 77000:30d16d1e5175 branch: 2.7 parent: 76994:e957b93571a8 user: Hynek Schlawack date: Wed May 16 18:02:54 2012 +0200 summary: #14692 Fix json docs to reflect changes in json.load The behaviour of the parse_constant callback changed insofar that 'null', 'true', 'false' don't trigger its call anymore. Patch by Serhiy Storchaka files: Doc/library/json.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -217,8 +217,8 @@ (e.g. :class:`float`). *parse_constant*, if specified, will be called with one of the following - strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``, ``'null'``, ``'true'``, - ``'false'``. This can be used to raise an exception if invalid JSON numbers + strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. + This can be used to raise an exception if invalid JSON numbers are encountered. To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 19:10:36 2012 From: python-checkins at python.org (hynek.schlawack) Date: Wed, 16 May 2012 19:10:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_=2314692_Fix_js?= =?utf8?q?on_docs_to_reflect_changes_in_json=2Eload?= Message-ID: http://hg.python.org/cpython/rev/4f27c4dc34ed changeset: 77001:4f27c4dc34ed branch: 3.2 parent: 76995:13900edf13be user: Hynek Schlawack date: Wed May 16 19:01:04 2012 +0200 summary: #14692 Fix json docs to reflect changes in json.load The behaviour of the parse_constant callback changed insofar that 'null', 'true', 'false' don't trigger its call anymore. Patch by Serhiy Storchaka files: Doc/library/json.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -209,8 +209,8 @@ (e.g. :class:`float`). *parse_constant*, if specified, will be called with one of the following - strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``, ``'null'``, ``'true'``, - ``'false'``. This can be used to raise an exception if invalid JSON numbers + strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. + This can be used to raise an exception if invalid JSON numbers are encountered. To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 19:10:36 2012 From: python-checkins at python.org (hynek.schlawack) Date: Wed, 16 May 2012 19:10:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314692_Fix_json_docs_to_reflect_changes_in_json=2Eload?= Message-ID: http://hg.python.org/cpython/rev/0f6a6f59b002 changeset: 77002:0f6a6f59b002 parent: 76999:0a1d7be10946 parent: 77001:4f27c4dc34ed user: Hynek Schlawack date: Wed May 16 19:08:36 2012 +0200 summary: #14692 Fix json docs to reflect changes in json.load The behaviour of the parse_constant callback changed insofar that 'null', 'true', 'false' don't trigger its call anymore. Patch by Serhiy Storchaka files: Doc/library/json.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -209,8 +209,8 @@ (e.g. :class:`float`). *parse_constant*, if specified, will be called with one of the following - strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``, ``'null'``, ``'true'``, - ``'false'``. This can be used to raise an exception if invalid JSON numbers + strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. + This can be used to raise an exception if invalid JSON numbers are encountered. To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 20:01:48 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 16 May 2012 20:01:48 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0ODMyOiAnZmly?= =?utf8?q?st=27_now_really_refers_to_first_arg_in_unittest_assertItemsEqua?= =?utf8?q?l?= Message-ID: http://hg.python.org/cpython/rev/ff52583a5576 changeset: 77003:ff52583a5576 branch: 2.7 parent: 77000:30d16d1e5175 user: R David Murray date: Wed May 16 14:01:03 2012 -0400 summary: #14832: 'first' now really refers to first arg in unittest assertItemsEqual This appears to have been a mixup introduced when we switched from 'expected/actual' to 'first/second'. The problem doesn't exist in the corresponding assertCountEqual method in Python3. files: Lib/unittest/case.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -872,7 +872,7 @@ - [0, 1, 1] and [1, 0, 1] compare equal. - [0, 0, 1] and [0, 1] compare unequal. """ - first_seq, second_seq = list(actual_seq), list(expected_seq) + first_seq, second_seq = list(expected_seq), list(actual_seq) with warnings.catch_warnings(): if sys.py3kwarning: # Silence Py3k warning raised during the sorting diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #14832: fixed the order of the argument references in the error + message produced by unittest's assertItemsEqual. + - Issue #14829: Fix bisect issues under 64-bit Windows. - Issue #14777: tkinter may return undecoded UTF-8 bytes as a string when -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 20:11:57 2012 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 May 2012 20:11:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Changes_in_=5Fmpd=5Fqexp=28?= =?utf8?b?KTo=?= Message-ID: http://hg.python.org/cpython/rev/db27a563fa9b changeset: 77004:db27a563fa9b parent: 77002:0f6a6f59b002 user: Stefan Krah date: Wed May 16 20:10:21 2012 +0200 summary: Changes in _mpd_qexp(): ----------------------- 1) Reduce the number of iterations in the Horner scheme for operands with a negative adjusted exponent. Previously the number was overestimated quite generously. 2) The function _mpd_get_exp_iterations() now has an ACL2 proof and is rewritten accordingly. 3) The proof relies on abs(op) > 9 * 10**(-prec-1), so operands without that property are now handled by the new function _mpd_qexp_check_one(). 4) The error analysis for the evaluation of the truncated Taylor series in Hull&Abrham's paper relies on the fact that the reduced operand 'r' has fewer than context.prec digits. Since the operands may have more than context.prec digits, a new ACL2 proof covers the case that r.digits > context.prec. To facilitate the proof, the Horner step now uses fma instead of rounding twice in multiply/add. Changes in mpd_qexp(): ---------------------- 1) Fix a bound in the correct rounding loop that was too optimistic. In practice results were always correctly rounded, because it is unlikely that the error in _mpd_qexp() ever reaches the theoretical maximum. files: Modules/_decimal/libmpdec/mpdecimal.c | 173 +++++++++---- 1 files changed, 122 insertions(+), 51 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3878,53 +3878,97 @@ } #endif -#if defined(_MSC_VER) - /* conversion from 'double' to 'mpd_ssize_t', possible loss of data */ - #pragma warning(disable:4244) -#endif +/* Pad the result with trailing zeros if it has fewer digits than prec. */ +static void +_mpd_zeropad(mpd_t *result, const mpd_context_t *ctx, uint32_t *status) +{ + if (!mpd_isspecial(result) && !mpd_iszero(result) && + result->digits < ctx->prec) { + mpd_ssize_t shift = ctx->prec - result->digits; + mpd_qshiftl(result, result, shift, status); + result->exp -= shift; + } +} + +/* Check if the result is guaranteed to be one. */ +static int +_mpd_qexp_check_one(mpd_t *result, const mpd_t *a, const mpd_context_t *ctx, + uint32_t *status) +{ + MPD_NEW_CONST(lim,0,-(ctx->prec+1),1,1,1,9); + MPD_NEW_SHARED(aa, a); + + mpd_set_positive(&aa); + + /* abs(a) <= 9 * 10**(-prec-1) */ + if (_mpd_cmp(&aa, &lim) <= 0) { + _settriple(result, 0, 1, 0); + _mpd_zeropad(result, ctx, status); + *status = MPD_Rounded|MPD_Inexact; + return 1; + } + + return 0; +} + /* * Get the number of iterations for the Horner scheme in _mpd_qexp(). */ static inline mpd_ssize_t -_mpd_get_exp_iterations(const mpd_t *a, mpd_ssize_t prec) -{ - mpd_uint_t dummy; - mpd_uint_t msdigits; - double f; - - /* 9 is MPD_RDIGITS for 32 bit platforms */ - _mpd_get_msdigits(&dummy, &msdigits, a, 9); - f = ((double)msdigits + 1) / mpd_pow10[mpd_word_digits(msdigits)]; +_mpd_get_exp_iterations(const mpd_t *r, mpd_ssize_t p) +{ + mpd_ssize_t log10pbyr; /* lower bound for log10(p / abs(r)) */ + mpd_ssize_t n; + + assert(p >= 10); + assert(!mpd_iszero(r)); + assert(-p < mpd_adjexp(r) && mpd_adjexp(r) <= -1); #ifdef CONFIG_64 - #ifdef USE_80BIT_LONG_DOUBLE - return ceill((1.435*(long double)prec - 1.182) - / log10l((long double)prec/f)); - #else - /* prec > floor((1ULL<<53) / 1.435) */ - if (prec > 6276793905742851LL) { + if (p > (mpd_ssize_t)(1ULL<<52)) { return MPD_SSIZE_MAX; } - return ceil((1.435*(double)prec - 1.182) / log10((double)prec/f)); - #endif -#else /* CONFIG_32 */ - return ceil((1.435*(double)prec - 1.182) / log10((double)prec/f)); - #if defined(_MSC_VER) - #pragma warning(default:4244) - #endif #endif + + /* + * Lower bound for log10(p / abs(r)): adjexp(p) - (adjexp(r) + 1) + * At this point (for CONFIG_64, CONFIG_32 is not problematic): + * 1) 10 <= p <= 2**52 + * 2) -p < adjexp(r) <= -1 + * 3) 1 <= log10pbyr <= 2**52 + 14 + */ + log10pbyr = (mpd_word_digits(p)-1) - (mpd_adjexp(r)+1); + + /* + * The numerator in the paper is 1.435 * p - 1.182, calculated + * exactly. We compensate for rounding errors by using 1.43503. + * ACL2 proofs: + * 1) exp-iter-approx-lower-bound: The term below evaluated + * in 53-bit floating point arithmetic is greater than or + * equal to the exact term used in the paper. + * 2) exp-iter-approx-upper-bound: The term below is less than + * or equal to 3/2 * p <= 3/2 * 2**52. + */ + n = (mpd_ssize_t)ceil((1.43503*(double)p - 1.182) / (double)log10pbyr); + return n >= 3 ? n : 3; } /* - * Internal function, specials have been dealt with. + * Internal function, specials have been dealt with. The result has a + * relative error of less than 0.5 * 10**(-ctx->prec). * * The algorithm is from Hull&Abrham, Variable Precision Exponential Function, * ACM Transactions on Mathematical Software, Vol. 12, No. 2, June 1986. * * Main differences: * - * - The number of iterations for the Horner scheme is calculated using the - * C log10() function. + * - The number of iterations for the Horner scheme is calculated using + * 53-bit floating point arithmetic. + * + * - In the error analysis for ER (relative error accumulated in the + * evaluation of the truncated series) the reduced operand r may + * have any number of digits. + * ACL2 proof: exponent-relative-error * * - The analysis for early abortion has been adapted for the mpd_t * ranges. @@ -3941,18 +3985,23 @@ assert(!mpd_isspecial(a)); + if (mpd_iszerocoeff(a)) { + _settriple(result, MPD_POS, 1, 0); + return; + } + /* - * We are calculating e^x = e^(r*10^t) = (e^r)^(10^t), where r < 1 and t >= 0. + * We are calculating e^x = e^(r*10^t) = (e^r)^(10^t), where abs(r) < 1 and t >= 0. * * If t > 0, we have: * - * (1) 0.1 <= r < 1, so e^r >= e^0.1. Overflow in the final power operation - * will occur when (e^0.1)^(10^t) > 10^(emax+1). If we consider MAX_EMAX, - * this will happen for t > 10 (32 bit) or (t > 19) (64 bit). + * (1) 0.1 <= r < 1, so e^0.1 <= e^r. If t > MAX_T, overflow occurs: * - * (2) -1 < r <= -0.1, so e^r > e^-1. Underflow in the final power operation - * will occur when (e^-1)^(10^t) < 10^(etiny-1). If we consider MIN_ETINY, - * this will also happen for t > 10 (32 bit) or (t > 19) (64 bit). + * MAX-EMAX+1 < log10(e^(0.1*10*t)) <= log10(e^(r*10^t)) < adjexp(e^(r*10^t))+1 + * + * (2) -1 < r <= -0.1, so e^r <= e^-0.1. It t > MAX_T, underflow occurs: + * + * adjexp(e^(r*10^t)) <= log10(e^(r*10^t)) <= log10(e^(-0.1*10^t) < MIN-ETINY */ #if defined(CONFIG_64) #define MPD_EXP_MAX_T 19 @@ -3974,20 +4023,33 @@ return; } + /* abs(a) <= 9 * 10**(-prec-1) */ + if (_mpd_qexp_check_one(result, a, ctx, status)) { + return; + } + mpd_maxcontext(&workctx); workctx.prec = ctx->prec + t + 2; - workctx.prec = (workctx.prec < 9) ? 9 : workctx.prec; + workctx.prec = (workctx.prec < 10) ? 10 : workctx.prec; workctx.round = MPD_ROUND_HALF_EVEN; - if ((n = _mpd_get_exp_iterations(a, workctx.prec)) == MPD_SSIZE_MAX) { + if (!mpd_qcopy(result, a, status)) { + return; + } + result->exp -= t; + + /* + * At this point: + * 1) 9 * 10**(-prec-1) < abs(a) + * 2) 9 * 10**(-prec-t-1) < abs(r) + * 3) log10(9) - prec - t - 1 < log10(abs(r)) < adjexp(abs(r)) + 1 + * 4) - prec - t - 2 < adjexp(abs(r)) <= -1 + */ + n = _mpd_get_exp_iterations(result, workctx.prec); + if (n == MPD_SSIZE_MAX) { mpd_seterror(result, MPD_Invalid_operation, status); /* GCOV_UNLIKELY */ - goto finish; /* GCOV_UNLIKELY */ - } - - if (!mpd_qcopy(result, a, status)) { - goto finish; - } - result->exp -= t; + return; /* GCOV_UNLIKELY */ + } _settriple(&sum, MPD_POS, 1, 0); @@ -3995,8 +4057,7 @@ word.data[0] = j; mpd_setdigits(&word); mpd_qdiv(&tmp, result, &word, &workctx, &workctx.status); - mpd_qmul(&sum, &sum, &tmp, &workctx, &workctx.status); - mpd_qadd(&sum, &sum, &one, &workctx, &workctx.status); + mpd_qfma(&sum, &sum, &tmp, &one, &workctx, &workctx.status); } #ifdef CONFIG_64 @@ -4013,8 +4074,8 @@ } #endif - -finish: + _mpd_zeropad(result, ctx, status); + mpd_del(&tmp); mpd_del(&sum); *status |= (workctx.status&MPD_Errors); @@ -4069,8 +4130,18 @@ workctx.prec = prec; _mpd_qexp(result, a, &workctx, status); _ssettriple(&ulp, MPD_POS, 1, - result->exp + result->digits-workctx.prec-1); - + result->exp + result->digits-workctx.prec); + + /* + * At this point: + * 1) abs(result - e**x) < 0.5 * 10**(-prec) * e**x + * 2) result - ulp < e**x < result + ulp + * 3) result - ulp < result < result + ulp + * + * If round(result-ulp)==round(result+ulp), then + * round(result)==round(e**x). Therefore the result + * is correctly rounded. + */ workctx.prec = ctx->prec; mpd_qadd(&t1, result, &ulp, &workctx, &workctx.status); mpd_qsub(&t2, result, &ulp, &workctx, &workctx.status); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 20:21:05 2012 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 May 2012 20:21:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_Visual_Studio_warning?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/87fa250c05cd changeset: 77005:87fa250c05cd user: Stefan Krah date: Wed May 16 20:20:03 2012 +0200 summary: Fix Visual Studio warning. files: Modules/_decimal/libmpdec/mpdecimal.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -7431,7 +7431,7 @@ if (x > 2711437152599294ULL) { return SIZE_MAX; } - return (double)x / log10(base) + 3; + return (size_t)((double)x / log10(base) + 3); #endif #else /* CONFIG_32 */ { -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 20:43:10 2012 From: python-checkins at python.org (stefan.krah) Date: Wed, 16 May 2012 20:43:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314779=3A_Get_sizeo?= =?utf8?q?f=28void_*=29_directly_rather_than_relying_on_sysconfig=2E?= Message-ID: http://hg.python.org/cpython/rev/967b06d56a69 changeset: 77006:967b06d56a69 user: Stefan Krah date: Wed May 16 20:41:56 2012 +0200 summary: Issue #14779: Get sizeof(void *) directly rather than relying on sysconfig. files: Lib/test/test_buffer.py | 6 ++---- Modules/_testbuffer.c | 7 +++++++ 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/Lib/test/test_buffer.py b/Lib/test/test_buffer.py --- a/Lib/test/test_buffer.py +++ b/Lib/test/test_buffer.py @@ -747,10 +747,8 @@ class TestBufferProtocol(unittest.TestCase): def setUp(self): - self.sizeof_void_p = get_config_var('SIZEOF_VOID_P') \ - if sys.platform != 'darwin' else None - if not self.sizeof_void_p: - self.sizeof_void_p = 8 if sys.maxsize > 2**32 else 4 + # The suboffsets tests need sizeof(void *). + self.sizeof_void_p = get_sizeof_void_p() def verify(self, result, obj=-1, itemsize={1}, fmt=-1, readonly={1}, diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c --- a/Modules/_testbuffer.c +++ b/Modules/_testbuffer.c @@ -2337,6 +2337,12 @@ return ret; } +static PyObject * +get_sizeof_void_p(PyObject *self) +{ + return PyLong_FromSize_t(sizeof(void *)); +} + static char get_ascii_order(PyObject *order) { @@ -2726,6 +2732,7 @@ static struct PyMethodDef _testbuffer_functions[] = { {"slice_indices", slice_indices, METH_VARARGS, NULL}, {"get_pointer", get_pointer, METH_VARARGS, NULL}, + {"get_sizeof_void_p", (PyCFunction)get_sizeof_void_p, METH_NOARGS, NULL}, {"get_contiguous", get_contiguous, METH_VARARGS, NULL}, {"is_contiguous", is_contiguous, METH_VARARGS, NULL}, {"cmp_contig", cmp_contig, METH_VARARGS, NULL}, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 21:44:39 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 16 May 2012 21:44:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314780=3A_urllib=2E?= =?utf8?q?request=2Eurlopen=28=29_now_has_a_=60cadefault=60_argument_to_us?= =?utf8?q?e?= Message-ID: http://hg.python.org/cpython/rev/f2ed5de1c568 changeset: 77007:f2ed5de1c568 user: Antoine Pitrou date: Wed May 16 21:40:01 2012 +0200 summary: Issue #14780: urllib.request.urlopen() now has a `cadefault` argument to use the default certificate store. Initial patch by James Oakley. files: Doc/library/urllib.request.rst | 15 ++++++++++++--- Lib/test/test_urllib2_localnet.py | 7 +++++++ Lib/urllib/request.py | 11 +++++++---- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 5 files changed, 30 insertions(+), 7 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -16,7 +16,7 @@ The :mod:`urllib.request` module defines the following functions: -.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None) +.. function:: urlopen(url, data=None[, timeout], *, cafile=None, capath=None, cadefault=True) Open the URL *url*, which can be either a string or a :class:`Request` object. @@ -53,9 +53,15 @@ point to a directory of hashed certificate files. More information can be found in :meth:`ssl.SSLContext.load_verify_locations`. + The *cadefault* parameter specifies whether to fall back to loading a + default certificate store defined by the underlying OpenSSL library if the + *cafile* and *capath* parameters are omitted. This will only work on + some non-Windows platforms. + .. warning:: - If neither *cafile* nor *capath* is specified, an HTTPS request - will not do any verification of the server's certificate. + If neither *cafile* nor *capath* is specified, and *cadefault* is False, + an HTTPS request will not do any verification of the server's + certificate. This function returns a file-like object that works as a :term:`context manager`, with two additional methods from the :mod:`urllib.response` module @@ -92,6 +98,9 @@ .. versionadded:: 3.2 *data* can be an iterable object. + .. versionchanged:: 3.3 + *cadefault* was added. + .. function:: install_opener(opener) Install an :class:`OpenerDirector` instance as the default global opener. diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -474,6 +474,13 @@ self.urlopen("https://localhost:%s/bizarre" % handler.port, cafile=CERT_fakehostname) + def test_https_with_cadefault(self): + handler = self.start_https_server(certfile=CERT_localhost) + # Self-signed cert should fail verification with system certificate store + with self.assertRaises(urllib.error.URLError) as cm: + self.urlopen("https://localhost:%s/bizarre" % handler.port, + cadefault=True) + def test_sending_headers(self): handler = self.start_server() req = urllib.request.Request("http://localhost:%s/" % handler.port, diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -135,16 +135,19 @@ _opener = None def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, - *, cafile=None, capath=None): + *, cafile=None, capath=None, cadefault=False): global _opener - if cafile or capath: + if cafile or capath or cadefault: if not _have_ssl: raise ValueError('SSL support not available') context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) context.options |= ssl.OP_NO_SSLv2 - if cafile or capath: + if cafile or capath or cadefault: context.verify_mode = ssl.CERT_REQUIRED - context.load_verify_locations(cafile, capath) + if cafile or capath: + context.load_verify_locations(cafile, capath) + else: + context.set_default_verify_paths() check_hostname = True else: check_hostname = False diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -746,6 +746,7 @@ John O'Connor Kevin O'Connor Tim O'Malley +James Oakley Jon Oberheide Pascal Oberndoerfer Jeffrey Ollie diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -34,6 +34,9 @@ Library ------- +- Issue #14780: urllib.request.urlopen() now has a ``cadefault`` argument + to use the default certificate store. Initial patch by James Oakley. + - Issue #14829: Fix bisect and range() indexing with large indices (>= 2 ** 32) under 64-bit Windows. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 16 22:51:36 2012 From: python-checkins at python.org (brett.cannon) Date: Wed, 16 May 2012 22:51:36 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Minor_cleanup_to_match_history?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/peps/rev/0c7ef034bf52 changeset: 4398:0c7ef034bf52 user: Brett Cannon date: Wed May 16 16:51:32 2012 -0400 summary: Minor cleanup to match history. files: pep-0352.txt | 12 +----------- 1 files changed, 1 insertions(+), 11 deletions(-) diff --git a/pep-0352.txt b/pep-0352.txt --- a/pep-0352.txt +++ b/pep-0352.txt @@ -232,20 +232,10 @@ - deprecate ``message`` attribute (see `Retracted Ideas`_) [done] -* Python 2.7 +* Python 2.7 [done] - deprecate raising exceptions that do not inherit from BaseException - - remove ``message`` attribute - -* Python 2.8 - - - deprecate catching exceptions that do not inherit from BaseException - -* Python 2.9 - - - deprecate ``__getitem__`` (optional) - * Python 3.0 [done] - drop everything that was deprecated above: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 17 00:23:41 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 17 May 2012 00:23:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_14813=3A_Fix_Visual_S?= =?utf8?q?tudio_2008_build_after_the_move_into_the_PC/VS9=2E0?= Message-ID: http://hg.python.org/cpython/rev/f6a207d86154 changeset: 77008:f6a207d86154 user: Stefan Krah date: Thu May 17 00:21:57 2012 +0200 summary: Issue 14813: Fix Visual Studio 2008 build after the move into the PC/VS9.0 directory. files: PC/VS9.0/_bz2.vcproj | 4 +- PC/VS9.0/_ctypes.vcproj | 48 +- PC/VS9.0/_ctypes_test.vcproj | 6 +- PC/VS9.0/_decimal.vcproj | 84 +- PC/VS9.0/_elementtree.vcproj | 56 +- PC/VS9.0/_hashlib.vcproj | 4 +- PC/VS9.0/_lzma.vcproj | 4 +- PC/VS9.0/_msi.vcproj | 4 +- PC/VS9.0/_multiprocessing.vcproj | 8 +- PC/VS9.0/_socket.vcproj | 6 +- PC/VS9.0/_sqlite3.vcproj | 38 +- PC/VS9.0/_ssl.vcproj | 4 +- PC/VS9.0/_testbuffer.vcproj | 2 +- PC/VS9.0/_testcapi.vcproj | 4 +- PC/VS9.0/_tkinter.vcproj | 6 +- PC/VS9.0/bdist_wininst.vcproj | 44 +- PC/VS9.0/kill_python.c | 178 +++++ PC/VS9.0/make_buildinfo.c | 195 +++++ PC/VS9.0/make_buildinfo.vcproj | 2 +- PC/VS9.0/make_versioninfo.vcproj | 28 +- PC/VS9.0/pcbuild.sln | 2 +- PC/VS9.0/pyexpat.vcproj | 30 +- PC/VS9.0/pyproject.vsprops | 6 +- PC/VS9.0/python.vcproj | 12 +- PC/VS9.0/python3dll.vcproj | 44 +- PC/VS9.0/pythoncore.vcproj | 624 +++++++++--------- PC/VS9.0/pythonw.vcproj | 6 +- PC/VS9.0/select.vcproj | 4 +- PC/VS9.0/sqlite3.vcproj | 2 +- PC/VS9.0/ssl.vcproj | 2 +- PC/VS9.0/unicodedata.vcproj | 8 +- PC/VS9.0/w9xpopen.vcproj | 4 +- PC/VS9.0/winsound.vcproj | 4 +- PC/VS9.0/xxlimited.vcproj | 4 +- 34 files changed, 925 insertions(+), 552 deletions(-) diff --git a/PC/VS9.0/_bz2.vcproj b/PC/VS9.0/_bz2.vcproj --- a/PC/VS9.0/_bz2.vcproj +++ b/PC/VS9.0/_bz2.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/_ctypes.vcproj b/PC/VS9.0/_ctypes.vcproj --- a/PC/VS9.0/_ctypes.vcproj +++ b/PC/VS9.0/_ctypes.vcproj @@ -42,7 +42,7 @@ /> @@ -559,39 +559,39 @@ Name="Source Files" > @@ -511,7 +511,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/_decimal.vcproj b/PC/VS9.0/_decimal.vcproj --- a/PC/VS9.0/_decimal.vcproj +++ b/PC/VS9.0/_decimal.vcproj @@ -1,7 +1,7 @@ @@ -605,67 +605,67 @@ Name="Source Files" > @@ -591,19 +591,19 @@ Name="Source Files" > diff --git a/PC/VS9.0/_hashlib.vcproj b/PC/VS9.0/_hashlib.vcproj --- a/PC/VS9.0/_hashlib.vcproj +++ b/PC/VS9.0/_hashlib.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/_lzma.vcproj b/PC/VS9.0/_lzma.vcproj --- a/PC/VS9.0/_lzma.vcproj +++ b/PC/VS9.0/_lzma.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/_msi.vcproj b/PC/VS9.0/_msi.vcproj --- a/PC/VS9.0/_msi.vcproj +++ b/PC/VS9.0/_msi.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/_multiprocessing.vcproj b/PC/VS9.0/_multiprocessing.vcproj --- a/PC/VS9.0/_multiprocessing.vcproj +++ b/PC/VS9.0/_multiprocessing.vcproj @@ -1,7 +1,7 @@ @@ -527,11 +527,11 @@ Name="Source Files" > diff --git a/PC/VS9.0/_socket.vcproj b/PC/VS9.0/_socket.vcproj --- a/PC/VS9.0/_socket.vcproj +++ b/PC/VS9.0/_socket.vcproj @@ -1,7 +1,7 @@ @@ -527,7 +527,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/_sqlite3.vcproj b/PC/VS9.0/_sqlite3.vcproj --- a/PC/VS9.0/_sqlite3.vcproj +++ b/PC/VS9.0/_sqlite3.vcproj @@ -527,43 +527,43 @@ Name="Header Files" > @@ -571,39 +571,39 @@ Name="Source Files" > diff --git a/PC/VS9.0/_ssl.vcproj b/PC/VS9.0/_ssl.vcproj --- a/PC/VS9.0/_ssl.vcproj +++ b/PC/VS9.0/_ssl.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/_testbuffer.vcproj b/PC/VS9.0/_testbuffer.vcproj --- a/PC/VS9.0/_testbuffer.vcproj +++ b/PC/VS9.0/_testbuffer.vcproj @@ -511,7 +511,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/_testcapi.vcproj b/PC/VS9.0/_testcapi.vcproj --- a/PC/VS9.0/_testcapi.vcproj +++ b/PC/VS9.0/_testcapi.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/_tkinter.vcproj b/PC/VS9.0/_tkinter.vcproj --- a/PC/VS9.0/_tkinter.vcproj +++ b/PC/VS9.0/_tkinter.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/bdist_wininst.vcproj b/PC/VS9.0/bdist_wininst.vcproj --- a/PC/VS9.0/bdist_wininst.vcproj +++ b/PC/VS9.0/bdist_wininst.vcproj @@ -20,7 +20,7 @@ @@ -247,7 +247,7 @@ Filter="h;hpp;hxx;hm;inl" > @@ -256,11 +256,11 @@ Filter="ico;cur;bmp;dlg;rc2;rct;bin;rgs;gif;jpg;jpeg;jpe" > diff --git a/PC/VS9.0/kill_python.c b/PC/VS9.0/kill_python.c new file mode 100644 --- /dev/null +++ b/PC/VS9.0/kill_python.c @@ -0,0 +1,178 @@ +/* + * Helper program for killing lingering python[_d].exe processes before + * building, thus attempting to avoid build failures due to files being + * locked. + */ + +#include +#include +#include +#include + +#pragma comment(lib, "psapi") + +#ifdef _DEBUG +#define PYTHON_EXE (L"python_d.exe") +#define PYTHON_EXE_LEN (12) +#define KILL_PYTHON_EXE (L"kill_python_d.exe") +#define KILL_PYTHON_EXE_LEN (17) +#else +#define PYTHON_EXE (L"python.exe") +#define PYTHON_EXE_LEN (10) +#define KILL_PYTHON_EXE (L"kill_python.exe") +#define KILL_PYTHON_EXE_LEN (15) +#endif + +int +main(int argc, char **argv) +{ + HANDLE hp, hsp, hsm; /* process, snapshot processes, snapshot modules */ + DWORD dac, our_pid; + size_t len; + wchar_t path[MAX_PATH+1]; + + MODULEENTRY32W me; + PROCESSENTRY32W pe; + + me.dwSize = sizeof(MODULEENTRY32W); + pe.dwSize = sizeof(PROCESSENTRY32W); + + memset(path, 0, MAX_PATH+1); + + our_pid = GetCurrentProcessId(); + + hsm = CreateToolhelp32Snapshot(TH32CS_SNAPMODULE, our_pid); + if (hsm == INVALID_HANDLE_VALUE) { + printf("CreateToolhelp32Snapshot[1] failed: %d\n", GetLastError()); + return 1; + } + + if (!Module32FirstW(hsm, &me)) { + printf("Module32FirstW[1] failed: %d\n", GetLastError()); + CloseHandle(hsm); + return 1; + } + + /* + * Enumerate over the modules for the current process in order to find + * kill_process[_d].exe, then take a note of the directory it lives in. + */ + do { + if (_wcsnicmp(me.szModule, KILL_PYTHON_EXE, KILL_PYTHON_EXE_LEN)) + continue; + + len = wcsnlen_s(me.szExePath, MAX_PATH) - KILL_PYTHON_EXE_LEN; + wcsncpy_s(path, MAX_PATH+1, me.szExePath, len); + + break; + + } while (Module32NextW(hsm, &me)); + + CloseHandle(hsm); + + if (path == NULL) { + printf("failed to discern directory of running process\n"); + return 1; + } + + /* + * Take a snapshot of system processes. Enumerate over the snapshot, + * looking for python processes. When we find one, verify it lives + * in the same directory we live in. If it does, kill it. If we're + * unable to kill it, treat this as a fatal error and return 1. + * + * The rationale behind this is that we're called at the start of the + * build process on the basis that we'll take care of killing any + * running instances, such that the build won't encounter permission + * denied errors during linking. If we can't kill one of the processes, + * we can't provide this assurance, and the build shouldn't start. + */ + + hsp = CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS, 0); + if (hsp == INVALID_HANDLE_VALUE) { + printf("CreateToolhelp32Snapshot[2] failed: %d\n", GetLastError()); + return 1; + } + + if (!Process32FirstW(hsp, &pe)) { + printf("Process32FirstW failed: %d\n", GetLastError()); + CloseHandle(hsp); + return 1; + } + + dac = PROCESS_QUERY_INFORMATION | PROCESS_VM_READ | PROCESS_TERMINATE; + do { + + /* + * XXX TODO: if we really wanted to be fancy, we could check the + * modules for all processes (not just the python[_d].exe ones) + * and see if any of our DLLs are loaded (i.e. python33[_d].dll), + * as that would also inhibit our ability to rebuild the solution. + * Not worth loosing sleep over though; for now, a simple check + * for just the python executable should be sufficient. + */ + + if (_wcsnicmp(pe.szExeFile, PYTHON_EXE, PYTHON_EXE_LEN)) + /* This isn't a python process. */ + continue; + + /* It's a python process, so figure out which directory it's in... */ + hsm = CreateToolhelp32Snapshot(TH32CS_SNAPMODULE, pe.th32ProcessID); + if (hsm == INVALID_HANDLE_VALUE) + /* + * If our module snapshot fails (which will happen if we don't own + * the process), just ignore it and continue. (It seems different + * versions of Windows return different values for GetLastError() + * in this situation; it's easier to just ignore it and move on vs. + * stopping the build for what could be a false positive.) + */ + continue; + + if (!Module32FirstW(hsm, &me)) { + printf("Module32FirstW[2] failed: %d\n", GetLastError()); + CloseHandle(hsp); + CloseHandle(hsm); + return 1; + } + + do { + if (_wcsnicmp(me.szModule, PYTHON_EXE, PYTHON_EXE_LEN)) + /* Wrong module, we're looking for python[_d].exe... */ + continue; + + if (_wcsnicmp(path, me.szExePath, len)) + /* Process doesn't live in our directory. */ + break; + + /* Python process residing in the right directory, kill it! */ + hp = OpenProcess(dac, FALSE, pe.th32ProcessID); + if (!hp) { + printf("OpenProcess failed: %d\n", GetLastError()); + CloseHandle(hsp); + CloseHandle(hsm); + return 1; + } + + if (!TerminateProcess(hp, 1)) { + printf("TerminateProcess failed: %d\n", GetLastError()); + CloseHandle(hsp); + CloseHandle(hsm); + CloseHandle(hp); + return 1; + } + + CloseHandle(hp); + break; + + } while (Module32NextW(hsm, &me)); + + CloseHandle(hsm); + + } while (Process32NextW(hsp, &pe)); + + CloseHandle(hsp); + + return 0; +} + +/* vi: set ts=8 sw=4 sts=4 expandtab */ diff --git a/PC/VS9.0/make_buildinfo.c b/PC/VS9.0/make_buildinfo.c new file mode 100644 --- /dev/null +++ b/PC/VS9.0/make_buildinfo.c @@ -0,0 +1,195 @@ +#include +#include +#include +#include +#include + +#define CMD_SIZE 500 + +/* This file creates the getbuildinfo.o object, by first + invoking subwcrev.exe (if found), and then invoking cl.exe. + As a side effect, it might generate PCBuild\getbuildinfo2.c + also. If this isn't a subversion checkout, or subwcrev isn't + found, it compiles ..\\..\\Modules\\getbuildinfo.c instead. + + Currently, subwcrev.exe is found from the registry entries + of TortoiseSVN. + + No attempt is made to place getbuildinfo.o into the proper + binary directory. This isn't necessary, as this tool is + invoked as a pre-link step for pythoncore, so that overwrites + any previous getbuildinfo.o. + + However, if a second argument is provided, this will be used + as a temporary directory where any getbuildinfo2.c and + getbuildinfo.o files are put. This is useful if multiple + configurations are being built in parallel, to avoid them + trampling each other's files. + +*/ + +int make_buildinfo2(const char *tmppath) +{ + struct _stat st; + HKEY hTortoise; + char command[CMD_SIZE+1]; + DWORD type, size; + if (_stat(".svn", &st) < 0) + return 0; + /* Allow suppression of subwcrev.exe invocation if a no_subwcrev file is present. */ + if (_stat("no_subwcrev", &st) == 0) + return 0; + if (RegOpenKey(HKEY_LOCAL_MACHINE, "Software\\TortoiseSVN", &hTortoise) != ERROR_SUCCESS && + RegOpenKey(HKEY_CURRENT_USER, "Software\\TortoiseSVN", &hTortoise) != ERROR_SUCCESS) + /* Tortoise not installed */ + return 0; + command[0] = '"'; /* quote the path to the executable */ + size = sizeof(command) - 1; + if (RegQueryValueEx(hTortoise, "Directory", 0, &type, command+1, &size) != ERROR_SUCCESS || + type != REG_SZ) + /* Registry corrupted */ + return 0; + strcat_s(command, CMD_SIZE, "bin\\subwcrev.exe"); + if (_stat(command+1, &st) < 0) + /* subwcrev.exe not part of the release */ + return 0; + strcat_s(command, CMD_SIZE, "\" ..\\.. ..\\..\\Modules\\getbuildinfo.c \""); + strcat_s(command, CMD_SIZE, tmppath); /* quoted tmppath */ + strcat_s(command, CMD_SIZE, "getbuildinfo2.c\""); + + puts(command); fflush(stdout); + if (system(command) < 0) + return 0; + return 1; +} + +const char DELIMS[] = { " \n" }; + +int get_mercurial_info(char * hgbranch, char * hgtag, char * hgrev, int size) +{ + int result = 0; + char filename[CMD_SIZE]; + char cmdline[CMD_SIZE]; + + strcpy_s(filename, CMD_SIZE, "tmpXXXXXX"); + if (_mktemp_s(filename, CMD_SIZE) == 0) { + int rc; + + strcpy_s(cmdline, CMD_SIZE, "hg id -bit > "); + strcat_s(cmdline, CMD_SIZE, filename); + rc = system(cmdline); + if (rc == 0) { + FILE * fp; + + if (fopen_s(&fp, filename, "r") == 0) { + char * cp = fgets(cmdline, CMD_SIZE, fp); + + if (cp) { + char * context = NULL; + char * tp = strtok_s(cp, DELIMS, &context); + if (tp) { + strcpy_s(hgrev, size, tp); + tp = strtok_s(NULL, DELIMS, &context); + if (tp) { + strcpy_s(hgbranch, size, tp); + tp = strtok_s(NULL, DELIMS, &context); + if (tp) { + strcpy_s(hgtag, size, tp); + result = 1; + } + } + } + } + fclose(fp); + } + } + _unlink(filename); + } + return result; +} + +int main(int argc, char*argv[]) +{ + char command[CMD_SIZE] = "cl.exe -c -D_WIN32 -DUSE_DL_EXPORT -D_WINDOWS -DWIN32 -D_WINDLL "; + char tmppath[CMD_SIZE] = ""; + int do_unlink, result; + char *tmpdir = NULL; + if (argc <= 2 || argc > 3) { + fprintf(stderr, "make_buildinfo $(ConfigurationName) [tmpdir]\n"); + return EXIT_FAILURE; + } + if (strcmp(argv[1], "Release") == 0) { + strcat_s(command, CMD_SIZE, "-MD "); + } + else if (strcmp(argv[1], "Debug") == 0) { + strcat_s(command, CMD_SIZE, "-D_DEBUG -MDd "); + } + else if (strcmp(argv[1], "ReleaseItanium") == 0) { + strcat_s(command, CMD_SIZE, "-MD /USECL:MS_ITANIUM "); + } + else if (strcmp(argv[1], "ReleaseAMD64") == 0) { + strcat_s(command, CMD_SIZE, "-MD "); + strcat_s(command, CMD_SIZE, "-MD /USECL:MS_OPTERON "); + } + else { + fprintf(stderr, "unsupported configuration %s\n", argv[1]); + return EXIT_FAILURE; + } + if (argc > 2) { + tmpdir = argv[2]; + strcat_s(tmppath, _countof(tmppath), tmpdir); + /* Hack fix for bad command line: If the command is issued like this: + * $(SolutionDir)make_buildinfo.exe" Debug "$(IntDir)" + * we will get a trailing quote because IntDir ends with a backslash that then + * escapes the final ". To simplify the life for developers, catch that problem + * here by cutting it off. + * The proper command line, btw is: + * $(SolutionDir)make_buildinfo.exe" Debug "$(IntDir)\" + * Hooray for command line parsing on windows. + */ + if (strlen(tmppath) > 0 && tmppath[strlen(tmppath)-1] == '"') + tmppath[strlen(tmppath)-1] = '\0'; + strcat_s(tmppath, _countof(tmppath), "\\"); + } + + if ((do_unlink = make_buildinfo2(tmppath))) { + strcat_s(command, CMD_SIZE, "\""); + strcat_s(command, CMD_SIZE, tmppath); + strcat_s(command, CMD_SIZE, "getbuildinfo2.c\" -DSUBWCREV "); + } + else { + char hgtag[CMD_SIZE]; + char hgbranch[CMD_SIZE]; + char hgrev[CMD_SIZE]; + + if (get_mercurial_info(hgbranch, hgtag, hgrev, CMD_SIZE)) { + strcat_s(command, CMD_SIZE, "-DHGBRANCH=\\\""); + strcat_s(command, CMD_SIZE, hgbranch); + strcat_s(command, CMD_SIZE, "\\\""); + + strcat_s(command, CMD_SIZE, " -DHGTAG=\\\""); + strcat_s(command, CMD_SIZE, hgtag); + strcat_s(command, CMD_SIZE, "\\\""); + + strcat_s(command, CMD_SIZE, " -DHGVERSION=\\\""); + strcat_s(command, CMD_SIZE, hgrev); + strcat_s(command, CMD_SIZE, "\\\" "); + } + strcat_s(command, CMD_SIZE, "..\\..\\Modules\\getbuildinfo.c"); + } + strcat_s(command, CMD_SIZE, " -Fo\""); + strcat_s(command, CMD_SIZE, tmppath); + strcat_s(command, CMD_SIZE, "getbuildinfo.o\" -I..\\..\\Include -I..\\..\\PC"); + puts(command); fflush(stdout); + result = system(command); + if (do_unlink) { + command[0] = '\0'; + strcat_s(command, CMD_SIZE, "\""); + strcat_s(command, CMD_SIZE, tmppath); + strcat_s(command, CMD_SIZE, "getbuildinfo2.c\""); + _unlink(command); + } + if (result < 0) + return EXIT_FAILURE; + return 0; +} diff --git a/PC/VS9.0/make_buildinfo.vcproj b/PC/VS9.0/make_buildinfo.vcproj --- a/PC/VS9.0/make_buildinfo.vcproj +++ b/PC/VS9.0/make_buildinfo.vcproj @@ -1,7 +1,7 @@ @@ -314,7 +314,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/pcbuild.sln b/PC/VS9.0/pcbuild.sln --- a/PC/VS9.0/pcbuild.sln +++ b/PC/VS9.0/pcbuild.sln @@ -29,7 +29,7 @@ EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{553EC33E-9816-4996-A660-5D6186A0B0B3}" ProjectSection(SolutionItems) = preProject - ..\Modules\getbuildinfo.c = ..\Modules\getbuildinfo.c + ..\..\Modules\getbuildinfo.c = ..\..\Modules\getbuildinfo.c readme.txt = readme.txt EndProjectSection EndProject diff --git a/PC/VS9.0/pyexpat.vcproj b/PC/VS9.0/pyexpat.vcproj --- a/PC/VS9.0/pyexpat.vcproj +++ b/PC/VS9.0/pyexpat.vcproj @@ -1,7 +1,7 @@ @@ -531,19 +531,19 @@ Name="Source Files" > diff --git a/PC/VS9.0/pyproject.vsprops b/PC/VS9.0/pyproject.vsprops --- a/PC/VS9.0/pyproject.vsprops +++ b/PC/VS9.0/pyproject.vsprops @@ -11,7 +11,7 @@ Optimization="2" InlineFunctionExpansion="1" EnableIntrinsicFunctions="true" - AdditionalIncludeDirectories="..\Include; ..\PC" + AdditionalIncludeDirectories="..\..\Include; ..\..\PC" PreprocessorDefinitions="_WIN32" StringPooling="true" ExceptionHandling="0" @@ -34,7 +34,7 @@ /> @@ -627,7 +627,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/python3dll.vcproj b/PC/VS9.0/python3dll.vcproj --- a/PC/VS9.0/python3dll.vcproj +++ b/PC/VS9.0/python3dll.vcproj @@ -1,7 +1,7 @@ @@ -54,7 +54,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="NDEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -129,7 +129,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="NDEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -207,7 +207,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="_DEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -285,7 +285,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="_DEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -359,7 +359,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="NDEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -434,7 +434,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="NDEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -509,7 +509,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="NDEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -584,7 +584,7 @@ Name="VCResourceCompilerTool" PreprocessorDefinitions="NDEBUG" Culture="1033" - AdditionalIncludeDirectories="..\Include" + AdditionalIncludeDirectories="..\..\Include" /> @@ -991,222 +991,222 @@ Name="Modules" > @@ -1214,31 +1214,31 @@ Name="zlib" > @@ -1374,71 +1374,71 @@ Name="cjkcodecs" > @@ -1447,175 +1447,175 @@ Name="Objects" > @@ -1623,59 +1623,59 @@ Name="Parser" > @@ -1683,31 +1683,31 @@ Name="PC" > @@ -1715,191 +1715,191 @@ Name="Python" > @@ -1907,7 +1907,7 @@ Name="Resource Files" > diff --git a/PC/VS9.0/pythonw.vcproj b/PC/VS9.0/pythonw.vcproj --- a/PC/VS9.0/pythonw.vcproj +++ b/PC/VS9.0/pythonw.vcproj @@ -1,7 +1,7 @@ @@ -608,7 +608,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/select.vcproj b/PC/VS9.0/select.vcproj --- a/PC/VS9.0/select.vcproj +++ b/PC/VS9.0/select.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/sqlite3.vcproj b/PC/VS9.0/sqlite3.vcproj --- a/PC/VS9.0/sqlite3.vcproj +++ b/PC/VS9.0/sqlite3.vcproj @@ -1,7 +1,7 @@ @@ -523,7 +523,7 @@ Name="Source Files" > diff --git a/PC/VS9.0/w9xpopen.vcproj b/PC/VS9.0/w9xpopen.vcproj --- a/PC/VS9.0/w9xpopen.vcproj +++ b/PC/VS9.0/w9xpopen.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/winsound.vcproj b/PC/VS9.0/winsound.vcproj --- a/PC/VS9.0/winsound.vcproj +++ b/PC/VS9.0/winsound.vcproj @@ -1,7 +1,7 @@ diff --git a/PC/VS9.0/xxlimited.vcproj b/PC/VS9.0/xxlimited.vcproj --- a/PC/VS9.0/xxlimited.vcproj +++ b/PC/VS9.0/xxlimited.vcproj @@ -1,7 +1,7 @@ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 01:02:23 2012 From: python-checkins at python.org (ned.deily) Date: Thu, 17 May 2012 01:02:23 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_myself_to_=27release_m?= =?utf8?q?anagement=27_interest_area=2C_e=2Eg=2E?= Message-ID: http://hg.python.org/devguide/rev/b13c72380297 changeset: 512:b13c72380297 user: Ned Deily date: Wed May 16 16:02:08 2012 -0700 summary: Add myself to 'release management' interest area, e.g. current role of producing OS X installers for releases. files: experts.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -315,7 +315,7 @@ packaging tarek, lemburg, alexis, eric.araujo py3 transition benjamin.peterson release management tarek, lemburg, benjamin.peterson, barry, loewis, - gvanrossum, anthonybaxter, eric.araujo + gvanrossum, anthonybaxter, eric.araujo, ned.deily str.format eric.smith testing michael.foord, pitrou, ezio.melotti test coverage ncoghlan, giampaolo.rodola -- Repository URL: http://hg.python.org/devguide From solipsis at pitrou.net Thu May 17 05:48:35 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 17 May 2012 05:48:35 +0200 Subject: [Python-checkins] Daily reference leaks (f6a207d86154): sum=0 Message-ID: results for f6a207d86154 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogPk8uaq', '-x'] From python-checkins at python.org Thu May 17 12:08:58 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 17 May 2012 12:08:58 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_link_to_rationale_for_allo?= =?utf8?q?wing_namespace_packages_in_the_standard_library=2E?= Message-ID: http://hg.python.org/peps/rev/b65cf749cacc changeset: 4399:b65cf749cacc user: Eric V. Smith date: Thu May 17 06:08:31 2012 -0400 summary: Add link to rationale for allowing namespace packages in the standard library. files: pep-0420.txt | 14 ++++++++++++-- 1 files changed, 12 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -326,6 +326,12 @@ 4. This will also be addressed in PEP 395. +The inclusion of namespace packages in the standard library was +motivated by Martin v. L?wis, who wanted the ``encodings`` package to +become a namespace package. While this PEP allows for standard +library packages to become namespaces, it defers a decision on +``encodings``. + ``find_module`` versus ``find_loader`` -------------------------------------- @@ -343,7 +349,7 @@ implement the ``find_loader`` method, described above. The use case for supporting multiple portions per ``find_loader`` call -is given in [6]_. +is given in [7]_. Module reprs @@ -403,7 +409,11 @@ .. [5] Nick Coghlan's response to his initial objections (http://mail.python.org/pipermail/import-sig/2012-April/000464.html) -.. [6] Use case for multiple portions per ``find_loader`` call +.. [6] Martin v. L?wis's suggestion to make ``encodings`` a namespace + package + (http://mail.python.org/pipermail/import-sig/2012-May/000540.html) + +.. [7] Use case for multiple portions per ``find_loader`` call (http://mail.python.org/pipermail/import-sig/2012-May/000585.html) Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 17 12:11:55 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 17 May 2012 12:11:55 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_forgotten_footnote_referen?= =?utf8?b?Y2Uu?= Message-ID: http://hg.python.org/peps/rev/f4130ae40e4b changeset: 4400:f4130ae40e4b user: Eric V. Smith date: Thu May 17 06:11:42 2012 -0400 summary: Add forgotten footnote reference. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -328,7 +328,7 @@ The inclusion of namespace packages in the standard library was motivated by Martin v. L?wis, who wanted the ``encodings`` package to -become a namespace package. While this PEP allows for standard +become a namespace package [6]_. While this PEP allows for standard library packages to become namespaces, it defers a decision on ``encodings``. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 17 12:53:21 2012 From: python-checkins at python.org (barry.warsaw) Date: Thu, 17 May 2012 12:53:21 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Spellchecked=2E?= Message-ID: http://hg.python.org/peps/rev/f257aa2796bd changeset: 4401:f257aa2796bd user: Barry Warsaw date: Thu May 17 06:53:13 2012 -0400 summary: Spellchecked. files: pep-0001.txt | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -42,7 +42,7 @@ provides general guidelines or information to the Python community, but does not propose a new feature. Informational PEPs do not necessarily represent a Python community consensus or - recommendation, so users and implementors are free to ignore + recommendation, so users and implementers are free to ignore Informational PEPs or follow their advice. 3. A **Process** PEP describes a process surrounding Python, or @@ -81,10 +81,10 @@ recommended that a single PEP contain a single key proposal or new idea. Small enhancements or patches often don't need a PEP and can be injected into the Python development work flow with a -patch submission to the Python `issue tracker`_. The more focussed the +patch submission to the Python `issue tracker`_. The more focused the PEP, the more successful it tends to be. The PEP editor reserves the -right to reject PEP proposals if they appear too unfocussed or too -broad. If in doubt, split your PEP into several well-focussed ones. +right to reject PEP proposals if they appear too unfocused or too +broad. If in doubt, split your PEP into several well-focused ones. Each PEP must have a champion -- someone who writes the PEP using the style and format described below, shepherds the discussions in the @@ -164,7 +164,7 @@ Once the authors have completed a PEP, they must inform the PEP editors that it is ready for review. PEPs are reviewed by the BDFL and his chosen consultants, who may accept or reject a PEP or send it back to -the author(s) for revision. For a PEP that is pre-determined to be +the author(s) for revision. For a PEP that is predetermined to be acceptable (e.g., it is an obvious win as-is and/or its implementation has already been checked in) the BDFL may also initiate a PEP review, first notifying the PEP author(s) and giving them a chance to make @@ -232,7 +232,7 @@ In general, Standards track PEPs are no longer modified after they have reached the Final state. Once a PEP has been completed, the Language and Standard Library References become the formal documentation of the expected -behaviour. +behavior. Informational and Process PEPs may be updated over time to reflect changes to development practices and other details. The precise process followed in @@ -254,7 +254,7 @@ being addressed. 3. Copyright/public domain -- Each PEP must either be explicitly - labelled as placed in the public domain (see this PEP as an + labeled as placed in the public domain (see this PEP as an example) or licensed under the `Open Publication License`_. 4. Specification -- The technical specification should describe the @@ -515,7 +515,7 @@ list for PEP changes, and correct any structure, grammar, spelling, or markup mistakes we see. -The editors don't pass judgement on PEPs. We merely do the +The editors don't pass judgment on PEPs. We merely do the administrative & editorial part. Except for times like this, there's relatively low volume. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 17 15:15:16 2012 From: python-checkins at python.org (r.david.murray) Date: Thu, 17 May 2012 15:15:16 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0ODIzOiBTaW1w?= =?utf8?q?lify_threading=2ELock=2Eacquire_argument_discussion=2E?= Message-ID: http://hg.python.org/cpython/rev/6ab4128acccc changeset: 77009:6ab4128acccc branch: 3.2 parent: 77001:4f27c4dc34ed user: R David Murray date: Thu May 17 09:13:30 2012 -0400 summary: #14823: Simplify threading.Lock.acquire argument discussion. files: Doc/library/threading.rst | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-) diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -403,15 +403,12 @@ Acquire a lock, blocking or non-blocking. - When invoked without arguments, block until the lock is unlocked, then set it to - locked, and return true. + When invoked with the *blocking* argument set to ``True`` (the default), + block until the lock is unlocked, then set it to locked and return ``True``. - When invoked with the *blocking* argument set to true, do the same thing as when - called without arguments, and return true. - - When invoked with the *blocking* argument set to false, do not block. If a call - without an argument would block, return false immediately; otherwise, do the - same thing as when called without arguments, and return true. + When invoked with the *blocking* argument set to ``False``, do not block. + If a call with *blocking* set to ``True`` would block, return ``False`` + immediately; otherwise, set the lock to locked and return ``True``. When invoked with the floating-point *timeout* argument set to a positive value, block for at most the number of seconds specified by *timeout* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 15:15:17 2012 From: python-checkins at python.org (r.david.murray) Date: Thu, 17 May 2012 15:15:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314823=3A_Simplify_threading=2ELock=2Eacquire_argument_dis?= =?utf8?q?cussion=2E?= Message-ID: http://hg.python.org/cpython/rev/b5e95bb79ba3 changeset: 77010:b5e95bb79ba3 parent: 77008:f6a207d86154 parent: 77009:6ab4128acccc user: R David Murray date: Thu May 17 09:14:28 2012 -0400 summary: #14823: Simplify threading.Lock.acquire argument discussion. files: Doc/library/threading.rst | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-) diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -426,15 +426,12 @@ Acquire a lock, blocking or non-blocking. - When invoked without arguments, block until the lock is unlocked, then set it to - locked, and return true. + When invoked with the *blocking* argument set to ``True`` (the default), + block until the lock is unlocked, then set it to locked and return ``True``. - When invoked with the *blocking* argument set to true, do the same thing as when - called without arguments, and return true. - - When invoked with the *blocking* argument set to false, do not block. If a call - without an argument would block, return false immediately; otherwise, do the - same thing as when called without arguments, and return true. + When invoked with the *blocking* argument set to ``False``, do not block. + If a call with *blocking* set to ``True`` would block, return ``False`` + immediately; otherwise, set the lock to locked and return ``True``. When invoked with the floating-point *timeout* argument set to a positive value, block for at most the number of seconds specified by *timeout* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 15:15:18 2012 From: python-checkins at python.org (r.david.murray) Date: Thu, 17 May 2012 15:15:18 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0ODIzOiBTaW1w?= =?utf8?q?lify_threading=2ELock=2Eacquire_argument_discussion=2E?= Message-ID: http://hg.python.org/cpython/rev/251463919f3c changeset: 77011:251463919f3c branch: 2.7 parent: 77003:ff52583a5576 user: R David Murray date: Thu May 17 09:15:02 2012 -0400 summary: #14823: Simplify threading.Lock.acquire argument discussion. files: Doc/library/threading.rst | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-) diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -401,15 +401,12 @@ Acquire a lock, blocking or non-blocking. - When invoked without arguments, block until the lock is unlocked, then set it to - locked, and return true. + When invoked with the *blocking* argument set to ``True`` (the default), + block until the lock is unlocked, then set it to locked and return ``True``. - When invoked with the *blocking* argument set to true, do the same thing as when - called without arguments, and return true. - - When invoked with the *blocking* argument set to false, do not block. If a call - without an argument would block, return false immediately; otherwise, do the - same thing as when called without arguments, and return true. + When invoked with the *blocking* argument set to ``False``, do not block. + If a call with *blocking* set to ``True`` would block, return ``False`` + immediately; otherwise, set the lock to locked and return ``True``. .. method:: Lock.release() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 15:32:09 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 17 May 2012 15:32:09 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Typo=2E?= Message-ID: http://hg.python.org/peps/rev/2ed614d32a5e changeset: 4402:2ed614d32a5e user: Eric V. Smith date: Thu May 17 09:31:53 2012 -0400 summary: Typo. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -361,7 +361,7 @@ this in its value. The only exception was that PEP 302 reserved missing ``__file__`` attributes to built-in modules, and in CPython, this assumption was baked into the module object's implementation. Because of this -restriction, some module contained contrived ``__file__`` values that did not +restriction, some modules contained contrived ``__file__`` values that did not reflect file system paths, and which could cause unexpected problems later (e.g. ``os.path.join()`` on a non-path ``__file__`` would return gibberish). -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 17 17:39:41 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 May 2012 17:39:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_mention_of_decoding_opt?= =?utf8?q?imizations_in_the_what=27s_new_document=2E?= Message-ID: http://hg.python.org/cpython/rev/3430d7329a3b changeset: 77012:3430d7329a3b parent: 77010:b5e95bb79ba3 user: Antoine Pitrou date: Thu May 17 17:37:02 2012 +0200 summary: Add mention of decoding optimizations in the what's new document. files: Doc/whatsnew/3.3.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1260,6 +1260,10 @@ * repeating a single ASCII letter and getting a substring of a ASCII strings is 4 times faster +* UTF-8 and UTF-16 decoding is now 2x to 4x faster. + + (contributed by Serhiy Storchaka, :issue:`14624` and :issue:`14738`.) + Build and C API Changes ======================= -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 19:03:04 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 May 2012 19:03:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=239260=3A_A_finer-gr?= =?utf8?q?ained_import_lock=2E?= Message-ID: http://hg.python.org/cpython/rev/edb9ce3a6c2e changeset: 77013:edb9ce3a6c2e user: Antoine Pitrou date: Thu May 17 18:55:59 2012 +0200 summary: Issue #9260: A finer-grained import lock. Most of the import sequence now uses per-module locks rather than the global import lock, eliminating well-known issues with threads and imports. files: Doc/c-api/import.rst | 14 +- Doc/library/imp.rst | 31 ++- Lib/importlib/_bootstrap.py | 191 +++++++++++++++++- Lib/importlib/test/test_locks.py | 115 +++++++++++ Lib/pydoc.py | 2 +- Lib/test/lock_tests.py | 12 +- Lib/test/test_pkg.py | 2 + Lib/test/test_threaded_import.py | 19 +- Lib/token.py | 2 +- Misc/NEWS | 4 + Python/import.c | 88 ++++---- Python/importlib.h | Bin 12 files changed, 398 insertions(+), 82 deletions(-) diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst --- a/Doc/c-api/import.rst +++ b/Doc/c-api/import.rst @@ -30,13 +30,13 @@ .. c:function:: PyObject* PyImport_ImportModuleNoBlock(const char *name) - This version of :c:func:`PyImport_ImportModule` does not block. It's intended - to be used in C functions that import other modules to execute a function. - The import may block if another thread holds the import lock. The function - :c:func:`PyImport_ImportModuleNoBlock` never blocks. It first tries to fetch - the module from sys.modules and falls back to :c:func:`PyImport_ImportModule` - unless the lock is held, in which case the function will raise an - :exc:`ImportError`. + This function is a deprecated alias of :c:func:`PyImport_ImportModule`. + + .. versionchanged:: 3.3 + This function used to fail immediately when the import lock was held + by another thread. In Python 3.3 though, the locking scheme switched + to per-module locks for most purposes, so this function's special + behaviour isn't needed anymore. .. c:function:: PyObject* PyImport_ImportModuleEx(char *name, PyObject *globals, PyObject *locals, PyObject *fromlist) diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -112,18 +112,29 @@ Return ``True`` if the import lock is currently held, else ``False``. On platforms without threads, always return ``False``. - On platforms with threads, a thread executing an import holds an internal lock - until the import is complete. This lock blocks other threads from doing an - import until the original import completes, which in turn prevents other threads - from seeing incomplete module objects constructed by the original thread while - in the process of completing its import (and the imports, if any, triggered by - that). + On platforms with threads, a thread executing an import first holds a + global import lock, then sets up a per-module lock for the rest of the + import. This blocks other threads from importing the same module until + the original import completes, preventing other threads from seeing + incomplete module objects constructed by the original thread. An + exception is made for circular imports, which by construction have to + expose an incomplete module object at some point. + + .. note:: + Locking semantics of imports are an implementation detail which may + vary from release to release. However, Python ensures that circular + imports work without any deadlocks. + + .. versionchanged:: 3.3 + In Python 3.3, the locking scheme has changed to per-module locks for + the most part. .. function:: acquire_lock() - Acquire the interpreter's import lock for the current thread. This lock should - be used by import hooks to ensure thread-safety when importing modules. + Acquire the interpreter's global import lock for the current thread. + This lock should be used by import hooks to ensure thread-safety when + importing modules. Once a thread has acquired the import lock, the same thread may acquire it again without blocking; the thread must release it once for each time it has @@ -134,8 +145,8 @@ .. function:: release_lock() - Release the interpreter's import lock. On platforms without threads, this - function does nothing. + Release the interpreter's global import lock. On platforms without + threads, this function does nothing. .. function:: reload(module) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -159,6 +159,145 @@ return type(_io)(name) +# Module-level locking ######################################################## + +# A dict mapping module names to weakrefs of _ModuleLock instances +_module_locks = {} +# A dict mapping thread ids to _ModuleLock instances +_blocking_on = {} + + +class _DeadlockError(RuntimeError): + pass + + +class _ModuleLock: + """A recursive lock implementation which is able to detect deadlocks + (e.g. thread 1 trying to take locks A then B, and thread 2 trying to + take locks B then A). + """ + + def __init__(self, name): + self.lock = _thread.allocate_lock() + self.wakeup = _thread.allocate_lock() + self.name = name + self.owner = None + self.count = 0 + self.waiters = 0 + + def has_deadlock(self): + # Deadlock avoidance for concurrent circular imports. + me = _thread.get_ident() + tid = self.owner + while True: + lock = _blocking_on.get(tid) + if lock is None: + return False + tid = lock.owner + if tid == me: + return True + + def acquire(self): + """ + Acquire the module lock. If a potential deadlock is detected, + a _DeadlockError is raised. + Otherwise, the lock is always acquired and True is returned. + """ + tid = _thread.get_ident() + _blocking_on[tid] = self + try: + while True: + with self.lock: + if self.count == 0 or self.owner == tid: + self.owner = tid + self.count += 1 + return True + if self.has_deadlock(): + raise _DeadlockError("deadlock detected by %r" % self) + if self.wakeup.acquire(False): + self.waiters += 1 + # Wait for a release() call + self.wakeup.acquire() + self.wakeup.release() + finally: + del _blocking_on[tid] + + def release(self): + tid = _thread.get_ident() + with self.lock: + if self.owner != tid: + raise RuntimeError("cannot release un-acquired lock") + assert self.count > 0 + self.count -= 1 + if self.count == 0: + self.owner = None + if self.waiters: + self.waiters -= 1 + self.wakeup.release() + + def __repr__(self): + return "_ModuleLock(%r) at %d" % (self.name, id(self)) + + +class _DummyModuleLock: + """A simple _ModuleLock equivalent for Python builds without + multi-threading support.""" + + def __init__(self, name): + self.name = name + self.count = 0 + + def acquire(self): + self.count += 1 + return True + + def release(self): + if self.count == 0: + raise RuntimeError("cannot release un-acquired lock") + self.count -= 1 + + def __repr__(self): + return "_DummyModuleLock(%r) at %d" % (self.name, id(self)) + + +# The following two functions are for consumption by Python/import.c. + +def _get_module_lock(name): + """Get or create the module lock for a given module name. + + Should only be called with the import lock taken.""" + lock = None + if name in _module_locks: + lock = _module_locks[name]() + if lock is None: + if _thread is None: + lock = _DummyModuleLock(name) + else: + lock = _ModuleLock(name) + def cb(_): + del _module_locks[name] + _module_locks[name] = _weakref.ref(lock, cb) + return lock + +def _lock_unlock_module(name): + """Release the global import lock, and acquires then release the + module lock for a given module name. + This is used to ensure a module is completely initialized, in the + event it is being imported by another thread. + + Should only be called with the import lock taken.""" + lock = _get_module_lock(name) + _imp.release_lock() + try: + lock.acquire() + except _DeadlockError: + # Concurrent circular import, we'll accept a partially initialized + # module object. + pass + else: + lock.release() + + # Finder/loader utility code ################################################## _PYCACHE = '__pycache__' @@ -264,12 +403,15 @@ else: module.__package__ = fullname.rpartition('.')[0] try: + module.__initializing__ = True # If __package__ was not set above, __import__() will do it later. return fxn(self, module, *args, **kwargs) except: if not is_reload: del sys.modules[fullname] raise + finally: + module.__initializing__ = False _wrap(module_for_loader_wrapper, fxn) return module_for_loader_wrapper @@ -932,7 +1074,8 @@ if not sys.meta_path: _warnings.warn('sys.meta_path is empty', ImportWarning) for finder in sys.meta_path: - loader = finder.find_module(name, path) + with _ImportLockContext(): + loader = finder.find_module(name, path) if loader is not None: # The parent import may have already imported this module. if name not in sys.modules: @@ -962,8 +1105,7 @@ _ERR_MSG = 'No module named {!r}' -def _find_and_load(name, import_): - """Find and load the module.""" +def _find_and_load_unlocked(name, import_): path = None parent = name.rpartition('.')[0] if parent: @@ -1009,6 +1151,19 @@ return module +def _find_and_load(name, import_): + """Find and load the module, and release the import lock.""" + try: + lock = _get_module_lock(name) + finally: + _imp.release_lock() + lock.acquire() + try: + return _find_and_load_unlocked(name, import_) + finally: + lock.release() + + def _gcd_import(name, package=None, level=0): """Import and return the module based on its name, the package the call is being made from, and the level adjustment. @@ -1021,17 +1176,17 @@ _sanity_check(name, package, level) if level > 0: name = _resolve_name(name, package, level) - with _ImportLockContext(): - try: - module = sys.modules[name] - if module is None: - message = ("import of {} halted; " - "None in sys.modules".format(name)) - raise ImportError(message, name=name) - return module - except KeyError: - pass # Don't want to chain the exception + _imp.acquire_lock() + if name not in sys.modules: return _find_and_load(name, _gcd_import) + module = sys.modules[name] + if module is None: + _imp.release_lock() + message = ("import of {} halted; " + "None in sys.modules".format(name)) + raise ImportError(message, name=name) + _lock_unlock_module(name) + return module def _handle_fromlist(module, fromlist, import_): @@ -1149,7 +1304,17 @@ continue else: raise ImportError('importlib requires posix or nt') + + try: + thread_module = BuiltinImporter.load_module('_thread') + except ImportError: + # Python was built without threads + thread_module = None + weakref_module = BuiltinImporter.load_module('_weakref') + setattr(self_module, '_os', os_module) + setattr(self_module, '_thread', thread_module) + setattr(self_module, '_weakref', weakref_module) setattr(self_module, 'path_sep', path_sep) setattr(self_module, 'path_separators', set(path_separators)) # Constants diff --git a/Lib/importlib/test/test_locks.py b/Lib/importlib/test/test_locks.py new file mode 100644 --- /dev/null +++ b/Lib/importlib/test/test_locks.py @@ -0,0 +1,115 @@ +from importlib import _bootstrap +import time +import unittest +import weakref + +from test import support + +try: + import threading +except ImportError: + threading = None +else: + from test import lock_tests + + +LockType = _bootstrap._ModuleLock +DeadlockError = _bootstrap._DeadlockError + + +if threading is not None: + class ModuleLockAsRLockTests(lock_tests.RLockTests): + locktype = staticmethod(lambda: LockType("some_lock")) + + # _is_owned() unsupported + test__is_owned = None + # acquire(blocking=False) unsupported + test_try_acquire = None + test_try_acquire_contended = None + # `with` unsupported + test_with = None + # acquire(timeout=...) unsupported + test_timeout = None + # _release_save() unsupported + test_release_save_unacquired = None + +else: + class ModuleLockAsRLockTests(unittest.TestCase): + pass + + + at unittest.skipUnless(threading, "threads needed for this test") +class DeadlockAvoidanceTests(unittest.TestCase): + + def run_deadlock_avoidance_test(self, create_deadlock): + NLOCKS = 10 + locks = [LockType(str(i)) for i in range(NLOCKS)] + pairs = [(locks[i], locks[(i+1)%NLOCKS]) for i in range(NLOCKS)] + if create_deadlock: + NTHREADS = NLOCKS + else: + NTHREADS = NLOCKS - 1 + barrier = threading.Barrier(NTHREADS) + results = [] + def _acquire(lock): + """Try to acquire the lock. Return True on success, False on deadlock.""" + try: + lock.acquire() + except DeadlockError: + return False + else: + return True + def f(): + a, b = pairs.pop() + ra = _acquire(a) + barrier.wait() + rb = _acquire(b) + results.append((ra, rb)) + if rb: + b.release() + if ra: + a.release() + lock_tests.Bunch(f, NTHREADS).wait_for_finished() + self.assertEqual(len(results), NTHREADS) + return results + + def test_deadlock(self): + results = self.run_deadlock_avoidance_test(True) + # One of the threads detected a potential deadlock on its second + # acquire() call. + self.assertEqual(results.count((True, False)), 1) + self.assertEqual(results.count((True, True)), len(results) - 1) + + def test_no_deadlock(self): + results = self.run_deadlock_avoidance_test(False) + self.assertEqual(results.count((True, False)), 0) + self.assertEqual(results.count((True, True)), len(results)) + + +class LifetimeTests(unittest.TestCase): + + def test_lock_lifetime(self): + name = "xyzzy" + self.assertNotIn(name, _bootstrap._module_locks) + lock = _bootstrap._get_module_lock(name) + self.assertIn(name, _bootstrap._module_locks) + wr = weakref.ref(lock) + del lock + support.gc_collect() + self.assertNotIn(name, _bootstrap._module_locks) + self.assertIs(wr(), None) + + def test_all_locks(self): + support.gc_collect() + self.assertEqual(0, len(_bootstrap._module_locks)) + + + at support.reap_threads +def test_main(): + support.run_unittest(ModuleLockAsRLockTests, + DeadlockAvoidanceTests, + LifetimeTests) + + +if __name__ == '__main__': + test_main() diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -167,7 +167,7 @@ if name in {'__builtins__', '__doc__', '__file__', '__path__', '__module__', '__name__', '__slots__', '__package__', '__cached__', '__author__', '__credits__', '__date__', - '__version__', '__qualname__'}: + '__version__', '__qualname__', '__initializing__'}: return 0 # Private names are hidden, but special names are displayed. if name.startswith('__') and name.endswith('__'): return 1 diff --git a/Lib/test/lock_tests.py b/Lib/test/lock_tests.py --- a/Lib/test/lock_tests.py +++ b/Lib/test/lock_tests.py @@ -247,7 +247,6 @@ # Cannot release an unacquired lock lock = self.locktype() self.assertRaises(RuntimeError, lock.release) - self.assertRaises(RuntimeError, lock._release_save) lock.acquire() lock.acquire() lock.release() @@ -255,6 +254,17 @@ lock.release() lock.release() self.assertRaises(RuntimeError, lock.release) + + def test_release_save_unacquired(self): + # Cannot _release_save an unacquired lock + lock = self.locktype() + self.assertRaises(RuntimeError, lock._release_save) + lock.acquire() + lock.acquire() + lock.release() + lock.acquire() + lock.release() + lock.release() self.assertRaises(RuntimeError, lock._release_save) def test_different_thread(self): diff --git a/Lib/test/test_pkg.py b/Lib/test/test_pkg.py --- a/Lib/test/test_pkg.py +++ b/Lib/test/test_pkg.py @@ -23,6 +23,8 @@ def fixdir(lst): if "__builtins__" in lst: lst.remove("__builtins__") + if "__initializing__" in lst: + lst.remove("__initializing__") return lst diff --git a/Lib/test/test_threaded_import.py b/Lib/test/test_threaded_import.py --- a/Lib/test/test_threaded_import.py +++ b/Lib/test/test_threaded_import.py @@ -12,7 +12,7 @@ import shutil import unittest from test.support import ( - verbose, import_module, run_unittest, TESTFN, reap_threads) + verbose, import_module, run_unittest, TESTFN, reap_threads, forget) threading = import_module('threading') def task(N, done, done_tasks, errors): @@ -187,7 +187,7 @@ contents = contents % {'delay': delay} with open(os.path.join(TESTFN, name + ".py"), "wb") as f: f.write(contents.encode('utf-8')) - self.addCleanup(sys.modules.pop, name, None) + self.addCleanup(forget, name) results = [] def import_ab(): @@ -204,6 +204,21 @@ t2.join() self.assertEqual(set(results), {'a', 'b'}) + def test_side_effect_import(self): + code = """if 1: + import threading + def target(): + import random + t = threading.Thread(target=target) + t.start() + t.join()""" + sys.path.insert(0, os.curdir) + self.addCleanup(sys.path.remove, os.curdir) + with open(TESTFN + ".py", "wb") as f: + f.write(code.encode('utf-8')) + self.addCleanup(forget, TESTFN) + __import__(TESTFN) + @reap_threads def test_main(): diff --git a/Lib/token.py b/Lib/token.py --- a/Lib/token.py +++ b/Lib/token.py @@ -70,7 +70,7 @@ tok_name = {value: name for name, value in globals().items() - if isinstance(value, int)} + if isinstance(value, int) and not name.startswith('_')} __all__.extend(tok_name.values()) def ISTERMINAL(x): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #9260: A finer-grained import lock. Most of the import sequence + now uses per-module locks rather than the global import lock, eliminating + well-known issues with threads and imports. + - Issue #14624: UTF-16 decoding is now 3x to 4x faster on various inputs. Patch by Serhiy Storchaka. diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -1370,47 +1370,7 @@ PyObject * PyImport_ImportModuleNoBlock(const char *name) { - PyObject *nameobj, *modules, *result; -#ifdef WITH_THREAD - long me; -#endif - - /* Try to get the module from sys.modules[name] */ - modules = PyImport_GetModuleDict(); - if (modules == NULL) - return NULL; - - nameobj = PyUnicode_FromString(name); - if (nameobj == NULL) - return NULL; - result = PyDict_GetItem(modules, nameobj); - if (result != NULL) { - Py_DECREF(nameobj); - Py_INCREF(result); - return result; - } - PyErr_Clear(); -#ifdef WITH_THREAD - /* check the import lock - * me might be -1 but I ignore the error here, the lock function - * takes care of the problem */ - me = PyThread_get_thread_ident(); - if (import_lock_thread == -1 || import_lock_thread == me) { - /* no thread or me is holding the lock */ - result = PyImport_Import(nameobj); - } - else { - PyErr_Format(PyExc_ImportError, - "Failed to import %R because the import lock" - "is held by another thread.", - nameobj); - result = NULL; - } -#else - result = PyImport_Import(nameobj); -#endif - Py_DECREF(nameobj); - return result; + return PyImport_ImportModule(name); } @@ -1420,11 +1380,13 @@ int level) { _Py_IDENTIFIER(__import__); + _Py_IDENTIFIER(__initializing__); _Py_IDENTIFIER(__package__); _Py_IDENTIFIER(__path__); _Py_IDENTIFIER(__name__); _Py_IDENTIFIER(_find_and_load); _Py_IDENTIFIER(_handle_fromlist); + _Py_IDENTIFIER(_lock_unlock_module); _Py_static_string(single_dot, "."); PyObject *abs_name = NULL; PyObject *builtins_import = NULL; @@ -1607,16 +1569,48 @@ goto error_with_unlock; } else if (mod != NULL) { + PyObject *value; + int initializing = 0; + Py_INCREF(mod); + /* Only call _bootstrap._lock_unlock_module() if __initializing__ is true. */ + value = _PyObject_GetAttrId(mod, &PyId___initializing__); + if (value == NULL) + PyErr_Clear(); + else { + initializing = PyObject_IsTrue(value); + Py_DECREF(value); + if (initializing == -1) + PyErr_Clear(); + } + if (initializing > 0) { + /* _bootstrap._lock_unlock_module() releases the import lock */ + value = _PyObject_CallMethodObjIdArgs(interp->importlib, + &PyId__lock_unlock_module, abs_name, + NULL); + if (value == NULL) + goto error; + Py_DECREF(value); + } + else { +#ifdef WITH_THREAD + if (_PyImport_ReleaseLock() < 0) { + PyErr_SetString(PyExc_RuntimeError, "not holding the import lock"); + goto error; + } +#endif + } } else { + /* _bootstrap._find_and_load() releases the import lock */ mod = _PyObject_CallMethodObjIdArgs(interp->importlib, &PyId__find_and_load, abs_name, builtins_import, NULL); if (mod == NULL) { - goto error_with_unlock; + goto error; } } + /* From now on we don't hold the import lock anymore. */ if (PyObject_Not(fromlist)) { if (level == 0 || PyUnicode_GET_LENGTH(name) > 0) { @@ -1625,12 +1619,12 @@ PyObject *borrowed_dot = _PyUnicode_FromId(&single_dot); if (borrowed_dot == NULL) { - goto error_with_unlock; + goto error; } partition = PyUnicode_Partition(name, borrowed_dot); if (partition == NULL) { - goto error_with_unlock; + goto error; } if (PyUnicode_GET_LENGTH(PyTuple_GET_ITEM(partition, 1)) == 0) { @@ -1638,7 +1632,7 @@ Py_DECREF(partition); final_mod = mod; Py_INCREF(mod); - goto exit_with_unlock; + goto error; } front = PyTuple_GET_ITEM(partition, 0); @@ -1657,7 +1651,7 @@ abs_name_len - cut_off); Py_DECREF(front); if (to_return == NULL) { - goto error_with_unlock; + goto error; } final_mod = PyDict_GetItem(interp->modules, to_return); @@ -1683,8 +1677,8 @@ fromlist, builtins_import, NULL); } + goto error; - exit_with_unlock: error_with_unlock: #ifdef WITH_THREAD if (_PyImport_ReleaseLock() < 0) { diff --git a/Python/importlib.h b/Python/importlib.h index cf5619a6c4b0587815b87145eae5bf212ec7e5f1..493cd1aac17510f62321955f9d7a45836ef23a7d GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 19:03:05 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 May 2012 19:03:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_import_lock-related_fu?= =?utf8?q?nctions_to_a_separate_doc_section=2E?= Message-ID: http://hg.python.org/cpython/rev/31ac156000c8 changeset: 77014:31ac156000c8 user: Antoine Pitrou date: Thu May 17 19:00:35 2012 +0200 summary: Move import lock-related functions to a separate doc section. files: Doc/library/imp.rst | 85 ++++++++++++++++---------------- 1 files changed, 43 insertions(+), 42 deletions(-) diff --git a/Doc/library/imp.rst b/Doc/library/imp.rst --- a/Doc/library/imp.rst +++ b/Doc/library/imp.rst @@ -107,48 +107,6 @@ in ``sys.modules``. -.. function:: lock_held() - - Return ``True`` if the import lock is currently held, else ``False``. On - platforms without threads, always return ``False``. - - On platforms with threads, a thread executing an import first holds a - global import lock, then sets up a per-module lock for the rest of the - import. This blocks other threads from importing the same module until - the original import completes, preventing other threads from seeing - incomplete module objects constructed by the original thread. An - exception is made for circular imports, which by construction have to - expose an incomplete module object at some point. - - .. note:: - Locking semantics of imports are an implementation detail which may - vary from release to release. However, Python ensures that circular - imports work without any deadlocks. - - .. versionchanged:: 3.3 - In Python 3.3, the locking scheme has changed to per-module locks for - the most part. - - -.. function:: acquire_lock() - - Acquire the interpreter's global import lock for the current thread. - This lock should be used by import hooks to ensure thread-safety when - importing modules. - - Once a thread has acquired the import lock, the same thread may acquire it - again without blocking; the thread must release it once for each time it has - acquired it. - - On platforms without threads, this function does nothing. - - -.. function:: release_lock() - - Release the interpreter's global import lock. On platforms without - threads, this function does nothing. - - .. function:: reload(module) Reload a previously imported *module*. The argument must be a module object, so @@ -246,6 +204,49 @@ magic number, as returned by :func:`get_magic`. +The following functions help interact with the import system's internal +locking mechanism. Locking semantics of imports are an implementation +detail which may vary from release to release. However, Python ensures +that circular imports work without any deadlocks. + +.. versionchanged:: 3.3 + In Python 3.3, the locking scheme has changed to per-module locks for + the most part. A global import lock is kept for some critical tasks, + such as initializing the per-module locks. + + +.. function:: lock_held() + + Return ``True`` if the global import lock is currently held, else + ``False``. On platforms without threads, always return ``False``. + + On platforms with threads, a thread executing an import first holds a + global import lock, then sets up a per-module lock for the rest of the + import. This blocks other threads from importing the same module until + the original import completes, preventing other threads from seeing + incomplete module objects constructed by the original thread. An + exception is made for circular imports, which by construction have to + expose an incomplete module object at some point. + +.. function:: acquire_lock() + + Acquire the interpreter's global import lock for the current thread. + This lock should be used by import hooks to ensure thread-safety when + importing modules. + + Once a thread has acquired the import lock, the same thread may acquire it + again without blocking; the thread must release it once for each time it has + acquired it. + + On platforms without threads, this function does nothing. + + +.. function:: release_lock() + + Release the interpreter's global import lock. On platforms without + threads, this function does nothing. + + The following constants with integer values, defined in this module, are used to indicate the search result of :func:`find_module`. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 19:53:34 2012 From: python-checkins at python.org (ross.lagerwall) Date: Thu, 17 May 2012 19:53:34 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313031=3A_Small_spe?= =?utf8?q?ed-up_for_tarfile_when_unzipping_tarfiles=2E?= Message-ID: http://hg.python.org/cpython/rev/c62fa6892424 changeset: 77015:c62fa6892424 user: Ross Lagerwall date: Thu May 17 19:49:27 2012 +0200 summary: Issue #13031: Small speed-up for tarfile when unzipping tarfiles. Patch by Justin Peel. files: Lib/tarfile.py | 4 ++-- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -245,8 +245,8 @@ the high bit set. So we calculate two checksums, unsigned and signed. """ - unsigned_chksum = 256 + sum(struct.unpack("148B", buf[:148]) + struct.unpack("356B", buf[156:512])) - signed_chksum = 256 + sum(struct.unpack("148b", buf[:148]) + struct.unpack("356b", buf[156:512])) + unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf)) + signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf)) return unsigned_chksum, signed_chksum def copyfileobj(src, dst, length=None): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -776,6 +776,7 @@ Joe Peterson Randy Pausch Samuele Pedroni +Justin Peel Marcel van der Peijl Berker Peksag Steven Pemberton diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ Library ------- +- Issue #13031: Small speed-up for tarfile when unzipping tarfiles. + Patch by Justin Peel. + - Issue #14780: urllib.request.urlopen() now has a ``cadefault`` argument to use the default certificate store. Initial patch by James Oakley. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 21:07:19 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 May 2012 21:07:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_the_test_completely_cl?= =?utf8?q?ean_up_after_itself=2E?= Message-ID: http://hg.python.org/cpython/rev/423a26e8ffbb changeset: 77016:423a26e8ffbb parent: 77014:31ac156000c8 user: Antoine Pitrou date: Thu May 17 21:02:54 2012 +0200 summary: Make the test completely clean up after itself. files: Lib/test/test_threaded_import.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_threaded_import.py b/Lib/test/test_threaded_import.py --- a/Lib/test/test_threaded_import.py +++ b/Lib/test/test_threaded_import.py @@ -12,7 +12,7 @@ import shutil import unittest from test.support import ( - verbose, import_module, run_unittest, TESTFN, reap_threads, forget) + verbose, import_module, run_unittest, TESTFN, reap_threads, forget, unlink) threading = import_module('threading') def task(N, done, done_tasks, errors): @@ -214,8 +214,10 @@ t.join()""" sys.path.insert(0, os.curdir) self.addCleanup(sys.path.remove, os.curdir) - with open(TESTFN + ".py", "wb") as f: + filename = TESTFN + ".py" + with open(filename, "wb") as f: f.write(code.encode('utf-8')) + self.addCleanup(unlink, filename) self.addCleanup(forget, TESTFN) __import__(TESTFN) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 21:07:19 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 May 2012 21:07:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/8e3425396912 changeset: 77017:8e3425396912 parent: 77016:423a26e8ffbb parent: 77015:c62fa6892424 user: Antoine Pitrou date: Thu May 17 21:04:49 2012 +0200 summary: Merge files: Lib/tarfile.py | 4 ++-- Misc/ACKS | 1 + Misc/NEWS | 3 +++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/tarfile.py b/Lib/tarfile.py --- a/Lib/tarfile.py +++ b/Lib/tarfile.py @@ -245,8 +245,8 @@ the high bit set. So we calculate two checksums, unsigned and signed. """ - unsigned_chksum = 256 + sum(struct.unpack("148B", buf[:148]) + struct.unpack("356B", buf[156:512])) - signed_chksum = 256 + sum(struct.unpack("148b", buf[:148]) + struct.unpack("356b", buf[156:512])) + unsigned_chksum = 256 + sum(struct.unpack_from("148B8x356B", buf)) + signed_chksum = 256 + sum(struct.unpack_from("148b8x356b", buf)) return unsigned_chksum, signed_chksum def copyfileobj(src, dst, length=None): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -776,6 +776,7 @@ Joe Peterson Randy Pausch Samuele Pedroni +Justin Peel Marcel van der Peijl Berker Peksag Steven Pemberton diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ Library ------- +- Issue #13031: Small speed-up for tarfile when unzipping tarfiles. + Patch by Justin Peel. + - Issue #14780: urllib.request.urlopen() now has a ``cadefault`` argument to use the default certificate store. Initial patch by James Oakley. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 17 21:16:12 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 17 May 2012 21:16:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_a_mention_of_the_new_im?= =?utf8?q?port_locks_in_whatsnew=2E?= Message-ID: http://hg.python.org/cpython/rev/364289cc7891 changeset: 77018:364289cc7891 user: Antoine Pitrou date: Thu May 17 21:13:45 2012 +0200 summary: Add a mention of the new import locks in whatsnew. files: Doc/whatsnew/3.3.rst | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -573,6 +573,23 @@ .. XXX mention new error messages for passing wrong number of arguments to functions +A Finer-Grained Import Lock +=========================== + +Previous versions of CPython have always relied on a global import lock. +This led to unexpected annoyances, such as deadlocks when importing a module +would trigger code execution in a different thread as a side-effect. +Clumsy workarounds were sometimes employed, such as the +:c:func:`PyImport_ImportModuleNoBlock` C API function. + +In Python 3.3, importing a module takes a per-module lock. This correctly +serializes importation of a given module from multiple threads (preventing +the exposure of incompletely initialized modules), while eliminating the +aforementioned annoyances. + +(contributed by Antoine Pitrou in :issue:`9260`.) + + New and Improved Modules ======================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 01:42:14 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 18 May 2012 01:42:14 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Remove_inaccurate_sections_on_?= =?utf8?q?Tkinter_and_OS_X_Framework_builds=2E?= Message-ID: http://hg.python.org/peps/rev/33d229040efa changeset: 4403:33d229040efa parent: 4394:51369a92dd8a user: Carl Meyer date: Wed May 16 09:03:41 2012 -0600 summary: Remove inaccurate sections on Tkinter and OS X Framework builds. files: pep-0405.txt | 12 ------------ 1 files changed, 0 insertions(+), 12 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -482,20 +482,6 @@ really because there's no supporting concept in ``Python/sysconfig``. -OS X Framework builds ---------------------- - -There have been some reports that the reference implementation does -not work on an OS X framework build of Python, but it seems to work -for us. This needs further investigation. - - -tkinter -------- - -Tkinter apps currently do not work within a virtual environment. - - Reference Implementation ======================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 18 01:42:15 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 18 May 2012 01:42:15 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Update_link_to_reference_imple?= =?utf8?q?mentation=2E?= Message-ID: http://hg.python.org/peps/rev/17b8403f1c73 changeset: 4404:17b8403f1c73 user: Carl Meyer date: Wed May 16 09:04:05 2012 -0600 summary: Update link to reference implementation. files: pep-0405.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -491,7 +491,7 @@ installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create a virtual environment. -.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv +.. _a clone of the CPython Mercurial repository: http://hg.python.org/sandbox/vsajip#venv Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 18 01:42:16 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 18 May 2012 01:42:16 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merge_from_upstream=2E?= Message-ID: http://hg.python.org/peps/rev/8f3f0d075adc changeset: 4405:8f3f0d075adc parent: 4404:17b8403f1c73 parent: 4397:3cb12850bd6d user: Carl Meyer date: Wed May 16 09:04:22 2012 -0600 summary: Merge from upstream. files: pep-0409.txt | 5 + pep-0415.txt | 26 +- pep-0420.txt | 170 +++++++++++--- pep-0421.txt | 425 ++++++++++++++++++++------------------ pep-3144.txt | 43 +++- 5 files changed, 410 insertions(+), 259 deletions(-) diff --git a/pep-0409.txt b/pep-0409.txt --- a/pep-0409.txt +++ b/pep-0409.txt @@ -8,6 +8,7 @@ Content-Type: text/x-rst Created: 26-Jan-2012 Post-History: 30-Aug-2002, 01-Feb-2012, 03-Feb-2012 +Superseded-By: 415 Resolution: http://mail.python.org/pipermail/python-dev/2012-February/116136.html @@ -87,6 +88,10 @@ Implementation Discussion ========================= +Note: after acceptance of this PEP, a cleaner implementation mechanism +was proposed and accepted in PEP 415. Refer to that PEP for more +details on the implementation actually used in Python 3.3. + Currently, ``None`` is the default for both ``__context__`` and ``__cause__``. In order to support ``raise ... from None`` (which would set ``__cause__`` to ``None``) we need a different default value for ``__cause__``. Several ideas diff --git a/pep-0415.txt b/pep-0415.txt --- a/pep-0415.txt +++ b/pep-0415.txt @@ -1,24 +1,35 @@ PEP: 415 -Title: Implementing PEP 409 differently +Title: Implement context suppression with exception attributes Version: $Revision$ Last-Modified: $Date$ Author: Benjamin Peterson BDFL-Delegate: Nick Coghlan -Status: Draft +Status: Final Type: Standards Track Content-Type: text/x-rst Created: 26-Feb-2012 Python-Version: 3.3 Post-History: 26-Feb-2012 +Replaces: 409 +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119467.html Abstract ======== -PEP 409 allows PEP 3134 exception contexts and causes to be suppressed when the -exception is printed. This is done using the ``raise exc from None`` -syntax. This PEP proposes to implement context and cause suppression -differently. +PEP 409 introduced support for the ``raise exc from None`` construct to +allow the display of the exception context to be explicitly suppressed. +This PEP retains the language level changes already implemented in PEP 409, +but replaces the underlying implementation mechanism with a simpler approach +based on a new ``__suppress_context__`` attribute on all ``BaseException`` +instances. + + +PEP Acceptance +============== + +This PEP was accepted by Nick Coghlan on the 14th of May, 2012. + Rationale ========= @@ -40,6 +51,7 @@ ``__cause__`` should be set to ``Ellipsis``. Using ``Ellipsis`` by default for ``__cause__`` makes it asymmetrical with ``__context__``. + Proposal ======== @@ -62,6 +74,7 @@ where ``exc.__cause__ = cause`` implicitly sets ``exc.__suppress_context__``. + Patches ======= @@ -74,6 +87,7 @@ .. _issue 14133: http://bugs.python.org/issue14133 + Copyright ========= diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -34,10 +34,12 @@ * "vendor package" refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). + * "regular package" refers to packages as they are implemented in + Python 3.2 and earlier. * "portion" refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. - * "regular package" refers to packages as they are implemented in - Python 3.2 and earlier. + * "legacy portion" refers to a portion that uses ``__path__`` + manipulation in order to implement namespace packages. This PEP defines a new type of package, the "namespace package". @@ -76,6 +78,10 @@ ``setup.py``, so that distribution developers don't need to put the magic ``__path__`` modification into ``__init__.py`` themselves. +See PEP 402's "The Problem" section [2]_ for more details on the +motivation for namespace packages. Note that PEP 402 has been +rejected, but the motivating use cases are still valid. + Rationale ========= @@ -135,48 +141,115 @@ least one directory was recorded, then a namespace package is created. The new namespace package: - * Has a ``__path__`` attribute set to the list of directories that - were found and recorded during the scan. + * Has a ``__path__`` attribute set to an iterable of the path strings + that were found and recorded during the scan. * Does not have a ``__file__`` attribute. -There is no mechanism to automatically recompute the ``__path__`` if -``sys.path`` is altered after a namespace package has already been -created. However, existing namespace utilities (like -``pkgutil.extend_path``) can be used to update them explicitly if -desired. - Note that if "import foo" is executed and "foo" is found as a namespace package (using the above rules), then "foo" is immediately created as a package. The creation of the namespace package is not deferred until a sub-level import occurs. A namespace package is not fundamentally different from a regular -package. It is just a different way of creating packages. Once a +package. It is just a different way of creating packages. Once a namespace package is created, there is no functional difference -between it and a regular package. The only observable difference is -that the namespace package's ``__file__`` attribute will end with a -path separator (typically a slash or backslash, depending on the -platform). +between it and a regular package. -Impact on Import Finders and Loaders +Dynamic path computation +------------------------ + +A namespace package's ``__path__`` will be recomputed if the value of +the parent path changes. In order for this feature to work, the parent +path must be modified in-place, not replaced with a new object. For +example, for top-level namespace packages, this will work:: + + sys.path.append('new-dir') + +But this will not:: + + sys.path = sys.path + ['new-dir'] + +Impact on import finders and loaders ------------------------------------ PEP 302 defines "finders" that are called to search path elements. -These finders' ``find_module`` methods currently return either a -"loader" object or None. For a finder to contribute to namespace -packages, ``find_module`` will return a third type: a string. This is -the string that will be recorded and later used as a component of the -namespace module's ``__path__``, as described above. This string must -not contain a trailing path separator. +These finders' ``find_module`` methods return either a "loader" object +or ``None``. + +For a finder to contribute to namespace packages, it must implement a +new ``find_loader(fullname)`` method. ``fullname`` has the same +meaning as for ``find_module``. ``find_loader`` always returns a +2-tuple of ``(loader, )``. ``loader`` may +be ``None``, in which case ```` (which may +be empty) is added to the list of recorded path entries and path +searching continues. If ``loader`` is not ``None``, it is immediately +used to load a module or regular package. + +Even if ``loader`` is returned and is not ``None``, +```` must still contain the path entries for +the package. This allows code such as ``pkgutil.extend_path()`` to +compute path entries for packages that it does not load. + +Note that multiple path entries per finder are allowed. This is to +support the case where a finder discovers multiple namespace portions +for a given ``fullname``. Many finders will support only a single +namespace package portion per ``find_loader`` call, in which case this +iterable will contain only a single string. + +The import machinery will call ``find_loader`` if it exists, else fall +back to ``find_module``. Legacy finders which implement +``find_module`` but not ``find_loader`` will be unable to contribute +portions to a namespace package. The specification expands PEP 302 loaders to include an optional method called ``module_repr()`` which if present, is used to generate module object reprs. See the section below for further details. -If an existing finder is not updated to support returning a string -from ``find_module``, the only impact is that such a finder will be -unable to provide portions of a namespace package. +Differences between namespace packages and regular packages +----------------------------------------------------------- + +Namespace packages and regular packages are very similar. The +differences are: + + * Portions of namespace packages need not all come from the same + directory structure, or even from the same loader. Regular packages + are self-contained: all parts live in the same directory hierarchy. + + * Namespace packages have no ``__file__`` attribute. + + * Namespace packages' ``__path__`` attribute is a read-only iterable + of strings, which is automatically updated when the parent path is + modified. + + * Namespace packages have no ``__init__.py`` module. + + * Namespace packages have a different type of object for their + ``__loader__`` attribute. + + +Namespace packages in the standard library +------------------------------------------ + +It is possible, and this PEP explicitly allows, that parts of the +standard library be implemented as namespace packages. When and if +any standard library packages become namespace packages is outside the +scope of this PEP. + + +Migrating from legacy namespace packages +---------------------------------------- + +As described above, prior to this PEP ``pkgutil.extend_path()`` was +used by legacy portions to create namespace packages. Because it is +likely not practical for all existing portions of a namespace package +to be migrated to this PEP at once, ``extend_path()`` will be modified +to also recognize PEP 420 namespace packages. This will allow some +portions of a namespace to be legacy portions while others are +migrated to PEP 420. These hybrid namespace packages will not have +the dynamic path computation that normal namespace packages have, +since ``extend_path()`` never provided this functionality in the past. + Packaging Implications ====================== @@ -201,7 +274,7 @@ "foo" directories would be in directories that are on ``sys.path``. "foo/bar" would be in one of these sys.path entries, and "foo/baz" would be in the other. Upon removal of "foo.bar", the "foo/bar" and -corresponding "foo" directories can be completely removed. But +corresponding "foo" directories can be completely removed. But "foo/baz" and its corresponding "foo" directory cannot be removed. It is also possible to have the "foo.bar" portion installed in a @@ -212,7 +285,7 @@ ========== At PyCon 2012, we had a discussion about namespace packages at which -PEP 382 and PEP 402 were rejected, to be replaced by this PEP [2]_. +PEP 382 and PEP 402 were rejected, to be replaced by this PEP [3]_. There is no intention to remove support of regular packages. If a developer knows that her package will never be a portion of a @@ -227,7 +300,7 @@ imported as a namespace package, whereas in prior Python versions an ImportWarning would be raised. -Nick Coghlan presented a list of his objections to this proposal [3]_. +Nick Coghlan presented a list of his objections to this proposal [4]_. They are: 1. Implicit package directories go against the Zen of Python. @@ -239,9 +312,9 @@ layouts. 4. Implicit package directories will permanently entrench current - newbie-hostile behaviour in ``__main__``. + newbie-hostile behavior in ``__main__``. -Nick gave a detailed response [4]_, which is summarized here: +Nick gave a detailed response [5]_, which is summarized here: 1. The practicality of this PEP wins over other proposals and the status quo. @@ -253,11 +326,25 @@ 4. This will also be addressed in PEP 395. -Phillip Eby asked about auto-updating of ``__path__``, instead of it -being a simple list [5]_. It is the intent of this PEP to get the -simplest possible solution working. It will be possible at a later -date to add such features. Several possible ways to do so were -discussed in the referenced email thread. +``find_module`` versus ``find_loader`` +-------------------------------------- + +An early draft of this PEP specified a change to the ``find_module`` +method in order to support namespace packages. It would be modified +to return a string in the case where a namespace package portion was +discovered. + +However, this caused a problem with existing code outside of the +standard library which calls ``find_module``. Because this code would +not be upgraded in concert with changes required by this PEP, it would +fail when it would receive unexpected return values from +``find_module``. Because of this incompatibility, this PEP now +specifies that finders that want to provide namespace portions must +implement the ``find_loader`` method, described above. + +The use case for supporting multiple portions per ``find_loader`` call +is given in [6]_. + Module reprs ============ @@ -304,17 +391,20 @@ .. [1] PEP 420 branch (http://hg.python.org/features/pep-420) -.. [2] PyCon 2012 Namespace Package discussion outcome +.. [2] PEP 402's description of use cases for namespace packages + (http://www.python.org/dev/peps/pep-0402/#the-problem) + +.. [3] PyCon 2012 Namespace Package discussion outcome (http://mail.python.org/pipermail/import-sig/2012-March/000421.html) -.. [3] Nick Coghlan's objection to the lack of marker files or directories +.. [4] Nick Coghlan's objection to the lack of marker files or directories (http://mail.python.org/pipermail/import-sig/2012-March/000423.html) -.. [4] Nick Coghlan's response to his initial objections +.. [5] Nick Coghlan's response to his initial objections (http://mail.python.org/pipermail/import-sig/2012-April/000464.html) -.. [5] Phillip Eby's question about auto-updating __path__ - (http://mail.python.org/pipermail/import-sig/2012-April/000468.html) +.. [6] Use case for multiple portions per ``find_loader`` call + (http://mail.python.org/pipermail/import-sig/2012-May/000585.html) Copyright ========= diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -34,7 +34,7 @@ viable alternate implementations of Python. Consider, however, the nearly two decades of CPython-centric Python -(i.e. most of its existence). That focus had understandably +(i.e. most of its existence). That focus has understandably contributed to quite a few CPython-specific artifacts both in the standard library and exposed in the interpreter. Though the core developers have made an effort in recent years to address this, quite @@ -51,41 +51,50 @@ ======== We will add a new attribute to the ``sys`` module, called -``sys.implementation``, as an instance of a new type to contain -implementation-specific information. +``sys.implementation``, as an object with attribute-access (as opposed +to a mapping). It will contain contain implementation-specific +information. The attributes of this object will remain fixed during interpreter execution and through the course of an implementation version. This ensures behaviors don't change between versions which depend on -variables in ``sys.implementation``. +attributes of ``sys.implementation``. -The object will have each of the attributes described in the `Required -Variables`_ section below. Any other per-implementation values may be -stored in ``sys.implementation.metadata``. However, nothing in the -standard library will rely on ``sys.implementation.metadata``. -Examples of possible metadata values are described in the `Example -Metadata Values`_ section. +The object has each of the attributes described in the `Required +Attributes`_ section below. Those attribute names will never start +with an underscore. The standard library and the language definition +will rely only on those required attributes. -This proposal takes a conservative approach in requiring only four -variables. As more become appropriate, they may be added with -discretion. +This proposal takes a conservative approach in requiring only a small +number of attributes. As more become appropriate, they may be added +with discretion, as described in `Adding New Required Attributes`_. +While this PEP places no other constraints on ``sys.implementation``, +it also recommends that no one rely on capabilities outside those +described here. The only exception to that recommendation is for +attributes starting with an underscore. Implementors may use those +as appropriate to store per-implementation data. -Required Variables ------------------- -These are variables in ``sys.implementation`` on which the standard -library would rely, with the exception of ``metadata``, meaning -implementers must define them: +Required Attributes +------------------- + +These are attributes in ``sys.implementation`` on which the standard +library and language definition will rely, meaning implementers must +define them: **name** - This is the common name of the implementation (case sensitive). - Examples include 'PyPy', 'Jython', 'IronPython', and 'CPython'. + A lower-case identifer representing the implementation. Examples + include 'pypy', 'jython', 'ironpython', and 'cpython'. **version** - This is the version of the implementation, as opposed to the - version of the language it implements. This value conforms to the - format described in `Version Format`_. + The version of the implementation, as opposed to the version of the + language it implements. This value conforms to the format described + in `Version Format`_. + +**hexversion** + The version of the implementation in the same hexadecimal format as + ``sys.hexversion``. **cache_tag** A string used for the PEP 3147 cache tag [#cachetag]_. It would @@ -94,17 +103,21 @@ different cache tag. If ``cache_tag`` is set to None, it indicates that module caching should be disabled. -**metadata** - Any other values that an implementation wishes to specify, - particularly informational ones. Neither the standard library nor - the language specification will rely on implementation metadata. - Also see the list of `Example Metadata Values`_. - Adding New Required Attributes ------------------------------ -XXX PEP? something lighter? +In time more required attributes will be added to +``sys.implementation``. However, each must have a meaningful use case +across all Python implementations in order to be considered. This is +made most clear by a use case in the standard library or language +specification. + +All proposals for new required attributes will go through the normal +PEP process. Such a PEP need not be long, just long enough. It will +need to sufficiently spell out the rationale for the new attribute, +its use cases, and the impact it will have on the various Python +implemenations. Version Format @@ -112,59 +125,14 @@ A main point of ``sys.implementation`` is to contain information that will be used internally in the standard library. In order to -facilitate the usefulness of a version variable, its value should be -in a consistent format across implementations. +facilitate the usefulness of the version attribute, its value should +be in a consistent format across implementations. -As such, the format of ``sys.implementation.version`` must follow that +As such, the format of ``sys.implementation.version`` will follow that of ``sys.version_info``, which is effectively a named tuple. It is a familiar format and generally consistent with normal version format conventions. -XXX The following is not exactly true: - -Keep in mind, however, that ``sys.implementation.version`` is the -version of the Python *implementation*, while ``sys.version_info`` -(and friends) is the version of the Python language. - - -Example Metadata Values ------------------------ - -These are the sorts of values an implementation may put into -``sys.implementation.metadata``. However, these names and -descriptions are only examples and are not being proposed here. If -they later have meaningful uses cases, they can be added by following -the process described in `Adding New Required Attributes`_. - -**vcs_url** - The URL pointing to the main VCS repository for the implementation - project. - -**vcs_revision_id** - A value that identifies the VCS revision of the implementation that - is currently running. - -**build_toolchain** - Identifies the tools used to build the interpreter. - -**build_date** - The timestamp of when the interpreter was built. - -**homepage** - The URL of the implementation's website. - -**site_prefix** - The preferred site prefix for this implementation. - -**runtime** - The run-time environment in which the interpreter is running, as - in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* - Executable". - -**gc_type** - The type of garbage collection used, like "reference counting" or - "mark and sweep". - Rationale ========= @@ -179,87 +147,41 @@ makes explicit that which was implicit. -Why a Custom Type? ------------------- +Type Considerations +------------------- -A dedicated class, of which ``sys.implementation`` is an instance, would -facilitate the dotted access of a "named" tuple. At the same time, it -allows us to avoid the problems of the other approaches (see below), -like confusion about ordering and iteration. +It's very easy to get bogged down in discussions about the type of +``sys.implementation``. However, its purpose is to support the +standard library and language definition. As such, there isn't much +that really matters regarding its type, as opposed to a feature that +would be more generally used. Thus characteristics like immutability +and sequence-ness have been disregarded. -The alternatives to a dictionary are considered separately here: +The only real choice has been between an object with attribute access +and a mapping with item access. This PEP espouses dotted access to +reflect the relatively fixed nature of the namespace. -**Dictionary** -A dictionary reflects a simple namespace with item access. It -maps names to values and that's all. It also reflects the more variable -nature of ``sys.implementation``. +Non-Required Attributes +----------------------- -However, a simple dictionary does not set expectations very well about -the nature of ``sys.implementation``. The custom type approach, with -a fixed set of required attributes, does a better job of this. +Earlier versions of this PEP included a required attribute called +``metadata`` that held any non-required, per-implementation data +[#Nick]_. However, this proved to be an unnecessary addition +considering the purpose of ``sys.implementation``. -**Named Tuple** - -Another close alternative is a namedtuple or a structseq or some other -tuple type with dotted access (a la ``sys.version_info``). This type -is immutable and simple. It is a well established pattern for -implementation-specific variables in Python. Dotted access on a -namespace is also very convenient. - -Fallback lookup may favor dicts:: - - cache_tag = sys.implementation.get('cache_tag') - - vs. - - cache_tag = getattr(sys.implementation.get, 'cache_tag', None) - -However, this is mitigated by having ``sys.implementation.metadata``. - -One problem with using a named tuple is that ``sys.implementation`` does -not have meaning as a sequence. Also, unlike other similar ``sys`` -variables, it has a far greater potential to change over time. - -If a named tuple were used, we'd be very clear in the documentation -that the length and order of the value are not reliable. Iterability -would not be guaranteed. - -**Module** - -Using a module instead of a dict is another option. It has similar -characteristics to an instance, but with a slight hint of immutability -(at least by convention). Such a module could be a stand-alone sub- -module of ``sys`` or added on, like ``os.path``. Unlike a concrete -class, no new type would be necessary. This is a pretty close fit to -what we need. - -The downside is that the module type is much less conducive to -extension, making it more difficult to address the weaknesses of using -an instance of a concrete class. - - -Why metadata? -------------- - -``sys.implementation.metadata`` will hold any optional, strictly- -informational, or per-implementation data. This allows us to restrict -``sys.implementation`` to the required attributes. In that way, its -type can reflect the more stable namespace and -``sys.implementation.metadata`` (as a dict) can reflect the less -certain namespace. - -``sys.implementation.metadata`` is the place an implementation can put -values that must be built-in, "without having to pollute the main sys -namespace" [#Nick]_. +Ultimately, non-required attributes are virtually ignored in this PEP. +They have no impact other than that careless use may collide with +future required attributes. That, however, is but a marginal concern +for ``sys.implementation``. Why a Part of ``sys``? ---------------------- -The ``sys`` module should hold the new namespace because ``sys`` is -the depot for interpreter-centric variables and functions. Many -implementation-specific variables are already found in ``sys``. +The ``sys`` module holds the new namespace because ``sys`` is the depot +for interpreter-centric variables and functions. Many +implementation-specific attributes are already found in ``sys``. Why Strict Constraints on Any of the Values? @@ -269,7 +191,8 @@ ``sys.implementation`` are intended for use by the standard library. Constraining those values, essentially specifying an API for them, allows them to be used consistently, regardless of how they are -otherwise implemented. +otherwise implemented. However, care should be take to not +over-specify the constraints. Discussion @@ -281,6 +204,9 @@ ``imp.get_tag()`` [#revived]_. Discussion has been ongoing [#feedback]_. The messages in `issue #14673`_ are also relevant. +A good part of the recent discussion centered on the type to use for +``sys.implementation``. + Use-cases ========= @@ -290,10 +216,10 @@ "explicit is better than implicit" -The ``platform`` module determines the python implementation by -looking for clues in a couple different ``sys`` variables [#guess]_. -However, this approach is fragile, requiring changes to the standard -library each time an implementation changes. Beyond that, support in +The ``platform`` module determines the python implementation by looking +for clues in a couple different ``sys`` variables [#guess]_. However, +this approach is fragile, requiring changes to the standard library +each time an implementation changes. Beyond that, support in ``platform`` is limited to those implementations that core developers have blessed by special-casing them in the ``platform`` module. @@ -302,7 +228,7 @@ module. Another concern is that the ``platform`` module is part of the stdlib, -which ideally would minimize implementation details such as would be +which ideally should minimize implementation details such as would be moved to ``sys.implementation``. Any overlap between ``sys.implementation`` and the ``platform`` module @@ -359,47 +285,108 @@ treatment of the java environment in the standard library [#os_name]_ [#javatest]_. Unfortunately it masks the os name that would otherwise go there. ``sys.implementation`` would help obviate the need for this -special case. +special case. Currently Jython sets ``os._name`` for the normal +``os.name`` value. -Feedback From Other Python Implementers +The Problem With ``sys.(version|version_info|hexversion)`` +---------------------------------------------------------- + +Earlier versions of this PEP made the mistake of calling +``sys.version_info`` (and friends) the version of the Python language, +in contrast to the implementation. However, this is not the case. +Instead, it is the version of the CPython implementation. Incidently, +the first two components of ``sys.version_info`` (major and minor) also +reflect the version of the language definition. + +As Barry Warsaw noted, the "semantics of sys.version_info have been +sufficiently squishy in the past" [#Barry]_. With +``sys.implementation`` we have the opportunity to improve this +situation by first establishing an explicit location for the version of +the implementation. + +This PEP makes no other effort to directly clarify the semantics of +``sys.version_info``. Regardless, having an explicit version for the +implementation will definitely help to clarify the distinction from the +language version. + + +Feedback From Other Python Implementors ======================================= IronPython ---------- -XXX +Jeff Hardy responded to a request for feedback [#ironpython]_. He +said, "I'll probably add it the day after it's approved" +[#jeff_hardy_2012]_. He also gave useful feedback on both the type of +``sys.implementation`` and on the ``metadata`` attribute (which has +since been removed from the PEP). Jython ------ -XXX +In 2009 Frank Wierzbicki said this (relative to Jython implementing the +required attributes) [#frank_wierzbicki_2009]_:: + + Speaking for Jython, so far it looks like something we would adopt + soonish after it was accepted (it looks pretty useful to me). PyPy ---- -XXX +Some of the PyPy developers have responded to a request for feedback +[#pypy]_. Armin Rigo said the following [#armin_rigo_2012]_:: + + For myself, I can only say that it looks like a good idea, which we + will happily adhere to when we migrate to Python 3.3. + +He also expressed support for keeping the required list small. Both +Armin and Laura Creighton indicated that an effort to better catalog +Python's implementation would be welcome. Such an effort, for which +this PEP is a small start, will be considered separately. Past Efforts ============ -PEP 3139 --------- +``PEP 3139`` +------------ -This PEP from 2008 recommended a clean-up of the ``sys`` module in +PEP 3139, from 2008, recommended a clean-up of the ``sys`` module in part by extracting implementation-specific variables and functions -into a separate module. PEP 421 is a much lighter version of that +into a separate module. PEP 421 is less ambitious version of that idea. While PEP 3139 was rejected, its goals are reflected in PEP 421 to a large extent, though with a much lighter approach. -PEP 399 -------- +``PEP 399`` +----------- -This informational PEP dictates policy regarding the standard library, -helping to make it friendlier to alternate implementations. PEP 421 -is proposed in that same spirit. +PEP 399 dictates policy regarding the standard library, helping to make +it friendlier to alternate implementations. PEP 421 is proposed in +that same spirit. + + +The Bigger Picture +================== + +It's worth noting again that this PEP is a small part of a larger +on-going effort to identify the implementation-specific parts of Python +and mitigate their impact on alternate implementations. + +``sys.implementation`` is a focal point for implementation-specific +data, acting as a nexus for cooperation between the language, the +standard library, and the different implementations. As time goes by +it is feasible that ``sys.implementation`` will assume current +attributes of ``sys`` and other builtin/stdlib modules, where +appropriate. In this way, it is a PEP 3137-lite, but starting as +small as possible. + +However, as already noted, many other efforts predate +``sys.implementation``. Neither is it necessarily a major part of the +effort. Rather, consider it as part of the infrastructure of the +effort to make Python friendler to alternate implementations. Alternatives @@ -409,50 +396,49 @@ straightforward, no alternatives have been considered for this PEP. +Examples of Other Attributes +============================ + +These are examples only and not part of the proposal. Most of them +were suggested during previous discussions, but did not fit into the +goals of this PEP. (See `Adding New Required Attributes`_ if they get +you excited.) + +**common_name** + The case-sensitive name by which the implementation is known. + +**vcs_url** + A URL for the main VCS repository for the implementation project. + +**vcs_revision_id** + A value that identifies the VCS revision of the implementation. + +**build_toolchain** + The tools used to build the interpreter. + +**build_date** + The timestamp of when the interpreter was built. + +**homepage** + The URL of the implementation's website. + +**site_prefix** + The preferred site prefix for the implementation. + +**runtime** + The run-time environment in which the interpreter is running, as + in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime* + Executable". + +**gc_type** + The type of garbage collection used, like "reference counting" or + "mark and sweep". + + Open Issues =========== -* What are the long-term objectives for ``sys.implementation``? - - - possibly pull in implementation details from the main ``sys`` - namespace and elsewhere (PEP 3137 lite). - -* What is the process for introducing new required variables? PEP? - -* Is the ``sys.version_info`` format the right one here? - -* Should ``sys.implementation.hexversion`` be part of the PEP? - -* Does ``sys.(version|version_info|hexversion)`` need to better - reflect the version of the language spec? Micro version, series, - and release seem like implementation-specific values. - -* Alternatives to the approach dictated by this PEP? - -* Do we really want to commit to using a dict for - ``sys.implementation``? - - Backward compatibility issues will make it difficult to change our - minds later. - - The type we use ultimately depends on how general we expect the - consumption of ``sys.implementation`` to be. If its practicality is - oriented toward internal use then the data structure is not as - critical. However, ``sys.implementation`` is intended to have a - non-localized impact across the standard library and the - interpreter. It is better to *not* make hacking it become an - attractive nuisance, regardless of our intentions for usage. - -* Should ``sys.implementation`` and its values be immutable? A benefit - of an immutable type is it communicates that the value is not - expected to change and should not be manipulated. - -* Should ``sys.implementation`` be strictly disallowed to have methods? - Classes often imply the presence (or possibility) of methods, which - may be misleading in this case. - -* Should ``sys.implementation`` implement the collections.abc.Mapping - interface? +Currently none. Implementation @@ -468,11 +454,33 @@ http://mail.python.org/pipermail/python-dev/2009-October/092893.html .. [#revived] The initial 2012 discussion: - http://mail.python.org/pipermail/python-ideas/2012-April/014878.html + http://mail.python.org/pipermail/python-ideas/2012-March/014555.html + (and http://mail.python.org/pipermail/python-ideas/2012-April/014878.html) .. [#feedback] Feedback on the PEP: http://mail.python.org/pipermail/python-ideas/2012-April/014954.html +.. [#ironpython] Feedback from the IronPython developers: + http://mail.python.org/pipermail/ironpython-users/2012-May/015980.html + +.. [#dino_viehland_2009] (2009) Dino Viehland offers his opinion: + http://mail.python.org/pipermail/python-dev/2009-October/092894.html + +.. [#jeff_hardy_2012] (2012) Jeff Hardy offers his opinion: + http://mail.python.org/pipermail/ironpython-users/2012-May/015981.html + +.. [#jython] Feedback from the Jython developers: + ??? + +.. [#frank_wierzbicki_2009] (2009) Frank Wierzbicki offers his opinion: + http://mail.python.org/pipermail/python-dev/2009-October/092974.html + +.. [#pypy] Feedback from the PyPy developers: + http://mail.python.org/pipermail/pypy-dev/2012-May/009883.html + +.. [#armin_rigo_2012] (2012) Armin Rigo offers his opinion: + http://mail.python.org/pipermail/pypy-dev/2012-May/009884.html + .. [#guess] The ``platform`` code which divines the implementation name: http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247 @@ -497,6 +505,9 @@ .. [#Nick] Nick Coghlan's proposal for ``sys.implementation.metadata``: http://mail.python.org/pipermail/python-ideas/2012-May/014984.html +.. [#Barry] Feedback from Barry Warsaw: + http://mail.python.org/pipermail/python-dev/2012-May/119374.html + .. _issue #14673: http://bugs.python.org/issue14673 .. _Lib/test/support.py: http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py diff --git a/pep-3144.txt b/pep-3144.txt --- a/pep-3144.txt +++ b/pep-3144.txt @@ -5,30 +5,36 @@ Author: Peter Moody BDFL-Delegate: Nick Coghlan Discussions-To: -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/plain Created: 6-Feb-2012 Python-Version: 3.3 - +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119474.html Abstract: This PEP proposes a design and for an IP address manipulation module for python. + +PEP Acceptance: + + This PEP was accepted by Nick Coghlan on the 15th of May, 2012. + + Motivation: Several very good IP address modules for python already exist. - The truth is that all of the struggle with the balance between + The truth is that all of them struggle with the balance between adherence to Pythonic principals and the shorthand upon which - network engineers and administrators rely. I believe ipaddr - strikes the right balance. + network engineers and administrators rely. ipaddress aims to + strike the right balance. Rationale: - The existance of several Python IP address manipulation moduels is + The existence of several Python IP address manipulation modules is evidence of an outstanding need for the functionality this module seeks to provide. @@ -55,6 +61,23 @@ * A few attributes were renamed to disambiguate their purpose as well. (eg. network, network_address) + * A number of methods and functions which returned containers in ipaddr now + return iterators. This includes, subnets, address_exclude, + summarize_address_range and collapse_address_list. + + + Due to the backwards incompatible API changes between ipaddress and ipaddr, + the proposal is to add the module using the new provisional API status: + + * http://docs.python.org/dev/glossary.html#term-provisional-package + + + Relevant messages on python-dev: + + * http://mail.python.org/pipermail/python-dev/2012-January/116016.html + * http://mail.python.org/pipermail/python-dev/2012-February/116656.html + * http://mail.python.org/pipermail/python-dev/2012-February/116688.html + Specification: @@ -101,8 +124,7 @@ _BaseV4 - Provides methods and variables (eg, _max_prefixlen) common to all IPv4 classes. - _BaseV6 - Provides methods and variables common to all IPv6 - classes. + _BaseV6 - Provides methods and variables common to all IPv6 classes. Comparisons between objects of differing IP versions results in a TypeError [1]. Additionally, comparisons of objects with @@ -115,7 +137,10 @@ Reference Implementation: The current reference implementation can be found at: - http://code.google.com/p/ipaddr-py/downloads/detail?name=3144.tar.gz + http://code.google.com/p/ipaddress-py/source/browse/ipaddress.py + + Or see the tarball to include the README and unittests. + http://code.google.com/p/ipaddress-py/downloads/detail?name=ipaddress-1.0.tar.gz More information about using the reference implementation can be found at: http://code.google.com/p/ipaddr-py/wiki/Using3144 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 18 01:42:17 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 18 May 2012 01:42:17 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Updates_to_PEP_405=2E?= Message-ID: http://hg.python.org/peps/rev/7b790ef85d26 changeset: 4406:7b790ef85d26 user: Carl Meyer date: Thu May 17 12:41:30 2012 -0600 summary: Updates to PEP 405. files: pep-0405.txt | 108 +++++++++++++++++++++++--------------- 1 files changed, 66 insertions(+), 42 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -222,15 +222,32 @@ those files when Python is run from the venv. +Sysconfig install schemes and user-site +--------------------------------------- + +This approach explicitly chooses not to introduce a new sysconfig +install scheme for venvs. Rather, by modifying ``sys.prefix`` we +ensure that existing install schemes which base locations on +``sys.prefix`` will simply work in a venv. Installation to other +install schemes (for instance, the user-site schemes) whose paths are +not relative to ``sys.prefix``, will not be affected by a venv at all. + +It may be feasible to create an alternative implementation of Python +virtual environments based on a virtual-specific sysconfig scheme, but +it would be less robust, as it would require more code to be aware of +whether it is operating within a virtual environment or not. + + Copies versus symlinks ---------------------- The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs on Windows). -Symlinking is preferable where possible, because in the case of an -upgrade to the underlying Python installation, a Python executable -copied in a venv might become out-of-sync with the installed standard -library and require manual upgrade. +or symlinked Python binary (and other needed DLLs and the ``Include`` +directory on Windows). Symlinking is preferable where possible, +because in the case of an upgrade to the underlying Python +installation, a Python executable copied in a venv might become +out-of-sync with the installed standard library and require manual +upgrade. There are some cross-platform difficulties with symlinks: @@ -251,11 +268,50 @@ never work there, and has no advantages). On Windows, if ``--symlink`` is not used, this means that if the -underlying Python installation is upgraded, the Python binary and DLLs -in the venv should be updated, or there could be issues of mismatch -with the upgraded standard library. The pyvenv script accepts a -``--upgrade`` option for easily performing this upgrade on an existing -venv. +underlying Python installation is upgraded, the Python binary, DLLs, +and include files in the venv should be updated, or there could be +issues of mismatch with the upgraded standard library. The pyvenv +script accepts a ``--upgrade`` option for easily performing this +upgrade on an existing venv. + + +Include files +------------- + +Current virtualenv handles include files in this way: + +On POSIX systems where the installed Python's include files are found +in ``${base_prefix}/include/pythonX.X``, virtualenv creates +``${venv}/include/`` and symlink ``${base_prefix}/include/pythonX.X`` +to ``${venv}/include/pythonX.X``. On Windows, where Python's include +files are found in ``{{ sys.prefix }}/Include`` and symlinks are not +reliably available, virtualenv copies ``{{ sys.prefix }}/Include`` to +``${venv}/Include``. This ensures that extension modules built and +installed within the virtualenv will always find the Python header +files they need in the expected location relative to ``sys.prefix``. + +This solution is not ideal when an extension module installs its own +header files, as the default installation location for those header +files may be a symlink to a system directory that may not be +writable. One installer, pip, explicitly works around this by +installing header files to a nonstandard location +``${venv}/include/site/pythonX.X/``, as in Python there's currently no +standard abstraction for a site-specific include directory. + +This PEP proposes a slightly different approach, though one with +essentially the same effect and the same set of advantages and +disadvantages. Rather than symlinking or copying include files into +the venv, we simply modify the sysconfig schemes so that header files +are always looked for relative to ``base_prefix`` rather than +``prefix``. (We also create an ``include/`` directory within the venv + +Better handling of include files in distutils/packaging and, by +extension, pyvenv, is an area that may deserve its own future PEP. For +now, we propose that the behavior of virtualenv has thus far proved +itself to be at least "good enough" in practice. + +[Open question: should pyvenv instead simply copy the behavior of +virtualenv entirely, to avoid introducing unexpected new issues here?] API @@ -449,39 +505,6 @@ locating and parsing the ``pyvenv.cfg`` file, if it is present. -Open Questions -============== - -What about include files? -------------------------- - -For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in -``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default -in ``$VE/include/pythonX.Y``. With virtualenv, everything works -because the PythonX.Y include is symlinked, so everything that's -needed is in ``$VE/include``. At the moment the reference -implementation doesn't do anything with include files, besides -creating the include directory; this might need to change, to -copy/symlink ``$VE/include/pythonX.Y``. - -As in Python there's no abstraction for a site-specific include -directory, other than for platform-specific stuff, then the user -expectation would seem to be that all include files anyone could ever -want should be found in one of just two locations, with sysconfig -labels "include" & "platinclude". - -There's another issue: what if includes are Python-version-specific? -For example, SIP installs by default into ``$VE/include/pythonX.Y`` -rather than ``$VE/include``, presumably because there's -version-specific stuff in there - but even if that's not the case with -SIP, it could be the case with some other package. And the problem -that gives is that you can't just symlink the ``include/pythonX.Y`` -directory, but actually have to provide a writable directory and -symlink/copy the contents from the system ``include/pythonX.Y``. Of -course this is not hard to do, but it does seem inelegant. OTOH it's -really because there's no supporting concept in ``Python/sysconfig``. - - Reference Implementation ======================== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 18 01:42:18 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 18 May 2012 01:42:18 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Oops=2C_we=27re_not_copying_in?= =?utf8?q?cludes_into_the_venv=2E?= Message-ID: http://hg.python.org/peps/rev/8cbef1f07cb9 changeset: 4407:8cbef1f07cb9 user: Carl Meyer date: Thu May 17 13:10:08 2012 -0600 summary: Oops, we're not copying includes into the venv. files: pep-0405.txt | 21 ++++++++++----------- 1 files changed, 10 insertions(+), 11 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -242,12 +242,11 @@ ---------------------- The technique in this PEP works equally well in general with a copied -or symlinked Python binary (and other needed DLLs and the ``Include`` -directory on Windows). Symlinking is preferable where possible, -because in the case of an upgrade to the underlying Python -installation, a Python executable copied in a venv might become -out-of-sync with the installed standard library and require manual -upgrade. +or symlinked Python binary (and other needed DLLs on Windows). +Symlinking is preferable where possible, because in the case of an +upgrade to the underlying Python installation, a Python executable +copied in a venv might become out-of-sync with the installed standard +library and require manual upgrade. There are some cross-platform difficulties with symlinks: @@ -268,11 +267,11 @@ never work there, and has no advantages). On Windows, if ``--symlink`` is not used, this means that if the -underlying Python installation is upgraded, the Python binary, DLLs, -and include files in the venv should be updated, or there could be -issues of mismatch with the upgraded standard library. The pyvenv -script accepts a ``--upgrade`` option for easily performing this -upgrade on an existing venv. +underlying Python installation is upgraded, the Python binary and DLLs +in the venv should be updated, or there could be issues of mismatch +with the upgraded standard library. The pyvenv script accepts a +``--upgrade`` option for easily performing this upgrade on an existing +venv. Include files -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 18 01:42:19 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 18 May 2012 01:42:19 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merged_Carl_Meyer=27s_updates_to_PEP_405=2E?= Message-ID: http://hg.python.org/peps/rev/f5182f1d9c5c changeset: 4408:f5182f1d9c5c parent: 4402:2ed614d32a5e parent: 4407:8cbef1f07cb9 user: Vinay Sajip date: Fri May 18 00:42:11 2012 +0100 summary: Merged Carl Meyer's updates to PEP 405. files: pep-0405.txt | 101 +++++++++++++++++++++----------------- 1 files changed, 56 insertions(+), 45 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -222,6 +222,22 @@ those files when Python is run from the venv. +Sysconfig install schemes and user-site +--------------------------------------- + +This approach explicitly chooses not to introduce a new sysconfig +install scheme for venvs. Rather, by modifying ``sys.prefix`` we +ensure that existing install schemes which base locations on +``sys.prefix`` will simply work in a venv. Installation to other +install schemes (for instance, the user-site schemes) whose paths are +not relative to ``sys.prefix``, will not be affected by a venv at all. + +It may be feasible to create an alternative implementation of Python +virtual environments based on a virtual-specific sysconfig scheme, but +it would be less robust, as it would require more code to be aware of +whether it is operating within a virtual environment or not. + + Copies versus symlinks ---------------------- @@ -258,6 +274,45 @@ venv. +Include files +------------- + +Current virtualenv handles include files in this way: + +On POSIX systems where the installed Python's include files are found +in ``${base_prefix}/include/pythonX.X``, virtualenv creates +``${venv}/include/`` and symlink ``${base_prefix}/include/pythonX.X`` +to ``${venv}/include/pythonX.X``. On Windows, where Python's include +files are found in ``{{ sys.prefix }}/Include`` and symlinks are not +reliably available, virtualenv copies ``{{ sys.prefix }}/Include`` to +``${venv}/Include``. This ensures that extension modules built and +installed within the virtualenv will always find the Python header +files they need in the expected location relative to ``sys.prefix``. + +This solution is not ideal when an extension module installs its own +header files, as the default installation location for those header +files may be a symlink to a system directory that may not be +writable. One installer, pip, explicitly works around this by +installing header files to a nonstandard location +``${venv}/include/site/pythonX.X/``, as in Python there's currently no +standard abstraction for a site-specific include directory. + +This PEP proposes a slightly different approach, though one with +essentially the same effect and the same set of advantages and +disadvantages. Rather than symlinking or copying include files into +the venv, we simply modify the sysconfig schemes so that header files +are always looked for relative to ``base_prefix`` rather than +``prefix``. (We also create an ``include/`` directory within the venv + +Better handling of include files in distutils/packaging and, by +extension, pyvenv, is an area that may deserve its own future PEP. For +now, we propose that the behavior of virtualenv has thus far proved +itself to be at least "good enough" in practice. + +[Open question: should pyvenv instead simply copy the behavior of +virtualenv entirely, to avoid introducing unexpected new issues here?] + + API --- @@ -449,53 +504,6 @@ locating and parsing the ``pyvenv.cfg`` file, if it is present. -Open Questions -============== - -What about include files? -------------------------- - -For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in -``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default -in ``$VE/include/pythonX.Y``. With virtualenv, everything works -because the PythonX.Y include is symlinked, so everything that's -needed is in ``$VE/include``. At the moment the reference -implementation doesn't do anything with include files, besides -creating the include directory; this might need to change, to -copy/symlink ``$VE/include/pythonX.Y``. - -As in Python there's no abstraction for a site-specific include -directory, other than for platform-specific stuff, then the user -expectation would seem to be that all include files anyone could ever -want should be found in one of just two locations, with sysconfig -labels "include" & "platinclude". - -There's another issue: what if includes are Python-version-specific? -For example, SIP installs by default into ``$VE/include/pythonX.Y`` -rather than ``$VE/include``, presumably because there's -version-specific stuff in there - but even if that's not the case with -SIP, it could be the case with some other package. And the problem -that gives is that you can't just symlink the ``include/pythonX.Y`` -directory, but actually have to provide a writable directory and -symlink/copy the contents from the system ``include/pythonX.Y``. Of -course this is not hard to do, but it does seem inelegant. OTOH it's -really because there's no supporting concept in ``Python/sysconfig``. - - -OS X Framework builds ---------------------- - -There have been some reports that the reference implementation does -not work on an OS X framework build of Python, but it seems to work -for us. This needs further investigation. - - -tkinter -------- - -Tkinter apps currently do not work within a virtual environment. - - Reference Implementation ======================== @@ -505,7 +513,7 @@ installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create a virtual environment. -.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv +.. _a clone of the CPython Mercurial repository: http://hg.python.org/sandbox/vsajip#venv Copyright -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Fri May 18 05:48:05 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 18 May 2012 05:48:05 +0200 Subject: [Python-checkins] Daily reference leaks (364289cc7891): sum=0 Message-ID: results for 364289cc7891 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogbXAq6V', '-x'] From python-checkins at python.org Fri May 18 13:59:13 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 13:59:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Use_SSL_directo?= =?utf8?q?ry_from_properties_file=2E?= Message-ID: http://hg.python.org/cpython/rev/749f5045ce59 changeset: 77019:749f5045ce59 branch: 2.7 parent: 77011:251463919f3c user: Martin v. L?wis date: Fri May 18 13:58:30 2012 +0200 summary: Use SSL directory from properties file. files: PCbuild/build_ssl.py | 40 ++++++------------------------- 1 files changed, 8 insertions(+), 32 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -64,37 +64,13 @@ print(" Please install ActivePerl and ensure it appears on your path") return None -# Locate the best SSL directory given a few roots to look into. -def find_best_ssl_dir(sources): - candidates = [] - for s in sources: - try: - # note: do not abspath s; the build will fail if any - # higher up directory name has spaces in it. - fnames = os.listdir(s) - except os.error: - fnames = [] - for fname in fnames: - fqn = os.path.join(s, fname) - if os.path.isdir(fqn) and fname.startswith("openssl-"): - candidates.append(fqn) - # Now we have all the candidates, locate the best. - best_parts = [] - best_name = None - for c in candidates: - parts = re.split("[.-]", os.path.basename(c))[1:] - # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers - if len(parts) >= 4: - continue - if parts > best_parts: - best_parts = parts - best_name = c - if best_name is not None: - print("Found an SSL directory at '%s'" % (best_name,)) - else: - print("Could not find an SSL directory in '%s'" % (sources,)) - sys.stdout.flush() - return best_name +# Fetch SSL directory from VC properties +def get_ssl_dir(): + propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.vsprops')) + with open(propfile) as f: + m = re.search('openssl-([^"]+)"', f.read()) + return "..\..\openssl-"+m.group(1) + def create_makefile64(makefile, m32): """Create and fix makefile for 64bit @@ -190,7 +166,7 @@ print("No Perl installation was found. Existing Makefiles are used.") sys.stdout.flush() # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live. - ssl_dir = find_best_ssl_dir(("..\\..",)) + ssl_dir = get_ssl_dir() if ssl_dir is None: sys.exit(1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 13:59:33 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 18 May 2012 13:59:33 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_outdated_statements_?= =?utf8?q?about_threading_and_imports=2E?= Message-ID: http://hg.python.org/cpython/rev/565734c9b66d changeset: 77020:565734c9b66d parent: 77018:364289cc7891 user: Antoine Pitrou date: Fri May 18 13:57:04 2012 +0200 summary: Remove outdated statements about threading and imports. files: Doc/library/multiprocessing.rst | 4 +-- Doc/library/threading.rst | 23 --------------------- 2 files changed, 1 insertions(+), 26 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -120,9 +120,7 @@ print(q.get()) # prints "[42, None, 'hello']" p.join() - Queues are thread and process safe, but note that they must never - be instantiated as a side effect of importing a module: this can lead - to a deadlock! (see :ref:`threaded-imports`) + Queues are thread and process safe. **Pipes** diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst --- a/Doc/library/threading.rst +++ b/Doc/library/threading.rst @@ -996,27 +996,3 @@ Currently, :class:`Lock`, :class:`RLock`, :class:`Condition`, :class:`Semaphore`, and :class:`BoundedSemaphore` objects may be used as :keyword:`with` statement context managers. - - -.. _threaded-imports: - -Importing in threaded code --------------------------- - -While the import machinery is thread-safe, there are two key restrictions on -threaded imports due to inherent limitations in the way that thread-safety is -provided: - -* Firstly, other than in the main module, an import should not have the - side effect of spawning a new thread and then waiting for that thread in - any way. Failing to abide by this restriction can lead to a deadlock if - the spawned thread directly or indirectly attempts to import a module. -* Secondly, all import attempts must be completed before the interpreter - starts shutting itself down. This can be most easily achieved by only - performing imports from non-daemon threads created through the threading - module. Daemon threads and threads created directly with the thread - module will require some other form of synchronization to ensure they do - not attempt imports after system shutdown has commenced. Failure to - abide by this restriction will lead to intermittent exceptions and - crashes during interpreter shutdown (as the late imports attempt to - access machinery which is no longer in a valid state). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:07:26 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:07:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Stop_refetching?= =?utf8?q?_OpenSSL=2E?= Message-ID: http://hg.python.org/cpython/rev/fcc8066cdc4d changeset: 77021:fcc8066cdc4d branch: 2.7 parent: 77019:749f5045ce59 user: Martin v. L?wis date: Fri May 18 14:07:28 2012 +0200 summary: Stop refetching OpenSSL. files: Tools/buildbot/external-common.bat | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -15,7 +15,7 @@ @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1 @rem if exist db-4.4.20 rd /s/q db-4.4.20 @rem if exist db-4.7.25.0 rd /s/q db-4.7.25.0 -if exist openssl-0.9.8x rd /s/q openssl-0.9.8x + at rem if exist openssl-0.9.8x rd /s/q openssl-0.9.8x @rem if exist sqlite-3.6.21 rd /s/q sqlite-3.6.21 @rem bzip -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:25:54 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:25:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fetch_openssl_d?= =?utf8?q?irectory_from_pyproject=2Evsprops=2E?= Message-ID: http://hg.python.org/cpython/rev/59fc47a5b0a4 changeset: 77022:59fc47a5b0a4 branch: 3.2 parent: 76929:7b64365d414f user: Martin v. L?wis date: Fri May 18 14:16:53 2012 +0200 summary: Fetch openssl directory from pyproject.vsprops. files: PCbuild/build_ssl.py | 40 ++++++------------------------- 1 files changed, 8 insertions(+), 32 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -63,37 +63,13 @@ print(" Please install ActivePerl and ensure it appears on your path") return None -# Locate the best SSL directory given a few roots to look into. -def find_best_ssl_dir(sources): - candidates = [] - for s in sources: - try: - # note: do not abspath s; the build will fail if any - # higher up directory name has spaces in it. - fnames = os.listdir(s) - except os.error: - fnames = [] - for fname in fnames: - fqn = os.path.join(s, fname) - if os.path.isdir(fqn) and fname.startswith("openssl-"): - candidates.append(fqn) - # Now we have all the candidates, locate the best. - best_parts = [] - best_name = None - for c in candidates: - parts = re.split("[.-]", os.path.basename(c))[1:] - # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers - if len(parts) >= 4: - continue - if parts > best_parts: - best_parts = parts - best_name = c - if best_name is not None: - print("Found an SSL directory at '%s'" % (best_name,)) - else: - print("Could not find an SSL directory in '%s'" % (sources,)) - sys.stdout.flush() - return best_name +# Fetch SSL directory from VC properties +def get_ssl_dir(): + propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.vsprops')) + with open(propfile) as f: + m = re.search('openssl-([^"]+)"', f.read()) + return "..\..\openssl-"+m.group(1) + def create_makefile64(makefile, m32): """Create and fix makefile for 64bit @@ -202,7 +178,7 @@ print("No Perl installation was found. Existing Makefiles are used.") sys.stdout.flush() # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live. - ssl_dir = find_best_ssl_dir(("..\\..",)) + ssl_dir = get_ssl_dir() if ssl_dir is None: sys.exit(1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:25:55 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:25:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_another_set?= =?utf8?q?_of_quotes_to_make_cmd=2Eexe_happy=2E?= Message-ID: http://hg.python.org/cpython/rev/640a0300ba85 changeset: 77023:640a0300ba85 branch: 3.2 user: Martin v. L?wis date: Fri May 18 14:17:43 2012 +0200 summary: Add another set of quotes to make cmd.exe happy. files: PCbuild/build_ssl.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -46,7 +46,7 @@ # is available. def find_working_perl(perls): for perl in perls: - fh = os.popen('"%s" -e "use Win32;"' % perl) + fh = os.popen('""%s" -e "use Win32;""' % perl) fh.read() rc = fh.close() if rc: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:25:56 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:25:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_3=2E2_build=5Fssl_changes=2E?= Message-ID: http://hg.python.org/cpython/rev/7d4521568fa7 changeset: 77024:7d4521568fa7 parent: 76930:5298b39fa6c0 parent: 77023:640a0300ba85 user: Martin v. L?wis date: Fri May 18 14:18:26 2012 +0200 summary: Merge 3.2 build_ssl changes. files: PCbuild/build_ssl.py | 42 ++++++------------------------- 1 files changed, 9 insertions(+), 33 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -46,7 +46,7 @@ # is available. def find_working_perl(perls): for perl in perls: - fh = os.popen('"%s" -e "use Win32;"' % perl) + fh = os.popen('""%s" -e "use Win32;""' % perl) fh.read() rc = fh.close() if rc: @@ -63,37 +63,13 @@ print(" Please install ActivePerl and ensure it appears on your path") return None -# Locate the best SSL directory given a few roots to look into. -def find_best_ssl_dir(sources): - candidates = [] - for s in sources: - try: - # note: do not abspath s; the build will fail if any - # higher up directory name has spaces in it. - fnames = os.listdir(s) - except os.error: - fnames = [] - for fname in fnames: - fqn = os.path.join(s, fname) - if os.path.isdir(fqn) and fname.startswith("openssl-"): - candidates.append(fqn) - # Now we have all the candidates, locate the best. - best_parts = [] - best_name = None - for c in candidates: - parts = re.split("[.-]", os.path.basename(c))[1:] - # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers - if len(parts) >= 4: - continue - if parts > best_parts: - best_parts = parts - best_name = c - if best_name is not None: - print("Found an SSL directory at '%s'" % (best_name,)) - else: - print("Could not find an SSL directory in '%s'" % (sources,)) - sys.stdout.flush() - return best_name +# Fetch SSL directory from VC properties +def get_ssl_dir(): + propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.vsprops')) + with open(propfile) as f: + m = re.search('openssl-([^"]+)"', f.read()) + return "..\..\openssl-"+m.group(1) + def create_makefile64(makefile, m32): """Create and fix makefile for 64bit @@ -202,7 +178,7 @@ print("No Perl installation was found. Existing Makefiles are used.") sys.stdout.flush() # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live. - ssl_dir = find_best_ssl_dir(("..\\..",)) + ssl_dir = get_ssl_dir() if ssl_dir is None: sys.exit(1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:25:57 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:25:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Port_to_VS_2010=2E?= Message-ID: http://hg.python.org/cpython/rev/b46b7da9f5fa changeset: 77025:b46b7da9f5fa user: Martin v. L?wis date: Fri May 18 14:20:04 2012 +0200 summary: Port to VS 2010. files: PCbuild/build_ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -65,9 +65,9 @@ # Fetch SSL directory from VC properties def get_ssl_dir(): - propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.vsprops')) + propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.props')) with open(propfile) as f: - m = re.search('openssl-([^"]+)"', f.read()) + m = re.search('openssl-([^<]+)<', f.read()) return "..\..\openssl-"+m.group(1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:25:58 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:25:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/d2a847cd7d5a changeset: 77026:d2a847cd7d5a parent: 77020:565734c9b66d parent: 77025:b46b7da9f5fa user: Martin v. L?wis date: Fri May 18 14:23:13 2012 +0200 summary: merge heads files: PCbuild/build_ssl.py | 42 ++++++------------------------- 1 files changed, 9 insertions(+), 33 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -46,7 +46,7 @@ # is available. def find_working_perl(perls): for perl in perls: - fh = os.popen('"%s" -e "use Win32;"' % perl) + fh = os.popen('""%s" -e "use Win32;""' % perl) fh.read() rc = fh.close() if rc: @@ -63,37 +63,13 @@ print(" Please install ActivePerl and ensure it appears on your path") return None -# Locate the best SSL directory given a few roots to look into. -def find_best_ssl_dir(sources): - candidates = [] - for s in sources: - try: - # note: do not abspath s; the build will fail if any - # higher up directory name has spaces in it. - fnames = os.listdir(s) - except os.error: - fnames = [] - for fname in fnames: - fqn = os.path.join(s, fname) - if os.path.isdir(fqn) and fname.startswith("openssl-"): - candidates.append(fqn) - # Now we have all the candidates, locate the best. - best_parts = [] - best_name = None - for c in candidates: - parts = re.split("[.-]", os.path.basename(c))[1:] - # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers - if len(parts) >= 4: - continue - if parts > best_parts: - best_parts = parts - best_name = c - if best_name is not None: - print("Found an SSL directory at '%s'" % (best_name,)) - else: - print("Could not find an SSL directory in '%s'" % (sources,)) - sys.stdout.flush() - return best_name +# Fetch SSL directory from VC properties +def get_ssl_dir(): + propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.props')) + with open(propfile) as f: + m = re.search('openssl-([^<]+)<', f.read()) + return "..\..\openssl-"+m.group(1) + def create_makefile64(makefile, m32): """Create and fix makefile for 64bit @@ -202,7 +178,7 @@ print("No Perl installation was found. Existing Makefiles are used.") sys.stdout.flush() # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live. - ssl_dir = find_best_ssl_dir(("..\\..",)) + ssl_dir = get_ssl_dir() if ssl_dir is None: sys.exit(1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:25:59 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:25:59 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?= =?utf8?q?_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/43f0c3c9187d changeset: 77027:43f0c3c9187d branch: 3.2 parent: 77009:6ab4128acccc parent: 77023:640a0300ba85 user: Martin v. L?wis date: Fri May 18 14:24:09 2012 +0200 summary: merge heads files: PCbuild/build_ssl.py | 42 ++++++------------------------- 1 files changed, 9 insertions(+), 33 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -46,7 +46,7 @@ # is available. def find_working_perl(perls): for perl in perls: - fh = os.popen('"%s" -e "use Win32;"' % perl) + fh = os.popen('""%s" -e "use Win32;""' % perl) fh.read() rc = fh.close() if rc: @@ -63,37 +63,13 @@ print(" Please install ActivePerl and ensure it appears on your path") return None -# Locate the best SSL directory given a few roots to look into. -def find_best_ssl_dir(sources): - candidates = [] - for s in sources: - try: - # note: do not abspath s; the build will fail if any - # higher up directory name has spaces in it. - fnames = os.listdir(s) - except os.error: - fnames = [] - for fname in fnames: - fqn = os.path.join(s, fname) - if os.path.isdir(fqn) and fname.startswith("openssl-"): - candidates.append(fqn) - # Now we have all the candidates, locate the best. - best_parts = [] - best_name = None - for c in candidates: - parts = re.split("[.-]", os.path.basename(c))[1:] - # eg - openssl-0.9.7-beta1 - ignore all "beta" or any other qualifiers - if len(parts) >= 4: - continue - if parts > best_parts: - best_parts = parts - best_name = c - if best_name is not None: - print("Found an SSL directory at '%s'" % (best_name,)) - else: - print("Could not find an SSL directory in '%s'" % (sources,)) - sys.stdout.flush() - return best_name +# Fetch SSL directory from VC properties +def get_ssl_dir(): + propfile = (os.path.join(os.path.dirname(__file__), 'pyproject.vsprops')) + with open(propfile) as f: + m = re.search('openssl-([^"]+)"', f.read()) + return "..\..\openssl-"+m.group(1) + def create_makefile64(makefile, m32): """Create and fix makefile for 64bit @@ -202,7 +178,7 @@ print("No Perl installation was found. Existing Makefiles are used.") sys.stdout.flush() # Look for SSL 2 levels up from pcbuild - ie, same place zlib etc all live. - ssl_dir = find_best_ssl_dir(("..\\..",)) + ssl_dir = get_ssl_dir() if ssl_dir is None: sys.exit(1) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 14:26:00 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 14:26:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/61ec31dec331 changeset: 77028:61ec31dec331 parent: 77026:d2a847cd7d5a parent: 77027:43f0c3c9187d user: Martin v. L?wis date: Fri May 18 14:25:54 2012 +0200 summary: merge 3.2 files: -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Fri May 18 15:16:09 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 18 May 2012 23:16:09 +1000 Subject: [Python-checkins] cpython: Remove outdated statements about threading and imports. In-Reply-To: References: Message-ID: I know you fixed the deadlock problem, but the warnings about shutdown misbehaviour are still valid. -- Sent from my phone, thus the relative brevity :) On May 18, 2012 9:59 PM, "antoine.pitrou" wrote: > http://hg.python.org/cpython/rev/565734c9b66d > changeset: 77020:565734c9b66d > parent: 77018:364289cc7891 > user: Antoine Pitrou > date: Fri May 18 13:57:04 2012 +0200 > summary: > Remove outdated statements about threading and imports. > > files: > Doc/library/multiprocessing.rst | 4 +-- > Doc/library/threading.rst | 23 --------------------- > 2 files changed, 1 insertions(+), 26 deletions(-) > > > diff --git a/Doc/library/multiprocessing.rst > b/Doc/library/multiprocessing.rst > --- a/Doc/library/multiprocessing.rst > +++ b/Doc/library/multiprocessing.rst > @@ -120,9 +120,7 @@ > print(q.get()) # prints "[42, None, 'hello']" > p.join() > > - Queues are thread and process safe, but note that they must never > - be instantiated as a side effect of importing a module: this can lead > - to a deadlock! (see :ref:`threaded-imports`) > + Queues are thread and process safe. > > **Pipes** > > diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst > --- a/Doc/library/threading.rst > +++ b/Doc/library/threading.rst > @@ -996,27 +996,3 @@ > Currently, :class:`Lock`, :class:`RLock`, :class:`Condition`, > :class:`Semaphore`, and :class:`BoundedSemaphore` objects may be used as > :keyword:`with` statement context managers. > - > - > -.. _threaded-imports: > - > -Importing in threaded code > --------------------------- > - > -While the import machinery is thread-safe, there are two key restrictions > on > -threaded imports due to inherent limitations in the way that > thread-safety is > -provided: > - > -* Firstly, other than in the main module, an import should not have the > - side effect of spawning a new thread and then waiting for that thread in > - any way. Failing to abide by this restriction can lead to a deadlock if > - the spawned thread directly or indirectly attempts to import a module. > -* Secondly, all import attempts must be completed before the interpreter > - starts shutting itself down. This can be most easily achieved by only > - performing imports from non-daemon threads created through the threading > - module. Daemon threads and threads created directly with the thread > - module will require some other form of synchronization to ensure they do > - not attempt imports after system shutdown has commenced. Failure to > - abide by this restriction will lead to intermittent exceptions and > - crashes during interpreter shutdown (as the late imports attempt to > - access machinery which is no longer in a valid state). > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-checkins at python.org Fri May 18 15:34:36 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 15:34:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Upgrade_OpenSSL?= =?utf8?q?_to_1=2E0=2E0j?= Message-ID: http://hg.python.org/cpython/rev/d86e5daab9ce changeset: 77029:d86e5daab9ce branch: 3.2 parent: 77027:43f0c3c9187d user: Martin v. L?wis date: Fri May 18 15:28:01 2012 +0200 summary: Upgrade OpenSSL to 1.0.0j files: Misc/NEWS | 2 ++ PC/VC6/readme.txt | 5 ++--- PC/VS8.0/pyproject.vsprops | 2 +- PCbuild/pyproject.vsprops | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 4 ++-- 6 files changed, 9 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -235,6 +235,8 @@ Build ----- +- The Windows build now uses OpenSSL 1.0.0j and bzip2 1.0.6. + - Issue #14557: Fix extensions build on HP-UX. Patch by Adi Roiban. - Issue #14437: Fix building the _io module under Cygwin. diff --git a/PC/VC6/readme.txt b/PC/VC6/readme.txt --- a/PC/VC6/readme.txt +++ b/PC/VC6/readme.txt @@ -153,10 +153,9 @@ Unpack into the "dist" directory, retaining the folder name from the archive - for example, the latest stable OpenSSL will install as - dist/openssl-1.0.0a + dist/openssl-1.0.0j - You can (theoretically) use any version of OpenSSL you like - the - build process will automatically select the latest version. + You need to use version 1.0.0j of OpenSSL. You can install the NASM assembler from http://www.nasm.us/ diff --git a/PC/VS8.0/pyproject.vsprops b/PC/VS8.0/pyproject.vsprops --- a/PC/VS8.0/pyproject.vsprops +++ b/PC/VS8.0/pyproject.vsprops @@ -58,7 +58,7 @@ /> http://hg.python.org/cpython/rev/26818230822a changeset: 77030:26818230822a branch: 3.2 user: Martin v. L?wis date: Fri May 18 15:28:43 2012 +0200 summary: Drop double quoting again. I'm at a loss when to quote and when not. files: PCbuild/build_ssl.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -46,7 +46,7 @@ # is available. def find_working_perl(perls): for perl in perls: - fh = os.popen('""%s" -e "use Win32;""' % perl) + fh = os.popen('"%s" -e "use Win32;"' % perl) fh.read() rc = fh.close() if rc: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 15:34:38 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 15:34:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/608105fd0edc changeset: 77031:608105fd0edc parent: 77028:61ec31dec331 parent: 77030:26818230822a user: Martin v. L?wis date: Fri May 18 15:34:22 2012 +0200 summary: merge 3.2 files: PC/VC6/readme.txt | 5 ++--- PC/VS8.0/pyproject.vsprops | 2 +- PC/VS9.0/pyproject.vsprops | 2 +- PCbuild/build_ssl.py | 2 +- PCbuild/pyproject.props | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 4 ++-- 7 files changed, 9 insertions(+), 10 deletions(-) diff --git a/PC/VC6/readme.txt b/PC/VC6/readme.txt --- a/PC/VC6/readme.txt +++ b/PC/VC6/readme.txt @@ -153,10 +153,9 @@ Unpack into the "dist" directory, retaining the folder name from the archive - for example, the latest stable OpenSSL will install as - dist/openssl-1.0.0a + dist/openssl-1.0.0j - You can (theoretically) use any version of OpenSSL you like - the - build process will automatically select the latest version. + You need to use version 1.0.0j of OpenSSL. You can install the NASM assembler from http://www.nasm.us/ diff --git a/PC/VS8.0/pyproject.vsprops b/PC/VS8.0/pyproject.vsprops --- a/PC/VS8.0/pyproject.vsprops +++ b/PC/VS8.0/pyproject.vsprops @@ -58,7 +58,7 @@ /> $(externalsDir)\sqlite-3.7.12 $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.3 - $(externalsDir)\openssl-1.0.0a + $(externalsDir)\openssl-1.0.0j $(externalsDir)\tcltk $(externalsDir)\tcltk64 $(tcltkDir)\lib\tcl85.lib;$(tcltkDir)\lib\tk85.lib diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -148,7 +148,7 @@ Get the source code through - svn export http://svn.python.org/projects/external/openssl-1.0.0a + svn export http://svn.python.org/projects/external/openssl-1.0.0j ** NOTE: if you use the Tools\buildbot\external(-amd64).bat approach for obtaining external sources then you don't need to manually get the source diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -14,7 +14,7 @@ @rem if exist tk8.4.16 rd /s/q tk8.4.16 @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1 @rem if exist db-4.4.20 rd /s/q db-4.4.20 - at rem if exist openssl-1.0.0a rd /s/q openssl-1.0.0a + at rem if exist openssl-1.0.0j rd /s/q openssl-1.0.0j @rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12 @rem bzip @@ -24,7 +24,7 @@ ) @rem OpenSSL -if not exist openssl-1.0.0a svn export http://svn.python.org/projects/external/openssl-1.0.0a +if not exist openssl-1.0.0j svn export http://svn.python.org/projects/external/openssl-1.0.0j @rem tcl/tk if not exist tcl-8.5.11.0 ( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 16:08:13 2012 From: python-checkins at python.org (barry.warsaw) Date: Fri, 18 May 2012 16:08:13 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Make_it_clear_that_the_PEP_edi?= =?utf8?q?tors_can_only_review_a_completed_PEP_for?= Message-ID: http://hg.python.org/peps/rev/f3358939e05e changeset: 4409:f3358939e05e user: Barry Warsaw date: Fri May 18 10:08:09 2012 -0400 summary: Make it clear that the PEP editors can only review a completed PEP for style and consistency. The BDFL (by way of python-dev) is the final arbiter of the PEP for its acceptance into the language. files: pep-0001.txt | 18 ++++++++++-------- 1 files changed, 10 insertions(+), 8 deletions(-) diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -161,14 +161,16 @@ PEP Review & Resolution ----------------------- -Once the authors have completed a PEP, they must inform the PEP editors -that it is ready for review. PEPs are reviewed by the BDFL and his -chosen consultants, who may accept or reject a PEP or send it back to -the author(s) for revision. For a PEP that is predetermined to be -acceptable (e.g., it is an obvious win as-is and/or its implementation -has already been checked in) the BDFL may also initiate a PEP review, -first notifying the PEP author(s) and giving them a chance to make -revisions. +Once the authors have completed a PEP, they may request a review for +style and consistency from the PEP editors. However, the content and +final acceptance of the PEP must be requested of the BDFL, usually via +an email to the python-dev mailing list. PEPs are reviewed by the +BDFL and his chosen consultants, who may accept or reject a PEP or +send it back to the author(s) for revision. For a PEP that is +predetermined to be acceptable (e.g., it is an obvious win as-is +and/or its implementation has already been checked in) the BDFL may +also initiate a PEP review, first notifying the PEP author(s) and +giving them a chance to make revisions. The final authority for PEP approval is the BDFL. However, whenever a new PEP is put forward, any core developer that believes they are suitably -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 18 16:26:01 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 16:26:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Assume_nasm=2E?= Message-ID: http://hg.python.org/cpython/rev/224ca86e3919 changeset: 77032:224ca86e3919 user: Martin v. L?wis date: Fri May 18 16:25:04 2012 +0200 summary: Assume nasm. files: PCbuild/build_ssl.py | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/PCbuild/build_ssl.py b/PCbuild/build_ssl.py --- a/PCbuild/build_ssl.py +++ b/PCbuild/build_ssl.py @@ -228,9 +228,9 @@ # Now run make. if arch == "amd64": - rc = os.system("ml64 -c -Foms\\uptable.obj ms\\uptable.asm") + rc = os.system("nasm -f win64 -DNEAR -Ox -g ms\\uptable.asm") if rc: - print("ml64 assembler has failed.") + print("nasm assembler has failed.") sys.exit(rc) copy(r"crypto\buildinf_%s.h" % arch, r"crypto\buildinf.h") -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 16:29:43 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 16:29:43 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Upgrade_OpenSSL_to_1=2E0=2E?= =?utf8?q?1c?= Message-ID: http://hg.python.org/cpython/rev/32a864682720 changeset: 77033:32a864682720 user: Martin v. L?wis date: Fri May 18 16:29:33 2012 +0200 summary: Upgrade OpenSSL to 1.0.1c files: Misc/NEWS | 2 ++ PC/VC6/readme.txt | 4 ++-- PC/VS8.0/pyproject.vsprops | 2 +- PC/VS9.0/pyproject.vsprops | 2 +- PCbuild/pyproject.props | 2 +- PCbuild/readme.txt | 2 +- Tools/buildbot/external-common.bat | 7 +++++-- 7 files changed, 13 insertions(+), 8 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -135,6 +135,8 @@ Build ----- +- Upgrade Windows library versions: bzip 1.0.6, OpenSSL 1.0.1c. + - Issue #14693: Under non-Windows platforms, hashlib's fallback modules are always compiled, even if OpenSSL is present at build time. diff --git a/PC/VC6/readme.txt b/PC/VC6/readme.txt --- a/PC/VC6/readme.txt +++ b/PC/VC6/readme.txt @@ -153,9 +153,9 @@ Unpack into the "dist" directory, retaining the folder name from the archive - for example, the latest stable OpenSSL will install as - dist/openssl-1.0.0j + dist/openssl-1.0.1c - You need to use version 1.0.0j of OpenSSL. + You need to use version 1.0.1c of OpenSSL. You can install the NASM assembler from http://www.nasm.us/ diff --git a/PC/VS8.0/pyproject.vsprops b/PC/VS8.0/pyproject.vsprops --- a/PC/VS8.0/pyproject.vsprops +++ b/PC/VS8.0/pyproject.vsprops @@ -58,7 +58,7 @@ /> $(externalsDir)\sqlite-3.7.12 $(externalsDir)\bzip2-1.0.6 $(externalsDir)\xz-5.0.3 - $(externalsDir)\openssl-1.0.0j + $(externalsDir)\openssl-1.0.1c $(externalsDir)\tcltk $(externalsDir)\tcltk64 $(tcltkDir)\lib\tcl85.lib;$(tcltkDir)\lib\tk85.lib diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -148,7 +148,7 @@ Get the source code through - svn export http://svn.python.org/projects/external/openssl-1.0.0j + svn export http://svn.python.org/projects/external/openssl-1.0.1c ** NOTE: if you use the Tools\buildbot\external(-amd64).bat approach for obtaining external sources then you don't need to manually get the source diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -14,7 +14,7 @@ @rem if exist tk8.4.16 rd /s/q tk8.4.16 @rem if exist tk-8.4.18.1 rd /s/q tk-8.4.18.1 @rem if exist db-4.4.20 rd /s/q db-4.4.20 - at rem if exist openssl-1.0.0j rd /s/q openssl-1.0.0j + at rem if exist openssl-1.0.1c rd /s/q openssl-1.0.1c @rem if exist sqlite-3.7.12 rd /s/q sqlite-3.7.12 @rem bzip @@ -24,7 +24,10 @@ ) @rem OpenSSL -if not exist openssl-1.0.0j svn export http://svn.python.org/projects/external/openssl-1.0.0j +if not exist openssl-1.0.0c ( + rd /s/q openssl-1.0.0j + svn export http://svn.python.org/projects/external/openssl-1.0.0c +) @rem tcl/tk if not exist tcl-8.5.11.0 ( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 16:43:03 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 18 May 2012 16:43:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo=2E?= Message-ID: http://hg.python.org/cpython/rev/9a2281086142 changeset: 77034:9a2281086142 user: Martin v. L?wis date: Fri May 18 16:42:52 2012 +0200 summary: Fix typo. files: Tools/buildbot/external-common.bat | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat --- a/Tools/buildbot/external-common.bat +++ b/Tools/buildbot/external-common.bat @@ -24,9 +24,9 @@ ) @rem OpenSSL -if not exist openssl-1.0.0c ( +if not exist openssl-1.0.1c ( rd /s/q openssl-1.0.0j - svn export http://svn.python.org/projects/external/openssl-1.0.0c + svn export http://svn.python.org/projects/external/openssl-1.0.1c ) @rem tcl/tk -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 17:41:12 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 18 May 2012 17:41:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2312098=3A_Make_multiproce?= =?utf8?q?ssing=27s_child_processes_inherit_sys=2Eflags_on_Windows?= Message-ID: http://hg.python.org/cpython/rev/347735ec92eb changeset: 77035:347735ec92eb user: Richard Oudkerk date: Fri May 18 14:28:02 2012 +0100 summary: #12098: Make multiprocessing's child processes inherit sys.flags on Windows Initial patch by Sergey Mezentsev. files: Lib/multiprocessing/forking.py | 3 +- Lib/multiprocessing/util.py | 31 +++++++++++++++++ Lib/test/support.py | 20 +---------- Lib/test/test_multiprocessing.py | 35 +++++++++++++++++++- Misc/NEWS | 4 ++ 5 files changed, 73 insertions(+), 20 deletions(-) diff --git a/Lib/multiprocessing/forking.py b/Lib/multiprocessing/forking.py --- a/Lib/multiprocessing/forking.py +++ b/Lib/multiprocessing/forking.py @@ -324,7 +324,8 @@ return [sys.executable, '--multiprocessing-fork'] else: prog = 'from multiprocessing.forking import main; main()' - return [_python_exe, '-c', prog, '--multiprocessing-fork'] + opts = util._args_from_interpreter_flags() + return [_python_exe] + opts + ['-c', prog, '--multiprocessing-fork'] def main(): diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -7,6 +7,7 @@ # Licensed to PSF under a Contributor Agreement. # +import sys import functools import itertools import weakref @@ -295,3 +296,33 @@ register_after_fork(self, lambda obj : obj.__dict__.clear()) def __reduce__(self): return type(self), () + +# +# Get options for python to produce the same sys.flags +# + +def _args_from_interpreter_flags(): + """Return a list of command-line arguments reproducing the current + settings in sys.flags and sys.warnoptions.""" + flag_opt_map = { + 'debug': 'd', + # 'inspect': 'i', + # 'interactive': 'i', + 'optimize': 'O', + 'dont_write_bytecode': 'B', + 'no_user_site': 's', + 'no_site': 'S', + 'ignore_environment': 'E', + 'verbose': 'v', + 'bytes_warning': 'b', + 'quiet': 'q', + 'hash_randomization': 'R', + } + args = [] + for flag, opt in flag_opt_map.items(): + v = getattr(sys.flags, flag) + if v > 0: + args.append('-' + opt * v) + for opt in sys.warnoptions: + args.append('-W' + opt) + return args diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -1596,24 +1596,8 @@ def args_from_interpreter_flags(): """Return a list of command-line arguments reproducing the current settings in sys.flags and sys.warnoptions.""" - flag_opt_map = { - 'bytes_warning': 'b', - 'dont_write_bytecode': 'B', - 'hash_randomization': 'R', - 'ignore_environment': 'E', - 'no_user_site': 's', - 'no_site': 'S', - 'optimize': 'O', - 'verbose': 'v', - } - args = [] - for flag, opt in flag_opt_map.items(): - v = getattr(sys.flags, flag) - if v > 0: - args.append('-' + opt * v) - for opt in sys.warnoptions: - args.append('-W' + opt) - return args + from multiprocessing.util import _args_from_interpreter_flags + return _args_from_interpreter_flags() #============================================================ # Support for assertions about logging. diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py --- a/Lib/test/test_multiprocessing.py +++ b/Lib/test/test_multiprocessing.py @@ -2814,8 +2814,41 @@ with self.assertRaises(ValueError): multiprocessing.connection.Listener('/var/test.pipe') +# +# Issue 12098: check sys.flags of child matches that for parent +# + +class TestFlags(unittest.TestCase): + @classmethod + def run_in_grandchild(cls, conn): + conn.send(tuple(sys.flags)) + + @classmethod + def run_in_child(cls): + import json + r, w = multiprocessing.Pipe(duplex=False) + p = multiprocessing.Process(target=cls.run_in_grandchild, args=(w,)) + p.start() + grandchild_flags = r.recv() + p.join() + r.close() + w.close() + flags = (tuple(sys.flags), grandchild_flags) + print(json.dumps(flags)) + + def test_flags(self): + import json, subprocess + # start child process using unusual flags + prog = ('from test.test_multiprocessing import TestFlags; ' + + 'TestFlags.run_in_child()') + data = subprocess.check_output( + [sys.executable, '-E', '-S', '-O', '-c', prog]) + child_flags, grandchild_flags = json.loads(data.decode('ascii')) + self.assertEqual(child_flags, grandchild_flags) + testcases_other = [OtherTest, TestInvalidHandle, TestInitializers, - TestStdinBadfiledescriptor, TestWait, TestInvalidFamily] + TestStdinBadfiledescriptor, TestWait, TestInvalidFamily, + TestFlags] # # diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,10 @@ Library ------- +- Issue #12098: multiprocessing on Windows now starts child processes + using the same sys.flags as the current process. Initial patch by + Sergey Mezentsev. + - Issue #13031: Small speed-up for tarfile when unzipping tarfiles. Patch by Justin Peel. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 18:36:02 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 18 May 2012 18:36:02 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Move_private_function_=5Far?= =?utf8?q?gs=5Ffrom=5Finterpreter=5Fflags=28=29_to_subprocess=2Epy=2C_so?= Message-ID: http://hg.python.org/cpython/rev/2034b3de1144 changeset: 77036:2034b3de1144 user: Antoine Pitrou date: Fri May 18 18:33:07 2012 +0200 summary: Move private function _args_from_interpreter_flags() to subprocess.py, so that it can be imported when threads are disabled. (followup to issue #12098) files: Lib/multiprocessing/util.py | 30 +----------------------- Lib/subprocess.py | 31 +++++++++++++++++++++++++ Lib/test/support.py | 3 +- Tools/scripts/run_tests.py | 6 ++++- 4 files changed, 38 insertions(+), 32 deletions(-) diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -14,6 +14,7 @@ import atexit import threading # we want threading to install it's # cleanup function before multiprocessing does +from subprocess import _args_from_interpreter_flags from multiprocessing.process import current_process, active_children @@ -297,32 +298,3 @@ def __reduce__(self): return type(self), () -# -# Get options for python to produce the same sys.flags -# - -def _args_from_interpreter_flags(): - """Return a list of command-line arguments reproducing the current - settings in sys.flags and sys.warnoptions.""" - flag_opt_map = { - 'debug': 'd', - # 'inspect': 'i', - # 'interactive': 'i', - 'optimize': 'O', - 'dont_write_bytecode': 'B', - 'no_user_site': 's', - 'no_site': 'S', - 'ignore_environment': 'E', - 'verbose': 'v', - 'bytes_warning': 'b', - 'quiet': 'q', - 'hash_randomization': 'R', - } - args = [] - for flag, opt in flag_opt_map.items(): - v = getattr(sys.flags, flag) - if v > 0: - args.append('-' + opt * v) - for opt in sys.warnoptions: - args.append('-W' + opt) - return args diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -475,6 +475,37 @@ continue +# XXX This function is only used by multiprocessing and the test suite, +# but it's here so that it can be imported when Python is compiled without +# threads. + +def _args_from_interpreter_flags(): + """Return a list of command-line arguments reproducing the current + settings in sys.flags and sys.warnoptions.""" + flag_opt_map = { + 'debug': 'd', + # 'inspect': 'i', + # 'interactive': 'i', + 'optimize': 'O', + 'dont_write_bytecode': 'B', + 'no_user_site': 's', + 'no_site': 'S', + 'ignore_environment': 'E', + 'verbose': 'v', + 'bytes_warning': 'b', + 'quiet': 'q', + 'hash_randomization': 'R', + } + args = [] + for flag, opt in flag_opt_map.items(): + v = getattr(sys.flags, flag) + if v > 0: + args.append('-' + opt * v) + for opt in sys.warnoptions: + args.append('-W' + opt) + return args + + def call(*popenargs, timeout=None, **kwargs): """Run command with arguments. Wait for command to complete or timeout, then return the returncode attribute. diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -1596,8 +1596,7 @@ def args_from_interpreter_flags(): """Return a list of command-line arguments reproducing the current settings in sys.flags and sys.warnoptions.""" - from multiprocessing.util import _args_from_interpreter_flags - return _args_from_interpreter_flags() + return subprocess._args_from_interpreter_flags() #============================================================ # Support for assertions about logging. diff --git a/Tools/scripts/run_tests.py b/Tools/scripts/run_tests.py --- a/Tools/scripts/run_tests.py +++ b/Tools/scripts/run_tests.py @@ -10,6 +10,10 @@ import os import sys import test.support +try: + import threading +except ImportError: + threading = None def is_multiprocess_flag(arg): @@ -34,7 +38,7 @@ ]) if sys.platform == 'win32': args.append('-n') # Silence alerts under Windows - if not any(is_multiprocess_flag(arg) for arg in regrtest_args): + if threading and not any(is_multiprocess_flag(arg) for arg in regrtest_args): args.extend(['-j', '0']) # Use all CPU cores if not any(is_resource_use_flag(arg) for arg in regrtest_args): args.extend(['-u', 'all,-largefile,-audio,-gui']) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 18:36:02 2012 From: python-checkins at python.org (antoine.pitrou) Date: Fri, 18 May 2012 18:36:02 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_whitespace?= Message-ID: http://hg.python.org/cpython/rev/61f91faf11a4 changeset: 77037:61f91faf11a4 user: Antoine Pitrou date: Fri May 18 18:33:32 2012 +0200 summary: Fix whitespace files: Lib/multiprocessing/util.py | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -297,4 +297,3 @@ register_after_fork(self, lambda obj : obj.__dict__.clear()) def __reduce__(self): return type(self), () - -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 20:30:21 2012 From: python-checkins at python.org (petri.lehtinen) Date: Fri, 18 May 2012 20:30:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_time=2Etime?= =?utf8?q?=28=29_references_in_the_time_module_docs?= Message-ID: http://hg.python.org/cpython/rev/40900f791469 changeset: 77038:40900f791469 branch: 2.7 parent: 77021:fcc8066cdc4d user: Petri Lehtinen date: Fri May 18 21:19:17 2012 +0300 summary: Fix time.time() references in the time module docs Closes #14842. files: Doc/library/time.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -71,9 +71,9 @@ the units in which their value or argument is expressed. E.g. on most Unix systems, the clock "ticks" only 50 or 100 times a second. -* On the other hand, the precision of :func:`time` and :func:`sleep` is better +* On the other hand, the precision of :func:`.time` and :func:`sleep` is better than their Unix equivalents: times are expressed as floating point numbers, - :func:`time` returns the most accurate time available (using Unix + :func:`.time` returns the most accurate time available (using Unix :c:func:`gettimeofday` where available), and :func:`sleep` will accept a time with a nonzero fraction (Unix :c:func:`select` is used to implement this, where available). @@ -164,7 +164,7 @@ Convert a time expressed in seconds since the epoch to a string representing local time. If *secs* is not provided or :const:`None`, the current time as - returned by :func:`time` is used. ``ctime(secs)`` is equivalent to + returned by :func:`.time` is used. ``ctime(secs)`` is equivalent to ``asctime(localtime(secs))``. Locale information is not used by :func:`ctime`. .. versionchanged:: 2.1 @@ -183,7 +183,7 @@ Convert a time expressed in seconds since the epoch to a :class:`struct_time` in UTC in which the dst flag is always zero. If *secs* is not provided or - :const:`None`, the current time as returned by :func:`time` is used. Fractions + :const:`None`, the current time as returned by :func:`.time` is used. Fractions of a second are ignored. See above for a description of the :class:`struct_time` object. See :func:`calendar.timegm` for the inverse of this function. @@ -198,7 +198,7 @@ .. function:: localtime([secs]) Like :func:`gmtime` but converts to local time. If *secs* is not provided or - :const:`None`, the current time as returned by :func:`time` is used. The dst + :const:`None`, the current time as returned by :func:`.time` is used. The dst flag is set to ``1`` when DST applies to the given time. .. versionchanged:: 2.1 @@ -213,7 +213,7 @@ This is the inverse function of :func:`localtime`. Its argument is the :class:`struct_time` or full 9-tuple (since the dst flag is needed; use ``-1`` as the dst flag if it is unknown) which expresses the time in *local* time, not - UTC. It returns a floating point number, for compatibility with :func:`time`. + UTC. It returns a floating point number, for compatibility with :func:`.time`. If the input value cannot be represented as a valid time, either :exc:`OverflowError` or :exc:`ValueError` will be raised (which depends on whether the invalid value is caught by Python or the underlying C libraries). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 20:30:22 2012 From: python-checkins at python.org (petri.lehtinen) Date: Fri, 18 May 2012 20:30:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_time=2Etime?= =?utf8?q?=28=29_references_in_the_time_module_docs?= Message-ID: http://hg.python.org/cpython/rev/d15f01b0c1a0 changeset: 77039:d15f01b0c1a0 branch: 3.2 parent: 77030:26818230822a user: Petri Lehtinen date: Fri May 18 21:19:17 2012 +0300 summary: Fix time.time() references in the time module docs Closes #14842. files: Doc/library/time.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -81,9 +81,9 @@ the units in which their value or argument is expressed. E.g. on most Unix systems, the clock "ticks" only 50 or 100 times a second. -* On the other hand, the precision of :func:`time` and :func:`sleep` is better +* On the other hand, the precision of :func:`.time` and :func:`sleep` is better than their Unix equivalents: times are expressed as floating point numbers, - :func:`time` returns the most accurate time available (using Unix + :func:`.time` returns the most accurate time available (using Unix :c:func:`gettimeofday` where available), and :func:`sleep` will accept a time with a nonzero fraction (Unix :c:func:`select` is used to implement this, where available). @@ -177,7 +177,7 @@ Convert a time expressed in seconds since the epoch to a string representing local time. If *secs* is not provided or :const:`None`, the current time as - returned by :func:`time` is used. ``ctime(secs)`` is equivalent to + returned by :func:`.time` is used. ``ctime(secs)`` is equivalent to ``asctime(localtime(secs))``. Locale information is not used by :func:`ctime`. @@ -190,7 +190,7 @@ Convert a time expressed in seconds since the epoch to a :class:`struct_time` in UTC in which the dst flag is always zero. If *secs* is not provided or - :const:`None`, the current time as returned by :func:`time` is used. Fractions + :const:`None`, the current time as returned by :func:`.time` is used. Fractions of a second are ignored. See above for a description of the :class:`struct_time` object. See :func:`calendar.timegm` for the inverse of this function. @@ -199,7 +199,7 @@ .. function:: localtime([secs]) Like :func:`gmtime` but converts to local time. If *secs* is not provided or - :const:`None`, the current time as returned by :func:`time` is used. The dst + :const:`None`, the current time as returned by :func:`.time` is used. The dst flag is set to ``1`` when DST applies to the given time. @@ -208,7 +208,7 @@ This is the inverse function of :func:`localtime`. Its argument is the :class:`struct_time` or full 9-tuple (since the dst flag is needed; use ``-1`` as the dst flag if it is unknown) which expresses the time in *local* time, not - UTC. It returns a floating point number, for compatibility with :func:`time`. + UTC. It returns a floating point number, for compatibility with :func:`.time`. If the input value cannot be represented as a valid time, either :exc:`OverflowError` or :exc:`ValueError` will be raised (which depends on whether the invalid value is caught by Python or the underlying C libraries). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 20:30:28 2012 From: python-checkins at python.org (petri.lehtinen) Date: Fri, 18 May 2012 20:30:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Fix_time=2Etime=28=29_references_in_the_time_module_docs?= Message-ID: http://hg.python.org/cpython/rev/6286dd856252 changeset: 77040:6286dd856252 parent: 77037:61f91faf11a4 parent: 77039:d15f01b0c1a0 user: Petri Lehtinen date: Fri May 18 21:27:27 2012 +0300 summary: Fix time.time() references in the time module docs Closes #14842. files: Doc/library/time.rst | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/time.rst b/Doc/library/time.rst --- a/Doc/library/time.rst +++ b/Doc/library/time.rst @@ -62,9 +62,9 @@ the units in which their value or argument is expressed. E.g. on most Unix systems, the clock "ticks" only 50 or 100 times a second. -* On the other hand, the precision of :func:`time` and :func:`sleep` is better +* On the other hand, the precision of :func:`.time` and :func:`sleep` is better than their Unix equivalents: times are expressed as floating point numbers, - :func:`time` returns the most accurate time available (using Unix + :func:`.time` returns the most accurate time available (using Unix :c:func:`gettimeofday` where available), and :func:`sleep` will accept a time with a nonzero fraction (Unix :c:func:`select` is used to implement this, where available). @@ -256,7 +256,7 @@ Convert a time expressed in seconds since the epoch to a string representing local time. If *secs* is not provided or :const:`None`, the current time as - returned by :func:`time` is used. ``ctime(secs)`` is equivalent to + returned by :func:`.time` is used. ``ctime(secs)`` is equivalent to ``asctime(localtime(secs))``. Locale information is not used by :func:`ctime`. @@ -284,7 +284,7 @@ Convert a time expressed in seconds since the epoch to a :class:`struct_time` in UTC in which the dst flag is always zero. If *secs* is not provided or - :const:`None`, the current time as returned by :func:`time` is used. Fractions + :const:`None`, the current time as returned by :func:`.time` is used. Fractions of a second are ignored. See above for a description of the :class:`struct_time` object. See :func:`calendar.timegm` for the inverse of this function. @@ -293,7 +293,7 @@ .. function:: localtime([secs]) Like :func:`gmtime` but converts to local time. If *secs* is not provided or - :const:`None`, the current time as returned by :func:`time` is used. The dst + :const:`None`, the current time as returned by :func:`.time` is used. The dst flag is set to ``1`` when DST applies to the given time. @@ -302,7 +302,7 @@ This is the inverse function of :func:`localtime`. Its argument is the :class:`struct_time` or full 9-tuple (since the dst flag is needed; use ``-1`` as the dst flag if it is unknown) which expresses the time in *local* time, not - UTC. It returns a floating point number, for compatibility with :func:`time`. + UTC. It returns a floating point number, for compatibility with :func:`.time`. If the input value cannot be represented as a valid time, either :exc:`OverflowError` or :exc:`ValueError` will be raised (which depends on whether the invalid value is caught by Python or the underlying C libraries). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 21:03:22 2012 From: python-checkins at python.org (petri.lehtinen) Date: Fri, 18 May 2012 21:03:22 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0Nzk4OiBweWNs?= =?utf8?q?br_now_raises_ImportError_instead_of_KeyError_for_missing_packag?= =?utf8?q?es?= Message-ID: http://hg.python.org/cpython/rev/2d2079593212 changeset: 77041:2d2079593212 branch: 2.7 parent: 77038:40900f791469 user: Petri Lehtinen date: Fri May 18 21:51:11 2012 +0300 summary: #14798: pyclbr now raises ImportError instead of KeyError for missing packages files: Lib/pyclbr.py | 2 ++ Lib/test/test_pyclbr.py | 5 +++++ Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Lib/pyclbr.py b/Lib/pyclbr.py --- a/Lib/pyclbr.py +++ b/Lib/pyclbr.py @@ -128,6 +128,8 @@ parent = _readmodule(package, path, inpackage) if inpackage is not None: package = "%s.%s" % (inpackage, package) + if not '__path__' in parent: + raise ImportError('No package named {}'.format(package)) return _readmodule(submodule, parent['__path__'], package) # Search the path for the module diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -188,6 +188,11 @@ cm('email.parser') cm('test.test_pyclbr') + def test_issue_14798(self): + # test ImportError is raised when the first part of a dotted name is + # not a package + self.assertRaises(ImportError, pyclbr.readmodule_ex, 'asyncore.foo') + def test_main(): run_unittest(PyclbrTest) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -297,6 +297,7 @@ Dan Gass Andrew Gaul Stephen M. Gava +Xavier de Gaye Harry Henry Gebel Marius Gedminas Thomas Gellekum diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,10 @@ Library ------- +- Issue #14798: Fix the functions in pyclbr to raise an ImportError + when the first part of a dotted name is not a package. Patch by + Xavier de Gaye. + - Issue #14832: fixed the order of the argument references in the error message produced by unittest's assertItemsEqual. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 21:03:23 2012 From: python-checkins at python.org (petri.lehtinen) Date: Fri, 18 May 2012 21:03:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0Nzk4OiBweWNs?= =?utf8?q?br_now_raises_ImportError_instead_of_KeyError_for_missing_packag?= =?utf8?q?es?= Message-ID: http://hg.python.org/cpython/rev/895246f1a06a changeset: 77042:895246f1a06a branch: 3.2 parent: 77039:d15f01b0c1a0 user: Petri Lehtinen date: Fri May 18 21:51:11 2012 +0300 summary: #14798: pyclbr now raises ImportError instead of KeyError for missing packages files: Lib/pyclbr.py | 2 ++ Lib/test/test_pyclbr.py | 5 +++++ Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Lib/pyclbr.py b/Lib/pyclbr.py --- a/Lib/pyclbr.py +++ b/Lib/pyclbr.py @@ -128,6 +128,8 @@ parent = _readmodule(package, path, inpackage) if inpackage is not None: package = "%s.%s" % (inpackage, package) + if not '__path__' in parent: + raise ImportError('No package named {}'.format(package)) return _readmodule(submodule, parent['__path__'], package) # Search the path for the module diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -167,6 +167,11 @@ cm('email.parser') cm('test.test_pyclbr') + def test_issue_14798(self): + # test ImportError is raised when the first part of a dotted name is + # not a package + self.assertRaises(ImportError, pyclbr.readmodule_ex, 'asyncore.foo') + def test_main(): run_unittest(PyclbrTest) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -332,6 +332,7 @@ Dan Gass Andrew Gaul Stephen M. Gava +Xavier de Gaye Harry Henry Gebel Marius Gedminas Thomas Gellekum diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,10 @@ Library ------- +- Issue #14798: Fix the functions in pyclbr to raise an ImportError + when the first part of a dotted name is not a package. Patch by + Xavier de Gaye. + - Issue #14829: Fix bisect and range() indexing with large indices (>= 2 ** 32) under 64-bit Windows. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 18 21:03:24 2012 From: python-checkins at python.org (petri.lehtinen) Date: Fri, 18 May 2012 21:03:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314798=3A_pyclbr_now_raises_ImportError_instead_of_KeyErro?= =?utf8?q?r_for_missing_packages?= Message-ID: http://hg.python.org/cpython/rev/2f51c15bbc56 changeset: 77043:2f51c15bbc56 parent: 77040:6286dd856252 parent: 77042:895246f1a06a user: Petri Lehtinen date: Fri May 18 21:51:11 2012 +0300 summary: #14798: pyclbr now raises ImportError instead of KeyError for missing packages files: Lib/pyclbr.py | 2 ++ Lib/test/test_pyclbr.py | 5 +++++ Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 12 insertions(+), 0 deletions(-) diff --git a/Lib/pyclbr.py b/Lib/pyclbr.py --- a/Lib/pyclbr.py +++ b/Lib/pyclbr.py @@ -130,6 +130,8 @@ parent = _readmodule(package, path, inpackage) if inpackage is not None: package = "%s.%s" % (inpackage, package) + if not '__path__' in parent: + raise ImportError('No package named {}'.format(package)) return _readmodule(submodule, parent['__path__'], package) # Search the path for the module diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -167,6 +167,11 @@ cm('email.parser') cm('test.test_pyclbr') + def test_issue_14798(self): + # test ImportError is raised when the first part of a dotted name is + # not a package + self.assertRaises(ImportError, pyclbr.readmodule_ex, 'asyncore.foo') + def test_main(): run_unittest(PyclbrTest) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -361,6 +361,7 @@ Dan Gass Andrew Gaul Stephen M. Gava +Xavier de Gaye Harry Henry Gebel Marius Gedminas Thomas Gellekum diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,10 @@ Library ------- +- Issue #14798: Fix the functions in pyclbr to raise an ImportError + when the first part of a dotted name is not a package. Patch by + Xavier de Gaye. + - Issue #12098: multiprocessing on Windows now starts child processes using the same sys.flags as the current process. Initial patch by Sergey Mezentsev. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 02:13:00 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 19 May 2012 02:13:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Issue9374_-_Gen?= =?utf8?q?eric_parsing_of_query_and_fragment_portion_of_urls_for_any_schem?= =?utf8?q?e?= Message-ID: http://hg.python.org/cpython/rev/79e6ff3d9afd changeset: 77044:79e6ff3d9afd branch: 2.7 parent: 77041:2d2079593212 user: Senthil Kumaran date: Sat May 19 08:10:40 2012 +0800 summary: Issue9374 - Generic parsing of query and fragment portion of urls for any scheme files: Lib/test/test_urlparse.py | 4 ++++ Lib/urlparse.py | 11 ++--------- Misc/NEWS | 3 +++ 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -493,6 +493,10 @@ ('s3','foo.com','/stuff','','','')) self.assertEqual(urlparse.urlparse("x-newscheme://foo.com/stuff"), ('x-newscheme','foo.com','/stuff','','','')) + self.assertEqual(urlparse.urlparse("x-newscheme://foo.com/stuff?query#fragment"), + ('x-newscheme','foo.com','/stuff','','query','fragment')) + self.assertEqual(urlparse.urlparse("x-newscheme://foo.com/stuff?query"), + ('x-newscheme','foo.com','/stuff','','query','')) def test_withoutscheme(self): # Test urlparse without scheme diff --git a/Lib/urlparse.py b/Lib/urlparse.py --- a/Lib/urlparse.py +++ b/Lib/urlparse.py @@ -40,16 +40,9 @@ 'imap', 'wais', 'file', 'mms', 'https', 'shttp', 'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', '', 'svn', 'svn+ssh', 'sftp','nfs','git', 'git+ssh'] -non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', - 'telnet', 'wais', 'imap', 'snews', 'sip', 'sips'] uses_params = ['ftp', 'hdl', 'prospero', 'http', 'imap', 'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips', 'mms', '', 'sftp'] -uses_query = ['http', 'wais', 'imap', 'https', 'shttp', 'mms', - 'gopher', 'rtsp', 'rtspu', 'sip', 'sips', ''] -uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', - 'nntp', 'wais', 'https', 'shttp', 'snews', - 'file', 'prospero', ''] # Characters valid in scheme names scheme_chars = ('abcdefghijklmnopqrstuvwxyz' @@ -204,9 +197,9 @@ if (('[' in netloc and ']' not in netloc) or (']' in netloc and '[' not in netloc)): raise ValueError("Invalid IPv6 URL") - if allow_fragments and scheme in uses_fragment and '#' in url: + if allow_fragments and '#' in url: url, fragment = url.split('#', 1) - if scheme in uses_query and '?' in url: + if '?' in url: url, query = url.split('?', 1) v = SplitResult(scheme, netloc, url, query, fragment) _parse_cache[key] = v diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #9374: Generic parsing of query and fragment portions of url for any + scheme. Supported both by RFC3986 and RFC2396. + - Issue #14798: Fix the functions in pyclbr to raise an ImportError when the first part of a dotted name is not a package. Patch by Xavier de Gaye. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 02:13:01 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 19 May 2012 02:13:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Issue9374_-_Gen?= =?utf8?q?eric_parsing_of_query_and_fragment_portion_of_urls_for_any_schem?= =?utf8?q?e?= Message-ID: http://hg.python.org/cpython/rev/a9d43e21f7d8 changeset: 77045:a9d43e21f7d8 branch: 3.2 parent: 77042:895246f1a06a user: Senthil Kumaran date: Sat May 19 08:12:00 2012 +0800 summary: Issue9374 - Generic parsing of query and fragment portion of urls for any scheme files: Lib/test/test_urlparse.py | 9 +++++++++ Lib/urllib/parse.py | 11 ++--------- Misc/NEWS | 3 +++ 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -636,11 +636,20 @@ ('s3', 'foo.com', '/stuff', '', '', '')) self.assertEqual(urllib.parse.urlparse("x-newscheme://foo.com/stuff"), ('x-newscheme', 'foo.com', '/stuff', '', '', '')) + self.assertEqual(urllib.parse.urlparse("x-newscheme://foo.com/stuff?query#fragment"), + ('x-newscheme', 'foo.com', '/stuff', '', 'query', 'fragment')) + self.assertEqual(urllib.parse.urlparse("x-newscheme://foo.com/stuff?query"), + ('x-newscheme', 'foo.com', '/stuff', '', 'query', '')) + # And for bytes... self.assertEqual(urllib.parse.urlparse(b"s3://foo.com/stuff"), (b's3', b'foo.com', b'/stuff', b'', b'', b'')) self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff"), (b'x-newscheme', b'foo.com', b'/stuff', b'', b'', b'')) + self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff?query#fragment"), + (b'x-newscheme', b'foo.com', b'/stuff', b'', b'query', b'fragment')) + self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff?query"), + (b'x-newscheme', b'foo.com', b'/stuff', b'', b'query', b'')) def test_mixed_types_rejected(self): # Several functions that process either strings or ASCII encoded bytes diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -44,16 +44,9 @@ 'imap', 'wais', 'file', 'mms', 'https', 'shttp', 'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', '', 'svn', 'svn+ssh', 'sftp', 'nfs', 'git', 'git+ssh'] -non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', - 'telnet', 'wais', 'imap', 'snews', 'sip', 'sips'] uses_params = ['ftp', 'hdl', 'prospero', 'http', 'imap', 'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips', 'mms', '', 'sftp'] -uses_query = ['http', 'wais', 'imap', 'https', 'shttp', 'mms', - 'gopher', 'rtsp', 'rtspu', 'sip', 'sips', ''] -uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', - 'nntp', 'wais', 'https', 'shttp', 'snews', - 'file', 'prospero', ''] # Characters valid in scheme names scheme_chars = ('abcdefghijklmnopqrstuvwxyz' @@ -357,9 +350,9 @@ if (('[' in netloc and ']' not in netloc) or (']' in netloc and '[' not in netloc)): raise ValueError("Invalid IPv6 URL") - if allow_fragments and scheme in uses_fragment and '#' in url: + if allow_fragments and '#' in url: url, fragment = url.split('#', 1) - if scheme in uses_query and '?' in url: + if '?' in url: url, query = url.split('?', 1) v = SplitResult(scheme, netloc, url, query, fragment) _parse_cache[key] = v diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ Library ------- +- Issue #9374: Generic parsing of query and fragment portions of url for any + scheme. Supported both by RFC3986 and RFC2396. + - Issue #14798: Fix the functions in pyclbr to raise an ImportError when the first part of a dotted name is not a package. Patch by Xavier de Gaye. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 02:13:04 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 19 May 2012 02:13:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue9374_-_Generic_parsing_of_query_and_fragment_portion_of?= =?utf8?q?_urls_for_any_scheme?= Message-ID: http://hg.python.org/cpython/rev/152c78b94e41 changeset: 77046:152c78b94e41 parent: 77043:2f51c15bbc56 parent: 77045:a9d43e21f7d8 user: Senthil Kumaran date: Sat May 19 08:12:46 2012 +0800 summary: Issue9374 - Generic parsing of query and fragment portion of urls for any scheme files: Lib/test/test_urlparse.py | 9 +++++++++ Lib/urllib/parse.py | 11 ++--------- Misc/NEWS | 3 +++ 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -636,11 +636,20 @@ ('s3', 'foo.com', '/stuff', '', '', '')) self.assertEqual(urllib.parse.urlparse("x-newscheme://foo.com/stuff"), ('x-newscheme', 'foo.com', '/stuff', '', '', '')) + self.assertEqual(urllib.parse.urlparse("x-newscheme://foo.com/stuff?query#fragment"), + ('x-newscheme', 'foo.com', '/stuff', '', 'query', 'fragment')) + self.assertEqual(urllib.parse.urlparse("x-newscheme://foo.com/stuff?query"), + ('x-newscheme', 'foo.com', '/stuff', '', 'query', '')) + # And for bytes... self.assertEqual(urllib.parse.urlparse(b"s3://foo.com/stuff"), (b's3', b'foo.com', b'/stuff', b'', b'', b'')) self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff"), (b'x-newscheme', b'foo.com', b'/stuff', b'', b'', b'')) + self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff?query#fragment"), + (b'x-newscheme', b'foo.com', b'/stuff', b'', b'query', b'fragment')) + self.assertEqual(urllib.parse.urlparse(b"x-newscheme://foo.com/stuff?query"), + (b'x-newscheme', b'foo.com', b'/stuff', b'', b'query', b'')) def test_mixed_types_rejected(self): # Several functions that process either strings or ASCII encoded bytes diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -44,16 +44,9 @@ 'imap', 'wais', 'file', 'mms', 'https', 'shttp', 'snews', 'prospero', 'rtsp', 'rtspu', 'rsync', '', 'svn', 'svn+ssh', 'sftp', 'nfs', 'git', 'git+ssh'] -non_hierarchical = ['gopher', 'hdl', 'mailto', 'news', - 'telnet', 'wais', 'imap', 'snews', 'sip', 'sips'] uses_params = ['ftp', 'hdl', 'prospero', 'http', 'imap', 'https', 'shttp', 'rtsp', 'rtspu', 'sip', 'sips', 'mms', '', 'sftp'] -uses_query = ['http', 'wais', 'imap', 'https', 'shttp', 'mms', - 'gopher', 'rtsp', 'rtspu', 'sip', 'sips', ''] -uses_fragment = ['ftp', 'hdl', 'http', 'gopher', 'news', - 'nntp', 'wais', 'https', 'shttp', 'snews', - 'file', 'prospero', ''] # Characters valid in scheme names scheme_chars = ('abcdefghijklmnopqrstuvwxyz' @@ -357,9 +350,9 @@ if (('[' in netloc and ']' not in netloc) or (']' in netloc and '[' not in netloc)): raise ValueError("Invalid IPv6 URL") - if allow_fragments and scheme in uses_fragment and '#' in url: + if allow_fragments and '#' in url: url, fragment = url.split('#', 1) - if scheme in uses_query and '?' in url: + if '?' in url: url, query = url.split('?', 1) v = SplitResult(scheme, netloc, url, query, fragment) _parse_cache[key] = v diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ Library ------- +- Issue #9374: Generic parsing of query and fragment portions of url for any + scheme. Supported both by RFC3986 and RFC2396. + - Issue #14798: Fix the functions in pyclbr to raise an ImportError when the first part of a dotted name is not a package. Patch by Xavier de Gaye. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat May 19 05:50:04 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 19 May 2012 05:50:04 +0200 Subject: [Python-checkins] Daily reference leaks (152c78b94e41): sum=0 Message-ID: results for 152c78b94e41 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog06sdkO', '-x'] From python-checkins at python.org Sat May 19 10:59:16 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 19 May 2012 10:59:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_Issue14721?= =?utf8?q?=3A_Send_Content-length=3A_0_for_empty_body_=28=29_in_the_http?= =?utf8?q?=2Erequest?= Message-ID: http://hg.python.org/cpython/rev/57f1d13c2cd4 changeset: 77047:57f1d13c2cd4 branch: 2.7 parent: 77044:79e6ff3d9afd user: Senthil Kumaran date: Sat May 19 16:52:21 2012 +0800 summary: Fix Issue14721: Send Content-length: 0 for empty body () in the http.request files: Lib/httplib.py | 2 +- Lib/test/test_httplib.py | 28 ++++++++++++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 32 insertions(+), 1 deletions(-) diff --git a/Lib/httplib.py b/Lib/httplib.py --- a/Lib/httplib.py +++ b/Lib/httplib.py @@ -989,7 +989,7 @@ self.putrequest(method, url, **skips) - if body and ('content-length' not in header_names): + if body is not None and 'content-length' not in header_names: self._set_content_length(body) for hdr, value in headers.iteritems(): self.putheader(hdr, value) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -90,6 +90,34 @@ conn.request('POST', '/', body, headers) self.assertEqual(conn._buffer.count[header.lower()], 1) + def test_content_length_0(self): + + class ContentLengthChecker(list): + def __init__(self): + list.__init__(self) + self.content_length = None + def append(self, item): + kv = item.split(':', 1) + if len(kv) > 1 and kv[0].lower() == 'content-length': + self.content_length = kv[1].strip() + list.append(self, item) + + # POST with empty body + conn = httplib.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn._buffer = ContentLengthChecker() + conn.request('POST', '/', '') + self.assertEqual(conn._buffer.content_length, '0', + 'Header Content-Length not set') + + # PUT request with empty body + conn = httplib.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn._buffer = ContentLengthChecker() + conn.request('PUT', '/', '') + self.assertEqual(conn._buffer.content_length, '0', + 'Header Content-Length not set') + def test_putheader(self): conn = httplib.HTTPConnection('example.com') conn.sock = FakeSocket(None) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #14721: Send proper header, Content-length: 0 when the body is an empty + string ''. Initial Patch contributed by Arve Knudsen. + - Issue #9374: Generic parsing of query and fragment portions of url for any scheme. Supported both by RFC3986 and RFC2396. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 10:59:16 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 19 May 2012 10:59:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_Issue14721?= =?utf8?q?=3A_Send_Content-length=3A_0_for_empty_body_=28=29_in_the_http?= =?utf8?q?=2Eclient?= Message-ID: http://hg.python.org/cpython/rev/6da1ab5f777d changeset: 77048:6da1ab5f777d branch: 3.2 parent: 77045:a9d43e21f7d8 user: Senthil Kumaran date: Sat May 19 16:58:09 2012 +0800 summary: Fix Issue14721: Send Content-length: 0 for empty body () in the http.client requests files: Lib/http/client.py | 2 +- Lib/test/test_httplib.py | 28 ++++++++++++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 32 insertions(+), 1 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -997,7 +997,7 @@ self.putrequest(method, url, **skips) - if body and ('content-length' not in header_names): + if body is not None and ('content-length' not in header_names): self._set_content_length(body) for hdr, value in headers.items(): self.putheader(hdr, value) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -99,6 +99,34 @@ conn.request('POST', '/', body, headers) self.assertEqual(conn._buffer.count[header.lower()], 1) + def test_content_length_0(self): + + class ContentLengthChecker(list): + def __init__(self): + list.__init__(self) + self.content_length = None + def append(self, item): + kv = item.split(b':', 1) + if len(kv) > 1 and kv[0].lower() == b'content-length': + self.content_length = kv[1].strip() + list.append(self, item) + + # POST with empty body + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn._buffer = ContentLengthChecker() + conn.request('POST', '/', '') + self.assertEqual(conn._buffer.content_length, b'0', + 'Header Content-Length not set') + + # PUT request with empty body + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn._buffer = ContentLengthChecker() + conn.request('PUT', '/', '') + self.assertEqual(conn._buffer.content_length, b'0', + 'Header Content-Length not set') + def test_putheader(self): conn = client.HTTPConnection('example.com') conn.sock = FakeSocket(None) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ Library ------- +- Issue #14721: Send the correct 'Content-length: 0' header when the body is an + empty string ''. Initial Patch contributed by Arve Knudsen. + - Issue #9374: Generic parsing of query and fragment portions of url for any scheme. Supported both by RFC3986 and RFC2396. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 10:59:17 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 19 May 2012 10:59:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_-_Fix_Issue14721=3A_Send_Content-length=3A_0_for_empty?= =?utf8?q?_body_=28=29_in_the?= Message-ID: http://hg.python.org/cpython/rev/732d70746fc0 changeset: 77049:732d70746fc0 parent: 77046:152c78b94e41 parent: 77048:6da1ab5f777d user: Senthil Kumaran date: Sat May 19 16:58:45 2012 +0800 summary: merge - Fix Issue14721: Send Content-length: 0 for empty body () in the http.client requests files: Lib/http/client.py | 2 +- Lib/test/test_httplib.py | 28 ++++++++++++++++++++++++++++ Misc/NEWS | 3 +++ 3 files changed, 32 insertions(+), 1 deletions(-) diff --git a/Lib/http/client.py b/Lib/http/client.py --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -1076,7 +1076,7 @@ self.putrequest(method, url, **skips) - if body and ('content-length' not in header_names): + if body is not None and ('content-length' not in header_names): self._set_content_length(body) for hdr, value in headers.items(): self.putheader(hdr, value) diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py --- a/Lib/test/test_httplib.py +++ b/Lib/test/test_httplib.py @@ -99,6 +99,34 @@ conn.request('POST', '/', body, headers) self.assertEqual(conn._buffer.count[header.lower()], 1) + def test_content_length_0(self): + + class ContentLengthChecker(list): + def __init__(self): + list.__init__(self) + self.content_length = None + def append(self, item): + kv = item.split(b':', 1) + if len(kv) > 1 and kv[0].lower() == b'content-length': + self.content_length = kv[1].strip() + list.append(self, item) + + # POST with empty body + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn._buffer = ContentLengthChecker() + conn.request('POST', '/', '') + self.assertEqual(conn._buffer.content_length, b'0', + 'Header Content-Length not set') + + # PUT request with empty body + conn = client.HTTPConnection('example.com') + conn.sock = FakeSocket(None) + conn._buffer = ContentLengthChecker() + conn.request('PUT', '/', '') + self.assertEqual(conn._buffer.content_length, b'0', + 'Header Content-Length not set') + def test_putheader(self): conn = client.HTTPConnection('example.com') conn.sock = FakeSocket(None) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ Library ------- +- Issue #14721: Send the correct 'Content-length: 0' header when the body is an + empty string ''. Initial Patch contributed by Arve Knudsen. + - Issue #9374: Generic parsing of query and fragment portions of url for any scheme. Supported both by RFC3986 and RFC2396. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 13:49:05 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sat, 19 May 2012 13:49:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313152=3A_Allow_to_specif?= =?utf8?q?y_a_custom_tabsize_for_expanding_tabs_in_textwrap?= Message-ID: http://hg.python.org/cpython/rev/d38e821c1b80 changeset: 77050:d38e821c1b80 user: Hynek Schlawack date: Sat May 19 13:33:11 2012 +0200 summary: #13152: Allow to specify a custom tabsize for expanding tabs in textwrap Patch by John Feuerstein. files: Doc/library/textwrap.rst | 9 +++++++++ Lib/test/test_textwrap.py | 8 ++++++++ Lib/textwrap.py | 13 +++++++++---- Misc/NEWS | 3 +++ 4 files changed, 29 insertions(+), 4 deletions(-) diff --git a/Doc/library/textwrap.rst b/Doc/library/textwrap.rst --- a/Doc/library/textwrap.rst +++ b/Doc/library/textwrap.rst @@ -107,6 +107,15 @@ expanded to spaces using the :meth:`expandtabs` method of *text*. + .. attribute:: tabsize + + (default: ``8``) If :attr:`expand_tabs` is true, then all tab characters + in *text* will be expanded to zero or more spaces, depending on the + current column and the given tab size. + + .. versionadded:: 3.3 + + .. attribute:: replace_whitespace (default: ``True``) If true, each whitespace character (as defined by diff --git a/Lib/test/test_textwrap.py b/Lib/test/test_textwrap.py --- a/Lib/test/test_textwrap.py +++ b/Lib/test/test_textwrap.py @@ -91,6 +91,14 @@ result = wrapper.fill(text) self.check(result, '\n'.join(expect)) + text = "\tTest\tdefault\t\ttabsize." + expect = [" Test default tabsize."] + self.check_wrap(text, 80, expect) + + text = "\tTest\tcustom\t\ttabsize." + expect = [" Test custom tabsize."] + self.check_wrap(text, 80, expect, tabsize=4) + def test_fix_sentence_endings(self): wrapper = TextWrapper(60, fix_sentence_endings=True) diff --git a/Lib/textwrap.py b/Lib/textwrap.py --- a/Lib/textwrap.py +++ b/Lib/textwrap.py @@ -39,8 +39,11 @@ of wrapped output; also counts towards each line's width. expand_tabs (default: true) Expand tabs in input text to spaces before further processing. - Each tab will become 1 .. 8 spaces, depending on its position in - its line. If false, each tab is treated as a single character. + Each tab will become 0 .. 'tabsize' spaces, depending on its position + in its line. If false, each tab is treated as a single character. + tabsize (default: 8) + Expand tabs in input text to 0 .. 'tabsize' spaces, unless + 'expand_tabs' is false. replace_whitespace (default: true) Replace all whitespace characters in the input text by spaces after tab expansion. Note that if expand_tabs is false and @@ -100,7 +103,8 @@ fix_sentence_endings=False, break_long_words=True, drop_whitespace=True, - break_on_hyphens=True): + break_on_hyphens=True, + tabsize=8): self.width = width self.initial_indent = initial_indent self.subsequent_indent = subsequent_indent @@ -110,6 +114,7 @@ self.break_long_words = break_long_words self.drop_whitespace = drop_whitespace self.break_on_hyphens = break_on_hyphens + self.tabsize = tabsize # -- Private methods ----------------------------------------------- @@ -123,7 +128,7 @@ becomes " foo bar baz". """ if self.expand_tabs: - text = text.expandtabs() + text = text.expandtabs(self.tabsize) if self.replace_whitespace: text = text.translate(self.unicode_whitespace_trans) return text diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -38,6 +38,9 @@ Library ------- +- Issue #13152: Allow to specify a custom tabsize for expanding tabs in + textwrap. Patch by John Feuerstein. + - Issue #14721: Send the correct 'Content-length: 0' header when the body is an empty string ''. Initial Patch contributed by Arve Knudsen. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 16:16:31 2012 From: python-checkins at python.org (ezio.melotti) Date: Sat, 19 May 2012 16:16:31 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0MDcyOiBGaXgg?= =?utf8?q?parsing_of_tel_URIs_in_urlparse_by_making_the_check_for_ports?= Message-ID: http://hg.python.org/cpython/rev/ff0fd7b26219 changeset: 77051:ff0fd7b26219 branch: 2.7 parent: 77047:57f1d13c2cd4 user: Ezio Melotti date: Sat May 19 17:12:17 2012 +0300 summary: #14072: Fix parsing of tel URIs in urlparse by making the check for ports stricter. files: Lib/test/test_urlparse.py | 7 +++++++ Lib/urlparse.py | 12 ++++++------ Misc/NEWS | 3 +++ 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -437,6 +437,13 @@ self.assertEqual(p.port, 80) self.assertEqual(p.geturl(), url) + def test_issue14072(self): + p1 = urlparse.urlsplit('tel:+31-641044153') + self.assertEqual(p1.scheme, 'tel') + self.assertEqual(p1.path, '+31-641044153') + p2 = urlparse.urlsplit('tel:+31641044153') + self.assertEqual(p2.scheme, 'tel') + self.assertEqual(p2.path, '+31641044153') def test_attributes_bad_port(self): """Check handling of non-integer ports.""" diff --git a/Lib/urlparse.py b/Lib/urlparse.py --- a/Lib/urlparse.py +++ b/Lib/urlparse.py @@ -185,12 +185,12 @@ if c not in scheme_chars: break else: - try: - # make sure "url" is not actually a port number (in which case - # "scheme" is really part of the path - _testportnum = int(url[i+1:]) - except ValueError: - scheme, url = url[:i].lower(), url[i+1:] + # make sure "url" is not actually a port number (in which case + # "scheme" is really part of the path) + rest = url[i+1:] + if not rest or any(c not in '0123456789' for c in rest): + # not a port number + scheme, url = url[:i].lower(), rest if url[:2] == '//': netloc, url = _splitnetloc(url, 2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -63,6 +63,9 @@ - Issue #14721: Send proper header, Content-length: 0 when the body is an empty string ''. Initial Patch contributed by Arve Knudsen. +- Issue #14072: Fix parsing of 'tel' URIs in urlparse by making the check for + ports stricter. + - Issue #9374: Generic parsing of query and fragment portions of url for any scheme. Supported both by RFC3986 and RFC2396. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 16:16:32 2012 From: python-checkins at python.org (ezio.melotti) Date: Sat, 19 May 2012 16:16:32 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0MDcyOiBGaXgg?= =?utf8?q?parsing_of_tel_URIs_in_urlparse_by_making_the_check_for_ports?= Message-ID: http://hg.python.org/cpython/rev/9f6b7576c08c changeset: 77052:9f6b7576c08c branch: 3.2 parent: 77048:6da1ab5f777d user: Ezio Melotti date: Sat May 19 17:15:19 2012 +0300 summary: #14072: Fix parsing of tel URIs in urlparse by making the check for ports stricter. files: Lib/test/test_urlparse.py | 7 +++++++ Lib/urllib/parse.py | 12 ++++++------ Misc/NEWS | 3 +++ 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -806,6 +806,13 @@ encoding='utf-8') self.assertRaises(TypeError, urllib.parse.quote, b'foo', errors='strict') + def test_issue14072(self): + p1 = urllib.parse.urlsplit('tel:+31-641044153') + self.assertEqual(p1.scheme, 'tel') + self.assertEqual(p1.path, '+31-641044153') + p2 = urllib.parse.urlsplit('tel:+31641044153') + self.assertEqual(p2.scheme, 'tel') + self.assertEqual(p2.path, '+31641044153') def test_main(): support.run_unittest(UrlParseTestCase) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -338,12 +338,12 @@ if c not in scheme_chars: break else: - try: - # make sure "url" is not actually a port number (in which case - # "scheme" is really part of the path - _testportnum = int(url[i+1:]) - except ValueError: - scheme, url = url[:i].lower(), url[i+1:] + # make sure "url" is not actually a port number (in which case + # "scheme" is really part of the path) + rest = url[i+1:] + if not rest or any(c not in '0123456789' for c in rest): + # not a port number + scheme, url = url[:i].lower(), rest if url[:2] == '//': netloc, url = _splitnetloc(url, 2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -66,6 +66,9 @@ - Issue #14721: Send the correct 'Content-length: 0' header when the body is an empty string ''. Initial Patch contributed by Arve Knudsen. +- Issue #14072: Fix parsing of 'tel' URIs in urlparse by making the check for + ports stricter. + - Issue #9374: Generic parsing of query and fragment portions of url for any scheme. Supported both by RFC3986 and RFC2396. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 16:16:33 2012 From: python-checkins at python.org (ezio.melotti) Date: Sat, 19 May 2012 16:16:33 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314072=3A_merge_with_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/b78c67665a7f changeset: 77053:b78c67665a7f parent: 77050:d38e821c1b80 parent: 77052:9f6b7576c08c user: Ezio Melotti date: Sat May 19 17:16:22 2012 +0300 summary: #14072: merge with 3.2. files: Lib/test/test_urlparse.py | 7 +++++++ Lib/urllib/parse.py | 12 ++++++------ Misc/NEWS | 3 +++ 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -806,6 +806,13 @@ encoding='utf-8') self.assertRaises(TypeError, urllib.parse.quote, b'foo', errors='strict') + def test_issue14072(self): + p1 = urllib.parse.urlsplit('tel:+31-641044153') + self.assertEqual(p1.scheme, 'tel') + self.assertEqual(p1.path, '+31-641044153') + p2 = urllib.parse.urlsplit('tel:+31641044153') + self.assertEqual(p2.scheme, 'tel') + self.assertEqual(p2.path, '+31641044153') def test_main(): support.run_unittest(UrlParseTestCase) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -338,12 +338,12 @@ if c not in scheme_chars: break else: - try: - # make sure "url" is not actually a port number (in which case - # "scheme" is really part of the path - _testportnum = int(url[i+1:]) - except ValueError: - scheme, url = url[:i].lower(), url[i+1:] + # make sure "url" is not actually a port number (in which case + # "scheme" is really part of the path) + rest = url[i+1:] + if not rest or any(c not in '0123456789' for c in rest): + # not a port number + scheme, url = url[:i].lower(), rest if url[:2] == '//': netloc, url = _splitnetloc(url, 2) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ - Issue #14721: Send the correct 'Content-length: 0' header when the body is an empty string ''. Initial Patch contributed by Arve Knudsen. +- Issue #14072: Fix parsing of 'tel' URIs in urlparse by making the check for + ports stricter. + - Issue #9374: Generic parsing of query and fragment portions of url for any scheme. Supported both by RFC3986 and RFC2396. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 17:41:50 2012 From: python-checkins at python.org (petri.lehtinen) Date: Sat, 19 May 2012 17:41:50 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0NDk0OiBEb2N1?= =?utf8?q?ment_that_absolute_imports_became_default_in_3=2E0_instead_of_2?= =?utf8?b?Ljcu?= Message-ID: http://hg.python.org/cpython/rev/cc9e5ddd8220 changeset: 77054:cc9e5ddd8220 branch: 2.7 parent: 77051:ff0fd7b26219 user: Petri Lehtinen date: Sat May 19 18:34:06 2012 +0300 summary: #14494: Document that absolute imports became default in 3.0 instead of 2.7. files: Doc/library/__future__.rst | 2 +- Lib/__future__.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -75,7 +75,7 @@ | division | 2.2.0a2 | 3.0 | :pep:`238`: | | | | | *Changing the Division Operator* | +------------------+-------------+--------------+---------------------------------------------+ -| absolute_import | 2.5.0a1 | 2.7 | :pep:`328`: | +| absolute_import | 2.5.0a1 | 3.0 | :pep:`328`: | | | | | *Imports: Multi-Line and Absolute/Relative* | +------------------+-------------+--------------+---------------------------------------------+ | with_statement | 2.5.0a1 | 2.6 | :pep:`343`: | diff --git a/Lib/__future__.py b/Lib/__future__.py --- a/Lib/__future__.py +++ b/Lib/__future__.py @@ -112,7 +112,7 @@ CO_FUTURE_DIVISION) absolute_import = _Feature((2, 5, 0, "alpha", 1), - (2, 7, 0, "alpha", 0), + (3, 0, 0, "alpha", 0), CO_FUTURE_ABSOLUTE_IMPORT) with_statement = _Feature((2, 5, 0, "alpha", 1), diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -544,6 +544,7 @@ Vladimir Marangozov David Marek Doug Marien +Sven Marnach Alex Martelli Anthony Martin S?bastien Martini diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,10 @@ Core and Builtins ----------------- +- Issue #14494: Fix __future__.py and its documentation to note that + absolute imports are the default behavior in 3.0 instead of 2.7. + Patch by Sven Marnach. + - Issue #14761: Fix potential leak on an error case in the import machinery. - Issue #14699: Fix calling the classmethod descriptor directly. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 17:41:51 2012 From: python-checkins at python.org (petri.lehtinen) Date: Sat, 19 May 2012 17:41:51 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NDk0OiBEb2N1?= =?utf8?q?ment_that_absolute_imports_became_default_in_3=2E0_instead_of_2?= =?utf8?b?Ljcu?= Message-ID: http://hg.python.org/cpython/rev/7cdc1392173f changeset: 77055:7cdc1392173f branch: 3.2 parent: 77052:9f6b7576c08c user: Petri Lehtinen date: Sat May 19 18:34:06 2012 +0300 summary: #14494: Document that absolute imports became default in 3.0 instead of 2.7. files: Doc/library/__future__.rst | 2 +- Lib/__future__.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -75,7 +75,7 @@ | division | 2.2.0a2 | 3.0 | :pep:`238`: | | | | | *Changing the Division Operator* | +------------------+-------------+--------------+---------------------------------------------+ -| absolute_import | 2.5.0a1 | 2.7 | :pep:`328`: | +| absolute_import | 2.5.0a1 | 3.0 | :pep:`328`: | | | | | *Imports: Multi-Line and Absolute/Relative* | +------------------+-------------+--------------+---------------------------------------------+ | with_statement | 2.5.0a1 | 2.6 | :pep:`343`: | diff --git a/Lib/__future__.py b/Lib/__future__.py --- a/Lib/__future__.py +++ b/Lib/__future__.py @@ -114,7 +114,7 @@ CO_FUTURE_DIVISION) absolute_import = _Feature((2, 5, 0, "alpha", 1), - (2, 7, 0, "alpha", 0), + (3, 0, 0, "alpha", 0), CO_FUTURE_ABSOLUTE_IMPORT) with_statement = _Feature((2, 5, 0, "alpha", 1), diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -598,6 +598,7 @@ Vladimir Marangozov David Marek Doug Marien +Sven Marnach Alex Martelli Anthony Martin Owen Martin diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #14494: Fix __future__.py and its documentation to note that + absolute imports are the default behavior in 3.0 instead of 2.7. + Patch by Sven Marnach. + - Issue #14761: Fix potential leak on an error case in the import machinery. - Issue #14699: Fix calling the classmethod descriptor directly. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 17:41:53 2012 From: python-checkins at python.org (petri.lehtinen) Date: Sat, 19 May 2012 17:41:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314494=3A_Document_that_absolute_imports_became_default_in?= =?utf8?q?_3=2E0_instead_of_2=2E7=2E?= Message-ID: http://hg.python.org/cpython/rev/26661d9bbb36 changeset: 77056:26661d9bbb36 parent: 77053:b78c67665a7f parent: 77055:7cdc1392173f user: Petri Lehtinen date: Sat May 19 18:38:27 2012 +0300 summary: #14494: Document that absolute imports became default in 3.0 instead of 2.7. files: Doc/library/__future__.rst | 2 +- Lib/__future__.py | 2 +- Misc/NEWS | 4 ++++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -75,7 +75,7 @@ | division | 2.2.0a2 | 3.0 | :pep:`238`: | | | | | *Changing the Division Operator* | +------------------+-------------+--------------+---------------------------------------------+ -| absolute_import | 2.5.0a1 | 2.7 | :pep:`328`: | +| absolute_import | 2.5.0a1 | 3.0 | :pep:`328`: | | | | | *Imports: Multi-Line and Absolute/Relative* | +------------------+-------------+--------------+---------------------------------------------+ | with_statement | 2.5.0a1 | 2.6 | :pep:`343`: | diff --git a/Lib/__future__.py b/Lib/__future__.py --- a/Lib/__future__.py +++ b/Lib/__future__.py @@ -114,7 +114,7 @@ CO_FUTURE_DIVISION) absolute_import = _Feature((2, 5, 0, "alpha", 1), - (2, 7, 0, "alpha", 0), + (3, 0, 0, "alpha", 0), CO_FUTURE_ABSOLUTE_IMPORT) with_statement = _Feature((2, 5, 0, "alpha", 1), diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,10 @@ Core and Builtins ----------------- +- Issue #14494: Fix __future__.py and its documentation to note that + absolute imports are the default behavior in 3.0 instead of 2.7. + Patch by Sven Marnach. + - Issue #9260: A finer-grained import lock. Most of the import sequence now uses per-module locks rather than the global import lock, eliminating well-known issues with threads and imports. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 17:53:07 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 19 May 2012 17:53:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Ignore_exceptio?= =?utf8?q?ns_on_test_handler_closing=2E?= Message-ID: http://hg.python.org/cpython/rev/9ab9bf20a4ea changeset: 77057:9ab9bf20a4ea branch: 2.7 parent: 77051:ff0fd7b26219 user: Vinay Sajip date: Sat May 19 16:50:07 2012 +0100 summary: Ignore exceptions on test handler closing. files: Lib/test/test_logging.py | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1907,7 +1907,10 @@ h.handle(r) finally: remover.join() - h.close() + try: + h.close() + except ValueError: + pass if os.path.exists(fn): os.unlink(fn) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 17:53:08 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 19 May 2012 17:53:08 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf8?q?_Merged_upstream_changes=2E?= Message-ID: http://hg.python.org/cpython/rev/4641d8d99a7d changeset: 77058:4641d8d99a7d branch: 2.7 parent: 77057:9ab9bf20a4ea parent: 77054:cc9e5ddd8220 user: Vinay Sajip date: Sat May 19 16:52:33 2012 +0100 summary: Merged upstream changes. files: Doc/library/__future__.rst | 2 +- Lib/__future__.py | 2 +- Misc/ACKS | 1 + Misc/NEWS | 4 ++++ 4 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Doc/library/__future__.rst b/Doc/library/__future__.rst --- a/Doc/library/__future__.rst +++ b/Doc/library/__future__.rst @@ -75,7 +75,7 @@ | division | 2.2.0a2 | 3.0 | :pep:`238`: | | | | | *Changing the Division Operator* | +------------------+-------------+--------------+---------------------------------------------+ -| absolute_import | 2.5.0a1 | 2.7 | :pep:`328`: | +| absolute_import | 2.5.0a1 | 3.0 | :pep:`328`: | | | | | *Imports: Multi-Line and Absolute/Relative* | +------------------+-------------+--------------+---------------------------------------------+ | with_statement | 2.5.0a1 | 2.6 | :pep:`343`: | diff --git a/Lib/__future__.py b/Lib/__future__.py --- a/Lib/__future__.py +++ b/Lib/__future__.py @@ -112,7 +112,7 @@ CO_FUTURE_DIVISION) absolute_import = _Feature((2, 5, 0, "alpha", 1), - (2, 7, 0, "alpha", 0), + (3, 0, 0, "alpha", 0), CO_FUTURE_ABSOLUTE_IMPORT) with_statement = _Feature((2, 5, 0, "alpha", 1), diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -544,6 +544,7 @@ Vladimir Marangozov David Marek Doug Marien +Sven Marnach Alex Martelli Anthony Martin S?bastien Martini diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,10 @@ Core and Builtins ----------------- +- Issue #14494: Fix __future__.py and its documentation to note that + absolute imports are the default behavior in 3.0 instead of 2.7. + Patch by Sven Marnach. + - Issue #14761: Fix potential leak on an error case in the import machinery. - Issue #14699: Fix calling the classmethod descriptor directly. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 18:34:26 2012 From: python-checkins at python.org (nick.coghlan) Date: Sat, 19 May 2012 18:34:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314588=3A_added_a_P?= =?utf8?q?EP_3115_compliant_dynamic_type_creation_mechanism?= Message-ID: http://hg.python.org/cpython/rev/befd56673c80 changeset: 77059:befd56673c80 parent: 77056:26661d9bbb36 user: Nick Coghlan date: Sun May 20 02:34:13 2012 +1000 summary: Close #14588: added a PEP 3115 compliant dynamic type creation mechanism files: Doc/library/functions.rst | 10 +- Doc/library/types.rst | 67 +++++- Doc/reference/datamodel.rst | 157 ++++++++++---- Doc/whatsnew/3.3.rst | 4 + Lib/test/test_types.py | 251 +++++++++++++++++++++++- Lib/types.py | 58 +++++ Misc/NEWS | 11 + 7 files changed, 497 insertions(+), 61 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1324,10 +1324,12 @@ Accordingly, :func:`super` is undefined for implicit lookups using statements or operators such as ``super()[name]``. - Also note that :func:`super` is not limited to use inside methods. The two - argument form specifies the arguments exactly and makes the appropriate - references. The zero argument form automatically searches the stack frame - for the class (``__class__``) and the first argument. + Also note that, aside from the zero argument form, :func:`super` is not + limited to use inside methods. The two argument form specifies the + arguments exactly and makes the appropriate references. The zero + argument form only works inside a class definition, as the compiler fills + in the necessary details to correctly retrieve the class being defined, + as well as accessing the current instance for ordinary methods. For practical suggestions on how to design cooperative classes using :func:`super`, see `guide to using super() diff --git a/Doc/library/types.rst b/Doc/library/types.rst --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -1,5 +1,5 @@ -:mod:`types` --- Names for built-in types -========================================= +:mod:`types` --- Dynamic type creation and names for built-in types +=================================================================== .. module:: types :synopsis: Names for built-in types. @@ -8,20 +8,69 @@ -------------- -This module defines names for some object types that are used by the standard +This module defines utility function to assist in dynamic creation of +new types. + +It also defines names for some object types that are used by the standard Python interpreter, but not exposed as builtins like :class:`int` or -:class:`str` are. Also, it does not include some of the types that arise -transparently during processing such as the ``listiterator`` type. +:class:`str` are. -Typical use is for :func:`isinstance` or :func:`issubclass` checks. -The module defines the following names: +Dynamic Type Creation +--------------------- + +.. function:: new_class(name, bases=(), kwds=None, exec_body=None) + + Creates a class object dynamically using the appropriate metaclass. + + The arguments are the components that make up a class definition: the + class name, the base classes (in order), the keyword arguments (such as + ``metaclass``) and the callback function to populate the class namespace. + + The *exec_body* callback should accept the class namespace as its sole + argument and update the namespace directly with the class contents. + +.. function:: prepare_class(name, bases=(), kwds=None) + + Calculates the appropriate metaclass and creates the class namespace. + + The arguments are the components that make up a class definition: the + class name, the base classes (in order) and the keyword arguments (such as + ``metaclass``). + + The return value is a 3-tuple: ``metaclass, namespace, kwds`` + + *metaclass* is the appropriate metaclass + *namespace* is the prepared class namespace + *kwds* is an updated copy of the passed in *kwds* argument with any + ``'metaclass'`` entry removed. If no *kwds* argument is passed in, this + will be an empty dict. + + +.. seealso:: + + :pep:`3115` - Metaclasses in Python 3000 + Introduced the ``__prepare__`` namespace hook + + +Standard Interpreter Types +-------------------------- + +This module provides names for many of the types that are required to +implement a Python interpreter. It deliberately avoids including some of +the types that arise only incidentally during processing such as the +``listiterator`` type. + +Typical use is of these names is for :func:`isinstance` or +:func:`issubclass` checks. + +Standard names are defined for the following types: .. data:: FunctionType LambdaType - The type of user-defined functions and functions created by :keyword:`lambda` - expressions. + The type of user-defined functions and functions created by + :keyword:`lambda` expressions. .. data:: GeneratorType diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1550,53 +1550,115 @@ Customizing class creation -------------------------- -By default, classes are constructed using :func:`type`. A class definition is -read into a separate namespace and the value of class name is bound to the -result of ``type(name, bases, dict)``. - -When the class definition is read, if a callable ``metaclass`` keyword argument -is passed after the bases in the class definition, the callable given will be -called instead of :func:`type`. If other keyword arguments are passed, they -will also be passed to the metaclass. This allows classes or functions to be -written which monitor or alter the class creation process: - -* Modifying the class dictionary prior to the class being created. - -* Returning an instance of another class -- essentially performing the role of a - factory function. - -These steps will have to be performed in the metaclass's :meth:`__new__` method --- :meth:`type.__new__` can then be called from this method to create a class -with different properties. This example adds a new element to the class -dictionary before creating the class:: - - class metacls(type): - def __new__(mcs, name, bases, dict): - dict['foo'] = 'metacls was here' - return type.__new__(mcs, name, bases, dict) - -You can of course also override other class methods (or add new methods); for -example defining a custom :meth:`__call__` method in the metaclass allows custom -behavior when the class is called, e.g. not always creating a new instance. - -If the metaclass has a :meth:`__prepare__` attribute (usually implemented as a -class or static method), it is called before the class body is evaluated with -the name of the class and a tuple of its bases for arguments. It should return -an object that supports the mapping interface that will be used to store the -namespace of the class. The default is a plain dictionary. This could be used, -for example, to keep track of the order that class attributes are declared in by -returning an ordered dictionary. - -The appropriate metaclass is determined by the following precedence rules: - -* If the ``metaclass`` keyword argument is passed with the bases, it is used. - -* Otherwise, if there is at least one base class, its metaclass is used. - -* Otherwise, the default metaclass (:class:`type`) is used. +By default, classes are constructed using :func:`type`. The class body is +executed in a new namespace and the class name is bound locally to the +result of ``type(name, bases, namespace)``. + +The class creation process can be customised by passing the ``metaclass`` +keyword argument in the class definition line, or by inheriting from an +existing class that included such an argument. In the following example, +both ``MyClass`` and ``MySubclass`` are instances of ``Meta``:: + + class Meta(type): + pass + + class MyClass(metaclass=Meta): + pass + + class MySubclass(MyClass): + pass + +Any other keyword arguments that are specified in the class definition are +passed through to all metaclass operations described below. + +When a class definition is executed, the following steps occur: + +* the appropriate metaclass is determined +* the class namespace is prepared +* the class body is executed +* the class object is created + +Determining the appropriate metaclass +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The appropriate metaclass for a class definition is determined as follows: + +* if no bases and no explicit metaclass are given, then :func:`type` is used +* if an explicit metaclass is given and it is *not* an instance of + :func:`type`, then it is used directly as the metaclass +* if an instance of :func:`type` is given as the explicit metaclass, or + bases are defined, then the most derived metaclass is used + +The most derived metaclass is selected from the explicitly specified +metaclass (if any) and the metaclasses (i.e. ``type(cls)``) of all specified +base classes. The most derived metaclass is one which is a subtype of *all* +of these candidate metaclasses. If none of the candidate metaclasses meets +that criterion, then the class definition will fail with ``TypeError``. + + +Preparing the class namespace +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once the appropriate metaclass has been identified, then the class namespace +is prepared. If the metaclass has a ``__prepare__`` attribute, it is called +as ``namespace = metaclass.__prepare__(name, bases, **kwds)`` (where the +additional keyword arguments, if any, come from the class definition). + +If the metaclass has no ``__prepare__`` attribute, then the class namespace +is initialised as an empty :func:`dict` instance. + +.. seealso:: + + :pep:`3115` - Metaclasses in Python 3000 + Introduced the ``__prepare__`` namespace hook + + +Executing the class body +^^^^^^^^^^^^^^^^^^^^^^^^ + +The class body is executed (approximately) as +``exec(body, globals(), namespace)``. The key difference from a normal +call to :func:`exec` is that lexical scoping allows the class body (including +any methods) to reference names from the current and outer scopes when the +class definition occurs inside a function. + +However, even when the class definition occurs inside the function, methods +defined inside the class still cannot see names defined at the class scope. +Class variables must be accessed through the first parameter of instance or +class methods, and cannot be accessed at all from static methods. + + +Creating the class object +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once the class namespace has been populated by executing the class body, +the class object is created by calling +``metaclass(name, bases, namespace, **kwds)`` (the additional keywords +passed here are the same as those passed to ``__prepate__``). + +This class object is the one that will be referenced by the zero-argument +form of :func:`super`. ``__class__`` is an implicit closure reference +created by the compiler if any methods in a class body refer to either +``__class__`` or ``super``. This allows the zero argument form of +:func:`super` to correctly identify the class being defined based on +lexical scoping, while the class or instance that was used to make the +current call is identified based on the first argument passed to the method. + +After the class object is created, any class decorators included in the +function definition are invoked and the resulting object is bound in the +local namespace to the name of the class. + +.. seealso:: + + :pep:`3135` - New super + Describes the implicit ``__class__`` closure reference + + +Metaclass example +^^^^^^^^^^^^^^^^^ The potential uses for metaclasses are boundless. Some ideas that have been -explored including logging, interface checking, automatic delegation, automatic +explored include logging, interface checking, automatic delegation, automatic property creation, proxies, frameworks, and automatic resource locking/synchronization. @@ -1609,9 +1671,9 @@ def __prepare__(metacls, name, bases, **kwds): return collections.OrderedDict() - def __new__(cls, name, bases, classdict): - result = type.__new__(cls, name, bases, dict(classdict)) - result.members = tuple(classdict) + def __new__(cls, name, bases, namespace, **kwds): + result = type.__new__(cls, name, bases, dict(namespace)) + result.members = tuple(namespace) return result class A(metaclass=OrderedClass): diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1239,6 +1239,10 @@ (:issue:`14386`) +The new functions `types.new_class` and `types.prepare_class` provide support +for PEP 3115 compliant dynamic type creation. (:issue:`14588`) + + urllib ------ diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -747,8 +747,257 @@ self.assertEqual(copy['key1'], 27) +class ClassCreationTests(unittest.TestCase): + + class Meta(type): + def __init__(cls, name, bases, ns, **kw): + super().__init__(name, bases, ns) + @staticmethod + def __new__(mcls, name, bases, ns, **kw): + return super().__new__(mcls, name, bases, ns) + @classmethod + def __prepare__(mcls, name, bases, **kw): + ns = super().__prepare__(name, bases) + ns["y"] = 1 + ns.update(kw) + return ns + + def test_new_class_basics(self): + C = types.new_class("C") + self.assertEqual(C.__name__, "C") + self.assertEqual(C.__bases__, (object,)) + + def test_new_class_subclass(self): + C = types.new_class("C", (int,)) + self.assertTrue(issubclass(C, int)) + + def test_new_class_meta(self): + Meta = self.Meta + settings = {"metaclass": Meta, "z": 2} + # We do this twice to make sure the passed in dict isn't mutated + for i in range(2): + C = types.new_class("C" + str(i), (), settings) + self.assertIsInstance(C, Meta) + self.assertEqual(C.y, 1) + self.assertEqual(C.z, 2) + + def test_new_class_exec_body(self): + Meta = self.Meta + def func(ns): + ns["x"] = 0 + C = types.new_class("C", (), {"metaclass": Meta, "z": 2}, func) + self.assertIsInstance(C, Meta) + self.assertEqual(C.x, 0) + self.assertEqual(C.y, 1) + self.assertEqual(C.z, 2) + + def test_new_class_exec_body(self): + #Test that keywords are passed to the metaclass: + def meta_func(name, bases, ns, **kw): + return name, bases, ns, kw + res = types.new_class("X", + (int, object), + dict(metaclass=meta_func, x=0)) + self.assertEqual(res, ("X", (int, object), {}, {"x": 0})) + + def test_new_class_defaults(self): + # Test defaults/keywords: + C = types.new_class("C", (), {}, None) + self.assertEqual(C.__name__, "C") + self.assertEqual(C.__bases__, (object,)) + + def test_new_class_meta_with_base(self): + Meta = self.Meta + def func(ns): + ns["x"] = 0 + C = types.new_class(name="C", + bases=(int,), + kwds=dict(metaclass=Meta, z=2), + exec_body=func) + self.assertTrue(issubclass(C, int)) + self.assertIsInstance(C, Meta) + self.assertEqual(C.x, 0) + self.assertEqual(C.y, 1) + self.assertEqual(C.z, 2) + + # Many of the following tests are derived from test_descr.py + def test_prepare_class(self): + # Basic test of metaclass derivation + expected_ns = {} + class A(type): + def __new__(*args, **kwargs): + return type.__new__(*args, **kwargs) + + def __prepare__(*args): + return expected_ns + + B = types.new_class("B", (object,)) + C = types.new_class("C", (object,), {"metaclass": A}) + + # The most derived metaclass of D is A rather than type. + meta, ns, kwds = types.prepare_class("D", (B, C), {"metaclass": type}) + self.assertIs(meta, A) + self.assertIs(ns, expected_ns) + self.assertEqual(len(kwds), 0) + + def test_metaclass_derivation(self): + # issue1294232: correct metaclass calculation + new_calls = [] # to check the order of __new__ calls + class AMeta(type): + def __new__(mcls, name, bases, ns): + new_calls.append('AMeta') + return super().__new__(mcls, name, bases, ns) + @classmethod + def __prepare__(mcls, name, bases): + return {} + + class BMeta(AMeta): + def __new__(mcls, name, bases, ns): + new_calls.append('BMeta') + return super().__new__(mcls, name, bases, ns) + @classmethod + def __prepare__(mcls, name, bases): + ns = super().__prepare__(name, bases) + ns['BMeta_was_here'] = True + return ns + + A = types.new_class("A", (), {"metaclass": AMeta}) + self.assertEqual(new_calls, ['AMeta']) + new_calls.clear() + + B = types.new_class("B", (), {"metaclass": BMeta}) + # BMeta.__new__ calls AMeta.__new__ with super: + self.assertEqual(new_calls, ['BMeta', 'AMeta']) + new_calls.clear() + + C = types.new_class("C", (A, B)) + # The most derived metaclass is BMeta: + self.assertEqual(new_calls, ['BMeta', 'AMeta']) + new_calls.clear() + # BMeta.__prepare__ should've been called: + self.assertIn('BMeta_was_here', C.__dict__) + + # The order of the bases shouldn't matter: + C2 = types.new_class("C2", (B, A)) + self.assertEqual(new_calls, ['BMeta', 'AMeta']) + new_calls.clear() + self.assertIn('BMeta_was_here', C2.__dict__) + + # Check correct metaclass calculation when a metaclass is declared: + D = types.new_class("D", (C,), {"metaclass": type}) + self.assertEqual(new_calls, ['BMeta', 'AMeta']) + new_calls.clear() + self.assertIn('BMeta_was_here', D.__dict__) + + E = types.new_class("E", (C,), {"metaclass": AMeta}) + self.assertEqual(new_calls, ['BMeta', 'AMeta']) + new_calls.clear() + self.assertIn('BMeta_was_here', E.__dict__) + + def test_metaclass_override_function(self): + # Special case: the given metaclass isn't a class, + # so there is no metaclass calculation. + class A(metaclass=self.Meta): + pass + + marker = object() + def func(*args, **kwargs): + return marker + + X = types.new_class("X", (), {"metaclass": func}) + Y = types.new_class("Y", (object,), {"metaclass": func}) + Z = types.new_class("Z", (A,), {"metaclass": func}) + self.assertIs(marker, X) + self.assertIs(marker, Y) + self.assertIs(marker, Z) + + def test_metaclass_override_callable(self): + # The given metaclass is a class, + # but not a descendant of type. + new_calls = [] # to check the order of __new__ calls + prepare_calls = [] # to track __prepare__ calls + class ANotMeta: + def __new__(mcls, *args, **kwargs): + new_calls.append('ANotMeta') + return super().__new__(mcls) + @classmethod + def __prepare__(mcls, name, bases): + prepare_calls.append('ANotMeta') + return {} + + class BNotMeta(ANotMeta): + def __new__(mcls, *args, **kwargs): + new_calls.append('BNotMeta') + return super().__new__(mcls) + @classmethod + def __prepare__(mcls, name, bases): + prepare_calls.append('BNotMeta') + return super().__prepare__(name, bases) + + A = types.new_class("A", (), {"metaclass": ANotMeta}) + self.assertIs(ANotMeta, type(A)) + self.assertEqual(prepare_calls, ['ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['ANotMeta']) + new_calls.clear() + + B = types.new_class("B", (), {"metaclass": BNotMeta}) + self.assertIs(BNotMeta, type(B)) + self.assertEqual(prepare_calls, ['BNotMeta', 'ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['BNotMeta', 'ANotMeta']) + new_calls.clear() + + C = types.new_class("C", (A, B)) + self.assertIs(BNotMeta, type(C)) + self.assertEqual(prepare_calls, ['BNotMeta', 'ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['BNotMeta', 'ANotMeta']) + new_calls.clear() + + C2 = types.new_class("C2", (B, A)) + self.assertIs(BNotMeta, type(C2)) + self.assertEqual(prepare_calls, ['BNotMeta', 'ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['BNotMeta', 'ANotMeta']) + new_calls.clear() + + # This is a TypeError, because of a metaclass conflict: + # BNotMeta is neither a subclass, nor a superclass of type + with self.assertRaises(TypeError): + D = types.new_class("D", (C,), {"metaclass": type}) + + E = types.new_class("E", (C,), {"metaclass": ANotMeta}) + self.assertIs(BNotMeta, type(E)) + self.assertEqual(prepare_calls, ['BNotMeta', 'ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['BNotMeta', 'ANotMeta']) + new_calls.clear() + + F = types.new_class("F", (object(), C)) + self.assertIs(BNotMeta, type(F)) + self.assertEqual(prepare_calls, ['BNotMeta', 'ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['BNotMeta', 'ANotMeta']) + new_calls.clear() + + F2 = types.new_class("F2", (C, object())) + self.assertIs(BNotMeta, type(F2)) + self.assertEqual(prepare_calls, ['BNotMeta', 'ANotMeta']) + prepare_calls.clear() + self.assertEqual(new_calls, ['BNotMeta', 'ANotMeta']) + new_calls.clear() + + # TypeError: BNotMeta is neither a + # subclass, nor a superclass of int + with self.assertRaises(TypeError): + X = types.new_class("X", (C, int())) + with self.assertRaises(TypeError): + X = types.new_class("X", (int(), C)) + + def test_main(): - run_unittest(TypesTests, MappingProxyTests) + run_unittest(TypesTests, MappingProxyTests, ClassCreationTests) if __name__ == '__main__': test_main() diff --git a/Lib/types.py b/Lib/types.py --- a/Lib/types.py +++ b/Lib/types.py @@ -40,3 +40,61 @@ MemberDescriptorType = type(FunctionType.__globals__) del sys, _f, _g, _C, # Not for export + + +# Provide a PEP 3115 compliant mechanism for class creation +def new_class(name, bases=(), kwds=None, exec_body=None): + """Create a class object dynamically using the appropriate metaclass.""" + meta, ns, kwds = prepare_class(name, bases, kwds) + if exec_body is not None: + exec_body(ns) + return meta(name, bases, ns, **kwds) + +def prepare_class(name, bases=(), kwds=None): + """Call the __prepare__ method of the appropriate metaclass. + + Returns (metaclass, namespace, kwds) as a 3-tuple + + *metaclass* is the appropriate metaclass + *namespace* is the prepared class namespace + *kwds* is an updated copy of the passed in kwds argument with any + 'metaclass' entry removed. If no kwds argument is passed in, this will + be an empty dict. + """ + if kwds is None: + kwds = {} + else: + kwds = dict(kwds) # Don't alter the provided mapping + if 'metaclass' in kwds: + meta = kwds.pop('metaclass') + else: + if bases: + meta = type(bases[0]) + else: + meta = type + if isinstance(meta, type): + # when meta is a type, we first determine the most-derived metaclass + # instead of invoking the initial candidate directly + meta = _calculate_meta(meta, bases) + if hasattr(meta, '__prepare__'): + ns = meta.__prepare__(name, bases, **kwds) + else: + ns = {} + return meta, ns, kwds + +def _calculate_meta(meta, bases): + """Calculate the most derived metaclass.""" + winner = meta + for base in bases: + base_meta = type(base) + if issubclass(winner, base_meta): + continue + if issubclass(base_meta, winner): + winner = base_meta + continue + # else: + raise TypeError("metaclass conflict: " + "the metaclass of a derived class " + "must be a (non-strict) subclass " + "of the metaclasses of all its bases") + return winner diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,10 @@ Library ------- +- Issue #14588: The types module now provide new_class() and prepare_class() + functions to support PEP 3115 compliant dynamic class creation. Patch by + Daniel Urban and Nick Coghlan. + - Issue #13152: Allow to specify a custom tabsize for expanding tabs in textwrap. Patch by John Feuerstein. @@ -166,6 +170,13 @@ - Issue #13210: Windows build now uses VS2010, ported from VS2008. +Documentation +------------- + +- Issue #14588: The language reference now accurately documents the Python 3 + class definition process. Patch by Nick Coghlan. + + What's New in Python 3.3.0 Alpha 3? =================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 23:11:38 2012 From: python-checkins at python.org (kristjan.jonsson) Date: Sat, 19 May 2012 23:11:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Clean_up_the_PCBuild_projec?= =?utf8?q?t_files=2C_removing_redundant_settings_and?= Message-ID: http://hg.python.org/cpython/rev/40165fa89db8 changeset: 77060:40165fa89db8 user: Kristj?n Valur J?nsson date: Sat May 19 21:10:14 2012 +0000 summary: Clean up the PCBuild project files, removing redundant settings and use "references" to link to dependent projects. Update readme and batch files. files: PCbuild/_bz2.vcxproj | 6 -- PCbuild/_ctypes.vcxproj | 21 +----- PCbuild/_ctypes_test.vcxproj | 7 -- PCbuild/_decimal.vcxproj | 21 ++----- PCbuild/_elementtree.vcxproj | 6 -- PCbuild/_hashlib.vcxproj | 5 - PCbuild/_lzma.vcxproj | 6 -- PCbuild/_msi.vcxproj | 6 -- PCbuild/_multiprocessing.vcxproj | 6 -- PCbuild/_socket.vcxproj | 6 -- PCbuild/_sqlite3.vcxproj | 6 -- PCbuild/_ssl.vcxproj | 5 - PCbuild/_testbuffer.vcxproj | 22 +------ PCbuild/_testcapi.vcxproj | 6 -- PCbuild/_tkinter.vcxproj | 5 - PCbuild/debug.props | 7 +- PCbuild/env.bat | 8 ++- PCbuild/kill_python.vcxproj | 6 -- PCbuild/make_buildinfo.vcxproj | 2 - PCbuild/make_versioninfo.vcxproj | 10 ++- PCbuild/pcbuild.sln | 4 +- PCbuild/pyd.props | 6 +- PCbuild/pyd_d.props | 18 +----- PCbuild/pyexpat.vcxproj | 6 -- PCbuild/pyproject.props | 11 ++- PCbuild/python.vcxproj | 8 -- PCbuild/pythoncore.vcxproj | 46 +++++----------- PCbuild/pythonw.vcxproj | 8 -- PCbuild/readme.txt | 18 ++---- PCbuild/release.props | 6 +- PCbuild/select.vcxproj | 6 -- PCbuild/sqlite3.vcxproj | 3 - PCbuild/unicodedata.vcxproj | 6 -- PCbuild/vs9to10.py | 56 ++++++++++++++++++++ PCbuild/w9xpopen.vcxproj | 24 ++++++++ PCbuild/winsound.vcxproj | 6 -- PCbuild/x64.props | 2 +- PCbuild/xxlimited.vcxproj | 18 ++--- 38 files changed, 159 insertions(+), 260 deletions(-) diff --git a/PCbuild/_bz2.vcxproj b/PCbuild/_bz2.vcxproj --- a/PCbuild/_bz2.vcxproj +++ b/PCbuild/_bz2.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_ctypes.vcxproj b/PCbuild/_ctypes.vcxproj --- a/PCbuild/_ctypes.vcxproj +++ b/PCbuild/_ctypes.vcxproj @@ -148,18 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - $(OutDirPGI)\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - .pyd @@ -178,7 +166,6 @@ 0x1D1A0000 - $(OutDir)python33_d.lib;%(AdditionalDependencies) @@ -202,7 +189,6 @@ /EXPORT:DllGetClassObject,PRIVATE /EXPORT:DllCanUnloadNow,PRIVATE %(AdditionalOptions) NotSet 0x1D1A0000 - $(OutDir)python33.lib;%(AdditionalDependencies) @@ -227,7 +213,6 @@ NotSet 0x1D1A0000 MachineX64 - $(OutDir)python33.lib;%(AdditionalDependencies) @@ -252,7 +237,6 @@ NotSet 0x1D1A0000 MachineX64 - $(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) @@ -299,6 +283,11 @@ $(IntDir)win64.obj;%(Outputs) + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + + diff --git a/PCbuild/_ctypes_test.vcxproj b/PCbuild/_ctypes_test.vcxproj --- a/PCbuild/_ctypes_test.vcxproj +++ b/PCbuild/_ctypes_test.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd @@ -190,7 +184,6 @@ {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} - false diff --git a/PCbuild/_decimal.vcxproj b/PCbuild/_decimal.vcxproj --- a/PCbuild/_decimal.vcxproj +++ b/PCbuild/_decimal.vcxproj @@ -113,6 +113,8 @@ + + @@ -146,20 +148,6 @@ AllRules.ruleset - .pyd - .pyd - $(SolutionDir)\amd64\ - $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ - .pyd - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - $(OutDirPGI)\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - .pyd @@ -313,6 +301,11 @@ $(IntDir)vcdiv64.obj;%(Outputs) + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + + diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj --- a/PCbuild/_elementtree.vcxproj +++ b/PCbuild/_elementtree.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_hashlib.vcxproj b/PCbuild/_hashlib.vcxproj --- a/PCbuild/_hashlib.vcxproj +++ b/PCbuild/_hashlib.vcxproj @@ -148,11 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_lzma.vcxproj b/PCbuild/_lzma.vcxproj --- a/PCbuild/_lzma.vcxproj +++ b/PCbuild/_lzma.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_msi.vcxproj b/PCbuild/_msi.vcxproj --- a/PCbuild/_msi.vcxproj +++ b/PCbuild/_msi.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_multiprocessing.vcxproj b/PCbuild/_multiprocessing.vcxproj --- a/PCbuild/_multiprocessing.vcxproj +++ b/PCbuild/_multiprocessing.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_socket.vcxproj b/PCbuild/_socket.vcxproj --- a/PCbuild/_socket.vcxproj +++ b/PCbuild/_socket.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj --- a/PCbuild/_sqlite3.vcxproj +++ b/PCbuild/_sqlite3.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_ssl.vcxproj b/PCbuild/_ssl.vcxproj --- a/PCbuild/_ssl.vcxproj +++ b/PCbuild/_ssl.vcxproj @@ -148,11 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_testbuffer.vcxproj b/PCbuild/_testbuffer.vcxproj --- a/PCbuild/_testbuffer.vcxproj +++ b/PCbuild/_testbuffer.vcxproj @@ -148,19 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - $(OutDirPGI)\ - $(ProjectName) - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - .pyd @@ -173,7 +160,6 @@ 0x1e1F0000 - $(OutDir)python33_d.lib;%(AdditionalDependencies) @@ -187,7 +173,6 @@ 0x1e1F0000 - $(OutDir)python33.lib;%(AdditionalDependencies) @@ -202,7 +187,6 @@ 0x1e1F0000 MachineX64 - $(OutDir)python33.lib;%(AdditionalDependencies) @@ -217,12 +201,16 @@ 0x1e1F0000 MachineX64 - $(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + + diff --git a/PCbuild/_testcapi.vcxproj b/PCbuild/_testcapi.vcxproj --- a/PCbuild/_testcapi.vcxproj +++ b/PCbuild/_testcapi.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/_tkinter.vcxproj b/PCbuild/_tkinter.vcxproj --- a/PCbuild/_tkinter.vcxproj +++ b/PCbuild/_tkinter.vcxproj @@ -148,11 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/debug.props b/PCbuild/debug.props --- a/PCbuild/debug.props +++ b/PCbuild/debug.props @@ -1,6 +1,7 @@ ? + _d $(OutDir)kill_python_d.exe @@ -11,11 +12,11 @@ _DEBUG;%(PreprocessorDefinitions) - - $(OutDir)$(TargetName)$(TargetExt) - + + $(PyDebugExt) + $(KillPythonExe) diff --git a/PCbuild/env.bat b/PCbuild/env.bat --- a/PCbuild/env.bat +++ b/PCbuild/env.bat @@ -1,5 +1,9 @@ @echo off -set VS9=%ProgramFiles%\Microsoft Visual Studio 9.0 +set VS10=%ProgramFiles(x86)%\Microsoft Visual Studio 10.0 +IF EXIST "%VS10%" GOTO ok +set VS10=%ProgramFiles%\Microsoft Visual Studio 10.0 +:ok + echo Build environments: x86, ia64, amd64, x86_amd64, x86_ia64 echo. -call "%VS9%\VC\vcvarsall.bat" %1 +call "%VS10%\VC\vcvarsall.bat" %1 diff --git a/PCbuild/kill_python.vcxproj b/PCbuild/kill_python.vcxproj --- a/PCbuild/kill_python.vcxproj +++ b/PCbuild/kill_python.vcxproj @@ -82,12 +82,6 @@ AllRules.ruleset - .exe - .exe - .exe - .exe - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ diff --git a/PCbuild/make_buildinfo.vcxproj b/PCbuild/make_buildinfo.vcxproj --- a/PCbuild/make_buildinfo.vcxproj +++ b/PCbuild/make_buildinfo.vcxproj @@ -30,7 +30,6 @@ AllRules.ruleset - .exe @@ -41,7 +40,6 @@ $(OutDir)make_buildinfo.exe - $(TargetDir)$(TargetName).pdb Console diff --git a/PCbuild/make_versioninfo.vcxproj b/PCbuild/make_versioninfo.vcxproj --- a/PCbuild/make_versioninfo.vcxproj +++ b/PCbuild/make_versioninfo.vcxproj @@ -54,9 +54,15 @@ + + + + + + @@ -73,8 +79,6 @@ AllRules.ruleset - .exe - .exe @@ -97,7 +101,6 @@ $(SolutionDir)make_versioninfo.exe - $(TargetDir)$(TargetName).pdb Console 0x1d000000 @@ -151,7 +154,6 @@ $(SolutionDir)make_versioninfo_d.exe - $(TargetDir)$(TargetName).pdb Console 0x1d000000 diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -547,8 +547,8 @@ {885D4898-D08D-4091-9C40-C700CFE3FC5A}.Release|Win32.Build.0 = Release|Win32 {885D4898-D08D-4091-9C40-C700CFE3FC5A}.Release|x64.ActiveCfg = Release|x64 {885D4898-D08D-4091-9C40-C700CFE3FC5A}.Release|x64.Build.0 = Release|x64 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|Win32.ActiveCfg = Debug|Win32 - {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|x64.ActiveCfg = Debug|x64 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|Win32.ActiveCfg = Release|Win32 + {F749B822-B489-4CA5-A3AD-CE078F5F338A}.Debug|x64.ActiveCfg = Release|x64 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|Win32.Build.0 = PGInstrument|Win32 {F749B822-B489-4CA5-A3AD-CE078F5F338A}.PGInstrument|x64.ActiveCfg = PGInstrument|x64 diff --git a/PCbuild/pyd.props b/PCbuild/pyd.props --- a/PCbuild/pyd.props +++ b/PCbuild/pyd.props @@ -1,13 +1,14 @@ ? + - <_ProjectFileVersion>10.0.30319.1 false false + .pyd @@ -15,9 +16,6 @@ MultiThreadedDLL - $(OutDir)$(ProjectName).pyd - $(OutDir)$(ProjectName).pdb - $(OutDir)$(TargetName).lib diff --git a/PCbuild/pyd_d.props b/PCbuild/pyd_d.props --- a/PCbuild/pyd_d.props +++ b/PCbuild/pyd_d.props @@ -1,12 +1,10 @@ ? + - - - $(SolutionDir)python_d.exe - + <_ProjectFileVersion>10.0.30319.1 false @@ -23,19 +21,11 @@ Py_BUILD_CORE_MODULE;%(PreprocessorDefinitions) MultiThreadedDebugDLL - - $(OutDir)$(ProjectName)_d.pyd - $(OutDir)$(ProjectName)_d.pdb - $(OutDir)$(TargetName).lib - + - - - $(PythonExe) - - + \ No newline at end of file diff --git a/PCbuild/pyexpat.vcxproj b/PCbuild/pyexpat.vcxproj --- a/PCbuild/pyexpat.vcxproj +++ b/PCbuild/pyexpat.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -1,8 +1,9 @@ ? - python33 - $(SolutionDir)\python.exe + python33$(PyDebugExt) + $(SolutionDir)python$(PyDebugExt).exe + $(OutDir)kill_python$(PyDebugExt).exe ..\.. $(externalsDir)\sqlite-3.7.12 $(externalsDir)\bzip2-1.0.6 @@ -17,10 +18,9 @@ <_ProjectFileVersion>10.0.30319.1 - $(SolutionDir)\ + $(SolutionDir) $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ false - .dll @@ -59,6 +59,9 @@ $(PythonExe) + + $(KillPythonExe) + $(externalsDir) diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj --- a/PCbuild/python.vcxproj +++ b/PCbuild/python.vcxproj @@ -156,14 +156,6 @@ AllRules.ruleset - .exe - .exe - .exe - .exe - .exe - .exe - .exe - .exe diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -77,51 +77,51 @@ + - + - + - + - + - + - + - + - @@ -150,15 +150,14 @@ AllRules.ruleset - $(PyDllName)_d + $(PyDllName) + $(PyDllName) + $(PyDllName) $(PyDllName) - $(PyDllName)_d + $(PyDllName) + $(PyDllName) + $(PyDllName) $(PyDllName) - .dll - $(PyDllName) - $(PyDllName) - $(PyDllName) - $(PyDllName) @@ -180,9 +179,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName).pdb 0x1e000000 - $(OutDir)$(PyDllName).lib @@ -208,9 +205,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName).pdb 0x1e000000 - $(OutDir)$(PyDllName).lib @@ -234,11 +229,8 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) - $(OutDir)$(PyDllName)_d.dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName)_d.pdb 0x1e000000 - $(OutDir)$(PyDllName)_d.lib @@ -267,9 +259,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName)_d.dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName)_d.pdb 0x1e000000 - $(OutDir)$(PyDllName)_d.lib @@ -292,9 +282,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName).pdb 0x1e000000 - $(OutDirPGI)$(PyDllName).lib @@ -320,9 +308,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName).pdb 0x1e000000 - $(OutDirPGI)$(PyDllName).lib MachineX64 @@ -346,9 +332,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName).pdb 0x1e000000 - $(OutDirPGI)$(PyDllName).lib @@ -374,9 +358,7 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) - $(OutDir)$(PyDllName).pdb 0x1e000000 - $(OutDirPGI)$(PyDllName).lib MachineX64 diff --git a/PCbuild/pythonw.vcxproj b/PCbuild/pythonw.vcxproj --- a/PCbuild/pythonw.vcxproj +++ b/PCbuild/pythonw.vcxproj @@ -149,14 +149,6 @@ AllRules.ruleset - .exe - .exe - .exe - .pyd - .exe - .exe - .exe - .exe diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -1,4 +1,4 @@ -Building Python using VC++ 9.0 +Building Python using VC++ 10.0 ------------------------------ This directory is used to build Python for Win32 and x64 platforms, e.g. @@ -62,17 +62,11 @@ C RUNTIME --------- -Visual Studio 2008 uses version 9 of the C runtime (MSVCRT9). The executables -are linked to a CRT "side by side" assembly which must be present on the target -machine. This is avalible under the VC/Redist folder of your visual studio -distribution. On XP and later operating systems that support -side-by-side assemblies it is not enough to have the msvcrt90.dll present, -it has to be there as a whole assembly, that is, a folder with the .dll -and a .manifest. Also, a check is made for the correct version. -Therefore, one should distribute this assembly with the dlls, and keep -it in the same directory. For compatibility with older systems, one should -also set the PATH to this directory so that the dll can be found. -For more info, see the Readme in the VC/Redist folder. +Visual Studio 2010 uses version 10 of the C runtime (MSVCRT9). The executables +no longer use the "Side by Side" assemblies used in previous versions of the +compiler. This simplifies distribution of applications. +The run time libraries are avalible under the VC/Redist folder of your visual studio +distribution. For more info, see the Readme in the VC/Redist folder. SUBPROJECTS ----------- diff --git a/PCbuild/release.props b/PCbuild/release.props --- a/PCbuild/release.props +++ b/PCbuild/release.props @@ -1,7 +1,7 @@ ? - $(OutDir)kill_python.exe + <_ProjectFileVersion>10.0.30319.1 @@ -12,8 +12,8 @@ - - $(KillPythonExe) + + $(PyDebugExt) \ No newline at end of file diff --git a/PCbuild/select.vcxproj b/PCbuild/select.vcxproj --- a/PCbuild/select.vcxproj +++ b/PCbuild/select.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/sqlite3.vcxproj b/PCbuild/sqlite3.vcxproj --- a/PCbuild/sqlite3.vcxproj +++ b/PCbuild/sqlite3.vcxproj @@ -150,9 +150,6 @@ AllRules.ruleset - .dll - .dll - .dll diff --git a/PCbuild/unicodedata.vcxproj b/PCbuild/unicodedata.vcxproj --- a/PCbuild/unicodedata.vcxproj +++ b/PCbuild/unicodedata.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/vs9to10.py b/PCbuild/vs9to10.py new file mode 100644 --- /dev/null +++ b/PCbuild/vs9to10.py @@ -0,0 +1,56 @@ +#Run this file after automatic convertsion of the VisualStudio 2008 solution by VisualStudio 2010. +#This can be done whenever the 2008 solution changes. +#It will make the necessary cleanup and updates to the vcxproj files +#the .props files need to be maintained by hand if the .vsprops files change + +from __future__ import with_statement +import sys +import os +import os.path + +def vs9to10(src, dest): + for name in os.listdir(src): + path, ext = os.path.splitext(name) + if ext.lower() not in ('.vcxproj',): + continue + + filename = os.path.normpath(os.path.join(src, name)) + destname = os.path.normpath(os.path.join(dest, name)) + print("%s -> %s" % (filename, destname)) + + lines = [] + lastline = b"" + importgroup = False + with open(filename, 'rb') as fin: + for line in fin: + #remove redundant linker output info + if b"" in line: + continue + if b"" in line: + continue + if b"" in line and b"" in line: + continue + + #add new property sheet to the pythoncore + if importgroup and "pythoncore" in name.lower(): + if b"" in line: + if b"debug.props" in lastline: + lines.append(b' \r\n') + elif b"pythoncore" not in lastline: + lines.append(b' \r\n') + if b"" in line: + importgroup = False + lines.append(line) + lastline = line + with open(destname, 'wb') as fout: + for line in lines: + fout.write(line) + +if __name__ == "__main__": + src = "." if len(sys.argv) < 2 else sys.argv[1] + name = os.path.basename(os.path.abspath(src)) + dest = os.path.abspath(os.path.join(src, "..", name + "Upd")) + os.makedirs(dest) + vs9to10(src, dest) diff --git a/PCbuild/w9xpopen.vcxproj b/PCbuild/w9xpopen.vcxproj --- a/PCbuild/w9xpopen.vcxproj +++ b/PCbuild/w9xpopen.vcxproj @@ -84,27 +84,51 @@ + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/PCbuild/winsound.vcxproj b/PCbuild/winsound.vcxproj --- a/PCbuild/winsound.vcxproj +++ b/PCbuild/winsound.vcxproj @@ -148,12 +148,6 @@ AllRules.ruleset - .pyd - .pyd - .pyd - .pyd - .pyd - .pyd diff --git a/PCbuild/x64.props b/PCbuild/x64.props --- a/PCbuild/x64.props +++ b/PCbuild/x64.props @@ -6,7 +6,7 @@ <_ProjectFileVersion>10.0.30319.1 <_PropertySheetDisplayName>amd64 - $(SolutionDir)\amd64\ + $(SolutionDir)amd64\ $(SolutionDir)$(PlatformName)-temp-$(Configuration)\$(ProjectName)\ diff --git a/PCbuild/xxlimited.vcxproj b/PCbuild/xxlimited.vcxproj --- a/PCbuild/xxlimited.vcxproj +++ b/PCbuild/xxlimited.vcxproj @@ -117,12 +117,6 @@ AllRules.ruleset - .pyd - .pyd - $(SolutionDir)$(PlatformName)-pgo\ - $(SolutionDir)$(PlatformName)-temp-pgi\$(ProjectName)\ - .pyd - .pyd @@ -132,7 +126,6 @@ wsock32.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 - $(OutDir)$(ProjectName).pyd @@ -140,7 +133,7 @@ X64 - wsock32.lib;$(SolutionDir)\$(PlatformShortName)\python33.lib;%(AdditionalDependencies) + wsock32.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 @@ -153,7 +146,6 @@ wsock32.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 - $(OutDirPGI)$(ProjectName).pyd @@ -161,11 +153,10 @@ X64 - wsock32.lib;$(OutDir)python33.lib;%(AdditionalDependencies) + wsock32.lib;%(AdditionalDependencies) libc;%(IgnoreSpecificDefaultLibraries) 0x1D110000 MachineX64 - $(OutDirPGI)$(ProjectName).pyd @@ -192,6 +183,11 @@ + + + {cf7ac3d1-e2df-41d2-bea6-1e2556cdea26} + + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 19 23:12:22 2012 From: python-checkins at python.org (ezio.melotti) Date: Sat, 19 May 2012 23:12:22 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_=2314860=3A_mention_that_t?= =?utf8?q?est=2Eregrtest_is_required_for_2=2E7=2E__Patch_by_Marc?= Message-ID: http://hg.python.org/devguide/rev/13e87e1715f8 changeset: 513:13e87e1715f8 user: Ezio Melotti date: Sun May 20 00:12:15 2012 +0300 summary: #14860: mention that test.regrtest is required for 2.7. Patch by Marc Abramowitz. files: index.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -48,7 +48,8 @@ and :menuselection:`Build --> Build Solution`. 3. :doc:`Run the tests ` with ``./python -m test -j3`` (use :file:`./python.exe` on :ref:`most ` Mac OS X systems and - :file:`PCbuild\\python_d.exe` on Windows). + :file:`PCbuild\\python_d.exe` on Windows; replace ``test`` with + ``test.regrtest`` for 2.7). 4. Make the :doc:`patch `. 5. Submit it to the `issue tracker`_. -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sat May 19 23:25:48 2012 From: python-checkins at python.org (kristjan.jonsson) Date: Sat, 19 May 2012 23:25:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_missing_files_from_t?= =?utf8?q?he_PCbuild_pythoncore_project=2E__This_avoids_a?= Message-ID: http://hg.python.org/cpython/rev/17efda0a2b2b changeset: 77061:17efda0a2b2b user: Kristj?n Valur J?nsson date: Sat May 19 21:25:41 2012 +0000 summary: Remove missing files from the PCbuild pythoncore project. This avoids a "missing dependency", causing pythocore to be rebuilt every time. files: PCbuild/pythoncore.vcxproj | 4 ---- PCbuild/pythoncore.vcxproj.filters | 12 ------------ 2 files changed, 0 insertions(+), 16 deletions(-) diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -389,7 +389,6 @@ - @@ -425,7 +424,6 @@ - @@ -446,7 +444,6 @@ - @@ -454,7 +451,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -111,9 +111,6 @@ Include - - Include - Include @@ -219,9 +216,6 @@ Include - - Include - Include @@ -282,9 +276,6 @@ Include - - Include - Include @@ -306,9 +297,6 @@ Modules - - Modules - Modules -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 05:33:37 2012 From: python-checkins at python.org (eli.bendersky) Date: Sun, 20 May 2012 05:33:37 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314849=3A_setup_Ele?= =?utf8?q?ment_data_members_to_be_assignable_in_subclasses?= Message-ID: http://hg.python.org/cpython/rev/41a9d24d075e changeset: 77062:41a9d24d075e user: Eli Bendersky date: Sun May 20 06:33:29 2012 +0300 summary: Issue #14849: setup Element data members to be assignable in subclasses files: Lib/test/test_xml_etree.py | 4 ++++ Modules/_elementtree.c | 25 ++++++++++++------------- 2 files changed, 16 insertions(+), 13 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1914,6 +1914,10 @@ self.assertIsInstance(mye, MyElement) self.assertEqual(mye.tag, 'foo') + # test that attribute assignment works (issue 14849) + mye.text = "joe" + self.assertEqual(mye.text, "joe") + def test_Element_subclass_constructor(self): class MyElement(ET.Element): def __init__(self, tag, attrib={}, **extra): diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -1640,16 +1640,15 @@ return res; } -static int -element_setattr(ElementObject* self, const char* name, PyObject* value) +static PyObject* +element_setattro(ElementObject* self, PyObject* nameobj, PyObject* value) { - if (value == NULL) { - PyErr_SetString( - PyExc_AttributeError, - "can't delete element attributes" - ); - return -1; - } + char *name = ""; + if (PyUnicode_Check(nameobj)) + name = _PyUnicode_AsString(nameobj); + + if (name == NULL) + return NULL; if (strcmp(name, "tag") == 0) { Py_DECREF(self->tag); @@ -1671,10 +1670,10 @@ Py_INCREF(self->extra->attrib); } else { PyErr_SetString(PyExc_AttributeError, name); - return -1; + return NULL; } - return 0; + return NULL; } static PySequenceMethods element_as_sequence = { @@ -1700,7 +1699,7 @@ (destructor)element_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ - (setattrfunc)element_setattr, /* tp_setattr */ + 0, /* tp_setattr */ 0, /* tp_reserved */ (reprfunc)element_repr, /* tp_repr */ 0, /* tp_as_number */ @@ -1710,7 +1709,7 @@ 0, /* tp_call */ 0, /* tp_str */ (getattrofunc)element_getattro, /* tp_getattro */ - 0, /* tp_setattro */ + (setattrofunc)element_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun May 20 05:50:45 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 20 May 2012 05:50:45 +0200 Subject: [Python-checkins] Daily reference leaks (17efda0a2b2b): sum=0 Message-ID: results for 17efda0a2b2b on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogwFKsEW', '-x'] From python-checkins at python.org Sun May 20 06:08:45 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 06:08:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A__Issue_=2314426?= =?utf8?q?=3A_Correct_the_Date_format_in_Expires_attribute_of_Set-Cookie?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/1db41c410f63 changeset: 77063:1db41c410f63 branch: 2.7 parent: 77058:4641d8d99a7d user: Senthil Kumaran date: Sun May 20 12:02:44 2012 +0800 summary: Issue #14426: Correct the Date format in Expires attribute of Set-Cookie. Patch by Federico Reghenzani and M?te Invert files: Lib/Cookie.py | 2 +- Lib/test/test_cookie.py | 8 ++++---- Misc/NEWS | 3 +++ 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/Lib/Cookie.py b/Lib/Cookie.py --- a/Lib/Cookie.py +++ b/Lib/Cookie.py @@ -390,7 +390,7 @@ from time import gmtime, time now = time() year, month, day, hh, mm, ss, wd, y, z = gmtime(now + future) - return "%s, %02d-%3s-%4d %02d:%02d:%02d GMT" % \ + return "%s, %02d %3s %4d %02d:%02d:%02d GMT" % \ (weekdayname[wd], day, monthname[month], year, hh, mm, ss) diff --git a/Lib/test/test_cookie.py b/Lib/test/test_cookie.py --- a/Lib/test/test_cookie.py +++ b/Lib/test/test_cookie.py @@ -64,13 +64,13 @@ # loading 'expires' C = Cookie.SimpleCookie() - C.load('Customer="W"; expires=Wed, 01-Jan-2010 00:00:00 GMT') + C.load('Customer="W"; expires=Wed, 01 Jan 2010 00:00:00 GMT') self.assertEqual(C['Customer']['expires'], - 'Wed, 01-Jan-2010 00:00:00 GMT') + 'Wed, 01 Jan 2010 00:00:00 GMT') C = Cookie.SimpleCookie() - C.load('Customer="W"; expires=Wed, 01-Jan-98 00:00:00 GMT') + C.load('Customer="W"; expires=Wed, 01 Jan 98 00:00:00 GMT') self.assertEqual(C['Customer']['expires'], - 'Wed, 01-Jan-98 00:00:00 GMT') + 'Wed, 01 Jan 98 00:00:00 GMT') def test_extended_encode(self): # Issue 9824: some browsers don't follow the standard; we now diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue #14426: Correct the Date format in Expires attribute of Set-Cookie + Header in Cookie.py. + - Issue #14721: Send proper header, Content-length: 0 when the body is an empty string ''. Initial Patch contributed by Arve Knudsen. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 06:08:46 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 06:08:46 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A__Issue_=2314426?= =?utf8?q?=3A_Correct_the_Date_format_in_Expires_attribute_of_Set-Cookie?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/a5fa068143eb changeset: 77064:a5fa068143eb branch: 3.2 parent: 77055:7cdc1392173f user: Senthil Kumaran date: Sun May 20 12:05:16 2012 +0800 summary: Issue #14426: Correct the Date format in Expires attribute of Set-Cookie. Patch by Federico Reghenzani and M?te Invert files: Lib/http/cookies.py | 2 +- Lib/test/test_http_cookies.py | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Lib/http/cookies.py b/Lib/http/cookies.py --- a/Lib/http/cookies.py +++ b/Lib/http/cookies.py @@ -301,7 +301,7 @@ from time import gmtime, time now = time() year, month, day, hh, mm, ss, wd, y, z = gmtime(now + future) - return "%s, %02d-%3s-%4d %02d:%02d:%02d GMT" % \ + return "%s, %02d %3s %4d %02d:%02d:%02d GMT" % \ (weekdayname[wd], day, monthname[month], year, hh, mm, ss) diff --git a/Lib/test/test_http_cookies.py b/Lib/test/test_http_cookies.py --- a/Lib/test/test_http_cookies.py +++ b/Lib/test/test_http_cookies.py @@ -86,13 +86,13 @@ # loading 'expires' C = cookies.SimpleCookie() - C.load('Customer="W"; expires=Wed, 01-Jan-2010 00:00:00 GMT') + C.load('Customer="W"; expires=Wed, 01 Jan 2010 00:00:00 GMT') self.assertEqual(C['Customer']['expires'], - 'Wed, 01-Jan-2010 00:00:00 GMT') + 'Wed, 01 Jan 2010 00:00:00 GMT') C = cookies.SimpleCookie() - C.load('Customer="W"; expires=Wed, 01-Jan-98 00:00:00 GMT') + C.load('Customer="W"; expires=Wed, 01 Jan 98 00:00:00 GMT') self.assertEqual(C['Customer']['expires'], - 'Wed, 01-Jan-98 00:00:00 GMT') + 'Wed, 01 Jan 98 00:00:00 GMT') # 'max-age' C = cookies.SimpleCookie('Customer="WILE_E_COYOTE"') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 06:08:47 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 06:08:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_news_for_Issue1?= =?utf8?q?4426?= Message-ID: http://hg.python.org/cpython/rev/32cf38bfb36f changeset: 77065:32cf38bfb36f branch: 3.2 user: Senthil Kumaran date: Sun May 20 12:06:24 2012 +0800 summary: news for Issue14426 files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14426: Correct the Date format in Expires attribute of Set-Cookie + Header in Cookie.py. + - Issue #14721: Send the correct 'Content-length: 0' header when the body is an empty string ''. Initial Patch contributed by Arve Knudsen. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 06:08:49 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 06:08:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A__Issue_=2314426=3A_Correct_the_Date_format_in_Expires_attrib?= =?utf8?q?ute_of_Set-Cookie=2E?= Message-ID: http://hg.python.org/cpython/rev/8e476cca25ff changeset: 77066:8e476cca25ff parent: 77062:41a9d24d075e parent: 77064:a5fa068143eb user: Senthil Kumaran date: Sun May 20 12:06:51 2012 +0800 summary: Issue #14426: Correct the Date format in Expires attribute of Set-Cookie. Patch by Federico Reghenzani and M?te Invert files: Lib/http/cookies.py | 2 +- Lib/test/test_http_cookies.py | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Lib/http/cookies.py b/Lib/http/cookies.py --- a/Lib/http/cookies.py +++ b/Lib/http/cookies.py @@ -301,7 +301,7 @@ from time import gmtime, time now = time() year, month, day, hh, mm, ss, wd, y, z = gmtime(now + future) - return "%s, %02d-%3s-%4d %02d:%02d:%02d GMT" % \ + return "%s, %02d %3s %4d %02d:%02d:%02d GMT" % \ (weekdayname[wd], day, monthname[month], year, hh, mm, ss) diff --git a/Lib/test/test_http_cookies.py b/Lib/test/test_http_cookies.py --- a/Lib/test/test_http_cookies.py +++ b/Lib/test/test_http_cookies.py @@ -95,13 +95,13 @@ # loading 'expires' C = cookies.SimpleCookie() - C.load('Customer="W"; expires=Wed, 01-Jan-2010 00:00:00 GMT') + C.load('Customer="W"; expires=Wed, 01 Jan 2010 00:00:00 GMT') self.assertEqual(C['Customer']['expires'], - 'Wed, 01-Jan-2010 00:00:00 GMT') + 'Wed, 01 Jan 2010 00:00:00 GMT') C = cookies.SimpleCookie() - C.load('Customer="W"; expires=Wed, 01-Jan-98 00:00:00 GMT') + C.load('Customer="W"; expires=Wed, 01 Jan 98 00:00:00 GMT') self.assertEqual(C['Customer']['expires'], - 'Wed, 01-Jan-98 00:00:00 GMT') + 'Wed, 01 Jan 98 00:00:00 GMT') # 'max-age' C = cookies.SimpleCookie('Customer="WILE_E_COYOTE"') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 06:08:50 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 06:08:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_news_for_Issue14426?= Message-ID: http://hg.python.org/cpython/rev/90ffd6c98057 changeset: 77067:90ffd6c98057 parent: 77066:8e476cca25ff parent: 77065:32cf38bfb36f user: Senthil Kumaran date: Sun May 20 12:08:29 2012 +0800 summary: news for Issue14426 files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,9 @@ Library ------- +- Issue #14426: Correct the Date format in Expires attribute of Set-Cookie + Header in Cookie.py. + - Issue #14588: The types module now provide new_class() and prepare_class() functions to support PEP 3115 compliant dynamic class creation. Patch by Daniel Urban and Nick Coghlan. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 06:21:03 2012 From: python-checkins at python.org (raymond.hettinger) Date: Sun, 20 May 2012 06:21:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_change_to_default_lru?= =?utf8?q?_size=2E__Set_default_to_a_power_of_two=2E?= Message-ID: http://hg.python.org/cpython/rev/1cd147d9c645 changeset: 77068:1cd147d9c645 user: Raymond Hettinger date: Sat May 19 21:20:48 2012 -0700 summary: Minor change to default lru size. Set default to a power of two. files: Doc/library/functools.rst | 2 +- Lib/functools.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -40,7 +40,7 @@ .. versionadded:: 3.2 -.. decorator:: lru_cache(maxsize=100, typed=False) +.. decorator:: lru_cache(maxsize=128, typed=False) Decorator to wrap a function with a memoizing callable that saves up to the *maxsize* most recent calls. It can save time when an expensive or I/O bound diff --git a/Lib/functools.py b/Lib/functools.py --- a/Lib/functools.py +++ b/Lib/functools.py @@ -166,7 +166,7 @@ def __hash__(self): return self.hashvalue -def lru_cache(maxsize=100, typed=False): +def lru_cache(maxsize=128, typed=False): """Least-recently-used cache decorator. If *maxsize* is set to None, the LRU features are disabled and the cache -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 08:36:12 2012 From: python-checkins at python.org (ned.deily) Date: Sun, 20 May 2012 08:36:12 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0NTcy?= =?utf8?q?=3A_Prevent_build_failures_with_pre-3=2E5=2E0_versions_of?= Message-ID: http://hg.python.org/cpython/rev/e31597b3bd15 changeset: 77069:e31597b3bd15 branch: 2.7 parent: 77063:1db41c410f63 user: Ned Deily date: Sat May 19 23:35:05 2012 -0700 summary: Issue #14572: Prevent build failures with pre-3.5.0 versions of sqlite3, such as was shipped with Centos 5 and Mac OS X 10.4. files: Misc/NEWS | 3 +++ Modules/_sqlite/connection.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue #14572: Prevent build failures with pre-3.5.0 versions of + sqlite3, such as was shipped with Centos 5 and Mac OS X 10.4. + - Issue #14426: Correct the Date format in Expires attribute of Set-Cookie Header in Cookie.py. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -549,7 +549,7 @@ } else if (py_val == Py_None) { sqlite3_result_null(context); } else if (PyInt_Check(py_val)) { - sqlite3_result_int64(context, (sqlite3_int64)PyInt_AsLong(py_val)); + sqlite3_result_int64(context, (sqlite_int64)PyInt_AsLong(py_val)); } else if (PyLong_Check(py_val)) { sqlite3_result_int64(context, PyLong_AsLongLong(py_val)); } else if (PyFloat_Check(py_val)) { @@ -580,7 +580,7 @@ sqlite3_value* cur_value; PyObject* cur_py_value; const char* val_str; - sqlite3_int64 val_int; + sqlite_int64 val_int; Py_ssize_t buflen; void* raw_buffer; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 10:15:25 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 20 May 2012 10:15:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo_in_new_metaclass_d?= =?utf8?q?ocs?= Message-ID: http://hg.python.org/cpython/rev/9385dae3cc6d changeset: 77070:9385dae3cc6d parent: 77068:1cd147d9c645 user: Nick Coghlan date: Sun May 20 18:15:11 2012 +1000 summary: Fix typo in new metaclass docs files: Doc/reference/datamodel.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1634,7 +1634,7 @@ Once the class namespace has been populated by executing the class body, the class object is created by calling ``metaclass(name, bases, namespace, **kwds)`` (the additional keywords -passed here are the same as those passed to ``__prepate__``). +passed here are the same as those passed to ``__prepare__``). This class object is the one that will be referenced by the zero-argument form of :func:`super`. ``__class__`` is an implicit closure reference -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 10:31:01 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 20 May 2012 10:31:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Describe_the_default_hash_c?= =?utf8?q?orrectly=2C_and_mark_a_couple_of_CPython?= Message-ID: http://hg.python.org/cpython/rev/71989b8349ca changeset: 77071:71989b8349ca user: Nick Coghlan date: Sun May 20 18:30:49 2012 +1000 summary: Describe the default hash correctly, and mark a couple of CPython implementation details files: Doc/reference/datamodel.rst | 20 ++++++++++++++++---- 1 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -35,12 +35,19 @@ Every object has an identity, a type and a value. An object's *identity* never changes once it has been created; you may think of it as the object's address in memory. The ':keyword:`is`' operator compares the identity of two objects; the -:func:`id` function returns an integer representing its identity (currently -implemented as its address). An object's :dfn:`type` is also unchangeable. [#]_ +:func:`id` function returns an integer representing its identity. + +.. impl-detail:: + + For CPython, ``id(x)`` is the memory address where ``x`` is stored. + An object's type determines the operations that the object supports (e.g., "does it have a length?") and also defines the possible values for objects of that type. The :func:`type` function returns an object's type (which is an object -itself). The *value* of some objects can change. Objects whose value can +itself). Like its identity, an object's :dfn:`type` is also unchangeable. +[#]_ + +The *value* of some objects can change. Objects whose value can change are said to be *mutable*; objects whose value is unchangeable once they are created are called *immutable*. (The value of an immutable container object that contains a reference to a mutable object can change when the latter's value @@ -1258,7 +1265,12 @@ User-defined classes have :meth:`__eq__` and :meth:`__hash__` methods by default; with them, all objects compare unequal (except with themselves) - and ``x.__hash__()`` returns ``id(x)``. + and ``x.__hash__()`` returns an appropriate value such that ``x == y`` + implies both that ``x is y`` and ``hash(x) == hash(y)``. + + .. impl-detail:: + + CPython uses ``hash(id(x))`` as the default hash for class instances. Classes which inherit a :meth:`__hash__` method from a parent class but change the meaning of :meth:`__eq__` such that the hash value returned is no -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 10:42:23 2012 From: python-checkins at python.org (martin.v.loewis) Date: Sun, 20 May 2012 10:42:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzg3Njc6?= =?utf8?q?_Restore_building_with_--disable-unicode=2E?= Message-ID: http://hg.python.org/cpython/rev/d7aff4423172 changeset: 77072:d7aff4423172 branch: 2.7 parent: 77069:e31597b3bd15 user: Martin v. L?wis date: Sun May 20 10:42:17 2012 +0200 summary: Issue #8767: Restore building with --disable-unicode. Original patch by Stefano Taschini. files: Lib/glob.py | 10 +++++++++- Lib/locale.py | 10 +++++++++- Lib/posixpath.py | 12 ++++++++++-- Lib/test/script_helper.py | 8 +++++++- Lib/test/test_support.py | 2 +- Lib/textwrap.py | 12 ++++++++++-- Lib/unittest/case.py | 6 +++++- Misc/ACKS | 1 + Misc/NEWS | 3 +++ Objects/object.c | 2 ++ Python/bltinmodule.c | 2 ++ Python/peephole.c | 2 ++ configure | 1 + configure.ac | 1 + 14 files changed, 63 insertions(+), 9 deletions(-) diff --git a/Lib/glob.py b/Lib/glob.py --- a/Lib/glob.py +++ b/Lib/glob.py @@ -5,6 +5,14 @@ import re import fnmatch +try: + _unicode = unicode +except NameError: + # If Python is built without Unicode support, the unicode type + # will not exist. Fake one. + class _unicode(object): + pass + __all__ = ["glob", "iglob"] def glob(pathname): @@ -49,7 +57,7 @@ def glob1(dirname, pattern): if not dirname: dirname = os.curdir - if isinstance(pattern, unicode) and not isinstance(dirname, unicode): + if isinstance(pattern, _unicode) and not isinstance(dirname, unicode): dirname = unicode(dirname, sys.getfilesystemencoding() or sys.getdefaultencoding()) try: diff --git a/Lib/locale.py b/Lib/locale.py --- a/Lib/locale.py +++ b/Lib/locale.py @@ -18,6 +18,14 @@ import operator import functools +try: + _unicode = unicode +except NameError: + # If Python is built without Unicode support, the unicode type + # will not exist. Fake one. + class _unicode(object): + pass + # Try importing the _locale module. # # If this fails, fall back on a basic 'C' locale emulation. @@ -353,7 +361,7 @@ """ # Normalize the locale name and extract the encoding - if isinstance(localename, unicode): + if isinstance(localename, _unicode): localename = localename.encode('ascii') fullname = localename.translate(_ascii_lower_map) if ':' in fullname: diff --git a/Lib/posixpath.py b/Lib/posixpath.py --- a/Lib/posixpath.py +++ b/Lib/posixpath.py @@ -17,6 +17,14 @@ import warnings from genericpath import * +try: + _unicode = unicode +except NameError: + # If Python is built without Unicode support, the unicode type + # will not exist. Fake one. + class _unicode(object): + pass + __all__ = ["normcase","isabs","join","splitdrive","split","splitext", "basename","dirname","commonprefix","getsize","getmtime", "getatime","getctime","islink","exists","lexists","isdir","isfile", @@ -312,7 +320,7 @@ def normpath(path): """Normalize path, eliminating double slashes, etc.""" # Preserve unicode (if path is unicode) - slash, dot = (u'/', u'.') if isinstance(path, unicode) else ('/', '.') + slash, dot = (u'/', u'.') if isinstance(path, _unicode) else ('/', '.') if path == '': return dot initial_slashes = path.startswith('/') @@ -341,7 +349,7 @@ def abspath(path): """Return an absolute path.""" if not isabs(path): - if isinstance(path, unicode): + if isinstance(path, _unicode): cwd = os.getcwdu() else: cwd = os.getcwd() diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py --- a/Lib/test/script_helper.py +++ b/Lib/test/script_helper.py @@ -10,7 +10,13 @@ import py_compile import contextlib import shutil -import zipfile +try: + import zipfile +except ImportError: + # If Python is build without Unicode support, importing _io will + # fail, which, in turn, means that zipfile cannot be imported + # Most of this module can then still be used. + pass from test.test_support import strip_python_stderr diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -405,7 +405,7 @@ the CWD, an error is raised. If it's True, only a warning is raised and the original CWD is used. """ - if isinstance(name, unicode): + if have_unicode and isinstance(name, unicode): try: name = name.encode(sys.getfilesystemencoding() or 'ascii') except UnicodeEncodeError: diff --git a/Lib/textwrap.py b/Lib/textwrap.py --- a/Lib/textwrap.py +++ b/Lib/textwrap.py @@ -9,6 +9,14 @@ import string, re +try: + _unicode = unicode +except NameError: + # If Python is built without Unicode support, the unicode type + # will not exist. Fake one. + class _unicode(object): + pass + # Do the right thing with boolean values for all known Python versions # (so this module can be copied to projects that don't depend on Python # 2.3, e.g. Optik and Docutils) by uncommenting the block of code below. @@ -147,7 +155,7 @@ if self.replace_whitespace: if isinstance(text, str): text = text.translate(self.whitespace_trans) - elif isinstance(text, unicode): + elif isinstance(text, _unicode): text = text.translate(self.unicode_whitespace_trans) return text @@ -167,7 +175,7 @@ 'use', ' ', 'the', ' ', '-b', ' ', option!' otherwise. """ - if isinstance(text, unicode): + if isinstance(text, _unicode): if self.break_on_hyphens: pat = self.wordsep_re_uni else: diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -202,7 +202,11 @@ self.addTypeEqualityFunc(tuple, 'assertTupleEqual') self.addTypeEqualityFunc(set, 'assertSetEqual') self.addTypeEqualityFunc(frozenset, 'assertSetEqual') - self.addTypeEqualityFunc(unicode, 'assertMultiLineEqual') + try: + self.addTypeEqualityFunc(unicode, 'assertMultiLineEqual') + except NameError: + # No unicode support in this build + pass def addTypeEqualityFunc(self, typeobj, function): """Add a type specific assertEqual style function to compare a type. diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -839,6 +839,7 @@ Geoff Talvola William Tanksley Christian Tanzer +Stefano Taschini Steven Taschuk Monty Taylor Amy Taylor diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -191,6 +191,9 @@ Build ----- +- Issue #8767: Restore building with --disable-unicode. + Patch by Stefano Taschini. + - Build against bzip2 1.0.6 and openssl 0.9.8x on Windows. - Issue #14557: Fix extensions build on HP-UX. Patch by Adi Roiban. diff --git a/Objects/object.c b/Objects/object.c --- a/Objects/object.c +++ b/Objects/object.c @@ -2111,8 +2111,10 @@ if (PyType_Ready(&PySet_Type) < 0) Py_FatalError("Can't initialize set type"); +#ifdef Py_USING_UNICODE if (PyType_Ready(&PyUnicode_Type) < 0) Py_FatalError("Can't initialize unicode type"); +#endif if (PyType_Ready(&PySlice_Type) < 0) Py_FatalError("Can't initialize slice type"); diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -1578,6 +1578,7 @@ Py_CLEAR(str_newline); return NULL; } +#ifdef Py_USING_UNICODE unicode_newline = PyUnicode_FromString("\n"); if (unicode_newline == NULL) { Py_CLEAR(str_newline); @@ -1591,6 +1592,7 @@ Py_CLEAR(unicode_space); return NULL; } +#endif } if (!PyArg_ParseTupleAndKeywords(dummy_args, kwds, "|OOO:print", kwlist, &sep, &end, &file)) diff --git a/Python/peephole.c b/Python/peephole.c --- a/Python/peephole.c +++ b/Python/peephole.c @@ -135,6 +135,7 @@ will return a surrogate. In both the cases skip the optimization in order to produce compatible pycs. */ +#ifdef Py_USING_UNICODE if (newconst != NULL && PyUnicode_Check(v) && PyUnicode_Check(newconst)) { Py_UNICODE ch = PyUnicode_AS_UNICODE(newconst)[0]; @@ -147,6 +148,7 @@ return 0; } } +#endif break; case BINARY_LSHIFT: newconst = PyNumber_Lshift(v, w); diff --git a/configure b/configure --- a/configure +++ b/configure @@ -12522,6 +12522,7 @@ $as_echo "#define Py_UNICODE_SIZE 4" >>confdefs.h ;; +no) ;; # To allow --disable-unicode *) as_fn_error $? "invalid value for --enable-unicode. Use either ucs2 or ucs4 (lowercase)." "$LINENO" 5 ;; esac diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -3776,6 +3776,7 @@ ucs4) unicode_size="4" AC_DEFINE(Py_UNICODE_SIZE,4) ;; +no) ;; # To allow --disable-unicode *) AC_MSG_ERROR([invalid value for --enable-unicode. Use either ucs2 or ucs4 (lowercase).]) ;; esac -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 10:53:54 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 20 May 2012 10:53:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Temporarily_dis?= =?utf8?q?abled_test=5Frace_on_Windows=2E?= Message-ID: http://hg.python.org/cpython/rev/235ee0a32ab5 changeset: 77073:235ee0a32ab5 branch: 2.7 user: Vinay Sajip date: Sun May 20 09:53:13 2012 +0100 summary: Temporarily disabled test_race on Windows. files: Lib/test/test_logging.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1877,6 +1877,7 @@ class HandlerTest(BaseTest): + @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') @unittest.skipUnless(threading, 'Threading required for this test.') def test_race(self): # Issue #14632 refers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:00:21 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 11:00:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_for_issue14?= =?utf8?q?426_-_buildbots_here_I_come?= Message-ID: http://hg.python.org/cpython/rev/f457419552e3 changeset: 77074:f457419552e3 branch: 2.7 parent: 77072:d7aff4423172 user: Senthil Kumaran date: Sun May 20 16:56:24 2012 +0800 summary: Fix for issue14426 - buildbots here I come files: Lib/Cookie.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/Cookie.py b/Lib/Cookie.py --- a/Lib/Cookie.py +++ b/Lib/Cookie.py @@ -539,7 +539,7 @@ r"(?P" # Start of group 'val' r'"(?:[^\\"]|\\.)*"' # Any doublequoted string r"|" # or - r"\w{3},\s[\w\d-]{9,11}\s[\d:]{8}\sGMT" # Special case for "expires" attr + r"\w{3},\s[\s\w\d-]{9,11}\s[\d:]{8}\sGMT" # Special case for "expires" attr r"|" # or ""+ _LegalCharsPatt +"*" # Any word or empty string r")" # End of group 'val' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:00:22 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 11:00:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_for_issue14?= =?utf8?q?426_-_buildbots_here_I_come?= Message-ID: http://hg.python.org/cpython/rev/f9d31d6977f1 changeset: 77075:f9d31d6977f1 branch: 3.2 parent: 77065:32cf38bfb36f user: Senthil Kumaran date: Sun May 20 16:58:30 2012 +0800 summary: Fix for issue14426 - buildbots here I come files: Lib/http/cookies.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/http/cookies.py b/Lib/http/cookies.py --- a/Lib/http/cookies.py +++ b/Lib/http/cookies.py @@ -439,7 +439,7 @@ (?P # Start of group 'val' "(?:[^\\"]|\\.)*" # Any doublequoted string | # or - \w{3},\s[\w\d-]{9,11}\s[\d:]{8}\sGMT # Special case for "expires" attr + \w{3},\s[\w\d\s-]{9,11}\s[\d:]{8}\sGMT # Special case for "expires" attr | # or """ + _LegalCharsPatt + r"""* # Any word or empty string ) # End of group 'val' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:00:23 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 11:00:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_merge_-_Fix_for_issue14426_-_buildbots_here_I_come?= Message-ID: http://hg.python.org/cpython/rev/ca2a793d6c7b changeset: 77076:ca2a793d6c7b parent: 77071:71989b8349ca parent: 77075:f9d31d6977f1 user: Senthil Kumaran date: Sun May 20 16:58:59 2012 +0800 summary: merge - Fix for issue14426 - buildbots here I come files: Lib/http/cookies.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/http/cookies.py b/Lib/http/cookies.py --- a/Lib/http/cookies.py +++ b/Lib/http/cookies.py @@ -439,7 +439,7 @@ (?P # Start of group 'val' "(?:[^\\"]|\\.)*" # Any doublequoted string | # or - \w{3},\s[\w\d-]{9,11}\s[\d:]{8}\sGMT # Special case for "expires" attr + \w{3},\s[\w\d\s-]{9,11}\s[\d:]{8}\sGMT # Special case for "expires" attr | # or """ + _LegalCharsPatt + r"""* # Any word or empty string ) # End of group 'val' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:00:26 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sun, 20 May 2012 11:00:26 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?= =?utf8?q?_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/bd42068caf00 changeset: 77077:bd42068caf00 branch: 2.7 parent: 77074:f457419552e3 parent: 77073:235ee0a32ab5 user: Senthil Kumaran date: Sun May 20 16:59:51 2012 +0800 summary: merge heads files: Lib/test/test_logging.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1877,6 +1877,7 @@ class HandlerTest(BaseTest): + @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') @unittest.skipUnless(threading, 'Threading required for this test.') def test_race(self): # Issue #14632 refers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:36:53 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 20 May 2012 11:36:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Clarify_a_paragraph_in_the_?= =?utf8?q?new_metaclass_docs?= Message-ID: http://hg.python.org/cpython/rev/c431bac4f880 changeset: 77078:c431bac4f880 parent: 77076:ca2a793d6c7b user: Nick Coghlan date: Sun May 20 19:36:40 2012 +1000 summary: Clarify a paragraph in the new metaclass docs files: Doc/reference/datamodel.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1656,9 +1656,9 @@ lexical scoping, while the class or instance that was used to make the current call is identified based on the first argument passed to the method. -After the class object is created, any class decorators included in the -function definition are invoked and the resulting object is bound in the -local namespace to the name of the class. +After the class object is created, it is passed to the class decorators +included in the class definition (if any) and the resulting object is bound +in the local namespace as the defined class. .. seealso:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:42:15 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 20 May 2012 11:42:15 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Starting_on_the_PEP_3144_i?= =?utf8?q?ntegration?= Message-ID: http://hg.python.org/devguide/rev/67f3b6fdae5a changeset: 514:67f3b6fdae5a user: Nick Coghlan date: Sun May 20 19:42:05 2012 +1000 summary: Starting on the PEP 3144 integration files: experts.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -130,6 +130,7 @@ importlib brett.cannon inspect io pitrou, benjamin.peterson, stutzbach +ipaddress pmoody, ncoghlan itertools rhettinger json bob.ippolito (inactive), ezio.melotti, rhettinger keyword -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Sun May 20 11:43:35 2012 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 20 May 2012 11:43:35 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312760=3A_Add_some_?= =?utf8?q?mising_documentation_about_the_new_=60x=60_exclusive?= Message-ID: http://hg.python.org/cpython/rev/ef406c4c6463 changeset: 77079:ef406c4c6463 parent: 77076:ca2a793d6c7b user: Charles-Fran?ois Natali date: Sun May 20 11:41:53 2012 +0200 summary: Issue #12760: Add some mising documentation about the new `x` exclusive creation flag to open(). files: Doc/library/functions.rst | 15 ++++++++++----- 1 files changed, 10 insertions(+), 5 deletions(-) diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -797,17 +797,19 @@ *mode* is an optional string that specifies the mode in which the file is opened. It defaults to ``'r'`` which means open for reading in text mode. Other common values are ``'w'`` for writing (truncating the file if it - already exists), and ``'a'`` for appending (which on *some* Unix systems, - means that *all* writes append to the end of the file regardless of the - current seek position). In text mode, if *encoding* is not specified the - encoding used is platform dependent. (For reading and writing raw bytes use - binary mode and leave *encoding* unspecified.) The available modes are: + already exists), ``'x'`` for exclusive creation and ``'a'`` for appending + (which on *some* Unix systems, means that *all* writes append to the end of + the file regardless of the current seek position). In text mode, if + *encoding* is not specified the encoding used is platform dependent. (For + reading and writing raw bytes use binary mode and leave *encoding* + unspecified.) The available modes are: ========= =============================================================== Character Meaning --------- --------------------------------------------------------------- ``'r'`` open for reading (default) ``'w'`` open for writing, truncating the file first + ``'x'`` open for exclusive creation, failing if the file already exists ``'a'`` open for writing, appending to the end of the file if it exists ``'b'`` binary mode ``'t'`` text mode (default) @@ -898,6 +900,7 @@ .. versionchanged:: 3.3 The *opener* parameter was added. + The ``'x'`` mode was added. The type of file object returned by the :func:`open` function depends on the mode. When :func:`open` is used to open a file in a text mode (``'w'``, @@ -926,6 +929,8 @@ .. versionchanged:: 3.3 :exc:`IOError` used to be raised, it is now an alias of :exc:`OSError`. + :exc:`FileExistsError` is now raised if the file opened in exclusive + creation mode (``'x'``) already exists. .. XXX works for bytes too, but should it? -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 11:43:36 2012 From: python-checkins at python.org (charles-francois.natali) Date: Sun, 20 May 2012 11:43:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?b?KTogTWVyZ2Uu?= Message-ID: http://hg.python.org/cpython/rev/6c4b39edee94 changeset: 77080:6c4b39edee94 parent: 77079:ef406c4c6463 parent: 77078:c431bac4f880 user: Charles-Fran?ois Natali date: Sun May 20 11:43:28 2012 +0200 summary: Merge. files: Doc/reference/datamodel.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1656,9 +1656,9 @@ lexical scoping, while the class or instance that was used to make the current call is identified based on the first argument passed to the method. -After the class object is created, any class decorators included in the -function definition are invoked and the resulting object is bound in the -local namespace to the name of the class. +After the class object is created, it is passed to the class decorators +included in the class definition (if any) and the resulting object is bound +in the local namespace as the defined class. .. seealso:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 12:09:12 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sun, 20 May 2012 12:09:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Document_when_j?= =?utf8?q?son=2Eload=27s_parse=5Fconstant_behaviour_changed?= Message-ID: http://hg.python.org/cpython/rev/aad7cf67b9ea changeset: 77081:aad7cf67b9ea branch: 2.7 parent: 77077:bd42068caf00 user: Hynek Schlawack date: Sun May 20 11:50:41 2012 +0200 summary: Document when json.load's parse_constant behaviour changed It doesn't get called on 'null', 'true', 'false' since f686aced02a3. files: Doc/library/json.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -221,6 +221,9 @@ This can be used to raise an exception if invalid JSON numbers are encountered. + .. versionchanged:: 2.7 + *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. + To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments will be passed to the constructor of the class. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 12:09:13 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sun, 20 May 2012 12:09:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Document_when_j?= =?utf8?q?son=2Eload=27s_parse=5Fconstant_behaviour_changed?= Message-ID: http://hg.python.org/cpython/rev/b2dce31d6aec changeset: 77082:b2dce31d6aec branch: 3.2 parent: 77075:f9d31d6977f1 user: Hynek Schlawack date: Sun May 20 12:03:17 2012 +0200 summary: Document when json.load's parse_constant behaviour changed It doesn't get called on 'null', 'true', 'false' since f686aced02a3. files: Doc/library/json.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -213,6 +213,9 @@ This can be used to raise an exception if invalid JSON numbers are encountered. + .. versionchanged:: 2.7 + *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. + To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments will be passed to the constructor of the class. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 12:09:14 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sun, 20 May 2012 12:09:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Document_when_json=2Eload=27s_parse=5Fconstant_behaviour_cha?= =?utf8?q?nged?= Message-ID: http://hg.python.org/cpython/rev/625f580a4a7a changeset: 77083:625f580a4a7a parent: 77080:6c4b39edee94 parent: 77082:b2dce31d6aec user: Hynek Schlawack date: Sun May 20 12:04:01 2012 +0200 summary: Document when json.load's parse_constant behaviour changed It doesn't get called on 'null', 'true', 'false' since f686aced02a3. files: Doc/library/json.rst | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -213,6 +213,9 @@ This can be used to raise an exception if invalid JSON numbers are encountered. + .. versionchanged:: 2.7 + *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. + To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` kwarg; otherwise :class:`JSONDecoder` is used. Additional keyword arguments will be passed to the constructor of the class. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 13:02:16 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 20 May 2012 13:02:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_addition_?= =?utf8?q?of_the_ipaddress_module_=28stage_1_-_code_and_tests=29?= Message-ID: http://hg.python.org/cpython/rev/82a649047272 changeset: 77084:82a649047272 user: Nick Coghlan date: Sun May 20 21:01:57 2012 +1000 summary: Issue #14814: addition of the ipaddress module (stage 1 - code and tests) files: Doc/whatsnew/3.3.rst | 9 + Lib/ipaddress.py | 2193 ++++++++++++++++++++++++ Lib/test/test_ipaddress.py | 1142 ++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 2 + 5 files changed, 3347 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -851,6 +851,15 @@ (Contributed by David Townshend in :issue:`12760`) +ipaddress +--------- + +The new :mod:`ipaddress` module provides tools for creating and manipulating +objects representing IPv4 and IPv6 addresses, networks and interfaces (i.e. +an IP address associated with a specific IP subnet). + +(Contributed by Google and Peter Moody in :pep:`3144`) + lzma ---- diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py new file mode 100644 --- /dev/null +++ b/Lib/ipaddress.py @@ -0,0 +1,2193 @@ +#!/usr/bin/python3 +# +# Copyright 2007 Google Inc. +# Licensed to PSF under a Contributor Agreement. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. See the License for the specific language governing +# permissions and limitations under the License. + +"""A fast, lightweight IPv4/IPv6 manipulation library in Python. + +This library is used to create/poke/manipulate IPv4 and IPv6 addresses +and networks. + +""" + +__version__ = '1.0' + +import struct + +IPV4LENGTH = 32 +IPV6LENGTH = 128 + + +class AddressValueError(ValueError): + """A Value Error related to the address.""" + + +class NetmaskValueError(ValueError): + """A Value Error related to the netmask.""" + + +def ip_address(address, version=None): + """Take an IP string/int and return an object of the correct type. + + Args: + address: A string or integer, the IP address. Either IPv4 or + IPv6 addresses may be supplied; integers less than 2**32 will + be considered to be IPv4 by default. + version: An Integer, 4 or 6. If set, don't try to automatically + determine what the IP address type is. important for things + like ip_address(1), which could be IPv4, '192.0.2.1', or IPv6, + '2001:db8::1'. + + Returns: + An IPv4Address or IPv6Address object. + + Raises: + ValueError: if the string passed isn't either a v4 or a v6 + address. + + """ + if version: + if version == 4: + return IPv4Address(address) + elif version == 6: + return IPv6Address(address) + + try: + return IPv4Address(address) + except (AddressValueError, NetmaskValueError): + pass + + try: + return IPv6Address(address) + except (AddressValueError, NetmaskValueError): + pass + + raise ValueError('%r does not appear to be an IPv4 or IPv6 address' % + address) + + +def ip_network(address, version=None, strict=True): + """Take an IP string/int and return an object of the correct type. + + Args: + address: A string or integer, the IP network. Either IPv4 or + IPv6 networks may be supplied; integers less than 2**32 will + be considered to be IPv4 by default. + version: An Integer, if set, don't try to automatically + determine what the IP address type is. important for things + like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, + '2001:db8::1/128'. + + Returns: + An IPv4Network or IPv6Network object. + + Raises: + ValueError: if the string passed isn't either a v4 or a v6 + address. Or if the network has host bits set. + + """ + if version: + if version == 4: + return IPv4Network(address, strict) + elif version == 6: + return IPv6Network(address, strict) + + try: + return IPv4Network(address, strict) + except (AddressValueError, NetmaskValueError): + pass + + try: + return IPv6Network(address, strict) + except (AddressValueError, NetmaskValueError): + pass + + raise ValueError('%r does not appear to be an IPv4 or IPv6 network' % + address) + + +def ip_interface(address, version=None): + """Take an IP string/int and return an object of the correct type. + + Args: + address: A string or integer, the IP address. Either IPv4 or + IPv6 addresses may be supplied; integers less than 2**32 will + be considered to be IPv4 by default. + version: An Integer, if set, don't try to automatically + determine what the IP address type is. important for things + like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, + '2001:db8::1/128'. + + Returns: + An IPv4Network or IPv6Network object. + + Raises: + ValueError: if the string passed isn't either a v4 or a v6 + address. + + Notes: + The IPv?Interface classes describe an Address on a particular + Network, so they're basically a combination of both the Address + and Network classes. + """ + if version: + if version == 4: + return IPv4Interface(address) + elif version == 6: + return IPv6Interface(address) + + try: + return IPv4Interface(address) + except (AddressValueError, NetmaskValueError): + pass + + try: + return IPv6Interface(address) + except (AddressValueError, NetmaskValueError): + pass + + raise ValueError('%r does not appear to be an IPv4 or IPv6 network' % + address) + + +def v4_int_to_packed(address): + """The binary representation of this address. + + Args: + address: An integer representation of an IPv4 IP address. + + Returns: + The binary representation of this address. + + Raises: + ValueError: If the integer is too large to be an IPv4 IP + address. + """ + if address > _BaseV4._ALL_ONES: + raise ValueError('Address too large for IPv4') + return struct.pack('!I', address) + + +def v6_int_to_packed(address): + """The binary representation of this address. + + Args: + address: An integer representation of an IPv4 IP address. + + Returns: + The binary representation of this address. + """ + return struct.pack('!QQ', address >> 64, address & (2**64 - 1)) + + +def _find_address_range(addresses): + """Find a sequence of addresses. + + Args: + addresses: a list of IPv4 or IPv6 addresses. + + Returns: + A tuple containing the first and last IP addresses in the sequence. + + """ + first = last = addresses[0] + for ip in addresses[1:]: + if ip._ip == last._ip + 1: + last = ip + else: + break + return (first, last) + +def _get_prefix_length(number1, number2, bits): + """Get the number of leading bits that are same for two numbers. + + Args: + number1: an integer. + number2: another integer. + bits: the maximum number of bits to compare. + + Returns: + The number of leading bits that are the same for two numbers. + + """ + for i in range(bits): + if number1 >> i == number2 >> i: + return bits - i + return 0 + +def _count_righthand_zero_bits(number, bits): + """Count the number of zero bits on the right hand side. + + Args: + number: an integer. + bits: maximum number of bits to count. + + Returns: + The number of zero bits on the right hand side of the number. + + """ + if number == 0: + return bits + for i in range(bits): + if (number >> i) % 2: + return i + + +def summarize_address_range(first, last): + """Summarize a network range given the first and last IP addresses. + + Example: + >>> summarize_address_range(IPv4Address('192.0.2.0'), + IPv4Address('192.0.2.130')) + [IPv4Network('192.0.2.0/25'), IPv4Network('192.0.2.128/31'), + IPv4Network('192.0.2.130/32')] + + Args: + first: the first IPv4Address or IPv6Address in the range. + last: the last IPv4Address or IPv6Address in the range. + + Returns: + An iterator of the summarized IPv(4|6) network objects. + + Raise: + TypeError: + If the first and last objects are not IP addresses. + If the first and last objects are not the same version. + ValueError: + If the last object is not greater than the first. + If the version is not 4 or 6. + + """ + if not (isinstance(first, _BaseAddress) and isinstance(last, _BaseAddress)): + raise TypeError('first and last must be IP addresses, not networks') + if first.version != last.version: + raise TypeError("%s and %s are not of the same version" % ( + str(first), str(last))) + if first > last: + raise ValueError('last IP address must be greater than first') + + networks = [] + + if first.version == 4: + ip = IPv4Network + elif first.version == 6: + ip = IPv6Network + else: + raise ValueError('unknown IP version') + + ip_bits = first._max_prefixlen + first_int = first._ip + last_int = last._ip + while first_int <= last_int: + nbits = _count_righthand_zero_bits(first_int, ip_bits) + current = None + while nbits >= 0: + addend = 2**nbits - 1 + current = first_int + addend + nbits -= 1 + if current <= last_int: + break + prefix = _get_prefix_length(first_int, current, ip_bits) + net = ip('%s/%d' % (str(first), prefix)) + yield net + #networks.append(net) + if current == ip._ALL_ONES: + break + first_int = current + 1 + first = ip_address(first_int, version=first._version) + +def _collapse_addresses_recursive(addresses): + """Loops through the addresses, collapsing concurrent netblocks. + + Example: + + ip1 = IPv4Network('192.0.2.0/26') + ip2 = IPv4Network('192.0.2.64/26') + ip3 = IPv4Network('192.0.2.128/26') + ip4 = IPv4Network('192.0.2.192/26') + + _collapse_addresses_recursive([ip1, ip2, ip3, ip4]) -> + [IPv4Network('192.0.2.0/24')] + + This shouldn't be called directly; it is called via + collapse_addresses([]). + + Args: + addresses: A list of IPv4Network's or IPv6Network's + + Returns: + A list of IPv4Network's or IPv6Network's depending on what we were + passed. + + """ + ret_array = [] + optimized = False + + for cur_addr in addresses: + if not ret_array: + ret_array.append(cur_addr) + continue + if (cur_addr.network_address >= ret_array[-1].network_address and + cur_addr.broadcast_address <= ret_array[-1].broadcast_address): + optimized = True + elif cur_addr == list(ret_array[-1].supernet().subnets())[1]: + ret_array.append(ret_array.pop().supernet()) + optimized = True + else: + ret_array.append(cur_addr) + + if optimized: + return _collapse_addresses_recursive(ret_array) + + return ret_array + + +def collapse_addresses(addresses): + """Collapse a list of IP objects. + + Example: + collapse_addresses([IPv4Network('192.0.2.0/25'), + IPv4Network('192.0.2.128/25')]) -> + [IPv4Network('192.0.2.0/24')] + + Args: + addresses: An iterator of IPv4Network or IPv6Network objects. + + Returns: + An iterator of the collapsed IPv(4|6)Network objects. + + Raises: + TypeError: If passed a list of mixed version objects. + + """ + i = 0 + addrs = [] + ips = [] + nets = [] + + # split IP addresses and networks + for ip in addresses: + if isinstance(ip, _BaseAddress): + if ips and ips[-1]._version != ip._version: + raise TypeError("%s and %s are not of the same version" % ( + str(ip), str(ips[-1]))) + ips.append(ip) + elif ip._prefixlen == ip._max_prefixlen: + if ips and ips[-1]._version != ip._version: + raise TypeError("%s and %s are not of the same version" % ( + str(ip), str(ips[-1]))) + try: + ips.append(ip.ip) + except AttributeError: + ips.append(ip.network_address) + else: + if nets and nets[-1]._version != ip._version: + raise TypeError("%s and %s are not of the same version" % ( + str(ip), str(ips[-1]))) + nets.append(ip) + + # sort and dedup + ips = sorted(set(ips)) + nets = sorted(set(nets)) + + while i < len(ips): + (first, last) = _find_address_range(ips[i:]) + i = ips.index(last) + 1 + addrs.extend(summarize_address_range(first, last)) + + return iter(_collapse_addresses_recursive(sorted( + addrs + nets, key=_BaseNetwork._get_networks_key))) + + +def get_mixed_type_key(obj): + """Return a key suitable for sorting between networks and addresses. + + Address and Network objects are not sortable by default; they're + fundamentally different so the expression + + IPv4Address('192.0.2.0') <= IPv4Network('192.0.2.0/24') + + doesn't make any sense. There are some times however, where you may wish + to have ipaddress sort these for you anyway. If you need to do this, you + can use this function as the key= argument to sorted(). + + Args: + obj: either a Network or Address object. + Returns: + appropriate key. + + """ + if isinstance(obj, _BaseNetwork): + return obj._get_networks_key() + elif isinstance(obj, _BaseAddress): + return obj._get_address_key() + return NotImplemented + + +class _IPAddressBase(object): + + """The mother class.""" + + @property + def exploded(self): + """Return the longhand version of the IP address as a string.""" + return self._explode_shorthand_ip_string() + + @property + def compressed(self): + """Return the shorthand version of the IP address as a string.""" + return str(self) + + def _ip_int_from_prefix(self, prefixlen=None): + """Turn the prefix length netmask into a int for comparison. + + Args: + prefixlen: An integer, the prefix length. + + Returns: + An integer. + + """ + if not prefixlen and prefixlen != 0: + prefixlen = self._prefixlen + return self._ALL_ONES ^ (self._ALL_ONES >> prefixlen) + + def _prefix_from_ip_int(self, ip_int, mask=32): + """Return prefix length from the decimal netmask. + + Args: + ip_int: An integer, the IP address. + mask: The netmask. Defaults to 32. + + Returns: + An integer, the prefix length. + + """ + while mask: + if ip_int & 1 == 1: + break + ip_int >>= 1 + mask -= 1 + + return mask + + def _ip_string_from_prefix(self, prefixlen=None): + """Turn a prefix length into a dotted decimal string. + + Args: + prefixlen: An integer, the netmask prefix length. + + Returns: + A string, the dotted decimal netmask string. + + """ + if not prefixlen: + prefixlen = self._prefixlen + return self._string_from_ip_int(self._ip_int_from_prefix(prefixlen)) + + +class _BaseAddress(_IPAddressBase): + + """A generic IP object. + + This IP class contains the version independent methods which are + used by single IP addresses. + + """ + + def __init__(self, address): + if (not isinstance(address, bytes) + and '/' in str(address)): + raise AddressValueError(address) + + def __index__(self): + return self._ip + + def __int__(self): + return self._ip + + def __hex__(self): + return hex(self._ip) + + def __eq__(self, other): + try: + return (self._ip == other._ip + and self._version == other._version) + except AttributeError: + return NotImplemented + + def __ne__(self, other): + eq = self.__eq__(other) + if eq is NotImplemented: + return NotImplemented + return not eq + + def __le__(self, other): + gt = self.__gt__(other) + if gt is NotImplemented: + return NotImplemented + return not gt + + def __ge__(self, other): + lt = self.__lt__(other) + if lt is NotImplemented: + return NotImplemented + return not lt + + def __lt__(self, other): + if self._version != other._version: + raise TypeError('%s and %s are not of the same version' % ( + str(self), str(other))) + if not isinstance(other, _BaseAddress): + raise TypeError('%s and %s are not of the same type' % ( + str(self), str(other))) + if self._ip != other._ip: + return self._ip < other._ip + return False + + def __gt__(self, other): + if self._version != other._version: + raise TypeError('%s and %s are not of the same version' % ( + str(self), str(other))) + if not isinstance(other, _BaseAddress): + raise TypeError('%s and %s are not of the same type' % ( + str(self), str(other))) + if self._ip != other._ip: + return self._ip > other._ip + return False + + # Shorthand for Integer addition and subtraction. This is not + # meant to ever support addition/subtraction of addresses. + def __add__(self, other): + if not isinstance(other, int): + return NotImplemented + return ip_address(int(self) + other, version=self._version) + + def __sub__(self, other): + if not isinstance(other, int): + return NotImplemented + return ip_address(int(self) - other, version=self._version) + + def __repr__(self): + return '%s(%r)' % (self.__class__.__name__, str(self)) + + def __str__(self): + return '%s' % self._string_from_ip_int(self._ip) + + def __hash__(self): + return hash(hex(int(self._ip))) + + def _get_address_key(self): + return (self._version, self) + + @property + def version(self): + raise NotImplementedError('BaseIP has no version') + + +class _BaseNetwork(_IPAddressBase): + + """A generic IP object. + + This IP class contains the version independent methods which are + used by networks. + + """ + + def __init__(self, address): + self._cache = {} + + def __index__(self): + return int(self.network_address) ^ self.prefixlen + + def __int__(self): + return int(self.network_address) + + def __repr__(self): + return '%s(%r)' % (self.__class__.__name__, str(self)) + + def hosts(self): + """Generate Iterator over usable hosts in a network. + + This is like __iter__ except it doesn't return the network + or broadcast addresses. + + """ + cur = int(self.network_address) + 1 + bcast = int(self.broadcast_address) - 1 + while cur <= bcast: + cur += 1 + yield ip_address(cur - 1, version=self._version) + + def __iter__(self): + cur = int(self.network_address) + bcast = int(self.broadcast_address) + while cur <= bcast: + cur += 1 + yield ip_address(cur - 1, version=self._version) + + def __getitem__(self, n): + network = int(self.network_address) + broadcast = int(self.broadcast_address) + if n >= 0: + if network + n > broadcast: + raise IndexError + return ip_address(network + n, version=self._version) + else: + n += 1 + if broadcast + n < network: + raise IndexError + return ip_address(broadcast + n, version=self._version) + + def __lt__(self, other): + if self._version != other._version: + raise TypeError('%s and %s are not of the same version' % ( + str(self), str(other))) + if not isinstance(other, _BaseNetwork): + raise TypeError('%s and %s are not of the same type' % ( + str(self), str(other))) + if self.network_address != other.network_address: + return self.network_address < other.network_address + if self.netmask != other.netmask: + return self.netmask < other.netmask + return False + + def __gt__(self, other): + if self._version != other._version: + raise TypeError('%s and %s are not of the same version' % ( + str(self), str(other))) + if not isinstance(other, _BaseNetwork): + raise TypeError('%s and %s are not of the same type' % ( + str(self), str(other))) + if self.network_address != other.network_address: + return self.network_address > other.network_address + if self.netmask != other.netmask: + return self.netmask > other.netmask + return False + + def __le__(self, other): + gt = self.__gt__(other) + if gt is NotImplemented: + return NotImplemented + return not gt + + def __ge__(self, other): + lt = self.__lt__(other) + if lt is NotImplemented: + return NotImplemented + return not lt + + def __eq__(self, other): + if not isinstance(other, _BaseNetwork): + raise TypeError('%s and %s are not of the same type' % ( + str(self), str(other))) + return (self._version == other._version and + self.network_address == other.network_address and + int(self.netmask) == int(other.netmask)) + + def __ne__(self, other): + eq = self.__eq__(other) + if eq is NotImplemented: + return NotImplemented + return not eq + + def __str__(self): + return '%s/%s' % (str(self.ip), + str(self._prefixlen)) + + def __hash__(self): + return hash(int(self.network_address) ^ int(self.netmask)) + + def __contains__(self, other): + # always false if one is v4 and the other is v6. + if self._version != other._version: + return False + # dealing with another network. + if isinstance(other, _BaseNetwork): + return False + # dealing with another address + else: + # address + return (int(self.network_address) <= int(other._ip) <= + int(self.broadcast_address)) + + def overlaps(self, other): + """Tell if self is partly contained in other.""" + return self.network_address in other or ( + self.broadcast_address in other or ( + other.network_address in self or ( + other.broadcast_address in self))) + + @property + def broadcast_address(self): + x = self._cache.get('broadcast_address') + if x is None: + x = ip_address(int(self.network_address) | int(self.hostmask), + version=self._version) + self._cache['broadcast_address'] = x + return x + + @property + def hostmask(self): + x = self._cache.get('hostmask') + if x is None: + x = ip_address(int(self.netmask) ^ self._ALL_ONES, + version=self._version) + self._cache['hostmask'] = x + return x + + @property + def network(self): + return ip_network('%s/%d' % (str(self.network_address), + self.prefixlen)) + + @property + def with_prefixlen(self): + return '%s/%d' % (str(self.ip), self._prefixlen) + + @property + def with_netmask(self): + return '%s/%s' % (str(self.ip), str(self.netmask)) + + @property + def with_hostmask(self): + return '%s/%s' % (str(self.ip), str(self.hostmask)) + + @property + def num_addresses(self): + """Number of hosts in the current subnet.""" + return int(self.broadcast_address) - int(self.network_address) + 1 + + @property + def version(self): + raise NotImplementedError('BaseNet has no version') + + @property + def prefixlen(self): + return self._prefixlen + + def address_exclude(self, other): + """Remove an address from a larger block. + + For example: + + addr1 = ip_network('192.0.2.0/28') + addr2 = ip_network('192.0.2.1/32') + addr1.address_exclude(addr2) = + [IPv4Network('192.0.2.0/32'), IPv4Network('192.0.2.2/31'), + IPv4Network('192.0.2.4/30'), IPv4Network('192.0.2.8/29')] + + or IPv6: + + addr1 = ip_network('2001:db8::1/32') + addr2 = ip_network('2001:db8::1/128') + addr1.address_exclude(addr2) = + [ip_network('2001:db8::1/128'), + ip_network('2001:db8::2/127'), + ip_network('2001:db8::4/126'), + ip_network('2001:db8::8/125'), + ... + ip_network('2001:db8:8000::/33')] + + Args: + other: An IPv4Network or IPv6Network object of the same type. + + Returns: + An iterator of the the IPv(4|6)Network objects which is self + minus other. + + Raises: + TypeError: If self and other are of difffering address + versions, or if other is not a network object. + ValueError: If other is not completely contained by self. + + """ + if not self._version == other._version: + raise TypeError("%s and %s are not of the same version" % ( + str(self), str(other))) + + if not isinstance(other, _BaseNetwork): + raise TypeError("%s is not a network object" % str(other)) + + if not (other.network_address >= self.network_address and + other.broadcast_address <= self.broadcast_address): + raise ValueError('%s not contained in %s' % (str(other), str(self))) + + if other == self: + raise StopIteration + + ret_addrs = [] + + # Make sure we're comparing the network of other. + other = ip_network('%s/%s' % (str(other.network_address), + str(other.prefixlen)), + version=other._version) + + s1, s2 = self.subnets() + while s1 != other and s2 != other: + if (other.network_address >= s1.network_address and + other.broadcast_address <= s1.broadcast_address): + yield s2 + s1, s2 = s1.subnets() + elif (other.network_address >= s2.network_address and + other.broadcast_address <= s2.broadcast_address): + yield s1 + s1, s2 = s2.subnets() + else: + # If we got here, there's a bug somewhere. + raise AssertionError('Error performing exclusion: ' + 's1: %s s2: %s other: %s' % + (str(s1), str(s2), str(other))) + if s1 == other: + yield s2 + elif s2 == other: + yield s1 + else: + # If we got here, there's a bug somewhere. + raise AssertionError('Error performing exclusion: ' + 's1: %s s2: %s other: %s' % + (str(s1), str(s2), str(other))) + + def compare_networks(self, other): + """Compare two IP objects. + + This is only concerned about the comparison of the integer + representation of the network addresses. This means that the + host bits aren't considered at all in this method. If you want + to compare host bits, you can easily enough do a + 'HostA._ip < HostB._ip' + + Args: + other: An IP object. + + Returns: + If the IP versions of self and other are the same, returns: + + -1 if self < other: + eg: IPv4Network('192.0.2.0/25') < IPv4Network('192.0.2.128/25') + IPv6Network('2001:db8::1000/124') < + IPv6Network('2001:db8::2000/124') + 0 if self == other + eg: IPv4Network('192.0.2.0/24') == IPv4Network('192.0.2.0/24') + IPv6Network('2001:db8::1000/124') == + IPv6Network('2001:db8::1000/124') + 1 if self > other + eg: IPv4Network('192.0.2.128/25') > IPv4Network('192.0.2.0/25') + IPv6Network('2001:db8::2000/124') > + IPv6Network('2001:db8::1000/124') + + Raises: + TypeError if the IP versions are different. + + """ + # does this need to raise a ValueError? + if self._version != other._version: + raise TypeError('%s and %s are not of the same type' % ( + str(self), str(other))) + # self._version == other._version below here: + if self.network_address < other.network_address: + return -1 + if self.network_address > other.network_address: + return 1 + # self.network_address == other.network_address below here: + if self.netmask < other.netmask: + return -1 + if self.netmask > other.netmask: + return 1 + return 0 + + def _get_networks_key(self): + """Network-only key function. + + Returns an object that identifies this address' network and + netmask. This function is a suitable "key" argument for sorted() + and list.sort(). + + """ + return (self._version, self.network_address, self.netmask) + + def subnets(self, prefixlen_diff=1, new_prefix=None): + """The subnets which join to make the current subnet. + + In the case that self contains only one IP + (self._prefixlen == 32 for IPv4 or self._prefixlen == 128 + for IPv6), yield an iterator with just ourself. + + Args: + prefixlen_diff: An integer, the amount the prefix length + should be increased by. This should not be set if + new_prefix is also set. + new_prefix: The desired new prefix length. This must be a + larger number (smaller prefix) than the existing prefix. + This should not be set if prefixlen_diff is also set. + + Returns: + An iterator of IPv(4|6) objects. + + Raises: + ValueError: The prefixlen_diff is too small or too large. + OR + prefixlen_diff and new_prefix are both set or new_prefix + is a smaller number than the current prefix (smaller + number means a larger network) + + """ + if self._prefixlen == self._max_prefixlen: + yield self + return + + if new_prefix is not None: + if new_prefix < self._prefixlen: + raise ValueError('new prefix must be longer') + if prefixlen_diff != 1: + raise ValueError('cannot set prefixlen_diff and new_prefix') + prefixlen_diff = new_prefix - self._prefixlen + + if prefixlen_diff < 0: + raise ValueError('prefix length diff must be > 0') + new_prefixlen = self._prefixlen + prefixlen_diff + + if not self._is_valid_netmask(str(new_prefixlen)): + raise ValueError( + 'prefix length diff %d is invalid for netblock %s' % ( + new_prefixlen, str(self))) + + first = ip_network('%s/%s' % (str(self.network_address), + str(self._prefixlen + prefixlen_diff)), + version=self._version) + + yield first + current = first + while True: + broadcast = current.broadcast_address + if broadcast == self.broadcast_address: + return + new_addr = ip_address(int(broadcast) + 1, version=self._version) + current = ip_network('%s/%s' % (str(new_addr), str(new_prefixlen)), + version=self._version) + + yield current + + def masked(self): + """Return the network object with the host bits masked out.""" + return ip_network('%s/%d' % (self.network_address, self._prefixlen), + version=self._version) + + def supernet(self, prefixlen_diff=1, new_prefix=None): + """The supernet containing the current network. + + Args: + prefixlen_diff: An integer, the amount the prefix length of + the network should be decreased by. For example, given a + /24 network and a prefixlen_diff of 3, a supernet with a + /21 netmask is returned. + + Returns: + An IPv4 network object. + + Raises: + ValueError: If self.prefixlen - prefixlen_diff < 0. I.e., you have a + negative prefix length. + OR + If prefixlen_diff and new_prefix are both set or new_prefix is a + larger number than the current prefix (larger number means a + smaller network) + + """ + if self._prefixlen == 0: + return self + + if new_prefix is not None: + if new_prefix > self._prefixlen: + raise ValueError('new prefix must be shorter') + if prefixlen_diff != 1: + raise ValueError('cannot set prefixlen_diff and new_prefix') + prefixlen_diff = self._prefixlen - new_prefix + + + if self.prefixlen - prefixlen_diff < 0: + raise ValueError( + 'current prefixlen is %d, cannot have a prefixlen_diff of %d' % + (self.prefixlen, prefixlen_diff)) + # TODO (pmoody): optimize this. + t = ip_network('%s/%d' % (str(self.network_address), + self.prefixlen - prefixlen_diff), + version=self._version, strict=False) + return ip_network('%s/%d' % (str(t.network_address), t.prefixlen), + version=t._version) + + +class _BaseV4(object): + + """Base IPv4 object. + + The following methods are used by IPv4 objects in both single IP + addresses and networks. + + """ + + # Equivalent to 255.255.255.255 or 32 bits of 1's. + _ALL_ONES = (2**IPV4LENGTH) - 1 + _DECIMAL_DIGITS = frozenset('0123456789') + + def __init__(self, address): + self._version = 4 + self._max_prefixlen = IPV4LENGTH + + def _explode_shorthand_ip_string(self): + return str(self) + + def _ip_int_from_string(self, ip_str): + """Turn the given IP string into an integer for comparison. + + Args: + ip_str: A string, the IP ip_str. + + Returns: + The IP ip_str as an integer. + + Raises: + AddressValueError: if ip_str isn't a valid IPv4 Address. + + """ + octets = ip_str.split('.') + if len(octets) != 4: + raise AddressValueError(ip_str) + + packed_ip = 0 + for oc in octets: + try: + packed_ip = (packed_ip << 8) | self._parse_octet(oc) + except ValueError: + raise AddressValueError(ip_str) + return packed_ip + + def _parse_octet(self, octet_str): + """Convert a decimal octet into an integer. + + Args: + octet_str: A string, the number to parse. + + Returns: + The octet as an integer. + + Raises: + ValueError: if the octet isn't strictly a decimal from [0..255]. + + """ + # Whitelist the characters, since int() allows a lot of bizarre stuff. + if not self._DECIMAL_DIGITS.issuperset(octet_str): + raise ValueError + octet_int = int(octet_str, 10) + # Disallow leading zeroes, because no clear standard exists on + # whether these should be interpreted as decimal or octal. + if octet_int > 255 or (octet_str[0] == '0' and len(octet_str) > 1): + raise ValueError + return octet_int + + def _string_from_ip_int(self, ip_int): + """Turns a 32-bit integer into dotted decimal notation. + + Args: + ip_int: An integer, the IP address. + + Returns: + The IP address as a string in dotted decimal notation. + + """ + octets = [] + for _ in range(4): + octets.insert(0, str(ip_int & 0xFF)) + ip_int >>= 8 + return '.'.join(octets) + + @property + def max_prefixlen(self): + return self._max_prefixlen + + @property + def version(self): + return self._version + + @property + def is_reserved(self): + """Test if the address is otherwise IETF reserved. + + Returns: + A boolean, True if the address is within the + reserved IPv4 Network range. + + """ + reserved_network = IPv4Network('240.0.0.0/4') + if isinstance(self, _BaseAddress): + return self in reserved_network + return (self.network_address in reserved_network and + self.broadcast_address in reserved_network) + + @property + def is_private(self): + """Test if this address is allocated for private networks. + + Returns: + A boolean, True if the address is reserved per RFC 1918. + + """ + private_10 = IPv4Network('10.0.0.0/8') + private_172 = IPv4Network('172.16.0.0/12') + private_192 = IPv4Network('192.168.0.0/16') + if isinstance(self, _BaseAddress): + return (self in private_10 or self in private_172 or + self in private_192) + else: + return ((self.network_address in private_10 and + self.broadcast_address in private_10) or + (self.network_address in private_172 and + self.broadcast_address in private_172) or + (self.network_address in private_192 and + self.broadcast_address in private_192)) + + @property + def is_multicast(self): + """Test if the address is reserved for multicast use. + + Returns: + A boolean, True if the address is multicast. + See RFC 3171 for details. + + """ + multicast_network = IPv4Network('224.0.0.0/4') + if isinstance(self, _BaseAddress): + return self in IPv4Network('224.0.0.0/4') + return (self.network_address in multicast_network and + self.broadcast_address in multicast_network) + + @property + def is_unspecified(self): + """Test if the address is unspecified. + + Returns: + A boolean, True if this is the unspecified address as defined in + RFC 5735 3. + + """ + unspecified_address = IPv4Address('0.0.0.0') + if isinstance(self, _BaseAddress): + return self in unspecified_address + return (self.network_address == self.broadcast_address == + unspecified_address) + + @property + def is_loopback(self): + """Test if the address is a loopback address. + + Returns: + A boolean, True if the address is a loopback per RFC 3330. + + """ + loopback_address = IPv4Network('127.0.0.0/8') + if isinstance(self, _BaseAddress): + return self in loopback_address + + return (self.network_address in loopback_address and + self.broadcast_address in loopback_address) + + @property + def is_link_local(self): + """Test if the address is reserved for link-local. + + Returns: + A boolean, True if the address is link-local per RFC 3927. + + """ + linklocal_network = IPv4Network('169.254.0.0/16') + if isinstance(self, _BaseAddress): + return self in linklocal_network + return (self.network_address in linklocal_network and + self.broadcast_address in linklocal_network) + + +class IPv4Address(_BaseV4, _BaseAddress): + + """Represent and manipulate single IPv4 Addresses.""" + + def __init__(self, address): + + """ + Args: + address: A string or integer representing the IP + + Additionally, an integer can be passed, so + IPv4Address('192.0.2.1') == IPv4Address(3221225985). + or, more generally + IPv4Address(int(IPv4Address('192.0.2.1'))) == + IPv4Address('192.0.2.1') + + Raises: + AddressValueError: If ipaddressisn't a valid IPv4 address. + + """ + _BaseAddress.__init__(self, address) + _BaseV4.__init__(self, address) + + # Efficient constructor from integer. + if isinstance(address, int): + self._ip = address + if address < 0 or address > self._ALL_ONES: + raise AddressValueError(address) + return + + # Constructing from a packed address + if isinstance(address, bytes) and len(address) == 4: + self._ip = struct.unpack('!I', address)[0] + return + + # Assume input argument to be string or any object representation + # which converts into a formatted IP string. + addr_str = str(address) + self._ip = self._ip_int_from_string(addr_str) + + @property + def packed(self): + """The binary representation of this address.""" + return v4_int_to_packed(self._ip) + + +class IPv4Interface(IPv4Address): + + # the valid octets for host and netmasks. only useful for IPv4. + _valid_mask_octets = set((255, 254, 252, 248, 240, 224, 192, 128, 0)) + + def __init__(self, address): + if isinstance(address, (bytes, int)): + IPv4Address.__init__(self, address) + self.network = IPv4Network(self._ip) + self._prefixlen = self._max_prefixlen + return + + addr = str(address).split('/') + if len(addr) > 2: + raise AddressValueError(address) + IPv4Address.__init__(self, addr[0]) + + self.network = IPv4Network(address, strict=False) + self._prefixlen = self.network._prefixlen + + self.netmask = self.network.netmask + self.hostmask = self.network.hostmask + + + def __str__(self): + return '%s/%d' % (self._string_from_ip_int(self._ip), + self.network.prefixlen) + + def __eq__(self, other): + try: + return (IPv4Address.__eq__(self, other) and + self.network == other.network) + except AttributeError: + return NotImplemented + + def __hash__(self): + return self._ip ^ self._prefixlen ^ int(self.network.network_address) + + def _is_valid_netmask(self, netmask): + """Verify that the netmask is valid. + + Args: + netmask: A string, either a prefix or dotted decimal + netmask. + + Returns: + A boolean, True if the prefix represents a valid IPv4 + netmask. + + """ + mask = netmask.split('.') + if len(mask) == 4: + if [x for x in mask if int(x) not in self._valid_mask_octets]: + return False + if [y for idx, y in enumerate(mask) if idx > 0 and + y > mask[idx - 1]]: + return False + return True + try: + netmask = int(netmask) + except ValueError: + return False + return 0 <= netmask <= self._max_prefixlen + + def _is_hostmask(self, ip_str): + """Test if the IP string is a hostmask (rather than a netmask). + + Args: + ip_str: A string, the potential hostmask. + + Returns: + A boolean, True if the IP string is a hostmask. + + """ + bits = ip_str.split('.') + try: + parts = [int(x) for x in bits if int(x) in self._valid_mask_octets] + except ValueError: + return False + if len(parts) != len(bits): + return False + if parts[0] < parts[-1]: + return True + return False + + + @property + def prefixlen(self): + return self._prefixlen + + @property + def ip(self): + return IPv4Address(self._ip) + + @property + def with_prefixlen(self): + return self + + @property + def with_netmask(self): + return '%s/%s' % (self._string_from_ip_int(self._ip), + self.netmask) + @property + def with_hostmask(self): + return '%s/%s' % (self._string_from_ip_int(self._ip), + self.hostmask) + + +class IPv4Network(_BaseV4, _BaseNetwork): + + """This class represents and manipulates 32-bit IPv4 network + addresses.. + + Attributes: [examples for IPv4Network('192.0.2.0/27')] + .network_address: IPv4Address('192.0.2.0') + .hostmask: IPv4Address('0.0.0.31') + .broadcast_address: IPv4Address('192.0.2.32') + .netmask: IPv4Address('255.255.255.224') + .prefixlen: 27 + + """ + + # the valid octets for host and netmasks. only useful for IPv4. + _valid_mask_octets = set((255, 254, 252, 248, 240, 224, 192, 128, 0)) + + def __init__(self, address, strict=True): + + """Instantiate a new IPv4 network object. + + Args: + address: A string or integer representing the IP [& network]. + '192.0.2.0/24' + '192.0.2.0/255.255.255.0' + '192.0.0.2/0.0.0.255' + are all functionally the same in IPv4. Similarly, + '192.0.2.1' + '192.0.2.1/255.255.255.255' + '192.0.2.1/32' + are also functionaly equivalent. That is to say, failing to + provide a subnetmask will create an object with a mask of /32. + + If the mask (portion after the / in the argument) is given in + dotted quad form, it is treated as a netmask if it starts with a + non-zero field (e.g. /255.0.0.0 == /8) and as a hostmask if it + starts with a zero field (e.g. 0.255.255.255 == /8), with the + single exception of an all-zero mask which is treated as a + netmask == /0. If no mask is given, a default of /32 is used. + + Additionally, an integer can be passed, so + IPv4Network('192.0.2.1') == IPv4Network(3221225985) + or, more generally + IPv4Interface(int(IPv4Interface('192.0.2.1'))) == + IPv4Interface('192.0.2.1') + + Raises: + AddressValueError: If ipaddressisn't a valid IPv4 address. + NetmaskValueError: If the netmask isn't valid for + an IPv4 address. + ValueError: If strict was True and a network address was not + supplied. + + """ + + _BaseV4.__init__(self, address) + _BaseNetwork.__init__(self, address) + + # Constructing from a packed address + if isinstance(address, bytes) and len(address) == 4: + self.network_address = IPv4Address( + struct.unpack('!I', address)[0]) + self._prefixlen = self._max_prefixlen + self.netmask = IPv4Address(self._ALL_ONES) + #fixme: address/network test here + return + + # Efficient constructor from integer. + if isinstance(address, int): + self._prefixlen = self._max_prefixlen + self.netmask = IPv4Address(self._ALL_ONES) + if address < 0 or address > self._ALL_ONES: + raise AddressValueError(address) + self.network_address = IPv4Address(address) + #fixme: address/network test here. + return + + # Assume input argument to be string or any object representation + # which converts into a formatted IP prefix string. + addr = str(address).split('/') + self.network_address = IPv4Address(self._ip_int_from_string(addr[0])) + + if len(addr) > 2: + raise AddressValueError(address) + + if len(addr) == 2: + mask = addr[1].split('.') + + if len(mask) == 4: + # We have dotted decimal netmask. + if self._is_valid_netmask(addr[1]): + self.netmask = IPv4Address(self._ip_int_from_string( + addr[1])) + elif self._is_hostmask(addr[1]): + self.netmask = IPv4Address( + self._ip_int_from_string(addr[1]) ^ self._ALL_ONES) + else: + raise NetmaskValueError('%s is not a valid netmask' + % addr[1]) + + self._prefixlen = self._prefix_from_ip_int(int(self.netmask)) + else: + # We have a netmask in prefix length form. + if not self._is_valid_netmask(addr[1]): + raise NetmaskValueError(addr[1]) + self._prefixlen = int(addr[1]) + self.netmask = IPv4Address(self._ip_int_from_prefix( + self._prefixlen)) + else: + self._prefixlen = self._max_prefixlen + self.netmask = IPv4Address(self._ip_int_from_prefix( + self._prefixlen)) + + if strict: + if (IPv4Address(int(self.network_address) & int(self.netmask)) != + self.network_address): + raise ValueError('%s has host bits set' % self) + self.network_address = IPv4Address(int(self.network_address) & + int(self.netmask)) + + if self._prefixlen == (self._max_prefixlen - 1): + self.hosts = self.__iter__ + + @property + def packed(self): + """The binary representation of this address.""" + return v4_int_to_packed(self.network_address) + + def __str__(self): + return '%s/%d' % (str(self.network_address), + self.prefixlen) + + def _is_valid_netmask(self, netmask): + """Verify that the netmask is valid. + + Args: + netmask: A string, either a prefix or dotted decimal + netmask. + + Returns: + A boolean, True if the prefix represents a valid IPv4 + netmask. + + """ + mask = netmask.split('.') + if len(mask) == 4: + if [x for x in mask if int(x) not in self._valid_mask_octets]: + return False + if [y for idx, y in enumerate(mask) if idx > 0 and + y > mask[idx - 1]]: + return False + return True + try: + netmask = int(netmask) + except ValueError: + return False + return 0 <= netmask <= self._max_prefixlen + + def _is_hostmask(self, ip_str): + """Test if the IP string is a hostmask (rather than a netmask). + + Args: + ip_str: A string, the potential hostmask. + + Returns: + A boolean, True if the IP string is a hostmask. + + """ + bits = ip_str.split('.') + try: + parts = [int(x) for x in bits if int(x) in self._valid_mask_octets] + except ValueError: + return False + if len(parts) != len(bits): + return False + if parts[0] < parts[-1]: + return True + return False + + @property + def with_prefixlen(self): + return '%s/%d' % (str(self.network_address), self._prefixlen) + + @property + def with_netmask(self): + return '%s/%s' % (str(self.network_address), str(self.netmask)) + + @property + def with_hostmask(self): + return '%s/%s' % (str(self.network_address), str(self.hostmask)) + + +class _BaseV6(object): + + """Base IPv6 object. + + The following methods are used by IPv6 objects in both single IP + addresses and networks. + + """ + + _ALL_ONES = (2**IPV6LENGTH) - 1 + _HEXTET_COUNT = 8 + _HEX_DIGITS = frozenset('0123456789ABCDEFabcdef') + + def __init__(self, address): + self._version = 6 + self._max_prefixlen = IPV6LENGTH + + def _ip_int_from_string(self, ip_str): + """Turn an IPv6 ip_str into an integer. + + Args: + ip_str: A string, the IPv6 ip_str. + + Returns: + An int, the IPv6 address + + Raises: + AddressValueError: if ip_str isn't a valid IPv6 Address. + + """ + parts = ip_str.split(':') + + # An IPv6 address needs at least 2 colons (3 parts). + if len(parts) < 3: + raise AddressValueError(ip_str) + + # If the address has an IPv4-style suffix, convert it to hexadecimal. + if '.' in parts[-1]: + ipv4_int = IPv4Address(parts.pop())._ip + parts.append('%x' % ((ipv4_int >> 16) & 0xFFFF)) + parts.append('%x' % (ipv4_int & 0xFFFF)) + + # An IPv6 address can't have more than 8 colons (9 parts). + if len(parts) > self._HEXTET_COUNT + 1: + raise AddressValueError(ip_str) + + # Disregarding the endpoints, find '::' with nothing in between. + # This indicates that a run of zeroes has been skipped. + try: + skip_index, = ( + [i for i in range(1, len(parts) - 1) if not parts[i]] or + [None]) + except ValueError: + # Can't have more than one '::' + raise AddressValueError(ip_str) + + # parts_hi is the number of parts to copy from above/before the '::' + # parts_lo is the number of parts to copy from below/after the '::' + if skip_index is not None: + # If we found a '::', then check if it also covers the endpoints. + parts_hi = skip_index + parts_lo = len(parts) - skip_index - 1 + if not parts[0]: + parts_hi -= 1 + if parts_hi: + raise AddressValueError(ip_str) # ^: requires ^:: + if not parts[-1]: + parts_lo -= 1 + if parts_lo: + raise AddressValueError(ip_str) # :$ requires ::$ + parts_skipped = self._HEXTET_COUNT - (parts_hi + parts_lo) + if parts_skipped < 1: + raise AddressValueError(ip_str) + else: + # Otherwise, allocate the entire address to parts_hi. The endpoints + # could still be empty, but _parse_hextet() will check for that. + if len(parts) != self._HEXTET_COUNT: + raise AddressValueError(ip_str) + parts_hi = len(parts) + parts_lo = 0 + parts_skipped = 0 + + try: + # Now, parse the hextets into a 128-bit integer. + ip_int = 0 + for i in range(parts_hi): + ip_int <<= 16 + ip_int |= self._parse_hextet(parts[i]) + ip_int <<= 16 * parts_skipped + for i in range(-parts_lo, 0): + ip_int <<= 16 + ip_int |= self._parse_hextet(parts[i]) + return ip_int + except ValueError: + raise AddressValueError(ip_str) + + def _parse_hextet(self, hextet_str): + """Convert an IPv6 hextet string into an integer. + + Args: + hextet_str: A string, the number to parse. + + Returns: + The hextet as an integer. + + Raises: + ValueError: if the input isn't strictly a hex number from [0..FFFF]. + + """ + # Whitelist the characters, since int() allows a lot of bizarre stuff. + if not self._HEX_DIGITS.issuperset(hextet_str): + raise ValueError + hextet_int = int(hextet_str, 16) + if hextet_int > 0xFFFF: + raise ValueError + return hextet_int + + def _compress_hextets(self, hextets): + """Compresses a list of hextets. + + Compresses a list of strings, replacing the longest continuous + sequence of "0" in the list with "" and adding empty strings at + the beginning or at the end of the string such that subsequently + calling ":".join(hextets) will produce the compressed version of + the IPv6 address. + + Args: + hextets: A list of strings, the hextets to compress. + + Returns: + A list of strings. + + """ + best_doublecolon_start = -1 + best_doublecolon_len = 0 + doublecolon_start = -1 + doublecolon_len = 0 + for index in range(len(hextets)): + if hextets[index] == '0': + doublecolon_len += 1 + if doublecolon_start == -1: + # Start of a sequence of zeros. + doublecolon_start = index + if doublecolon_len > best_doublecolon_len: + # This is the longest sequence of zeros so far. + best_doublecolon_len = doublecolon_len + best_doublecolon_start = doublecolon_start + else: + doublecolon_len = 0 + doublecolon_start = -1 + + if best_doublecolon_len > 1: + best_doublecolon_end = (best_doublecolon_start + + best_doublecolon_len) + # For zeros at the end of the address. + if best_doublecolon_end == len(hextets): + hextets += [''] + hextets[best_doublecolon_start:best_doublecolon_end] = [''] + # For zeros at the beginning of the address. + if best_doublecolon_start == 0: + hextets = [''] + hextets + + return hextets + + def _string_from_ip_int(self, ip_int=None): + """Turns a 128-bit integer into hexadecimal notation. + + Args: + ip_int: An integer, the IP address. + + Returns: + A string, the hexadecimal representation of the address. + + Raises: + ValueError: The address is bigger than 128 bits of all ones. + + """ + if not ip_int and ip_int != 0: + ip_int = int(self._ip) + + if ip_int > self._ALL_ONES: + raise ValueError('IPv6 address is too large') + + hex_str = '%032x' % ip_int + hextets = [] + for x in range(0, 32, 4): + hextets.append('%x' % int(hex_str[x:x+4], 16)) + + hextets = self._compress_hextets(hextets) + return ':'.join(hextets) + + def _explode_shorthand_ip_string(self): + """Expand a shortened IPv6 address. + + Args: + ip_str: A string, the IPv6 address. + + Returns: + A string, the expanded IPv6 address. + + """ + if isinstance(self, IPv6Network): + ip_str = str(self.network_address) + elif isinstance(self, IPv6Interface): + ip_str = str(self.ip) + else: + ip_str = str(self) + + ip_int = self._ip_int_from_string(ip_str) + parts = [] + for i in range(self._HEXTET_COUNT): + parts.append('%04x' % (ip_int & 0xFFFF)) + ip_int >>= 16 + parts.reverse() + if isinstance(self, (_BaseNetwork, IPv6Interface)): + return '%s/%d' % (':'.join(parts), self.prefixlen) + return ':'.join(parts) + + @property + def max_prefixlen(self): + return self._max_prefixlen + + @property + def packed(self): + """The binary representation of this address.""" + return v6_int_to_packed(self._ip) + + @property + def version(self): + return self._version + + @property + def is_multicast(self): + """Test if the address is reserved for multicast use. + + Returns: + A boolean, True if the address is a multicast address. + See RFC 2373 2.7 for details. + + """ + multicast_network = IPv6Network('ff00::/8') + if isinstance(self, _BaseAddress): + return self in multicast_network + return (self.network_address in multicast_network and + self.broadcast_address in multicast_network) + + @property + def is_reserved(self): + """Test if the address is otherwise IETF reserved. + + Returns: + A boolean, True if the address is within one of the + reserved IPv6 Network ranges. + + """ + reserved_networks = [IPv6Network('::/8'), IPv6Network('100::/8'), + IPv6Network('200::/7'), IPv6Network('400::/6'), + IPv6Network('800::/5'), IPv6Network('1000::/4'), + IPv6Network('4000::/3'), IPv6Network('6000::/3'), + IPv6Network('8000::/3'), IPv6Network('A000::/3'), + IPv6Network('C000::/3'), IPv6Network('E000::/4'), + IPv6Network('F000::/5'), IPv6Network('F800::/6'), + IPv6Network('FE00::/9')] + + if isinstance(self, _BaseAddress): + return len([x for x in reserved_networks if self in x]) > 0 + return len([x for x in reserved_networks if self.network_address in x + and self.broadcast_address in x]) > 0 + + @property + def is_link_local(self): + """Test if the address is reserved for link-local. + + Returns: + A boolean, True if the address is reserved per RFC 4291. + + """ + linklocal_network = IPv6Network('fe80::/10') + if isinstance(self, _BaseAddress): + return self in linklocal_network + return (self.network_address in linklocal_network and + self.broadcast_address in linklocal_network) + + @property + def is_site_local(self): + """Test if the address is reserved for site-local. + + Note that the site-local address space has been deprecated by RFC 3879. + Use is_private to test if this address is in the space of unique local + addresses as defined by RFC 4193. + + Returns: + A boolean, True if the address is reserved per RFC 3513 2.5.6. + + """ + sitelocal_network = IPv6Network('fec0::/10') + if isinstance(self, _BaseAddress): + return self in sitelocal_network + return (self.network_address in sitelocal_network and + self.broadcast_address in sitelocal_network) + + @property + def is_private(self): + """Test if this address is allocated for private networks. + + Returns: + A boolean, True if the address is reserved per RFC 4193. + + """ + private_network = IPv6Network('fc00::/7') + if isinstance(self, _BaseAddress): + return self in private_network + return (self.network_address in private_network and + self.broadcast_address in private_network) + + + @property + def ipv4_mapped(self): + """Return the IPv4 mapped address. + + Returns: + If the IPv6 address is a v4 mapped address, return the + IPv4 mapped address. Return None otherwise. + + """ + if (self._ip >> 32) != 0xFFFF: + return None + return IPv4Address(self._ip & 0xFFFFFFFF) + + @property + def teredo(self): + """Tuple of embedded teredo IPs. + + Returns: + Tuple of the (server, client) IPs or None if the address + doesn't appear to be a teredo address (doesn't start with + 2001::/32) + + """ + if (self._ip >> 96) != 0x20010000: + return None + return (IPv4Address((self._ip >> 64) & 0xFFFFFFFF), + IPv4Address(~self._ip & 0xFFFFFFFF)) + + @property + def sixtofour(self): + """Return the IPv4 6to4 embedded address. + + Returns: + The IPv4 6to4-embedded address if present or None if the + address doesn't appear to contain a 6to4 embedded address. + + """ + if (self._ip >> 112) != 0x2002: + return None + return IPv4Address((self._ip >> 80) & 0xFFFFFFFF) + + @property + def is_unspecified(self): + """Test if the address is unspecified. + + Returns: + A boolean, True if this is the unspecified address as defined in + RFC 2373 2.5.2. + + """ + if isinstance(self, (IPv6Network, IPv6Interface)): + return int(self.network_address) == 0 and getattr( + self, '_prefixlen', 128) == 128 + return self._ip == 0 + + @property + def is_loopback(self): + """Test if the address is a loopback address. + + Returns: + A boolean, True if the address is a loopback address as defined in + RFC 2373 2.5.3. + + """ + if isinstance(self, IPv6Network): + return int(self.network) == 1 and getattr( + self, '_prefixlen', 128) == 128 + elif isinstance(self, IPv6Interface): + return int(self.network.network_address) == 1 and getattr( + self, '_prefixlen', 128) == 128 + return self._ip == 1 + + +class IPv6Address(_BaseV6, _BaseAddress): + + """Represent and manipulate single IPv6 Addresses. + """ + + def __init__(self, address): + """Instantiate a new IPv6 address object. + + Args: + address: A string or integer representing the IP + + Additionally, an integer can be passed, so + IPv6Address('2001:db8::') == + IPv6Address(42540766411282592856903984951653826560) + or, more generally + IPv6Address(int(IPv6Address('2001:db8::'))) == + IPv6Address('2001:db8::') + + Raises: + AddressValueError: If address isn't a valid IPv6 address. + + """ + _BaseAddress.__init__(self, address) + _BaseV6.__init__(self, address) + + # Efficient constructor from integer. + if isinstance(address, int): + self._ip = address + if address < 0 or address > self._ALL_ONES: + raise AddressValueError(address) + return + + # Constructing from a packed address + if isinstance(address, bytes) and len(address) == 16: + tmp = struct.unpack('!QQ', address) + self._ip = (tmp[0] << 64) | tmp[1] + return + + # Assume input argument to be string or any object representation + # which converts into a formatted IP string. + addr_str = str(address) + if not addr_str: + raise AddressValueError('') + + self._ip = self._ip_int_from_string(addr_str) + + +class IPv6Interface(IPv6Address): + + def __init__(self, address): + if isinstance(address, (bytes, int)): + IPv6Address.__init__(self, address) + self.network = IPv6Network(self._ip) + self._prefixlen = self._max_prefixlen + return + + addr = str(address).split('/') + IPv6Address.__init__(self, addr[0]) + self.network = IPv6Network(address, strict=False) + self.netmask = self.network.netmask + self._prefixlen = self.network._prefixlen + self.hostmask = self.network.hostmask + + + def __str__(self): + return '%s/%d' % (self._string_from_ip_int(self._ip), + self.network.prefixlen) + + def __eq__(self, other): + try: + return (IPv6Address.__eq__(self, other) and + self.network == other.network) + except AttributeError: + return NotImplemented + + def __hash__(self): + return self._ip ^ self._prefixlen ^ int(self.network.network_address) + + @property + def prefixlen(self): + return self._prefixlen + @property + def ip(self): + return IPv6Address(self._ip) + + @property + def with_prefixlen(self): + return self + + @property + def with_netmask(self): + return self.with_prefixlen + @property + def with_hostmask(self): + return '%s/%s' % (self._string_from_ip_int(self._ip), + self.hostmask) + + +class IPv6Network(_BaseV6, _BaseNetwork): + + """This class represents and manipulates 128-bit IPv6 networks. + + Attributes: [examples for IPv6('2001:db8::1000/124')] + .network_address: IPv6Address('2001:db8::1000') + .hostmask: IPv6Address('::f') + .broadcast_address: IPv6Address('2001:db8::100f') + .netmask: IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff:ffff:fff0') + .prefixlen: 124 + + """ + + def __init__(self, address, strict=True): + """Instantiate a new IPv6 Network object. + + Args: + address: A string or integer representing the IPv6 network or the IP + and prefix/netmask. + '2001:db8::/128' + '2001:db8:0000:0000:0000:0000:0000:0000/128' + '2001:db8::' + are all functionally the same in IPv6. That is to say, + failing to provide a subnetmask will create an object with + a mask of /128. + + Additionally, an integer can be passed, so + IPv6Network('2001:db8::') == + IPv6Network(42540766411282592856903984951653826560) + or, more generally + IPv6Network(int(IPv6Network('2001:db8::'))) == + IPv6Network('2001:db8::') + + strict: A boolean. If true, ensure that we have been passed + A true network address, eg, 2001:db8::1000/124 and not an + IP address on a network, eg, 2001:db8::1/124. + + Raises: + AddressValueError: If address isn't a valid IPv6 address. + NetmaskValueError: If the netmask isn't valid for + an IPv6 address. + ValueError: If strict was True and a network address was not + supplied. + + """ + _BaseV6.__init__(self, address) + _BaseNetwork.__init__(self, address) + + # Efficient constructor from integer. + if isinstance(address, int): + if address < 0 or address > self._ALL_ONES: + raise AddressValueError(address) + self.network_address = IPv6Address(address) + self._prefixlen = self._max_prefixlen + self.netmask = IPv6Address(self._ALL_ONES) + if strict: + if (IPv6Address(int(self.network_address) & + int(self.netmask)) != self.network_address): + raise ValueError('%s has host bits set' % str(self)) + self.network_address = IPv6Address(int(self.network_address) & + int(self.netmask)) + return + + # Constructing from a packed address + if isinstance(address, bytes) and len(address) == 16: + tmp = struct.unpack('!QQ', address) + self.network_address = IPv6Address((tmp[0] << 64) | tmp[1]) + self._prefixlen = self._max_prefixlen + self.netmask = IPv6Address(self._ALL_ONES) + if strict: + if (IPv6Address(int(self.network_address) & + int(self.netmask)) != self.network_address): + raise ValueError('%s has host bits set' % str(self)) + self.network_address = IPv6Address(int(self.network_address) & + int(self.netmask)) + return + + # Assume input argument to be string or any object representation + # which converts into a formatted IP prefix string. + addr = str(address).split('/') + + if len(addr) > 2: + raise AddressValueError(address) + + self.network_address = IPv6Address(self._ip_int_from_string(addr[0])) + + if len(addr) == 2: + if self._is_valid_netmask(addr[1]): + self._prefixlen = int(addr[1]) + else: + raise NetmaskValueError(addr[1]) + else: + self._prefixlen = self._max_prefixlen + + self.netmask = IPv6Address(self._ip_int_from_prefix(self._prefixlen)) + if strict: + if (IPv6Address(int(self.network_address) & int(self.netmask)) != + self.network_address): + raise ValueError('%s has host bits set' % str(self)) + self.network_address = IPv6Address(int(self.network_address) & + int(self.netmask)) + + if self._prefixlen == (self._max_prefixlen - 1): + self.hosts = self.__iter__ + + def __str__(self): + return '%s/%d' % (str(self.network_address), + self.prefixlen) + + def _is_valid_netmask(self, prefixlen): + """Verify that the netmask/prefixlen is valid. + + Args: + prefixlen: A string, the netmask in prefix length format. + + Returns: + A boolean, True if the prefix represents a valid IPv6 + netmask. + + """ + try: + prefixlen = int(prefixlen) + except ValueError: + return False + return 0 <= prefixlen <= self._max_prefixlen + + @property + def with_netmask(self): + return self.with_prefixlen + + @property + def with_prefixlen(self): + return '%s/%d' % (str(self.network_address), self._prefixlen) + + @property + def with_netmask(self): + return '%s/%s' % (str(self.network_address), str(self.netmask)) + + @property + def with_hostmask(self): + return '%s/%s' % (str(self.network_address), str(self.hostmask)) diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_ipaddress.py @@ -0,0 +1,1142 @@ +#!/usr/bin/python3 +# +# Copyright 2007 Google Inc. +# Licensed to PSF under a Contributor Agreement. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unittest for ipaddressmodule.""" + + +import unittest +import time +import ipaddress + +# Compatibility function to cast str to bytes objects +_cb = lambda bytestr: bytes(bytestr, 'charmap') + +class IpaddrUnitTest(unittest.TestCase): + + def setUp(self): + self.ipv4_address = ipaddress.IPv4Address('1.2.3.4') + self.ipv4_interface = ipaddress.IPv4Interface('1.2.3.4/24') + self.ipv4_network = ipaddress.IPv4Network('1.2.3.0/24') + #self.ipv4_hostmask = ipaddress.IPv4Interface('10.0.0.1/0.255.255.255') + self.ipv6_address = ipaddress.IPv6Interface( + '2001:658:22a:cafe:200:0:0:1') + self.ipv6_interface = ipaddress.IPv6Interface( + '2001:658:22a:cafe:200:0:0:1/64') + self.ipv6_network = ipaddress.IPv6Network('2001:658:22a:cafe::/64') + + def testRepr(self): + self.assertEqual("IPv4Interface('1.2.3.4/32')", + repr(ipaddress.IPv4Interface('1.2.3.4'))) + self.assertEqual("IPv6Interface('::1/128')", + repr(ipaddress.IPv6Interface('::1'))) + + # issue57 + def testAddressIntMath(self): + self.assertEqual(ipaddress.IPv4Address('1.1.1.1') + 255, + ipaddress.IPv4Address('1.1.2.0')) + self.assertEqual(ipaddress.IPv4Address('1.1.1.1') - 256, + ipaddress.IPv4Address('1.1.0.1')) + self.assertEqual(ipaddress.IPv6Address('::1') + (2**16 - 2), + ipaddress.IPv6Address('::ffff')) + self.assertEqual(ipaddress.IPv6Address('::ffff') - (2**16 - 2), + ipaddress.IPv6Address('::1')) + + def testInvalidStrings(self): + def AssertInvalidIP(ip_str): + self.assertRaises(ValueError, ipaddress.ip_address, ip_str) + AssertInvalidIP("") + AssertInvalidIP("016.016.016.016") + AssertInvalidIP("016.016.016") + AssertInvalidIP("016.016") + AssertInvalidIP("016") + AssertInvalidIP("000.000.000.000") + AssertInvalidIP("000") + AssertInvalidIP("0x0a.0x0a.0x0a.0x0a") + AssertInvalidIP("0x0a.0x0a.0x0a") + AssertInvalidIP("0x0a.0x0a") + AssertInvalidIP("0x0a") + AssertInvalidIP("42.42.42.42.42") + AssertInvalidIP("42.42.42") + AssertInvalidIP("42.42") + AssertInvalidIP("42") + AssertInvalidIP("42..42.42") + AssertInvalidIP("42..42.42.42") + AssertInvalidIP("42.42.42.42.") + AssertInvalidIP("42.42.42.42...") + AssertInvalidIP(".42.42.42.42") + AssertInvalidIP("...42.42.42.42") + AssertInvalidIP("42.42.42.-0") + AssertInvalidIP("42.42.42.+0") + AssertInvalidIP(".") + AssertInvalidIP("...") + AssertInvalidIP("bogus") + AssertInvalidIP("bogus.com") + AssertInvalidIP("192.168.0.1.com") + AssertInvalidIP("12345.67899.-54321.-98765") + AssertInvalidIP("257.0.0.0") + AssertInvalidIP("42.42.42.-42") + AssertInvalidIP("3ffe::1.net") + AssertInvalidIP("3ffe::1::1") + AssertInvalidIP("1::2::3::4:5") + AssertInvalidIP("::7:6:5:4:3:2:") + AssertInvalidIP(":6:5:4:3:2:1::") + AssertInvalidIP("2001::db:::1") + AssertInvalidIP("FEDC:9878") + AssertInvalidIP("+1.+2.+3.4") + AssertInvalidIP("1.2.3.4e0") + AssertInvalidIP("::7:6:5:4:3:2:1:0") + AssertInvalidIP("7:6:5:4:3:2:1:0::") + AssertInvalidIP("9:8:7:6:5:4:3::2:1") + AssertInvalidIP("0:1:2:3::4:5:6:7") + AssertInvalidIP("3ffe:0:0:0:0:0:0:0:1") + AssertInvalidIP("3ffe::10000") + AssertInvalidIP("3ffe::goog") + AssertInvalidIP("3ffe::-0") + AssertInvalidIP("3ffe::+0") + AssertInvalidIP("3ffe::-1") + AssertInvalidIP(":") + AssertInvalidIP(":::") + AssertInvalidIP("::1.2.3") + AssertInvalidIP("::1.2.3.4.5") + AssertInvalidIP("::1.2.3.4:") + AssertInvalidIP("1.2.3.4::") + AssertInvalidIP("2001:db8::1:") + AssertInvalidIP(":2001:db8::1") + AssertInvalidIP(":1:2:3:4:5:6:7") + AssertInvalidIP("1:2:3:4:5:6:7:") + AssertInvalidIP(":1:2:3:4:5:6:") + + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Interface, '') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv4Interface, + 'google.com') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv4Interface, + '::1.2.3.4') + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv6Interface, '') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Interface, + 'google.com') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Interface, + '1.2.3.4') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Interface, + 'cafe:cafe::/128/190') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Interface, + '1234:axy::b') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Address, + '1234:axy::b') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Address, + '2001:db8:::1') + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Address, + '2001:888888::1') + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Address(1)._ip_int_from_string, + '1.a.2.3') + self.assertEqual(False, ipaddress.IPv4Interface(1)._is_hostmask( + '1.a.2.3')) + + def testGetNetwork(self): + self.assertEqual(int(self.ipv4_network.network_address), 16909056) + self.assertEqual(str(self.ipv4_network.network_address), '1.2.3.0') + + self.assertEqual(int(self.ipv6_network.network_address), + 42540616829182469433403647294022090752) + self.assertEqual(str(self.ipv6_network.network_address), + '2001:658:22a:cafe::') + self.assertEqual(str(self.ipv6_network.hostmask), + '::ffff:ffff:ffff:ffff') + + def testBadVersionComparison(self): + # These should always raise TypeError + v4addr = ipaddress.ip_address('1.1.1.1') + v4net = ipaddress.ip_network('1.1.1.1') + v6addr = ipaddress.ip_address('::1') + v6net = ipaddress.ip_address('::1') + + self.assertRaises(TypeError, v4addr.__lt__, v6addr) + self.assertRaises(TypeError, v4addr.__gt__, v6addr) + self.assertRaises(TypeError, v4net.__lt__, v6net) + self.assertRaises(TypeError, v4net.__gt__, v6net) + + self.assertRaises(TypeError, v6addr.__lt__, v4addr) + self.assertRaises(TypeError, v6addr.__gt__, v4addr) + self.assertRaises(TypeError, v6net.__lt__, v4net) + self.assertRaises(TypeError, v6net.__gt__, v4net) + + def testMixedTypeComparison(self): + v4addr = ipaddress.ip_address('1.1.1.1') + v4net = ipaddress.ip_network('1.1.1.1/32') + v6addr = ipaddress.ip_address('::1') + v6net = ipaddress.ip_network('::1/128') + + self.assertFalse(v4net.__contains__(v6net)) + self.assertFalse(v6net.__contains__(v4net)) + + self.assertRaises(TypeError, lambda: v4addr < v4net) + self.assertRaises(TypeError, lambda: v4addr > v4net) + self.assertRaises(TypeError, lambda: v4net < v4addr) + self.assertRaises(TypeError, lambda: v4net > v4addr) + + self.assertRaises(TypeError, lambda: v6addr < v6net) + self.assertRaises(TypeError, lambda: v6addr > v6net) + self.assertRaises(TypeError, lambda: v6net < v6addr) + self.assertRaises(TypeError, lambda: v6net > v6addr) + + # with get_mixed_type_key, you can sort addresses and network. + self.assertEqual([v4addr, v4net], + sorted([v4net, v4addr], + key=ipaddress.get_mixed_type_key)) + self.assertEqual([v6addr, v6net], + sorted([v6net, v6addr], + key=ipaddress.get_mixed_type_key)) + + def testIpFromInt(self): + self.assertEqual(self.ipv4_interface._ip, + ipaddress.IPv4Interface(16909060)._ip) + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Interface, 2**32) + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Interface, -1) + + ipv4 = ipaddress.ip_network('1.2.3.4') + ipv6 = ipaddress.ip_network('2001:658:22a:cafe:200:0:0:1') + self.assertEqual(ipv4, ipaddress.ip_network(int(ipv4))) + self.assertEqual(ipv6, ipaddress.ip_network(int(ipv6))) + + v6_int = 42540616829182469433547762482097946625 + self.assertEqual(self.ipv6_interface._ip, + ipaddress.IPv6Interface(v6_int)._ip) + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv6Interface, 2**128) + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv6Interface, -1) + + self.assertEqual(ipaddress.ip_network(self.ipv4_address._ip).version, 4) + self.assertEqual(ipaddress.ip_network(self.ipv6_address._ip).version, 6) + + def testIpFromPacked(self): + ip = ipaddress.ip_network + + self.assertEqual(self.ipv4_interface._ip, + ipaddress.ip_interface(_cb('\x01\x02\x03\x04'))._ip) + self.assertEqual(ip('255.254.253.252'), + ip(_cb('\xff\xfe\xfd\xfc'))) + self.assertRaises(ValueError, ipaddress.ip_network, _cb('\x00' * 3)) + self.assertRaises(ValueError, ipaddress.ip_network, _cb('\x00' * 5)) + self.assertEqual(self.ipv6_interface.ip, + ipaddress.ip_interface( + _cb('\x20\x01\x06\x58\x02\x2a\xca\xfe' + '\x02\x00\x00\x00\x00\x00\x00\x01')).ip) + self.assertEqual(ip('ffff:2:3:4:ffff::'), + ip(_cb('\xff\xff\x00\x02\x00\x03\x00\x04' + + '\xff\xff' + '\x00' * 6))) + self.assertEqual(ip('::'), + ip(_cb('\x00' * 16))) + self.assertRaises(ValueError, ip, _cb('\x00' * 15)) + self.assertRaises(ValueError, ip, _cb('\x00' * 17)) + + def testGetIp(self): + self.assertEqual(int(self.ipv4_interface.ip), 16909060) + self.assertEqual(str(self.ipv4_interface.ip), '1.2.3.4') + + self.assertEqual(int(self.ipv6_interface.ip), + 42540616829182469433547762482097946625) + self.assertEqual(str(self.ipv6_interface.ip), + '2001:658:22a:cafe:200::1') + + def testGetNetmask(self): + self.assertEqual(int(self.ipv4_network.netmask), 4294967040) + self.assertEqual(str(self.ipv4_network.netmask), '255.255.255.0') + self.assertEqual(int(self.ipv6_network.netmask), + 340282366920938463444927863358058659840) + self.assertEqual(self.ipv6_network.prefixlen, 64) + + def testZeroNetmask(self): + ipv4_zero_netmask = ipaddress.IPv4Interface('1.2.3.4/0') + self.assertEqual(int(ipv4_zero_netmask.network.netmask), 0) + self.assertTrue(ipv4_zero_netmask.network._is_valid_netmask( + str(0))) + + ipv6_zero_netmask = ipaddress.IPv6Interface('::1/0') + self.assertEqual(int(ipv6_zero_netmask.network.netmask), 0) + self.assertTrue(ipv6_zero_netmask.network._is_valid_netmask( + str(0))) + + def testGetBroadcast(self): + self.assertEqual(int(self.ipv4_network.broadcast_address), 16909311) + self.assertEqual(str(self.ipv4_network.broadcast_address), '1.2.3.255') + + self.assertEqual(int(self.ipv6_network.broadcast_address), + 42540616829182469451850391367731642367) + self.assertEqual(str(self.ipv6_network.broadcast_address), + '2001:658:22a:cafe:ffff:ffff:ffff:ffff') + + def testGetPrefixlen(self): + self.assertEqual(self.ipv4_interface.prefixlen, 24) + self.assertEqual(self.ipv6_interface.prefixlen, 64) + + def testGetSupernet(self): + self.assertEqual(self.ipv4_network.supernet().prefixlen, 23) + self.assertEqual(str(self.ipv4_network.supernet().network_address), + '1.2.2.0') + self.assertEqual( + ipaddress.IPv4Interface('0.0.0.0/0').network.supernet(), + ipaddress.IPv4Network('0.0.0.0/0')) + + self.assertEqual(self.ipv6_network.supernet().prefixlen, 63) + self.assertEqual(str(self.ipv6_network.supernet().network_address), + '2001:658:22a:cafe::') + self.assertEqual(ipaddress.IPv6Interface('::0/0').network.supernet(), + ipaddress.IPv6Network('::0/0')) + + def testGetSupernet3(self): + self.assertEqual(self.ipv4_network.supernet(3).prefixlen, 21) + self.assertEqual(str(self.ipv4_network.supernet(3).network_address), + '1.2.0.0') + + self.assertEqual(self.ipv6_network.supernet(3).prefixlen, 61) + self.assertEqual(str(self.ipv6_network.supernet(3).network_address), + '2001:658:22a:caf8::') + + def testGetSupernet4(self): + self.assertRaises(ValueError, self.ipv4_network.supernet, + prefixlen_diff=2, new_prefix=1) + self.assertRaises(ValueError, self.ipv4_network.supernet, new_prefix=25) + self.assertEqual(self.ipv4_network.supernet(prefixlen_diff=2), + self.ipv4_network.supernet(new_prefix=22)) + + self.assertRaises(ValueError, self.ipv6_network.supernet, + prefixlen_diff=2, new_prefix=1) + self.assertRaises(ValueError, self.ipv6_network.supernet, new_prefix=65) + self.assertEqual(self.ipv6_network.supernet(prefixlen_diff=2), + self.ipv6_network.supernet(new_prefix=62)) + + def testHosts(self): + self.assertEqual([ipaddress.IPv4Address('2.0.0.0'), + ipaddress.IPv4Address('2.0.0.1')], + list(ipaddress.ip_network('2.0.0.0/31').hosts())) + + def testFancySubnetting(self): + self.assertEqual(sorted(self.ipv4_network.subnets(prefixlen_diff=3)), + sorted(self.ipv4_network.subnets(new_prefix=27))) + self.assertRaises(ValueError, list, + self.ipv4_network.subnets(new_prefix=23)) + self.assertRaises(ValueError, list, + self.ipv4_network.subnets(prefixlen_diff=3, + new_prefix=27)) + self.assertEqual(sorted(self.ipv6_network.subnets(prefixlen_diff=4)), + sorted(self.ipv6_network.subnets(new_prefix=68))) + self.assertRaises(ValueError, list, + self.ipv6_network.subnets(new_prefix=63)) + self.assertRaises(ValueError, list, + self.ipv6_network.subnets(prefixlen_diff=4, + new_prefix=68)) + + def testGetSubnets(self): + self.assertEqual(list(self.ipv4_network.subnets())[0].prefixlen, 25) + self.assertEqual(str(list( + self.ipv4_network.subnets())[0].network_address), + '1.2.3.0') + self.assertEqual(str(list( + self.ipv4_network.subnets())[1].network_address), + '1.2.3.128') + + self.assertEqual(list(self.ipv6_network.subnets())[0].prefixlen, 65) + + def testGetSubnetForSingle32(self): + ip = ipaddress.IPv4Network('1.2.3.4/32') + subnets1 = [str(x) for x in ip.subnets()] + subnets2 = [str(x) for x in ip.subnets(2)] + self.assertEqual(subnets1, ['1.2.3.4/32']) + self.assertEqual(subnets1, subnets2) + + def testGetSubnetForSingle128(self): + ip = ipaddress.IPv6Network('::1/128') + subnets1 = [str(x) for x in ip.subnets()] + subnets2 = [str(x) for x in ip.subnets(2)] + self.assertEqual(subnets1, ['::1/128']) + self.assertEqual(subnets1, subnets2) + + def testSubnet2(self): + ips = [str(x) for x in self.ipv4_network.subnets(2)] + self.assertEqual( + ips, + ['1.2.3.0/26', '1.2.3.64/26', '1.2.3.128/26', '1.2.3.192/26']) + + ipsv6 = [str(x) for x in self.ipv6_network.subnets(2)] + self.assertEqual( + ipsv6, + ['2001:658:22a:cafe::/66', + '2001:658:22a:cafe:4000::/66', + '2001:658:22a:cafe:8000::/66', + '2001:658:22a:cafe:c000::/66']) + + def testSubnetFailsForLargeCidrDiff(self): + self.assertRaises(ValueError, list, + self.ipv4_interface.network.subnets(9)) + self.assertRaises(ValueError, list, + self.ipv4_network.subnets(9)) + self.assertRaises(ValueError, list, + self.ipv6_interface.network.subnets(65)) + self.assertRaises(ValueError, list, + self.ipv6_network.subnets(65)) + + def testSupernetFailsForLargeCidrDiff(self): + self.assertRaises(ValueError, + self.ipv4_interface.network.supernet, 25) + self.assertRaises(ValueError, + self.ipv6_interface.network.supernet, 65) + + def testSubnetFailsForNegativeCidrDiff(self): + self.assertRaises(ValueError, list, + self.ipv4_interface.network.subnets(-1)) + self.assertRaises(ValueError, list, + self.ipv4_network.network.subnets(-1)) + self.assertRaises(ValueError, list, + self.ipv6_interface.network.subnets(-1)) + self.assertRaises(ValueError, list, + self.ipv6_network.subnets(-1)) + + def testGetNum_Addresses(self): + self.assertEqual(self.ipv4_network.num_addresses, 256) + self.assertEqual(list(self.ipv4_network.subnets())[0].num_addresses, 128) + self.assertEqual(self.ipv4_network.supernet().num_addresses, 512) + + self.assertEqual(self.ipv6_network.num_addresses, 18446744073709551616) + self.assertEqual(list(self.ipv6_network.subnets())[0].num_addresses, + 9223372036854775808) + self.assertEqual(self.ipv6_network.supernet().num_addresses, + 36893488147419103232) + + def testContains(self): + self.assertTrue(ipaddress.IPv4Interface('1.2.3.128/25') in + self.ipv4_network) + self.assertFalse(ipaddress.IPv4Interface('1.2.4.1/24') in + self.ipv4_network) + # We can test addresses and string as well. + addr1 = ipaddress.IPv4Address('1.2.3.37') + self.assertTrue(addr1 in self.ipv4_network) + # issue 61, bad network comparison on like-ip'd network objects + # with identical broadcast addresses. + self.assertFalse(ipaddress.IPv4Network('1.1.0.0/16').__contains__( + ipaddress.IPv4Network('1.0.0.0/15'))) + + def testBadAddress(self): + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv4Interface, + 'poop') + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Interface, '1.2.3.256') + + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Interface, + 'poopv6') + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Interface, '1.2.3.4/32/24') + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv4Interface, '10/8') + self.assertRaises(ipaddress.AddressValueError, + ipaddress.IPv6Interface, '10/8') + + + def testBadNetMask(self): + self.assertRaises(ipaddress.NetmaskValueError, + ipaddress.IPv4Interface, '1.2.3.4/') + self.assertRaises(ipaddress.NetmaskValueError, + ipaddress.IPv4Interface, '1.2.3.4/33') + self.assertRaises(ipaddress.NetmaskValueError, + ipaddress.IPv4Interface, '1.2.3.4/254.254.255.256') + self.assertRaises(ipaddress.NetmaskValueError, + ipaddress.IPv4Interface, '1.1.1.1/240.255.0.0') + self.assertRaises(ipaddress.NetmaskValueError, + ipaddress.IPv6Interface, '::1/') + self.assertRaises(ipaddress.NetmaskValueError, + ipaddress.IPv6Interface, '::1/129') + + def testNth(self): + self.assertEqual(str(self.ipv4_network[5]), '1.2.3.5') + self.assertRaises(IndexError, self.ipv4_network.__getitem__, 256) + + self.assertEqual(str(self.ipv6_network[5]), + '2001:658:22a:cafe::5') + + def testGetitem(self): + # http://code.google.com/p/ipaddr-py/issues/detail?id=15 + addr = ipaddress.IPv4Network('172.31.255.128/255.255.255.240') + self.assertEqual(28, addr.prefixlen) + addr_list = list(addr) + self.assertEqual('172.31.255.128', str(addr_list[0])) + self.assertEqual('172.31.255.128', str(addr[0])) + self.assertEqual('172.31.255.143', str(addr_list[-1])) + self.assertEqual('172.31.255.143', str(addr[-1])) + self.assertEqual(addr_list[-1], addr[-1]) + + def testEqual(self): + self.assertTrue(self.ipv4_interface == + ipaddress.IPv4Interface('1.2.3.4/24')) + self.assertFalse(self.ipv4_interface == + ipaddress.IPv4Interface('1.2.3.4/23')) + self.assertFalse(self.ipv4_interface == + ipaddress.IPv6Interface('::1.2.3.4/24')) + self.assertFalse(self.ipv4_interface == '') + self.assertFalse(self.ipv4_interface == []) + self.assertFalse(self.ipv4_interface == 2) + + self.assertTrue(self.ipv6_interface == + ipaddress.IPv6Interface('2001:658:22a:cafe:200::1/64')) + self.assertFalse(self.ipv6_interface == + ipaddress.IPv6Interface('2001:658:22a:cafe:200::1/63')) + self.assertFalse(self.ipv6_interface == + ipaddress.IPv4Interface('1.2.3.4/23')) + self.assertFalse(self.ipv6_interface == '') + self.assertFalse(self.ipv6_interface == []) + self.assertFalse(self.ipv6_interface == 2) + + def testNotEqual(self): + self.assertFalse(self.ipv4_interface != + ipaddress.IPv4Interface('1.2.3.4/24')) + self.assertTrue(self.ipv4_interface != + ipaddress.IPv4Interface('1.2.3.4/23')) + self.assertTrue(self.ipv4_interface != + ipaddress.IPv6Interface('::1.2.3.4/24')) + self.assertTrue(self.ipv4_interface != '') + self.assertTrue(self.ipv4_interface != []) + self.assertTrue(self.ipv4_interface != 2) + + self.assertTrue(self.ipv4_address != + ipaddress.IPv4Address('1.2.3.5')) + self.assertTrue(self.ipv4_address != '') + self.assertTrue(self.ipv4_address != []) + self.assertTrue(self.ipv4_address != 2) + + self.assertFalse(self.ipv6_interface != + ipaddress.IPv6Interface('2001:658:22a:cafe:200::1/64')) + self.assertTrue(self.ipv6_interface != + ipaddress.IPv6Interface('2001:658:22a:cafe:200::1/63')) + self.assertTrue(self.ipv6_interface != + ipaddress.IPv4Interface('1.2.3.4/23')) + self.assertTrue(self.ipv6_interface != '') + self.assertTrue(self.ipv6_interface != []) + self.assertTrue(self.ipv6_interface != 2) + + self.assertTrue(self.ipv6_address != + ipaddress.IPv4Address('1.2.3.4')) + self.assertTrue(self.ipv6_address != '') + self.assertTrue(self.ipv6_address != []) + self.assertTrue(self.ipv6_address != 2) + + def testSlash32Constructor(self): + self.assertEqual(str(ipaddress.IPv4Interface( + '1.2.3.4/255.255.255.255')), '1.2.3.4/32') + + def testSlash128Constructor(self): + self.assertEqual(str(ipaddress.IPv6Interface('::1/128')), + '::1/128') + + def testSlash0Constructor(self): + self.assertEqual(str(ipaddress.IPv4Interface('1.2.3.4/0.0.0.0')), + '1.2.3.4/0') + + def testCollapsing(self): + # test only IP addresses including some duplicates + ip1 = ipaddress.IPv4Address('1.1.1.0') + ip2 = ipaddress.IPv4Address('1.1.1.1') + ip3 = ipaddress.IPv4Address('1.1.1.2') + ip4 = ipaddress.IPv4Address('1.1.1.3') + ip5 = ipaddress.IPv4Address('1.1.1.4') + ip6 = ipaddress.IPv4Address('1.1.1.0') + # check that addreses are subsumed properly. + collapsed = ipaddress.collapse_addresses( + [ip1, ip2, ip3, ip4, ip5, ip6]) + self.assertEqual(list(collapsed), [ipaddress.IPv4Network('1.1.1.0/30'), + ipaddress.IPv4Network('1.1.1.4/32')]) + + # test a mix of IP addresses and networks including some duplicates + ip1 = ipaddress.IPv4Address('1.1.1.0') + ip2 = ipaddress.IPv4Address('1.1.1.1') + ip3 = ipaddress.IPv4Address('1.1.1.2') + ip4 = ipaddress.IPv4Address('1.1.1.3') + #ip5 = ipaddress.IPv4Interface('1.1.1.4/30') + #ip6 = ipaddress.IPv4Interface('1.1.1.4/30') + # check that addreses are subsumed properly. + collapsed = ipaddress.collapse_addresses([ip1, ip2, ip3, ip4]) + self.assertEqual(list(collapsed), [ipaddress.IPv4Network('1.1.1.0/30')]) + + # test only IP networks + ip1 = ipaddress.IPv4Network('1.1.0.0/24') + ip2 = ipaddress.IPv4Network('1.1.1.0/24') + ip3 = ipaddress.IPv4Network('1.1.2.0/24') + ip4 = ipaddress.IPv4Network('1.1.3.0/24') + ip5 = ipaddress.IPv4Network('1.1.4.0/24') + # stored in no particular order b/c we want CollapseAddr to call [].sort + ip6 = ipaddress.IPv4Network('1.1.0.0/22') + # check that addreses are subsumed properly. + collapsed = ipaddress.collapse_addresses([ip1, ip2, ip3, ip4, ip5, + ip6]) + self.assertEqual(list(collapsed), [ipaddress.IPv4Network('1.1.0.0/22'), + ipaddress.IPv4Network('1.1.4.0/24')]) + + # test that two addresses are supernet'ed properly + collapsed = ipaddress.collapse_addresses([ip1, ip2]) + self.assertEqual(list(collapsed), [ipaddress.IPv4Network('1.1.0.0/23')]) + + # test same IP networks + ip_same1 = ip_same2 = ipaddress.IPv4Network('1.1.1.1/32') + self.assertEqual(list(ipaddress.collapse_addresses( + [ip_same1, ip_same2])), + [ip_same1]) + + # test same IP addresses + ip_same1 = ip_same2 = ipaddress.IPv4Address('1.1.1.1') + self.assertEqual(list(ipaddress.collapse_addresses( + [ip_same1, ip_same2])), + [ipaddress.ip_network('1.1.1.1/32')]) + ip1 = ipaddress.IPv6Network('2001::/100') + ip2 = ipaddress.IPv6Network('2001::/120') + ip3 = ipaddress.IPv6Network('2001::/96') + # test that ipv6 addresses are subsumed properly. + collapsed = ipaddress.collapse_addresses([ip1, ip2, ip3]) + self.assertEqual(list(collapsed), [ip3]) + + # the toejam test + ip1 = ipaddress.ip_address('1.1.1.1') + ip2 = ipaddress.ip_address('::1') + self.assertRaises(TypeError, ipaddress.collapse_addresses, + [ip1, ip2]) + + def testSummarizing(self): + #ip = ipaddress.ip_address + #ipnet = ipaddress.ip_network + summarize = ipaddress.summarize_address_range + ip1 = ipaddress.ip_address('1.1.1.0') + ip2 = ipaddress.ip_address('1.1.1.255') + # test a /24 is sumamrized properly + self.assertEqual(list(summarize(ip1, ip2))[0], + ipaddress.ip_network('1.1.1.0/24')) + # test an IPv4 range that isn't on a network byte boundary + ip2 = ipaddress.ip_address('1.1.1.8') + self.assertEqual(list(summarize(ip1, ip2)), + [ipaddress.ip_network('1.1.1.0/29'), + ipaddress.ip_network('1.1.1.8')]) + + ip1 = ipaddress.ip_address('1::') + ip2 = ipaddress.ip_address('1:ffff:ffff:ffff:ffff:ffff:ffff:ffff') + # test a IPv6 is sumamrized properly + self.assertEqual(list(summarize(ip1, ip2))[0], + ipaddress.ip_network('1::/16')) + # test an IPv6 range that isn't on a network byte boundary + ip2 = ipaddress.ip_address('2::') + self.assertEqual(list(summarize(ip1, ip2)), + [ipaddress.ip_network('1::/16'), + ipaddress.ip_network('2::/128')]) + + # test exception raised when first is greater than last + self.assertRaises(ValueError, list, + summarize(ipaddress.ip_address('1.1.1.0'), + ipaddress.ip_address('1.1.0.0'))) + # test exception raised when first and last aren't IP addresses + self.assertRaises(TypeError, list, + summarize(ipaddress.ip_network('1.1.1.0'), + ipaddress.ip_network('1.1.0.0'))) + self.assertRaises(TypeError, list, + summarize(ipaddress.ip_network('1.1.1.0'), + ipaddress.ip_network('1.1.0.0'))) + # test exception raised when first and last are not same version + self.assertRaises(TypeError, list, + summarize(ipaddress.ip_address('::'), + ipaddress.ip_network('1.1.0.0'))) + + def testAddressComparison(self): + self.assertTrue(ipaddress.ip_address('1.1.1.1') <= + ipaddress.ip_address('1.1.1.1')) + self.assertTrue(ipaddress.ip_address('1.1.1.1') <= + ipaddress.ip_address('1.1.1.2')) + self.assertTrue(ipaddress.ip_address('::1') <= + ipaddress.ip_address('::1')) + self.assertTrue(ipaddress.ip_address('::1') <= + ipaddress.ip_address('::2')) + + def testNetworkComparison(self): + # ip1 and ip2 have the same network address + ip1 = ipaddress.IPv4Network('1.1.1.0/24') + ip2 = ipaddress.IPv4Network('1.1.1.1/32') + ip3 = ipaddress.IPv4Network('1.1.2.0/24') + + self.assertTrue(ip1 < ip3) + self.assertTrue(ip3 > ip2) + + #self.assertEqual(ip1.compare_networks(ip2), 0) + #self.assertTrue(ip1._get_networks_key() == ip2._get_networks_key()) + self.assertEqual(ip1.compare_networks(ip3), -1) + self.assertTrue(ip1._get_networks_key() < ip3._get_networks_key()) + + ip1 = ipaddress.IPv6Network('2001:2000::/96') + ip2 = ipaddress.IPv6Network('2001:2001::/96') + ip3 = ipaddress.IPv6Network('2001:ffff:2000::/96') + + self.assertTrue(ip1 < ip3) + self.assertTrue(ip3 > ip2) + self.assertEqual(ip1.compare_networks(ip3), -1) + self.assertTrue(ip1._get_networks_key() < ip3._get_networks_key()) + + # Test comparing different protocols. + # Should always raise a TypeError. + ipv6 = ipaddress.IPv6Interface('::/0') + ipv4 = ipaddress.IPv4Interface('0.0.0.0/0') + self.assertRaises(TypeError, ipv4.__lt__, ipv6) + self.assertRaises(TypeError, ipv4.__gt__, ipv6) + self.assertRaises(TypeError, ipv6.__lt__, ipv4) + self.assertRaises(TypeError, ipv6.__gt__, ipv4) + + # Regression test for issue 19. + ip1 = ipaddress.ip_network('10.1.2.128/25') + self.assertFalse(ip1 < ip1) + self.assertFalse(ip1 > ip1) + ip2 = ipaddress.ip_network('10.1.3.0/24') + self.assertTrue(ip1 < ip2) + self.assertFalse(ip2 < ip1) + self.assertFalse(ip1 > ip2) + self.assertTrue(ip2 > ip1) + ip3 = ipaddress.ip_network('10.1.3.0/25') + self.assertTrue(ip2 < ip3) + self.assertFalse(ip3 < ip2) + self.assertFalse(ip2 > ip3) + self.assertTrue(ip3 > ip2) + + # Regression test for issue 28. + ip1 = ipaddress.ip_network('10.10.10.0/31') + ip2 = ipaddress.ip_network('10.10.10.0') + ip3 = ipaddress.ip_network('10.10.10.2/31') + ip4 = ipaddress.ip_network('10.10.10.2') + sorted = [ip1, ip2, ip3, ip4] + unsorted = [ip2, ip4, ip1, ip3] + unsorted.sort() + self.assertEqual(sorted, unsorted) + unsorted = [ip4, ip1, ip3, ip2] + unsorted.sort() + self.assertEqual(sorted, unsorted) + self.assertRaises(TypeError, ip1.__lt__, + ipaddress.ip_address('10.10.10.0')) + self.assertRaises(TypeError, ip2.__lt__, + ipaddress.ip_address('10.10.10.0')) + + # <=, >= + self.assertTrue(ipaddress.ip_network('1.1.1.1') <= + ipaddress.ip_network('1.1.1.1')) + self.assertTrue(ipaddress.ip_network('1.1.1.1') <= + ipaddress.ip_network('1.1.1.2')) + self.assertFalse(ipaddress.ip_network('1.1.1.2') <= + ipaddress.ip_network('1.1.1.1')) + self.assertTrue(ipaddress.ip_network('::1') <= + ipaddress.ip_network('::1')) + self.assertTrue(ipaddress.ip_network('::1') <= + ipaddress.ip_network('::2')) + self.assertFalse(ipaddress.ip_network('::2') <= + ipaddress.ip_network('::1')) + + def testStrictNetworks(self): + self.assertRaises(ValueError, ipaddress.ip_network, '192.168.1.1/24') + self.assertRaises(ValueError, ipaddress.ip_network, '::1/120') + + def testOverlaps(self): + other = ipaddress.IPv4Network('1.2.3.0/30') + other2 = ipaddress.IPv4Network('1.2.2.0/24') + other3 = ipaddress.IPv4Network('1.2.2.64/26') + self.assertTrue(self.ipv4_network.overlaps(other)) + self.assertFalse(self.ipv4_network.overlaps(other2)) + self.assertTrue(other2.overlaps(other3)) + + def testEmbeddedIpv4(self): + ipv4_string = '192.168.0.1' + ipv4 = ipaddress.IPv4Interface(ipv4_string) + v4compat_ipv6 = ipaddress.IPv6Interface('::%s' % ipv4_string) + self.assertEqual(int(v4compat_ipv6.ip), int(ipv4.ip)) + v4mapped_ipv6 = ipaddress.IPv6Interface('::ffff:%s' % ipv4_string) + self.assertNotEqual(v4mapped_ipv6.ip, ipv4.ip) + self.assertRaises(ipaddress.AddressValueError, ipaddress.IPv6Interface, + '2001:1.1.1.1:1.1.1.1') + + # Issue 67: IPv6 with embedded IPv4 address not recognized. + def testIPv6AddressTooLarge(self): + # RFC4291 2.5.5.2 + self.assertEqual(ipaddress.ip_address('::FFFF:192.0.2.1'), + ipaddress.ip_address('::FFFF:c000:201')) + # RFC4291 2.2 (part 3) x::d.d.d.d + self.assertEqual(ipaddress.ip_address('FFFF::192.0.2.1'), + ipaddress.ip_address('FFFF::c000:201')) + + def testIPVersion(self): + self.assertEqual(self.ipv4_address.version, 4) + self.assertEqual(self.ipv6_address.version, 6) + + def testMaxPrefixLength(self): + self.assertEqual(self.ipv4_interface.max_prefixlen, 32) + self.assertEqual(self.ipv6_interface.max_prefixlen, 128) + + def testPacked(self): + self.assertEqual(self.ipv4_address.packed, + _cb('\x01\x02\x03\x04')) + self.assertEqual(ipaddress.IPv4Interface('255.254.253.252').packed, + _cb('\xff\xfe\xfd\xfc')) + self.assertEqual(self.ipv6_address.packed, + _cb('\x20\x01\x06\x58\x02\x2a\xca\xfe' + '\x02\x00\x00\x00\x00\x00\x00\x01')) + self.assertEqual(ipaddress.IPv6Interface('ffff:2:3:4:ffff::').packed, + _cb('\xff\xff\x00\x02\x00\x03\x00\x04\xff\xff' + + '\x00' * 6)) + self.assertEqual(ipaddress.IPv6Interface('::1:0:0:0:0').packed, + _cb('\x00' * 6 + '\x00\x01' + '\x00' * 8)) + + def testIpStrFromPrefixlen(self): + ipv4 = ipaddress.IPv4Interface('1.2.3.4/24') + self.assertEqual(ipv4._ip_string_from_prefix(), '255.255.255.0') + self.assertEqual(ipv4._ip_string_from_prefix(28), '255.255.255.240') + + def testIpType(self): + ipv4net = ipaddress.ip_network('1.2.3.4') + ipv4addr = ipaddress.ip_address('1.2.3.4') + ipv6net = ipaddress.ip_network('::1.2.3.4') + ipv6addr = ipaddress.ip_address('::1.2.3.4') + self.assertEqual(ipaddress.IPv4Network, type(ipv4net)) + self.assertEqual(ipaddress.IPv4Address, type(ipv4addr)) + self.assertEqual(ipaddress.IPv6Network, type(ipv6net)) + self.assertEqual(ipaddress.IPv6Address, type(ipv6addr)) + + def testReservedIpv4(self): + # test networks + self.assertEqual(True, ipaddress.ip_interface( + '224.1.1.1/31').is_multicast) + self.assertEqual(False, ipaddress.ip_network('240.0.0.0').is_multicast) + + self.assertEqual(True, ipaddress.ip_interface( + '192.168.1.1/17').is_private) + self.assertEqual(False, ipaddress.ip_network('192.169.0.0').is_private) + self.assertEqual(True, ipaddress.ip_network( + '10.255.255.255').is_private) + self.assertEqual(False, ipaddress.ip_network('11.0.0.0').is_private) + self.assertEqual(True, ipaddress.ip_network( + '172.31.255.255').is_private) + self.assertEqual(False, ipaddress.ip_network('172.32.0.0').is_private) + + self.assertEqual(True, + ipaddress.ip_interface( + '169.254.100.200/24').is_link_local) + self.assertEqual(False, + ipaddress.ip_interface( + '169.255.100.200/24').is_link_local) + + self.assertEqual(True, + ipaddress.ip_network( + '127.100.200.254/32').is_loopback) + self.assertEqual(True, ipaddress.ip_network( + '127.42.0.0/16').is_loopback) + self.assertEqual(False, ipaddress.ip_network('128.0.0.0').is_loopback) + + # test addresses + self.assertEqual(True, ipaddress.ip_address('224.1.1.1').is_multicast) + self.assertEqual(False, ipaddress.ip_address('240.0.0.0').is_multicast) + + self.assertEqual(True, ipaddress.ip_address('192.168.1.1').is_private) + self.assertEqual(False, ipaddress.ip_address('192.169.0.0').is_private) + self.assertEqual(True, ipaddress.ip_address( + '10.255.255.255').is_private) + self.assertEqual(False, ipaddress.ip_address('11.0.0.0').is_private) + self.assertEqual(True, ipaddress.ip_address( + '172.31.255.255').is_private) + self.assertEqual(False, ipaddress.ip_address('172.32.0.0').is_private) + + self.assertEqual(True, + ipaddress.ip_address('169.254.100.200').is_link_local) + self.assertEqual(False, + ipaddress.ip_address('169.255.100.200').is_link_local) + + self.assertEqual(True, + ipaddress.ip_address('127.100.200.254').is_loopback) + self.assertEqual(True, ipaddress.ip_address('127.42.0.0').is_loopback) + self.assertEqual(False, ipaddress.ip_address('128.0.0.0').is_loopback) + self.assertEqual(True, ipaddress.ip_network('0.0.0.0').is_unspecified) + + def testReservedIpv6(self): + + self.assertEqual(True, ipaddress.ip_network('ffff::').is_multicast) + self.assertEqual(True, ipaddress.ip_network(2**128-1).is_multicast) + self.assertEqual(True, ipaddress.ip_network('ff00::').is_multicast) + self.assertEqual(False, ipaddress.ip_network('fdff::').is_multicast) + + self.assertEqual(True, ipaddress.ip_network('fecf::').is_site_local) + self.assertEqual(True, ipaddress.ip_network( + 'feff:ffff:ffff:ffff::').is_site_local) + self.assertEqual(False, ipaddress.ip_network( + 'fbf:ffff::').is_site_local) + self.assertEqual(False, ipaddress.ip_network('ff00::').is_site_local) + + self.assertEqual(True, ipaddress.ip_network('fc00::').is_private) + self.assertEqual(True, ipaddress.ip_network( + 'fc00:ffff:ffff:ffff::').is_private) + self.assertEqual(False, ipaddress.ip_network('fbff:ffff::').is_private) + self.assertEqual(False, ipaddress.ip_network('fe00::').is_private) + + self.assertEqual(True, ipaddress.ip_network('fea0::').is_link_local) + self.assertEqual(True, ipaddress.ip_network( + 'febf:ffff::').is_link_local) + self.assertEqual(False, ipaddress.ip_network( + 'fe7f:ffff::').is_link_local) + self.assertEqual(False, ipaddress.ip_network('fec0::').is_link_local) + + self.assertEqual(True, ipaddress.ip_interface('0:0::0:01').is_loopback) + self.assertEqual(False, ipaddress.ip_interface('::1/127').is_loopback) + self.assertEqual(False, ipaddress.ip_network('::').is_loopback) + self.assertEqual(False, ipaddress.ip_network('::2').is_loopback) + + self.assertEqual(True, ipaddress.ip_network('0::0').is_unspecified) + self.assertEqual(False, ipaddress.ip_network('::1').is_unspecified) + self.assertEqual(False, ipaddress.ip_network('::/127').is_unspecified) + + # test addresses + self.assertEqual(True, ipaddress.ip_address('ffff::').is_multicast) + self.assertEqual(True, ipaddress.ip_address(2**128-1).is_multicast) + self.assertEqual(True, ipaddress.ip_address('ff00::').is_multicast) + self.assertEqual(False, ipaddress.ip_address('fdff::').is_multicast) + + self.assertEqual(True, ipaddress.ip_address('fecf::').is_site_local) + self.assertEqual(True, ipaddress.ip_address( + 'feff:ffff:ffff:ffff::').is_site_local) + self.assertEqual(False, ipaddress.ip_address( + 'fbf:ffff::').is_site_local) + self.assertEqual(False, ipaddress.ip_address('ff00::').is_site_local) + + self.assertEqual(True, ipaddress.ip_address('fc00::').is_private) + self.assertEqual(True, ipaddress.ip_address( + 'fc00:ffff:ffff:ffff::').is_private) + self.assertEqual(False, ipaddress.ip_address('fbff:ffff::').is_private) + self.assertEqual(False, ipaddress.ip_address('fe00::').is_private) + + self.assertEqual(True, ipaddress.ip_address('fea0::').is_link_local) + self.assertEqual(True, ipaddress.ip_address( + 'febf:ffff::').is_link_local) + self.assertEqual(False, ipaddress.ip_address( + 'fe7f:ffff::').is_link_local) + self.assertEqual(False, ipaddress.ip_address('fec0::').is_link_local) + + self.assertEqual(True, ipaddress.ip_address('0:0::0:01').is_loopback) + self.assertEqual(True, ipaddress.ip_address('::1').is_loopback) + self.assertEqual(False, ipaddress.ip_address('::2').is_loopback) + + self.assertEqual(True, ipaddress.ip_address('0::0').is_unspecified) + self.assertEqual(False, ipaddress.ip_address('::1').is_unspecified) + + # some generic IETF reserved addresses + self.assertEqual(True, ipaddress.ip_address('100::').is_reserved) + self.assertEqual(True, ipaddress.ip_network('4000::1/128').is_reserved) + + def testIpv4Mapped(self): + self.assertEqual(ipaddress.ip_address('::ffff:192.168.1.1').ipv4_mapped, + ipaddress.ip_address('192.168.1.1')) + self.assertEqual(ipaddress.ip_address('::c0a8:101').ipv4_mapped, None) + self.assertEqual(ipaddress.ip_address('::ffff:c0a8:101').ipv4_mapped, + ipaddress.ip_address('192.168.1.1')) + + def testAddrExclude(self): + addr1 = ipaddress.ip_network('10.1.1.0/24') + addr2 = ipaddress.ip_network('10.1.1.0/26') + addr3 = ipaddress.ip_network('10.2.1.0/24') + addr4 = ipaddress.ip_address('10.1.1.0') + self.assertEqual(sorted(list(addr1.address_exclude(addr2))), + [ipaddress.ip_network('10.1.1.64/26'), + ipaddress.ip_network('10.1.1.128/25')]) + self.assertRaises(ValueError, list, addr1.address_exclude(addr3)) + self.assertRaises(TypeError, list, addr1.address_exclude(addr4)) + self.assertEqual(list(addr1.address_exclude(addr1)), []) + + def testHash(self): + self.assertEqual(hash(ipaddress.ip_network('10.1.1.0/24')), + hash(ipaddress.ip_network('10.1.1.0/24'))) + self.assertEqual(hash(ipaddress.ip_address('10.1.1.0')), + hash(ipaddress.ip_address('10.1.1.0'))) + # i70 + self.assertEqual(hash(ipaddress.ip_address('1.2.3.4')), + hash(ipaddress.ip_address( + int(ipaddress.ip_address('1.2.3.4')._ip)))) + ip1 = ipaddress.ip_address('10.1.1.0') + ip2 = ipaddress.ip_address('1::') + dummy = {} + dummy[self.ipv4_address] = None + dummy[self.ipv6_address] = None + dummy[ip1] = None + dummy[ip2] = None + self.assertTrue(self.ipv4_address in dummy) + self.assertTrue(ip2 in dummy) + + def testCopyConstructor(self): + addr1 = ipaddress.ip_network('10.1.1.0/24') + addr2 = ipaddress.ip_network(addr1) + addr3 = ipaddress.ip_interface('2001:658:22a:cafe:200::1/64') + addr4 = ipaddress.ip_interface(addr3) + addr5 = ipaddress.IPv4Address('1.1.1.1') + addr6 = ipaddress.IPv6Address('2001:658:22a:cafe:200::1') + + self.assertEqual(addr1, addr2) + self.assertEqual(addr3, addr4) + self.assertEqual(addr5, ipaddress.IPv4Address(addr5)) + self.assertEqual(addr6, ipaddress.IPv6Address(addr6)) + + def testCompressIPv6Address(self): + test_addresses = { + '1:2:3:4:5:6:7:8': '1:2:3:4:5:6:7:8/128', + '2001:0:0:4:0:0:0:8': '2001:0:0:4::8/128', + '2001:0:0:4:5:6:7:8': '2001::4:5:6:7:8/128', + '2001:0:3:4:5:6:7:8': '2001:0:3:4:5:6:7:8/128', + '2001:0:3:4:5:6:7:8': '2001:0:3:4:5:6:7:8/128', + '0:0:3:0:0:0:0:ffff': '0:0:3::ffff/128', + '0:0:0:4:0:0:0:ffff': '::4:0:0:0:ffff/128', + '0:0:0:0:5:0:0:ffff': '::5:0:0:ffff/128', + '1:0:0:4:0:0:7:8': '1::4:0:0:7:8/128', + '0:0:0:0:0:0:0:0': '::/128', + '0:0:0:0:0:0:0:0/0': '::/0', + '0:0:0:0:0:0:0:1': '::1/128', + '2001:0658:022a:cafe:0000:0000:0000:0000/66': + '2001:658:22a:cafe::/66', + '::1.2.3.4': '::102:304/128', + '1:2:3:4:5:ffff:1.2.3.4': '1:2:3:4:5:ffff:102:304/128', + '::7:6:5:4:3:2:1': '0:7:6:5:4:3:2:1/128', + '::7:6:5:4:3:2:0': '0:7:6:5:4:3:2:0/128', + '7:6:5:4:3:2:1::': '7:6:5:4:3:2:1:0/128', + '0:6:5:4:3:2:1::': '0:6:5:4:3:2:1:0/128', + } + for uncompressed, compressed in list(test_addresses.items()): + self.assertEqual(compressed, str(ipaddress.IPv6Interface( + uncompressed))) + + def testExplodeShortHandIpStr(self): + addr1 = ipaddress.IPv6Interface('2001::1') + addr2 = ipaddress.IPv6Address('2001:0:5ef5:79fd:0:59d:a0e5:ba1') + addr3 = ipaddress.IPv6Network('2001::/96') + self.assertEqual('2001:0000:0000:0000:0000:0000:0000:0001/128', + addr1.exploded) + self.assertEqual('0000:0000:0000:0000:0000:0000:0000:0001/128', + ipaddress.IPv6Interface('::1/128').exploded) + # issue 77 + self.assertEqual('2001:0000:5ef5:79fd:0000:059d:a0e5:0ba1', + addr2.exploded) + self.assertEqual('2001:0000:0000:0000:0000:0000:0000:0000/96', + addr3.exploded) + + def testIntRepresentation(self): + self.assertEqual(16909060, int(self.ipv4_address)) + self.assertEqual(42540616829182469433547762482097946625, + int(self.ipv6_address)) + + def testHexRepresentation(self): + self.assertEqual(hex(0x1020304), + hex(self.ipv4_address)) + + self.assertEqual(hex(0x20010658022ACAFE0200000000000001), + hex(self.ipv6_address)) + + def testForceVersion(self): + self.assertEqual(ipaddress.ip_network(1).version, 4) + self.assertEqual(ipaddress.ip_network(1, version=6).version, 6) + + def testWithStar(self): + self.assertEqual(str(self.ipv4_interface.with_prefixlen), "1.2.3.4/24") + self.assertEqual(str(self.ipv4_interface.with_netmask), + "1.2.3.4/255.255.255.0") + self.assertEqual(str(self.ipv4_interface.with_hostmask), + "1.2.3.4/0.0.0.255") + + self.assertEqual(str(self.ipv6_interface.with_prefixlen), + '2001:658:22a:cafe:200::1/64') + # rfc3513 sec 2.3 says that ipv6 only uses cidr notation for + # subnets + self.assertEqual(str(self.ipv6_interface.with_netmask), + '2001:658:22a:cafe:200::1/64') + # this probably don't make much sense, but it's included for + # compatibility with ipv4 + self.assertEqual(str(self.ipv6_interface.with_hostmask), + '2001:658:22a:cafe:200::1/::ffff:ffff:ffff:ffff') + + def testNetworkElementCaching(self): + # V4 - make sure we're empty + self.assertFalse('network_address' in self.ipv4_network._cache) + self.assertFalse('broadcast_address' in self.ipv4_network._cache) + self.assertFalse('hostmask' in self.ipv4_network._cache) + + # V4 - populate and test + self.assertEqual(self.ipv4_network.network_address, + ipaddress.IPv4Address('1.2.3.0')) + self.assertEqual(self.ipv4_network.broadcast_address, + ipaddress.IPv4Address('1.2.3.255')) + self.assertEqual(self.ipv4_network.hostmask, + ipaddress.IPv4Address('0.0.0.255')) + + # V4 - check we're cached + self.assertTrue('broadcast_address' in self.ipv4_network._cache) + self.assertTrue('hostmask' in self.ipv4_network._cache) + + # V6 - make sure we're empty + self.assertFalse('broadcast_address' in self.ipv6_network._cache) + self.assertFalse('hostmask' in self.ipv6_network._cache) + + # V6 - populate and test + self.assertEqual(self.ipv6_network.network_address, + ipaddress.IPv6Address('2001:658:22a:cafe::')) + self.assertEqual(self.ipv6_interface.network.network_address, + ipaddress.IPv6Address('2001:658:22a:cafe::')) + + self.assertEqual( + self.ipv6_network.broadcast_address, + ipaddress.IPv6Address('2001:658:22a:cafe:ffff:ffff:ffff:ffff')) + self.assertEqual(self.ipv6_network.hostmask, + ipaddress.IPv6Address('::ffff:ffff:ffff:ffff')) + self.assertEqual( + self.ipv6_interface.network.broadcast_address, + ipaddress.IPv6Address('2001:658:22a:cafe:ffff:ffff:ffff:ffff')) + self.assertEqual(self.ipv6_interface.network.hostmask, + ipaddress.IPv6Address('::ffff:ffff:ffff:ffff')) + + # V6 - check we're cached + self.assertTrue('broadcast_address' in self.ipv6_network._cache) + self.assertTrue('hostmask' in self.ipv6_network._cache) + self.assertTrue('broadcast_address' in self.ipv6_interface.network._cache) + self.assertTrue('hostmask' in self.ipv6_interface.network._cache) + + def testTeredo(self): + # stolen from wikipedia + server = ipaddress.IPv4Address('65.54.227.120') + client = ipaddress.IPv4Address('192.0.2.45') + teredo_addr = '2001:0000:4136:e378:8000:63bf:3fff:fdd2' + self.assertEqual((server, client), + ipaddress.ip_address(teredo_addr).teredo) + bad_addr = '2000::4136:e378:8000:63bf:3fff:fdd2' + self.assertFalse(ipaddress.ip_address(bad_addr).teredo) + bad_addr = '2001:0001:4136:e378:8000:63bf:3fff:fdd2' + self.assertFalse(ipaddress.ip_address(bad_addr).teredo) + + # i77 + teredo_addr = ipaddress.IPv6Address('2001:0:5ef5:79fd:0:59d:a0e5:ba1') + self.assertEqual((ipaddress.IPv4Address('94.245.121.253'), + ipaddress.IPv4Address('95.26.244.94')), + teredo_addr.teredo) + + + def testsixtofour(self): + sixtofouraddr = ipaddress.ip_address('2002:ac1d:2d64::1') + bad_addr = ipaddress.ip_address('2000:ac1d:2d64::1') + self.assertEqual(ipaddress.IPv4Address('172.29.45.100'), + sixtofouraddr.sixtofour) + self.assertFalse(bad_addr.sixtofour) + + +if __name__ == '__main__': + unittest.main() diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -702,6 +702,7 @@ Doug Moen The Dragon De Monsyne Skip Montanaro +Peter Moody Paul Moore Derek Morr James A Morrison diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,8 @@ Library ------- +- PEP 3144, Issue #14814: Added the ipaddress module + - Issue #14426: Correct the Date format in Expires attribute of Set-Cookie Header in Cookie.py. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 13:22:45 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 20 May 2012 13:22:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_an_incorrect_impleme?= =?utf8?q?ntation_detail?= Message-ID: http://hg.python.org/cpython/rev/6195c7397d9e changeset: 77085:6195c7397d9e user: Nick Coghlan date: Sun May 20 21:22:27 2012 +1000 summary: Remove an incorrect implementation detail files: Doc/reference/datamodel.rst | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1262,16 +1262,11 @@ immutable (if the object's hash value changes, it will be in the wrong hash bucket). - User-defined classes have :meth:`__eq__` and :meth:`__hash__` methods by default; with them, all objects compare unequal (except with themselves) and ``x.__hash__()`` returns an appropriate value such that ``x == y`` implies both that ``x is y`` and ``hash(x) == hash(y)``. - .. impl-detail:: - - CPython uses ``hash(id(x))`` as the default hash for class instances. - Classes which inherit a :meth:`__hash__` method from a parent class but change the meaning of :meth:`__eq__` such that the hash value returned is no longer appropriate (e.g. by switching to a value-based concept of equality -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 13:27:16 2012 From: python-checkins at python.org (kristjan.jonsson) Date: Sun, 20 May 2012 13:27:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_14821=3A?= Message-ID: http://hg.python.org/cpython/rev/2e96629c6dab changeset: 77086:2e96629c6dab user: Kristj?n Valur J?nsson date: Sun May 20 11:25:48 2012 +0000 summary: Issue 14821: If a dependency is expressed both in the .sln file and the .vcxproj file msbuild is confused. Removing the .sln dependencies which are deprecated anyway. See: http://blogs.msdn.com/b/visualstudio/archive/2010/12/21/incorrect-solution-build-ordering-when-using-msbuild-exe.aspx files: PCbuild/pcbuild.sln | 12 ------------ 1 files changed, 0 insertions(+), 12 deletions(-) diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln --- a/PCbuild/pcbuild.sln +++ b/PCbuild/pcbuild.sln @@ -21,14 +21,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "winsound", "winsound.vcxproj", "{28B5D777-DDF2-4B6B-B34F-31D938813856}" EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_decimal", "_decimal.vcxproj", "{0E9791DB-593A-465F-98BC-681011311617}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes", "_ctypes.vcxproj", "{0E9791DB-593A-465F-98BC-681011311618}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_ctypes_test", "_ctypes_test.vcxproj", "{9EC7190A-249F-4180-A900-548FDCF3055F}" EndProject @@ -71,14 +65,8 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "python3dll", "python3dll.vcxproj", "{885D4898-D08D-4091-9C40-C700CFE3FC5A}" EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "xxlimited", "xxlimited.vcxproj", "{F749B822-B489-4CA5-A3AD-CE078F5F338A}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection EndProject Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_testbuffer", "_testbuffer.vcxproj", "{A2697BD3-28C1-4AEC-9106-8B748639FD16}" - ProjectSection(ProjectDependencies) = postProject - {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} - EndProjectSection EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 16:37:40 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 20 May 2012 16:37:40 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogRml4ZXMgIzE0ODY0?= =?utf8?q?=3A_Added_documentation_on_how_to_undo_the_effects_of_a?= Message-ID: http://hg.python.org/cpython/rev/4973c90ce9e6 changeset: 77087:4973c90ce9e6 branch: 2.7 parent: 77081:aad7cf67b9ea user: Vinay Sajip date: Sun May 20 15:35:00 2012 +0100 summary: Fixes #14864: Added documentation on how to undo the effects of a logging.disable() call. files: Doc/library/logging.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -814,7 +814,8 @@ effect is to disable all logging calls of severity *lvl* and below, so that if you call it with a value of INFO, then all INFO and DEBUG events would be discarded, whereas those of severity WARNING and above would be processed - according to the logger's effective level. + according to the logger's effective level. To undo the effect of a call to + ``logging.disable(lvl)``, call ``logging.disable(logging.NOTSET)``. .. function:: addLevelName(lvl, levelName) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 16:37:41 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 20 May 2012 16:37:41 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogRml4ZXMgIzE0ODY0?= =?utf8?q?=3A_Added_documentation_on_how_to_undo_the_effects_of_a?= Message-ID: http://hg.python.org/cpython/rev/c30170a168b3 changeset: 77088:c30170a168b3 branch: 3.2 parent: 77082:b2dce31d6aec user: Vinay Sajip date: Sun May 20 15:36:17 2012 +0100 summary: Fixes #14864: Added documentation on how to undo the effects of a logging.disable() call. files: Doc/library/logging.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -942,7 +942,8 @@ effect is to disable all logging calls of severity *lvl* and below, so that if you call it with a value of INFO, then all INFO and DEBUG events would be discarded, whereas those of severity WARNING and above would be processed - according to the logger's effective level. + according to the logger's effective level. To undo the effect of a call to + ``logging.disable(lvl)``, call ``logging.disable(logging.NOTSET)``. .. function:: addLevelName(lvl, levelName) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 16:37:42 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 20 May 2012 16:37:42 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Closes_=2314864=3A_Added_documentation_on_how_to_undo_the_ef?= =?utf8?q?fects_of_a?= Message-ID: http://hg.python.org/cpython/rev/76445d7e613f changeset: 77089:76445d7e613f parent: 77086:2e96629c6dab parent: 77088:c30170a168b3 user: Vinay Sajip date: Sun May 20 15:37:33 2012 +0100 summary: Closes #14864: Added documentation on how to undo the effects of a logging.disable() call. Thanks to user Guillaume for the suggestion. files: Doc/library/logging.rst | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -962,7 +962,8 @@ effect is to disable all logging calls of severity *lvl* and below, so that if you call it with a value of INFO, then all INFO and DEBUG events would be discarded, whereas those of severity WARNING and above would be processed - according to the logger's effective level. + according to the logger's effective level. To undo the effect of a call to + ``logging.disable(lvl)``, call ``logging.disable(logging.NOTSET)``. .. function:: addLevelName(lvl, levelName) -- Repository URL: http://hg.python.org/cpython From hs at ox.cx Sun May 20 18:27:06 2012 From: hs at ox.cx (Hynek Schlawack) Date: Sun, 20 May 2012 18:27:06 +0200 Subject: [Python-checkins] cpython (3.2): Document when json.load's parse_constant behaviour changed In-Reply-To: <4FB911D1.1070800@udel.edu> References: <4FB911D1.1070800@udel.edu> Message-ID: <4FB91B5A.2060701@ox.cx> Hello Terry, > The Python 3 docs document Python 3. Any version notices indicate when > the change occurred in Python 3 after 3.0. Notices about Python 2 are > not useful to someone writing multi-version Python 3 code. I would guess > that the above should be 3.2, but it could theoretically have been 3.1. Thanks for pointing that out. I?ve just checked it and 3.1 has the new behavior already (3.0 doesn?t). Corrections are in the works. Cheers, Hynek From python-checkins at python.org Sun May 20 18:36:48 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sun, 20 May 2012 18:36:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_=60versionc?= =?utf8?q?hanged=60_tags_for_json=2Eload?= Message-ID: http://hg.python.org/cpython/rev/1bee3da9a305 changeset: 77090:1bee3da9a305 branch: 3.2 parent: 77088:c30170a168b3 user: Hynek Schlawack date: Sun May 20 18:32:53 2012 +0200 summary: Fix `versionchanged` tags for json.load `versionchanged` tags about 2.7 are useless in 3.x branches. files: Doc/library/json.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -213,7 +213,7 @@ This can be used to raise an exception if invalid JSON numbers are encountered. - .. versionchanged:: 2.7 + .. versionchanged:: 3.1 *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 20 18:36:49 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sun, 20 May 2012 18:36:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Fix_=60versionchanged=60_tags_for_json=2Eload?= Message-ID: http://hg.python.org/cpython/rev/e0f997a7aaa5 changeset: 77091:e0f997a7aaa5 parent: 77089:76445d7e613f parent: 77090:1bee3da9a305 user: Hynek Schlawack date: Sun May 20 18:34:11 2012 +0200 summary: Fix `versionchanged` tags for json.load `versionchanged` tags about 2.7 are useless in 3.x branches. files: Doc/library/json.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/json.rst b/Doc/library/json.rst --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -213,7 +213,7 @@ This can be used to raise an exception if invalid JSON numbers are encountered. - .. versionchanged:: 2.7 + .. versionchanged:: 3.1 *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. To use a custom :class:`JSONDecoder` subclass, specify it with the ``cls`` -- Repository URL: http://hg.python.org/cpython From tjreedy at udel.edu Sun May 20 17:46:25 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 20 May 2012 11:46:25 -0400 Subject: [Python-checkins] cpython (3.2): Document when json.load's parse_constant behaviour changed In-Reply-To: References: Message-ID: <4FB911D1.1070800@udel.edu> On 5/20/2012 6:09 AM, hynek.schlawack wrote: > http://hg.python.org/cpython/rev/b2dce31d6aec > changeset: 77082:b2dce31d6aec > branch: 3.2 > parent: 77075:f9d31d6977f1 > user: Hynek Schlawack > date: Sun May 20 12:03:17 2012 +0200 > summary: > Document when json.load's parse_constant behaviour changed > + .. versionchanged:: 2.7 > + *parse_constant* doesn't get called on 'null', 'true', 'false' anymore. The Python 3 docs document Python 3. Any version notices indicate when the change occurred in Python 3 after 3.0. Notices about Python 2 are not useful to someone writing multi-version Python 3 code. I would guess that the above should be 3.2, but it could theoretically have been 3.1. tjr From tjreedy at udel.edu Sun May 20 19:26:22 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 20 May 2012 13:26:22 -0400 Subject: [Python-checkins] cpython (merge 3.2 -> default): Fix `versionchanged` tags for json.load In-Reply-To: References: Message-ID: <4FB9293E.7030303@udel.edu> > Fix `versionchanged` tags for json.load > > `versionchanged` tags about 2.7 are useless in 3.x branches. It also works the other way around. 3.1 came out before 2.7, but putting 'version changed 3.1' in the 2.7 docs would not help people writing mulit-version Py 2 code. From python-checkins at python.org Mon May 21 00:00:05 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 21 May 2012 00:00:05 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Peter_Moody_is_now_a_devel?= =?utf8?q?oper=2E?= Message-ID: http://hg.python.org/devguide/rev/0bbe32fc7ec2 changeset: 515:0bbe32fc7ec2 user: Antoine Pitrou date: Sun May 20 23:57:28 2012 +0200 summary: Peter Moody is now a developer. files: developers.rst | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/developers.rst b/developers.rst --- a/developers.rst +++ b/developers.rst @@ -24,6 +24,10 @@ Permissions History ------------------- +- Peter Moody was given push privileges on May 20 2012 by Antoine Pitrou + for authorship and maintenance of the ipaddress module (accepted in PEP + 3144 by Nick Coghlan). + - Hynek Schlawack was given push privileges on May 14 2012 by Antoine Pitrou for general contributions. -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon May 21 01:39:22 2012 From: python-checkins at python.org (eric.smith) Date: Mon, 21 May 2012 01:39:22 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Normalize_=27file_system=27=2E?= Message-ID: http://hg.python.org/peps/rev/48691c487c96 changeset: 4410:48691c487c96 parent: 4402:2ed614d32a5e user: Eric V. Smith date: Thu May 17 10:31:43 2012 -0400 summary: Normalize 'file system'. files: pep-0420.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -98,7 +98,7 @@ an ``__init__.py`` file, so long as each portion correctly initializes the namespace package. However, Linux distribution vendors (amongst others) prefer to combine the separate portions and install them all -into the *same* filesystem directory. This creates a potential for +into the *same* file system directory. This creates a potential for conflict, as the portions are now attempting to provide the *same* file on the target system - something that is not allowed by many package managers. Allowing implicit namespace packages means that the @@ -308,7 +308,7 @@ 2. Implicit package directories pose awkward backwards compatibility challenges. - 3. Implicit package directories introduce ambiguity into filesystem + 3. Implicit package directories introduce ambiguity into file system layouts. 4. Implicit package directories will permanently entrench current -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 21 01:39:23 2012 From: python-checkins at python.org (eric.smith) Date: Mon, 21 May 2012 01:39:23 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Corrections_from_Guido=2E?= Message-ID: http://hg.python.org/peps/rev/512bae470453 changeset: 4411:512bae470453 user: Eric V. Smith date: Sun May 20 19:39:00 2012 -0400 summary: Corrections from Guido. files: pep-0420.txt | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -85,7 +85,7 @@ Rationale ========= -The current imperative approach to namespace packages has lead to +The current imperative approach to namespace packages has led to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports ``*.pkg`` files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and @@ -128,7 +128,9 @@ imported and returned. * If not, but ``/foo.{py,pyc,so,pyd}`` is found, a module - is imported and returned. + is imported and returned. The exact list of extension varies by + platform and whether the -O flag is specified. The list here is + representative. * If not, but ``/foo`` is found and is a directory, it is recorded and the scan continues with the next directory in the @@ -314,7 +316,8 @@ 4. Implicit package directories will permanently entrench current newbie-hostile behavior in ``__main__``. -Nick gave a detailed response [5]_, which is summarized here: +Nick later gave a detailed response to his own objections[5]_, which +is summarized here: 1. The practicality of this PEP wins over other proposals and the status quo. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 21 01:39:23 2012 From: python-checkins at python.org (eric.smith) Date: Mon, 21 May 2012 01:39:23 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merge=2E?= Message-ID: http://hg.python.org/peps/rev/6432585f38ed changeset: 4412:6432585f38ed parent: 4411:512bae470453 parent: 4409:f3358939e05e user: Eric V. Smith date: Sun May 20 19:39:16 2012 -0400 summary: Merge. files: pep-0001.txt | 18 +++--- pep-0405.txt | 101 +++++++++++++++++++++----------------- 2 files changed, 66 insertions(+), 53 deletions(-) diff --git a/pep-0001.txt b/pep-0001.txt --- a/pep-0001.txt +++ b/pep-0001.txt @@ -161,14 +161,16 @@ PEP Review & Resolution ----------------------- -Once the authors have completed a PEP, they must inform the PEP editors -that it is ready for review. PEPs are reviewed by the BDFL and his -chosen consultants, who may accept or reject a PEP or send it back to -the author(s) for revision. For a PEP that is predetermined to be -acceptable (e.g., it is an obvious win as-is and/or its implementation -has already been checked in) the BDFL may also initiate a PEP review, -first notifying the PEP author(s) and giving them a chance to make -revisions. +Once the authors have completed a PEP, they may request a review for +style and consistency from the PEP editors. However, the content and +final acceptance of the PEP must be requested of the BDFL, usually via +an email to the python-dev mailing list. PEPs are reviewed by the +BDFL and his chosen consultants, who may accept or reject a PEP or +send it back to the author(s) for revision. For a PEP that is +predetermined to be acceptable (e.g., it is an obvious win as-is +and/or its implementation has already been checked in) the BDFL may +also initiate a PEP review, first notifying the PEP author(s) and +giving them a chance to make revisions. The final authority for PEP approval is the BDFL. However, whenever a new PEP is put forward, any core developer that believes they are suitably diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -222,6 +222,22 @@ those files when Python is run from the venv. +Sysconfig install schemes and user-site +--------------------------------------- + +This approach explicitly chooses not to introduce a new sysconfig +install scheme for venvs. Rather, by modifying ``sys.prefix`` we +ensure that existing install schemes which base locations on +``sys.prefix`` will simply work in a venv. Installation to other +install schemes (for instance, the user-site schemes) whose paths are +not relative to ``sys.prefix``, will not be affected by a venv at all. + +It may be feasible to create an alternative implementation of Python +virtual environments based on a virtual-specific sysconfig scheme, but +it would be less robust, as it would require more code to be aware of +whether it is operating within a virtual environment or not. + + Copies versus symlinks ---------------------- @@ -258,6 +274,45 @@ venv. +Include files +------------- + +Current virtualenv handles include files in this way: + +On POSIX systems where the installed Python's include files are found +in ``${base_prefix}/include/pythonX.X``, virtualenv creates +``${venv}/include/`` and symlink ``${base_prefix}/include/pythonX.X`` +to ``${venv}/include/pythonX.X``. On Windows, where Python's include +files are found in ``{{ sys.prefix }}/Include`` and symlinks are not +reliably available, virtualenv copies ``{{ sys.prefix }}/Include`` to +``${venv}/Include``. This ensures that extension modules built and +installed within the virtualenv will always find the Python header +files they need in the expected location relative to ``sys.prefix``. + +This solution is not ideal when an extension module installs its own +header files, as the default installation location for those header +files may be a symlink to a system directory that may not be +writable. One installer, pip, explicitly works around this by +installing header files to a nonstandard location +``${venv}/include/site/pythonX.X/``, as in Python there's currently no +standard abstraction for a site-specific include directory. + +This PEP proposes a slightly different approach, though one with +essentially the same effect and the same set of advantages and +disadvantages. Rather than symlinking or copying include files into +the venv, we simply modify the sysconfig schemes so that header files +are always looked for relative to ``base_prefix`` rather than +``prefix``. (We also create an ``include/`` directory within the venv + +Better handling of include files in distutils/packaging and, by +extension, pyvenv, is an area that may deserve its own future PEP. For +now, we propose that the behavior of virtualenv has thus far proved +itself to be at least "good enough" in practice. + +[Open question: should pyvenv instead simply copy the behavior of +virtualenv entirely, to avoid introducing unexpected new issues here?] + + API --- @@ -449,53 +504,6 @@ locating and parsing the ``pyvenv.cfg`` file, if it is present. -Open Questions -============== - -What about include files? -------------------------- - -For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in -``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default -in ``$VE/include/pythonX.Y``. With virtualenv, everything works -because the PythonX.Y include is symlinked, so everything that's -needed is in ``$VE/include``. At the moment the reference -implementation doesn't do anything with include files, besides -creating the include directory; this might need to change, to -copy/symlink ``$VE/include/pythonX.Y``. - -As in Python there's no abstraction for a site-specific include -directory, other than for platform-specific stuff, then the user -expectation would seem to be that all include files anyone could ever -want should be found in one of just two locations, with sysconfig -labels "include" & "platinclude". - -There's another issue: what if includes are Python-version-specific? -For example, SIP installs by default into ``$VE/include/pythonX.Y`` -rather than ``$VE/include``, presumably because there's -version-specific stuff in there - but even if that's not the case with -SIP, it could be the case with some other package. And the problem -that gives is that you can't just symlink the ``include/pythonX.Y`` -directory, but actually have to provide a writable directory and -symlink/copy the contents from the system ``include/pythonX.Y``. Of -course this is not hard to do, but it does seem inelegant. OTOH it's -really because there's no supporting concept in ``Python/sysconfig``. - - -OS X Framework builds ---------------------- - -There have been some reports that the reference implementation does -not work on an OS X framework build of Python, but it seems to work -for us. This needs further investigation. - - -tkinter -------- - -Tkinter apps currently do not work within a virtual environment. - - Reference Implementation ======================== @@ -505,7 +513,7 @@ installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create a virtual environment. -.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv +.. _a clone of the CPython Mercurial repository: http://hg.python.org/sandbox/vsajip#venv Copyright -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Mon May 21 05:51:37 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 21 May 2012 05:51:37 +0200 Subject: [Python-checkins] Daily reference leaks (e0f997a7aaa5): sum=0 Message-ID: results for e0f997a7aaa5 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogkZRvm4', '-x'] From python-checkins at python.org Mon May 21 10:33:59 2012 From: python-checkins at python.org (hynek.schlawack) Date: Mon, 21 May 2012 10:33:59 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_myself_as_a_shutil_=22?= =?utf8?q?expert=22?= Message-ID: http://hg.python.org/devguide/rev/295ba819f3cb changeset: 516:295ba819f3cb user: Hynek Schlawack date: Mon May 21 10:33:36 2012 +0200 summary: Add myself as a shutil "expert" files: experts.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -191,7 +191,7 @@ select shelve shlex -shutil tarek +shutil tarek, hynek signal site smtpd giampaolo.rodola -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon May 21 13:43:14 2012 From: python-checkins at python.org (hynek.schlawack) Date: Mon, 21 May 2012 13:43:14 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0ODA0OiBSZW1v?= =?utf8?q?ve_=5B=5D_around_optional_arguments_with_default_values?= Message-ID: http://hg.python.org/cpython/rev/d13fdd97cc8e changeset: 77092:d13fdd97cc8e branch: 3.2 parent: 77090:1bee3da9a305 user: Hynek Schlawack date: Mon May 21 11:01:54 2012 +0200 summary: #14804: Remove [] around optional arguments with default values files: Doc/library/curses.rst | 2 +- Doc/library/email.generator.rst | 2 +- Doc/library/http.cookiejar.rst | 2 +- Doc/library/nis.rst | 6 +++--- Doc/library/os.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/select.rst | 4 ++-- Doc/library/sqlite3.rst | 2 +- 8 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -1091,7 +1091,7 @@ rendition (as set by :meth:`bkgdset`) merged into them. -.. method:: window.scroll([lines=1]) +.. method:: window.scroll(lines=1) Scroll the screen or scrolling region upward by *lines* lines. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -178,7 +178,7 @@ representing the part. -.. class:: DecodedGenerator(outfp[, mangle_from_=True, maxheaderlen=78, fmt=None) +.. class:: DecodedGenerator(outfp, mangle_from_=True, maxheaderlen=78, fmt=None) This class, derived from :class:`Generator` walks through all the subparts of a message. If the subpart is of main type :mimetype:`text`, then it prints the diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -700,7 +700,7 @@ The :class:`Cookie` class also defines the following method: -.. method:: Cookie.is_expired([now=None]) +.. method:: Cookie.is_expired(now=None) True if cookie has passed the time at which the server requested it should expire. If *now* is given (in seconds since the epoch), return whether the diff --git a/Doc/library/nis.rst b/Doc/library/nis.rst --- a/Doc/library/nis.rst +++ b/Doc/library/nis.rst @@ -17,7 +17,7 @@ The :mod:`nis` module defines the following functions: -.. function:: match(key, mapname[, domain=default_domain]) +.. function:: match(key, mapname, domain=default_domain) Return the match for *key* in map *mapname*, or raise an error (:exc:`nis.error`) if there is none. Both should be strings, *key* is 8-bit @@ -30,7 +30,7 @@ unspecified, lookup is in the default NIS domain. -.. function:: cat(mapname[, domain=default_domain]) +.. function:: cat(mapname, domain=default_domain) Return a dictionary mapping *key* to *value* such that ``match(key, mapname)==value``. Note that both keys and values of the dictionary are @@ -42,7 +42,7 @@ unspecified, lookup is in the default NIS domain. -.. function:: maps([domain=default_domain]) +.. function:: maps(domain=default_domain) Return a list of all valid maps. diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1145,7 +1145,7 @@ Availability: Unix. -.. function:: mknod(filename[, mode=0o600[, device]]) +.. function:: mknod(filename, mode=0o600, device=0) Create a filesystem node (file, device special file or named pipe) named *filename*. *mode* specifies both the permissions to use and the type of node diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -277,7 +277,7 @@ simple calculations. -.. method:: oss_audio_device.setparameters(format, nchannels, samplerate [, strict=False]) +.. method:: oss_audio_device.setparameters(format, nchannels, samplerate, strict=False) Set the key audio sampling parameters---sample format, number of channels, and sampling rate---in one method call. *format*, *nchannels*, and *samplerate* diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -181,7 +181,7 @@ Remove a registered file descriptor from the epoll object. -.. method:: epoll.poll([timeout=-1[, maxevents=-1]]) +.. method:: epoll.poll(timeout=-1, maxevents=-1) Wait for events. timeout in seconds (float) @@ -285,7 +285,7 @@ Create a kqueue object from a given file descriptor. -.. method:: kqueue.control(changelist, max_events[, timeout=None]) -> eventlist +.. method:: kqueue.control(changelist, max_events, timeout=None) -> eventlist Low level interface to kevent diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -514,7 +514,7 @@ or :const:`None` when no more data is available. -.. method:: Cursor.fetchmany([size=cursor.arraysize]) +.. method:: Cursor.fetchmany(size=cursor.arraysize) Fetches the next set of rows of a query result, returning a list. An empty list is returned when no more rows are available. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 13:43:15 2012 From: python-checkins at python.org (hynek.schlawack) Date: Mon, 21 May 2012 13:43:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314804=3A_Remove_=5B=5D_around_optional_arguments_with_def?= =?utf8?q?ault_values?= Message-ID: http://hg.python.org/cpython/rev/2293eba03348 changeset: 77093:2293eba03348 parent: 77091:e0f997a7aaa5 parent: 77092:d13fdd97cc8e user: Hynek Schlawack date: Mon May 21 13:35:03 2012 +0200 summary: #14804: Remove [] around optional arguments with default values files: Doc/library/curses.rst | 2 +- Doc/library/email.generator.rst | 2 +- Doc/library/http.cookiejar.rst | 2 +- Doc/library/nis.rst | 6 +++--- Doc/library/os.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/select.rst | 4 ++-- Doc/library/sqlite3.rst | 2 +- 8 files changed, 11 insertions(+), 11 deletions(-) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -1120,7 +1120,7 @@ rendition (as set by :meth:`bkgdset`) merged into them. -.. method:: window.scroll([lines=1]) +.. method:: window.scroll(lines=1) Scroll the screen or scrolling region upward by *lines* lines. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -197,7 +197,7 @@ representing the part. -.. class:: DecodedGenerator(outfp[, mangle_from_=True, maxheaderlen=78, fmt=None) +.. class:: DecodedGenerator(outfp, mangle_from_=True, maxheaderlen=78, fmt=None) This class, derived from :class:`Generator` walks through all the subparts of a message. If the subpart is of main type :mimetype:`text`, then it prints the diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -707,7 +707,7 @@ The :class:`Cookie` class also defines the following method: -.. method:: Cookie.is_expired([now=None]) +.. method:: Cookie.is_expired(now=None) True if cookie has passed the time at which the server requested it should expire. If *now* is given (in seconds since the epoch), return whether the diff --git a/Doc/library/nis.rst b/Doc/library/nis.rst --- a/Doc/library/nis.rst +++ b/Doc/library/nis.rst @@ -17,7 +17,7 @@ The :mod:`nis` module defines the following functions: -.. function:: match(key, mapname[, domain=default_domain]) +.. function:: match(key, mapname, domain=default_domain) Return the match for *key* in map *mapname*, or raise an error (:exc:`nis.error`) if there is none. Both should be strings, *key* is 8-bit @@ -30,7 +30,7 @@ unspecified, lookup is in the default NIS domain. -.. function:: cat(mapname[, domain=default_domain]) +.. function:: cat(mapname, domain=default_domain) Return a dictionary mapping *key* to *value* such that ``match(key, mapname)==value``. Note that both keys and values of the dictionary are @@ -42,7 +42,7 @@ unspecified, lookup is in the default NIS domain. -.. function:: maps([domain=default_domain]) +.. function:: maps(domain=default_domain) Return a list of all valid maps. diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1786,7 +1786,7 @@ Availability: Unix. -.. function:: mknod(filename[, mode=0o600[, device]]) +.. function:: mknod(filename, mode=0o600, device=0) Create a filesystem node (file, device special file or named pipe) named *filename*. *mode* specifies both the permissions to use and the type of node diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -281,7 +281,7 @@ simple calculations. -.. method:: oss_audio_device.setparameters(format, nchannels, samplerate [, strict=False]) +.. method:: oss_audio_device.setparameters(format, nchannels, samplerate, strict=False) Set the key audio sampling parameters---sample format, number of channels, and sampling rate---in one method call. *format*, *nchannels*, and *samplerate* diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -267,7 +267,7 @@ Remove a registered file descriptor from the epoll object. -.. method:: epoll.poll([timeout=-1[, maxevents=-1]]) +.. method:: epoll.poll(timeout=-1, maxevents=-1) Wait for events. timeout in seconds (float) @@ -371,7 +371,7 @@ Create a kqueue object from a given file descriptor. -.. method:: kqueue.control(changelist, max_events[, timeout=None]) -> eventlist +.. method:: kqueue.control(changelist, max_events, timeout=None) -> eventlist Low level interface to kevent diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -526,7 +526,7 @@ or :const:`None` when no more data is available. -.. method:: Cursor.fetchmany([size=cursor.arraysize]) +.. method:: Cursor.fetchmany(size=cursor.arraysize) Fetches the next set of rows of a query result, returning a list. An empty list is returned when no more rows are available. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 13:43:16 2012 From: python-checkins at python.org (hynek.schlawack) Date: Mon, 21 May 2012 13:43:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314804=3A_Remove_=5B=5D_a?= =?utf8?q?round_optional_arguments_with_default_values?= Message-ID: http://hg.python.org/cpython/rev/8b7cb7e4ed8b changeset: 77094:8b7cb7e4ed8b user: Hynek Schlawack date: Mon May 21 13:41:25 2012 +0200 summary: #14804: Remove [] around optional arguments with default values 3.3 specific additions to d13fdd97cc8e. files: Doc/library/packaging.compiler.rst | 32 +++++++------- Doc/library/packaging.fancy_getopt.rst | 6 +- Doc/library/shutil.rst | 10 ++-- 3 files changed, 24 insertions(+), 24 deletions(-) diff --git a/Doc/library/packaging.compiler.rst b/Doc/library/packaging.compiler.rst --- a/Doc/library/packaging.compiler.rst +++ b/Doc/library/packaging.compiler.rst @@ -255,7 +255,7 @@ that the runtime linker may search by default. - .. method:: CCompiler.define_macro(name[, value=None]) + .. method:: CCompiler.define_macro(name, value=None) Define a preprocessor macro for all compilations driven by this compiler object. The optional parameter *value* should be a string; if it is not @@ -298,7 +298,7 @@ (a list) to do the job. - .. method:: CCompiler.find_library_file(dirs, lib[, debug=0]) + .. method:: CCompiler.find_library_file(dirs, lib, debug=0) Search the specified list of directories for a static or shared library file *lib* and return the full path to that file. If *debug* is true, look for a @@ -306,7 +306,7 @@ ``None`` if *lib* wasn't found in any of the specified directories. - .. method:: CCompiler.has_function(funcname [, includes=None, include_dirs=None, libraries=None, library_dirs=None]) + .. method:: CCompiler.has_function(funcname, includes=None, include_dirs=None, libraries=None, library_dirs=None) Return a boolean indicating whether *funcname* is supported on the current platform. The optional arguments can be used to augment the compilation @@ -361,7 +361,7 @@ The following methods invoke stages in the build process. - .. method:: CCompiler.compile(sources[, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None]) + .. method:: CCompiler.compile(sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None) Compile one or more source files. Generates object files (e.g. transforms a :file:`.c` file to a :file:`.o` file.) @@ -405,7 +405,7 @@ Raises :exc:`CompileError` on failure. - .. method:: CCompiler.create_static_lib(objects, output_libname[, output_dir=None, debug=0, target_lang=None]) + .. method:: CCompiler.create_static_lib(objects, output_libname, output_dir=None, debug=0, target_lang=None) Link a bunch of stuff together to create a static library file. The "bunch of stuff" consists of the list of object files supplied as *objects*, the extra @@ -427,7 +427,7 @@ Raises :exc:`LibError` on failure. - .. method:: CCompiler.link(target_desc, objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + .. method:: CCompiler.link(target_desc, objects, output_filename, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None) Link a bunch of stuff together to create an executable or shared library file. @@ -469,28 +469,28 @@ Raises :exc:`LinkError` on failure. - .. method:: CCompiler.link_executable(objects, output_progname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None]) + .. method:: CCompiler.link_executable(objects, output_progname, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None) Link an executable. *output_progname* is the name of the file executable, while *objects* are a list of object filenames to link in. Other arguments are as for the :meth:`link` method. - .. method:: CCompiler.link_shared_lib(objects, output_libname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + .. method:: CCompiler.link_shared_lib(objects, output_libname, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None) Link a shared library. *output_libname* is the name of the output library, while *objects* is a list of object filenames to link in. Other arguments are as for the :meth:`link` method. - .. method:: CCompiler.link_shared_object(objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) + .. method:: CCompiler.link_shared_object(objects, output_filename, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None) Link a shared object. *output_filename* is the name of the shared object that will be created, while *objects* is a list of object filenames to link in. Other arguments are as for the :meth:`link` method. - .. method:: CCompiler.preprocess(source[, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None]) + .. method:: CCompiler.preprocess(source, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None) Preprocess a single C/C++ source file, named in *source*. Output will be written to file named *output_file*, or *stdout* if *output_file* not supplied. @@ -505,14 +505,14 @@ use by the various concrete subclasses. - .. method:: CCompiler.executable_filename(basename[, strip_dir=0, output_dir='']) + .. method:: CCompiler.executable_filename(basename, strip_dir=0, output_dir='') Returns the filename of the executable for the given *basename*. Typically for non-Windows platforms this is the same as the basename, while Windows will get a :file:`.exe` added. - .. method:: CCompiler.library_filename(libname[, lib_type='static', strip_dir=0, output_dir='']) + .. method:: CCompiler.library_filename(libname, lib_type='static', strip_dir=0, output_dir='') Returns the filename for the given library name on the current platform. On Unix a library with *lib_type* of ``'static'`` will typically be of the form @@ -520,18 +520,18 @@ :file:`liblibname.so`. - .. method:: CCompiler.object_filenames(source_filenames[, strip_dir=0, output_dir='']) + .. method:: CCompiler.object_filenames(source_filenames, strip_dir=0, output_dir='') Returns the name of the object files for the given source files. *source_filenames* should be a list of filenames. - .. method:: CCompiler.shared_object_filename(basename[, strip_dir=0, output_dir='']) + .. method:: CCompiler.shared_object_filename(basename, strip_dir=0, output_dir='') Returns the name of a shared object file for the given file name *basename*. - .. method:: CCompiler.execute(func, args[, msg=None, level=1]) + .. method:: CCompiler.execute(func, args, msg=None, level=1) Invokes :func:`packaging.util.execute` This method invokes a Python function *func* with the given arguments *args*, after logging and taking into account @@ -544,7 +544,7 @@ the given command. XXX see also. - .. method:: CCompiler.mkpath(name[, mode=511]) + .. method:: CCompiler.mkpath(name, mode=511) Invokes :func:`packaging.dir_util.mkpath`. This creates a directory and any missing ancestor directories. XXX see also. diff --git a/Doc/library/packaging.fancy_getopt.rst b/Doc/library/packaging.fancy_getopt.rst --- a/Doc/library/packaging.fancy_getopt.rst +++ b/Doc/library/packaging.fancy_getopt.rst @@ -33,7 +33,7 @@ ``sys.argv[1:]`` if you pass ``None`` as *args*. -.. class:: FancyGetopt([option_table=None]) +.. class:: FancyGetopt(option_table=None) The option_table is a list of 3-tuples: ``(long_option, short_option, help_string)`` @@ -46,7 +46,7 @@ The :class:`FancyGetopt` class provides the following methods: -.. method:: FancyGetopt.getopt([args=None, object=None]) +.. method:: FancyGetopt.getopt(args=None, object=None) Parse command-line options in args. Store as attributes on *object*. @@ -67,7 +67,7 @@ yet. -.. method:: FancyGetopt.generate_help([header=None]) +.. method:: FancyGetopt.generate_help(header=None) Generate help text (a list of strings, one per suggested line of output) from the option table for this :class:`FancyGetopt` object. diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -47,7 +47,7 @@ be copied. -.. function:: copyfile(src, dst[, symlinks=False]) +.. function:: copyfile(src, dst, symlinks=False) Copy the contents (no metadata) of the file named *src* to a file named *dst*. *dst* must be the complete target file name; look at @@ -67,7 +67,7 @@ Added *symlinks* argument. -.. function:: copymode(src, dst[, symlinks=False]) +.. function:: copymode(src, dst, symlinks=False) Copy the permission bits from *src* to *dst*. The file contents, owner, and group are unaffected. *src* and *dst* are path names given as strings. If @@ -78,7 +78,7 @@ .. versionchanged:: 3.3 Added *symlinks* argument. -.. function:: copystat(src, dst[, symlinks=False]) +.. function:: copystat(src, dst, symlinks=False) Copy the permission bits, last access time, last modification time, and flags from *src* to *dst*. The file contents, owner, and group are unaffected. *src* @@ -89,7 +89,7 @@ .. versionchanged:: 3.3 Added *symlinks* argument. -.. function:: copy(src, dst[, symlinks=False])) +.. function:: copy(src, dst, symlinks=False)) Copy the file *src* to the file or directory *dst*. If *dst* is a directory, a file with the same basename as *src* is created (or overwritten) in the @@ -100,7 +100,7 @@ .. versionchanged:: 3.3 Added *symlinks* argument. -.. function:: copy2(src, dst[, symlinks=False]) +.. function:: copy2(src, dst, symlinks=False) Similar to :func:`shutil.copy`, but metadata is copied as well. This is similar to the Unix command :program:`cp -p`. If *symlinks* is true, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 14:54:57 2012 From: python-checkins at python.org (nick.coghlan) Date: Mon, 21 May 2012 14:54:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2313585=3A_add_conte?= =?utf8?q?xtlib=2EExitStack_to_replace_the_ill-fated?= Message-ID: http://hg.python.org/cpython/rev/8ef66c73b1e1 changeset: 77095:8ef66c73b1e1 user: Nick Coghlan date: Mon May 21 22:54:43 2012 +1000 summary: Close #13585: add contextlib.ExitStack to replace the ill-fated contextlib.nested API files: Doc/library/contextlib.rst | 279 +++++++++++++++++++++++- Doc/whatsnew/3.3.rst | 15 + Lib/contextlib.py | 126 ++++++++++- Lib/test/test_contextlib.py | 123 ++++++++++ Misc/NEWS | 2 + 5 files changed, 539 insertions(+), 6 deletions(-) diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -12,8 +12,11 @@ statement. For more information see also :ref:`typecontextmanager` and :ref:`context-managers`. -Functions provided: +Utilities +--------- + +Functions and classes provided: .. decorator:: contextmanager @@ -168,6 +171,280 @@ .. versionadded:: 3.2 +.. class:: ExitStack() + + A context manager that is designed to make it easy to programmatically + combine other context managers and cleanup functions, especially those + that are optional or otherwise driven by input data. + + For example, a set of files may easily be handled in a single with + statement as follows:: + + with ExitStack() as stack: + files = [stack.enter_context(open(fname)) for fname in filenames] + # All opened files will automatically be closed at the end of + # the with statement, even if attempts to open files later + # in the list throw an exception + + Each instance maintains a stack of registered callbacks that are called in + reverse order when the instance is closed (either explicitly or implicitly + at the end of a ``with`` statement). Note that callbacks are *not* invoked + implicitly when the context stack instance is garbage collected. + + This stack model is used so that context managers that acquire their + resources in their ``__init__`` method (such as file objects) can be + handled correctly. + + Since registered callbacks are invoked in the reverse order of + registration, this ends up behaving as if multiple nested ``with`` + statements had been used with the registered set of callbacks. This even + extends to exception handling - if an inner callback suppresses or replaces + an exception, then outer callbacks will be passed arguments based on that + updated state. + + This is a relatively low level API that takes care of the details of + correctly unwinding the stack of exit callbacks. It provides a suitable + foundation for higher level context managers that manipulate the exit + stack in application specific ways. + + .. method:: enter_context(cm) + + Enters a new context manager and adds its :meth:`__exit__` method to + the callback stack. The return value is the result of the context + manager's own :meth:`__enter__` method. + + These context managers may suppress exceptions just as they normally + would if used directly as part of a ``with`` statement. + + .. method:: push(exit) + + Adds a context manager's :meth:`__exit__` method to the callback stack. + + As ``__enter__`` is *not* invoked, this method can be used to cover + part of an :meth:`__enter__` implementation with a context manager's own + :meth:`__exit__` method. + + If passed an object that is not a context manager, this method assumes + it is a callback with the same signature as a context manager's + :meth:`__exit__` method and adds it directly to the callback stack. + + By returning true values, these callbacks can suppress exceptions the + same way context manager :meth:`__exit__` methods can. + + The passed in object is returned from the function, allowing this + method to be used is a function decorator. + + .. method:: callback(callback, *args, **kwds) + + Accepts an arbitrary callback function and arguments and adds it to + the callback stack. + + Unlike the other methods, callbacks added this way cannot suppress + exceptions (as they are never passed the exception details). + + The passed in callback is returned from the function, allowing this + method to be used is a function decorator. + + .. method:: pop_all() + + Transfers the callback stack to a fresh :class:`ExitStack` instance + and returns it. No callbacks are invoked by this operation - instead, + they will now be invoked when the new stack is closed (either + explicitly or implicitly). + + For example, a group of files can be opened as an "all or nothing" + operation as follows:: + + with ExitStack() as stack: + files = [stack.enter_context(open(fname)) for fname in filenames] + close_files = stack.pop_all().close + # If opening any file fails, all previously opened files will be + # closed automatically. If all files are opened successfully, + # they will remain open even after the with statement ends. + # close_files() can then be invoked explicitly to close them all + + .. method:: close() + + Immediately unwinds the callback stack, invoking callbacks in the + reverse order of registration. For any context managers and exit + callbacks registered, the arguments passed in will indicate that no + exception occurred. + + .. versionadded:: 3.3 + + +Examples and Recipes +-------------------- + +This section describes some examples and recipes for making effective use of +the tools provided by :mod:`contextlib`. + + +Cleaning up in an ``__enter__`` implementation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted in the documentation of :meth:`ExitStack.push`, this +method can be useful in cleaning up an already allocated resource if later +steps in the :meth:`__enter__` implementation fail. + +Here's an example of doing this for a context manager that accepts resource +acquisition and release functions, along with an optional validation function, +and maps them to the context management protocol:: + + from contextlib import contextmanager, ExitStack + + class ResourceManager(object): + + def __init__(self, acquire_resource, release_resource, check_resource_ok=None): + self.acquire_resource = acquire_resource + self.release_resource = release_resource + if check_resource_ok is None: + def check_resource_ok(resource): + return True + self.check_resource_ok = check_resource_ok + + @contextmanager + def _cleanup_on_error(self): + with ExitStack() as stack: + stack.push(self) + yield + # The validation check passed and didn't raise an exception + # Accordingly, we want to keep the resource, and pass it + # back to our caller + stack.pop_all() + + def __enter__(self): + resource = self.acquire_resource() + with self._cleanup_on_error(): + if not self.check_resource_ok(resource): + msg = "Failed validation for {!r}" + raise RuntimeError(msg.format(resource)) + return resource + + def __exit__(self, *exc_details): + # We don't need to duplicate any of our resource release logic + self.release_resource() + + +Replacing any use of ``try-finally`` and flag variables +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A pattern you will sometimes see is a ``try-finally`` statement with a flag +variable to indicate whether or not the body of the ``finally`` clause should +be executed. In its simplest form (that can't already be handled just by +using an ``except`` clause instead), it looks something like this:: + + cleanup_needed = True + try: + result = perform_operation() + if result: + cleanup_needed = False + finally: + if cleanup_needed: + cleanup_resources() + +As with any ``try`` statement based code, this can cause problems for +development and review, because the setup code and the cleanup code can end +up being separated by arbitrarily long sections of code. + +:class:`ExitStack` makes it possible to instead register a callback for +execution at the end of a ``with`` statement, and then later decide to skip +executing that callback:: + + from contextlib import ExitStack + + with ExitStack() as stack: + stack.callback(cleanup_resources) + result = perform_operation() + if result: + stack.pop_all() + +This allows the intended cleanup up behaviour to be made explicit up front, +rather than requiring a separate flag variable. + +If a particular application uses this pattern a lot, it can be simplified +even further by means of a small helper class:: + + from contextlib import ExitStack + + class Callback(ExitStack): + def __init__(self, callback, *args, **kwds): + super(Callback, self).__init__() + self.callback(callback, *args, **kwds) + + def cancel(self): + self.pop_all() + + with Callback(cleanup_resources) as cb: + result = perform_operation() + if result: + cb.cancel() + +If the resource cleanup isn't already neatly bundled into a standalone +function, then it is still possible to use the decorator form of +:meth:`ExitStack.callback` to declare the resource cleanup in +advance:: + + from contextlib import ExitStack + + with ExitStack() as stack: + @stack.callback + def cleanup_resources(): + ... + result = perform_operation() + if result: + stack.pop_all() + +Due to the way the decorator protocol works, a callback function +declared this way cannot take any parameters. Instead, any resources to +be released must be accessed as closure variables + + +Using a context manager as a function decorator +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:class:`ContextDecorator` makes it possible to use a context manager in +both an ordinary ``with`` statement and also as a function decorator. + +For example, it is sometimes useful to wrap functions or groups of statements +with a logger that can track the time of entry and time of exit. Rather than +writing both a function decorator and a context manager for the task, +inheriting from :class:`ContextDecorator` provides both capabilities in a +single definition:: + + from contextlib import ContextDecorator + import logging + + logging.basicConfig(level=logging.INFO) + + class track_entry_and_exit(ContextDecorator): + def __init__(self, name): + self.name = name + + def __enter__(self): + logging.info('Entering: {}'.format(name)) + + def __exit__(self, exc_type, exc, exc_tb): + logging.info('Exiting: {}'.format(name)) + +Instances of this class can be used as both a context manager:: + + with track_entry_and_exit('widget loader'): + print('Some time consuming activity goes here') + load_widget() + +And also as a function decorator:: + + @track_entry_and_exit('widget loader') + def activity(): + print('Some time consuming activity goes here') + load_widget() + +Note that there is one additional limitation when using context managers +as function decorators: there's no way to access the return value of +:meth:`__enter__`. If that value is needed, then it is still necessary to use +an explicit ``with`` statement. + .. seealso:: :pep:`0343` - The "with" statement diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -696,6 +696,21 @@ .. XXX addition of __slots__ to ABCs not recorded here: internal detail +contextlib +---------- + +:class:`~collections.ExitStack` now provides a solid foundation for +programmatic manipulation of context managers and similar cleanup +functionality. Unlike the previous ``contextlib.nested`` API (which was +deprecated and removed), the new API is designed to work correctly +regardless of whether context managers acquire their resources in +their ``__init`` method (for example, file objects) or in their +``__enter__`` method (for example, synchronisation objects from the +:mod:`threading` module). + +(:issue:`13585`) + + crypt ----- diff --git a/Lib/contextlib.py b/Lib/contextlib.py --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -1,9 +1,10 @@ """Utilities for with-statement contexts. See PEP 343.""" import sys +from collections import deque from functools import wraps -__all__ = ["contextmanager", "closing", "ContextDecorator"] +__all__ = ["contextmanager", "closing", "ContextDecorator", "ExitStack"] class ContextDecorator(object): @@ -12,12 +13,12 @@ def _recreate_cm(self): """Return a recreated instance of self. - Allows otherwise one-shot context managers like + Allows an otherwise one-shot context manager like _GeneratorContextManager to support use as - decorators via implicit recreation. + a decorator via implicit recreation. - Note: this is a private interface just for _GCM in 3.2 but will be - renamed and documented for third party use in 3.3 + This is a private interface just for _GeneratorContextManager. + See issue #11647 for details. """ return self @@ -138,3 +139,118 @@ return self.thing def __exit__(self, *exc_info): self.thing.close() + + +# Inspired by discussions on http://bugs.python.org/issue13585 +class ExitStack(object): + """Context manager for dynamic management of a stack of exit callbacks + + For example: + + with ExitStack() as stack: + files = [stack.enter_context(open(fname)) for fname in filenames] + # All opened files will automatically be closed at the end of + # the with statement, even if attempts to open files later + # in the list throw an exception + + """ + def __init__(self): + self._exit_callbacks = deque() + + def pop_all(self): + """Preserve the context stack by transferring it to a new instance""" + new_stack = type(self)() + new_stack._exit_callbacks = self._exit_callbacks + self._exit_callbacks = deque() + return new_stack + + def _push_cm_exit(self, cm, cm_exit): + """Helper to correctly register callbacks to __exit__ methods""" + def _exit_wrapper(*exc_details): + return cm_exit(cm, *exc_details) + _exit_wrapper.__self__ = cm + self.push(_exit_wrapper) + + def push(self, exit): + """Registers a callback with the standard __exit__ method signature + + Can suppress exceptions the same way __exit__ methods can. + + Also accepts any object with an __exit__ method (registering a call + to the method instead of the object itself) + """ + # We use an unbound method rather than a bound method to follow + # the standard lookup behaviour for special methods + _cb_type = type(exit) + try: + exit_method = _cb_type.__exit__ + except AttributeError: + # Not a context manager, so assume its a callable + self._exit_callbacks.append(exit) + else: + self._push_cm_exit(exit, exit_method) + return exit # Allow use as a decorator + + def callback(self, callback, *args, **kwds): + """Registers an arbitrary callback and arguments. + + Cannot suppress exceptions. + """ + def _exit_wrapper(exc_type, exc, tb): + callback(*args, **kwds) + # We changed the signature, so using @wraps is not appropriate, but + # setting __wrapped__ may still help with introspection + _exit_wrapper.__wrapped__ = callback + self.push(_exit_wrapper) + return callback # Allow use as a decorator + + def enter_context(self, cm): + """Enters the supplied context manager + + If successful, also pushes its __exit__ method as a callback and + returns the result of the __enter__ method. + """ + # We look up the special methods on the type to match the with statement + _cm_type = type(cm) + _exit = _cm_type.__exit__ + result = _cm_type.__enter__(cm) + self._push_cm_exit(cm, _exit) + return result + + def close(self): + """Immediately unwind the context stack""" + self.__exit__(None, None, None) + + def __enter__(self): + return self + + def __exit__(self, *exc_details): + if not self._exit_callbacks: + return + # This looks complicated, but it is really just + # setting up a chain of try-expect statements to ensure + # that outer callbacks still get invoked even if an + # inner one throws an exception + def _invoke_next_callback(exc_details): + # Callbacks are removed from the list in FIFO order + # but the recursion means they're invoked in LIFO order + cb = self._exit_callbacks.popleft() + if not self._exit_callbacks: + # Innermost callback is invoked directly + return cb(*exc_details) + # More callbacks left, so descend another level in the stack + try: + suppress_exc = _invoke_next_callback(exc_details) + except: + suppress_exc = cb(*sys.exc_info()) + # Check if this cb suppressed the inner exception + if not suppress_exc: + raise + else: + # Check if inner cb suppressed the original exception + if suppress_exc: + exc_details = (None, None, None) + suppress_exc = cb(*exc_details) or suppress_exc + return suppress_exc + # Kick off the recursive chain + return _invoke_next_callback(exc_details) diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -370,6 +370,129 @@ self.assertEqual(state, [1, 'something else', 999]) +class TestExitStack(unittest.TestCase): + + def test_no_resources(self): + with ExitStack(): + pass + + def test_callback(self): + expected = [ + ((), {}), + ((1,), {}), + ((1,2), {}), + ((), dict(example=1)), + ((1,), dict(example=1)), + ((1,2), dict(example=1)), + ] + result = [] + def _exit(*args, **kwds): + """Test metadata propagation""" + result.append((args, kwds)) + with ExitStack() as stack: + for args, kwds in reversed(expected): + if args and kwds: + f = stack.callback(_exit, *args, **kwds) + elif args: + f = stack.callback(_exit, *args) + elif kwds: + f = stack.callback(_exit, **kwds) + else: + f = stack.callback(_exit) + self.assertIs(f, _exit) + for wrapper in stack._exit_callbacks: + self.assertIs(wrapper.__wrapped__, _exit) + self.assertNotEqual(wrapper.__name__, _exit.__name__) + self.assertIsNone(wrapper.__doc__, _exit.__doc__) + self.assertEqual(result, expected) + + def test_push(self): + exc_raised = ZeroDivisionError + def _expect_exc(exc_type, exc, exc_tb): + self.assertIs(exc_type, exc_raised) + def _suppress_exc(*exc_details): + return True + def _expect_ok(exc_type, exc, exc_tb): + self.assertIsNone(exc_type) + self.assertIsNone(exc) + self.assertIsNone(exc_tb) + class ExitCM(object): + def __init__(self, check_exc): + self.check_exc = check_exc + def __enter__(self): + self.fail("Should not be called!") + def __exit__(self, *exc_details): + self.check_exc(*exc_details) + with ExitStack() as stack: + stack.push(_expect_ok) + self.assertIs(stack._exit_callbacks[-1], _expect_ok) + cm = ExitCM(_expect_ok) + stack.push(cm) + self.assertIs(stack._exit_callbacks[-1].__self__, cm) + stack.push(_suppress_exc) + self.assertIs(stack._exit_callbacks[-1], _suppress_exc) + cm = ExitCM(_expect_exc) + stack.push(cm) + self.assertIs(stack._exit_callbacks[-1].__self__, cm) + stack.push(_expect_exc) + self.assertIs(stack._exit_callbacks[-1], _expect_exc) + stack.push(_expect_exc) + self.assertIs(stack._exit_callbacks[-1], _expect_exc) + 1/0 + + def test_enter_context(self): + class TestCM(object): + def __enter__(self): + result.append(1) + def __exit__(self, *exc_details): + result.append(3) + + result = [] + cm = TestCM() + with ExitStack() as stack: + @stack.callback # Registered first => cleaned up last + def _exit(): + result.append(4) + self.assertIsNotNone(_exit) + stack.enter_context(cm) + self.assertIs(stack._exit_callbacks[-1].__self__, cm) + result.append(2) + self.assertEqual(result, [1, 2, 3, 4]) + + def test_close(self): + result = [] + with ExitStack() as stack: + @stack.callback + def _exit(): + result.append(1) + self.assertIsNotNone(_exit) + stack.close() + result.append(2) + self.assertEqual(result, [1, 2]) + + def test_pop_all(self): + result = [] + with ExitStack() as stack: + @stack.callback + def _exit(): + result.append(3) + self.assertIsNotNone(_exit) + new_stack = stack.pop_all() + result.append(1) + result.append(2) + new_stack.close() + self.assertEqual(result, [1, 2, 3]) + + def test_instance_bypass(self): + class Example(object): pass + cm = Example() + cm.__exit__ = object() + stack = ExitStack() + self.assertRaises(AttributeError, stack.enter_context, cm) + stack.push(cm) + self.assertIs(stack._exit_callbacks[-1], cm) + + # This is needed to make the test actually run under regrtest.py! def test_main(): support.run_unittest(__name__) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,8 @@ Library ------- +- Issue #13585: Added contextlib.ExitStack + - PEP 3144, Issue #14814: Added the ipaddress module - Issue #14426: Correct the Date format in Expires attribute of Set-Cookie -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 15:03:45 2012 From: python-checkins at python.org (nick.coghlan) Date: Mon, 21 May 2012 15:03:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314136_by_cleaning_?= =?utf8?q?up_the_PEP_409_command_line_test_=28patch_by_Ethan?= Message-ID: http://hg.python.org/cpython/rev/02f13577b6fe changeset: 77096:02f13577b6fe user: Nick Coghlan date: Mon May 21 23:03:30 2012 +1000 summary: Close #14136 by cleaning up the PEP 409 command line test (patch by Ethan Furman) files: Lib/test/test_cmd_line_script.py | 19 +++++++ Lib/test/test_raise.py | 53 +------------------- 2 files changed, 20 insertions(+), 52 deletions(-) diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py --- a/Lib/test/test_cmd_line_script.py +++ b/Lib/test/test_cmd_line_script.py @@ -7,6 +7,7 @@ import os.path import py_compile +import textwrap from test import support from test.script_helper import ( make_pkg, make_script, make_zip_pkg, make_zip_script, @@ -286,6 +287,24 @@ self._check_output(script_name, rc, out, script_name, script_name, '', '') + def test_pep_409_verbiage(self): + # Make sure PEP 409 syntax properly suppresses + # the context of an exception + script = textwrap.dedent("""\ + try: + raise ValueError + except: + raise NameError from None + """) + with temp_dir() as script_dir: + script_name = _make_test_script(script_dir, 'script', script) + exitcode, stdout, stderr = assert_python_failure(script_name) + text = stderr.decode('ascii').split('\n') + self.assertEqual(len(text), 4) + self.assertTrue(text[0].startswith('Traceback')) + self.assertTrue(text[1].startswith(' File ')) + self.assertTrue(text[3].startswith('NameError')) + def test_main(): support.run_unittest(CmdLineTest) support.reap_children() diff --git a/Lib/test/test_raise.py b/Lib/test/test_raise.py --- a/Lib/test/test_raise.py +++ b/Lib/test/test_raise.py @@ -3,27 +3,13 @@ """Tests for the raise statement.""" -from test import support, script_helper +from test import support import re import sys import types import unittest -try: - from resource import setrlimit, RLIMIT_CORE, error as resource_error -except ImportError: - prepare_subprocess = None -else: - def prepare_subprocess(): - # don't create core file - try: - setrlimit(RLIMIT_CORE, (0, 0)) - except (ValueError, resource_error): - pass - - - def get_tb(): try: raise OSError() @@ -224,43 +210,6 @@ class TestTraceback(unittest.TestCase): - def get_output(self, code, filename=None): - """ - Run the specified code in Python (in a new child process) and read the - output from the standard error or from a file (if filename is set). - Return the output lines as a list. - """ - options = {} - if prepare_subprocess: - options['preexec_fn'] = prepare_subprocess - process = script_helper.spawn_python('-c', code, **options) - stdout, stderr = process.communicate() - exitcode = process.wait() - output = support.strip_python_stderr(stdout) - output = output.decode('ascii', 'backslashreplace') - if filename: - self.assertEqual(output, '') - with open(filename, "rb") as fp: - output = fp.read() - output = output.decode('ascii', 'backslashreplace') - output = re.sub('Current thread 0x[0-9a-f]+', - 'Current thread XXX', - output) - return output.splitlines(), exitcode - - def test_traceback_verbiage(self): - code = """ -try: - raise ValueError -except: - raise NameError from None -""" - text, exitcode = self.get_output(code) - self.assertEqual(len(text), 3) - self.assertTrue(text[0].startswith('Traceback')) - self.assertTrue(text[1].startswith(' File ')) - self.assertTrue(text[2].startswith('NameError')) - def test_sets_traceback(self): try: raise IndexError() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 19:25:39 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 21 May 2012 19:25:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_outdated_statement?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/836a3ef19e34 changeset: 77097:836a3ef19e34 user: Antoine Pitrou date: Mon May 21 19:23:00 2012 +0200 summary: Remove outdated statement. files: Doc/whatsnew/3.3.rst | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -1455,11 +1455,6 @@ .. XXX add a point about hash randomization and that it's always on in 3.3 -* :issue:`14205`: A dict lookup now raises a :exc:`RuntimeError` if the dict is - modified during the lookup. If you implement your own comparison function for - objects used as dict keys and the dict is shared by multiple threads, access - to the dict should be protected by a lock. - * :issue:`12326`: On Linux, sys.platform doesn't contain the major version anymore. It is now always 'linux', instead of 'linux2' or 'linux3' depending on the Linux version used to build Python. Replace sys.platform == 'linux2' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 20:16:42 2012 From: python-checkins at python.org (georg.brandl) Date: Mon, 21 May 2012 20:16:42 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_3144_and_415_are_implement?= =?utf8?b?ZWQu?= Message-ID: http://hg.python.org/peps/rev/0bb29342d3e8 changeset: 4413:0bb29342d3e8 user: Georg Brandl date: Mon May 21 19:51:16 2012 +0200 summary: PEP 3144 and 415 are implemented. files: pep-0398.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -65,9 +65,11 @@ * PEP 409: Suppressing exception context * PEP 412: Key-Sharing Dictionary * PEP 414: Explicit Unicode Literal for Python 3.3 +* PEP 415: Implement context suppression with exception attributes * PEP 417: Including mock in the Standard Library * PEP 418: Add monotonic time, performance counter, and process time functions * PEP 3118: Revising the buffer protocol (protocol semantics finalised) +* PEP 3144: IP Address manipulation library * PEP 3151: Reworking the OS and IO exception hierarchy * PEP 3155: Qualified name for classes and functions @@ -84,11 +86,9 @@ * PEP 362: Function Signature Object * PEP 397: Python launcher for Windows * PEP 405: Python Virtual Environments -* PEP 415: Implementing PEP 409 differently * PEP 420: Implicit Namespace Packages * PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library -* PEP 3144: IP Address manipulation library * PEP 3154: Pickle protocol version 4 (Note that these are not accepted yet and even if they are, they might -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 21 20:28:56 2012 From: python-checkins at python.org (georg.brandl) Date: Mon, 21 May 2012 20:28:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_useless_indentation?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/35c3946657ce changeset: 77098:35c3946657ce user: Georg Brandl date: Mon May 21 20:28:58 2012 +0200 summary: Avoid useless indentation. files: Doc/library/http.client.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst --- a/Doc/library/http.client.rst +++ b/Doc/library/http.client.rst @@ -382,8 +382,8 @@ | | | :rfc:`6585`, Section 6 | +------------------------------------------+---------+-----------------------------------------------------------------------+ - .. versionchanged:: 3.3 - Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. +.. versionchanged:: 3.3 + Added codes ``428``, ``429``, ``431`` and ``511`` from :rfc:`6585`. .. data:: responses -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 21 20:31:45 2012 From: python-checkins at python.org (georg.brandl) Date: Mon, 21 May 2012 20:31:45 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_More_interest_areas=2E?= Message-ID: http://hg.python.org/devguide/rev/2802b03060b2 changeset: 517:2802b03060b2 user: Georg Brandl date: Mon May 21 20:31:44 2012 +0200 summary: More interest areas. files: experts.rst | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -299,7 +299,7 @@ bsd bug tracker ezio.melotti buildbots -bytecode pitrou +bytecode pitrou, georg.brandl context managers ncoghlan data formats mark.dickinson, georg.brandl database lemburg @@ -316,12 +316,13 @@ packaging tarek, lemburg, alexis, eric.araujo py3 transition benjamin.peterson release management tarek, lemburg, benjamin.peterson, barry, loewis, - gvanrossum, anthonybaxter, eric.araujo, ned.deily + gvanrossum, anthonybaxter, eric.araujo, ned.deily, + georg.brandl str.format eric.smith testing michael.foord, pitrou, ezio.melotti test coverage ncoghlan, giampaolo.rodola threads pitrou time and dates lemburg, belopolsky unicode lemburg, ezio.melotti, haypo, benjamin.peterson -version control eric.araujo +version control eric.araujo, georg.brandl ================== =========== -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Mon May 21 21:56:02 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 21:56:02 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Factor_out_repeated_code?= =?utf8?q?_used_to_call_pysetup_from_tests=2E?= Message-ID: http://hg.python.org/distutils2/rev/1c09e0c94dd3 changeset: 1342:1c09e0c94dd3 user: ?ric Araujo date: Sun May 20 00:45:41 2012 -0400 summary: Factor out repeated code used to call pysetup from tests. The whole assert_python_* business was unfriendly for readers, now the code needed to have the equivalent of pysetup run from tests is moved to helper methods and the tests can have much clearer code. Also remove Python 2.4 compatibility (using -c instead of -m). files: distutils2/tests/test_run.py | 42 +++++++++-------------- 1 files changed, 17 insertions(+), 25 deletions(-) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -62,28 +62,32 @@ def get_pythonpath(self): pythonpath = os.environ.get('PYTHONPATH') - d2parent = os.path.dirname(os.path.dirname(__file__)) + d2parent = os.path.dirname(os.path.dirname(__file__)) # XXX buggy if pythonpath is not None: pythonpath = os.pathsep.join((pythonpath, d2parent)) else: pythonpath = d2parent return pythonpath + def call_pysetup(self, *args): + _, out, err = assert_python_ok('-m', 'distutils2.run', *args, + PYTHONPATH=self.get_pythonpath()) + return out, err + + def call_pysetup_fail(self, *args): + _, out, err = assert_python_failure('-m', 'distutils2.run', *args, + PYTHONPATH=self.get_pythonpath()) + return out, err + def test_show_help(self): # smoke test, just makes sure some help is displayed - status, out, err = assert_python_ok( - '-c', 'from distutils2.run import main; main()', '--help', - PYTHONPATH=self.get_pythonpath()) - self.assertEqual(status, 0) + out, err = self.call_pysetup('--help') self.assertGreater(out, '') self.assertEqual(err, '') def test_list_commands(self): - status, out, err = assert_python_ok( - '-c', 'from distutils2.run import main; main()', 'run', - '--list-commands', PYTHONPATH=self.get_pythonpath()) + out, err = self.call_pysetup('run', '--list-commands') # check that something is displayed - self.assertEqual(status, 0) self.assertGreater(out, '') self.assertEqual(err, '') @@ -97,10 +101,7 @@ # TODO test that custom commands don't break --list-commands def test_unknown_command_option(self): - status, out, err = assert_python_failure( - '-c', 'from distutils2.run import main; main()', 'run', 'build', - '--unknown', PYTHONPATH=self.get_pythonpath()) - self.assertEqual(status, 1) + out, err = self.call_pysetup_fail('run', 'build', '--unknown') self.assertGreater(out, '') # sadly this message comes straight from the getopt module and can't be # modified to use repr instead of str for the unknown option; to be @@ -109,28 +110,19 @@ 'error: option --unknown not recognized') def test_invalid_command(self): - status, out, err = assert_python_failure( - '-c', 'from distutils2.run import main; main()', 'run', - 'com#mand', PYTHONPATH=self.get_pythonpath()) - self.assertEqual(status, 1) + out, err = self.call_pysetup_fail('run', 'com#mand') self.assertGreater(out, 1) self.assertEqual(err.splitlines()[-1], "error: invalid command name 'com#mand'") def test_unknown_command(self): - status, out, err = assert_python_failure( - '-c', 'from distutils2.run import main; main()', 'run', - 'invalid_command', PYTHONPATH=self.get_pythonpath()) - self.assertEqual(status, 1) + out, err = self.call_pysetup_fail('run', 'invalid_command') self.assertGreater(out, 1) self.assertEqual(err.splitlines()[-1], "error: command 'invalid_command' not recognized") def test_unknown_action(self): - status, out, err = assert_python_failure( - '-c', 'from distutils2.run import main; main()', 'invalid_action', - PYTHONPATH=self.get_pythonpath()) - self.assertEqual(status, 1) + out, err = self.call_pysetup_fail('invalid_action') self.assertGreater(out, 1) self.assertEqual(err.splitlines()[-1], "error: action 'invalid_action' not recognized") -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 21:56:02 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 21:56:02 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Tighten_tests_for_=23133?= =?utf8?q?99?= Message-ID: http://hg.python.org/distutils2/rev/7d4098ce0087 changeset: 1343:7d4098ce0087 user: ?ric Araujo date: Sun May 20 00:53:08 2012 -0400 summary: Tighten tests for #13399 files: distutils2/tests/test_run.py | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -106,26 +106,26 @@ # sadly this message comes straight from the getopt module and can't be # modified to use repr instead of str for the unknown option; to be # changed when the command line parsers are replaced by something clean - self.assertEqual(err.splitlines()[-1], - 'error: option --unknown not recognized') + self.assertEqual(err.splitlines(), + ['error: option --unknown not recognized']) def test_invalid_command(self): out, err = self.call_pysetup_fail('run', 'com#mand') self.assertGreater(out, 1) - self.assertEqual(err.splitlines()[-1], - "error: invalid command name 'com#mand'") + self.assertEqual(err.splitlines(), + ["error: invalid command name 'com#mand'"]) def test_unknown_command(self): out, err = self.call_pysetup_fail('run', 'invalid_command') self.assertGreater(out, 1) - self.assertEqual(err.splitlines()[-1], - "error: command 'invalid_command' not recognized") + self.assertEqual(err.splitlines(), + ["error: command 'invalid_command' not recognized"]) def test_unknown_action(self): out, err = self.call_pysetup_fail('invalid_action') self.assertGreater(out, 1) - self.assertEqual(err.splitlines()[-1], - "error: action 'invalid_action' not recognized") + self.assertEqual(err.splitlines(), + ["error: action 'invalid_action' not recognized"]) def test_suite(): -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 21:56:02 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 21:56:02 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Have_pysetup_read_setup?= =?utf8?q?=2Ecfg_early_to_find_custom_commands_=28=2314733=29=2E?= Message-ID: http://hg.python.org/distutils2/rev/70831fde0ec4 changeset: 1344:70831fde0ec4 user: ?ric Araujo date: Mon May 21 15:51:11 2012 -0400 summary: Have pysetup read setup.cfg early to find custom commands (#14733). Independently found and fixed by me (with help from Luis Rojas) and Janusz Lewandowski. files: CHANGES.txt | 2 + CONTRIBUTORS.txt | 1 + distutils2/run.py | 23 +++++------- distutils2/tests/test_run.py | 42 +++++++++++++++++++---- 4 files changed, 47 insertions(+), 21 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt --- a/CHANGES.txt +++ b/CHANGES.txt @@ -19,6 +19,8 @@ - #13399: Display error message instead of unhandled traceback for missing actions, commands and options [patrice] - #10374: Recreate scripts everytime build_scripts is called [pierre paul] +- #14733: Have pysetup read setup.cfg early enough to find custom commands + [?ric, janusz] 1.0a4 - 2012-03-13 ------------------ diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -48,6 +48,7 @@ - Pierre Paul Lefebvre - Tshepang Lekhonkhobe - Alain Leufroy +- Janusz Lewandowski - Martin von L?wis - Hugo Lopes Tavares - Guillermo L?pez-Anglada diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -10,6 +10,7 @@ from distutils2.dist import Distribution from distutils2.util import _is_archive_file, generate_setup_py from distutils2.command import get_command_class, STANDARD_COMMANDS +from distutils2.command.cmd import Command from distutils2.install import install, install_local_project, remove from distutils2.database import get_distribution, get_distributions from distutils2.depgraph import generate_graph @@ -254,6 +255,13 @@ parser = dispatcher.parser args = args[1:] + # Find and parse the config file(s): they will override options from + # the setup script, but be overridden by the command line. + # XXX call the functions from config and kill the Distribution class + # (merging it into Dispatcher) + dist = Distribution() + dist.parse_config_files() + commands = STANDARD_COMMANDS # FIXME display extra commands if args == ['--list-commands']: @@ -269,16 +277,6 @@ if args is None: return - # create the Distribution class - # need to feed setup.cfg here ! - dist = Distribution() - - # Find and parse the config file(s): they will override options from - # the setup script, but be overridden by the command line. - - # XXX still need to be extracted from Distribution - dist.parse_config_files() - for cmd in dispatcher.commands: # FIXME need to catch MetadataMissingError here (from the check command # e.g.)--or catch any exception, print an error message and exit with 1 @@ -547,9 +545,8 @@ def _show_help(self, parser, global_options_=True, display_options_=True, commands=[]): - # late import because of mutual dependence between these modules - from distutils2.command.cmd import Command - + # XXX want to print to stdout when help is requested (--help) but to + # stderr in case of error! print 'Usage: pysetup [options] action [action_options]' print if global_options_: diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -2,6 +2,7 @@ import os import sys +import textwrap from StringIO import StringIO from distutils2 import install @@ -31,14 +32,7 @@ support.LoggingCatcher, unittest.TestCase): - def setUp(self): - super(RunTestCase, self).setUp() - self.old_argv = sys.argv, sys.argv[:] - - def tearDown(self): - sys.argv = self.old_argv[0] - sys.argv[:] = self.old_argv[1] - super(RunTestCase, self).tearDown() + maxDiff = None # TODO restore the tests removed six months ago and port them to pysetup @@ -127,6 +121,38 @@ self.assertEqual(err.splitlines(), ["error: action 'invalid_action' not recognized"]) + def test_setupcfg_parsing(self): + # #14733: pysetup used to parse setup.cfg too late + project_dir = self.mkdtemp() + os.chdir(project_dir) + custompy = textwrap.dedent( + """\ + from distutils2.command.cmd import Command + + class custom(Command): + + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + print 'custom: ok' + """) + setupcfg = textwrap.dedent( + """\ + [global] + commands = custom.custom + """) + self.write_file('custom.py', custompy) + self.write_file('setup.cfg', setupcfg) + + out, err = self.call_pysetup('run', 'custom') + self.assertEqual(out.splitlines(), ['custom: ok']) + def test_suite(): return unittest.makeSuite(RunTestCase) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 21:56:29 2012 From: python-checkins at python.org (barry.warsaw) Date: Mon, 21 May 2012 21:56:29 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_some_clarification_and_exa?= =?utf8?q?mples_based_on_Guido=27s_feedback=2E?= Message-ID: http://hg.python.org/peps/rev/385f1b81d55a changeset: 4414:385f1b81d55a user: Barry Warsaw date: Mon May 21 15:56:20 2012 -0400 summary: Add some clarification and examples based on Guido's feedback. files: pep-0420.txt | 49 ++++++++++++++++++++++++++++++++++++++- 1 files changed, 47 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -386,14 +386,59 @@ * If the module has an ``__loader__`` and that loader has a ``module_repr()`` method, call it with a single argument, which is the module object. The value returned is used as the module's repr. - * Exceptions from ``module_repr()`` are ignored, and the following steps - are used instead. + * If an exception occurs in ``module_repr()``, the exception is + caught and discarded, and the calculation of the module's repr + continues as if ``module_repr()`` did not exist. * If the module has an ``__file__`` attribute, this is used as part of the module's repr. * If the module has no ``__file__`` but does have an ``__loader__``, then the loader's repr is used as part of the module's repr. * Otherwise, just use the module's ``__name__`` in the repr. +Here is a snippet showing how namespace module reprs are calculated +from its loader:: + + class NamespaceLoader: + @classmethod + def module_repr(cls, module): + return "".format(module.__name__) + +Built-in module reprs would no longer need to be hard-coded, but +instead would come from their loader as well:: + + class BuiltinImporter: + @classmethod + def module_repr(cls, module): + return "".format(module.__name__) + +Here are some example reprs of different types of modules with +different sets of the related attributes:: + + >>> import email + >>> email + + >>> m = type(email)('foo') + >>> m + + >>> m.__file__ = 'zippy:/de/do/dah' + >>> m + + >>> class Loader: pass + ... + >>> m.__loader__ = Loader + >>> del m.__file__ + >>> m + )> + >>> class NewLoader: + ... @classmethod + ... def module_repr(cls, module): + ... return '' + ... + >>> m.__loader__ = NewLoader + >>> m + + >>> + References ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Mon May 21 23:03:35 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:35 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Use_test_skip_decorator_?= =?utf8?q?instead_of_private_module_attribute?= Message-ID: http://hg.python.org/distutils2/rev/ea2d092556c2 changeset: 1345:ea2d092556c2 user: ?ric Araujo date: Mon May 21 16:20:43 2012 -0400 summary: Use test skip decorator instead of private module attribute files: distutils2/tests/test_command_check.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/distutils2/tests/test_command_check.py b/distutils2/tests/test_command_check.py --- a/distutils2/tests/test_command_check.py +++ b/distutils2/tests/test_command_check.py @@ -1,7 +1,6 @@ """Tests for distutils.command.check.""" from distutils2.command.check import check -from distutils2.metadata import _HAS_DOCUTILS from distutils2.errors import PackagingSetupError, MetadataMissingError from distutils2.tests import unittest, support @@ -101,7 +100,7 @@ self._run(metadata, strict=True) self.assertEqual(self.get_logs(), []) - @unittest.skipUnless(_HAS_DOCUTILS, "requires docutils") + @support.require_docutils def test_check_restructuredtext(self): # let's see if it detects broken rest in description broken_rest = 'title\n===\n\ntest' -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 23:03:35 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:35 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_overlooked_thinkos_i?= =?utf8?q?n_one_test_file?= Message-ID: http://hg.python.org/distutils2/rev/fa61f93eafc7 changeset: 1346:fa61f93eafc7 user: ?ric Araujo date: Mon May 21 16:26:46 2012 -0400 summary: Fix overlooked thinkos in one test file files: distutils2/tests/test_run.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -105,19 +105,19 @@ def test_invalid_command(self): out, err = self.call_pysetup_fail('run', 'com#mand') - self.assertGreater(out, 1) + self.assertGreater(out, '') self.assertEqual(err.splitlines(), ["error: invalid command name 'com#mand'"]) def test_unknown_command(self): out, err = self.call_pysetup_fail('run', 'invalid_command') - self.assertGreater(out, 1) + self.assertGreater(out, '') self.assertEqual(err.splitlines(), ["error: command 'invalid_command' not recognized"]) def test_unknown_action(self): out, err = self.call_pysetup_fail('invalid_action') - self.assertGreater(out, 1) + self.assertGreater(out, '') self.assertEqual(err.splitlines(), ["error: action 'invalid_action' not recognized"]) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 23:03:35 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:35 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Also_check_stderr_in_one?= =?utf8?q?_test?= Message-ID: http://hg.python.org/distutils2/rev/47ab2e3cefc6 changeset: 1347:47ab2e3cefc6 user: ?ric Araujo date: Mon May 21 16:39:27 2012 -0400 summary: Also check stderr in one test files: distutils2/tests/test_run.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -151,6 +151,7 @@ self.write_file('setup.cfg', setupcfg) out, err = self.call_pysetup('run', 'custom') + self.assertEqual(err.splitlines(), ['running custom']) self.assertEqual(out.splitlines(), ['custom: ok']) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 23:03:35 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:35 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Minor_simplification_in_?= =?utf8?q?one_test?= Message-ID: http://hg.python.org/distutils2/rev/46827ba519b1 changeset: 1348:46827ba519b1 user: ?ric Araujo date: Mon May 21 16:48:14 2012 -0400 summary: Minor simplification in one test files: distutils2/tests/test_pypi_wrapper.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/distutils2/tests/test_pypi_wrapper.py b/distutils2/tests/test_pypi_wrapper.py --- a/distutils2/tests/test_pypi_wrapper.py +++ b/distutils2/tests/test_pypi_wrapper.py @@ -23,7 +23,7 @@ def test_wrapper(self): index = Indexes._indexes['one'] - func = switch_index_if_fails(getattr(index, 'test'), Indexes) + func = switch_index_if_fails(index.test, Indexes) self.assertEqual(func(), 'OK') -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 23:03:36 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:36 +0200 Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_typo?= Message-ID: http://hg.python.org/distutils2/rev/747eec42e7ae changeset: 1350:747eec42e7ae parent: 1348:46827ba519b1 user: ?ric Araujo date: Mon May 21 17:01:44 2012 -0400 summary: Fix typo files: distutils2/tests/test_command_check.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/distutils2/tests/test_command_check.py b/distutils2/tests/test_command_check.py --- a/distutils2/tests/test_command_check.py +++ b/distutils2/tests/test_command_check.py @@ -100,7 +100,7 @@ self._run(metadata, strict=True) self.assertEqual(self.get_logs(), []) - @support.require_docutils + @support.requires_docutils def test_check_restructuredtext(self): # let's see if it detects broken rest in description broken_rest = 'title\n===\n\ntest' -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 23:03:36 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:36 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?= =?utf8?q?n3=29=3A_Merge_from_default?= Message-ID: http://hg.python.org/distutils2/rev/0291648eb2b2 changeset: 1351:0291648eb2b2 branch: python3 parent: 1349:1eff97fce288 parent: 1350:747eec42e7ae user: ?ric Araujo date: Mon May 21 17:03:13 2012 -0400 summary: Merge from default files: distutils2/tests/test_command_check.py | 3 +-- distutils2/tests/test_pypi_wrapper.py | 2 +- distutils2/tests/test_run.py | 1 + 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/distutils2/tests/test_command_check.py b/distutils2/tests/test_command_check.py --- a/distutils2/tests/test_command_check.py +++ b/distutils2/tests/test_command_check.py @@ -1,7 +1,6 @@ """Tests for distutils.command.check.""" from distutils2.command.check import check -from distutils2.metadata import _HAS_DOCUTILS from distutils2.errors import PackagingSetupError, MetadataMissingError from distutils2.tests import unittest, support @@ -101,7 +100,7 @@ self._run(metadata, strict=True) self.assertEqual(self.get_logs(), []) - @unittest.skipUnless(_HAS_DOCUTILS, "requires docutils") + @support.requires_docutils def test_check_restructuredtext(self): # let's see if it detects broken rest in description broken_rest = 'title\n===\n\ntest' diff --git a/distutils2/tests/test_pypi_wrapper.py b/distutils2/tests/test_pypi_wrapper.py --- a/distutils2/tests/test_pypi_wrapper.py +++ b/distutils2/tests/test_pypi_wrapper.py @@ -23,7 +23,7 @@ def test_wrapper(self): index = Indexes._indexes['one'] - func = switch_index_if_fails(getattr(index, 'test'), Indexes) + func = switch_index_if_fails(index.test, Indexes) self.assertEqual(func(), 'OK') diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -151,6 +151,7 @@ self.write_file('setup.cfg', setupcfg) out, err = self.call_pysetup('run', 'custom') + self.assertEqual(err.splitlines(), [b'running custom']) self.assertEqual(out.splitlines(), [b'custom: ok']) -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Mon May 21 23:03:36 2012 From: python-checkins at python.org (eric.araujo) Date: Mon, 21 May 2012 23:03:36 +0200 Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?= =?utf8?q?n3=29=3A_Merge_default?= Message-ID: http://hg.python.org/distutils2/rev/1eff97fce288 changeset: 1349:1eff97fce288 branch: python3 parent: 1291:7d1a7251d771 parent: 1344:70831fde0ec4 user: ?ric Araujo date: Mon May 21 16:54:12 2012 -0400 summary: Merge default files: .hgignore | 17 +- .hgsigs | 1 + .hgtags | 1 + CHANGES.txt | 25 +- CONTRIBUTORS.txt | 6 + DEVNOTES.txt | 8 +- MANIFEST.in | 1 - README.txt | 2 +- distutils2/__init__.py | 2 +- distutils2/_trove.py | 11 +- distutils2/command/build_scripts.py | 15 +- distutils2/command/cmd.py | 16 +- distutils2/command/register.py | 1 + distutils2/compiler/msvc9compiler.py | 2 +- distutils2/compiler/msvccompiler.py | 2 +- distutils2/database.py | 94 +- distutils2/install.py | 32 +- distutils2/metadata.py | 33 +- distutils2/pypi/simple.py | 4 +- distutils2/pypi/wrapper.py | 5 +- distutils2/run.py | 42 +- distutils2/tests/requires.txt | 15 + distutils2/tests/support.py | 8 +- distutils2/tests/test_command_bdist.py | 15 +- distutils2/tests/test_command_build_scripts.py | 71 +- distutils2/tests/test_command_register.py | 26 +- distutils2/tests/test_database.py | 70 +- distutils2/tests/test_install.py | 26 +- distutils2/tests/test_metadata.py | 37 +- distutils2/tests/test_msvc9compiler.py | 2 +- distutils2/tests/test_pypi_wrapper.py | 37 + distutils2/tests/test_run.py | 95 +- distutils2/tests/test_util.py | 57 +- distutils2/tests/test_version.py | 10 +- distutils2/util.py | 80 + distutils2/version.py | 18 +- docs/Makefile | 89 - docs/design/configfile.rst | 131 - docs/design/pep-0376.txt | 687 --- docs/design/wiki.rst | 620 --- docs/make.bat | 113 - docs/source/_static/depgraph_big.png | Bin docs/source/conf.py | 199 - docs/source/contributing.rst | 25 - docs/source/devresources.rst | 49 - docs/source/distutils/apiref.rst | 2010 ---------- docs/source/distutils/builtdist.rst | 452 -- docs/source/distutils/commandhooks.rst | 31 - docs/source/distutils/commandref.rst | 58 - docs/source/distutils/configfile.rst | 130 - docs/source/distutils/examples.rst | 332 - docs/source/distutils/extending.rst | 95 - docs/source/distutils/index.rst | 33 - docs/source/distutils/introduction.rst | 193 - docs/source/distutils/newcommands.rst | 144 - docs/source/distutils/packageindex.rst | 104 - docs/source/distutils/setupscript.rst | 681 --- docs/source/distutils/sourcedist.rst | 270 - docs/source/distutils/uploading.rst | 80 - docs/source/images/depgraph_output.png | Bin docs/source/index.rst | 89 - docs/source/install/index.rst | 988 ---- docs/source/library/distutils2.depgraph.rst | 122 - docs/source/library/distutils2.index.client.rst | 20 - docs/source/library/distutils2.index.dist.rst | 106 - docs/source/library/distutils2.index.rst | 31 - docs/source/library/distutils2.index.simple.rst | 141 - docs/source/library/distutils2.index.xmlrpc.rst | 124 - docs/source/library/distutils2.install.rst | 23 - docs/source/library/distutils2.metadata.rst | 93 - docs/source/library/distutils2.rst | 44 - docs/source/library/distutils2.tests.pypi_server.rst | 89 - docs/source/library/distutils2.version.rst | 69 - docs/source/library/pkgutil.rst | 330 - docs/source/setupcfg-spec.rst | 308 - docs/source/setupcfg.rst | 544 -- docs/source/tutorial.rst | 129 - setup.cfg | 26 +- setup.py | 2 +- tox.ini | 22 +- 80 files changed, 664 insertions(+), 10049 deletions(-) diff --git a/.hgignore b/.hgignore --- a/.hgignore +++ b/.hgignore @@ -1,18 +1,15 @@ syntax: glob *.py[co] __pycache__/ -*.so -configure.cache +*.swp +bin/ +include/ +lib/ +man/ +Distutils2.egg-info MANIFEST build/ dist/ -_static/ -_build/ -*.swp +.tox/ .coverage -.tox -lib -include -bin nosetests.xml -Distutils2.egg-info diff --git a/.hgsigs b/.hgsigs new file mode 100644 --- /dev/null +++ b/.hgsigs @@ -0,0 +1,1 @@ +27910fcea9ed851af1f932caa58fdc0f811b748f 0 iEYEABECAAYFAk9fbhkACgkQ8s4Pc9+KtnaVdACfTNegvVs1BXtKgM+oAq/9gsuZtGwAnA04B7Zc/ICRkbpejyS4xzPTv8jw diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -5,3 +5,4 @@ 7c8e61aa51f4748286964bc1405bd4169c270f46 1.0a3 7c8e61aa51f4748286964bc1405bd4169c270f46 1.0a3 d930ae6caab58bec92683235aa88d06bbc07ae36 1.0a3 +1e4d52d83e95c14e3f0bd2179a81ac1023ef32e9 1.0a4 diff --git a/CHANGES.txt b/CHANGES.txt --- a/CHANGES.txt +++ b/CHANGES.txt @@ -8,8 +8,21 @@ (and last name initial when needed) are given for each item; see CONTRIBUTORS.txt for full names. Bug numbers refer to http://bugs.python.org/. +1.0a5 - 2012-xx-xx +------------------ -1.0a4 - 2012-02-?? +- #14294: Let Metadata convert setuptools-style requires.txt [preston] +- #14270: Add dest_dir parameter to install functions [mathieu] +- #13166: Add __str__ to database.*Distribution for nicer output [guillaume] +- #13614: Fix register failure with invalid rst in description [julien c, + mathieu, pierre paul, ?ric] +- #13399: Display error message instead of unhandled traceback for missing + actions, commands and options [patrice] +- #10374: Recreate scripts everytime build_scripts is called [pierre paul] +- #14733: Have pysetup read setup.cfg early enough to find custom commands + [?ric, janusz] + +1.0a4 - 2012-03-13 ------------------ - Remove type check for commands in favor of minimal duck type check [tarek] @@ -111,8 +124,8 @@ guillermoo, ?ric] - Add 'pysetup generate-setup' to expose util.generate_setup_py [tarek] - #11092: Make sdist always include setup.cfg [?ric] -- #12246: Don?t try to install something when running from an uninstalled Python - built in its checkout [tschepang, ?ric] +- #12246: Don't try to install something when running from an uninstalled Python + built in its checkout [tshepang, ?ric] - Add packaging.util.split_multiline [julien m, erik] - #11595: Fix assorted bugs in packaging.util.cfg_to_args [erik, ?ric] - #12240: Allow multiple setup hooks [erik, ?ric] @@ -169,6 +182,12 @@ - #13974: add test for util.set_platform [tshepang] - #6884: Fix MANIFEST.in parsing bugs on Windows [?ric, nadeem] - #11841: Fix comparison bug with 'rc' versions [filip] +- #14263: Fix function name lookup in d2.pypi.wrapper [tarek] +- #14264: Stop removing trailing zeroes in versions [tarek] +- #14268: Fix small error in a test [tarek] +- Drop support for Python 2.4 [tarek, ?ric] +- #13009: Out-of-date documentation removed, people should look at + http://docs.python.org/dev/packaging [?ric] 1.0a3 - 2010-10-08 diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt --- a/CONTRIBUTORS.txt +++ b/CONTRIBUTORS.txt @@ -20,6 +20,7 @@ - Francisco Mart?n Brugu? - Nicolas Cadou - Godefroid Chapelle +- Julien Courteau - Christophe Combelles - Jason R. Coombs - Pierre-Yves David @@ -31,19 +32,23 @@ - Boris Feld - Andrew Francis - Hallvard B Furuseth +- Patrice Gauthier - Yannick Gingras - Filip Gruszczy?ski - Walker Hale IV - Alexandre Hamelin - Kelsey Hightower - Thomas Holmes +- Preston Holmes - Christian Hudon - Julien Jehannet - Jeremy Kloth - Amos Latteier - Mathieu Leduc-Hamel +- Pierre Paul Lefebvre - Tshepang Lekhonkhobe - Alain Leufroy +- Janusz Lewandowski - Martin von L?wis - Hugo Lopes Tavares - Guillermo L?pez-Anglada @@ -59,6 +64,7 @@ - Ga?l Pasgrimaud - George Peristerakis - Mathieu Perreault +- Guillaume Pratte - Sean Reifschneider - Antoine Reversat - Arc Riley diff --git a/DEVNOTES.txt b/DEVNOTES.txt --- a/DEVNOTES.txt +++ b/DEVNOTES.txt @@ -11,7 +11,7 @@ Repo: http://hg.python.org/cpython (default branch) More info: http://wiki.python.org/moin/Distutils/Contributing -- Distutils2 runs on Python from 2.4 to 2.7, so make sure you don't use code +- Distutils2 runs on Python from 2.5 to 2.7, so make sure you don't use code that doesn't work under one of these Python versions. The version in the "python3" branch is compatible with all version from 3.1 to 3.3. @@ -21,12 +21,8 @@ have a look or use a diff tool with the same file in distutils or packaging from Python 3.3. If you can't run tests, let someone else do the merge. -- For 2.4, you need to run "python2.4 setup.py build" before you can try out - pysetup or run tests (unless you use the runtests.py script which will call - "setup.py build" automatically). - - Always run tests.sh before you commit a change. This implies that you have - all Python versions installed from 2.4 to 2.7, as well as 3.1-.3.3 if you + all Python versions installed from 2.5 to 2.7, as well as 3.1-.3.3 if you merge into the python3 branch. Be sure to also have docutils installed on all Python versions to avoid skipping tests. diff --git a/MANIFEST.in b/MANIFEST.in --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,2 +1,1 @@ -recursive-include docs *.bat *.rst *.py *.png Makefile *.txt recursive-include distutils2/_backport *.c *.h *.cfg diff --git a/README.txt b/README.txt --- a/README.txt +++ b/README.txt @@ -10,7 +10,7 @@ - Developers of packaging-related tools who need a support library to build on -Authors will have to write a :file:`setup.cfg` file and run a few +Authors will have to write a ``setup.cfg`` file and run a few commands to package and distribute their code. End users will be able to search for, install and remove Python projects with the included ``pysetup`` program. Last, developers will be able to reuse classes and diff --git a/distutils2/__init__.py b/distutils2/__init__.py --- a/distutils2/__init__.py +++ b/distutils2/__init__.py @@ -13,5 +13,5 @@ __all__ = ['__version__', 'logger'] -__version__ = "1.0a3" +__version__ = "1.0a5.dev0" logger = getLogger('distutils2') diff --git a/distutils2/_trove.py b/distutils2/_trove.py --- a/distutils2/_trove.py +++ b/distutils2/_trove.py @@ -114,6 +114,7 @@ 'License :: OSI Approved :: Motosoto License', 'License :: OSI Approved :: Mozilla Public License 1.0 (MPL)', 'License :: OSI Approved :: Mozilla Public License 1.1 (MPL 1.1)', +'License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)', 'License :: OSI Approved :: Nethack General Public License', 'License :: OSI Approved :: Nokia Open Source License', 'License :: OSI Approved :: Open Group Test Suite License', @@ -149,6 +150,7 @@ 'Natural Language :: Esperanto', 'Natural Language :: Finnish', 'Natural Language :: French', +'Natural Language :: Galician', 'Natural Language :: German', 'Natural Language :: Greek', 'Natural Language :: Hebrew', @@ -193,9 +195,14 @@ 'Operating System :: Microsoft :: MS-DOS', 'Operating System :: Microsoft :: Windows', 'Operating System :: Microsoft :: Windows :: Windows 3.1 or Earlier', +'Operating System :: Microsoft :: Windows :: Windows 7', 'Operating System :: Microsoft :: Windows :: Windows 95/98/2000', 'Operating System :: Microsoft :: Windows :: Windows CE', 'Operating System :: Microsoft :: Windows :: Windows NT/2000', +'Operating System :: Microsoft :: Windows :: Windows Server 2003', +'Operating System :: Microsoft :: Windows :: Windows Server 2008', +'Operating System :: Microsoft :: Windows :: Windows Vista', +'Operating System :: Microsoft :: Windows :: Windows XP', 'Operating System :: OS/2', 'Operating System :: OS Independent', 'Operating System :: Other OS', @@ -263,10 +270,12 @@ 'Programming Language :: Python :: 2.5', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', +'Programming Language :: Python :: 2 :: Only', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.0', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.2', +'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: Implementation', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: IronPython', @@ -449,8 +458,8 @@ 'Topic :: Printing', 'Topic :: Religion', 'Topic :: Scientific/Engineering', +'Topic :: Scientific/Engineering :: Artificial Intelligence', 'Topic :: Scientific/Engineering :: Artificial Life', -'Topic :: Scientific/Engineering :: Artificial Intelligence', 'Topic :: Scientific/Engineering :: Astronomy', 'Topic :: Scientific/Engineering :: Atmospheric Science', 'Topic :: Scientific/Engineering :: Bio-Informatics', diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py --- a/distutils2/command/build_scripts.py +++ b/distutils2/command/build_scripts.py @@ -5,7 +5,7 @@ from tokenize import detect_encoding from distutils2.command.cmd import Command -from distutils2.util import convert_path, newer +from distutils2.util import convert_path from distutils2 import logger from distutils2.compat import Mixin2to3 from distutils2._backport import sysconfig @@ -21,17 +21,12 @@ user_options = [ ('build-dir=', 'd', "directory to build (copy) to"), - ('force', 'f', "forcibly build everything (ignore file timestamps"), ('executable=', 'e', "specify final destination interpreter path"), ] - boolean_options = ['force'] - - def initialize_options(self): self.build_dir = None self.scripts = None - self.force = None self.executable = None self.outfiles = None self.use_2to3 = False @@ -42,7 +37,7 @@ self.set_undefined_options('build', ('build_scripts', 'build_dir'), 'use_2to3', 'use_2to3_fixers', - 'convert_2to3_doctests', 'force', + 'convert_2to3_doctests', 'executable') self.scripts = self.distribution.scripts @@ -62,6 +57,8 @@ ie. starts with "\#!" and contains "python"), then adjust the first line to refer to the current Python interpreter as we copy. """ + # XXX use self.execute(shutil.rmtree, ...) + self.rmpath(self.build_dir) self.mkpath(self.build_dir) outfiles = [] for script in self.scripts: @@ -70,10 +67,6 @@ outfile = os.path.join(self.build_dir, os.path.basename(script)) outfiles.append(outfile) - if not self.force and not newer(script, outfile): - logger.debug("not copying %s (up-to-date)", script) - continue - # Always open the file, but ignore failures in dry-run mode -- # that way, we'll get accurate feedback if we can read the # script. diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py --- a/distutils2/command/cmd.py +++ b/distutils2/command/cmd.py @@ -5,7 +5,7 @@ from distutils2 import util from distutils2 import logger from distutils2.errors import PackagingOptionError -from distutils2._backport.shutil import copyfile, move, make_archive +from distutils2._backport.shutil import copyfile, move, make_archive, rmtree class Command: @@ -365,6 +365,20 @@ return os.makedirs(name, mode) + def rmpath(self, name, dry_run=None): + if dry_run is None: + dry_run = self.dry_run + name = os.path.normpath(name) + if not os.path.isdir(name) or name == '': + return + if dry_run: + head = '' + for part in name.split(os.sep): + logger.info("removing directory %s%s", head, part) + head += part + os.sep + return + rmtree(name) + def copy_file(self, infile, outfile, preserve_mode=True, preserve_times=True, link=None, level=1): """Copy a file respecting dry-run and force flags. diff --git a/distutils2/command/register.py b/distutils2/command/register.py --- a/distutils2/command/register.py +++ b/distutils2/command/register.py @@ -81,6 +81,7 @@ def classifiers(self): ''' Fetch the list of classifiers from the server. ''' + # TODO use _trove module response = urllib.request.urlopen(self.repository+'?:action=list_classifiers') logger.info(response.read()) diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py --- a/distutils2/compiler/msvc9compiler.py +++ b/distutils2/compiler/msvc9compiler.py @@ -20,7 +20,7 @@ from distutils2 import logger from distutils2.util import get_platform -import winreg +import _winreg as winreg RegOpenKeyEx = winreg.OpenKeyEx RegEnumKey = winreg.EnumKey diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py --- a/distutils2/compiler/msvccompiler.py +++ b/distutils2/compiler/msvccompiler.py @@ -19,7 +19,7 @@ _can_read_reg = False try: - import winreg + import _winreg as winreg _can_read_reg = True hkey_mod = winreg diff --git a/distutils2/database.py b/distutils2/database.py --- a/distutils2/database.py +++ b/distutils2/database.py @@ -1,17 +1,16 @@ """PEP 376 implementation.""" import os -import re import csv import sys import zipimport from io import StringIO from hashlib import md5 -from distutils2 import logger from distutils2.errors import PackagingError from distutils2.version import suggest_normalized_version, VersionPredicate from distutils2.metadata import Metadata +from distutils2.util import parse_requires __all__ = [ @@ -157,6 +156,9 @@ return '' % ( self.name, self.version, self.path) + def __str__(self): + return "%s %s" % (self.name, self.version) + def _get_records(self, local=False): results = [] with self.get_distinfo_file('RECORD') as record: @@ -291,12 +293,6 @@ """A :class:`distutils2.metadata.Metadata` instance loaded with the distribution's ``METADATA`` file.""" - _REQUIREMENT = re.compile( - r'(?P[-A-Za-z0-9_.]+)\s*' - r'(?P(?:<|<=|!=|==|>=|>)[-A-Za-z0-9_.]+)?\s*' - r'(?P(?:\s*,\s*(?:<|<=|!=|==|>=|>)[-A-Za-z0-9_.]+)*)\s*' - r'(?P\[.*\])?') - def __init__(self, path): self.path = path if _cache_enabled and path in _cache_path_egg: @@ -305,33 +301,14 @@ self.version = self.metadata['Version'] return - # reused from Distribute's pkg_resources - def yield_lines(strs): - """Yield non-empty/non-comment lines of a ``basestring`` - or sequence""" - if isinstance(strs, str): - for s in strs.splitlines(): - s = s.strip() - # skip blank lines/comments - if s and not s.startswith('#'): - yield s - else: - for ss in strs: - for s in yield_lines(ss): - yield s - requires = None if path.endswith('.egg'): if os.path.isdir(path): meta_path = os.path.join(path, 'EGG-INFO', 'PKG-INFO') self.metadata = Metadata(path=meta_path) - try: - req_path = os.path.join(path, 'EGG-INFO', 'requires.txt') - with open(req_path, 'r') as fp: - requires = fp.read() - except IOError: - requires = None + req_path = os.path.join(path, 'EGG-INFO', 'requires.txt') + requires = parse_requires(req_path) else: # FIXME handle the case where zipfile is not available zipf = zipimport.zipimporter(path) @@ -348,11 +325,8 @@ elif path.endswith('.egg-info'): if os.path.isdir(path): path = os.path.join(path, 'PKG-INFO') - try: - with open(os.path.join(path, 'requires.txt'), 'r') as fp: - requires = fp.read() - except IOError: - requires = None + req_path = os.path.join(path, 'requires.txt') + requires = parse_requires(req_path) self.metadata = Metadata(path=path) self.name = self.metadata['Name'] self.version = self.metadata['Version'] @@ -361,46 +335,13 @@ raise ValueError('path must end with .egg-info or .egg, got %r' % path) - if requires is not None: + if requires: if self.metadata['Metadata-Version'] == '1.1': # we can't have 1.1 metadata *and* Setuptools requires for field in ('Obsoletes', 'Requires', 'Provides'): - del self.metadata[field] - - reqs = [] - - if requires is not None: - for line in yield_lines(requires): - if line.startswith('['): - logger.warning( - 'extensions in requires.txt are not supported ' - '(used by %r %s)', self.name, self.version) - break - else: - match = self._REQUIREMENT.match(line.strip()) - if not match: - # this happens when we encounter extras; since they - # are written at the end of the file we just exit - break - else: - if match.group('extras'): - msg = ('extra requirements are not supported ' - '(used by %r %s)', self.name, self.version) - logger.warning(msg, self.name) - name = match.group('name') - version = None - if match.group('first'): - version = match.group('first') - if match.group('rest'): - version += match.group('rest') - version = version.replace(' ', '') # trim spaces - if version is None: - reqs.append(name) - else: - reqs.append('%s (%s)' % (name, version)) - - if len(reqs) > 0: - self.metadata['Requires-Dist'] += reqs + if field in self.metadata: + del self.metadata[field] + self.metadata['Requires-Dist'] += requires if _cache_enabled: _cache_path_egg[self.path] = self @@ -409,6 +350,9 @@ return '' % ( self.name, self.version, self.path) + def __str__(self): + return "%s %s" % (self.name, self.version) + def list_installed_files(self, local=False): def _md5(path): @@ -476,7 +420,7 @@ return '-'.join([name, normalized_version]) + file_extension -def get_distributions(use_egg_info=False, paths=None): +def get_distributions(use_egg_info=True, paths=None): """ Provides an iterator that looks for ``.dist-info`` directories in ``sys.path`` and returns :class:`Distribution` instances for each one of @@ -503,7 +447,7 @@ yield dist -def get_distribution(name, use_egg_info=False, paths=None): +def get_distribution(name, use_egg_info=True, paths=None): """ Scans all elements in ``sys.path`` and looks for all directories ending with ``.dist-info``. Returns a :class:`Distribution` @@ -538,7 +482,7 @@ return None -def obsoletes_distribution(name, version=None, use_egg_info=False): +def obsoletes_distribution(name, version=None, use_egg_info=True): """ Iterates over all distributions to find which distributions obsolete *name*. @@ -572,7 +516,7 @@ break -def provides_distribution(name, version=None, use_egg_info=False): +def provides_distribution(name, version=None, use_egg_info=True): """ Iterates over all distributions to find which distributions provide *name*. If a *version* is provided, it will be used to filter the results. Scans diff --git a/distutils2/install.py b/distutils2/install.py --- a/distutils2/install.py +++ b/distutils2/install.py @@ -58,10 +58,12 @@ yield old, new -def _run_distutils_install(path): +def _run_distutils_install(path, dest): # backward compat: using setuptools or plain-distutils + # FIXME pass dest argument to the command cmd = '%s setup.py install --record=%s' record_file = os.path.join(path, 'RECORD') + # FIXME use subprocess os.system(cmd % (sys.executable, record_file)) if not os.path.exists(record_file): raise ValueError('failed to install') @@ -69,7 +71,7 @@ egginfo_to_distinfo(record_file, remove_egginfo=True) -def _run_setuptools_install(path): +def _run_setuptools_install(path, dest): cmd = '%s setup.py install --record=%s --single-version-externally-managed' record_file = os.path.join(path, 'RECORD') @@ -80,12 +82,12 @@ egginfo_to_distinfo(record_file, remove_egginfo=True) -def _run_packaging_install(path): +def _run_packaging_install(path, dest): # XXX check for a valid setup.cfg? dist = Distribution() dist.parse_config_files() try: - dist.run_command('install_dist') + dist.run_command('install_dist', {'prefix': (None, dest)}) name = dist.metadata['Name'] return database.get_distribution(name) is not None except (IOError, os.error, PackagingError, CCompilerError) as msg: @@ -106,11 +108,13 @@ if where is None: raise ValueError('Cannot locate the unpacked archive') - return _run_install_from_archive(where) + return _run_install_from_archive(where, path) def install_local_project(path): - """Install a distribution from a source directory. + """Install a distribution from a source directory or archive. + + If *path* is an archive, it will be unarchived first. If the source directory contains a setup.py install using distutils1. If a setup.cfg is found, install using the install_dist command. @@ -134,14 +138,14 @@ return False -def _run_install_from_archive(source_dir): +def _run_install_from_archive(source_dir, dest_dir): # XXX need a better way for item in os.listdir(source_dir): fullpath = os.path.join(source_dir, item) if os.path.isdir(fullpath): source_dir = fullpath break - return _run_install_from_dir(source_dir) + return _run_install_from_dir(source_dir, dest_dir) install_methods = { @@ -150,15 +154,14 @@ 'distutils': _run_distutils_install} -def _run_install_from_dir(source_dir): +def _run_install_from_dir(source_dir, dest_dir=None): old_dir = os.getcwd() os.chdir(source_dir) - install_method = get_install_method(source_dir) - func = install_methods[install_method] try: + install_method = get_install_method(source_dir) func = install_methods[install_method] try: - func(source_dir) + func(source_dir, dest_dir) return True except ValueError as err: # failed to install @@ -183,7 +186,7 @@ installed_dists = [] for dist in dists: - logger.info('Installing %r %s...', dist.name, dist.version) + logger.info('Installing %s...', dist) try: _install_dist(dist, path) installed_dists.append(dist) @@ -244,7 +247,8 @@ temp_dir = tempfile.mkdtemp() for dist in remove: files = dist.list_installed_files() - temp_files[dist] = _move_files(files, temp_dir) + paths = [path for path, md5, size in files] + temp_files[dist] = _move_files(paths, temp_dir) try: if install: install_dists(install, install_path, paths) diff --git a/distutils2/metadata.py b/distutils2/metadata.py --- a/distutils2/metadata.py +++ b/distutils2/metadata.py @@ -34,6 +34,8 @@ def system_message(self, level, message, *children, **kwargs): self.messages.append((level, message, children, kwargs)) + return nodes.system_message(message, level=level, type=self. + levels[level], *children, **kwargs) _HAS_DOCUTILS = True except ImportError: @@ -48,7 +50,7 @@ # preferred version. Hopefully will be changed # to 1.2 once PEP 345 is supported everywhere -PKG_INFO_PREFERRED_VERSION = '1.0' +PKG_INFO_PREFERRED_VERSION = '1.1' _LINE_PREFIX = re.compile('\n \|') _241_FIELDS = ('Metadata-Version', 'Name', 'Version', 'Platform', @@ -101,7 +103,12 @@ return True return False - keys = list(fields) + keys = [] + for key, value in fields.items(): + if value in ([], 'UNKNOWN', None): + continue + keys.append(key) + possible_versions = ['1.0', '1.1', '1.2'] # first let's try to see if a field is not part of one of the version @@ -214,8 +221,9 @@ self.read_file(fileobj) elif mapping is not None: self.update(mapping) + self.set_metadata_version() - def _set_best_version(self): + def set_metadata_version(self): self._fields['Metadata-Version'] = _best_version(self._fields) def _write_field(self, file, name, value): @@ -233,7 +241,6 @@ del self._fields[field_name] except KeyError: raise KeyError(name) - self._set_best_version() def __contains__(self, name): return (name in self._fields or @@ -332,6 +339,7 @@ value = msg[field] if value is not None and value != 'UNKNOWN': self.set(field, value) + self.set_metadata_version() def write(self, filepath): """Write the metadata fields to filepath.""" @@ -340,7 +348,8 @@ def write_file(self, fileobject): """Write the PKG-INFO format data to a file object.""" - self._set_best_version() + self.set_metadata_version() + for field in _version2fieldlist(self['Metadata-Version']): values = self.get(field) if field in _ELEMENTSFIELD: @@ -367,14 +376,6 @@ Keys that don't match a metadata field or that have an empty value are dropped. """ - # XXX the code should just use self.set, which does tbe same checks and - # conversions already, but that would break packaging.pypi: it uses the - # update method, which does not call _set_best_version (which set - # does), and thus allows having a Metadata object (as long as you don't - # modify or write it) with extra fields from PyPI that are not fields - # defined in Metadata PEPs. to solve it, the best_version system - # should be reworked so that it's called only for writing, or in a new - # strict mode, or with a new, more lax Metadata subclass in p7g.pypi def _set(key, value): if key in _ATTR2FIELD and value: self.set(self._convert_name(key), value) @@ -435,7 +436,6 @@ value = self._remove_line_prefix(value) self._fields[name] = value - self._set_best_version() def get(self, name, default=_MISSING): """Get a metadata field.""" @@ -475,8 +475,10 @@ return value def check(self, strict=False, restructuredtext=False): - """Check if the metadata is compliant. If strict is False then raise if + """Check if the metadata is compliant. If strict is True then raise if no Name or Version are provided""" + self.set_metadata_version() + # XXX should check the versions (if the file was loaded) missing, warnings = [], [] @@ -521,6 +523,7 @@ Field names will be converted to use the underscore-lowercase style instead of hyphen-mixed case (i.e. home_page instead of Home-page). """ + self.set_metadata_version() data = { 'metadata_version': self['Metadata-Version'], 'name': self['Name'], diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py --- a/distutils2/pypi/simple.py +++ b/distutils2/pypi/simple.py @@ -196,8 +196,8 @@ def get_release(self, requirements, prefer_final=None): """Return only one release that fulfill the given requirements""" predicate = get_version_predicate(requirements) - release = self.get_releases(predicate, prefer_final)\ - .get_last(predicate) + releases = self.get_releases(predicate, prefer_final) + release = releases.get_last(predicate) if not release: raise ReleaseNotFound("No release matches the given criterias") return release diff --git a/distutils2/pypi/wrapper.py b/distutils2/pypi/wrapper.py --- a/distutils2/pypi/wrapper.py +++ b/distutils2/pypi/wrapper.py @@ -25,8 +25,9 @@ exception = None methods = [func] for f in wrapper._indexes.values(): - if f != func.__self__ and hasattr(f, func.__name__): - methods.append(getattr(f, func.__name__)) + func_name = func.__func__.__name__ + if f != func.__self__ and hasattr(f, func_name): + methods.append(getattr(f, func_name)) for method in methods: try: response = method(*args, **kwargs) diff --git a/distutils2/run.py b/distutils2/run.py --- a/distutils2/run.py +++ b/distutils2/run.py @@ -10,6 +10,7 @@ from distutils2.dist import Distribution from distutils2.util import _is_archive_file, generate_setup_py from distutils2.command import get_command_class, STANDARD_COMMANDS +from distutils2.command.cmd import Command from distutils2.install import install, install_local_project, remove from distutils2.database import get_distribution, get_distributions from distutils2.depgraph import generate_graph @@ -255,6 +256,13 @@ parser = dispatcher.parser args = args[1:] + # Find and parse the config file(s): they will override options from + # the setup script, but be overridden by the command line. + # XXX call the functions from config and kill the Distribution class + # (merging it into Dispatcher) + dist = Distribution() + dist.parse_config_files() + commands = STANDARD_COMMANDS # FIXME display extra commands if args == ['--list-commands']: @@ -270,16 +278,6 @@ if args is None: return - # create the Distribution class - # need to feed setup.cfg here ! - dist = Distribution() - - # Find and parse the config file(s): they will override options from - # the setup script, but be overridden by the command line. - - # XXX still need to be extracted from Distribution - dist.parse_config_files() - for cmd in dispatcher.commands: # FIXME need to catch MetadataMissingError here (from the check command # e.g.)--or catch any exception, print an error message and exit with 1 @@ -309,7 +307,7 @@ number = 0 for dist in results: - print('%r %s (from %r)' % (dist.name, dist.version, dist.path)) + print('%s (from %r)' % (dist, dist.path)) number += 1 if number == 0: @@ -388,6 +386,7 @@ self.parser.set_negative_aliases(negative_opt) # FIXME this parses everything, including command options (e.g. "run # build -i" errors with "option -i not recognized") + # FIXME unknown options are not detected args = self.parser.getopt(args=args, object=self) # if first arg is "run", we have some commands @@ -398,8 +397,8 @@ allowed = [action[0] for action in actions] + [None] if self.action not in allowed: - msg = 'Unrecognized action "%s"' % self.action - raise PackagingArgError(msg) + self.show_help() + sys.exit('error: action %r not recognized' % self.action) self._set_logger() self.args = args @@ -436,7 +435,8 @@ # Pull the current command from the head of the command line command = args[0] if not command_re.match(command): - raise SystemExit("invalid command name %r" % (command,)) + self.show_help() # TODO list only commands, not actions + sys.exit('error: invalid command name %r' % command) self.commands.append(command) # Dig up the command class that implements this command, so we @@ -445,7 +445,8 @@ try: cmd_class = get_command_class(command) except PackagingModuleError as msg: - raise PackagingArgError(msg) + self.show_help() # TODO list only commands, not actions + sys.exit('error: command %r not recognized' % command) # XXX We want to push this in distutils2.command # @@ -486,7 +487,11 @@ cmd_class.user_options + help_options) parser.set_negative_aliases(_negative_opt) - args, opts = parser.getopt(args[1:]) + try: + args, opts = parser.getopt(args[1:]) + except PackagingArgError as msg: + self.show_help() + sys.exit('error: %s' % msg) if hasattr(opts, 'help') and opts.help: self._show_command_help(cmd_class) @@ -541,9 +546,8 @@ def _show_help(self, parser, global_options_=True, display_options_=True, commands=[]): - # late import because of mutual dependence between these modules - from distutils2.command.cmd import Command - + # XXX want to print to stdout when help is requested (--help) but to + # stderr in case of error! print('Usage: pysetup [options] action [action_options]') print() if global_options_: diff --git a/distutils2/tests/requires.txt b/distutils2/tests/requires.txt new file mode 100644 --- /dev/null +++ b/distutils2/tests/requires.txt @@ -0,0 +1,15 @@ +setuptools +zope.browser +zope.component +zope.configuration +zope.contenttype >= 3.5 +zope.event +zope.exceptions +zope.i18n +zope.interface +zope.location +zope.proxy +zope.security + +[test] +zope.testing \ No newline at end of file diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py --- a/distutils2/tests/support.py +++ b/distutils2/tests/support.py @@ -45,6 +45,10 @@ import zlib except ImportError: zlib = None +try: + import docutils +except ImportError: + docutils = None from distutils2.dist import Distribution from distutils2.util import resolve_name @@ -62,7 +66,7 @@ # misc. functions and decorators 'fake_dec', 'create_distribution', 'use_command', 'copy_xxmodule_c', 'fixup_build_ext', - 'skip_2to3_optimize', + 'skip_2to3_optimize', 'requires_docutils', # imported from this module for backport purposes 'unittest', 'requires_zlib', 'skip_unless_symlink', ] @@ -404,6 +408,8 @@ requires_zlib = unittest.skipUnless(zlib, 'requires zlib') +requires_docutils = unittest.skipUnless(docutils, 'requires docutils') + def unlink(filename): try: diff --git a/distutils2/tests/test_command_bdist.py b/distutils2/tests/test_command_bdist.py --- a/distutils2/tests/test_command_bdist.py +++ b/distutils2/tests/test_command_bdist.py @@ -1,7 +1,9 @@ """Tests for distutils.command.bdist.""" import os +from distutils2.errors import PackagingPlatformError +from distutils2.command.bdist import bdist, show_formats + from test.support import captured_stdout -from distutils2.command.bdist import bdist, show_formats from distutils2.tests import unittest, support @@ -42,6 +44,17 @@ self.assertTrue(subcmd.skip_build, '%s should take --skip-build from bdist' % name) + def test_unsupported_platform(self): + _os_name = os.name + try: + os.name = 'some-obscure-os' + + dist = self.create_dist()[1] + cmd = bdist(dist) + self.assertRaises(PackagingPlatformError, cmd.ensure_finalized) + finally: + os.name = _os_name + def test_show_formats(self): with captured_stdout() as stdout: show_formats() diff --git a/distutils2/tests/test_command_build_scripts.py b/distutils2/tests/test_command_build_scripts.py --- a/distutils2/tests/test_command_build_scripts.py +++ b/distutils2/tests/test_command_build_scripts.py @@ -20,7 +20,7 @@ cmd.finalize_options() - self.assertTrue(cmd.force) + self.assertFalse(cmd.force) self.assertEqual(cmd.build_dir, "/foo/bar") def test_build(self): @@ -38,13 +38,14 @@ for name in expected: self.assertIn(name, built) - def get_build_scripts_cmd(self, target, scripts): + def get_build_scripts_cmd(self, target, scripts, + executable=sys.executable): dist = Distribution() dist.scripts = scripts dist.command_obj["build"] = support.DummyCommand( build_scripts=target, - force=True, - executable=sys.executable, + force=False, + executable=executable, use_2to3=False, use_2to3_fixers=None, convert_2to3_doctests=None @@ -79,10 +80,8 @@ target = self.mkdtemp() expected = self.write_sample_scripts(source) - - cmd = self.get_build_scripts_cmd(target, - [os.path.join(source, fn) - for fn in expected]) + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, fn) for fn in expected]) cmd.finalize_options() # http://bugs.python.org/issue4524 @@ -102,6 +101,62 @@ for name in expected: self.assertIn(name, built) + def test_build_dir_recreated(self): + source = self.mkdtemp() + target = self.mkdtemp() + self.write_script(source, 'taunt', '#! /usr/bin/python') + + built = os.path.join(target, 'taunt') + + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, 'taunt')], 'pythona') + cmd.finalize_options() + cmd.run() + + with open(built) as fp: + firstline = fp.readline().strip() + self.assertEqual(firstline, '#!pythona') + + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, 'taunt')], 'pythonx') + cmd.finalize_options() + cmd.run() + + with open(built) as fp: + firstline = fp.readline().strip() + self.assertEqual(firstline, '#!pythonx') + + def test_build_old_scripts_deleted(self): + source = self.mkdtemp() + target = self.mkdtemp() + + expected = ['script1.py', 'script2.py'] + self.write_script(source, "script1.py", + ("#! /usr/bin/env python2.3\n" + "pass\n")) + self.write_script(source, "script2.py", + ("#!/usr/bin/python\n" + "pass\n")) + + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, fn) for fn in expected]) + cmd.finalize_options() + cmd.run() + + built = sorted(os.listdir(target)) + self.assertEqual(built, expected) + + # if we run build_scripts with a different list of scripts, the old + # ones used to be left over in the build directory and installed anyway + cmd = self.get_build_scripts_cmd( + target, [os.path.join(source, 'script1.py')]) + cmd.finalize_options() + cmd.run() + + built = os.listdir(target) + self.assertEqual(built, ['script1.py']) + + def test_suite(): return unittest.makeSuite(BuildScriptsTestCase) diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py --- a/distutils2/tests/test_command_register.py +++ b/distutils2/tests/test_command_register.py @@ -5,14 +5,8 @@ import urllib.error import urllib.parse -try: - import docutils - DOCUTILS_SUPPORT = True -except ImportError: - DOCUTILS_SUPPORT = False - from distutils2.tests import unittest, support -from distutils2.tests.support import Inputs +from distutils2.tests.support import Inputs, requires_docutils from distutils2.command import register as register_module from distutils2.command.register import register from distutils2.errors import PackagingSetupError @@ -187,7 +181,7 @@ self.assertEqual(headers['Content-length'], '298') self.assertIn(b'tarek', req.data) - @unittest.skipUnless(DOCUTILS_SUPPORT, 'needs docutils') + @requires_docutils def test_strict(self): # testing the strict option: when on, the register command stops if the # metadata is incomplete or if description contains bad reST @@ -252,6 +246,22 @@ self.assertEqual(data['metadata_version'], '1.2') self.assertEqual(data['requires_dist'], ['lxml']) + @requires_docutils + def test_register_invalid_long_description(self): + description = ':funkie:`str`' # mimic Sphinx-specific markup + metadata = {'Home-page': 'xxx', 'Author': 'xxx', + 'Author-email': 'xxx', + 'Name': 'xxx', 'Version': 'xxx', + 'Description': description} + cmd = self._get_cmd(metadata) + cmd.ensure_finalized() + cmd.strict = True + inputs = Inputs('2', 'tarek', 'tarek at ziade.org') + register_module.input = inputs + with self.assertRaises(PackagingSetupError) as e: + cmd.run() + self.assertIn('funkie', str(e.exception)) + def test_suite(): return unittest.makeSuite(RegisterTestCase) diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py --- a/distutils2/tests/test_database.py +++ b/distutils2/tests/test_database.py @@ -75,12 +75,14 @@ attributes are used in test methods. See source code for details. """ + def _get_dist_path(self, distdir): + here = os.path.abspath(os.path.dirname(__file__)) + return os.path.join(here, 'fake_dists', distdir) + def test_instantiation(self): # check that useful attributes are here name, version, distdir = self.sample_dist - here = os.path.abspath(os.path.dirname(__file__)) - dist_path = os.path.join(here, 'fake_dists', distdir) - + dist_path = self._get_dist_path(distdir) dist = self.dist = self.cls(dist_path) self.assertEqual(dist.path, dist_path) self.assertEqual(dist.name, name) @@ -96,6 +98,17 @@ self.assertIn(self.cls.__name__, repr(dist)) @requires_zlib + def test_str(self): + name, version, distdir = self.sample_dist + dist = self.cls(self._get_dist_path(distdir)) + self.assertEqual(name, dist.name) + # Sanity test: dist.name is a unicode string, + # but str output contains no u prefix. + self.assertIsInstance(dist.name, str) + self.assertEqual(version, dist.version) + self.assertEqual(str(dist), self.expected_str_output) + + @requires_zlib def test_comparison(self): # tests for __eq__ and __hash__ dist = self.cls(self.dirs[0]) @@ -123,6 +136,7 @@ cls = Distribution sample_dist = 'choxie', '2.0.0.9', 'choxie-2.0.0.9.dist-info' + expected_str_output = 'choxie 2.0.0.9' def setUp(self): super(TestDistribution, self).setUp() @@ -251,6 +265,7 @@ cls = EggInfoDistribution sample_dist = 'bacon', '0.1', 'bacon-0.1.egg-info' + expected_str_output = 'bacon 0.1' def setUp(self): super(TestEggInfoDistribution, self).setUp() @@ -316,7 +331,7 @@ found_dists = [] # Verify the fake dists have been found. - dists = [dist for dist in get_distributions()] + dists = [dist for dist in get_distributions(use_egg_info=False)] for dist in dists: self.assertIsInstance(dist, Distribution) if (dist.name in dict(fake_dists) and @@ -367,10 +382,10 @@ # Verify that it does not find egg-info distributions, when not # instructed to - self.assertIsNone(get_distribution('bacon')) - self.assertIsNone(get_distribution('cheese')) - self.assertIsNone(get_distribution('strawberry')) - self.assertIsNone(get_distribution('banana')) + self.assertIsNone(get_distribution('bacon', use_egg_info=False)) + self.assertIsNone(get_distribution('cheese', use_egg_info=False)) + self.assertIsNone(get_distribution('strawberry', use_egg_info=False)) + self.assertIsNone(get_distribution('banana', use_egg_info=False)) # Now check that it works well in both situations, when egg-info # is a file and directory respectively. @@ -404,24 +419,29 @@ # Test for looking up distributions by what they provide checkLists = lambda x, y: self.assertEqual(sorted(x), sorted(y)) - l = [dist.name for dist in provides_distribution('truffles')] + l = [dist.name for dist in provides_distribution('truffles', + use_egg_info=False)] checkLists(l, ['choxie', 'towel-stuff']) - l = [dist.name for dist in provides_distribution('truffles', '1.0')] + l = [dist.name for dist in provides_distribution('truffles', '1.0', + use_egg_info=False)] checkLists(l, ['choxie']) l = [dist.name for dist in provides_distribution('truffles', '1.0', use_egg_info=True)] checkLists(l, ['choxie', 'cheese']) - l = [dist.name for dist in provides_distribution('truffles', '1.1.2')] + l = [dist.name for dist in provides_distribution('truffles', '1.1.2', + use_egg_info=False)] checkLists(l, ['towel-stuff']) - l = [dist.name for dist in provides_distribution('truffles', '1.1')] + l = [dist.name for dist in provides_distribution('truffles', '1.1', + use_egg_info=False)] checkLists(l, ['towel-stuff']) l = [dist.name for dist in provides_distribution('truffles', - '!=1.1,<=2.0')] + '!=1.1,<=2.0', + use_egg_info=False)] checkLists(l, ['choxie']) l = [dist.name for dist in provides_distribution('truffles', @@ -429,17 +449,20 @@ use_egg_info=True)] checkLists(l, ['choxie', 'bacon', 'cheese']) - l = [dist.name for dist in provides_distribution('truffles', '>1.0')] + l = [dist.name for dist in provides_distribution('truffles', '>1.0', + use_egg_info=False)] checkLists(l, ['towel-stuff']) - l = [dist.name for dist in provides_distribution('truffles', '>1.5')] + l = [dist.name for dist in provides_distribution('truffles', '>1.5', + use_egg_info=False)] checkLists(l, []) l = [dist.name for dist in provides_distribution('truffles', '>1.5', use_egg_info=True)] checkLists(l, ['bacon']) - l = [dist.name for dist in provides_distribution('truffles', '>=1.0')] + l = [dist.name for dist in provides_distribution('truffles', '>=1.0', + use_egg_info=False)] checkLists(l, ['choxie', 'towel-stuff']) l = [dist.name for dist in provides_distribution('strawberry', '0.6', @@ -471,28 +494,33 @@ # Test looking for distributions based on what they obsolete checkLists = lambda x, y: self.assertEqual(sorted(x), sorted(y)) - l = [dist.name for dist in obsoletes_distribution('truffles', '1.0')] + l = [dist.name for dist in obsoletes_distribution('truffles', '1.0', + use_egg_info=False)] checkLists(l, []) l = [dist.name for dist in obsoletes_distribution('truffles', '1.0', use_egg_info=True)] checkLists(l, ['cheese', 'bacon']) - l = [dist.name for dist in obsoletes_distribution('truffles', '0.8')] + l = [dist.name for dist in obsoletes_distribution('truffles', '0.8', + use_egg_info=False)] checkLists(l, ['choxie']) l = [dist.name for dist in obsoletes_distribution('truffles', '0.8', use_egg_info=True)] checkLists(l, ['choxie', 'cheese']) - l = [dist.name for dist in obsoletes_distribution('truffles', '0.9.6')] + l = [dist.name for dist in obsoletes_distribution('truffles', '0.9.6', + use_egg_info=False)] checkLists(l, ['choxie', 'towel-stuff']) l = [dist.name for dist in obsoletes_distribution('truffles', - '0.5.2.3')] + '0.5.2.3', + use_egg_info=False)] checkLists(l, ['choxie', 'towel-stuff']) - l = [dist.name for dist in obsoletes_distribution('truffles', '0.2')] + l = [dist.name for dist in obsoletes_distribution('truffles', '0.2', + use_egg_info=False)] checkLists(l, ['towel-stuff']) @requires_zlib diff --git a/distutils2/tests/test_install.py b/distutils2/tests/test_install.py --- a/distutils2/tests/test_install.py +++ b/distutils2/tests/test_install.py @@ -1,6 +1,7 @@ """Tests for the distutils2.install module.""" import os import logging + from tempfile import mkstemp from distutils2 import install @@ -57,7 +58,7 @@ def list_installed_files(self, **args): if self._files: - return self._real_files + return [(path, 'md5', 0) for path in self._real_files] def get_install(self, **args): return self.list_installed_files() @@ -258,6 +259,25 @@ for key in expect: self.assertEqual(expect[key], dict1[key]) + def test_install_custom_dir(self): + dest = self.mkdtemp() + + project_dir, dist = self.create_dist( + name='Spamlib', version='0.1', + data_files={'spamd': '{scripts}/spamd'}) + + dist.name = MagicMock(return_value='Spamlib') + dist.version = MagicMock(return_value='0.1') + dist.unpack = MagicMock(return_value=project_dir) + + self.write_file([project_dir, 'setup.cfg'], + ("[metadata]\n" + "name = mypackage\n" + "version = 0.1.0\n")) + + install.install_from_infos(dest, install=[dist]) + self.assertEqual(len(os.listdir(dest)), 1) + def test_install_dists_rollback(self): # if one of the distribution installation fails, call uninstall on all # installed distributions. @@ -312,7 +332,7 @@ # assert that the files have been removed for dist in dists: - for f in dist.list_installed_files(): + for f, md5, size in dist.list_installed_files(): self.assertFalse(os.path.exists(f)) finally: install.install_dists = old_install_dists @@ -338,7 +358,7 @@ # assert that the files are in the same place # assert that the files have been removed for dist in remove: - for f in dist.list_installed_files(): + for f, md5, size in dist.list_installed_files(): self.assertTrue(os.path.exists(f)) dist._unlink_installed_files() finally: diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py --- a/distutils2/tests/test_metadata.py +++ b/distutils2/tests/test_metadata.py @@ -10,7 +10,7 @@ from distutils2.tests import unittest from distutils2.tests.support import (LoggingCatcher, TempdirManager, - EnvironRestorer) + EnvironRestorer, requires_docutils) class MetadataTestCase(LoggingCatcher, @@ -48,7 +48,7 @@ self.assertEqual(len(m.items()), 22) m = Metadata(mapping=dict(name='Test', version='1.0')) - self.assertEqual(len(m.items()), 11) + self.assertEqual(len(m.items()), 17) d = dict(m.items()) self.assertRaises(TypeError, Metadata, @@ -250,27 +250,32 @@ self.assertNotIn('Obsoletes', metadata) metadata['Classifier'] = ['ok'] + metadata.set_metadata_version() self.assertEqual(metadata['Metadata-Version'], '1.1') metadata = Metadata() metadata['Download-URL'] = 'ok' + metadata.set_metadata_version() self.assertEqual(metadata['Metadata-Version'], '1.1') metadata = Metadata() metadata['Obsoletes'] = 'ok' + metadata.set_metadata_version() self.assertEqual(metadata['Metadata-Version'], '1.1') del metadata['Obsoletes'] metadata['Obsoletes-Dist'] = 'ok' + metadata.set_metadata_version() self.assertEqual(metadata['Metadata-Version'], '1.2') - - self.assertRaises(MetadataConflictError, metadata.set, - 'Obsoletes', 'ok') + metadata.set('Obsoletes', 'ok') + self.assertRaises(MetadataConflictError, + metadata.set_metadata_version) del metadata['Obsoletes'] del metadata['Obsoletes-Dist'] + metadata.set_metadata_version() metadata['Version'] = '1' - self.assertEqual(metadata['Metadata-Version'], '1.0') + self.assertEqual(metadata['Metadata-Version'], '1.1') # make sure the _best_version function works okay with # non-conflicting fields from 1.1 and 1.2 (i.e. we want only the @@ -279,18 +284,25 @@ metadata = Metadata() metadata['Requires-Python'] = '3' metadata['Classifier'] = ['Programming language :: Python :: 3'] + metadata.set_metadata_version() self.assertEqual(metadata['Metadata-Version'], '1.2') PKG_INFO = os.path.join(os.path.dirname(__file__), 'SETUPTOOLS-PKG-INFO') metadata = Metadata(PKG_INFO) - self.assertEqual(metadata['Metadata-Version'], '1.0') + self.assertEqual(metadata['Metadata-Version'], '1.1') PKG_INFO = os.path.join(os.path.dirname(__file__), 'SETUPTOOLS-PKG-INFO2') metadata = Metadata(PKG_INFO) self.assertEqual(metadata['Metadata-Version'], '1.1') + # make sure an empty list for Obsoletes and Requires-dist gets ignored + metadata['Obsoletes'] = [] + metadata['Requires-dist'] = [] + metadata.set_metadata_version() + self.assertEqual(metadata['Metadata-Version'], '1.1') + # Update the _fields dict directly to prevent 'Metadata-Version' # from being updated by the _set_best_version() method. metadata._fields['Metadata-Version'] = '1.618' @@ -345,10 +357,21 @@ folded_desc = desc.replace('\n', '\n' + (7 * ' ') + '|') self.assertIn(folded_desc, out.getvalue()) + @requires_docutils + def test_description_invalid_rst(self): + # make sure bad rst is well handled in the description attribute + metadata = Metadata() + description = ':funkie:`str`' # mimic Sphinx-specific markup + metadata['description'] = description + missing, warnings = metadata.check(restructuredtext=True) + warning = warnings[0][1] + self.assertIn('funkie', warning) + def test_project_url(self): metadata = Metadata() metadata['Project-URL'] = [('one', 'http://ok')] self.assertEqual(metadata['Project-URL'], [('one', 'http://ok')]) + metadata.set_metadata_version() self.assertEqual(metadata['Metadata-Version'], '1.2') # make sure this particular field is handled properly when written diff --git a/distutils2/tests/test_msvc9compiler.py b/distutils2/tests/test_msvc9compiler.py --- a/distutils2/tests/test_msvc9compiler.py +++ b/distutils2/tests/test_msvc9compiler.py @@ -101,7 +101,7 @@ v = Reg.get_value(path, 'dragfullwindows') self.assertIn(v, ('0', '1', '2')) - import winreg + import _winreg as winreg HKCU = winreg.HKEY_CURRENT_USER keys = Reg.read_keys(HKCU, 'xxxx') self.assertEqual(keys, None) diff --git a/distutils2/tests/test_pypi_wrapper.py b/distutils2/tests/test_pypi_wrapper.py new file mode 100644 --- /dev/null +++ b/distutils2/tests/test_pypi_wrapper.py @@ -0,0 +1,37 @@ +"""Tests for the distutils2.pypi.wrapper module.""" + + +from distutils2.tests import unittest +from distutils2.pypi.wrapper import switch_index_if_fails + + +class Index1(object): + def test(self): + raise Exception("boo") + + +class Index2(object): + def test(self): + return 'OK' + + +class Indexes(object): + _indexes = {'one': Index1(), 'two': Index2()} + + +class TestPyPIWrapper(unittest.TestCase): + + def test_wrapper(self): + index = Indexes._indexes['one'] + func = switch_index_if_fails(getattr(index, 'test'), Indexes) + self.assertEqual(func(), 'OK') + + +def test_suite(): + suite = unittest.TestSuite() + suite.addTest(unittest.makeSuite(TestPyPIWrapper)) + return suite + + +if __name__ == '__main__': + unittest.main(defaultTest='test_suite') diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py --- a/distutils2/tests/test_run.py +++ b/distutils2/tests/test_run.py @@ -2,13 +2,14 @@ import os import sys +import textwrap from io import StringIO from distutils2 import install from distutils2.tests import unittest, support from distutils2.run import main -from distutils2.tests.support import assert_python_ok +from distutils2.tests.support import assert_python_ok, assert_python_failure # setup script that uses __file__ setup_using___file__ = """\ @@ -31,14 +32,7 @@ support.LoggingCatcher, unittest.TestCase): - def setUp(self): - super(RunTestCase, self).setUp() - self.old_argv = sys.argv, sys.argv[:] - - def tearDown(self): - sys.argv = self.old_argv[0] - sys.argv[:] = self.old_argv[1] - super(RunTestCase, self).tearDown() + maxDiff = None # TODO restore the tests removed six months ago and port them to pysetup @@ -62,28 +56,32 @@ def get_pythonpath(self): pythonpath = os.environ.get('PYTHONPATH') - d2parent = os.path.dirname(os.path.dirname(__file__)) + d2parent = os.path.dirname(os.path.dirname(__file__)) # XXX buggy if pythonpath is not None: pythonpath = os.pathsep.join((pythonpath, d2parent)) else: pythonpath = d2parent return pythonpath + def call_pysetup(self, *args): + _, out, err = assert_python_ok('-m', 'distutils2.run', *args, + PYTHONPATH=self.get_pythonpath()) + return out, err + + def call_pysetup_fail(self, *args): + _, out, err = assert_python_failure('-m', 'distutils2.run', *args, + PYTHONPATH=self.get_pythonpath()) + return out, err + def test_show_help(self): # smoke test, just makes sure some help is displayed - status, out, err = assert_python_ok( - '-m', 'distutils2.run', '--help', - PYTHONPATH=self.get_pythonpath()) - self.assertEqual(status, 0) + out, err = self.call_pysetup('--help') self.assertGreater(out, b'') self.assertEqual(err, b'') def test_list_commands(self): - status, out, err = assert_python_ok( - '-m', 'distutils2.run', 'run', - '--list-commands', PYTHONPATH=self.get_pythonpath()) + out, err = self.call_pysetup('run', '--list-commands') # check that something is displayed - self.assertEqual(status, 0) self.assertGreater(out, b'') self.assertEqual(err, b'') @@ -94,7 +92,66 @@ self.assertTrue(build_position, out) self.assertLess(check_position, build_position, out) - # TODO test that custom commands don't break --list-commands + # TODO test that custom commands don't break --list-commands + + def test_unknown_command_option(self): + out, err = self.call_pysetup_fail('run', 'build', '--unknown') + self.assertGreater(out, b'') + # sadly this message comes straight from the getopt module and can't be + # modified to use repr instead of str for the unknown option; to be + # changed when the command line parsers are replaced by something clean + self.assertEqual(err.splitlines(), + [b'error: option --unknown not recognized']) + + def test_invalid_command(self): + out, err = self.call_pysetup_fail('run', 'com#mand') + self.assertGreater(out, b'') + self.assertEqual(err.splitlines(), + [b"error: invalid command name 'com#mand'"]) + + def test_unknown_command(self): + out, err = self.call_pysetup_fail('run', 'invalid_command') + self.assertGreater(out, b'') + self.assertEqual(err.splitlines(), + [b"error: command 'invalid_command' not recognized"]) + + def test_unknown_action(self): + out, err = self.call_pysetup_fail('invalid_action') + self.assertGreater(out, b'') + self.assertEqual(err.splitlines(), + [b"error: action 'invalid_action' not recognized"]) + + def test_setupcfg_parsing(self): + # #14733: pysetup used to parse setup.cfg too late + project_dir = self.mkdtemp() + os.chdir(project_dir) + custompy = textwrap.dedent( + """\ + from distutils2.command.cmd import Command + + class custom(Command): + + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + print('custom: ok') + """) + setupcfg = textwrap.dedent( + """\ + [global] + commands = custom.custom + """) + self.write_file('custom.py', custompy) + self.write_file('setup.cfg', setupcfg) + + out, err = self.call_pysetup('run', 'custom') + self.assertEqual(out.splitlines(), [b'custom: ok']) def test_suite(): diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py --- a/distutils2/tests/test_util.py +++ b/distutils2/tests/test_util.py @@ -19,7 +19,8 @@ get_compiler_versions, _MAC_OS_X_LD_VERSION, byte_compile, find_packages, spawn, get_pypirc_path, generate_pypirc, read_pypirc, resolve_name, iglob, RICH_GLOB, egginfo_to_distinfo, is_setuptools, is_distutils, is_packaging, - get_install_method, cfg_to_args, generate_setup_py, encode_multipart) + get_install_method, cfg_to_args, generate_setup_py, encode_multipart, + parse_requires) from distutils2.tests import support, unittest from distutils2.tests.test_config import SETUP_CFG @@ -377,6 +378,15 @@ self.assertEqual(sorted(res), ['pkg1', 'pkg1.pkg3', 'pkg1.pkg3.pkg6', 'pkg5']) + def test_parse_requires(self): + req_file = os.path.join(os.path.dirname(__file__), 'requires.txt') + expected_requires = ['setuptools', 'zope.browser', 'zope.component', + 'zope.configuration', 'zope.contenttype', 'zope.event', + 'zope.exceptions', 'zope.i18n', 'zope.interface', + 'zope.location', 'zope.proxy', 'zope.security'] + requires = parse_requires(req_file) + self.assertEqual(requires, expected_requires) + def test_resolve_name(self): # test raw module name tmpdir = self.mkdtemp() @@ -744,6 +754,31 @@ self.assertRaises(ValueError, iglob, pattern) +PKG_INFO = '''\ +Metadata-Version: 1.1 +Name: hello +Version: 0.1.1 +Summary: Hello World +Home-page: https://example.com +Author: John Doe +Author-email: j.doe at example.com +License: UNKNOWN +Download-URL: https://example.com/tarball/master +Description: UNKNOWN +Platform: Any +Classifier: Development Status :: 3 - Alpha +Classifier: License :: OSI Approved :: Apache Software License +Classifier: Programming Language :: Python +Classifier: Programming Language :: Python :: 2 +Classifier: Programming Language :: Python :: 2.7 +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.2 +Classifier: Intended Audience :: Developers +Classifier: Environment :: Console +Provides: hello +''' + + class EggInfoToDistInfoTestCase(support.TempdirManager, support.LoggingCatcher, unittest.TestCase): @@ -762,10 +797,14 @@ dirs = [egginfo] files = ['hello.py', 'hello.pyc'] extra_metadata = ['dependency_links.txt', 'entry_points.txt', - 'not-zip-safe', 'PKG-INFO', 'top_level.txt', - 'SOURCES.txt'] + 'not-zip-safe', ('PKG-INFO', PKG_INFO), + 'top_level.txt', 'SOURCES.txt'] for f in extra_metadata: - files.append(os.path.join(egginfo, f)) + if isinstance(f, tuple): + f, content = f + else: + content = 'XXX' + files.append((os.path.join(egginfo, f), content)) tempdir, record_file = self.build_dist_tree(files, dirs) distinfo_path = os.path.join(tempdir, distinfo) @@ -784,7 +823,7 @@ distinfo = 'hello-0.1.1-py3.3.dist-info' egginfo = 'hello-0.1.1-py3.3.egg-info' # egginfo is a file in distutils which contains the metadata - files = ['hello.py', 'hello.pyc', egginfo] + files = ['hello.py', 'hello.pyc', (egginfo, PKG_INFO)] tempdir, record_file = self.build_dist_tree(files, dirs=[]) distinfo_path = os.path.join(tempdir, distinfo) @@ -792,6 +831,7 @@ metadata_file_paths = self.get_metadata_file_paths(distinfo_path) egginfo_to_distinfo(record_file) + # test that directories and files get created self.assertTrue(os.path.isdir(distinfo_path)) self.assertTrue(os.path.isfile(egginfo_path)) @@ -808,9 +848,14 @@ os.makedirs(path) dir_paths.append(path) for f in files: + if isinstance(f, (list, tuple)): + f, content = f + else: + content = '' + path = os.path.join(tempdir, f) with open(path, 'w') as _f: - _f.write(f) + _f.write(content) file_paths.append(path) with open(record_file_path, 'w') as record_file: diff --git a/distutils2/tests/test_version.py b/distutils2/tests/test_version.py --- a/distutils2/tests/test_version.py +++ b/distutils2/tests/test_version.py @@ -18,8 +18,8 @@ (V('1.2c4'), '1.2c4'), (V('4.17rc2'), '4.17rc2'), (V('1.2.3.4'), '1.2.3.4'), - (V('1.2.3.4.0b3'), '1.2.3.4b3'), - (V('1.2.0.0.0'), '1.2'), + (V('1.2.3.4.0b3', drop_trailing_zeros=True), '1.2.3.4b3'), + (V('1.2.0.0.0', drop_trailing_zeros=True), '1.2'), (V('1.0.dev345'), '1.0.dev345'), (V('1.0.post456.dev623'), '1.0.post456.dev623')) @@ -249,6 +249,12 @@ for version in other_versions: self.assertFalse(V(version).is_final) + def test_micro_predicate(self): + self.assertNotEqual(V('3.4.0'), V('3.4')) + predicate = VersionPredicate('zope.event (3.4.0)') + self.assertTrue(predicate.match('3.4.0')) + self.assertFalse(predicate.match('3.4.1')) + class VersionWhiteBoxTestCase(unittest.TestCase): diff --git a/distutils2/util.py b/distutils2/util.py --- a/distutils2/util.py +++ b/distutils2/util.py @@ -19,6 +19,7 @@ from distutils2.errors import (PackagingPlatformError, PackagingFileError, PackagingExecError, InstallationException, PackagingInternalError) +from distutils2.metadata import Metadata from distutils2._backport import shutil, sysconfig from distutils2._backport.misc import cache_from_source @@ -1152,6 +1153,69 @@ return record_path +def parse_requires(req_path): + """Create a list of dependencies from a requires.txt file. + + *req_path* must be the path to a setuptools-produced requires.txt file. + """ + + # reused from Distribute's pkg_resources + def yield_lines(strs): + """Yield non-empty/non-comment lines of a string or sequence""" + if isinstance(strs, str): + for s in strs.splitlines(): + s = s.strip() + # skip blank lines/comments + if s and not s.startswith('#'): + yield s + else: + for ss in strs: + for s in yield_lines(ss): + yield s + + _REQUIREMENT = re.compile( + r'(?P[-A-Za-z0-9_.]+)\s*' + r'(?P(?:<|<=|!=|==|>=|>)[-A-Za-z0-9_.]+)?\s*' + r'(?P(?:\s*,\s*(?:<|<=|!=|==|>=|>)[-A-Za-z0-9_.]+)*)\s*' + r'(?P\[.*\])?') + + reqs = [] + try: + with open(req_path, 'r') as fp: + requires = fp.read() + except IOError: + return None + + for line in yield_lines(requires): + if line.startswith('['): + logger.warning('extensions in requires.txt are not supported') + break + else: + match = _REQUIREMENT.match(line.strip()) + if not match: + # this happens when we encounter extras; since they + # are written at the end of the file we just exit + break + else: + if match.group('extras'): + # msg = ('extra requirements are not supported ' + # '(used by %r %s)', self.name, self.version) + msg = 'extra requirements are not supported' + logger.warning(msg) + name = match.group('name') + version = None + if match.group('first'): + version = match.group('first') + if match.group('rest'): + version += match.group('rest') + version = version.replace(' ', '') # trim spaces + if version is None: + reqs.append(name) + else: + reqs.append('%s (%s)' % (name, version)) + return reqs + + def egginfo_to_distinfo(record_file, installer=_DEFAULT_INSTALLER, requested=False, remove_egginfo=False): """Create files and directories required for PEP 376 @@ -1180,6 +1244,22 @@ metadata_path = distinfo['metadata_path'] logger.info('creating %s', metadata_path) shutil.copy2(distinfo['metadata'], metadata_path) + # add requirements and output metadata + requires = None + req_path = os.path.join(distinfo_dir, 'requires.txt') + requires = parse_requires(req_path) + + # adapting the metadata + metadata = Metadata(path=metadata_path) + if metadata['Provides'] != []: + metadata['Provides-Dist'] = metadata['Provides'] + metadata['Provides'] = [] + + if requires is not None: + # create a metadata instance to handle the reqs injection + metadata['Requires-Dist'] = requires + + metadata.write(metadata_path) installer_path = distinfo['installer_path'] logger.info('creating %s', installer_path) diff --git a/distutils2/version.py b/distutils2/version.py --- a/distutils2/version.py +++ b/distutils2/version.py @@ -57,7 +57,8 @@ 1.2a # release level must have a release serial 1.2.3b """ - def __init__(self, s, error_on_huge_major_num=True): + def __init__(self, s, error_on_huge_major_num=True, + drop_trailing_zeros=False): """Create a NormalizedVersion instance from a version string. @param s {str} The version string. @@ -74,8 +75,12 @@ and, e.g. downstream Linux package managers, will forever remove the possibility of using a version number like "1.0" (i.e. where the major number is less than that huge major number). + @param drop_trailing_zeros {bool} Whether to drop trailing zeros + + from the returned list. Default True. """ self.is_final = True # by default, consider a version as final. + self.drop_trailing_zeros = drop_trailing_zeros self._parse(s, error_on_huge_major_num) @classmethod @@ -93,7 +98,7 @@ parts = [] # main version - block = self._parse_numdots(groups['version'], s, False, 2) + block = self._parse_numdots(groups['version'], s, 2) extraversion = groups.get('extraversion') if extraversion not in ('', None): block += self._parse_numdots(extraversion[1:], s) @@ -130,16 +135,13 @@ raise HugeMajorVersionNumError("huge major version number, %r, " "which might cause future problems: %r" % (self.parts[0][0], s)) - def _parse_numdots(self, s, full_ver_str, drop_trailing_zeros=True, - pad_zeros_length=0): + def _parse_numdots(self, s, full_ver_str, pad_zeros_length=0): """Parse 'N.N.N' sequences, return a list of ints. @param s {str} 'N.N.N...' sequence to be parsed @param full_ver_str {str} The full version string from which this comes. Used for error strings. - @param drop_trailing_zeros {bool} Whether to drop trailing zeros - from the returned list. Default True. - @param pad_zeros_length {int} The length to which to pad the + @param pad_zeros_length {int} The length to which to pad the returned list with zeros, if necessary. Default 0. """ nums = [] @@ -148,7 +150,7 @@ raise IrrationalVersionError("cannot have leading zero in " "version number segment: '%s' in %r" % (n, full_ver_str)) nums.append(int(n)) - if drop_trailing_zeros: + if self.drop_trailing_zeros: while nums and nums[-1] == 0: nums.pop() while len(nums) < pad_zeros_length: diff --git a/docs/Makefile b/docs/Makefile deleted file mode 100644 --- a/docs/Makefile +++ /dev/null @@ -1,89 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = build - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source - -.PHONY: help clean html dirhtml pickle json htmlhelp qthelp latex changes linkcheck doctest - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - -clean: - -rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Distutils2.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Distutils2.qhc" - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \ - "run these through (pdf)latex." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." diff --git a/docs/design/configfile.rst b/docs/design/configfile.rst deleted file mode 100644 --- a/docs/design/configfile.rst +++ /dev/null @@ -1,132 +0,0 @@ -.. _setup-config: - -************************************ -Writing the Setup Configuration File -************************************ - -Often, it's not possible to write down everything needed to build a distribution -*a priori*: you may need to get some information from the user, or from the -user's system, in order to proceed. As long as that information is fairly -simple---a list of directories to search for C header files or libraries, for -example---then providing a configuration file, :file:`setup.cfg`, for users to -edit is a cheap and easy way to solicit it. Configuration files also let you -provide default values for any command option, which the installer can then -override either on the command line or by editing the config file. - -The setup configuration file is a useful middle-ground between the setup script ----which, ideally, would be opaque to installers [#]_---and the command line to -the setup script, which is outside of your control and entirely up to the -installer. In fact, :file:`setup.cfg` (and any other Distutils configuration -files present on the target system) are processed after the contents of the -setup script, but before the command line. This has several useful -consequences: - -.. If you have more advanced needs, such as determining which extensions to - build based on what capabilities are present on the target system, then you - need the Distutils auto-configuration facility. This started to appear in - Distutils 0.9 but, as of this writing, isn't mature or stable enough yet - for real-world use. - -* installers can override some of what you put in :file:`setup.py` by editing - :file:`setup.cfg` - -* you can provide non-standard defaults for options that are not easily set in - :file:`setup.py` - -* installers can override anything in :file:`setup.cfg` using the command-line - options to :file:`setup.py` - -The basic syntax of the configuration file is simple:: - - [command] - option=value - ... - -where *command* is one of the Distutils commands (e.g. :command:`build_py`, -:command:`install`), and *option* is one of the options that command supports. -Any number of options can be supplied for each command, and any number of -command sections can be included in the file. Blank lines are ignored, as are -comments, which run from a ``'#'`` character until the end of the line. Long -option values can be split across multiple lines simply by indenting the -continuation lines. - -You can find out the list of options supported by a particular command with the -universal :option:`--help` option, e.g. :: - - > python setup.py --help build_ext - [...] - Options for 'build_ext' command: - --build-lib (-b) directory for compiled extension modules - --build-temp (-t) directory for temporary files (build by-products) - --inplace (-i) ignore build-lib and put compiled extensions into the - source directory alongside your pure Python modules - --include-dirs (-I) list of directories to search for header files - --define (-D) C preprocessor macros to define - --undef (-U) C preprocessor macros to undefine - --swig-opts list of SWIG command-line options - [...] - -.. XXX do we want to support ``setup.py --help metadata``? - -Note that an option spelled :option:`--foo-bar` on the command line is spelled -:option:`foo_bar` in configuration files. - -For example, say you want your extensions to be built "in-place"---that is, you -have an extension :mod:`pkg.ext`, and you want the compiled extension file -(:file:`ext.so` on Unix, say) to be put in the same source directory as your -pure Python modules :mod:`pkg.mod1` and :mod:`pkg.mod2`. You can always use the -:option:`--inplace` option on the command line to ensure this:: - - python setup.py build_ext --inplace - -But this requires that you always specify the :command:`build_ext` command -explicitly, and remember to provide :option:`--inplace`. An easier way is to -"set and forget" this option, by encoding it in :file:`setup.cfg`, the -configuration file for this distribution:: - - [build_ext] - inplace=1 - -This will affect all builds of this module distribution, whether or not you -explicitly specify :command:`build_ext`. If you include :file:`setup.cfg` in -your source distribution, it will also affect end-user builds---which is -probably a bad idea for this option, since always building extensions in-place -would break installation of the module distribution. In certain peculiar cases, -though, modules are built right in their installation directory, so this is -conceivably a useful ability. (Distributing extensions that expect to be built -in their installation directory is almost always a bad idea, though.) - -Another example: certain commands take a lot of options that don't change from -run to run; for example, :command:`bdist_rpm` needs to know everything required -to generate a "spec" file for creating an RPM distribution. Some of this -information comes from the setup script, and some is automatically generated by -the Distutils (such as the list of files installed). But some of it has to be -supplied as options to :command:`bdist_rpm`, which would be very tedious to do -on the command line for every run. Hence, here is a snippet from the Distutils' -own :file:`setup.cfg`:: - - [bdist_rpm] - release = 1 - packager = Greg Ward - doc_files = CHANGES.txt - README.txt - USAGE.txt - doc/ - examples/ - -Note that the :option:`doc_files` option is simply a whitespace-separated string -split across multiple lines for readability. - - -.. seealso:: - - :ref:`inst-config-syntax` in "Installing Python Modules" - More information on the configuration files is available in the manual for - system administrators. - - -.. rubric:: Footnotes - -.. [#] This ideal probably won't be achieved until auto-configuration is fully - supported by the Distutils. - diff --git a/docs/design/pep-0376.txt b/docs/design/pep-0376.txt deleted file mode 100644 --- a/docs/design/pep-0376.txt +++ /dev/null @@ -1,698 +0,0 @@ -PEP: 376 -Title: Changing the .egg-info structure -Version: $Revision: 75414 $ -Last-Modified: $Date: 2009-10-14 14:39:17 -0400 (Wed, 14 Oct 2009) $ -Author: Tarek Ziad? -Status: Draft -Type: Standards Track -Content-Type: text/x-rst -Created: 22-Feb-2009 -Python-Version: 2.7, 3.2 -Post-History: - - -Abstract -======== - -This PEP proposes various enhancements for Distutils: - -- A new format for the .egg-info structure. -- Some APIs to read the metadata of a distribution. -- A replacement PEP 262. -- An uninstall feature. - -Definitions -=========== - -A **distribution** is a collection of files, which can be Python modules, -extensions, or data. A distribution is managed by a special module called -`setup.py` which contains a call to the `distutils.core.setup` function. -The arguments passed to that function describe the distribution, like -its `name`, its `version`, and so on. - -Distutils provides, among other things, **commands** that can be called -through the shell using the `setup.py` script. An `sdist` command is provided -for instance to create a source distribution archive. An `install` command -is also provided to perform an installation of the distribution in the Python -installation the script is invoked with:: - - $ python setup.py install - -See the Distutils [#distutils]_ documentation for more information. - -Once installed, the elements are located in various places in the system, like: - -- In Python's site-packages (Python modules, Python modules organized into - packages, Extensions, etc.) -- In Python's `include` directory. -- In Python's `bin` or `Script` directory. -- Etc. - -Rationale -========= - -There are two problems right now in the way distributions are installed in -Python: - -- There are too many ways to do it. -- There is no API to get the metadata of installed distributions. - -How distributions are installed -------------------------------- - -Right now, when a distribution is installed in Python, the elements it -contains are installed in various directories. - -The pure Python code, for instance, is installed in the `purelib` directory -which is located in the Python installation at ``lib/python2.6/site-packages`` -for example under Unix-like systems or Mac OS X, and in ``Lib\site-packages`` -under Windows. This is done with the Distutils `install` command, which calls -various subcommands. - -The `install_egg_info` subcommand is called during this process in order to -create an `.egg-info` file in the `purelib` directory. - -For example, for the `docutils` distribution, which contains one package an -extra module and executable scripts, three elements are installed in -`site-packages`: - -- `docutils`: The ``docutils`` package. -- `roman.py`: An extra module used by `docutils`. -- `docutils-0.5-py2.6.egg-info`: A file containing the distribution metadata - as described in PEP 314 [#pep314]_. This file corresponds to the file - called `PKG-INFO`, built by the `sdist` command. - -Some executable scripts, such as `rst2html.py`, are also be added in the -`bin` directory of the Python installation. - -Another project called `setuptools` [#setuptools]_ has two other formats -to install distributions, called `EggFormats` [#eggformats]_: - -- a self-contained `.egg` directory, that contains all the distribution files - and the distribution metadata in a file called `PKG-INFO` in a subdirectory - called `EGG-INFO`. `setuptools` creates other fils in that directory that can - be considered as complementary metadata. - -- a `.egg-info` directory installed in `site-packages`, that contains the same - files `EGG-INFO` has in the `.egg` format. - -The first format is automatically used when you install a distribution that -uses the ``setuptools.setup`` function in its setup.py file, instead of -the ``distutils.core.setup`` one. - -The `setuptools` project also provides an executable script called -`easy_install` [#easyinstall]_ that installs all distributions, including -distutils-based ones in self-contained `.egg` directories. - -If you want to have a standalone `.egg.info` directory distributions, e.g. -the second `setuptools` format, you have to force it when you work -with a setuptools-based distribution or with the `easy_install` script. -You can force it by using the `-?single-version-externally-managed` option -**or** the `--root` option. - -This option is used by : - -- the `pip` [#pip]_ installer -- the Fedora packagers [#fedora]_. -- the Debian packagers [#debian]_. - -Uninstall information ---------------------- - -Distutils doesn't provide an `uninstall` command. If you want to uninstall -a distribution, you have to be a power user and remove the various elements -that were installed, and then look over the `.pth` file to clean them if -necessary. - -And the process differs depending on the tools you have used to install the -distribution and if the distribution's `setup.py` uses Distutils or -Setuptools. - -Under some circumstances, you might not be able to know for sure that you -have removed everything, or that you didn't break another distribution by -removing a file that is shared among several distributions. - -But there's a common behavior: when you install a distribution, files are -copied in your system. And it's possible to keep track of these files for -later removal. - -What this PEP proposes ----------------------- - -To address those issues, this PEP proposes a few changes: - -- A new `.egg-info` structure using a directory, based on one format of - the `EggFormats` standard from `setuptools`. -- New APIs in `pkgutil` to be able to query the information of installed - distributions. -- A de-facto replacement for PEP 262 -- An uninstall function and an uninstall script in Distutils. - - -.egg-info becomes a directory -============================= - -As explained earlier, the `EggFormats` standard from `setuptools` proposes two -formats to install the metadata information of a distribution: - -- A self-contained directory that can be zipped or left unzipped and contains - the distribution files *and* an `.egg-info` directory containing the - metadata. - -- A distinct `.egg-info` directory located in the site-packages directory, - with the metadata inside. - -This PEP proposes to keep just one format and make it the standard way to -install the metadata of a distribution : a distinct `.egg-info` directory -located in the site-packages directory, containing the metadata. - -This `.egg-info` directory contains a `PKG-INFO` file built by the -`write_pkg_file` method of the `Distribution` class in Distutils. - -This change does not impact Python itself because the metadata files are not -used anywhere yet in the standard library besides Distutils. - -It does impact the `setuptools` and `pip` projects, but given the fact that -they already work with a directory that contains a `PKG-INFO` file, the change -will have no deep consequences. - -Let's take an example of the new format with the `docutils` distribution. -The elements installed in `site-packages` are:: - - - docutils/ - - roman.py - - docutils-0.5.egg-info/ - PKG-INFO - -The syntax of the egg-info directory name is as follows:: - - name + '-' + version + '.egg-info' - -The egg-info directory name is created using a new function called -``egginfo_dirname(name, version)`` added to ``pkgutil``. ``name`` is -converted to a standard distribution name by replacing any runs of -non-alphanumeric characters with a single '-'. ``version`` is converted -to a standard version string. Spaces become dots, and all other -non-alphanumeric characters (except dots) become dashes, with runs of -multiple dashes condensed to a single dash. Both attributes are then -converted into their filename-escaped form, i.e. any '-' characters are -replaced with '_' other than the one in 'egg-info' and the one -separating the name from the version number. - -Examples:: - - >>> egginfo_dirname('docutils', '0.5') - 'docutils-0.5.egg-info' - - >>> egginfo_dirname('python-ldap', '2.5') - 'python_ldap-2.5.egg-info' - - >>> egginfo_dirname('python-ldap', '2.5 a---5') - 'python_ldap-2.5.a_5.egg-info' - -Adding a RECORD file in the .egg-info directory -=============================================== - -A `RECORD` file is added inside the `.egg-info` directory at installation -time when installing a source distribution using the `install` command. -Notice that when installing a binary distribution created with `bdist` command -or a `bdist`-based command, the `RECORD` file will be installed as well since -these commands use the `install` command to create a binary distributions. - -The `RECORD` file holds the list of installed files. These correspond -to the files listed by the `record` option of the `install` command, and will -be generated by default. This allows the implementation of an uninstallation -feature, as explained later in this PEP. The `install` command also provides -an option to prevent the `RECORD` file from being written and this option -should be used when creating system packages. - -Third-party installation tools also should not overwrite or delete files -that are not in a RECORD file without prompting or warning. - -This RECORD file is inspired from PEP 262 FILES [#pep262]_. - -The RECORD format ------------------ - -The `RECORD` file is a CSV file, composed of records, one line per -installed file. The ``csv`` module is used to read the file, with -these options: - -- field delimiter : `,` -- quoting char : `"`. -- line terminator : ``os.linesep`` (so ``\r\n`` or ``\n``) - -Each record is composed of three elements. - -- the file's full **path** - - - if the installed file is located in the directory where the `.egg-info` - directory of the package is located, it's a '/'-separated relative - path, no matter what the target system is. This makes this information - cross-compatible and allows simple installations to be relocatable. - - - if the installed file is located under ``sys.prefix`` or - `sys.exec_prefix``, it's a it's a '/'-separated relative path prefixed - by the `$PREFIX` or the `$EXEC_PREFIX` string. The `install` command - decides which prefix to use depending on the files. For instance if - it's an executable script defined in the `scripts` option of the - setup script, `$EXEC_PREFIX` will be used. If `install` doesn't know - which prefix to use, `$PREFIX` is preferred. - -- the **MD5** hash of the file, encoded in hex. Notice that `pyc` and `pyo` - generated files don't have any hash because they are automatically produced - from `py` files. So checking the hash of the corresponding `py` file is - enough to decide if the file and its associated `pyc` or `pyo` files have - changed. - -- the file's size in bytes - -The ``csv`` module is used to generate this file, so the field separator is -",". Any "," characters found within a field is escaped automatically by ``csv``. - -When the file is read, the `U` option is used so the universal newline -support (see PEP 278 [#pep278]_) is activated, avoiding any trouble -reading a file produced on a platform that uses a different new line -terminator. - -Example -------- - -Back to our `docutils` example, we now have:: - - - docutils/ - - roman.py - - docutils-0.5.egg-info/ - PKG-INFO - RECORD - -And the RECORD file contains (extract):: - - docutils/__init__.py,b690274f621402dda63bf11ba5373bf2,9544 - docutils/core.py,9c4b84aff68aa55f2e9bf70481b94333,66188 - roman.py,a4b84aff68aa55f2e9bf70481b943D3,234 - $EXEC_PREFIX/bin/rst2html.py,a4b84aff68aa55f2e9bf70481b943D3,234 - docutils-0.5.egg-info/PKG-INFO,6fe57de576d749536082d8e205b77748,195 - docutils-0.5.egg-info/RECORD - -Notice that: - -- the `RECORD` file can't contain a hash of itself and is just mentioned here -- `docutils` and `docutils-0.5.egg-info` are located in `site-packages` so the file - paths are relative to it. - -Adding an INSTALLER file in the .egg-info directory -=================================================== - -The `install` command has a new option called `installer`. This option -is the name of the tool used to invoke the installation. It's an normalized -lower-case string matching `[a-z0-9_\-\.]`. - - $ python setup.py install --installer=pkg-system - -It defaults to `distutils` if not provided. - -When a distribution is installed, the INSTALLER file is generated in the -`.egg-info` directory with this value, to keep track of **who** installed the -distribution. The file is a single-line text file. - -Adding a REQUESTED file in the .egg-info directory -================================================== - -If a distribution is installed by direct user request (the usual -case), a file REQUESTED is added to the .egg-info directory of the -installed distribution. The REQUESTED file may be empty, or may -contain a marker comment line beginning with the "#" character. - -If an install tool installs a distribution automatically, as a -dependency of another distribution, the REQUESTED file should not be -created. - -The ``install`` command of distutils by default creates the REQUESTED -file. It accepts --requested and --no-requested options to explicitly -specify whether the file is created. - -If a package that was already installed on the system as a dependency -is later installed by name, the distutils ``install`` command will -create the REQUESTED file in the .egg-info directory of the existing -installation. - -Rationale ---------- - -Some install tools automatically detect unfulfilled dependencies and -install them. These tools may also want to be able to alert the user -if distributions are left on the system in an "orphan" state: -installed as a dependency of another distribution, which has since -been removed. - -In order to provide information about orphaned dependencies, knowing -the dependency graph for installed distributions does not suffice (a -package that is not required by any other package may be on the system -because the user needs it directly). It is also necessary to know, for -each installed package, one additional bit of information: whether it -was installed "by request" or solely as a dependency. - -Each (un)install tool could of course record that bit in its own -separate metadata cache, but this will break down if multiple tools -are used to work with installed packages on the same system. If -distutils takes care of this bit of metadata in a standard way, -multiple tools can cooperate and correctly handle orphaned -dependencies. - -(In contrast, it is not necessary for distutils to record or manage -the full dependency graph for installed packages, because the list of -installed packages and their dependency metadata, standardized in PEP -345, allow any tool to independently calculate the dependency graph.) - -An (un)installer tool which works with dependencies could use the -REQUESTED metadata for orphan detection as follows: an orphaned -distribution is any installed distribution that doesn't have a -REQUESTED file and is not required by any other installed -distribution. - -The availability of the REQUESTED metadata of course does not obligate -any tool to provide this orphan-detection feature, or to implement it -in a certain way; for instance, distutils has no opinion about whether -tools should automatically remove newly-orphaned dependencies at -uninstall time. - - -New APIs in pkgutil -=================== - -To use the `.egg-info` directory content, we need to add in the standard -library a set of APIs. The best place to put these APIs is `pkgutil`. - -Query functions ---------------- - -The new functions added in the ``pkgutil`` are : - -- ``get_distributions()`` -> iterator of ``Distribution`` instances. - - Provides an iterator that looks for ``.egg-info`` directories in - ``sys.path`` and returns ``Distribution`` instances for - each one of them. - -- ``get_distribution(name)`` -> ``Distribution`` or None. - - Scans all elements in ``sys.path`` and looks for all directories ending with - ``.egg-info``. Returns a ``Distribution`` corresponding to the - ``.egg-info`` directory that contains a PKG-INFO that matches `name` - for the `name` metadata. - - Notice that there should be at most one result. The first result found - is returned. If the directory is not found, returns None. - -- ``get_file_users(path)`` -> iterator of ``Distribution`` instances. - - Iterates over all distributions to find out which distributions uses ``path``. - ``path`` can be a local absolute path or a relative '/'-separated path. - -Distribution class ------------------- - -A new class called ``Distribution`` is created with the path of the -`.egg-info` directory provided to the constructor. It reads the metadata -contained in `PKG-INFO` when it is instantiated. - -``Distribution(path)`` -> instance - - Creates a ``Distribution`` instance for the given ``path``. - -``Distribution`` provides the following attributes: - -- ``name``: The name of the distribution. - -- ``metadata``: A ``Metadata`` instance loaded with the - distribution's PKG-INFO file. - -- ``requested``: A boolean that indicates whether the REQUESTED - metadata file is present (in other words, whether the package was - installed by user request). - -And following methods: - -- ``get_installed_files(local=False)`` -> iterator of (path, md5, size) - - Iterates over the `RECORD` entries and return a tuple ``(path, md5, size)`` - for each line. If ``local`` is ``True``, the path is transformed into a - local absolute path. Otherwise the raw value from `RECORD` is returned. - - A local absolute path is an absolute path in which occurrences of '/' - have been replaced by the system separator given by ``os.sep``. - -- ``uses(path)`` -> Boolean - - Returns ``True`` if ``path`` is listed in `RECORD`. ``path`` - can be a local absolute path or a relative '/'-separated path. - -- ``get_egginfo_file(path, binary=False)`` -> file object - - Returns a file located under the `.egg-info` directory. - - Returns a ``file`` instance for the file pointed by ``path``. - - ``path`` has to be a '/'-separated path relative to the `.egg-info` - directory or an absolute path. - - If ``path`` is an absolute path and doesn't start with the `.egg-info` - directory path, a ``DistutilsError`` is raised. - - If ``binary`` is ``True``, opens the file in read-only binary mode (`rb`), - otherwise opens it in read-only mode (`r`). - -- ``get_egginfo_files(local=False)`` -> iterator of paths - - Iterates over the `RECORD` entries and return paths for each line if the path - is pointing a file located in the `.egg-info` directory or one of its - subdirectory. - - If ``local`` is ``True``, each path is transformed into a - local absolute path. Otherwise the raw value from `RECORD` is returned. - - -Notice that the API is organized in five classes that work with directories -and Zip files (so it works with files included in Zip files, see PEP 273 for -more details [#pep273]_). These classes are described in the documentation -of the prototype implementation for interested readers [#prototype]_. - -Usage example -------------- - -Let's use some of the new APIs with our `docutils` example:: - - >>> from pkgutil import get_distribution, get_file_users - >>> dist = get_distribution('docutils') - >>> dist.name - 'docutils' - >>> dist.metadata.version - '0.5' - - >>> for path, hash, size in dist.get_installed_files():: - ... print '%s %s %d' % (path, hash, size) - ... - docutils/__init__.py b690274f621402dda63bf11ba5373bf2 9544 - docutils/core.py 9c4b84aff68aa55f2e9bf70481b94333 66188 - roman.py a4b84aff68aa55f2e9bf70481b943D3 234 - /usr/local/bin/rst2html.py a4b84aff68aa55f2e9bf70481b943D3 234 - docutils-0.5.egg-info/PKG-INFO 6fe57de576d749536082d8e205b77748 195 - docutils-0.5.egg-info/RECORD None None - - >>> dist.uses('docutils/core.py') - True - - >>> dist.uses('/usr/local/bin/rst2html.py') - True - - >>> dist.get_egginfo_file('PKG-INFO') - - - >>> dist.requested - True - -PEP 262 replacement -=================== - -In the past an attempt was made to create a installation database (see PEP 262 -[#pep262]_). - -Extract from PEP 262 Requirements: - - " We need a way to figure out what distributions, and what versions of - those distributions, are installed on a system..." - - -Since the APIs proposed in the current PEP provide everything needed to meet -this requirement, PEP 376 replaces PEP 262 and becomes the official -`installation database` standard. - -The new version of PEP 345 (XXX work in progress) extends the Metadata -standard and fullfills the requirements described in PEP 262, like the -`REQUIRES` section. - -Adding an Uninstall function -============================ - -Distutils already provides a very basic way to install a distribution, which -is running the `install` command over the `setup.py` script of the -distribution. - -Distutils will provide a very basic ``uninstall`` function, that is added -in ``distutils.util`` and takes the name of the distribution to uninstall -as its argument. ``uninstall`` uses the APIs described earlier and remove all -unique files, as long as their hash didn't change. Then it removes empty -directories left behind. - -``uninstall`` returns a list of uninstalled files:: - - >>> from distutils.util import uninstall - >>> uninstall('docutils') - ['/opt/local/lib/python2.6/site-packages/docutils/core.py', - ... - '/opt/local/lib/python2.6/site-packages/docutils/__init__.py'] - -If the distribution is not found, a ``DistutilsUninstallError`` is be raised. - -Filtering ---------- - -To make it a reference API for third-party projects that wish to control -how `uninstall` works, a second callable argument can be used. It's -called for each file that is removed. If the callable returns `True`, the -file is removed. If it returns False, it's left alone. - -Examples:: - - >>> def _remove_and_log(path): - ... logging.info('Removing %s' % path) - ... return True - ... - >>> uninstall('docutils', _remove_and_log) - - >>> def _dry_run(path): - ... logging.info('Removing %s (dry run)' % path) - ... return False - ... - >>> uninstall('docutils', _dry_run) - -Of course, a third-party tool can use ``pkgutil`` APIs to implement -its own uninstall feature. - -Installer marker ----------------- - -As explained earlier in this PEP, the `install` command adds an `INSTALLER` -file in the `.egg-info` directory with the name of the installer. - -To avoid removing distributions that where installed by another packaging system, -the ``uninstall`` function takes an extra argument ``installer`` which default -to ``distutils``. - -When called, ``uninstall`` controls that the ``INSTALLER`` file matches -this argument. If not, it raises a ``DistutilsUninstallError``:: - - >>> uninstall('docutils') - Traceback (most recent call last): - ... - DistutilsUninstallError: docutils was installed by 'cool-pkg-manager' - - >>> uninstall('docutils', installer='cool-pkg-manager') - -This allows a third-party application to use the ``uninstall`` function -and strongly suggest that no other program remove a distribution it has -previously installed. This is useful when a third-party program that relies -on Distutils APIs does extra steps on the system at installation time, -it has to undo at uninstallation time. - -Adding an Uninstall script -========================== - -An `uninstall` script is added in Distutils. and is used like this:: - - $ python -m distutils.uninstall packagename - -Notice that script doesn't control if the removal of a distribution breaks -another distribution. Although it makes sure that all the files it removes -are not used by any other distribution, by using the uninstall function. - -Also note that this uninstall script pays no attention to the -REQUESTED metadata; that is provided only for use by external tools to -provide more advanced dependency management. - -Backward compatibility and roadmap -================================== - -These changes don't introduce any compatibility problems with the previous -version of Distutils, and will also work with existing third-party tools. - -The plan is to include the functionality outlined in this PEP in distutils for -Python 2.7 and Python 3.2. A backport of the new distutils for 2.5, 2.6, 3.0 -and 3.1 is provided so people can benefit from these new features. - -Distributions installed using existing, pre-standardization formats do not have -the necessary metadata available for the new API, and thus will be -ignored. Third-party tools may of course continue to support previous -formats in addition to the new format, in order to ease the transition. - - -References -========== - -.. [#distutils] - http://docs.python.org/distutils - -.. [#pep262] - http://www.python.org/dev/peps/pep-0262 - -.. [#pep314] - http://www.python.org/dev/peps/pep-0314 - -.. [#setuptools] - http://peak.telecommunity.com/DevCenter/setuptools - -.. [#easyinstall] - http://peak.telecommunity.com/DevCenter/EasyInstall - -.. [#pip] - http://pypi.python.org/pypi/pip - -.. [#eggformats] - http://peak.telecommunity.com/DevCenter/EggFormats - -.. [#pep273] - http://www.python.org/dev/peps/pep-0273 - -.. [#pep278] - http://www.python.org/dev/peps/pep-0278 - -.. [#fedora] - http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools - -.. [#debian] - http://wiki.debian.org/DebianPython/NewPolicy - -.. [#prototype] - http://bitbucket.org/tarek/pep376/ - -Acknowledgements -================ - -Jim Fulton, Ian Bicking, Phillip Eby, and many people at Pycon and Distutils-SIG. - -Copyright -========= - -This document has been placed in the public domain. - - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: diff --git a/docs/design/wiki.rst b/docs/design/wiki.rst deleted file mode 100644 --- a/docs/design/wiki.rst +++ /dev/null @@ -1,637 +0,0 @@ -====================== -PyCon packaging sprint -====================== - -.. contents:: - -We sprinted on Distutils2 at the US Pycon 2010 sessions. This is one -of the results of that work. - -Distutils is used by both developers and packagers. - -:developers: People who write code in python. The code may be - distributed as an sdist or a variety of binary versions. The - developer presently uses setup.py to tell distutils how to build - the sdist and binary packages. -:packagers: People who package sdists into binary packages for a Linux - distribution. (Is there an equivalent role for people who make - Windows binary packages? Or do developers generally do that for - their own packages?) - ----------------------- -Problems for packagers ----------------------- - -We identified specific problems packagers face when creating deb/rpm: - -The problem this document is intended to solve: **want to lay out files across the filesystem in a FHS-compliant way** - -Problems that are fixed simply by not using setuptools: - -- get rid of ez_setup.py - -- breakages using ez_setup => Some setup.py scripts import ez_setup. - ez_setup requires a specific version of setuptools but doesn't - actually need it -- lesser versions work fine. - -Problems that are out of scope, at least for now: - -- Requires that are for a specific version (or range) because of bugs - (not API change/features) but the distro packages have backported - the fixes needed to an earlier version: - -- differences between distribution names of packages e.g. "what are - the packages of NumPy called?" - -- Egg version (specified in a Require section in setup.py) doesn't - match the distro package version - https://www.redhat.com/archives/fedora-python-devel-list/2008-June/msg00002.html - -We want to solve these issues for Linux distribution packages without -negatively impacting Windows, OS X, or pure-python -(i.e. easy_install/pip/virtualenv) installs. - -Example problem with current distutils -====================================== - -In -http://docs.python.org/distutils/setupscript.html#installing-additional-files -there is this example of supplying an initscript within a setup.py:: - - setup(..., - data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), - ('config', ['cfg/data.cfg']), - ('/etc/init.d', ['init-script'])] - ) - -This suffers from several problems: - -1. The file hardcodes "/etc/init.d" as the directory for installation - of the initscript. However, this varies between different Linux - distributions. The above example assumes SysV init, but isn't - correct for systems using Upstart (the initscripts are likely to - need to be different for this case). Even within systems using - SysV init, the content of the script can vary between different - distributions - -2. The FHS mandates that configuration files go below /etc, but on a - Windows box that's meaningless. - -3. The file is a python script: if we want to extract data from it, we - have to either run it, or somehow parse it; we would need to - sandbox. We would prefer a declarative minilanguage for specifying - this data. - -Similarly: documentation files, .h files (e.g. NumPy) - -We want a system that: - -1. is easy for developers, does not require an "install" or "build" - phase during the edit/test development loop in a working copy - -2. supports both FHS-compliant scattering across the filesystem - hierarchy, and Windows, and virtualenv-style trees. - ------------------------ -Problems for Developers ------------------------ - -Package/file lists in multiple places -===================================== - -* MANIFEST -* MANIFEST.in -* setup.py::data_files -* setup.py::package_data -* setup.py::packages -* setup.py::scripts - -Replace all of these with settings in ``setup.cfg`` to specify runtime -files (python code, resource files) and files that belong in the sdist -but are not wanted for runtime. - -No idea where Linux distros want files -====================================== - -Sometimes programmers want to do the right thing for people wanting to -package their programs in Linux distributions, but they don't know -where they belong. Making matters worse, the files can go in -different places on different Linux distributions or on Windows and -MacOS. Placing the files in the wrong place can lead to errors at -runtime, for instance, if the file needs to be writable by the module -but it's placed on a read-only filesystem. - -This PEP attempts to deal with this by categorizing files so -developers can properly mark what properties their files need and -using an API to access the files, abstracting the different file -locations on different platforms. - -Hard to extend the build commands -================================= - -* distutils documentation is very poor - -* distutils build commands are classes with special method names -- - why not simple functions? - -* how do you extend the data allowed to be set in entries setup()? - -* build commands sometimes need to act on the same arguments. No way - to pass these between them right now. - - ------------------------------------ -Proposed solution for placing files ------------------------------------ - -This solution attempts to make several pieces of building and -installing better. It merges the many file lists into a single file, -simplifies (or eliminates the need for) setup.py, and allows packagers -to place resource files in locations appropriate to their -distribution. - -This solution comes in three pieces: - -1. A ``resources`` section in ``setup.cfg`` that maps resource files - to their categories (and optionally subdirectory prefixes within - those categories) - -2. A ``sysconfig.cfg`` file at the system Python level that maps - categories to a position on the filesystem - -3. A simple ``pkgutil.open()`` API to access resources from code - -Rationale -========= - -1. The evidence (from ``__file__`` usage) is strong that package devs - want to think in terms of paths within their local dev tree. They - don't want to worry about categorizing or finding their static - files elsewhere. - -2. Package devs are more likely to use an API that makes them think - less and type less. - -3. Package devs are more likely to accept patches from packagers if - that patch only touches a single .cfg file, rather than touching - every single ``pkgutil.open()`` call throughout their code. - -Therefore, the ``pkgutil.open()`` call should accept a simple path -relative to the package/distribution root. The ``resources`` section -in ``setup.cfg`` uses globs to categorize those files: -forward-thinking package devs can write this section, or packagers can -do it for them and submit patches. - - -"resources" section in setup.cfg -================================ - -The setup.py file has many sections that need to list files. We plan -to remove those lists to ``setup.cfg``. The ``resources`` section of -``setup.cfg`` replaces the current ``package_data``, ``data_files``, -and ``extra_files`` options in ``setup.py``. - -There are three pieces of information that are needed for resource -files: - -1. Position in the source tree - (e.g. 'mailman/database/schemas/schema.cfg', 'mywidget/jquery.js') - -* Position when installed - (e.g. '/etc/mailman/database/schemas/schema.cfg', - '/usr/share/mywidget-1.1/javascript/jquery.js'). For simple - virtualenv-style installations, this may well be the same as (1). - -* Key used when referencing the resource from code. Ideally, this - could be the same as (1), but because of difficulties in finding - "distribution root" at runtime from a ``pkgutil.open()`` call, it - will instead have to be a combination of "module name" and "path - relative to module", similar to what ``pkg_resources`` does. - -The information that the developer is concerned with: -* Position in the source tree -* Key used in referencing it - -The information the downstream packager (RPM/deb/sysadmin) cares about are: -* Position when installed -* Key used in referencing it - -Example -------- - -We have a source tree with the following files:: - - mailman-1.0/ - README - some.tpl - some-new-semantic.sns - mailman/ - database/ - mailman.db - schemas/ - blah.schema - etc/ - my.cnf - foo/ - some/ - path/ - bar/ - my.cfg - other.cfg - developer-docs/ - index.txt - api/ - toc.txt - -Here's where we want the files to end up in a typical Linux distribution: - -== ==================================== =================================================================================================== -## Relative path in source tree Final full installed path -== ==================================== =================================================================================================== -1 mailman/database/schemas/blah.schema /var/mailman/schemas/blah.schema -2 some.tpl /var/mailman/templates/some.tpl -3 path/to/some.tpl /var/mailman/templates/path/to/some.tpl ! -4 mailman/database/mailman.db /var/mailman/database/mailman.db ! -5 developer-docs/index.txt /usr/share/doc/mailman/developer-docs/index.txt -6 developer-docs/api/toc.txt /usr/share/doc/mailman/developer-docs/api/toc.txt -7 README /usr/share/doc/mailman/README -8 mailman/etc/my.cnf /etc/mailman/my.cnf -9 mailman/foo/some/path/bar/my.cfg /etc/mailman/baz/some/path/bar/my.cfg AND - /etc/mailman/hmm/some/path/bar/my.cfg + - emit a warning -10 mailman/foo/some/path/other.cfg /etc/mailman/some/path/other.cfg ! -11 some-new-semantic.sns /var/funky/mailman/some-new-semantic.sns -== ==================================== =================================================================================================== - -The numbers in the above placements are referenced below. - -setup.cfg -~~~~~~~~~ - -The setup.cfg file allows the developer and/or packager to mark what -categories the files belong to. These are drawn from the types of -files that the FHS and GNU coding standards define:: - - [resources] - # path glob category placement from above table - - mailman/database/schemas/* = {appdata}/schemas # 1 - **/*.tpl = {appdata}/templates # 2, 3 # does NOT flatten folder structure in destination - developer-docs/**/*.txt = {doc} # 5, 6 - README = {doc} # 7 - mailman/etc/* = {config} # 8 - mailman/foo/**/bar/*.cfg = {config}/baz # 9 - mailman/foo/**/*.cfg = {config}/hmm # 9, 10 - some-new-semantic.sns = {funky-crazy-category} # 11 - -The glob definitions are relative paths that match files from the top -of the source tree (the location of ``setup.cfg``). Forward slashes -(only) are used as path separator. - -:"*": is a glob that matches any characters within a file or directory -name -:"**": is a recursive glob that matches any (or no) characters within a file -or directory name as well as a forward slash (thus an arbitrarily deep -number of directories) - -The "category" value both categorizes the files and allows for placing -them in a more fine-grained subdirectory within a category. This value -must begin with a {category}; raw absolute or relative paths are not -allowed. - -The full Python 3 string interpolation language is not supported, only -simple {category} substitutions. The {category} is looked up in a -system-level Python ``sysconfig.cfg`` file, where operating system -vendors and system administrators can define where in the filesystem -various types of files are placed. The category paths will generally -include a {distribution.name} variable, to isolate one package's files -of a given type from other packages. - -As can be seen from the examples above, explicitly-matched directory -prefixes are stripped from the relative path before it is appended to -the category location. Glob matches are never stripped (to avoid -flattening hierarchies and overwriting files). In the -``mailman/foo/\*\*/\*.cfg`` example, ``mailman/foo`` is removed, but -not any directories matched by the recursive glob: see entries 9 and -10 in the example table. - -sysconfig.cfg -~~~~~~~~~~~~~ - -This is a system-wide Python configuration file (TODO: can be -overridden by e.g. virtualenv) that defines where on the filesystem -resources will actually be installed. A sample ``sysconfig.cfg`` can -be found in the ``distutils2`` repository at -``src/distutils2/_backport/sysconfig.cfg`` [3]. - -Links - -.. [1] Filesystem Hierarchy Standard http://www.pathname.com/fhs/ -.. [2] Rationale from the FHS which makes the distinctions between parts of the filesystem: http://www.pathname.com/fhs/pub/fhs-2.3.html#THEFILESYSTEM -.. [3] sample sysconfig.cfg: http://bitbucket.org/tarek/distutils2/src/tip/src/distutils2/_backport/sysconfig.cfg - -What happens? -~~~~~~~~~~~~~ - -As an example, ``mailman/database/schemas/blah.schema``: - -1. The file ``mailman/database/schemas/blah.schema`` in the source - tree matches ``mailman/database/schemas/*`` within the - ``resources`` stanza of the setup.cfg, which has right-hand side - ``{appdata}/schemas`` - -2. The ``*`` in the left-hand-side matches ``blah.schema``, and the - initial ``mailman/database/schemas/`` is stripped, so the - installation path for the file is mapped to - ``{appdata}/schemas/blah.schema`` - -3. The label ``appdata`` is listed in the ``sysconfig.cfg`` section - for the ``posix_prefix`` installation scheme as installed to - ``/usr/share/{distribution.name}``. This expands out to: - ``/usr/share/mailman`` - -4. The result is that the source file - ``mailman/database/schemas/blah.schema`` is installed to - ``/var/mailman/schemas/blah.schema``, and this mapping is recorded - in a RESOURCES file in the installation metadata for the - distribution. - -5. The source code can open the file at runtime via the API call - ``pkgutil.open('mailman', 'database/schemas/blah.schema')`` (where - the first argument is an importable Python package name, and the - second is a path relative to the location of that package), and - pkgutil will (using the RESOURCES mapping) open it from - ``/var/mailman/schemas/blah.schema``. - -6. If the package is not installed, and thus has no RESOURCES mapping, - ``pkgutil.open('mailman', - 'database/schemas/blah.schema')`` - -1. The file `mailman/database/schemas/blah.schema` in the source tree matches `mailman/database/schemas/*` within the data clause of the setup.cfg, so it is treated as having the label `{data}`. -2. The clause specified a prefix path, so the installation path for the file is mapped to "schemas/blah.schema" -3. The label "data" is listed in the [resource_variables] stanza as being installed to "/var/mailman" -4. The result is that the source file "mailman/database/schemas/blah.schema" is installed within the rpm/deb to "/var/mailman/schemas/blah.schema" -5. The source code can still open the file via an API using pkgutil.open('mailman', 'database/schemas/blah.schema') and have the underlying system open it from "/var/mailman/schemas/blah.schema". - - -Advice to packagers for fixing file locations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two places where you might need to change things in order to -customize the locations that files are installed into. The setup.cfg file can -be patched if the files are miscategorized. For instance someone marks a help -file that is used by the app at runtime as {doc} when it should be marked as -{help}. These types of patches should be submitted to the upstream project. -The resource_variables file can be changed to define different locations for -all apps on a system. This should usually be done once in a systemwide file -for the distribution. Changing this again may require the system packager to -rebuild all their Python modules to change the file location. There is API in -pkgutil to allow adding categories to the resource_variables file from -a script that should be used instead of trying to edit the file with raw text -processing. - -Open issues -=========== - -* setup.cfg is in the distribution, not in the module. Thus, in the unbuilt-egg - case, how can we find the distro when all we have is a module name? It would - be nice to not need an equivalent of ``setup.py develop``. Can we just walk up - the folder hierarchy from the module until we find a setup.cfg? A setup.cfg is - necessary if you use distutils2, is it not? - - -> information found in setup.cfg will be put in the *FILES* file upon - installation in the egg-info directory. - IOW in the unbuit-egg case, we would need to create that dir, then use - pkgutil APIs. - -* If sysconfig.cfg lands in Python 2.7, what happens when we run distutils2 in - 2.4? - - -> A backport of sysconfig.cfg is provided within the distutils2 distribution. - -* Our new glob-based [resources] section is much more compact (and consistent - with other systems, like bash) than the explicit MANIFEST.in directives, but - they don't offer some of the old features. Is it okay to lose exclude, - global-exclude, and recursive-exclude? What do graft and prune do, and do we - cover their behavior? I think we could probably use a [resource:exclude] - section with additional exclude globs in it. - - -> let's list the use cases we don't cover, and see - -API -=== - -pkgutil.open -------------- - -Returns a file object for the resource. - -:: - - pkgutil.open('STRING_NAME_FOR_PACKAGE', 'filename/with/path/relative/to/the/source/package/directory') - Example: - pkgutil.open('mailman', 'database/schemas/blah.schema') - - -* First argument is the string name for a python package. -* Second argument is the directory path relative to the python package's directory. -* At install (or build) time we create a metadata file that maps from the source tree files to the files in their installed locations on the filesystem. -* pkgutil.open() consults the metadata file to decide where to find the resource. If the metadata file is not found (as in a package before the egg is built), open() falls back to traversing the given relative path starting from the root of the calling package (using __name__). -* pkgutil.open() calls from nested packages aren't a problem because, after all, we pass the desired 'module_name' to start from as the first arg. - -* ? Do we still need this? Default behavior: alongside the package files (if the real-installed-locations metadata file does not exist). Or if the package is installed without any resource_variables specified. ?? - -pkgutil.filename ------------------ - -Returns a resource's filename with the full path. - -:: - - pkgutil.filename('STRING_NAME_FOR_PACKAGE', 'filename/with/path/relative/to/the/source/package/directory') - Example: - pkgutil.filename('mailman', 'database/schemas/blah.schema') - '/usr/share/mailman/schemas/blah.schema' - -pkgutil.add_category ---------------------- - -Adds a new category to the resource variables filename. - -:: - - pkgutil.add_category('CATEGORY', 'LOCATION') - Example: - pkgutil.add_category('lockdir', '{statedir}/lock') - -Using the API allows the parser to protect from adding duplicate categories. - ----- -Todo ----- - -These need to be worked in or discarded somehow - - * Differences between applications and packages - - Applications sometimes want a private library (for instance to do their commandline parsing) - -Ideally, for every resource file, the developer (or the defaults) have classified with a "label" *TODO*: we don't have a default classifier right now: for instance:: - - **/*.txt = doc - **/*.png = data - **/*.jpg = data - **/*.gif = data - **/*.cfg = config - - -Similar to i18n: marking of strings for translatability: gives you an ID, and a default value -Analagous to gettext: parse the source, figure out the resources - -[X] Per-distro (per site ?): label placement file, mapping labels to system locations *I think this is done* - -[X] What strings are valid as labels? only strings that are valid Python 2 identifiers: ([_A-Za-z][_A-Za-z0-9]*) TODO: doublecheck this! *Obsolete* We have gotten rid of the labels - -[X] So now, when it comes to building a deb/rpm, we have another file: "label placement" which maps from labels to rules about placement on the filesystem, written once by each linux distribution: *I think this is done* - - - -How Debian does a .install file -=============================== - -In `packagename.install`:: - - etc/* etc - usr/* usr - Products/statusmessages/* usr/share/zope/Products/statusmessages3 - -Each line has 2 parts: -* A glob of the source path within the deb -* Where it should land within the fakeroot (which corresponds to the final installed path) - -This gives the packager the opportunity to both move and rename things, and it's fairly concise. - - -Building different packages from one source -=========================================== - -?? Do we want to do this?? - -Use case --------- - -Split the docs into a separate sdist from the code so that people can download them separately. -(Matthias) - -Another use case ----------------- - -Need to split submodule into its own binary package (essentially converting top-level to a namespace package). - - - -Alternate ways of specifying labels ------------------------------------ -Noufal's:: - - [mailman] - data = *.txt, README - data.persistent = sqlite.db - ----------------------- - -Tarek's:: - - [files] - - data = - mailman/database/schemas/* - *.txt - README - - data.persistent = sqlite.db - ----------------------- - -Toshio's:: - - [resources] - *.jpg = data - -Alternative Label Idea -====================== - -labels for different resource types: images, manpages, script, config files etc, javascript, schema, sql, data files - -(those labels impose some other issues - what would one do to differences in statically servable on a webserver versus gtkbuilder can find it) - -pkg_resources already provides software with an API:: - - pkg_resource.open(label='javascript', name='jquery.js') - -Then we have the ability for Linux distros to place the different labels in FHS-compliant (or whatever) locations on the filesystem, mapping each label to a fs path:: - - pkg_resource.open(label='config', name=') - - pkg_resource.resource_stream(pkgname='mailman', label='config', victim='schema.cfg') - pkg_resource.resource_stream('mailman', 'mailman.config', 'schema.cfg') - pkg_resource.resource_stream('mailman', 'mailman.config', 'schema.cfg', label='config') - -analogy with logging: - -- with logging: developer sets up streams of data; sysadmin decides what to do with each log stream -- with packaging: developer sets up streams of data; packaging system decides where to put each one - -developer: - -(1) everything's within my local working copy; look within it; want to be able to quickly hack on stuff without having to "install" somewhere, for fast edit/test loop -(2) "setup.py sdist" has given us a zipfile, put it on pypi, someone uses buildout on it -(3) as (2) but a distribution has moved things to FHS-compliant location - - - pkgutil.open(pkgname='mailman.database.schemas', filename='schema.cfg') # <-- Does this work with our examples below? - - pkgutil.open(pkgname='mailman', label='data', filename='schemas/schema.cfg') - -(It won't be easy to get package devs to use this API; __file__ feels less magic than some strange call from pkgutil. The simpler the API call and the more "builtin" it looks, the better.) - -*TODO* Can we make sane defaults? For instance, can pkgname default to the pkgname or modulename that the call is being made from? - -*TODO* can we match things against a range of packages/paths - -(1) ./mailman/config/schema.cfg -(2) . -(3) /etc/mailman/database/schemas/schema.cfg - -mapping from labels to dirs:: - - distro_dict = { - 'config':'/etc', - 'mandir':'/usr/share/mandir', - } - -Another old syntax proposal -=========================== -:: - - [resources] - - # data are composed of two elements - # 1. the path relative to the package - # 2. an optional prefix path that will replace the explicit (non-glob) initial path from (1) - - data = - mailman/database/schemas/* schemas/ - **/*.tpl templates/ - - data.persistent = - mailman/database/mailman.db database/ - - doc = - developer-docs/**/*.txt - README - - config = - mailman/etc/* . # all files in mailman/etc/* copied to - mailman/foo/**/*.cfg foo # all .cfg files below mailman/foo/SOME/PATH/TO/FILE/ will get copied to foo/SOME/PATH/TO/FILE/ - mailman/foo/**/*.cfg - mailman/foo/**/bar/*.cfg baz diff --git a/docs/make.bat b/docs/make.bat deleted file mode 100644 --- a/docs/make.bat +++ /dev/null @@ -1,113 +0,0 @@ - at ECHO OFF - -REM Command file for Sphinx documentation - -set SPHINXBUILD=sphinx-build -set BUILDDIR=build -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% source -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% -) - -if "%1" == "" goto help - -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. changes to make an overview over all changed/added/deprecated items - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\Distutils2.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\Distutils2.ghc - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -:end diff --git a/docs/source/_static/depgraph_big.png b/docs/source/_static/depgraph_big.png deleted file mode 100644 Binary file docs/source/_static/depgraph_big.png has changed diff --git a/docs/source/conf.py b/docs/source/conf.py deleted file mode 100644 --- a/docs/source/conf.py +++ /dev/null @@ -1,199 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Distutils2 documentation build configuration file, created by -# sphinx-quickstart on Sun Feb 28 15:23:06 2010. -# -# This file is execfile()d with the current directory set to its containing -# dir. It requires Sphinx 1.0. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import os -import sys -import sphinx - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -#sys.path.append(os.path.abspath('.')) -sys.path.append(os.path.abspath(os.path.join(__file__, '..', '..', '..'))) - -# -- General configuration ----------------------------------------------------- - -# Add any Sphinx extension module names here, as strings. They can be extensions -# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix of source filenames. -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'Distutils2' -copyright = u'2010, Tarek Ziad? and contributors' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = '1.0a3' -# The full version, including alpha/beta/rc tags. -release = '1.0a3+' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -#language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of documents that shouldn't be included in the build. -#unused_docs = [] - -# List of directories, relative to source directory, that shouldn't be searched -# for source files. -exclude_trees = [] - -# The reST default role (used for this markup: `text`) to use for all documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -modindex_common_prefix = ['distutils2.', 'distutils2.command.'] - - -# -- Options for HTML output --------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. Major themes that come with -# Sphinx are currently 'default' and 'sphinxdoc'. -html_theme = 'default' - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -if sphinx.__version__[:3] >= '1.0': - html_theme_options = {'collapsiblesidebar': True} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -html_title = 'Distutils2' - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_use_modindex = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = '' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'Distutils2doc' - - -# -- Options for LaTeX output -------------------------------------------------- - -# The paper size ('letter' or 'a4'). -#latex_paper_size = 'letter' - -# The font size ('10pt', '11pt' or '12pt'). -#latex_font_size = '10pt' - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, documentclass [howto/manual]). -latex_documents = [ - ('index', 'Distutils2.tex', u'Distutils2 Documentation', - u'Tarek Ziad? and contributors', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# Additional stuff for the LaTeX preamble. -#latex_preamble = '' - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_use_modindex = True diff --git a/docs/source/contributing.rst b/docs/source/contributing.rst deleted file mode 100644 --- a/docs/source/contributing.rst +++ /dev/null @@ -1,27 +0,0 @@ -========================== -Contributing to Distutils2 -========================== - ----------------- -Reporting Issues ----------------- - -When using, testing or developping distutils2, you may encounter issues. Please report to the following sections to know how these issues should be reported. - -Please keep in mind that this guide is intended to ease the triage and fixing processes by giving the maximum information to the developers. It should not be viewed as mandatory, only advisory ;). - - -- Go to http://bugs.python.org/ (you'll need a Python Bugs account), then "Issues" > "Create ticket". -- **Title**: write in a short summary of the issue. You may prefix the issue title with ?component:?, where component can be something like installer, sdist, setup.cfg, etc., or add it at the end of your title if the normal flow of the sentence allows it. This will ease up later searches. -- **Components**: choose "Distutils2" -- **Version**: choose "3rd party" -- **Body**: explain how to reproduce the bug: What you want to do, what code you write, what happens, what should happen, how to fix it (if you have an idea). - * You should always test with the tip of the main repository, not releases. - * Mention the versions of Python you tested with. d2 supports 2.4 to 2.7. - * If relevant, mention the version of your operating system (for example with issues related to C extensions). - * When referencing commits, be careful to use the universal changeset identifiers (12 characters, for instance c3cf81fc64db), not the local sequential numbers (for example 925) that are not shared among clones. - * Try to be as concise as possible, but not too much. - * If useful, paste tracebacks. - * If useful, attach setup.cfg or other files (binary files like archives are not very convenient, better to stick to text). - -Issues related to PyPI are reported via email to the **catalog-sig at python.org** mailing list, not within bugs.python.org. diff --git a/docs/source/devresources.rst b/docs/source/devresources.rst deleted file mode 100644 --- a/docs/source/devresources.rst +++ /dev/null @@ -1,49 +0,0 @@ -=================== -Developer Resources -=================== - - -Source code -~~~~~~~~~~~ - -* Main repo: http://hg.python.org/distutils2 -* For contributors at: http://bitbucket.org/tarek/distutils2 - -Dependencies -~~~~~~~~~~~~ - -* unittest2 -* If your operating system splits core python and a python-dev (or -devel) - packages, install the dev package too: It contains header files needed by - tests. - -Issue Tracker -~~~~~~~~~~~~~ - -Using the `distutils2` component at the `python.org bug tracker `_, - -Mailing List -~~~~~~~~~~~~ - -http://groups.google.com/group/the-fellowship-of-the-packaging - -more general discussion at distutils-sig at python.org - -IRC Channel -~~~~~~~~~~~ - -#distutils on irc.freenode.net - -Documentation -~~~~~~~~~~~~~ - -This documentation is present in the docs/source directory of the above source -code. - -Additional useful information available at: - -* `Hitchhikers guide to packaging `_ - - - - diff --git a/docs/source/distutils/apiref.rst b/docs/source/distutils/apiref.rst deleted file mode 100644 --- a/docs/source/distutils/apiref.rst +++ /dev/null @@ -1,2010 +0,0 @@ -.. _api-reference: - -************* -API Reference -************* - - -:mod:`distutils2.core` --- Core Distutils functionality -======================================================= - -.. module:: distutils2.core - :synopsis: The core Distutils functionality - - -The :mod:`distutils2.core` module is the only module that needs to be installed -to use the Distutils. It provides the :func:`setup` (which is called from the -setup script). Indirectly provides the :class:`distutils2.dist.Distribution` and -:class:`distutils2.cmd.Command` class. - - -.. function:: setup(arguments) - - The basic do-everything function that does most everything you could ever ask - for from a Distutils method. - - The setup function takes a large number of arguments. These are laid out in - the following table. - - +--------------------+--------------------------------+-------------------------------------------------------------+ - | argument name | value | type | - +====================+================================+=============================================================+ - | *name* | The name of the package | a string | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *version* | The version number of the | See :mod:`distutils2.version` | - | | package | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *summary* | A single line describing the | a string | - | | package | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *description* | Longer description of the | a string | - | | package | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *author* | The name of the package author | a string | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *author_email* | The email address of the | a string | - | | package author | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *maintainer* | The name of the current | a string | - | | maintainer, if different from | | - | | the author | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *maintainer_email* | The email address of the | | - | | current maintainer, if | | - | | different from the author | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *home_page* | A URL for the package | a URL | - | | (homepage) | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *download_url* | A URL to download the package | a URL | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *packages* | A list of Python packages that | a list of strings | - | | distutils will manipulate | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *py_modules* | A list of Python modules that | a list of strings | - | | distutils will manipulate | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *scripts* | A list of standalone script | a list of strings | - | | files to be built and | | - | | installed | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *ext_modules* | A list of Python extensions to | A list of instances of | - | | be built | :class:`distutils2.extension.Extension` | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *classifiers* | A list of categories for the | The list of available | - | | package | categorizations is at | - | | | http://pypi.python.org/pypi?:action=list_classifiers. | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *distclass* | the :class:`Distribution` | A subclass of | - | | class to use | :class:`distutils2.dist.Distribution` | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *script_name* | The name of the setup.py | a string | - | | script - defaults to | | - | | ``sys.argv[0]`` | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *script_args* | Arguments to supply to the | a list of strings | - | | setup script | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *options* | default options for the setup | a string | - | | script | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *license* | The license for the package | a string | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *keywords* | Descriptive metadata, see | | - | | :PEP:`314` | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *platforms* | | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *cmdclass* | A mapping of command names to | a dictionary | - | | :class:`Command` subclasses | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *data_files* | A list of data files to | a list | - | | install | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *package_dir* | A mapping of package to | a dictionary | - | | directory names | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - | *extra_path* | Information about an | a string, 1-tuple or 2-tuple | - | | intervening directory the | | - | | install directory and the | | - | | actual installation directory. | | - | | | | - | | If the value is a string is is | | - | | treated as a comma-separated | | - | | tuple. | | - | | | | - | | If the value is a 2-tuple, | | - | | the first element is the | | - | | ``.pth`` file and the second | | - | | is the name of the intervening | | - | | directory. | | - | | | | - | | If the value is a 1-tuple that | | - | | element is both the name of | | - | | the ``.pth`` file and the | | - | | intervening directory. | | - +--------------------+--------------------------------+-------------------------------------------------------------+ - - - -.. function:: run_setup(script_name[, script_args=None, stop_after='run']) - - Run a setup script in a somewhat controlled environment, and return the - :class:`distutils2.dist.Distribution` instance that drives things. This is - useful if you need to find out the distribution metadata (passed as keyword - args from *script* to :func:`setup`), or the contents of the config files or - command line. - - *script_name* is a file that will be run with :func:`execfile` - ``sys.argv[0]`` will be replaced with *script* for the duration of the call. - *script_args* is a list of strings; if supplied, ``sys.argv[1:]`` will be - replaced by *script_args* for the duration of the call. - - *stop_after* tells :func:`setup` when to stop processing; possible values: - - +---------------+---------------------------------------------+ - | value | description | - +===============+=============================================+ - | *init* | Stop after the :class:`Distribution` | - | | instance has been created and populated | - | | with the keyword arguments to :func:`setup` | - +---------------+---------------------------------------------+ - | *config* | Stop after config files have been parsed | - | | (and their data stored in the | - | | :class:`Distribution` instance) | - +---------------+---------------------------------------------+ - | *commandline* | Stop after the command line | - | | (``sys.argv[1:]`` or *script_args*) have | - | | been parsed (and the data stored in the | - | | :class:`Distribution` instance.) | - +---------------+---------------------------------------------+ - | *run* | Stop after all commands have been run (the | - | | same as if :func:`setup` had been called in | - | | the usual way). This is the default value. | - +---------------+---------------------------------------------+ - -In addition, the :mod:`distutils2.core` module exposed a number of classes that -live elsewhere. - -* :class:`~distutils.extension.Extension` from :mod:`distutils2.extension` - -* :class:`~distutils.command.cmd.Command` from :mod:`distutils2.command.cmd` - -* :class:`~distutils.dist.Distribution` from :mod:`distutils2.dist` - -A short description of each of these follows, but see the relevant module for -the full reference. - - -.. class:: Extension - - The Extension class describes a single C or C++extension module in a setup - script. It accepts the following keyword arguments in its constructor - - +------------------------+--------------------------------+---------------------------+ - | argument name | value | type | - +========================+================================+===========================+ - | *name* | the full name of the | string | - | | extension, including any | | - | | packages --- i.e. *not* a | | - | | filename or pathname, but | | - | | Python dotted name | | - +------------------------+--------------------------------+---------------------------+ - | *sources* | list of source filenames, | string | - | | relative to the distribution | | - | | root (where the setup script | | - | | lives), in Unix form (slash- | | - | | separated) for portability. | | - | | Source files may be C, C++, | | - | | SWIG (.i), platform-specific | | - | | resource files, or whatever | | - | | else is recognized by the | | - | | :command:`build_ext` command | | - | | as source for a Python | | - | | extension. | | - +------------------------+--------------------------------+---------------------------+ - | *include_dirs* | list of directories to search | string | - | | for C/C++ header files (in | | - | | Unix form for portability) | | - +------------------------+--------------------------------+---------------------------+ - | *define_macros* | list of macros to define; each | (string, string) tuple or | - | | macro is defined using a | (name, ``None``) | - | | 2-tuple ``(name, value)``, | | - | | where *value* is | | - | | either the string to define it | | - | | to or ``None`` to define it | | - | | without a particular value | | - | | (equivalent of ``#define FOO`` | | - | | in source or :option:`-DFOO` | | - | | on Unix C compiler command | | - | | line) | | - +------------------------+--------------------------------+---------------------------+ - | *undef_macros* | list of macros to undefine | string | - | | explicitly | | - +------------------------+--------------------------------+---------------------------+ - | *library_dirs* | list of directories to search | string | - | | for C/C++ libraries at link | | - | | time | | - +------------------------+--------------------------------+---------------------------+ - | *libraries* | list of library names (not | string | - | | filenames or paths) to link | | - | | against | | - +------------------------+--------------------------------+---------------------------+ - | *runtime_library_dirs* | list of directories to search | string | - | | for C/C++ libraries at run | | - | | time (for shared extensions, | | - | | this is when the extension is | | - | | loaded) | | - +------------------------+--------------------------------+---------------------------+ - | *extra_objects* | list of extra files to link | string | - | | with (e.g. object files not | | - | | implied by 'sources', static | | - | | library that must be | | - | | explicitly specified, binary | | - | | resource files, etc.) | | - +------------------------+--------------------------------+---------------------------+ - | *extra_compile_args* | any extra platform- and | string | - | | compiler-specific information | | - | | to use when compiling the | | - | | source files in 'sources'. For | | - | | platforms and compilers where | | - | | a command line makes sense, | | - | | this is typically a list of | | - | | command-line arguments, but | | - | | for other platforms it could | | - | | be anything. | | - +------------------------+--------------------------------+---------------------------+ - | *extra_link_args* | any extra platform- and | string | - | | compiler-specific information | | - | | to use when linking object | | - | | files together to create the | | - | | extension (or to create a new | | - | | static Python interpreter). | | - | | Similar interpretation as for | | - | | 'extra_compile_args'. | | - +------------------------+--------------------------------+---------------------------+ - | *export_symbols* | list of symbols to be exported | string | - | | from a shared extension. Not | | - | | used on all platforms, and not | | - | | generally necessary for Python | | - | | extensions, which typically | | - | | export exactly one symbol: | | - | | ``init`` + extension_name. | | - +------------------------+--------------------------------+---------------------------+ - | *depends* | list of files that the | string | - | | extension depends on | | - +------------------------+--------------------------------+---------------------------+ - | *language* | extension language (i.e. | string | - | | ``'c'``, ``'c++'``, | | - | | ``'objc'``). Will be detected | | - | | from the source extensions if | | - | | not provided. | | - +------------------------+--------------------------------+---------------------------+ - - -.. class:: Distribution - - A :class:`Distribution` describes how to build, install and package up a - Python software package. - - See the :func:`setup` function for a list of keyword arguments accepted by - the Distribution constructor. :func:`setup` creates a Distribution instance. - - -.. class:: Command - - A :class:`Command` class (or rather, an instance of one of its subclasses) - implement a single distutils command. - - -:mod:`distutils2.ccompiler` --- CCompiler base class -==================================================== - -.. module:: distutils2.ccompiler - :synopsis: Abstract CCompiler class - - -This module provides the abstract base class for the :class:`CCompiler` -classes. A :class:`CCompiler` instance can be used for all the compile and -link steps needed to build a single project. Methods are provided to set -options for the compiler --- macro definitions, include directories, link path, -libraries and the like. - -This module provides the following functions. - - -.. function:: gen_lib_options(compiler, library_dirs, runtime_library_dirs, libraries) - - Generate linker options for searching library directories and linking with - specific libraries. *libraries* and *library_dirs* are, respectively, lists - of library names (not filenames!) and search directories. Returns a list of - command-line options suitable for use with some compiler (depending on the - two format strings passed in). - - -.. function:: gen_preprocess_options(macros, include_dirs) - - Generate C preprocessor options (:option:`-D`, :option:`-U`, :option:`-I`) as - used by at least two types of compilers: the typical Unix compiler and Visual - C++. *macros* is the usual thing, a list of 1- or 2-tuples, where ``(name,)`` - means undefine (:option:`-U`) macro *name*, and ``(name, value)`` means - define (:option:`-D`) macro *name* to *value*. *include_dirs* is just a list - of directory names to be added to the header file search path (:option:`-I`). - Returns a list of command-line options suitable for either Unix compilers or - Visual C++. - - -.. function:: get_default_compiler(osname, platform) - - Determine the default compiler to use for the given platform. - - *osname* should be one of the standard Python OS names (i.e. the ones - returned by ``os.name``) and *platform* the common value returned by - ``sys.platform`` for the platform in question. - - The default values are ``os.name`` and ``sys.platform``. - - -.. function:: new_compiler(plat=None, compiler=None, verbose=0, dry_run=0, force=0) - - Factory function to generate an instance of some CCompiler subclass for the - supplied platform/compiler combination. *plat* defaults to ``os.name`` (e.g. - ``'posix'``, ``'nt'``), and *compiler* defaults to the default compiler for - that platform. Currently only ``'posix'`` and ``'nt'`` are supported, and the - default compilers are "traditional Unix interface" (:class:`UnixCCompiler` - class) and Visual C++ (:class:`MSVCCompiler` class). Note that it's perfectly - possible to ask for a Unix compiler object under Windows, and a Microsoft - compiler object under Unix---if you supply a value for *compiler*, *plat* is - ignored. - - .. % Is the posix/nt only thing still true? Mac OS X seems to work, and - .. % returns a UnixCCompiler instance. How to document this... hmm. - - -.. function:: show_compilers() - - Print list of available compilers (used by the :option:`--help-compiler` - options to :command:`build`, :command:`build_ext`, :command:`build_clib`). - - -.. class:: CCompiler([verbose=0, dry_run=0, force=0]) - - The abstract base class :class:`CCompiler` defines the interface that must be - implemented by real compiler classes. The class also has some utility - methods used by several compiler classes. - - The basic idea behind a compiler abstraction class is that each instance can - be used for all the compile/link steps in building a single project. Thus, - attributes common to all of those compile and link steps --- include - directories, macros to define, libraries to link against, etc. --- are - attributes of the compiler instance. To allow for variability in how - individual files are treated, most of those attributes may be varied on a - per-compilation or per-link basis. - - The constructor for each subclass creates an instance of the Compiler object. - Flags are *verbose* (show verbose output), *dry_run* (don't actually execute - the steps) and *force* (rebuild everything, regardless of dependencies). All - of these flags default to ``0`` (off). Note that you probably don't want to - instantiate :class:`CCompiler` or one of its subclasses directly - use the - :func:`distutils2.CCompiler.new_compiler` factory function instead. - - The following methods allow you to manually alter compiler options for the - instance of the Compiler class. - - - .. method:: CCompiler.add_include_dir(dir) - - Add *dir* to the list of directories that will be searched for header - files. The compiler is instructed to search directories in the order in - which they are supplied by successive calls to :meth:`add_include_dir`. - - - .. method:: CCompiler.set_include_dirs(dirs) - - Set the list of directories that will be searched to *dirs* (a list of - strings). Overrides any preceding calls to :meth:`add_include_dir`; - subsequent calls to :meth:`add_include_dir` add to the list passed to - :meth:`set_include_dirs`. This does not affect any list of standard - include directories that the compiler may search by default. - - - .. method:: CCompiler.add_library(libname) - - Add *libname* to the list of libraries that will be included in all links - driven by this compiler object. Note that *libname* should *not* be the - name of a file containing a library, but the name of the library itself: - the actual filename will be inferred by the linker, the compiler, or the - compiler class (depending on the platform). - - The linker will be instructed to link against libraries in the order they - were supplied to :meth:`add_library` and/or :meth:`set_libraries`. It is - perfectly valid to duplicate library names; the linker will be instructed - to link against libraries as many times as they are mentioned. - - - .. method:: CCompiler.set_libraries(libnames) - - Set the list of libraries to be included in all links driven by this - compiler object to *libnames* (a list of strings). This does not affect - any standard system libraries that the linker may include by default. - - - .. method:: CCompiler.add_library_dir(dir) - - Add *dir* to the list of directories that will be searched for libraries - specified to :meth:`add_library` and :meth:`set_libraries`. The linker - will be instructed to search for libraries in the order they are supplied - to :meth:`add_library_dir` and/or :meth:`set_library_dirs`. - - - .. method:: CCompiler.set_library_dirs(dirs) - - Set the list of library search directories to *dirs* (a list of strings). - This does not affect any standard library search path that the linker may - search by default. - - - .. method:: CCompiler.add_runtime_library_dir(dir) - - Add *dir* to the list of directories that will be searched for shared - libraries at runtime. - - - .. method:: CCompiler.set_runtime_library_dirs(dirs) - - Set the list of directories to search for shared libraries at runtime to - *dirs* (a list of strings). This does not affect any standard search path - that the runtime linker may search by default. - - - .. method:: CCompiler.define_macro(name[, value=None]) - - Define a preprocessor macro for all compilations driven by this compiler - object. The optional parameter *value* should be a string; if it is not - supplied, then the macro will be defined without an explicit value and the - exact outcome depends on the compiler used (XXX true? does ANSI say - anything about this?) - - - .. method:: CCompiler.undefine_macro(name) - - Undefine a preprocessor macro for all compilations driven by this compiler - object. If the same macro is defined by :meth:`define_macro` and - undefined by :meth:`undefine_macro` the last call takes precedence - (including multiple redefinitions or undefinitions). If the macro is - redefined/undefined on a per-compilation basis (i.e. in the call to - :meth:`compile`), then that takes precedence. - - - .. method:: CCompiler.add_link_object(object) - - Add *object* to the list of object files (or analogues, such as explicitly - named library files or the output of "resource compilers") to be included - in every link driven by this compiler object. - - - .. method:: CCompiler.set_link_objects(objects) - - Set the list of object files (or analogues) to be included in every link - to *objects*. This does not affect any standard object files that the - linker may include by default (such as system libraries). - - The following methods implement methods for autodetection of compiler - options, providing some functionality similar to GNU :program:`autoconf`. - - - .. method:: CCompiler.detect_language(sources) - - Detect the language of a given file, or list of files. Uses the instance - attributes :attr:`language_map` (a dictionary), and :attr:`language_order` - (a list) to do the job. - - - .. method:: CCompiler.find_library_file(dirs, lib[, debug=0]) - - Search the specified list of directories for a static or shared library file - *lib* and return the full path to that file. If *debug* is true, look for a - debugging version (if that makes sense on the current platform). Return - ``None`` if *lib* wasn't found in any of the specified directories. - - - .. method:: CCompiler.has_function(funcname [, includes=None, include_dirs=None, libraries=None, library_dirs=None]) - - Return a boolean indicating whether *funcname* is supported on the current - platform. The optional arguments can be used to augment the compilation - environment by providing additional include files and paths and libraries and - paths. - - - .. method:: CCompiler.library_dir_option(dir) - - Return the compiler option to add *dir* to the list of directories searched for - libraries. - - - .. method:: CCompiler.library_option(lib) - - Return the compiler option to add *dir* to the list of libraries linked into the - shared library or executable. - - - .. method:: CCompiler.runtime_library_dir_option(dir) - - Return the compiler option to add *dir* to the list of directories searched for - runtime libraries. - - - .. method:: CCompiler.set_executables(**args) - - Define the executables (and options for them) that will be run to perform the - various stages of compilation. The exact set of executables that may be - specified here depends on the compiler class (via the 'executables' class - attribute), but most will have: - - +--------------+------------------------------------------+ - | attribute | description | - +==============+==========================================+ - | *compiler* | the C/C++ compiler | - +--------------+------------------------------------------+ - | *linker_so* | linker used to create shared objects and | - | | libraries | - +--------------+------------------------------------------+ - | *linker_exe* | linker used to create binary executables | - +--------------+------------------------------------------+ - | *archiver* | static library creator | - +--------------+------------------------------------------+ - - On platforms with a command line (Unix, DOS/Windows), each of these is a string - that will be split into executable name and (optional) list of arguments. - (Splitting the string is done similarly to how Unix shells operate: words are - delimited by spaces, but quotes and backslashes can override this. See - :func:`distutils2.util.split_quoted`.) - - The following methods invoke stages in the build process. - - - .. method:: CCompiler.compile(sources[, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None]) - - Compile one or more source files. Generates object files (e.g. transforms a - :file:`.c` file to a :file:`.o` file.) - - *sources* must be a list of filenames, most likely C/C++ files, but in reality - anything that can be handled by a particular compiler and compiler class (e.g. - :class:`MSVCCompiler` can handle resource files in *sources*). Return a list of - object filenames, one per source filename in *sources*. Depending on the - implementation, not all source files will necessarily be compiled, but all - corresponding object filenames will be returned. - - If *output_dir* is given, object files will be put under it, while retaining - their original path component. That is, :file:`foo/bar.c` normally compiles to - :file:`foo/bar.o` (for a Unix implementation); if *output_dir* is *build*, then - it would compile to :file:`build/foo/bar.o`. - - *macros*, if given, must be a list of macro definitions. A macro definition is - either a ``(name, value)`` 2-tuple or a ``(name,)`` 1-tuple. The former defines - a macro; if the value is ``None``, the macro is defined without an explicit - value. The 1-tuple case undefines a macro. Later - definitions/redefinitions/undefinitions take precedence. - - *include_dirs*, if given, must be a list of strings, the directories to add to - the default include file search path for this compilation only. - - *debug* is a boolean; if true, the compiler will be instructed to output debug - symbols in (or alongside) the object file(s). - - *extra_preargs* and *extra_postargs* are implementation-dependent. On platforms - that have the notion of a command line (e.g. Unix, DOS/Windows), they are most - likely lists of strings: extra command-line arguments to prepend/append to the - compiler command line. On other platforms, consult the implementation class - documentation. In any event, they are intended as an escape hatch for those - occasions when the abstract compiler framework doesn't cut the mustard. - - *depends*, if given, is a list of filenames that all targets depend on. If a - source file is older than any file in depends, then the source file will be - recompiled. This supports dependency tracking, but only at a coarse - granularity. - - Raises :exc:`CompileError` on failure. - - - .. method:: CCompiler.create_static_lib(objects, output_libname[, output_dir=None, debug=0, target_lang=None]) - - Link a bunch of stuff together to create a static library file. The "bunch of - stuff" consists of the list of object files supplied as *objects*, the extra - object files supplied to :meth:`add_link_object` and/or - :meth:`set_link_objects`, the libraries supplied to :meth:`add_library` and/or - :meth:`set_libraries`, and the libraries supplied as *libraries* (if any). - - *output_libname* should be a library name, not a filename; the filename will be - inferred from the library name. *output_dir* is the directory where the library - file will be put. XXX defaults to what? - - *debug* is a boolean; if true, debugging information will be included in the - library (note that on most platforms, it is the compile step where this matters: - the *debug* flag is included here just for consistency). - - *target_lang* is the target language for which the given objects are being - compiled. This allows specific linkage time treatment of certain languages. - - Raises :exc:`LibError` on failure. - - - .. method:: CCompiler.link(target_desc, objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) - - Link a bunch of stuff together to create an executable or shared library file. - - The "bunch of stuff" consists of the list of object files supplied as *objects*. - *output_filename* should be a filename. If *output_dir* is supplied, - *output_filename* is relative to it (i.e. *output_filename* can provide - directory components if needed). - - *libraries* is a list of libraries to link against. These are library names, - not filenames, since they're translated into filenames in a platform-specific - way (e.g. *foo* becomes :file:`libfoo.a` on Unix and :file:`foo.lib` on - DOS/Windows). However, they can include a directory component, which means the - linker will look in that specific directory rather than searching all the normal - locations. - - *library_dirs*, if supplied, should be a list of directories to search for - libraries that were specified as bare library names (i.e. no directory - component). These are on top of the system default and those supplied to - :meth:`add_library_dir` and/or :meth:`set_library_dirs`. *runtime_library_dirs* - is a list of directories that will be embedded into the shared library and used - to search for other shared libraries that \*it\* depends on at run-time. (This - may only be relevant on Unix.) - - *export_symbols* is a list of symbols that the shared library will export. - (This appears to be relevant only on Windows.) - - *debug* is as for :meth:`compile` and :meth:`create_static_lib`, with the - slight distinction that it actually matters on most platforms (as opposed to - :meth:`create_static_lib`, which includes a *debug* flag mostly for form's - sake). - - *extra_preargs* and *extra_postargs* are as for :meth:`compile` (except of - course that they supply command-line arguments for the particular linker being - used). - - *target_lang* is the target language for which the given objects are being - compiled. This allows specific linkage time treatment of certain languages. - - Raises :exc:`LinkError` on failure. - - - .. method:: CCompiler.link_executable(objects, output_progname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, target_lang=None]) - - Link an executable. *output_progname* is the name of the file executable, while - *objects* are a list of object filenames to link in. Other arguments are as for - the :meth:`link` method. - - - .. method:: CCompiler.link_shared_lib(objects, output_libname[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) - - Link a shared library. *output_libname* is the name of the output library, - while *objects* is a list of object filenames to link in. Other arguments are - as for the :meth:`link` method. - - - .. method:: CCompiler.link_shared_object(objects, output_filename[, output_dir=None, libraries=None, library_dirs=None, runtime_library_dirs=None, export_symbols=None, debug=0, extra_preargs=None, extra_postargs=None, build_temp=None, target_lang=None]) - - Link a shared object. *output_filename* is the name of the shared object that - will be created, while *objects* is a list of object filenames to link in. - Other arguments are as for the :meth:`link` method. - - - .. method:: CCompiler.preprocess(source[, output_file=None, macros=None, include_dirs=None, extra_preargs=None, extra_postargs=None]) - - Preprocess a single C/C++ source file, named in *source*. Output will be written - to file named *output_file*, or *stdout* if *output_file* not supplied. - *macros* is a list of macro definitions as for :meth:`compile`, which will - augment the macros set with :meth:`define_macro` and :meth:`undefine_macro`. - *include_dirs* is a list of directory names that will be added to the default - list, in the same way as :meth:`add_include_dir`. - - Raises :exc:`PreprocessError` on failure. - - The following utility methods are defined by the :class:`CCompiler` class, for - use by the various concrete subclasses. - - - .. method:: CCompiler.executable_filename(basename[, strip_dir=0, output_dir='']) - - Returns the filename of the executable for the given *basename*. Typically for - non-Windows platforms this is the same as the basename, while Windows will get - a :file:`.exe` added. - - - .. method:: CCompiler.library_filename(libname[, lib_type='static', strip_dir=0, output_dir='']) - - Returns the filename for the given library name on the current platform. On Unix - a library with *lib_type* of ``'static'`` will typically be of the form - :file:`liblibname.a`, while a *lib_type* of ``'dynamic'`` will be of the form - :file:`liblibname.so`. - - - .. method:: CCompiler.object_filenames(source_filenames[, strip_dir=0, output_dir='']) - - Returns the name of the object files for the given source files. - *source_filenames* should be a list of filenames. - - - .. method:: CCompiler.shared_object_filename(basename[, strip_dir=0, output_dir='']) - - Returns the name of a shared object file for the given file name *basename*. - - - .. method:: CCompiler.execute(func, args[, msg=None, level=1]) - - Invokes :func:`distutils2.util.execute` This method invokes a Python function - *func* with the given arguments *args*, after logging and taking into account - the *dry_run* flag. XXX see also. - - - .. method:: CCompiler.spawn(cmd) - - Invokes :func:`distutils2.util.spawn`. This invokes an external process to run - the given command. XXX see also. - - - .. method:: CCompiler.mkpath(name[, mode=511]) - - Invokes :func:`distutils2.dir_util.mkpath`. This creates a directory and any - missing ancestor directories. XXX see also. - - - .. method:: CCompiler.move_file(src, dst) - - Invokes :meth:`distutils2.file_util.move_file`. Renames *src* to *dst*. XXX see - also. - - - .. method:: CCompiler.announce(msg[, level=1]) - - Write a message using :func:`distutils2.log.debug`. XXX see also. - - - .. method:: CCompiler.warn(msg) - - Write a warning message *msg* to standard error. - - - .. method:: CCompiler.debug_print(msg) - - If the *debug* flag is set on this :class:`CCompiler` instance, print *msg* to - standard output, otherwise do nothing. - -.. % \subsection{Compiler-specific modules} -.. % -.. % The following modules implement concrete subclasses of the abstract -.. % \class{CCompiler} class. They should not be instantiated directly, but should -.. % be created using \function{distutils.ccompiler.new_compiler()} factory -.. % function. - - -:mod:`distutils2.unixccompiler` --- Unix C Compiler -=================================================== - -.. module:: distutils2.unixccompiler - :synopsis: UNIX C Compiler - - -This module provides the :class:`UnixCCompiler` class, a subclass of -:class:`CCompiler` that handles the typical Unix-style command-line C compiler: - -* macros defined with :option:`-Dname[=value]` - -* macros undefined with :option:`-Uname` - -* include search directories specified with :option:`-Idir` - -* libraries specified with :option:`-llib` - -* library search directories specified with :option:`-Ldir` - -* compile handled by :program:`cc` (or similar) executable with :option:`-c` - option: compiles :file:`.c` to :file:`.o` - -* link static library handled by :program:`ar` command (possibly with - :program:`ranlib`) - -* link shared library handled by :program:`cc` :option:`-shared` - - -:mod:`distutils2.msvccompiler` --- Microsoft Compiler -===================================================== - -.. module:: distutils2.msvccompiler - :synopsis: Microsoft Compiler - - -This module provides :class:`MSVCCompiler`, an implementation of the abstract -:class:`CCompiler` class for Microsoft Visual Studio. Typically, extension -modules need to be compiled with the same compiler that was used to compile -Python. For Python 2.3 and earlier, the compiler was Visual Studio 6. For Python -2.4 and 2.5, the compiler is Visual Studio .NET 2003. The AMD64 and Itanium -binaries are created using the Platform SDK. - -:class:`MSVCCompiler` will normally choose the right compiler, linker etc. on -its own. To override this choice, the environment variables *DISTUTILS_USE_SDK* -and *MSSdk* must be both set. *MSSdk* indicates that the current environment has -been setup by the SDK's ``SetEnv.Cmd`` script, or that the environment variables -had been registered when the SDK was installed; *DISTUTILS_USE_SDK* indicates -that the distutils user has made an explicit choice to override the compiler -selection by :class:`MSVCCompiler`. - - -:mod:`distutils2.bcppcompiler` --- Borland Compiler -=================================================== - -.. module:: distutils2.bcppcompiler - - -This module provides :class:`BorlandCCompiler`, an subclass of the abstract -:class:`CCompiler` class for the Borland C++ compiler. - - -:mod:`distutils2.cygwincompiler` --- Cygwin Compiler -==================================================== - -.. module:: distutils2.cygwinccompiler - - -This module provides the :class:`CygwinCCompiler` class, a subclass of -:class:`UnixCCompiler` that handles the Cygwin port of the GNU C compiler to -Windows. It also contains the Mingw32CCompiler class which handles the mingw32 -port of GCC (same as cygwin in no-cygwin mode). - - -:mod:`distutils2.emxccompiler` --- OS/2 EMX Compiler -==================================================== - -.. module:: distutils2.emxccompiler - :synopsis: OS/2 EMX Compiler support - - -This module provides the EMXCCompiler class, a subclass of -:class:`UnixCCompiler` that handles the EMX port of the GNU C compiler to OS/2. - - -:mod:`distutils2.archive_util` --- Archiving utilities -====================================================== - -.. module:: distutils2.archive_util - :synopsis: Utility functions for creating archive files (tarballs, zip files, ...) - - -This module provides a few functions for creating archive files, such as -tarballs or zipfiles. - - -.. function:: make_archive(base_name, format[, root_dir=None, base_dir=None, verbose=0, dry_run=0]) - - Create an archive file (e.g. ``zip`` or ``tar``). *base_name* is the name of - the file to create, minus any format-specific extension; *format* is the - archive format: one of ``zip``, ``tar``, ``ztar``, or ``gztar``. *root_dir* - is a directory that will be the root directory of the archive; i.e. we - typically - ``chdir`` into *root_dir* before creating the archive. *base_dir* is the - directory where we start archiving from; i.e. *base_dir* will be the common - prefix of all files and directories in the archive. *root_dir* and *base_dir* - both default to the current directory. Returns the name of the archive file. - - .. XXX This should be changed to support bz2 files. - - -.. function:: make_tarball(base_name, base_dir[, compress='gzip', verbose=0, dry_run=0]) - - 'Create an (optional compressed) archive as a tar file from all files in and - under *base_dir*. *compress* must be ``'gzip'`` (the default), ``'compress'``, - ``'bzip2'``, or ``None``. Both :program:`tar` and the compression utility named - by *compress* must be on the default program search path, so this is probably - Unix-specific. The output tar file will be named :file:`base_dir.tar`, - possibly plus the appropriate compression extension (:file:`.gz`, :file:`.bz2` - or :file:`.Z`). Return the output filename. - - .. TODO update to reflect use of the :mod:`tarfile` module. - - -.. function:: make_zipfile(base_name, base_dir[, verbose=0, dry_run=0]) - - Create a zip file from all files in and under *base_dir*. The output zip file - will be named *base_name* + :file:`.zip`. Uses either the :mod:`zipfile` Python - module (if available) or the InfoZIP :file:`zip` utility (if installed and - found on the default search path). If neither tool is available, raises - :exc:`DistutilsExecError`. Returns the name of the output zip file. - - -:mod:`distutils2.dep_util` --- Dependency checking -================================================== - -.. module:: distutils2.dep_util - :synopsis: Utility functions for simple dependency checking - - -This module provides functions for performing simple, timestamp-based -dependency of files and groups of files; also, functions based entirely on such -timestamp dependency analysis. - - -.. function:: newer(source, target) - - Return true if *source* exists and is more recently modified than *target*, - or if *source* exists and *target* doesn't. Return false if both exist and - *target* is the same age or newer than *source*. Raise - :exc:`DistutilsFileError` if *source* does not exist. - - -.. function:: newer_pairwise(sources, targets) - - Walk two filename lists in parallel, testing if each source is newer than its - corresponding target. Return a pair of lists (*sources*, *targets*) where - source is newer than target, according to the semantics of :func:`newer` - - .. % % equivalent to a listcomp... - - -.. function:: newer_group(sources, target[, missing='error']) - - Return true if *target* is out-of-date with respect to any file listed in - *sources*. In other words, if *target* exists and is newer than every file - in *sources*, return false; otherwise return true. *missing* controls what - we do when a source file is missing; the default (``'error'``) is to blow up - with an :exc:`OSError` from inside :func:`os.stat`; if it is ``'ignore'``, we - silently drop any missing source files; if it is ``'newer'``, any missing - source files make us assume that *target* is out-of-date (this is handy in - "dry-run" mode: it'll make you pretend to carry out commands that wouldn't - work because inputs are missing, but that doesn't matter because you're not - actually going to run the commands). - - -:mod:`distutils2.dir_util` --- Directory tree operations -======================================================== - -.. module:: distutils2.dir_util - :synopsis: Utility functions for operating on directories and directory trees - - -This module provides functions for operating on directories and trees of -directories. - - -.. function:: mkpath(name[, mode=0777, verbose=0, dry_run=0]) - - Create a directory and any missing ancestor directories. If the directory - already exists (or if *name* is the empty string, which means the current - directory, which of course exists), then do nothing. Raise - :exc:`DistutilsFileError` if unable to create some directory along the way - (e.g. some sub-path exists, but is a file rather than a directory). If - *verbose* is true, print a one-line summary of each mkdir to stdout. Return - the list of directories actually created. - - -.. function:: create_tree(base_dir, files[, mode=0777, verbose=0, dry_run=0]) - - Create all the empty directories under *base_dir* needed to put *files* - there. *base_dir* is just the a name of a directory which doesn't necessarily - exist yet; *files* is a list of filenames to be interpreted relative to - *base_dir*. *base_dir* + the directory portion of every file in *files* will - be created if it doesn't already exist. *mode*, *verbose* and *dry_run* - flags are as for :func:`mkpath`. - - -.. function:: copy_tree(src, dst[, preserve_mode=1, preserve_times=1, preserve_symlinks=0, update=0, verbose=0, dry_run=0]) - - Copy an entire directory tree *src* to a new location *dst*. Both *src* and - *dst* must be directory names. If *src* is not a directory, raise - :exc:`DistutilsFileError`. If *dst* does not exist, it is created with - :func:`mkpath`. The end result of the copy is that every file in *src* is - copied to *dst*, and directories under *src* are recursively copied to - *dst*. Return the list of files that were copied or might have been copied, - using their output name. The return value is unaffected by *update* or - *dry_run*: it is simply the list of all files under *src*, with the names - changed to be under *dst*. - - *preserve_mode* and *preserve_times* are the same as for :func:`copy_file` - in :mod:`distutils2.file_util`; note that they only apply to regular files, - not to directories. If *preserve_symlinks* is true, symlinks will be copied - as symlinks (on platforms that support them!); otherwise (the default), the - destination of the symlink will be copied. *update* and *verbose* are the - same as for :func:`copy_file`. - - -.. function:: remove_tree(directory[, verbose=0, dry_run=0]) - - Recursively remove *directory* and all files and directories underneath it. - Any errors are ignored (apart from being reported to ``sys.stdout`` if - *verbose* is true). - -.. XXX Some of this could be replaced with the shutil module? - - -:mod:`distutils2.file_util` --- Single file operations -====================================================== - -.. module:: distutils2.file_util - :synopsis: Utility functions for operating on single files - - -This module contains some utility functions for operating on individual files. - - -.. function:: copy_file(src, dst[, preserve_mode=1, preserve_times=1, update=0, link=None, verbose=0, dry_run=0]) - - Copy file *src* to *dst*. If *dst* is a directory, then *src* is copied there - with the same name; otherwise, it must be a filename. (If the file exists, it - will be ruthlessly clobbered.) If *preserve_mode* is true (the default), the - file's mode (type and permission bits, or whatever is analogous on the - current platform) is copied. If *preserve_times* is true (the default), the - last-modified and last-access times are copied as well. If *update* is true, - *src* will only be copied if *dst* does not exist, or if *dst* does exist but - is older than *src*. - - *link* allows you to make hard links (using :func:`os.link`) or symbolic - links (using :func:`os.symlink`) instead of copying: set it to ``'hard'`` or - ``'sym'``; if it is ``None`` (the default), files are copied. Don't set - *link* on systems that don't support it: :func:`copy_file` doesn't check if - hard or symbolic linking is available. It uses :func:`_copy_file_contents` - to copy file contents. - - Return a tuple ``(dest_name, copied)``: *dest_name* is the actual name of - the output file, and *copied* is true if the file was copied (or would have - been copied, if *dry_run* true). - - .. % XXX if the destination file already exists, we clobber it if - .. % copying, but blow up if linking. Hmmm. And I don't know what - .. % macostools.copyfile() does. Should definitely be consistent, and - .. % should probably blow up if destination exists and we would be - .. % changing it (i.e. it's not already a hard/soft link to src OR - .. % (not update) and (src newer than dst)). - - -.. function:: move_file(src, dst[, verbose, dry_run]) - - Move file *src* to *dst*. If *dst* is a directory, the file will be moved - into it with the same name; otherwise, *src* is just renamed to *dst*. - Returns the new full name of the file. - - .. warning:: - - Handles cross-device moves on Unix using :func:`copy_file`. What about - other systems? - - -.. function:: write_file(filename, contents) - - Create a file called *filename* and write *contents* (a sequence of strings - without line terminators) to it. - -:mod:`distutils2.metadata` --- Metadata handling -================================================================ - -.. module:: distutils2.metadata - -.. FIXME CPython/stdlib docs don't use autoclass, write doc manually here - -.. autoclass:: distutils2.metadata.Metadata - :members: - -:mod:`distutils2.util` --- Miscellaneous other utility functions -================================================================ - -.. module:: distutils2.util - :synopsis: Miscellaneous other utility functions - - -This module contains other assorted bits and pieces that don't fit into any -other utility module. - - -.. function:: get_platform() - - Return a string that identifies the current platform. This is used mainly to - distinguish platform-specific build directories and platform-specific built - distributions. Typically includes the OS name and version and the - architecture (as supplied by 'os.uname()'), although the exact information - included depends on the OS; e.g. for IRIX the architecture isn't particularly - important (IRIX only runs on SGI hardware), but for Linux the kernel version - isn't particularly important. - - Examples of returned values: - - * ``linux-i586`` - * ``linux-alpha`` - * ``solaris-2.6-sun4u`` - * ``irix-5.3`` - * ``irix64-6.2`` - - For non-POSIX platforms, currently just returns ``sys.platform``. - - For Mac OS X systems the OS version reflects the minimal version on which - binaries will run (that is, the value of ``MACOSX_DEPLOYMENT_TARGET`` - during the build of Python), not the OS version of the current system. - - For universal binary builds on Mac OS X the architecture value reflects - the univeral binary status instead of the architecture of the current - processor. For 32-bit universal binaries the architecture is ``fat``, - for 64-bit universal binaries the architecture is ``fat64``, and - for 4-way universal binaries the architecture is ``universal``. Starting - from Python 2.7 and Python 3.2 the architecture ``fat3`` is used for - a 3-way universal build (ppc, i386, x86_64) and ``intel`` is used for - a univeral build with the i386 and x86_64 architectures - - Examples of returned values on Mac OS X: - - * ``macosx-10.3-ppc`` - - * ``macosx-10.3-fat`` - - * ``macosx-10.5-universal`` - - * ``macosx-10.6-intel`` - - .. XXX reinvention of platform module? - - -.. function:: convert_path(pathname) - - Return 'pathname' as a name that will work on the native filesystem, i.e. - split it on '/' and put it back together again using the current directory - separator. Needed because filenames in the setup script are always supplied - in Unix style, and have to be converted to the local convention before we - can actually use them in the filesystem. Raises :exc:`ValueError` on - non-Unix-ish systems if *pathname* either starts or ends with a slash. - - -.. function:: change_root(new_root, pathname) - - Return *pathname* with *new_root* prepended. If *pathname* is relative, this - is equivalent to ``os.path.join(new_root,pathname)`` Otherwise, it requires - making *pathname* relative and then joining the two, which is tricky on - DOS/Windows. - - -.. function:: check_environ() - - Ensure that 'os.environ' has all the environment variables we guarantee that - users can use in config files, command-line options, etc. Currently this - includes: - - * :envvar:`HOME` - user's home directory (Unix only) - * :envvar:`PLAT` - description of the current platform, including hardware - and OS (see :func:`get_platform`) - - -.. function:: subst_vars(s, local_vars) - - Perform shell/Perl-style variable substitution on *s*. Every occurrence of - ``$`` followed by a name is considered a variable, and variable is - substituted by the value found in the *local_vars* dictionary, or in - ``os.environ`` if it's not in *local_vars*. *os.environ* is first - checked/augmented to guarantee that it contains certain values: see - :func:`check_environ`. Raise :exc:`ValueError` for any variables not found - in either *local_vars* or ``os.environ``. - - Note that this is not a fully-fledged string interpolation function. A valid - ``$variable`` can consist only of upper and lower case letters, numbers and - an underscore. No { } or ( ) style quoting is available. - - -.. function:: split_quoted(s) - - Split a string up according to Unix shell-like rules for quotes and - backslashes. In short: words are delimited by spaces, as long as those spaces - are not escaped by a backslash, or inside a quoted string. Single and double - quotes are equivalent, and the quote characters can be backslash-escaped. - The backslash is stripped from any two-character escape sequence, leaving - only the escaped character. The quote characters are stripped from any - quoted string. Returns a list of words. - - .. % Should probably be moved into the standard library. - - -.. function:: execute(func, args[, msg=None, verbose=0, dry_run=0]) - - Perform some action that affects the outside world (for instance, writing to - the filesystem). Such actions are special because they are disabled by the - *dry_run* flag. This method takes care of all that bureaucracy for you; - all you have to do is supply the function to call and an argument tuple for - it (to embody the "external action" being performed), and an optional message - to print. - - -.. function:: strtobool(val) - - Convert a string representation of truth to true (1) or false (0). - - True values are ``y``, ``yes``, ``t``, ``true``, ``on`` and ``1``; false - values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``. Raises - :exc:`ValueError` if *val* is anything else. - -.. TODO Add :term: markup to bytecode when merging into the stdlib - -.. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None]) - - Byte-compile a collection of Python source files to either :file:`.pyc` or - :file:`.pyo` files in the same directory. *py_files* is a list of files to - compile; any files that don't end in :file:`.py` are silently skipped. - *optimize* must be one of the following: - - * ``0`` - don't optimize (generate :file:`.pyc`) - * ``1`` - normal optimization (like ``python -O``) - * ``2`` - extra optimization (like ``python -OO``) - - If *force* is true, all files are recompiled regardless of timestamps. - - The source filename encoded in each bytecode file defaults to the filenames - listed in *py_files*; you can modify these with *prefix* and *basedir*. - *prefix* is a string that will be stripped off of each source filename, and - *base_dir* is a directory name that will be prepended (after *prefix* is - stripped). You can supply either or both (or neither) of *prefix* and - *base_dir*, as you wish. - - If *dry_run* is true, doesn't actually do anything that would affect the - filesystem. - - Byte-compilation is either done directly in this interpreter process with the - standard :mod:`py_compile` module, or indirectly by writing a temporary - script and executing it. Normally, you should let :func:`byte_compile` - figure out to use direct compilation or not (see the source for details). - The *direct* flag is used by the script generated in indirect mode; unless - you know what you're doing, leave it set to ``None``. - - -.. function:: rfc822_escape(header) - - Return a version of *header* escaped for inclusion in an :rfc:`822` header, by - ensuring there are 8 spaces space after each newline. Note that it does no - other modification of the string. - - .. % this _can_ be replaced - -.. % \subsection{Distutils objects} - - -:mod:`distutils2.dist` --- The Distribution class -================================================= - -.. module:: distutils2.dist - :synopsis: Provides the Distribution class, which represents the module - distribution being built/installed/distributed - - -This module provides the :class:`Distribution` class, which represents the -module distribution being built/installed/distributed. - - -:mod:`distutils2.extension` --- The Extension class -=================================================== - -.. module:: distutils2.extension - :synopsis: Provides the Extension class, used to describe C/C++ extension - modules in setup scripts - - -This module provides the :class:`Extension` class, used to describe C/C++ -extension modules in setup scripts. - -.. % \subsection{Ungrouped modules} -.. % The following haven't been moved into a more appropriate section yet. - - -:mod:`distutils2.debug` --- Distutils debug mode -================================================ - -.. module:: distutils2.debug - :synopsis: Provides the debug flag for distutils - - -This module provides the DEBUG flag. - - -:mod:`distutils2.errors` --- Distutils exceptions -================================================= - -.. module:: distutils2.errors - :synopsis: Provides standard distutils exceptions - - -Provides exceptions used by the Distutils modules. Note that Distutils modules -may raise standard exceptions; in particular, SystemExit is usually raised for -errors that are obviously the end-user's fault (e.g. bad command-line arguments). - -This module is safe to use in ``from ... import *`` mode; it only exports -symbols whose names start with ``Distutils`` and end with ``Error``. - - -:mod:`distutils2.fancy_getopt` --- Wrapper around the standard getopt module -============================================================================ - -.. module:: distutils2.fancy_getopt - :synopsis: Additional getopt functionality - - -This module provides a wrapper around the standard :mod:`getopt` module that -provides the following additional features: - -* short and long options are tied together - -* options have help strings, so :func:`fancy_getopt` could potentially create a - complete usage summary - -* options set attributes of a passed-in object - -* boolean options can have "negative aliases" --- e.g. if :option:`--quiet` is - the "negative alias" of :option:`--verbose`, then :option:`--quiet` on the - command line sets *verbose* to false. - -.. XXX Should be replaced with :mod:`argparse`. - - -.. function:: fancy_getopt(options, negative_opt, object, args) - - Wrapper function. *options* is a list of ``(long_option, short_option, - help_string)`` 3-tuples as described in the constructor for - :class:`FancyGetopt`. *negative_opt* should be a dictionary mapping option names - to option names, both the key and value should be in the *options* list. - *object* is an object which will be used to store values (see the :meth:`getopt` - method of the :class:`FancyGetopt` class). *args* is the argument list. Will use - ``sys.argv[1:]`` if you pass ``None`` as *args*. - - -.. function:: wrap_text(text, width) - - Wraps *text* to less than *width* wide. - - .. XXX Should be replaced with :mod:`textwrap` (which is available in Python - 2.3 and later). - - -.. class:: FancyGetopt([option_table=None]) - - The option_table is a list of 3-tuples: ``(long_option, short_option, - help_string)`` - - If an option takes an argument, its *long_option* should have ``'='`` appended; - *short_option* should just be a single character, no ``':'`` in any case. - *short_option* should be ``None`` if a *long_option* doesn't have a - corresponding *short_option*. All option tuples must have long options. - -The :class:`FancyGetopt` class provides the following methods: - - -.. method:: FancyGetopt.getopt([args=None, object=None]) - - Parse command-line options in args. Store as attributes on *object*. - - If *args* is ``None`` or not supplied, uses ``sys.argv[1:]``. If *object* is - ``None`` or not supplied, creates a new :class:`OptionDummy` instance, stores - option values there, and returns a tuple ``(args, object)``. If *object* is - supplied, it is modified in place and :func:`getopt` just returns *args*; in - both cases, the returned *args* is a modified copy of the passed-in *args* list, - which is left untouched. - - .. % and args returned are? - - -.. method:: FancyGetopt.get_option_order() - - Returns the list of ``(option, value)`` tuples processed by the previous run of - :meth:`getopt` Raises :exc:`RuntimeError` if :meth:`getopt` hasn't been called - yet. - - -.. method:: FancyGetopt.generate_help([header=None]) - - Generate help text (a list of strings, one per suggested line of output) from - the option table for this :class:`FancyGetopt` object. - - If supplied, prints the supplied *header* at the top of the help. - - -:mod:`distutils2.filelist` --- The FileList class -================================================= - -.. module:: distutils2.filelist - :synopsis: The FileList class, used for poking about the file system and - building lists of files. - - -This module provides the :class:`FileList` class, used for poking about the -filesystem and building lists of files. - -.. TODO move to util - - -:mod:`distutils2.log` --- Simple PEP 282-style logging -====================================================== - -.. module:: distutils2.log - :synopsis: A simple logging mechanism, 282-style - - -.. XXX Should be replaced with standard :mod:`logging` module. - - - -:mod:`distutils2.spawn` --- Spawn a sub-process -=============================================== - -.. module:: distutils2.spawn - :synopsis: Provides the spawn() function - - -This module provides the :func:`spawn` function, a front-end to various -platform-specific functions for launching another program in a sub-process. -Also provides :func:`find_executable` to search the path for a given executable -name. - - -:mod:`distutils2.sysconfig` --- System configuration information -================================================================ - -.. module:: distutils2.sysconfig - :synopsis: Low-level access to configuration information of the Python interpreter. -.. moduleauthor:: Fred L. Drake, Jr. -.. moduleauthor:: Greg Ward -.. sectionauthor:: Fred L. Drake, Jr. - - -The :mod:`distutils2.sysconfig` module provides access to Python's low-level -configuration information. The specific configuration variables available -depend heavily on the platform and configuration. The specific variables depend -on the build process for the specific version of Python being run; the variables -are those found in the :file:`Makefile` and configuration header that are -installed with Python on Unix systems. The configuration header is called -:file:`pyconfig.h` for Python versions starting with 2.2, and :file:`config.h` -for earlier versions of Python. - -Some additional functions are provided which perform some useful manipulations -for other parts of the :mod:`distutils2` package. - - -.. data:: PREFIX - - The result of ``os.path.normpath(sys.prefix)``. - - -.. data:: EXEC_PREFIX - - The result of ``os.path.normpath(sys.exec_prefix)``. - - -.. function:: get_config_var(name) - - Return the value of a single variable. This is equivalent to - ``get_config_vars().get(name)``. - - -.. function:: get_config_vars(...) - - Return a set of variable definitions. If there are no arguments, this returns a - dictionary mapping names of configuration variables to values. If arguments are - provided, they should be strings, and the return value will be a sequence giving - the associated values. If a given name does not have a corresponding value, - ``None`` will be included for that variable. - - -.. function:: get_config_h_filename() - - Return the full path name of the configuration header. For Unix, this will be - the header generated by the :program:`configure` script; for other platforms the - header will have been supplied directly by the Python source distribution. The - file is a platform-specific text file. - - -.. function:: get_makefile_filename() - - Return the full path name of the :file:`Makefile` used to build Python. For - Unix, this will be a file generated by the :program:`configure` script; the - meaning for other platforms will vary. The file is a platform-specific text - file, if it exists. This function is only useful on POSIX platforms. - - -.. function:: get_python_inc([plat_specific[, prefix]]) - - Return the directory for either the general or platform-dependent C include - files. If *plat_specific* is true, the platform-dependent include directory is - returned; if false or omitted, the platform-independent directory is returned. - If *prefix* is given, it is used as either the prefix instead of - :const:`PREFIX`, or as the exec-prefix instead of :const:`EXEC_PREFIX` if - *plat_specific* is true. - - -.. function:: get_python_lib([plat_specific[, standard_lib[, prefix]]]) - - Return the directory for either the general or platform-dependent library - installation. If *plat_specific* is true, the platform-dependent include - directory is returned; if false or omitted, the platform-independent directory - is returned. If *prefix* is given, it is used as either the prefix instead of - :const:`PREFIX`, or as the exec-prefix instead of :const:`EXEC_PREFIX` if - *plat_specific* is true. If *standard_lib* is true, the directory for the - standard library is returned rather than the directory for the installation of - third-party extensions. - -The following function is only intended for use within the :mod:`distutils2` -package. - - -.. function:: customize_compiler(compiler) - - Do any platform-specific customization of a - :class:`distutils2.ccompiler.CCompiler` instance. - - This function is only needed on Unix at this time, but should be called - consistently to support forward-compatibility. It inserts the information that - varies across Unix flavors and is stored in Python's :file:`Makefile`. This - information includes the selected compiler, compiler and linker options, and the - extension used by the linker for shared objects. - -This function is even more special-purpose, and should only be used from -Python's own build procedures. - - -.. function:: set_python_build() - - Inform the :mod:`distutils2.sysconfig` module that it is being used as part of - the build process for Python. This changes a lot of relative locations for - files, allowing them to be located in the build area rather than in an installed - Python. - - -:mod:`distutils2.text_file` --- The TextFile class -================================================== - -.. module:: distutils2.text_file - :synopsis: provides the TextFile class, a simple interface to text files - - -This module provides the :class:`TextFile` class, which gives an interface to -text files that (optionally) takes care of stripping comments, ignoring blank -lines, and joining lines with backslashes. - - -.. class:: TextFile([filename=None, file=None, **options]) - - This class provides a file-like object that takes care of all the things you - commonly want to do when processing a text file that has some line-by-line - syntax: strip comments (as long as ``#`` is your comment character), skip blank - lines, join adjacent lines by escaping the newline (i.e. backslash at end of - line), strip leading and/or trailing whitespace. All of these are optional and - independently controllable. - - The class provides a :meth:`warn` method so you can generate warning messages - that report physical line number, even if the logical line in question spans - multiple physical lines. Also provides :meth:`unreadline` for implementing - line-at-a-time lookahead. - - :class:`TextFile` instances are create with either *filename*, *file*, or both. - :exc:`RuntimeError` is raised if both are ``None``. *filename* should be a - string, and *file* a file object (or something that provides :meth:`readline` - and :meth:`close` methods). It is recommended that you supply at least - *filename*, so that :class:`TextFile` can include it in warning messages. If - *file* is not supplied, :class:`TextFile` creates its own using the - :func:`open` built-in function. - - The options are all boolean, and affect the values returned by :meth:`readline` - - +------------------+--------------------------------+---------+ - | option name | description | default | - +==================+================================+=========+ - | *strip_comments* | strip from ``'#'`` to end-of- | true | - | | line, as well as any | | - | | whitespace leading up to the | | - | | ``'#'``\ ---unless it is | | - | | escaped by a backslash | | - +------------------+--------------------------------+---------+ - | *lstrip_ws* | strip leading whitespace from | false | - | | each line before returning it | | - +------------------+--------------------------------+---------+ - | *rstrip_ws* | strip trailing whitespace | true | - | | (including line terminator!) | | - | | from each line before | | - | | returning it. | | - +------------------+--------------------------------+---------+ - | *skip_blanks* | skip lines that are empty | true | - | | \*after\* stripping comments | | - | | and whitespace. (If both | | - | | lstrip_ws and rstrip_ws are | | - | | false, then some lines may | | - | | consist of solely whitespace: | | - | | these will \*not\* be skipped, | | - | | even if *skip_blanks* is | | - | | true.) | | - +------------------+--------------------------------+---------+ - | *join_lines* | if a backslash is the last | false | - | | non-newline character on a | | - | | line after stripping comments | | - | | and whitespace, join the | | - | | following line to it to form | | - | | one logical line; if N | | - | | consecutive lines end with a | | - | | backslash, then N+1 physical | | - | | lines will be joined to form | | - | | one logical line. | | - +------------------+--------------------------------+---------+ - | *collapse_join* | strip leading whitespace from | false | - | | lines that are joined to their | | - | | predecessor; only matters if | | - | | ``(join_lines and not | | - | | lstrip_ws)`` | | - +------------------+--------------------------------+---------+ - - Note that since *rstrip_ws* can strip the trailing newline, the semantics of - :meth:`readline` must differ from those of the built-in file object's - :meth:`readline` method! In particular, :meth:`readline` returns ``None`` for - end-of-file: an empty string might just be a blank line (or an all-whitespace - line), if *rstrip_ws* is true but *skip_blanks* is not. - - - .. method:: TextFile.open(filename) - - Open a new file *filename*. This overrides any *file* or *filename* - constructor arguments. - - - .. method:: TextFile.close() - - Close the current file and forget everything we know about it (including the - filename and the current line number). - - - .. method:: TextFile.warn(msg[,line=None]) - - Print (to stderr) a warning message tied to the current logical line in the - current file. If the current logical line in the file spans multiple physical - lines, the warning refers to the whole range, such as ``"lines 3-5"``. If - *line* is supplied, it overrides the current line number; it may be a list or - tuple to indicate a range of physical lines, or an integer for a single - physical line. - - - .. method:: TextFile.readline() - - Read and return a single logical line from the current file (or from an internal - buffer if lines have previously been "unread" with :meth:`unreadline`). If the - *join_lines* option is true, this may involve reading multiple physical lines - concatenated into a single string. Updates the current line number, so calling - :meth:`warn` after :meth:`readline` emits a warning about the physical line(s) - just read. Returns ``None`` on end-of-file, since the empty string can occur - if *rstrip_ws* is true but *strip_blanks* is not. - - - .. method:: TextFile.readlines() - - Read and return the list of all logical lines remaining in the current file. - This updates the current line number to the last line of the file. - - - .. method:: TextFile.unreadline(line) - - Push *line* (a string) onto an internal buffer that will be checked by future - :meth:`readline` calls. Handy for implementing a parser with line-at-a-time - lookahead. Note that lines that are "unread" with :meth:`unreadline` are not - subsequently re-cleansed (whitespace stripped, or whatever) when read with - :meth:`readline`. If multiple calls are made to :meth:`unreadline` before a call - to :meth:`readline`, the lines will be returned most in most recent first order. - - -:mod:`distutils2.version` --- Version number classes -==================================================== - -.. module:: distutils2.version - :synopsis: implements classes that represent module version numbers. - - -.. % todo -.. % \section{Distutils Commands} -.. % -.. % This part of Distutils implements the various Distutils commands, such -.. % as \code{build}, \code{install} \&c. Each command is implemented as a -.. % separate module, with the command name as the name of the module. - - -:mod:`distutils2.cmd` --- Abstract base class for Distutils commands -==================================================================== - -.. module:: distutils2.cmd - :synopsis: This module provides the abstract base class Command. This class - is subclassed by the modules in the distutils.command subpackage. - - -This module supplies the abstract base class :class:`Command`. - - -.. class:: Command(dist) - - Abstract base class for defining command classes, the "worker bees" of the - Distutils. A useful analogy for command classes is to think of them as - subroutines with local variables called *options*. The options are declared - in :meth:`initialize_options` and defined (given their final values) in - :meth:`finalize_options`, both of which must be defined by every command - class. The distinction between the two is necessary because option values - might come from the outside world (command line, config file, ...), and any - options dependent on other options must be computed after these outside - influences have been processed --- hence :meth:`finalize_options`. The body - of the subroutine, where it does all its work based on the values of its - options, is the :meth:`run` method, which must also be implemented by every - command class. - - The class constructor takes a single argument *dist*, a :class:`Distribution` - instance. - - -.. % todo - -:mod:`distutils2.command.check` --- Check the metadata of a package -=================================================================== - -.. module:: distutils2.command.check - :synopsis: Check the metadata of a package - - -The ``check`` command performs some tests on the metadata of a package. -For example, it verifies that all required metadata are provided as -the arguments passed to the :func:`setup` function. - -.. % todo - -Creating a new Distutils command -================================ - -This section outlines the steps to create a new Distutils command. - -A new command lives in a module in the :mod:`distutils2.command` package. There -is a sample template in that directory called :file:`command_template`. Copy -this file to a new module with the same name as the new command you're -implementing. This module should implement a class with the same name as the -module (and the command). So, for instance, to create the command -``peel_banana`` (so that users can run ``setup.py peel_banana``), you'd copy -:file:`command_template` to :file:`distutils2/command/peel_banana.py`, then edit -it so that it's implementing the class :class:`peel_banana`, a subclass of -:class:`distutils2.cmd.Command`. - -Subclasses of :class:`Command` must define the following methods. - -.. method:: Command.initialize_options() - - Set default values for all the options that this command supports. Note that - these defaults may be overridden by other commands, by the setup script, by - config files, or by the command line. Thus, this is not the place to code - dependencies between options; generally, :meth:`initialize_options` - implementations are just a bunch of ``self.foo = None`` assignments. - - -.. method:: Command.finalize_options() - - Set final values for all the options that this command supports. This is - always called as late as possible, i.e. after any option assignments from the - command line or from other commands have been done. Thus, this is the place - to to code option dependencies: if *foo* depends on *bar*, then it is safe to - set *foo* from *bar* as long as *foo* still has the same value it was - assigned in :meth:`initialize_options`. - - -.. method:: Command.run() - - A command's raison d'etre: carry out the action it exists to perform, - controlled by the options initialized in :meth:`initialize_options`, - customized by other commands, the setup script, the command line, and config - files, and finalized in :meth:`finalize_options`. All terminal output and - filesystem interaction should be done by :meth:`run`. - - -.. attribute:: Command.sub_commands - - *sub_commands* formalizes the notion of a "family" of commands, - e.g. ``install`` as the parent with sub-commands ``install_lib``, - ``install_headers``, etc. The parent of a family of commands defines - *sub_commands* as a class attribute; it's a list of 2-tuples ``(command_name, - predicate)``, with *command_name* a string and *predicate* a function, a - string or ``None``. *predicate* is a method of the parent command that - determines whether the corresponding command is applicable in the current - situation. (E.g. ``install_headers`` is only applicable if we have any C - header files to install.) If *predicate* is ``None``, that command is always - applicable. - - *sub_commands* is usually defined at the *end* of a class, because - predicates can be methods of the class, so they must already have been - defined. The canonical example is the :command:`install` command. - - -:mod:`distutils2.command` --- Individual Distutils commands -=========================================================== - -.. module:: distutils2.command - :synopsis: This subpackage contains one module for each standard Distutils command. - - -.. % \subsubsection{Individual Distutils commands} -.. % todo - - -:mod:`distutils2.command.bdist` --- Build a binary installer -============================================================ - -.. module:: distutils2.command.bdist - :synopsis: Build a binary installer for a package - - -.. % todo - - -:mod:`distutils2.command.bdist_dumb` --- Build a "dumb" installer -================================================================= - -.. module:: distutils2.command.bdist_dumb - :synopsis: Build a "dumb" installer - a simple archive of files - - -.. % todo - - -:mod:`distutils2.command.bdist_msi` --- Build a Microsoft Installer binary package -================================================================================== - -.. module:: distutils2.command.bdist_msi - :synopsis: Build a binary distribution as a Windows MSI file - -.. class:: bdist_msi(Command) - - Builds a `Windows Installer`_ (.msi) binary package. - - .. _Windows Installer: http://msdn.microsoft.com/en-us/library/cc185688(VS.85).aspx - - In most cases, the ``bdist_msi`` installer is a better choice than the - ``bdist_wininst`` installer, because it provides better support for - Win64 platforms, allows administrators to perform non-interactive - installations, and allows installation through group policies. - - -:mod:`distutils2.command.bdist_wininst` --- Build a Windows installer -===================================================================== - -.. module:: distutils2.command.bdist_wininst - :synopsis: Build a Windows installer - - -.. % todo - - -:mod:`distutils2.command.sdist` --- Build a source distribution -=============================================================== - -.. module:: distutils2.command.sdist - :synopsis: Build a source distribution - - -.. % todo - - -:mod:`distutils2.command.build` --- Build all files of a package -================================================================ - -.. module:: distutils2.command.build - :synopsis: Build all files of a package - - -.. % todo - - -:mod:`distutils2.command.build_clib` --- Build any C libraries in a package -=========================================================================== - -.. module:: distutils2.command.build_clib - :synopsis: Build any C libraries in a package - - -.. % todo - - -:mod:`distutils2.command.build_ext` --- Build any extensions in a package -========================================================================= - -.. module:: distutils2.command.build_ext - :synopsis: Build any extensions in a package - - -.. % todo - - -:mod:`distutils2.command.build_py` --- Build the .py/.pyc files of a package -============================================================================ - -.. module:: distutils2.command.build_py - :synopsis: Build the .py/.pyc files of a package - - -.. class:: build_py(Command) - - -:mod:`distutils2.command.build_scripts` --- Build the scripts of a package -========================================================================== - -.. module:: distutils2.command.build_scripts - :synopsis: Build the scripts of a package - - -.. % todo - - -:mod:`distutils2.command.clean` --- Clean a package build area -============================================================== - -.. module:: distutils2.command.clean - :synopsis: Clean a package build area - - -.. % todo - - -:mod:`distutils2.command.config` --- Perform package configuration -================================================================== - -.. module:: distutils2.command.config - :synopsis: Perform package configuration - - -.. % todo - - -:mod:`distutils2.command.install` --- Install a package -======================================================= - -.. module:: distutils2.command.install - :synopsis: Install a package - - -.. % todo - - -:mod:`distutils2.command.install_data` --- Install data files from a package -============================================================================ - -.. module:: distutils2.command.install_data - :synopsis: Install data files from a package - - -.. % todo - - -:mod:`distutils2.command.install_headers` --- Install C/C++ header files from a package -======================================================================================= - -.. module:: distutils2.command.install_headers - :synopsis: Install C/C++ header files from a package - - -.. % todo - - -:mod:`distutils2.command.install_lib` --- Install library files from a package -============================================================================== - -.. module:: distutils2.command.install_lib - :synopsis: Install library files from a package - - -.. % todo - - -:mod:`distutils2.command.install_scripts` --- Install script files from a package -================================================================================= - -.. module:: distutils2.command.install_scripts - :synopsis: Install script files from a package - - -.. % todo - - -:mod:`distutils2.command.register` --- Register a module with the Python Package Index -====================================================================================== - -.. module:: distutils2.command.register - :synopsis: Register a module with the Python Package Index - - -The ``register`` command registers the package with the Python Package Index. -This is described in more detail in :PEP:`301`. - -.. % todo diff --git a/docs/source/distutils/builtdist.rst b/docs/source/distutils/builtdist.rst deleted file mode 100644 --- a/docs/source/distutils/builtdist.rst +++ /dev/null @@ -1,454 +0,0 @@ -.. _built-dist: - -**************************** -Creating Built Distributions -**************************** - -A "built distribution" is what you're probably used to thinking of either as a -"binary package" or an "installer" (depending on your background). It's not -necessarily binary, though, because it might contain only Python source code -and/or byte-code; and we don't call it a package, because that word is already -spoken for in Python. (And "installer" is a term specific to the world of -mainstream desktop systems.) - -A built distribution is how you make life as easy as possible for installers of -your module distribution: for users of RPM-based Linux systems, it's a binary -RPM; for Windows users, it's an executable installer; for Debian-based Linux -users, it's a Debian package; and so forth. Obviously, no one person will be -able to create built distributions for every platform under the sun, so the -Distutils are designed to enable module developers to concentrate on their -specialty---writing code and creating source distributions---while an -intermediary species called *packagers* springs up to turn source distributions -into built distributions for as many platforms as there are packagers. - -Of course, the module developer could be his own packager; or the packager could -be a volunteer "out there" somewhere who has access to a platform which the -original developer does not; or it could be software periodically grabbing new -source distributions and turning them into built distributions for as many -platforms as the software has access to. Regardless of who they are, a packager -uses the setup script and the :command:`bdist` command family to generate built -distributions. - -As a simple example, if I run the following command in the Distutils source -tree:: - - python setup.py bdist - -then the Distutils builds my module distribution (the Distutils itself in this -case), does a "fake" installation (also in the :file:`build` directory), and -creates the default type of built distribution for my platform. The default -format for built distributions is a "dumb" tar file on Unix, and a simple -executable installer on Windows. (That tar file is considered "dumb" because it -has to be unpacked in a specific location to work.) - -Thus, the above command on a Unix system creates -:file:`Distutils-1.0.{plat}.tar.gz`; unpacking this tarball from the right place -installs the Distutils just as though you had downloaded the source distribution -and run ``python setup.py install``. (The "right place" is either the root of -the filesystem or Python's :file:`{prefix}` directory, depending on the options -given to the :command:`bdist_dumb` command; the default is to make dumb -distributions relative to :file:`{prefix}`.) - -Obviously, for pure Python distributions, this isn't any simpler than just -running ``python setup.py install``\ ---but for non-pure distributions, which -include extensions that would need to be compiled, it can mean the difference -between someone being able to use your extensions or not. And creating "smart" -built distributions, such as an RPM package or an executable installer for -Windows, is far more convenient for users even if your distribution doesn't -include any extensions. - -The :command:`bdist` command has a :option:`--formats` option, similar to the -:command:`sdist` command, which you can use to select the types of built -distribution to generate: for example, :: - - python setup.py bdist --format=zip - -would, when run on a Unix system, create :file:`Distutils-1.0.{plat}.zip`\ ----again, this archive would be unpacked from the root directory to install the -Distutils. - -The available formats for built distributions are: - -+-------------+------------------------------+---------+ -| Format | Description | Notes | -+=============+==============================+=========+ -| ``gztar`` | gzipped tar file | (1),(3) | -| | (:file:`.tar.gz`) | | -+-------------+------------------------------+---------+ -| ``ztar`` | compressed tar file | \(3) | -| | (:file:`.tar.Z`) | | -+-------------+------------------------------+---------+ -| ``tar`` | tar file (:file:`.tar`) | \(3) | -+-------------+------------------------------+---------+ -| ``zip`` | zip file (:file:`.zip`) | (2),(4) | -+-------------+------------------------------+---------+ -| ``rpm`` | RPM | \(5) | -+-------------+------------------------------+---------+ -| ``pkgtool`` | Solaris :program:`pkgtool` | | -+-------------+------------------------------+---------+ -| ``sdux`` | HP-UX :program:`swinstall` | | -+-------------+------------------------------+---------+ -| ``rpm`` | RPM | \(5) | -+-------------+------------------------------+---------+ -| ``wininst`` | self-extracting ZIP file for | \(4) | -| | Windows | | -+-------------+------------------------------+---------+ -| ``msi`` | Microsoft Installer. | | -+-------------+------------------------------+---------+ - - -Notes: - -(1) - default on Unix - -(2) - default on Windows - -(3) - requires external utilities: :program:`tar` and possibly one of :program:`gzip`, - :program:`bzip2`, or :program:`compress` - -(4) - requires either external :program:`zip` utility or :mod:`zipfile` module (part - of the standard Python library since Python 1.6) - -(5) - requires external :program:`rpm` utility, version 3.0.4 or better (use ``rpm - --version`` to find out which version you have) - -You don't have to use the :command:`bdist` command with the :option:`--formats` -option; you can also use the command that directly implements the format you're -interested in. Some of these :command:`bdist` "sub-commands" actually generate -several similar formats; for instance, the :command:`bdist_dumb` command -generates all the "dumb" archive formats (``tar``, ``ztar``, ``gztar``, and -``zip``), and :command:`bdist_rpm` generates both binary and source RPMs. The -:command:`bdist` sub-commands, and the formats generated by each, are: - -+--------------------------+-----------------------+ -| Command | Formats | -+==========================+=======================+ -| :command:`bdist_dumb` | tar, ztar, gztar, zip | -+--------------------------+-----------------------+ -| :command:`bdist_rpm` | rpm, srpm | -+--------------------------+-----------------------+ -| :command:`bdist_wininst` | wininst | -+--------------------------+-----------------------+ -| :command:`bdist_msi` | msi | -+--------------------------+-----------------------+ - -The following sections give details on the individual :command:`bdist_\*` -commands. - - -.. _creating-dumb: - -Creating dumb built distributions -================================= - -.. XXX Need to document absolute vs. prefix-relative packages here, but first - I have to implement it! - - -.. _creating-rpms: - -Creating RPM packages -===================== - -The RPM format is used by many popular Linux distributions, including Red Hat, -SuSE, and Mandrake. If one of these (or any of the other RPM-based Linux -distributions) is your usual environment, creating RPM packages for other users -of that same distribution is trivial. Depending on the complexity of your module -distribution and differences between Linux distributions, you may also be able -to create RPMs that work on different RPM-based distributions. - -The usual way to create an RPM of your module distribution is to run the -:command:`bdist_rpm` command:: - - python setup.py bdist_rpm - -or the :command:`bdist` command with the :option:`--format` option:: - - python setup.py bdist --formats=rpm - -The former allows you to specify RPM-specific options; the latter allows you to -easily specify multiple formats in one run. If you need to do both, you can -explicitly specify multiple :command:`bdist_\*` commands and their options:: - - python setup.py bdist_rpm --packager="John Doe " \ - bdist_wininst --target-version="2.0" - -Creating RPM packages is driven by a :file:`.spec` file, much as using the -Distutils is driven by the setup script. To make your life easier, the -:command:`bdist_rpm` command normally creates a :file:`.spec` file based on the -information you supply in the setup script, on the command line, and in any -Distutils configuration files. Various options and sections in the -:file:`.spec` file are derived from options in the setup script as follows: - -+------------------------------------------+----------------------------------------------+ -| RPM :file:`.spec` file option or section | Distutils setup script option | -+==========================================+==============================================+ -| Name | :option:`name` | -+------------------------------------------+----------------------------------------------+ -| Summary (in preamble) | :option:`description` | -+------------------------------------------+----------------------------------------------+ -| Version | :option:`version` | -+------------------------------------------+----------------------------------------------+ -| Vendor | :option:`author` and :option:`author_email`, | -| | or --- & :option:`maintainer` and | -| | :option:`maintainer_email` | -+------------------------------------------+----------------------------------------------+ -| Copyright | :option:`license` | -+------------------------------------------+----------------------------------------------+ -| Url | :option:`url` | -+------------------------------------------+----------------------------------------------+ -| %description (section) | :option:`long_description` | -+------------------------------------------+----------------------------------------------+ - -Additionally, there are many options in :file:`.spec` files that don't have -corresponding options in the setup script. Most of these are handled through -options to the :command:`bdist_rpm` command as follows: - -+-------------------------------+-----------------------------+-------------------------+ -| RPM :file:`.spec` file option | :command:`bdist_rpm` option | default value | -| or section | | | -+===============================+=============================+=========================+ -| Release | :option:`release` | "1" | -+-------------------------------+-----------------------------+-------------------------+ -| Group | :option:`group` | "Development/Libraries" | -+-------------------------------+-----------------------------+-------------------------+ -| Vendor | :option:`vendor` | (see above) | -+-------------------------------+-----------------------------+-------------------------+ -| Packager | :option:`packager` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| Provides | :option:`provides` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| Requires | :option:`requires` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| Conflicts | :option:`conflicts` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| Obsoletes | :option:`obsoletes` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| Distribution | :option:`distribution_name` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| BuildRequires | :option:`build_requires` | (none) | -+-------------------------------+-----------------------------+-------------------------+ -| Icon | :option:`icon` | (none) | -+-------------------------------+-----------------------------+-------------------------+ - -Obviously, supplying even a few of these options on the command line would be -tedious and error-prone, so it's usually best to put them in the setup -configuration file, :file:`setup.cfg`\ ---see section :ref:`setup-config`. If -you distribute or package many Python module distributions, you might want to -put options that apply to all of them in your personal Distutils configuration -file (:file:`~/.pydistutils.cfg`). If you want to temporarily disable -this file, you can pass the --no-user-cfg option to setup.py. - -There are three steps to building a binary RPM package, all of which are -handled automatically by the Distutils: - -#. create a :file:`.spec` file, which describes the package (analogous to the - Distutils setup script; in fact, much of the information in the setup script - winds up in the :file:`.spec` file) - -#. create the source RPM - -#. create the "binary" RPM (which may or may not contain binary code, depending - on whether your module distribution contains Python extensions) - -Normally, RPM bundles the last two steps together; when you use the Distutils, -all three steps are typically bundled together. - -If you wish, you can separate these three steps. You can use the -:option:`--spec-only` option to make :command:`bdist_rpm` just create the -:file:`.spec` file and exit; in this case, the :file:`.spec` file will be -written to the "distribution directory"---normally :file:`dist/`, but -customizable with the :option:`--dist-dir` option. (Normally, the :file:`.spec` -file winds up deep in the "build tree," in a temporary directory created by -:command:`bdist_rpm`.) - -.. % \XXX{this isn't implemented yet---is it needed?!} -.. % You can also specify a custom \file{.spec} file with the -.. % \longprogramopt{spec-file} option; used in conjunction with -.. % \longprogramopt{spec-only}, this gives you an opportunity to customize -.. % the \file{.spec} file manually: -.. % -.. % \ begin{verbatim} -.. % > python setup.py bdist_rpm --spec-only -.. % # ...edit dist/FooBar-1.0.spec -.. % > python setup.py bdist_rpm --spec-file=dist/FooBar-1.0.spec -.. % \ end{verbatim} -.. % -.. % (Although a better way to do this is probably to override the standard -.. % \command{bdist\_rpm} command with one that writes whatever else you want -.. % to the \file{.spec} file.) - - -.. _creating-wininst: - -Creating Windows Installers -=========================== - -Executable installers are the natural format for binary distributions on -Windows. They display a nice graphical user interface, display some information -about the module distribution to be installed taken from the metadata in the -setup script, let the user select a few options, and start or cancel the -installation. - -Since the metadata is taken from the setup script, creating Windows installers -is usually as easy as running:: - - python setup.py bdist_wininst - -or the :command:`bdist` command with the :option:`--formats` option:: - - python setup.py bdist --formats=wininst - -If you have a pure module distribution (only containing pure Python modules and -packages), the resulting installer will be version independent and have a name -like :file:`foo-1.0.win32.exe`. These installers can even be created on Unix -platforms or Mac OS X. - -If you have a non-pure distribution, the extensions can only be created on a -Windows platform, and will be Python version dependent. The installer filename -will reflect this and now has the form :file:`foo-1.0.win32-py2.0.exe`. You -have to create a separate installer for every Python version you want to -support. - -.. TODO Add :term: markup to bytecode when merging into the stdlib - -The installer will try to compile pure modules into bytecode after installation -on the target system in normal and optimizing mode. If you don't want this to -happen for some reason, you can run the :command:`bdist_wininst` command with -the :option:`--no-target-compile` and/or the :option:`--no-target-optimize` -option. - -By default the installer will display the cool "Python Powered" logo when it is -run, but you can also supply your own 152x261 bitmap which must be a Windows -:file:`.bmp` file with the :option:`--bitmap` option. - -The installer will also display a large title on the desktop background window -when it is run, which is constructed from the name of your distribution and the -version number. This can be changed to another text by using the -:option:`--title` option. - -The installer file will be written to the "distribution directory" --- normally -:file:`dist/`, but customizable with the :option:`--dist-dir` option. - -.. _cross-compile-windows: - -Cross-compiling on Windows -========================== - -Starting with Python 2.6, distutils is capable of cross-compiling between -Windows platforms. In practice, this means that with the correct tools -installed, you can use a 32bit version of Windows to create 64bit extensions -and vice-versa. - -To build for an alternate platform, specify the :option:`--plat-name` option -to the build command. Valid values are currently 'win32', 'win-amd64' and -'win-ia64'. For example, on a 32bit version of Windows, you could execute:: - - python setup.py build --plat-name=win-amd64 - -to build a 64bit version of your extension. The Windows Installers also -support this option, so the command:: - - python setup.py build --plat-name=win-amd64 bdist_wininst - -would create a 64bit installation executable on your 32bit version of Windows. - -To cross-compile, you must download the Python source code and cross-compile -Python itself for the platform you are targetting - it is not possible from a -binary installtion of Python (as the .lib etc file for other platforms are -not included.) In practice, this means the user of a 32 bit operating -system will need to use Visual Studio 2008 to open the -:file:`PCBuild/PCbuild.sln` solution in the Python source tree and build the -"x64" configuration of the 'pythoncore' project before cross-compiling -extensions is possible. - -Note that by default, Visual Studio 2008 does not install 64bit compilers or -tools. You may need to reexecute the Visual Studio setup process and select -these tools (using Control Panel->[Add/Remove] Programs is a convenient way to -check or modify your existing install.) - -.. _postinstallation-script: - -The Postinstallation script ---------------------------- - -Starting with Python 2.3, a postinstallation script can be specified with the -:option:`--install-script` option. The basename of the script must be -specified, and the script filename must also be listed in the scripts argument -to the setup function. - -This script will be run at installation time on the target system after all the -files have been copied, with ``argv[1]`` set to :option:`-install`, and again at -uninstallation time before the files are removed with ``argv[1]`` set to -:option:`-remove`. - -The installation script runs embedded in the windows installer, every output -(``sys.stdout``, ``sys.stderr``) is redirected into a buffer and will be -displayed in the GUI after the script has finished. - -Some functions especially useful in this context are available as additional -built-in functions in the installation script. - - -.. function:: directory_created(path) - file_created(path) - - These functions should be called when a directory or file is created by the - postinstall script at installation time. It will register *path* with the - uninstaller, so that it will be removed when the distribution is uninstalled. - To be safe, directories are only removed if they are empty. - - -.. function:: get_special_folder_path(csidl_string) - - This function can be used to retrieve special folder locations on Windows like - the Start Menu or the Desktop. It returns the full path to the folder. - *csidl_string* must be one of the following strings:: - - "CSIDL_APPDATA" - - "CSIDL_COMMON_STARTMENU" - "CSIDL_STARTMENU" - - "CSIDL_COMMON_DESKTOPDIRECTORY" - "CSIDL_DESKTOPDIRECTORY" - - "CSIDL_COMMON_STARTUP" - "CSIDL_STARTUP" - - "CSIDL_COMMON_PROGRAMS" - "CSIDL_PROGRAMS" - - "CSIDL_FONTS" - - If the folder cannot be retrieved, :exc:`OSError` is raised. - - Which folders are available depends on the exact Windows version, and probably - also the configuration. For details refer to Microsoft's documentation of the - c:function:`SHGetSpecialFolderPath` function. - - -.. function:: create_shortcut(target, description, filename[, arguments[, workdir[, iconpath[, iconindex]]]]) - - This function creates a shortcut. *target* is the path to the program to be - started by the shortcut. *description* is the description of the shortcut. - *filename* is the title of the shortcut that the user will see. *arguments* - specifies the command-line arguments, if any. *workdir* is the working directory - for the program. *iconpath* is the file containing the icon for the shortcut, - and *iconindex* is the index of the icon in the file *iconpath*. Again, for - details consult the Microsoft documentation for the :class:`IShellLink` - interface. - - -Vista User Access Control (UAC) -=============================== - -Starting with Python 2.6, bdist_wininst supports a :option:`--user-access-control` -option. The default is 'none' (meaning no UAC handling is done), and other -valid values are 'auto' (meaning prompt for UAC elevation if Python was -installed for all users) and 'force' (meaning always prompt for elevation). diff --git a/docs/source/distutils/commandhooks.rst b/docs/source/distutils/commandhooks.rst deleted file mode 100644 --- a/docs/source/distutils/commandhooks.rst +++ /dev/null @@ -1,31 +0,0 @@ -============= -Command hooks -============= - -Distutils2 provides a way of extending its commands by the use of pre- and -post- command hooks. The hooks are simple Python functions (or any callable -objects) and are specified in the config file using their full qualified names. -The pre-hooks are run after the command is finalized (its options are -processed), but before it is run. The post-hooks are run after the command -itself. Both types of hooks receive an instance of the command object. - -Sample usage of hooks -===================== - -Firstly, you need to make sure your hook is present in the path. This is usually -done by dropping them to the same folder where `setup.py` file lives :: - - # file: myhooks.py - def my_install_hook(install_cmd): - print "Oh la la! Someone is installing my project!" - -Then, you need to point to it in your `setup.cfg` file, under the appropriate -command section :: - - [install] - pre-hook.project = myhooks.my_install_hook - -The hooks defined in different config files (system-wide, user-wide and -package-wide) do not override each other as long as they are specified with -different aliases (additional names after the dot). The alias in the example -above is ``project``. diff --git a/docs/source/distutils/commandref.rst b/docs/source/distutils/commandref.rst deleted file mode 100644 --- a/docs/source/distutils/commandref.rst +++ /dev/null @@ -1,60 +0,0 @@ -.. _reference: - -***************** -Command Reference -***************** - -.. % \section{Building modules: the \protect\command{build} command family} -.. % \label{build-cmds} -.. % \subsubsection{\protect\command{build}} -.. % \label{build-cmd} -.. % \subsubsection{\protect\command{build\_py}} -.. % \label{build-py-cmd} -.. % \subsubsection{\protect\command{build\_ext}} -.. % \label{build-ext-cmd} -.. % \subsubsection{\protect\command{build\_clib}} -.. % \label{build-clib-cmd} - - -.. _install-cmd: - -Installing modules: the :command:`install` command family -========================================================= - -The install command ensures that the build commands have been run and then runs -the subcommands :command:`install_lib`, :command:`install_data` and -:command:`install_scripts`. - -.. % \subsubsection{\protect\command{install\_lib}} -.. % \label{install-lib-cmd} - - -.. _install-data-cmd: - -:command:`install_data` ------------------------ - -This command installs all data files provided with the distribution. - - -.. _install-scripts-cmd: - -:command:`install_scripts` --------------------------- - -This command installs all (Python) scripts in the distribution. - -.. % \subsection{Cleaning up: the \protect\command{clean} command} -.. % \label{clean-cmd} - - -.. % \section{Creating a built distribution: the -.. % \protect\command{bdist} command family} -.. % \label{bdist-cmds} - -.. % \subsection{\protect\command{bdist}} -.. % \subsection{\protect\command{bdist\_dumb}} -.. % \subsection{\protect\command{bdist\_rpm}} -.. % \subsection{\protect\command{bdist\_wininst}} - - diff --git a/docs/source/distutils/configfile.rst b/docs/source/distutils/configfile.rst deleted file mode 100644 --- a/docs/source/distutils/configfile.rst +++ /dev/null @@ -1,131 +0,0 @@ -.. _setup-config: - -************************************ -Writing the Setup Configuration File -************************************ - -Often, it's not possible to write down everything needed to build a distribution -*a priori*: you may need to get some information from the user, or from the -user's system, in order to proceed. As long as that information is fairly -simple---a list of directories to search for C header files or libraries, for -example---then providing a configuration file, :file:`setup.cfg`, for users to -edit is a cheap and easy way to solicit it. Configuration files also let you -provide default values for any command option, which the installer can then -override either on the command line or by editing the config file. - -The setup configuration file is a useful middle-ground between the setup script ----which, ideally, would be opaque to installers [#]_---and the command line to -the setup script, which is outside of your control and entirely up to the -installer. In fact, :file:`setup.cfg` (and any other Distutils configuration -files present on the target system) are processed after the contents of the -setup script, but before the command line. This has several useful -consequences: - -.. If you have more advanced needs, such as determining which extensions to - build based on what capabilities are present on the target system, then you - need the Distutils auto-configuration facility. This started to appear in - Distutils 0.9 but, as of this writing, isn't mature or stable enough yet - for real-world use. - -* installers can override some of what you put in :file:`setup.py` by editing - :file:`setup.cfg` - -* you can provide non-standard defaults for options that are not easily set in - :file:`setup.py` - -* installers can override anything in :file:`setup.cfg` using the command-line - options to :file:`setup.py` - -The basic syntax of the configuration file is simple:: - - [command] - option = value - ... - -where *command* is one of the Distutils commands (e.g. :command:`build_py`, -:command:`install`), and *option* is one of the options that command supports. -Any number of options can be supplied for each command, and any number of -command sections can be included in the file. Blank lines are ignored, as are -comments, which run from a ``'#'`` character until the end of the line. Long -option values can be split across multiple lines simply by indenting the -continuation lines. - -You can find out the list of options supported by a particular command with the -universal :option:`--help` option, e.g. :: - - > python setup.py --help build_ext - [...] - Options for 'build_ext' command: - --build-lib (-b) directory for compiled extension modules - --build-temp (-t) directory for temporary files (build by-products) - --inplace (-i) ignore build-lib and put compiled extensions into the - source directory alongside your pure Python modules - --include-dirs (-I) list of directories to search for header files - --define (-D) C preprocessor macros to define - --undef (-U) C preprocessor macros to undefine - --swig-opts list of SWIG command-line options - [...] - -.. XXX do we want to support ``setup.py --help metadata``? - -Note that an option spelled :option:`--foo-bar` on the command line is spelled -:option:`foo_bar` in configuration files. - -For example, say you want your extensions to be built "in-place"---that is, you -have an extension :mod:`pkg.ext`, and you want the compiled extension file -(:file:`ext.so` on Unix, say) to be put in the same source directory as your -pure Python modules :mod:`pkg.mod1` and :mod:`pkg.mod2`. You can always use the -:option:`--inplace` option on the command line to ensure this:: - - python setup.py build_ext --inplace - -But this requires that you always specify the :command:`build_ext` command -explicitly, and remember to provide :option:`--inplace`. An easier way is to -"set and forget" this option, by encoding it in :file:`setup.cfg`, the -configuration file for this distribution:: - - [build_ext] - inplace = 1 - -This will affect all builds of this module distribution, whether or not you -explicitly specify :command:`build_ext`. If you include :file:`setup.cfg` in -your source distribution, it will also affect end-user builds---which is -probably a bad idea for this option, since always building extensions in-place -would break installation of the module distribution. In certain peculiar cases, -though, modules are built right in their installation directory, so this is -conceivably a useful ability. (Distributing extensions that expect to be built -in their installation directory is almost always a bad idea, though.) - -Another example: certain commands take a lot of options that don't change from -run to run; for example, :command:`bdist_rpm` needs to know everything required -to generate a "spec" file for creating an RPM distribution. Some of this -information comes from the setup script, and some is automatically generated by -the Distutils (such as the list of files installed). But some of it has to be -supplied as options to :command:`bdist_rpm`, which would be very tedious to do -on the command line for every run. Hence, here is a snippet from the Distutils' -own :file:`setup.cfg`:: - - [bdist_rpm] - release = 1 - packager = Greg Ward - doc_files = CHANGES.txt - README.txt - USAGE.txt - doc/ - examples/ - -Note that the :option:`doc_files` option is simply a whitespace-separated string -split across multiple lines for readability. - - -.. seealso:: - - :ref:`inst-config-syntax` in "Installing Python Projects" - More information on the configuration files is available in the manual for - system administrators. - - -.. rubric:: Footnotes - -.. [#] This ideal probably won't be achieved until auto-configuration is fully - supported by the Distutils. diff --git a/docs/source/distutils/examples.rst b/docs/source/distutils/examples.rst deleted file mode 100644 --- a/docs/source/distutils/examples.rst +++ /dev/null @@ -1,332 +0,0 @@ -.. _examples: - -******** -Examples -******** - -This chapter provides a number of basic examples to help get started with -Distutils2. - - -.. _pure-mod: - -Pure Python distribution (by module) -==================================== - -If you're just distributing a couple of modules, especially if they don't live -in a particular package, you can specify them individually using the -:option:`py_modules` option in the setup script. - -In the simplest case, you'll have two files to worry about: a setup script and -the single module you're distributing, :file:`foo.py` in this example:: - - / - setup.py - foo.py - -(In all diagrams in this section, ** will refer to the distribution root -directory.) A minimal setup script to describe this situation would be:: - - from distutils2.core import setup - setup(name='foo', - version='1.0', - py_modules=['foo']) - -Note that the name of the distribution is specified independently with the -:option:`name` option, and there's no rule that says it has to be the same as -the name of the sole module in the distribution (although that's probably a good -convention to follow). However, the distribution name is used to generate -filenames, so you should stick to letters, digits, underscores, and hyphens. - -Since :option:`py_modules` is a list, you can of course specify multiple -modules, e.g. if you're distributing modules :mod:`foo` and :mod:`bar`, your -setup might look like this:: - - / - setup.py - foo.py - bar.py - -and the setup script might be :: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - py_modules=['foo', 'bar']) - -You can put module source files into another directory, but if you have enough -modules to do that, it's probably easier to specify modules by package rather -than listing them individually. - - -.. _pure-pkg: - -Pure Python distribution (by package) -===================================== - -If you have more than a couple of modules to distribute, especially if they are -in multiple packages, it's probably easier to specify whole packages rather than -individual modules. This works even if your modules are not in a package; you -can just tell the Distutils to process modules from the root package, and that -works the same as any other package (except that you don't have to have an -:file:`__init__.py` file). - -The setup script from the last example could also be written as :: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - packages=['']) - -(The empty string stands for the root package.) - -If those two files are moved into a subdirectory, but remain in the root -package, e.g.:: - - / - setup.py - src/ - foo.py - bar.py - -then you would still specify the root package, but you have to tell the -Distutils where source files in the root package live:: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - package_dir={'': 'src'}, - packages=['']) - -More typically, though, you will want to distribute multiple modules in the same -package (or in sub-packages). For example, if the :mod:`foo` and :mod:`bar` -modules belong in package :mod:`foobar`, one way to lay out your source tree is - -:: - - / - setup.py - foobar/ - __init__.py - foo.py - bar.py - -This is in fact the default layout expected by the Distutils, and the one that -requires the least work to describe in your setup script:: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - packages=['foobar']) - -If you want to put modules in directories not named for their package, then you -need to use the :option:`package_dir` option again. For example, if the -:file:`src` directory holds modules in the :mod:`foobar` package:: - - / - setup.py - src/ - __init__.py - foo.py - bar.py - -an appropriate setup script would be :: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - package_dir={'foobar': 'src'}, - packages=['foobar']) - -Or, you might put modules from your main package right in the distribution -root:: - - / - setup.py - __init__.py - foo.py - bar.py - -in which case your setup script would be :: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - package_dir={'foobar': ''}, - packages=['foobar']) - -(The empty string also stands for the current directory.) - -If you have sub-packages, they must be explicitly listed in :option:`packages`, -but any entries in :option:`package_dir` automatically extend to sub-packages. -(In other words, the Distutils does *not* scan your source tree, trying to -figure out which directories correspond to Python packages by looking for -:file:`__init__.py` files.) Thus, if the default layout grows a sub-package:: - - / - setup.py - foobar/ - __init__.py - foo.py - bar.py - subfoo/ - __init__.py - blah.py - -then the corresponding setup script would be :: - - from distutils2.core import setup - setup(name='foobar', - version='1.0', - packages=['foobar', 'foobar.subfoo']) - -(Again, the empty string in :option:`package_dir` stands for the current -directory.) - - -.. _single-ext: - -Single extension module -======================= - -Extension modules are specified using the :option:`ext_modules` option. -:option:`package_dir` has no effect on where extension source files are found; -it only affects the source for pure Python modules. The simplest case, a -single extension module in a single C source file, is:: - - / - setup.py - foo.c - -If the :mod:`foo` extension belongs in the root package, the setup script for -this could be :: - - from distutils2.core import setup, Extension - setup(name='foobar', - version='1.0', - ext_modules=[Extension('foo', ['foo.c'])]) - -If the extension actually belongs in a package, say :mod:`foopkg`, then - -With exactly the same source tree layout, this extension can be put in the -:mod:`foopkg` package simply by changing the name of the extension:: - - from distutils2.core import setup, Extension - setup(name='foobar', - version='1.0', - packages=['foopkg'], - ext_modules=[Extension('foopkg.foo', ['foo.c'])]) - - -Checking metadata -================= - -The ``check`` command allows you to verify if your project's metadata -meets the minimum requirements to build a distribution. - -To run it, just call it using your :file:`setup.py` script. If something is -missing, ``check`` will display a warning. - -Let's take an example with a simple script:: - - from distutils2.core import setup - - setup(name='foobar') - -Running the ``check`` command will display some warnings:: - - $ python setup.py check - running check - warning: check: missing required metadata: version, home_page - warning: check: missing metadata: either (author and author_email) or - (maintainer and maintainer_email) must be supplied - - -If you use the reStructuredText syntax in the ``long_description`` field and -`Docutils `_ is installed you can check if -the syntax is fine with the ``check`` command, using the ``restructuredtext`` -option. - -For example, if the :file:`setup.py` script is changed like this:: - - from distutils2.core import setup - - desc = """\ - Welcome to foobar! - =============== - - This is the description of the ``foobar`` project. - """ - - setup(name='foobar', - version='1.0', - author=u'Tarek Ziad?', - author_email='tarek at ziade.org', - summary='Foobar utilities' - description=desc, - home_page='http://example.com') - -Where the long description is broken, ``check`` will be able to detect it -by using the :mod:`docutils` parser:: - - $ python setup.py check --restructuredtext - running check - warning: check: Title underline too short. (line 2) - warning: check: Could not finish the parsing. - - -.. _reading-metadata: - -Reading the metadata -==================== - -The :func:`distutils2.core.setup` function provides a command-line interface -that allows you to query the metadata fields of a project through the -:file:`setup.py` script of a given project:: - - $ python setup.py --name - foobar - -This call reads the ``name`` metadata by running the -:func:`distutils2.core.setup` function. When a source or binary -distribution is created with Distutils, the metadata fields are written -in a static file called :file:`PKG-INFO`. When a Distutils-based project is -installed in Python, the :file:`PKG-INFO` file is copied alongside the modules -and packages of the distribution under :file:`NAME-VERSION-pyX.X.egg-info`, -where ``NAME`` is the name of the project, ``VERSION`` its version as defined -in the Metadata, and ``pyX.X`` the major and minor version of Python like -``2.7`` or ``3.2``. - -You can read back this static file, by using the -:class:`distutils2.dist.Metadata` class and its -:func:`read_pkg_file` method:: - - >>> from distutils2.metadata import Metadata - >>> metadata = Metadata() - >>> metadata.read_pkg_file(open('distribute-0.6.8-py2.7.egg-info')) - >>> metadata.name - 'distribute' - >>> metadata.version - '0.6.8' - >>> metadata.description - 'Easily download, build, install, upgrade, and uninstall Python packages' - -Notice that the class can also be instantiated with a metadata file path to -loads its values:: - - >>> pkg_info_path = 'distribute-0.6.8-py2.7.egg-info' - >>> Metadata(pkg_info_path).name - 'distribute' - - -.. XXX These comments have been here for at least ten years. Write the - sections or delete the comments (we can maybe ask Greg Ward about - the planned contents). (Unindent to make them section titles) - - .. multiple-ext:: - - Multiple extension modules - ========================== - - Putting it all together - ======================= diff --git a/docs/source/distutils/extending.rst b/docs/source/distutils/extending.rst deleted file mode 100644 --- a/docs/source/distutils/extending.rst +++ /dev/null @@ -1,95 +0,0 @@ -.. _extending-distutils: - -******************* -Extending Distutils -******************* - -Distutils can be extended in various ways. Most extensions take the form of new -commands or replacements for existing commands. New commands may be written to -support new types of platform-specific packaging, for example, while -replacements for existing commands may be made to modify details of how the -command operates on a package. - -Most extensions of the distutils are made within :file:`setup.py` scripts that -want to modify existing commands; many simply add a few file extensions that -should be copied into packages in addition to :file:`.py` files as a -convenience. - -Most distutils command implementations are subclasses of the -:class:`distutils2.cmd.Command` class. New commands may directly inherit from -:class:`Command`, while replacements often derive from :class:`Command` -indirectly, directly subclassing the command they are replacing. Commands are -required to derive from :class:`Command`. - -.. .. _extend-existing: - Extending existing commands - =========================== - - -.. .. _new-commands: - Writing new commands - ==================== - - -Integrating new commands -======================== - -There are different ways to integrate new command implementations into -distutils. The most difficult is to lobby for the inclusion of the new features -in distutils itself, and wait for (and require) a version of Python that -provides that support. This is really hard for many reasons. - -The most common, and possibly the most reasonable for most needs, is to include -the new implementations with your :file:`setup.py` script, and cause the -:func:`distutils2.core.setup` function use them:: - - from distutils2.core import setup - from distutils2.command.build_py import build_py as _build_py - - class build_py(_build_py): - """Specialized Python source builder.""" - - # implement whatever needs to be different... - - setup(..., cmdclass={'build_py': build_py}) - -This approach is most valuable if the new implementations must be used to use a -particular package, as everyone interested in the package will need to have the -new command implementation. - -Beginning with Python 2.4, a third option is available, intended to allow new -commands to be added which can support existing :file:`setup.py` scripts without -requiring modifications to the Python installation. This is expected to allow -third-party extensions to provide support for additional packaging systems, but -the commands can be used for anything distutils commands can be used for. A new -configuration option, :option:`command_packages` (command-line option -:option:`--command-packages`), can be used to specify additional packages to be -searched for modules implementing commands. Like all distutils options, this -can be specified on the command line or in a configuration file. This option -can only be set in the ``[global]`` section of a configuration file, or before -any commands on the command line. If set in a configuration file, it can be -overridden from the command line; setting it to an empty string on the command -line causes the default to be used. This should never be set in a configuration -file provided with a package. - -This new option can be used to add any number of packages to the list of -packages searched for command implementations; multiple package names should be -separated by commas. When not specified, the search is only performed in the -:mod:`distutils2.command` package. When :file:`setup.py` is run with the option -:option:`--command-packages` :option:`distcmds,buildcmds`, however, the packages -:mod:`distutils2.command`, :mod:`distcmds`, and :mod:`buildcmds` will be searched -in that order. New commands are expected to be implemented in modules of the -same name as the command by classes sharing the same name. Given the example -command-line option above, the command :command:`bdist_openpkg` could be -implemented by the class :class:`distcmds.bdist_openpkg.bdist_openpkg` or -:class:`buildcmds.bdist_openpkg.bdist_openpkg`. - - -Adding new distribution types -============================= - -Commands that create distributions (files in the :file:`dist/` directory) need -to add ``(command, filename)`` pairs to ``self.distribution.dist_files`` so that -:command:`upload` can upload it to PyPI. The *filename* in the pair contains no -path information, only the name of the file itself. In dry-run mode, pairs -should still be added to represent what would have been created. diff --git a/docs/source/distutils/index.rst b/docs/source/distutils/index.rst deleted file mode 100644 --- a/docs/source/distutils/index.rst +++ /dev/null @@ -1,33 +0,0 @@ -.. _distutils-index: - -############################### - Distributing Python Projects -############################### - -:Authors: Greg Ward, Anthony Baxter and Distutils2 contributors -:Email: distutils-sig at python.org -:Release: |version| -:Date: |today| - -This document describes the Python Distribution Utilities ("Distutils2") from -the developer's point of view, describing how to use the Distutils to make -Python applications, packages or modules easily available to a wider audience -with very little overhead for build/release/install mechanics. - -.. toctree:: - :maxdepth: 2 - :numbered: - - introduction - setupscript - configfile - sourcedist - builtdist - packageindex - uploading - examples - extending - commandhooks - commandref - newcommands - apiref diff --git a/docs/source/distutils/introduction.rst b/docs/source/distutils/introduction.rst deleted file mode 100644 --- a/docs/source/distutils/introduction.rst +++ /dev/null @@ -1,193 +0,0 @@ -.. _distutils-intro: - -***************************** -An Introduction to Distutils2 -***************************** - -This document covers using Distutils2 to distribute your Python modules, -concentrating on the role of developer/distributor. If you're looking for -information on installing Python modules you should refer to the -:ref:`install-index` chapter. - -Throughout this documentation, the terms "Distutils", "the Distutils" and -"Distutils2" will be used interchangeably. - -.. _distutils-concepts: - -Concepts & Terminology -====================== - -Using Distutils is quite simple both for module developers and for -users/administrators installing third-party modules. As a developer, your -responsibilities (apart from writing solid, well-documented and well-tested -code, of course!) are: - -* writing a setup script (:file:`setup.py` by convention) - -* (optional) writing a setup configuration file - -* creating a source distribution - -* (optional) creating one or more "built" (binary) distributions of your - project - -All of these tasks are covered in this document. - -Not all module developers have access to multiple platforms, so one cannot -expect them to create buildt distributions for every platform. To remedy -this, it is hoped that intermediaries called *packagers* will arise to address -this need. Packagers take source distributions released by module developers, -build them on one or more platforms and release the resulting built -distributions. Thus, users on a greater range of platforms will be able to -install the most popular Python modules in the most natural way for their -platform without having to run a setup script or compile a single line of code. - - -.. _distutils-simple-example: - -A Simple Example -================ - -A setup script is usually quite simple, although since it's written in Python -there are no arbitrary limits to what you can do with it, though you should be -careful about putting expensive operations in your setup script. -Unlike, say, Autoconf-style configure scripts the setup script may be run -multiple times in the course of building and installing a module -distribution. - -If all you want to do is distribute a module called :mod:`foo`, contained in a -file :file:`foo.py`, then your setup script can be as simple as:: - - from distutils2.core import setup - setup(name='foo', - version='1.0', - py_modules=['foo']) - -Some observations: - -* most information that you supply to the Distutils is supplied as keyword - arguments to the :func:`setup` function - -* those keyword arguments fall into two categories: package metadata (name, - version number, etc.) and information about what's in the package (a list - of pure Python modules in this case) - -* modules are specified by module name, not filename (the same will hold true - for packages and extensions) - -* it's recommended that you supply a little more metadata than we have in the - example. In particular your name, email address and a URL for the - project if appropriate (see section :ref:`setup-script` for an example) - -To create a source distribution for this module you would create a setup -script, :file:`setup.py`, containing the above code and run:: - - python setup.py sdist - -which will create an archive file (e.g., tarball on Unix, ZIP file on Windows) -containing your setup script :file:`setup.py`, and your module :file:`foo.py`. -The archive file will be named :file:`foo-1.0.tar.gz` (or :file:`.zip`), and -will unpack into a directory :file:`foo-1.0`. - -If an end-user wishes to install your :mod:`foo` module all he has to do is -download :file:`foo-1.0.tar.gz` (or :file:`.zip`), unpack it, and from the -:file:`foo-1.0` directory run :: - - python setup.py install - -which will copy :file:`foo.py` to the appropriate directory for -third-party modules in their Python installation. - -This simple example demonstrates some fundamental concepts of Distutils. -First, both developers and installers have the same basic user interface, i.e. -the setup script. The difference is which Distutils *commands* they use: the -:command:`sdist` command is almost exclusively for module developers, while -:command:`install` is more often used by installers (although some developers -will want to install their own code occasionally). - -If you want to make things really easy for your users, you can create more -than one built distributions for them. For instance, if you are running on a -Windows machine and want to make things easy for other Windows users, you can -create an executable installer (the most appropriate type of built distribution -for this platform) with the :command:`bdist_wininst` command. For example:: - - python setup.py bdist_wininst - -will create an executable installer, :file:`foo-1.0.win32.exe`, in the current -directory. You can find out what distribution formats are available at any time -by running :: - - python setup.py bdist --help-formats - - -.. _python-terms: - -General Python terminology -========================== - -If you're reading this document, you probably have a good idea of what Python -modules, extensions and so forth are. Nevertheless, just to be sure that -everyone is on the same page, here's a quick overview of Python terms: - -module - The basic unit of code reusability in Python: a block of code imported by - some other code. Three types of modules are important to us here: pure - Python modules, extension modules and packages. - -pure Python module - A module written in Python and contained in a single :file:`.py` file (and - possibly associated :file:`.pyc` and/or :file:`.pyo` files). Sometimes - referred to as a "pure module." - -extension module - A module written in the low-level language of the Python implementation: C/C++ - for Python, Java for Jython. Typically contained in a single dynamically - loaded pre-compiled file, e.g. a shared object (:file:`.so`) file for Python - extensions on Unix, a DLL (given the :file:`.pyd` extension) for Python - extensions on Windows, or a Java class file for Jython extensions. Note that - currently Distutils only handles C/C++ extensions for Python. - -package - A module that contains other modules, typically contained in a directory of - the filesystem and distinguished from other directories by the presence of a - file :file:`__init__.py`. - -root package - The root of the hierarchy of packages. (This isn't really a package, - since it doesn't have an :file:`__init__.py` file. But... we have to - call it something, right?) The vast majority of the standard library is - in the root package, as are many small standalone third-party modules that - don't belong to a larger module collection. Unlike regular packages, - modules in the root package can be found in many directories: in fact, - every directory listed in ``sys.path`` contributes modules to the root - package. - - -.. _distutils-term: - -Distutils-specific terminology -============================== - -The following terms apply more specifically to the domain of distributing Python -modules using Distutils: - -module distribution - A collection of Python modules distributed together as a single downloadable - resource and meant to be installed all as one. Examples of some well-known - module distributions are Numeric Python, PyXML, PIL (the Python Imaging - Library) or mxBase. (Module distributions would be called a *package*, - except that term is already taken in the Python context: a single module - distribution may contain zero, one, or many Python packages.) - -pure module distribution - A module distribution that contains only pure Python modules and packages. - Sometimes referred to as a "pure distribution." - -non-pure module distribution - A module distribution that contains at least one extension module. Sometimes - referred to as a "non-pure distribution." - -distribution root - The top-level directory of your source tree (or source distribution). The - directory where :file:`setup.py` exists. Generally :file:`setup.py` will - be run from this directory. diff --git a/docs/source/distutils/newcommands.rst b/docs/source/distutils/newcommands.rst deleted file mode 100644 --- a/docs/source/distutils/newcommands.rst +++ /dev/null @@ -1,144 +0,0 @@ -======== -Commands -======== - -Distutils2 provides a set of commands that are not present in distutils itself. -You might recognize some of them from other projects, like Distribute or -Setuptools. - -``test`` - Build package and run a unittest suite -================================================= - -When doing test-driven development, or running automated builds that need -testing before they are deployed for downloading or use, it's often useful -to be able to run a project's unit tests without actually deploying the project -anywhere. The ``test`` command runs -project's unit tests without actually deploying it, by temporarily putting the -project's source on ``sys.path``, after first running ``build_ext -i`` -to ensure that any C extensions are built. - -You can use this command in one of two ways: either by specifying a -unittest-compatible test suite for your project (or any callable that returns -it) or by passing a test runner function that will run your tests and display -results in the console. Both options: ``suite`` and ``runner`` accept dotted -names that will be resolved into actual objects. - -If none of these options are specified, distutils2 will try to perform test -discovery using either unittest2 (if installed) or unittest (if running on -recent Python version). - -``--suite=NAME, -s NAME`` - Specify the test suite (or module, class, or method) to be run - (for example ``my_package.tests.all_tests``). The default for this option can be - set by in your ``setup.cfg`` file. - -.. code-block:: ini - - [test] - suite = my_package.tests.all_tests - -``--runner=NAME, -r NAME`` - Specify the test runner to be called. - - -``upload`` - Upload source and/or binary distributions to PyPI -============================================================== - -The Python Package Index (PyPI) not only stores the package info, but also the -package data if the author of the package wishes to. The distutils command -:command:`upload` pushes the distribution files to PyPI. - -The command is invoked immediately after building one or more distribution -files. For example, the command :: - - python setup.py sdist bdist_wininst upload - -will cause the source distribution and the Windows installer to be uploaded to -PyPI. Note that these will be uploaded even if they are built using an earlier -invocation of :file:`setup.py`, but that only distributions named on the command -line for the invocation including the :command:`upload` command are uploaded. - -The :command:`upload` command uses the username, password, and repository URL -from the :file:`$HOME/.pypirc` file . If a :command:`register` command was -previously called in the same command, and if the password was entered in the -prompt, :command:`upload` will reuse the entered password. This is useful if -you do not want to store a clear text password in the :file:`$HOME/.pypirc` -file. - -The ``upload`` command has a few options worth noting: - -``--sign, -s`` - Sign each uploaded file using GPG (GNU Privacy Guard). The ``gpg`` program - must be available for execution on the system ``PATH``. - -``--identity=NAME, -i NAME`` - Specify the identity or key name for GPG to use when signing. The value of - this option will be passed through the ``--local-user`` option of the - ``gpg`` program. - -``--show-response`` - Display the full response text from server; this is useful for debugging - PyPI problems. - -``--repository=URL, -r URL`` - The URL of the repository to upload to. Defaults to - http://pypi.python.org/pypi (i.e., the main PyPI installation). - - -``upload_docs`` - Upload package documentation to PyPI -====================================================== - -PyPI now supports uploading project documentation to the dedicated URL -http://packages.python.org//. - -The ``upload_docs`` command will create the necessary zip file out of a -documentation directory and will post to the repository. - -Note that to upload the documentation of a project, the corresponding version -must already be registered with PyPI, using the distutils ``register`` -command -- just like the ``upload`` command. - -Assuming there is an ``Example`` project with documentation in the -subdirectory ``docs``, for example:: - - Example/ - |-- example.py - |-- setup.cfg - |-- setup.py - |-- docs - | |-- build - | | `-- html - | | | |-- index.html - | | | `-- tips_tricks.html - | |-- conf.py - | |-- index.txt - | `-- tips_tricks.txt - -You can simply pass the documentation directory path to the ``upload_docs`` -command:: - - python setup.py upload_docs --upload-dir=docs/build/html - -As with any other command, you can define useful -defaults in the ``setup.cfg`` of your Python project, for example: - -.. code-block:: ini - - [upload_docs] - upload-dir = docs/build/html - -The ``upload_docs`` command has the following options: - -``--upload-dir`` - The directory to be uploaded to the repository. By default documentation - is searched for in ``docs`` (or ``doc``) directory in project root. - -``--show-response`` - Display the full response text from server; this is useful for debugging - PyPI problems. - -``--repository=URL, -r URL`` - The URL of the repository to upload to. Defaults to - http://pypi.python.org/pypi (i.e., the main PyPI installation). - - diff --git a/docs/source/distutils/packageindex.rst b/docs/source/distutils/packageindex.rst deleted file mode 100644 --- a/docs/source/distutils/packageindex.rst +++ /dev/null @@ -1,104 +0,0 @@ -.. _package-index: - -********************************** -Registering with the Package Index -********************************** - -The Python Package Index (PyPI) holds metadata describing distributions -packaged with distutils. The distutils command :command:`register` is used to -submit your distribution's metadata to the index. It is invoked as follows:: - - python setup.py register - -Distutils will respond with the following prompt:: - - running register - We need to know who you are, so please choose either: - 1. use your existing login, - 2. register as a new user, - 3. have the server generate a new password for you (and email it to you), or - 4. quit - Your selection [default 1]: - -Note: if your username and password are saved locally, you will not see this -menu. - -If you have not registered with PyPI, then you will need to do so now. You -should choose option 2, and enter your details as required. Soon after -submitting your details, you will receive an email which will be used to confirm -your registration. - -Once you are registered, you may choose option 1 from the menu. You will be -prompted for your PyPI username and password, and :command:`register` will then -submit your metadata to the index. - -You may submit any number of versions of your distribution to the index. If you -alter the metadata for a particular version, you may submit it again and the -index will be updated. - -PyPI holds a record for each (name, version) combination submitted. The first -user to submit information for a given name is designated the Owner of that -name. They may submit changes through the :command:`register` command or through -the web interface. They may also designate other users as Owners or Maintainers. -Maintainers may edit the package information, but not designate other Owners or -Maintainers. - -By default PyPI will list all versions of a given package. To hide certain -versions, the Hidden property should be set to yes. This must be edited through -the web interface. - - -.. _pypirc: - -The .pypirc file -================ - -The format of the :file:`.pypirc` file is as follows:: - - [distutils] - index-servers = - pypi - - [pypi] - repository: - username: - password: - -The *distutils* section defines a *index-servers* variable that lists the -name of all sections describing a repository. - -Each section describing a repository defines three variables: - -- *repository*, that defines the url of the PyPI server. Defaults to - ``http://www.python.org/pypi``. -- *username*, which is the registered username on the PyPI server. -- *password*, that will be used to authenticate. If omitted the user - will be prompt to type it when needed. - -If you want to define another server a new section can be created and -listed in the *index-servers* variable:: - - [distutils] - index-servers = - pypi - other - - [pypi] - repository: - username: - password: - - [other] - repository: http://example.com/pypi - username: - password: - -:command:`register` can then be called with the -r option to point the -repository to work with:: - - python setup.py register -r http://example.com/pypi - -For convenience, the name of the section that describes the repository -may also be used:: - - python setup.py register -r other diff --git a/docs/source/distutils/setupscript.rst b/docs/source/distutils/setupscript.rst deleted file mode 100644 --- a/docs/source/distutils/setupscript.rst +++ /dev/null @@ -1,686 +0,0 @@ -.. _setup-script: - -************************ -Writing the Setup Script -************************ - -The setup script is the center of all activity in building, distributing, and -installing modules using Distutils. The main purpose of the setup script is -to describe your module distribution to Distutils, so that the various -commands that operate on your modules do the right thing. As we saw in section -:ref:`distutils-simple-example`, the setup script consists mainly of a -call to :func:`setup` where the most information is supplied as -keyword arguments to :func:`setup`. - -Here's a slightly more involved example, which we'll follow for the next couple -of sections: a setup script that could be used for Distutils2 itself:: - - #!/usr/bin/env python - - from distutils2.core import setup, find_packages - - setup(name='Distutils2', - version='1.0', - summary='Python Distribution Utilities', - keywords=['packaging', 'distutils2'], - author=u'Tarek Ziad?', - author_email='tarek at ziade.org', - home_page='http://bitbucket.org/tarek/distutils2/wiki/Home', - license='PSF', - packages=find_packages()) - - -There are only two differences between this and the trivial one-file -distribution presented in section :ref:`distutils-simple-example`: more -metadata and the specification of pure Python modules by package rather than -by module. This is important since Ristutils consist of a couple of dozen -modules split into (so far) two packages; an explicit list of every module -would be tedious to generate and difficult to maintain. For more information -on the additional metadata, see section :ref:`metadata`. - -Note that any pathnames (files or directories) supplied in the setup script -should be written using the Unix convention, i.e. slash-separated. The -Distutils will take care of converting this platform-neutral representation into -whatever is appropriate on your current platform before actually using the -pathname. This makes your setup script portable across operating systems, which -of course is one of the major goals of the Distutils. In this spirit, all -pathnames in this document are slash-separated. - -This, of course, only applies to pathnames given to Distutils functions. If -you, for example, use standard Python functions such as :func:`glob.glob` or -:func:`os.listdir` to specify files, you should be careful to write portable -code instead of hardcoding path separators:: - - glob.glob(os.path.join('mydir', 'subdir', '*.html')) - os.listdir(os.path.join('mydir', 'subdir')) - - -.. _listing-packages: - -Listing whole packages -====================== - -The :option:`packages` option tells the Distutils to process (build, distribute, -install, etc.) all pure Python modules found in each package mentioned in the -:option:`packages` list. In order to do this, of course, there has to be a -correspondence between package names and directories in the filesystem. The -default correspondence is the most obvious one, i.e. package :mod:`distutils2` is -found in the directory :file:`distutils2` relative to the distribution root. -Thus, when you say ``packages = ['foo']`` in your setup script, you are -promising that the Distutils will find a file :file:`foo/__init__.py` (which -might be spelled differently on your system, but you get the idea) relative to -the directory where your setup script lives. If you break this promise, the -Distutils will issue a warning but still process the broken package anyways. - -If you use a different convention to lay out your source directory, that's no -problem: you just have to supply the :option:`package_dir` option to tell the -Distutils about your convention. For example, say you keep all Python source -under :file:`lib`, so that modules in the "root package" (i.e., not in any -package at all) are in :file:`lib`, modules in the :mod:`foo` package are in -:file:`lib/foo`, and so forth. Then you would put :: - - package_dir = {'': 'lib'} - -in your setup script. The keys to this dictionary are package names, and an -empty package name stands for the root package. The values are directory names -relative to your distribution root. In this case, when you say ``packages = -['foo']``, you are promising that the file :file:`lib/foo/__init__.py` exists. - -Another possible convention is to put the :mod:`foo` package right in -:file:`lib`, the :mod:`foo.bar` package in :file:`lib/bar`, etc. This would be -written in the setup script as :: - - package_dir = {'foo': 'lib'} - -A ``package: dir`` entry in the :option:`package_dir` dictionary implicitly -applies to all packages below *package*, so the :mod:`foo.bar` case is -automatically handled here. In this example, having ``packages = ['foo', -'foo.bar']`` tells the Distutils to look for :file:`lib/__init__.py` and -:file:`lib/bar/__init__.py`. (Keep in mind that although :option:`package_dir` -applies recursively, you must explicitly list all packages in -:option:`packages`: the Distutils will *not* recursively scan your source tree -looking for any directory with an :file:`__init__.py` file.) - - -.. _listing-modules: - -Listing individual modules -========================== - -For a small module distribution, you might prefer to list all modules rather -than listing packages---especially the case of a single module that goes in the -"root package" (i.e., no package at all). This simplest case was shown in -section :ref:`distutils-simple-example`; here is a slightly more involved -example:: - - py_modules = ['mod1', 'pkg.mod2'] - -This describes two modules, one of them in the "root" package, the other in the -:mod:`pkg` package. Again, the default package/directory layout implies that -these two modules can be found in :file:`mod1.py` and :file:`pkg/mod2.py`, and -that :file:`pkg/__init__.py` exists as well. And again, you can override the -package/directory correspondence using the :option:`package_dir` option. - - -.. _describing-extensions: - -Describing extension modules -============================ - -Just as writing Python extension modules is a bit more complicated than writing -pure Python modules, describing them to the Distutils is a bit more complicated. -Unlike pure modules, it's not enough just to list modules or packages and expect -the Distutils to go out and find the right files; you have to specify the -extension name, source file(s), and any compile/link requirements (include -directories, libraries to link with, etc.). - -.. XXX read over this section - -All of this is done through another keyword argument to :func:`setup`, the -:option:`ext_modules` option. :option:`ext_modules` is just a list of -:class:`Extension` instances, each of which describes a single extension module. -Suppose your distribution includes a single extension, called :mod:`foo` and -implemented by :file:`foo.c`. If no additional instructions to the -compiler/linker are needed, describing this extension is quite simple:: - - Extension('foo', ['foo.c']) - -The :class:`Extension` class can be imported from :mod:`distutils2.core` along -with :func:`setup`. Thus, the setup script for a module distribution that -contains only this one extension and nothing else might be:: - - from distutils2.core import setup, Extension - setup(name='foo', - version='1.0', - ext_modules=[Extension('foo', ['foo.c'])]) - -The :class:`Extension` class (actually, the underlying extension-building -machinery implemented by the :command:`build_ext` command) supports a great deal -of flexibility in describing Python extensions, which is explained in the -following sections. - - -Extension names and packages ----------------------------- - -The first argument to the :class:`Extension` constructor is always the name of -the extension, including any package names. For example, :: - - Extension('foo', ['src/foo1.c', 'src/foo2.c']) - -describes an extension that lives in the root package, while :: - - Extension('pkg.foo', ['src/foo1.c', 'src/foo2.c']) - -describes the same extension in the :mod:`pkg` package. The source files and -resulting object code are identical in both cases; the only difference is where -in the filesystem (and therefore where in Python's namespace hierarchy) the -resulting extension lives. - -If you have a number of extensions all in the same package (or all under the -same base package), use the :option:`ext_package` keyword argument to -:func:`setup`. For example, :: - - setup(..., - ext_package='pkg', - ext_modules=[Extension('foo', ['foo.c']), - Extension('subpkg.bar', ['bar.c'])]) - -will compile :file:`foo.c` to the extension :mod:`pkg.foo`, and :file:`bar.c` to -:mod:`pkg.subpkg.bar`. - - -Extension source files ----------------------- - -The second argument to the :class:`Extension` constructor is a list of source -files. Since the Distutils currently only support C, C++, and Objective-C -extensions, these are normally C/C++/Objective-C source files. (Be sure to use -appropriate extensions to distinguish C++\ source files: :file:`.cc` and -:file:`.cpp` seem to be recognized by both Unix and Windows compilers.) - -However, you can also include SWIG interface (:file:`.i`) files in the list; the -:command:`build_ext` command knows how to deal with SWIG extensions: it will run -SWIG on the interface file and compile the resulting C/C++ file into your -extension. - -.. XXX SWIG support is rough around the edges and largely untested! - -This warning notwithstanding, options to SWIG can be currently passed like -this:: - - setup(..., - ext_modules=[Extension('_foo', ['foo.i'], - swig_opts=['-modern', '-I../include'])], - py_modules=['foo']) - -Or on the command line like this:: - - > python setup.py build_ext --swig-opts="-modern -I../include" - -On some platforms, you can include non-source files that are processed by the -compiler and included in your extension. Currently, this just means Windows -message text (:file:`.mc`) files and resource definition (:file:`.rc`) files for -Visual C++. These will be compiled to binary resource (:file:`.res`) files and -linked into the executable. - - -Preprocessor options --------------------- - -Three optional arguments to :class:`Extension` will help if you need to specify -include directories to search or preprocessor macros to define/undefine: -``include_dirs``, ``define_macros``, and ``undef_macros``. - -For example, if your extension requires header files in the :file:`include` -directory under your distribution root, use the ``include_dirs`` option:: - - Extension('foo', ['foo.c'], include_dirs=['include']) - -You can specify absolute directories there; if you know that your extension will -only be built on Unix systems with X11R6 installed to :file:`/usr`, you can get -away with :: - - Extension('foo', ['foo.c'], include_dirs=['/usr/include/X11']) - -You should avoid this sort of non-portable usage if you plan to distribute your -code: it's probably better to write C code like :: - - #include - -If you need to include header files from some other Python extension, you can -take advantage of the fact that header files are installed in a consistent way -by the Distutils :command:`install_header` command. For example, the Numerical -Python header files are installed (on a standard Unix installation) to -:file:`/usr/local/include/python1.5/Numerical`. (The exact location will differ -according to your platform and Python installation.) Since the Python include -directory---\ :file:`/usr/local/include/python1.5` in this case---is always -included in the search path when building Python extensions, the best approach -is to write C code like :: - - #include - -.. TODO check if it's d2.sysconfig or the new sysconfig module now - -If you must put the :file:`Numerical` include directory right into your header -search path, though, you can find that directory using the Distutils -:mod:`distutils2.sysconfig` module:: - - from distutils2.sysconfig import get_python_inc - incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical') - setup(..., - Extension(..., include_dirs=[incdir])) - -Even though this is quite portable---it will work on any Python installation, -regardless of platform---it's probably easier to just write your C code in the -sensible way. - -You can define and undefine preprocessor macros with the ``define_macros`` and -``undef_macros`` options. ``define_macros`` takes a list of ``(name, value)`` -tuples, where ``name`` is the name of the macro to define (a string) and -``value`` is its value: either a string or ``None``. (Defining a macro ``FOO`` -to ``None`` is the equivalent of a bare ``#define FOO`` in your C source: with -most compilers, this sets ``FOO`` to the string ``1``.) ``undef_macros`` is -just a list of macros to undefine. - -For example:: - - Extension(..., - define_macros=[('NDEBUG', '1'), - ('HAVE_STRFTIME', None)], - undef_macros=['HAVE_FOO', 'HAVE_BAR']) - -is the equivalent of having this at the top of every C source file:: - - #define NDEBUG 1 - #define HAVE_STRFTIME - #undef HAVE_FOO - #undef HAVE_BAR - - -Library options ---------------- - -You can also specify the libraries to link against when building your extension, -and the directories to search for those libraries. The ``libraries`` option is -a list of libraries to link against, ``library_dirs`` is a list of directories -to search for libraries at link-time, and ``runtime_library_dirs`` is a list of -directories to search for shared (dynamically loaded) libraries at run-time. - -For example, if you need to link against libraries known to be in the standard -library search path on target systems :: - - Extension(..., - libraries=['gdbm', 'readline']) - -If you need to link with libraries in a non-standard location, you'll have to -include the location in ``library_dirs``:: - - Extension(..., - library_dirs=['/usr/X11R6/lib'], - libraries=['X11', 'Xt']) - -(Again, this sort of non-portable construct should be avoided if you intend to -distribute your code.) - -.. XXX Should mention clib libraries here or somewhere else! - - -Other options -------------- - -There are still some other options which can be used to handle special cases. - -The :option:`optional` option is a boolean; if it is true, -a build failure in the extension will not abort the build process, but -instead simply not install the failing extension. - -The :option:`extra_objects` option is a list of object files to be passed to the -linker. These files must not have extensions, as the default extension for the -compiler is used. - -:option:`extra_compile_args` and :option:`extra_link_args` can be used to -specify additional command-line options for the respective compiler and linker -command lines. - -:option:`export_symbols` is only useful on Windows. It can contain a list of -symbols (functions or variables) to be exported. This option is not needed when -building compiled extensions: Distutils will automatically add ``initmodule`` -to the list of exported symbols. - -The :option:`depends` option is a list of files that the extension depends on -(for example header files). The build command will call the compiler on the -sources to rebuild extension if any on this files has been modified since the -previous build. - -Relationships between Distributions and Packages -================================================ - -.. FIXME rewrite to update to PEP 345 (but without dist/release confusion) - -A distribution may relate to packages in three specific ways: - -#. It can require packages or modules. - -#. It can provide packages or modules. - -#. It can obsolete packages or modules. - -These relationships can be specified using keyword arguments to the -:func:`distutils2.core.setup` function. - -Dependencies on other Python modules and packages can be specified by supplying -the *requires* keyword argument to :func:`setup`. The value must be a list of -strings. Each string specifies a package that is required, and optionally what -versions are sufficient. - -To specify that any version of a module or package is required, the string -should consist entirely of the module or package name. Examples include -``'mymodule'`` and ``'xml.parsers.expat'``. - -If specific versions are required, a sequence of qualifiers can be supplied in -parentheses. Each qualifier may consist of a comparison operator and a version -number. The accepted comparison operators are:: - - < > == - <= >= != - -These can be combined by using multiple qualifiers separated by commas (and -optional whitespace). In this case, all of the qualifiers must be matched; a -logical AND is used to combine the evaluations. - -Let's look at a bunch of examples: - -+-------------------------+----------------------------------------------+ -| Requires Expression | Explanation | -+=========================+==============================================+ -| ``==1.0`` | Only version ``1.0`` is compatible | -+-------------------------+----------------------------------------------+ -| ``>1.0, !=1.5.1, <2.0`` | Any version after ``1.0`` and before ``2.0`` | -| | is compatible, except ``1.5.1`` | -+-------------------------+----------------------------------------------+ - -Now that we can specify dependencies, we also need to be able to specify what we -provide that other distributions can require. This is done using the *provides* -keyword argument to :func:`setup`. The value for this keyword is a list of -strings, each of which names a Python module or package, and optionally -identifies the version. If the version is not specified, it is assumed to match -that of the distribution. - -Some examples: - -+---------------------+----------------------------------------------+ -| Provides Expression | Explanation | -+=====================+==============================================+ -| ``mypkg`` | Provide ``mypkg``, using the distribution | -| | version | -+---------------------+----------------------------------------------+ -| ``mypkg (1.1)`` | Provide ``mypkg`` version 1.1, regardless of | -| | the distribution version | -+---------------------+----------------------------------------------+ - -A package can declare that it obsoletes other packages using the *obsoletes* -keyword argument. The value for this is similar to that of the *requires* -keyword: a list of strings giving module or package specifiers. Each specifier -consists of a module or package name optionally followed by one or more version -qualifiers. Version qualifiers are given in parentheses after the module or -package name. - -The versions identified by the qualifiers are those that are obsoleted by the -distribution being described. If no qualifiers are given, all versions of the -named module or package are understood to be obsoleted. - -.. _distutils-installing-scripts: - -Installing Scripts -================== - -So far we have been dealing with pure and non-pure Python modules, which are -usually not run by themselves but imported by scripts. - -Scripts are files containing Python source code, intended to be started from the -command line. Scripts don't require Distutils to do anything very complicated. -The only clever feature is that if the first line of the script starts with -``#!`` and contains the word "python", the Distutils will adjust the first line -to refer to the current interpreter location. By default, it is replaced with -the current interpreter location. The :option:`--executable` (or :option:`-e`) -option will allow the interpreter path to be explicitly overridden. - -The :option:`scripts` option simply is a list of files to be handled in this -way. From the PyXML setup script:: - - setup(..., - scripts=['scripts/xmlproc_parse', 'scripts/xmlproc_val']) - -All the scripts will also be added to the ``MANIFEST`` file if no template is -provided. See :ref:`manifest`. - -.. _distutils-installing-package-data: - -Installing Package Data -======================= - -Often, additional files need to be installed into a package. These files are -often data that's closely related to the package's implementation, or text files -containing documentation that might be of interest to programmers using the -package. These files are called :dfn:`package data`. - -Package data can be added to packages using the ``package_data`` keyword -argument to the :func:`setup` function. The value must be a mapping from -package name to a list of relative path names that should be copied into the -package. The paths are interpreted as relative to the directory containing the -package (information from the ``package_dir`` mapping is used if appropriate); -that is, the files are expected to be part of the package in the source -directories. They may contain glob patterns as well. - -The path names may contain directory portions; any necessary directories will be -created in the installation. - -For example, if a package should contain a subdirectory with several data files, -the files can be arranged like this in the source tree:: - - setup.py - src/ - mypkg/ - __init__.py - module.py - data/ - tables.dat - spoons.dat - forks.dat - -The corresponding call to :func:`setup` might be:: - - setup(..., - packages=['mypkg'], - package_dir={'mypkg': 'src/mypkg'}, - package_data={'mypkg': ['data/*.dat']}) - - -All the files that match ``package_data`` will be added to the ``MANIFEST`` -file if no template is provided. See :ref:`manifest`. - - -.. _distutils-additional-files: - -Installing Additional Files -=========================== - -The :option:`data_files` option can be used to specify additional files needed -by the module distribution: configuration files, message catalogs, data files, -anything which doesn't fit in the previous categories. - -:option:`data_files` specifies a sequence of (*directory*, *files*) pairs in the -following way:: - - setup(..., - data_files=[('bitmaps', ['bm/b1.gif', 'bm/b2.gif']), - ('config', ['cfg/data.cfg']), - ('/etc/init.d', ['init-script'])]) - -Note that you can specify the directory names where the data files will be -installed, but you cannot rename the data files themselves. - -Each (*directory*, *files*) pair in the sequence specifies the installation -directory and the files to install there. If *directory* is a relative path, it -is interpreted relative to the installation prefix (Python's ``sys.prefix`` for -pure-Python packages, ``sys.exec_prefix`` for packages that contain extension -modules). Each file name in *files* is interpreted relative to the -:file:`setup.py` script at the top of the package source distribution. No -directory information from *files* is used to determine the final location of -the installed file; only the name of the file is used. - -You can specify the :option:`data_files` options as a simple sequence of files -without specifying a target directory, but this is not recommended, and the -:command:`install` command will print a warning in this case. To install data -files directly in the target directory, an empty string should be given as the -directory. - -All the files that match ``data_files`` will be added to the ``MANIFEST`` file -if no template is provided. See :ref:`manifest`. - - - -.. _metadata: - -Metadata reference -================== - -The setup script may include additional metadata beyond the name and version. -This table describes required and additional information: - -+----------------------+---------------------------+-----------------+--------+ -| Meta-Data | Description | Value | Notes | -+======================+===========================+=================+========+ -| ``name`` | name of the project | short string | \(1) | -+----------------------+---------------------------+-----------------+--------+ -| ``version`` | version of this release | short string | (1)(2) | -+----------------------+---------------------------+-----------------+--------+ -| ``author`` | project author's name | short string | \(3) | -+----------------------+---------------------------+-----------------+--------+ -| ``author_email`` | email address of the | email address | \(3) | -| | project author | | | -+----------------------+---------------------------+-----------------+--------+ -| ``maintainer`` | project maintainer's name | short string | \(3) | -+----------------------+---------------------------+-----------------+--------+ -| ``maintainer_email`` | email address of the | email address | \(3) | -| | project maintainer | | | -+----------------------+---------------------------+-----------------+--------+ -| ``home_page`` | home page for the project | URL | \(1) | -+----------------------+---------------------------+-----------------+--------+ -| ``summary`` | short description of the | short string | | -| | project | | | -+----------------------+---------------------------+-----------------+--------+ -| ``description`` | longer description of the | long string | \(5) | -| | project | | | -+----------------------+---------------------------+-----------------+--------+ -| ``download_url`` | location where the | URL | | -| | project may be downloaded | | | -+----------------------+---------------------------+-----------------+--------+ -| ``classifiers`` | a list of classifiers | list of strings | \(4) | -+----------------------+---------------------------+-----------------+--------+ -| ``platforms`` | a list of platforms | list of strings | | -+----------------------+---------------------------+-----------------+--------+ -| ``license`` | license for the release | short string | \(6) | -+----------------------+---------------------------+-----------------+--------+ - -Notes: - -(1) - These fields are required. - -(2) - It is recommended that versions take the form *major.minor[.patch[.sub]]*. - -(3) - Either the author or the maintainer must be identified. - -(4) - The list of classifiers is available from the `PyPI website - `_. See also :mod:`distutils2.mkcfg`. - -(5) - The ``description`` field is used by PyPI when you are registering a - release, to build its PyPI page. - -(6) - The ``license`` field is a text indicating the license covering the - distribution where the license is not a selection from the "License" Trove - classifiers. See the ``Classifier`` field. Notice that - there's a ``licence`` distribution option which is deprecated but still - acts as an alias for ``license``. - -'short string' - A single line of text, not more than 200 characters. - -'long string' - Multiple lines of plain text in reStructuredText format (see - http://docutils.sf.net/). - -'list of strings' - See below. - -In Python 2.x, "string value" means a unicode object. If a byte string (str or -bytes) is given, it has to be valid ASCII. - -.. TODO move this section to the version document, keep a summary, add a link - -Encoding the version information is an art in itself. Python projects generally -adhere to the version format *major.minor[.patch][sub]*. The major number is 0 -for initial, experimental releases of software. It is incremented for releases -that represent major milestones in a project. The minor number is incremented -when important new features are added to the project. The patch number -increments when bug-fix releases are made. Additional trailing version -information is sometimes used to indicate sub-releases. These are -"a1,a2,...,aN" (for alpha releases, where functionality and API may change), -"b1,b2,...,bN" (for beta releases, which only fix bugs) and "pr1,pr2,...,prN" -(for final pre-release release testing). Some examples: - -0.1.0 - the first, experimental release of a project - -1.0.1a2 - the second alpha release of the first patch version of 1.0 - -:option:`classifiers` are specified in a Python list:: - - setup(..., - classifiers=[ - 'Development Status :: 4 - Beta', - 'Environment :: Console', - 'Environment :: Web Environment', - 'Intended Audience :: End Users/Desktop', - 'Intended Audience :: Developers', - 'Intended Audience :: System Administrators', - 'License :: OSI Approved :: Python Software Foundation License', - 'Operating System :: MacOS :: MacOS X', - 'Operating System :: Microsoft :: Windows', - 'Operating System :: POSIX', - 'Programming Language :: Python', - 'Topic :: Communications :: Email', - 'Topic :: Office/Business', - 'Topic :: Software Development :: Bug Tracking', - ]) - - -Debugging the setup script -========================== - -Sometimes things go wrong, and the setup script doesn't do what the developer -wants. - -Distutils catches any exceptions when running the setup script, and print a -simple error message before the script is terminated. The motivation for this -behaviour is to not confuse administrators who don't know much about Python and -are trying to install a project. If they get a big long traceback from deep -inside the guts of Distutils, they may think the project or the Python -installation is broken because they don't read all the way down to the bottom -and see that it's a permission problem. - -.. FIXME DISTUTILS_DEBUG is dead, document logging/warnings here - -On the other hand, this doesn't help the developer to find the cause of the -failure. For this purpose, the DISTUTILS_DEBUG environment variable can be set -to anything except an empty string, and Distutils2 will now print detailed -information about what it is doing, and prints the full traceback in case an -exception occurs. diff --git a/docs/source/distutils/sourcedist.rst b/docs/source/distutils/sourcedist.rst deleted file mode 100644 --- a/docs/source/distutils/sourcedist.rst +++ /dev/null @@ -1,272 +0,0 @@ -.. _source-dist: - -****************************** -Creating a Source Distribution -****************************** - -As shown in section :ref:`distutils-simple-example`, you use the :command:`sdist` command -to create a source distribution. In the simplest case, :: - - python setup.py sdist - -(assuming you haven't specified any :command:`sdist` options in the setup script -or config file), :command:`sdist` creates the archive of the default format for -the current platform. The default format is a gzip'ed tar file -(:file:`.tar.gz`) on Unix, and ZIP file on Windows. - -You can specify as many formats as you like using the :option:`--formats` -option, for example:: - - python setup.py sdist --formats=gztar,zip - -to create a gzipped tarball and a zip file. The available formats are: - -+-----------+-------------------------+---------+ -| Format | Description | Notes | -+===========+=========================+=========+ -| ``zip`` | zip file (:file:`.zip`) | (1),(3) | -+-----------+-------------------------+---------+ -| ``gztar`` | gzip'ed tar file | \(2) | -| | (:file:`.tar.gz`) | | -+-----------+-------------------------+---------+ -| ``bztar`` | bzip2'ed tar file | | -| | (:file:`.tar.bz2`) | | -+-----------+-------------------------+---------+ -| ``ztar`` | compressed tar file | \(4) | -| | (:file:`.tar.Z`) | | -+-----------+-------------------------+---------+ -| ``tar`` | tar file (:file:`.tar`) | | -+-----------+-------------------------+---------+ - -Notes: - -(1) - default on Windows - -(2) - default on Unix - -(3) - requires either external :program:`zip` utility or :mod:`zipfile` module (part - of the standard Python library since Python 1.6) - -(4) - requires the :program:`compress` program. Notice that this format is now - pending for deprecation and will be removed in the future versions of Python. - -When using any ``tar`` format (``gztar``, ``bztar``, ``ztar`` or -``tar``) under Unix, you can specify the ``owner`` and ``group`` names -that will be set for each member of the archive. - -For example, if you want all files of the archive to be owned by root:: - - python setup.py sdist --owner=root --group=root - - -.. _manifest: - -Specifying the files to distribute -================================== - -If you don't supply an explicit list of files (or instructions on how to -generate one), the :command:`sdist` command puts a minimal default set into the -source distribution: - -* all Python source files implied by the :option:`py_modules` and - :option:`packages` options - -* all C source files mentioned in the :option:`ext_modules` or - :option:`libraries` options - -* scripts identified by the :option:`scripts` option - See :ref:`distutils-installing-scripts`. - -* anything that looks like a test script: :file:`test/test\*.py` (currently, the - Distutils don't do anything with test scripts except include them in source - distributions, but in the future there will be a standard for testing Python - module distributions) - -* The configuration file :file:`setup.cfg` - -* all files that matches the ``package_data`` metadata. - See :ref:`distutils-installing-package-data`. - -* all files that matches the ``data_files`` metadata. - See :ref:`distutils-additional-files`. - -.. Warning:: - In Distutils2, setup.py and README (or README.txt) files are not more - included in source distribution by default - -Sometimes this is enough, but usually you will want to specify additional files -to distribute. The typical way to do this is to write a *manifest template*, -called :file:`MANIFEST.in` by default. The manifest template is just a list of -instructions for how to generate your manifest file, :file:`MANIFEST`, which is -the exact list of files to include in your source distribution. The -:command:`sdist` command processes this template and generates a manifest based -on its instructions and what it finds in the filesystem. - -If you prefer to roll your own manifest file, the format is simple: one filename -per line, regular files (or symlinks to them) only. If you do supply your own -:file:`MANIFEST`, you must specify everything: the default set of files -described above does not apply in this case. - -:file:`MANIFEST` files start with a comment indicating they are generated. -Files without this comment are not overwritten or removed. - -See :ref:`manifest_template` section for a syntax reference. - -.. _manifest-options: - -Manifest-related options -======================== - -The normal course of operations for the :command:`sdist` command is as follows: - -* if the manifest file, :file:`MANIFEST` doesn't exist, read :file:`MANIFEST.in` - and create the manifest - -* if neither :file:`MANIFEST` nor :file:`MANIFEST.in` exist, create a manifest - with just the default file set - -* if either :file:`MANIFEST.in` or the setup script (:file:`setup.py`) are more - recent than :file:`MANIFEST`, recreate :file:`MANIFEST` by reading - :file:`MANIFEST.in` - -* use the list of files now in :file:`MANIFEST` (either just generated or read - in) to create the source distribution archive(s) - -There are a couple of options that modify this behaviour. First, use the -:option:`--no-defaults` and :option:`--no-prune` to disable the standard -"include" and "exclude" sets. - -Second, you might just want to (re)generate the manifest, but not create a -source distribution:: - - python setup.py sdist --manifest-only - -:option:`-o` is a shortcut for :option:`--manifest-only`. - -.. _manifest_template: - -The MANIFEST.in template -======================== - -A :file:`MANIFEST.in` file can be added in a project to define the list of -files to include in the distribution built by the :command:`sdist` command. - -When :command:`sdist` is run, it will look for the :file:`MANIFEST.in` file -and interpret it to generate the :file:`MANIFEST` file that contains the -list of files that will be included in the package. - -This mechanism can be used when the default list of files is not enough. -(See :ref:`manifest`). - -Principle ---------- - -The manifest template has one command per line, where each command specifies a -set of files to include or exclude from the source distribution. For an -example, let's look at the Distutils' own manifest template:: - - include *.txt - recursive-include examples *.txt *.py - prune examples/sample?/build - -The meanings should be fairly clear: include all files in the distribution root -matching :file:`\*.txt`, all files anywhere under the :file:`examples` directory -matching :file:`\*.txt` or :file:`\*.py`, and exclude all directories matching -:file:`examples/sample?/build`. All of this is done *after* the standard -include set, so you can exclude files from the standard set with explicit -instructions in the manifest template. (Or, you can use the -:option:`--no-defaults` option to disable the standard set entirely.) - -The order of commands in the manifest template matters: initially, we have the -list of default files as described above, and each command in the template adds -to or removes from that list of files. Once we have fully processed the -manifest template, we remove files that should not be included in the source -distribution: - -* all files in the Distutils "build" tree (default :file:`build/`) - -* all files in directories named :file:`RCS`, :file:`CVS`, :file:`.svn`, - :file:`.hg`, :file:`.git`, :file:`.bzr` or :file:`_darcs` - -Now we have our complete list of files, which is written to the manifest for -future reference, and then used to build the source distribution archive(s). - -You can disable the default set of included files with the -:option:`--no-defaults` option, and you can disable the standard exclude set -with :option:`--no-prune`. - -Following the Distutils' own manifest template, let's trace how the -:command:`sdist` command builds the list of files to include in the Distutils -source distribution: - -#. include all Python source files in the :file:`distutils2` and - :file:`distutils2/command` subdirectories (because packages corresponding to - those two directories were mentioned in the :option:`packages` option in the - setup script---see section :ref:`setup-script`) - -#. include :file:`README.txt`, :file:`setup.py`, and :file:`setup.cfg` (standard - files) - -#. include :file:`test/test\*.py` (standard files) - -#. include :file:`\*.txt` in the distribution root (this will find - :file:`README.txt` a second time, but such redundancies are weeded out later) - -#. include anything matching :file:`\*.txt` or :file:`\*.py` in the sub-tree - under :file:`examples`, - -#. exclude all files in the sub-trees starting at directories matching - :file:`examples/sample?/build`\ ---this may exclude files included by the - previous two steps, so it's important that the ``prune`` command in the manifest - template comes after the ``recursive-include`` command - -#. exclude the entire :file:`build` tree, and any :file:`RCS`, :file:`CVS`, - :file:`.svn`, :file:`.hg`, :file:`.git`, :file:`.bzr` and :file:`_darcs` - directories - -Just like in the setup script, file and directory names in the manifest template -should always be slash-separated; the Distutils will take care of converting -them to the standard representation on your platform. That way, the manifest -template is portable across operating systems. - -Commands --------- - -The manifest template commands are: - -+-------------------------------------------+-----------------------------------------------+ -| Command | Description | -+===========================================+===============================================+ -| :command:`include pat1 pat2 ...` | include all files matching any of the listed | -| | patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`exclude pat1 pat2 ...` | exclude all files matching any of the listed | -| | patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`recursive-include dir pat1 pat2 | include all files under *dir* matching any of | -| ...` | the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`recursive-exclude dir pat1 pat2 | exclude all files under *dir* matching any of | -| ...` | the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`global-include pat1 pat2 ...` | include all files anywhere in the source tree | -| | matching --- & any of the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`global-exclude pat1 pat2 ...` | exclude all files anywhere in the source tree | -| | matching --- & any of the listed patterns | -+-------------------------------------------+-----------------------------------------------+ -| :command:`prune dir` | exclude all files under *dir* | -+-------------------------------------------+-----------------------------------------------+ -| :command:`graft dir` | include all files under *dir* | -+-------------------------------------------+-----------------------------------------------+ - -The patterns here are Unix-style "glob" patterns: ``*`` matches any sequence of -regular filename characters, ``?`` matches any single regular filename -character, and ``[range]`` matches any of the characters in *range* (e.g., -``a-z``, ``a-zA-Z``, ``a-f0-9_.``). The definition of "regular filename -character" is platform-specific: on Unix it is anything except slash; on Windows -anything except backslash or colon. diff --git a/docs/source/distutils/uploading.rst b/docs/source/distutils/uploading.rst deleted file mode 100644 --- a/docs/source/distutils/uploading.rst +++ /dev/null @@ -1,80 +0,0 @@ -.. _package-upload: - -*************************************** -Uploading Packages to the Package Index -*************************************** - -The Python Package Index (PyPI) not only stores the package info, but also the -package data if the author of the package wishes to. The distutils command -:command:`upload` pushes the distribution files to PyPI. - -The command is invoked immediately after building one or more distribution -files. For example, the command :: - - python setup.py sdist bdist_wininst upload - -will cause the source distribution and the Windows installer to be uploaded to -PyPI. Note that these will be uploaded even if they are built using an earlier -invocation of :file:`setup.py`, but that only distributions named on the command -line for the invocation including the :command:`upload` command are uploaded. - -The :command:`upload` command uses the username, password, and repository URL -from the :file:`$HOME/.pypirc` file (see section :ref:`pypirc` for more on this -file). If a :command:`register` command was previously called in the same -command, and if the password was entered in the prompt, :command:`upload` will -reuse the entered password. This is useful if you do not want to store a clear -text password in the :file:`$HOME/.pypirc` file. - -You can specify another PyPI server with the :option:`--repository=*url*` -option:: - - python setup.py sdist bdist_wininst upload -r http://example.com/pypi - -See section :ref:`pypirc` for more on defining several servers. - -You can use the :option:`--sign` option to tell :command:`upload` to sign each -uploaded file using GPG (GNU Privacy Guard). The :program:`gpg` program must -be available for execution on the system :envvar:`PATH`. You can also specify -which key to use for signing using the :option:`--identity=*name*` option. - -Other :command:`upload` options include :option:`--repository=` or -:option:`--repository=
` where *url* is the url of the server and -*section* the name of the section in :file:`$HOME/.pypirc`, and -:option:`--show-response` (which displays the full response text from the PyPI -server for help in debugging upload problems). - -PyPI package display -==================== - -The ``description`` field plays a special role at PyPI. It is used by -the server to display a home page for the registered package. - -If you use the `reStructuredText `_ -syntax for this field, PyPI will parse it and display an HTML output for -the package home page. - -The ``description`` field can be filled from a text file located in the -project:: - - from distutils2.core import setup - - fp = open('README.txt') - try: - description = fp.read() - finally: - fp.close() - - setup(name='Distutils2', - description=description) - -In that case, :file:`README.txt` is a regular reStructuredText text file located -in the root of the package besides :file:`setup.py`. - -To prevent registering broken reStructuredText content, you can use the -:program:`rst2html` program that is provided by the :mod:`docutils` package -and check the ``description`` from the command line:: - - $ python setup.py --description | rst2html.py > output.html - -:mod:`docutils` will display a warning if there's something wrong with your -syntax. diff --git a/docs/source/images/depgraph_output.png b/docs/source/images/depgraph_output.png deleted file mode 100644 Binary file docs/source/images/depgraph_output.png has changed diff --git a/docs/source/index.rst b/docs/source/index.rst deleted file mode 100644 --- a/docs/source/index.rst +++ /dev/null @@ -1,91 +0,0 @@ -.. Distutils2 doc site master file - This doc serves as project homepage and documentation preview - (on distutils2.notmyidea.org) until distutils2 is merged back into - the Python standard library. - This file should contain the root `toctree` directive. - - -Welcome to Distutils2! -====================== - -Distutils2 is the new, improved version of the Python Distribution Utilities, -a library used to package, distribute, build and install Python projects. - -- `Origin of the project`__ -- `Main code repository`__ -- `Clones of this repository`__ -- `Bug tracker`__ (some bugs may be assigned only to Distutils) -- `Teams that worked on Distutils2 for Google Summer of Code 2010`__ (links to - their code repositories and weblogs) - -.. __: http://tarekziade.wordpress.com/2010/03/03/ - the-fate-of-distutils-pycon-summit-packaging-sprint-detailed-report/ -.. __: http://bitbucket.org/tarek/distutils2/ -.. __: http://bitbucket.org/tarek/distutils2/descendants/ -.. __: http://bugs.python.org/issue?%40sort0=activity&%40sortdir0=on&%40sort1= - &%40group0=priority&%40group1=&%40columns=title%2Cid%2Cactivity%2Cstatus - &%40filter=components%2Cstatus&status=1&components=25&%40pagesize=50 - &%40startwith=0 -.. __: http://bitbucket.org/tarek/distutils2/wiki/GSoC_2010_teams - -If you?re looking for information on how to contribute, head to -:doc:`devresources`, and be sure to have a look at :doc:`contributing`. - - -Documentation -------------- - -These documents are the in-development version of Distutils2's documentation, -started from the existing Distutils documentation and updated by the -Distutils2 group (GSoC students, mentors, volunteers). - -Please remember that this is a work in progress. The documentation is not -complete, not spell-checked, and uses different styles. - -The documentation is split in four sections, as in the standard Python docs: - -:doc:`install/index` - A guide for for end-users wanting to install a Python application or - library. - -:doc:`setupcfg-spec` - Specifications of setup.cfg, the most important file for developers. - -:doc:`setupcfg` - Documenation about using setup.cfg - -:doc:`tutorial` - A tutorial for Python developers to discover Distutils2 main features. - -:doc:`distutils/index` - A guide for Python developers wanting to package and distribute their - project. - -:doc:`library/distutils2` - A reference for developers wanting to use standalone building blocks like - :mod:`~distutils2.version` or :mod:`~distutils2.metadata`, or extend - Distutils2 itself. Since :PEP:`376` is partly implemented in the - :mod:`pkgutil` module, its updated documentation is also included: - :doc:`library/pkgutil`. - - -.. toctree:: - :hidden: - - devresources - install/index - setupcfg-spec - setupcfg - tutorial - distutils/index - library/distutils2 - library/pkgutil - contributing - - -Indices and tables ------------------- - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` diff --git a/docs/source/install/index.rst b/docs/source/install/index.rst deleted file mode 100644 --- a/docs/source/install/index.rst +++ /dev/null @@ -1,1004 +0,0 @@ -.. highlightlang:: none - -.. _install-index: - -****************************** - Installing Python Projects -****************************** - -:Author: Greg Ward and Distutils2 contributors -:Release: |version| -:Date: |today| - -.. TODO: Fill in XXX comments - -.. The audience for this document includes people who don't know anything - about Python and aren't about to learn the language just in order to - install and maintain it for their users, i.e. system administrators. - Thus, I have to be sure to explain the basics at some point: - sys.path and PYTHONPATH at least. Should probably give pointers to - other docs on "import site", PYTHONSTARTUP, PYTHONHOME, etc. - - Finally, it might be useful to include all the material from my "Care - and Feeding of a Python Installation" talk in here somewhere. Yow! - -.. topic:: Abstract - - This document describes the Python Distribution Utilities ("Distutils2") from - the end-user's point-of-view, describing how to extend the capabilities of a - standard Python installation by building and installing third-party Python - modules and extensions. - - -.. _inst-intro: - -Introduction -============ - -Although Python's extensive standard library covers many programming needs, -there often comes a time when you need to add some new functionality to your -Python installation in the form of third-party modules. This might be necessary -to support your own programming, or to support an application that you want to -use and that happens to be written in Python. - -In the past, there has been little support for adding third-party modules to an -existing Python installation. With the introduction of the Python Distribution -Utilities (Distutils for short) in Python 2.0, this changed. - -This document is aimed primarily at the people who need to install third-party -Python modules: end-users and system administrators who just need to get some -Python application running, and existing Python programmers who want to add some -new goodies to their toolbox. You don't need to know Python to read this -document; there will be some brief forays into using Python's interactive mode -to explore your installation, but that's it. If you're looking for information -on how to distribute your own Python modules so that others may use them, see -the :ref:`distutils-index` manual. - - -.. _inst-trivial-install: - -Best case: trivial installation -------------------------------- - -In the best case, someone will have prepared a special version of the module -distribution you want to install that is targeted specifically at your platform -and is installed just like any other software on your platform. For example, -the module developer might make an executable installer available for Windows -users, an RPM package for users of RPM-based Linux systems (Red Hat, SuSE, -Mandrake, and many others), a Debian package for users of Debian-based Linux -systems, and so forth. - -In that case, you would download the installer appropriate to your platform and -do the obvious thing with it: run it if it's an executable installer, ``rpm ---install`` it if it's an RPM, etc. You don't need to run Python or a setup -script, you don't need to compile anything---you might not even need to read any -instructions (although it's always a good idea to do so anyways). - -Of course, things will not always be that easy. You might be interested in a -module distribution that doesn't have an easy-to-use installer for your -platform. In that case, you'll have to start with the source distribution -released by the module's author/maintainer. Installing from a source -distribution is not too hard, as long as the modules are packaged in the -standard way. The bulk of this document is about building and installing -modules from standard source distributions. - - -.. _inst-new-standard: - -The new standard: Distutils ---------------------------- - -If you download a module source distribution, you can tell pretty quickly if it -was packaged and distributed in the standard way, i.e. using the Distutils. -First, the distribution's name and version number will be featured prominently -in the name of the downloaded archive, e.g. :file:`foo-1.0.tar.gz` or -:file:`widget-0.9.7.zip`. Next, the archive will unpack into a similarly-named -directory: :file:`foo-1.0` or :file:`widget-0.9.7`. Additionally, the -distribution will contain a setup script :file:`setup.py`, and a file named -:file:`README.txt` or possibly just :file:`README`, which should explain that -building and installing the module distribution is a simple matter of running :: - - python setup.py install - -If all these things are true, then you already know how to build and install the -modules you've just downloaded: Run the command above. Unless you need to -install things in a non-standard way or customize the build process, you don't -really need this manual. Or rather, the above command is everything you need to -get out of this manual. - - -.. _inst-standard-install: - -Standard Build and Install -========================== - -As described in section :ref:`inst-new-standard`, building and installing a module -distribution using the Distutils is usually one simple command:: - - python setup.py install - -On Unix, you'd run this command from a shell prompt; on Windows, you have to -open a command prompt window ("DOS box") and do it there; on Mac OS X, you open -a :command:`Terminal` window to get a shell prompt. - - -.. _inst-platform-variations: - -Platform variations -------------------- - -You should always run the setup command from the distribution root directory, -i.e. the top-level subdirectory that the module source distribution unpacks -into. For example, if you've just downloaded a module source distribution -:file:`foo-1.0.tar.gz` onto a Unix system, the normal thing to do is:: - - gunzip -c foo-1.0.tar.gz | tar xf - # unpacks into directory foo-1.0 - cd foo-1.0 - python setup.py install - -On Windows, you'd probably download :file:`foo-1.0.zip`. If you downloaded the -archive file to :file:`C:\\Temp`, then it would unpack into -:file:`C:\\Temp\\foo-1.0`; you can use either a archive manipulator with a -graphical user interface (such as WinZip) or a command-line tool (such as -:program:`unzip` or :program:`pkunzip`) to unpack the archive. Then, open a -command prompt window ("DOS box"), and run:: - - cd c:\Temp\foo-1.0 - python setup.py install - - -.. _inst-splitting-up: - -Splitting the job up --------------------- - -Running ``setup.py install`` builds and installs all modules in one run. If you -prefer to work incrementally---especially useful if you want to customize the -build process, or if things are going wrong---you can use the setup script to do -one thing at a time. This is particularly helpful when the build and install -will be done by different users---for example, you might want to build a module -distribution and hand it off to a system administrator for installation (or do -it yourself, with super-user privileges). - -For example, you can build everything in one step, and then install everything -in a second step, by invoking the setup script twice:: - - python setup.py build - python setup.py install - -If you do this, you will notice that running the :command:`install` command -first runs the :command:`build` command, which---in this case---quickly notices -that it has nothing to do, since everything in the :file:`build` directory is -up-to-date. - -You may not need this ability to break things down often if all you do is -install modules downloaded off the 'net, but it's very handy for more advanced -tasks. If you get into distributing your own Python modules and extensions, -you'll run lots of individual Distutils commands on their own. - - -.. _inst-how-build-works: - -How building works ------------------- - -As implied above, the :command:`build` command is responsible for putting the -files to install into a *build directory*. By default, this is :file:`build` -under the distribution root; if you're excessively concerned with speed, or want -to keep the source tree pristine, you can change the build directory with the -:option:`--build-base` option. For example:: - - python setup.py build --build-base /tmp/pybuild/foo-1.0 - -(Or you could do this permanently with a directive in your system or personal -Distutils configuration file; see section :ref:`inst-config-files`.) Normally, -this isn't necessary. - -The default layout for the build tree is as follows:: - - --- build/ --- lib/ - or - --- build/ --- lib./ - temp./ - -where ```` expands to a brief description of the current OS/hardware -platform and Python version. The first form, with just a :file:`lib` directory, -is used for "pure module distributions"---that is, module distributions that -include only pure Python modules. If a module distribution contains any -extensions (modules written in C/C++), then the second form, with two ```` -directories, is used. In that case, the :file:`temp.{plat}` directory holds -temporary files generated by the compile/link process that don't actually get -installed. In either case, the :file:`lib` (or :file:`lib.{plat}`) directory -contains all Python modules (pure Python and extensions) that will be installed. - -In the future, more directories will be added to handle Python scripts, -documentation, binary executables, and whatever else is needed to handle the job -of installing Python modules and applications. - - -.. _inst-how-install-works: - -How installation works ----------------------- - -After the :command:`build` command runs (whether you run it explicitly, or the -:command:`install` command does it for you), the work of the :command:`install` -command is relatively simple: all it has to do is copy everything under -:file:`build/lib` (or :file:`build/lib.{plat}`) to your chosen installation -directory. - -If you don't choose an installation directory---i.e., if you just run ``setup.py -install``\ ---then the :command:`install` command installs to the standard -location for third-party Python modules. This location varies by platform and -by how you built/installed Python itself. On Unix (and Mac OS X, which is also -Unix-based), it also depends on whether the module distribution being installed -is pure Python or contains extensions ("non-pure"): - -+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ -| Platform | Standard installation location | Default value | Notes | -+=================+=====================================================+==================================================+=======+ -| Unix (pure) | :file:`{prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1) | -+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ -| Unix (non-pure) | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :file:`/usr/local/lib/python{X.Y}/site-packages` | \(1) | -+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ -| Windows | :file:`{prefix}` | :file:`C:\\Python` | \(2) | -+-----------------+-----------------------------------------------------+--------------------------------------------------+-------+ - -Notes: - -(1) - Most Linux distributions include Python as a standard part of the system, so - :file:`{prefix}` and :file:`{exec-prefix}` are usually both :file:`/usr` on - Linux. If you build Python yourself on Linux (or any Unix-like system), the - default :file:`{prefix}` and :file:`{exec-prefix}` are :file:`/usr/local`. - -(2) - The default installation directory on Windows was :file:`C:\\Program - Files\\Python` under Python 1.6a1, 1.5.2, and earlier. - -:file:`{prefix}` and :file:`{exec-prefix}` stand for the directories that Python -is installed to, and where it finds its libraries at run-time. They are always -the same under Windows, and very often the same under Unix and Mac OS X. You -can find out what your Python installation uses for :file:`{prefix}` and -:file:`{exec-prefix}` by running Python in interactive mode and typing a few -simple commands. Under Unix, just type ``python`` at the shell prompt. Under -Windows, choose :menuselection:`Start --> Programs --> Python X.Y --> -Python (command line)`. Once the interpreter is started, you type Python code -at the prompt. For example, on my Linux system, I type the three Python -statements shown below, and get the output as shown, to find out my -:file:`{prefix}` and :file:`{exec-prefix}`:: - - Python 2.4 (#26, Aug 7 2004, 17:19:02) - Type "help", "copyright", "credits" or "license" for more information. - >>> import sys - >>> sys.prefix - '/usr' - >>> sys.exec_prefix - '/usr' - -If you don't want to install modules to the standard location, or if you don't -have permission to write there, then you need to read about alternate -installations in section :ref:`inst-alt-install`. If you want to customize your -installation directories more heavily, see section :ref:`inst-custom-install` on -custom installations. - - -.. _inst-alt-install: - -Alternate Installation -====================== - -Often, it is necessary or desirable to install modules to a location other than -the standard location for third-party Python modules. For example, on a Unix -system you might not have permission to write to the standard third-party module -directory. Or you might wish to try out a module before making it a standard -part of your local Python installation. This is especially true when upgrading -a distribution already present: you want to make sure your existing base of -scripts still works with the new version before actually upgrading. - -The Distutils :command:`install` command is designed to make installing module -distributions to an alternate location simple and painless. The basic idea is -that you supply a base directory for the installation, and the -:command:`install` command picks a set of directories (called an *installation -scheme*) under this base directory in which to install files. The details -differ across platforms, so read whichever of the following sections applies to -you. - - -.. _inst-alt-install-prefix: - -Alternate installation: the home scheme ---------------------------------------- - -The idea behind the "home scheme" is that you build and maintain a personal -stash of Python modules. This scheme's name is derived from the idea of a -"home" directory on Unix, since it's not unusual for a Unix user to make their -home directory have a layout similar to :file:`/usr/` or :file:`/usr/local/`. -This scheme can be used by anyone, regardless of the operating system they -are installing for. - -Installing a new module distribution is as simple as :: - - python setup.py install --home - -where you can supply any directory you like for the :option:`--home` option. On -Unix, lazy typists can just type a tilde (``~``); the :command:`install` command -will expand this to your home directory:: - - python setup.py install --home ~ - -The :option:`--home` option defines the installation base directory. Files are -installed to the following directories under the installation base as follows: - -+------------------------------+---------------------------+-----------------------------+ -| Type of file | Installation Directory | Override option | -+==============================+===========================+=============================+ -| pure module distribution | :file:`{home}/lib/python` | :option:`--install-purelib` | -+------------------------------+---------------------------+-----------------------------+ -| non-pure module distribution | :file:`{home}/lib/python` | :option:`--install-platlib` | -+------------------------------+---------------------------+-----------------------------+ -| scripts | :file:`{home}/bin` | :option:`--install-scripts` | -+------------------------------+---------------------------+-----------------------------+ -| data | :file:`{home}/share` | :option:`--install-data` | -+------------------------------+---------------------------+-----------------------------+ - - -.. _inst-alt-install-home: - -Alternate installation: Unix (the prefix scheme) ------------------------------------------------- - -The "prefix scheme" is useful when you wish to use one Python installation to -perform the build/install (i.e., to run the setup script), but install modules -into the third-party module directory of a different Python installation (or -something that looks like a different Python installation). If this sounds a -trifle unusual, it is---that's why the "home scheme" comes first. However, -there are at least two known cases where the prefix scheme will be useful. - -First, consider that many Linux distributions put Python in :file:`/usr`, rather -than the more traditional :file:`/usr/local`. This is entirely appropriate, -since in those cases Python is part of "the system" rather than a local add-on. -However, if you are installing Python modules from source, you probably want -them to go in :file:`/usr/local/lib/python2.{X}` rather than -:file:`/usr/lib/python2.{X}`. This can be done with :: - - /usr/bin/python setup.py install --prefix /usr/local - -Another possibility is a network filesystem where the name used to write to a -remote directory is different from the name used to read it: for example, the -Python interpreter accessed as :file:`/usr/local/bin/python` might search for -modules in :file:`/usr/local/lib/python2.{X}`, but those modules would have to -be installed to, say, :file:`/mnt/{@server}/export/lib/python2.{X}`. This could -be done with :: - - /usr/local/bin/python setup.py install --prefix=/mnt/@server/export - -In either case, the :option:`--prefix` option defines the installation base, and -the :option:`--exec-prefix` option defines the platform-specific installation -base, which is used for platform-specific files. (Currently, this just means -non-pure module distributions, but could be expanded to C libraries, binary -executables, etc.) If :option:`--exec-prefix` is not supplied, it defaults to -:option:`--prefix`. Files are installed as follows: - -+------------------------------+-----------------------------------------------------+-----------------------------+ -| Type of file | Installation Directory | Override option | -+==============================+=====================================================+=============================+ -| pure module distribution | :file:`{prefix}/lib/python{X.Y}/site-packages` | :option:`--install-purelib` | -+------------------------------+-----------------------------------------------------+-----------------------------+ -| non-pure module distribution | :file:`{exec-prefix}/lib/python{X.Y}/site-packages` | :option:`--install-platlib` | -+------------------------------+-----------------------------------------------------+-----------------------------+ -| scripts | :file:`{prefix}/bin` | :option:`--install-scripts` | -+------------------------------+-----------------------------------------------------+-----------------------------+ -| data | :file:`{prefix}/share` | :option:`--install-data` | -+------------------------------+-----------------------------------------------------+-----------------------------+ - -There is no requirement that :option:`--prefix` or :option:`--exec-prefix` -actually point to an alternate Python installation; if the directories listed -above do not already exist, they are created at installation time. - -Incidentally, the real reason the prefix scheme is important is simply that a -standard Unix installation uses the prefix scheme, but with :option:`--prefix` -and :option:`--exec-prefix` supplied by Python itself as ``sys.prefix`` and -``sys.exec_prefix``. Thus, you might think you'll never use the prefix scheme, -but every time you run ``python setup.py install`` without any other options, -you're using it. - -Note that installing extensions to an alternate Python installation has no -effect on how those extensions are built: in particular, the Python header files -(:file:`Python.h` and friends) installed with the Python interpreter used to run -the setup script will be used in compiling extensions. It is your -responsibility to ensure that the interpreter used to run extensions installed -in this way is compatible with the interpreter used to build them. The best way -to do this is to ensure that the two interpreters are the same version of Python -(possibly different builds, or possibly copies of the same build). (Of course, -if your :option:`--prefix` and :option:`--exec-prefix` don't even point to an -alternate Python installation, this is immaterial.) - - -.. _inst-alt-install-windows: - -Alternate installation: Windows (the prefix scheme) ---------------------------------------------------- - -Windows has no concept of a user's home directory, and since the standard Python -installation under Windows is simpler than under Unix, the :option:`--prefix` -option has traditionally been used to install additional packages in separate -locations on Windows. :: - - python setup.py install --prefix "\Temp\Python" - -to install modules to the :file:`\\Temp\\Python` directory on the current drive. - -The installation base is defined by the :option:`--prefix` option; the -:option:`--exec-prefix` option is not supported under Windows. Files are -installed as follows: - -+------------------------------+---------------------------+-----------------------------+ -| Type of file | Installation Directory | Override option | -+==============================+===========================+=============================+ -| pure module distribution | :file:`{prefix}` | :option:`--install-purelib` | -+------------------------------+---------------------------+-----------------------------+ -| non-pure module distribution | :file:`{prefix}` | :option:`--install-platlib` | -+------------------------------+---------------------------+-----------------------------+ -| scripts | :file:`{prefix}\\Scripts` | :option:`--install-scripts` | -+------------------------------+---------------------------+-----------------------------+ -| data | :file:`{prefix}\\Data` | :option:`--install-data` | -+------------------------------+---------------------------+-----------------------------+ - - -.. _inst-custom-install: - -Custom Installation -=================== - -Sometimes, the alternate installation schemes described in section -:ref:`inst-alt-install` just don't do what you want. You might want to tweak -just one or two directories while keeping everything under the same base -directory, or you might want to completely redefine the installation scheme. -In either case, you're creating a *custom installation scheme*. - -You probably noticed the column of "override options" in the tables describing -the alternate installation schemes above. Those options are how you define a -custom installation scheme. These override options can be relative, absolute, -or explicitly defined in terms of one of the installation base directories. -(There are two installation base directories, and they are normally the same--- -they only differ when you use the Unix "prefix scheme" and supply different -:option:`--prefix` and :option:`--exec-prefix` options.) - -For example, say you're installing a module distribution to your home directory -under Unix---but you want scripts to go in :file:`~/scripts` rather than -:file:`~/bin`. As you might expect, you can override this directory with the -:option:`--install-scripts` option; in this case, it makes most sense to supply -a relative path, which will be interpreted relative to the installation base -directory (your home directory, in this case):: - - python setup.py install --home ~ --install-scripts scripts - -Another Unix example: suppose your Python installation was built and installed -with a prefix of :file:`/usr/local/python`, so under a standard installation -scripts will wind up in :file:`/usr/local/python/bin`. If you want them in -:file:`/usr/local/bin` instead, you would supply this absolute directory for the -:option:`--install-scripts` option:: - - python setup.py install --install-scripts /usr/local/bin - -(This performs an installation using the "prefix scheme," where the prefix is -whatever your Python interpreter was installed with--- :file:`/usr/local/python` -in this case.) - -If you maintain Python on Windows, you might want third-party modules to live in -a subdirectory of :file:`{prefix}`, rather than right in :file:`{prefix}` -itself. This is almost as easy as customizing the script installation directory ----you just have to remember that there are two types of modules to worry about, -pure modules and non-pure modules (i.e., modules from a non-pure distribution). -For example:: - - python setup.py install --install-purelib Site --install-platlib Site - -The specified installation directories are relative to :file:`{prefix}`. Of -course, you also have to ensure that these directories are in Python's module -search path, such as by putting a :file:`.pth` file in :file:`{prefix}`. See -section :ref:`inst-search-path` to find out how to modify Python's search path. - -If you want to define an entire installation scheme, you just have to supply all -of the installation directory options. The recommended way to do this is to -supply relative paths; for example, if you want to maintain all Python -module-related files under :file:`python` in your home directory, and you want a -separate directory for each platform that you use your home directory from, you -might define the following installation scheme:: - - python setup.py install --home ~ \ - --install-purelib python/lib \ - --install-platlib python/'lib.$PLAT' \ - --install-scripts python/scripts - --install-data python/data - -or, equivalently, :: - - python setup.py install --home ~/python \ - --install-purelib lib \ - --install-platlib 'lib.$PLAT' \ - --install-scripts scripts - --install-data data - -``$PLAT`` is not (necessarily) an environment variable---it will be expanded by -the Distutils as it parses your command line options, just as it does when -parsing your configuration file(s). - -Obviously, specifying the entire installation scheme every time you install a -new module distribution would be very tedious. Thus, you can put these options -into your Distutils config file (see section :ref:`inst-config-files`):: - - [install] - install-base = $HOME - install-purelib = python/lib - install-platlib = python/lib.$PLAT - install-scripts = python/scripts - install-data = python/data - -or, equivalently, :: - - [install] - install-base = $HOME/python - install-purelib = lib - install-platlib = lib.$PLAT - install-scripts = scripts - install-data = data - -Note that these two are *not* equivalent if you supply a different installation -base directory when you run the setup script. For example, :: - - python setup.py install --install-base /tmp - -would install pure modules to :file:`{/tmp/python/lib}` in the first case, and -to :file:`{/tmp/lib}` in the second case. (For the second case, you probably -want to supply an installation base of :file:`/tmp/python`.) - -You probably noticed the use of ``$HOME`` and ``$PLAT`` in the sample -configuration file input. These are Distutils configuration variables, which -bear a strong resemblance to environment variables. In fact, you can use -environment variables in config files on platforms that have such a notion but -the Distutils additionally define a few extra variables that may not be in your -environment, such as ``$PLAT``. (And of course, on systems that don't have -environment variables, such as Mac OS 9, the configuration variables supplied by -the Distutils are the only ones you can use.) See section :ref:`inst-config-files` -for details. - -.. XXX need some Windows examples---when would custom installation schemes be - needed on those platforms? - - -.. XXX I'm not sure where this section should go. - -.. _inst-search-path: - -Modifying Python's Search Path ------------------------------- - -When the Python interpreter executes an :keyword:`import` statement, it searches -for both Python code and extension modules along a search path. A default value -for the path is configured into the Python binary when the interpreter is built. -You can determine the path by importing the :mod:`sys` module and printing the -value of ``sys.path``. :: - - $ python - Python 2.2 (#11, Oct 3 2002, 13:31:27) - [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 - Type "help", "copyright", "credits" or "license" for more information. - >>> import sys - >>> sys.path - ['', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2', - '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload', - '/usr/local/lib/python2.3/site-packages'] - >>> - -The null string in ``sys.path`` represents the current working directory. - -The expected convention for locally installed packages is to put them in the -:file:`{...}/site-packages/` directory, but you may want to install Python -modules into some arbitrary directory. For example, your site may have a -convention of keeping all software related to the web server under :file:`/www`. -Add-on Python modules might then belong in :file:`/www/python`, and in order to -import them, this directory must be added to ``sys.path``. There are several -different ways to add the directory. - -The most convenient way is to add a path configuration file to a directory -that's already on Python's path, usually to the :file:`.../site-packages/` -directory. Path configuration files have an extension of :file:`.pth`, and each -line must contain a single path that will be appended to ``sys.path``. (Because -the new paths are appended to ``sys.path``, modules in the added directories -will not override standard modules. This means you can't use this mechanism for -installing fixed versions of standard modules.) - -Paths can be absolute or relative, in which case they're relative to the -directory containing the :file:`.pth` file. See the documentation of -the :mod:`site` module for more information. - -A slightly less convenient way is to edit the :file:`site.py` file in Python's -standard library, and modify ``sys.path``. :file:`site.py` is automatically -imported when the Python interpreter is executed, unless the :option:`-S` switch -is supplied to suppress this behaviour. So you could simply edit -:file:`site.py` and add two lines to it:: - - import sys - sys.path.append('/www/python/') - -However, if you reinstall the same major version of Python (perhaps when -upgrading from 2.2 to 2.2.2, for example) :file:`site.py` will be overwritten by -the stock version. You'd have to remember that it was modified and save a copy -before doing the installation. - -There are two environment variables that can modify ``sys.path``. -:envvar:`PYTHONHOME` sets an alternate value for the prefix of the Python -installation. For example, if :envvar:`PYTHONHOME` is set to ``/www/python``, -the search path will be set to ``['', '/www/python/lib/pythonX.Y/', -'/www/python/lib/pythonX.Y/plat-linux2', ...]``. - -The :envvar:`PYTHONPATH` variable can be set to a list of paths that will be -added to the beginning of ``sys.path``. For example, if :envvar:`PYTHONPATH` is -set to ``/www/python:/opt/py``, the search path will begin with -``['/www/python', '/opt/py']``. (Note that directories must exist in order to -be added to ``sys.path``; the :mod:`site` module removes paths that don't -exist.) - -Finally, ``sys.path`` is just a regular Python list, so any Python application -can modify it by adding or removing entries. - - -.. _inst-config-files: - -Distutils Configuration Files -============================= - -As mentioned above, you can use Distutils configuration files to record personal -or site preferences for any Distutils options. That is, any option to any -command can be stored in one of two or three (depending on your platform) -configuration files, which will be consulted before the command-line is parsed. -This means that configuration files will override default values, and the -command-line will in turn override configuration files. Furthermore, if -multiple configuration files apply, values from "earlier" files are overridden -by "later" files. - - -.. _inst-config-filenames: - -Location and names of config files ----------------------------------- - -The names and locations of the configuration files vary slightly across -platforms. On Unix and Mac OS X, the three configuration files (in the order -they are processed) are: - -+--------------+----------------------------------------------------------+-------+ -| Type of file | Location and filename | Notes | -+==============+==========================================================+=======+ -| system | :file:`{prefix}/lib/python{ver}/distutils/distutils.cfg` | \(1) | -+--------------+----------------------------------------------------------+-------+ -| personal | :file:`$HOME/.pydistutils.cfg` | \(2) | -+--------------+----------------------------------------------------------+-------+ -| local | :file:`setup.cfg` | \(3) | -+--------------+----------------------------------------------------------+-------+ - -And on Windows, the configuration files are: - -+--------------+-------------------------------------------------+-------+ -| Type of file | Location and filename | Notes | -+==============+=================================================+=======+ -| system | :file:`{prefix}\\Lib\\distutils\\distutils.cfg` | \(4) | -+--------------+-------------------------------------------------+-------+ -| personal | :file:`%HOME%\\pydistutils.cfg` | \(5) | -+--------------+-------------------------------------------------+-------+ -| local | :file:`setup.cfg` | \(3) | -+--------------+-------------------------------------------------+-------+ - -On all platforms, the "personal" file can be temporarily disabled by -passing the `--no-user-cfg` option. - -Notes: - -(1) - Strictly speaking, the system-wide configuration file lives in the directory - where the Distutils are installed; under Python 1.6 and later on Unix, this - is as shown. For Python 1.5.2, the Distutils will normally be installed to - :file:`{prefix}/lib/python1.5/site-packages/distutils`, so the system - configuration file should be put there under Python 1.5.2. - -(2) - On Unix, if the :envvar:`HOME` environment variable is not defined, the - user's home directory will be determined with the :func:`getpwuid` function - from the standard :mod:`pwd` module. This is done by the - :func:`os.path.expanduser` function used by Distutils. - -(3) - I.e., in the current directory (usually the location of the setup script). - -(4) - (See also note (1).) Python's default installation prefix is - :file:`C:\\Python`, so the system configuration file is normally - :file:`C:\\Python\\Lib\\distutils\\distutils.cfg`. - -(5) - On Windows, if the :envvar:`HOME` environment variable is not defined, - :envvar:`USERPROFILE` then :envvar:`HOMEDRIVE` and :envvar:`HOMEPATH` will - be tried. This is done by the :func:`os.path.expanduser` function used - by Distutils. - - -.. _inst-config-syntax: - -Syntax of config files ----------------------- - -The Distutils configuration files all have the same syntax. The config files -are grouped into sections. There is one section for each Distutils command, -plus a ``global`` section for global options that affect every command. Each -section consists of one option per line, specified as ``option = value``. - -For example, the following is a complete config file that just forces all -commands to run quietly by default:: - - [global] - verbose = 0 - -If this is installed as the system config file, it will affect all processing -of any Python module distribution by any user on the current system. If it is -installed as your personal config file (on systems that support them), it will -affect only module distributions processed by you. And if it is used as the -:file:`setup.cfg` for a particular module distribution, it affects only that -distribution. - -You could override the default "build base" directory and make the -:command:`build\*` commands always forcibly rebuild all files with the -following:: - - [build] - build-base = blib - force = 1 - -which corresponds to the command-line arguments :: - - python setup.py build --build-base blib --force - -except that including the :command:`build` command on the command-line means -that command will be run. Including a particular command in config files has no -such implication; it only means that if the command is run, the options in the -config file will apply. (Or if other commands that derive values from it are -run, they will use the values in the config file.) - -You can find out the complete list of options for any command using the -:option:`--help` option, e.g.:: - - python setup.py build --help - -and you can find out the complete list of global options by using -:option:`--help` without a command:: - - python setup.py --help - -See also the "Reference" section of the "Distributing Python Modules" manual. - - -.. _inst-building-ext: - -Building Extensions: Tips and Tricks -==================================== - -Whenever possible, the Distutils try to use the configuration information made -available by the Python interpreter used to run the :file:`setup.py` script. -For example, the same compiler and linker flags used to compile Python will also -be used for compiling extensions. Usually this will work well, but in -complicated situations this might be inappropriate. This section discusses how -to override the usual Distutils behaviour. - - -.. _inst-tweak-flags: - -Tweaking compiler/linker flags ------------------------------- - -Compiling a Python extension written in C or C++ will sometimes require -specifying custom flags for the compiler and linker in order to use a particular -library or produce a special kind of object code. This is especially true if the -extension hasn't been tested on your platform, or if you're trying to -cross-compile Python. - -.. TODO update to new setup.cfg - -In the most general case, the extension author might have foreseen that -compiling the extensions would be complicated, and provided a :file:`Setup` file -for you to edit. This will likely only be done if the module distribution -contains many separate extension modules, or if they often require elaborate -sets of compiler flags in order to work. - -A :file:`Setup` file, if present, is parsed in order to get a list of extensions -to build. Each line in a :file:`Setup` describes a single module. Lines have -the following structure:: - - module ... [sourcefile ...] [cpparg ...] [library ...] - - -Let's examine each of the fields in turn. - -* *module* is the name of the extension module to be built, and should be a - valid Python identifier. You can't just change this in order to rename a module - (edits to the source code would also be needed), so this should be left alone. - -* *sourcefile* is anything that's likely to be a source code file, at least - judging by the filename. Filenames ending in :file:`.c` are assumed to be - written in C, filenames ending in :file:`.C`, :file:`.cc`, and :file:`.c++` are - assumed to be C++, and filenames ending in :file:`.m` or :file:`.mm` are assumed - to be in Objective C. - -* *cpparg* is an argument for the C preprocessor, and is anything starting with - :option:`-I`, :option:`-D`, :option:`-U` or :option:`-C`. - -* *library* is anything ending in :file:`.a` or beginning with :option:`-l` or - :option:`-L`. - -If a particular platform requires a special library on your platform, you can -add it by editing the :file:`Setup` file and running ``python setup.py build``. -For example, if the module defined by the line :: - - foo foomodule.c - -must be linked with the math library :file:`libm.a` on your platform, simply add -:option:`-lm` to the line:: - - foo foomodule.c -lm - -Arbitrary switches intended for the compiler or the linker can be supplied with -the :option:`-Xcompiler` *arg* and :option:`-Xlinker` *arg* options:: - - foo foomodule.c -Xcompiler -o32 -Xlinker -shared -lm - -The next option after :option:`-Xcompiler` and :option:`-Xlinker` will be -appended to the proper command line, so in the above example the compiler will -be passed the :option:`-o32` option, and the linker will be passed -:option:`-shared`. If a compiler option requires an argument, you'll have to -supply multiple :option:`-Xcompiler` options; for example, to pass ``-x c++`` -the :file:`Setup` file would have to contain ``-Xcompiler -x -Xcompiler c++``. - -Compiler flags can also be supplied through setting the :envvar:`CFLAGS` -environment variable. If set, the contents of :envvar:`CFLAGS` will be added to -the compiler flags specified in the :file:`Setup` file. - - -.. _inst-non-ms-compilers: - -Using non-Microsoft compilers on Windows ----------------------------------------- - -.. sectionauthor:: Rene Liebscher - - - -Borland/CodeGear C++ -^^^^^^^^^^^^^^^^^^^^ - -This subsection describes the necessary steps to use Distutils with the Borland -C++ compiler version 5.5. First you have to know that Borland's object file -format (OMF) is different from the format used by the Python version you can -download from the Python or ActiveState Web site. (Python is built with -Microsoft Visual C++, which uses COFF as the object file format.) For this -reason you have to convert Python's library :file:`python25.lib` into the -Borland format. You can do this as follows: - -.. Should we mention that users have to create cfg-files for the compiler? -.. see also http://community.borland.com/article/0,1410,21205,00.html - -:: - - coff2omf python25.lib python25_bcpp.lib - -The :file:`coff2omf` program comes with the Borland compiler. The file -:file:`python25.lib` is in the :file:`Libs` directory of your Python -installation. If your extension uses other libraries (zlib, ...) you have to -convert them too. - -The converted files have to reside in the same directories as the normal -libraries. - -How does Distutils manage to use these libraries with their changed names? If -the extension needs a library (eg. :file:`foo`) Distutils checks first if it -finds a library with suffix :file:`_bcpp` (eg. :file:`foo_bcpp.lib`) and then -uses this library. In the case it doesn't find such a special library it uses -the default name (:file:`foo.lib`.) [#]_ - -To let Distutils compile your extension with Borland C++ you now have to type:: - - python setup.py build --compiler bcpp - -If you want to use the Borland C++ compiler as the default, you could specify -this in your personal or system-wide configuration file for Distutils (see -section :ref:`inst-config-files`.) - - -.. seealso:: - - `C++Builder Compiler `_ - Information about the free C++ compiler from Borland, including links to the - download pages. - - `Creating Python Extensions Using Borland's Free Compiler `_ - Document describing how to use Borland's free command-line C++ compiler to build - Python. - - -GNU C / Cygwin / MinGW -^^^^^^^^^^^^^^^^^^^^^^ - -This section describes the necessary steps to use Distutils with the GNU C/C++ -compilers in their Cygwin and MinGW distributions. [#]_ For a Python interpreter -that was built with Cygwin, everything should work without any of these -following steps. - -Not all extensions can be built with MinGW or Cygwin, but many can. Extensions -most likely to not work are those that use C++ or depend on Microsoft Visual C -extensions. - -To let Distutils compile your extension with Cygwin you have to type:: - - python setup.py build --compiler=cygwin - -and for Cygwin in no-cygwin mode [#]_ or for MinGW type:: - - python setup.py build --compiler=mingw32 - -If you want to use any of these options/compilers as default, you should -consider writing it in your personal or system-wide configuration file for -Distutils (see section :ref:`inst-config-files`.) - -Older Versions of Python and MinGW -"""""""""""""""""""""""""""""""""" -The following instructions only apply if you're using a version of Python -inferior to 2.4.1 with a MinGW inferior to 3.0.0 (with -binutils-2.13.90-20030111-1). - -These compilers require some special libraries. This task is more complex than -for Borland's C++, because there is no program to convert the library. First -you have to create a list of symbols which the Python DLL exports. (You can find -a good program for this task at -http://www.emmestech.com/software/pexports-0.43/download_pexports.html). - -.. I don't understand what the next line means. --amk -.. (inclusive the references on data structures.) - -:: - - pexports python25.dll > python25.def - -The location of an installed :file:`python25.dll` will depend on the -installation options and the version and language of Windows. In a "just for -me" installation, it will appear in the root of the installation directory. In -a shared installation, it will be located in the system directory. - -Then you can create from these information an import library for gcc. :: - - /cygwin/bin/dlltool --dllname python25.dll --def python25.def --output-lib libpython25.a - -The resulting library has to be placed in the same directory as -:file:`python25.lib`. (Should be the :file:`libs` directory under your Python -installation directory.) - -If your extension uses other libraries (zlib,...) you might have to convert -them too. The converted files have to reside in the same directories as the -normal libraries do. - - -.. seealso:: - - `Building Python modules on MS Windows platform with MinGW `_ - Information about building the required libraries for the MinGW - environment. - - -.. rubric:: Footnotes - -.. [#] This also means you could replace all existing COFF-libraries with - OMF-libraries of the same name. - -.. [#] Check http://sources.redhat.com/cygwin/ and http://www.mingw.org/ for - more information - -.. [#] Then you have no POSIX emulation available, but you also don't need - :file:`cygwin1.dll`. diff --git a/docs/source/library/distutils2.depgraph.rst b/docs/source/library/distutils2.depgraph.rst deleted file mode 100644 --- a/docs/source/library/distutils2.depgraph.rst +++ /dev/null @@ -1,125 +0,0 @@ -=============================== -Dependency Graph Builder Module -=============================== - -Introduction ------------- - -This module provides the means to analyse the dependencies between various -distributions and furthermore to create a graph representing the relationships -from a list of :class:`distutils2._backport.pkgutil.Distribution` and -:class:`distutils2._backport.pkgutil.EggInfoDistribution` instances. The graph -is represented by the :class:`distutils2.depgraph.DependencyGraph` class that -keeps internally an adjacency list. Several functions are provided that -generate a graph in different manners. First, all of them are documented and -then several use case examples are provided along with -`graphviz `_ illustrations -of the generated graphs. - -API ---- - -.. automodule:: distutils2.depgraph - :members: - -Example Usage -------------- - -Depict all Dependenciess in the System -++++++++++++++++++++++++++++++++++++++ - -First, we shall generate a graph of all the distributions on the system -and then create an image out of it using the tools provided by -`graphviz `_. For obtaining the list of -installed distributions, we will use the functions provided by the module -:mod:`distutils2._backport.pkgutil`:: - - from distutils2._backport.pkgutil import get_distributions - from distutils2.depgraph import generate_graph - - dists = list(pkgutil.get_distributions()) - graph = generate_graph(dists) - -Now, it would be of interest to print out the missing requirements. This -can be done as follows:: - - for dist, reqs in graph.missing.iteritems(): - if len(reqs) > 0: - reqs_s = " ,".join(reqs) - print("Missing dependencies for %s: %s" % (dist.name, reqs_s)) - -Example output on my configuration is: - -.. code-block:: none - - Missing dependencies for TurboCheetah: Cheetah - Missing dependencies for TurboGears: ConfigObj, DecoratorTools, RuleDispatch - Missing dependencies for jockey: PyKDE4.kdecore, PyKDE4.kdeui, PyQt4.QtCore, PyQt4.QtGui - Missing dependencies for TurboKid: kid - Missing dependencies for TurboJson: DecoratorTools, RuleDispatch - -Now, we proceed with generating a graphical representation of the graph. First -we write it to a file, and then we generate a PNG image using the ``dot`` -command-line tool:: - - from distutils2.depgraph import graph_to_dot - f = open('output.dot', 'w') - # we only show the interesting distributions, skipping the disconnected ones - graph_to_dot(graph, f, skip_disconnected=True) - -Now, we can create the actual picture using: - -.. code-block:: bash - - $ dot -Tpng output.dot > output.png - -An example output image is: - -.. figure:: ../images/depgraph_output.png - :alt: An example dot output - -If you want to include ``.egg`` and ``.egg-info`` distributions as well, then -the code requires only one change, namely the line:: - - dists = list(pkgutil.get_distributions()) - -has to be replaced with:: - - dists = list(pkgutil.get_distributions(use_egg_info=True)) - -Then, on most platforms, a richer graph is obtained because at the moment most -distributions are provided in the ``.egg``/``.egg-info`` rather than the -``.dist-info`` format. An example of a more involved graph for illustrative -reasons can be seen `here <_static/depgraph_big.png>`_. - - -List all Dependent Distributions -++++++++++++++++++++++++++++++++ - -We will list all distributions that are dependent on some given distibution. -This time, ``.egg``/``.egg-info`` distributions will be considered as well:: - - from distutils2._backport.pkgutil import get_distributions - from distutils2.depgraph import dependent_dists - import sys - - dists = list(get_distributions(use_egg_info=True)) - dist = None - for d in dists: - if d.name == 'bacon': - dist = d - break - if dist is None: - print('No such distribution in the system') - sys.exit(1) - deps = dependent_dists(dists, dist) - deps_s = ", ".join([x.name for x in deps]) - print("The following distributions depend on %s: %s" % (dist.name, deps_s)) - -And this is example output with the dependency relationships as in the -`previous section <_static/depgraph_big.png>`_: - -.. code-block:: none - - The following distributions depend on bacon: towel-stuff, choxie, grammar - diff --git a/docs/source/library/distutils2.index.client.rst b/docs/source/library/distutils2.index.client.rst deleted file mode 100644 --- a/docs/source/library/distutils2.index.client.rst +++ /dev/null @@ -1,20 +0,0 @@ -=============================== -High level API to Query indexes -=============================== - -Distutils2 provides a high level API to query indexes, search for releases and -distributions, no matters the underlying API you want to use. - -The aim of this module is to choose the best way to query the API, using the -less possible XML-RPC, and when possible the simple index. - -API -=== - -The client comes with the common methods "find_projects", "get_release" and -"get_releases", which helps to query the servers, and returns -:class:`distutils2.index.dist.ReleaseInfo`, and -:class:`distutils2.index.dist.ReleasesList` objects. - -.. autoclass:: distutils2.index.wrapper.ClientWrapper - :members: diff --git a/docs/source/library/distutils2.index.dist.rst b/docs/source/library/distutils2.index.dist.rst deleted file mode 100644 --- a/docs/source/library/distutils2.index.dist.rst +++ /dev/null @@ -1,112 +0,0 @@ -================================================== -Representation of informations coming from indexes -================================================== - -Informations coming from indexes are represented by the classes present in the -:mod:`distutils2.index.dist` module. - -API -=== - -Keep in mind that each project (eg. FooBar) can have several releases -(eg. 1.1, 1.2, 1.3), and each of these releases can be provided in multiple -distributions (eg. a source distribution, a binary one, etc). - -ReleaseInfo ------------- - -Each release have a project name, a project version and contain project -metadata. In addition, releases contain the distributions too. - -These informations are stored in :class:`distutils2.index.dist.ReleaseInfo` -objects. - -.. autoclass:: distutils2.index.dist.ReleaseInfo - :members: - -DistInfo ---------- - -:class:`distutils2.index.dist.DistInfo` is a simple class that contains -informations related to distributions. It's mainly about the URLs where those -distributions can be found. - -.. autoclass:: distutils2.index.dist.DistInfo - :members: - -ReleasesList ------------- - -The :mod:`~distutils2.index.dist` module also provides another class, to work -with lists of :class:`distutils2.index.dist.ReleaseInfo` classes. It allow to -filter and order results. - -.. autoclass:: distutils2.index.dist.ReleasesList - :members: - -Exemple usages -=============== - -Build a list of releases, and order them ----------------------------------------- - -Assuming we have a list of releases:: - - >>> from distutils2.index.dist import ReleaseList, ReleaseInfo - >>> fb10 = ReleaseInfo("FooBar", "1.0") - >>> fb11 = ReleaseInfo("FooBar", "1.1") - >>> fb11a = ReleaseInfo("FooBar", "1.1a1") - >>> ReleasesList("FooBar", [fb11, fb11a, fb10]) - >>> releases.sort_releases() - >>> releases.get_versions() - ['1.1', '1.1a1', '1.0'] - >>> releases.add_release("1.2a1") - >>> releases.get_versions() - ['1.1', '1.1a1', '1.0', '1.2a1'] - >>> releases.sort_releases() - ['1.2a1', '1.1', '1.1a1', '1.0'] - >>> releases.sort_releases(prefer_final=True) - >>> releases.get_versions() - ['1.1', '1.0', '1.2a1', '1.1a1'] - - -Add distribution related informations to releases -------------------------------------------------- - -It's easy to add distribution informatons to releases:: - - >>> from distutils2.index.dist import ReleaseList, ReleaseInfo - >>> r = ReleaseInfo("FooBar", "1.0") - >>> r.add_distribution("sdist", url="http://example.org/foobar-1.0.tar.gz") - >>> r.dists - {'sdist': FooBar 1.0 sdist} - >>> r['sdist'].url - {'url': 'http://example.org/foobar-1.0.tar.gz', 'hashname': None, 'hashval': - None, 'is_external': True} - -Getting attributes from the dist objects ------------------------------------------ - -To abstract a maximum the way of querying informations to the indexes, -attributes and releases informations can be retrieved directly from the objects -returned by the indexes. - -For instance, if you have a release instance that does not contain the metadata -attribute, it can be fetched by using the "fetch_metadata" method:: - - >>> r = Release("FooBar", "1.1") - >>> print r.metadata - None # metadata field is actually set to "None" - >>> r.fetch_metadata() - - -.. XXX add proper roles to these constructs - -Like this, it's possible to retrieve project's releases (`fetch_releases`), -releases metadata (`fetch_metadata` and releases distributions -(`fetch_distributions` informations). - -Internally, this is possible because while retrieving for the first time -informations about projects, releases or distributions, a reference to the -client used is stored in the objects (can be accessed using the object -`_index` attribute. diff --git a/docs/source/library/distutils2.index.rst b/docs/source/library/distutils2.index.rst deleted file mode 100644 --- a/docs/source/library/distutils2.index.rst +++ /dev/null @@ -1,31 +0,0 @@ -=================================== -Query Python Package Indexes (PyPI) -=================================== - -Distutils2 queries PyPI to get information about projects or download those -projects. The low-level facilities used internally are also part of the public -API destined to be used by other tools. - -The :mod:`distutils2.index` package provides those facilities, which can be -used to access informations about Python projects registered at indexes, the -main one being PyPI, located ad http://pypi.python.org/. - -There is two ways to retrieve data from these indexes: using the *simple* API, -and using *XML-RPC*. The first one is a set of HTML pages avalaibles at -`http://pypi.python.org/simple/`, and the second one contains a set of XML-RPC -methods. Mirrors typically implement the simple interface. - -If you dont care about which API to use, the best thing to do is to let -distutils2 decide this for you, by using :class:`distutils2.index.ClientWrapper`. - -Of course, you can rely too on :class:`distutils2.index.simple.Crawler` and -:class:`distutils2.index.xmlrpc.Client` if you need to use a specific API. - -.. toctree:: - :maxdepth: 2 - :numbered: - - distutils2.index.client - distutils2.index.dist - distutils2.index.simple - distutils2.index.xmlrpc diff --git a/docs/source/library/distutils2.index.simple.rst b/docs/source/library/distutils2.index.simple.rst deleted file mode 100644 --- a/docs/source/library/distutils2.index.simple.rst +++ /dev/null @@ -1,143 +0,0 @@ -========================================= -Querying indexes via the simple index API -========================================= - -`distutils2.index.simple` can process Python Package Indexes, and provides -useful informations about distributions. It also can crawl local indexes, for -instance. - -You should use `distutils2.index.simple` for: - - * Search distributions by name and versions. - * Process index external pages. - * Download distributions by name and versions. - -And should not be used to: - - * Things that will end up in too long index processing (like "finding all - distributions with a specific version, no matters the name") - -API ---- - -.. autoclass:: distutils2.index.simple.Crawler - :members: - - -Usage Exemples ---------------- - -To help you understand how using the `Crawler` class, here are some basic -usages. - -Request the simple index to get a specific distribution -++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -Supposing you want to scan an index to get a list of distributions for -the "foobar" project. You can use the "get_releases" method for that. -The get_releases method will browse the project page, and return :class:`ReleaseInfo` -objects for each found link that rely on downloads. :: - - >>> from distutils2.index.simple import Crawler - >>> crawler = Crawler() - >>> crawler.get_releases("FooBar") - [, ] - -Note that you also can request the client about specific versions, using version -specifiers (described in `PEP 345 -`_):: - - >>> client.get_releases("FooBar < 1.2") - [, ] - -`get_releases` returns a list of :class:`ReleaseInfo`, but you also can get the best -distribution that fullfil your requirements, using "get_release":: - - >>> client.get_release("FooBar < 1.2") - - -Download distributions -+++++++++++++++++++++++ - -As it can get the urls of distributions provided by PyPI, the `Crawler` -client also can download the distributions and put it for you in a temporary -destination:: - - >>> client.download("foobar") - /tmp/temp_dir/foobar-1.2.tar.gz - -You also can specify the directory you want to download to:: - - >>> client.download("foobar", "/path/to/my/dir") - /path/to/my/dir/foobar-1.2.tar.gz - -While downloading, the md5 of the archive will be checked, if not matches, it -will try another time, then if fails again, raise `MD5HashDoesNotMatchError`. - -Internally, that's not the Crawler which download the distributions, but the -`DistributionInfo` class. Please refer to this documentation for more details. - -Following PyPI external links -++++++++++++++++++++++++++++++ - -The default behavior for distutils2 is to *not* follow the links provided -by HTML pages in the "simple index", to find distributions related -downloads. - -It's possible to tell the PyPIClient to follow external links by setting the -`follow_externals` attribute, on instantiation or after:: - - >>> client = Crawler(follow_externals=True) - -or :: - - >>> client = Crawler() - >>> client.follow_externals = True - -Working with external indexes, and mirrors -+++++++++++++++++++++++++++++++++++++++++++ - -The default `Crawler` behavior is to rely on the Python Package index stored -on PyPI (http://pypi.python.org/simple). - -As you can need to work with a local index, or private indexes, you can specify -it using the index_url parameter:: - - >>> client = Crawler(index_url="file://filesystem/path/") - -or :: - - >>> client = Crawler(index_url="http://some.specific.url/") - -You also can specify mirrors to fallback on in case the first index_url you -provided doesnt respond, or not correctly. The default behavior for -`Crawler` is to use the list provided by Python.org DNS records, as -described in the :PEP:`381` about mirroring infrastructure. - -If you don't want to rely on these, you could specify the list of mirrors you -want to try by specifying the `mirrors` attribute. It's a simple iterable:: - - >>> mirrors = ["http://first.mirror","http://second.mirror"] - >>> client = Crawler(mirrors=mirrors) - -Searching in the simple index -+++++++++++++++++++++++++++++ - -It's possible to search for projects with specific names in the package index. -Assuming you want to find all projects containing the "distutils" keyword:: - - >>> c.search_projects("distutils") - [, , , , , , ] - -You can also search the projects starting with a specific text, or ending with -that text, using a wildcard.:: - - >>> c.search_projects("distutils*") - [, , ] - - >>> c.search_projects("*distutils") - [, , , , ] diff --git a/docs/source/library/distutils2.index.xmlrpc.rst b/docs/source/library/distutils2.index.xmlrpc.rst deleted file mode 100644 --- a/docs/source/library/distutils2.index.xmlrpc.rst +++ /dev/null @@ -1,128 +0,0 @@ -========================= -Query indexes via XML-RPC -========================= - -Indexes can be queried using XML-RPC calls, and Distutils2 provides a simple -way to interface with XML-RPC. - -You should **use** XML-RPC when: - - * Searching the index for projects **on other fields than project - names**. For instance, you can search for projects based on the - author_email field. - * Searching all the versions that have existed for a project. - * you want to retrive METADATAs informations from releases or - distributions. - -You should **avoid using** XML-RPC method calls when: - - * Retrieving the last version of a project - * Getting the projects with a specific name and version. - * The simple index can match your needs - -When dealing with indexes, keep in mind that the index queriers will always -return you :class:`distutils2.index.ReleaseInfo` and -:class:`distutils2.index.ReleasesList` objects. - -Some methods here share common APIs with the one you can find on -:class:`distutils2.index.simple`, internally, :class:`distutils2.index.client` -is inherited by :class:`distutils2.index.xmlrpc.Client` - -API -==== - -.. autoclass:: distutils2.index.xmlrpc.Client - :members: - -Usage examples -=============== - -Use case described here are use case that are not common to the other clients. -If you want to see all the methods, please refer to API or to usage examples -described in :class:`distutils2.index.client.Client` - -Finding releases ----------------- - -It's a common use case to search for "things" within the index. -We can basically search for projects by their name, which is the -most used way for users (eg. "give me the last version of the FooBar project"). -This can be accomplished using the following syntax:: - - >>> client = xmlrpc.Client() - >>> client.get_release("Foobar (<= 1.3)) - - >>> client.get_releases("FooBar (<= 1.3)") - [FooBar 1.1, FooBar 1.1.1, FooBar 1.2, FooBar 1.2.1] - -And we also can find for specific fields:: - - >>> client.search_projects(field=value) - -You could specify the operator to use, default is "or":: - - >>> client.search_projects(field=value, operator="and") - -The specific fields you can search are: - - * name - * version - * author - * author_email - * maintainer - * maintainer_email - * home_page - * license - * summary - * description - * keywords - * platform - * download_url - -Getting metadata informations ------------------------------ - -XML-RPC is a prefered way to retrieve metadata informations from indexes. -It's really simple to do so:: - - >>> client = xmlrpc.Client() - >>> client.get_metadata("FooBar", "1.1") - - -Assuming we already have a :class:`distutils2.index.ReleaseInfo` object defined, -it's possible to pass it ot the xmlrpc client to retrieve and complete its -metadata:: - - >>> foobar11 = ReleaseInfo("FooBar", "1.1") - >>> client = xmlrpc.Client() - >>> returned_release = client.get_metadata(release=foobar11) - >>> returned_release - - -Get all the releases of a project ---------------------------------- - -To retrieve all the releases for a project, you can build them using -`get_releases`:: - - >>> client = xmlrpc.Client() - >>> client.get_releases("FooBar") - [, , ] - -Get informations about distributions ------------------------------------- - -Indexes have informations about projects, releases **and** distributions. -If you're not familiar with those, please refer to the documentation of -:mod:`distutils2.index.dist`. - -It's possible to retrive informations about distributions, e.g "what are the -existing distributions for this release ? How to retrieve them ?":: - - >>> client = xmlrpc.Client() - >>> release = client.get_distributions("FooBar", "1.1") - >>> release.dists - {'sdist': , 'bdist': } - -As you see, this does not return a list of distributions, but a release, -because a release can be used like a list of distributions. diff --git a/docs/source/library/distutils2.install.rst b/docs/source/library/distutils2.install.rst deleted file mode 100644 --- a/docs/source/library/distutils2.install.rst +++ /dev/null @@ -1,25 +0,0 @@ -================== -Installation tools -================== - -In addition to the install commands, distutils2 provides a set of tools to deal -with installation of distributions. - -Basically, they're intended to download the distribution from indexes, to -resolve the dependencies, and to provide a safe way to install all the -distributions. - -You can find those tools in :module distutils2.install_tools:. - - -API ---- - -.. automodule:: distutils2.install - :members: - -Example usage --------------- - -Get the scheme of what's gonna be installed if we install "foobar": - diff --git a/docs/source/library/distutils2.metadata.rst b/docs/source/library/distutils2.metadata.rst deleted file mode 100644 --- a/docs/source/library/distutils2.metadata.rst +++ /dev/null @@ -1,93 +0,0 @@ -======== -Metadata -======== - -.. module:: distutils2.metadata - -Distutils2 provides a :class:`~distutils2.metadata.Metadata` class that can read and -write metadata files. This class is compatible with all metadata versions: - -* 1.0: :PEP:`241` -* 1.1: :PEP:`314` -* 1.2: :PEP:`345` - -The :PEP:`345` implementation supports the micro-language for the environment -markers, and displays warnings when versions that are supposed to be -:PEP:`386` are violating the scheme. - - -Reading metadata -================ - -The :class:`~distutils2.metadata.Metadata` class can be instantiated with the path of -the metadata file, and provides a dict-like interface to the values:: - - >>> from distutils2.metadata import Metadata - >>> metadata = Metadata('PKG-INFO') - >>> metadata.keys()[:5] - ('Metadata-Version', 'Name', 'Version', 'Platform', 'Supported-Platform') - >>> metadata['Name'] - 'CLVault' - >>> metadata['Version'] - '0.5' - >>> metadata['Requires-Dist'] - ["pywin32; sys.platform == 'win32'", "Sphinx"] - -The fields that supports environment markers can be automatically ignored if -the object is instantiated using the ``platform_dependent`` option. -:class:`~distutils2.metadata.Metadata` will interpret in the case the markers and will -automatically remove the fields that are not compliant with the running -environment. Here's an example under Mac OS X. The win32 dependency -we saw earlier is ignored:: - - >>> from distutils2.metadata import Metadata - >>> metadata = Metadata('PKG-INFO', platform_dependent=True) - >>> metadata['Requires-Dist'] - ['bar'] - -If you want to provide your own execution context, let's say to test the -metadata under a particular environment that is not the current environment, -you can provide your own values in the ``execution_context`` option, which -is the dict that may contain one or more keys of the context the micro-language -expects. - -Here's an example, simulating a win32 environment:: - - >>> from distutils2.metadata import Metadata - >>> context = {'sys.platform': 'win32'} - >>> metadata = Metadata('PKG-INFO', platform_dependent=True, - ... execution_context=context) - ... - >>> metadata['Requires-Dist'] = ["pywin32; sys.platform == 'win32'", - ... "Sphinx"] - ... - >>> metadata['Requires-Dist'] - ['pywin32', 'Sphinx'] - - -Writing metadata -================ - -Writing metadata can be done using the ``write`` API:: - - >>> metadata.write('/to/my/PKG-INFO') - -The class will pick the best version for the metadata, depending on the values -provided. If all the values provided exist in all versions, the class will -use :attr:`metadata.PKG_INFO_PREFERRED_VERSION`. It is set by default to 1.0. - - -Conflict checking and best version -================================== - -Some fields in :PEP:`345` have to follow a version scheme in their versions -predicate. When the scheme is violated, a warning is emitted:: - - >>> from distutils2.metadata import Metadata - >>> metadata = Metadata() - >>> metadata['Requires-Dist'] = ['Funky (Groovie)'] - "Funky (Groovie)" is not a valid predicate - >>> metadata['Requires-Dist'] = ['Funky (1.2)'] - - -.. TODO talk about check() diff --git a/docs/source/library/distutils2.rst b/docs/source/library/distutils2.rst deleted file mode 100644 --- a/docs/source/library/distutils2.rst +++ /dev/null @@ -1,44 +0,0 @@ -:mod:`distutils2` --- Packaging support -======================================= - -.. module:: distutils2 - :synopsis: Packaging system and building blocks for other packaging systems -.. sectionauthor:: Distutils2 contributors - - -The :mod:`distutils2` package provides support for building, packaging, -distributing and installing additional projects into a Python installation. -Projects may be include modules, extension modules, packages and scripts. -:mod:`distutils2` also provides building blocks for other packaging systems -that do not use the command system. - -This manual is the reference documentation for those standalone building -blocks and for extending Distutils2. If you're looking for the user-centric -guides to install a project or package your own code, head to `See also`__. - - -.. toctree:: - :maxdepth: 2 - :numbered: - - distutils2.version - distutils2.metadata - distutils2.depgraph - distutils2.install - distutils2.index - distutils2.tests.pypi_server - - -.. __: - -.. seealso:: - - :doc:`../distutils/index` - The manual for developers of Python projects who want to package and - distribute them. This describes how to use :mod:`distutils2` to make - projects easily found and added to an existing Python installation. - - :doc:`../install/index` - A user-centered manual which includes information on adding projects - into an existing Python installation. You do not need to be a Python - programmer to read this manual. diff --git a/docs/source/library/distutils2.tests.pypi_server.rst b/docs/source/library/distutils2.tests.pypi_server.rst deleted file mode 100644 --- a/docs/source/library/distutils2.tests.pypi_server.rst +++ /dev/null @@ -1,89 +0,0 @@ -============== -Test Framework -============== - -When you are testing code that works with distutils, you might find these tools -useful. - -``PyPIServer`` -============== - -PyPIServer is a class that implements an HTTP server running in a separate -thread. All it does is record the requests for further inspection. The recorded -data is available under ``requests`` attribute. The default -HTTP response can be overriden with the ``default_response_status``, -``default_response_headers`` and ``default_response_data`` attributes. - -By default, when accessing the server with urls beginning with `/simple/`, -the server also record your requests, but will look for files under -the `/tests/pypiserver/simple/` path. - -You can tell the sever to serve static files for other paths. This could be -accomplished by using the `static_uri_paths` parameter, as below:: - - server = PyPIServer(static_uri_paths=["first_path", "second_path"]) - -You need to create the content that will be served under the -`/tests/pypiserver/default` path. If you want to serve content from another -place, you also can specify another filesystem path (wich need to be under -`tests/pypiserver/`. This will replace the default behavior of the server, and -it will not serve content from the `default` dir :: - - server = PyPIServer(static_filesystem_paths=["path/to/your/dir"]) - -If you just need to add some paths to the existing ones, you can do as shown, -keeping in mind that the server will alwas try to load paths in reverse order -(e.g here, try "another/super/path" then the default one) :: - - server = PyPIServer(test_static_path="another/super/path") - server = PyPIServer("another/super/path") - # or - server.static_filesystem_paths.append("another/super/path") - -As a result of what, in your tests, while you need to use the PyPIServer, in -order to isolates the test cases, the best practice is to place the common files -in the `default` folder, and to create a directory for each specific test case:: - - server = PyPIServer(static_filesystem_paths = ["default", "test_pypi_server"], - static_uri_paths=["simple", "external"]) - -``PyPIServerTestCase`` -====================== - -``PyPIServerTestCase`` is a test case class with setUp and tearDown methods that -take care of a single PyPIServer instance attached as a ``pypi`` attribute on -the test class. Use it as one of the base classes in your test case:: - - class UploadTestCase(PyPIServerTestCase): - def test_something(self): - cmd = self.prepare_command() - cmd.ensure_finalized() - cmd.repository = self.pypi.full_address - cmd.run() - - environ, request_data = self.pypi.requests[-1] - self.assertEqual(request_data, EXPECTED_REQUEST_DATA) - -The ``use_pypi_server`` decorator -================================= - -You also can use a decorator for your tests, if you do not need the same server -instance along all you test case. So, you can specify, for each test method, -some initialisation parameters for the server. - -For this, you need to add a `server` parameter to your method, like this:: - - class SampleTestCase(TestCase): - @use_pypi_server() - def test_somthing(self, server): - # your tests goes here - ... - -The decorator will instantiate the server for you, and run and stop it just -before and after your method call. You also can pass the server initializer, -just like this:: - - class SampleTestCase(TestCase): - @use_pypi_server("test_case_name") - def test_something(self, server): - ... diff --git a/docs/source/library/distutils2.version.rst b/docs/source/library/distutils2.version.rst deleted file mode 100644 --- a/docs/source/library/distutils2.version.rst +++ /dev/null @@ -1,70 +0,0 @@ -====================== -Working with versions -====================== - -Distutils2 ships with a python package capable to work with version numbers. -It's an implementation of version specifiers `as defined in PEP 345 -`_ about -Metadatas. - -NormalizedVersion -================= - -A Normalized version is a specific version of a distribution, as -described in the PEP 345. So, you can work with the `NormalizedVersion` like -this:: - - >>> NormalizedVersion("1.2b1") - NormalizedVersion('1.2b1') - -If you try to use irrational version specifiers, an `IrrationalVersionError` -will be raised:: - - >>> NormalizedVersion("irrational_version_number") - ... - IrrationalVersionError: irrational_version_number - -You can compare NormalizedVersion objects, like this:: - - >>> NormalizedVersion("1.2b1") < NormalizedVersion("1.2") - True - -NormalizedVersion is used internally by `VersionPredicate` to do his stuff. - -Suggest a normalized version ----------------------------- - -Before this version scheme, there was existing others ones. Distutils2 provides -a tool that suggests a normalized version: the `suggest_normalized_version` -function:: - - >>> suggest_normalized_version('2.1-rc1') - 2.1c1 - -If `suggest_normalized_version` can't actually suggest you a version, it will -return `None`:: - - >>> print suggest_normalized_version('not a version') - None - -Dealing with version predicates -=============================== - -`VersionPredicate` knows how to parse stuff like "ProjectName (>=version)", the -class also provides a `match` method to test if a version number is the version -predicate:: - - >>> version = VersionPredicate("ProjectName (<1.2,>1.0") - >>> version.match("1.2.1") - False - >>> version.match("1.1.1") - True - -You could test if a predicate is a valid one, usng the `is_valid_predicate` -function:: - - >>> is_valid_predicate('FooBar (1.1)') - True - >>> is_valid_predicate('FooBar (+1.1)') - False - diff --git a/docs/source/library/pkgutil.rst b/docs/source/library/pkgutil.rst deleted file mode 100644 --- a/docs/source/library/pkgutil.rst +++ /dev/null @@ -1,333 +0,0 @@ -:mod:`pkgutil` --- Package utilities -==================================== - -.. module:: pkgutil - :synopsis: Utilities to support packages. - -This module provides utilities to manipulate packages: support for the -Importer protocol defined in :PEP:`302` and implementation of the API -described in :PEP:`376` to work with the database of installed Python -distributions. - -Import system utilities ------------------------ - -.. function:: extend_path(path, name) - - Extend the search path for the modules which comprise a package. Intended - use is to place the following code in a package's :file:`__init__.py`:: - - from pkgutil import extend_path - __path__ = extend_path(__path__, __name__) - - This will add to the package's ``__path__`` all subdirectories of directories - on :data:`sys.path` named after the package. This is useful if one wants to - distribute different parts of a single logical package as multiple - directories. - - It also looks for :file:`\*.pkg` files beginning where ``*`` matches the - *name* argument. This feature is similar to :file:`\*.pth` files (see the - :mod:`site` module for more information), except that it doesn't special-case - lines starting with ``import``. A :file:`\*.pkg` file is trusted at face - value: apart from checking for duplicates, all entries found in a - :file:`\*.pkg` file are added to the path, regardless of whether they exist - on the filesystem. (This is a feature.) - - If the input path is not a list (as is the case for frozen packages) it is - returned unchanged. The input path is not modified; an extended copy is - returned. Items are only appended to the copy at the end. - - It is assumed that :data:`sys.path` is a sequence. Items of :data:`sys.path` - that are not strings referring to existing directories are ignored. Unicode - items on :data:`sys.path` that cause errors when used as filenames may cause - this function to raise an exception (in line with :func:`os.path.isdir` - behavior). - - -.. class:: ImpImporter(dirname=None) - - :pep:`302` Importer that wraps Python's "classic" import algorithm. - - If *dirname* is a string, a :pep:`302` importer is created that searches that - directory. If *dirname* is ``None``, a :pep:`302` importer is created that - searches the current :data:`sys.path`, plus any modules that are frozen or - built-in. - - Note that :class:`ImpImporter` does not currently support being used by - placement on :data:`sys.meta_path`. - - -.. class:: ImpLoader(fullname, file, filename, etc) - - :pep:`302` Loader that wraps Python's "classic" import algorithm. - - -.. function:: find_loader(fullname) - - Find a :pep:`302` "loader" object for *fullname*. - - If *fullname* contains dots, path must be the containing package's - ``__path__``. Returns ``None`` if the module cannot be found or imported. - This function uses :func:`iter_importers`, and is thus subject to the same - limitations regarding platform-specific special import locations such as the - Windows registry. - - -.. function:: get_importer(path_item) - - Retrieve a :pep:`302` importer for the given *path_item*. - - The returned importer is cached in :data:`sys.path_importer_cache` if it was - newly created by a path hook. - - If there is no importer, a wrapper around the basic import machinery is - returned. This wrapper is never inserted into the importer cache (None is - inserted instead). - - The cache (or part of it) can be cleared manually if a rescan of - :data:`sys.path_hooks` is necessary. - - -.. function:: get_loader(module_or_name) - - Get a :pep:`302` "loader" object for *module_or_name*. - - If the module or package is accessible via the normal import mechanism, a - wrapper around the relevant part of that machinery is returned. Returns - ``None`` if the module cannot be found or imported. If the named module is - not already imported, its containing package (if any) is imported, in order - to establish the package ``__path__``. - - This function uses :func:`iter_importers`, and is thus subject to the same - limitations regarding platform-specific special import locations such as the - Windows registry. - - -.. function:: iter_importers(fullname='') - - Yield :pep:`302` importers for the given module name. - - If fullname contains a '.', the importers will be for the package containing - fullname, otherwise they will be importers for :data:`sys.meta_path`, - :data:`sys.path`, and Python's "classic" import machinery, in that order. If - the named module is in a package, that package is imported as a side effect - of invoking this function. - - Non-:pep:`302` mechanisms (e.g. the Windows registry) used by the standard - import machinery to find files in alternative locations are partially - supported, but are searched *after* :data:`sys.path`. Normally, these - locations are searched *before* :data:`sys.path`, preventing :data:`sys.path` - entries from shadowing them. - - For this to cause a visible difference in behaviour, there must be a module - or package name that is accessible via both :data:`sys.path` and one of the - non-:pep:`302` file system mechanisms. In this case, the emulation will find - the former version, while the builtin import mechanism will find the latter. - - Items of the following types can be affected by this discrepancy: - ``imp.C_EXTENSION``, ``imp.PY_SOURCE``, ``imp.PY_COMPILED``, - ``imp.PKG_DIRECTORY``. - - -.. function:: iter_modules(path=None, prefix='') - - Yields ``(module_loader, name, ispkg)`` for all submodules on *path*, or, if - path is ``None``, all top-level modules on :data:`sys.path`. - - *path* should be either ``None`` or a list of paths to look for modules in. - - *prefix* is a string to output on the front of every module name on output. - - -.. function:: walk_packages(path=None, prefix='', onerror=None) - - Yields ``(module_loader, name, ispkg)`` for all modules recursively on - *path*, or, if path is ``None``, all accessible modules. - - *path* should be either ``None`` or a list of paths to look for modules in. - - *prefix* is a string to output on the front of every module name on output. - - Note that this function must import all *packages* (*not* all modules!) on - the given *path*, in order to access the ``__path__`` attribute to find - submodules. - - *onerror* is a function which gets called with one argument (the name of the - package which was being imported) if any exception occurs while trying to - import a package. If no *onerror* function is supplied, :exc:`ImportError`\s - are caught and ignored, while all other exceptions are propagated, - terminating the search. - - Examples:: - - # list all modules python can access - walk_packages() - - # list all submodules of ctypes - walk_packages(ctypes.__path__, ctypes.__name__ + '.') - - -.. function:: get_data(package, resource) - - Get a resource from a package. - - This is a wrapper for the :pep:`302` loader :func:`get_data` API. The - *package* argument should be the name of a package, in standard module format - (``foo.bar``). The *resource* argument should be in the form of a relative - filename, using ``/`` as the path separator. The parent directory name - ``..`` is not allowed, and nor is a rooted name (starting with a ``/``). - - The function returns a binary string that is the contents of the specified - resource. - - For packages located in the filesystem, which have already been imported, - this is the rough equivalent of:: - - d = os.path.dirname(sys.modules[package].__file__) - data = open(os.path.join(d, resource), 'rb').read() - - If the package cannot be located or loaded, or it uses a :pep:`302` loader - which does not support :func:`get_data`, then ``None`` is returned. - - -Installed distributions database --------------------------------- - -Installed Python distributions are represented by instances of -:class:`~distutils2._backport.pkgutil.Distribution`, or its subclass -:class:`~distutils2._backport.pkgutil.EggInfoDistribution` for legacy ``.egg`` -and ``.egg-info`` formats). Most functions also provide an extra argument -``use_egg_info`` to take legacy distributions into account. - -.. TODO write docs here, don't rely on automodule - classes: Distribution and descendents - functions: provides, obsoletes, replaces, etc. - -Caching -+++++++ - -For performance purposes, the list of distributions is being internally -cached. It is enabled by default, but you can turn it off or clear -it using :func:`~distutils2._backport.pkgutil.enable_cache`, -:func:`~distutils2._backport.pkgutil.disable_cache` and -:func:`~distutils2._backport.pkgutil.clear_cache`. - - -Examples --------- - -Print all information about a distribution -++++++++++++++++++++++++++++++++++++++++++ - -Given a path to a ``.dist-info`` distribution, we shall print out all -information that can be obtained using functions provided in this module:: - - from distutils2._backport import pkgutil - import sys - - path = raw_input() # read the path from the keyboard - # first create the Distribution instance - try: - dist = pkgutil.Distribution(path) - except IOError: - print('No such distribution') - sys.exit(1) - - print('Information about %s' % dist.name) - print('Files') - print('=====') - for (path, md5, size) in dist.get_installed_files(): - print('* Path: %s' % path) - print(' Hash %s, Size: %s bytes' % (md5, size)) - print('Metadata') - print('========') - for key, value in dist.metadata.items(): - print('%20s: %s' % (key, value)) - print('Extra') - print('=====') - if dist.requested: - print('* It was installed by user request') - else: - print('* It was installed as a dependency') - -If we save the script above as ``print_info.py`` and we are intested in the -distribution located at -``/home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9`` -then by typing in the console: - -.. code-block:: bash - - $ echo /home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info | python print_info.py - -we get the following output: - -.. code-block:: none - - Information about choxie - Files - ===== - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/truffles.py - Hash 5e052db6a478d06bad9ae033e6bc08af, Size: 111 bytes - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/chocolate.py - Hash ac56bf496d8d1d26f866235b95f31030, Size: 214 bytes - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9/choxie/__init__.py - Hash 416aab08dfa846f473129e89a7625bbc, Size: 25 bytes - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/INSTALLER - Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/METADATA - Hash 696a209967fef3c8b8f5a7bb10386385, Size: 225 bytes - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/REQUESTED - Hash d41d8cd98f00b204e9800998ecf8427e, Size: 0 bytes - * Path: ../home/josip/dev/distutils2/src/distutils2/_backport/tests/fake_dists/choxie-2.0.0.9.dist-info/RECORD - Hash None, Size: None bytes - Metadata - ======== - Metadata-Version: 1.2 - Name: choxie - Version: 2.0.0.9 - Platform: [] - Supported-Platform: UNKNOWN - Summary: Chocolate with a kick! - Description: UNKNOWN - Keywords: [] - Home-page: UNKNOWN - Author: UNKNOWN - Author-email: UNKNOWN - Maintainer: UNKNOWN - Maintainer-email: UNKNOWN - License: UNKNOWN - Classifier: [] - Download-URL: UNKNOWN - Obsoletes-Dist: ['truffles (<=0.8,>=0.5)', 'truffles (<=0.9,>=0.6)'] - Project-URL: [] - Provides-Dist: ['truffles (1.0)'] - Requires-Dist: ['towel-stuff (0.1)'] - Requires-Python: UNKNOWN - Requires-External: [] - Extra - ===== - * It was installed as a dependency - -Find out obsoleted distributions -++++++++++++++++++++++++++++++++ - -Now, we take tackle a different problem, we are interested in finding out -which distributions have been obsoleted. This can be easily done as follows:: - - from distutils2._backport import pkgutil - - # iterate over all distributions in the system - for dist in pkgutil.get_distributions(): - name = dist.name - version = dist.metadata['Version'] - # find out which distributions obsolete this name/version combination - for obsoleted_by in pkgutil.obsoletes_distribution(name, version): - print('%s(%s) is obsoleted by %s' % (name, version, obsoleted_by.name)) - -This is how the output might look like: - -.. code-block:: none - - strawberry(0.6) is obsoleted by choxie - grammar(1.0a4) is obsoleted by towel-stuff - diff --git a/docs/source/setupcfg-spec.rst b/docs/source/setupcfg-spec.rst deleted file mode 100644 --- a/docs/source/setupcfg-spec.rst +++ /dev/null @@ -1,308 +0,0 @@ -=================== -setup.cfg file v0.9 -=================== - -This document describes the :file:`setup.cfg`, a ini-like file used by -Distutils2 to replace the :file:`setup.py` file. - -Syntax -====== - -The configuration file is an ini-based file. Variables name can be -assigned values, and grouped into sections. A line that starts with "#" is -commented out. Empty lines are also removed. - -Example:: - - [section1] - # comment - name = value - name2 = "other value" - - [section2] - foo = bar - - -Values conversion -::::::::::::::::: - -Here are a set of rules for converting values: - -- If value is quoted with " chars, it's a string. This notation is useful to - include "=" characters in the value. In case the value contains a " - character, it must be escaped with a "\" character. -- If the value is "true" or "false" --no matter what the case is--, it's - converted to a boolean, or 0 and 1 when the language does not have a - boolean type. -- A value can contains multiple lines. When read, lines are converted into a - sequence of values. Each new line for a multiple lines value must start with - a least one space or tab character. These indentation characters will be - stripped. -- all other values are considered as strings - -Examples:: - - [section] - foo = one - two - three - - bar = false - baz = 1.3 - boo = "ok" - beee = "wqdqw pojpj w\"ddq" - -Extending files -::::::::::::::: - -An INI file can extend another file. For this, a "DEFAULT" section must contain -an "extends" variable that can point to one or several INI files which will be -merged to the current file by adding new sections and values. - -If the file pointed in "extends" contains section/variable names that already -exist in the original file, they will not override existing ones. - -file_one.ini:: - - [section1] - name2 = "other value" - - [section2] - foo = baz - bas = bar - -file_two.ini:: - - [DEFAULT] - extends = file_one.ini - - [section2] - foo = bar - -Result:: - - [section1] - name2 = "other value" - - [section2] - foo = bar - bas = bar - -To point several files, the multi-line notation can be used: - - [DEFAULT] - extends = file_one.ini - file_two.ini - -When several files are provided, they are processed sequentially. So if the -first one has a value that is also present in the second, the second one will -be ignored. This means that the configuration goes from the most specialized to -the most common. - -**Tools will need to provide a way to produce a canonical version of the -file**. This will be useful to publish a single file. - - -Description of sections and fields -================================== - -Each section contains a description of its options. - -- Options that are marked *\*multi* can have multiple values, one value per - line. -- Options that are marked *\*optional* can be omited. - - -The sections are: - -global - Global options for Distutils2. - -metadata - The metadata section contains the metadata for the project as described in - :PEP:`345`. - -files - Declaration of package files included in the project. - -`command` sections - Redefinition of user options for Distutils2 commands. - - -global -:::::: - -Contains global options for Distutils2. This section is shared with Distutils1 -(legacy version distributed in python 2.X standard library). - - -- **commands**: Defined Distutils2 command. A command is defined by its fully - qualified name. - - Examples:: - - [global] - commands = - package.sdist.CustomSdistCommand - - *\*optional* *\*multi* - -- **compilers**: Defined Distutils2 compiler. A compiler is defined by its fully - qualified name. - - Example:: - - [global] - compiler = - package.compiler.CustomCCompiler - - *\*optional* *\*multi* - -- **setup_hook**: defines a callable that will be called right after the - :file:`setup.cfg` file is read. The callable receives the configuration - in form of a mapping and can make some changes to it. *\*optional* - - Example:: - - [global] - setup_hook = - distutils2.tests.test_config.hook - - -metadata -:::::::: - -The metadata section contains the metadata for the project as described in -:PEP:`345`. Field names are case-insensitive. - -- metadata-version -- name -- version -- platform -- supported-platform -- summary -- description -- keywords -- home-page -- download-url -- author -- author-email -- maintainer -- maintainer-email -- license -- classifiers -- requires-dist -- provides-dist -- obsoletes-dist -- requires-python -- requires-externals -- project-url - -There's one extra field, to replace description with a path to a text file: - -- description-file: path to a text file that will be used to replace description - - -files -::::: - -This section describes the files included in the project. - -- **packages_root**: the root directory containing all packages. If not provided - Distutils2 will use the current directory. *\*optional* -- **packages**: a list of packages the project includes *\*optional* *\*multi* -- **modules**: a list of packages the project includes *\*optional* *\*multi* -- **scripts**: a list of scripts the project includes *\*optional* *\*multi* -- **extra_files**: a list of patterns to include extra files *\*optional* *\*multi* -- **resources**: a list of data files. *\*optional* *\*multi* - -Example:: - - [files] - packages_root = src - packages = - pypi2rpm - pypi2rpm.command - - scripts = - pypi2rpm/pypi2rpm.py - - extra_files = - setup.py - README - - resources = - source1 = destination1 - source2 = destination2 - doc/* = {doc} - scripts/foo.sh = {datadir}/scripts/{distribution.name} - -extensions -:::::::::: - - -XXX - -Command sections -::::::::::::::::: - -Each Distutils2 command can have its own user options defined in :file:`setup.cfg` - -Example:: - - [sdist] - manifest-builders = package.module.Maker - - -To override the build class in order to generate Python3 code from your Python2 base:: - - [build_py] - use-2to3 = True - -Extensibility -============= - -Every section can define new variable that are not part of the specification. -They are called **extensions**. - -An extension field starts with *X-*. - -Example:: - - [metadata] - ... - X-Debian-Name = python-distribute - -Changes in the specification -============================ - -The version scheme for this specification is **MAJOR.MINOR**. -Changes in the specification will increment the version. - -- minor version changes (1.x): backwards compatible - - new fields and sections (both optional and mandatory) can be added - - optional fields can be removed -- major channges (2.X): backwards-incompatible - - mandatory fields/sections are removed - - fields change their meaning - -As a consequence, a tool written to consume 1.X (say, X=5) has these -properties: - -- reading 1.Y, YX is also possible. The tool will just ignore the new - fields (even if they are mandatory in that version) - If optional fields were removed, the tool will just consider them absent. -- reading 2.X is not possible; the tool should refuse to interpret - the file. - -A tool written to produce 1.X should have these properties: -- it will write all mandatory fields -- it may write optional fields - -Acks -==== - -XXX - diff --git a/docs/source/setupcfg.rst b/docs/source/setupcfg.rst deleted file mode 100644 --- a/docs/source/setupcfg.rst +++ /dev/null @@ -1,549 +0,0 @@ -================== -The setup.cfg file -================== - -This document describes the :file:`setup.cfg`, a ini-like file used by -Distutils2 to replace the :file:`setup.py` file. - -Each section contains a description of its options. - -- Options that are marked *\*multi* can have multiple values, one value per - line. -- Options that are marked *\*optional* can be omited. -- Options that are marked *\*environ* can use environment markers, as described - in :PEP:`345`. - - -The sections are: - -global - Global options for Distutils2. - -metadata - The metadata section contains the metadata for the project as described in - :PEP:`345`. - -files - Declaration of package files included in the project. - -`command` sections - Redefinition of user options for Distutils2 commands. - - -global -====== - -Contains global options for Distutils2. This section is shared with Distutils1 -(legacy version distributed in python 2.X standard library). - - -- **commands**: Defined Distutils2 command. A command is defined by its fully - qualified name. - - Examples:: - - [global] - commands = - package.sdist.CustomSdistCommand - - *\*optional* *\*multi* - -- **compilers**: Defined Distutils2 compiler. A compiler is defined by its fully - qualified name. - - Example:: - - [global] - compiler = - package.compiler.CustomCCompiler - - *\*optional* *\*multi* - -- **setup_hook**: defines a callable that will be called right after the - :file:`setup.cfg` file is read. The callable receives the configuration - in form of a mapping and can make some changes to it. *\*optional* - - Example:: - - [global] - setup_hook = - distutils2.tests.test_config.hook - - -metadata -======== - -The metadata section contains the metadata for the project as described in -:PEP:`345`. - -.. Note:: - Field names are case-insensitive. - -Fields: - -- **name**: Name of the project. -- **version**: Version of the project. Must comply with :PEP:`386`. -- **platform**: Platform specification describing an operating system supported - by the distribution which is not listed in the "Operating System" Trove - classifiers (:PEP:`301`). *\*multi* *\*optional* -- **supported-platform**: Binary distributions containing a PKG-INFO file will - use the Supported-Platform field in their metadata to specify the OS and - CPU for which the binary distribution was compiled. The semantics of - the Supported-Platform field are freeform. *\*multi* *\*optional* -- **summary**: A one-line summary of what the distribution does. - (Used to be called *description* in Distutils1.) -- **description**: A longer description. (Used to be called *long_description* - in Distutils1.) A file can be provided in the *description-file* field. - *\*optional* -- **description-file**: path to a text file that will be used for the - **description** field. *\*optional* -- **keywords**: A list of additional keywords to be used to assist searching - for the distribution in a larger catalog. Comma or space-separated. *\*optional* -- **home-page**: The URL for the distribution's home page. -- **download-url**: The URL from which this version of the distribution - can be downloaded. *\*optional* -- **author**: Author's name. *\*optional* -- **author-email**: Author's e-mail. *\*optional* -- **maintainer**: Maintainer's name. *\*optional* -- **maintainer-email**: Maintainer's e-mail. *\*optional* -- **license**: A text indicating the term of uses, when a trove classifier does - not match. *\*optional*. -- **classifiers**: Classification for the distribution, as described in PEP 301. - *\*optional* *\*multi* *\*environ* -- **requires-dist**: name of another distutils project required as a dependency. - The format is *name (version)* where version is an optional - version declaration, as described in PEP 345. *\*optional* *\*multi* *\*environ* -- **provides-dist**: name of another distutils project contained whithin this - distribution. Same format than *requires-dist*. *\*optional* *\*multi* *\*environ* -- **obsoletes-dist**: name of another distutils project this version obsoletes. - Same format than *requires-dist*. *\*optional* *\*multi* *\*environ* -- **requires-python**: Specifies the Python version the distribution requires. - The value is a version number, as described in PEP 345. - *\*optional* *\*multi* *\*environ* -- **requires-externals**: a dependency in the system. This field is free-form, - and just a hint for downstream maintainers. *\*optional* *\*multi* *\*environ* -- **project-url**: A label, followed by a browsable URL for the project. - "label, url". The label is limited to 32 signs. *\*optional* *\*multi* - - -Example:: - - [metadata] - name = pypi2rpm - version = 0.1 - author = Tarek Ziade - author-email = tarek at ziade.org - summary = Script that transforms a sdist archive into a rpm archive - description-file = README - home-page = http://bitbucket.org/tarek/pypi2rpm - project-url: RSS feed, https://bitbucket.org/tarek/pypi2rpm/rss - - classifier = Development Status :: 3 - Alpha - License :: OSI Approved :: Mozilla Public License 1.1 (MPL 1.1) - -.. Note:: - Some metadata fields seen in :PEP:`345` are automatically generated - (for instance Metadata-Version value). - - -files -===== - -This section describes the files included in the project. - -- **packages_root**: the root directory containing all packages. If not provided - Distutils2 will use the current directory. *\*optional* -- **packages**: a list of packages the project includes *\*optional* *\*multi* -- **modules**: a list of packages the project includes *\*optional* *\*multi* -- **scripts**: a list of scripts the project includes *\*optional* *\*multi* -- **extra_files**: a list of patterns to include extra files *\*optional* *\*multi* - -Example:: - - [files] - packages_root = src - packages = - pypi2rpm - pypi2rpm.command - - scripts = - pypi2rpm/pypi2rpm.py - - extra_files = - setup.py - README - -.. Note:: - In Distutils2, setup.cfg will be implicitly included. - -Resources -========= - -This section describes the files used by the project which must not be installed in the same place that python modules or libraries, they are called **resources**. They are for example documentation files, script files, databases, etc... - -For declaring resources, you must use this notation :: - - source = destination - -Data-files are declared in the **resources** field in the **file** section, for example:: - - [files] - resources = - source1 = destination1 - source2 = destination2 - -The **source** part of the declaration are relative paths of resources files (using unix path separator **/**). For example, if you've this source tree:: - - foo/ - doc/ - doc.man - scripts/ - foo.sh - -Your setup.cfg will look like:: - - [files] - resources = - doc/doc.man = destination_doc - scripts/foo.sh = destination_scripts - -The final paths where files will be placed are composed by : **source** + **destination**. In the previous example, **doc/doc.man** will be placed in **destination_doc/doc/doc.man** and **scripts/foo.sh** will be placed in **destination_scripts/scripts/foo.sh**. (If you want more control on the final path, take a look at base_prefix_). - -The **destination** part of resources declaration are paths with categories. Indeed, it's generally a bad idea to give absolute path as it will be cross incompatible. So, you must use resources categories in your **destination** declaration. Categories will be replaced by their real path at the installation time. Using categories is all benefit, your declaration will be simpler, cross platform and it will allow packager to place resources files where they want without breaking your code. - -Categories can be specified by using this syntax:: - - {category} - -Default categories are:: - -* config -* appdata -* appdata.arch -* appdata.persistent -* appdata.disposable -* help -* icon -* scripts -* doc -* info -* man - -A special category also exists **{distribution.name}** that will be replaced by the name of the distribution, but as most of the defaults categories use them, so it's not necessary to add **{distribution.name}** into your destination. - -If you use categories in your declarations, and you are encouraged to do, final path will be:: - - source + destination_expanded - -.. _example_final_path: - -For example, if you have this setup.cfg:: - - [metadata] - name = foo - - [files] - resources = - doc/doc.man = {doc} - -And if **{doc}** is replaced by **{datadir}/doc/{distribution.name}**, final path will be:: - - {datadir}/doc/foo/doc/doc.man - -Where {datafir} category will be platform-dependent. - - -More control on source part ---------------------------- - -Glob syntax -___________ - -When you declare source file, you can use a glob-like syntax to match multiples file, for example:: - - scripts/* = {script} - -Will match all the files in the scripts directory and placed them in the script category. - -Glob tokens are: - - * * : match all files. - * ? : match any character. - * ** : match any level of tree recursion (even 0). - * {} : will match any part separated by comma (example : {sh,bat}). - -TODO :: - - Add an example - -Order of declaration -____________________ - -The order of declaration is important if one file match multiple rules. The last rules matched by file is used, this is useful if you have this source tree:: - - foo/ - doc/ - index.rst - setup.rst - documentation.txt - doc.tex - README - -And you want all the files in the doc directory to be placed in {doc} category, but README must be placed in {help} category, instead of listing all the files one by one, you can declare them in this way:: - - [files] - resources = - doc/* = {doc} - doc/README = {help} - -Exclude -_______ - -You can exclude some files of resources declaration by giving no destination, it can be useful if you have a non-resources file in the same directory of resources files:: - - foo/ - doc/ - RELEASES - doc.tex - documentation.txt - docu.rst - -Your **file** section will be:: - - [files] - resources = - doc/* = {doc} - doc/RELEASES = - -More control on destination part --------------------------------- - -.. _base_prefix: - -Define a base-prefix -____________________ - -When you define your resources, you can have more control of how the final path is compute. - -By default, the final path is:: - - destination + source - -This can generate long paths, for example (example_final_path_):: - - {datadir}/doc/foo/doc/doc.man - -When you declare your source, you can use a separator to split the source in **prefix** **suffix**. The supported separator are : - - * Whitespace - -So, for example, if you have this source:: - - docs/ doc.man - -The **prefix** is "docs/" and the **suffix** is "doc.html". - -.. note:: - - Separator can be placed after a path separator or replace it. So theses two sources are equivalent:: - - docs/ doc.man - docs doc.man - -.. note:: - - Glob syntax is working the same way with standard source and splitted source. So theses rules:: - - docs/* - docs/ * - docs * - - Will match all the files in the docs directory. - -When you use splitted source, the final path is compute in this way:: - - destination + prefix - -So for example, if you have this setup.cfg:: - - [metadata] - name = foo - - [files] - resources = - doc/ doc.man = {doc} - -And if **{doc}** is replaced by **{datadir}/doc/{distribution.name}**, final path will be:: - - {datadir}/doc/foo/doc.man - - -Overwrite paths for categories ------------------------------- - -.. warning:: - - This part is intended for system administrator or packager. - -The real paths of categories are registered in the *sysconfig.cfg* file installed in your python installation. The format of this file is INI-like. The content of the file is organized into several sections : - - * globals : Standard categories's paths. - * posix_prefix : Standard paths for categories and installation paths for posix system. - * other one... - -Standard categories's paths are platform independent, they generally refers to other categories, which are platform dependent. Sysconfig module will choose these category from sections matching os.name. For example:: - - doc = {datadir}/doc/{distribution.name} - -It refers to datadir category, which can be different between platforms. In posix system, it may be:: - - datadir = /usr/share - -So the final path will be:: - - doc = /usr/share/doc/{distribution.name} - -The platform dependent categories are : - - * confdir - * datadir - * libdir - * base - -Define extra-categories ------------------------ - -Examples --------- - -.. note:: - - These examples are incremental but works unitarily. - -Resources in root dir -_____________________ - -Source tree:: - - babar-1.0/ - README - babar.sh - launch.sh - babar.py - -Setup.cfg:: - - [files] - resources = - README = {doc} - *.sh = {scripts} - -So babar.sh and launch.sh will be placed in {scripts} directory. - -Now let's move all the scripts into a scripts directory. - -Resources in sub-directory -__________________________ - -Source tree:: - - babar-1.1/ - README - scripts/ - babar.sh - launch.sh - LAUNCH - babar.py - -Setup.cfg:: - - [files] - resources = - README = {doc} - scripts/ LAUNCH = {doc} - scripts/ *.sh = {scripts} - -It's important to use the separator after scripts/ to install all the bash scripts into {scripts} instead of {scripts}/scripts. - -Now let's add some docs. - -Resources in multiple sub-directories -_____________________________________ - -Source tree:: - - babar-1.2/ - README - scripts/ - babar.sh - launch.sh - LAUNCH - docs/ - api - man - babar.py - -Setup.cfg:: - - [files] - resources = - README = {doc} - scripts/ LAUNCH = {doc} - scripts/ *.sh = {scripts} - doc/ * = {doc} - doc/ man = {man} - -You want to place all the file in the docs script into {doc} category, instead of man, which must be placed into {man} category, we will use the order of declaration of globs to choose the destination, the last glob that match the file is used. - -Now let's add some scripts for windows users. - -Complete example -________________ - -Source tree:: - - babar-1.3/ - README - doc/ - api - man - scripts/ - babar.sh - launch.sh - babar.bat - launch.bat - LAUNCH - -Setup.cfg:: - - [files] - resources = - README = {doc} - scripts/ LAUNCH = {doc} - scripts/ *.{sh,bat} = {scripts} - doc/ * = {doc} - doc/ man = {man} - -We use brace expansion syntax to place all the bash and batch scripts into {scripts} category. - -.. Warning:: - In Distutils2, setup.py and README (or README.txt) files are not more - included in source distribution by default - -`command` sections -================== - -Each Distutils2 command can have its own user options defined in :file:`setup.cfg` - -Example:: - - [sdist] - manifest-builders = package.module.Maker - - -To override the build class in order to generate Python3 code from your Python2 base:: - - [build_py] - use-2to3 = True - - diff --git a/docs/source/tutorial.rst b/docs/source/tutorial.rst deleted file mode 100644 --- a/docs/source/tutorial.rst +++ /dev/null @@ -1,129 +0,0 @@ -=================== -Distutils2 tutorial -=================== - -Welcome to the Distutils2 tutorial ! We will learn here how to use Distutils2 -to package your project. - - -Installing Distutils2 -===================== - -Quite simple, my dear reader:: - - $ pip install Distutils2 - -Or.. grab it at PyPI and run:: - - $ python setup.py install - - - -Getting started -=============== - -Distutils2 works with the *setup.cfg* file. It contains all the metadata for -your project, as defined in PEP 345, but also declare what your project -contains. - -Let's say you have a project called *CLVault* containing one package called -*clvault*, and a few scripts inside. You can use the *mkcfg* script to create -a *setup.cfg* file for the project. The script will ask you a few questions:: - - $ mkdir CLVault - $ cd CLVault - $ python -m distutils2.mkcfg - Project name [CLVault]: - Current version number: 0.1 - Package description: - >Command-line utility to store and retrieve passwords - Author name: Tarek Ziade - Author e-mail address: tarek at ziade.org - Project Home Page: http://bitbucket.org/tarek/clvault - Do you want to add a package ? (y/n): y - Package name: clvault - Do you want to add a package ? (y/n): n - Do you want to set Trove classifiers? (y/n): y - Please select the project status: - - 1 - Planning - 2 - Pre-Alpha - 3 - Alpha - 4 - Beta - 5 - Production/Stable - 6 - Mature - 7 - Inactive - - Status: 3 - What license do you use: GPL - Matching licenses: - - 1) License :: OSI Approved :: GNU General Public License (GPL) - 2) License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL) - - Type the number of the license you wish to use or ? to try again:: 1 - Do you want to set other trove identifiers (y/n) [n]: n - Wrote "setup.cfg". - - -A setup.cfg file is created, containing the metadata of your project and the -list of the packages it contains:: - - $ more setup.cfg - [metadata] - name = CLVault - version = 0.1 - author = Tarek Ziade - author_email = tarek at ziade.org - description = Command-line utility to store and retrieve passwords - home_page = http://bitbucket.org/tarek/clvault - - classifier = Development Status :: 3 - Alpha - License :: OSI Approved :: GNU General Public License (GPL) - - [files] - - packages = clvault - - -Our project will depend on the *keyring* project. Let's add it in the -[metadata] section:: - - [metadata] - ... - requires_dist = - keyring - - - -Running commands -================ - -You can run useful commands on your project once the setup.cfg file is ready: - -- sdist: creates a source distribution -- register: register your project to PyPI -- upload: upload the distribution to PyPI -- install: install it - -All commands are run using the run script:: - - $ python -m distutils2.run install - $ python -m distutils2.run sdist - $ python -m distutils2.run upload - -If you want to push a source distribution of your project at PyPI, do:: - - $ python -m distutils2.run sdist register upload - - -Installing the project -====================== - -People will have to manually run the distutils2 install command:: - - $ python -m distutils2.run install - - - - diff --git a/setup.cfg b/setup.cfg --- a/setup.cfg +++ b/setup.cfg @@ -1,9 +1,10 @@ [metadata] name = Distutils2 -version = 1.0a3 +version = 1.0a5.dev0 summary = Python Packaging Library description-file = README.txt -home-page = http://bitbucket.org/tarek/distutils2/wiki/Home +home-page = http://wiki.python.org/moin/Distutils2 +download-url = http://pypi.python.org/pypi/Distutils2 author = The Fellowship of the Packaging author-email = the-fellowship-of-the-packaging at googlegroups.com # we set a custom license field in addition to the classifier below @@ -16,6 +17,10 @@ License :: OSI Approved :: Python Software Foundation License Operating System :: OS Independent Programming Language :: Python + Programming Language :: Python :: 3 + Programming Language :: Python :: 3.1 + Programming Language :: Python :: 3.2 + Programming Language :: Python :: Implementation :: CPython Topic :: Software Development :: Libraries :: Python Modules Topic :: System :: Archiving :: Packaging Topic :: System :: Systems Administration @@ -37,14 +42,23 @@ distutils2.tests = xxmodule.c scripts = pysetup +extra_files = + README.txt + LICENSE.txt + CHANGES.txt + CONTRIBUTORS.txt + DEVNOTES.txt + setup.py + Makefile + runtests.py + check.sh + tests.sh + tox.ini + scan_pypi_versions.py -# TODO build hashlib for Python < 2.4 # TODO add all test data files # FIXME cfg_to_args should support comments in multi-line fields [build_ext] # needed so that tests work without mucking with sys.path inplace = 1 - -[upload_docs] -upload-dir = docs/build/html diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -15,7 +15,7 @@ def cfg_to_args(path='setup.cfg'): - opts_to_args = { + opts_to_args = { 'metadata': ( ('name', 'name', None), ('version', 'version', None), diff --git a/tox.ini b/tox.ini --- a/tox.ini +++ b/tox.ini @@ -1,6 +1,6 @@ [tox] #distshare={homedir}/.tox/distshare -envlist=py24,py25,py26,py26-s,py27 +envlist=py31,py32,py33 [tox:hudson] #distshare={toxworkdir}/distshare @@ -8,22 +8,10 @@ [testenv] commands= + # explicit argument to exclude _backport/tests; + # their coverage is not interesting nosetests --with-xunit distutils2/tests deps= - unittest2 + docutils + unittest2-py3k nose - -[testenv:py26-s] -distribute=False - -[testenv:py27] -basepython=python2.7 -commands= - nosetests --with-xunit distutils2/tests - -[testenv:py24] -basepython=python2.4 -commands= - rm -f distutils2/_backport/_hashlib.so - python setup.py build_ext -f - nosetests --with-xunit distutils2/tests -- Repository URL: http://hg.python.org/distutils2 From python-checkins at python.org Tue May 22 00:22:29 2012 From: python-checkins at python.org (ezio.melotti) Date: Tue, 22 May 2012 00:22:29 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Update_my_interests=2E?= Message-ID: http://hg.python.org/devguide/rev/df9a8035cca3 changeset: 518:df9a8035cca3 user: Ezio Melotti date: Mon May 21 16:22:09 2012 -0600 summary: Update my interests. files: experts.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -218,7 +218,7 @@ telnetlib tempfile georg.brandl, ncoghlan termios -test +test ezio.melotti textwrap georg.brandl threading pitrou time belopolsky @@ -303,7 +303,7 @@ context managers ncoghlan data formats mark.dickinson, georg.brandl database lemburg -devguide ncoghlan, eric.araujo +devguide ncoghlan, eric.araujo, ezio.melotti documentation georg.brandl, ezio.melotti, eric.araujo GUI i18n lemburg, eric.araujo @@ -324,5 +324,5 @@ threads pitrou time and dates lemburg, belopolsky unicode lemburg, ezio.melotti, haypo, benjamin.peterson -version control eric.araujo, georg.brandl +version control eric.araujo, georg.brandl, ezio.melotti ================== =========== -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Tue May 22 00:36:19 2012 From: python-checkins at python.org (antoine.pitrou) Date: Tue, 22 May 2012 00:36:19 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_interests?= Message-ID: http://hg.python.org/devguide/rev/b1c1d15271c0 changeset: 519:b1c1d15271c0 user: Antoine Pitrou date: Tue May 22 00:33:42 2012 +0200 summary: Add interests files: experts.rst | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-) diff --git a/experts.rst b/experts.rst --- a/experts.rst +++ b/experts.rst @@ -132,7 +132,7 @@ io pitrou, benjamin.peterson, stutzbach ipaddress pmoody, ncoghlan itertools rhettinger -json bob.ippolito (inactive), ezio.melotti, rhettinger +json bob.ippolito (inactive), ezio.melotti, rhettinger, pitrou keyword lib2to3 benjamin.peterson linecache @@ -197,7 +197,7 @@ smtpd giampaolo.rodola smtplib sndhdr -socket +socket pitrou socketserver spwd sqlite3 ghaering @@ -298,7 +298,7 @@ autoconf/makefiles bsd bug tracker ezio.melotti -buildbots +buildbots pitrou bytecode pitrou, georg.brandl context managers ncoghlan data formats mark.dickinson, georg.brandl @@ -312,7 +312,7 @@ locale lemburg, loewis mathematics mark.dickinson, eric.smith, lemburg, stutzbach memory management tim_one, lemburg -networking giampaolo.rodola +networking giampaolo.rodola, pitrou packaging tarek, lemburg, alexis, eric.araujo py3 transition benjamin.peterson release management tarek, lemburg, benjamin.peterson, barry, loewis, @@ -323,6 +323,6 @@ test coverage ncoghlan, giampaolo.rodola threads pitrou time and dates lemburg, belopolsky -unicode lemburg, ezio.melotti, haypo, benjamin.peterson +unicode lemburg, ezio.melotti, haypo, benjamin.peterson, pitrou version control eric.araujo, georg.brandl, ezio.melotti ================== =========== -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Tue May 22 01:54:05 2012 From: python-checkins at python.org (ezio.melotti) Date: Tue, 22 May 2012 01:54:05 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0ODc1OiBVc2Ug?= =?utf8?q?float=28=27inf=27=29_instead_of_float=28=271e66666=27=29_in_the_?= =?utf8?q?json_module=2E?= Message-ID: http://hg.python.org/cpython/rev/c13066f752a8 changeset: 77099:c13066f752a8 branch: 2.7 parent: 77087:4973c90ce9e6 user: Ezio Melotti date: Mon May 21 17:46:55 2012 -0600 summary: #14875: Use float('inf') instead of float('1e66666') in the json module. files: Lib/json/encoder.py | 3 +-- Misc/NEWS | 4 +++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/Lib/json/encoder.py b/Lib/json/encoder.py --- a/Lib/json/encoder.py +++ b/Lib/json/encoder.py @@ -27,8 +27,7 @@ ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i)) #ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,)) -# Assume this produces an infinity on all machines (probably not guaranteed) -INFINITY = float('1e66666') +INFINITY = float('inf') FLOAT_REPR = repr def encode_basestring(s): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,8 @@ Library ------- +- Issue #14875: Use float('inf') instead of float('1e66666') in the json module. + - Issue #14572: Prevent build failures with pre-3.5.0 versions of sqlite3, such as was shipped with Centos 5 and Mac OS X 10.4. @@ -119,7 +121,7 @@ - Issue #13684: Fix httplib tunnel issue of infinite loops for certain sites which send EOF without trailing \r\n. - + - Issue #14308: Fix an exception when a "dummy" thread is in the threading module's active list after a fork(). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 01:54:06 2012 From: python-checkins at python.org (ezio.melotti) Date: Tue, 22 May 2012 01:54:06 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0ODc1OiBVc2Ug?= =?utf8?q?float=28=27inf=27=29_instead_of_float=28=271e66666=27=29_in_the_?= =?utf8?q?json_module=2E?= Message-ID: http://hg.python.org/cpython/rev/06f21c6e0f2b changeset: 77100:06f21c6e0f2b branch: 3.2 parent: 77092:d13fdd97cc8e user: Ezio Melotti date: Mon May 21 17:49:06 2012 -0600 summary: #14875: Use float('inf') instead of float('1e66666') in the json module. files: Lib/json/encoder.py | 3 +-- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/json/encoder.py b/Lib/json/encoder.py --- a/Lib/json/encoder.py +++ b/Lib/json/encoder.py @@ -27,8 +27,7 @@ ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i)) #ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,)) -# Assume this produces an infinity on all machines (probably not guaranteed) -INFINITY = float('1e66666') +INFINITY = float('inf') FLOAT_REPR = repr def encode_basestring(s): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,8 @@ Library ------- +- Issue #14875: Use float('inf') instead of float('1e66666') in the json module. + - Issue #14426: Correct the Date format in Expires attribute of Set-Cookie Header in Cookie.py. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 01:54:07 2012 From: python-checkins at python.org (ezio.melotti) Date: Tue, 22 May 2012 01:54:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314875=3A_merge_with_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/f5f5db593d99 changeset: 77101:f5f5db593d99 parent: 77098:35c3946657ce parent: 77100:06f21c6e0f2b user: Ezio Melotti date: Mon May 21 17:53:42 2012 -0600 summary: #14875: merge with 3.2. files: Lib/json/encoder.py | 3 +-- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/json/encoder.py b/Lib/json/encoder.py --- a/Lib/json/encoder.py +++ b/Lib/json/encoder.py @@ -27,8 +27,7 @@ ESCAPE_DCT.setdefault(chr(i), '\\u{0:04x}'.format(i)) #ESCAPE_DCT.setdefault(chr(i), '\\u%04x' % (i,)) -# Assume this produces an infinity on all machines (probably not guaranteed) -INFINITY = float('1e66666') +INFINITY = float('inf') FLOAT_REPR = repr def encode_basestring(s): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,8 @@ Library ------- +- Issue #14875: Use float('inf') instead of float('1e66666') in the json module. + - Issue #13585: Added contextlib.ExitStack - PEP 3144, Issue #14814: Added the ipaddress module -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Tue May 22 05:51:01 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 22 May 2012 05:51:01 +0200 Subject: [Python-checkins] Daily reference leaks (f5f5db593d99): sum=3 Message-ID: results for f5f5db593d99 on branch "default" -------------------------------------------- test_dbm leaked [0, 2, 0] references, sum=2 test_xmlrpc leaked [1, 0, 0] references, sum=1 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog3NabCZ', '-x'] From python-checkins at python.org Tue May 22 10:34:01 2012 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 22 May 2012 10:34:01 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE0ODA0OiBSZW1v?= =?utf8?q?ve_=5B=5D_around_optional_arguments_with_default_values?= Message-ID: http://hg.python.org/cpython/rev/a36666c52115 changeset: 77102:a36666c52115 branch: 2.7 parent: 77099:c13066f752a8 user: Hynek Schlawack date: Tue May 22 10:27:40 2012 +0200 summary: #14804: Remove [] around optional arguments with default values Mostly just mechanical removal of []. In some rare cases I've pulled the default value up into the argument list. files: Doc/library/bdb.rst | 4 +- Doc/library/csv.rst | 6 ++-- Doc/library/difflib.rst | 4 +- Doc/library/dl.rst | 2 +- Doc/library/formatter.rst | 4 +- Doc/library/functions.rst | 4 +- Doc/library/httplib.rst | 2 +- Doc/library/itertools.rst | 2 +- Doc/library/mailbox.rst | 16 +++++++------- Doc/library/os.rst | 4 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/parser.rst | 2 +- Doc/library/pickletools.rst | 2 +- Doc/library/plistlib.rst | 4 +- Doc/library/profile.rst | 2 +- Doc/library/pyclbr.rst | 4 +- Doc/library/shelve.rst | 8 +++--- Doc/library/shutil.rst | 2 +- Doc/library/struct.rst | 2 +- Doc/library/timeit.rst | 4 +- Doc/library/trace.rst | 2 +- Doc/library/ttk.rst | 24 ++++++++++---------- Doc/library/webbrowser.rst | 4 +- Doc/library/wsgiref.rst | 10 ++++---- Doc/library/xml.dom.minidom.rst | 2 +- 25 files changed, 61 insertions(+), 61 deletions(-) diff --git a/Doc/library/bdb.rst b/Doc/library/bdb.rst --- a/Doc/library/bdb.rst +++ b/Doc/library/bdb.rst @@ -20,7 +20,7 @@ The :mod:`bdb` module also defines two classes: -.. class:: Breakpoint(self, file, line[, temporary=0[, cond=None [, funcname=None]]]) +.. class:: Breakpoint(self, file, line, temporary=0, cond=None , funcname=None) This class implements temporary breakpoints, ignore counts, disabling and (re-)enabling, and conditionals. @@ -245,7 +245,7 @@ breakpoints. These methods return a string containing an error message if something went wrong, or ``None`` if all is well. - .. method:: set_break(filename, lineno[, temporary=0[, cond[, funcname]]]) + .. method:: set_break(filename, lineno, temporary=0, cond=None, funcname=None) Set a new breakpoint. If the *lineno* line doesn't exist for the *filename* passed as argument, return an error message. The *filename* diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -162,7 +162,7 @@ The :mod:`csv` module defines the following classes: -.. class:: DictReader(csvfile[, fieldnames=None[, restkey=None[, restval=None[, dialect='excel'[, *args, **kwds]]]]]) +.. class:: DictReader(csvfile, fieldnames=None, restkey=None, restval=None, dialect='excel', *args, **kwds) Create an object which operates like a regular reader but maps the information read into a dict whose keys are given by the optional *fieldnames* parameter. @@ -175,7 +175,7 @@ the underlying :class:`reader` instance. -.. class:: DictWriter(csvfile, fieldnames[, restval=''[, extrasaction='raise'[, dialect='excel'[, *args, **kwds]]]]) +.. class:: DictWriter(csvfile, fieldnames, restval='', extrasaction='raise', dialect='excel', *args, **kwds) Create an object which operates like a regular writer but maps dictionaries onto output rows. The *fieldnames* parameter identifies the order in which values in @@ -219,7 +219,7 @@ The :class:`Sniffer` class provides two methods: - .. method:: sniff(sample[, delimiters=None]) + .. method:: sniff(sample, delimiters=None) Analyze the given *sample* and return a :class:`Dialect` subclass reflecting the parameters found. If the optional *delimiters* parameter diff --git a/Doc/library/difflib.rst b/Doc/library/difflib.rst --- a/Doc/library/difflib.rst +++ b/Doc/library/difflib.rst @@ -84,7 +84,7 @@ The constructor for this class is: - .. function:: __init__([tabsize][, wrapcolumn][, linejunk][, charjunk]) + .. function:: __init__(tabsize=8, wrapcolumn=None, linejunk=None, charjunk=IS_CHARACTER_JUNK) Initializes instance of :class:`HtmlDiff`. @@ -344,7 +344,7 @@ The :class:`SequenceMatcher` class has this constructor: -.. class:: SequenceMatcher([isjunk[, a[, b[, autojunk=True]]]]) +.. class:: SequenceMatcher(isjunk=None, a='', b='', autojunk=True) Optional argument *isjunk* must be ``None`` (the default) or a one-argument function that takes a sequence element and returns true if and only if the diff --git a/Doc/library/dl.rst b/Doc/library/dl.rst --- a/Doc/library/dl.rst +++ b/Doc/library/dl.rst @@ -31,7 +31,7 @@ The :mod:`dl` module defines the following function: -.. function:: open(name[, mode=RTLD_LAZY]) +.. function:: open(name, mode=RTLD_LAZY) Open a shared object file, and return a handle. Mode signifies late binding (:const:`RTLD_LAZY`) or immediate binding (:const:`RTLD_NOW`). Default is diff --git a/Doc/library/formatter.rst b/Doc/library/formatter.rst --- a/Doc/library/formatter.rst +++ b/Doc/library/formatter.rst @@ -341,10 +341,10 @@ output. -.. class:: DumbWriter([file[, maxcol=72]]) +.. class:: DumbWriter(file=None, maxcol=72) Simple writer class which writes output on the file object passed in as *file* - or, if *file* is omitted, on standard output. The output is simply word-wrapped + or, if *file* is None, on standard output. The output is simply word-wrapped to the number of columns specified by *maxcol*. This class is suitable for reflowing a sequence of paragraphs. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -354,7 +354,7 @@ Using :func:`divmod` with complex numbers is deprecated. -.. function:: enumerate(sequence[, start=0]) +.. function:: enumerate(sequence, start=0) Return an enumerate object. *sequence* must be a sequence, an :term:`iterator`, or some other object which supports iteration. The @@ -912,7 +912,7 @@ accidents.) -.. function:: print([object, ...][, sep=' '][, end='\\n'][, file=sys.stdout]) +.. function:: print([object, ...], sep=' ', end='\\n', file=sys.stdout) Print *object*\(s) to the stream *file*, separated by *sep* and followed by *end*. *sep*, *end* and *file*, if present, must be given as keyword diff --git a/Doc/library/httplib.rst b/Doc/library/httplib.rst --- a/Doc/library/httplib.rst +++ b/Doc/library/httplib.rst @@ -89,7 +89,7 @@ *source_address* was added. -.. class:: HTTPResponse(sock[, debuglevel=0][, strict=0]) +.. class:: HTTPResponse(sock, debuglevel=0, strict=0) Class whose instances are returned upon successful connection. Not instantiated directly by user. diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -627,7 +627,7 @@ break -.. function:: tee(iterable[, n=2]) +.. function:: tee(iterable, n=2) Return *n* independent iterators from a single iterable. Equivalent to:: diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -154,7 +154,7 @@ when the :class:`Mailbox` instance was initialized. - .. method:: get(key[, default=None]) + .. method:: get(key, default=None) __getitem__(key) Return a representation of the message corresponding to *key*. If no such @@ -278,7 +278,7 @@ ^^^^^^^^^^^^^^^^ -.. class:: Maildir(dirname[, factory=rfc822.Message[, create=True]]) +.. class:: Maildir(dirname, factory=rfc822.Message, create=True) A subclass of :class:`Mailbox` for mailboxes in Maildir format. Parameter *factory* is a callable object that accepts a file-like message representation @@ -423,7 +423,7 @@ ^^^^^^^^^^^^^ -.. class:: mbox(path[, factory=None[, create=True]]) +.. class:: mbox(path, factory=None, create=True) A subclass of :class:`Mailbox` for mailboxes in mbox format. Parameter *factory* is a callable object that accepts a file-like message representation (which @@ -483,7 +483,7 @@ ^^^^^^^^^^^ -.. class:: MH(path[, factory=None[, create=True]]) +.. class:: MH(path, factory=None, create=True) A subclass of :class:`Mailbox` for mailboxes in MH format. Parameter *factory* is a callable object that accepts a file-like message representation (which @@ -613,7 +613,7 @@ ^^^^^^^^^^^^^^ -.. class:: Babyl(path[, factory=None[, create=True]]) +.. class:: Babyl(path, factory=None, create=True) A subclass of :class:`Mailbox` for mailboxes in Babyl format. Parameter *factory* is a callable object that accepts a file-like message representation @@ -689,7 +689,7 @@ ^^^^^^^^^^^^^ -.. class:: MMDF(path[, factory=None[, create=True]]) +.. class:: MMDF(path, factory=None, create=True) A subclass of :class:`Mailbox` for mailboxes in MMDF format. Parameter *factory* is a callable object that accepts a file-like message representation (which @@ -987,7 +987,7 @@ are excluded. - .. method:: set_from(from_[, time_=None]) + .. method:: set_from(from_, time_=None) Set the "From " line to *from_*, which should be specified without a leading "From " or trailing newline. For convenience, *time_* may be @@ -1358,7 +1358,7 @@ are excluded. - .. method:: set_from(from_[, time_=None]) + .. method:: set_from(from_, time_=None) Set the "From " line to *from_*, which should be specified without a leading "From " or trailing newline. For convenience, *time_* may be diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1181,7 +1181,7 @@ doesn't open the FIFO --- it just creates the rendezvous point. -.. function:: mknod(filename[, mode=0600, device]) +.. function:: mknod(filename, mode=0600, device=0) Create a filesystem node (file, device special file or named pipe) named *filename*. *mode* specifies both the permissions to use and the type of node to @@ -1583,7 +1583,7 @@ Availability: Unix, Windows. -.. function:: walk(top[, topdown=True [, onerror=None[, followlinks=False]]]) +.. function:: walk(top, topdown=True, onerror=None, followlinks=False) .. index:: single: directory; walking diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -275,7 +275,7 @@ simple calculations. -.. method:: oss_audio_device.setparameters(format, nchannels, samplerate [, strict=False]) +.. method:: oss_audio_device.setparameters(format, nchannels, samplerate, strict=False) Set the key audio sampling parameters---sample format, number of channels, and sampling rate---in one method call. *format*, *nchannels*, and *samplerate* diff --git a/Doc/library/parser.rst b/Doc/library/parser.rst --- a/Doc/library/parser.rst +++ b/Doc/library/parser.rst @@ -200,7 +200,7 @@ information is omitted if the flag is false or omitted. -.. function:: compilest(ast[, filename='']) +.. function:: compilest(ast, filename='') .. index:: builtin: eval diff --git a/Doc/library/pickletools.rst b/Doc/library/pickletools.rst --- a/Doc/library/pickletools.rst +++ b/Doc/library/pickletools.rst @@ -20,7 +20,7 @@ probably won't find the :mod:`pickletools` module relevant. -.. function:: dis(pickle[, out=None, memo=None, indentlevel=4]) +.. function:: dis(pickle, out=None, memo=None, indentlevel=4) Outputs a symbolic disassembly of the pickle to the file-like object *out*, defaulting to ``sys.stdout``. *pickle* can be a string or a file-like object. diff --git a/Doc/library/plistlib.rst b/Doc/library/plistlib.rst --- a/Doc/library/plistlib.rst +++ b/Doc/library/plistlib.rst @@ -74,7 +74,7 @@ -.. function:: readPlistFromResource(path[, restype='plst'[, resid=0]]) +.. function:: readPlistFromResource(path, restype='plst', resid=0) Read a plist from the resource with type *restype* from the resource fork of *path*. Availability: Mac OS X. @@ -84,7 +84,7 @@ In Python 3.x, this function has been removed. -.. function:: writePlistToResource(rootObject, path[, restype='plst'[, resid=0]]) +.. function:: writePlistToResource(rootObject, path, restype='plst', resid=0) Write *rootObject* as a resource with type *restype* to the resource fork of *path*. Availability: Mac OS X. diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -298,7 +298,7 @@ :synopsis: Statistics object for use with the profiler. -.. class:: Stats(filename[, stream=sys.stdout[, ...]]) +.. class:: Stats(filename, stream=sys.stdout[, ...]) This class constructor creates an instance of a "statistics object" from a *filename* (or set of filenames). :class:`Stats` objects are manipulated by diff --git a/Doc/library/pyclbr.rst b/Doc/library/pyclbr.rst --- a/Doc/library/pyclbr.rst +++ b/Doc/library/pyclbr.rst @@ -19,7 +19,7 @@ modules. -.. function:: readmodule(module[, path=None]) +.. function:: readmodule(module, path=None) Read a module and return a dictionary mapping class names to class descriptor objects. The parameter *module* should be the name of a @@ -28,7 +28,7 @@ of ``sys.path``, which is used to locate module source code. -.. function:: readmodule_ex(module[, path=None]) +.. function:: readmodule_ex(module, path=None) Like :func:`readmodule`, but the returned dictionary, in addition to mapping class names to class descriptor objects, also maps top-level diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst --- a/Doc/library/shelve.rst +++ b/Doc/library/shelve.rst @@ -18,7 +18,7 @@ lots of shared sub-objects. The keys are ordinary strings. -.. function:: open(filename[, flag='c'[, protocol=None[, writeback=False]]]) +.. function:: open(filename, flag='c', protocol=None, writeback=False) Open a persistent dictionary. The filename specified is the base filename for the underlying database. As a side-effect, an extension may be added to the @@ -100,7 +100,7 @@ implementation used. -.. class:: Shelf(dict[, protocol=None[, writeback=False]]) +.. class:: Shelf(dict, protocol=None, writeback=False) A subclass of :class:`UserDict.DictMixin` which stores pickled values in the *dict* object. @@ -118,7 +118,7 @@ memory and make sync and close take a long time. -.. class:: BsdDbShelf(dict[, protocol=None[, writeback=False]]) +.. class:: BsdDbShelf(dict, protocol=None, writeback=False) A subclass of :class:`Shelf` which exposes :meth:`first`, :meth:`!next`, :meth:`previous`, :meth:`last` and :meth:`set_location` which are available in @@ -129,7 +129,7 @@ the same interpretation as for the :class:`Shelf` class. -.. class:: DbfilenameShelf(filename[, flag='c'[, protocol=None[, writeback=False]]]) +.. class:: DbfilenameShelf(filename, flag='c', protocol=None, writeback=False) A subclass of :class:`Shelf` which accepts a *filename* instead of a dict-like object. The underlying file will be opened using :func:`anydbm.open`. By diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -96,7 +96,7 @@ .. versionadded:: 2.6 -.. function:: copytree(src, dst[, symlinks=False[, ignore=None]]) +.. function:: copytree(src, dst, symlinks=False, ignore=None) Recursively copy an entire directory tree rooted at *src*. The destination directory, named by *dst*, must not already exist; it will be created as diff --git a/Doc/library/struct.rst b/Doc/library/struct.rst --- a/Doc/library/struct.rst +++ b/Doc/library/struct.rst @@ -386,7 +386,7 @@ (``len(string)`` must equal :attr:`self.size`). - .. method:: unpack_from(buffer[, offset=0]) + .. method:: unpack_from(buffer, offset=0) Identical to the :func:`unpack_from` function, using the compiled format. (``len(buffer[offset:])`` must be at least :attr:`self.size`). diff --git a/Doc/library/timeit.rst b/Doc/library/timeit.rst --- a/Doc/library/timeit.rst +++ b/Doc/library/timeit.rst @@ -114,7 +114,7 @@ time. This means that other processes running on the same computer may interfere with the timing. -.. function:: repeat(stmt[, setup[, timer[, repeat=3 [, number=1000000]]]]) +.. function:: repeat(stmt, setup='pass', timer=default_timer, repeat=3 , number=1000000) Create a :class:`Timer` instance with the given statement, setup code and timer function and run its :meth:`repeat` method with the given repeat count and @@ -123,7 +123,7 @@ .. versionadded:: 2.6 -.. function:: timeit(stmt[, setup[, timer[, number=1000000]]]) +.. function:: timeit(stmt, setup='pass', timer=default_timer, number=1000000) Create a :class:`Timer` instance with the given statement, setup code and timer function and run its :meth:`timeit` method with *number* executions. diff --git a/Doc/library/trace.rst b/Doc/library/trace.rst --- a/Doc/library/trace.rst +++ b/Doc/library/trace.rst @@ -149,7 +149,7 @@ the current tracing parameters. *cmd* must be a string or code object, suitable for passing into :func:`exec`. - .. method:: runctx(cmd[, globals=None[, locals=None]]) + .. method:: runctx(cmd, globals=None, locals=None) Execute the command and gather statistics from the execution with the current tracing parameters, in the defined global and local diff --git a/Doc/library/ttk.rst b/Doc/library/ttk.rst --- a/Doc/library/ttk.rst +++ b/Doc/library/ttk.rst @@ -265,7 +265,7 @@ *x* and *y* are pixel coordinates relative to the widget. - .. method:: instate(statespec[, callback=None[, *args[, **kw]]]) + .. method:: instate(statespec, callback=None, *args, **kw) Test the widget's state. If a callback is not specified, returns True if the widget state matches *statespec* and False otherwise. If callback @@ -523,7 +523,7 @@ omitted, returns the widget name of the currently selected pane. - .. method:: tab(tab_id[, option=None[, **kw]]) + .. method:: tab(tab_id, option=None, **kw) Query or modify the options of the specific *tab_id*. @@ -846,7 +846,7 @@ .. class:: Treeview - .. method:: bbox(item[, column=None]) + .. method:: bbox(item, column=None) Returns the bounding box (relative to the treeview widget's window) of the specified *item* in the form (x, y, width, height). @@ -873,7 +873,7 @@ *item*'s children. - .. method:: column(column[, option=None[, **kw]]) + .. method:: column(column, option=None, **kw) Query or modify the options for the specified *column*. @@ -928,7 +928,7 @@ the current focus item, or '' if there is none. - .. method:: heading(column[, option=None[, **kw]]) + .. method:: heading(column, option=None, **kw) Query or modify the heading options for the specified *column*. @@ -1001,7 +1001,7 @@ Returns the integer index of *item* within its parent's list of children. - .. method:: insert(parent, index[, iid=None[, **kw]]) + .. method:: insert(parent, index, iid=None, **kw) Creates a new item and returns the item identifier of the newly created item. @@ -1096,7 +1096,7 @@ Toggle the selection state of each item in *items*. - .. method:: set(item[, column=None[, value=None]]) + .. method:: set(item, column=None, value=None) With one argument, returns a dictionary of column/value pairs for the specified *item*. With two arguments, returns the current value of the @@ -1104,14 +1104,14 @@ *column* in given *item* to the specified *value*. - .. method:: tag_bind(tagname[, sequence=None[, callback=None]]) + .. method:: tag_bind(tagname, sequence=None, callback=None) Bind a callback for the given event *sequence* to the tag *tagname*. When an event is delivered to an item, the callbacks for each of the item's tags option are called. - .. method:: tag_configure(tagname[, option=None[, **kw]]) + .. method:: tag_configure(tagname, option=None, **kw) Query or modify the options for the specified *tagname*. @@ -1220,7 +1220,7 @@ foreground option, for example, you would get a blue foreground when the widget is in the active or pressed states. - .. method:: lookup(style, option[, state=None[, default=None]]) + .. method:: lookup(style, option, state=None, default=None) Returns the value specified for *option* in *style*. @@ -1235,7 +1235,7 @@ print ttk.Style().lookup("TButton", "font") - .. method:: layout(style[, layoutspec=None]) + .. method:: layout(style, layoutspec=None) Define the widget layout for given *style*. If *layoutspec* is omitted, return the layout specification for given style. @@ -1318,7 +1318,7 @@ Returns the list of *elementname*'s options. - .. method:: theme_create(themename[, parent=None[, settings=None]]) + .. method:: theme_create(themename, parent=None, settings=None) Create a new theme. diff --git a/Doc/library/webbrowser.rst b/Doc/library/webbrowser.rst --- a/Doc/library/webbrowser.rst +++ b/Doc/library/webbrowser.rst @@ -48,7 +48,7 @@ The following functions are defined: -.. function:: open(url[, new=0[, autoraise=True]]) +.. function:: open(url, new=0, autoraise=True) Display *url* using the default browser. If *new* is 0, the *url* is opened in the same browser window if possible. If *new* is 1, a new browser window @@ -178,7 +178,7 @@ module-level convenience functions: -.. method:: controller.open(url[, new=0[, autoraise=True]]) +.. method:: controller.open(url, new=0, autoraise=True) Display *url* using the browser handled by this controller. If *new* is 1, a new browser window is opened if possible. If *new* is 2, a new browser page ("tab") diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -59,7 +59,7 @@ found, and "http" otherwise. -.. function:: request_uri(environ [, include_query=1]) +.. function:: request_uri(environ, include_query=1) Return the full request URI, optionally including the query string, using the algorithm found in the "URL Reconstruction" section of :pep:`333`. If @@ -148,7 +148,7 @@ :rfc:`2616`. -.. class:: FileWrapper(filelike [, blksize=8192]) +.. class:: FileWrapper(filelike, blksize=8192) A wrapper to convert a file-like object to an :term:`iterator`. The resulting objects support both :meth:`__getitem__` and :meth:`__iter__` iteration styles, for @@ -271,7 +271,7 @@ :mod:`wsgiref.util`.) -.. function:: make_server(host, port, app [, server_class=WSGIServer [, handler_class=WSGIRequestHandler]]) +.. function:: make_server(host, port, app, server_class=WSGIServer, handler_class=WSGIRequestHandler) Create a new WSGI server listening on *host* and *port*, accepting connections for *app*. The return value is an instance of the supplied *server_class*, and @@ -460,7 +460,7 @@ environment. -.. class:: BaseCGIHandler(stdin, stdout, stderr, environ [, multithread=True [, multiprocess=False]]) +.. class:: BaseCGIHandler(stdin, stdout, stderr, environ, multithread=True, multiprocess=False) Similar to :class:`CGIHandler`, but instead of using the :mod:`sys` and :mod:`os` modules, the CGI environment and I/O streams are specified explicitly. @@ -475,7 +475,7 @@ instead of :class:`SimpleHandler`. -.. class:: SimpleHandler(stdin, stdout, stderr, environ [,multithread=True [, multiprocess=False]]) +.. class:: SimpleHandler(stdin, stdout, stderr, environ, multithread=True, multiprocess=False) Similar to :class:`BaseCGIHandler`, but designed for use with HTTP origin servers. If you are writing an HTTP server implementation, you will probably diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -129,7 +129,7 @@ to discard children of that node. -.. method:: Node.writexml(writer[, indent=""[, addindent=""[, newl=""]]]) +.. method:: Node.writexml(writer, indent="", addindent="", newl="") Write XML to the writer object. The writer should have a :meth:`write` method which matches that of the file object interface. The *indent* parameter is the -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Tue May 22 14:08:50 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 22 May 2012 22:08:50 +1000 Subject: [Python-checkins] cpython (2.7): #14804: Remove [] around optional arguments with default values In-Reply-To: References: Message-ID: On Tue, May 22, 2012 at 6:34 PM, hynek.schlawack wrote: > http://hg.python.org/cpython/rev/a36666c52115 > changeset: ? 77102:a36666c52115 > branch: ? ? ?2.7 > parent: ? ? ?77099:c13066f752a8 > user: ? ? ? ?Hynek Schlawack > date: ? ? ? ?Tue May 22 10:27:40 2012 +0200 > summary: > ?#14804: Remove [] around optional arguments with default values > > Mostly just mechanical removal of []. In some rare cases I've pulled the > default value up into the argument list. Be a little careful with this - "[]" is the right notation when the function doesn't support keyword arguments. At least one of the updated signatures is incorrect: > diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst > --- a/Doc/library/itertools.rst > +++ b/Doc/library/itertools.rst > @@ -627,7 +627,7 @@ > ? ? ? ? ? ? ? ? ? break > > > -.. function:: tee(iterable[, n=2]) > +.. function:: tee(iterable, n=2) >>> itertools.tee([], n=2) Traceback (most recent call last): File "", line 1, in TypeError: tee() takes no keyword arguments Since calling "tee(itr, n=2)" doesn't add really any clarity over "tee(itr, 2)", it's unlikely this function will ever gain keyword argument support (since supporting keyword arguments *is* slower than supporting only positional arguments for functions written in C. The change is probably valid for the pure Python modules, and the builtins looked right, but be wary of any extension modules in the list. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python-checkins at python.org Tue May 22 15:02:16 2012 From: python-checkins at python.org (nick.coghlan) Date: Tue, 22 May 2012 15:02:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_the_versionadded_tags_f?= =?utf8?q?or_a_couple_of_my_recent_changes?= Message-ID: http://hg.python.org/cpython/rev/8056cb9d6eff changeset: 77103:8056cb9d6eff parent: 77101:f5f5db593d99 user: Nick Coghlan date: Tue May 22 23:02:00 2012 +1000 summary: Fix the versionadded tags for a couple of my recent changes files: Doc/library/contextlib.rst | 4 ++-- Doc/library/types.rst | 3 +++ 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -207,6 +207,8 @@ foundation for higher level context managers that manipulate the exit stack in application specific ways. + .. versionadded:: 3.3 + .. method:: enter_context(cm) Enters a new context manager and adds its :meth:`__exit__` method to @@ -270,8 +272,6 @@ callbacks registered, the arguments passed in will indicate that no exception occurred. - .. versionadded:: 3.3 - Examples and Recipes -------------------- diff --git a/Doc/library/types.rst b/Doc/library/types.rst --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -30,6 +30,8 @@ The *exec_body* callback should accept the class namespace as its sole argument and update the namespace directly with the class contents. + .. versionadded:: 3.3 + .. function:: prepare_class(name, bases=(), kwds=None) Calculates the appropriate metaclass and creates the class namespace. @@ -46,6 +48,7 @@ ``'metaclass'`` entry removed. If no *kwds* argument is passed in, this will be an empty dict. + .. versionadded:: 3.3 .. seealso:: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 15:04:54 2012 From: python-checkins at python.org (nick.coghlan) Date: Tue, 22 May 2012 15:04:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo?= Message-ID: http://hg.python.org/cpython/rev/38b218b68d7f changeset: 77104:38b218b68d7f user: Nick Coghlan date: Tue May 22 23:04:42 2012 +1000 summary: Fix typo files: Doc/whatsnew/3.3.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -704,7 +704,7 @@ functionality. Unlike the previous ``contextlib.nested`` API (which was deprecated and removed), the new API is designed to work correctly regardless of whether context managers acquire their resources in -their ``__init`` method (for example, file objects) or in their +their ``__init__`` method (for example, file objects) or in their ``__enter__`` method (for example, synchronisation objects from the :mod:`threading` module). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 15:23:06 2012 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 22 May 2012 15:23:06 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogUmVzdG9yZSBbXSB3?= =?utf8?q?here_default_arguments_are_not_keywords?= Message-ID: http://hg.python.org/cpython/rev/086afe7b61f5 changeset: 77105:086afe7b61f5 branch: 2.7 parent: 77102:a36666c52115 user: Hynek Schlawack date: Tue May 22 15:22:14 2012 +0200 summary: Restore [] where default arguments are not keywords Reverts some changes of a36666c52115. In the case of os's `mknod(filename[, mode=0600[, device=0]])`, I have nested the [] as setting mode doesn't require to set device (but setting device requires to set mode). files: Doc/library/dl.rst | 2 +- Doc/library/itertools.rst | 2 +- Doc/library/os.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/dl.rst b/Doc/library/dl.rst --- a/Doc/library/dl.rst +++ b/Doc/library/dl.rst @@ -31,7 +31,7 @@ The :mod:`dl` module defines the following function: -.. function:: open(name, mode=RTLD_LAZY) +.. function:: open(name[, mode=RTLD_LAZY]) Open a shared object file, and return a handle. Mode signifies late binding (:const:`RTLD_LAZY`) or immediate binding (:const:`RTLD_NOW`). Default is diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -627,7 +627,7 @@ break -.. function:: tee(iterable, n=2) +.. function:: tee(iterable[, n=2]) Return *n* independent iterators from a single iterable. Equivalent to:: diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1181,7 +1181,7 @@ doesn't open the FIFO --- it just creates the rendezvous point. -.. function:: mknod(filename, mode=0600, device=0) +.. function:: mknod(filename[, mode=0600[, device=0]]) Create a filesystem node (file, device special file or named pipe) named *filename*. *mode* specifies both the permissions to use and the type of node to diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -275,7 +275,7 @@ simple calculations. -.. method:: oss_audio_device.setparameters(format, nchannels, samplerate, strict=False) +.. method:: oss_audio_device.setparameters(format, nchannels, samplerate[, strict=False]) Set the key audio sampling parameters---sample format, number of channels, and sampling rate---in one method call. *format*, *nchannels*, and *samplerate* -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 16:17:06 2012 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 22 May 2012 16:17:06 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogUmVzdG9yZSBbXSB3?= =?utf8?q?here_default_arguments_are_not_keywords?= Message-ID: http://hg.python.org/cpython/rev/4fcdde674861 changeset: 77106:4fcdde674861 branch: 3.2 parent: 77100:06f21c6e0f2b user: Hynek Schlawack date: Tue May 22 16:12:18 2012 +0200 summary: Restore [] where default arguments are not keywords Reverts some changes of d13fdd97cc8e. files: Doc/library/curses.rst | 2 +- Doc/library/os.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/select.rst | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -1091,7 +1091,7 @@ rendition (as set by :meth:`bkgdset`) merged into them. -.. method:: window.scroll(lines=1) +.. method:: window.scroll([lines=1]) Scroll the screen or scrolling region upward by *lines* lines. diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1145,7 +1145,7 @@ Availability: Unix. -.. function:: mknod(filename, mode=0o600, device=0) +.. function:: mknod(filename[, mode=0o600[, device=0]]) Create a filesystem node (file, device special file or named pipe) named *filename*. *mode* specifies both the permissions to use and the type of node diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -277,7 +277,7 @@ simple calculations. -.. method:: oss_audio_device.setparameters(format, nchannels, samplerate, strict=False) +.. method:: oss_audio_device.setparameters(format, nchannels, samplerate[, strict=False]) Set the key audio sampling parameters---sample format, number of channels, and sampling rate---in one method call. *format*, *nchannels*, and *samplerate* diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -285,7 +285,7 @@ Create a kqueue object from a given file descriptor. -.. method:: kqueue.control(changelist, max_events, timeout=None) -> eventlist +.. method:: kqueue.control(changelist, max_events[, timeout=None]) -> eventlist Low level interface to kevent -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 16:17:07 2012 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 22 May 2012 16:17:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Restore_=5B=5D_where_default_arguments_are_not_keywords?= Message-ID: http://hg.python.org/cpython/rev/dff6c506c2f1 changeset: 77107:dff6c506c2f1 parent: 77104:38b218b68d7f parent: 77106:4fcdde674861 user: Hynek Schlawack date: Tue May 22 16:14:56 2012 +0200 summary: Restore [] where default arguments are not keywords files: Doc/library/curses.rst | 2 +- Doc/library/os.rst | 2 +- Doc/library/ossaudiodev.rst | 2 +- Doc/library/select.rst | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -1120,7 +1120,7 @@ rendition (as set by :meth:`bkgdset`) merged into them. -.. method:: window.scroll(lines=1) +.. method:: window.scroll([lines=1]) Scroll the screen or scrolling region upward by *lines* lines. diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -1786,7 +1786,7 @@ Availability: Unix. -.. function:: mknod(filename, mode=0o600, device=0) +.. function:: mknod(filename[, mode=0o600[, device=0]]) Create a filesystem node (file, device special file or named pipe) named *filename*. *mode* specifies both the permissions to use and the type of node diff --git a/Doc/library/ossaudiodev.rst b/Doc/library/ossaudiodev.rst --- a/Doc/library/ossaudiodev.rst +++ b/Doc/library/ossaudiodev.rst @@ -281,7 +281,7 @@ simple calculations. -.. method:: oss_audio_device.setparameters(format, nchannels, samplerate, strict=False) +.. method:: oss_audio_device.setparameters(format, nchannels, samplerate[, strict=False]) Set the key audio sampling parameters---sample format, number of channels, and sampling rate---in one method call. *format*, *nchannels*, and *samplerate* diff --git a/Doc/library/select.rst b/Doc/library/select.rst --- a/Doc/library/select.rst +++ b/Doc/library/select.rst @@ -371,7 +371,7 @@ Create a kqueue object from a given file descriptor. -.. method:: kqueue.control(changelist, max_events, timeout=None) -> eventlist +.. method:: kqueue.control(changelist, max_events[, timeout=None]) -> eventlist Low level interface to kevent -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 18:05:18 2012 From: python-checkins at python.org (brian.curtin) Date: Tue, 22 May 2012 18:05:18 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_build=2Ebat_for_VS20?= =?utf8?q?10?= Message-ID: http://hg.python.org/cpython/rev/1ea3e1bc3d47 changeset: 77108:1ea3e1bc3d47 user: Brian Curtin date: Tue May 22 11:04:32 2012 -0500 summary: Update build.bat for VS2010 files: PCbuild/build.bat | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/PCbuild/build.bat b/PCbuild/build.bat --- a/PCbuild/build.bat +++ b/PCbuild/build.bat @@ -5,14 +5,15 @@ setlocal set platf=Win32 set conf=Release -set build= +set target=build +set dir=%~dp0 :CheckOpts if "%1"=="-c" (set conf=%2) & shift & shift & goto CheckOpts if "%1"=="-p" (set platf=%2) & shift & shift & goto CheckOpts -if "%1"=="-r" (set build=/rebuild) & shift & goto CheckOpts +if "%1"=="-r" (set target=rebuild) & shift & goto CheckOpts if "%1"=="-d" (set conf=Debug) & shift & goto CheckOpts -set cmd=vcbuild /useenv pcbuild.sln %build% "%conf%|%platf%" +set cmd=msbuild /p:useenv=true %dir%pcbuild.sln /t:%target% /p:Configuration=%conf% /p:Platform=%platf% echo %cmd% %cmd% -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 19:52:39 2012 From: python-checkins at python.org (petri.lehtinen) Date: Tue, 22 May 2012 19:52:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314472=3A_Update_=2Egitig?= =?utf8?q?nore?= Message-ID: http://hg.python.org/cpython/rev/fb5da2a5d9da changeset: 77109:fb5da2a5d9da user: Petri Lehtinen date: Tue May 22 20:48:16 2012 +0300 summary: #14472: Update .gitignore Patch by Matej Cepl. files: .gitignore | 19 +++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 2 ++ 3 files changed, 22 insertions(+), 0 deletions(-) diff --git a/.gitignore b/.gitignore --- a/.gitignore +++ b/.gitignore @@ -5,14 +5,18 @@ *.pyd *.pyo *.rej +*.swp *~ +.gdb_history Doc/build/ Doc/tools/docutils/ +Doc/tools/jinja/ Doc/tools/jinja2/ Doc/tools/pygments/ Doc/tools/sphinx/ Lib/lib2to3/*.pickle Lib/_sysconfigdata.py +Lib/plat-mac/errors.rsrc.df.rsrc Makefile Makefile.pre Misc/python.pc @@ -31,19 +35,34 @@ PCbuild/*.o PCbuild/*.pdb PCbuild/Win32-temp-* +PCbuild/amd64/ +.purify Parser/pgen __pycache__ autom4te.cache build/ +buildno +config.cache +config.log +config.status +config.status.lineno +core +db_home config.log config.status libpython*.a libpython*.so* +platform pybuilddir.txt pyconfig.h python +python.exe python-gdb.py +python.exe-gdb.py +reflog.txt +.svn/ tags +TAGS .coverage coverage/ htmlcov/ diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -161,6 +161,7 @@ Donn Cave Charles Cazabon Per Cederqvist +Matej Cepl Octavian Cerna Pascal Chambon John Chandler diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -172,6 +172,8 @@ Build ----- +- Issue #14472: Update .gitignore. Patch by Matej Cepl. + - Upgrade Windows library versions: bzip 1.0.6, OpenSSL 1.0.1c. - Issue #14693: Under non-Windows platforms, hashlib's fallback modules are -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 20:07:53 2012 From: python-checkins at python.org (petri.lehtinen) Date: Tue, 22 May 2012 20:07:53 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0NDcyOiBVcGRh?= =?utf8?q?te_=2Egitignore?= Message-ID: http://hg.python.org/cpython/rev/8652dd5c2a14 changeset: 77110:8652dd5c2a14 branch: 3.2 parent: 77106:4fcdde674861 user: Petri Lehtinen date: Tue May 22 20:48:16 2012 +0300 summary: #14472: Update .gitignore Patch by Matej Cepl. files: .gitignore | 19 +++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 2 ++ 3 files changed, 22 insertions(+), 0 deletions(-) diff --git a/.gitignore b/.gitignore --- a/.gitignore +++ b/.gitignore @@ -5,13 +5,17 @@ *.pyd *.pyo *.rej +*.swp *~ +.gdb_history Doc/build/ Doc/tools/docutils/ +Doc/tools/jinja/ Doc/tools/jinja2/ Doc/tools/pygments/ Doc/tools/sphinx/ Lib/lib2to3/*.pickle +Lib/plat-mac/errors.rsrc.df.rsrc Makefile Makefile.pre Misc/python.pc @@ -30,20 +34,35 @@ PCbuild/*.o PCbuild/*.pdb PCbuild/Win32-temp-* +PCbuild/amd64/ +.purify Parser/pgen Parser/pgen.stamp __pycache__ autom4te.cache build/ +buildno +config.cache +config.log +config.status +config.status.lineno +core +db_home config.log config.status libpython*.a libpython*.so* +platform pybuilddir.txt pyconfig.h python +python.exe python-gdb.py +python.exe-gdb.py +reflog.txt +.svn/ tags +TAGS .coverage coverage/ htmlcov/ diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -150,6 +150,7 @@ Donn Cave Charles Cazabon Per Cederqvist +Matej Cepl Octavian Cerna Pascal Chambon John Chandler diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -257,6 +257,8 @@ Build ----- +- Issue #14472: Update .gitignore. Patch by Matej Cepl. + - The Windows build now uses OpenSSL 1.0.0j and bzip2 1.0.6. - Issue #14557: Fix extensions build on HP-UX. Patch by Adi Roiban. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 22 20:07:54 2012 From: python-checkins at python.org (petri.lehtinen) Date: Tue, 22 May 2012 20:07:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_branch_=273=2E2=27?= Message-ID: http://hg.python.org/cpython/rev/5e6964705419 changeset: 77111:5e6964705419 parent: 77109:fb5da2a5d9da parent: 77110:8652dd5c2a14 user: Petri Lehtinen date: Tue May 22 21:05:30 2012 +0300 summary: Merge branch '3.2' files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 02:35:49 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 23 May 2012 02:35:49 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Added_dynamic_path_computation?= =?utf8?q?_rationale=2C_specification=2C_and_discussion=2E?= Message-ID: http://hg.python.org/peps/rev/f2b8f61b39a1 changeset: 4415:f2b8f61b39a1 user: Eric V. Smith date: Tue May 22 20:35:42 2012 -0400 summary: Added dynamic path computation rationale, specification, and discussion. files: pep-0420.txt | 92 +++++++++++++++++++++++++++++++++------ 1 files changed, 77 insertions(+), 15 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -21,6 +21,7 @@ documented in PEP 382 and PEP 402. Those PEPs have since been rejected in favor of this one. An implementation of this PEP is at [1]_. + Terminology =========== @@ -43,6 +44,7 @@ This PEP defines a new type of package, the "namespace package". + Namespace packages today ======================== @@ -78,9 +80,10 @@ ``setup.py``, so that distribution developers don't need to put the magic ``__path__`` modification into ``__init__.py`` themselves. -See PEP 402's "The Problem" section [2]_ for more details on the -motivation for namespace packages. Note that PEP 402 has been -rejected, but the motivating use cases are still valid. +See PEP 402's "The Problem" section [2]_ for additional motivations +for namespace packages. Note that PEP 402 has been rejected, but the +motivating use cases are still valid. + Rationale ========= @@ -107,6 +110,36 @@ directory or split across multiple directories as distributions see fit. +A namespace package will not be constrained by a fixed ``__path__``, +computed from the parent path at namespace package creation time. +Consider the standard library ``encodings`` package: + + 1. Suppose that ``encodings`` becomes a namespace package. + + 2. It sometimes gets imported during interpreter startup to + initialize the standard io streams. + + 3. An application modifies ``sys.path`` after startup and wants to + contribute additional encodings. + + 4. An attempt is made to import an ``encodings`` portion that is + found on a path added in step 3. + +If the import system was restricted to only finding portions along the +value of ``sys.path`` that existed at the time the ``encodings`` +namespace package was created, the additional paths added in step 3 +would never be searched for the additional portions imported in step +4. In addition, if step 2 were sometimes skipped (due to some runtime +flag or other condition), then the path items added in step 3 would +indeed be used the first time a portion was imported. Thus this PEP +requires that the list of path entries be dynamically computed when +each portion is loaded. It is expected that the import machinery will +do this efficiently by caching ``__path__`` values and only refreshing +them when it detects that the parent path has changed. In the case of +a top-level package like ``encodings``, this parent path would be +``sys.path``. + + Specification ============= @@ -161,16 +194,15 @@ Dynamic path computation ------------------------ -A namespace package's ``__path__`` will be recomputed if the value of -the parent path changes. In order for this feature to work, the parent -path must be modified in-place, not replaced with a new object. For -example, for top-level namespace packages, this will work:: +The import machinery will behave as if a namespace package's +``__path__`` is recomputed before each portion is loaded. - sys.path.append('new-dir') - -But this will not:: - - sys.path = sys.path + ['new-dir'] +For performance reasons, it is expected that this will be achieved by +detecting that the parent path has changed. If no change has taken +place, then no ``__path__`` recomputation is required. The +implementation must ensure that changes to the contents of the parent +path are detected, as well as detecting the replacement of the parent +path with a new path entry list object. Impact on import finders and loaders ------------------------------------ @@ -354,6 +386,33 @@ The use case for supporting multiple portions per ``find_loader`` call is given in [7]_. +Dynamic path computation +------------------------ + +Guido raised a concern that automatic dynamic path computation was an +unnecessary feature [8]_. Later in that thread, Philip Eby and Nick +Coghlan presented arguments as to why dynamic computation would +minimize surprise to Python users. The conclusion of that discussion +has been included in this PEP's Rationale section. + +An earlier version of this PEP required that dynamic path computation +could only take affect if the parent path object were modified +in-place. That is, this would work:: + + sys.path.append('new-dir') + +But this would not:: + + sys.path = sys.path + ['new-dir'] + +In the same thread [8]_, it was pointed out that this restriction is +not required. If the parent path is looked up by name instead of by +holding a reference to it, then there is no restriction on how the +parent path is modified or replaced. For a top-level namespace +package, the lookup would be module ``sys`` then attribute ``path``. +For a namespace package nested inside a module ``foo``, the lookup +would be module ``foo`` then attribute ``__path__``. + Module reprs ============ @@ -424,7 +483,7 @@ >>> m >>> class Loader: pass - ... + ... >>> m.__loader__ = Loader >>> del m.__file__ >>> m @@ -433,11 +492,11 @@ ... @classmethod ... def module_repr(cls, module): ... return '' - ... + ... >>> m.__loader__ = NewLoader >>> m - >>> + >>> References @@ -464,6 +523,9 @@ .. [7] Use case for multiple portions per ``find_loader`` call (http://mail.python.org/pipermail/import-sig/2012-May/000585.html) +.. [8] Discussion about dynamic path computation + (http://mail.python.org/pipermail/python-dev/2012-May/119560.html) + Copyright ========= -- Repository URL: http://hg.python.org/peps From ncoghlan at gmail.com Wed May 23 02:42:22 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 May 2012 10:42:22 +1000 Subject: [Python-checkins] peps: Added dynamic path computation rationale, specification, and discussion. In-Reply-To: References: Message-ID: On Wed, May 23, 2012 at 10:35 AM, eric.smith wrote: > + ?4. An attempt is made to import an ``encodings`` portion that is > + ? ? found on a path added in step 3. I'd phrase this as something like "import an encoding from an ``encodings`` portion". You don't really import namespace package portions directly - you import the modules and packages they contain. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python-checkins at python.org Wed May 23 02:54:06 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 23 May 2012 02:54:06 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Improvement_suggested_by_Nick?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/peps/rev/9ba1568e8935 changeset: 4416:9ba1568e8935 user: Eric V. Smith date: Tue May 22 20:54:00 2012 -0400 summary: Improvement suggested by Nick. files: pep-0420.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -120,10 +120,10 @@ initialize the standard io streams. 3. An application modifies ``sys.path`` after startup and wants to - contribute additional encodings. + contribute additional encodings from new path entries. - 4. An attempt is made to import an ``encodings`` portion that is - found on a path added in step 3. + 4. An attempt is made to import an encoding from an ``encodings`` + portion that is found on a path entry added in step 3. If the import system was restricted to only finding portions along the value of ``sys.path`` that existed at the time the ``encodings`` -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 23 02:57:47 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 23 May 2012 02:57:47 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Make_it_clear_that_the_lookups?= =?utf8?q?_are_being_done_via_strings=2E?= Message-ID: http://hg.python.org/peps/rev/9a340aeb9950 changeset: 4417:9a340aeb9950 user: Eric V. Smith date: Tue May 22 20:57:41 2012 -0400 summary: Make it clear that the lookups are being done via strings. files: pep-0420.txt | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -409,9 +409,9 @@ not required. If the parent path is looked up by name instead of by holding a reference to it, then there is no restriction on how the parent path is modified or replaced. For a top-level namespace -package, the lookup would be module ``sys`` then attribute ``path``. -For a namespace package nested inside a module ``foo``, the lookup -would be module ``foo`` then attribute ``__path__``. +package, the lookup would be a module ``"sys"`` then an attribute +``"path"``. For a namespace package nested inside a module ``foo``, +the lookup would be module ``"foo"`` then attribute ``"__path__"``. Module reprs -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 23 03:00:24 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 23 May 2012 03:00:24 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Improve_wording=2E?= Message-ID: http://hg.python.org/peps/rev/51123c9a56f4 changeset: 4418:51123c9a56f4 user: Eric V. Smith date: Tue May 22 21:00:16 2012 -0400 summary: Improve wording. files: pep-0420.txt | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -409,9 +409,10 @@ not required. If the parent path is looked up by name instead of by holding a reference to it, then there is no restriction on how the parent path is modified or replaced. For a top-level namespace -package, the lookup would be a module ``"sys"`` then an attribute +package, the lookup would be a module ``"sys"`` then its attribute ``"path"``. For a namespace package nested inside a module ``foo``, -the lookup would be module ``"foo"`` then attribute ``"__path__"``. +the lookup would be module ``"foo"`` then its attribute +``"__path__"``. Module reprs -- Repository URL: http://hg.python.org/peps From solipsis at pitrou.net Wed May 23 05:50:49 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 23 May 2012 05:50:49 +0200 Subject: [Python-checkins] Daily reference leaks (5e6964705419): sum=0 Message-ID: results for 5e6964705419 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogpBFJCQ', '-x'] From python-checkins at python.org Wed May 23 06:10:11 2012 From: python-checkins at python.org (eli.bendersky) Date: Wed, 23 May 2012 06:10:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_s/tabs/spaces=2C_and_clean_?= =?utf8?q?trailing_whitespace?= Message-ID: http://hg.python.org/cpython/rev/c84f896e76e9 changeset: 77112:c84f896e76e9 user: Eli Bendersky date: Wed May 23 07:09:08 2012 +0300 summary: s/tabs/spaces, and clean trailing whitespace files: Include/structmember.h | 55 ++++++++++++++--------------- 1 files changed, 27 insertions(+), 28 deletions(-) diff --git a/Include/structmember.h b/Include/structmember.h --- a/Include/structmember.h +++ b/Include/structmember.h @@ -16,42 +16,41 @@ pointer is NULL. */ typedef struct PyMemberDef { - /* Current version, use this */ - char *name; - int type; - Py_ssize_t offset; - int flags; - char *doc; + char *name; + int type; + Py_ssize_t offset; + int flags; + char *doc; } PyMemberDef; /* Types */ -#define T_SHORT 0 -#define T_INT 1 -#define T_LONG 2 -#define T_FLOAT 3 -#define T_DOUBLE 4 -#define T_STRING 5 -#define T_OBJECT 6 +#define T_SHORT 0 +#define T_INT 1 +#define T_LONG 2 +#define T_FLOAT 3 +#define T_DOUBLE 4 +#define T_STRING 5 +#define T_OBJECT 6 /* XXX the ordering here is weird for binary compatibility */ -#define T_CHAR 7 /* 1-character string */ -#define T_BYTE 8 /* 8-bit signed int */ +#define T_CHAR 7 /* 1-character string */ +#define T_BYTE 8 /* 8-bit signed int */ /* unsigned variants: */ -#define T_UBYTE 9 -#define T_USHORT 10 -#define T_UINT 11 -#define T_ULONG 12 +#define T_UBYTE 9 +#define T_USHORT 10 +#define T_UINT 11 +#define T_ULONG 12 /* Added by Jack: strings contained in the structure */ -#define T_STRING_INPLACE 13 +#define T_STRING_INPLACE 13 /* Added by Lillo: bools contained in the structure (assumed char) */ -#define T_BOOL 14 +#define T_BOOL 14 -#define T_OBJECT_EX 16 /* Like T_OBJECT, but raises AttributeError - when the value is NULL, instead of - converting to None. */ +#define T_OBJECT_EX 16 /* Like T_OBJECT, but raises AttributeError + when the value is NULL, instead of + converting to None. */ #ifdef HAVE_LONG_LONG -#define T_LONGLONG 17 +#define T_LONGLONG 17 #define T_ULONGLONG 18 #endif /* HAVE_LONG_LONG */ @@ -60,10 +59,10 @@ /* Flags */ -#define READONLY 1 -#define READ_RESTRICTED 2 +#define READONLY 1 +#define READ_RESTRICTED 2 #define PY_WRITE_RESTRICTED 4 -#define RESTRICTED (READ_RESTRICTED | PY_WRITE_RESTRICTED) +#define RESTRICTED (READ_RESTRICTED | PY_WRITE_RESTRICTED) /* Current API, use this */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 11:55:30 2012 From: python-checkins at python.org (hynek.schlawack) Date: Wed, 23 May 2012 11:55:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314885=3A_Make_support=2E?= =?utf8?q?skip=5Funless=5Fxattr_check_also_tempfile?= Message-ID: http://hg.python.org/cpython/rev/ab94ed2a8012 changeset: 77113:ab94ed2a8012 user: Hynek Schlawack date: Wed May 23 11:22:44 2012 +0200 summary: #14885: Make support.skip_unless_xattr check also tempfile There is a rare edge case where the filesystem used by the tempfile functions (usually /tmp) doesn't support xattrs while the one used by TESTFN (the current directory, so likely to be below /home) does. This causes the xattr related test_shutil tests fail. skip_unless_xattr now checks both. I have also added skip_unless_xattr to __all__ where it has been missing. files: Lib/test/support.py | 28 ++++++++++++++++++---------- 1 files changed, 18 insertions(+), 10 deletions(-) diff --git a/Lib/test/support.py b/Lib/test/support.py --- a/Lib/test/support.py +++ b/Lib/test/support.py @@ -24,6 +24,7 @@ import fnmatch import logging.handlers import struct +import tempfile try: import _thread, threading @@ -51,23 +52,25 @@ lzma = None __all__ = [ - "Error", "TestFailed", "ResourceDenied", "import_module", - "verbose", "use_resources", "max_memuse", "record_original_stdout", + "Error", "TestFailed", "ResourceDenied", "import_module", "verbose", + "use_resources", "max_memuse", "record_original_stdout", "get_original_stdout", "unload", "unlink", "rmtree", "forget", "is_resource_enabled", "requires", "requires_freebsd_version", - "requires_linux_version", "requires_mac_ver", "find_unused_port", "bind_port", - "IPV6_ENABLED", "is_jython", "TESTFN", "HOST", "SAVEDCWD", "temp_cwd", - "findfile", "create_empty_file", "sortdict", "check_syntax_error", "open_urlresource", - "check_warnings", "CleanImport", "EnvironmentVarGuard", "TransientResource", - "captured_stdout", "captured_stdin", "captured_stderr", "time_out", - "socket_peer_reset", "ioerror_peer_reset", "run_with_locale", 'temp_umask', + "requires_linux_version", "requires_mac_ver", "find_unused_port", + "bind_port", "IPV6_ENABLED", "is_jython", "TESTFN", "HOST", "SAVEDCWD", + "temp_cwd", "findfile", "create_empty_file", "sortdict", + "check_syntax_error", "open_urlresource", "check_warnings", "CleanImport", + "EnvironmentVarGuard", "TransientResource", "captured_stdout", + "captured_stdin", "captured_stderr", "time_out", "socket_peer_reset", + "ioerror_peer_reset", "run_with_locale", 'temp_umask', "transient_internet", "set_memlimit", "bigmemtest", "bigaddrspacetest", "BasicTestRunner", "run_unittest", "run_doctest", "threading_setup", "threading_cleanup", "reap_children", "cpython_only", "check_impl_detail", "get_attribute", "swap_item", "swap_attr", "requires_IEEE_754", "TestHandler", "Matcher", "can_symlink", "skip_unless_symlink", - "import_fresh_module", "requires_zlib", "PIPE_MAX_SIZE", "failfast", - "anticipate_failure", "run_with_tz", "requires_bz2", "requires_lzma" + "skip_unless_xattr", "import_fresh_module", "requires_zlib", + "PIPE_MAX_SIZE", "failfast", "anticipate_failure", "run_with_tz", + "requires_bz2", "requires_lzma" ] class Error(Exception): @@ -1694,9 +1697,13 @@ if not hasattr(os, "setxattr"): can = False else: + tmp_fp, tmp_name = tempfile.mkstemp() try: with open(TESTFN, "wb") as fp: try: + # TESTFN & tempfile may use different file systems with + # different capabilities + os.fsetxattr(tmp_fp, b"user.test", b"") os.fsetxattr(fp.fileno(), b"user.test", b"") # Kernels < 2.6.39 don't respect setxattr flags. kernel_version = platform.release() @@ -1706,6 +1713,7 @@ can = False finally: unlink(TESTFN) + unlink(tmp_name) _can_xattr = can return can -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 14:20:12 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 23 May 2012 14:20:12 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Improved_wording=2E?= Message-ID: http://hg.python.org/peps/rev/5cef0e854065 changeset: 4419:5cef0e854065 user: Eric V. Smith date: Wed May 23 08:20:02 2012 -0400 summary: Improved wording. files: pep-0420.txt | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -409,10 +409,10 @@ not required. If the parent path is looked up by name instead of by holding a reference to it, then there is no restriction on how the parent path is modified or replaced. For a top-level namespace -package, the lookup would be a module ``"sys"`` then its attribute -``"path"``. For a namespace package nested inside a module ``foo``, -the lookup would be module ``"foo"`` then its attribute -``"__path__"``. +package, the lookup would be the module named ``"sys"`` then its +attribute ``"path"``. For a namespace package nested inside a package +``foo``, the lookup would be for the module named ``"foo"`` then its +attribute ``"__path__"``. Module reprs -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 23 14:27:12 2012 From: python-checkins at python.org (eric.smith) Date: Wed, 23 May 2012 14:27:12 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fixed_Phillip_Eby=27s_name=2E?= Message-ID: http://hg.python.org/peps/rev/557df76025a1 changeset: 4420:557df76025a1 user: Eric V. Smith date: Wed May 23 08:27:06 2012 -0400 summary: Fixed Phillip Eby's name. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -390,7 +390,7 @@ ------------------------ Guido raised a concern that automatic dynamic path computation was an -unnecessary feature [8]_. Later in that thread, Philip Eby and Nick +unnecessary feature [8]_. Later in that thread, PJ Eby and Nick Coghlan presented arguments as to why dynamic computation would minimize surprise to Python users. The conclusion of that discussion has been included in this PEP's Rationale section. -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 23 17:43:13 2012 From: python-checkins at python.org (florent.xicluna) Date: Wed, 23 May 2012 17:43:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_duplicat?= =?utf8?q?e_entries_in_Misc/NEWS=2E?= Message-ID: http://hg.python.org/cpython/rev/de5cb14e929e changeset: 77114:de5cb14e929e branch: 3.2 parent: 77110:8652dd5c2a14 user: Florent Xicluna date: Wed May 23 17:42:50 2012 +0200 summary: Remove duplicate entries in Misc/NEWS. files: Misc/NEWS | 18 ------------------ 1 files changed, 0 insertions(+), 18 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -287,26 +287,6 @@ Tools/parser/unparse.py. -What's New in Python 3.2.3 release candidate 2? -=============================================== - -*Release date: XX-Mar-2012* - -Library -------- - -- Issue #6884: Fix long-standing bugs with MANIFEST.in parsing in distutils - on Windows. - -Extension Modules ------------------ - -- Issue #14234: CVE-2012-0876: Randomize hashes of xml attributes in the hash - table internal to the pyexpat module's copy of the expat library to avoid a - denial of service due to hash collisions. Patch by David Malcolm with some - modifications by the expat project. - - What's New in Python 3.2.3? =========================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 20:16:50 2012 From: python-checkins at python.org (eli.bendersky) Date: Wed, 23 May 2012 20:16:50 +0200 Subject: [Python-checkins] =?utf8?q?devguide=3A_Issue_=2314884=3A_fixed_a_?= =?utf8?q?couple_of_typos_in_the_Windows_build_instructions=2E_Patch?= Message-ID: http://hg.python.org/devguide/rev/e55c65fc3cb4 changeset: 520:e55c65fc3cb4 user: Eli Bendersky date: Wed May 23 21:15:35 2012 +0300 summary: Issue #14884: fixed a couple of typos in the Windows build instructions. Patch by Michael Driscoll files: setup.rst | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/setup.rst b/setup.rst --- a/setup.rst +++ b/setup.rst @@ -171,9 +171,10 @@ http://msdn.microsoft.com/en-us/library/hs24szh9%28v=VS.90%29.aspx . To build from the Visual Studio GUI, open pcbuild.sln to load the project -files and choose the Build Solution option from the Build menu, often +files and choose the Build Solution option from either the Build or Debug menu +(depending on your Visual Studio version), which is often associated with the F7 key. Make sure you have chosen the "Debug" option from -the build configuration drop-down first. +the "Solution Configurations" drop-down on the toolbar first. Once built you might want to set Python as a startup project. Pressing F5 in Visual Studio, or choosing Start Debugging from the Debug menu, will launch -- Repository URL: http://hg.python.org/devguide From python-checkins at python.org Wed May 23 20:38:07 2012 From: python-checkins at python.org (petri.lehtinen) Date: Wed, 23 May 2012 20:38:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314862=3A_Add_missing_nam?= =?utf8?b?ZXMgdG8gb3MuX19hbGxfXw==?= Message-ID: http://hg.python.org/cpython/rev/352147bbefdb changeset: 77115:352147bbefdb parent: 77113:ab94ed2a8012 user: Petri Lehtinen date: Wed May 23 21:36:16 2012 +0300 summary: #14862: Add missing names to os.__all__ files: Lib/os.py | 11 +++++++++-- Misc/NEWS | 2 ++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/Lib/os.py b/Lib/os.py --- a/Lib/os.py +++ b/Lib/os.py @@ -30,8 +30,9 @@ # Note: more names are added to __all__ later. __all__ = ["altsep", "curdir", "pardir", "sep", "pathsep", "linesep", - "defpath", "name", "path", "devnull", - "SEEK_SET", "SEEK_CUR", "SEEK_END"] + "defpath", "name", "path", "devnull", "SEEK_SET", "SEEK_CUR", + "SEEK_END", "fsencode", "fsdecode", "get_exec_path", "fdopen", + "popen", "extsep"] def _exists(name): return name in globals() @@ -50,6 +51,7 @@ from posix import * try: from posix import _exit + __all__.append('_exit') except ImportError: pass import posixpath as path @@ -64,6 +66,7 @@ from nt import * try: from nt import _exit + __all__.append('_exit') except ImportError: pass import ntpath as path @@ -78,6 +81,7 @@ from os2 import * try: from os2 import _exit + __all__.append('_exit') except ImportError: pass if sys.version.find('EMX GCC') == -1: @@ -96,6 +100,7 @@ from ce import * try: from ce import _exit + __all__.append('_exit') except ImportError: pass # We can use the standard Windows path. @@ -700,6 +705,8 @@ P_WAIT = 0 P_NOWAIT = P_NOWAITO = 1 + __all__.extend(["P_WAIT", "P_NOWAIT", "P_NOWAITO"]) + # XXX Should we support P_DETACH? I suppose it could fork()**2 # and close the std I/O streams. Also, P_OVERLAY is the same # as execv*()? diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,8 @@ Library ------- +- Issue #14862: Add missing names to os.__all__ + - Issue #14875: Use float('inf') instead of float('1e66666') in the json module. - Issue #13585: Added contextlib.ExitStack -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 22:27:17 2012 From: python-checkins at python.org (sandro.tosi) Date: Wed, 23 May 2012 22:27:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_improve_d?= =?utf8?q?ocstrings_and_arguments_value_handling=2C_as_per_Terry_J=2E?= Message-ID: http://hg.python.org/cpython/rev/0be296605165 changeset: 77116:0be296605165 user: Sandro Tosi date: Wed May 23 22:26:55 2012 +0200 summary: Issue #14814: improve docstrings and arguments value handling, as per Terry J. Reedy's comments files: Lib/ipaddress.py | 65 +++++++++++++++---------- Lib/test/test_ipaddress.py | 18 +++++++ 2 files changed, 56 insertions(+), 27 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -45,8 +45,8 @@ address: A string or integer, the IP address. Either IPv4 or IPv6 addresses may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An Integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. important for things + version: An integer, 4 or 6. If set, don't try to automatically + determine what the IP address type is. Important for things like ip_address(1), which could be IPv4, '192.0.2.1', or IPv6, '2001:db8::1'. @@ -54,15 +54,17 @@ An IPv4Address or IPv6Address object. Raises: - ValueError: if the string passed isn't either a v4 or a v6 - address. + ValueError: if the *address* passed isn't either a v4 or a v6 + address, or if the version is not None, 4, or 6. """ - if version: + if version is not None: if version == 4: return IPv4Address(address) elif version == 6: return IPv6Address(address) + else: + raise ValueError() try: return IPv4Address(address) @@ -85,8 +87,8 @@ address: A string or integer, the IP network. Either IPv4 or IPv6 networks may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An Integer, if set, don't try to automatically - determine what the IP address type is. important for things + version: An integer, 4 or 6. If set, don't try to automatically + determine what the IP address type is. Important for things like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, '2001:db8::1/128'. @@ -95,14 +97,17 @@ Raises: ValueError: if the string passed isn't either a v4 or a v6 - address. Or if the network has host bits set. + address. Or if the network has host bits set. Or if the version + is not None, 4, or 6. """ - if version: + if version is not None: if version == 4: return IPv4Network(address, strict) elif version == 6: return IPv6Network(address, strict) + else: + raise ValueError() try: return IPv4Network(address, strict) @@ -125,28 +130,30 @@ address: A string or integer, the IP address. Either IPv4 or IPv6 addresses may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An Integer, if set, don't try to automatically - determine what the IP address type is. important for things - like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, + version: An integer, 4 or 6. If set, don't try to automatically + determine what the IP address type is. Important for things + like ip_interface(1), which could be IPv4, '192.0.2.1/32', or IPv6, '2001:db8::1/128'. Returns: - An IPv4Network or IPv6Network object. + An IPv4Interface or IPv6Interface object. Raises: ValueError: if the string passed isn't either a v4 or a v6 - address. + address. Or if the version is not None, 4, or 6. Notes: The IPv?Interface classes describe an Address on a particular Network, so they're basically a combination of both the Address and Network classes. """ - if version: + if version is not None: if version == 4: return IPv4Interface(address) elif version == 6: return IPv6Interface(address) + else: + raise ValueError() try: return IPv4Interface(address) @@ -163,40 +170,44 @@ def v4_int_to_packed(address): - """The binary representation of this address. + """Represent an address as 4 packed bytes in network (big-endian) order. Args: address: An integer representation of an IPv4 IP address. Returns: - The binary representation of this address. + The integer address packed as 4 bytes in network (big-endian) order. Raises: - ValueError: If the integer is too large to be an IPv4 IP - address. + ValueError: If the integer is negative or too large to be an + IPv4 IP address. """ - if address > _BaseV4._ALL_ONES: - raise ValueError('Address too large for IPv4') - return struct.pack('!I', address) + try: + return struct.pack('!I', address) + except: + raise ValueError("Address negative or too large for IPv4") def v6_int_to_packed(address): - """The binary representation of this address. + """Represent an address as 16 packed bytes in network (big-endian) order. Args: address: An integer representation of an IPv4 IP address. Returns: - The binary representation of this address. + The integer address packed as 16 bytes in network (big-endian) order. """ - return struct.pack('!QQ', address >> 64, address & (2**64 - 1)) + try: + return struct.pack('!QQ', address >> 64, address & (2**64 - 1)) + except: + raise ValueError("Address negative or too large for IPv6") def _find_address_range(addresses): - """Find a sequence of addresses. + """Find a sequence of IPv#Address. Args: - addresses: a list of IPv4 or IPv6 addresses. + addresses: a list of IPv#Address objects. Returns: A tuple containing the first and last IP addresses in the sequence. diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -780,6 +780,12 @@ self.assertEqual(self.ipv4_address.version, 4) self.assertEqual(self.ipv6_address.version, 6) + with self.assertRaises(ValueError): + ipaddress.ip_address('1', version=[]) + + with self.assertRaises(ValueError): + ipaddress.ip_address('1', version=5) + def testMaxPrefixLength(self): self.assertEqual(self.ipv4_interface.max_prefixlen, 32) self.assertEqual(self.ipv6_interface.max_prefixlen, 128) @@ -1048,6 +1054,11 @@ self.assertEqual(ipaddress.ip_network(1).version, 4) self.assertEqual(ipaddress.ip_network(1, version=6).version, 6) + with self.assertRaises(ValueError): + ipaddress.ip_network(1, version='l') + with self.assertRaises(ValueError): + ipaddress.ip_network(1, version=3) + def testWithStar(self): self.assertEqual(str(self.ipv4_interface.with_prefixlen), "1.2.3.4/24") self.assertEqual(str(self.ipv4_interface.with_netmask), @@ -1137,6 +1148,13 @@ sixtofouraddr.sixtofour) self.assertFalse(bad_addr.sixtofour) + def testIpInterfaceVersion(self): + with self.assertRaises(ValueError): + ipaddress.ip_interface(1, version=123) + + with self.assertRaises(ValueError): + ipaddress.ip_interface(1, version='') + if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 23:17:44 2012 From: python-checkins at python.org (sandro.tosi) Date: Wed, 23 May 2012 23:17:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_minor_imp?= =?utf8?q?rovements_as_suggested_by_Hynek_Schlawack?= Message-ID: http://hg.python.org/cpython/rev/f4f2139202c5 changeset: 77117:f4f2139202c5 user: Sandro Tosi date: Wed May 23 23:17:22 2012 +0200 summary: Issue #14814: minor improvements as suggested by Hynek Schlawack files: Lib/ipaddress.py | 15 +++++++++------ 1 files changed, 9 insertions(+), 6 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1,5 +1,3 @@ -#!/usr/bin/python3 -# # Copyright 2007 Google Inc. # Licensed to PSF under a Contributor Agreement. # @@ -146,6 +144,7 @@ The IPv?Interface classes describe an Address on a particular Network, so they're basically a combination of both the Address and Network classes. + """ if version is not None: if version == 4: @@ -181,6 +180,7 @@ Raises: ValueError: If the integer is negative or too large to be an IPv4 IP address. + """ try: return struct.pack('!I', address) @@ -196,6 +196,7 @@ Returns: The integer address packed as 16 bytes in network (big-endian) order. + """ try: return struct.pack('!QQ', address >> 64, address & (2**64 - 1)) @@ -221,6 +222,7 @@ break return (first, last) + def _get_prefix_length(number1, number2, bits): """Get the number of leading bits that are same for two numbers. @@ -238,6 +240,7 @@ return bits - i return 0 + def _count_righthand_zero_bits(number, bits): """Count the number of zero bits on the right hand side. @@ -319,6 +322,7 @@ first_int = current + 1 first = ip_address(first_int, version=first._version) + def _collapse_addresses_recursive(addresses): """Loops through the addresses, collapsing concurrent netblocks. @@ -632,8 +636,8 @@ def hosts(self): """Generate Iterator over usable hosts in a network. - This is like __iter__ except it doesn't return the network - or broadcast addresses. + This is like __iter__ except it doesn't return the network + or broadcast addresses. """ cur = int(self.network_address) + 1 @@ -1965,8 +1969,7 @@ class IPv6Address(_BaseV6, _BaseAddress): - """Represent and manipulate single IPv6 Addresses. - """ + """Represent and manipulate single IPv6 Addresses.""" def __init__(self, address): """Instantiate a new IPv6 address object. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 23 23:19:20 2012 From: python-checkins at python.org (antoine.pitrou) Date: Wed, 23 May 2012 23:19:20 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0ODg4?= =?utf8?q?=3A_Fix_misbehaviour_of_the_=5Fmd5_module_when_called_on_data_la?= =?utf8?q?rger?= Message-ID: http://hg.python.org/cpython/rev/290d970c011d changeset: 77118:290d970c011d branch: 2.7 parent: 77105:086afe7b61f5 user: Antoine Pitrou date: Wed May 23 23:16:14 2012 +0200 summary: Issue #14888: Fix misbehaviour of the _md5 module when called on data larger than 2**32 bytes. files: Misc/NEWS | 3 +++ Modules/md5module.c | 17 ++++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue #14888: Fix misbehaviour of the _md5 module when called on data + larger than 2**32 bytes. + - Issue #14875: Use float('inf') instead of float('1e66666') in the json module. - Issue #14572: Prevent build failures with pre-3.5.0 versions of diff --git a/Modules/md5module.c b/Modules/md5module.c --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -262,6 +262,8 @@ { md5object *md5p; Py_buffer view = { 0 }; + Py_ssize_t n; + unsigned char *buf; if (!PyArg_ParseTuple(args, "|s*:new", &view)) return NULL; @@ -271,9 +273,18 @@ return NULL; } - if (view.len > 0) { - md5_append(&md5p->md5, (unsigned char*)view.buf, - Py_SAFE_DOWNCAST(view.len, Py_ssize_t, unsigned int)); + n = view.len; + buf = (unsigned char *) view.buf; + while (n > 0) { + Py_ssize_t nbytes; + if (n > INT_MAX) + nbytes = INT_MAX; + else + nbytes = n; + md5_append(&md5p->md5, buf, + Py_SAFE_DOWNCAST(nbytes, Py_ssize_t, unsigned int)); + buf += nbytes; + n -= nbytes; } PyBuffer_Release(&view); -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Thu May 24 05:50:52 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 24 May 2012 05:50:52 +0200 Subject: [Python-checkins] Daily reference leaks (f4f2139202c5): sum=0 Message-ID: results for f4f2139202c5 on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogMHQ440', '-x'] From python-checkins at python.org Thu May 24 11:14:21 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 24 May 2012 11:14:21 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Make_pep-0402=2Etxt_non-execut?= =?utf8?q?able=2E?= Message-ID: http://hg.python.org/peps/rev/d5ab78b95f41 changeset: 4421:d5ab78b95f41 user: Eric V. Smith date: Thu May 24 05:14:14 2012 -0400 summary: Make pep-0402.txt non-executable. files: pep-0402.txt | 0 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/pep-0402.txt b/pep-0402.txt old mode 100755 new mode 100644 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 12:10:30 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 24 May 2012 12:10:30 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Added_examples=2E?= Message-ID: http://hg.python.org/peps/rev/b0ad3d1dc002 changeset: 4422:b0ad3d1dc002 user: Eric V. Smith date: Thu May 24 06:10:10 2012 -0400 summary: Added examples. files: pep-0420.txt | 94 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 94 insertions(+), 0 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -315,6 +315,100 @@ directory on ``sys.path``, and have the "foo.baz" portion provided in a zip file, also on ``sys.path``. + +Examples +======== + +Nested namespace packages +------------------------- + +This example uses the following directory structure:: + + Lib/test/namspace_pkgs + parent1 + parent + child + one.py + parent2 + parent + child + two.py + +Here, both parent and child are namespace packages: Portions of them +exist in different directories, and they do not have ``__init__.py`` +files. + +Here we add the parent directories to ``sys.path``, and show that the +portions are correctly found:: + + >>> import sys + >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] + >>> import parent.child.one + >>> parent.__path__ + _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent']) + >>> parent.child.__path__ + _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent/child', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent/child']) + >>> import parent.child.two + +Dynamic path computation +------------------------ + +This example uses a similar directory structure, but adds a third +portion:: + + Lib/test/namspace_pkgs + parent1 + parent + child + one.py + parent2 + parent + child + two.py + parent3 + parent + child + three.py + +We add the first two parent paths to ``sys.path``. The third portion +is added dynamically to the parent's ``__path__``, and the third +portion is found when it is imported:: + + # add the first two parent paths to sys.path + >>> import sys + >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] + + # parent.child.one can be imported, because parent1/parent was added to sys.path: + >>> import parent.child.one + >>> parent.__path__ + _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent']) + + # parent.child.__path__ contains parent1/parent/child and parent2/parent/child, but not parent3/parent/child: + >>> parent.child.__path__ + _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent/child', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent/child']) + + # parent.child.two can be imported, because parent2/parent was added to sys.path: + >>> import parent.child.two + + # we cannot import parent.child.three, because parent3 is not in the path: + >>> import parent.child.three + Traceback (most recent call last): + File "", line 1, in + File "", line 1286, in _find_and_load + File "", line 1250, in _find_and_load_unlocked + ImportError: No module named 'parent.child.three' + + # now add parent3 to the parent's __path__: + >>> parent.__path__.append('Lib/test/namespace_pkgs/parent3/parent') + + # and now parent.child.three can be imported: + >>> import parent.child.three + + # and parent3/parent/child has dynamically been added to parent.child.__path__ + >>> parent.child.__path__ + _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent/child', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent/child', 'Lib/test/namespace_pkgs/parent3/parent/child']) + + Discussion ========== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 12:15:40 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 24 May 2012 12:15:40 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Clarifications=2E?= Message-ID: http://hg.python.org/peps/rev/a3fe77e6714c changeset: 4423:a3fe77e6714c user: Eric V. Smith date: Thu May 24 06:15:35 2012 -0400 summary: Clarifications. files: pep-0420.txt | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -370,9 +370,9 @@ child three.py -We add the first two parent paths to ``sys.path``. The third portion -is added dynamically to the parent's ``__path__``, and the third -portion is found when it is imported:: +We add the first two parent paths to ``sys.path``. The third +``parent`` portion is added dynamically to ``parent.__path__``, and +the third portion is then found when it is imported:: # add the first two parent paths to sys.path >>> import sys @@ -398,7 +398,7 @@ File "", line 1250, in _find_and_load_unlocked ImportError: No module named 'parent.child.three' - # now add parent3 to the parent's __path__: + # now add the third parent portion to parent.__path__: >>> parent.__path__.append('Lib/test/namespace_pkgs/parent3/parent') # and now parent.child.three can be imported: -- Repository URL: http://hg.python.org/peps From ncoghlan at gmail.com Thu May 24 14:47:14 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 24 May 2012 22:47:14 +1000 Subject: [Python-checkins] peps: Added examples. In-Reply-To: References: Message-ID: On Thu, May 24, 2012 at 8:10 PM, eric.smith wrote: > + ? Lib/test/namspace_pkgs Typo: s/namspace/namespace/ > +Here we add the parent directories to ``sys.path``, and show that the > +portions are correctly found:: > + > + ? ?>>> import sys > + ? ?>>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] The trailing "/parent" shouldn't be there on either of these paths. The comments that refer back to these also need the same adjustment. > + ? Lib/test/namspace_pkgs Same typo as above. > + ? ?# add the first two parent paths to sys.path > + ? ?>>> import sys > + ? ?>>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] Again, need to lose the last directory from these paths and the comments that refer to them. > + ? ?# now add parent3 to the parent's __path__: > + ? ?>>> parent.__path__.append('Lib/test/namespace_pkgs/parent3/parent') This modification is incorrect, it should be: sys.path.append('Lib/test/namespace_pkgs/parent3') and both parent.__path__ and parent.child.__path__ should pick up their extra portions on the next import attempt. Also, I suggest renaming "parent1", "parent2" and "parent3" to "project1", "project2" and "project3". Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From eric at trueblade.com Thu May 24 14:52:14 2012 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 24 May 2012 08:52:14 -0400 Subject: [Python-checkins] peps: Added examples. In-Reply-To: References: Message-ID: <4FBE2EFE.6020205@trueblade.com> On 05/24/2012 08:47 AM, Nick Coghlan wrote: > On Thu, May 24, 2012 at 8:10 PM, eric.smith wrote: >> + Lib/test/namspace_pkgs > > Typo: s/namspace/namespace/ Thanks. > The trailing "/parent" shouldn't be there on either of these paths. > The comments that refer back to these also need the same adjustment. Hmm, you're right. But I took this from a command session. These commands really did work. I'll investigate. > Also, I suggest renaming "parent1", "parent2" and "parent3" to > "project1", "project2" and "project3". Excellent idea. Thanks Eric. From python-checkins at python.org Thu May 24 15:23:36 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 24 May 2012 15:23:36 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_errors_found_by_Nick=2E?= Message-ID: http://hg.python.org/peps/rev/ad4b5d2924f3 changeset: 4424:ad4b5d2924f3 user: Eric V. Smith date: Thu May 24 09:23:31 2012 -0400 summary: Fix errors found by Nick. files: pep-0420.txt | 43 +++++++++++++++++++++------------------ 1 files changed, 23 insertions(+), 20 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -324,12 +324,12 @@ This example uses the following directory structure:: - Lib/test/namspace_pkgs - parent1 + Lib/test/namespace_pkgs + project1 parent child one.py - parent2 + project2 parent child two.py @@ -342,13 +342,14 @@ portions are correctly found:: >>> import sys - >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] + >>> sys.path += ['Lib/test/namespace_pkgs/project1', 'Lib/test/namespace_pkgs/project2'] >>> import parent.child.one >>> parent.__path__ - _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent']) + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent']) >>> parent.child.__path__ - _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent/child', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent/child']) + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child']) >>> import parent.child.two + >>> Dynamic path computation ------------------------ @@ -356,16 +357,16 @@ This example uses a similar directory structure, but adds a third portion:: - Lib/test/namspace_pkgs - parent1 + Lib/test/namespace_pkgs + project1 parent child one.py - parent2 + project2 parent child two.py - parent3 + project3 parent child three.py @@ -376,21 +377,21 @@ # add the first two parent paths to sys.path >>> import sys - >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] + >>> sys.path += ['Lib/test/namespace_pkgs/project1', 'Lib/test/namespace_pkgs/project2'] - # parent.child.one can be imported, because parent1/parent was added to sys.path: + # parent.child.one can be imported, because project1 was added to sys.path: >>> import parent.child.one >>> parent.__path__ - _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent']) + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent']) - # parent.child.__path__ contains parent1/parent/child and parent2/parent/child, but not parent3/parent/child: + # parent.child.__path__ contains project1/parent/child and project2/parent/child, but not project3/parent/child: >>> parent.child.__path__ - _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent/child', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent/child']) + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child']) - # parent.child.two can be imported, because parent2/parent was added to sys.path: + # parent.child.two can be imported, because project2 was added to sys.path: >>> import parent.child.two - # we cannot import parent.child.three, because parent3 is not in the path: + # we cannot import parent.child.three, because project3 is not in the path: >>> import parent.child.three Traceback (most recent call last): File "", line 1, in @@ -399,14 +400,16 @@ ImportError: No module named 'parent.child.three' # now add the third parent portion to parent.__path__: - >>> parent.__path__.append('Lib/test/namespace_pkgs/parent3/parent') + >>> parent.__path__.append('Lib/test/namespace_pkgs/project3/parent') + >>> parent.__path__ + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent', 'Lib/test/namespace_pkgs/project3/parent']) # and now parent.child.three can be imported: >>> import parent.child.three - # and parent3/parent/child has dynamically been added to parent.child.__path__ + # and project3/parent/child has dynamically been added to parent.child.__path__ >>> parent.child.__path__ - _NamespacePath(['/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent1/parent/child', '/home/eric/local/python/pep-420/Lib/test/namespace_pkgs/parent2/parent/child', 'Lib/test/namespace_pkgs/parent3/parent/child']) + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child', 'Lib/test/namespace_pkgs/project3/parent/child']) Discussion -- Repository URL: http://hg.python.org/peps From eric at trueblade.com Thu May 24 15:22:46 2012 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 24 May 2012 09:22:46 -0400 Subject: [Python-checkins] peps: Added examples. In-Reply-To: References: Message-ID: <4FBE3626.1030605@trueblade.com> On 05/24/2012 08:47 AM, Nick Coghlan wrote: >> + >>> import sys >> + >>> sys.path += ['Lib/test/namespace_pkgs/parent1/parent', 'Lib/test/namespace_pkgs/parent2/parent'] > > The trailing "/parent" shouldn't be there on either of these paths. > The comments that refer back to these also need the same adjustment. True. I had stupidly started with PYTHONPATH including the parent1 and parent2 directories, so this really wasn't doing anything. > >> + # now add parent3 to the parent's __path__: >> + >>> parent.__path__.append('Lib/test/namespace_pkgs/parent3/parent') > > This modification is incorrect, it should be: > sys.path.append('Lib/test/namespace_pkgs/parent3') Either one will work. >>> import unittest >>> unittest.__path__ ['/home/eric/local/python/pep-420/Lib/unittest'] Note that unittest.__path__ ends in Lib/unittest, despite it being only Lib that was on PYTHONPATH. The same holds true for modifying parent.__path__: it needs to list the directory names where child portions will be found. >>> sys.path += ['Lib/test/namespace_pkgs/project1', 'Lib/test/namespace_pkgs/project2'] >>> import parent >>> parent.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent']) Here, the paths contain the places from which portions of parent will be loaded. So to find more portions of parent in project3, it's project3/parent that has to be added to parent.__path__. Maybe it would be clearer if I included other modules in parent itself, not just in parent.child. Possibly I am being too tricky here by modifying parent.__path__, and I should just modify sys.path again, as you suggest. But I was trying to show that modifying parent.__path__ will also work. Eric. From python-checkins at python.org Thu May 24 15:57:54 2012 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 24 May 2012 15:57:54 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0MDM2?= =?utf8?q?=3A_return_None_when_port_in_urlparse_cross_65535?= Message-ID: http://hg.python.org/cpython/rev/988903cf24c5 changeset: 77119:988903cf24c5 branch: 2.7 user: Senthil Kumaran date: Thu May 24 21:54:34 2012 +0800 summary: Issue #14036: return None when port in urlparse cross 65535 files: Lib/test/test_urlparse.py | 5 +++++ Lib/urlparse.py | 8 +++++--- Misc/NEWS | 3 +++ 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -437,6 +437,11 @@ self.assertEqual(p.port, 80) self.assertEqual(p.geturl(), url) + # Verify an illegal port of value greater than 65535 is set as None + url = "http://www.python.org:65536" + p = urlparse.urlsplit(url) + self.assertEqual(p.port, None) + def test_issue14072(self): p1 = urlparse.urlsplit('tel:+31-641044153') self.assertEqual(p1.scheme, 'tel') diff --git a/Lib/urlparse.py b/Lib/urlparse.py --- a/Lib/urlparse.py +++ b/Lib/urlparse.py @@ -97,9 +97,11 @@ netloc = self.netloc.split('@')[-1].split(']')[-1] if ':' in netloc: port = netloc.split(':')[1] - return int(port, 10) - else: - return None + port = int(port, 10) + # verify legal port + if (0 <= port <= 65535): + return port + return None from collections import namedtuple diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue #14036: Add an additional check to validate that port in urlparse does + not go in illegal range and returns None. + - Issue #14888: Fix misbehaviour of the _md5 module when called on data larger than 2**32 bytes. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 24 15:57:55 2012 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 24 May 2012 15:57:55 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0MDM2?= =?utf8?q?=3A_return_None_when_port_in_urlparse_cross_65535?= Message-ID: http://hg.python.org/cpython/rev/d769e64aed79 changeset: 77120:d769e64aed79 branch: 3.2 parent: 77114:de5cb14e929e user: Senthil Kumaran date: Thu May 24 21:56:17 2012 +0800 summary: Issue #14036: return None when port in urlparse cross 65535 files: Lib/test/test_urlparse.py | 5 +++++ Lib/urllib/parse.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -524,6 +524,11 @@ self.assertEqual(p.port, 80) self.assertEqual(p.geturl(), url) + # Verify an illegal port is returned as None + url = b"HTTP://WWW.PYTHON.ORG:65536/doc/#frag" + p = urllib.parse.urlsplit(url) + self.assertEqual(p.port, None) + def test_attributes_bad_port(self): """Check handling of non-integer ports.""" p = urllib.parse.urlsplit("http://www.example.net:foo") diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -143,6 +143,9 @@ port = self._hostinfo[1] if port is not None: port = int(port, 10) + # Return None on an illegal port + if not ( 0 <= port <= 65535): + return None return port diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14036: Add an additional check to validate that port in urlparse does + not go in illegal range and returns None. + - Issue #14875: Use float('inf') instead of float('1e66666') in the json module. - Issue #14426: Correct the Date format in Expires attribute of Set-Cookie -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 24 15:57:56 2012 From: python-checkins at python.org (senthil.kumaran) Date: Thu, 24 May 2012 15:57:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314036=3A_return_None_when_port_in_urlparse_cross_65?= =?utf8?q?535?= Message-ID: http://hg.python.org/cpython/rev/b4d257c64db7 changeset: 77121:b4d257c64db7 parent: 77117:f4f2139202c5 parent: 77120:d769e64aed79 user: Senthil Kumaran date: Thu May 24 21:57:38 2012 +0800 summary: Issue #14036: return None when port in urlparse cross 65535 files: Lib/test/test_urlparse.py | 5 +++++ Lib/urllib/parse.py | 3 +++ Misc/NEWS | 3 +++ 3 files changed, 11 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -524,6 +524,11 @@ self.assertEqual(p.port, 80) self.assertEqual(p.geturl(), url) + # Verify an illegal port is returned as None + url = b"HTTP://WWW.PYTHON.ORG:65536/doc/#frag" + p = urllib.parse.urlsplit(url) + self.assertEqual(p.port, None) + def test_attributes_bad_port(self): """Check handling of non-integer ports.""" p = urllib.parse.urlsplit("http://www.example.net:foo") diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -143,6 +143,9 @@ port = self._hostinfo[1] if port is not None: port = int(port, 10) + # Return None on an illegal port + if not ( 0 <= port <= 65535): + return None return port diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,9 @@ Library ------- +- Issue #14036: Add an additional check to validate that port in urlparse does + not go in illegal range and returns None. + - Issue #14862: Add missing names to os.__all__ - Issue #14875: Use float('inf') instead of float('1e66666') in the json module. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 24 20:24:41 2012 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 24 May 2012 20:24:41 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Accept_PEP_418=2E_Congrats_Vic?= =?utf8?q?tor_and_others!?= Message-ID: http://hg.python.org/peps/rev/5579d601a9a9 changeset: 4425:5579d601a9a9 parent: 4322:a9a6a0851544 user: Guido van Rossum date: Sun Apr 29 07:39:18 2012 -0700 summary: Accept PEP 418. Congrats Victor and others! files: pep-0418.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0418.txt b/pep-0418.txt --- a/pep-0418.txt +++ b/pep-0418.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Cameron Simpson , Jim Jewett , Stephen J. Turnbull , Victor Stinner -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 26-March-2012 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 20:24:42 2012 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 24 May 2012 20:24:42 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Null_merge_to_get_rid_of_old_head=2E?= Message-ID: http://hg.python.org/peps/rev/9f0e3e034355 changeset: 4426:9f0e3e034355 parent: 4424:ad4b5d2924f3 parent: 4425:5579d601a9a9 user: Guido van Rossum date: Thu May 24 11:24:33 2012 -0700 summary: Null merge to get rid of old head. files: -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 20:32:03 2012 From: python-checkins at python.org (guido.van.rossum) Date: Thu, 24 May 2012 20:32:03 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Accept_PEP_420=2E?= Message-ID: http://hg.python.org/peps/rev/96e7d8b480ff changeset: 4427:96e7d8b480ff user: Guido van Rossum date: Thu May 24 11:31:45 2012 -0700 summary: Accept PEP 420. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -3,7 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Eric V. Smith -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 19-Apr-2012 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 20:50:23 2012 From: python-checkins at python.org (barry.warsaw) Date: Thu, 24 May 2012 20:50:23 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Add_Resolution_header?= Message-ID: http://hg.python.org/peps/rev/f2500baa8a51 changeset: 4428:f2500baa8a51 user: Barry Warsaw date: Thu May 24 14:24:40 2012 -0400 summary: Add Resolution header files: pep-0420.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -4,6 +4,7 @@ Last-Modified: $Date$ Author: Eric V. Smith Status: Accepted +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119651.html Type: Standards Track Content-Type: text/x-rst Created: 19-Apr-2012 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 20:52:57 2012 From: python-checkins at python.org (petri.lehtinen) Date: Thu, 24 May 2012 20:52:57 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE0ODYzOiBVcGRh?= =?utf8?q?te_the_documentation_of_os=2Efdopen=28=29?= Message-ID: http://hg.python.org/cpython/rev/9ef2cb56926d changeset: 77122:9ef2cb56926d branch: 3.2 parent: 77120:d769e64aed79 user: Petri Lehtinen date: Thu May 24 21:44:07 2012 +0300 summary: #14863: Update the documentation of os.fdopen() There's no bufsize argument anymore, and os.fdopen() is only a very thin wrapper around open() anymore. Change the documentation to reflect that. files: Doc/library/os.rst | 22 ++++++---------------- Misc/NEWS | 3 +++ 2 files changed, 9 insertions(+), 16 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -545,22 +545,12 @@ These functions create new :term:`file objects `. (See also :func:`open`.) -.. function:: fdopen(fd[, mode[, bufsize]]) - - .. index:: single: I/O control; buffering - - Return an open file object connected to the file descriptor *fd*. The *mode* - and *bufsize* arguments have the same meaning as the corresponding arguments to - the built-in :func:`open` function. - - When specified, the *mode* argument must start with one of the letters - ``'r'``, ``'w'``, or ``'a'``, otherwise a :exc:`ValueError` is raised. - - On Unix, when the *mode* argument starts with ``'a'``, the *O_APPEND* flag is - set on the file descriptor (which the :c:func:`fdopen` implementation already - does on most platforms). - - Availability: Unix, Windows. +.. function:: fdopen(fd, *args, **kwargs) + + Return an open file object connected to the file descriptor *fd*. + This is an alias of :func:`open` and accepts the same arguments. + The only difference is that the first argument of :func:`fdopen` + must always be an integer. .. _os-fd-ops: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14863: Update the documentation of os.fdopen() to reflect the + fact that it's only a thin wrapper around open() anymore. + - Issue #14036: Add an additional check to validate that port in urlparse does not go in illegal range and returns None. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 24 20:53:01 2012 From: python-checkins at python.org (petri.lehtinen) Date: Thu, 24 May 2012 20:53:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=2314863=3A_Update_the_documentation_of_os=2Efdopen=28=29?= Message-ID: http://hg.python.org/cpython/rev/f27e098a774a changeset: 77123:f27e098a774a parent: 77121:b4d257c64db7 parent: 77122:9ef2cb56926d user: Petri Lehtinen date: Thu May 24 21:49:59 2012 +0300 summary: #14863: Update the documentation of os.fdopen() files: Doc/library/os.rst | 27 +++++++-------------------- Misc/NEWS | 3 +++ 2 files changed, 10 insertions(+), 20 deletions(-) diff --git a/Doc/library/os.rst b/Doc/library/os.rst --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -600,26 +600,13 @@ These functions create new :term:`file objects `. (See also :func:`open`.) -.. function:: fdopen(fd[, mode[, bufsize]]) - - .. index:: single: I/O control; buffering - - Return an open file object connected to the file descriptor *fd*. The *mode* - and *bufsize* arguments have the same meaning as the corresponding arguments to - the built-in :func:`open` function. - - When specified, the *mode* argument must start with one of the letters - ``'r'``, ``'w'``, ``'x'`` or ``'a'``, otherwise a :exc:`ValueError` is - raised. - - On Unix, when the *mode* argument starts with ``'a'``, the *O_APPEND* flag is - set on the file descriptor (which the :c:func:`fdopen` implementation already - does on most platforms). - - Availability: Unix, Windows. - - .. versionchanged:: 3.3 - The ``'x'`` mode was added. +.. function:: fdopen(fd, *args, **kwargs) + + Return an open file object connected to the file descriptor *fd*. + This is an alias of :func:`open` and accepts the same arguments. + The only difference is that the first argument of :func:`fdopen` + must always be an integer. + .. _os-fd-ops: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -42,6 +42,9 @@ Library ------- +- Issue #14863: Update the documentation of os.fdopen() to reflect the + fact that it's only a thin wrapper around open() anymore. + - Issue #14036: Add an additional check to validate that port in urlparse does not go in illegal range and returns None. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 24 22:11:43 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 24 May 2012 22:11:43 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_memory_cons?= =?utf8?q?traint_for_test=5Fdecodeascii?= Message-ID: http://hg.python.org/cpython/rev/5107782c4dd8 changeset: 77124:5107782c4dd8 branch: 2.7 parent: 77119:988903cf24c5 user: Antoine Pitrou date: Thu May 24 22:08:51 2012 +0200 summary: Fix memory constraint for test_decodeascii files: Lib/test/test_bigmem.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_bigmem.py b/Lib/test/test_bigmem.py --- a/Lib/test/test_bigmem.py +++ b/Lib/test/test_bigmem.py @@ -118,7 +118,7 @@ except MemoryError: pass # acceptable on 32-bit - @precisionbigmemtest(size=_2G-1, memuse=2) + @precisionbigmemtest(size=_2G-1, memuse=4) def test_decodeascii(self, size): return self.basic_encode_test(size, 'ascii', c='A') -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Thu May 24 22:13:33 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 May 2012 06:13:33 +1000 Subject: [Python-checkins] peps: Added examples. In-Reply-To: <4FBE3626.1030605@trueblade.com> References: <4FBE3626.1030605@trueblade.com> Message-ID: On May 24, 2012 11:29 PM, "Eric V. Smith" wrote: > > Possibly I am being too tricky here by modifying parent.__path__, and I > should just modify sys.path again, as you suggest. But I was trying to > show that modifying parent.__path__ will also work. Modifying namespace package __path__ attributes directly seems like a good way to accidentally break the auto-updating. We probably don't want to encourage that. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu May 24 22:17:04 2012 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 24 May 2012 16:17:04 -0400 Subject: [Python-checkins] peps: Added examples. In-Reply-To: References: <4FBE3626.1030605@trueblade.com> Message-ID: <4FBE9740.2060004@trueblade.com> On 5/24/2012 4:13 PM, Nick Coghlan wrote: > > On May 24, 2012 11:29 PM, "Eric V. Smith" > wrote: >> >> Possibly I am being too tricky here by modifying parent.__path__, and I >> should just modify sys.path again, as you suggest. But I was trying to >> show that modifying parent.__path__ will also work. > > Modifying namespace package __path__ attributes directly seems like a > good way to accidentally break the auto-updating. We probably don't want > to encourage that. Well, it works if you modify it in place (so I added .extend() to _NamespacePath). But I agree, it's not a great example. I'll modify it. From python-checkins at python.org Thu May 24 22:33:05 2012 From: python-checkins at python.org (antoine.pitrou) Date: Thu, 24 May 2012 22:33:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_other_test_?= =?utf8?q?requirements=2E?= Message-ID: http://hg.python.org/cpython/rev/13197ce81b0d changeset: 77125:13197ce81b0d branch: 2.7 user: Antoine Pitrou date: Thu May 24 22:30:19 2012 +0200 summary: Fix other test requirements. files: Lib/test/test_bigmem.py | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_bigmem.py b/Lib/test/test_bigmem.py --- a/Lib/test/test_bigmem.py +++ b/Lib/test/test_bigmem.py @@ -485,7 +485,7 @@ self.assertEqual(s.count('.'), 3) self.assertEqual(s.count('-'), size * 2) - @bigmemtest(minsize=_2G + 10, memuse=2) + @bigmemtest(minsize=_2G + 10, memuse=5) def test_repr_small(self, size): s = '-' * size s = repr(s) @@ -497,7 +497,6 @@ # repr() will create a string four times as large as this 'binary # string', but we don't want to allocate much more than twice # size in total. (We do extra testing in test_repr_large()) - size = size // 5 * 2 s = '\x00' * size s = repr(s) self.assertEqual(len(s), size * 4 + 2) @@ -541,7 +540,7 @@ self.assertEqual(len(s), size * 2) self.assertEqual(s.count('.'), size * 2) - @bigmemtest(minsize=_2G + 20, memuse=1) + @bigmemtest(minsize=_2G + 20, memuse=2) def test_slice_and_getitem(self, size): SUBSTR = '0123456789' sublen = len(SUBSTR) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 24 23:17:19 2012 From: python-checkins at python.org (nick.coghlan) Date: Thu, 24 May 2012 23:17:19 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_The_PEP_0_generator_requires_t?= =?utf8?q?hat_headers_appear_in_the_order_given_in_PEP_1?= Message-ID: http://hg.python.org/peps/rev/5e1662a81157 changeset: 4429:5e1662a81157 user: Nick Coghlan date: Fri May 25 07:17:07 2012 +1000 summary: The PEP 0 generator requires that headers appear in the order given in PEP 1 files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -4,12 +4,12 @@ Last-Modified: $Date$ Author: Eric V. Smith Status: Accepted -Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119651.html Type: Standards Track Content-Type: text/x-rst Created: 19-Apr-2012 Python-Version: 3.3 Post-History: +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119651.html Abstract ======== -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 23:46:21 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 24 May 2012 23:46:21 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Improved_example_to_just_focus?= =?utf8?q?_on_sys=2Epath_manipulation=2E?= Message-ID: http://hg.python.org/peps/rev/4032557ae02c changeset: 4430:4032557ae02c user: Eric V. Smith date: Thu May 24 17:46:07 2012 -0400 summary: Improved example to just focus on sys.path manipulation. files: pep-0420.txt | 22 ++++++++++++++-------- 1 files changed, 14 insertions(+), 8 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -372,9 +372,11 @@ child three.py -We add the first two parent paths to ``sys.path``. The third -``parent`` portion is added dynamically to ``parent.__path__``, and -the third portion is then found when it is imported:: +We add ``project1`` and ``project2`` to ``sys.path``, then import +``parent.child.one`` and ``parent.child.two``. Then we add the +``project3`` to ``sys.path`` and when ``parent.child.three`` is +imported, ``project3/parent`` is automatically added to +``parent.__path__``. # add the first two parent paths to sys.path >>> import sys @@ -400,17 +402,21 @@ File "", line 1250, in _find_and_load_unlocked ImportError: No module named 'parent.child.three' - # now add the third parent portion to parent.__path__: - >>> parent.__path__.append('Lib/test/namespace_pkgs/project3/parent') - >>> parent.__path__ - _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent', 'Lib/test/namespace_pkgs/project3/parent']) + # now add project3 to sys.path: + >>> sys.path.append('Lib/test/namespace_pkgs/project3') # and now parent.child.three can be imported: >>> import parent.child.three - # and project3/parent/child has dynamically been added to parent.child.__path__ + # project3/parent has been added to parent.__path__: + >>> parent.__path__ + _NamespacePath(['Lib/test/namespace_pkgs/project1/parent', 'Lib/test/namespace_pkgs/project2/parent', 'Lib/test/namespace_pkgs/project3/parent']) + + # and project3/parent/child has been added to parent.child.__path__ >>> parent.child.__path__ _NamespacePath(['Lib/test/namespace_pkgs/project1/parent/child', 'Lib/test/namespace_pkgs/project2/parent/child', 'Lib/test/namespace_pkgs/project3/parent/child']) + >>> + Discussion -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Thu May 24 23:59:06 2012 From: python-checkins at python.org (eric.smith) Date: Thu, 24 May 2012 23:59:06 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_markup=2E?= Message-ID: http://hg.python.org/peps/rev/f6f5604b0946 changeset: 4431:f6f5604b0946 user: Eric V. Smith date: Thu May 24 17:58:59 2012 -0400 summary: Fix markup. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -376,7 +376,7 @@ ``parent.child.one`` and ``parent.child.two``. Then we add the ``project3`` to ``sys.path`` and when ``parent.child.three`` is imported, ``project3/parent`` is automatically added to -``parent.__path__``. +``parent.__path__``:: # add the first two parent paths to sys.path >>> import sys -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 02:22:22 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 25 May 2012 02:22:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_issue_14660=3A_Implement_PE?= =?utf8?q?P_420=2C_namespace_packages=2E?= Message-ID: http://hg.python.org/cpython/rev/702009f3c0b1 changeset: 77126:702009f3c0b1 parent: 77123:f27e098a774a user: Eric V. Smith date: Thu May 24 20:21:04 2012 -0400 summary: issue 14660: Implement PEP 420, namespace packages. files: Lib/importlib/_bootstrap.py | 154 +++++- Lib/importlib/test/frozen/test_loader.py | 28 +- Lib/importlib/test/source/test_finder.py | 19 - Lib/pkgutil.py | 26 +- Lib/test/namespace_pkgs/both_portions/foo/one.py | 1 + Lib/test/namespace_pkgs/both_portions/foo/two.py | 1 + Lib/test/namespace_pkgs/missing_directory.zip | Bin Lib/test/namespace_pkgs/nested_portion1.zip | Bin Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py | 0 Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py | 1 + Lib/test/namespace_pkgs/portion1/foo/one.py | 1 + Lib/test/namespace_pkgs/portion2/foo/two.py | 1 + Lib/test/namespace_pkgs/project1/parent/child/one.py | 1 + Lib/test/namespace_pkgs/project2/parent/child/two.py | 1 + Lib/test/namespace_pkgs/project3/parent/child/three.py | 1 + Lib/test/namespace_pkgs/top_level_portion1.zip | Bin Lib/test/test_frozen.py | 2 +- Lib/test/test_import.py | 6 - Lib/test/test_module.py | 91 +++ Lib/test/test_namespace_pkgs.py | 239 ++++++++++ Lib/test/test_pkgutil.py | 64 +- Misc/NEWS | 2 + Modules/zipimport.c | 148 +++++- Objects/moduleobject.c | 33 +- Python/importlib.h | Bin 25 files changed, 736 insertions(+), 84 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -468,6 +468,10 @@ """ @classmethod + def module_repr(cls, module): + return "".format(module.__name__) + + @classmethod def find_module(cls, fullname, path=None): """Find the built-in module. @@ -521,6 +525,10 @@ """ @classmethod + def module_repr(cls, m): + return "".format(m.__name__) + + @classmethod def find_module(cls, fullname, path=None): """Find a frozen module.""" return cls if _imp.is_frozen(fullname) else None @@ -533,7 +541,10 @@ """Load a frozen module.""" is_reload = fullname in sys.modules try: - return _imp.init_frozen(fullname) + m = _imp.init_frozen(fullname) + # Let our own module_repr() method produce a suitable repr. + del m.__file__ + return m except: if not is_reload and fullname in sys.modules: del sys.modules[fullname] @@ -875,6 +886,79 @@ return None +class _NamespacePath: + """Represents a namespace package's path. It uses the module name + to find its parent module, and from there it looks up the parent's + __path__. When this changes, the module's own path is recomputed, + using path_finder. For top-leve modules, the parent module's path + is sys.path.""" + + def __init__(self, name, path, path_finder): + self._name = name + self._path = path + self._last_parent_path = tuple(self._get_parent_path()) + self._path_finder = path_finder + + def _find_parent_path_names(self): + """Returns a tuple of (parent-module-name, parent-path-attr-name)""" + parent, dot, me = self._name.rpartition('.') + if dot == '': + # This is a top-level module. sys.path contains the parent path. + return 'sys', 'path' + # Not a top-level module. parent-module.__path__ contains the + # parent path. + return parent, '__path__' + + def _get_parent_path(self): + parent_module_name, path_attr_name = self._find_parent_path_names() + return getattr(sys.modules[parent_module_name], path_attr_name) + + def _recalculate(self): + # If the parent's path has changed, recalculate _path + parent_path = tuple(self._get_parent_path()) # Make a copy + if parent_path != self._last_parent_path: + loader, new_path = self._path_finder(self._name, parent_path) + # Note that no changes are made if a loader is returned, but we + # do remember the new parent path + if loader is None: + self._path = new_path + self._last_parent_path = parent_path # Save the copy + return self._path + + def __iter__(self): + return iter(self._recalculate()) + + def __len__(self): + return len(self._recalculate()) + + def __repr__(self): + return "_NamespacePath({0!r})".format(self._path) + + def __contains__(self, item): + return item in self._recalculate() + + def append(self, item): + self._path.append(item) + + +class NamespaceLoader: + def __init__(self, name, path, path_finder): + self._path = _NamespacePath(name, path, path_finder) + + @classmethod + def module_repr(cls, module): + return "".format(module.__name__) + + @set_package + @set_loader + @module_for_loader + def load_module(self, module): + """Load a namespace module.""" + _verbose_message('namespace module loaded with path {!r}', self._path) + module.__path__ = self._path + return module + + # Finders ##################################################################### class PathFinder: @@ -916,19 +1000,46 @@ return finder @classmethod + def _get_loader(cls, fullname, path): + """Find the loader or namespace_path for this module/package name.""" + # If this ends up being a namespace package, namespace_path is + # the list of paths that will become its __path__ + namespace_path = [] + for entry in path: + finder = cls._path_importer_cache(entry) + if finder is not None: + if hasattr(finder, 'find_loader'): + loader, portions = finder.find_loader(fullname) + else: + loader = finder.find_module(fullname) + portions = [] + if loader is not None: + # We found a loader: return it immediately. + return (loader, namespace_path) + # This is possibly part of a namespace package. + # Remember these path entries (if any) for when we + # create a namespace package, and continue iterating + # on path. + namespace_path.extend(portions) + else: + return (None, namespace_path) + + @classmethod def find_module(cls, fullname, path=None): """Find the module on sys.path or 'path' based on sys.path_hooks and sys.path_importer_cache.""" if path is None: path = sys.path - for entry in path: - finder = cls._path_importer_cache(entry) - if finder is not None: - loader = finder.find_module(fullname) - if loader: - return loader + loader, namespace_path = cls._get_loader(fullname, path) + if loader is not None: + return loader else: - return None + if namespace_path: + # We found at least one namespace path. Return a + # loader which can create the namespace package. + return NamespaceLoader(fullname, namespace_path, cls._get_loader) + else: + return None class FileFinder: @@ -942,8 +1053,8 @@ def __init__(self, path, *details): """Initialize with the path to search on and a variable number of - 3-tuples containing the loader, file suffixes the loader recognizes, and - a boolean of whether the loader handles packages.""" + 3-tuples containing the loader, file suffixes the loader recognizes, + and a boolean of whether the loader handles packages.""" packages = [] modules = [] for loader, suffixes, supports_packages in details: @@ -964,6 +1075,19 @@ def find_module(self, fullname): """Try to find a loader for the specified module.""" + # Call find_loader(). If it returns a string (indicating this + # is a namespace package portion), generate a warning and + # return None. + loader, portions = self.find_loader(fullname) + assert len(portions) in [0, 1] + if loader is None and len(portions): + msg = "Not importing directory {}: missing __init__" + _warnings.warn(msg.format(portions[0]), ImportWarning) + return loader + + def find_loader(self, fullname): + """Try to find a loader for the specified module, or the namespace + package portions. Returns (loader, list-of-portions).""" tail_module = fullname.rpartition('.')[2] try: mtime = _os.stat(self.path).st_mtime @@ -987,17 +1111,17 @@ init_filename = '__init__' + suffix full_path = _path_join(base_path, init_filename) if _path_isfile(full_path): - return loader(fullname, full_path) + return (loader(fullname, full_path), [base_path]) else: - msg = "Not importing directory {}: missing __init__" - _warnings.warn(msg.format(base_path), ImportWarning) + # A namespace package, return the path + return (None, [base_path]) # Check for a file w/ a proper suffix exists. for suffix, loader in self.modules: if cache_module + suffix in cache: full_path = _path_join(self.path, tail_module + suffix) if _path_isfile(full_path): - return loader(fullname, full_path) - return None + return (loader(fullname, full_path), []) + return (None, []) def _fill_cache(self): """Fill the cache of potential modules and packages for this directory.""" diff --git a/Lib/importlib/test/frozen/test_loader.py b/Lib/importlib/test/frozen/test_loader.py --- a/Lib/importlib/test/frozen/test_loader.py +++ b/Lib/importlib/test/frozen/test_loader.py @@ -10,38 +10,46 @@ def test_module(self): with util.uncache('__hello__'), captured_stdout() as stdout: module = machinery.FrozenImporter.load_module('__hello__') - check = {'__name__': '__hello__', '__file__': '', - '__package__': '', '__loader__': machinery.FrozenImporter} + check = {'__name__': '__hello__', + '__package__': '', + '__loader__': machinery.FrozenImporter, + } for attr, value in check.items(): self.assertEqual(getattr(module, attr), value) self.assertEqual(stdout.getvalue(), 'Hello world!\n') + self.assertFalse(hasattr(module, '__file__')) def test_package(self): with util.uncache('__phello__'), captured_stdout() as stdout: module = machinery.FrozenImporter.load_module('__phello__') - check = {'__name__': '__phello__', '__file__': '', - '__package__': '__phello__', '__path__': ['__phello__'], - '__loader__': machinery.FrozenImporter} + check = {'__name__': '__phello__', + '__package__': '__phello__', + '__path__': ['__phello__'], + '__loader__': machinery.FrozenImporter, + } for attr, value in check.items(): attr_value = getattr(module, attr) self.assertEqual(attr_value, value, "for __phello__.%s, %r != %r" % (attr, attr_value, value)) self.assertEqual(stdout.getvalue(), 'Hello world!\n') + self.assertFalse(hasattr(module, '__file__')) def test_lacking_parent(self): with util.uncache('__phello__', '__phello__.spam'), \ captured_stdout() as stdout: module = machinery.FrozenImporter.load_module('__phello__.spam') - check = {'__name__': '__phello__.spam', '__file__': '', + check = {'__name__': '__phello__.spam', '__package__': '__phello__', - '__loader__': machinery.FrozenImporter} + '__loader__': machinery.FrozenImporter, + } for attr, value in check.items(): attr_value = getattr(module, attr) self.assertEqual(attr_value, value, "for __phello__.spam.%s, %r != %r" % (attr, attr_value, value)) self.assertEqual(stdout.getvalue(), 'Hello world!\n') + self.assertFalse(hasattr(module, '__file__')) def test_module_reuse(self): with util.uncache('__hello__'), captured_stdout() as stdout: @@ -51,6 +59,12 @@ self.assertEqual(stdout.getvalue(), 'Hello world!\nHello world!\n') + def test_module_repr(self): + with util.uncache('__hello__'), captured_stdout(): + module = machinery.FrozenImporter.load_module('__hello__') + self.assertEqual(repr(module), + "") + def test_state_after_failure(self): # No way to trigger an error in a frozen module. pass diff --git a/Lib/importlib/test/source/test_finder.py b/Lib/importlib/test/source/test_finder.py --- a/Lib/importlib/test/source/test_finder.py +++ b/Lib/importlib/test/source/test_finder.py @@ -106,36 +106,17 @@ loader = self.import_(pkg_dir, 'pkg.sub') self.assertTrue(hasattr(loader, 'load_module')) - # [sub empty] - def test_empty_sub_directory(self): - context = source_util.create_modules('pkg.__init__', 'pkg.sub.__init__') - with warnings.catch_warnings(): - warnings.simplefilter("error", ImportWarning) - with context as mapping: - os.unlink(mapping['pkg.sub.__init__']) - pkg_dir = os.path.dirname(mapping['pkg.__init__']) - with self.assertRaises(ImportWarning): - self.import_(pkg_dir, 'pkg.sub') - # [package over modules] def test_package_over_module(self): name = '_temp' loader = self.run_test(name, {'{0}.__init__'.format(name), name}) self.assertTrue('__init__' in loader.get_filename(name)) - def test_failure(self): with source_util.create_modules('blah') as mapping: nothing = self.import_(mapping['.root'], 'sdfsadsadf') self.assertTrue(nothing is None) - # [empty dir] - def test_empty_dir(self): - with warnings.catch_warnings(): - warnings.simplefilter("error", ImportWarning) - with self.assertRaises(ImportWarning): - self.run_test('pkg', {'pkg.__init__'}, unlink={'pkg.__init__'}) - def test_empty_string_for_dir(self): # The empty string from sys.path means to search in the cwd. finder = machinery.FileFinder('', (machinery.SourceFileLoader, diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py --- a/Lib/pkgutil.py +++ b/Lib/pkgutil.py @@ -515,19 +515,29 @@ pname = os.path.join(*name.split('.')) # Reconstitute as relative path sname_pkg = name + ".pkg" - init_py = "__init__.py" path = path[:] # Start with a copy of the existing path for dir in sys.path: - if not isinstance(dir, str) or not os.path.isdir(dir): + if not isinstance(dir, str): continue - subdir = os.path.join(dir, pname) - # XXX This may still add duplicate entries to path on - # case-insensitive filesystems - initfile = os.path.join(subdir, init_py) - if subdir not in path and os.path.isfile(initfile): - path.append(subdir) + + finder = get_importer(dir) + if finder is not None: + # Is this finder PEP 420 compliant? + if hasattr(finder, 'find_loader'): + loader, portions = finder.find_loader(name) + else: + # No, no need to call it + loader = None + portions = [] + + for portion in portions: + # XXX This may still add duplicate entries to path on + # case-insensitive filesystems + if portion not in path: + path.append(portion) + # XXX Is this the right thing for subpackages like zope.app? # It looks for a file named "zope.app.pkg" pkgfile = os.path.join(dir, sname_pkg) diff --git a/Lib/test/namespace_pkgs/both_portions/foo/one.py b/Lib/test/namespace_pkgs/both_portions/foo/one.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/both_portions/foo/one.py @@ -0,0 +1,1 @@ +attr = 'both_portions foo one' diff --git a/Lib/test/namespace_pkgs/both_portions/foo/two.py b/Lib/test/namespace_pkgs/both_portions/foo/two.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/both_portions/foo/two.py @@ -0,0 +1,1 @@ +attr = 'both_portions foo two' diff --git a/Lib/test/namespace_pkgs/missing_directory.zip b/Lib/test/namespace_pkgs/missing_directory.zip new file mode 100644 index 0000000000000000000000000000000000000000..836a9106bcdd5d171dcf54386bb51e0f7b6a520c GIT binary patch [stripped] diff --git a/Lib/test/namespace_pkgs/nested_portion1.zip b/Lib/test/namespace_pkgs/nested_portion1.zip new file mode 100644 index 0000000000000000000000000000000000000000..8d22406f23758f3d0d138cd49c651c5c52e1f84f GIT binary patch [stripped] diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py b/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py new file mode 100644 diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py b/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py @@ -0,0 +1,1 @@ +attr = 'portion1 foo one' diff --git a/Lib/test/namespace_pkgs/portion1/foo/one.py b/Lib/test/namespace_pkgs/portion1/foo/one.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/portion1/foo/one.py @@ -0,0 +1,1 @@ +attr = 'portion1 foo one' diff --git a/Lib/test/namespace_pkgs/portion2/foo/two.py b/Lib/test/namespace_pkgs/portion2/foo/two.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/portion2/foo/two.py @@ -0,0 +1,1 @@ +attr = 'portion2 foo two' diff --git a/Lib/test/namespace_pkgs/project1/parent/child/one.py b/Lib/test/namespace_pkgs/project1/parent/child/one.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/project1/parent/child/one.py @@ -0,0 +1,1 @@ +attr = 'parent child one' diff --git a/Lib/test/namespace_pkgs/project2/parent/child/two.py b/Lib/test/namespace_pkgs/project2/parent/child/two.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/project2/parent/child/two.py @@ -0,0 +1,1 @@ +attr = 'parent child two' diff --git a/Lib/test/namespace_pkgs/project3/parent/child/three.py b/Lib/test/namespace_pkgs/project3/parent/child/three.py new file mode 100644 --- /dev/null +++ b/Lib/test/namespace_pkgs/project3/parent/child/three.py @@ -0,0 +1,1 @@ +attr = 'parent child three' diff --git a/Lib/test/namespace_pkgs/top_level_portion1.zip b/Lib/test/namespace_pkgs/top_level_portion1.zip new file mode 100644 index 0000000000000000000000000000000000000000..3b866c914ad2f2fe348405799a482235854bac10 GIT binary patch [stripped] diff --git a/Lib/test/test_frozen.py b/Lib/test/test_frozen.py --- a/Lib/test/test_frozen.py +++ b/Lib/test/test_frozen.py @@ -7,7 +7,7 @@ class FrozenTests(unittest.TestCase): module_attrs = frozenset(['__builtins__', '__cached__', '__doc__', - '__file__', '__loader__', '__name__', + '__loader__', '__name__', '__package__']) package_attrs = frozenset(list(module_attrs) + ['__path__']) diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py --- a/Lib/test/test_import.py +++ b/Lib/test/test_import.py @@ -286,12 +286,6 @@ import test.support as y self.assertIs(y, test.support, y.__name__) - def test_import_initless_directory_warning(self): - with check_warnings(('', ImportWarning)): - # Just a random non-package directory we always expect to be - # somewhere in sys.path... - self.assertRaises(ImportError, __import__, "site-packages") - def test_import_by_filename(self): path = os.path.abspath(TESTFN) encoding = sys.getfilesystemencoding() diff --git a/Lib/test/test_module.py b/Lib/test/test_module.py --- a/Lib/test/test_module.py +++ b/Lib/test/test_module.py @@ -5,6 +5,15 @@ import sys ModuleType = type(sys) +class FullLoader: + @classmethod + def module_repr(cls, m): + return "".format(m.__name__) + +class BareLoader: + pass + + class ModuleTests(unittest.TestCase): def test_uninitialized(self): # An uninitialized module has no __dict__ or __name__, @@ -80,8 +89,90 @@ gc_collect() self.assertEqual(destroyed, [1]) + def test_module_repr_minimal(self): + # reprs when modules have no __file__, __name__, or __loader__ + m = ModuleType('foo') + del m.__name__ + self.assertEqual(repr(m), "") + + def test_module_repr_with_name(self): + m = ModuleType('foo') + self.assertEqual(repr(m), "") + + def test_module_repr_with_name_and_filename(self): + m = ModuleType('foo') + m.__file__ = '/tmp/foo.py' + self.assertEqual(repr(m), "") + + def test_module_repr_with_filename_only(self): + m = ModuleType('foo') + del m.__name__ + m.__file__ = '/tmp/foo.py' + self.assertEqual(repr(m), "") + + def test_module_repr_with_bare_loader_but_no_name(self): + m = ModuleType('foo') + del m.__name__ + # Yes, a class not an instance. + m.__loader__ = BareLoader + self.assertEqual( + repr(m), ")>") + + def test_module_repr_with_full_loader_but_no_name(self): + # m.__loader__.module_repr() will fail because the module has no + # m.__name__. This exception will get suppressed and instead the + # loader's repr will be used. + m = ModuleType('foo') + del m.__name__ + # Yes, a class not an instance. + m.__loader__ = FullLoader + self.assertEqual( + repr(m), ")>") + + def test_module_repr_with_bare_loader(self): + m = ModuleType('foo') + # Yes, a class not an instance. + m.__loader__ = BareLoader + self.assertEqual( + repr(m), ")>") + + def test_module_repr_with_full_loader(self): + m = ModuleType('foo') + # Yes, a class not an instance. + m.__loader__ = FullLoader + self.assertEqual( + repr(m), "") + + def test_module_repr_with_bare_loader_and_filename(self): + # Because the loader has no module_repr(), use the file name. + m = ModuleType('foo') + # Yes, a class not an instance. + m.__loader__ = BareLoader + m.__file__ = '/tmp/foo.py' + self.assertEqual(repr(m), "") + + def test_module_repr_with_full_loader_and_filename(self): + # Even though the module has an __file__, use __loader__.module_repr() + m = ModuleType('foo') + # Yes, a class not an instance. + m.__loader__ = FullLoader + m.__file__ = '/tmp/foo.py' + self.assertEqual(repr(m), "") + + def test_module_repr_builtin(self): + self.assertEqual(repr(sys), "") + + def test_module_repr_source(self): + r = repr(unittest) + self.assertEqual(r[:25], "") + + # frozen and namespace module reprs are tested in importlib. + + def test_main(): run_unittest(ModuleTests) + if __name__ == '__main__': test_main() diff --git a/Lib/test/test_namespace_pkgs.py b/Lib/test/test_namespace_pkgs.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_namespace_pkgs.py @@ -0,0 +1,239 @@ +import sys +import contextlib +import unittest +import os + +import importlib.test.util +from test.support import run_unittest + +# needed tests: +# +# need to test when nested, so that the top-level path isn't sys.path +# need to test dynamic path detection, both at top-level and nested +# with dynamic path, check when a loader is returned on path reload (that is, +# trying to switch from a namespace package to a regular package) + + + at contextlib.contextmanager +def sys_modules_context(): + """ + Make sure sys.modules is the same object and has the same content + when exiting the context as when entering. + + Similar to importlib.test.util.uncache, but doesn't require explicit + names. + """ + sys_modules_saved = sys.modules + sys_modules_copy = sys.modules.copy() + try: + yield + finally: + sys.modules = sys_modules_saved + sys.modules.clear() + sys.modules.update(sys_modules_copy) + + + at contextlib.contextmanager +def namespace_tree_context(**kwargs): + """ + Save import state and sys.modules cache and restore it on exit. + Typical usage: + + >>> with namespace_tree_context(path=['/tmp/xxyy/portion1', + ... '/tmp/xxyy/portion2']): + ... pass + """ + # use default meta_path and path_hooks unless specified otherwise + kwargs.setdefault('meta_path', sys.meta_path) + kwargs.setdefault('path_hooks', sys.path_hooks) + import_context = importlib.test.util.import_state(**kwargs) + with import_context, sys_modules_context(): + yield + +class NamespacePackageTest(unittest.TestCase): + """ + Subclasses should define self.root and self.paths (under that root) + to be added to sys.path. + """ + root = os.path.join(os.path.dirname(__file__), 'namespace_pkgs') + + def setUp(self): + self.resolved_paths = [ + os.path.join(self.root, path) for path in self.paths + ] + self.ctx = namespace_tree_context(path=self.resolved_paths) + self.ctx.__enter__() + + def tearDown(self): + # TODO: will we ever want to pass exc_info to __exit__? + self.ctx.__exit__(None, None, None) + +class SingleNamespacePackage(NamespacePackageTest): + paths = ['portion1'] + + def test_simple_package(self): + import foo.one + self.assertEqual(foo.one.attr, 'portion1 foo one') + + def test_cant_import_other(self): + with self.assertRaises(ImportError): + import foo.two + + def test_module_repr(self): + import foo.one + self.assertEqual(repr(foo), "") + + +class DynamicPatheNamespacePackage(NamespacePackageTest): + paths = ['portion1'] + + def test_dynamic_path(self): + # Make sure only 'foo.one' can be imported + import foo.one + self.assertEqual(foo.one.attr, 'portion1 foo one') + + with self.assertRaises(ImportError): + import foo.two + + # Now modify sys.path + sys.path.append(os.path.join(self.root, 'portion2')) + + # And make sure foo.two is now importable + import foo.two + self.assertEqual(foo.two.attr, 'portion2 foo two') + + +class CombinedNamespacePackages(NamespacePackageTest): + paths = ['both_portions'] + + def test_imports(self): + import foo.one + import foo.two + self.assertEqual(foo.one.attr, 'both_portions foo one') + self.assertEqual(foo.two.attr, 'both_portions foo two') + + +class SeparatedNamespacePackages(NamespacePackageTest): + paths = ['portion1', 'portion2'] + + def test_imports(self): + import foo.one + import foo.two + self.assertEqual(foo.one.attr, 'portion1 foo one') + self.assertEqual(foo.two.attr, 'portion2 foo two') + + +class SeparatedOverlappingNamespacePackages(NamespacePackageTest): + paths = ['portion1', 'both_portions'] + + def test_first_path_wins(self): + import foo.one + import foo.two + self.assertEqual(foo.one.attr, 'portion1 foo one') + self.assertEqual(foo.two.attr, 'both_portions foo two') + + def test_first_path_wins_again(self): + sys.path.reverse() + import foo.one + import foo.two + self.assertEqual(foo.one.attr, 'both_portions foo one') + self.assertEqual(foo.two.attr, 'both_portions foo two') + + def test_first_path_wins_importing_second_first(self): + import foo.two + import foo.one + self.assertEqual(foo.one.attr, 'portion1 foo one') + self.assertEqual(foo.two.attr, 'both_portions foo two') + + +class SingleZipNamespacePackage(NamespacePackageTest): + paths = ['top_level_portion1.zip'] + + def test_simple_package(self): + import foo.one + self.assertEqual(foo.one.attr, 'portion1 foo one') + + def test_cant_import_other(self): + with self.assertRaises(ImportError): + import foo.two + + +class SeparatedZipNamespacePackages(NamespacePackageTest): + paths = ['top_level_portion1.zip', 'portion2'] + + def test_imports(self): + import foo.one + import foo.two + self.assertEqual(foo.one.attr, 'portion1 foo one') + self.assertEqual(foo.two.attr, 'portion2 foo two') + self.assertIn('top_level_portion1.zip', foo.one.__file__) + self.assertNotIn('.zip', foo.two.__file__) + + +class SingleNestedZipNamespacePackage(NamespacePackageTest): + paths = ['nested_portion1.zip/nested_portion1'] + + def test_simple_package(self): + import foo.one + self.assertEqual(foo.one.attr, 'portion1 foo one') + + def test_cant_import_other(self): + with self.assertRaises(ImportError): + import foo.two + + +class SeparatedNestedZipNamespacePackages(NamespacePackageTest): + paths = ['nested_portion1.zip/nested_portion1', 'portion2'] + + def test_imports(self): + import foo.one + import foo.two + self.assertEqual(foo.one.attr, 'portion1 foo one') + self.assertEqual(foo.two.attr, 'portion2 foo two') + fn = os.path.join('nested_portion1.zip', 'nested_portion1') + self.assertIn(fn, foo.one.__file__) + self.assertNotIn('.zip', foo.two.__file__) + + +class LegacySupport(NamespacePackageTest): + paths = ['not_a_namespace_pkg', 'portion1', 'portion2', 'both_portions'] + + def test_non_namespace_package_takes_precedence(self): + import foo.one + with self.assertRaises(ImportError): + import foo.two + self.assertIn('__init__', foo.__file__) + self.assertNotIn('namespace', str(foo.__loader__).lower()) + + +class ZipWithMissingDirectory(NamespacePackageTest): + paths = ['missing_directory.zip'] + + @unittest.expectedFailure + def test_missing_directory(self): + # This will fail because missing_directory.zip contains: + # Length Date Time Name + # --------- ---------- ----- ---- + # 29 2012-05-03 18:13 foo/one.py + # 0 2012-05-03 20:57 bar/ + # 38 2012-05-03 20:57 bar/two.py + # --------- ------- + # 67 3 files + + # Because there is no 'foo/', the zipimporter currently doesn't + # know that foo is a namespace package + + import foo.one + + def test_present_directory(self): + # This succeeds because there is a "bar/" in the zip file + import bar.two + self.assertEqual(bar.two.attr, 'missing_directory foo two') + + +def test_main(): + run_unittest(*NamespacePackageTest.__subclasses__()) + + +if __name__ == "__main__": + test_main() diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py --- a/Lib/test/test_pkgutil.py +++ b/Lib/test/test_pkgutil.py @@ -138,10 +138,11 @@ del sys.modules['foo'] +# These tests, especially the setup and cleanup, are hideous. They +# need to be cleaned up once issue 14715 is addressed. class ExtendPathTests(unittest.TestCase): def create_init(self, pkgname): dirname = tempfile.mkdtemp() - self.addCleanup(shutil.rmtree, dirname) sys.path.insert(0, dirname) pkgdir = os.path.join(dirname, pkgname) @@ -156,22 +157,12 @@ with open(module_name, 'w') as fl: print('value={}'.format(value), file=fl) - def setUp(self): - # Create 2 directories on sys.path - self.pkgname = 'foo' - self.dirname_0 = self.create_init(self.pkgname) - self.dirname_1 = self.create_init(self.pkgname) - - def tearDown(self): - del sys.path[0] - del sys.path[0] - del sys.modules['foo'] - del sys.modules['foo.bar'] - del sys.modules['foo.baz'] - def test_simple(self): - self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) - self.create_submodule(self.dirname_1, self.pkgname, 'baz', 1) + pkgname = 'foo' + dirname_0 = self.create_init(pkgname) + dirname_1 = self.create_init(pkgname) + self.create_submodule(dirname_0, pkgname, 'bar', 0) + self.create_submodule(dirname_1, pkgname, 'baz', 1) import foo.bar import foo.baz # Ensure we read the expected values @@ -180,8 +171,45 @@ # Ensure the path is set up correctly self.assertEqual(sorted(foo.__path__), - sorted([os.path.join(self.dirname_0, self.pkgname), - os.path.join(self.dirname_1, self.pkgname)])) + sorted([os.path.join(dirname_0, pkgname), + os.path.join(dirname_1, pkgname)])) + + # Cleanup + shutil.rmtree(dirname_0) + shutil.rmtree(dirname_1) + del sys.path[0] + del sys.path[0] + del sys.modules['foo'] + del sys.modules['foo.bar'] + del sys.modules['foo.baz'] + + def test_mixed_namespace(self): + pkgname = 'foo' + dirname_0 = self.create_init(pkgname) + dirname_1 = self.create_init(pkgname) + self.create_submodule(dirname_0, pkgname, 'bar', 0) + # Turn this into a PEP 420 namespace package + os.unlink(os.path.join(dirname_0, pkgname, '__init__.py')) + self.create_submodule(dirname_1, pkgname, 'baz', 1) + import foo.bar + import foo.baz + # Ensure we read the expected values + self.assertEqual(foo.bar.value, 0) + self.assertEqual(foo.baz.value, 1) + + # Ensure the path is set up correctly + self.assertEqual(sorted(foo.__path__), + sorted([os.path.join(dirname_0, pkgname), + os.path.join(dirname_1, pkgname)])) + + # Cleanup + shutil.rmtree(dirname_0) + shutil.rmtree(dirname_1) + del sys.path[0] + del sys.path[0] + del sys.modules['foo'] + del sys.modules['foo.bar'] + del sys.modules['foo.baz'] # XXX: test .pkg files diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #14660 (PEP 420): Namespace packages. Implemented by Eric Smith. + - Issue #14494: Fix __future__.py and its documentation to note that absolute imports are the default behavior in 3.0 instead of 2.7. Patch by Sven Marnach. diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -259,6 +259,29 @@ MI_PACKAGE }; +/* Does this path represent a directory? + on error, return < 0 + if not a dir, return 0 + if a dir, return 1 +*/ +static int +check_is_directory(ZipImporter *self, PyObject* prefix, PyObject *path) +{ + PyObject *dirpath; + PyObject *item; + + /* See if this is a "directory". If so, it's eligible to be part + of a namespace package. We test by seeing if the name, with an + appended path separator, exists. */ + dirpath = PyUnicode_FromFormat("%U%U%c", prefix, path, SEP); + if (dirpath == NULL) + return -1; + /* If dirpath is present in self->files, we have a directory. */ + item = PyDict_GetItem(self->files, dirpath); + Py_DECREF(dirpath); + return item != NULL; +} + /* Return some information about a module. */ static enum zi_module_info get_module_info(ZipImporter *self, PyObject *fullname) @@ -296,6 +319,46 @@ return MI_NOT_FOUND; } +/* The guts of "find_loader" and "find_module". Return values: + -1: error + 0: no loader or namespace portions found + 1: module/package found + 2: namespace portion found: *namespace_portion will point to the name +*/ +static int +find_loader(ZipImporter *self, PyObject *fullname, PyObject **namespace_portion) +{ + enum zi_module_info mi; + + *namespace_portion = NULL; + + mi = get_module_info(self, fullname); + if (mi == MI_ERROR) + return -1; + if (mi == MI_NOT_FOUND) { + /* Not a module or regular package. See if this is a directory, and + therefore possibly a portion of a namespace package. */ + int is_dir = check_is_directory(self, self->prefix, fullname); + if (is_dir < 0) + return -1; + if (is_dir) { + /* This is possibly a portion of a namespace + package. Return the string representing its path, + without a trailing separator. */ + *namespace_portion = PyUnicode_FromFormat("%U%c%U%U", + self->archive, SEP, + self->prefix, fullname); + if (*namespace_portion == NULL) + return -1; + return 2; + } + return 0; + } + /* This is a module or package. */ + return 1; +} + + /* Check whether we can satisfy the import of the module named by 'fullname'. Return self if we can, None if we can't. */ static PyObject * @@ -304,21 +367,78 @@ ZipImporter *self = (ZipImporter *)obj; PyObject *path = NULL; PyObject *fullname; - enum zi_module_info mi; + PyObject* namespace_portion = NULL; if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", &fullname, &path)) - return NULL; + goto error; - mi = get_module_info(self, fullname); - if (mi == MI_ERROR) - return NULL; - if (mi == MI_NOT_FOUND) { + switch (find_loader(self, fullname, &namespace_portion)) { + case -1: /* Error */ + goto error; + case 0: /* Not found, return None */ + Py_INCREF(Py_None); + return Py_None; + case 1: /* Return self */ + Py_INCREF(self); + return (PyObject *)self; + case 2: /* A namespace portion, but not allowed via + find_module, so return None */ + Py_DECREF(namespace_portion); Py_INCREF(Py_None); return Py_None; } - Py_INCREF(self); - return (PyObject *)self; + /* Can't get here. */ + assert(0); + return NULL; +error: + Py_XDECREF(namespace_portion); + return NULL; +} + + +/* Check whether we can satisfy the import of the module named by + 'fullname', or whether it could be a portion of a namespace + package. Return self if we can load it, a string containing the + full path if it's a possible namespace portion, None if we + can't load it. */ +static PyObject * +zipimporter_find_loader(PyObject *obj, PyObject *args) +{ + ZipImporter *self = (ZipImporter *)obj; + PyObject *path = NULL; + PyObject *fullname; + PyObject *result = NULL; + PyObject *namespace_portion = NULL; + + if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", + &fullname, &path)) + goto error; + + switch (find_loader(self, fullname, &namespace_portion)) { + case -1: /* Error */ + goto error; + case 0: /* Not found, return (None, []) */ + if (!(result = Py_BuildValue("O[]", Py_None))) + goto error; + return result; + case 1: /* Return (self, []) */ + if (!(result = Py_BuildValue("O[]", self))) + goto error; + return result; + case 2: /* Return (None, [namespace_portion]) */ + if (!(result = Py_BuildValue("O[O]", Py_None, namespace_portion))) + goto error; + return result; + } + /* Can't get here. */ + assert(0); + return NULL; + +error: + Py_XDECREF(namespace_portion); + Py_XDECREF(result); + return NULL; } /* Load and return the module named by 'fullname'. */ @@ -558,6 +678,16 @@ The optional 'path' argument is ignored -- it's there for compatibility\n\ with the importer protocol."); +PyDoc_STRVAR(doc_find_loader, +"find_loader(fullname, path=None) -> self, str or None.\n\ +\n\ +Search for a module specified by 'fullname'. 'fullname' must be the\n\ +fully qualified (dotted) module name. It returns the zipimporter\n\ +instance itself if the module was found, a string containing the\n\ +full path name if it's possibly a portion of a namespace package,\n\ +or None otherwise. The optional 'path' argument is ignored -- it's\n\ + there for compatibility with the importer protocol."); + PyDoc_STRVAR(doc_load_module, "load_module(fullname) -> module.\n\ \n\ @@ -599,6 +729,8 @@ static PyMethodDef zipimporter_methods[] = { {"find_module", zipimporter_find_module, METH_VARARGS, doc_find_module}, + {"find_loader", zipimporter_find_loader, METH_VARARGS, + doc_find_loader}, {"load_module", zipimporter_load_module, METH_VARARGS, doc_load_module}, {"get_data", zipimporter_get_data, METH_VARARGS, diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -366,8 +366,28 @@ static PyObject * module_repr(PyModuleObject *m) { - PyObject *name, *filename, *repr; + PyObject *name, *filename, *repr, *loader = NULL; + /* See if the module has an __loader__. If it does, give the loader the + * first shot at producing a repr for the module. + */ + if (m->md_dict != NULL) { + loader = PyDict_GetItemString(m->md_dict, "__loader__"); + } + if (loader != NULL) { + repr = PyObject_CallMethod(loader, "module_repr", "(O)", + (PyObject *)m, NULL); + if (repr == NULL) { + PyErr_Clear(); + } + else { + return repr; + } + } + /* __loader__.module_repr(m) did not provide us with a repr. Next, see if + * the module has an __file__. If it doesn't then use repr(__loader__) if + * it exists, otherwise, just use module.__name__. + */ name = PyModule_GetNameObject((PyObject *)m); if (name == NULL) { PyErr_Clear(); @@ -378,8 +398,17 @@ filename = PyModule_GetFilenameObject((PyObject *)m); if (filename == NULL) { PyErr_Clear(); - repr = PyUnicode_FromFormat("", name); + /* There's no m.__file__, so if there was an __loader__, use that in + * the repr, otherwise, the only thing you can use is m.__name__ + */ + if (loader == NULL) { + repr = PyUnicode_FromFormat("", name); + } + else { + repr = PyUnicode_FromFormat("", name, loader); + } } + /* Finally, use m.__file__ */ else { repr = PyUnicode_FromFormat("", name, filename); Py_DECREF(filename); diff --git a/Python/importlib.h b/Python/importlib.h index 493cd1aac17510f62321955f9d7a45836ef23a7d..bdb3644db479da7d7f6b949be135c6b1c3d1fc33 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 02:22:23 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 25 May 2012 02:22:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Whitespace_cleanup=2E?= Message-ID: http://hg.python.org/cpython/rev/9b1790de6efa changeset: 77127:9b1790de6efa user: Eric V. Smith date: Thu May 24 20:22:10 2012 -0400 summary: Whitespace cleanup. files: Lib/importlib/test/frozen/test_loader.py | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/importlib/test/frozen/test_loader.py b/Lib/importlib/test/frozen/test_loader.py --- a/Lib/importlib/test/frozen/test_loader.py +++ b/Lib/importlib/test/frozen/test_loader.py @@ -11,7 +11,7 @@ with util.uncache('__hello__'), captured_stdout() as stdout: module = machinery.FrozenImporter.load_module('__hello__') check = {'__name__': '__hello__', - '__package__': '', + '__package__': '', '__loader__': machinery.FrozenImporter, } for attr, value in check.items(): @@ -22,8 +22,8 @@ def test_package(self): with util.uncache('__phello__'), captured_stdout() as stdout: module = machinery.FrozenImporter.load_module('__phello__') - check = {'__name__': '__phello__', - '__package__': '__phello__', + check = {'__name__': '__phello__', + '__package__': '__phello__', '__path__': ['__phello__'], '__loader__': machinery.FrozenImporter, } @@ -62,7 +62,7 @@ def test_module_repr(self): with util.uncache('__hello__'), captured_stdout(): module = machinery.FrozenImporter.load_module('__hello__') - self.assertEqual(repr(module), + self.assertEqual(repr(module), "") def test_state_after_failure(self): -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Fri May 25 05:46:52 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Fri, 25 May 2012 05:46:52 +0200 Subject: [Python-checkins] Daily reference leaks (9b1790de6efa): sum=21 Message-ID: results for 9b1790de6efa on branch "default" -------------------------------------------- test_namespace_pkgs leaked [7, 7, 7] references, sum=21 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog31CF0Q', '-x'] From python-checkins at python.org Fri May 25 07:35:38 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 07:35:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_plug_ref_leak?= Message-ID: http://hg.python.org/cpython/rev/7dad810efbfe changeset: 77128:7dad810efbfe user: Benjamin Peterson date: Thu May 24 22:35:39 2012 -0700 summary: plug ref leak files: Modules/zipimport.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -429,6 +429,7 @@ case 2: /* Return (None, [namespace_portion]) */ if (!(result = Py_BuildValue("O[O]", Py_None, namespace_portion))) goto error; + Py_DECREF(namespace_portion); return result; } /* Can't get here. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 07:54:25 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 07:54:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_simplify_and_rewrite_the_zi?= =?utf8?q?pimport_part_of_702009f3c0b1_a_bit?= Message-ID: http://hg.python.org/cpython/rev/a47d32a28662 changeset: 77129:a47d32a28662 user: Benjamin Peterson date: Thu May 24 22:54:15 2012 -0700 summary: simplify and rewrite the zipimport part of 702009f3c0b1 a bit files: Modules/zipimport.c | 92 ++++++++++++++------------------ 1 files changed, 41 insertions(+), 51 deletions(-) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -319,13 +319,20 @@ return MI_NOT_FOUND; } +typedef enum { + fl_error, + fl_not_found, + fl_module_found, + fl_ns_found +} find_loader_result; + /* The guts of "find_loader" and "find_module". Return values: -1: error 0: no loader or namespace portions found 1: module/package found 2: namespace portion found: *namespace_portion will point to the name */ -static int +static find_loader_result find_loader(ZipImporter *self, PyObject *fullname, PyObject **namespace_portion) { enum zi_module_info mi; @@ -334,7 +341,7 @@ mi = get_module_info(self, fullname); if (mi == MI_ERROR) - return -1; + return fl_error; if (mi == MI_NOT_FOUND) { /* Not a module or regular package. See if this is a directory, and therefore possibly a portion of a namespace package. */ @@ -349,13 +356,13 @@ self->archive, SEP, self->prefix, fullname); if (*namespace_portion == NULL) - return -1; - return 2; + return fl_error; + return fl_ns_found; } - return 0; + return fl_not_found; } /* This is a module or package. */ - return 1; + return fl_module_found; } @@ -367,32 +374,26 @@ ZipImporter *self = (ZipImporter *)obj; PyObject *path = NULL; PyObject *fullname; - PyObject* namespace_portion = NULL; + PyObject *namespace_portion = NULL; + PyObject *result = NULL; - if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", - &fullname, &path)) - goto error; + if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", &fullname, &path)) + return NULL; switch (find_loader(self, fullname, &namespace_portion)) { - case -1: /* Error */ - goto error; - case 0: /* Not found, return None */ - Py_INCREF(Py_None); - return Py_None; - case 1: /* Return self */ - Py_INCREF(self); - return (PyObject *)self; - case 2: /* A namespace portion, but not allowed via - find_module, so return None */ + case fl_ns_found: + /* A namespace portion is not allowed via find_module, so return None. */ Py_DECREF(namespace_portion); - Py_INCREF(Py_None); - return Py_None; + /* FALL THROUGH */ + case fl_error: + case fl_not_found: + result = Py_None; + break; + case fl_module_found: + result = (PyObject *)self; + break; } - /* Can't get here. */ - assert(0); - return NULL; -error: - Py_XDECREF(namespace_portion); + Py_XINCREF(result); return NULL; } @@ -411,35 +412,24 @@ PyObject *result = NULL; PyObject *namespace_portion = NULL; - if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", - &fullname, &path)) - goto error; + if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", &fullname, &path)) + return NULL; switch (find_loader(self, fullname, &namespace_portion)) { - case -1: /* Error */ - goto error; - case 0: /* Not found, return (None, []) */ - if (!(result = Py_BuildValue("O[]", Py_None))) - goto error; - return result; - case 1: /* Return (self, []) */ - if (!(result = Py_BuildValue("O[]", self))) - goto error; - return result; - case 2: /* Return (None, [namespace_portion]) */ - if (!(result = Py_BuildValue("O[O]", Py_None, namespace_portion))) - goto error; + case fl_error: + return NULL; + case fl_not_found: /* Not found, return (None, []) */ + result = Py_BuildValue("O[]", Py_None); + break; + case fl_module_found: /* Return (self, []) */ + result = Py_BuildValue("O[]", self); + break; + case fl_ns_found: /* Return (None, [namespace_portion]) */ + result = Py_BuildValue("O[O]", Py_None, namespace_portion); Py_DECREF(namespace_portion); return result; } - /* Can't get here. */ - assert(0); - return NULL; - -error: - Py_XDECREF(namespace_portion); - Py_XDECREF(result); - return NULL; + return result; } /* Load and return the module named by 'fullname'. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 07:58:47 2012 From: python-checkins at python.org (larry.hastings) Date: Fri, 25 May 2012 07:58:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314889=3A_PyBytes?= =?utf8?q?=5FFromObject=28bytes=29_now_just_increfs_and_returns=2E?= Message-ID: http://hg.python.org/cpython/rev/f8cd75e8a1d6 changeset: 77130:f8cd75e8a1d6 user: Larry Hastings date: Thu May 24 22:58:30 2012 -0700 summary: Issue #14889: PyBytes_FromObject(bytes) now just increfs and returns. Previously, if you passed in a bytes object, it would create a whole new object. files: Objects/bytesobject.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2577,6 +2577,12 @@ PyErr_BadInternalCall(); return NULL; } + + if (PyBytes_CheckExact(x)) { + Py_INCREF(x); + return x; + } + /* Use the modern buffer interface */ if (PyObject_CheckBuffer(x)) { Py_buffer view; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 09:19:39 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 09:19:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_actually_return_the_result?= Message-ID: http://hg.python.org/cpython/rev/828e9d559c55 changeset: 77131:828e9d559c55 user: Benjamin Peterson date: Fri May 25 00:19:40 2012 -0700 summary: actually return the result files: Modules/zipimport.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -394,7 +394,7 @@ break; } Py_XINCREF(result); - return NULL; + return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 09:22:03 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 09:22:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_return_NULL_on_error?= Message-ID: http://hg.python.org/cpython/rev/9554a27b5b19 changeset: 77132:9554a27b5b19 user: Benjamin Peterson date: Fri May 25 00:22:04 2012 -0700 summary: return NULL on error files: Modules/zipimport.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -381,11 +381,12 @@ return NULL; switch (find_loader(self, fullname, &namespace_portion)) { + case fl_error: + return NULL; case fl_ns_found: /* A namespace portion is not allowed via find_module, so return None. */ Py_DECREF(namespace_portion); /* FALL THROUGH */ - case fl_error: case fl_not_found: result = Py_None; break; @@ -393,7 +394,7 @@ result = (PyObject *)self; break; } - Py_XINCREF(result); + Py_INCREF(result); return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 09:26:30 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 09:26:30 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_use_PyDict=5FContains?= Message-ID: http://hg.python.org/cpython/rev/17f48832e749 changeset: 77133:17f48832e749 user: Benjamin Peterson date: Fri May 25 00:24:42 2012 -0700 summary: use PyDict_Contains files: Modules/zipimport.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -268,7 +268,7 @@ check_is_directory(ZipImporter *self, PyObject* prefix, PyObject *path) { PyObject *dirpath; - PyObject *item; + int res; /* See if this is a "directory". If so, it's eligible to be part of a namespace package. We test by seeing if the name, with an @@ -277,9 +277,9 @@ if (dirpath == NULL) return -1; /* If dirpath is present in self->files, we have a directory. */ - item = PyDict_GetItem(self->files, dirpath); + res = PyDict_Contains(self->files, dirpath); Py_DECREF(dirpath); - return item != NULL; + return res; } /* Return some information about a module. */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 10:30:07 2012 From: python-checkins at python.org (hynek.schlawack) Date: Fri, 25 May 2012 10:30:07 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzQ4NDE6IEZpeCBG?= =?utf8?q?ileIO_constructor_to_honor_closefd_when_called_repeatedly?= Message-ID: http://hg.python.org/cpython/rev/83d500908ffc changeset: 77134:83d500908ffc branch: 2.7 parent: 77125:13197ce81b0d user: Hynek Schlawack date: Fri May 25 09:24:18 2012 +0200 summary: #4841: Fix FileIO constructor to honor closefd when called repeatedly Patch by Victor Stinner. files: Lib/test/test_io.py | 13 +++++++++++++ Modules/_io/fileio.c | 10 +++++++--- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -593,6 +593,19 @@ self.assertEqual(rawio.read(2), None) self.assertEqual(rawio.read(2), b"") + def test_fileio_closefd(self): + # Issue #4841 + with self.open(__file__, 'rb') as f1, \ + self.open(__file__, 'rb') as f2: + fileio = self.FileIO(f1.fileno(), closefd=False) + # .__init__() must not close f1 + fileio.__init__(f2.fileno(), closefd=False) + f1.readline() + # .close() must not close f2 + fileio.close() + f2.readline() + + class CIOTest(IOTest): def test_IOBase_finalize(self): diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -198,9 +198,13 @@ assert(PyFileIO_Check(oself)); if (self->fd >= 0) { - /* Have to close the existing file first. */ - if (internal_close(self) < 0) - return -1; + if (self->closefd) { + /* Have to close the existing file first. */ + if (internal_close(self) < 0) + return -1; + } + else + self->fd = -1; } if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|si:fileio", -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 10:30:09 2012 From: python-checkins at python.org (hynek.schlawack) Date: Fri, 25 May 2012 10:30:09 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzQ4NDE6IEZpeCBG?= =?utf8?q?ileIO_constructor_to_honor_closefd_when_called_repeatedly?= Message-ID: http://hg.python.org/cpython/rev/8a58670048c9 changeset: 77135:8a58670048c9 branch: 3.2 parent: 77122:9ef2cb56926d user: Hynek Schlawack date: Fri May 25 10:05:53 2012 +0200 summary: #4841: Fix FileIO constructor to honor closefd when called repeatedly Patch by Victor Stinner. files: Lib/test/test_io.py | 13 +++++++++++++ Modules/_io/fileio.c | 10 +++++++--- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -634,6 +634,19 @@ for obj in test: self.assertTrue(hasattr(obj, "__dict__")) + def test_fileio_closefd(self): + # Issue #4841 + with self.open(__file__, 'rb') as f1, \ + self.open(__file__, 'rb') as f2: + fileio = self.FileIO(f1.fileno(), closefd=False) + # .__init__() must not close f1 + fileio.__init__(f2.fileno(), closefd=False) + f1.readline() + # .close() must not close f2 + fileio.close() + f2.readline() + + class CIOTest(IOTest): def test_IOBase_finalize(self): diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -227,9 +227,13 @@ assert(PyFileIO_Check(oself)); if (self->fd >= 0) { - /* Have to close the existing file first. */ - if (internal_close(self) < 0) - return -1; + if (self->closefd) { + /* Have to close the existing file first. */ + if (internal_close(self) < 0) + return -1; + } + else + self->fd = -1; } if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|si:fileio", -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 10:30:17 2012 From: python-checkins at python.org (hynek.schlawack) Date: Fri, 25 May 2012 10:30:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_=234841=3A_Fix_FileIO_constructor_to_honor_closefd_when_call?= =?utf8?q?ed_repeatedly?= Message-ID: http://hg.python.org/cpython/rev/d2b511592cc5 changeset: 77136:d2b511592cc5 parent: 77133:17f48832e749 parent: 77135:8a58670048c9 user: Hynek Schlawack date: Fri May 25 10:27:43 2012 +0200 summary: #4841: Fix FileIO constructor to honor closefd when called repeatedly Patch by Victor Stinner. files: Lib/test/test_io.py | 13 +++++++++++++ Modules/_io/fileio.c | 10 +++++++--- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -643,6 +643,19 @@ with self.open("non-existent", "r", opener=opener) as f: self.assertEqual(f.read(), "egg\n") + def test_fileio_closefd(self): + # Issue #4841 + with self.open(__file__, 'rb') as f1, \ + self.open(__file__, 'rb') as f2: + fileio = self.FileIO(f1.fileno(), closefd=False) + # .__init__() must not close f1 + fileio.__init__(f2.fileno(), closefd=False) + f1.readline() + # .close() must not close f2 + fileio.close() + f2.readline() + + class CIOTest(IOTest): def test_IOBase_finalize(self): diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -230,9 +230,13 @@ assert(PyFileIO_Check(oself)); if (self->fd >= 0) { - /* Have to close the existing file first. */ - if (internal_close(self) < 0) - return -1; + if (self->closefd) { + /* Have to close the existing file first. */ + if (internal_close(self) < 0) + return -1; + } + else + self->fd = -1; } if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|siO:fileio", -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 10:34:47 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 25 May 2012 10:34:47 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Edit_include_files_section_of_?= =?utf8?q?PEP_405=2E?= Message-ID: http://hg.python.org/peps/rev/adbc23f61cd0 changeset: 4432:adbc23f61cd0 parent: 4428:f2500baa8a51 user: Carl Meyer date: Thu May 24 14:18:29 2012 -0600 summary: Edit include files section of PEP 405. files: pep-0405.txt | 13 ++++++------- 1 files changed, 6 insertions(+), 7 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -299,19 +299,18 @@ This PEP proposes a slightly different approach, though one with essentially the same effect and the same set of advantages and -disadvantages. Rather than symlinking or copying include files into -the venv, we simply modify the sysconfig schemes so that header files -are always looked for relative to ``base_prefix`` rather than -``prefix``. (We also create an ``include/`` directory within the venv +disadvantages. Rather than symlinking or copying include files into the +venv, we simply modify the sysconfig schemes so that header files are +always looked for relative to ``base_prefix`` rather than +``prefix``. (We also create an ``include/`` directory within the venv, +so installers have somewhere to put include files installed within the +env). Better handling of include files in distutils/packaging and, by extension, pyvenv, is an area that may deserve its own future PEP. For now, we propose that the behavior of virtualenv has thus far proved itself to be at least "good enough" in practice. -[Open question: should pyvenv instead simply copy the behavior of -virtualenv entirely, to avoid introducing unexpected new issues here?] - API --- -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 10:34:48 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 25 May 2012 10:34:48 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Edits_from_ncoghlan_review=2E?= Message-ID: http://hg.python.org/peps/rev/3b7e6e0316be changeset: 4433:3b7e6e0316be user: Carl Meyer date: Thu May 24 15:46:52 2012 -0600 summary: Edits from ncoghlan review. files: pep-0405.txt | 78 ++++++++++++++++++++------------------- 1 files changed, 40 insertions(+), 38 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -78,12 +78,13 @@ found, falling back to the build-time prefix hardcoded in the binary. This PEP proposes to add a new first step to this search. If a -``pyvenv.cfg`` file is found either adjacent to the Python executable, -or one directory above it, this file is scanned for lines of the form -``key = value``. If a ``home`` key is found, this signifies that the -Python binary belongs to a virtual environment, and the value of the -``home`` key is the directory containing the Python executable used to -create this virtual environment. +``pyvenv.cfg`` file is found either adjacent to the Python executable or +one directory above it (if the executable is a symlink, it is not +dereferenced), this file is scanned for lines of the form ``key = +value``. If a ``home`` key is found, this signifies that the Python +binary belongs to a virtual environment, and the value of the ``home`` +key is the directory containing the Python executable used to create +this virtual environment. In this case, prefix-finding continues as normal using the value of the ``home`` key as the effective Python binary location, which finds @@ -95,14 +96,16 @@ prefix-finding continues normally, and ``sys.prefix`` will be equal to ``sys.base_prefix``.) +Also, ``sys.base_exec_prefix`` is added, and handled similarly with +regard to ``sys.exec_prefix``. (``sys.exec_prefix`` is the equivalent of +``sys.prefix``, but for platform-specific files; by default it has the +same value as ``sys.prefix``.) + The ``site`` and ``sysconfig`` standard-library modules are modified such that the standard library and header files are are found relative -to ``sys.base_prefix``, while site-package directories ("purelib" and -"platlib", in ``sysconfig`` terms) are still found relative to -``sys.prefix``. - -(Also, ``sys.base_exec_prefix`` is added, and handled similarly with -regard to ``sys.exec_prefix``.) +to ``sys.base_prefix`` / ``sys.base_exec_prefix``, while site-package +directories ("purelib" and "platlib", in ``sysconfig`` terms) are still +found relative to ``sys.prefix`` / ``sys.exec_prefix``. Thus, a Python virtual environment in its simplest form would consist of nothing more than a copy or symlink of the Python binary @@ -147,12 +150,12 @@ directories that don't exist already) and places a ``pyvenv.cfg`` file in it with a ``home`` key pointing to the Python installation the command was run from. It also creates a ``bin/`` (or ``Scripts`` on -Windows) subdirectory containing a copy (or symlink) of the -``python3`` executable, and the ``pysetup3`` script from the -``packaging`` standard library module (to facilitate easy installation -of packages from PyPI into the new virtualenv). And it creates an -(initially empty) ``lib/pythonX.Y/site-packages`` (or -``Lib\site-packages`` on Windows) subdirectory. +Windows) subdirectory containing a copy (or symlink) of the ``python3`` +executable, and the ``pysetup3`` script from the ``packaging`` standard +library module (to facilitate easy installation of packages from PyPI +into the new venv). And it creates an (initially empty) +``lib/pythonX.Y/site-packages`` (or ``Lib\site-packages`` on Windows) +subdirectory. If the target directory already exists an error will be raised, unless the ``--clear`` option was provided, in which case the target @@ -160,27 +163,29 @@ proceed as usual. The created ``pyvenv.cfg`` file also includes the -``include-system-site-packages`` key, set to ``true`` if ``venv`` is +``include-system-site-packages`` key, set to ``true`` if ``pyvenv`` is run with the ``--system-site-packages`` option, ``false`` by default. Multiple paths can be given to ``pyvenv``, in which case an identical -virtualenv will be created, according to the given options, at each +venv will be created, according to the given options, at each provided path. -The ``venv`` module also provides "shell activation scripts" for POSIX -and Windows systems which simply add the virtual environment's ``bin`` -(or ``Scripts``) directory to the front of the user's shell PATH. -This is not strictly necessary for use of a virtual environment (as an -explicit path to the venv's python binary or scripts can just as well -be used), but it is convenient. +The ``venv`` module also places "shell activation scripts" for POSIX and +Windows systems in the ``bin`` or ``Scripts`` directory of the +venv. These scripts simply add the virtual environment's ``bin`` (or +``Scripts``) directory to the front of the user's shell PATH. This is +not strictly necessary for use of a virtual environment (as an explicit +path to the venv's python binary or scripts can just as well be used), +but it is convenient. In order to allow ``pysetup`` and other Python package managers to install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing -virtual environments in ``sysconfig`` beyond using ``sys.site_prefix`` -in place of ``sys.prefix``, the internal virtual environment layout -mimics the layout of the Python installation itself on each platform. -So a typical virtual environment layout on a POSIX system would be:: +virtual environments in ``sysconfig`` beyond using ``sys.base_prefix`` +in place of ``sys.prefix`` where appropriate, the internal virtual +environment layout mimics the layout of the Python installation itself +on each platform. So a typical virtual environment layout on a POSIX +system would be:: pyvenv.cfg bin/python3 @@ -301,10 +306,9 @@ essentially the same effect and the same set of advantages and disadvantages. Rather than symlinking or copying include files into the venv, we simply modify the sysconfig schemes so that header files are -always looked for relative to ``base_prefix`` rather than -``prefix``. (We also create an ``include/`` directory within the venv, -so installers have somewhere to put include files installed within the -env). +always sought relative to ``base_prefix`` rather than ``prefix``. (We +also create an ``include/`` directory within the venv, so installers +have somewhere to put include files installed within the env). Better handling of include files in distutils/packaging and, by extension, pyvenv, is an area that may deserve its own future PEP. For @@ -507,10 +511,8 @@ ======================== The reference implementation is found in `a clone of the CPython -Mercurial repository`_. To test it, build and install it (the virtual -environment tool currently does not run from a source tree). From the -installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create -a virtual environment. +Mercurial repository`_. To test it, build and run ``bin/pyvenv +/path/to/new/venv`` to create a virtual environment. .. _a clone of the CPython Mercurial repository: http://hg.python.org/sandbox/vsajip#venv -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 10:34:52 2012 From: python-checkins at python.org (vinay.sajip) Date: Fri, 25 May 2012 10:34:52 +0200 Subject: [Python-checkins] =?utf8?q?peps_=28merge_default_-=3E_default=29?= =?utf8?q?=3A_Merged_updates_by_Carl_following_Nick=27s_review=2E?= Message-ID: http://hg.python.org/peps/rev/8c17dbfbef26 changeset: 4434:8c17dbfbef26 parent: 4431:f6f5604b0946 parent: 4433:3b7e6e0316be user: Vinay Sajip date: Fri May 25 09:34:41 2012 +0100 summary: Merged updates by Carl following Nick's review. files: pep-0405.txt | 85 ++++++++++++++++++++------------------- 1 files changed, 43 insertions(+), 42 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -9,7 +9,7 @@ Content-Type: text/x-rst Created: 13-Jun-2011 Python-Version: 3.3 -Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012 +Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012, 24-May-2012 Abstract @@ -78,12 +78,13 @@ found, falling back to the build-time prefix hardcoded in the binary. This PEP proposes to add a new first step to this search. If a -``pyvenv.cfg`` file is found either adjacent to the Python executable, -or one directory above it, this file is scanned for lines of the form -``key = value``. If a ``home`` key is found, this signifies that the -Python binary belongs to a virtual environment, and the value of the -``home`` key is the directory containing the Python executable used to -create this virtual environment. +``pyvenv.cfg`` file is found either adjacent to the Python executable or +one directory above it (if the executable is a symlink, it is not +dereferenced), this file is scanned for lines of the form ``key = +value``. If a ``home`` key is found, this signifies that the Python +binary belongs to a virtual environment, and the value of the ``home`` +key is the directory containing the Python executable used to create +this virtual environment. In this case, prefix-finding continues as normal using the value of the ``home`` key as the effective Python binary location, which finds @@ -95,14 +96,16 @@ prefix-finding continues normally, and ``sys.prefix`` will be equal to ``sys.base_prefix``.) +Also, ``sys.base_exec_prefix`` is added, and handled similarly with +regard to ``sys.exec_prefix``. (``sys.exec_prefix`` is the equivalent of +``sys.prefix``, but for platform-specific files; by default it has the +same value as ``sys.prefix``.) + The ``site`` and ``sysconfig`` standard-library modules are modified such that the standard library and header files are are found relative -to ``sys.base_prefix``, while site-package directories ("purelib" and -"platlib", in ``sysconfig`` terms) are still found relative to -``sys.prefix``. - -(Also, ``sys.base_exec_prefix`` is added, and handled similarly with -regard to ``sys.exec_prefix``.) +to ``sys.base_prefix`` / ``sys.base_exec_prefix``, while site-package +directories ("purelib" and "platlib", in ``sysconfig`` terms) are still +found relative to ``sys.prefix`` / ``sys.exec_prefix``. Thus, a Python virtual environment in its simplest form would consist of nothing more than a copy or symlink of the Python binary @@ -147,12 +150,12 @@ directories that don't exist already) and places a ``pyvenv.cfg`` file in it with a ``home`` key pointing to the Python installation the command was run from. It also creates a ``bin/`` (or ``Scripts`` on -Windows) subdirectory containing a copy (or symlink) of the -``python3`` executable, and the ``pysetup3`` script from the -``packaging`` standard library module (to facilitate easy installation -of packages from PyPI into the new virtualenv). And it creates an -(initially empty) ``lib/pythonX.Y/site-packages`` (or -``Lib\site-packages`` on Windows) subdirectory. +Windows) subdirectory containing a copy (or symlink) of the ``python3`` +executable, and the ``pysetup3`` script from the ``packaging`` standard +library module (to facilitate easy installation of packages from PyPI +into the new venv). And it creates an (initially empty) +``lib/pythonX.Y/site-packages`` (or ``Lib\site-packages`` on Windows) +subdirectory. If the target directory already exists an error will be raised, unless the ``--clear`` option was provided, in which case the target @@ -160,27 +163,29 @@ proceed as usual. The created ``pyvenv.cfg`` file also includes the -``include-system-site-packages`` key, set to ``true`` if ``venv`` is +``include-system-site-packages`` key, set to ``true`` if ``pyvenv`` is run with the ``--system-site-packages`` option, ``false`` by default. Multiple paths can be given to ``pyvenv``, in which case an identical -virtualenv will be created, according to the given options, at each +venv will be created, according to the given options, at each provided path. -The ``venv`` module also provides "shell activation scripts" for POSIX -and Windows systems which simply add the virtual environment's ``bin`` -(or ``Scripts``) directory to the front of the user's shell PATH. -This is not strictly necessary for use of a virtual environment (as an -explicit path to the venv's python binary or scripts can just as well -be used), but it is convenient. +The ``venv`` module also places "shell activation scripts" for POSIX and +Windows systems in the ``bin`` or ``Scripts`` directory of the +venv. These scripts simply add the virtual environment's ``bin`` (or +``Scripts``) directory to the front of the user's shell PATH. This is +not strictly necessary for use of a virtual environment (as an explicit +path to the venv's python binary or scripts can just as well be used), +but it is convenient. In order to allow ``pysetup`` and other Python package managers to install packages into the virtual environment the same way they would install into a normal Python installation, and avoid special-casing -virtual environments in ``sysconfig`` beyond using ``sys.site_prefix`` -in place of ``sys.prefix``, the internal virtual environment layout -mimics the layout of the Python installation itself on each platform. -So a typical virtual environment layout on a POSIX system would be:: +virtual environments in ``sysconfig`` beyond using ``sys.base_prefix`` +in place of ``sys.prefix`` where appropriate, the internal virtual +environment layout mimics the layout of the Python installation itself +on each platform. So a typical virtual environment layout on a POSIX +system would be:: pyvenv.cfg bin/python3 @@ -299,19 +304,17 @@ This PEP proposes a slightly different approach, though one with essentially the same effect and the same set of advantages and -disadvantages. Rather than symlinking or copying include files into -the venv, we simply modify the sysconfig schemes so that header files -are always looked for relative to ``base_prefix`` rather than -``prefix``. (We also create an ``include/`` directory within the venv +disadvantages. Rather than symlinking or copying include files into the +venv, we simply modify the sysconfig schemes so that header files are +always sought relative to ``base_prefix`` rather than ``prefix``. (We +also create an ``include/`` directory within the venv, so installers +have somewhere to put include files installed within the env). Better handling of include files in distutils/packaging and, by extension, pyvenv, is an area that may deserve its own future PEP. For now, we propose that the behavior of virtualenv has thus far proved itself to be at least "good enough" in practice. -[Open question: should pyvenv instead simply copy the behavior of -virtualenv entirely, to avoid introducing unexpected new issues here?] - API --- @@ -508,10 +511,8 @@ ======================== The reference implementation is found in `a clone of the CPython -Mercurial repository`_. To test it, build and install it (the virtual -environment tool currently does not run from a source tree). From the -installed Python, run ``bin/pyvenv /path/to/new/virtualenv`` to create -a virtual environment. +Mercurial repository`_. To test it, build and run ``bin/pyvenv +/path/to/new/venv`` to create a virtual environment. .. _a clone of the CPython Mercurial repository: http://hg.python.org/sandbox/vsajip#venv -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 14:06:41 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 25 May 2012 14:06:41 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0ODgx?= =?utf8?q?=3A_Allow_normal_non-main_thread_to_spawn_a_dummy_process?= Message-ID: http://hg.python.org/cpython/rev/1f5d2642929a changeset: 77137:1f5d2642929a branch: 2.7 parent: 77134:83d500908ffc user: Richard Oudkerk date: Fri May 25 12:56:33 2012 +0100 summary: Issue #14881: Allow normal non-main thread to spawn a dummy process Fix suggested by Itay Brandes files: Lib/multiprocessing/dummy/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/dummy/__init__.py b/Lib/multiprocessing/dummy/__init__.py --- a/Lib/multiprocessing/dummy/__init__.py +++ b/Lib/multiprocessing/dummy/__init__.py @@ -70,7 +70,8 @@ def start(self): assert self._parent is current_process() self._start_called = True - self._parent._children[self] = None + if hasattr(self._parent, '_children'): + self._parent._children[self] = None threading.Thread.start(self) @property -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 14:06:42 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 25 May 2012 14:06:42 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0ODgx?= =?utf8?q?=3A_Allow_normal_non-main_thread_to_spawn_a_dummy_process?= Message-ID: http://hg.python.org/cpython/rev/0528ec18e230 changeset: 77138:0528ec18e230 branch: 3.2 parent: 77135:8a58670048c9 user: Richard Oudkerk date: Fri May 25 12:57:58 2012 +0100 summary: Issue #14881: Allow normal non-main thread to spawn a dummy process Fix suggested by Itay Brandes files: Lib/multiprocessing/dummy/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/dummy/__init__.py b/Lib/multiprocessing/dummy/__init__.py --- a/Lib/multiprocessing/dummy/__init__.py +++ b/Lib/multiprocessing/dummy/__init__.py @@ -70,7 +70,8 @@ def start(self): assert self._parent is current_process() self._start_called = True - self._parent._children[self] = None + if hasattr(self._parent, '_children'): + self._parent._children[self] = None threading.Thread.start(self) @property -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 14:06:42 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 25 May 2012 14:06:42 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge?= Message-ID: http://hg.python.org/cpython/rev/9373ca8c6c55 changeset: 77139:9373ca8c6c55 parent: 77136:d2b511592cc5 parent: 77138:0528ec18e230 user: Richard Oudkerk date: Fri May 25 13:04:20 2012 +0100 summary: Merge files: Lib/multiprocessing/dummy/__init__.py | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Lib/multiprocessing/dummy/__init__.py b/Lib/multiprocessing/dummy/__init__.py --- a/Lib/multiprocessing/dummy/__init__.py +++ b/Lib/multiprocessing/dummy/__init__.py @@ -68,7 +68,8 @@ def start(self): assert self._parent is current_process() self._start_called = True - self._parent._children[self] = None + if hasattr(self._parent, '_children'): + self._parent._children[self] = None threading.Thread.start(self) @property -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 14:11:26 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 25 May 2012 14:11:26 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Fixed_markup=2E?= Message-ID: http://hg.python.org/peps/rev/7c3a3d8097c5 changeset: 4435:7c3a3d8097c5 user: Eric V. Smith date: Fri May 25 08:11:19 2012 -0400 summary: Fixed markup. files: pep-0420.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0420.txt b/pep-0420.txt --- a/pep-0420.txt +++ b/pep-0420.txt @@ -452,7 +452,7 @@ 4. Implicit package directories will permanently entrench current newbie-hostile behavior in ``__main__``. -Nick later gave a detailed response to his own objections[5]_, which +Nick later gave a detailed response to his own objections [5]_, which is summarized here: 1. The practicality of this PEP wins over other proposals and the -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 14:29:00 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 25 May 2012 14:29:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312091=3A_simplify_?= =?utf8?q?ApplyResult_and_MapResult_with_threading=2EEvent?= Message-ID: http://hg.python.org/cpython/rev/57d6265beaaa changeset: 77140:57d6265beaaa user: Richard Oudkerk date: Fri May 25 13:26:53 2012 +0100 summary: Issue #12091: simplify ApplyResult and MapResult with threading.Event Patch by Charles-Fran?ois Natali files: Lib/multiprocessing/pool.py | 39 +++++------------------- 1 files changed, 9 insertions(+), 30 deletions(-) diff --git a/Lib/multiprocessing/pool.py b/Lib/multiprocessing/pool.py --- a/Lib/multiprocessing/pool.py +++ b/Lib/multiprocessing/pool.py @@ -526,32 +526,26 @@ class ApplyResult(object): def __init__(self, cache, callback, error_callback): - self._cond = threading.Condition(threading.Lock()) + self._event = threading.Event() self._job = next(job_counter) self._cache = cache - self._ready = False self._callback = callback self._error_callback = error_callback cache[self._job] = self def ready(self): - return self._ready + return self._event.is_set() def successful(self): - assert self._ready + assert self.ready() return self._success def wait(self, timeout=None): - self._cond.acquire() - try: - if not self._ready: - self._cond.wait(timeout) - finally: - self._cond.release() + self._event.wait(timeout) def get(self, timeout=None): self.wait(timeout) - if not self._ready: + if not self.ready(): raise TimeoutError if self._success: return self._value @@ -564,12 +558,7 @@ self._callback(self._value) if self._error_callback and not self._success: self._error_callback(self._value) - self._cond.acquire() - try: - self._ready = True - self._cond.notify() - finally: - self._cond.release() + self._event.set() del self._cache[self._job] # @@ -586,7 +575,7 @@ self._chunksize = chunksize if chunksize <= 0: self._number_left = 0 - self._ready = True + self._event.set() else: self._number_left = length//chunksize + bool(length % chunksize) @@ -599,24 +588,14 @@ if self._callback: self._callback(self._value) del self._cache[self._job] - self._cond.acquire() - try: - self._ready = True - self._cond.notify() - finally: - self._cond.release() + self._event.set() else: self._success = False self._value = result if self._error_callback: self._error_callback(self._value) del self._cache[self._job] - self._cond.acquire() - try: - self._ready = True - self._cond.notify() - finally: - self._cond.release() + self._event.set() # # Class whose instances are returned by `Pool.imap()` -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 14:58:21 2012 From: python-checkins at python.org (richard.oudkerk) Date: Fri, 25 May 2012 14:58:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314548=3A_Make_mult?= =?utf8?q?iprocessing_finalizers_check_pid_before_running?= Message-ID: http://hg.python.org/cpython/rev/59567c117b0e changeset: 77141:59567c117b0e user: Richard Oudkerk date: Fri May 25 13:54:53 2012 +0100 summary: Issue #14548: Make multiprocessing finalizers check pid before running This protects from possibilty of gc running just after fork. files: Lib/multiprocessing/util.py | 12 +++++++++--- Misc/NEWS | 3 +++ 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -9,6 +9,7 @@ import sys import functools +import os import itertools import weakref import atexit @@ -161,6 +162,7 @@ self._args = args self._kwargs = kwargs or {} self._key = (exitpriority, next(_finalizer_counter)) + self._pid = os.getpid() _finalizer_registry[self._key] = self @@ -177,9 +179,13 @@ except KeyError: sub_debug('finalizer no longer registered') else: - sub_debug('finalizer calling %s with args %s and kwargs %s', - self._callback, self._args, self._kwargs) - res = self._callback(*self._args, **self._kwargs) + if self._pid != os.getpid(): + sub_debug('finalizer ignored because different process') + res = None + else: + sub_debug('finalizer calling %s with args %s and kwargs %s', + self._callback, self._args, self._kwargs) + res = self._callback(*self._args, **self._kwargs) self._weakref = self._callback = self._args = \ self._kwargs = self._key = None return res diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #14548: Make multiprocessing finalizers check pid before + running to cope with possibility of gc running just after fork. + - Issue #14863: Update the documentation of os.fdopen() to reflect the fact that it's only a thin wrapper around open() anymore. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 15:21:34 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 25 May 2012 15:21:34 +0200 (CEST) Subject: [Python-checkins] r88964 - tracker/instances/python-dev/detectors/rietveldreactor.py Message-ID: <3VzSly6Y7XzNfP@mail.python.org> Author: martin.v.loewis Date: Fri May 25 15:21:34 2012 New Revision: 88964 Log: Skip rietveldreactor if no rietveld tables are present. Modified: tracker/instances/python-dev/detectors/rietveldreactor.py Modified: tracker/instances/python-dev/detectors/rietveldreactor.py ============================================================================== --- tracker/instances/python-dev/detectors/rietveldreactor.py (original) +++ tracker/instances/python-dev/detectors/rietveldreactor.py Fri May 25 15:21:34 2012 @@ -56,6 +56,11 @@ c.execute("update codereview_issue set cc=%s where id=%s", (cc, nodeid)) def init(db): + c = db.cursor + c.execute("select table_name from information_schema.tables where table_name='auth_user'") + if not c.fetchall(): + # Rietveld tables not present + return db.user.react('create', create_django_user) db.user.react('set', update_django_user) db.issue.react('set', update_issue_cc) From python-checkins at python.org Fri May 25 15:57:14 2012 From: python-checkins at python.org (nick.coghlan) Date: Fri, 25 May 2012 15:57:14 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Record_acceptance_of_PEP_405?= Message-ID: http://hg.python.org/peps/rev/5a26cc296e83 changeset: 4436:5a26cc296e83 user: Nick Coghlan date: Fri May 25 23:57:03 2012 +1000 summary: Record acceptance of PEP 405 files: pep-0405.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0405.txt b/pep-0405.txt --- a/pep-0405.txt +++ b/pep-0405.txt @@ -4,12 +4,13 @@ Last-Modified: $Date$ Author: Carl Meyer BDFL-Delegate: Nick Coghlan -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 13-Jun-2011 Python-Version: 3.3 Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012, 24-May-2012 +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119668.html Abstract -- Repository URL: http://hg.python.org/peps From brett at python.org Fri May 25 16:31:37 2012 From: brett at python.org (Brett Cannon) Date: Fri, 25 May 2012 10:31:37 -0400 Subject: [Python-checkins] cpython: issue 14660: Implement PEP 420, namespace packages. In-Reply-To: References: Message-ID: Is documentation coming in a separate commit? On Thu, May 24, 2012 at 8:22 PM, eric.smith wrote: > http://hg.python.org/cpython/rev/702009f3c0b1 > changeset: 77126:702009f3c0b1 > parent: 77123:f27e098a774a > user: Eric V. Smith > date: Thu May 24 20:21:04 2012 -0400 > summary: > issue 14660: Implement PEP 420, namespace packages. > > files: > Lib/importlib/_bootstrap.py | 154 +++++- > Lib/importlib/test/frozen/test_loader.py | 28 +- > Lib/importlib/test/source/test_finder.py | 19 - > Lib/pkgutil.py | 26 +- > Lib/test/namespace_pkgs/both_portions/foo/one.py | 1 + > Lib/test/namespace_pkgs/both_portions/foo/two.py | 1 + > Lib/test/namespace_pkgs/missing_directory.zip | Bin > Lib/test/namespace_pkgs/nested_portion1.zip | Bin > Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py | 0 > Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py | 1 + > Lib/test/namespace_pkgs/portion1/foo/one.py | 1 + > Lib/test/namespace_pkgs/portion2/foo/two.py | 1 + > Lib/test/namespace_pkgs/project1/parent/child/one.py | 1 + > Lib/test/namespace_pkgs/project2/parent/child/two.py | 1 + > Lib/test/namespace_pkgs/project3/parent/child/three.py | 1 + > Lib/test/namespace_pkgs/top_level_portion1.zip | Bin > Lib/test/test_frozen.py | 2 +- > Lib/test/test_import.py | 6 - > Lib/test/test_module.py | 91 +++ > Lib/test/test_namespace_pkgs.py | 239 > ++++++++++ > Lib/test/test_pkgutil.py | 64 +- > Misc/NEWS | 2 + > Modules/zipimport.c | 148 +++++- > Objects/moduleobject.c | 33 +- > Python/importlib.h | Bin > 25 files changed, 736 insertions(+), 84 deletions(-) > > > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -468,6 +468,10 @@ > """ > > @classmethod > + def module_repr(cls, module): > + return "".format(module.__name__) > + > + @classmethod > def find_module(cls, fullname, path=None): > """Find the built-in module. > > @@ -521,6 +525,10 @@ > """ > > @classmethod > + def module_repr(cls, m): > + return "".format(m.__name__) > + > + @classmethod > def find_module(cls, fullname, path=None): > """Find a frozen module.""" > return cls if _imp.is_frozen(fullname) else None > @@ -533,7 +541,10 @@ > """Load a frozen module.""" > is_reload = fullname in sys.modules > try: > - return _imp.init_frozen(fullname) > + m = _imp.init_frozen(fullname) > + # Let our own module_repr() method produce a suitable repr. > + del m.__file__ > + return m > except: > if not is_reload and fullname in sys.modules: > del sys.modules[fullname] > @@ -875,6 +886,79 @@ > return None > > > +class _NamespacePath: > + """Represents a namespace package's path. It uses the module name > + to find its parent module, and from there it looks up the parent's > + __path__. When this changes, the module's own path is recomputed, > + using path_finder. For top-leve modules, the parent module's path > + is sys.path.""" > + > + def __init__(self, name, path, path_finder): > + self._name = name > + self._path = path > + self._last_parent_path = tuple(self._get_parent_path()) > + self._path_finder = path_finder > + > + def _find_parent_path_names(self): > + """Returns a tuple of (parent-module-name, > parent-path-attr-name)""" > + parent, dot, me = self._name.rpartition('.') > + if dot == '': > + # This is a top-level module. sys.path contains the parent > path. > + return 'sys', 'path' > + # Not a top-level module. parent-module.__path__ contains the > + # parent path. > + return parent, '__path__' > + > + def _get_parent_path(self): > + parent_module_name, path_attr_name = > self._find_parent_path_names() > + return getattr(sys.modules[parent_module_name], path_attr_name) > + > + def _recalculate(self): > + # If the parent's path has changed, recalculate _path > + parent_path = tuple(self._get_parent_path()) # Make a copy > + if parent_path != self._last_parent_path: > + loader, new_path = self._path_finder(self._name, parent_path) > + # Note that no changes are made if a loader is returned, but > we > + # do remember the new parent path > + if loader is None: > + self._path = new_path > + self._last_parent_path = parent_path # Save the copy > + return self._path > + > + def __iter__(self): > + return iter(self._recalculate()) > + > + def __len__(self): > + return len(self._recalculate()) > + > + def __repr__(self): > + return "_NamespacePath({0!r})".format(self._path) > + > + def __contains__(self, item): > + return item in self._recalculate() > + > + def append(self, item): > + self._path.append(item) > + > + > +class NamespaceLoader: > + def __init__(self, name, path, path_finder): > + self._path = _NamespacePath(name, path, path_finder) > + > + @classmethod > + def module_repr(cls, module): > + return "".format(module.__name__) > + > + @set_package > + @set_loader > + @module_for_loader > + def load_module(self, module): > + """Load a namespace module.""" > + _verbose_message('namespace module loaded with path {!r}', > self._path) > + module.__path__ = self._path > + return module > + > + > # Finders > ##################################################################### > > class PathFinder: > @@ -916,19 +1000,46 @@ > return finder > > @classmethod > + def _get_loader(cls, fullname, path): > + """Find the loader or namespace_path for this module/package > name.""" > + # If this ends up being a namespace package, namespace_path is > + # the list of paths that will become its __path__ > + namespace_path = [] > + for entry in path: > + finder = cls._path_importer_cache(entry) > + if finder is not None: > + if hasattr(finder, 'find_loader'): > + loader, portions = finder.find_loader(fullname) > + else: > + loader = finder.find_module(fullname) > + portions = [] > + if loader is not None: > + # We found a loader: return it immediately. > + return (loader, namespace_path) > + # This is possibly part of a namespace package. > + # Remember these path entries (if any) for when we > + # create a namespace package, and continue iterating > + # on path. > + namespace_path.extend(portions) > + else: > + return (None, namespace_path) > + > + @classmethod > def find_module(cls, fullname, path=None): > """Find the module on sys.path or 'path' based on sys.path_hooks > and > sys.path_importer_cache.""" > if path is None: > path = sys.path > - for entry in path: > - finder = cls._path_importer_cache(entry) > - if finder is not None: > - loader = finder.find_module(fullname) > - if loader: > - return loader > + loader, namespace_path = cls._get_loader(fullname, path) > + if loader is not None: > + return loader > else: > - return None > + if namespace_path: > + # We found at least one namespace path. Return a > + # loader which can create the namespace package. > + return NamespaceLoader(fullname, namespace_path, > cls._get_loader) > + else: > + return None > > > class FileFinder: > @@ -942,8 +1053,8 @@ > > def __init__(self, path, *details): > """Initialize with the path to search on and a variable number of > - 3-tuples containing the loader, file suffixes the loader > recognizes, and > - a boolean of whether the loader handles packages.""" > + 3-tuples containing the loader, file suffixes the loader > recognizes, > + and a boolean of whether the loader handles packages.""" > packages = [] > modules = [] > for loader, suffixes, supports_packages in details: > @@ -964,6 +1075,19 @@ > > def find_module(self, fullname): > """Try to find a loader for the specified module.""" > + # Call find_loader(). If it returns a string (indicating this > + # is a namespace package portion), generate a warning and > + # return None. > + loader, portions = self.find_loader(fullname) > + assert len(portions) in [0, 1] > + if loader is None and len(portions): > + msg = "Not importing directory {}: missing __init__" > + _warnings.warn(msg.format(portions[0]), ImportWarning) > + return loader > + > + def find_loader(self, fullname): > + """Try to find a loader for the specified module, or the namespace > + package portions. Returns (loader, list-of-portions).""" > tail_module = fullname.rpartition('.')[2] > try: > mtime = _os.stat(self.path).st_mtime > @@ -987,17 +1111,17 @@ > init_filename = '__init__' + suffix > full_path = _path_join(base_path, init_filename) > if _path_isfile(full_path): > - return loader(fullname, full_path) > + return (loader(fullname, full_path), [base_path]) > else: > - msg = "Not importing directory {}: missing __init__" > - _warnings.warn(msg.format(base_path), ImportWarning) > + # A namespace package, return the path > + return (None, [base_path]) > # Check for a file w/ a proper suffix exists. > for suffix, loader in self.modules: > if cache_module + suffix in cache: > full_path = _path_join(self.path, tail_module + suffix) > if _path_isfile(full_path): > - return loader(fullname, full_path) > - return None > + return (loader(fullname, full_path), []) > + return (None, []) > > def _fill_cache(self): > """Fill the cache of potential modules and packages for this > directory.""" > diff --git a/Lib/importlib/test/frozen/test_loader.py > b/Lib/importlib/test/frozen/test_loader.py > --- a/Lib/importlib/test/frozen/test_loader.py > +++ b/Lib/importlib/test/frozen/test_loader.py > @@ -10,38 +10,46 @@ > def test_module(self): > with util.uncache('__hello__'), captured_stdout() as stdout: > module = machinery.FrozenImporter.load_module('__hello__') > - check = {'__name__': '__hello__', '__file__': '', > - '__package__': '', '__loader__': > machinery.FrozenImporter} > + check = {'__name__': '__hello__', > + '__package__': '', > + '__loader__': machinery.FrozenImporter, > + } > for attr, value in check.items(): > self.assertEqual(getattr(module, attr), value) > self.assertEqual(stdout.getvalue(), 'Hello world!\n') > + self.assertFalse(hasattr(module, '__file__')) > > def test_package(self): > with util.uncache('__phello__'), captured_stdout() as stdout: > module = machinery.FrozenImporter.load_module('__phello__') > - check = {'__name__': '__phello__', '__file__': '', > - '__package__': '__phello__', '__path__': > ['__phello__'], > - '__loader__': machinery.FrozenImporter} > + check = {'__name__': '__phello__', > + '__package__': '__phello__', > + '__path__': ['__phello__'], > + '__loader__': machinery.FrozenImporter, > + } > for attr, value in check.items(): > attr_value = getattr(module, attr) > self.assertEqual(attr_value, value, > "for __phello__.%s, %r != %r" % > (attr, attr_value, value)) > self.assertEqual(stdout.getvalue(), 'Hello world!\n') > + self.assertFalse(hasattr(module, '__file__')) > > def test_lacking_parent(self): > with util.uncache('__phello__', '__phello__.spam'), \ > captured_stdout() as stdout: > module = > machinery.FrozenImporter.load_module('__phello__.spam') > - check = {'__name__': '__phello__.spam', '__file__': > '', > + check = {'__name__': '__phello__.spam', > '__package__': '__phello__', > - '__loader__': machinery.FrozenImporter} > + '__loader__': machinery.FrozenImporter, > + } > for attr, value in check.items(): > attr_value = getattr(module, attr) > self.assertEqual(attr_value, value, > "for __phello__.spam.%s, %r != %r" % > (attr, attr_value, value)) > self.assertEqual(stdout.getvalue(), 'Hello world!\n') > + self.assertFalse(hasattr(module, '__file__')) > > def test_module_reuse(self): > with util.uncache('__hello__'), captured_stdout() as stdout: > @@ -51,6 +59,12 @@ > self.assertEqual(stdout.getvalue(), > 'Hello world!\nHello world!\n') > > + def test_module_repr(self): > + with util.uncache('__hello__'), captured_stdout(): > + module = machinery.FrozenImporter.load_module('__hello__') > + self.assertEqual(repr(module), > + "") > + > def test_state_after_failure(self): > # No way to trigger an error in a frozen module. > pass > diff --git a/Lib/importlib/test/source/test_finder.py > b/Lib/importlib/test/source/test_finder.py > --- a/Lib/importlib/test/source/test_finder.py > +++ b/Lib/importlib/test/source/test_finder.py > @@ -106,36 +106,17 @@ > loader = self.import_(pkg_dir, 'pkg.sub') > self.assertTrue(hasattr(loader, 'load_module')) > > - # [sub empty] > - def test_empty_sub_directory(self): > - context = source_util.create_modules('pkg.__init__', > 'pkg.sub.__init__') > - with warnings.catch_warnings(): > - warnings.simplefilter("error", ImportWarning) > - with context as mapping: > - os.unlink(mapping['pkg.sub.__init__']) > - pkg_dir = os.path.dirname(mapping['pkg.__init__']) > - with self.assertRaises(ImportWarning): > - self.import_(pkg_dir, 'pkg.sub') > - > # [package over modules] > def test_package_over_module(self): > name = '_temp' > loader = self.run_test(name, {'{0}.__init__'.format(name), name}) > self.assertTrue('__init__' in loader.get_filename(name)) > > - > def test_failure(self): > with source_util.create_modules('blah') as mapping: > nothing = self.import_(mapping['.root'], 'sdfsadsadf') > self.assertTrue(nothing is None) > > - # [empty dir] > - def test_empty_dir(self): > - with warnings.catch_warnings(): > - warnings.simplefilter("error", ImportWarning) > - with self.assertRaises(ImportWarning): > - self.run_test('pkg', {'pkg.__init__'}, > unlink={'pkg.__init__'}) > - > def test_empty_string_for_dir(self): > # The empty string from sys.path means to search in the cwd. > finder = machinery.FileFinder('', (machinery.SourceFileLoader, > diff --git a/Lib/pkgutil.py b/Lib/pkgutil.py > --- a/Lib/pkgutil.py > +++ b/Lib/pkgutil.py > @@ -515,19 +515,29 @@ > > pname = os.path.join(*name.split('.')) # Reconstitute as relative path > sname_pkg = name + ".pkg" > - init_py = "__init__.py" > > path = path[:] # Start with a copy of the existing path > > for dir in sys.path: > - if not isinstance(dir, str) or not os.path.isdir(dir): > + if not isinstance(dir, str): > continue > - subdir = os.path.join(dir, pname) > - # XXX This may still add duplicate entries to path on > - # case-insensitive filesystems > - initfile = os.path.join(subdir, init_py) > - if subdir not in path and os.path.isfile(initfile): > - path.append(subdir) > + > + finder = get_importer(dir) > + if finder is not None: > + # Is this finder PEP 420 compliant? > + if hasattr(finder, 'find_loader'): > + loader, portions = finder.find_loader(name) > + else: > + # No, no need to call it > + loader = None > + portions = [] > + > + for portion in portions: > + # XXX This may still add duplicate entries to path on > + # case-insensitive filesystems > + if portion not in path: > + path.append(portion) > + > # XXX Is this the right thing for subpackages like zope.app? > # It looks for a file named "zope.app.pkg" > pkgfile = os.path.join(dir, sname_pkg) > diff --git a/Lib/test/namespace_pkgs/both_portions/foo/one.py > b/Lib/test/namespace_pkgs/both_portions/foo/one.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/both_portions/foo/one.py > @@ -0,0 +1,1 @@ > +attr = 'both_portions foo one' > diff --git a/Lib/test/namespace_pkgs/both_portions/foo/two.py > b/Lib/test/namespace_pkgs/both_portions/foo/two.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/both_portions/foo/two.py > @@ -0,0 +1,1 @@ > +attr = 'both_portions foo two' > diff --git a/Lib/test/namespace_pkgs/missing_directory.zip > b/Lib/test/namespace_pkgs/missing_directory.zip > new file mode 100644 > index > 0000000000000000000000000000000000000000..836a9106bcdd5d171dcf54386bb51e0f7b6a520c > GIT binary patch > [stripped] > diff --git a/Lib/test/namespace_pkgs/nested_portion1.zip > b/Lib/test/namespace_pkgs/nested_portion1.zip > new file mode 100644 > index > 0000000000000000000000000000000000000000..8d22406f23758f3d0d138cd49c651c5c52e1f84f > GIT binary patch > [stripped] > diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py > b/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/__init__.py > new file mode 100644 > diff --git a/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py > b/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/not_a_namespace_pkg/foo/one.py > @@ -0,0 +1,1 @@ > +attr = 'portion1 foo one' > diff --git a/Lib/test/namespace_pkgs/portion1/foo/one.py > b/Lib/test/namespace_pkgs/portion1/foo/one.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/portion1/foo/one.py > @@ -0,0 +1,1 @@ > +attr = 'portion1 foo one' > diff --git a/Lib/test/namespace_pkgs/portion2/foo/two.py > b/Lib/test/namespace_pkgs/portion2/foo/two.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/portion2/foo/two.py > @@ -0,0 +1,1 @@ > +attr = 'portion2 foo two' > diff --git a/Lib/test/namespace_pkgs/project1/parent/child/one.py > b/Lib/test/namespace_pkgs/project1/parent/child/one.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/project1/parent/child/one.py > @@ -0,0 +1,1 @@ > +attr = 'parent child one' > diff --git a/Lib/test/namespace_pkgs/project2/parent/child/two.py > b/Lib/test/namespace_pkgs/project2/parent/child/two.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/project2/parent/child/two.py > @@ -0,0 +1,1 @@ > +attr = 'parent child two' > diff --git a/Lib/test/namespace_pkgs/project3/parent/child/three.py > b/Lib/test/namespace_pkgs/project3/parent/child/three.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/namespace_pkgs/project3/parent/child/three.py > @@ -0,0 +1,1 @@ > +attr = 'parent child three' > diff --git a/Lib/test/namespace_pkgs/top_level_portion1.zip > b/Lib/test/namespace_pkgs/top_level_portion1.zip > new file mode 100644 > index > 0000000000000000000000000000000000000000..3b866c914ad2f2fe348405799a482235854bac10 > GIT binary patch > [stripped] > diff --git a/Lib/test/test_frozen.py b/Lib/test/test_frozen.py > --- a/Lib/test/test_frozen.py > +++ b/Lib/test/test_frozen.py > @@ -7,7 +7,7 @@ > class FrozenTests(unittest.TestCase): > > module_attrs = frozenset(['__builtins__', '__cached__', '__doc__', > - '__file__', '__loader__', '__name__', > + '__loader__', '__name__', > '__package__']) > package_attrs = frozenset(list(module_attrs) + ['__path__']) > > diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py > --- a/Lib/test/test_import.py > +++ b/Lib/test/test_import.py > @@ -286,12 +286,6 @@ > import test.support as y > self.assertIs(y, test.support, y.__name__) > > - def test_import_initless_directory_warning(self): > - with check_warnings(('', ImportWarning)): > - # Just a random non-package directory we always expect to be > - # somewhere in sys.path... > - self.assertRaises(ImportError, __import__, "site-packages") > - > def test_import_by_filename(self): > path = os.path.abspath(TESTFN) > encoding = sys.getfilesystemencoding() > diff --git a/Lib/test/test_module.py b/Lib/test/test_module.py > --- a/Lib/test/test_module.py > +++ b/Lib/test/test_module.py > @@ -5,6 +5,15 @@ > import sys > ModuleType = type(sys) > > +class FullLoader: > + @classmethod > + def module_repr(cls, m): > + return "".format(m.__name__) > + > +class BareLoader: > + pass > + > + > class ModuleTests(unittest.TestCase): > def test_uninitialized(self): > # An uninitialized module has no __dict__ or __name__, > @@ -80,8 +89,90 @@ > gc_collect() > self.assertEqual(destroyed, [1]) > > + def test_module_repr_minimal(self): > + # reprs when modules have no __file__, __name__, or __loader__ > + m = ModuleType('foo') > + del m.__name__ > + self.assertEqual(repr(m), "") > + > + def test_module_repr_with_name(self): > + m = ModuleType('foo') > + self.assertEqual(repr(m), "") > + > + def test_module_repr_with_name_and_filename(self): > + m = ModuleType('foo') > + m.__file__ = '/tmp/foo.py' > + self.assertEqual(repr(m), "") > + > + def test_module_repr_with_filename_only(self): > + m = ModuleType('foo') > + del m.__name__ > + m.__file__ = '/tmp/foo.py' > + self.assertEqual(repr(m), "") > + > + def test_module_repr_with_bare_loader_but_no_name(self): > + m = ModuleType('foo') > + del m.__name__ > + # Yes, a class not an instance. > + m.__loader__ = BareLoader > + self.assertEqual( > + repr(m), " 'test.test_module.BareLoader'>)>") > + > + def test_module_repr_with_full_loader_but_no_name(self): > + # m.__loader__.module_repr() will fail because the module has no > + # m.__name__. This exception will get suppressed and instead the > + # loader's repr will be used. > + m = ModuleType('foo') > + del m.__name__ > + # Yes, a class not an instance. > + m.__loader__ = FullLoader > + self.assertEqual( > + repr(m), " 'test.test_module.FullLoader'>)>") > + > + def test_module_repr_with_bare_loader(self): > + m = ModuleType('foo') > + # Yes, a class not an instance. > + m.__loader__ = BareLoader > + self.assertEqual( > + repr(m), " 'test.test_module.BareLoader'>)>") > + > + def test_module_repr_with_full_loader(self): > + m = ModuleType('foo') > + # Yes, a class not an instance. > + m.__loader__ = FullLoader > + self.assertEqual( > + repr(m), "") > + > + def test_module_repr_with_bare_loader_and_filename(self): > + # Because the loader has no module_repr(), use the file name. > + m = ModuleType('foo') > + # Yes, a class not an instance. > + m.__loader__ = BareLoader > + m.__file__ = '/tmp/foo.py' > + self.assertEqual(repr(m), "") > + > + def test_module_repr_with_full_loader_and_filename(self): > + # Even though the module has an __file__, use > __loader__.module_repr() > + m = ModuleType('foo') > + # Yes, a class not an instance. > + m.__loader__ = FullLoader > + m.__file__ = '/tmp/foo.py' > + self.assertEqual(repr(m), "") > + > + def test_module_repr_builtin(self): > + self.assertEqual(repr(sys), "") > + > + def test_module_repr_source(self): > + r = repr(unittest) > + self.assertEqual(r[:25], " + self.assertEqual(r[-13:], "__init__.py'>") > + > + # frozen and namespace module reprs are tested in importlib. > + > + > def test_main(): > run_unittest(ModuleTests) > > + > if __name__ == '__main__': > test_main() > diff --git a/Lib/test/test_namespace_pkgs.py > b/Lib/test/test_namespace_pkgs.py > new file mode 100644 > --- /dev/null > +++ b/Lib/test/test_namespace_pkgs.py > @@ -0,0 +1,239 @@ > +import sys > +import contextlib > +import unittest > +import os > + > +import importlib.test.util > +from test.support import run_unittest > + > +# needed tests: > +# > +# need to test when nested, so that the top-level path isn't sys.path > +# need to test dynamic path detection, both at top-level and nested > +# with dynamic path, check when a loader is returned on path reload (that > is, > +# trying to switch from a namespace package to a regular package) > + > + > + at contextlib.contextmanager > +def sys_modules_context(): > + """ > + Make sure sys.modules is the same object and has the same content > + when exiting the context as when entering. > + > + Similar to importlib.test.util.uncache, but doesn't require explicit > + names. > + """ > + sys_modules_saved = sys.modules > + sys_modules_copy = sys.modules.copy() > + try: > + yield > + finally: > + sys.modules = sys_modules_saved > + sys.modules.clear() > + sys.modules.update(sys_modules_copy) > + > + > + at contextlib.contextmanager > +def namespace_tree_context(**kwargs): > + """ > + Save import state and sys.modules cache and restore it on exit. > + Typical usage: > + > + >>> with namespace_tree_context(path=['/tmp/xxyy/portion1', > + ... '/tmp/xxyy/portion2']): > + ... pass > + """ > + # use default meta_path and path_hooks unless specified otherwise > + kwargs.setdefault('meta_path', sys.meta_path) > + kwargs.setdefault('path_hooks', sys.path_hooks) > + import_context = importlib.test.util.import_state(**kwargs) > + with import_context, sys_modules_context(): > + yield > + > +class NamespacePackageTest(unittest.TestCase): > + """ > + Subclasses should define self.root and self.paths (under that root) > + to be added to sys.path. > + """ > + root = os.path.join(os.path.dirname(__file__), 'namespace_pkgs') > + > + def setUp(self): > + self.resolved_paths = [ > + os.path.join(self.root, path) for path in self.paths > + ] > + self.ctx = namespace_tree_context(path=self.resolved_paths) > + self.ctx.__enter__() > + > + def tearDown(self): > + # TODO: will we ever want to pass exc_info to __exit__? > + self.ctx.__exit__(None, None, None) > + > +class SingleNamespacePackage(NamespacePackageTest): > + paths = ['portion1'] > + > + def test_simple_package(self): > + import foo.one > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + > + def test_cant_import_other(self): > + with self.assertRaises(ImportError): > + import foo.two > + > + def test_module_repr(self): > + import foo.one > + self.assertEqual(repr(foo), "") > + > + > +class DynamicPatheNamespacePackage(NamespacePackageTest): > + paths = ['portion1'] > + > + def test_dynamic_path(self): > + # Make sure only 'foo.one' can be imported > + import foo.one > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + > + with self.assertRaises(ImportError): > + import foo.two > + > + # Now modify sys.path > + sys.path.append(os.path.join(self.root, 'portion2')) > + > + # And make sure foo.two is now importable > + import foo.two > + self.assertEqual(foo.two.attr, 'portion2 foo two') > + > + > +class CombinedNamespacePackages(NamespacePackageTest): > + paths = ['both_portions'] > + > + def test_imports(self): > + import foo.one > + import foo.two > + self.assertEqual(foo.one.attr, 'both_portions foo one') > + self.assertEqual(foo.two.attr, 'both_portions foo two') > + > + > +class SeparatedNamespacePackages(NamespacePackageTest): > + paths = ['portion1', 'portion2'] > + > + def test_imports(self): > + import foo.one > + import foo.two > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + self.assertEqual(foo.two.attr, 'portion2 foo two') > + > + > +class SeparatedOverlappingNamespacePackages(NamespacePackageTest): > + paths = ['portion1', 'both_portions'] > + > + def test_first_path_wins(self): > + import foo.one > + import foo.two > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + self.assertEqual(foo.two.attr, 'both_portions foo two') > + > + def test_first_path_wins_again(self): > + sys.path.reverse() > + import foo.one > + import foo.two > + self.assertEqual(foo.one.attr, 'both_portions foo one') > + self.assertEqual(foo.two.attr, 'both_portions foo two') > + > + def test_first_path_wins_importing_second_first(self): > + import foo.two > + import foo.one > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + self.assertEqual(foo.two.attr, 'both_portions foo two') > + > + > +class SingleZipNamespacePackage(NamespacePackageTest): > + paths = ['top_level_portion1.zip'] > + > + def test_simple_package(self): > + import foo.one > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + > + def test_cant_import_other(self): > + with self.assertRaises(ImportError): > + import foo.two > + > + > +class SeparatedZipNamespacePackages(NamespacePackageTest): > + paths = ['top_level_portion1.zip', 'portion2'] > + > + def test_imports(self): > + import foo.one > + import foo.two > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + self.assertEqual(foo.two.attr, 'portion2 foo two') > + self.assertIn('top_level_portion1.zip', foo.one.__file__) > + self.assertNotIn('.zip', foo.two.__file__) > + > + > +class SingleNestedZipNamespacePackage(NamespacePackageTest): > + paths = ['nested_portion1.zip/nested_portion1'] > + > + def test_simple_package(self): > + import foo.one > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + > + def test_cant_import_other(self): > + with self.assertRaises(ImportError): > + import foo.two > + > + > +class SeparatedNestedZipNamespacePackages(NamespacePackageTest): > + paths = ['nested_portion1.zip/nested_portion1', 'portion2'] > + > + def test_imports(self): > + import foo.one > + import foo.two > + self.assertEqual(foo.one.attr, 'portion1 foo one') > + self.assertEqual(foo.two.attr, 'portion2 foo two') > + fn = os.path.join('nested_portion1.zip', 'nested_portion1') > + self.assertIn(fn, foo.one.__file__) > + self.assertNotIn('.zip', foo.two.__file__) > + > + > +class LegacySupport(NamespacePackageTest): > + paths = ['not_a_namespace_pkg', 'portion1', 'portion2', > 'both_portions'] > + > + def test_non_namespace_package_takes_precedence(self): > + import foo.one > + with self.assertRaises(ImportError): > + import foo.two > + self.assertIn('__init__', foo.__file__) > + self.assertNotIn('namespace', str(foo.__loader__).lower()) > + > + > +class ZipWithMissingDirectory(NamespacePackageTest): > + paths = ['missing_directory.zip'] > + > + @unittest.expectedFailure > + def test_missing_directory(self): > + # This will fail because missing_directory.zip contains: > + # Length Date Time Name > + # --------- ---------- ----- ---- > + # 29 2012-05-03 18:13 foo/one.py > + # 0 2012-05-03 20:57 bar/ > + # 38 2012-05-03 20:57 bar/two.py > + # --------- ------- > + # 67 3 files > + > + # Because there is no 'foo/', the zipimporter currently doesn't > + # know that foo is a namespace package > + > + import foo.one > + > + def test_present_directory(self): > + # This succeeds because there is a "bar/" in the zip file > + import bar.two > + self.assertEqual(bar.two.attr, 'missing_directory foo two') > + > + > +def test_main(): > + run_unittest(*NamespacePackageTest.__subclasses__()) > + > + > +if __name__ == "__main__": > + test_main() > diff --git a/Lib/test/test_pkgutil.py b/Lib/test/test_pkgutil.py > --- a/Lib/test/test_pkgutil.py > +++ b/Lib/test/test_pkgutil.py > @@ -138,10 +138,11 @@ > del sys.modules['foo'] > > > +# These tests, especially the setup and cleanup, are hideous. They > +# need to be cleaned up once issue 14715 is addressed. > class ExtendPathTests(unittest.TestCase): > def create_init(self, pkgname): > dirname = tempfile.mkdtemp() > - self.addCleanup(shutil.rmtree, dirname) > sys.path.insert(0, dirname) > > pkgdir = os.path.join(dirname, pkgname) > @@ -156,22 +157,12 @@ > with open(module_name, 'w') as fl: > print('value={}'.format(value), file=fl) > > - def setUp(self): > - # Create 2 directories on sys.path > - self.pkgname = 'foo' > - self.dirname_0 = self.create_init(self.pkgname) > - self.dirname_1 = self.create_init(self.pkgname) > - > - def tearDown(self): > - del sys.path[0] > - del sys.path[0] > - del sys.modules['foo'] > - del sys.modules['foo.bar'] > - del sys.modules['foo.baz'] > - > def test_simple(self): > - self.create_submodule(self.dirname_0, self.pkgname, 'bar', 0) > - self.create_submodule(self.dirname_1, self.pkgname, 'baz', 1) > + pkgname = 'foo' > + dirname_0 = self.create_init(pkgname) > + dirname_1 = self.create_init(pkgname) > + self.create_submodule(dirname_0, pkgname, 'bar', 0) > + self.create_submodule(dirname_1, pkgname, 'baz', 1) > import foo.bar > import foo.baz > # Ensure we read the expected values > @@ -180,8 +171,45 @@ > > # Ensure the path is set up correctly > self.assertEqual(sorted(foo.__path__), > - sorted([os.path.join(self.dirname_0, > self.pkgname), > - os.path.join(self.dirname_1, > self.pkgname)])) > + sorted([os.path.join(dirname_0, pkgname), > + os.path.join(dirname_1, pkgname)])) > + > + # Cleanup > + shutil.rmtree(dirname_0) > + shutil.rmtree(dirname_1) > + del sys.path[0] > + del sys.path[0] > + del sys.modules['foo'] > + del sys.modules['foo.bar'] > + del sys.modules['foo.baz'] > + > + def test_mixed_namespace(self): > + pkgname = 'foo' > + dirname_0 = self.create_init(pkgname) > + dirname_1 = self.create_init(pkgname) > + self.create_submodule(dirname_0, pkgname, 'bar', 0) > + # Turn this into a PEP 420 namespace package > + os.unlink(os.path.join(dirname_0, pkgname, '__init__.py')) > + self.create_submodule(dirname_1, pkgname, 'baz', 1) > + import foo.bar > + import foo.baz > + # Ensure we read the expected values > + self.assertEqual(foo.bar.value, 0) > + self.assertEqual(foo.baz.value, 1) > + > + # Ensure the path is set up correctly > + self.assertEqual(sorted(foo.__path__), > + sorted([os.path.join(dirname_0, pkgname), > + os.path.join(dirname_1, pkgname)])) > + > + # Cleanup > + shutil.rmtree(dirname_0) > + shutil.rmtree(dirname_1) > + del sys.path[0] > + del sys.path[0] > + del sys.modules['foo'] > + del sys.modules['foo.bar'] > + del sys.modules['foo.baz'] > > # XXX: test .pkg files > > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -10,6 +10,8 @@ > Core and Builtins > ----------------- > > +- Issue #14660 (PEP 420): Namespace packages. Implemented by Eric Smith. > + > - Issue #14494: Fix __future__.py and its documentation to note that > absolute imports are the default behavior in 3.0 instead of 2.7. > Patch by Sven Marnach. > diff --git a/Modules/zipimport.c b/Modules/zipimport.c > --- a/Modules/zipimport.c > +++ b/Modules/zipimport.c > @@ -259,6 +259,29 @@ > MI_PACKAGE > }; > > +/* Does this path represent a directory? > + on error, return < 0 > + if not a dir, return 0 > + if a dir, return 1 > +*/ > +static int > +check_is_directory(ZipImporter *self, PyObject* prefix, PyObject *path) > +{ > + PyObject *dirpath; > + PyObject *item; > + > + /* See if this is a "directory". If so, it's eligible to be part > + of a namespace package. We test by seeing if the name, with an > + appended path separator, exists. */ > + dirpath = PyUnicode_FromFormat("%U%U%c", prefix, path, SEP); > + if (dirpath == NULL) > + return -1; > + /* If dirpath is present in self->files, we have a directory. */ > + item = PyDict_GetItem(self->files, dirpath); > + Py_DECREF(dirpath); > + return item != NULL; > +} > + > /* Return some information about a module. */ > static enum zi_module_info > get_module_info(ZipImporter *self, PyObject *fullname) > @@ -296,6 +319,46 @@ > return MI_NOT_FOUND; > } > > +/* The guts of "find_loader" and "find_module". Return values: > + -1: error > + 0: no loader or namespace portions found > + 1: module/package found > + 2: namespace portion found: *namespace_portion will point to the name > +*/ > +static int > +find_loader(ZipImporter *self, PyObject *fullname, PyObject > **namespace_portion) > +{ > + enum zi_module_info mi; > + > + *namespace_portion = NULL; > + > + mi = get_module_info(self, fullname); > + if (mi == MI_ERROR) > + return -1; > + if (mi == MI_NOT_FOUND) { > + /* Not a module or regular package. See if this is a directory, > and > + therefore possibly a portion of a namespace package. */ > + int is_dir = check_is_directory(self, self->prefix, fullname); > + if (is_dir < 0) > + return -1; > + if (is_dir) { > + /* This is possibly a portion of a namespace > + package. Return the string representing its path, > + without a trailing separator. */ > + *namespace_portion = PyUnicode_FromFormat("%U%c%U%U", > + self->archive, SEP, > + self->prefix, > fullname); > + if (*namespace_portion == NULL) > + return -1; > + return 2; > + } > + return 0; > + } > + /* This is a module or package. */ > + return 1; > +} > + > + > /* Check whether we can satisfy the import of the module named by > 'fullname'. Return self if we can, None if we can't. */ > static PyObject * > @@ -304,21 +367,78 @@ > ZipImporter *self = (ZipImporter *)obj; > PyObject *path = NULL; > PyObject *fullname; > - enum zi_module_info mi; > + PyObject* namespace_portion = NULL; > > if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", > &fullname, &path)) > - return NULL; > + goto error; > > - mi = get_module_info(self, fullname); > - if (mi == MI_ERROR) > - return NULL; > - if (mi == MI_NOT_FOUND) { > + switch (find_loader(self, fullname, &namespace_portion)) { > + case -1: /* Error */ > + goto error; > + case 0: /* Not found, return None */ > + Py_INCREF(Py_None); > + return Py_None; > + case 1: /* Return self */ > + Py_INCREF(self); > + return (PyObject *)self; > + case 2: /* A namespace portion, but not allowed via > + find_module, so return None */ > + Py_DECREF(namespace_portion); > Py_INCREF(Py_None); > return Py_None; > } > - Py_INCREF(self); > - return (PyObject *)self; > + /* Can't get here. */ > + assert(0); > + return NULL; > +error: > + Py_XDECREF(namespace_portion); > + return NULL; > +} > + > + > +/* Check whether we can satisfy the import of the module named by > + 'fullname', or whether it could be a portion of a namespace > + package. Return self if we can load it, a string containing the > + full path if it's a possible namespace portion, None if we > + can't load it. */ > +static PyObject * > +zipimporter_find_loader(PyObject *obj, PyObject *args) > +{ > + ZipImporter *self = (ZipImporter *)obj; > + PyObject *path = NULL; > + PyObject *fullname; > + PyObject *result = NULL; > + PyObject *namespace_portion = NULL; > + > + if (!PyArg_ParseTuple(args, "U|O:zipimporter.find_module", > + &fullname, &path)) > + goto error; > + > + switch (find_loader(self, fullname, &namespace_portion)) { > + case -1: /* Error */ > + goto error; > + case 0: /* Not found, return (None, []) */ > + if (!(result = Py_BuildValue("O[]", Py_None))) > + goto error; > + return result; > + case 1: /* Return (self, []) */ > + if (!(result = Py_BuildValue("O[]", self))) > + goto error; > + return result; > + case 2: /* Return (None, [namespace_portion]) */ > + if (!(result = Py_BuildValue("O[O]", Py_None, namespace_portion))) > + goto error; > + return result; > + } > + /* Can't get here. */ > + assert(0); > + return NULL; > + > +error: > + Py_XDECREF(namespace_portion); > + Py_XDECREF(result); > + return NULL; > } > > /* Load and return the module named by 'fullname'. */ > @@ -558,6 +678,16 @@ > The optional 'path' argument is ignored -- it's there for compatibility\n\ > with the importer protocol."); > > +PyDoc_STRVAR(doc_find_loader, > +"find_loader(fullname, path=None) -> self, str or None.\n\ > +\n\ > +Search for a module specified by 'fullname'. 'fullname' must be the\n\ > +fully qualified (dotted) module name. It returns the zipimporter\n\ > +instance itself if the module was found, a string containing the\n\ > +full path name if it's possibly a portion of a namespace package,\n\ > +or None otherwise. The optional 'path' argument is ignored -- it's\n\ > + there for compatibility with the importer protocol."); > + > PyDoc_STRVAR(doc_load_module, > "load_module(fullname) -> module.\n\ > \n\ > @@ -599,6 +729,8 @@ > static PyMethodDef zipimporter_methods[] = { > {"find_module", zipimporter_find_module, METH_VARARGS, > doc_find_module}, > + {"find_loader", zipimporter_find_loader, METH_VARARGS, > + doc_find_loader}, > {"load_module", zipimporter_load_module, METH_VARARGS, > doc_load_module}, > {"get_data", zipimporter_get_data, METH_VARARGS, > diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c > --- a/Objects/moduleobject.c > +++ b/Objects/moduleobject.c > @@ -366,8 +366,28 @@ > static PyObject * > module_repr(PyModuleObject *m) > { > - PyObject *name, *filename, *repr; > + PyObject *name, *filename, *repr, *loader = NULL; > > + /* See if the module has an __loader__. If it does, give the loader > the > + * first shot at producing a repr for the module. > + */ > + if (m->md_dict != NULL) { > + loader = PyDict_GetItemString(m->md_dict, "__loader__"); > + } > + if (loader != NULL) { > + repr = PyObject_CallMethod(loader, "module_repr", "(O)", > + (PyObject *)m, NULL); > + if (repr == NULL) { > + PyErr_Clear(); > + } > + else { > + return repr; > + } > + } > + /* __loader__.module_repr(m) did not provide us with a repr. Next, > see if > + * the module has an __file__. If it doesn't then use > repr(__loader__) if > + * it exists, otherwise, just use module.__name__. > + */ > name = PyModule_GetNameObject((PyObject *)m); > if (name == NULL) { > PyErr_Clear(); > @@ -378,8 +398,17 @@ > filename = PyModule_GetFilenameObject((PyObject *)m); > if (filename == NULL) { > PyErr_Clear(); > - repr = PyUnicode_FromFormat("", name); > + /* There's no m.__file__, so if there was an __loader__, use that > in > + * the repr, otherwise, the only thing you can use is m.__name__ > + */ > + if (loader == NULL) { > + repr = PyUnicode_FromFormat("", name); > + } > + else { > + repr = PyUnicode_FromFormat("", name, loader); > + } > } > + /* Finally, use m.__file__ */ > else { > repr = PyUnicode_FromFormat("", name, filename); > Py_DECREF(filename); > diff --git a/Python/importlib.h b/Python/importlib.h > index > 493cd1aac17510f62321955f9d7a45836ef23a7d..bdb3644db479da7d7f6b949be135c6b1c3d1fc33 > GIT binary patch > [stripped] > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri May 25 16:41:30 2012 From: barry at python.org (Barry Warsaw) Date: Fri, 25 May 2012 10:41:30 -0400 Subject: [Python-checkins] cpython: issue 14660: Implement PEP 420, namespace packages. In-Reply-To: References: Message-ID: <20120525104130.19b416fb@limelight.wooz.org> On May 25, 2012, at 10:31 AM, Brett Cannon wrote: >Is documentation coming in a separate commit? Yes. I've been reworking the import machinery documentation; it's a work-in-progress on the pep-420 feature clone ('importdocs' branch). I made some good progress and then got side-tracked, but I'm planning on getting back to it soon. -Barry From python-checkins at python.org Fri May 25 17:25:35 2012 From: python-checkins at python.org (eric.smith) Date: Fri, 25 May 2012 17:25:35 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Added_test_for_namespace_pa?= =?utf8?q?ckage_dynamic_path_updates=2E?= Message-ID: http://hg.python.org/cpython/rev/70ec1f671c19 changeset: 77142:70ec1f671c19 user: Eric V. Smith date: Fri May 25 11:25:27 2012 -0400 summary: Added test for namespace package dynamic path updates. files: Lib/test/test_namespace_pkgs.py | 45 +++++++++++++++++++++ 1 files changed, 45 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_namespace_pkgs.py b/Lib/test/test_namespace_pkgs.py --- a/Lib/test/test_namespace_pkgs.py +++ b/Lib/test/test_namespace_pkgs.py @@ -206,6 +206,51 @@ self.assertNotIn('namespace', str(foo.__loader__).lower()) +class DynamicPathCalculation(NamespacePackageTest): + paths = ['project1', 'project2'] + + def test_project3_fails(self): + import parent.child.one + self.assertEqual(len(parent.__path__), 2) + self.assertEqual(len(parent.child.__path__), 2) + import parent.child.two + self.assertEqual(len(parent.__path__), 2) + self.assertEqual(len(parent.child.__path__), 2) + + self.assertEqual(parent.child.one.attr, 'parent child one') + self.assertEqual(parent.child.two.attr, 'parent child two') + + with self.assertRaises(ImportError): + import parent.child.three + + self.assertEqual(len(parent.__path__), 2) + self.assertEqual(len(parent.child.__path__), 2) + + def test_project3_succeeds(self): + import parent.child.one + self.assertEqual(len(parent.__path__), 2) + self.assertEqual(len(parent.child.__path__), 2) + import parent.child.two + self.assertEqual(len(parent.__path__), 2) + self.assertEqual(len(parent.child.__path__), 2) + + self.assertEqual(parent.child.one.attr, 'parent child one') + self.assertEqual(parent.child.two.attr, 'parent child two') + + with self.assertRaises(ImportError): + import parent.child.three + + # now add project3 + sys.path.append(os.path.join(self.root, 'project3')) + import parent.child.three + + # the paths dynamically get longer, to include the new directories + self.assertEqual(len(parent.__path__), 3) + self.assertEqual(len(parent.child.__path__), 3) + + self.assertEqual(parent.child.three.attr, 'parent child three') + + class ZipWithMissingDirectory(NamespacePackageTest): paths = ['missing_directory.zip'] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 17:31:40 2012 From: python-checkins at python.org (nick.coghlan) Date: Fri, 25 May 2012 17:31:40 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Placeholder_in_What=27s_New?= =?utf8?q?_for_the_major_packaging_infrastructure_changes?= Message-ID: http://hg.python.org/cpython/rev/1997345ce391 changeset: 77143:1997345ce391 user: Nick Coghlan date: Sat May 26 01:31:25 2012 +1000 summary: Placeholder in What's New for the major packaging infrastructure changes files: Doc/whatsnew/3.3.rst | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -48,6 +48,26 @@ This article explains the new features in Python 3.3, compared to 3.2. +.. note:: Alpha users should be aware that this document is currently in + draft form. It will be updated substantially as Python 3.3 moves towards + release, so it's worth checking back even after reading earlier versions. + + +New packaging infrastructure +============================ + +The standard library's packaging infrastructure has been updated to adopt +some of the features developed by the wider community. + +* the :mod:`packaging` package and ``pysetup`` script (inspired by + ``setuptools``, ``distribute``, ``distutil2`` and ``pip``) +* the :mod:`venv` module and ``pyvenv`` script (inspired by ``virtualenv``) + (Note: at time of writing, :pep:`405` is accepted, but not yet implemented) +* native support for package directories that don't require ``__init__.py`` + marker files and can automatically span multiple path segments + (inspired by various third party approaches to namespace packages, as + described in :pep:`420`) + .. pep-3118-update: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 18:50:18 2012 From: python-checkins at python.org (georg.brandl) Date: Fri, 25 May 2012 18:50:18 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_420_is_implemented=2E?= Message-ID: http://hg.python.org/peps/rev/44e2e3f54912 changeset: 4437:44e2e3f54912 user: Georg Brandl date: Fri May 25 18:50:24 2012 +0200 summary: PEP 420 is implemented. files: pep-0398.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -68,6 +68,7 @@ * PEP 415: Implement context suppression with exception attributes * PEP 417: Including mock in the Standard Library * PEP 418: Add monotonic time, performance counter, and process time functions +* PEP 420: Implicit Namespace Packages * PEP 3118: Revising the buffer protocol (protocol semantics finalised) * PEP 3144: IP Address manipulation library * PEP 3151: Reworking the OS and IO exception hierarchy @@ -86,7 +87,6 @@ * PEP 362: Function Signature Object * PEP 397: Python launcher for Windows * PEP 405: Python Virtual Environments -* PEP 420: Implicit Namespace Packages * PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3154: Pickle protocol version 4 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 18:52:19 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 25 May 2012 18:52:19 +0200 (CEST) Subject: [Python-checkins] r88965 - tracker/instances/python-dev/extensions/openid_login.py Message-ID: <3VzYR71JcHzMNZ@mail.python.org> Author: martin.v.loewis Date: Fri May 25 18:52:18 2012 New Revision: 88965 Log: Drop import of M2Crypto; this shouldn't be used anymore. Modified: tracker/instances/python-dev/extensions/openid_login.py Modified: tracker/instances/python-dev/extensions/openid_login.py ============================================================================== --- tracker/instances/python-dev/extensions/openid_login.py (original) +++ tracker/instances/python-dev/extensions/openid_login.py Fri May 25 18:52:18 2012 @@ -2,7 +2,6 @@ from roundup.cgi.actions import Action, LoginAction, RegisterAction from roundup.cgi.exceptions import * from roundup import date, password -from M2Crypto.SSL.Checker import NoCertificate providers = {} for p in ( From python-checkins at python.org Fri May 25 18:53:45 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 25 May 2012 18:53:45 +0200 (CEST) Subject: [Python-checkins] r88966 - tracker/instances/python-dev/rietveld Message-ID: <3VzYSn5f7hzMNZ@mail.python.org> Author: martin.v.loewis Date: Fri May 25 18:53:45 2012 New Revision: 88966 Log: Move rietveld to Mercurial. Removed: tracker/instances/python-dev/rietveld/ From python-checkins at python.org Fri May 25 18:59:38 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 25 May 2012 18:59:38 +0200 (CEST) Subject: [Python-checkins] r88967 - tracker/instances/python-dev Message-ID: <3VzYbZ3svHzNT5@mail.python.org> Author: martin.v.loewis Date: Fri May 25 18:59:38 2012 New Revision: 88967 Log: Switch gae2django from svn to hg. Modified: tracker/instances/python-dev/ (props changed) From python-checkins at python.org Fri May 25 19:26:49 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 19:26:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_capitialize_enum_members?= Message-ID: http://hg.python.org/cpython/rev/4a6df7a2dd9d changeset: 77144:4a6df7a2dd9d user: Benjamin Peterson date: Fri May 25 10:22:29 2012 -0700 summary: capitialize enum members files: Modules/zipimport.c | 34 ++++++++++++++++---------------- 1 files changed, 17 insertions(+), 17 deletions(-) diff --git a/Modules/zipimport.c b/Modules/zipimport.c --- a/Modules/zipimport.c +++ b/Modules/zipimport.c @@ -320,10 +320,10 @@ } typedef enum { - fl_error, - fl_not_found, - fl_module_found, - fl_ns_found + FL_ERROR, + FL_NOT_FOUND, + FL_MODULE_FOUND, + FL_NS_FOUND } find_loader_result; /* The guts of "find_loader" and "find_module". Return values: @@ -341,7 +341,7 @@ mi = get_module_info(self, fullname); if (mi == MI_ERROR) - return fl_error; + return FL_ERROR; if (mi == MI_NOT_FOUND) { /* Not a module or regular package. See if this is a directory, and therefore possibly a portion of a namespace package. */ @@ -356,13 +356,13 @@ self->archive, SEP, self->prefix, fullname); if (*namespace_portion == NULL) - return fl_error; - return fl_ns_found; + return FL_ERROR; + return FL_NS_FOUND; } - return fl_not_found; + return FL_NOT_FOUND; } /* This is a module or package. */ - return fl_module_found; + return FL_MODULE_FOUND; } @@ -381,16 +381,16 @@ return NULL; switch (find_loader(self, fullname, &namespace_portion)) { - case fl_error: + case FL_ERROR: return NULL; - case fl_ns_found: + case FL_NS_FOUND: /* A namespace portion is not allowed via find_module, so return None. */ Py_DECREF(namespace_portion); /* FALL THROUGH */ - case fl_not_found: + case FL_NOT_FOUND: result = Py_None; break; - case fl_module_found: + case FL_MODULE_FOUND: result = (PyObject *)self; break; } @@ -417,15 +417,15 @@ return NULL; switch (find_loader(self, fullname, &namespace_portion)) { - case fl_error: + case FL_ERROR: return NULL; - case fl_not_found: /* Not found, return (None, []) */ + case FL_NOT_FOUND: /* Not found, return (None, []) */ result = Py_BuildValue("O[]", Py_None); break; - case fl_module_found: /* Return (self, []) */ + case FL_MODULE_FOUND: /* Return (self, []) */ result = Py_BuildValue("O[]", self); break; - case fl_ns_found: /* Return (None, [namespace_portion]) */ + case FL_NS_FOUND: /* Return (None, [namespace_portion]) */ result = Py_BuildValue("O[O]", Py_None, namespace_portion); Py_DECREF(namespace_portion); return result; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 19:26:49 2012 From: python-checkins at python.org (benjamin.peterson) Date: Fri, 25 May 2012 19:26:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_capitialize_utime_statuses?= Message-ID: http://hg.python.org/cpython/rev/789469caffe5 changeset: 77145:789469caffe5 user: Benjamin Peterson date: Fri May 25 10:26:47 2012 -0700 summary: capitialize utime statuses files: Modules/posixmodule.c | 40 +++++++++++++++--------------- 1 files changed, 20 insertions(+), 20 deletions(-) diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -3661,11 +3661,11 @@ */ typedef enum { - utime_success = 0, - utime_parse_failure = 1, - utime_times_and_ns_collision = 2, - utime_times_conversion_failure = 3, - utime_ns_conversion_failure = 4, + UTIME_SUCCESS = 0, + UTIME_PARSE_FAILURE = 1, + UTIME_TIMES_AND_NS_COLLISION = 2, + UTIME_TIMES_CONVERSION_FAILURE = 3, + UTIME_NS_CONVERSION_FAILURE = 4, } utime_status; static utime_status @@ -3697,14 +3697,14 @@ format, kwlist, ua->path, ×, &ns); if (!parse_result) - return utime_parse_failure; + return UTIME_PARSE_FAILURE; if (times && ns) { PyErr_Format(PyExc_RuntimeError, "%s: you may specify either 'times'" " or 'ns' but not both", ua->function_name); - return_value = utime_times_and_ns_collision; + return_value = UTIME_TIMES_AND_NS_COLLISION; goto fail; } @@ -3714,7 +3714,7 @@ "%s: 'times' must be either" " a tuple of two ints or None", ua->function_name); - return_value = utime_times_conversion_failure; + return_value = UTIME_TIMES_CONVERSION_FAILURE; goto fail; } ua->now = 0; @@ -3722,10 +3722,10 @@ &ua->atime_s, &ua->atime_ns) == -1 || _PyTime_ObjectToTimespec(PyTuple_GET_ITEM(times, 1), &ua->mtime_s, &ua->mtime_ns) == -1) { - return_value = utime_times_conversion_failure; + return_value = UTIME_TIMES_CONVERSION_FAILURE; goto fail; } - return utime_success; + return UTIME_SUCCESS; } if (ns) { @@ -3733,7 +3733,7 @@ PyErr_Format(PyExc_TypeError, "%s: 'ns' must be a tuple of two ints", ua->function_name); - return_value = utime_ns_conversion_failure; + return_value = UTIME_NS_CONVERSION_FAILURE; goto fail; } ua->now = 0; @@ -3741,15 +3741,15 @@ &ua->atime_s, &ua->atime_ns) || !split_py_long_to_s_and_ns(PyTuple_GET_ITEM(ns, 1), &ua->mtime_s, &ua->mtime_ns)) { - return_value = utime_ns_conversion_failure; + return_value = UTIME_NS_CONVERSION_FAILURE; goto fail; } - return utime_success; + return UTIME_SUCCESS; } /* either times=None, or neither times nor ns was specified. use "now". */ ua->now = 1; - return utime_success; + return UTIME_SUCCESS; fail: if (ua->converter) @@ -3786,7 +3786,7 @@ switch (utime_read_time_arguments(&ua)) { default: return NULL; - case utime_success: { + case UTIME_SUCCESS: { wchar_t *wpath = PyUnicode_AsUnicode(upath); if (wpath == NULL) return NULL; @@ -3799,7 +3799,7 @@ return win32_error_object("utime", upath); break; } - case utime_parse_failure: { + case UTIME_PARSE_FAILURE: { const char *apath; /* Drop the argument parsing error as narrow strings are also valid. */ @@ -3807,7 +3807,7 @@ ua.path_format = 'y'; ua.path = (PyObject **)&apath; - if (utime_read_time_arguments(&ua) != utime_success) + if (utime_read_time_arguments(&ua) != UTIME_SUCCESS) return NULL; if (win32_warn_bytes_api()) return NULL; @@ -3862,7 +3862,7 @@ ua.path = &opath; ua.converter = PyUnicode_FSConverter; - if (utime_read_time_arguments(&ua) != utime_success) + if (utime_read_time_arguments(&ua) != UTIME_SUCCESS) return NULL; path = PyBytes_AsString(opath); if (ua.now) { @@ -3915,7 +3915,7 @@ ua.path = (PyObject **)&fd; ua.first_argument_name = "fd"; - if (utime_read_time_arguments(&ua) != utime_success) + if (utime_read_time_arguments(&ua) != UTIME_SUCCESS) return NULL; if (ua.now) { @@ -3960,7 +3960,7 @@ ua.path = &opath; ua.converter = PyUnicode_FSConverter; - if (utime_read_time_arguments(&ua) != utime_success) + if (utime_read_time_arguments(&ua) != UTIME_SUCCESS) return NULL; path = PyBytes_AsString(opath); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Fri May 25 19:51:28 2012 From: python-checkins at python.org (martin.v.loewis) Date: Fri, 25 May 2012 19:51:28 +0200 (CEST) Subject: [Python-checkins] r88968 - tracker/instances/python-dev/extensions/rietveldlink.py Message-ID: <3VzZlN0kCqzNYY@mail.python.org> Author: martin.v.loewis Date: Fri May 25 19:51:27 2012 New Revision: 88968 Log: Deep-link to patch set. Modified: tracker/instances/python-dev/extensions/rietveldlink.py Modified: tracker/instances/python-dev/extensions/rietveldlink.py ============================================================================== --- tracker/instances/python-dev/extensions/rietveldlink.py (original) +++ tracker/instances/python-dev/extensions/rietveldlink.py Fri May 25 19:51:27 2012 @@ -1,7 +1,7 @@ def rietveldlink(request, issueid, fileid): patchset = request.client.db.file.get(fileid, 'patchset') if patchset and patchset != 'n/a': - return '/review/%s/show' % issueid + return '/review/%s/#ps%s' % (issueid, patchset) return "" def init(instance): From python-checkins at python.org Fri May 25 20:13:26 2012 From: python-checkins at python.org (barry.warsaw) Date: Fri, 25 May 2012 20:13:26 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Spell-checked=2E?= Message-ID: http://hg.python.org/peps/rev/d507076941b1 changeset: 4438:d507076941b1 user: Barry Warsaw date: Fri May 25 14:13:23 2012 -0400 summary: Spell-checked. files: pep-0421.txt | 15 +++++++-------- 1 files changed, 7 insertions(+), 8 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -52,8 +52,7 @@ We will add a new attribute to the ``sys`` module, called ``sys.implementation``, as an object with attribute-access (as opposed -to a mapping). It will contain contain implementation-specific -information. +to a mapping). It will contain implementation-specific information. The attributes of this object will remain fixed during interpreter execution and through the course of an implementation version. This @@ -72,7 +71,7 @@ While this PEP places no other constraints on ``sys.implementation``, it also recommends that no one rely on capabilities outside those described here. The only exception to that recommendation is for -attributes starting with an underscore. Implementors may use those +attributes starting with an underscore. Implementers may use those as appropriate to store per-implementation data. @@ -84,7 +83,7 @@ define them: **name** - A lower-case identifer representing the implementation. Examples + A lower-case identifier representing the implementation. Examples include 'pypy', 'jython', 'ironpython', and 'cpython'. **version** @@ -117,7 +116,7 @@ PEP process. Such a PEP need not be long, just long enough. It will need to sufficiently spell out the rationale for the new attribute, its use cases, and the impact it will have on the various Python -implemenations. +implementations. Version Format @@ -295,7 +294,7 @@ Earlier versions of this PEP made the mistake of calling ``sys.version_info`` (and friends) the version of the Python language, in contrast to the implementation. However, this is not the case. -Instead, it is the version of the CPython implementation. Incidently, +Instead, it is the version of the CPython implementation. Incidentally, the first two components of ``sys.version_info`` (major and minor) also reflect the version of the language definition. @@ -311,7 +310,7 @@ language version. -Feedback From Other Python Implementors +Feedback From Other Python Implementers ======================================= IronPython @@ -386,7 +385,7 @@ However, as already noted, many other efforts predate ``sys.implementation``. Neither is it necessarily a major part of the effort. Rather, consider it as part of the infrastructure of the -effort to make Python friendler to alternate implementations. +effort to make Python friendlier to alternate implementations. Alternatives -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 20:15:27 2012 From: python-checkins at python.org (barry.warsaw) Date: Fri, 25 May 2012 20:15:27 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Since_there_were_no_objections?= =?utf8?q?_to_my_2012-05-21_email=2C_claim_czarship=2E?= Message-ID: http://hg.python.org/peps/rev/6041ad0e6dfe changeset: 4439:6041ad0e6dfe user: Barry Warsaw date: Fri May 25 14:15:20 2012 -0400 summary: Since there were no objections to my 2012-05-21 email, claim czarship. files: pep-0421.txt | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -3,6 +3,7 @@ Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow +BDFL-Delegate: Barry Warsaw Status: Draft Type: Standards Track Content-Type: text/x-rst -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 20:47:07 2012 From: python-checkins at python.org (barry.warsaw) Date: Fri, 25 May 2012 20:47:07 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_The_PEP_is_accepted=2E?= Message-ID: http://hg.python.org/peps/rev/c9575ab4ddc7 changeset: 4440:c9575ab4ddc7 user: Barry Warsaw date: Fri May 25 14:47:04 2012 -0400 summary: The PEP is accepted. files: pep-0421.txt | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/pep-0421.txt b/pep-0421.txt --- a/pep-0421.txt +++ b/pep-0421.txt @@ -4,11 +4,12 @@ Last-Modified: $Date$ Author: Eric Snow BDFL-Delegate: Barry Warsaw -Status: Draft +Status: Accepted Type: Standards Track Content-Type: text/x-rst Created: 26-April-2012 Post-History: 26-April-2012 +Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119683.html Abstract -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Fri May 25 21:02:46 2012 From: python-checkins at python.org (r.david.murray) Date: Fri, 25 May 2012 21:02:46 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314731=3A_refactor_email_?= =?utf8?q?policy_framework=2E?= Message-ID: http://hg.python.org/cpython/rev/9388c671d52d changeset: 77146:9388c671d52d user: R David Murray date: Fri May 25 15:01:48 2012 -0400 summary: #14731: refactor email policy framework. This patch primarily does two things: (1) it adds some internal-interface methods to Policy that allow for Policy to control the parsing and folding of headers in such a way that we can construct a backward compatibility policy that is 100% compatible with the 3.2 API, while allowing a new policy to implement the email6 API. (2) it adds that backward compatibility policy and refactors the test suite so that the only differences between the 3.2 test_email.py file and the 3.3 test_email.py file is some small changes in test framework and the addition of tests for bugs fixed that apply to the 3.2 API. There are some additional teaks, such as moving just the code needed for the compatibility policy into _policybase, so that the library code can import only _policybase. That way the new code that will be added for email6 will only get imported when a non-compatibility policy is imported. files: Doc/library/email.generator.rst | 38 +- Doc/library/email.policy.rst | 294 ++++++++--- Lib/email/_policybase.py | 338 ++++++++++++++ Lib/email/architecture.rst | 216 ++++++++ Lib/email/feedparser.py | 24 +- Lib/email/generator.py | 87 +-- Lib/email/message.py | 64 +- Lib/email/parser.py | 4 +- Lib/email/policy.py | 172 +------ Lib/email/utils.py | 4 + Lib/test/test_email/__init__.py | 14 +- Lib/test/test_email/test_email.py | 182 +----- Lib/test/test_email/test_generator.py | 85 +++- Lib/test/test_email/test_parser.py | 276 +++++++++++ Lib/test/test_email/test_policy.py | 114 ++++- 15 files changed, 1392 insertions(+), 520 deletions(-) diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -32,8 +32,7 @@ :mod:`email.generator` module: -.. class:: Generator(outfp, mangle_from_=True, maxheaderlen=78, *, \ - policy=policy.default) +.. class:: Generator(outfp, mangle_from_=True, maxheaderlen=78, *, policy=None) The constructor for the :class:`Generator` class takes a :term:`file-like object` called *outfp* for an argument. *outfp* must support the :meth:`write` method @@ -55,8 +54,9 @@ The default is 78, as recommended (but not required) by :rfc:`2822`. The *policy* keyword specifies a :mod:`~email.policy` object that controls a - number of aspects of the generator's operation. The default policy - maintains backward compatibility. + number of aspects of the generator's operation. If no *policy* is specified, + then the *policy* attached to the message object passed to :attr:``flatten`` + is used. .. versionchanged:: 3.3 Added the *policy* keyword. @@ -80,19 +80,19 @@ Optional *linesep* specifies the line separator character used to terminate lines in the output. If specified it overrides the value - specified by the ``Generator``\'s ``policy``. + specified by the *msg*\'s or ``Generator``\'s ``policy``. - Because strings cannot represent non-ASCII bytes, ``Generator`` ignores - the value of the :attr:`~email.policy.Policy.must_be_7bit` - :mod:`~email.policy` setting and operates as if it were set ``True``. - This means that messages parsed with a Bytes parser that have a - :mailheader:`Content-Transfer-Encoding` of 8bit will be converted to a - use a 7bit Content-Transfer-Encoding. Non-ASCII bytes in the headers - will be :rfc:`2047` encoded with a charset of `unknown-8bit`. + Because strings cannot represent non-ASCII bytes, if the policy that + applies when ``flatten`` is run has :attr:`~email.policy.Policy.cte_type` + set to ``8bit``, ``Generator`` will operate as if it were set to + ``7bit``. This means that messages parsed with a Bytes parser that have + a :mailheader:`Content-Transfer-Encoding` of ``8bit`` will be converted + to a use a ``7bit`` Content-Transfer-Encoding. Non-ASCII bytes in the + headers will be :rfc:`2047` encoded with a charset of ``unknown-8bit``. .. versionchanged:: 3.2 - Added support for re-encoding 8bit message bodies, and the *linesep* - argument. + Added support for re-encoding ``8bit`` message bodies, and the + *linesep* argument. .. method:: clone(fp) @@ -149,13 +149,13 @@ at *msg* to the output file specified when the :class:`BytesGenerator` instance was created. Subparts are visited depth-first and the resulting text will be properly MIME encoded. If the :mod:`~email.policy` option - :attr:`~email.policy.Policy.must_be_7bit` is ``False`` (the default), + :attr:`~email.policy.Policy.cte_type` is ``8bit`` (the default), then any bytes with the high bit set in the original parsed message that have not been modified will be copied faithfully to the output. If - ``must_be_7bit`` is true, the bytes will be converted as needed using an - ASCII content-transfer-encoding. In particular, RFC-invalid non-ASCII - bytes in headers will be encoded using the MIME ``unknown-8bit`` - character set, thus rendering them RFC-compliant. + ``cte_type`` is ``7bit``, the bytes will be converted as needed + using an ASCII-compatible Content-Transfer-Encoding. In particular, + RFC-invalid non-ASCII bytes in headers will be encoded using the MIME + ``unknown-8bit`` character set, thus rendering them RFC-compliant. .. XXX: There should be a complementary option that just does the RFC compliance transformation but leaves CTE 8bit parts alone. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -23,81 +23,100 @@ control the behavior of various components of the email package during use. :class:`Policy` instances can be passed to various classes and methods in the email package to alter the default behavior. The settable values and their -defaults are described below. The :mod:`policy` module also provides some -pre-created :class:`Policy` instances. In addition to a :const:`default` -instance, there are instances tailored for certain applications. For example -there is an :const:`SMTP` :class:`Policy` with defaults appropriate for -generating output to be sent to an SMTP server. These are listed `below -`. +defaults are described below. -In general an application will only need to deal with setting the policy at the -input and output boundaries. Once parsed, a message is represented by a -:class:`~email.message.Message` object, which is designed to be independent of -the format that the message has "on the wire" when it is received, transmitted, -or displayed. Thus, a :class:`Policy` can be specified when parsing a message -to create a :class:`~email.message.Message`, and again when turning the -:class:`~email.message.Message` into some other representation. While often a -program will use the same :class:`Policy` for both input and output, the two -can be different. +There is a default policy used by all classes in the email package. This +policy is named :class:`Compat32`, with a corresponding pre-defined instance +named :const:`compat32`. It provides for complete backward compatibility (in +some cases, including bug compatibility) with the pre-Python3.3 version of the +email package. + +The first part of this documentation covers the features of :class:`Policy`, an +:term:`abstract base class` that defines the features that are common to all +policy objects, including :const:`compat32`. This includes certain hook +methods that are called internally by the email package, which a custom policy +could override to obtain different behavior. + +When a :class:`~email.message.Message` object is created, it acquires a policy. +By default this will be :const:`compat32`, but a different policy can be +specified. If the ``Message`` is created by a :mod:`~email.parser`, a policy +passed to the parser will be the policy used by the ``Message`` it creates. If +the ``Message`` is created by the program, then the policy can be specified +when it is created. When a ``Message`` is passed to a :mod:`~email.generator`, +the generator uses the policy from the ``Message`` by default, but you can also +pass a specific policy to the generator that will override the one stored on +the ``Message`` object. + +:class:`Policy` instances are immutable, but they can be cloned, accepting the +same keyword arguments as the class constructor and returning a new +:class:`Policy` instance that is a copy of the original but with the specified +attributes values changed. As an example, the following code could be used to read an email message from a file on disk and pass it to the system ``sendmail`` program on a Unix system:: >>> from email import msg_from_binary_file >>> from email.generator import BytesGenerator - >>> import email.policy >>> from subprocess import Popen, PIPE >>> with open('mymsg.txt', 'b') as f: - ... msg = msg_from_binary_file(f, policy=email.policy.mbox) + ... msg = msg_from_binary_file(f) >>> p = Popen(['sendmail', msg['To'][0].address], stdin=PIPE) - >>> g = BytesGenerator(p.stdin, policy=email.policy.SMTP) + >>> g = BytesGenerator(p.stdin, policy=msg.policy.clone(linesep='\r\n')) >>> g.flatten(msg) >>> p.stdin.close() >>> rc = p.wait() -.. XXX email.policy.mbox/MBOX does not exist yet +Here we are telling :class:`~email.generator.BytesGenerator` to use the RFC +correct line separator characters when creating the binary string to feed into +``sendmail's`` ``stdin``, where the default policy would use ``\n`` line +separators. Some email package methods accept a *policy* keyword argument, allowing the policy to be overridden for that method. For example, the following code uses -the :meth:`~email.message.Message.as_string` method of the *msg* object from the -previous example and re-write it to a file using the native line separators for -the platform on which it is running:: +the :meth:`~email.message.Message.as_string` method of the *msg* object from +the previous example and writes the message to a file using the native line +separators for the platform on which it is running:: >>> import os - >>> mypolicy = email.policy.Policy(linesep=os.linesep) >>> with open('converted.txt', 'wb') as f: - ... f.write(msg.as_string(policy=mypolicy)) - -Policy instances are immutable, but they can be cloned, accepting the same -keyword arguments as the class constructor and returning a new :class:`Policy` -instance that is a copy of the original but with the specified attributes -values changed. For example, the following creates an SMTP policy that will -raise any defects detected as errors:: - - >>> strict_SMTP = email.policy.SMTP.clone(raise_on_defect=True) + ... f.write(msg.as_string(policy=msg.policy.clone(linesep=os.linesep)) Policy objects can also be combined using the addition operator, producing a policy object whose settings are a combination of the non-default values of the summed objects:: - >>> strict_SMTP = email.policy.SMTP + email.policy.strict + >>> compat_SMTP = email.policy.clone(linesep='\r\n') + >>> compat_strict = email.policy.clone(raise_on_defect=True) + >>> compat_strict_SMTP = compat_SMTP + compat_strict This operation is not commutative; that is, the order in which the objects are added matters. To illustrate:: - >>> Policy = email.policy.Policy - >>> apolicy = Policy(max_line_length=100) + Policy(max_line_length=80) + >>> policy100 = compat32.clone(max_line_length=100) + >>> policy80 = compat32.clone(max_line_length=80) + >>> apolicy = policy100 + Policy80 >>> apolicy.max_line_length 80 - >>> apolicy = Policy(max_line_length=80) + Policy(max_line_length=100) + >>> apolicy = policy80 + policy100 >>> apolicy.max_line_length 100 .. class:: Policy(**kw) - The valid constructor keyword arguments are any of the attributes listed - below. + This is the :term:`abstract base class` for all policy classes. It provides + default implementations for a couple of trivial methods, as well as the + implementation of the immutability property, the :meth:`clone` method, and + the constructor semantics. + + The constructor of a policy class can be passed various keyword arguments. + The arguments that may be specified are any non-method properties on this + class, plus any additional non-method properties on the concrete class. A + value specified in the constructor will override the default value for the + corresponding attribute. + + This class defines the following properties, and thus values for the + following may be passed in the constructor of any policy class: .. attribute:: max_line_length @@ -110,18 +129,28 @@ The string to be used to terminate lines in serialized output. The default is ``\n`` because that's the internal end-of-line discipline used - by Python, though ``\r\n`` is required by the RFCs. See `Policy - Instances`_ for policies that use an RFC conformant linesep. Setting it - to :attr:`os.linesep` may also be useful. + by Python, though ``\r\n`` is required by the RFCs. - .. attribute:: must_be_7bit + .. attribute:: cte_type - If ``True``, data output by a bytes generator is limited to ASCII - characters. If :const:`False` (the default), then bytes with the high - bit set are preserved and/or allowed in certain contexts (for example, - where possible a content transfer encoding of ``8bit`` will be used). - String generators act as if ``must_be_7bit`` is ``True`` regardless of - the policy in effect, since a string cannot represent non-ASCII bytes. + Controls the type of Content Transfer Encodings that may be or are + required to be used. The possible values are: + + ======== =============================================================== + ``7bit`` all data must be "7 bit clean" (ASCII-only). This means that + where necessary data will be encoded using either + quoted-printable or base64 encoding. + + ``8bit`` data is not constrained to be 7 bit clean. Data in headers is + still required to be ASCII-only and so will be encoded (see + 'binary_fold' below for an exception), but body parts may use + the ``8bit`` CTE. + ======== =============================================================== + + A ``cte_type`` value of ``8bit`` only works with ``BytesGenerator``, not + ``Generator``, because strings cannot contain binary data. If a + ``Generator`` is operating under a policy that specifies + ``cte_type=8bit``, it will act as if ``cte_type`` is ``7bit``. .. attribute:: raise_on_defect @@ -129,56 +158,151 @@ :const:`False` (the default), defects will be passed to the :meth:`register_defect` method. - :mod:`Policy` object also have the following methods: + The following :class:`Policy` method is intended to be called by code using + the email library to create policy instances with custom settings: - .. method:: handle_defect(obj, defect) - - *obj* is the object on which to register the defect. *defect* should be - an instance of a subclass of :class:`~email.errors.Defect`. - If :attr:`raise_on_defect` - is ``True`` the defect is raised as an exception. Otherwise *obj* and - *defect* are passed to :meth:`register_defect`. This method is intended - to be called by parsers when they encounter defects, and will not be - called by code that uses the email library unless that code is - implementing an alternate parser. - - .. method:: register_defect(obj, defect) - - *obj* is the object on which to register the defect. *defect* should be - a subclass of :class:`~email.errors.Defect`. This method is part of the - public API so that custom ``Policy`` subclasses can implement alternate - handling of defects. The default implementation calls the ``append`` - method of the ``defects`` attribute of *obj*. - - .. method:: clone(obj, *kw) + .. method:: clone(**kw) Return a new :class:`Policy` instance whose attributes have the same values as the current instance, except where those attributes are given new values by the keyword arguments. + The remaining :class:`Policy` methods are called by the email package code, + and are not intended to be called by an application using the email package. + A custom policy must implement all of these methods. -Policy Instances -^^^^^^^^^^^^^^^^ + .. method:: handle_defect(obj, defect) -The following instances of :class:`Policy` provide defaults suitable for -specific common application domains. + Handle a *defect* found on *obj*. When the email package calls this + method, *defect* will always be a subclass of + :class:`~email.errors.Defect`. -.. data:: default + The default implementation checks the :attr:`raise_on_defect` flag. If + it is ``True``, *defect* is raised as an exception. If it is ``False`` + (the default), *obj* and *defect* are passed to :meth:`register_defect`. - An instance of :class:`Policy` with all defaults unchanged. + .. method:: register_defect(obj, defect) -.. data:: SMTP + Register a *defect* on *obj*. In the email package, *defect* will always + be a subclass of :class:`~email.errors.Defect`. - Output serialized from a message will conform to the email and SMTP - RFCs. The only changed attribute is :attr:`linesep`, which is set to - ``\r\n``. + The default implementation calls the ``append`` method of the ``defects`` + attribute of *obj*. When the email package calls :attr:`handle_defect`, + *obj* will normally have a ``defects`` attribute that has an ``append`` + method. Custom object types used with the email package (for example, + custom ``Message`` objects) should also provide such an attribute, + otherwise defects in parsed messages will raise unexpected errors. -.. data:: HTTP + .. method:: header_source_parse(sourcelines) - Suitable for use when serializing headers for use in HTTP traffic. - :attr:`linesep` is set to ``\r\n``, and :attr:`max_line_length` is set to - :const:`None` (unlimited). + The email package calls this method with a list of strings, each string + ending with the line separation characters found in the source being + parsed. The first line includes the field header name and separator. + All whitespace in the source is preserved. The method should return the + ``(name, value)`` tuple that is to be stored in the ``Message`` to + represent the parsed header. -.. data:: strict + If an implementation wishes to retain compatibility with the existing + email package policies, *name* should be the case preserved name (all + characters up to the '``:``' separator), while *value* should be the + unfolded value (all line separator characters removed, but whitespace + kept intact), stripped of leading whitespace. - :attr:`raise_on_defect` is set to :const:`True`. + *sourcelines* may contain surrogateescaped binary data. + + There is no default implementation + + .. method:: header_store_parse(name, value) + + The email package calls this method with the name and value provided by + the application program when the application program is modifying a + ``Message`` programmatically (as opposed to a ``Message`` created by a + parser). The method should return the ``(name, value)`` tuple that is to + be stored in the ``Message`` to represent the header. + + If an implementation wishes to retain compatibility with the existing + email package policies, the *name* and *value* should be strings or + string subclasses that do not change the content of the passed in + arguments. + + There is no default implementation + + .. method:: header_fetch_parse(name, value) + + The email package calls this method with the *name* and *value* currently + stored in the ``Message`` when that header is requested by the + application program, and whatever the method returns is what is passed + back to the application as the value of the header being retrieved. + Note that there may be more than one header with the same name stored in + the ``Message``; the method is passed the specific name and value of the + header destined to be returned to the application. + + *value* may contain surrogateescaped binary data. There should be no + surrogateescaped binary data in the value returned by the method. + + There is no default implementation + + .. method:: fold(name, value) + + The email package calls this method with the *name* and *value* currently + stored in the ``Message`` for a given header. The method should return a + string that represents that header "folded" correctly (according to the + policy settings) by composing the *name* with the *value* and inserting + :attr:`linesep` characters at the appropriate places. See :rfc:`5322` + for a discussion of the rules for folding email headers. + + *value* may contain surrogateescaped binary data. There should be no + surrogateescaped binary data in the string returned by the method. + + .. method:: fold_binary(name, value) + + The same as :meth:`fold`, except that the returned value should be a + bytes object rather than a string. + + *value* may contain surrogateescaped binary data. These could be + converted back into binary data in the returned bytes object. + + +.. class:: Compat32(**kw) + + This concrete :class:`Policy` is the backward compatibility policy. It + replicates the behavior of the email package in Python 3.2. The + :mod:`policy` module also defines an instance of this class, + :const:`compat32`, that is used as the default policy. Thus the default + behavior of the email package is to maintain compatibility with Python 3.2. + + The class provides the following concrete implementations of the + abstract methods of :class:`Policy`: + + .. method:: header_source_parse(sourcelines) + + The name is parsed as everything up to the '``:``' and returned + unmodified. The value is determined by stripping leading whitespace off + the remainder of the first line, joining all subsequent lines together, + and stripping any trailing carriage return or linefeed characters. + + .. method:: header_store_parse(name, value) + + The name and value are returned unmodified. + + .. method:: header_fetch_parse(name, value) + + If the value contains binary data, it is converted into a + :class:`~email.header.Header` object using the ``unknown-8bit`` charset. + Otherwise it is returned unmodified. + + .. method:: fold(name, value) + + Headers are folded using the :class:`~email.header.Header` folding + algorithm, which preserves existing line breaks in the value, and wraps + each resulting line to the ``max_line_length``. Non-ASCII binary data are + CTE encoded using the ``unknown-8bit`` charset. + + .. method:: fold_binary(name, value) + + Headers are folded using the :class:`~email.header.Header` folding + algorithm, which preserves existing line breaks in the value, and wraps + each resulting line to the ``max_line_length``. If ``cte_type`` is + ``7bit``, non-ascii binary data is CTE encoded using the ``unknown-8bit`` + charset. Otherwise the original source header is used, with its existing + line breaks and and any (RFC invalid) binary data it may contain. diff --git a/Lib/email/_policybase.py b/Lib/email/_policybase.py new file mode 100644 --- /dev/null +++ b/Lib/email/_policybase.py @@ -0,0 +1,338 @@ +"""Policy framework for the email package. + +Allows fine grained feature control of how the package parses and emits data. +""" + +import abc +from email import header +from email import charset as _charset +from email.utils import _has_surrogates + +__all__ = [ + 'Policy', + 'Compat32', + 'compat32', + ] + + +class _PolicyBase: + + """Policy Object basic framework. + + This class is useless unless subclassed. A subclass should define + class attributes with defaults for any values that are to be + managed by the Policy object. The constructor will then allow + non-default values to be set for these attributes at instance + creation time. The instance will be callable, taking these same + attributes keyword arguments, and returning a new instance + identical to the called instance except for those values changed + by the keyword arguments. Instances may be added, yielding new + instances with any non-default values from the right hand + operand overriding those in the left hand operand. That is, + + A + B == A() + + The repr of an instance can be used to reconstruct the object + if and only if the repr of the values can be used to reconstruct + those values. + + """ + + def __init__(self, **kw): + """Create new Policy, possibly overriding some defaults. + + See class docstring for a list of overridable attributes. + + """ + for name, value in kw.items(): + if hasattr(self, name): + super(_PolicyBase,self).__setattr__(name, value) + else: + raise TypeError( + "{!r} is an invalid keyword argument for {}".format( + name, self.__class__.__name__)) + + def __repr__(self): + args = [ "{}={!r}".format(name, value) + for name, value in self.__dict__.items() ] + return "{}({})".format(self.__class__.__name__, ', '.join(args)) + + def clone(self, **kw): + """Return a new instance with specified attributes changed. + + The new instance has the same attribute values as the current object, + except for the changes passed in as keyword arguments. + + """ + for attr, value in self.__dict__.items(): + if attr not in kw: + kw[attr] = value + return self.__class__(**kw) + + def __setattr__(self, name, value): + if hasattr(self, name): + msg = "{!r} object attribute {!r} is read-only" + else: + msg = "{!r} object has no attribute {!r}" + raise AttributeError(msg.format(self.__class__.__name__, name)) + + def __add__(self, other): + """Non-default values from right operand override those from left. + + The object returned is a new instance of the subclass. + + """ + return self.clone(**other.__dict__) + + +# Conceptually this isn't a subclass of ABCMeta, but since we want Policy to +# use ABCMeta as a metaclass *and* we want it to use this one as well, we have +# to make this one a subclas of ABCMeta. +class _DocstringExtenderMetaclass(abc.ABCMeta): + + def __new__(meta, classname, bases, classdict): + if classdict.get('__doc__') and classdict['__doc__'].startswith('+'): + classdict['__doc__'] = meta._append_doc(bases[0].__doc__, + classdict['__doc__']) + for name, attr in classdict.items(): + if attr.__doc__ and attr.__doc__.startswith('+'): + for cls in (cls for base in bases for cls in base.mro()): + doc = getattr(getattr(cls, name), '__doc__') + if doc: + attr.__doc__ = meta._append_doc(doc, attr.__doc__) + break + return super().__new__(meta, classname, bases, classdict) + + @staticmethod + def _append_doc(doc, added_doc): + added_doc = added_doc.split('\n', 1)[1] + return doc + '\n' + added_doc + + +class Policy(_PolicyBase, metaclass=_DocstringExtenderMetaclass): + + r"""Controls for how messages are interpreted and formatted. + + Most of the classes and many of the methods in the email package accept + Policy objects as parameters. A Policy object contains a set of values and + functions that control how input is interpreted and how output is rendered. + For example, the parameter 'raise_on_defect' controls whether or not an RFC + violation results in an error being raised or not, while 'max_line_length' + controls the maximum length of output lines when a Message is serialized. + + Any valid attribute may be overridden when a Policy is created by passing + it as a keyword argument to the constructor. Policy objects are immutable, + but a new Policy object can be created with only certain values changed by + calling the Policy instance with keyword arguments. Policy objects can + also be added, producing a new Policy object in which the non-default + attributes set in the right hand operand overwrite those specified in the + left operand. + + Settable attributes: + + raise_on_defect -- If true, then defects should be raised as errors. + Default: False. + + linesep -- string containing the value to use as separation + between output lines. Default '\n'. + + cte_type -- Type of allowed content transfer encodings + + 7bit -- ASCII only + 8bit -- Content-Transfer-Encoding: 8bit is allowed + + Default: 8bit. Also controls the disposition of + (RFC invalid) binary data in headers; see the + documentation of the binary_fold method. + + max_line_length -- maximum length of lines, excluding 'linesep', + during serialization. None or 0 means no line + wrapping is done. Default is 78. + + """ + + raise_on_defect = False + linesep = '\n' + cte_type = '8bit' + max_line_length = 78 + + def handle_defect(self, obj, defect): + """Based on policy, either raise defect or call register_defect. + + handle_defect(obj, defect) + + defect should be a Defect subclass, but in any case must be an + Exception subclass. obj is the object on which the defect should be + registered if it is not raised. If the raise_on_defect is True, the + defect is raised as an error, otherwise the object and the defect are + passed to register_defect. + + This method is intended to be called by parsers that discover defects. + The email package parsers always call it with Defect instances. + + """ + if self.raise_on_defect: + raise defect + self.register_defect(obj, defect) + + def register_defect(self, obj, defect): + """Record 'defect' on 'obj'. + + Called by handle_defect if raise_on_defect is False. This method is + part of the Policy API so that Policy subclasses can implement custom + defect handling. The default implementation calls the append method of + the defects attribute of obj. The objects used by the email package by + default that get passed to this method will always have a defects + attribute with an append method. + + """ + obj.defects.append(defect) + + @abc.abstractmethod + def header_source_parse(self, sourcelines): + """Given a list of linesep terminated strings constituting the lines of + a single header, return the (name, value) tuple that should be stored + in the model. The input lines should retain their terminating linesep + characters. The lines passed in by the email package may contain + surrogateescaped binary data. + """ + raise NotImplementedError + + @abc.abstractmethod + def header_store_parse(self, name, value): + """Given the header name and the value provided by the application + program, return the (name, value) that should be stored in the model. + """ + raise NotImplementedError + + @abc.abstractmethod + def header_fetch_parse(self, name, value): + """Given the header name and the value from the model, return the value + to be returned to the application program that is requesting that + header. The value passed in by the email package may contain + surrogateescaped binary data if the lines were parsed by a BytesParser. + The returned value should not contain any surrogateescaped data. + + """ + raise NotImplementedError + + @abc.abstractmethod + def fold(self, name, value): + """Given the header name and the value from the model, return a string + containing linesep characters that implement the folding of the header + according to the policy controls. The value passed in by the email + package may contain surrogateescaped binary data if the lines were + parsed by a BytesParser. The returned value should not contain any + surrogateescaped data. + + """ + raise NotImplementedError + + @abc.abstractmethod + def fold_binary(self, name, value): + """Given the header name and the value from the model, return binary + data containing linesep characters that implement the folding of the + header according to the policy controls. The value passed in by the + email package may contain surrogateescaped binary data. + + """ + raise NotImplementedError + + +class Compat32(Policy): + + """+ + This particular policy is the backward compatibility Policy. It + replicates the behavior of the email package version 5.1. + """ + + def _sanitize_header(self, name, value): + # If the header value contains surrogates, return a Header using + # the unknown-8bit charset to encode the bytes as encoded words. + if not isinstance(value, str): + # Assume it is already a header object + return value + if _has_surrogates(value): + return header.Header(value, charset=_charset.UNKNOWN8BIT, + header_name=name) + else: + return value + + def header_source_parse(self, sourcelines): + """+ + The name is parsed as everything up to the ':' and returned unmodified. + The value is determined by stripping leading whitespace off the + remainder of the first line, joining all subsequent lines together, and + stripping any trailing carriage return or linefeed characters. + + """ + name, value = sourcelines[0].split(':', 1) + value = value.lstrip(' \t') + ''.join(sourcelines[1:]) + return (name, value.rstrip('\r\n')) + + def header_store_parse(self, name, value): + """+ + The name and value are returned unmodified. + """ + return (name, value) + + def header_fetch_parse(self, name, value): + """+ + If the value contains binary data, it is converted into a Header object + using the unknown-8bit charset. Otherwise it is returned unmodified. + """ + return self._sanitize_header(name, value) + + def fold(self, name, value): + """+ + Headers are folded using the Header folding algorithm, which preserves + existing line breaks in the value, and wraps each resulting line to the + max_line_length. Non-ASCII binary data are CTE encoded using the + unknown-8bit charset. + + """ + return self._fold(name, value, sanitize=True) + + def fold_binary(self, name, value): + """+ + Headers are folded using the Header folding algorithm, which preserves + existing line breaks in the value, and wraps each resulting line to the + max_line_length. If cte_type is 7bit, non-ascii binary data is CTE + encoded using the unknown-8bit charset. Otherwise the original source + header is used, with its existing line breaks and/or binary data. + + """ + folded = self._fold(name, value, sanitize=self.cte_type=='7bit') + return folded.encode('ascii', 'surrogateescape') + + def _fold(self, name, value, sanitize): + parts = [] + parts.append('%s: ' % name) + if isinstance(value, str): + if _has_surrogates(value): + if sanitize: + h = header.Header(value, + charset=_charset.UNKNOWN8BIT, + header_name=name) + else: + # If we have raw 8bit data in a byte string, we have no idea + # what the encoding is. There is no safe way to split this + # string. If it's ascii-subset, then we could do a normal + # ascii split, but if it's multibyte then we could break the + # string. There's no way to know so the least harm seems to + # be to not split the string and risk it being too long. + parts.append(value) + h = None + else: + h = header.Header(value, header_name=name) + else: + # Assume it is a Header-like object. + h = value + if h is not None: + parts.append(h.encode(linesep=self.linesep, + maxlinelen=self.max_line_length)) + parts.append(self.linesep) + return ''.join(parts) + + +compat32 = Compat32() diff --git a/Lib/email/architecture.rst b/Lib/email/architecture.rst new file mode 100644 --- /dev/null +++ b/Lib/email/architecture.rst @@ -0,0 +1,216 @@ +:mod:`email` Package Architecture +================================= + +Overview +-------- + +The email package consists of three major components: + + Model + An object structure that represents an email message, and provides an + API for creating, querying, and modifying a message. + + Parser + Takes a sequence of characters or bytes and produces a model of the + email message represented by those characters or bytes. + + Generator + Takes a model and turns it into a sequence of characters or bytes. The + sequence can either be intended for human consumption (a printable + unicode string) or bytes suitable for transmission over the wire. In + the latter case all data is properly encoded using the content transfer + encodings specified by the relevant RFCs. + +Conceptually the package is organized around the model. The model provides both +"external" APIs intended for use by application programs using the library, +and "internal" APIs intended for use by the Parser and Generator components. +This division is intentionally a bit fuzy; the API described by this documentation +is all a public, stable API. This allows for an application with special needs +to implement its own parser and/or generator. + +In addition to the three major functional components, there is a third key +component to the architecture: + + Policy + An object that specifies various behavioral settings and carries + implementations of various behavior-controlling methods. + +The Policy framework provides a simple and convenient way to control the +behavior of the library, making it possible for the library to be used in a +very flexible fashion while leveraging the common code required to parse, +represent, and generate message-like objects. For example, in addition to the +default :rfc:`5322` email message policy, we also have a policy that manages +HTTP headers in a fashion compliant with :rfc:`2616`. Individual policy +controls, such as the maximum line length produced by the generator, can also +be controlled individually to meet specialized application requirements. + + +The Model +--------- + +The message model is implemented by the :class:`~email.message.Message` class. +The model divides a message into the two fundamental parts discussed by the +RFC: the header section and the body. The `Message` object acts as a +pseudo-dictionary of named headers. Its dictionary interface provides +convenient access to individual headers by name. However, all headers are kept +internally in an ordered list, so that the information about the order of the +headers in the original message is preserved. + +The `Message` object also has a `payload` that holds the body. A `payload` can +be one of two things: data, or a list of `Message` objects. The latter is used +to represent a multipart MIME message. Lists can be nested arbitrarily deeply +in order to represent the message, with all terminal leaves having non-list +data payloads. + + +Message Lifecycle +----------------- + +The general lifecyle of a message is: + + Creation + A `Message` object can be created by a Parser, or it can be + instantiated as an empty message by an application. + + Manipulation + The application may examine one or more headers, and/or the + payload, and it may modify one or more headers and/or + the payload. This may be done on the top level `Message` + object, or on any sub-object. + + Finalization + The Model is converted into a unicode or binary stream, + or the model is discarded. + + + +Header Policy Control During Lifecycle +-------------------------------------- + +One of the major controls exerted by the Policy is the management of headers +during the `Message` lifecycle. Most applications don't need to be aware of +this. + +A header enters the model in one of two ways: via a Parser, or by being set to +a specific value by an application program after the Model already exists. +Similarly, a header exits the model in one of two ways: by being serialized by +a Generator, or by being retrieved from a Model by an application program. The +Policy object provides hooks for all four of these pathways. + +The model storage for headers is a list of (name, value) tuples. + +The Parser identifies headers during parsing, and passes them to the +:meth:`~email.policy.Policy.header_source_parse` method of the Policy. The +result of that method is the (name, value) tuple to be stored in the model. + +When an application program supplies a header value (for example, through the +`Message` object `__setitem__` interface), the name and the value are passed to +the :meth:`~email.policy.Policy.header_store_parse` method of the Policy, which +returns the (name, value) tuple to be stored in the model. + +When an application program retrieves a header (through any of the dict or list +interfaces of `Message`), the name and value are passed to the +:meth:`~email.policy.Policy.header_fetch_parse` method of the Policy to +obtain the value returned to the application. + +When a Generator requests a header during serialization, the name and value are +passed to the :meth:`~email.policy.Policy.fold` method of the Policy, which +returns a string containing line breaks in the appropriate places. The +:meth:`~email.policy.Policy.cte_type` Policy control determines whether or +not Content Transfer Encoding is performed on the data in the header. There is +also a :meth:`~email.policy.Policy.binary_fold` method for use by generators +that produce binary output, which returns the folded header as binary data, +possibly folded at different places than the corresponding string would be. + + +Handling Binary Data +-------------------- + +In an ideal world all message data would conform to the RFCs, meaning that the +parser could decode the message into the idealized unicode message that the +sender originally wrote. In the real world, the email package must also be +able to deal with badly formatted messages, including messages containing +non-ASCII characters that either have no indicated character set or are not +valid characters in the indicated character set. + +Since email messages are *primarily* text data, and operations on message data +are primarily text operations (except for binary payloads of course), the model +stores all text data as unicode strings. Un-decodable binary inside text +data is handled by using the `surrogateescape` error handler of the ASCII +codec. As with the binary filenames the error handler was introduced to +handle, this allows the email package to "carry" the binary data received +during parsing along until the output stage, at which time it is regenerated +in its original form. + +This carried binary data is almost entirely an implementation detail. The one +place where it is visible in the API is in the "internal" API. A Parser must +do the `surrogateescape` encoding of binary input data, and pass that data to +the appropriate Policy method. The "internal" interface used by the Generator +to access header values preserves the `surrogateescaped` bytes. All other +interfaces convert the binary data either back into bytes or into a safe form +(losing information in some cases). + + +Backward Compatibility +---------------------- + +The :class:`~email.policy.Policy.Compat32` Policy provides backward +compatibility with version 5.1 of the email package. It does this via the +following implementation of the four+1 Policy methods described above: + +header_source_parse + Splits the first line on the colon to obtain the name, discards any spaces + after the colon, and joins the remainder of the line with all of the + remaining lines, preserving the linesep characters to obtain the value. + Trailing carriage return and/or linefeed characters are stripped from the + resulting value string. + +header_store_parse + Returns the name and value exactly as received from the application. + +header_fetch_parse + If the value contains any `surrogateescaped` binary data, return the value + as a :class:`~email.header.Header` object, using the character set + `unknown-8bit`. Otherwise just returns the value. + +fold + Uses :class:`~email.header.Header`'s folding to fold headers in the + same way the email5.1 generator did. + +binary_fold + Same as fold, but encodes to 'ascii'. + + +New Algorithm +------------- + +header_source_parse + Same as legacy behavior. + +header_store_parse + Same as legacy behavior. + +header_fetch_parse + If the value is already a header object, returns it. Otherwise, parses the + value using the new parser, and returns the resulting object as the value. + `surrogateescaped` bytes get turned into unicode unknown character code + points. + +fold + Uses the new header folding algorithm, respecting the policy settings. + surrogateescaped bytes are encoded using the ``unknown-8bit`` charset for + ``cte_type=7bit`` or ``8bit``. Returns a string. + + At some point there will also be a ``cte_type=unicode``, and for that + policy fold will serialize the idealized unicode message with RFC-like + folding, converting any surrogateescaped bytes into the unicode + unknown character glyph. + +binary_fold + Uses the new header folding algorithm, respecting the policy settings. + surrogateescaped bytes are encoded using the `unknown-8bit` charset for + ``cte_type=7bit``, and get turned back into bytes for ``cte_type=8bit``. + Returns bytes. + + At some point there will also be a ``cte_type=unicode``, and for that + policy binary_fold will serialize the message according to :rfc:``5335``. diff --git a/Lib/email/feedparser.py b/Lib/email/feedparser.py --- a/Lib/email/feedparser.py +++ b/Lib/email/feedparser.py @@ -25,7 +25,7 @@ from email import errors from email import message -from email import policy +from email._policybase import compat32 NLCRE = re.compile('\r\n|\r|\n') NLCRE_bol = re.compile('(\r\n|\r|\n)') @@ -135,7 +135,7 @@ class FeedParser: """A feed-style parser of email.""" - def __init__(self, _factory=message.Message, *, policy=policy.default): + def __init__(self, _factory=message.Message, *, policy=compat32): """_factory is called with no arguments to create a new message obj The policy keyword specifies a policy object that controls a number of @@ -145,6 +145,12 @@ """ self._factory = _factory self.policy = policy + try: + _factory(policy=self.policy) + self._factory_kwds = lambda: {'policy': self.policy} + except TypeError: + # Assume this is an old-style factory + self._factory_kwds = lambda: {} self._input = BufferedSubFile() self._msgstack = [] self._parse = self._parsegen().__next__ @@ -181,7 +187,7 @@ return root def _new_message(self): - msg = self._factory() + msg = self._factory(**self._factory_kwds()) if self._cur and self._cur.get_content_type() == 'multipart/digest': msg.set_default_type('message/rfc822') if self._msgstack: @@ -458,9 +464,7 @@ lastvalue.append(line) continue if lastheader: - # XXX reconsider the joining of folded lines - lhdr = EMPTYSTRING.join(lastvalue)[:-1].rstrip('\r\n') - self._cur[lastheader] = lhdr + self._cur.set_raw(*self.policy.header_source_parse(lastvalue)) lastheader, lastvalue = '', [] # Check for envelope header, i.e. unix-from if line.startswith('From '): @@ -487,16 +491,16 @@ i = line.find(':') if i < 0: defect = errors.MalformedHeaderDefect(line) + # XXX: fixme (defect not going through policy) self._cur.defects.append(defect) continue lastheader = line[:i] - lastvalue = [line[i+1:].lstrip()] + lastvalue = [line] # Done with all the lines, so handle the last header. if lastheader: - # XXX reconsider the joining of folded lines - self._cur[lastheader] = EMPTYSTRING.join(lastvalue).rstrip('\r\n') + self._cur.set_raw(*self.policy.header_source_parse(lastvalue)) - + class BytesFeedParser(FeedParser): """Like FeedParser, but feed accepts bytes.""" diff --git a/Lib/email/generator.py b/Lib/email/generator.py --- a/Lib/email/generator.py +++ b/Lib/email/generator.py @@ -13,9 +13,9 @@ import warnings from io import StringIO, BytesIO -from email import policy +from email._policybase import compat32 from email.header import Header -from email.message import _has_surrogates +from email.utils import _has_surrogates import email.charset as _charset UNDERSCORE = '_' @@ -36,7 +36,7 @@ # def __init__(self, outfp, mangle_from_=True, maxheaderlen=None, *, - policy=policy.default): + policy=None): """Create the generator for message flattening. outfp is the output file-like object for writing the message to. It @@ -60,8 +60,7 @@ """ self._fp = outfp self._mangle_from_ = mangle_from_ - self._maxheaderlen = (maxheaderlen if maxheaderlen is not None else - policy.max_line_length) + self.maxheaderlen = maxheaderlen self.policy = policy def write(self, s): @@ -87,20 +86,33 @@ # from the msg, and _encoded_XXX constants for operating on data that # has already been converted (to bytes in the BytesGenerator) and # inserted into a temporary buffer. - self._NL = linesep if linesep is not None else self.policy.linesep + policy = msg.policy if self.policy is None else self.policy + if linesep is not None: + policy = policy.clone(linesep=linesep) + if self.maxheaderlen is not None: + policy = policy.clone(max_line_length=self.maxheaderlen) + self._NL = policy.linesep self._encoded_NL = self._encode(self._NL) self._EMPTY = '' self._encoded_EMTPY = self._encode('') - if unixfrom: - ufrom = msg.get_unixfrom() - if not ufrom: - ufrom = 'From nobody ' + time.ctime(time.time()) - self.write(ufrom + self._NL) - self._write(msg) + p = self.policy + try: + self.policy = policy + if unixfrom: + ufrom = msg.get_unixfrom() + if not ufrom: + ufrom = 'From nobody ' + time.ctime(time.time()) + self.write(ufrom + self._NL) + self._write(msg) + finally: + self.policy = p def clone(self, fp): """Clone this generator with the exact same options.""" - return self.__class__(fp, self._mangle_from_, self._maxheaderlen) + return self.__class__(fp, + self._mangle_from_, + None, # Use policy setting, which we've adjusted + policy=self.policy) # # Protected interface - undocumented ;/ @@ -175,16 +187,8 @@ # def _write_headers(self, msg): - for h, v in msg.items(): - self.write('%s: ' % h) - if isinstance(v, Header): - self.write(v.encode( - maxlinelen=self._maxheaderlen, linesep=self._NL)+self._NL) - else: - # Header's got lots of smarts, so use it. - header = Header(v, maxlinelen=self._maxheaderlen, - header_name=h) - self.write(header.encode(linesep=self._NL)+self._NL) + for h, v in msg.raw_items(): + self.write(self.policy.fold(h, v)) # A blank line always separates headers from body self.write(self._NL) @@ -265,12 +269,12 @@ # The contents of signed parts has to stay unmodified in order to keep # the signature intact per RFC1847 2.1, so we disable header wrapping. # RDM: This isn't enough to completely preserve the part, but it helps. - old_maxheaderlen = self._maxheaderlen + p = self.policy + self.policy = p.clone(max_line_length=0) try: - self._maxheaderlen = 0 self._handle_multipart(msg) finally: - self._maxheaderlen = old_maxheaderlen + self.policy = p def _handle_message_delivery_status(self, msg): # We can't just write the headers directly to self's file object @@ -347,9 +351,9 @@ Functionally identical to the base Generator except that the output is bytes and not string. When surrogates were used in the input to encode bytes, these are decoded back to bytes for output. If the policy has - must_be_7bit set true, then the message is transformed such that the - non-ASCII bytes are properly content transfer encoded, using the - charset unknown-8bit. + cte_type set to 7bit, then the message is transformed such that the + non-ASCII bytes are properly content transfer encoded, using the charset + unknown-8bit. The outfp object must accept bytes in its write method. """ @@ -370,27 +374,8 @@ def _write_headers(self, msg): # This is almost the same as the string version, except for handling # strings with 8bit bytes. - for h, v in msg._headers: - self.write('%s: ' % h) - if isinstance(v, str): - if _has_surrogates(v): - if not self.policy.must_be_7bit: - # If we have raw 8bit data in a byte string, we have no idea - # what the encoding is. There is no safe way to split this - # string. If it's ascii-subset, then we could do a normal - # ascii split, but if it's multibyte then we could break the - # string. There's no way to know so the least harm seems to - # be to not split the string and risk it being too long. - self.write(v+NL) - continue - h = Header(v, charset=_charset.UNKNOWN8BIT, header_name=h) - else: - h = Header(v, header_name=h) - else: - # Assume it is a Header-like object. - h = v - self.write(h.encode(linesep=self._NL, - maxlinelen=self._maxheaderlen)+self._NL) + for h, v in msg.raw_items(): + self._fp.write(self.policy.fold_binary(h, v)) # A blank line always separates headers from body self.write(self._NL) @@ -399,7 +384,7 @@ # just write it back out. if msg._payload is None: return - if _has_surrogates(msg._payload) and not self.policy.must_be_7bit: + if _has_surrogates(msg._payload) and not self.policy.cte_type=='7bit': self.write(msg._payload) else: super(BytesGenerator,self)._handle_text(msg) diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -10,13 +10,12 @@ import uu import base64 import binascii -import warnings from io import BytesIO, StringIO # Intrapackage imports from email import utils from email import errors -from email import header +from email._policybase import compat32 from email import charset as _charset Charset = _charset.Charset @@ -26,24 +25,6 @@ # existence of which force quoting of the parameter value. tspecials = re.compile(r'[ \(\)<>@,;:\\"/\[\]\?=]') -# How to figure out if we are processing strings that come from a byte -# source with undecodable characters. -_has_surrogates = re.compile( - '([^\ud800-\udbff]|\A)[\udc00-\udfff]([^\udc00-\udfff]|\Z)').search - - -# Helper functions -def _sanitize_header(name, value): - # If the header value contains surrogates, return a Header using - # the unknown-8bit charset to encode the bytes as encoded words. - if not isinstance(value, str): - # Assume it is already a header object - return value - if _has_surrogates(value): - return header.Header(value, charset=_charset.UNKNOWN8BIT, - header_name=name) - else: - return value def _splitparam(param): # Split header parameters. BAW: this may be too simple. It isn't @@ -136,7 +117,8 @@ you must use the explicit API to set or get all the headers. Not all of the mapping methods are implemented. """ - def __init__(self): + def __init__(self, policy=compat32): + self.policy = policy self._headers = [] self._unixfrom = None self._payload = None @@ -246,7 +228,7 @@ cte = str(self.get('content-transfer-encoding', '')).lower() # payload may be bytes here. if isinstance(payload, str): - if _has_surrogates(payload): + if utils._has_surrogates(payload): bpayload = payload.encode('ascii', 'surrogateescape') if not decode: try: @@ -362,7 +344,7 @@ Note: this does not overwrite an existing header with the same field name. Use __delitem__() first to delete any existing headers. """ - self._headers.append((name, val)) + self._headers.append(self.policy.header_store_parse(name, val)) def __delitem__(self, name): """Delete all occurrences of a header, if present. @@ -401,7 +383,8 @@ Any fields deleted and re-inserted are always appended to the header list. """ - return [_sanitize_header(k, v) for k, v in self._headers] + return [self.policy.header_fetch_parse(k, v) + for k, v in self._headers] def items(self): """Get all the message's header fields and values. @@ -411,7 +394,8 @@ Any fields deleted and re-inserted are always appended to the header list. """ - return [(k, _sanitize_header(k, v)) for k, v in self._headers] + return [(k, self.policy.header_fetch_parse(k, v)) + for k, v in self._headers] def get(self, name, failobj=None): """Get a header value. @@ -422,10 +406,29 @@ name = name.lower() for k, v in self._headers: if k.lower() == name: - return _sanitize_header(k, v) + return self.policy.header_fetch_parse(k, v) return failobj # + # "Internal" methods (public API, but only intended for use by a parser + # or generator, not normal application code. + # + + def set_raw(self, name, value): + """Store name and value in the model without modification. + + This is an "internal" API, intended only for use by a parser. + """ + self._headers.append((name, value)) + + def raw_items(self): + """Return the (name, value) header pairs without modification. + + This is an "internal" API, intended only for use by a generator. + """ + return iter(self._headers.copy()) + + # # Additional useful stuff # @@ -442,7 +445,7 @@ name = name.lower() for k, v in self._headers: if k.lower() == name: - values.append(_sanitize_header(k, v)) + values.append(self.policy.header_fetch_parse(k, v)) if not values: return failobj return values @@ -475,7 +478,7 @@ parts.append(_formatparam(k.replace('_', '-'), v)) if _value is not None: parts.insert(0, _value) - self._headers.append((_name, SEMISPACE.join(parts))) + self[_name] = SEMISPACE.join(parts) def replace_header(self, _name, _value): """Replace a header. @@ -487,7 +490,7 @@ _name = _name.lower() for i, (k, v) in zip(range(len(self._headers)), self._headers): if k.lower() == _name: - self._headers[i] = (k, _value) + self._headers[i] = self.policy.header_store_parse(k, _value) break else: raise KeyError(_name) @@ -805,7 +808,8 @@ parts.append(k) else: parts.append('%s=%s' % (k, v)) - newheaders.append((h, SEMISPACE.join(parts))) + val = SEMISPACE.join(parts) + newheaders.append(self.policy.header_store_parse(h, val)) else: newheaders.append((h, v)) diff --git a/Lib/email/parser.py b/Lib/email/parser.py --- a/Lib/email/parser.py +++ b/Lib/email/parser.py @@ -11,12 +11,12 @@ from email.feedparser import FeedParser from email.message import Message -from email import policy +from email._policybase import compat32 class Parser: - def __init__(self, _class=Message, *, policy=policy.default): + def __init__(self, _class=Message, *, policy=compat32): """Parser of RFC 2822 and MIME email messages. Creates an in-memory object tree representing the email message, which diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -1,174 +1,12 @@ -"""Policy framework for the email package. - -Allows fine grained feature control of how the package parses and emits data. +"""This will be the home for the policy that hooks in the new +code that adds all the email6 features. """ -__all__ = [ - 'Policy', - 'default', - 'strict', - 'SMTP', - 'HTTP', - ] +from email._policybase import Policy, compat32, Compat32 +# XXX: temporarily derive everything from compat32. -class _PolicyBase: - - """Policy Object basic framework. - - This class is useless unless subclassed. A subclass should define - class attributes with defaults for any values that are to be - managed by the Policy object. The constructor will then allow - non-default values to be set for these attributes at instance - creation time. The instance will be callable, taking these same - attributes keyword arguments, and returning a new instance - identical to the called instance except for those values changed - by the keyword arguments. Instances may be added, yielding new - instances with any non-default values from the right hand - operand overriding those in the left hand operand. That is, - - A + B == A() - - The repr of an instance can be used to reconstruct the object - if and only if the repr of the values can be used to reconstruct - those values. - - """ - - def __init__(self, **kw): - """Create new Policy, possibly overriding some defaults. - - See class docstring for a list of overridable attributes. - - """ - for name, value in kw.items(): - if hasattr(self, name): - super(_PolicyBase,self).__setattr__(name, value) - else: - raise TypeError( - "{!r} is an invalid keyword argument for {}".format( - name, self.__class__.__name__)) - - def __repr__(self): - args = [ "{}={!r}".format(name, value) - for name, value in self.__dict__.items() ] - return "{}({})".format(self.__class__.__name__, ', '.join(args)) - - def clone(self, **kw): - """Return a new instance with specified attributes changed. - - The new instance has the same attribute values as the current object, - except for the changes passed in as keyword arguments. - - """ - for attr, value in self.__dict__.items(): - if attr not in kw: - kw[attr] = value - return self.__class__(**kw) - - def __setattr__(self, name, value): - if hasattr(self, name): - msg = "{!r} object attribute {!r} is read-only" - else: - msg = "{!r} object has no attribute {!r}" - raise AttributeError(msg.format(self.__class__.__name__, name)) - - def __add__(self, other): - """Non-default values from right operand override those from left. - - The object returned is a new instance of the subclass. - - """ - return self.clone(**other.__dict__) - - -class Policy(_PolicyBase): - - """Controls for how messages are interpreted and formatted. - - Most of the classes and many of the methods in the email package - accept Policy objects as parameters. A Policy object contains a set - of values and functions that control how input is interpreted and how - output is rendered. For example, the parameter 'raise_on_defect' - controls whether or not an RFC violation throws an error or not, - while 'max_line_length' controls the maximum length of output lines - when a Message is serialized. - - Any valid attribute may be overridden when a Policy is created by - passing it as a keyword argument to the constructor. Policy - objects are immutable, but a new Policy object can be created - with only certain values changed by calling the Policy instance - with keyword arguments. Policy objects can also be added, - producing a new Policy object in which the non-default attributes - set in the right hand operand overwrite those specified in the - left operand. - - Settable attributes: - - raise_on_defect -- If true, then defects should be raised - as errors. Default False. - - linesep -- string containing the value to use as - separation between output lines. Default '\n'. - - must_be_7bit -- output must contain only 7bit clean data. - Default False. - - max_line_length -- maximum length of lines, excluding 'linesep', - during serialization. None means no line - wrapping is done. Default is 78. - - Methods: - - register_defect(obj, defect) - defect is a Defect instance. The default implementation appends defect - to the objs 'defects' attribute. - - handle_defect(obj, defect) - intended to be called by parser code that finds a defect. If - raise_on_defect is True, defect is raised as an error, otherwise - register_defect is called. - - """ - - raise_on_defect = False - linesep = '\n' - must_be_7bit = False - max_line_length = 78 - - def handle_defect(self, obj, defect): - """Based on policy, either raise defect or call register_defect. - - handle_defect(obj, defect) - - defect should be a Defect subclass, but in any case must be an - Exception subclass. obj is the object on which the defect should be - registered if it is not raised. If the raise_on_defect is True, the - defect is raised as an error, otherwise the object and the defect are - passed to register_defect. - - This class is intended to be called by parsers that discover defects, - and will not be called from code using the library unless that code is - implementing an alternate parser. - - """ - if self.raise_on_defect: - raise defect - self.register_defect(obj, defect) - - def register_defect(self, obj, defect): - """Record 'defect' on 'obj'. - - Called by handle_defect if raise_on_defect is False. This method is - part of the Policy API so that Policy subclasses can implement custom - defect handling. The default implementation calls the append method - of the defects attribute of obj. - - """ - obj.defects.append(defect) - - -default = Policy() +default = compat32 strict = default.clone(raise_on_defect=True) SMTP = default.clone(linesep='\r\n') HTTP = default.clone(linesep='\r\n', max_line_length=None) diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -57,6 +57,10 @@ specialsre = re.compile(r'[][\\()<>@,:;".]') escapesre = re.compile(r'[\\"]') +# How to figure out if we are processing strings that come from a byte +# source with undecodable characters. +_has_surrogates = re.compile( + '([^\ud800-\udbff]|\A)[\udc00-\udfff]([^\udc00-\udfff]|\Z)').search # Helpers diff --git a/Lib/test/test_email/__init__.py b/Lib/test/test_email/__init__.py --- a/Lib/test/test_email/__init__.py +++ b/Lib/test/test_email/__init__.py @@ -3,6 +3,8 @@ import unittest import test.support import email +from email.message import Message +from email._policybase import compat32 from test.test_email import __file__ as landmark # Run all tests in package for '-m unittest test.test_email' @@ -36,16 +38,26 @@ class TestEmailBase(unittest.TestCase): maxDiff = None + # Currently the default policy is compat32. By setting that as the default + # here we make minimal changes in the test_email tests compared to their + # pre-3.3 state. + policy = compat32 def __init__(self, *args, **kw): super().__init__(*args, **kw) self.addTypeEqualityFunc(bytes, self.assertBytesEqual) + # Backward compatibility to minimize test_email test changes. ndiffAssertEqual = unittest.TestCase.assertEqual def _msgobj(self, filename): with openfile(filename) as fp: - return email.message_from_file(fp) + return email.message_from_file(fp, policy=self.policy) + + def _str_msg(self, string, message=Message, policy=None): + if policy is None: + policy = self.policy + return email.message_from_string(string, message, policy=policy) def _bytes_repr(self, b): return [repr(x) for x in b.splitlines(keepends=True)] diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -16,6 +16,7 @@ from itertools import chain import email +import email.policy from email.charset import Charset from email.header import Header, decode_header, make_header @@ -1805,11 +1806,7 @@ # Test some badly formatted messages -class TestNonConformantBase: - - def _msgobj(self, filename): - with openfile(filename) as fp: - return email.message_from_file(fp, policy=self.policy) +class TestNonConformant(TestEmailBase): def test_parse_missing_minor_type(self): eq = self.assertEqual @@ -1818,24 +1815,26 @@ eq(msg.get_content_maintype(), 'text') eq(msg.get_content_subtype(), 'plain') + # test_parser.TestMessageDefectDetectionBase def test_same_boundary_inner_outer(self): unless = self.assertTrue msg = self._msgobj('msg_15.txt') # XXX We can probably eventually do better inner = msg.get_payload(0) unless(hasattr(inner, 'defects')) - self.assertEqual(len(self.get_defects(inner)), 1) - unless(isinstance(self.get_defects(inner)[0], + self.assertEqual(len(inner.defects), 1) + unless(isinstance(inner.defects[0], errors.StartBoundaryNotFoundDefect)) + # test_parser.TestMessageDefectDetectionBase def test_multipart_no_boundary(self): unless = self.assertTrue msg = self._msgobj('msg_25.txt') unless(isinstance(msg.get_payload(), str)) - self.assertEqual(len(self.get_defects(msg)), 2) - unless(isinstance(self.get_defects(msg)[0], + self.assertEqual(len(msg.defects), 2) + unless(isinstance(msg.defects[0], errors.NoBoundaryInMultipartDefect)) - unless(isinstance(self.get_defects(msg)[1], + unless(isinstance(msg.defects[1], errors.MultipartInvariantViolationDefect)) multipart_msg = textwrap.dedent("""\ @@ -1861,27 +1860,26 @@ --===============3344438784458119861==-- """) + # test_parser.TestMessageDefectDetectionBase def test_multipart_invalid_cte(self): - msg = email.message_from_string( - self.multipart_msg.format("\nContent-Transfer-Encoding: base64"), - policy = self.policy) - self.assertEqual(len(self.get_defects(msg)), 1) - self.assertIsInstance(self.get_defects(msg)[0], + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) + self.assertEqual(len(msg.defects), 1) + self.assertIsInstance(msg.defects[0], errors.InvalidMultipartContentTransferEncodingDefect) + # test_parser.TestMessageDefectDetectionBase def test_multipart_no_cte_no_defect(self): - msg = email.message_from_string( - self.multipart_msg.format(''), - policy = self.policy) - self.assertEqual(len(self.get_defects(msg)), 0) - + msg = self._str_msg(self.multipart_msg.format('')) + self.assertEqual(len(msg.defects), 0) + + # test_parser.TestMessageDefectDetectionBase def test_multipart_valid_cte_no_defect(self): for cte in ('7bit', '8bit', 'BINary'): - msg = email.message_from_string( + msg = self._str_msg( self.multipart_msg.format( - "\nContent-Transfer-Encoding: {}".format(cte)), - policy = self.policy) - self.assertEqual(len(self.get_defects(msg)), 0) + "\nContent-Transfer-Encoding: {}".format(cte))) + self.assertEqual(len(msg.defects), 0) def test_invalid_content_type(self): eq = self.assertEqual @@ -1932,16 +1930,18 @@ counter to RFC 2822, there's no separating newline here """) + # test_parser.TestMessageDefectDetectionBase def test_lying_multipart(self): unless = self.assertTrue msg = self._msgobj('msg_41.txt') unless(hasattr(msg, 'defects')) - self.assertEqual(len(self.get_defects(msg)), 2) - unless(isinstance(self.get_defects(msg)[0], + self.assertEqual(len(msg.defects), 2) + unless(isinstance(msg.defects[0], errors.NoBoundaryInMultipartDefect)) - unless(isinstance(self.get_defects(msg)[1], + unless(isinstance(msg.defects[1], errors.MultipartInvariantViolationDefect)) + # test_parser.TestMessageDefectDetectionBase def test_missing_start_boundary(self): outer = self._msgobj('msg_42.txt') # The message structure is: @@ -1953,71 +1953,21 @@ # # [*] This message is missing its start boundary bad = outer.get_payload(1).get_payload(0) - self.assertEqual(len(self.get_defects(bad)), 1) - self.assertTrue(isinstance(self.get_defects(bad)[0], + self.assertEqual(len(bad.defects), 1) + self.assertTrue(isinstance(bad.defects[0], errors.StartBoundaryNotFoundDefect)) + # test_parser.TestMessageDefectDetectionBase def test_first_line_is_continuation_header(self): eq = self.assertEqual m = ' Line 1\nLine 2\nLine 3' - msg = email.message_from_string(m, policy=self.policy) + msg = email.message_from_string(m) eq(msg.keys(), []) eq(msg.get_payload(), 'Line 2\nLine 3') - eq(len(self.get_defects(msg)), 1) - self.assertTrue(isinstance(self.get_defects(msg)[0], + eq(len(msg.defects), 1) + self.assertTrue(isinstance(msg.defects[0], errors.FirstHeaderLineIsContinuationDefect)) - eq(self.get_defects(msg)[0].line, ' Line 1\n') - - -class TestNonConformant(TestNonConformantBase, TestEmailBase): - - policy=email.policy.default - - def get_defects(self, obj): - return obj.defects - - -class TestNonConformantCapture(TestNonConformantBase, TestEmailBase): - - class CapturePolicy(email.policy.Policy): - captured = None - def register_defect(self, obj, defect): - self.captured.append(defect) - - def setUp(self): - self.policy = self.CapturePolicy(captured=list()) - - def get_defects(self, obj): - return self.policy.captured - - -class TestRaisingDefects(TestEmailBase): - - def _msgobj(self, filename): - with openfile(filename) as fp: - return email.message_from_file(fp, policy=email.policy.strict) - - def test_same_boundary_inner_outer(self): - with self.assertRaises(errors.StartBoundaryNotFoundDefect): - self._msgobj('msg_15.txt') - - def test_multipart_no_boundary(self): - with self.assertRaises(errors.NoBoundaryInMultipartDefect): - self._msgobj('msg_25.txt') - - def test_lying_multipart(self): - with self.assertRaises(errors.NoBoundaryInMultipartDefect): - self._msgobj('msg_41.txt') - - - def test_missing_start_boundary(self): - with self.assertRaises(errors.StartBoundaryNotFoundDefect): - self._msgobj('msg_42.txt') - - def test_first_line_is_continuation_header(self): - m = ' Line 1\nLine 2\nLine 3' - with self.assertRaises(errors.FirstHeaderLineIsContinuationDefect): - msg = email.message_from_string(m, policy=email.policy.strict) + eq(msg.defects[0].line, ' Line 1\n') # Test RFC 2047 header encoding and decoding @@ -2610,6 +2560,13 @@ for subpart in msg.walk(): unless(isinstance(subpart, MyMessage)) + def test_custom_message_does_not_require_arguments(self): + class MyMessage(Message): + def __init__(self): + super().__init__() + msg = self._str_msg("Subject: test\n\ntest", MyMessage) + self.assertTrue(isinstance(msg, MyMessage)) + def test__all__(self): module = __import__('email') self.assertEqual(sorted(module.__all__), [ @@ -3137,25 +3094,6 @@ g.flatten(msg, linesep='\r\n') self.assertEqual(s.getvalue(), text) - def test_crlf_control_via_policy(self): - with openfile('msg_26.txt', newline='\n') as fp: - text = fp.read() - msg = email.message_from_string(text) - s = StringIO() - g = email.generator.Generator(s, policy=email.policy.SMTP) - g.flatten(msg) - self.assertEqual(s.getvalue(), text) - - def test_flatten_linesep_overrides_policy(self): - # msg_27 is lf separated - with openfile('msg_27.txt', newline='\n') as fp: - text = fp.read() - msg = email.message_from_string(text) - s = StringIO() - g = email.generator.Generator(s, policy=email.policy.SMTP) - g.flatten(msg, linesep='\n') - self.assertEqual(s.getvalue(), text) - maxDiff = None def test_multipart_digest_with_extra_mime_headers(self): @@ -3646,44 +3584,6 @@ s.getvalue(), 'Subject: =?utf-8?b?xb5sdcWlb3XEjWvDvSBrxa/FiA==?=\r\n\r\n') - def test_crlf_control_via_policy(self): - # msg_26 is crlf terminated - with openfile('msg_26.txt', 'rb') as fp: - text = fp.read() - msg = email.message_from_bytes(text) - s = BytesIO() - g = email.generator.BytesGenerator(s, policy=email.policy.SMTP) - g.flatten(msg) - self.assertEqual(s.getvalue(), text) - - def test_flatten_linesep_overrides_policy(self): - # msg_27 is lf separated - with openfile('msg_27.txt', 'rb') as fp: - text = fp.read() - msg = email.message_from_bytes(text) - s = BytesIO() - g = email.generator.BytesGenerator(s, policy=email.policy.SMTP) - g.flatten(msg, linesep='\n') - self.assertEqual(s.getvalue(), text) - - def test_must_be_7bit_handles_unknown_8bit(self): - msg = email.message_from_bytes(self.non_latin_bin_msg) - out = BytesIO() - g = email.generator.BytesGenerator(out, - policy=email.policy.default.clone(must_be_7bit=True)) - g.flatten(msg) - self.assertEqual(out.getvalue(), - self.non_latin_bin_msg_as7bit_wrapped.encode('ascii')) - - def test_must_be_7bit_transforms_8bit_cte(self): - msg = email.message_from_bytes(self.latin_bin_msg) - out = BytesIO() - g = email.generator.BytesGenerator(out, - policy=email.policy.default.clone(must_be_7bit=True)) - g.flatten(msg) - self.assertEqual(out.getvalue(), - self.latin_bin_msg_as7bit.encode('ascii')) - maxDiff = None diff --git a/Lib/test/test_email/test_generator.py b/Lib/test/test_email/test_generator.py --- a/Lib/test/test_email/test_generator.py +++ b/Lib/test/test_email/test_generator.py @@ -11,6 +11,8 @@ class TestGeneratorBase(): + policy = policy.compat32 + long_subject = { 0: textwrap.dedent("""\ To: whom_it_may_concern at example.com @@ -58,11 +60,11 @@ long_subject[100] = long_subject[0] def maxheaderlen_parameter_test(self, n): - msg = self.msgmaker(self.long_subject[0]) + msg = self.msgmaker(self.typ(self.long_subject[0])) s = self.ioclass() g = self.genclass(s, maxheaderlen=n) g.flatten(msg) - self.assertEqual(s.getvalue(), self.long_subject[n]) + self.assertEqual(s.getvalue(), self.typ(self.long_subject[n])) def test_maxheaderlen_parameter_0(self): self.maxheaderlen_parameter_test(0) @@ -77,11 +79,11 @@ self.maxheaderlen_parameter_test(20) def maxheaderlen_policy_test(self, n): - msg = self.msgmaker(self.long_subject[0]) + msg = self.msgmaker(self.typ(self.long_subject[0])) s = self.ioclass() g = self.genclass(s, policy=policy.default.clone(max_line_length=n)) g.flatten(msg) - self.assertEqual(s.getvalue(), self.long_subject[n]) + self.assertEqual(s.getvalue(), self.typ(self.long_subject[n])) def test_maxheaderlen_policy_0(self): self.maxheaderlen_policy_test(0) @@ -96,12 +98,12 @@ self.maxheaderlen_policy_test(20) def maxheaderlen_parm_overrides_policy_test(self, n): - msg = self.msgmaker(self.long_subject[0]) + msg = self.msgmaker(self.typ(self.long_subject[0])) s = self.ioclass() g = self.genclass(s, maxheaderlen=n, policy=policy.default.clone(max_line_length=10)) g.flatten(msg) - self.assertEqual(s.getvalue(), self.long_subject[n]) + self.assertEqual(s.getvalue(), self.typ(self.long_subject[n])) def test_maxheaderlen_parm_overrides_policy_0(self): self.maxheaderlen_parm_overrides_policy_test(0) @@ -115,21 +117,84 @@ def test_maxheaderlen_parm_overrides_policy_20(self): self.maxheaderlen_parm_overrides_policy_test(20) + def test_crlf_control_via_policy(self): + source = "Subject: test\r\n\r\ntest body\r\n" + expected = source + msg = self.msgmaker(self.typ(source)) + s = self.ioclass() + g = self.genclass(s, policy=policy.SMTP) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(expected)) + + def test_flatten_linesep_overrides_policy(self): + source = "Subject: test\n\ntest body\n" + expected = source + msg = self.msgmaker(self.typ(source)) + s = self.ioclass() + g = self.genclass(s, policy=policy.SMTP) + g.flatten(msg, linesep='\n') + self.assertEqual(s.getvalue(), self.typ(expected)) + class TestGenerator(TestGeneratorBase, TestEmailBase): - msgmaker = staticmethod(message_from_string) genclass = Generator ioclass = io.StringIO + typ = str + + def msgmaker(self, msg, policy=None): + policy = self.policy if policy is None else policy + return message_from_string(msg, policy=policy) class TestBytesGenerator(TestGeneratorBase, TestEmailBase): - msgmaker = staticmethod(message_from_bytes) genclass = BytesGenerator ioclass = io.BytesIO - long_subject = {key: x.encode('ascii') - for key, x in TestGeneratorBase.long_subject.items()} + typ = lambda self, x: x.encode('ascii') + + def msgmaker(self, msg, policy=None): + policy = self.policy if policy is None else policy + return message_from_bytes(msg, policy=policy) + + def test_cte_type_7bit_handles_unknown_8bit(self): + source = ("Subject: Maintenant je vous pr?sente mon " + "coll?gue\n\n").encode('utf-8') + expected = ('Subject: =?unknown-8bit?q?Maintenant_je_vous_pr=C3=A9sente_mon_' + 'coll=C3=A8gue?=\n\n').encode('ascii') + msg = message_from_bytes(source) + s = io.BytesIO() + g = BytesGenerator(s, policy=self.policy.clone(cte_type='7bit')) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) + + def test_cte_type_7bit_transforms_8bit_cte(self): + source = textwrap.dedent("""\ + From: foo at bar.com + To: Dinsdale + Subject: Nudge nudge, wink, wink + Mime-Version: 1.0 + Content-Type: text/plain; charset="latin-1" + Content-Transfer-Encoding: 8bit + + oh l? l?, know what I mean, know what I mean? + """).encode('latin1') + msg = message_from_bytes(source) + expected = textwrap.dedent("""\ + From: foo at bar.com + To: Dinsdale + Subject: Nudge nudge, wink, wink + Mime-Version: 1.0 + Content-Type: text/plain; charset="iso-8859-1" + Content-Transfer-Encoding: quoted-printable + + oh l=E0 l=E0, know what I mean, know what I mean? + """).encode('ascii') + s = io.BytesIO() + g = BytesGenerator(s, policy=self.policy.clone(cte_type='7bit', + linesep='\n')) + g.flatten(msg) + self.assertEqual(s.getvalue(), expected) if __name__ == '__main__': diff --git a/Lib/test/test_email/test_parser.py b/Lib/test/test_email/test_parser.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test_parser.py @@ -0,0 +1,276 @@ +import io +import email +import textwrap +import unittest +from email._policybase import Compat32 +from email import errors +from email.message import Message +from test.test_email import TestEmailBase + + +class TestCustomMessage(TestEmailBase): + + class MyMessage(Message): + def __init__(self, policy): + self.check_policy = policy + super().__init__() + + MyPolicy = TestEmailBase.policy.clone(linesep='boo') + + def test_custom_message_gets_policy_if_possible_from_string(self): + msg = email.message_from_string("Subject: bogus\n\nmsg\n", + self.MyMessage, + policy=self.MyPolicy) + self.assertTrue(isinstance(msg, self.MyMessage)) + self.assertIs(msg.check_policy, self.MyPolicy) + + def test_custom_message_gets_policy_if_possible_from_file(self): + source_file = io.StringIO("Subject: bogus\n\nmsg\n") + msg = email.message_from_file(source_file, + self.MyMessage, + policy=self.MyPolicy) + self.assertTrue(isinstance(msg, self.MyMessage)) + self.assertIs(msg.check_policy, self.MyPolicy) + + # XXX add tests for other functions that take Message arg. + + +class TestMessageDefectDetectionBase: + + dup_boundary_msg = textwrap.dedent("""\ + Subject: XX + From: xx at xx.dk + To: XX + Mime-version: 1.0 + Content-type: multipart/mixed; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: multipart/alternative; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/plain; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + text + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/html; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: image/gif; name="xx.gif"; + Content-disposition: attachment + Content-transfer-encoding: base64 + + Some removed base64 encoded chars. + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + """) + + def test_same_boundary_inner_outer(self): + # XXX better would be to actually detect the duplicate. + msg = self._str_msg(self.dup_boundary_msg) + inner = msg.get_payload(0) + self.assertTrue(hasattr(inner, 'defects')) + self.assertEqual(len(self.get_defects(inner)), 1) + self.assertTrue(isinstance(self.get_defects(inner)[0], + errors.StartBoundaryNotFoundDefect)) + + def test_same_boundary_inner_outer_raises_on_defect(self): + with self.assertRaises(errors.StartBoundaryNotFoundDefect): + self._str_msg(self.dup_boundary_msg, + policy=self.policy.clone(raise_on_defect=True)) + + no_boundary_msg = textwrap.dedent("""\ + Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) + From: foobar + Subject: broken mail + MIME-Version: 1.0 + Content-Type: multipart/report; report-type=delivery-status; + + --JAB03225.986577786/zinfandel.lacita.com + + One part + + --JAB03225.986577786/zinfandel.lacita.com + Content-Type: message/delivery-status + + Header: Another part + + --JAB03225.986577786/zinfandel.lacita.com-- + """) + + def test_multipart_no_boundary(self): + msg = self._str_msg(self.no_boundary_msg) + self.assertTrue(isinstance(msg.get_payload(), str)) + self.assertEqual(len(self.get_defects(msg)), 2) + self.assertTrue(isinstance(self.get_defects(msg)[0], + errors.NoBoundaryInMultipartDefect)) + self.assertTrue(isinstance(self.get_defects(msg)[1], + errors.MultipartInvariantViolationDefect)) + + def test_multipart_no_boundary_raise_on_defect(self): + with self.assertRaises(errors.NoBoundaryInMultipartDefect): + self._str_msg(self.no_boundary_msg, + policy=self.policy.clone(raise_on_defect=True)) + + multipart_msg = textwrap.dedent("""\ + Date: Wed, 14 Nov 2007 12:56:23 GMT + From: foo at bar.invalid + To: foo at bar.invalid + Subject: Content-Transfer-Encoding: base64 and multipart + MIME-Version: 1.0 + Content-Type: multipart/mixed; + boundary="===============3344438784458119861=="{} + + --===============3344438784458119861== + Content-Type: text/plain + + Test message + + --===============3344438784458119861== + Content-Type: application/octet-stream + Content-Transfer-Encoding: base64 + + YWJj + + --===============3344438784458119861==-- + """) + + def test_multipart_invalid_cte(self): + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) + self.assertEqual(len(self.get_defects(msg)), 1) + self.assertIsInstance(self.get_defects(msg)[0], + errors.InvalidMultipartContentTransferEncodingDefect) + + def test_multipart_invalid_cte_raise_on_defect(self): + with self.assertRaises( + errors.InvalidMultipartContentTransferEncodingDefect): + self._str_msg( + self.multipart_msg.format( + "\nContent-Transfer-Encoding: base64"), + policy=self.policy.clone(raise_on_defect=True)) + + def test_multipart_no_cte_no_defect(self): + msg = self._str_msg(self.multipart_msg.format('')) + self.assertEqual(len(self.get_defects(msg)), 0) + + def test_multipart_valid_cte_no_defect(self): + for cte in ('7bit', '8bit', 'BINary'): + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: "+cte)) + self.assertEqual(len(self.get_defects(msg)), 0, "cte="+cte) + + lying_multipart_msg = textwrap.dedent("""\ + From: "Allison Dunlap" + To: yyy at example.com + Subject: 64423 + Date: Sun, 11 Jul 2004 16:09:27 -0300 + MIME-Version: 1.0 + Content-Type: multipart/alternative; + + Blah blah blah + """) + + def test_lying_multipart(self): + msg = self._str_msg(self.lying_multipart_msg) + self.assertTrue(hasattr(msg, 'defects')) + self.assertEqual(len(self.get_defects(msg)), 2) + self.assertTrue(isinstance(self.get_defects(msg)[0], + errors.NoBoundaryInMultipartDefect)) + self.assertTrue(isinstance(self.get_defects(msg)[1], + errors.MultipartInvariantViolationDefect)) + + def test_lying_multipart_raise_on_defect(self): + with self.assertRaises(errors.NoBoundaryInMultipartDefect): + self._str_msg(self.lying_multipart_msg, + policy=self.policy.clone(raise_on_defect=True)) + + missing_start_boundary_msg = textwrap.dedent("""\ + Content-Type: multipart/mixed; boundary="AAA" + From: Mail Delivery Subsystem + To: yyy at example.com + + --AAA + + Stuff + + --AAA + Content-Type: message/rfc822 + + From: webmaster at python.org + To: zzz at example.com + Content-Type: multipart/mixed; boundary="BBB" + + --BBB-- + + --AAA-- + + """) + + def test_missing_start_boundary(self): + # The message structure is: + # + # multipart/mixed + # text/plain + # message/rfc822 + # multipart/mixed [*] + # + # [*] This message is missing its start boundary + outer = self._str_msg(self.missing_start_boundary_msg) + bad = outer.get_payload(1).get_payload(0) + self.assertEqual(len(self.get_defects(bad)), 1) + self.assertTrue(isinstance(self.get_defects(bad)[0], + errors.StartBoundaryNotFoundDefect)) + + def test_missing_start_boundary_raise_on_defect(self): + with self.assertRaises(errors.StartBoundaryNotFoundDefect): + self._str_msg(self.missing_start_boundary_msg, + policy=self.policy.clone(raise_on_defect=True)) + + def test_first_line_is_continuation_header(self): + msg = self._str_msg(' Line 1\nLine 2\nLine 3') + self.assertEqual(msg.keys(), []) + self.assertEqual(msg.get_payload(), 'Line 2\nLine 3') + self.assertEqual(len(self.get_defects(msg)), 1) + self.assertTrue(isinstance(self.get_defects(msg)[0], + errors.FirstHeaderLineIsContinuationDefect)) + self.assertEqual(self.get_defects(msg)[0].line, ' Line 1\n') + + def test_first_line_is_continuation_header_raise_on_defect(self): + with self.assertRaises(errors.FirstHeaderLineIsContinuationDefect): + self._str_msg(' Line 1\nLine 2\nLine 3', + policy=self.policy.clone(raise_on_defect=True)) + + +class TestMessageDefectDetection(TestMessageDefectDetectionBase, TestEmailBase): + + def get_defects(self, obj): + return obj.defects + + +class TestMessageDefectDetectionCapture(TestMessageDefectDetectionBase, + TestEmailBase): + + class CapturePolicy(Compat32): + captured = None + def register_defect(self, obj, defect): + self.captured.append(defect) + + def setUp(self): + self.policy = self.CapturePolicy(captured=list()) + + def get_defects(self, obj): + return self.policy.captured + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_policy.py b/Lib/test/test_email/test_policy.py --- a/Lib/test/test_email/test_policy.py +++ b/Lib/test/test_email/test_policy.py @@ -1,6 +1,10 @@ +import io import types +import textwrap import unittest import email.policy +import email.parser +import email.generator class PolicyAPITests(unittest.TestCase): @@ -11,14 +15,15 @@ policy_defaults = { 'max_line_length': 78, 'linesep': '\n', - 'must_be_7bit': False, + 'cte_type': '8bit', 'raise_on_defect': False, } # For each policy under test, we give here the values of the attributes # that are different from the defaults for that policy. policies = { - email.policy.Policy(): {}, + email.policy.Compat32(): {}, + email.policy.compat32: {}, email.policy.default: {}, email.policy.SMTP: {'linesep': '\r\n'}, email.policy.HTTP: {'linesep': '\r\n', 'max_line_length': None}, @@ -44,6 +49,18 @@ self.assertIn(attr, self.policy_defaults, "{} is not fully tested".format(attr)) + def test_abc(self): + with self.assertRaises(TypeError) as cm: + email.policy.Policy() + msg = str(cm.exception) + abstract_methods = ('fold', + 'fold_binary', + 'header_fetch_parse', + 'header_source_parse', + 'header_store_parse') + for method in abstract_methods: + self.assertIn(method, msg) + def test_policy_is_immutable(self): for policy in self.policies: for attr in self.policy_defaults: @@ -88,7 +105,7 @@ self.defects = [] obj = Dummy() defect = object() - policy = email.policy.Policy() + policy = email.policy.Compat32() policy.register_defect(obj, defect) self.assertEqual(obj.defects, [defect]) defect2 = object() @@ -117,7 +134,7 @@ email.policy.default.handle_defect(foo, defect2) self.assertEqual(foo.defects, [defect1, defect2]) - class MyPolicy(email.policy.Policy): + class MyPolicy(email.policy.Compat32): defects = None def __init__(self, *args, **kw): super().__init__(*args, defects=[], **kw) @@ -146,5 +163,94 @@ # For adding subclassed objects, make sure the usual rules apply (subclass # wins), but that the order still works (right overrides left). + +class TestPolicyPropagation(unittest.TestCase): + + # The abstract methods are used by the parser but not by the wrapper + # functions that call it, so if the exception gets raised we know that the + # policy was actually propagated all the way to feedparser. + class MyPolicy(email.policy.Policy): + def badmethod(self, *args, **kw): + raise Exception("test") + fold = fold_binary = header_fetch_parser = badmethod + header_source_parse = header_store_parse = badmethod + + def test_message_from_string(self): + with self.assertRaisesRegex(Exception, "^test$"): + email.message_from_string("Subject: test\n\n", + policy=self.MyPolicy) + + def test_message_from_bytes(self): + with self.assertRaisesRegex(Exception, "^test$"): + email.message_from_bytes(b"Subject: test\n\n", + policy=self.MyPolicy) + + def test_message_from_file(self): + f = io.StringIO('Subject: test\n\n') + with self.assertRaisesRegex(Exception, "^test$"): + email.message_from_file(f, policy=self.MyPolicy) + + def test_message_from_binary_file(self): + f = io.BytesIO(b'Subject: test\n\n') + with self.assertRaisesRegex(Exception, "^test$"): + email.message_from_binary_file(f, policy=self.MyPolicy) + + # These are redundant, but we need them for black-box completeness. + + def test_parser(self): + p = email.parser.Parser(policy=self.MyPolicy) + with self.assertRaisesRegex(Exception, "^test$"): + p.parsestr('Subject: test\n\n') + + def test_bytes_parser(self): + p = email.parser.BytesParser(policy=self.MyPolicy) + with self.assertRaisesRegex(Exception, "^test$"): + p.parsebytes(b'Subject: test\n\n') + + # Now that we've established that all the parse methods get the + # policy in to feedparser, we can use message_from_string for + # the rest of the propagation tests. + + def _make_msg(self, source='Subject: test\n\n', policy=None): + self.policy = email.policy.default.clone() if policy is None else policy + return email.message_from_string(source, policy=self.policy) + + def test_parser_propagates_policy_to_message(self): + msg = self._make_msg() + self.assertIs(msg.policy, self.policy) + + def test_parser_propagates_policy_to_sub_messages(self): + msg = self._make_msg(textwrap.dedent("""\ + Subject: mime test + MIME-Version: 1.0 + Content-Type: multipart/mixed, boundary="XXX" + + --XXX + Content-Type: text/plain + + test + --XXX + Content-Type: text/plain + + test2 + --XXX-- + """)) + for part in msg.walk(): + self.assertIs(part.policy, self.policy) + + def test_message_policy_propagates_to_generator(self): + msg = self._make_msg("Subject: test\nTo: foo\n\n", + policy=email.policy.default.clone(linesep='X')) + s = io.StringIO() + g = email.generator.Generator(s) + g.flatten(msg) + self.assertEqual(s.getvalue(), "Subject: testXTo: fooXX") + + def test_message_policy_used_by_as_string(self): + msg = self._make_msg("Subject: test\nTo: foo\n\n", + policy=email.policy.default.clone(linesep='X')) + self.assertEqual(msg.as_string(), "Subject: testXTo: fooXX") + + if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 00:00:08 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 00:00:08 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314731=3A_add_preliminary?= =?utf8?q?_What=27s_New_entry_for_policy_framework=2E?= Message-ID: http://hg.python.org/cpython/rev/8ba99b810b40 changeset: 77147:8ba99b810b40 user: R David Murray date: Fri May 25 17:59:56 2012 -0400 summary: #14731: add preliminary What's New entry for policy framework. files: Doc/whatsnew/3.3.rst | 55 ++++++++++++++++++++++++++++++++ 1 files changed, 55 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -555,6 +555,61 @@ in the `Porting Python code`_ section of this document. +New Email Package Features +========================== + +The email package now has a :mod:`~email.policy` framework. A +:class:`~email.policy.Policy` is an object with several methods and properties +that control how the email package behaves. The primary policy for Python 3.3 +is the :class:`~email.policy.Compat32` policy, which provides backward +compatibility with the email package in Python 3.2. A ``policy`` can be +specified when an email message is parsed by a :mod:`~email.parser`, or when a +:class:`~email.message.Message` object is created, or when an email is +serialized using a :mod:`~email.generator`. Unless overridden, a policy passed +to a ``parser`` is inherited by all the ``Message`` object and sub-objects +created by the ``parser``. By default a ``generator`` will use the policy of +the ``Message`` object it is serializing. The default policy is +:data:`~email.policy.compat32`. + +The minimum set of controls implemented by all ``policy`` objects are: + + =============== ======================================================= + max_line_length The maximum length, excluding the linesep character(s), + individual lines may have when a ``Message`` is + serialized. Defaults to 78. + + linesep The character used to separate individual lines when a + ``Message`` is serialized. Defaults to ``\n``. + + cte_type ``7bit`` or ``8bit``. ``8bit`` applies only to a + ``Bytes`` ``generator``, and means that non-ASCII may + be used where allowed by the protocol (or where it + exists in the original input). + + raise_on_defect Causes a ``parser`` to raise error when defects are + encountered instead of adding them to the ``Message`` + object's ``defects`` list. + =============== ======================================================= + +A new policy instance, with new settings, is created using the +:meth:`~email.policy.Policy.clone` method of policy objects. ``clone`` takes +any of the above controls as keyword arguments. Any control not specified in +the call retains its default value. Thus you can create a policy that uses +``\r\n`` linesep characters like this:: + + mypolicy = compat32.clone(linesep=`\r\n`) + +Policies can be used to make the generation of messages in the format needed by +your application simpler. Instead of having to remember to specify +``linesep='\r\n'`` in all the places you call a ``generator``, you can specify +it once, when you set the policy used by the ``parser`` or the ``Message``, +whichever your program uses to create ``Message`` objects. On the other hand, +if you need to generate messages in multiple forms, you can still specify the +parameters in the appropriate ``generator`` call. Or you can have custom +policy instances for your different cases, and pass those in when you create +the ``generator``. + + Other Language Changes ====================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 00:44:00 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 00:44:00 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2312586=3A_add_provisional?= =?utf8?q?_email_policy_with_new_header_parsing_and_folding=2E?= Message-ID: http://hg.python.org/cpython/rev/0189b9d2d6bc changeset: 77148:0189b9d2d6bc user: R David Murray date: Fri May 25 18:42:14 2012 -0400 summary: #12586: add provisional email policy with new header parsing and folding. When the new policies are used (and only when the new policies are explicitly used) headers turn into objects that have attributes based on their parsed values, and can be set using objects that encapsulate the values, as well as set directly from unicode strings. The folding algorithm then takes care of encoding unicode where needed, and folding according to the highest level syntactic objects. With this patch only date and time headers are parsed as anything other than unstructured, but that is all the helper methods in the existing API handle. I do plan to add more parsers, and complete the set specified in the RFC before the package becomes stable. files: Doc/library/email.policy.rst | 323 + Lib/email/_encoded_words.py | 211 + Lib/email/_header_value_parser.py | 2145 ++++++++ Lib/email/_headerregistry.py | 456 + Lib/email/_policybase.py | 12 +- Lib/email/errors.py | 43 +- Lib/email/generator.py | 11 +- Lib/email/policy.py | 173 +- Lib/email/utils.py | 7 + Lib/test/test_email/__init__.py | 6 + Lib/test/test_email/test__encoded_words.py | 187 + Lib/test/test_email/test__header_value_parser.py | 2466 ++++++++++ Lib/test/test_email/test__headerregistry.py | 717 ++ Lib/test/test_email/test_generator.py | 170 +- Lib/test/test_email/test_pickleable.py | 57 + Lib/test/test_email/test_policy.py | 126 +- 16 files changed, 6994 insertions(+), 116 deletions(-) diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -306,3 +306,326 @@ ``7bit``, non-ascii binary data is CTE encoded using the ``unknown-8bit`` charset. Otherwise the original source header is used, with its existing line breaks and and any (RFC invalid) binary data it may contain. + + +.. note:: + + The remainder of the classes documented below are included in the standard + library on a :term:`provisional basis `. Backwards + incompatible changes (up to and including removal of the feature) may occur + if deemed necessary by the core developers. + + +.. class:: EmailPolicy(**kw) + + This concrete :class:`Policy` provides behavior that is intended to be fully + compliant with the current email RFCs. These include (but are not limited + to) :rfc:`5322`, :rfc:`2047`, and the current MIME RFCs. + + This policy adds new header parsing and folding algorithms. Instead of + simple strings, headers are custom objects with custom attributes depending + on the type of the field. The parsing and folding algorithm fully implement + :rfc:`2047` and :rfc:`5322`. + + In addition to the settable attributes listed above that apply to all + policies, this policy adds the following additional attributes: + + .. attribute:: refold_source + + If the value for a header in the ``Message`` object originated from a + :mod:`~email.parser` (as opposed to being set by a program), this + attribute indicates whether or not a generator should refold that value + when transforming the message back into stream form. The possible values + are: + + ======== =============================================================== + ``none`` all source values use original folding + + ``long`` source values that have any line that is longer than + ``max_line_length`` will be refolded + + ``all`` all values are refolded. + ======== =============================================================== + + The default is ``long``. + + .. attribute:: header_factory + + A callable that takes two arguments, ``name`` and ``value``, where + ``name`` is a header field name and ``value`` is an unfolded header field + value, and returns a string-like object that represents that header. A + default ``header_factory`` is provided that understands some of the + :RFC:`5322` header field types. (Currently address fields and date + fields have special treatment, while all other fields are treated as + unstructured. This list will be completed before the extension is marked + stable.) + + The class provides the following concrete implementations of the abstract + methods of :class:`Policy`: + + .. method:: header_source_parse(sourcelines) + + The implementation of this method is the same as that for the + :class:`Compat32` policy. + + .. method:: header_store_parse(name, value) + + The name is returned unchanged. If the input value has a ``name`` + attribute and it matches *name* ignoring case, the value is returned + unchanged. Otherwise the *name* and *value* are passed to + ``header_factory``, and the resulting custom header object is returned as + the value. In this case a ``ValueError`` is raised if the input value + contains CR or LF characters. + + .. method:: header_fetch_parse(name, value) + + If the value has a ``name`` attribute, it is returned to unmodified. + Otherwise the *name*, and the *value* with any CR or LF characters + removed, are passed to the ``header_factory``, and the resulting custom + header object is returned. Any surrogateescaped bytes get turned into + the unicode unknown-character glyph. + + .. method:: fold(name, value) + + Header folding is controlled by the :attr:`refold_source` policy setting. + A value is considered to be a 'source value' if and only if it does not + have a ``name`` attribute (having a ``name`` attribute means it is a + header object of some sort). If a source value needs to be refolded + according to the policy, it is converted into a custom header object by + passing the *name* and the *value* with any CR and LF characters removed + to the ``header_factory``. Folding of a custom header object is done by + calling its ``fold`` method with the current policy. + + Source values are split into lines using :meth:`~str.splitlines`. If + the value is not to be refolded, the lines are rejoined using the + ``linesep`` from the policy and returned. The exception is lines + containing non-ascii binary data. In that case the value is refolded + regardless of the ``refold_source`` setting, which causes the binary data + to be CTE encoded using the ``unknown-8bit`` charset. + + .. method:: fold_binary(name, value) + + The same as :meth:`fold` if :attr:`cte_type` is ``7bit``, except that + the returned value is bytes. + + If :attr:`cte_type` is ``8bit``, non-ASCII binary data is converted back + into bytes. Headers with binary data are not refolded, regardless of the + ``refold_header`` setting, since there is no way to know whether the + binary data consists of single byte characters or multibyte characters. + +The following instances of :class:`EmailPolicy` provide defaults suitable for +specific application domains. Note that in the future the behavior of these +instances (in particular the ``HTTP` instance) may be adjusted to conform even +more closely to the RFCs relevant to their domains. + +.. data:: default + + An instance of ``EmailPolicy`` with all defaults unchanged. This policy + uses the standard Python ``\n`` line endings rather than the RFC-correct + ``\r\n``. + +.. data:: SMTP + + Suitable for serializing messages in conformance with the email RFCs. + Like ``default``, but with ``linesep`` set to ``\r\n``, which is RFC + compliant. + +.. data:: HTTP + + Suitable for serializing headers with for use in HTTP traffic. Like + ``SMTP`` except that ``max_line_length`` is set to ``None`` (unlimited). + +.. data:: strict + + Convenience instance. The same as ``default`` except that + ``raise_on_defect`` is set to ``True``. This allows any policy to be made + strict by writing:: + + somepolicy + policy.strict + +With all of these :class:`EmailPolicies <.EmailPolicy>`, the effective API of +the email package is changed from the Python 3.2 API in the following ways: + + * Setting a header on a :class:`~email.message.Message` results in that + header being parsed and a custom header object created. + + * Fetching a header value from a :class:`~email.message.Message` results + in that header being parsed and a custom header object created and + returned. + + * Any custom header object, or any header that is refolded due to the + policy settings, is folded using an algorithm that fully implements the + RFC folding algorithms, including knowing where encoded words are required + and allowed. + +From the application view, this means that any header obtained through the +:class:`~email.message.Message` is a custom header object with custom +attributes, whose string value is the fully decoded unicode value of the +header. Likewise, a header may be assigned a new value, or a new header +created, using a unicode string, and the policy will take care of converting +the unicode string into the correct RFC encoded form. + +The custom header objects and their attributes are described below. All custom +header objects are string subclasses, and their string value is the fully +decoded value of the header field (the part of the field after the ``:``) + + +.. class:: BaseHeader + + This is the base class for all custom header objects. It provides the + following attributes: + + .. attribute:: name + + The header field name (the portion of the field before the ':'). + + .. attribute:: defects + + A possibly empty list of :class:`~email.errors.MessageDefect` objects + that record any RFC violations found while parsing the header field. + + .. method:: fold(*, policy) + + Return a string containing :attr:`~email.policy.Policy.linesep` + characters as required to correctly fold the header according + to *policy*. A :attr:`~email.policy.Policy.cte_type` of + ``8bit`` will be treated as if it were ``7bit``, since strings + may not contain binary data. + + +.. class:: UnstructuredHeader + + The class used for any header that does not have a more specific + type. (The :mailheader:`Subject` header is an example of an + unstructured header.) It does not have any additional attributes. + + +.. class:: DateHeader + + The value of this type of header is a single date and time value. The + primary example of this type of header is the :mailheader:`Date` header. + + .. attribute:: datetime + + A :class:`~datetime.datetime` encoding the date and time from the + header value. + + The ``datetime`` will be a naive ``datetime`` if the value either does + not have a specified timezone (which would be a violation of the RFC) or + if the timezone is specified as ``-0000``. This timezone value indicates + that the date and time is to be considered to be in UTC, but with no + indication of the local timezone in which it was generated. (This + contrasts to ``+0000``, which indicates a date and time that really is in + the UTC ``0000`` timezone.) + + If the header value contains a valid timezone that is not ``-0000``, the + ``datetime`` will be an aware ``datetime`` having a + :class:`~datetime.tzinfo` set to the :class:`~datetime.timezone` + indicated by the header value. + + A ``datetime`` may also be assigned to a :mailheader:`Date` type header. + The resulting string value will use a timezone of ``-0000`` if the + ``datetime`` is naive, and the appropriate UTC offset if the ``datetime`` is + aware. + + +.. class:: AddressHeader + + This class is used for all headers that can contain addresses, whether they + are supposed to be singleton addresses or a list. + + .. attribute:: addresses + + A list of :class:`.Address` objects listing all of the addresses that + could be parsed out of the field value. + + .. attribute:: groups + + A list of :class:`.Group` objects. Every address in :attr:`.addresses` + appears in one of the group objects in the tuple. Addresses that are not + syntactically part of a group are represented by ``Group`` objects whose + ``name`` is ``None``. + + In addition to addresses in string form, any combination of + :class:`.Address` and :class:`.Group` objects, singly or in a list, may be + assigned to an address header. + + +.. class:: Address(display_name='', username='', domain='', addr_spec=None): + + The class used to represent an email address. The general form of an + address is:: + + [display_name] + + or:: + + username at domain + + where each part must conform to specific syntax rules spelled out in + :rfc:`5322`. + + As a convenience *addr_spec* can be specified instead of *username* and + *domain*, in which case *username* and *domain* will be parsed from the + *addr_spec*. An *addr_spec* must be a properly RFC quoted string; if it is + not ``Address`` will raise an error. Unicode characters are allowed and + will be property encoded when serialized. However, per the RFCs, unicode is + *not* allowed in the username portion of the address. + + .. attribute:: display_name + + The display name portion of the address, if any, with all quoting + removed. If the address does not have a display name, this attribute + will be an empty string. + + .. attribute:: username + + The ``username`` portion of the address, with all quoting removed. + + .. attribute:: domain + + The ``domain`` portion of the address. + + .. attribute:: addr_spec + + The ``username at domain`` portion of the address, correctly quoted + for use as a bare address (the second form shown above). This + attribute is not mutable. + + .. method:: __str__() + + The ``str`` value of the object is the address quoted according to + :rfc:`5322` rules, but with no Content Transfer Encoding of any non-ASCII + characters. + + +.. class:: Group(display_name=None, addresses=None) + + The class used to represent an address group. The general form of an + address group is:: + + display_name: [address-list]; + + As a convenience for processing lists of addresses that consist of a mixture + of groups and single addresses, a ``Group`` may also be used to represent + single addresses that are not part of a group by setting *display_name* to + ``None`` and providing a list of the single address as *addresses*. + + .. attribute:: display_name + + The ``display_name`` of the group. If it is ``None`` and there is + exactly one ``Address`` in ``addresses``, then the ``Group`` represents a + single address that is not in a group. + + .. attribute:: addresses + + A possibly empty tuple of :class:`.Address` objects representing the + addresses in the group. + + .. method:: __str__() + + The ``str`` value of a ``Group`` is formatted according to :rfc:`5322`, + but with no Content Transfer Encoding of any non-ASCII characters. If + ``display_name`` is none and there is a single ``Address`` in the + ``addresses` list, the ``str`` value will be the same as the ``str`` of + that single ``Address``. diff --git a/Lib/email/_encoded_words.py b/Lib/email/_encoded_words.py new file mode 100644 --- /dev/null +++ b/Lib/email/_encoded_words.py @@ -0,0 +1,211 @@ +""" Routines for manipulating RFC2047 encoded words. + +This is currently a package-private API, but will be considered for promotion +to a public API if there is demand. + +""" + +# An ecoded word looks like this: +# +# =?charset[*lang]?cte?encoded_string?= +# +# for more information about charset see the charset module. Here it is one +# of the preferred MIME charset names (hopefully; you never know when parsing). +# cte (Content Transfer Encoding) is either 'q' or 'b' (ignoring case). In +# theory other letters could be used for other encodings, but in practice this +# (almost?) never happens. There could be a public API for adding entries +# to to the CTE tables, but YAGNI for now. 'q' is Quoted Printable, 'b' is +# Base64. The meaning of encoded_string should be obvious. 'lang' is optional +# as indicated by the brackets (they are not part of the syntax) but is almost +# never encountered in practice. +# +# The general interface for a CTE decoder is that it takes the encoded_string +# as its argument, and returns a tuple (cte_decoded_string, defects). The +# cte_decoded_string is the original binary that was encoded using the +# specified cte. 'defects' is a list of MessageDefect instances indicating any +# problems encountered during conversion. 'charset' and 'lang' are the +# corresponding strings extracted from the EW, case preserved. +# +# The general interface for a CTE encoder is that it takes a binary sequence +# as input and returns the cte_encoded_string, which is an ascii-only string. +# +# Each decoder must also supply a length function that takes the binary +# sequence as its argument and returns the length of the resulting encoded +# string. +# +# The main API functions for the module are decode, which calls the decoder +# referenced by the cte specifier, and encode, which adds the appropriate +# RFC 2047 "chrome" to the encoded string, and can optionally automatically +# select the shortest possible encoding. See their docstrings below for +# details. + +import re +import base64 +import binascii +import functools +from string import ascii_letters, digits +from email import errors + +# +# Quoted Printable +# + +# regex based decoder. +_q_byte_subber = functools.partial(re.compile(br'=([a-fA-F0-9]{2})').sub, + lambda m: bytes([int(m.group(1), 16)])) + +def decode_q(encoded): + encoded = encoded.replace(b'_', b' ') + return _q_byte_subber(encoded), [] + + +# dict mapping bytes to their encoded form +class QByteMap(dict): + + safe = b'-!*+/' + ascii_letters.encode('ascii') + digits.encode('ascii') + + def __missing__(self, key): + if key in self.safe: + self[key] = chr(key) + else: + self[key] = "={:02X}".format(key) + return self[key] + +_q_byte_map = QByteMap() + +# In headers spaces are mapped to '_'. +_q_byte_map[ord(' ')] = '_' + +def encode_q(bstring): + return ''.join(_q_byte_map[x] for x in bstring) + +def len_q(bstring): + return sum(len(_q_byte_map[x]) for x in bstring) + + +# +# Base64 +# + +def decode_b(encoded): + defects = [] + pad_err = len(encoded) % 4 + if pad_err: + defects.append(errors.InvalidBase64PaddingDefect()) + padded_encoded = encoded + b'==='[:4-pad_err] + else: + padded_encoded = encoded + try: + return base64.b64decode(padded_encoded, validate=True), defects + except binascii.Error: + # Since we had correct padding, this must an invalid char error. + defects = [errors.InvalidBase64CharactersDefect()] + # The non-alphabet characters are ignored as far as padding + # goes, but we don't know how many there are. So we'll just + # try various padding lengths until something works. + for i in 0, 1, 2, 3: + try: + return base64.b64decode(encoded+b'='*i, validate=False), defects + except binascii.Error: + if i==0: + defects.append(errors.InvalidBase64PaddingDefect()) + else: + # This should never happen. + raise AssertionError("unexpected binascii.Error") + +def encode_b(bstring): + return base64.b64encode(bstring).decode('ascii') + +def len_b(bstring): + groups_of_3, leftover = divmod(len(bstring), 3) + # 4 bytes out for each 3 bytes (or nonzero fraction thereof) in. + return groups_of_3 * 4 + (4 if leftover else 0) + + +_cte_decoders = { + 'q': decode_q, + 'b': decode_b, + } + +def decode(ew): + """Decode encoded word and return (string, charset, lang, defects) tuple. + + An RFC 2047/2243 encoded word has the form: + + =?charset*lang?cte?encoded_string?= + + where '*lang' may be omitted but the other parts may not be. + + This function expects exactly such a string (that is, it does not check the + syntax and may raise errors if the string is not well formed), and returns + the encoded_string decoded first from its Content Transfer Encoding and + then from the resulting bytes into unicode using the specified charset. If + the cte-decoded string does not successfully decode using the specified + character set, a defect is added to the defects list and the unknown octets + are replaced by the unicode 'unknown' character \uFDFF. + + The specified charset and language are returned. The default for language, + which is rarely if ever encountered, is the empty string. + + """ + _, charset, cte, cte_string, _ = ew.split('?') + charset, _, lang = charset.partition('*') + cte = cte.lower() + # Recover the original bytes and do CTE decoding. + bstring = cte_string.encode('ascii', 'surrogateescape') + bstring, defects = _cte_decoders[cte](bstring) + # Turn the CTE decoded bytes into unicode. + try: + string = bstring.decode(charset) + except UnicodeError: + defects.append(errors.UndecodableBytesDefect("Encoded word " + "contains bytes not decodable using {} charset".format(charset))) + string = bstring.decode(charset, 'surrogateescape') + except LookupError: + string = bstring.decode('ascii', 'surrogateescape') + if charset.lower() != 'unknown-8bit': + defects.append(errors.CharsetError("Unknown charset {} " + "in encoded word; decoded as unknown bytes".format(charset))) + return string, charset, lang, defects + + +_cte_encoders = { + 'q': encode_q, + 'b': encode_b, + } + +_cte_encode_length = { + 'q': len_q, + 'b': len_b, + } + +def encode(string, charset='utf-8', encoding=None, lang=''): + """Encode string using the CTE encoding that produces the shorter result. + + Produces an RFC 2047/2243 encoded word of the form: + + =?charset*lang?cte?encoded_string?= + + where '*lang' is omitted unless the 'lang' parameter is given a value. + Optional argument charset (defaults to utf-8) specifies the charset to use + to encode the string to binary before CTE encoding it. Optional argument + 'encoding' is the cte specifier for the encoding that should be used ('q' + or 'b'); if it is None (the default) the encoding which produces the + shortest encoded sequence is used, except that 'q' is preferred if it is up + to five characters longer. Optional argument 'lang' (default '') gives the + RFC 2243 language string to specify in the encoded word. + + """ + if charset == 'unknown-8bit': + bstring = string.encode('ascii', 'surrogateescape') + else: + bstring = string.encode(charset) + if encoding is None: + qlen = _cte_encode_length['q'](bstring) + blen = _cte_encode_length['b'](bstring) + # Bias toward q. 5 is arbitrary. + encoding = 'q' if qlen - blen < 5 else 'b' + encoded = _cte_encoders[encoding](bstring) + if lang: + lang = '*' + lang + return "=?{}{}?{}?{}?=".format(charset, lang, encoding, encoded) diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py new file mode 100644 --- /dev/null +++ b/Lib/email/_header_value_parser.py @@ -0,0 +1,2145 @@ +"""Header value parser implementing various email-related RFC parsing rules. + +The parsing methods defined in this module implement various email related +parsing rules. Principal among them is RFC 5322, which is the followon +to RFC 2822 and primarily a clarification of the former. It also implements +RFC 2047 encoded word decoding. + +RFC 5322 goes to considerable trouble to maintain backward compatibility with +RFC 822 in the parse phase, while cleaning up the structure on the generation +phase. This parser supports correct RFC 5322 generation by tagging white space +as folding white space only when folding is allowed in the non-obsolete rule +sets. Actually, the parser is even more generous when accepting input than RFC +5322 mandates, following the spirit of Postel's Law, which RFC 5322 encourages. +Where possible deviations from the standard are annotated on the 'defects' +attribute of tokens that deviate. + +The general structure of the parser follows RFC 5322, and uses its terminology +where there is a direct correspondence. Where the implementation requires a +somewhat different structure than that used by the formal grammar, new terms +that mimic the closest existing terms are used. Thus, it really helps to have +a copy of RFC 5322 handy when studying this code. + +Input to the parser is a string that has already been unfolded according to +RFC 5322 rules. According to the RFC this unfolding is the very first step, and +this parser leaves the unfolding step to a higher level message parser, which +will have already detected the line breaks that need unfolding while +determining the beginning and end of each header. + +The output of the parser is a TokenList object, which is a list subclass. A +TokenList is a recursive data structure. The terminal nodes of the structure +are Terminal objects, which are subclasses of str. These do not correspond +directly to terminal objects in the formal grammar, but are instead more +practical higher level combinations of true terminals. + +All TokenList and Terminal objects have a 'value' attribute, which produces the +semantically meaningful value of that part of the parse subtree. The value of +all whitespace tokens (no matter how many sub-tokens they may contain) is a +single space, as per the RFC rules. This includes 'CFWS', which is herein +included in the general class of whitespace tokens. There is one exception to +the rule that whitespace tokens are collapsed into single spaces in values: in +the value of a 'bare-quoted-string' (a quoted-string with no leading or +trailing whitespace), any whitespace that appeared between the quotation marks +is preserved in the returned value. Note that in all Terminal strings quoted +pairs are turned into their unquoted values. + +All TokenList and Terminal objects also have a string value, which attempts to +be a "canonical" representation of the RFC-compliant form of the substring that +produced the parsed subtree, including minimal use of quoted pair quoting. +Whitespace runs are not collapsed. + +Comment tokens also have a 'content' attribute providing the string found +between the parens (including any nested comments) with whitespace preserved. + +All TokenList and Terminal objects have a 'defects' attribute which is a +possibly empty list all of the defects found while creating the token. Defects +may appear on any token in the tree, and a composite list of all defects in the +subtree is available through the 'all_defects' attribute of any node. (For +Terminal notes x.defects == x.all_defects.) + +Each object in a parse tree is called a 'token', and each has a 'token_type' +attribute that gives the name from the RFC 5322 grammar that it represents. +Not all RFC 5322 nodes are produced, and there is one non-RFC 5322 node that +may be produced: 'ptext'. A 'ptext' is a string of printable ascii characters. +It is returned in place of lists of (ctext/quoted-pair) and +(qtext/quoted-pair). + +XXX: provide complete list of token types. +""" + +import re +from email import _encoded_words as _ew +from email import errors +from email import utils + +# +# Useful constants and functions +# + +WSP = set(' \t') +CFWS_LEADER = WSP | set('(') +SPECIALS = set(r'()<>@,:;.\"[]') +ATOM_ENDS = SPECIALS | WSP +DOT_ATOM_ENDS = ATOM_ENDS - set('.') +# '.', '"', and '(' do not end phrases in order to support obs-phrase +PHRASE_ENDS = SPECIALS - set('."(') + +def quote_string(value): + return '"'+str(value).replace('\\', '\\\\').replace('"', r'\"')+'"' + +# +# Accumulator for header folding +# + +class _Folded: + + def __init__(self, maxlen, policy): + self.maxlen = maxlen + self.policy = policy + self.lastlen = 0 + self.stickyspace = None + self.firstline = True + self.done = [] + self.current = [] + + def newline(self): + self.done.extend(self.current) + self.done.append(self.policy.linesep) + self.current.clear() + self.lastlen = 0 + + def finalize(self): + if self.current: + self.newline() + + def __str__(self): + return ''.join(self.done) + + def append(self, stoken): + self.current.append(stoken) + + def append_if_fits(self, token, stoken=None): + if stoken is None: + stoken = str(token) + l = len(stoken) + if self.stickyspace is not None: + stickyspace_len = len(self.stickyspace) + if self.lastlen + stickyspace_len + l <= self.maxlen: + self.current.append(self.stickyspace) + self.lastlen += stickyspace_len + self.current.append(stoken) + self.lastlen += l + self.stickyspace = None + self.firstline = False + return True + if token.has_fws: + ws = token.pop_leading_fws() + if ws is not None: + self.stickyspace += str(ws) + stickyspace_len += len(ws) + token._fold(self) + return True + if stickyspace_len and l + 1 <= self.maxlen: + margin = self.maxlen - l + if 0 < margin < stickyspace_len: + trim = stickyspace_len - margin + self.current.append(self.stickyspace[:trim]) + self.stickyspace = self.stickyspace[trim:] + stickyspace_len = trim + self.newline() + self.current.append(self.stickyspace) + self.current.append(stoken) + self.lastlen = l + stickyspace_len + self.stickyspace = None + self.firstline = False + return True + if not self.firstline: + self.newline() + self.current.append(self.stickyspace) + self.current.append(stoken) + self.stickyspace = None + self.firstline = False + return True + if self.lastlen + l <= self.maxlen: + self.current.append(stoken) + self.lastlen += l + return True + if l < self.maxlen: + self.newline() + self.current.append(stoken) + self.lastlen = l + return True + return False + +# +# TokenList and its subclasses +# + +class TokenList(list): + + token_type = None + + def __init__(self, *args, **kw): + super().__init__(*args, **kw) + self.defects = [] + + def __str__(self): + return ''.join(str(x) for x in self) + + def __repr__(self): + return '{}({})'.format(self.__class__.__name__, + super().__repr__()) + + @property + def value(self): + return ''.join(x.value for x in self if x.value) + + @property + def all_defects(self): + return sum((x.all_defects for x in self), self.defects) + + # + # Folding API + # + # parts(): + # + # return a list of objects that constitute the "higher level syntactic + # objects" specified by the RFC as the best places to fold a header line. + # The returned objects must include leading folding white space, even if + # this means mutating the underlying parse tree of the object. Each object + # is only responsible for returning *its* parts, and should not drill down + # to any lower level except as required to meet the leading folding white + # space constraint. + # + # _fold(folded): + # + # folded: the result accumulator. This is an instance of _Folded. + # (XXX: I haven't finished factoring this out yet, the folding code + # pretty much uses this as a state object.) When the folded.current + # contains as much text as will fit, the _fold method should call + # folded.newline. + # folded.lastlen: the current length of the test stored in folded.current. + # folded.maxlen: The maximum number of characters that may appear on a + # folded line. Differs from the policy setting in that "no limit" is + # represented by +inf, which means it can be used in the trivially + # logical fashion in comparisons. + # + # Currently no subclasses implement parts, and I think this will remain + # true. A subclass only needs to implement _fold when the generic version + # isn't sufficient. _fold will need to be implemented primarily when it is + # possible for encoded words to appear in the specialized token-list, since + # there is no generic algorithm that can know where exactly the encoded + # words are allowed. A _fold implementation is responsible for filling + # lines in the same general way that the top level _fold does. It may, and + # should, call the _fold method of sub-objects in a similar fashion to that + # of the top level _fold. + # + # XXX: I'm hoping it will be possible to factor the existing code further + # to reduce redundancy and make the logic clearer. + + @property + def parts(self): + klass = self.__class__ + this = [] + for token in self: + if token.startswith_fws(): + if this: + yield this[0] if len(this)==1 else klass(this) + this.clear() + end_ws = token.pop_trailing_ws() + this.append(token) + if end_ws: + yield klass(this) + this = [end_ws] + if this: + yield this[0] if len(this)==1 else klass(this) + + def startswith_fws(self): + return self[0].startswith_fws() + + def pop_leading_fws(self): + if self[0].token_type == 'fws': + return self.pop(0) + return self[0].pop_leading_fws() + + def pop_trailing_ws(self): + if self[-1].token_type == 'cfws': + return self.pop(-1) + return self[-1].pop_trailing_ws() + + @property + def has_fws(self): + for part in self: + if part.has_fws: + return True + return False + + def has_leading_comment(self): + return self[0].has_leading_comment() + + @property + def comments(self): + comments = [] + for token in self: + comments.extend(token.comments) + return comments + + def fold(self, *, policy): + # max_line_length 0/None means no limit, ie: infinitely long. + maxlen = policy.max_line_length or float("+inf") + folded = _Folded(maxlen, policy) + self._fold(folded) + folded.finalize() + return str(folded) + + def as_encoded_word(self, charset): + # This works only for things returned by 'parts', which include + # the leading fws, if any, that should be used. + res = [] + ws = self.pop_leading_fws() + if ws: + res.append(ws) + trailer = self.pop(-1) if self[-1].token_type=='fws' else '' + res.append(_ew.encode(str(self), charset)) + res.append(trailer) + return ''.join(res) + + def cte_encode(self, charset, policy): + res = [] + for part in self: + res.append(part.cte_encode(charset, policy)) + return ''.join(res) + + def _fold(self, folded): + for part in self.parts: + tstr = str(part) + tlen = len(tstr) + try: + str(part).encode('us-ascii') + except UnicodeEncodeError: + if any(isinstance(x, errors.UndecodableBytesDefect) + for x in part.all_defects): + charset = 'unknown-8bit' + else: + # XXX: this should be a policy setting + charset = 'utf-8' + tstr = part.cte_encode(charset, folded.policy) + tlen = len(tstr) + if folded.append_if_fits(part, tstr): + continue + # Peel off the leading whitespace if any and make it sticky, to + # avoid infinite recursion. + ws = part.pop_leading_fws() + if ws is not None: + # Peel off the leading whitespace and make it sticky, to + # avoid infinite recursion. + folded.stickyspace = str(part.pop(0)) + if folded.append_if_fits(part): + continue + if part.has_fws: + part._fold(folded) + continue + # There are no fold points in this one; it is too long for a single + # line and can't be split...we just have to put it on its own line. + folded.append(tstr) + folded.newline() + + def pprint(self, indent=''): + print('\n'.join(self._pp(indent=''))) + + def ppstr(self, indent=''): + return '\n'.join(self._pp(indent='')) + + def _pp(self, indent=''): + yield '{}{}/{}('.format( + indent, + self.__class__.__name__, + self.token_type) + for token in self: + for line in token._pp(indent+' '): + yield line + if self.defects: + extra = ' Defects: {}'.format(self.defects) + else: + extra = '' + yield '{}){}'.format(indent, extra) + + +class WhiteSpaceTokenList(TokenList): + + @property + def value(self): + return ' ' + + @property + def comments(self): + return [x.content for x in self if x.token_type=='comment'] + + +class UnstructuredTokenList(TokenList): + + token_type = 'unstructured' + + def _fold(self, folded): + if any(x.token_type=='encoded-word' for x in self): + return self._fold_encoded(folded) + # Here we can have either a pure ASCII string that may or may not + # have surrogateescape encoded bytes, or a unicode string. + last_ew = None + for part in self.parts: + tstr = str(part) + is_ew = False + try: + str(part).encode('us-ascii') + except UnicodeEncodeError: + if any(isinstance(x, errors.UndecodableBytesDefect) + for x in part.all_defects): + charset = 'unknown-8bit' + else: + charset = 'utf-8' + if last_ew is not None: + # We've already done an EW, combine this one with it + # if there's room. + chunk = get_unstructured( + ''.join(folded.current[last_ew:]+[tstr])).as_encoded_word(charset) + oldlastlen = sum(len(x) for x in folded.current[:last_ew]) + schunk = str(chunk) + lchunk = len(schunk) + if oldlastlen + lchunk <= folded.maxlen: + del folded.current[last_ew:] + folded.append(schunk) + folded.lastlen = oldlastlen + lchunk + continue + tstr = part.as_encoded_word(charset) + is_ew = True + if folded.append_if_fits(part, tstr): + if is_ew: + last_ew = len(folded.current) - 1 + continue + if is_ew or last_ew: + # It's too big to fit on the line, but since we've + # got encoded words we can use encoded word folding. + part._fold_as_ew(folded) + continue + # Peel off the leading whitespace if any and make it sticky, to + # avoid infinite recursion. + ws = part.pop_leading_fws() + if ws is not None: + folded.stickyspace = str(ws) + if folded.append_if_fits(part): + continue + if part.has_fws: + part.fold(folded) + continue + # It can't be split...we just have to put it on its own line. + folded.append(tstr) + folded.newline() + last_ew = None + + def cte_encode(self, charset, policy): + res = [] + last_ew = None + for part in self: + spart = str(part) + try: + spart.encode('us-ascii') + res.append(spart) + except UnicodeEncodeError: + if last_ew is None: + res.append(part.cte_encode(charset, policy)) + last_ew = len(res) + else: + tl = get_unstructured(''.join(res[last_ew:] + [spart])) + res.append(tl.as_encoded_word()) + return ''.join(res) + + +class Phrase(TokenList): + + token_type = 'phrase' + + def _fold(self, folded): + # As with Unstructured, we can have pure ASCII with or without + # surrogateescape encoded bytes, or we could have unicode. But this + # case is more complicated, since we have to deal with the various + # sub-token types and how they can be composed in the face of + # unicode-that-needs-CTE-encoding, and the fact that if a token a + # comment that becomes a barrier across which we can't compose encoded + # words. + last_ew = None + for part in self.parts: + tstr = str(part) + tlen = len(tstr) + has_ew = False + try: + str(part).encode('us-ascii') + except UnicodeEncodeError: + if any(isinstance(x, errors.UndecodableBytesDefect) + for x in part.all_defects): + charset = 'unknown-8bit' + else: + charset = 'utf-8' + if last_ew is not None and not part.has_leading_comment(): + # We've already done an EW, let's see if we can combine + # this one with it. The last_ew logic ensures that all we + # have at this point is atoms, no comments or quoted + # strings. So we can treat the text between the last + # encoded word and the content of this token as + # unstructured text, and things will work correctly. But + # we have to strip off any trailing comment on this token + # first, and if it is a quoted string we have to pull out + # the content (we're encoding it, so it no longer needs to + # be quoted). + if part[-1].token_type == 'cfws' and part.comments: + remainder = part.pop(-1) + else: + remainder = '' + for i, token in enumerate(part): + if token.token_type == 'bare-quoted-string': + part[i] = UnstructuredTokenList(token[:]) + chunk = get_unstructured( + ''.join(folded.current[last_ew:]+[tstr])).as_encoded_word(charset) + schunk = str(chunk) + lchunk = len(schunk) + if last_ew + lchunk <= folded.maxlen: + del folded.current[last_ew:] + folded.append(schunk) + folded.lastlen = sum(len(x) for x in folded.current) + continue + tstr = part.as_encoded_word(charset) + tlen = len(tstr) + has_ew = True + if folded.append_if_fits(part, tstr): + if has_ew and not part.comments: + last_ew = len(folded.current) - 1 + elif part.comments or part.token_type == 'quoted-string': + # If a comment is involved we can't combine EWs. And if a + # quoted string is involved, it's not worth the effort to + # try to combine them. + last_ew = None + continue + part._fold(folded) + + def cte_encode(self, charset, policy): + res = [] + last_ew = None + is_ew = False + for part in self: + spart = str(part) + try: + spart.encode('us-ascii') + res.append(spart) + except UnicodeEncodeError: + is_ew = True + if last_ew is None: + if not part.comments: + last_ew = len(res) + res.append(part.cte_encode(charset, policy)) + elif not part.has_leading_comment(): + if part[-1].token_type == 'cfws' and part.comments: + remainder = part.pop(-1) + else: + remainder = '' + for i, token in enumerate(part): + if token.token_type == 'bare-quoted-string': + part[i] = UnstructuredTokenList(token[:]) + tl = get_unstructured(''.join(res[last_ew:] + [spart])) + res[last_ew:] = [tl.as_encoded_word(charset)] + if part.comments or (not is_ew and part.token_type == 'quoted-string'): + last_ew = None + return ''.join(res) + +class Word(TokenList): + + token_type = 'word' + + +class CFWSList(WhiteSpaceTokenList): + + token_type = 'cfws' + + def has_leading_comment(self): + return bool(self.comments) + + +class Atom(TokenList): + + token_type = 'atom' + + +class EncodedWord(TokenList): + + token_type = 'encoded-word' + cte = None + charset = None + lang = None + + @property + def encoded(self): + if self.cte is not None: + return self.cte + _ew.encode(str(self), self.charset) + + + +class QuotedString(TokenList): + + token_type = 'quoted-string' + + @property + def content(self): + for x in self: + if x.token_type == 'bare-quoted-string': + return x.value + + @property + def quoted_value(self): + res = [] + for x in self: + if x.token_type == 'bare-quoted-string': + res.append(str(x)) + else: + res.append(x.value) + return ''.join(res) + + +class BareQuotedString(QuotedString): + + token_type = 'bare-quoted-string' + + def __str__(self): + return quote_string(''.join(self)) + + @property + def value(self): + return ''.join(str(x) for x in self) + + +class Comment(WhiteSpaceTokenList): + + token_type = 'comment' + + def __str__(self): + return ''.join(sum([ + ["("], + [self.quote(x) for x in self], + [")"], + ], [])) + + def quote(self, value): + if value.token_type == 'comment': + return str(value) + return str(value).replace('\\', '\\\\').replace( + '(', '\(').replace( + ')', '\)') + + @property + def content(self): + return ''.join(str(x) for x in self) + + @property + def comments(self): + return [self.content] + +class AddressList(TokenList): + + token_type = 'address-list' + + @property + def addresses(self): + return [x for x in self if x.token_type=='address'] + + @property + def mailboxes(self): + return sum((x.mailboxes + for x in self if x.token_type=='address'), []) + + @property + def all_mailboxes(self): + return sum((x.all_mailboxes + for x in self if x.token_type=='address'), []) + + +class Address(TokenList): + + token_type = 'address' + + @property + def display_name(self): + if self[0].token_type == 'group': + return self[0].display_name + + @property + def mailboxes(self): + if self[0].token_type == 'mailbox': + return [self[0]] + elif self[0].token_type == 'invalid-mailbox': + return [] + return self[0].mailboxes + + @property + def all_mailboxes(self): + if self[0].token_type == 'mailbox': + return [self[0]] + elif self[0].token_type == 'invalid-mailbox': + return [self[0]] + return self[0].all_mailboxes + +class MailboxList(TokenList): + + token_type = 'mailbox-list' + + @property + def mailboxes(self): + return [x for x in self if x.token_type=='mailbox'] + + @property + def all_mailboxes(self): + return [x for x in self + if x.token_type in ('mailbox', 'invalid-mailbox')] + + +class GroupList(TokenList): + + token_type = 'group-list' + + @property + def mailboxes(self): + if not self or self[0].token_type != 'mailbox-list': + return [] + return self[0].mailboxes + + @property + def all_mailboxes(self): + if not self or self[0].token_type != 'mailbox-list': + return [] + return self[0].all_mailboxes + + +class Group(TokenList): + + token_type = "group" + + @property + def mailboxes(self): + if self[2].token_type != 'group-list': + return [] + return self[2].mailboxes + + @property + def all_mailboxes(self): + if self[2].token_type != 'group-list': + return [] + return self[2].all_mailboxes + + @property + def display_name(self): + return self[0].display_name + + +class NameAddr(TokenList): + + token_type = 'name-addr' + + @property + def display_name(self): + if len(self) == 1: + return None + return self[0].display_name + + @property + def local_part(self): + return self[-1].local_part + + @property + def domain(self): + return self[-1].domain + + @property + def route(self): + return self[-1].route + + @property + def addr_spec(self): + return self[-1].addr_spec + + +class AngleAddr(TokenList): + + token_type = 'angle-addr' + + @property + def local_part(self): + for x in self: + if x.token_type == 'addr-spec': + return x.local_part + + @property + def domain(self): + for x in self: + if x.token_type == 'addr-spec': + return x.domain + + @property + def route(self): + for x in self: + if x.token_type == 'obs-route': + return x.domains + + @property + def addr_spec(self): + for x in self: + if x.token_type == 'addr-spec': + return x.addr_spec + + +class ObsRoute(TokenList): + + token_type = 'obs-route' + + @property + def domains(self): + return [x.domain for x in self if x.token_type == 'domain'] + + +class Mailbox(TokenList): + + token_type = 'mailbox' + + @property + def display_name(self): + if self[0].token_type == 'name-addr': + return self[0].display_name + + @property + def local_part(self): + return self[0].local_part + + @property + def domain(self): + return self[0].domain + + @property + def route(self): + if self[0].token_type == 'name-addr': + return self[0].route + + @property + def addr_spec(self): + return self[0].addr_spec + + +class InvalidMailbox(TokenList): + + token_type = 'invalid-mailbox' + + @property + def display_name(self): + return None + + local_part = domain = route = addr_spec = display_name + + +class Domain(TokenList): + + token_type = 'domain' + + @property + def domain(self): + return ''.join(super().value.split()) + + +class DotAtom(TokenList): + + token_type = 'dot-atom' + + +class DotAtomText(TokenList): + + token_type = 'dot-atom-text' + + +class AddrSpec(TokenList): + + token_type = 'addr-spec' + + @property + def local_part(self): + return self[0].local_part + + @property + def domain(self): + if len(self) < 3: + return None + return self[-1].domain + + @property + def value(self): + if len(self) < 3: + return self[0].value + return self[0].value.rstrip()+self[1].value+self[2].value.lstrip() + + @property + def addr_spec(self): + nameset = set(self.local_part) + if len(nameset) > len(nameset-DOT_ATOM_ENDS): + lp = quote_string(self.local_part) + else: + lp = self.local_part + if self.domain is not None: + return lp + '@' + self.domain + return lp + + +class ObsLocalPart(TokenList): + + token_type = 'obs-local-part' + + +class DisplayName(Phrase): + + token_type = 'display-name' + + @property + def display_name(self): + res = TokenList(self) + if res[0].token_type == 'cfws': + res.pop(0) + else: + if res[0][0].token_type == 'cfws': + res[0] = TokenList(res[0][1:]) + if res[-1].token_type == 'cfws': + res.pop() + else: + if res[-1][-1].token_type == 'cfws': + res[-1] = TokenList(res[-1][:-1]) + return res.value + + @property + def value(self): + quote = False + if self.defects: + quote = True + else: + for x in self: + if x.token_type == 'quoted-string': + quote = True + if quote: + pre = post = '' + if self[0].token_type=='cfws' or self[0][0].token_type=='cfws': + pre = ' ' + if self[-1].token_type=='cfws' or self[-1][-1].token_type=='cfws': + post = ' ' + return pre+quote_string(self.display_name)+post + else: + return super().value + + +class LocalPart(TokenList): + + token_type = 'local-part' + + @property + def value(self): + if self[0].token_type == "quoted-string": + return self[0].quoted_value + else: + return self[0].value + + @property + def local_part(self): + # Strip whitespace from front, back, and around dots. + res = [DOT] + last = DOT + last_is_tl = False + for tok in self[0] + [DOT]: + if tok.token_type == 'cfws': + continue + if (last_is_tl and tok.token_type == 'dot' and + last[-1].token_type == 'cfws'): + res[-1] = TokenList(last[:-1]) + is_tl = isinstance(tok, TokenList) + if (is_tl and last.token_type == 'dot' and + tok[0].token_type == 'cfws'): + res.append(TokenList(tok[1:])) + else: + res.append(tok) + last = res[-1] + last_is_tl = is_tl + res = TokenList(res[1:-1]) + return res.value + + +class DomainLiteral(TokenList): + + token_type = 'domain-literal' + + @property + def domain(self): + return ''.join(super().value.split()) + + @property + def ip(self): + for x in self: + if x.token_type == 'ptext': + return x.value + + +class HeaderLabel(TokenList): + + token_type = 'header-label' + + +class Header(TokenList): + + token_type = 'header' + + def _fold(self, folded): + folded.append(str(self.pop(0))) + folded.lastlen = len(folded.current[0]) + # The first line of the header is different from all others: we don't + # want to start a new object on a new line if it has any fold points in + # it that would allow part of it to be on the first header line. + # Further, if the first fold point would fit on the new line, we want + # to do that, but if it doesn't we want to put it on the first line. + # Folded supports this via the stickyspace attribute. If this + # attribute is not None, it does the special handling. + folded.stickyspace = str(self.pop(0)) if self[0].token_type == 'cfws' else '' + rest = self.pop(0) + if self: + raise ValueError("Malformed Header token list") + rest._fold(folded) + + +# +# Terminal classes and instances +# + +class Terminal(str): + + def __new__(cls, value, token_type): + self = super().__new__(cls, value) + self.token_type = token_type + self.defects = [] + return self + + def __repr__(self): + return "{}({})".format(self.__class__.__name__, super().__repr__()) + + @property + def all_defects(self): + return list(self.defects) + + def _pp(self, indent=''): + return ["{}{}/{}({}){}".format( + indent, + self.__class__.__name__, + self.token_type, + super().__repr__(), + '' if not self.defects else ' {}'.format(self.defects), + )] + + def cte_encode(self, charset, policy): + value = str(self) + try: + value.encode('us-ascii') + return value + except UnicodeEncodeError: + return _ew.encode(value, charset) + + def pop_trailing_ws(self): + # This terminates the recursion. + return None + + def pop_leading_fws(self): + # This terminates the recursion. + return None + + @property + def comments(self): + return [] + + def has_leading_comment(self): + return False + + def __getnewargs__(self): + return(str(self), self.token_type) + + +class WhiteSpaceTerminal(Terminal): + + @property + def value(self): + return ' ' + + def startswith_fws(self): + return True + + has_fws = True + + +class ValueTerminal(Terminal): + + @property + def value(self): + return self + + def startswith_fws(self): + return False + + has_fws = False + + def as_encoded_word(self, charset): + return _ew.encode(str(self), charset) + + +class EWWhiteSpaceTerminal(WhiteSpaceTerminal): + + @property + def value(self): + return '' + + @property + def encoded(self): + return self[:] + + def __str__(self): + return '' + + has_fws = True + + +# XXX these need to become classes and used as instances so +# that a program can't change them in a parse tree and screw +# up other parse trees. Maybe should have tests for that, too. +DOT = ValueTerminal('.', 'dot') +ListSeparator = ValueTerminal(',', 'list-separator') +RouteComponentMarker = ValueTerminal('@', 'route-component-marker') + +# +# Parser +# + +"""Parse strings according to RFC822/2047/2822/5322 rules. + +This is a stateless parser. Each get_XXX function accepts a string and +returns either a Terminal or a TokenList representing the RFC object named +by the method and a string containing the remaining unparsed characters +from the input. Thus a parser method consumes the next syntactic construct +of a given type and returns a token representing the construct plus the +unparsed remainder of the input string. + +For example, if the first element of a structured header is a 'phrase', +then: + + phrase, value = get_phrase(value) + +returns the complete phrase from the start of the string value, plus any +characters left in the string after the phrase is removed. + +""" + +_wsp_splitter = re.compile(r'([{}]+)'.format(''.join(WSP))).split +_non_atom_end_matcher = re.compile(r"[^{}]+".format( + ''.join(ATOM_ENDS).replace('\\','\\\\').replace(']','\]'))).match +_non_printable_finder = re.compile(r"[\x00-\x20\x7F]").findall + +def _validate_xtext(xtext): + """If input token contains ASCII non-printables, register a defect.""" + + non_printables = _non_printable_finder(xtext) + if non_printables: + xtext.defects.append(errors.NonPrintableDefect(non_printables)) + if utils._has_surrogates(xtext): + xtext.defects.append(errors.UndecodableBytesDefect( + "Non-ASCII characters found in header token")) + +def _get_ptext_to_endchars(value, endchars): + """Scan printables/quoted-pairs until endchars and return unquoted ptext. + + This function turns a run of qcontent, ccontent-without-comments, or + dtext-with-quoted-printables into a single string by unquoting any + quoted printables. It returns the string, the remaining value, and + a flag that is True iff there were any quoted printables decoded. + + """ + fragment, *remainder = _wsp_splitter(value, 1) + vchars = [] + escape = False + had_qp = False + for pos in range(len(fragment)): + if fragment[pos] == '\\': + if escape: + escape = False + had_qp = True + else: + escape = True + continue + if escape: + escape = False + elif fragment[pos] in endchars: + break + vchars.append(fragment[pos]) + else: + pos = pos + 1 + return ''.join(vchars), ''.join([fragment[pos:]] + remainder), had_qp + +def _decode_ew_run(value): + """ Decode a run of RFC2047 encoded words. + + _decode_ew_run(value) -> (text, value, defects) + + Scans the supplied value for a run of tokens that look like they are RFC + 2047 encoded words, decodes those words into text according to RFC 2047 + rules (whitespace between encoded words is discarded), and returns the text + and the remaining value (including any leading whitespace on the remaining + value), as well as a list of any defects encountered while decoding. The + input value may not have any leading whitespace. + + """ + res = [] + defects = [] + last_ws = '' + while value: + try: + tok, ws, value = _wsp_splitter(value, 1) + except ValueError: + tok, ws, value = value, '', '' + if not (tok.startswith('=?') and tok.endswith('?=')): + return ''.join(res), last_ws + tok + ws + value, defects + text, charset, lang, new_defects = _ew.decode(tok) + res.append(text) + defects.extend(new_defects) + last_ws = ws + return ''.join(res), last_ws, defects + +def get_fws(value): + """FWS = 1*WSP + + This isn't the RFC definition. We're using fws to represent tokens where + folding can be done, but when we are parsing the *un*folding has already + been done so we don't need to watch out for CRLF. + + """ + newvalue = value.lstrip() + fws = WhiteSpaceTerminal(value[:len(value)-len(newvalue)], 'fws') + return fws, newvalue + +def get_encoded_word(value): + """ encoded-word = "=?" charset "?" encoding "?" encoded-text "?=" + + """ + ew = EncodedWord() + if not value.startswith('=?'): + raise errors.HeaderParseError( + "expected encoded word but found {}".format(value)) + tok, *remainder = value[2:].split('?=', 1) + if tok == value[2:]: + raise errors.HeaderParseError( + "expected encoded word but found {}".format(value)) + remstr = ''.join(remainder) + if remstr[:2].isdigit(): + rest, *remainder = remstr.split('?=', 1) + tok = tok + '?=' + rest + if len(tok.split()) > 1: + ew.defects.append(errors.InvalidHeaderDefect( + "whitespace inside encoded word")) + ew.cte = value + value = ''.join(remainder) + try: + text, charset, lang, defects = _ew.decode('=?' + tok + '?=') + except ValueError: + raise errors.HeaderParseError( + "encoded word format invalid: '{}'".format(ew.cte)) + ew.charset = charset + ew.lang = lang + ew.defects.extend(defects) + while text: + if text[0] in WSP: + token, text = get_fws(text) + ew.append(token) + continue + chars, *remainder = _wsp_splitter(text, 1) + vtext = ValueTerminal(chars, 'vtext') + _validate_xtext(vtext) + ew.append(vtext) + text = ''.join(remainder) + return ew, value + +def get_unstructured(value): + """unstructured = (*([FWS] vchar) *WSP) / obs-unstruct + obs-unstruct = *((*LF *CR *(obs-utext) *LF *CR)) / FWS) + obs-utext = %d0 / obs-NO-WS-CTL / LF / CR + + obs-NO-WS-CTL is control characters except WSP/CR/LF. + + So, basically, we have printable runs, plus control characters or nulls in + the obsolete syntax, separated by whitespace. Since RFC 2047 uses the + obsolete syntax in its specification, but requires whitespace on either + side of the encoded words, I can see no reason to need to separate the + non-printable-non-whitespace from the printable runs if they occur, so we + parse this into xtext tokens separated by WSP tokens. + + Because an 'unstructured' value must by definition constitute the entire + value, this 'get' routine does not return a remaining value, only the + parsed TokenList. + + """ + # XXX: but what about bare CR and LF? They might signal the start or + # end of an encoded word. YAGNI for now, since out current parsers + # will never send us strings with bard CR or LF. + + unstructured = UnstructuredTokenList() + while value: + if value[0] in WSP: + token, value = get_fws(value) + unstructured.append(token) + continue + if value.startswith('=?'): + try: + token, value = get_encoded_word(value) + except errors.HeaderParseError: + pass + else: + have_ws = True + if len(unstructured) > 0: + if unstructured[-1].token_type != 'fws': + unstructured.defects.append(errors.InvalidHeaderDefect( + "missing whitespace before encoded word")) + have_ws = False + if have_ws and len(unstructured) > 1: + if unstructured[-2].token_type == 'encoded-word': + unstructured[-1] = EWWhiteSpaceTerminal( + unstructured[-1], 'fws') + unstructured.append(token) + continue + tok, *remainder = _wsp_splitter(value, 1) + vtext = ValueTerminal(tok, 'vtext') + _validate_xtext(vtext) + unstructured.append(vtext) + value = ''.join(remainder) + return unstructured + +def get_qp_ctext(value): + """ctext = + + This is not the RFC ctext, since we are handling nested comments in comment + and unquoting quoted-pairs here. We allow anything except the '()' + characters, but if we find any ASCII other than the RFC defined printable + ASCII an NonPrintableDefect is added to the token's defects list. Since + quoted pairs are converted to their unquoted values, what is returned is + a 'ptext' token. In this case it is a WhiteSpaceTerminal, so it's value + is ' '. + + """ + ptext, value, _ = _get_ptext_to_endchars(value, '()') + ptext = WhiteSpaceTerminal(ptext, 'ptext') + _validate_xtext(ptext) + return ptext, value + +def get_qcontent(value): + """qcontent = qtext / quoted-pair + + We allow anything except the DQUOTE character, but if we find any ASCII + other than the RFC defined printable ASCII an NonPrintableDefect is + added to the token's defects list. Any quoted pairs are converted to their + unquoted values, so what is returned is a 'ptext' token. In this case it + is a ValueTerminal. + + """ + ptext, value, _ = _get_ptext_to_endchars(value, '"') + ptext = ValueTerminal(ptext, 'ptext') + _validate_xtext(ptext) + return ptext, value + +def get_atext(value): + """atext = + + We allow any non-ATOM_ENDS in atext, but add an InvalidATextDefect to + the token's defects list if we find non-atext characters. + """ + m = _non_atom_end_matcher(value) + if not m: + raise errors.HeaderParseError( + "expected atext but found '{}'".format(value)) + atext = m.group() + value = value[len(atext):] + atext = ValueTerminal(atext, 'atext') + _validate_xtext(atext) + return atext, value + +def get_bare_quoted_string(value): + """bare-quoted-string = DQUOTE *([FWS] qcontent) [FWS] DQUOTE + + A quoted-string without the leading or trailing white space. Its + value is the text between the quote marks, with whitespace + preserved and quoted pairs decoded. + """ + if value[0] != '"': + raise errors.HeaderParseError( + "expected '\"' but found '{}'".format(value)) + bare_quoted_string = BareQuotedString() + value = value[1:] + while value and value[0] != '"': + if value[0] in WSP: + token, value = get_fws(value) + else: + token, value = get_qcontent(value) + bare_quoted_string.append(token) + if not value: + bare_quoted_string.defects.append(errors.InvalidHeaderDefect( + "end of header inside quoted string")) + return bare_quoted_string, value + return bare_quoted_string, value[1:] + +def get_comment(value): + """comment = "(" *([FWS] ccontent) [FWS] ")" + ccontent = ctext / quoted-pair / comment + + We handle nested comments here, and quoted-pair in our qp-ctext routine. + """ + if value and value[0] != '(': + raise errors.HeaderParseError( + "expected '(' but found '{}'".format(value)) + comment = Comment() + value = value[1:] + while value and value[0] != ")": + if value[0] in WSP: + token, value = get_fws(value) + elif value[0] == '(': + token, value = get_comment(value) + else: + token, value = get_qp_ctext(value) + comment.append(token) + if not value: + comment.defects.append(errors.InvalidHeaderDefect( + "end of header inside comment")) + return comment, value + return comment, value[1:] + +def get_cfws(value): + """CFWS = (1*([FWS] comment) [FWS]) / FWS + + """ + cfws = CFWSList() + while value and value[0] in CFWS_LEADER: + if value[0] in WSP: + token, value = get_fws(value) + else: + token, value = get_comment(value) + cfws.append(token) + return cfws, value + +def get_quoted_string(value): + """quoted-string = [CFWS] [CFWS] + + 'bare-quoted-string' is an intermediate class defined by this + parser and not by the RFC grammar. It is the quoted string + without any attached CFWS. + """ + quoted_string = QuotedString() + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + quoted_string.append(token) + token, value = get_bare_quoted_string(value) + quoted_string.append(token) + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + quoted_string.append(token) + return quoted_string, value + +def get_atom(value): + """atom = [CFWS] 1*atext [CFWS] + + """ + atom = Atom() + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + atom.append(token) + if value and value[0] in ATOM_ENDS: + raise errors.HeaderParseError( + "expected atom but found '{}'".format(value)) + token, value = get_atext(value) + atom.append(token) + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + atom.append(token) + return atom, value + +def get_dot_atom_text(value): + """ dot-text = 1*atext *("." 1*atext) + + """ + dot_atom_text = DotAtomText() + if not value or value[0] in ATOM_ENDS: + raise errors.HeaderParseError("expected atom at a start of " + "dot-atom-text but found '{}'".format(value)) + while value and value[0] not in ATOM_ENDS: + token, value = get_atext(value) + dot_atom_text.append(token) + if value and value[0] == '.': + dot_atom_text.append(DOT) + value = value[1:] + if dot_atom_text[-1] is DOT: + raise errors.HeaderParseError("expected atom at end of dot-atom-text " + "but found '{}'".format('.'+value)) + return dot_atom_text, value + +def get_dot_atom(value): + """ dot-atom = [CFWS] dot-atom-text [CFWS] + + """ + dot_atom = DotAtom() + if value[0] in CFWS_LEADER: + token, value = get_cfws(value) + dot_atom.append(token) + token, value = get_dot_atom_text(value) + dot_atom.append(token) + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + dot_atom.append(token) + return dot_atom, value + +def get_word(value): + """word = atom / quoted-string + + Either atom or quoted-string may start with CFWS. We have to peel off this + CFWS first to determine which type of word to parse. Afterward we splice + the leading CFWS, if any, into the parsed sub-token. + + If neither an atom or a quoted-string is found before the next special, a + HeaderParseError is raised. + + The token returned is either an Atom or a QuotedString, as appropriate. + This means the 'word' level of the formal grammar is not represented in the + parse tree; this is because having that extra layer when manipulating the + parse tree is more confusing than it is helpful. + + """ + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + else: + leader = None + if value[0]=='"': + token, value = get_quoted_string(value) + elif value[0] in SPECIALS: + raise errors.HeaderParseError("Expected 'atom' or 'quoted-string' " + "but found '{}'".format(value)) + else: + token, value = get_atom(value) + if leader is not None: + token[:0] = [leader] + return token, value + +def get_phrase(value): + """ phrase = 1*word / obs-phrase + obs-phrase = word *(word / "." / CFWS) + + This means a phrase can be a sequence of words, periods, and CFWS in any + order as long as it starts with at least one word. If anything other than + words is detected, an ObsoleteHeaderDefect is added to the token's defect + list. We also accept a phrase that starts with CFWS followed by a dot; + this is registered as an InvalidHeaderDefect, since it is not supported by + even the obsolete grammar. + + """ + phrase = Phrase() + try: + token, value = get_word(value) + phrase.append(token) + except errors.HeaderParseError: + phrase.defects.append(errors.InvalidHeaderDefect( + "phrase does not start with word")) + while value and value[0] not in PHRASE_ENDS: + if value[0]=='.': + phrase.append(DOT) + phrase.defects.append(errors.ObsoleteHeaderDefect( + "period in 'phrase'")) + value = value[1:] + else: + try: + token, value = get_word(value) + except errors.HeaderParseError: + if value[0] in CFWS_LEADER: + token, value = get_cfws(value) + phrase.defects.append(errors.ObsoleteHeaderDefect( + "comment found without atom")) + else: + raise + phrase.append(token) + return phrase, value + +def get_local_part(value): + """ local-part = dot-atom / quoted-string / obs-local-part + + """ + local_part = LocalPart() + leader = None + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value: + raise errors.HeaderParseError( + "expected local-part but found '{}'".format(value)) + try: + token, value = get_dot_atom(value) + except errors.HeaderParseError: + try: + token, value = get_word(value) + except errors.HeaderParseError: + if value[0] != '\\' and value[0] in PHRASE_ENDS: + raise + token = TokenList() + if leader is not None: + token[:0] = [leader] + local_part.append(token) + if value and (value[0]=='\\' or value[0] not in PHRASE_ENDS): + obs_local_part, value = get_obs_local_part(str(local_part) + value) + if obs_local_part.token_type == 'invalid-obs-local-part': + local_part.defects.append(errors.InvalidHeaderDefect( + "local-part is not dot-atom, quoted-string, or obs-local-part")) + else: + local_part.defects.append(errors.ObsoleteHeaderDefect( + "local-part is not a dot-atom (contains CFWS)")) + local_part[0] = obs_local_part + try: + local_part.value.encode('ascii') + except UnicodeEncodeError: + local_part.defects.append(errors.NonASCIILocalPartDefect( + "local-part contains non-ASCII characters)")) + return local_part, value + +def get_obs_local_part(value): + """ obs-local-part = word *("." word) + """ + obs_local_part = ObsLocalPart() + last_non_ws_was_dot = False + while value and (value[0]=='\\' or value[0] not in PHRASE_ENDS): + if value[0] == '.': + if last_non_ws_was_dot: + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "invalid repeated '.'")) + obs_local_part.append(DOT) + last_non_ws_was_dot = True + value = value[1:] + continue + elif value[0]=='\\': + obs_local_part.append(ValueTerminal(value[0], + 'misplaced-special')) + value = value[1:] + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "'\\' character outside of quoted-string/ccontent")) + last_non_ws_was_dot = False + continue + if obs_local_part and obs_local_part[-1].token_type != 'dot': + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "missing '.' between words")) + try: + token, value = get_word(value) + last_non_ws_was_dot = False + except errors.HeaderParseError: + if value[0] not in CFWS_LEADER: + raise + token, value = get_cfws(value) + obs_local_part.append(token) + if (obs_local_part[0].token_type == 'dot' or + obs_local_part[0].token_type=='cfws' and + obs_local_part[1].token_type=='dot'): + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "Invalid leading '.' in local part")) + if (obs_local_part[-1].token_type == 'dot' or + obs_local_part[-1].token_type=='cfws' and + obs_local_part[-2].token_type=='dot'): + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "Invalid trailing '.' in local part")) + if obs_local_part.defects: + obs_local_part.token_type = 'invalid-obs-local-part' + return obs_local_part, value + +def get_dtext(value): + """ dtext = / obs-dtext + obs-dtext = obs-NO-WS-CTL / quoted-pair + + We allow anything except the excluded characters, but but if we find any + ASCII other than the RFC defined printable ASCII an NonPrintableDefect is + added to the token's defects list. Quoted pairs are converted to their + unquoted values, so what is returned is a ptext token, in this case a + ValueTerminal. If there were quoted-printables, an ObsoleteHeaderDefect is + added to the returned token's defect list. + + """ + ptext, value, had_qp = _get_ptext_to_endchars(value, '[]') + ptext = ValueTerminal(ptext, 'ptext') + if had_qp: + ptext.defects.append(errors.ObsoleteHeaderDefect( + "quoted printable found in domain-literal")) + _validate_xtext(ptext) + return ptext, value + +def _check_for_early_dl_end(value, domain_literal): + if value: + return False + domain_literal.append(errors.InvalidHeaderDefect( + "end of input inside domain-literal")) + domain_literal.append(ValueTerminal(']', 'domain-literal-end')) + return True + +def get_domain_literal(value): + """ domain-literal = [CFWS] "[" *([FWS] dtext) [FWS] "]" [CFWS] + + """ + domain_literal = DomainLiteral() + if value[0] in CFWS_LEADER: + token, value = get_cfws(value) + domain_literal.append(token) + if not value: + raise errors.HeaderParseError("expected domain-literal") + if value[0] != '[': + raise errors.HeaderParseError("expected '[' at start of domain-literal " + "but found '{}'".format(value)) + value = value[1:] + if _check_for_early_dl_end(value, domain_literal): + return domain_literal, value + domain_literal.append(ValueTerminal('[', 'domain-literal-start')) + if value[0] in WSP: + token, value = get_fws(value) + domain_literal.append(token) + token, value = get_dtext(value) + domain_literal.append(token) + if _check_for_early_dl_end(value, domain_literal): + return domain_literal, value + if value[0] in WSP: + token, value = get_fws(value) + domain_literal.append(token) + if _check_for_early_dl_end(value, domain_literal): + return domain_literal, value + if value[0] != ']': + raise errors.HeaderParseError("expected ']' at end of domain-literal " + "but found '{}'".format(value)) + domain_literal.append(ValueTerminal(']', 'domain-literal-end')) + value = value[1:] + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + domain_literal.append(token) + return domain_literal, value + +def get_domain(value): + """ domain = dot-atom / domain-literal / obs-domain + obs-domain = atom *("." atom)) + + """ + domain = Domain() + leader = None + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value: + raise errors.HeaderParseError( + "expected domain but found '{}'".format(value)) + if value[0] == '[': + token, value = get_domain_literal(value) + if leader is not None: + token[:0] = [leader] + domain.append(token) + return domain, value + try: + token, value = get_dot_atom(value) + except errors.HeaderParseError: + token, value = get_atom(value) + if leader is not None: + token[:0] = [leader] + domain.append(token) + if value and value[0] == '.': + domain.defects.append(errors.ObsoleteHeaderDefect( + "domain is not a dot-atom (contains CFWS)")) + if domain[0].token_type == 'dot-atom': + domain[:] = domain[0] + while value and value[0] == '.': + domain.append(DOT) + token, value = get_atom(value[1:]) + domain.append(token) + return domain, value + +def get_addr_spec(value): + """ addr-spec = local-part "@" domain + + """ + addr_spec = AddrSpec() + token, value = get_local_part(value) + addr_spec.append(token) + if not value or value[0] != '@': + addr_spec.defects.append(errors.InvalidHeaderDefect( + "add-spec local part with no domain")) + return addr_spec, value + addr_spec.append(ValueTerminal('@', 'address-at-symbol')) + token, value = get_domain(value[1:]) + addr_spec.append(token) + return addr_spec, value + +def get_obs_route(value): + """ obs-route = obs-domain-list ":" + obs-domain-list = *(CFWS / ",") "@" domain *("," [CFWS] ["@" domain]) + + Returns an obs-route token with the appropriate sub-tokens (that is, + there is no obs-domain-list in the parse tree). + """ + obs_route = ObsRoute() + while value and (value[0]==',' or value[0] in CFWS_LEADER): + if value[0] in CFWS_LEADER: + token, value = get_cfws(value) + obs_route.append(token) + elif value[0] == ',': + obs_route.append(ListSeparator) + value = value[1:] + if not value or value[0] != '@': + raise errors.HeaderParseError( + "expected obs-route domain but found '{}'".format(value)) + obs_route.append(RouteComponentMarker) + token, value = get_domain(value[1:]) + obs_route.append(token) + while value and value[0]==',': + obs_route.append(ListSeparator) + value = value[1:] + if not value: + break + if value[0] in CFWS_LEADER: + token, value = get_cfws(value) + obs_route.append(token) + if value[0] == '@': + obs_route.append(RouteComponentMarker) + token, value = get_domain(value[1:]) + obs_route.append(token) + if not value: + raise errors.HeaderParseError("end of header while parsing obs-route") + if value[0] != ':': + raise errors.HeaderParseError( "expected ':' marking end of " + "obs-route but found '{}'".format(value)) + obs_route.append(ValueTerminal(':', 'end-of-obs-route-marker')) + return obs_route, value[1:] + +def get_angle_addr(value): + """ angle-addr = [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr + obs-angle-addr = [CFWS] "<" obs-route addr-spec ">" [CFWS] + + """ + angle_addr = AngleAddr() + if value[0] in CFWS_LEADER: + token, value = get_cfws(value) + angle_addr.append(token) + if not value or value[0] != '<': + raise errors.HeaderParseError( + "expected angle-addr but found '{}'".format(value)) + angle_addr.append(ValueTerminal('<', 'angle-addr-start')) + value = value[1:] + try: + token, value = get_addr_spec(value) + except errors.HeaderParseError: + try: + token, value = get_obs_route(value) + angle_addr.defects.append(errors.ObsoleteHeaderDefect( + "obsolete route specification in angle-addr")) + except errors.HeaderParseError: + raise errors.HeaderParseError( + "expected addr-spec or but found '{}'".format(value)) + angle_addr.append(token) + token, value = get_addr_spec(value) + angle_addr.append(token) + if value and value[0] == '>': + value = value[1:] + else: + angle_addr.defects.append(errors.InvalidHeaderDefect( + "missing trailing '>' on angle-addr")) + angle_addr.append(ValueTerminal('>', 'angle-addr-end')) + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + angle_addr.append(token) + return angle_addr, value + +def get_display_name(value): + """ display-name = phrase + + Because this is simply a name-rule, we don't return a display-name + token containing a phrase, but rather a display-name token with + the content of the phrase. + + """ + display_name = DisplayName() + token, value = get_phrase(value) + display_name.extend(token[:]) + display_name.defects = token.defects[:] + return display_name, value + + +def get_name_addr(value): + """ name-addr = [display-name] angle-addr + + """ + name_addr = NameAddr() + # Both the optional display name and the angle-addr can start with cfws. + leader = None + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value: + raise errors.HeaderParseError( + "expected name-addr but found '{}'".format(leader)) + if value[0] != '<': + if value[0] in PHRASE_ENDS: + raise errors.HeaderParseError( + "expected name-addr but found '{}'".format(value)) + token, value = get_display_name(value) + if not value: + raise errors.HeaderParseError( + "expected name-addr but found '{}'".format(token)) + if leader is not None: + token[0][:0] = [leader] + leader = None + name_addr.append(token) + token, value = get_angle_addr(value) + if leader is not None: + token[:0] = [leader] + name_addr.append(token) + return name_addr, value + +def get_mailbox(value): + """ mailbox = name-addr / addr-spec + + """ + # The only way to figure out if we are dealing with a name-addr or an + # addr-spec is to try parsing each one. + mailbox = Mailbox() + try: + token, value = get_name_addr(value) + except errors.HeaderParseError: + try: + token, value = get_addr_spec(value) + except errors.HeaderParseError: + raise errors.HeaderParseError( + "expected mailbox but found '{}'".format(value)) + if any(isinstance(x, errors.InvalidHeaderDefect) + for x in token.all_defects): + mailbox.token_type = 'invalid-mailbox' + mailbox.append(token) + return mailbox, value + +def get_invalid_mailbox(value, endchars): + """ Read everything up to one of the chars in endchars. + + This is outside the formal grammar. The InvalidMailbox TokenList that is + returned acts like a Mailbox, but the data attributes are None. + + """ + invalid_mailbox = InvalidMailbox() + while value and value[0] not in endchars: + if value[0] in PHRASE_ENDS: + invalid_mailbox.append(ValueTerminal(value[0], + 'misplaced-special')) + value = value[1:] + else: + token, value = get_phrase(value) + invalid_mailbox.append(token) + return invalid_mailbox, value + +def get_mailbox_list(value): + """ mailbox-list = (mailbox *("," mailbox)) / obs-mbox-list + obs-mbox-list = *([CFWS] ",") mailbox *("," [mailbox / CFWS]) + + For this routine we go outside the formal grammar in order to improve error + handling. We recognize the end of the mailbox list only at the end of the + value or at a ';' (the group terminator). This is so that we can turn + invalid mailboxes into InvalidMailbox tokens and continue parsing any + remaining valid mailboxes. We also allow all mailbox entries to be null, + and this condition is handled appropriately at a higher level. + + """ + mailbox_list = MailboxList() + while value and value[0] != ';': + try: + token, value = get_mailbox(value) + mailbox_list.append(token) + except errors.HeaderParseError: + leader = None + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value or value[0] in ',;': + mailbox_list.append(leader) + mailbox_list.defects.append(errors.ObsoleteHeaderDefect( + "empty element in mailbox-list")) + else: + token, value = get_invalid_mailbox(value, ',;') + if leader is not None: + token[:0] = [leader] + mailbox_list.append(token) + mailbox_list.defects.append(errors.InvalidHeaderDefect( + "invalid mailbox in mailbox-list")) + elif value[0] == ',': + mailbox_list.defects.append(errors.ObsoleteHeaderDefect( + "empty element in mailbox-list")) + else: + token, value = get_invalid_mailbox(value, ',;') + if leader is not None: + token[:0] = [leader] + mailbox_list.append(token) + mailbox_list.defects.append(errors.InvalidHeaderDefect( + "invalid mailbox in mailbox-list")) + if value and value[0] not in ',;': + # Crap after mailbox; treat it as an invalid mailbox. + # The mailbox info will still be available. + mailbox = mailbox_list[-1] + mailbox.token_type = 'invalid-mailbox' + token, value = get_invalid_mailbox(value, ',;') + mailbox.extend(token) + mailbox_list.defects.append(errors.InvalidHeaderDefect( + "invalid mailbox in mailbox-list")) + if value and value[0] == ',': + mailbox_list.append(ListSeparator) + value = value[1:] + return mailbox_list, value + + +def get_group_list(value): + """ group-list = mailbox-list / CFWS / obs-group-list + obs-group-list = 1*([CFWS] ",") [CFWS] + + """ + group_list = GroupList() + if not value: + group_list.defects.append(errors.InvalidHeaderDefect( + "end of header before group-list")) + return group_list, value + leader = None + if value and value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value: + # This should never happen in email parsing, since CFWS-only is a + # legal alternative to group-list in a group, which is the only + # place group-list appears. + group_list.defects.append(errors.InvalidHeaderDefect( + "end of header in group-list")) + group_list.append(leader) + return group_list, value + if value[0] == ';': + group_list.append(leader) + return group_list, value + token, value = get_mailbox_list(value) + if len(token.all_mailboxes)==0: + if leader is not None: + group_list.append(leader) + group_list.extend(token) + group_list.defects.append(errors.ObsoleteHeaderDefect( + "group-list with empty entries")) + return group_list, value + if leader is not None: + token[:0] = [leader] + group_list.append(token) + return group_list, value + +def get_group(value): + """ group = display-name ":" [group-list] ";" [CFWS] + + """ + group = Group() + token, value = get_display_name(value) + if not value or value[0] != ':': + raise errors.HeaderParseError("expected ':' at end of group " + "display name but found '{}'".format(value)) + group.append(token) + group.append(ValueTerminal(':', 'group-display-name-terminator')) + value = value[1:] + if value and value[0] == ';': + group.append(ValueTerminal(';', 'group-terminator')) + return group, value[1:] + token, value = get_group_list(value) + group.append(token) + if not value: + group.defects.append(errors.InvalidHeaderDefect( + "end of header in group")) + if value[0] != ';': + raise errors.HeaderParseError( + "expected ';' at end of group but found {}".format(value)) + group.append(ValueTerminal(';', 'group-terminator')) + value = value[1:] + if value and value[0] in CFWS_LEADER: + token, value = get_cfws(value) + group.append(token) + return group, value + +def get_address(value): + """ address = mailbox / group + + Note that counter-intuitively, an address can be either a single address or + a list of addresses (a group). This is why the returned Address object has + a 'mailboxes' attribute which treats a single address as a list of length + one. When you need to differentiate between to two cases, extract the single + element, which is either a mailbox or a group token. + + """ + # The formal grammar isn't very helpful when parsing an address. mailbox + # and group, especially when allowing for obsolete forms, start off very + # similarly. It is only when you reach one of @, <, or : that you know + # what you've got. So, we try each one in turn, starting with the more + # likely of the two. We could perhaps make this more efficient by looking + # for a phrase and then branching based on the next character, but that + # would be a premature optimization. + address = Address() + try: + token, value = get_group(value) + except errors.HeaderParseError: + try: + token, value = get_mailbox(value) + except errors.HeaderParseError: + raise errors.HeaderParseError( + "expected address but found '{}'".format(value)) + address.append(token) + return address, value + +def get_address_list(value): + """ address_list = (address *("," address)) / obs-addr-list + obs-addr-list = *([CFWS] ",") address *("," [address / CFWS]) + + We depart from the formal grammar here by continuing to parse until the end + of the input, assuming the input to be entirely composed of an + address-list. This is always true in email parsing, and allows us + to skip invalid addresses to parse additional valid ones. + + """ + address_list = AddressList() + while value: + try: + token, value = get_address(value) + address_list.append(token) + except errors.HeaderParseError as err: + leader = None + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value or value[0] == ',': + address_list.append(leader) + address_list.defects.append(errors.ObsoleteHeaderDefect( + "address-list entry with no content")) + else: + token, value = get_invalid_mailbox(value, ',') + if leader is not None: + token[:0] = [leader] + address_list.append(Address([token])) + address_list.defects.append(errors.InvalidHeaderDefect( + "invalid address in address-list")) + elif value[0] == ',': + address_list.defects.append(errors.ObsoleteHeaderDefect( + "empty element in address-list")) + else: + token, value = get_invalid_mailbox(value, ',') + if leader is not None: + token[:0] = [leader] + address_list.append(Address([token])) + address_list.defects.append(errors.InvalidHeaderDefect( + "invalid address in address-list")) + if value and value[0] != ',': + # Crap after address; treat it as an invalid mailbox. + # The mailbox info will still be available. + mailbox = address_list[-1][0] + mailbox.token_type = 'invalid-mailbox' + token, value = get_invalid_mailbox(value, ',') + mailbox.extend(token) + address_list.defects.append(errors.InvalidHeaderDefect( + "invalid address in address-list")) + if value: # Must be a , at this point. + address_list.append(ValueTerminal(',', 'list-separator')) + value = value[1:] + return address_list, value diff --git a/Lib/email/_headerregistry.py b/Lib/email/_headerregistry.py new file mode 100644 --- /dev/null +++ b/Lib/email/_headerregistry.py @@ -0,0 +1,456 @@ +"""Representing and manipulating email headers via custom objects. + +This module provides an implementation of the HeaderRegistry API. +The implementation is designed to flexibly follow RFC5322 rules. + +Eventually HeaderRegistry will be a public API, but it isn't yet, +and will probably change some before that happens. + +""" + +from email import utils +from email import errors +from email import _header_value_parser as parser + +class Address: + + def __init__(self, display_name='', username='', domain='', addr_spec=None): + """Create an object represeting a full email address. + + An address can have a 'display_name', a 'username', and a 'domain'. In + addition to specifying the username and domain separately, they may be + specified together by using the addr_spec keyword *instead of* the + username and domain keywords. If an addr_spec string is specified it + must be properly quoted according to RFC 5322 rules; an error will be + raised if it is not. + + An Address object has display_name, username, domain, and addr_spec + attributes, all of which are read-only. The addr_spec and the string + value of the object are both quoted according to RFC5322 rules, but + without any Content Transfer Encoding. + + """ + # This clause with its potential 'raise' may only happen when an + # application program creates an Address object using an addr_spec + # keyword. The email library code itself must always supply username + # and domain. + if addr_spec is not None: + if username or domain: + raise TypeError("addrspec specified when username and/or " + "domain also specified") + a_s, rest = parser.get_addr_spec(addr_spec) + if rest: + raise ValueError("Invalid addr_spec; only '{}' " + "could be parsed from '{}'".format( + a_s, addr_spec)) + if a_s.all_defects: + raise a_s.all_defects[0] + username = a_s.local_part + domain = a_s.domain + self._display_name = display_name + self._username = username + self._domain = domain + + @property + def display_name(self): + return self._display_name + + @property + def username(self): + return self._username + + @property + def domain(self): + return self._domain + + @property + def addr_spec(self): + """The addr_spec (username at domain) portion of the address, quoted + according to RFC 5322 rules, but with no Content Transfer Encoding. + """ + nameset = set(self.username) + if len(nameset) > len(nameset-parser.DOT_ATOM_ENDS): + lp = parser.quote_string(self.username) + else: + lp = self.username + if self.domain: + return lp + '@' + self.domain + if not lp: + return '<>' + return lp + + def __repr__(self): + return "Address(display_name={!r}, username={!r}, domain={!r})".format( + self.display_name, self.username, self.domain) + + def __str__(self): + nameset = set(self.display_name) + if len(nameset) > len(nameset-parser.SPECIALS): + disp = parser.quote_string(self.display_name) + else: + disp = self.display_name + if disp: + addr_spec = '' if self.addr_spec=='<>' else self.addr_spec + return "{} <{}>".format(disp, addr_spec) + return self.addr_spec + + def __eq__(self, other): + if type(other) != type(self): + return False + return (self.display_name == other.display_name and + self.username == other.username and + self.domain == other.domain) + + +class Group: + + def __init__(self, display_name=None, addresses=None): + """Create an object representing an address group. + + An address group consists of a display_name followed by colon and an + list of addresses (see Address) terminated by a semi-colon. The Group + is created by specifying a display_name and a possibly empty list of + Address objects. A Group can also be used to represent a single + address that is not in a group, which is convenient when manipulating + lists that are a combination of Groups and individual Addresses. In + this case the display_name should be set to None. In particular, the + string representation of a Group whose display_name is None is the same + as the Address object, if there is one and only one Address object in + the addresses list. + + """ + self._display_name = display_name + self._addresses = tuple(addresses) if addresses else tuple() + + @property + def display_name(self): + return self._display_name + + @property + def addresses(self): + return self._addresses + + def __repr__(self): + return "Group(display_name={!r}, addresses={!r}".format( + self.display_name, self.addresses) + + def __str__(self): + if self.display_name is None and len(self.addresses)==1: + return str(self.addresses[0]) + disp = self.display_name + if disp is not None: + nameset = set(disp) + if len(nameset) > len(nameset-parser.SPECIALS): + disp = parser.quote_string(disp) + adrstr = ", ".join(str(x) for x in self.addresses) + adrstr = ' ' + adrstr if adrstr else adrstr + return "{}:{};".format(disp, adrstr) + + def __eq__(self, other): + if type(other) != type(self): + return False + return (self.display_name == other.display_name and + self.addresses == other.addresses) + + +# Header Classes # + +class BaseHeader(str): + + """Base class for message headers. + + Implements generic behavior and provides tools for subclasses. + + A subclass must define a classmethod named 'parse' that takes an unfolded + value string and a dictionary as its arguments. The dictionary will + contain one key, 'defects', initialized to an empty list. After the call + the dictionary must contain two additional keys: parse_tree, set to the + parse tree obtained from parsing the header, and 'decoded', set to the + string value of the idealized representation of the data from the value. + (That is, encoded words are decoded, and values that have canonical + representations are so represented.) + + The defects key is intended to collect parsing defects, which the message + parser will subsequently dispose of as appropriate. The parser should not, + insofar as practical, raise any errors. Defects should be added to the + list instead. The standard header parsers register defects for RFC + compliance issues, for obsolete RFC syntax, and for unrecoverable parsing + errors. + + The parse method may add additional keys to the dictionary. In this case + the subclass must define an 'init' method, which will be passed the + dictionary as its keyword arguments. The method should use (usually by + setting them as the value of similarly named attributes) and remove all the + extra keys added by its parse method, and then use super to call its parent + class with the remaining arguments and keywords. + + The subclass should also make sure that a 'max_count' attribute is defined + that is either None or 1. XXX: need to better define this API. + + """ + + def __new__(cls, name, value): + kwds = {'defects': []} + cls.parse(value, kwds) + if utils._has_surrogates(kwds['decoded']): + kwds['decoded'] = utils._sanitize(kwds['decoded']) + self = str.__new__(cls, kwds['decoded']) + del kwds['decoded'] + self.init(name, **kwds) + return self + + def init(self, name, *, parse_tree, defects): + self._name = name + self._parse_tree = parse_tree + self._defects = defects + + @property + def name(self): + return self._name + + @property + def defects(self): + return tuple(self._defects) + + def __reduce__(self): + return ( + _reconstruct_header, + ( + self.__class__.__name__, + self.__class__.__bases__, + str(self), + ), + self.__dict__) + + @classmethod + def _reconstruct(cls, value): + return str.__new__(cls, value) + + def fold(self, *, policy): + """Fold header according to policy. + + The parsed representation of the header is folded according to + RFC5322 rules, as modified by the policy. If the parse tree + contains surrogateescaped bytes, the bytes are CTE encoded using + the charset 'unknown-8bit". + + Any non-ASCII characters in the parse tree are CTE encoded using + charset utf-8. XXX: make this a policy setting. + + The returned value is an ASCII-only string possibly containing linesep + characters, and ending with a linesep character. The string includes + the header name and the ': ' separator. + + """ + # At some point we need to only put fws here if it was in the source. + header = parser.Header([ + parser.HeaderLabel([ + parser.ValueTerminal(self.name, 'header-name'), + parser.ValueTerminal(':', 'header-sep')]), + parser.CFWSList([parser.WhiteSpaceTerminal(' ', 'fws')]), + self._parse_tree]) + return header.fold(policy=policy) + + +def _reconstruct_header(cls_name, bases, value): + return type(cls_name, bases, {})._reconstruct(value) + + +class UnstructuredHeader: + + max_count = None + value_parser = staticmethod(parser.get_unstructured) + + @classmethod + def parse(cls, value, kwds): + kwds['parse_tree'] = cls.value_parser(value) + kwds['decoded'] = str(kwds['parse_tree']) + + +class UniqueUnstructuredHeader(UnstructuredHeader): + + max_count = 1 + + +class DateHeader: + + """Header whose value consists of a single timestamp. + + Provides an additional attribute, datetime, which is either an aware + datetime using a timezone, or a naive datetime if the timezone + in the input string is -0000. Also accepts a datetime as input. + The 'value' attribute is the normalized form of the timestamp, + which means it is the output of format_datetime on the datetime. + """ + + max_count = None + + # This is used only for folding, not for creating 'decoded'. + value_parser = staticmethod(parser.get_unstructured) + + @classmethod + def parse(cls, value, kwds): + if not value: + kwds['defects'].append(errors.HeaderMissingRequiredValue()) + kwds['datetime'] = None + kwds['decoded'] = '' + kwds['parse_tree'] = parser.TokenList() + return + if isinstance(value, str): + value = utils.parsedate_to_datetime(value) + kwds['datetime'] = value + kwds['decoded'] = utils.format_datetime(kwds['datetime']) + kwds['parse_tree'] = cls.value_parser(kwds['decoded']) + + def init(self, *args, **kw): + self._datetime = kw.pop('datetime') + super().init(*args, **kw) + + @property + def datetime(self): + return self._datetime + + +class UniqueDateHeader(DateHeader): + + max_count = 1 + + +class AddressHeader: + + max_count = None + + @staticmethod + def value_parser(value): + address_list, value = parser.get_address_list(value) + assert not value, 'this should not happen' + return address_list + + @classmethod + def parse(cls, value, kwds): + if isinstance(value, str): + # We are translating here from the RFC language (address/mailbox) + # to our API language (group/address). + kwds['parse_tree'] = address_list = cls.value_parser(value) + groups = [] + for addr in address_list.addresses: + groups.append(Group(addr.display_name, + [Address(mb.display_name or '', + mb.local_part or '', + mb.domain or '') + for mb in addr.all_mailboxes])) + defects = list(address_list.all_defects) + else: + # Assume it is Address/Group stuff + if not hasattr(value, '__iter__'): + value = [value] + groups = [Group(None, [item]) if not hasattr(item, 'addresses') + else item + for item in value] + defects = [] + kwds['groups'] = groups + kwds['defects'] = defects + kwds['decoded'] = ', '.join([str(item) for item in groups]) + if 'parse_tree' not in kwds: + kwds['parse_tree'] = cls.value_parser(kwds['decoded']) + + def init(self, *args, **kw): + self._groups = tuple(kw.pop('groups')) + self._addresses = None + super().init(*args, **kw) + + @property + def groups(self): + return self._groups + + @property + def addresses(self): + if self._addresses is None: + self._addresses = tuple([address for group in self._groups + for address in group.addresses]) + return self._addresses + + +class UniqueAddressHeader(AddressHeader): + + max_count = 1 + + +class SingleAddressHeader(AddressHeader): + + @property + def address(self): + if len(self.addresses)!=1: + raise ValueError(("value of single address header {} is not " + "a single address").format(self.name)) + return self.addresses[0] + + +class UniqueSingleAddressHeader(SingleAddressHeader): + + max_count = 1 + + +# The header factory # + +_default_header_map = { + 'subject': UniqueUnstructuredHeader, + 'date': UniqueDateHeader, + 'resent-date': DateHeader, + 'orig-date': UniqueDateHeader, + 'sender': UniqueSingleAddressHeader, + 'resent-sender': SingleAddressHeader, + 'to': UniqueAddressHeader, + 'resent-to': AddressHeader, + 'cc': UniqueAddressHeader, + 'resent-cc': AddressHeader, + 'bcc': UniqueAddressHeader, + 'resent-bcc': AddressHeader, + 'from': UniqueAddressHeader, + 'resent-from': AddressHeader, + 'reply-to': UniqueAddressHeader, + } + +class HeaderRegistry: + + """A header_factory and header registry.""" + + def __init__(self, base_class=BaseHeader, default_class=UnstructuredHeader, + use_default_map=True): + """Create a header_factory that works with the Policy API. + + base_class is the class that will be the last class in the created + header class's __bases__ list. default_class is the class that will be + used if "name" (see __call__) does not appear in the registry. + use_default_map controls whether or not the default mapping of names to + specialized classes is copied in to the registry when the factory is + created. The default is True. + + """ + self.registry = {} + self.base_class = base_class + self.default_class = default_class + if use_default_map: + self.registry.update(_default_header_map) + + def map_to_type(self, name, cls): + """Register cls as the specialized class for handling "name" headers. + + """ + self.registry[name.lower()] = cls + + def __getitem__(self, name): + cls = self.registry.get(name.lower(), self.default_class) + return type('_'+cls.__name__, (cls, self.base_class), {}) + + def __call__(self, name, value): + """Create a header instance for header 'name' from 'value'. + + Creates a header instance by creating a specialized class for parsing + and representing the specified header by combining the factory + base_class with a specialized class from the registry or the + default_class, and passing the name and value to the constructed + class's constructor. + + """ + return self[name](name, value) diff --git a/Lib/email/_policybase.py b/Lib/email/_policybase.py --- a/Lib/email/_policybase.py +++ b/Lib/email/_policybase.py @@ -64,10 +64,16 @@ except for the changes passed in as keyword arguments. """ + newpolicy = self.__class__.__new__(self.__class__) for attr, value in self.__dict__.items(): - if attr not in kw: - kw[attr] = value - return self.__class__(**kw) + object.__setattr__(newpolicy, attr, value) + for attr, value in kw.items(): + if not hasattr(self, attr): + raise TypeError( + "{!r} is an invalid keyword argument for {}".format( + attr, self.__class__.__name__)) + object.__setattr__(newpolicy, attr, value) + return newpolicy def __setattr__(self, name, value): if hasattr(self, name): diff --git a/Lib/email/errors.py b/Lib/email/errors.py --- a/Lib/email/errors.py +++ b/Lib/email/errors.py @@ -5,7 +5,6 @@ """email package exception classes.""" - class MessageError(Exception): """Base class for errors in the email package.""" @@ -30,9 +29,8 @@ """An illegal charset was given.""" - # These are parsing defects which the parser was able to work around. -class MessageDefect(Exception): +class MessageDefect(ValueError): """Base class for a message defect.""" def __init__(self, line=None): @@ -58,3 +56,42 @@ class InvalidMultipartContentTransferEncodingDefect(MessageDefect): """An invalid content transfer encoding was set on the multipart itself.""" + +class UndecodableBytesDefect(MessageDefect): + """Header contained bytes that could not be decoded""" + +class InvalidBase64PaddingDefect(MessageDefect): + """base64 encoded sequence had an incorrect length""" + +class InvalidBase64CharactersDefect(MessageDefect): + """base64 encoded sequence had characters not in base64 alphabet""" + +# These errors are specific to header parsing. + +class HeaderDefect(MessageDefect): + """Base class for a header defect.""" + +class InvalidHeaderDefect(HeaderDefect): + """Header is not valid, message gives details.""" + +class HeaderMissingRequiredValue(HeaderDefect): + """A header that must have a value had none""" + +class NonPrintableDefect(HeaderDefect): + """ASCII characters outside the ascii-printable range found""" + + def __init__(self, non_printables): + super().__init__(non_printables) + self.non_printables = non_printables + + def __str__(self): + return ("the following ASCII non-printables found in header: " + "{}".format(self.non_printables)) + +class ObsoleteHeaderDefect(HeaderDefect): + """Header uses syntax declared obsolete by RFC 5322""" + +class NonASCIILocalPartDefect(HeaderDefect): + """local_part contains non-ASCII characters""" + # This defect only occurs during unicode parsing, not when + # parsing messages decoded from binary. diff --git a/Lib/email/generator.py b/Lib/email/generator.py --- a/Lib/email/generator.py +++ b/Lib/email/generator.py @@ -95,9 +95,15 @@ self._encoded_NL = self._encode(self._NL) self._EMPTY = '' self._encoded_EMTPY = self._encode('') - p = self.policy + # Because we use clone (below) when we recursively process message + # subparts, and because clone uses the computed policy (not None), + # submessages will automatically get set to the computed policy when + # they are processed by this code. + old_gen_policy = self.policy + old_msg_policy = msg.policy try: self.policy = policy + msg.policy = policy if unixfrom: ufrom = msg.get_unixfrom() if not ufrom: @@ -105,7 +111,8 @@ self.write(ufrom + self._NL) self._write(msg) finally: - self.policy = p + self.policy = old_gen_policy + msg.policy = old_msg_policy def clone(self, fp): """Clone this generator with the exact same options.""" diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -2,11 +2,178 @@ code that adds all the email6 features. """ -from email._policybase import Policy, compat32, Compat32 +from email._policybase import Policy, Compat32, compat32 +from email.utils import _has_surrogates +from email._headerregistry import HeaderRegistry as _HeaderRegistry -# XXX: temporarily derive everything from compat32. +__all__ = [ + 'Compat32', + 'compat32', + 'Policy', + 'EmailPolicy', + 'default', + 'strict', + 'SMTP', + 'HTTP', + ] -default = compat32 +class EmailPolicy(Policy): + + """+ + PROVISIONAL + + The API extensions enabled by this this policy are currently provisional. + Refer to the documentation for details. + + This policy adds new header parsing and folding algorithms. Instead of + simple strings, headers are custom objects with custom attributes + depending on the type of the field. The folding algorithm fully + implements RFCs 2047 and 5322. + + In addition to the settable attributes listed above that apply to + all Policies, this policy adds the following additional attributes: + + refold_source -- if the value for a header in the Message object + came from the parsing of some source, this attribute + indicates whether or not a generator should refold + that value when transforming the message back into + stream form. The possible values are: + + none -- all source values use original folding + long -- source values that have any line that is + longer than max_line_length will be + refolded + all -- all values are refolded. + + The default is 'long'. + + header_factory -- a callable that takes two arguments, 'name' and + 'value', where 'name' is a header field name and + 'value' is an unfolded header field value, and + returns a string-like object that represents that + header. A default header_factory is provided that + understands some of the RFC5322 header field types. + (Currently address fields and date fields have + special treatment, while all other fields are + treated as unstructured. This list will be + completed before the extension is marked stable.) + """ + + refold_source = 'long' + header_factory = _HeaderRegistry() + + def __init__(self, **kw): + # Ensure that each new instance gets a unique header factory + # (as opposed to clones, which share the factory). + if 'header_factory' not in kw: + object.__setattr__(self, 'header_factory', _HeaderRegistry()) + super().__init__(**kw) + + # The logic of the next three methods is chosen such that it is possible to + # switch a Message object between a Compat32 policy and a policy derived + # from this class and have the results stay consistent. This allows a + # Message object constructed with this policy to be passed to a library + # that only handles Compat32 objects, or to receive such an object and + # convert it to use the newer style by just changing its policy. It is + # also chosen because it postpones the relatively expensive full rfc5322 + # parse until as late as possible when parsing from source, since in many + # applications only a few headers will actually be inspected. + + def header_source_parse(self, sourcelines): + """+ + The name is parsed as everything up to the ':' and returned unmodified. + The value is determined by stripping leading whitespace off the + remainder of the first line, joining all subsequent lines together, and + stripping any trailing carriage return or linefeed characters. (This + is the same as Compat32). + + """ + name, value = sourcelines[0].split(':', 1) + value = value.lstrip(' \t') + ''.join(sourcelines[1:]) + return (name, value.rstrip('\r\n')) + + def header_store_parse(self, name, value): + """+ + The name is returned unchanged. If the input value has a 'name' + attribute and it matches the name ignoring case, the value is returned + unchanged. Otherwise the name and value are passed to header_factory + method, and the resulting custom header object is returned as the + value. In this case a ValueError is raised if the input value contains + CR or LF characters. + + """ + if hasattr(value, 'name') and value.name.lower() == name.lower(): + return (name, value) + if len(value.splitlines())>1: + raise ValueError("Header values may not contain linefeed " + "or carriage return characters") + return (name, self.header_factory(name, value)) + + def header_fetch_parse(self, name, value): + """+ + If the value has a 'name' attribute, it is returned to unmodified. + Otherwise the name and the value with any linesep characters removed + are passed to the header_factory method, and the resulting custom + header object is returned. Any surrogateescaped bytes get turned + into the unicode unknown-character glyph. + + """ + if hasattr(value, 'name'): + return value + return self.header_factory(name, ''.join(value.splitlines())) + + def fold(self, name, value): + """+ + Header folding is controlled by the refold_source policy setting. A + value is considered to be a 'source value' if and only if it does not + have a 'name' attribute (having a 'name' attribute means it is a header + object of some sort). If a source value needs to be refolded according + to the policy, it is converted into a custom header object by passing + the name and the value with any linesep characters removed to the + header_factory method. Folding of a custom header object is done by + calling its fold method with the current policy. + + Source values are split into lines using splitlines. If the value is + not to be refolded, the lines are rejoined using the linesep from the + policy and returned. The exception is lines containing non-ascii + binary data. In that case the value is refolded regardless of the + refold_source setting, which causes the binary data to be CTE encoded + using the unknown-8bit charset. + + """ + return self._fold(name, value, refold_binary=True) + + def fold_binary(self, name, value): + """+ + The same as fold if cte_type is 7bit, except that the returned value is + bytes. + + If cte_type is 8bit, non-ASCII binary data is converted back into + bytes. Headers with binary data are not refolded, regardless of the + refold_header setting, since there is no way to know whether the binary + data consists of single byte characters or multibyte characters. + + """ + folded = self._fold(name, value, refold_binary=self.cte_type=='7bit') + return folded.encode('ascii', 'surrogateescape') + + def _fold(self, name, value, refold_binary=False): + if hasattr(value, 'name'): + return value.fold(policy=self) + maxlen = self.max_line_length if self.max_line_length else float('inf') + lines = value.splitlines() + refold = (self.refold_source == 'all' or + self.refold_source == 'long' and + (len(lines[0])+len(name)+2 > maxlen or + any(len(x) > maxlen for x in lines[1:]))) + if refold or refold_binary and _has_surrogates(value): + return self.header_factory(name, ''.join(lines)).fold(policy=self) + return name + ': ' + self.linesep.join(lines) + self.linesep + + +default = EmailPolicy() +# Make the default policy use the class default header_factory +del default.header_factory strict = default.clone(raise_on_defect=True) SMTP = default.clone(linesep='\r\n') HTTP = default.clone(linesep='\r\n', max_line_length=None) diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -62,6 +62,13 @@ _has_surrogates = re.compile( '([^\ud800-\udbff]|\A)[\udc00-\udfff]([^\udc00-\udfff]|\Z)').search +# How to deal with a string containing bytes before handing it to the +# application through the 'normal' interface. +def _sanitize(string): + # Turn any escaped bytes into unicode 'unknown' char. + original_bytes = string.encode('ascii', 'surrogateescape') + return original_bytes.decode('ascii', 'replace') + # Helpers diff --git a/Lib/test/test_email/__init__.py b/Lib/test/test_email/__init__.py --- a/Lib/test/test_email/__init__.py +++ b/Lib/test/test_email/__init__.py @@ -65,3 +65,9 @@ def assertBytesEqual(self, first, second, msg): """Our byte strings are really encoded strings; improve diff output""" self.assertEqual(self._bytes_repr(first), self._bytes_repr(second)) + + def assertDefectsEqual(self, actual, expected): + self.assertEqual(len(actual), len(expected), actual) + for i in range(len(actual)): + self.assertIsInstance(actual[i], expected[i], + 'item {}'.format(i)) diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test__encoded_words.py @@ -0,0 +1,187 @@ +import unittest +from email import _encoded_words as _ew +from email import errors +from test.test_email import TestEmailBase + + +class TestDecodeQ(TestEmailBase): + + def _test(self, source, ex_result, ex_defects=[]): + result, defects = _ew.decode_q(source) + self.assertEqual(result, ex_result) + self.assertDefectsEqual(defects, ex_defects) + + def test_no_encoded(self): + self._test(b'foobar', b'foobar') + + def test_spaces(self): + self._test(b'foo=20bar=20', b'foo bar ') + self._test(b'foo_bar_', b'foo bar ') + + def test_run_of_encoded(self): + self._test(b'foo=20=20=21=2Cbar', b'foo !,bar') + + +class TestDecodeB(TestEmailBase): + + def _test(self, source, ex_result, ex_defects=[]): + result, defects = _ew.decode_b(source) + self.assertEqual(result, ex_result) + self.assertDefectsEqual(defects, ex_defects) + + def test_simple(self): + self._test(b'Zm9v', b'foo') + + def test_missing_padding(self): + self._test(b'dmk', b'vi', [errors.InvalidBase64PaddingDefect]) + + def test_invalid_character(self): + self._test(b'dm\x01k===', b'vi', [errors.InvalidBase64CharactersDefect]) + + def test_invalid_character_and_bad_padding(self): + self._test(b'dm\x01k', b'vi', [errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect]) + + +class TestDecode(TestEmailBase): + + def test_wrong_format_input_raises(self): + with self.assertRaises(ValueError): + _ew.decode('=?badone?=') + with self.assertRaises(ValueError): + _ew.decode('=?') + with self.assertRaises(ValueError): + _ew.decode('') + + def _test(self, source, result, charset='us-ascii', lang='', defects=[]): + res, char, l, d = _ew.decode(source) + self.assertEqual(res, result) + self.assertEqual(char, charset) + self.assertEqual(l, lang) + self.assertDefectsEqual(d, defects) + + def test_simple_q(self): + self._test('=?us-ascii?q?foo?=', 'foo') + + def test_simple_b(self): + self._test('=?us-ascii?b?dmk=?=', 'vi') + + def test_q_case_ignored(self): + self._test('=?us-ascii?Q?foo?=', 'foo') + + def test_b_case_ignored(self): + self._test('=?us-ascii?B?dmk=?=', 'vi') + + def test_non_trivial_q(self): + self._test('=?latin-1?q?=20F=fcr=20Elise=20?=', ' F?r Elise ', 'latin-1') + + def test_q_escpaed_bytes_preserved(self): + self._test(b'=?us-ascii?q?=20\xACfoo?='.decode('us-ascii', + 'surrogateescape'), + ' \uDCACfoo', + defects = [errors.UndecodableBytesDefect]) + + def test_b_undecodable_bytes_ignored_with_defect(self): + self._test(b'=?us-ascii?b?dm\xACk?='.decode('us-ascii', + 'surrogateescape'), + 'vi', + defects = [ + errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect]) + + def test_b_invalid_bytes_ignored_with_defect(self): + self._test('=?us-ascii?b?dm\x01k===?=', + 'vi', + defects = [errors.InvalidBase64CharactersDefect]) + + def test_b_invalid_bytes_incorrect_padding(self): + self._test('=?us-ascii?b?dm\x01k?=', + 'vi', + defects = [ + errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect]) + + def test_b_padding_defect(self): + self._test('=?us-ascii?b?dmk?=', + 'vi', + defects = [errors.InvalidBase64PaddingDefect]) + + def test_nonnull_lang(self): + self._test('=?us-ascii*jive?q?test?=', 'test', lang='jive') + + def test_unknown_8bit_charset(self): + self._test('=?unknown-8bit?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'unknown-8bit', + defects = []) + + def test_unknown_charset(self): + self._test('=?foobar?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'foobar', + # XXX Should this be a new Defect instead? + defects = [errors.CharsetError]) + + +class TestEncodeQ(TestEmailBase): + + def _test(self, src, expected): + self.assertEqual(_ew.encode_q(src), expected) + + def test_all_safe(self): + self._test(b'foobar', 'foobar') + + def test_spaces(self): + self._test(b'foo bar ', 'foo_bar_') + + def test_run_of_encodables(self): + self._test(b'foo ,,bar', 'foo__=2C=2Cbar') + + +class TestEncodeB(TestEmailBase): + + def test_simple(self): + self.assertEqual(_ew.encode_b(b'foo'), 'Zm9v') + + def test_padding(self): + self.assertEqual(_ew.encode_b(b'vi'), 'dmk=') + + +class TestEncode(TestEmailBase): + + def test_q(self): + self.assertEqual(_ew.encode('foo', 'utf-8', 'q'), '=?utf-8?q?foo?=') + + def test_b(self): + self.assertEqual(_ew.encode('foo', 'utf-8', 'b'), '=?utf-8?b?Zm9v?=') + + def test_auto_q(self): + self.assertEqual(_ew.encode('foo', 'utf-8'), '=?utf-8?q?foo?=') + + def test_auto_q_if_short_mostly_safe(self): + self.assertEqual(_ew.encode('vi.', 'utf-8'), '=?utf-8?q?vi=2E?=') + + def test_auto_b_if_enough_unsafe(self): + self.assertEqual(_ew.encode('.....', 'utf-8'), '=?utf-8?b?Li4uLi4=?=') + + def test_auto_b_if_long_unsafe(self): + self.assertEqual(_ew.encode('vi.vi.vi.vi.vi.', 'utf-8'), + '=?utf-8?b?dmkudmkudmkudmkudmku?=') + + def test_auto_q_if_long_mostly_safe(self): + self.assertEqual(_ew.encode('vi vi vi.vi ', 'utf-8'), + '=?utf-8?q?vi_vi_vi=2Evi_?=') + + def test_utf8_default(self): + self.assertEqual(_ew.encode('foo'), '=?utf-8?q?foo?=') + + def test_lang(self): + self.assertEqual(_ew.encode('foo', lang='jive'), '=?utf-8*jive?q?foo?=') + + def test_unknown_8bit(self): + self.assertEqual(_ew.encode('foo\uDCACbar', charset='unknown-8bit'), + '=?unknown-8bit?q?foo=ACbar?=') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test__header_value_parser.py @@ -0,0 +1,2466 @@ +import string +import unittest +from email import _header_value_parser as parser +from email import errors +from email import policy +from test.test_email import TestEmailBase + +class TestTokens(TestEmailBase): + + # EWWhiteSpaceTerminal + + def test_EWWhiteSpaceTerminal(self): + x = parser.EWWhiteSpaceTerminal(' \t', 'fws') + self.assertEqual(x, ' \t') + self.assertEqual(str(x), '') + self.assertEqual(x.value, '') + self.assertEqual(x.encoded, ' \t') + + # UnstructuredTokenList + + def test_undecodable_bytes_error_preserved(self): + badstr = b"le pouf c\xaflebre".decode('ascii', 'surrogateescape') + unst = parser.get_unstructured(badstr) + self.assertDefectsEqual(unst.all_defects, [errors.UndecodableBytesDefect]) + parts = list(unst.parts) + self.assertDefectsEqual(parts[0].all_defects, []) + self.assertDefectsEqual(parts[1].all_defects, []) + self.assertDefectsEqual(parts[2].all_defects, [errors.UndecodableBytesDefect]) + + +class TestParser(TestEmailBase): + + # _wsp_splitter + + rfc_printable_ascii = bytes(range(33, 127)).decode('ascii') + rfc_atext_chars = (string.ascii_letters + string.digits + + "!#$%&\'*+-/=?^_`{}|~") + rfc_dtext_chars = rfc_printable_ascii.translate(str.maketrans('','',r'\[]')) + + def test__wsp_splitter_one_word(self): + self.assertEqual(parser._wsp_splitter('foo', 1), ['foo']) + + def test__wsp_splitter_two_words(self): + self.assertEqual(parser._wsp_splitter('foo def', 1), + ['foo', ' ', 'def']) + + def test__wsp_splitter_ws_runs(self): + self.assertEqual(parser._wsp_splitter('foo \t def jik', 1), + ['foo', ' \t ', 'def jik']) + + + # test harness + + def _test_get_x(self, method, input, string, value, defects, + remainder, comments=None): + token, rest = method(input) + self.assertEqual(str(token), string) + self.assertEqual(token.value, value) + self.assertDefectsEqual(token.all_defects, defects) + self.assertEqual(rest, remainder) + if comments is not None: + self.assertEqual(token.comments, comments) + return token + + # get_fws + + def test_get_fws_only(self): + fws = self._test_get_x(parser.get_fws, ' \t ', ' \t ', ' ', [], '') + self.assertEqual(fws.token_type, 'fws') + + def test_get_fws_space(self): + self._test_get_x(parser.get_fws, ' foo', ' ', ' ', [], 'foo') + + def test_get_fws_ws_run(self): + self._test_get_x(parser.get_fws, ' \t foo ', ' \t ', ' ', [], 'foo ') + + # get_encoded_word + + def test_get_encoded_word_missing_start_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('abc') + + def test_get_encoded_word_missing_end_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('=?abc') + + def test_get_encoded_word_missing_middle_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_encoded_word('=?abc?=') + + def test_get_encoded_word_valid_ew(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?this_is_a_test?= bird', + 'this is a test', + 'this is a test', + [], + ' bird') + + def test_get_encoded_word_internal_spaces(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?this is a test?= bird', + 'this is a test', + 'this is a test', + [errors.InvalidHeaderDefect], + ' bird') + + def test_get_encoded_word_gets_first(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first?= =?utf-8?q?second?=', + 'first', + 'first', + [], + ' =?utf-8?q?second?=') + + def test_get_encoded_word_gets_first_even_if_no_space(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first?==?utf-8?q?second?=', + 'first', + 'first', + [], + '=?utf-8?q?second?=') + + def test_get_encoded_word_sets_extra_attributes(self): + ew = self._test_get_x(parser.get_encoded_word, + '=?us-ascii*jive?q?first_second?=', + 'first second', + 'first second', + [], + '') + self.assertEqual(ew.encoded, '=?us-ascii*jive?q?first_second?=') + self.assertEqual(ew.charset, 'us-ascii') + self.assertEqual(ew.lang, 'jive') + + def test_get_encoded_word_lang_default_is_blank(self): + ew = self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first_second?=', + 'first second', + 'first second', + [], + '') + self.assertEqual(ew.encoded, '=?us-ascii?q?first_second?=') + self.assertEqual(ew.charset, 'us-ascii') + self.assertEqual(ew.lang, '') + + def test_get_encoded_word_non_printable_defect(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?first\x02second?=', + 'first\x02second', + 'first\x02second', + [errors.NonPrintableDefect], + '') + + def test_get_encoded_word_leading_internal_space(self): + self._test_get_x(parser.get_encoded_word, + '=?us-ascii?q?=20foo?=', + ' foo', + ' foo', + [], + '') + + # get_unstructured + + def _get_unst(self, value): + token = parser.get_unstructured(value) + return token, '' + + def test_get_unstructured_null(self): + self._test_get_x(self._get_unst, '', '', '', [], '') + + def test_get_unstructured_one_word(self): + self._test_get_x(self._get_unst, 'foo', 'foo', 'foo', [], '') + + def test_get_unstructured_normal_phrase(self): + self._test_get_x(self._get_unst, 'foo bar bird', + 'foo bar bird', + 'foo bar bird', + [], + '') + + def test_get_unstructured_normal_phrase_with_whitespace(self): + self._test_get_x(self._get_unst, 'foo \t bar bird', + 'foo \t bar bird', + 'foo bar bird', + [], + '') + + def test_get_unstructured_leading_whitespace(self): + self._test_get_x(self._get_unst, ' foo bar', + ' foo bar', + ' foo bar', + [], + '') + + def test_get_unstructured_trailing_whitespace(self): + self._test_get_x(self._get_unst, 'foo bar ', + 'foo bar ', + 'foo bar ', + [], + '') + + def test_get_unstructured_leading_and_trailing_whitespace(self): + self._test_get_x(self._get_unst, ' foo bar ', + ' foo bar ', + ' foo bar ', + [], + '') + + def test_get_unstructured_one_valid_ew_no_ws(self): + self._test_get_x(self._get_unst, '=?us-ascii?q?bar?=', + 'bar', + 'bar', + [], + '') + + def test_get_unstructured_one_ew_trailing_ws(self): + self._test_get_x(self._get_unst, '=?us-ascii?q?bar?= ', + 'bar ', + 'bar ', + [], + '') + + def test_get_unstructured_one_valid_ew_trailing_text(self): + self._test_get_x(self._get_unst, '=?us-ascii?q?bar?= bird', + 'bar bird', + 'bar bird', + [], + '') + + def test_get_unstructured_phrase_with_ew_in_middle_of_text(self): + self._test_get_x(self._get_unst, 'foo =?us-ascii?q?bar?= bird', + 'foo bar bird', + 'foo bar bird', + [], + '') + + def test_get_unstructured_phrase_with_two_ew(self): + self._test_get_x(self._get_unst, + 'foo =?us-ascii?q?bar?= =?us-ascii?q?bird?=', + 'foo barbird', + 'foo barbird', + [], + '') + + def test_get_unstructured_phrase_with_two_ew_trailing_ws(self): + self._test_get_x(self._get_unst, + 'foo =?us-ascii?q?bar?= =?us-ascii?q?bird?= ', + 'foo barbird ', + 'foo barbird ', + [], + '') + + def test_get_unstructured_phrase_with_ew_with_leading_ws(self): + self._test_get_x(self._get_unst, + ' =?us-ascii?q?bar?=', + ' bar', + ' bar', + [], + '') + + def test_get_unstructured_phrase_with_two_ew_extra_ws(self): + self._test_get_x(self._get_unst, + 'foo =?us-ascii?q?bar?= \t =?us-ascii?q?bird?=', + 'foo barbird', + 'foo barbird', + [], + '') + + def test_get_unstructured_two_ew_extra_ws_trailing_text(self): + self._test_get_x(self._get_unst, + '=?us-ascii?q?test?= =?us-ascii?q?foo?= val', + 'testfoo val', + 'testfoo val', + [], + '') + + def test_get_unstructured_ew_with_internal_ws(self): + self._test_get_x(self._get_unst, + '=?iso-8859-1?q?hello=20world?=', + 'hello world', + 'hello world', + [], + '') + + def test_get_unstructured_ew_with_internal_leading_ws(self): + self._test_get_x(self._get_unst, + ' =?us-ascii?q?=20test?= =?us-ascii?q?=20foo?= val', + ' test foo val', + ' test foo val', + [], + '') + + def test_get_unstructured_invaild_ew(self): + self._test_get_x(self._get_unst, + '=?test val', + '=?test val', + '=?test val', + [], + '') + + def test_get_unstructured_undecodable_bytes(self): + self._test_get_x(self._get_unst, + b'test \xACfoo val'.decode('ascii', 'surrogateescape'), + 'test \uDCACfoo val', + 'test \uDCACfoo val', + [errors.UndecodableBytesDefect], + '') + + def test_get_unstructured_undecodable_bytes_in_EW(self): + self._test_get_x(self._get_unst, + (b'=?us-ascii?q?=20test?= =?us-ascii?q?=20\xACfoo?=' + b' val').decode('ascii', 'surrogateescape'), + ' test \uDCACfoo val', + ' test \uDCACfoo val', + [errors.UndecodableBytesDefect]*2, + '') + + def test_get_unstructured_missing_base64_padding(self): + self._test_get_x(self._get_unst, + '=?utf-8?b?dmk?=', + 'vi', + 'vi', + [errors.InvalidBase64PaddingDefect], + '') + + def test_get_unstructured_invalid_base64_character(self): + self._test_get_x(self._get_unst, + '=?utf-8?b?dm\x01k===?=', + 'vi', + 'vi', + [errors.InvalidBase64CharactersDefect], + '') + + def test_get_unstructured_invalid_base64_character_and_bad_padding(self): + self._test_get_x(self._get_unst, + '=?utf-8?b?dm\x01k?=', + 'vi', + 'vi', + [errors.InvalidBase64CharactersDefect, + errors.InvalidBase64PaddingDefect], + '') + + def test_get_unstructured_no_whitespace_between_ews(self): + self._test_get_x(self._get_unst, + '=?utf-8?q?foo?==?utf-8?q?bar?=', + 'foobar', + 'foobar', + [errors.InvalidHeaderDefect], + '') + + # get_qp_ctext + + def test_get_qp_ctext_only(self): + ptext = self._test_get_x(parser.get_qp_ctext, + 'foobar', 'foobar', ' ', [], '') + self.assertEqual(ptext.token_type, 'ptext') + + def test_get_qp_ctext_all_printables(self): + with_qp = self.rfc_printable_ascii.replace('\\', '\\\\') + with_qp = with_qp. replace('(', r'\(') + with_qp = with_qp.replace(')', r'\)') + ptext = self._test_get_x(parser.get_qp_ctext, + with_qp, self.rfc_printable_ascii, ' ', [], '') + + def test_get_qp_ctext_two_words_gets_first(self): + self._test_get_x(parser.get_qp_ctext, + 'foo de', 'foo', ' ', [], ' de') + + def test_get_qp_ctext_following_wsp_preserved(self): + self._test_get_x(parser.get_qp_ctext, + 'foo \t\tde', 'foo', ' ', [], ' \t\tde') + + def test_get_qp_ctext_up_to_close_paren_only(self): + self._test_get_x(parser.get_qp_ctext, + 'foo)', 'foo', ' ', [], ')') + + def test_get_qp_ctext_wsp_before_close_paren_preserved(self): + self._test_get_x(parser.get_qp_ctext, + 'foo )', 'foo', ' ', [], ' )') + + def test_get_qp_ctext_close_paren_mid_word(self): + self._test_get_x(parser.get_qp_ctext, + 'foo)bar', 'foo', ' ', [], ')bar') + + def test_get_qp_ctext_up_to_open_paren_only(self): + self._test_get_x(parser.get_qp_ctext, + 'foo(', 'foo', ' ', [], '(') + + def test_get_qp_ctext_wsp_before_open_paren_preserved(self): + self._test_get_x(parser.get_qp_ctext, + 'foo (', 'foo', ' ', [], ' (') + + def test_get_qp_ctext_open_paren_mid_word(self): + self._test_get_x(parser.get_qp_ctext, + 'foo(bar', 'foo', ' ', [], '(bar') + + def test_get_qp_ctext_non_printables(self): + ptext = self._test_get_x(parser.get_qp_ctext, + 'foo\x00bar)', 'foo\x00bar', ' ', + [errors.NonPrintableDefect], ')') + self.assertEqual(ptext.defects[0].non_printables[0], '\x00') + + # get_qcontent + + def test_get_qcontent_only(self): + ptext = self._test_get_x(parser.get_qcontent, + 'foobar', 'foobar', 'foobar', [], '') + self.assertEqual(ptext.token_type, 'ptext') + + def test_get_qcontent_all_printables(self): + with_qp = self.rfc_printable_ascii.replace('\\', '\\\\') + with_qp = with_qp. replace('"', r'\"') + ptext = self._test_get_x(parser.get_qcontent, with_qp, + self.rfc_printable_ascii, + self.rfc_printable_ascii, [], '') + + def test_get_qcontent_two_words_gets_first(self): + self._test_get_x(parser.get_qcontent, + 'foo de', 'foo', 'foo', [], ' de') + + def test_get_qcontent_following_wsp_preserved(self): + self._test_get_x(parser.get_qcontent, + 'foo \t\tde', 'foo', 'foo', [], ' \t\tde') + + def test_get_qcontent_up_to_dquote_only(self): + self._test_get_x(parser.get_qcontent, + 'foo"', 'foo', 'foo', [], '"') + + def test_get_qcontent_wsp_before_close_paren_preserved(self): + self._test_get_x(parser.get_qcontent, + 'foo "', 'foo', 'foo', [], ' "') + + def test_get_qcontent_close_paren_mid_word(self): + self._test_get_x(parser.get_qcontent, + 'foo"bar', 'foo', 'foo', [], '"bar') + + def test_get_qcontent_non_printables(self): + ptext = self._test_get_x(parser.get_qcontent, + 'foo\x00fg"', 'foo\x00fg', 'foo\x00fg', + [errors.NonPrintableDefect], '"') + self.assertEqual(ptext.defects[0].non_printables[0], '\x00') + + # get_atext + + def test_get_atext_only(self): + atext = self._test_get_x(parser.get_atext, + 'foobar', 'foobar', 'foobar', [], '') + self.assertEqual(atext.token_type, 'atext') + + def test_get_atext_all_atext(self): + atext = self._test_get_x(parser.get_atext, self.rfc_atext_chars, + self.rfc_atext_chars, + self.rfc_atext_chars, [], '') + + def test_get_atext_two_words_gets_first(self): + self._test_get_x(parser.get_atext, + 'foo bar', 'foo', 'foo', [], ' bar') + + def test_get_atext_following_wsp_preserved(self): + self._test_get_x(parser.get_atext, + 'foo \t\tbar', 'foo', 'foo', [], ' \t\tbar') + + def test_get_atext_up_to_special(self): + self._test_get_x(parser.get_atext, + 'foo at bar', 'foo', 'foo', [], '@bar') + + def test_get_atext_non_printables(self): + atext = self._test_get_x(parser.get_atext, + 'foo\x00bar(', 'foo\x00bar', 'foo\x00bar', + [errors.NonPrintableDefect], '(') + self.assertEqual(atext.defects[0].non_printables[0], '\x00') + + # get_bare_quoted_string + + def test_get_bare_quoted_string_only(self): + bqs = self._test_get_x(parser.get_bare_quoted_string, + '"foo"', '"foo"', 'foo', [], '') + self.assertEqual(bqs.token_type, 'bare-quoted-string') + + def test_get_bare_quoted_string_must_start_with_dquote(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_bare_quoted_string('foo"') + with self.assertRaises(errors.HeaderParseError): + parser.get_bare_quoted_string(' "foo"') + + def test_get_bare_quoted_string_following_wsp_preserved(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo"\t bar', '"foo"', 'foo', [], '\t bar') + + def test_get_bare_quoted_string_multiple_words(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo bar moo"', '"foo bar moo"', 'foo bar moo', [], '') + + def test_get_bare_quoted_string_multiple_words_wsp_preserved(self): + self._test_get_x(parser.get_bare_quoted_string, + '" foo moo\t"', '" foo moo\t"', ' foo moo\t', [], '') + + def test_get_bare_quoted_string_end_dquote_mid_word(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo"bar', '"foo"', 'foo', [], 'bar') + + def test_get_bare_quoted_string_quoted_dquote(self): + self._test_get_x(parser.get_bare_quoted_string, + r'"foo\"in"a', r'"foo\"in"', 'foo"in', [], 'a') + + def test_get_bare_quoted_string_non_printables(self): + self._test_get_x(parser.get_bare_quoted_string, + '"a\x01a"', '"a\x01a"', 'a\x01a', + [errors.NonPrintableDefect], '') + + def test_get_bare_quoted_string_no_end_dquote(self): + self._test_get_x(parser.get_bare_quoted_string, + '"foo', '"foo"', 'foo', + [errors.InvalidHeaderDefect], '') + self._test_get_x(parser.get_bare_quoted_string, + '"foo ', '"foo "', 'foo ', + [errors.InvalidHeaderDefect], '') + + def test_get_bare_quoted_string_empty_quotes(self): + self._test_get_x(parser.get_bare_quoted_string, + '""', '""', '', [], '') + + # get_comment + + def test_get_comment_only(self): + comment = self._test_get_x(parser.get_comment, + '(comment)', '(comment)', ' ', [], '', ['comment']) + self.assertEqual(comment.token_type, 'comment') + + def test_get_comment_must_start_with_paren(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_comment('foo"') + with self.assertRaises(errors.HeaderParseError): + parser.get_comment(' (foo"') + + def test_get_comment_following_wsp_preserved(self): + self._test_get_x(parser.get_comment, + '(comment) \t', '(comment)', ' ', [], ' \t', ['comment']) + + def test_get_comment_multiple_words(self): + self._test_get_x(parser.get_comment, + '(foo bar) \t', '(foo bar)', ' ', [], ' \t', ['foo bar']) + + def test_get_comment_multiple_words_wsp_preserved(self): + self._test_get_x(parser.get_comment, + '( foo bar\t ) \t', '( foo bar\t )', ' ', [], ' \t', + [' foo bar\t ']) + + def test_get_comment_end_paren_mid_word(self): + self._test_get_x(parser.get_comment, + '(foo)bar', '(foo)', ' ', [], 'bar', ['foo']) + + def test_get_comment_quoted_parens(self): + self._test_get_x(parser.get_comment, + '(foo\) \(\)bar)', '(foo\) \(\)bar)', ' ', [], '', ['foo) ()bar']) + + def test_get_comment_non_printable(self): + self._test_get_x(parser.get_comment, + '(foo\x7Fbar)', '(foo\x7Fbar)', ' ', + [errors.NonPrintableDefect], '', ['foo\x7Fbar']) + + def test_get_comment_no_end_paren(self): + self._test_get_x(parser.get_comment, + '(foo bar', '(foo bar)', ' ', + [errors.InvalidHeaderDefect], '', ['foo bar']) + self._test_get_x(parser.get_comment, + '(foo bar ', '(foo bar )', ' ', + [errors.InvalidHeaderDefect], '', ['foo bar ']) + + def test_get_comment_nested_comment(self): + comment = self._test_get_x(parser.get_comment, + '(foo(bar))', '(foo(bar))', ' ', [], '', ['foo(bar)']) + self.assertEqual(comment[1].content, 'bar') + + def test_get_comment_nested_comment_wsp(self): + comment = self._test_get_x(parser.get_comment, + '(foo ( bar ) )', '(foo ( bar ) )', ' ', [], '', ['foo ( bar ) ']) + self.assertEqual(comment[2].content, ' bar ') + + def test_get_comment_empty_comment(self): + self._test_get_x(parser.get_comment, + '()', '()', ' ', [], '', ['']) + + def test_get_comment_multiple_nesting(self): + comment = self._test_get_x(parser.get_comment, + '(((((foo)))))', '(((((foo)))))', ' ', [], '', ['((((foo))))']) + for i in range(4, 0, -1): + self.assertEqual(comment[0].content, '('*(i-1)+'foo'+')'*(i-1)) + comment = comment[0] + self.assertEqual(comment.content, 'foo') + + def test_get_comment_missing_end_of_nesting(self): + self._test_get_x(parser.get_comment, + '(((((foo)))', '(((((foo)))))', ' ', + [errors.InvalidHeaderDefect]*2, '', ['((((foo))))']) + + def test_get_comment_qs_in_nested_comment(self): + comment = self._test_get_x(parser.get_comment, + '(foo (b\)))', '(foo (b\)))', ' ', [], '', ['foo (b\))']) + self.assertEqual(comment[2].content, 'b)') + + # get_cfws + + def test_get_cfws_only_ws(self): + cfws = self._test_get_x(parser.get_cfws, + ' \t \t', ' \t \t', ' ', [], '', []) + self.assertEqual(cfws.token_type, 'cfws') + + def test_get_cfws_only_comment(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo)', '(foo)', ' ', [], '', ['foo']) + self.assertEqual(cfws[0].content, 'foo') + + def test_get_cfws_only_mixed(self): + cfws = self._test_get_x(parser.get_cfws, + ' (foo ) ( bar) ', ' (foo ) ( bar) ', ' ', [], '', + ['foo ', ' bar']) + self.assertEqual(cfws[1].content, 'foo ') + self.assertEqual(cfws[3].content, ' bar') + + def test_get_cfws_ends_at_non_leader(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo) bar', '(foo) ', ' ', [], 'bar', ['foo']) + self.assertEqual(cfws[0].content, 'foo') + + def test_get_cfws_ends_at_non_printable(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo) \x07', '(foo) ', ' ', [], '\x07', ['foo']) + self.assertEqual(cfws[0].content, 'foo') + + def test_get_cfws_non_printable_in_comment(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo \x07) "test"', '(foo \x07) ', ' ', + [errors.NonPrintableDefect], '"test"', ['foo \x07']) + self.assertEqual(cfws[0].content, 'foo \x07') + + def test_get_cfws_header_ends_in_comment(self): + cfws = self._test_get_x(parser.get_cfws, + ' (foo ', ' (foo )', ' ', + [errors.InvalidHeaderDefect], '', ['foo ']) + self.assertEqual(cfws[1].content, 'foo ') + + def test_get_cfws_multiple_nested_comments(self): + cfws = self._test_get_x(parser.get_cfws, + '(foo (bar)) ((a)(a))', '(foo (bar)) ((a)(a))', ' ', [], + '', ['foo (bar)', '(a)(a)']) + self.assertEqual(cfws[0].comments, ['foo (bar)']) + self.assertEqual(cfws[2].comments, ['(a)(a)']) + + # get_quoted_string + + def test_get_quoted_string_only(self): + qs = self._test_get_x(parser.get_quoted_string, + '"bob"', '"bob"', 'bob', [], '') + self.assertEqual(qs.token_type, 'quoted-string') + self.assertEqual(qs.quoted_value, '"bob"') + self.assertEqual(qs.content, 'bob') + + def test_get_quoted_string_with_wsp(self): + qs = self._test_get_x(parser.get_quoted_string, + '\t "bob" ', '\t "bob" ', ' bob ', [], '') + self.assertEqual(qs.quoted_value, ' "bob" ') + self.assertEqual(qs.content, 'bob') + + def test_get_quoted_string_with_comments_and_wsp(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (foo) "bob"(bar)', ' (foo) "bob"(bar)', ' bob ', [], '') + self.assertEqual(qs[0][1].content, 'foo') + self.assertEqual(qs[2][0].content, 'bar') + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + def test_get_quoted_string_with_multiple_comments(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (foo) (bar) "bob"(bird)', ' (foo) (bar) "bob"(bird)', ' bob ', + [], '') + self.assertEqual(qs[0].comments, ['foo', 'bar']) + self.assertEqual(qs[2].comments, ['bird']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + def test_get_quoted_string_non_printable_in_comment(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (\x0A) "bob"', ' (\x0A) "bob"', ' bob', + [errors.NonPrintableDefect], '') + self.assertEqual(qs[0].comments, ['\x0A']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob"') + + def test_get_quoted_string_non_printable_in_qcontent(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "a\x0B"', ' (a) "a\x0B"', ' a\x0B', + [errors.NonPrintableDefect], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs.content, 'a\x0B') + self.assertEqual(qs.quoted_value, ' "a\x0B"') + + def test_get_quoted_string_internal_ws(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "foo bar "', ' (a) "foo bar "', ' foo bar ', + [], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs.content, 'foo bar ') + self.assertEqual(qs.quoted_value, ' "foo bar "') + + def test_get_quoted_string_header_ends_in_comment(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "bob" (a', ' (a) "bob" (a)', ' bob ', + [errors.InvalidHeaderDefect], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs[2].comments, ['a']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + def test_get_quoted_string_header_ends_in_qcontent(self): + qs = self._test_get_x(parser.get_quoted_string, + ' (a) "bob', ' (a) "bob"', ' bob', + [errors.InvalidHeaderDefect], '') + self.assertEqual(qs[0].comments, ['a']) + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob"') + + def test_get_quoted_string_no_quoted_string(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_quoted_string(' (ab) xyz') + + def test_get_quoted_string_qs_ends_at_noncfws(self): + qs = self._test_get_x(parser.get_quoted_string, + '\t "bob" fee', '\t "bob" ', ' bob ', [], 'fee') + self.assertEqual(qs.content, 'bob') + self.assertEqual(qs.quoted_value, ' "bob" ') + + # get_atom + + def test_get_atom_only(self): + atom = self._test_get_x(parser.get_atom, + 'bob', 'bob', 'bob', [], '') + self.assertEqual(atom.token_type, 'atom') + + def test_get_atom_with_wsp(self): + self._test_get_x(parser.get_atom, + '\t bob ', '\t bob ', ' bob ', [], '') + + def test_get_atom_with_comments_and_wsp(self): + atom = self._test_get_x(parser.get_atom, + ' (foo) bob(bar)', ' (foo) bob(bar)', ' bob ', [], '') + self.assertEqual(atom[0][1].content, 'foo') + self.assertEqual(atom[2][0].content, 'bar') + + def test_get_atom_with_multiple_comments(self): + atom = self._test_get_x(parser.get_atom, + ' (foo) (bar) bob(bird)', ' (foo) (bar) bob(bird)', ' bob ', + [], '') + self.assertEqual(atom[0].comments, ['foo', 'bar']) + self.assertEqual(atom[2].comments, ['bird']) + + def test_get_atom_non_printable_in_comment(self): + atom = self._test_get_x(parser.get_atom, + ' (\x0A) bob', ' (\x0A) bob', ' bob', + [errors.NonPrintableDefect], '') + self.assertEqual(atom[0].comments, ['\x0A']) + + def test_get_atom_non_printable_in_atext(self): + atom = self._test_get_x(parser.get_atom, + ' (a) a\x0B', ' (a) a\x0B', ' a\x0B', + [errors.NonPrintableDefect], '') + self.assertEqual(atom[0].comments, ['a']) + + def test_get_atom_header_ends_in_comment(self): + atom = self._test_get_x(parser.get_atom, + ' (a) bob (a', ' (a) bob (a)', ' bob ', + [errors.InvalidHeaderDefect], '') + self.assertEqual(atom[0].comments, ['a']) + self.assertEqual(atom[2].comments, ['a']) + + def test_get_atom_no_atom(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_atom(' (ab) ') + + def test_get_atom_no_atom_before_special(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_atom(' (ab) @') + + def test_get_atom_atom_ends_at_special(self): + atom = self._test_get_x(parser.get_atom, + ' (foo) bob(bar) @bang', ' (foo) bob(bar) ', ' bob ', [], '@bang') + self.assertEqual(atom[0].comments, ['foo']) + self.assertEqual(atom[2].comments, ['bar']) + + def test_get_atom_atom_ends_at_noncfws(self): + atom = self._test_get_x(parser.get_atom, + 'bob fred', 'bob ', 'bob ', [], 'fred') + + # get_dot_atom_text + + def test_get_dot_atom_text(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo.bar.bang', 'foo.bar.bang', 'foo.bar.bang', [], '') + self.assertEqual(dot_atom_text.token_type, 'dot-atom-text') + self.assertEqual(len(dot_atom_text), 5) + + def test_get_dot_atom_text_lone_atom_is_valid(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo', 'foo', 'foo', [], '') + + def test_get_dot_atom_text_raises_on_leading_dot(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('.foo.bar') + + def test_get_dot_atom_text_raises_on_trailing_dot(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('foo.bar.') + + def test_get_dot_atom_text_raises_on_leading_non_atext(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text(' foo.bar') + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('@foo.bar') + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom_text('"foo.bar"') + + def test_get_dot_atom_text_trailing_text_preserved(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo at bar', 'foo', 'foo', [], '@bar') + + def test_get_dot_atom_text_trailing_ws_preserved(self): + dot_atom_text = self._test_get_x(parser.get_dot_atom_text, + 'foo .bar', 'foo', 'foo', [], ' .bar') + + # get_dot_atom + + def test_get_dot_atom_only(self): + dot_atom = self._test_get_x(parser.get_dot_atom, + 'foo.bar.bing', 'foo.bar.bing', 'foo.bar.bing', [], '') + self.assertEqual(dot_atom.token_type, 'dot-atom') + self.assertEqual(len(dot_atom), 1) + + def test_get_dot_atom_with_wsp(self): + self._test_get_x(parser.get_dot_atom, + '\t foo.bar.bing ', '\t foo.bar.bing ', ' foo.bar.bing ', [], '') + + def test_get_dot_atom_with_comments_and_wsp(self): + self._test_get_x(parser.get_dot_atom, + ' (sing) foo.bar.bing (here) ', ' (sing) foo.bar.bing (here) ', + ' foo.bar.bing ', [], '') + + def test_get_dot_atom_space_ends_dot_atom(self): + self._test_get_x(parser.get_dot_atom, + ' (sing) foo.bar .bing (here) ', ' (sing) foo.bar ', + ' foo.bar ', [], '.bing (here) ') + + def test_get_dot_atom_no_atom_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom(' (foo) ') + + def test_get_dot_atom_leading_dot_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom(' (foo) .bar') + + def test_get_dot_atom_two_dots_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom('bar..bang') + + def test_get_dot_atom_trailing_dot_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_dot_atom(' (foo) bar.bang. foo') + + # get_word (if this were black box we'd repeat all the qs/atom tests) + + def test_get_word_atom_yields_atom(self): + word = self._test_get_x(parser.get_word, + ' (foo) bar (bang) :ah', ' (foo) bar (bang) ', ' bar ', [], ':ah') + self.assertEqual(word.token_type, 'atom') + self.assertEqual(word[0].token_type, 'cfws') + + def test_get_word_qs_yields_qs(self): + word = self._test_get_x(parser.get_word, + '"bar " (bang) ah', '"bar " (bang) ', 'bar ', [], 'ah') + self.assertEqual(word.token_type, 'quoted-string') + self.assertEqual(word[0].token_type, 'bare-quoted-string') + self.assertEqual(word[0].value, 'bar ') + self.assertEqual(word.content, 'bar ') + + def test_get_word_ends_at_dot(self): + self._test_get_x(parser.get_word, + 'foo.', 'foo', 'foo', [], '.') + + # get_phrase + + def test_get_phrase_simple(self): + phrase = self._test_get_x(parser.get_phrase, + '"Fred A. Johnson" is his name, oh.', + '"Fred A. Johnson" is his name', + 'Fred A. Johnson is his name', + [], + ', oh.') + self.assertEqual(phrase.token_type, 'phrase') + + def test_get_phrase_complex(self): + phrase = self._test_get_x(parser.get_phrase, + ' (A) bird (in (my|your)) "hand " is messy\t<>\t', + ' (A) bird (in (my|your)) "hand " is messy\t', + ' bird hand is messy ', + [], + '<>\t') + self.assertEqual(phrase[0][0].comments, ['A']) + self.assertEqual(phrase[0][2].comments, ['in (my|your)']) + + def test_get_phrase_obsolete(self): + phrase = self._test_get_x(parser.get_phrase, + 'Fred A.(weird).O Johnson', + 'Fred A.(weird).O Johnson', + 'Fred A. .O Johnson', + [errors.ObsoleteHeaderDefect]*3, + '') + self.assertEqual(len(phrase), 7) + self.assertEqual(phrase[3].comments, ['weird']) + + def test_get_phrase_pharse_must_start_with_word(self): + phrase = self._test_get_x(parser.get_phrase, + '(even weirder).name', + '(even weirder).name', + ' .name', + [errors.InvalidHeaderDefect] + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(len(phrase), 3) + self.assertEqual(phrase[0].comments, ['even weirder']) + + def test_get_phrase_ending_with_obsolete(self): + phrase = self._test_get_x(parser.get_phrase, + 'simple phrase.(with trailing comment):boo', + 'simple phrase.(with trailing comment)', + 'simple phrase. ', + [errors.ObsoleteHeaderDefect]*2, + ':boo') + self.assertEqual(len(phrase), 4) + self.assertEqual(phrase[3].comments, ['with trailing comment']) + + def get_phrase_cfws_only_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_phrase(' (foo) ') + + # get_local_part + + def test_get_local_part_simple(self): + local_part = self._test_get_x(parser.get_local_part, + 'dinsdale at python.org', 'dinsdale', 'dinsdale', [], '@python.org') + self.assertEqual(local_part.token_type, 'local-part') + self.assertEqual(local_part.local_part, 'dinsdale') + + def test_get_local_part_with_dot(self): + local_part = self._test_get_x(parser.get_local_part, + 'Fred.A.Johnson at python.org', + 'Fred.A.Johnson', + 'Fred.A.Johnson', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_with_whitespace(self): + local_part = self._test_get_x(parser.get_local_part, + ' Fred.A.Johnson @python.org', + ' Fred.A.Johnson ', + ' Fred.A.Johnson ', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_with_cfws(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo) Fred.A.Johnson (bar (bird)) @python.org', + ' (foo) Fred.A.Johnson (bar (bird)) ', + ' Fred.A.Johnson ', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + self.assertEqual(local_part[0][0].comments, ['foo']) + self.assertEqual(local_part[0][2].comments, ['bar (bird)']) + + def test_get_local_part_simple_quoted(self): + local_part = self._test_get_x(parser.get_local_part, + '"dinsdale"@python.org', '"dinsdale"', '"dinsdale"', [], '@python.org') + self.assertEqual(local_part.token_type, 'local-part') + self.assertEqual(local_part.local_part, 'dinsdale') + + def test_get_local_part_with_quoted_dot(self): + local_part = self._test_get_x(parser.get_local_part, + '"Fred.A.Johnson"@python.org', + '"Fred.A.Johnson"', + '"Fred.A.Johnson"', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_quoted_with_whitespace(self): + local_part = self._test_get_x(parser.get_local_part, + ' "Fred A. Johnson" @python.org', + ' "Fred A. Johnson" ', + ' "Fred A. Johnson" ', + [], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred A. Johnson') + + def test_get_local_part_quoted_with_cfws(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo) " Fred A. Johnson " (bar (bird)) @python.org', + ' (foo) " Fred A. Johnson " (bar (bird)) ', + ' " Fred A. Johnson " ', + [], + '@python.org') + self.assertEqual(local_part.local_part, ' Fred A. Johnson ') + self.assertEqual(local_part[0][0].comments, ['foo']) + self.assertEqual(local_part[0][2].comments, ['bar (bird)']) + + + def test_get_local_part_simple_obsolete(self): + local_part = self._test_get_x(parser.get_local_part, + 'Fred. A.Johnson at python.org', + 'Fred. A.Johnson', + 'Fred. A.Johnson', + [errors.ObsoleteHeaderDefect], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson') + + def test_get_local_part_complex_obsolete_1(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo )Fred (bar).(bird) A.(sheep)Johnson."and dogs "@python.org', + ' (foo )Fred (bar).(bird) A.(sheep)Johnson."and dogs "', + ' Fred . A. Johnson.and dogs ', + [errors.ObsoleteHeaderDefect], + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson.and dogs ') + + def test_get_local_part_complex_obsolete_invalid(self): + local_part = self._test_get_x(parser.get_local_part, + ' (foo )Fred (bar).(bird) A.(sheep)Johnson "and dogs"@python.org', + ' (foo )Fred (bar).(bird) A.(sheep)Johnson "and dogs"', + ' Fred . A. Johnson and dogs', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'Fred.A.Johnson and dogs') + + def test_get_local_part_no_part_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_local_part(' (foo) ') + + def test_get_local_part_special_instead_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_local_part(' (foo) @python.org') + + def test_get_local_part_trailing_dot(self): + local_part = self._test_get_x(parser.get_local_part, + ' borris. at python.org', + ' borris.', + ' borris.', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'borris.') + + def test_get_local_part_trailing_dot_with_ws(self): + local_part = self._test_get_x(parser.get_local_part, + ' borris. @python.org', + ' borris. ', + ' borris. ', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'borris.') + + def test_get_local_part_leading_dot(self): + local_part = self._test_get_x(parser.get_local_part, + '.borris at python.org', + '.borris', + '.borris', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, '.borris') + + def test_get_local_part_leading_dot_after_ws(self): + local_part = self._test_get_x(parser.get_local_part, + ' .borris at python.org', + ' .borris', + ' .borris', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, '.borris') + + def test_get_local_part_double_dot_raises(self): + local_part = self._test_get_x(parser.get_local_part, + ' borris.(foo).natasha at python.org', + ' borris.(foo).natasha', + ' borris. .natasha', + [errors.InvalidHeaderDefect]*2, + '@python.org') + self.assertEqual(local_part.local_part, 'borris..natasha') + + def test_get_local_part_quoted_strings_in_atom_list(self): + local_part = self._test_get_x(parser.get_local_part, + '""example" example"@example.com', + '""example" example"', + 'example example', + [errors.InvalidHeaderDefect]*3, + '@example.com') + self.assertEqual(local_part.local_part, 'example example') + + def test_get_local_part_valid_and_invalid_qp_in_atom_list(self): + local_part = self._test_get_x(parser.get_local_part, + r'"\\"example\\" example"@example.com', + r'"\\"example\\" example"', + r'\example\\ example', + [errors.InvalidHeaderDefect]*5, + '@example.com') + self.assertEqual(local_part.local_part, r'\example\\ example') + + def test_get_local_part_unicode_defect(self): + # Currently this only happens when parsing unicode, not when parsing + # stuff that was originally binary. + local_part = self._test_get_x(parser.get_local_part, + 'ex?mple at example.com', + 'ex?mple', + 'ex?mple', + [errors.NonASCIILocalPartDefect], + '@example.com') + self.assertEqual(local_part.local_part, 'ex?mple') + + # get_dtext + + def test_get_dtext_only(self): + dtext = self._test_get_x(parser.get_dtext, + 'foobar', 'foobar', 'foobar', [], '') + self.assertEqual(dtext.token_type, 'ptext') + + def test_get_dtext_all_dtext(self): + dtext = self._test_get_x(parser.get_dtext, self.rfc_dtext_chars, + self.rfc_dtext_chars, + self.rfc_dtext_chars, [], '') + + def test_get_dtext_two_words_gets_first(self): + self._test_get_x(parser.get_dtext, + 'foo bar', 'foo', 'foo', [], ' bar') + + def test_get_dtext_following_wsp_preserved(self): + self._test_get_x(parser.get_dtext, + 'foo \t\tbar', 'foo', 'foo', [], ' \t\tbar') + + def test_get_dtext_non_printables(self): + dtext = self._test_get_x(parser.get_dtext, + 'foo\x00bar]', 'foo\x00bar', 'foo\x00bar', + [errors.NonPrintableDefect], ']') + self.assertEqual(dtext.defects[0].non_printables[0], '\x00') + + def test_get_dtext_with_qp(self): + ptext = self._test_get_x(parser.get_dtext, + r'foo\]\[\\bar\b\e\l\l', + r'foo][\barbell', + r'foo][\barbell', + [errors.ObsoleteHeaderDefect], + '') + + def test_get_dtext_up_to_close_bracket_only(self): + self._test_get_x(parser.get_dtext, + 'foo]', 'foo', 'foo', [], ']') + + def test_get_dtext_wsp_before_close_bracket_preserved(self): + self._test_get_x(parser.get_dtext, + 'foo ]', 'foo', 'foo', [], ' ]') + + def test_get_dtext_close_bracket_mid_word(self): + self._test_get_x(parser.get_dtext, + 'foo]bar', 'foo', 'foo', [], ']bar') + + def test_get_dtext_up_to_open_bracket_only(self): + self._test_get_x(parser.get_dtext, + 'foo[', 'foo', 'foo', [], '[') + + def test_get_dtext_wsp_before_open_bracket_preserved(self): + self._test_get_x(parser.get_dtext, + 'foo [', 'foo', 'foo', [], ' [') + + def test_get_dtext_open_bracket_mid_word(self): + self._test_get_x(parser.get_dtext, + 'foo[bar', 'foo', 'foo', [], '[bar') + + # get_domain_literal + + def test_get_domain_literal_only(self): + domain_literal = domain_literal = self._test_get_x(parser.get_domain_literal, + '[127.0.0.1]', + '[127.0.0.1]', + '[127.0.0.1]', + [], + '') + self.assertEqual(domain_literal.token_type, 'domain-literal') + self.assertEqual(domain_literal.domain, '[127.0.0.1]') + self.assertEqual(domain_literal.ip, '127.0.0.1') + + def test_get_domain_literal_with_internal_ws(self): + domain_literal = self._test_get_x(parser.get_domain_literal, + '[ 127.0.0.1\t ]', + '[ 127.0.0.1\t ]', + '[ 127.0.0.1 ]', + [], + '') + self.assertEqual(domain_literal.domain, '[127.0.0.1]') + self.assertEqual(domain_literal.ip, '127.0.0.1') + + def test_get_domain_literal_with_surrounding_cfws(self): + domain_literal = self._test_get_x(parser.get_domain_literal, + '(foo)[ 127.0.0.1] (bar)', + '(foo)[ 127.0.0.1] (bar)', + ' [ 127.0.0.1] ', + [], + '') + self.assertEqual(domain_literal.domain, '[127.0.0.1]') + self.assertEqual(domain_literal.ip, '127.0.0.1') + + def test_get_domain_literal_no_start_char_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain_literal('(foo) ') + + def test_get_domain_literal_no_start_char_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain_literal('(foo) @') + + def test_get_domain_literal_bad_dtext_char_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain_literal('(foo) [abc[@') + + # get_domain + + def test_get_domain_regular_domain_only(self): + domain = self._test_get_x(parser.get_domain, + 'example.com', + 'example.com', + 'example.com', + [], + '') + self.assertEqual(domain.token_type, 'domain') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_domain_literal_only(self): + domain = self._test_get_x(parser.get_domain, + '[127.0.0.1]', + '[127.0.0.1]', + '[127.0.0.1]', + [], + '') + self.assertEqual(domain.token_type, 'domain') + self.assertEqual(domain.domain, '[127.0.0.1]') + + def test_get_domain_with_cfws(self): + domain = self._test_get_x(parser.get_domain, + '(foo) example.com(bar)\t', + '(foo) example.com(bar)\t', + ' example.com ', + [], + '') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_domain_literal_with_cfws(self): + domain = self._test_get_x(parser.get_domain, + '(foo)[127.0.0.1]\t(bar)', + '(foo)[127.0.0.1]\t(bar)', + ' [127.0.0.1] ', + [], + '') + self.assertEqual(domain.domain, '[127.0.0.1]') + + def test_get_domain_domain_with_cfws_ends_at_special(self): + domain = self._test_get_x(parser.get_domain, + '(foo)example.com\t(bar), next', + '(foo)example.com\t(bar)', + ' example.com ', + [], + ', next') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_domain_literal_with_cfws_ends_at_special(self): + domain = self._test_get_x(parser.get_domain, + '(foo)[127.0.0.1]\t(bar), next', + '(foo)[127.0.0.1]\t(bar)', + ' [127.0.0.1] ', + [], + ', next') + self.assertEqual(domain.domain, '[127.0.0.1]') + + def test_get_domain_obsolete(self): + domain = self._test_get_x(parser.get_domain, + '(foo) example . (bird)com(bar)\t', + '(foo) example . (bird)com(bar)\t', + ' example . com ', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(domain.domain, 'example.com') + + def test_get_domain_no_non_cfws_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain(" (foo)\t") + + def test_get_domain_no_atom_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_domain(" (foo)\t, broken") + + + # get_addr_spec + + def test_get_addr_spec_normal(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + 'dinsdale at example.com', + 'dinsdale at example.com', + 'dinsdale at example.com', + [], + '') + self.assertEqual(addr_spec.token_type, 'addr-spec') + self.assertEqual(addr_spec.local_part, 'dinsdale') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, 'dinsdale at example.com') + + def test_get_addr_spec_with_doamin_literal(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + 'dinsdale@[127.0.0.1]', + 'dinsdale@[127.0.0.1]', + 'dinsdale@[127.0.0.1]', + [], + '') + self.assertEqual(addr_spec.local_part, 'dinsdale') + self.assertEqual(addr_spec.domain, '[127.0.0.1]') + self.assertEqual(addr_spec.addr_spec, 'dinsdale@[127.0.0.1]') + + def test_get_addr_spec_with_cfws(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '(foo) dinsdale(bar)@ (bird) example.com (bog)', + '(foo) dinsdale(bar)@ (bird) example.com (bog)', + ' dinsdale at example.com ', + [], + '') + self.assertEqual(addr_spec.local_part, 'dinsdale') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, 'dinsdale at example.com') + + def test_get_addr_spec_with_qouoted_string_and_cfws(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '(foo) "roy a bug"(bar)@ (bird) example.com (bog)', + '(foo) "roy a bug"(bar)@ (bird) example.com (bog)', + ' "roy a bug"@example.com ', + [], + '') + self.assertEqual(addr_spec.local_part, 'roy a bug') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, '"roy a bug"@example.com') + + def test_get_addr_spec_ends_at_special(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '(foo) "roy a bug"(bar)@ (bird) example.com (bog) , next', + '(foo) "roy a bug"(bar)@ (bird) example.com (bog) ', + ' "roy a bug"@example.com ', + [], + ', next') + self.assertEqual(addr_spec.local_part, 'roy a bug') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, '"roy a bug"@example.com') + + def test_get_addr_spec_quoted_strings_in_atom_list(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + '""example" example"@example.com', + '""example" example"@example.com', + 'example example at example.com', + [errors.InvalidHeaderDefect]*3, + '') + self.assertEqual(addr_spec.local_part, 'example example') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, '"example example"@example.com') + + def test_get_addr_spec_dot_atom(self): + addr_spec = self._test_get_x(parser.get_addr_spec, + 'star.a.star at example.com', + 'star.a.star at example.com', + 'star.a.star at example.com', + [], + '') + self.assertEqual(addr_spec.local_part, 'star.a.star') + self.assertEqual(addr_spec.domain, 'example.com') + self.assertEqual(addr_spec.addr_spec, 'star.a.star at example.com') + + # get_obs_route + + def test_get_obs_route_simple(self): + obs_route = self._test_get_x(parser.get_obs_route, + '@example.com, @two.example.com:', + '@example.com, @two.example.com:', + '@example.com, @two.example.com:', + [], + '') + self.assertEqual(obs_route.token_type, 'obs-route') + self.assertEqual(obs_route.domains, ['example.com', 'two.example.com']) + + def test_get_obs_route_complex(self): + obs_route = self._test_get_x(parser.get_obs_route, + '(foo),, (blue)@example.com (bar), at two.(foo) example.com (bird):', + '(foo),, (blue)@example.com (bar), at two.(foo) example.com (bird):', + ' ,, @example.com , at two. example.com :', + [errors.ObsoleteHeaderDefect], # This is the obs-domain + '') + self.assertEqual(obs_route.token_type, 'obs-route') + self.assertEqual(obs_route.domains, ['example.com', 'two.example.com']) + + def test_get_obs_route_no_route_before_end_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) @example.com,') + + def test_get_obs_route_no_route_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) [abc],') + + def test_get_obs_route_no_route_before_special_raises2(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_obs_route('(foo) @example.com [abc],') + + # get_angle_addr + + def test_get_angle_addr_simple(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '', + '', + '', + [], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_with_cfws(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + ' (foo) (bar)', + ' (foo) (bar)', + ' ', + [], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_qs_and_domain_literal(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<"Fred Perfect"@[127.0.0.1]>', + '<"Fred Perfect"@[127.0.0.1]>', + '<"Fred Perfect"@[127.0.0.1]>', + [], + '') + self.assertEqual(angle_addr.local_part, 'Fred Perfect') + self.assertEqual(angle_addr.domain, '[127.0.0.1]') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, '"Fred Perfect"@[127.0.0.1]') + + def test_get_angle_addr_internal_cfws(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<(foo) dinsdale at example.com(bar)>', + '<(foo) dinsdale at example.com(bar)>', + '< dinsdale at example.com >', + [], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_obs_route(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '(foo)<@example.com, (bird) @two.example.com: dinsdale at example.com> (bar) ', + '(foo)<@example.com, (bird) @two.example.com: dinsdale at example.com> (bar) ', + ' <@example.com, @two.example.com: dinsdale at example.com> ', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertEqual(angle_addr.route, ['example.com', 'two.example.com']) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_missing_closing_angle(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '', + '', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_missing_closing_angle_with_cfws(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '', + '', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_ends_at_special(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + ' (foo), next', + ' (foo)', + ' ', + [], + ', next') + self.assertEqual(angle_addr.local_part, 'dinsdale') + self.assertEqual(angle_addr.domain, 'example.com') + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + + def test_get_angle_addr_no_angle_raise(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('(foo) ') + + def test_get_angle_addr_no_angle_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('(foo) , next') + + def test_get_angle_addr_no_angle_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('bar') + + def test_get_angle_addr_special_after_angle_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_angle_addr('(foo) <, bar') + + # get_display_name This is phrase but with a different value. + + def test_get_display_name_simple(self): + display_name = self._test_get_x(parser.get_display_name, + 'Fred A Johnson', + 'Fred A Johnson', + 'Fred A Johnson', + [], + '') + self.assertEqual(display_name.token_type, 'display-name') + self.assertEqual(display_name.display_name, 'Fred A Johnson') + + def test_get_display_name_complex1(self): + display_name = self._test_get_x(parser.get_display_name, + '"Fred A. Johnson" is his name, oh.', + '"Fred A. Johnson" is his name', + '"Fred A. Johnson is his name"', + [], + ', oh.') + self.assertEqual(display_name.token_type, 'display-name') + self.assertEqual(display_name.display_name, 'Fred A. Johnson is his name') + + def test_get_display_name_complex2(self): + display_name = self._test_get_x(parser.get_display_name, + ' (A) bird (in (my|your)) "hand " is messy\t<>\t', + ' (A) bird (in (my|your)) "hand " is messy\t', + ' "bird hand is messy" ', + [], + '<>\t') + self.assertEqual(display_name[0][0].comments, ['A']) + self.assertEqual(display_name[0][2].comments, ['in (my|your)']) + self.assertEqual(display_name.display_name, 'bird hand is messy') + + def test_get_display_name_obsolete(self): + display_name = self._test_get_x(parser.get_display_name, + 'Fred A.(weird).O Johnson', + 'Fred A.(weird).O Johnson', + '"Fred A. .O Johnson"', + [errors.ObsoleteHeaderDefect]*3, + '') + self.assertEqual(len(display_name), 7) + self.assertEqual(display_name[3].comments, ['weird']) + self.assertEqual(display_name.display_name, 'Fred A. .O Johnson') + + def test_get_display_name_pharse_must_start_with_word(self): + display_name = self._test_get_x(parser.get_display_name, + '(even weirder).name', + '(even weirder).name', + ' ".name"', + [errors.InvalidHeaderDefect] + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(len(display_name), 3) + self.assertEqual(display_name[0].comments, ['even weirder']) + self.assertEqual(display_name.display_name, '.name') + + def test_get_display_name_ending_with_obsolete(self): + display_name = self._test_get_x(parser.get_display_name, + 'simple phrase.(with trailing comment):boo', + 'simple phrase.(with trailing comment)', + '"simple phrase." ', + [errors.ObsoleteHeaderDefect]*2, + ':boo') + self.assertEqual(len(display_name), 4) + self.assertEqual(display_name[3].comments, ['with trailing comment']) + self.assertEqual(display_name.display_name, 'simple phrase.') + + # get_name_addr + + def test_get_name_addr_angle_addr_only(self): + name_addr = self._test_get_x(parser.get_name_addr, + '', + '', + '', + [], + '') + self.assertEqual(name_addr.token_type, 'name-addr') + self.assertIsNone(name_addr.display_name) + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_atom_name(self): + name_addr = self._test_get_x(parser.get_name_addr, + 'Dinsdale ', + 'Dinsdale ', + 'Dinsdale ', + [], + '') + self.assertEqual(name_addr.token_type, 'name-addr') + self.assertEqual(name_addr.display_name, 'Dinsdale') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_atom_name_with_cfws(self): + name_addr = self._test_get_x(parser.get_name_addr, + '(foo) Dinsdale (bar) (bird)', + '(foo) Dinsdale (bar) (bird)', + ' Dinsdale ', + [], + '') + self.assertEqual(name_addr.display_name, 'Dinsdale') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_name_with_cfws_and_dots(self): + name_addr = self._test_get_x(parser.get_name_addr, + '(foo) Roy.A.Bear (bar) (bird)', + '(foo) Roy.A.Bear (bar) (bird)', + ' "Roy.A.Bear" ', + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_qs_name(self): + name_addr = self._test_get_x(parser.get_name_addr, + '"Roy.A.Bear" ', + '"Roy.A.Bear" ', + '"Roy.A.Bear" ', + [], + '') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_with_route(self): + name_addr = self._test_get_x(parser.get_name_addr, + '"Roy.A.Bear" <@two.example.com: dinsdale at example.com>', + '"Roy.A.Bear" <@two.example.com: dinsdale at example.com>', + '"Roy.A.Bear" <@two.example.com: dinsdale at example.com>', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertEqual(name_addr.route, ['two.example.com']) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_ends_at_special(self): + name_addr = self._test_get_x(parser.get_name_addr, + '"Roy.A.Bear" , next', + '"Roy.A.Bear" ', + '"Roy.A.Bear" ', + [], + ', next') + self.assertEqual(name_addr.display_name, 'Roy.A.Bear') + self.assertEqual(name_addr.local_part, 'dinsdale') + self.assertEqual(name_addr.domain, 'example.com') + self.assertIsNone(name_addr.route) + self.assertEqual(name_addr.addr_spec, 'dinsdale at example.com') + + def test_get_name_addr_no_content_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr(' (foo) ') + + def test_get_name_addr_no_content_before_special_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr(' (foo) ,') + + def test_get_name_addr_no_angle_after_display_name_raises(self): + with self.assertRaises(errors.HeaderParseError): + parser.get_name_addr('foo bar') + + # get_mailbox + + def test_get_mailbox_addr_spec_only(self): + mailbox = self._test_get_x(parser.get_mailbox, + 'dinsdale at example.com', + 'dinsdale at example.com', + 'dinsdale at example.com', + [], + '') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertIsNone(mailbox.display_name) + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale at example.com') + + def test_get_mailbox_angle_addr_only(self): + mailbox = self._test_get_x(parser.get_mailbox, + '', + '', + '', + [], + '') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertIsNone(mailbox.display_name) + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale at example.com') + + def test_get_mailbox_name_addr(self): + mailbox = self._test_get_x(parser.get_mailbox, + '"Roy A. Bear" ', + '"Roy A. Bear" ', + '"Roy A. Bear" ', + [], + '') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertEqual(mailbox.display_name, 'Roy A. Bear') + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale at example.com') + + def test_get_mailbox_ends_at_special(self): + mailbox = self._test_get_x(parser.get_mailbox, + '"Roy A. Bear" , rest', + '"Roy A. Bear" ', + '"Roy A. Bear" ', + [], + ', rest') + self.assertEqual(mailbox.token_type, 'mailbox') + self.assertEqual(mailbox.display_name, 'Roy A. Bear') + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale at example.com') + + def test_get_mailbox_quoted_strings_in_atom_list(self): + mailbox = self._test_get_x(parser.get_mailbox, + '""example" example"@example.com', + '""example" example"@example.com', + 'example example at example.com', + [errors.InvalidHeaderDefect]*3, + '') + self.assertEqual(mailbox.local_part, 'example example') + self.assertEqual(mailbox.domain, 'example.com') + self.assertEqual(mailbox.addr_spec, '"example example"@example.com') + + # get_mailbox_list + + def test_get_mailbox_list_single_addr(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + 'dinsdale at example.com', + 'dinsdale at example.com', + 'dinsdale at example.com', + [], + '') + self.assertEqual(mailbox_list.token_type, 'mailbox-list') + self.assertEqual(len(mailbox_list.mailboxes), 1) + mailbox = mailbox_list.mailboxes[0] + self.assertIsNone(mailbox.display_name) + self.assertEqual(mailbox.local_part, 'dinsdale') + self.assertEqual(mailbox.domain, 'example.com') + self.assertIsNone(mailbox.route) + self.assertEqual(mailbox.addr_spec, 'dinsdale at example.com') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_two_simple_addr(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + 'dinsdale at example.com, dinsdale at test.example.com', + 'dinsdale at example.com, dinsdale at test.example.com', + 'dinsdale at example.com, dinsdale at test.example.com', + [], + '') + self.assertEqual(mailbox_list.token_type, 'mailbox-list') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale at example.com') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale at test.example.com') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_two_name_addr(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear" ,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" ,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" ,' + ' "Fred Flintstone" '), + [], + '') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale at example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale at test.example.com') + self.assertEqual(mailbox_list.mailboxes[1].display_name, + 'Fred Flintstone') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_two_complex(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('(foo) "Roy A. Bear" (bar),' + ' "Fred Flintstone" '), + ('(foo) "Roy A. Bear" (bar),' + ' "Fred Flintstone" '), + (' "Roy A. Bear" ,' + ' "Fred Flintstone" '), + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale at example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale at test.example.com') + self.assertEqual(mailbox_list.mailboxes[1].display_name, + 'Fred Flintstone') + self.assertEqual(mailbox_list.mailboxes, + mailbox_list.all_mailboxes) + + def test_get_mailbox_list_unparseable_mailbox_null(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear"[] dinsdale at example.com,' + ' "Fred Flintstone" '), + ('"Roy A. Bear"[] dinsdale at example.com,' + ' "Fred Flintstone" '), + ('"Roy A. Bear"[] dinsdale at example.com,' + ' "Fred Flintstone" '), + [errors.InvalidHeaderDefect, # the 'extra' text after the local part + errors.InvalidHeaderDefect, # the local part with no angle-addr + errors.ObsoleteHeaderDefect, # period in extra text (example.com) + errors.ObsoleteHeaderDefect], # (bird) in valid address. + '') + self.assertEqual(len(mailbox_list.mailboxes), 1) + self.assertEqual(len(mailbox_list.all_mailboxes), 2) + self.assertEqual(mailbox_list.all_mailboxes[0].token_type, + 'invalid-mailbox') + self.assertIsNone(mailbox_list.all_mailboxes[0].display_name) + self.assertEqual(mailbox_list.all_mailboxes[0].local_part, + 'Roy A. Bear') + self.assertIsNone(mailbox_list.all_mailboxes[0].domain) + self.assertEqual(mailbox_list.all_mailboxes[0].addr_spec, + '"Roy A. Bear"') + self.assertIs(mailbox_list.all_mailboxes[1], + mailbox_list.mailboxes[0]) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale at test.example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Fred Flintstone') + + def test_get_mailbox_list_junk_after_valid_address(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear" @@,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" @@,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" @@,' + ' "Fred Flintstone" '), + [errors.InvalidHeaderDefect], + '') + self.assertEqual(len(mailbox_list.mailboxes), 1) + self.assertEqual(len(mailbox_list.all_mailboxes), 2) + self.assertEqual(mailbox_list.all_mailboxes[0].addr_spec, + 'dinsdale at example.com') + self.assertEqual(mailbox_list.all_mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.all_mailboxes[0].token_type, + 'invalid-mailbox') + self.assertIs(mailbox_list.all_mailboxes[1], + mailbox_list.mailboxes[0]) + self.assertEqual(mailbox_list.mailboxes[0].addr_spec, + 'dinsdale at test.example.com') + self.assertEqual(mailbox_list.mailboxes[0].display_name, + 'Fred Flintstone') + + def test_get_mailbox_list_empty_list_element(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + ('"Roy A. Bear" , (bird),,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" , (bird),,' + ' "Fred Flintstone" '), + ('"Roy A. Bear" , ,,' + ' "Fred Flintstone" '), + [errors.ObsoleteHeaderDefect]*2, + '') + self.assertEqual(len(mailbox_list.mailboxes), 2) + self.assertEqual(mailbox_list.all_mailboxes, + mailbox_list.mailboxes) + self.assertEqual(mailbox_list.all_mailboxes[0].addr_spec, + 'dinsdale at example.com') + self.assertEqual(mailbox_list.all_mailboxes[0].display_name, + 'Roy A. Bear') + self.assertEqual(mailbox_list.mailboxes[1].addr_spec, + 'dinsdale at test.example.com') + self.assertEqual(mailbox_list.mailboxes[1].display_name, + 'Fred Flintstone') + + def test_get_mailbox_list_only_empty_elements(self): + mailbox_list = self._test_get_x(parser.get_mailbox_list, + '(foo),, (bar)', + '(foo),, (bar)', + ' ,, ', + [errors.ObsoleteHeaderDefect]*3, + '') + self.assertEqual(len(mailbox_list.mailboxes), 0) + self.assertEqual(mailbox_list.all_mailboxes, + mailbox_list.mailboxes) + + # get_group_list + + def test_get_group_list_cfws_only(self): + group_list = self._test_get_x(parser.get_group_list, + '(hidden);', + '(hidden)', + ' ', + [], + ';') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 0) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + + def test_get_group_list_mailbox_list(self): + group_list = self._test_get_x(parser.get_group_list, + 'dinsdale at example.org, "Fred A. Bear" ', + 'dinsdale at example.org, "Fred A. Bear" ', + 'dinsdale at example.org, "Fred A. Bear" ', + [], + '') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 2) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + self.assertEqual(group_list.mailboxes[1].display_name, + 'Fred A. Bear') + + def test_get_group_list_obs_group_list(self): + group_list = self._test_get_x(parser.get_group_list, + ', (foo),,(bar)', + ', (foo),,(bar)', + ', ,, ', + [errors.ObsoleteHeaderDefect], + '') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 0) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + + def test_get_group_list_comment_only_invalid(self): + group_list = self._test_get_x(parser.get_group_list, + '(bar)', + '(bar)', + ' ', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(group_list.token_type, 'group-list') + self.assertEqual(len(group_list.mailboxes), 0) + self.assertEqual(group_list.mailboxes, + group_list.all_mailboxes) + + # get_group + + def test_get_group_empty(self): + group = self._test_get_x(parser.get_group, + 'Monty Python:;', + 'Monty Python:;', + 'Monty Python:;', + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 0) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + + def test_get_troup_null_addr_spec(self): + group = self._test_get_x(parser.get_group, + 'foo: <>;', + 'foo: <>;', + 'foo: <>;', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(group.display_name, 'foo') + self.assertEqual(len(group.mailboxes), 0) + self.assertEqual(len(group.all_mailboxes), 1) + self.assertEqual(group.all_mailboxes[0].value, '<>') + + def test_get_group_cfws_only(self): + group = self._test_get_x(parser.get_group, + 'Monty Python: (hidden);', + 'Monty Python: (hidden);', + 'Monty Python: ;', + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 0) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + + def test_get_group_single_mailbox(self): + group = self._test_get_x(parser.get_group, + 'Monty Python: "Fred A. Bear" ;', + 'Monty Python: "Fred A. Bear" ;', + 'Monty Python: "Fred A. Bear" ;', + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 1) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + self.assertEqual(group.mailboxes[0].addr_spec, + 'dinsdale at example.com') + + def test_get_group_mixed_list(self): + group = self._test_get_x(parser.get_group, + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger , x at test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger , x at test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + ' Roger , x at test.example.com;'), + [], + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 3) + self.assertEqual(group.mailboxes, + group.all_mailboxes) + self.assertEqual(group.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(group.mailboxes[1].display_name, + 'Roger') + self.assertEqual(group.mailboxes[2].local_part, 'x') + + def test_get_group_one_invalid(self): + group = self._test_get_x(parser.get_group, + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger ping at exampele.com, x at test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + '(foo) Roger ping at exampele.com, x at test.example.com;'), + ('Monty Python: "Fred A. Bear" ,' + ' Roger ping at exampele.com, x at test.example.com;'), + [errors.InvalidHeaderDefect, # non-angle addr makes local part invalid + errors.InvalidHeaderDefect], # and its not obs-local either: no dots. + '') + self.assertEqual(group.token_type, 'group') + self.assertEqual(group.display_name, 'Monty Python') + self.assertEqual(len(group.mailboxes), 2) + self.assertEqual(len(group.all_mailboxes), 3) + self.assertEqual(group.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(group.mailboxes[1].local_part, 'x') + self.assertIsNone(group.all_mailboxes[1].display_name) + + # get_address + + def test_get_address_simple(self): + address = self._test_get_x(parser.get_address, + 'dinsdale at example.com', + 'dinsdale at example.com', + 'dinsdale at example.com', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].domain, + 'example.com') + self.assertEqual(address[0].token_type, + 'mailbox') + + def test_get_address_complex(self): + address = self._test_get_x(parser.get_address, + '(foo) "Fred A. Bear" <(bird)dinsdale at example.com>', + '(foo) "Fred A. Bear" <(bird)dinsdale at example.com>', + ' "Fred A. Bear" < dinsdale at example.com>', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].display_name, + 'Fred A. Bear') + self.assertEqual(address[0].token_type, + 'mailbox') + + def test_get_address_empty_group(self): + address = self._test_get_x(parser.get_address, + 'Monty Python:;', + 'Monty Python:;', + 'Monty Python:;', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 0) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address[0].token_type, + 'group') + self.assertEqual(address[0].display_name, + 'Monty Python') + + def test_get_address_group(self): + address = self._test_get_x(parser.get_address, + 'Monty Python: x at example.com, y at example.com;', + 'Monty Python: x at example.com, y at example.com;', + 'Monty Python: x at example.com, y at example.com;', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 2) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address[0].token_type, + 'group') + self.assertEqual(address[0].display_name, + 'Monty Python') + self.assertEqual(address.mailboxes[0].local_part, 'x') + + def test_get_address_quoted_local_part(self): + address = self._test_get_x(parser.get_address, + '"foo bar"@example.com', + '"foo bar"@example.com', + '"foo bar"@example.com', + [], + '') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].domain, + 'example.com') + self.assertEqual(address.mailboxes[0].local_part, + 'foo bar') + self.assertEqual(address[0].token_type, 'mailbox') + + def test_get_address_ends_at_special(self): + address = self._test_get_x(parser.get_address, + 'dinsdale at example.com, next', + 'dinsdale at example.com', + 'dinsdale at example.com', + [], + ', next') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 1) + self.assertEqual(address.mailboxes, + address.all_mailboxes) + self.assertEqual(address.mailboxes[0].domain, + 'example.com') + self.assertEqual(address[0].token_type, 'mailbox') + + def test_get_address_invalid_mailbox_invalid(self): + address = self._test_get_x(parser.get_address, + 'ping example.com, next', + 'ping example.com', + 'ping example.com', + [errors.InvalidHeaderDefect, # addr-spec with no domain + errors.InvalidHeaderDefect, # invalid local-part + errors.InvalidHeaderDefect, # missing .s in local-part + ], + ', next') + self.assertEqual(address.token_type, 'address') + self.assertEqual(len(address.mailboxes), 0) + self.assertEqual(len(address.all_mailboxes), 1) + self.assertIsNone(address.all_mailboxes[0].domain) + self.assertEqual(address.all_mailboxes[0].local_part, 'ping example.com') + self.assertEqual(address[0].token_type, 'invalid-mailbox') + + def test_get_address_quoted_strings_in_atom_list(self): + address = self._test_get_x(parser.get_address, + '""example" example"@example.com', + '""example" example"@example.com', + 'example example at example.com', + [errors.InvalidHeaderDefect]*3, + '') + self.assertEqual(address.all_mailboxes[0].local_part, 'example example') + self.assertEqual(address.all_mailboxes[0].domain, 'example.com') + self.assertEqual(address.all_mailboxes[0].addr_spec, '"example example"@example.com') + + + # get_address_list + + def test_get_address_list_mailboxes_simple(self): + address_list = self._test_get_x(parser.get_address_list, + 'dinsdale at example.com', + 'dinsdale at example.com', + 'dinsdale at example.com', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 1) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual([str(x) for x in address_list.mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list[0].token_type, 'address') + self.assertIsNone(address_list[0].display_name) + + def test_get_address_list_mailboxes_two_simple(self): + address_list = self._test_get_x(parser.get_address_list, + 'foo at example.com, "Fred A. Bar" ', + 'foo at example.com, "Fred A. Bar" ', + 'foo at example.com, "Fred A. Bar" ', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 2) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual([str(x) for x in address_list.mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].local_part, 'foo') + self.assertEqual(address_list.mailboxes[1].display_name, "Fred A. Bar") + + def test_get_address_list_mailboxes_complex(self): + address_list = self._test_get_x(parser.get_address_list, + ('"Roy A. Bear" , ' + '(ping) Foo ,' + 'Nobody Is. Special '), + ('"Roy A. Bear" , ' + '(ping) Foo ,' + 'Nobody Is. Special '), + ('"Roy A. Bear" , ' + 'Foo ,' + '"Nobody Is. Special" '), + [errors.ObsoleteHeaderDefect, # period in Is. + errors.ObsoleteHeaderDefect], # cfws in domain + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 3) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual([str(x) for x in address_list.mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list.mailboxes[0].token_type, 'mailbox') + self.assertEqual(address_list.addresses[0].token_type, 'address') + self.assertEqual(address_list.mailboxes[1].local_part, 'x') + self.assertEqual(address_list.mailboxes[2].display_name, + 'Nobody Is. Special') + + def test_get_address_list_mailboxes_invalid_addresses(self): + address_list = self._test_get_x(parser.get_address_list, + ('"Roy A. Bear" , ' + '(ping) Foo x at example.com[],' + 'Nobody Is. Special <(bird)example.(bad)com>'), + ('"Roy A. Bear" , ' + '(ping) Foo x at example.com[],' + 'Nobody Is. Special <(bird)example.(bad)com>'), + ('"Roy A. Bear" , ' + 'Foo x at example.com[],' + '"Nobody Is. Special" < example. com>'), + [errors.InvalidHeaderDefect, # invalid address in list + errors.InvalidHeaderDefect, # 'Foo x' local part invalid. + errors.InvalidHeaderDefect, # Missing . in 'Foo x' local part + errors.ObsoleteHeaderDefect, # period in 'Is.' disp-name phrase + errors.InvalidHeaderDefect, # no domain part in addr-spec + errors.ObsoleteHeaderDefect], # addr-spec has comment in it + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 1) + self.assertEqual(len(address_list.all_mailboxes), 3) + self.assertEqual([str(x) for x in address_list.all_mailboxes], + [str(x) for x in address_list.addresses]) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list.mailboxes[0].token_type, 'mailbox') + self.assertEqual(address_list.addresses[0].token_type, 'address') + self.assertEqual(address_list.addresses[1].token_type, 'address') + self.assertEqual(len(address_list.addresses[0].mailboxes), 1) + self.assertEqual(len(address_list.addresses[1].mailboxes), 0) + self.assertEqual(len(address_list.addresses[1].mailboxes), 0) + self.assertEqual( + address_list.addresses[1].all_mailboxes[0].local_part, 'Foo x') + self.assertEqual( + address_list.addresses[2].all_mailboxes[0].display_name, + "Nobody Is. Special") + + def test_get_address_list_group_empty(self): + address_list = self._test_get_x(parser.get_address_list, + 'Monty Python: ;', + 'Monty Python: ;', + 'Monty Python: ;', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 0) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual(len(address_list.addresses), 1) + self.assertEqual(address_list.addresses[0].token_type, 'address') + self.assertEqual(address_list.addresses[0].display_name, 'Monty Python') + self.assertEqual(len(address_list.addresses[0].mailboxes), 0) + + def test_get_address_list_group_simple(self): + address_list = self._test_get_x(parser.get_address_list, + 'Monty Python: dinsdale at example.com;', + 'Monty Python: dinsdale at example.com;', + 'Monty Python: dinsdale at example.com;', + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 1) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual(address_list.mailboxes[0].domain, 'example.com') + self.assertEqual(address_list.addresses[0].display_name, + 'Monty Python') + self.assertEqual(address_list.addresses[0].mailboxes[0].domain, + 'example.com') + + def test_get_address_list_group_and_mailboxes(self): + address_list = self._test_get_x(parser.get_address_list, + ('Monty Python: dinsdale at example.com, "Fred" ;, ' + 'Abe , Bee '), + ('Monty Python: dinsdale at example.com, "Fred" ;, ' + 'Abe , Bee '), + ('Monty Python: dinsdale at example.com, "Fred" ;, ' + 'Abe , Bee '), + [], + '') + self.assertEqual(address_list.token_type, 'address-list') + self.assertEqual(len(address_list.mailboxes), 4) + self.assertEqual(address_list.mailboxes, + address_list.all_mailboxes) + self.assertEqual(len(address_list.addresses), 3) + self.assertEqual(address_list.mailboxes[0].local_part, 'dinsdale') + self.assertEqual(address_list.addresses[0].display_name, + 'Monty Python') + self.assertEqual(address_list.addresses[0].mailboxes[0].domain, + 'example.com') + self.assertEqual(address_list.addresses[0].mailboxes[1].local_part, + 'flint') + self.assertEqual(address_list.addresses[1].mailboxes[0].local_part, + 'x') + self.assertEqual(address_list.addresses[2].mailboxes[0].local_part, + 'y') + self.assertEqual(str(address_list.addresses[1]), + str(address_list.mailboxes[2])) + + +class TestFolding(TestEmailBase): + + policy = policy.default + + def _test(self, tl, folded, policy=policy): + self.assertEqual(tl.fold(policy=policy), folded, tl.ppstr()) + + def test_simple_unstructured_no_folds(self): + self._test(parser.get_unstructured("This is a test"), + "This is a test\n") + + def test_simple_unstructured_folded(self): + self._test(parser.get_unstructured("This is also a test, but this " + "time there are enough words (and even some " + "symbols) to make it wrap; at least in theory."), + "This is also a test, but this time there are enough " + "words (and even some\n" + " symbols) to make it wrap; at least in theory.\n") + + def test_unstructured_with_unicode_no_folds(self): + self._test(parser.get_unstructured("h?bsch kleiner bei?t"), + "=?utf-8?q?h=C3=BCbsch_kleiner_bei=C3=9Ft?=\n") + + def test_one_ew_on_each_of_two_wrapped_lines(self): + self._test(parser.get_unstructured("Mein kleiner Kaktus ist sehr " + "h?bsch. Es hat viele Stacheln " + "und oft bei?t mich."), + "Mein kleiner Kaktus ist sehr =?utf-8?q?h=C3=BCbsch=2E?= " + "Es hat viele Stacheln\n" + " und oft =?utf-8?q?bei=C3=9Ft?= mich.\n") + + def test_ews_combined_before_wrap(self): + self._test(parser.get_unstructured("Mein Kaktus ist h?bsch. " + "Es bei?t mich. " + "And that's all I'm sayin."), + "Mein Kaktus ist =?utf-8?q?h=C3=BCbsch=2E__Es_bei=C3=9Ft?= " + "mich. And that's\n" + " all I'm sayin.\n") + + # XXX Need test of an encoded word so long that it needs to be wrapped + + def test_simple_address(self): + self._test(parser.get_address_list("abc ")[0], + "abc \n") + + def test_address_list_folding_at_commas(self): + self._test(parser.get_address_list('abc , ' + '"Fred Blunt" , ' + '"J.P.Cool" , ' + '"K<>y" , ' + 'Firesale , ' + '')[0], + 'abc , "Fred Blunt" ,\n' + ' "J.P.Cool" , "K<>y" ,\n' + ' Firesale , \n') + + def test_address_list_with_unicode_names(self): + self._test(parser.get_address_list( + 'H?bsch Kaktus , ' + 'bei?t bei?t ')[0], + '=?utf-8?q?H=C3=BCbsch?= Kaktus ,\n' + ' =?utf-8?q?bei=C3=9Ft_bei=C3=9Ft?= \n') + + def test_address_list_with_unicode_names_in_quotes(self): + self._test(parser.get_address_list( + '"H?bsch Kaktus" , ' + '"bei?t" bei?t ')[0], + '=?utf-8?q?H=C3=BCbsch?= Kaktus ,\n' + ' =?utf-8?q?bei=C3=9Ft_bei=C3=9Ft?= \n') + + # XXX Need tests with comments on various sides of a unicode token, + # and with unicode tokens in the comments. Spaces inside the quotes + # currently don't do the right thing. + + def test_initial_whitespace_splitting(self): + body = parser.get_unstructured(' ' + 'x'*77) + header = parser.Header([ + parser.HeaderLabel([parser.ValueTerminal('test:', 'atext')]), + parser.CFWSList([parser.WhiteSpaceTerminal(' ', 'fws')]), body]) + self._test(header, 'test: \n ' + 'x'*77 + '\n') + + def test_whitespace_splitting(self): + self._test(parser.get_unstructured('xxx ' + 'y'*77), + 'xxx \n ' + 'y'*77 + '\n') + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test__headerregistry.py b/Lib/test/test_email/test__headerregistry.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test__headerregistry.py @@ -0,0 +1,717 @@ +import datetime +import textwrap +import unittest +from email import errors +from email import policy +from test.test_email import TestEmailBase +from email import _headerregistry +# Address and Group are public but I'm not sure where to put them yet. +from email._headerregistry import Address, Group + + +class TestHeaderRegistry(TestEmailBase): + + def test_arbitrary_name_unstructured(self): + factory = _headerregistry.HeaderRegistry() + h = factory('foobar', 'test') + self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + + def test_name_case_ignored(self): + factory = _headerregistry.HeaderRegistry() + # Whitebox check that test is valid + self.assertNotIn('Subject', factory.registry) + h = factory('Subject', 'test') + self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, _headerregistry.UniqueUnstructuredHeader) + + class FooBase: + def __init__(self, *args, **kw): + pass + + def test_override_default_base_class(self): + factory = _headerregistry.HeaderRegistry(base_class=self.FooBase) + h = factory('foobar', 'test') + self.assertIsInstance(h, self.FooBase) + self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + + class FooDefault: + parse = _headerregistry.UnstructuredHeader.parse + + def test_override_default_class(self): + factory = _headerregistry.HeaderRegistry(default_class=self.FooDefault) + h = factory('foobar', 'test') + self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, self.FooDefault) + + def test_override_default_class_only_overrides_default(self): + factory = _headerregistry.HeaderRegistry(default_class=self.FooDefault) + h = factory('subject', 'test') + self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, _headerregistry.UniqueUnstructuredHeader) + + def test_dont_use_default_map(self): + factory = _headerregistry.HeaderRegistry(use_default_map=False) + h = factory('subject', 'test') + self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + + def test_map_to_type(self): + factory = _headerregistry.HeaderRegistry() + h1 = factory('foobar', 'test') + factory.map_to_type('foobar', _headerregistry.UniqueUnstructuredHeader) + h2 = factory('foobar', 'test') + self.assertIsInstance(h1, _headerregistry.BaseHeader) + self.assertIsInstance(h1, _headerregistry.UnstructuredHeader) + self.assertIsInstance(h2, _headerregistry.BaseHeader) + self.assertIsInstance(h2, _headerregistry.UniqueUnstructuredHeader) + + +class TestHeaderBase(TestEmailBase): + + factory = _headerregistry.HeaderRegistry() + + def make_header(self, name, value): + return self.factory(name, value) + + +class TestBaseHeaderFeatures(TestHeaderBase): + + def test_str(self): + h = self.make_header('subject', 'this is a test') + self.assertIsInstance(h, str) + self.assertEqual(h, 'this is a test') + self.assertEqual(str(h), 'this is a test') + + def test_substr(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(h[5:7], 'is') + + def test_has_name(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(h.name, 'subject') + + def _test_attr_ro(self, attr): + h = self.make_header('subject', 'this is a test') + with self.assertRaises(AttributeError): + setattr(h, attr, 'foo') + + def test_name_read_only(self): + self._test_attr_ro('name') + + def test_defects_read_only(self): + self._test_attr_ro('defects') + + def test_defects_is_tuple(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(len(h.defects), 0) + self.assertIsInstance(h.defects, tuple) + # Make sure it is still true when there are defects. + h = self.make_header('date', '') + self.assertEqual(len(h.defects), 1) + self.assertIsInstance(h.defects, tuple) + + # XXX: FIXME + #def test_CR_in_value(self): + # # XXX: this also re-raises the issue of embedded headers, + # # need test and solution for that. + # value = '\r'.join(['this is', ' a test']) + # h = self.make_header('subject', value) + # self.assertEqual(h, value) + # self.assertDefectsEqual(h.defects, [errors.ObsoleteHeaderDefect]) + + def test_RFC2047_value_decoded(self): + value = '=?utf-8?q?this_is_a_test?=' + h = self.make_header('subject', value) + self.assertEqual(h, 'this is a test') + + +class TestDateHeader(TestHeaderBase): + + datestring = 'Sun, 23 Sep 2001 20:10:55 -0700' + utcoffset = datetime.timedelta(hours=-7) + tz = datetime.timezone(utcoffset) + dt = datetime.datetime(2001, 9, 23, 20, 10, 55, tzinfo=tz) + + def test_parse_date(self): + h = self.make_header('date', self.datestring) + self.assertEqual(h, self.datestring) + self.assertEqual(h.datetime, self.dt) + self.assertEqual(h.datetime.utcoffset(), self.utcoffset) + self.assertEqual(h.defects, ()) + + def test_set_from_datetime(self): + h = self.make_header('date', self.dt) + self.assertEqual(h, self.datestring) + self.assertEqual(h.datetime, self.dt) + self.assertEqual(h.defects, ()) + + def test_date_header_properties(self): + h = self.make_header('date', self.datestring) + self.assertIsInstance(h, _headerregistry.UniqueDateHeader) + self.assertEqual(h.max_count, 1) + self.assertEqual(h.defects, ()) + + def test_resent_date_header_properties(self): + h = self.make_header('resent-date', self.datestring) + self.assertIsInstance(h, _headerregistry.DateHeader) + self.assertEqual(h.max_count, None) + self.assertEqual(h.defects, ()) + + def test_no_value_is_defect(self): + h = self.make_header('date', '') + self.assertEqual(len(h.defects), 1) + self.assertIsInstance(h.defects[0], errors.HeaderMissingRequiredValue) + + def test_datetime_read_only(self): + h = self.make_header('date', self.datestring) + with self.assertRaises(AttributeError): + h.datetime = 'foo' + + +class TestAddressHeader(TestHeaderBase): + + examples = { + + 'empty': + ('<>', + [errors.InvalidHeaderDefect], + '<>', + '', + '<>', + '', + '', + None), + + 'address_only': + ('zippy at pinhead.com', + [], + 'zippy at pinhead.com', + '', + 'zippy at pinhead.com', + 'zippy', + 'pinhead.com', + None), + + 'name_and_address': + ('Zaphrod Beblebrux ', + [], + 'Zaphrod Beblebrux ', + 'Zaphrod Beblebrux', + 'zippy at pinhead.com', + 'zippy', + 'pinhead.com', + None), + + 'quoted_local_part': + ('Zaphrod Beblebrux <"foo bar"@pinhead.com>', + [], + 'Zaphrod Beblebrux <"foo bar"@pinhead.com>', + 'Zaphrod Beblebrux', + '"foo bar"@pinhead.com', + 'foo bar', + 'pinhead.com', + None), + + 'quoted_parens_in_name': + (r'"A \(Special\) Person" ', + [], + '"A (Special) Person" ', + 'A (Special) Person', + 'person at dom.ain', + 'person', + 'dom.ain', + None), + + 'quoted_backslashes_in_name': + (r'"Arthur \\Backslash\\ Foobar" ', + [], + r'"Arthur \\Backslash\\ Foobar" ', + r'Arthur \Backslash\ Foobar', + 'person at dom.ain', + 'person', + 'dom.ain', + None), + + 'name_with_dot': + ('John X. Doe ', + [errors.ObsoleteHeaderDefect], + '"John X. Doe" ', + 'John X. Doe', + 'jxd at example.com', + 'jxd', + 'example.com', + None), + + 'quoted_strings_in_local_part': + ('""example" example"@example.com', + [errors.InvalidHeaderDefect]*3, + '"example example"@example.com', + '', + '"example example"@example.com', + 'example example', + 'example.com', + None), + + 'escaped_quoted_strings_in_local_part': + (r'"\"example\" example"@example.com', + [], + r'"\"example\" example"@example.com', + '', + r'"\"example\" example"@example.com', + r'"example" example', + 'example.com', + None), + + 'escaped_escapes_in_local_part': + (r'"\\"example\\" example"@example.com', + [errors.InvalidHeaderDefect]*5, + r'"\\example\\\\ example"@example.com', + '', + r'"\\example\\\\ example"@example.com', + r'\example\\ example', + 'example.com', + None), + + 'spaces_in_unquoted_local_part_collapsed': + ('merwok wok @example.com', + [errors.InvalidHeaderDefect]*2, + '"merwok wok"@example.com', + '', + '"merwok wok"@example.com', + 'merwok wok', + 'example.com', + None), + + 'spaces_around_dots_in_local_part_removed': + ('merwok. wok . wok at example.com', + [errors.ObsoleteHeaderDefect], + 'merwok.wok.wok at example.com', + '', + 'merwok.wok.wok at example.com', + 'merwok.wok.wok', + 'example.com', + None), + + } + + # XXX: Need many more examples, and in particular some with names in + # trailing comments, which aren't currently handled. comments in + # general are not handled yet. + + def _test_single_addr(self, source, defects, decoded, display_name, + addr_spec, username, domain, comment): + h = self.make_header('sender', source) + self.assertEqual(h, decoded) + self.assertDefectsEqual(h.defects, defects) + a = h.address + self.assertEqual(str(a), decoded) + self.assertEqual(len(h.groups), 1) + self.assertEqual([a], list(h.groups[0].addresses)) + self.assertEqual([a], list(h.addresses)) + self.assertEqual(a.display_name, display_name) + self.assertEqual(a.addr_spec, addr_spec) + self.assertEqual(a.username, username) + self.assertEqual(a.domain, domain) + # XXX: we have no comment support yet. + #self.assertEqual(a.comment, comment) + + for name in examples: + locals()['test_'+name] = ( + lambda self, name=name: + self._test_single_addr(*self.examples[name])) + + def _test_group_single_addr(self, source, defects, decoded, display_name, + addr_spec, username, domain, comment): + source = 'foo: {};'.format(source) + gdecoded = 'foo: {};'.format(decoded) if decoded else 'foo:;' + h = self.make_header('to', source) + self.assertEqual(h, gdecoded) + self.assertDefectsEqual(h.defects, defects) + self.assertEqual(h.groups[0].addresses, h.addresses) + self.assertEqual(len(h.groups), 1) + self.assertEqual(len(h.addresses), 1) + a = h.addresses[0] + self.assertEqual(str(a), decoded) + self.assertEqual(a.display_name, display_name) + self.assertEqual(a.addr_spec, addr_spec) + self.assertEqual(a.username, username) + self.assertEqual(a.domain, domain) + + for name in examples: + locals()['test_group_'+name] = ( + lambda self, name=name: + self._test_group_single_addr(*self.examples[name])) + + def test_simple_address_list(self): + value = ('Fred , foo at example.com, ' + '"Harry W. Hastings" ') + h = self.make_header('to', value) + self.assertEqual(h, value) + self.assertEqual(len(h.groups), 3) + self.assertEqual(len(h.addresses), 3) + for i in range(3): + self.assertEqual(h.groups[i].addresses[0], h.addresses[i]) + self.assertEqual(str(h.addresses[0]), 'Fred ') + self.assertEqual(str(h.addresses[1]), 'foo at example.com') + self.assertEqual(str(h.addresses[2]), + '"Harry W. Hastings" ') + self.assertEqual(h.addresses[2].display_name, + 'Harry W. Hastings') + + def test_complex_address_list(self): + examples = list(self.examples.values()) + source = ('dummy list:;, another: (empty);,' + + ', '.join([x[0] for x in examples[:4]]) + ', ' + + r'"A \"list\"": ' + + ', '.join([x[0] for x in examples[4:6]]) + ';,' + + ', '.join([x[0] for x in examples[6:]]) + ) + # XXX: the fact that (empty) disappears here is a potential API design + # bug. We don't currently have a way to preserve comments. + expected = ('dummy list:;, another:;, ' + + ', '.join([x[2] for x in examples[:4]]) + ', ' + + r'"A \"list\"": ' + + ', '.join([x[2] for x in examples[4:6]]) + ';, ' + + ', '.join([x[2] for x in examples[6:]]) + ) + + h = self.make_header('to', source) + self.assertEqual(h.split(','), expected.split(',')) + self.assertEqual(h, expected) + self.assertEqual(len(h.groups), 7 + len(examples) - 6) + self.assertEqual(h.groups[0].display_name, 'dummy list') + self.assertEqual(h.groups[1].display_name, 'another') + self.assertEqual(h.groups[6].display_name, 'A "list"') + self.assertEqual(len(h.addresses), len(examples)) + for i in range(4): + self.assertIsNone(h.groups[i+2].display_name) + self.assertEqual(str(h.groups[i+2].addresses[0]), examples[i][2]) + for i in range(7, 7 + len(examples) - 6): + self.assertIsNone(h.groups[i].display_name) + self.assertEqual(str(h.groups[i].addresses[0]), examples[i-1][2]) + for i in range(len(examples)): + self.assertEqual(str(h.addresses[i]), examples[i][2]) + self.assertEqual(h.addresses[i].addr_spec, examples[i][4]) + + def test_address_read_only(self): + h = self.make_header('sender', 'abc at xyz.com') + with self.assertRaises(AttributeError): + h.address = 'foo' + + def test_addresses_read_only(self): + h = self.make_header('sender', 'abc at xyz.com') + with self.assertRaises(AttributeError): + h.addresses = 'foo' + + def test_groups_read_only(self): + h = self.make_header('sender', 'abc at xyz.com') + with self.assertRaises(AttributeError): + h.groups = 'foo' + + def test_addresses_types(self): + source = 'me ' + h = self.make_header('to', source) + self.assertIsInstance(h.addresses, tuple) + self.assertIsInstance(h.addresses[0], Address) + + def test_groups_types(self): + source = 'me ' + h = self.make_header('to', source) + self.assertIsInstance(h.groups, tuple) + self.assertIsInstance(h.groups[0], Group) + + def test_set_from_Address(self): + h = self.make_header('to', Address('me', 'foo', 'example.com')) + self.assertEqual(h, 'me ') + + def test_set_from_Address_list(self): + h = self.make_header('to', [Address('me', 'foo', 'example.com'), + Address('you', 'bar', 'example.com')]) + self.assertEqual(h, 'me , you ') + + def test_set_from_Address_and_Group_list(self): + h = self.make_header('to', [Address('me', 'foo', 'example.com'), + Group('bing', [Address('fiz', 'z', 'b.com'), + Address('zif', 'f', 'c.com')]), + Address('you', 'bar', 'example.com')]) + self.assertEqual(h, 'me , bing: fiz , ' + 'zif ;, you ') + self.assertEqual(h.fold(policy=policy.default.clone(max_line_length=40)), + 'to: me ,\n' + ' bing: fiz , zif ;,\n' + ' you \n') + + def test_set_from_Group_list(self): + h = self.make_header('to', [Group('bing', [Address('fiz', 'z', 'b.com'), + Address('zif', 'f', 'c.com')])]) + self.assertEqual(h, 'bing: fiz , zif ;') + + +class TestAddressAndGroup(TestEmailBase): + + def _test_attr_ro(self, obj, attr): + with self.assertRaises(AttributeError): + setattr(obj, attr, 'foo') + + def test_address_display_name_ro(self): + self._test_attr_ro(Address('foo', 'bar', 'baz'), 'display_name') + + def test_address_username_ro(self): + self._test_attr_ro(Address('foo', 'bar', 'baz'), 'username') + + def test_address_domain_ro(self): + self._test_attr_ro(Address('foo', 'bar', 'baz'), 'domain') + + def test_group_display_name_ro(self): + self._test_attr_ro(Group('foo'), 'display_name') + + def test_group_addresses_ro(self): + self._test_attr_ro(Group('foo'), 'addresses') + + def test_address_from_username_domain(self): + a = Address('foo', 'bar', 'baz') + self.assertEqual(a.display_name, 'foo') + self.assertEqual(a.username, 'bar') + self.assertEqual(a.domain, 'baz') + self.assertEqual(a.addr_spec, 'bar at baz') + self.assertEqual(str(a), 'foo ') + + def test_address_from_addr_spec(self): + a = Address('foo', addr_spec='bar at baz') + self.assertEqual(a.display_name, 'foo') + self.assertEqual(a.username, 'bar') + self.assertEqual(a.domain, 'baz') + self.assertEqual(a.addr_spec, 'bar at baz') + self.assertEqual(str(a), 'foo ') + + def test_address_with_no_display_name(self): + a = Address(addr_spec='bar at baz') + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, 'bar') + self.assertEqual(a.domain, 'baz') + self.assertEqual(a.addr_spec, 'bar at baz') + self.assertEqual(str(a), 'bar at baz') + + def test_null_address(self): + a = Address() + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, '') + self.assertEqual(a.domain, '') + self.assertEqual(a.addr_spec, '<>') + self.assertEqual(str(a), '<>') + + def test_domain_only(self): + # This isn't really a valid address. + a = Address(domain='buzz') + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, '') + self.assertEqual(a.domain, 'buzz') + self.assertEqual(a.addr_spec, '@buzz') + self.assertEqual(str(a), '@buzz') + + def test_username_only(self): + # This isn't really a valid address. + a = Address(username='buzz') + self.assertEqual(a.display_name, '') + self.assertEqual(a.username, 'buzz') + self.assertEqual(a.domain, '') + self.assertEqual(a.addr_spec, 'buzz') + self.assertEqual(str(a), 'buzz') + + def test_display_name_only(self): + a = Address('buzz') + self.assertEqual(a.display_name, 'buzz') + self.assertEqual(a.username, '') + self.assertEqual(a.domain, '') + self.assertEqual(a.addr_spec, '<>') + self.assertEqual(str(a), 'buzz <>') + + def test_quoting(self): + # Ideally we'd check every special individually, but I'm not up for + # writing that many tests. + a = Address('Sara J.', 'bad name', 'example.com') + self.assertEqual(a.display_name, 'Sara J.') + self.assertEqual(a.username, 'bad name') + self.assertEqual(a.domain, 'example.com') + self.assertEqual(a.addr_spec, '"bad name"@example.com') + self.assertEqual(str(a), '"Sara J." <"bad name"@example.com>') + + def test_il8n(self): + a = Address('?ric', 'wok', 'ex?mple.com') + self.assertEqual(a.display_name, '?ric') + self.assertEqual(a.username, 'wok') + self.assertEqual(a.domain, 'ex?mple.com') + self.assertEqual(a.addr_spec, 'wok at ex?mple.com') + self.assertEqual(str(a), '?ric ') + + # XXX: there is an API design issue that needs to be solved here. + #def test_non_ascii_username_raises(self): + # with self.assertRaises(ValueError): + # Address('foo', 'w?k', 'example.com') + + def test_non_ascii_username_in_addr_spec_raises(self): + with self.assertRaises(ValueError): + Address('foo', addr_spec='w?k at example.com') + + def test_address_addr_spec_and_username_raises(self): + with self.assertRaises(TypeError): + Address('foo', username='bing', addr_spec='bar at baz') + + def test_address_addr_spec_and_domain_raises(self): + with self.assertRaises(TypeError): + Address('foo', domain='bing', addr_spec='bar at baz') + + def test_address_addr_spec_and_username_and_domain_raises(self): + with self.assertRaises(TypeError): + Address('foo', username='bong', domain='bing', addr_spec='bar at baz') + + def test_space_in_addr_spec_username_raises(self): + with self.assertRaises(ValueError): + Address('foo', addr_spec="bad name at example.com") + + def test_bad_addr_sepc_raises(self): + with self.assertRaises(ValueError): + Address('foo', addr_spec="name at ex[]ample.com") + + def test_empty_group(self): + g = Group('foo') + self.assertEqual(g.display_name, 'foo') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'foo:;') + + def test_empty_group_list(self): + g = Group('foo', addresses=[]) + self.assertEqual(g.display_name, 'foo') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'foo:;') + + def test_null_group(self): + g = Group() + self.assertIsNone(g.display_name) + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'None:;') + + def test_group_with_addresses(self): + addrs = [Address('b', 'b', 'c'), Address('a', 'b','c')] + g = Group('foo', addrs) + self.assertEqual(g.display_name, 'foo') + self.assertEqual(g.addresses, tuple(addrs)) + self.assertEqual(str(g), 'foo: b , a ;') + + def test_group_with_addresses_no_display_name(self): + addrs = [Address('b', 'b', 'c'), Address('a', 'b','c')] + g = Group(addresses=addrs) + self.assertIsNone(g.display_name) + self.assertEqual(g.addresses, tuple(addrs)) + self.assertEqual(str(g), 'None: b , a ;') + + def test_group_with_one_address_no_display_name(self): + addrs = [Address('b', 'b', 'c')] + g = Group(addresses=addrs) + self.assertIsNone(g.display_name) + self.assertEqual(g.addresses, tuple(addrs)) + self.assertEqual(str(g), 'b ') + + def test_display_name_quoting(self): + g = Group('foo.bar') + self.assertEqual(g.display_name, 'foo.bar') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), '"foo.bar":;') + + def test_display_name_blanks_not_quoted(self): + g = Group('foo bar') + self.assertEqual(g.display_name, 'foo bar') + self.assertEqual(g.addresses, tuple()) + self.assertEqual(str(g), 'foo bar:;') + + +class TestFolding(TestHeaderBase): + + def test_short_unstructured(self): + h = self.make_header('subject', 'this is a test') + self.assertEqual(h.fold(policy=self.policy), + 'subject: this is a test\n') + + def test_long_unstructured(self): + h = self.make_header('Subject', 'This is a long header ' + 'line that will need to be folded into two lines ' + 'and will demonstrate basic folding') + self.assertEqual(h.fold(policy=self.policy), + 'Subject: This is a long header line that will ' + 'need to be folded into two lines\n' + ' and will demonstrate basic folding\n') + + def test_unstructured_short_max_line_length(self): + h = self.make_header('Subject', 'this is a short header ' + 'that will be folded anyway') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + textwrap.dedent("""\ + Subject: this is a + short header that + will be folded + anyway + """)) + + def test_fold_unstructured_single_word(self): + h = self.make_header('Subject', 'test') + self.assertEqual(h.fold(policy=self.policy), 'Subject: test\n') + + def test_fold_unstructured_short(self): + h = self.make_header('Subject', 'test test test') + self.assertEqual(h.fold(policy=self.policy), + 'Subject: test test test\n') + + def test_fold_unstructured_with_overlong_word(self): + h = self.make_header('Subject', 'thisisaverylonglineconsistingofa' + 'singlewordthatwontfit') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Subject: thisisaverylonglineconsistingofasinglewordthatwontfit\n') + + def test_fold_unstructured_with_two_overlong_words(self): + h = self.make_header('Subject', 'thisisaverylonglineconsistingofa' + 'singlewordthatwontfit plusanotherverylongwordthatwontfit') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=20)), + 'Subject: thisisaverylonglineconsistingofasinglewordthatwontfit\n' + ' plusanotherverylongwordthatwontfit\n') + + def test_fold_unstructured_with_slightly_long_word(self): + h = self.make_header('Subject', 'thislongwordislessthanmaxlinelen') + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=35)), + 'Subject:\n thislongwordislessthanmaxlinelen\n') + + def test_fold_unstructured_with_commas(self): + # The old wrapper would fold this at the commas. + h = self.make_header('Subject', "This header is intended to " + "demonstrate, in a fairly susinct way, that we now do " + "not give a , special treatment in unstructured headers.") + self.assertEqual( + h.fold(policy=policy.default.clone(max_line_length=60)), + textwrap.dedent("""\ + Subject: This header is intended to demonstrate, in a fairly + susinct way, that we now do not give a , special treatment + in unstructured headers. + """)) + + def test_fold_address_list(self): + h = self.make_header('To', '"Theodore H. Perfect" , ' + '"My address is very long because my name is long" , ' + '"Only A. Friend" ') + self.assertEqual(h.fold(policy=self.policy), textwrap.dedent("""\ + To: "Theodore H. Perfect" , + "My address is very long because my name is long" , + "Only A. Friend" + """)) + + def test_fold_date_header(self): + h = self.make_header('Date', 'Sat, 2 Feb 2002 17:00:06 -0800') + self.assertEqual(h.fold(policy=self.policy), + 'Date: Sat, 02 Feb 2002 17:00:06 -0800\n') + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_generator.py b/Lib/test/test_email/test_generator.py --- a/Lib/test/test_email/test_generator.py +++ b/Lib/test/test_email/test_generator.py @@ -6,14 +6,16 @@ from email import policy from test.test_email import TestEmailBase -# XXX: move generator tests from test_email into here at some point. +class TestGeneratorBase: -class TestGeneratorBase(): + policy = policy.default - policy = policy.compat32 + def msgmaker(self, msg, policy=None): + policy = self.policy if policy is None else policy + return self.msgfunc(msg, policy=policy) - long_subject = { + refold_long_expected = { 0: textwrap.dedent("""\ To: whom_it_may_concern at example.com From: nobody_you_want_to_know at example.com @@ -23,33 +25,32 @@ None """), + # From is wrapped because wrapped it fits in 40. 40: textwrap.dedent("""\ To: whom_it_may_concern at example.com - From:\x20 + From: nobody_you_want_to_know at example.com Subject: We the willing led by the - unknowing are doing the - impossible for the ungrateful. We have - done so much for so long with so little - we are now qualified to do anything - with nothing. + unknowing are doing the impossible for + the ungrateful. We have done so much + for so long with so little we are now + qualified to do anything with nothing. None """), + # Neither to nor from fit even if put on a new line, + # so we leave them sticking out on the first line. 20: textwrap.dedent("""\ - To:\x20 - whom_it_may_concern at example.com - From:\x20 - nobody_you_want_to_know at example.com + To: whom_it_may_concern at example.com + From: nobody_you_want_to_know at example.com Subject: We the willing led by the unknowing are doing - the - impossible for the - ungrateful. We have - done so much for so - long with so little - we are now + the impossible for + the ungrateful. We + have done so much + for so long with so + little we are now qualified to do anything with nothing. @@ -57,65 +58,90 @@ None """), } - long_subject[100] = long_subject[0] + refold_long_expected[100] = refold_long_expected[0] - def maxheaderlen_parameter_test(self, n): - msg = self.msgmaker(self.typ(self.long_subject[0])) + refold_all_expected = refold_long_expected.copy() + refold_all_expected[0] = ( + "To: whom_it_may_concern at example.com\n" + "From: nobody_you_want_to_know at example.com\n" + "Subject: We the willing led by the unknowing are doing the " + "impossible for the ungrateful. We have done so much for " + "so long with so little we are now qualified to do anything " + "with nothing.\n" + "\n" + "None\n") + refold_all_expected[100] = ( + "To: whom_it_may_concern at example.com\n" + "From: nobody_you_want_to_know at example.com\n" + "Subject: We the willing led by the unknowing are doing the " + "impossible for the ungrateful. We have\n" + " done so much for so long with so little we are now qualified " + "to do anything with nothing.\n" + "\n" + "None\n") + + def _test_maxheaderlen_parameter(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() - g = self.genclass(s, maxheaderlen=n) + g = self.genclass(s, maxheaderlen=n, policy=self.policy) g.flatten(msg) - self.assertEqual(s.getvalue(), self.typ(self.long_subject[n])) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) - def test_maxheaderlen_parameter_0(self): - self.maxheaderlen_parameter_test(0) + for n in refold_long_expected: + locals()['test_maxheaderlen_parameter_' + str(n)] = ( + lambda self, n=n: + self._test_maxheaderlen_parameter(n)) - def test_maxheaderlen_parameter_100(self): - self.maxheaderlen_parameter_test(100) + def _test_max_line_length_policy(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(max_line_length=n)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) - def test_maxheaderlen_parameter_40(self): - self.maxheaderlen_parameter_test(40) + for n in refold_long_expected: + locals()['test_max_line_length_policy' + str(n)] = ( + lambda self, n=n: + self._test_max_line_length_policy(n)) - def test_maxheaderlen_parameter_20(self): - self.maxheaderlen_parameter_test(20) - - def maxheaderlen_policy_test(self, n): - msg = self.msgmaker(self.typ(self.long_subject[0])) - s = self.ioclass() - g = self.genclass(s, policy=policy.default.clone(max_line_length=n)) - g.flatten(msg) - self.assertEqual(s.getvalue(), self.typ(self.long_subject[n])) - - def test_maxheaderlen_policy_0(self): - self.maxheaderlen_policy_test(0) - - def test_maxheaderlen_policy_100(self): - self.maxheaderlen_policy_test(100) - - def test_maxheaderlen_policy_40(self): - self.maxheaderlen_policy_test(40) - - def test_maxheaderlen_policy_20(self): - self.maxheaderlen_policy_test(20) - - def maxheaderlen_parm_overrides_policy_test(self, n): - msg = self.msgmaker(self.typ(self.long_subject[0])) + def _test_maxheaderlen_parm_overrides_policy(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() g = self.genclass(s, maxheaderlen=n, - policy=policy.default.clone(max_line_length=10)) + policy=self.policy.clone(max_line_length=10)) g.flatten(msg) - self.assertEqual(s.getvalue(), self.typ(self.long_subject[n])) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) - def test_maxheaderlen_parm_overrides_policy_0(self): - self.maxheaderlen_parm_overrides_policy_test(0) + for n in refold_long_expected: + locals()['test_maxheaderlen_parm_overrides_policy' + str(n)] = ( + lambda self, n=n: + self._test_maxheaderlen_parm_overrides_policy(n)) - def test_maxheaderlen_parm_overrides_policy_100(self): - self.maxheaderlen_parm_overrides_policy_test(100) + def _test_refold_none_does_not_fold(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='none', + max_line_length=n)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[0])) - def test_maxheaderlen_parm_overrides_policy_40(self): - self.maxheaderlen_parm_overrides_policy_test(40) + for n in refold_long_expected: + locals()['test_refold_none_does_not_fold' + str(n)] = ( + lambda self, n=n: + self._test_refold_none_does_not_fold(n)) - def test_maxheaderlen_parm_overrides_policy_20(self): - self.maxheaderlen_parm_overrides_policy_test(20) + def _test_refold_all(self, n): + msg = self.msgmaker(self.typ(self.refold_long_expected[0])) + s = self.ioclass() + g = self.genclass(s, policy=self.policy.clone(refold_source='all', + max_line_length=n)) + g.flatten(msg) + self.assertEqual(s.getvalue(), self.typ(self.refold_all_expected[n])) + + for n in refold_long_expected: + locals()['test_refold_all' + str(n)] = ( + lambda self, n=n: + self._test_refold_all(n)) def test_crlf_control_via_policy(self): source = "Subject: test\r\n\r\ntest body\r\n" @@ -138,30 +164,24 @@ class TestGenerator(TestGeneratorBase, TestEmailBase): + msgfunc = staticmethod(message_from_string) genclass = Generator ioclass = io.StringIO typ = str - def msgmaker(self, msg, policy=None): - policy = self.policy if policy is None else policy - return message_from_string(msg, policy=policy) - class TestBytesGenerator(TestGeneratorBase, TestEmailBase): + msgfunc = staticmethod(message_from_bytes) genclass = BytesGenerator ioclass = io.BytesIO typ = lambda self, x: x.encode('ascii') - def msgmaker(self, msg, policy=None): - policy = self.policy if policy is None else policy - return message_from_bytes(msg, policy=policy) - def test_cte_type_7bit_handles_unknown_8bit(self): source = ("Subject: Maintenant je vous pr?sente mon " "coll?gue\n\n").encode('utf-8') - expected = ('Subject: =?unknown-8bit?q?Maintenant_je_vous_pr=C3=A9sente_mon_' - 'coll=C3=A8gue?=\n\n').encode('ascii') + expected = ('Subject: Maintenant je vous =?unknown-8bit?q?' + 'pr=C3=A9sente_mon_coll=C3=A8gue?=\n\n').encode('ascii') msg = message_from_bytes(source) s = io.BytesIO() g = BytesGenerator(s, policy=self.policy.clone(cte_type='7bit')) diff --git a/Lib/test/test_email/test_pickleable.py b/Lib/test/test_email/test_pickleable.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test_pickleable.py @@ -0,0 +1,57 @@ +import unittest +import textwrap +import copy +import pickle +from email import policy +from email import message_from_string +from email._headerregistry import HeaderRegistry +from test.test_email import TestEmailBase + +class TestPickleCopyHeader(TestEmailBase): + + unstructured = HeaderRegistry()('subject', 'this is a test') + + def test_deepcopy_unstructured(self): + h = copy.deepcopy(self.unstructured) + self.assertEqual(str(h), str(self.unstructured)) + + def test_pickle_unstructured(self): + p = pickle.dumps(self.unstructured) + h = pickle.loads(p) + self.assertEqual(str(h), str(self.unstructured)) + + address = HeaderRegistry()('from', 'frodo at mordor.net') + + def test_deepcopy_address(self): + h = copy.deepcopy(self.address) + self.assertEqual(str(h), str(self.address)) + + def test_pickle_address(self): + p = pickle.dumps(self.address) + h = pickle.loads(p) + self.assertEqual(str(h), str(self.address)) + + +class TestPickleCopyMessage(TestEmailBase): + + testmsg = message_from_string(textwrap.dedent("""\ + From: frodo at mordor.net + To: bilbo at underhill.org + Subject: help + + I think I forgot the ring. + """), policy=policy.default) + + def test_deepcopy(self): + msg2 = copy.deepcopy(self.testmsg) + self.assertEqual(msg2.as_string(), self.testmsg.as_string()) + + def test_pickle(self): + p = pickle.dumps(self.testmsg) + msg2 = pickle.loads(p) + self.assertEqual(msg2.as_string(), self.testmsg.as_string()) + + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_policy.py b/Lib/test/test_email/test_policy.py --- a/Lib/test/test_email/test_policy.py +++ b/Lib/test/test_email/test_policy.py @@ -5,49 +5,70 @@ import email.policy import email.parser import email.generator +from email import _headerregistry + +def make_defaults(base_defaults, differences): + defaults = base_defaults.copy() + defaults.update(differences) + return defaults class PolicyAPITests(unittest.TestCase): longMessage = True - # These default values are the ones set on email.policy.default. - # If any of these defaults change, the docs must be updated. - policy_defaults = { + # Base default values. + compat32_defaults = { 'max_line_length': 78, 'linesep': '\n', 'cte_type': '8bit', 'raise_on_defect': False, } + # These default values are the ones set on email.policy.default. + # If any of these defaults change, the docs must be updated. + policy_defaults = compat32_defaults.copy() + policy_defaults.update({ + 'raise_on_defect': False, + 'header_factory': email.policy.EmailPolicy.header_factory, + 'refold_source': 'long', + }) - # For each policy under test, we give here the values of the attributes - # that are different from the defaults for that policy. + # For each policy under test, we give here what we expect the defaults to + # be for that policy. The second argument to make defaults is the + # difference between the base defaults and that for the particular policy. + new_policy = email.policy.EmailPolicy() policies = { - email.policy.Compat32(): {}, - email.policy.compat32: {}, - email.policy.default: {}, - email.policy.SMTP: {'linesep': '\r\n'}, - email.policy.HTTP: {'linesep': '\r\n', 'max_line_length': None}, - email.policy.strict: {'raise_on_defect': True}, + email.policy.compat32: make_defaults(compat32_defaults, {}), + email.policy.default: make_defaults(policy_defaults, {}), + email.policy.SMTP: make_defaults(policy_defaults, + {'linesep': '\r\n'}), + email.policy.HTTP: make_defaults(policy_defaults, + {'linesep': '\r\n', + 'max_line_length': None}), + email.policy.strict: make_defaults(policy_defaults, + {'raise_on_defect': True}), + new_policy: make_defaults(policy_defaults, {}), } + # Creating a new policy creates a new header factory. There is a test + # later that proves this. + policies[new_policy]['header_factory'] = new_policy.header_factory def test_defaults(self): - for policy, changed_defaults in self.policies.items(): - expected = self.policy_defaults.copy() - expected.update(changed_defaults) + for policy, expected in self.policies.items(): for attr, value in expected.items(): self.assertEqual(getattr(policy, attr), value, ("change {} docs/docstrings if defaults have " "changed").format(policy)) def test_all_attributes_covered(self): - for attr in dir(email.policy.default): - if (attr.startswith('_') or - isinstance(getattr(email.policy.Policy, attr), - types.FunctionType)): - continue - else: - self.assertIn(attr, self.policy_defaults, - "{} is not fully tested".format(attr)) + for policy, expected in self.policies.items(): + for attr in dir(policy): + if (attr.startswith('_') or + isinstance(getattr(email.policy.EmailPolicy, attr), + types.FunctionType)): + continue + else: + self.assertIn(attr, expected, + "{} is not fully tested".format(attr)) def test_abc(self): with self.assertRaises(TypeError) as cm: @@ -62,18 +83,20 @@ self.assertIn(method, msg) def test_policy_is_immutable(self): - for policy in self.policies: - for attr in self.policy_defaults: + for policy, defaults in self.policies.items(): + for attr in defaults: with self.assertRaisesRegex(AttributeError, attr+".*read-only"): setattr(policy, attr, None) with self.assertRaisesRegex(AttributeError, 'no attribute.*foo'): policy.foo = None - def test_set_policy_attrs_when_calledl(self): - testattrdict = { attr: None for attr in self.policy_defaults } - for policyclass in self.policies: + def test_set_policy_attrs_when_cloned(self): + # None of the attributes has a default value of None, so we set them + # all to None in the clone call and check that it worked. + for policyclass, defaults in self.policies.items(): + testattrdict = {attr: None for attr in defaults} policy = policyclass.clone(**testattrdict) - for attr in self.policy_defaults: + for attr in defaults: self.assertIsNone(getattr(policy, attr)) def test_reject_non_policy_keyword_when_called(self): @@ -105,7 +128,7 @@ self.defects = [] obj = Dummy() defect = object() - policy = email.policy.Compat32() + policy = email.policy.EmailPolicy() policy.register_defect(obj, defect) self.assertEqual(obj.defects, [defect]) defect2 = object() @@ -134,7 +157,7 @@ email.policy.default.handle_defect(foo, defect2) self.assertEqual(foo.defects, [defect1, defect2]) - class MyPolicy(email.policy.Compat32): + class MyPolicy(email.policy.EmailPolicy): defects = None def __init__(self, *args, **kw): super().__init__(*args, defects=[], **kw) @@ -159,6 +182,49 @@ self.assertEqual(my_policy.defects, [defect1, defect2]) self.assertEqual(foo.defects, []) + def test_default_header_factory(self): + h = email.policy.default.header_factory('Test', 'test') + self.assertEqual(h.name, 'Test') + self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + self.assertIsInstance(h, _headerregistry.BaseHeader) + + class Foo: + parse = _headerregistry.UnstructuredHeader.parse + + def test_each_Policy_gets_unique_factory(self): + policy1 = email.policy.EmailPolicy() + policy2 = email.policy.EmailPolicy() + policy1.header_factory.map_to_type('foo', self.Foo) + h = policy1.header_factory('foo', 'test') + self.assertIsInstance(h, self.Foo) + self.assertNotIsInstance(h, _headerregistry.UnstructuredHeader) + h = policy2.header_factory('foo', 'test') + self.assertNotIsInstance(h, self.Foo) + self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + + def test_clone_copies_factory(self): + policy1 = email.policy.EmailPolicy() + policy2 = policy1.clone() + policy1.header_factory.map_to_type('foo', self.Foo) + h = policy1.header_factory('foo', 'test') + self.assertIsInstance(h, self.Foo) + h = policy2.header_factory('foo', 'test') + self.assertIsInstance(h, self.Foo) + + def test_new_factory_overrides_default(self): + mypolicy = email.policy.EmailPolicy() + myfactory = mypolicy.header_factory + newpolicy = mypolicy + email.policy.strict + self.assertEqual(newpolicy.header_factory, myfactory) + newpolicy = email.policy.strict + mypolicy + self.assertEqual(newpolicy.header_factory, myfactory) + + def test_adding_default_policies_preserves_default_factory(self): + newpolicy = email.policy.default + email.policy.strict + self.assertEqual(newpolicy.header_factory, + email.policy.EmailPolicy.header_factory) + self.assertEqual(newpolicy.__dict__, {'raise_on_defect': True}) + # XXX: Need subclassing tests. # For adding subclassed objects, make sure the usual rules apply (subclass # wins), but that the order still works (right overrides left). -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 03:55:44 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 26 May 2012 03:55:44 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0OTIw?= =?utf8?q?=3A_Fix_the_help=28urllib=2Eparse=29_failure_on_locale_C_termina?= =?utf8?q?ls=2E_Just?= Message-ID: http://hg.python.org/cpython/rev/ea25ce432343 changeset: 77149:ea25ce432343 branch: 3.2 parent: 77138:0528ec18e230 user: Senthil Kumaran date: Sat May 26 09:53:32 2012 +0800 summary: Issue #14920: Fix the help(urllib.parse) failure on locale C terminals. Just have ascii in help msg files: Lib/urllib/parse.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -707,7 +707,7 @@ def quote_from_bytes(bs, safe='/'): """Like quote(), but accepts a bytes object rather than a str, and does not perform string-to-bytes encoding. It always returns an ASCII string. - quote_from_bytes(b'abc def\xab') -> 'abc%20def%AB' + quote_from_bytes(b'abc def\x3f') -> 'abc%20def%3f' """ if not isinstance(bs, (bytes, bytearray)): raise TypeError("quote_from_bytes() expected bytes") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. + Have ascii characters in help. + - Issue #14863: Update the documentation of os.fdopen() to reflect the fact that it's only a thin wrapper around open() anymore. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 03:55:45 2012 From: python-checkins at python.org (senthil.kumaran) Date: Sat, 26 May 2012 03:55:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314920=3A_Fix_the_help=28urllib=2Eparse=29_failure_o?= =?utf8?q?n_locale_C_terminals=2E_Just?= Message-ID: http://hg.python.org/cpython/rev/cb62c958dd6a changeset: 77150:cb62c958dd6a parent: 77148:0189b9d2d6bc parent: 77149:ea25ce432343 user: Senthil Kumaran date: Sat May 26 09:55:28 2012 +0800 summary: Issue #14920: Fix the help(urllib.parse) failure on locale C terminals. Just have ascii in help msg files: Lib/urllib/parse.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -707,7 +707,7 @@ def quote_from_bytes(bs, safe='/'): """Like quote(), but accepts a bytes object rather than a str, and does not perform string-to-bytes encoding. It always returns an ASCII string. - quote_from_bytes(b'abc def\xab') -> 'abc%20def%AB' + quote_from_bytes(b'abc def\x3f') -> 'abc%20def%3f' """ if not isinstance(bs, (bytes, bytearray)): raise TypeError("quote_from_bytes() expected bytes") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,9 @@ Library ------- +- Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. + Have ascii characters in help. + - Issue #14548: Make multiprocessing finalizers check pid before running to cope with possibility of gc running just after fork. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 04:26:28 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 04:26:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2312586=3A_Expand_What=27s?= =?utf8?q?_New_email_entry_with_provisional_policy_features=2E?= Message-ID: http://hg.python.org/cpython/rev/331cceee3b45 changeset: 77151:331cceee3b45 user: R David Murray date: Fri May 25 22:25:56 2012 -0400 summary: #12586: Expand What's New email entry with provisional policy features. files: Doc/whatsnew/3.3.rst | 96 ++++++++++++++++++++++++++++++++ 1 files changed, 96 insertions(+), 0 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -558,6 +558,9 @@ New Email Package Features ========================== +Policy Framework +---------------- + The email package now has a :mod:`~email.policy` framework. A :class:`~email.policy.Policy` is an object with several methods and properties that control how the email package behaves. The primary policy for Python 3.3 @@ -610,6 +613,99 @@ the ``generator``. +Provisional Policy with New Header API +-------------------------------------- + +While the policy framework is worthwhile all by itself, the main motivation for +introducing it is to allow the creation of new policies that implement new +features for the email package in a way that maintains backward compatibility +for those who do not use the new policies. Because the new policies introduce a +new API, we are releasing them in Python 3.3 as a :term:`provisional policy +`. Backwards incompatible changes (up to and including +removal of the code) may occur if deemed necessary by the core developers. + +The new policies are instances of :class:`~email.policy.EmailPolicy`, +and add the following additional controls: + + =============== ======================================================= + refold_source Controls whether or not headers parsed by a + :mod:`~email.parser` are refolded by the + :mod:`~email.generator`. It can be ``none``, ``long``, + or ``all``. The default is ``long``, which means that + source headers with a line longer than + ``max_line_length`` get refolded. ``none`` means no + line get refolded, and ``all`` means that all lines + get refolded. + + header_factory A callable that take a ``name`` and ``value`` and + produces a custom header object. + =============== ======================================================= + +The ``header_factory`` is the key to the new features provided by the new +policies. When one of the new policies is used, any header retrieved from +a ``Message`` object is an object produced by the ``header_factory``, and any +time you set a header on a ``Message`` it becomes an object produced by +``header_factory``. All such header objects have a ``name`` attribute equal +to the header name. Address and Date headers have additional attributes +that give you access to the parsed data of the header. This means you can now +do things like this:: + + >>> m = Message(policy=SMTP) + >>> m['To'] = '?ric ' + >>> m['to'] + '?ric ' + >>> m['to'].addresses + (Address(display_name='?ric', username='foo', domain='example.com'),) + >>> m['to'].addresses[0].username + 'foo' + >>> m['to'].addresses[0].display_name + '?ric' + >>> m['Date'] = email.utils.localtime() + >>> m['Date'].datetime + datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT')) + >>> m['Date'] + 'Fri, 25 May 2012 21:44:27 -0400' + >>> print(m) + To: =?utf-8?q?=C3=89ric?= + Date: Fri, 25 May 2012 21:44:27 -0400 + +You will note that the unicode display name is automatically encoded as +``utf-8`` when the message is serialized, but that when the header is accessed +directly, you get the unicode version. This eliminates any need to deal with +the :mod:`email.header` :meth:`~email.header.decode_header` or +:meth:`~email.header.make_header` functions. + +You can also create addresses from parts:: + + >>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'), + ... Address('Sally', 'sally', 'example.com')]), + ... Address('Bonzo', addr_spec='bonz at laugh.com')] + >>> print(m) + To: =?utf-8?q?=C3=89ric?= + Date: Fri, 25 May 2012 21:44:27 -0400 + cc: pals: Bob , Sally ;, Bonzo + +Decoding to unicode is done automatically:: + + >>> m2 = message_from_string(str(m)) + >>> m2['to'] + '?ric ' + +When you parse a message, you can use the ``addresses`` and ``groups`` +attributes of the header objects to access the groups and individual +addresses:: + + >>> m2['cc'].addresses + (Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com')) + >>> m2['cc'].groups + (Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),)) + +In summary, if you use one of the new policies, header manipulation works the +way it ought to: your application works with unicode strings, and the email +package transparently encodes and decodes the unicode to and from the RFC +standard Content Transfer Encodings. + + Other Language Changes ====================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 04:33:50 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 04:33:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_news_entries_for_=23147?= =?utf8?q?31_and_=2312586=2E?= Message-ID: http://hg.python.org/cpython/rev/09e97829ed1e changeset: 77152:09e97829ed1e user: R David Murray date: Fri May 25 22:33:36 2012 -0400 summary: Add news entries for #14731 and #12586. files: Misc/NEWS | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -44,6 +44,14 @@ Library ------- +- Issue #12586: Added new provisional policies that implement convenient + unicode support for email headers. See What's New for details. + +- Issue #14731: Refactored email Policy framework to support full backward + compatibility with Python 3.2 by default yet allow for the introduction of + new features through new policies. Note that Policy.must_be_7bit is renamed + to cte_type. + - Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. Have ascii characters in help. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 04:48:48 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 26 May 2012 04:48:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Updated_=2Ehgeol_for_additi?= =?utf8?q?onal_binaries=2E?= Message-ID: http://hg.python.org/cpython/rev/3c8f8dc62ef1 changeset: 77153:3c8f8dc62ef1 parent: 77148:0189b9d2d6bc user: Vinay Sajip date: Sat May 26 03:25:23 2012 +0100 summary: Updated .hgeol for additional binaries. files: .hgeol | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/.hgeol b/.hgeol --- a/.hgeol +++ b/.hgeol @@ -32,6 +32,8 @@ Lib/test/sndhdrdata/sndhdr.* = BIN Lib/test/test_email/data/msg_26.txt = BIN +Lib/venv/scripts/nt/* = BIN + # All other files (which presumably are human-editable) are "native". # This must be the last rule! -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 04:48:49 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 26 May 2012 04:48:49 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Implemented_PEP_405_=28Pyth?= =?utf8?q?on_virtual_environments=29=2E?= Message-ID: http://hg.python.org/cpython/rev/294f8aeb4d4b changeset: 77154:294f8aeb4d4b user: Vinay Sajip date: Sat May 26 03:45:29 2012 +0100 summary: Implemented PEP 405 (Python virtual environments). files: Doc/library/development.rst | 1 + Doc/library/sys.rst | 28 + Doc/library/venv.rst | 193 ++++ Lib/distutils/sysconfig.py | 40 +- Lib/gettext.py | 2 +- Lib/idlelib/EditorWindow.py | 6 +- Lib/packaging/command/build_ext.py | 5 +- Lib/pydoc.py | 2 +- Lib/site.py | 88 ++- Lib/subprocess.py | 2 +- Lib/sysconfig.cfg | 24 +- Lib/sysconfig.py | 29 +- Lib/test/regrtest.py | 2 +- Lib/test/test_cmd.py | 2 +- Lib/test/test_doctest.py | 2 +- Lib/test/test_subprocess.py | 4 + Lib/test/test_sys.py | 4 + Lib/test/test_sysconfig.py | 11 +- Lib/test/test_trace.py | 4 +- Lib/test/test_venv.py | 139 +++ Lib/tkinter/_fix.py | 4 +- Lib/trace.py | 8 +- Lib/venv/__init__.py | 502 +++++++++++++ Lib/venv/__main__.py | 10 + Lib/venv/scripts/nt/Activate.ps1 | 34 + Lib/venv/scripts/nt/Deactivate.ps1 | 19 + Lib/venv/scripts/nt/activate.bat | 31 + Lib/venv/scripts/nt/deactivate.bat | 17 + Lib/venv/scripts/nt/pysetup3-script.py | 11 + Lib/venv/scripts/nt/pysetup3.exe | Bin Lib/venv/scripts/posix/activate | 76 + Lib/venv/scripts/posix/pysetup3 | 11 + Mac/Makefile.in | 4 +- Mac/Tools/pythonw.c | 12 + Makefile.pre.in | 3 + Modules/getpath.c | 86 ++ PC/getpathp.c | 81 ++ Python/sysmodule.c | 4 + Tools/msi/msi.py | 1 + Tools/scripts/pyvenv | 11 + setup.py | 7 +- 41 files changed, 1454 insertions(+), 66 deletions(-) diff --git a/Doc/library/development.rst b/Doc/library/development.rst --- a/Doc/library/development.rst +++ b/Doc/library/development.rst @@ -23,3 +23,4 @@ unittest.mock-examples.rst 2to3.rst test.rst + venv.rst diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -29,6 +29,26 @@ command line, see the :mod:`fileinput` module. +.. data:: base_exec_prefix + + Set during Python startup, before ``site.py`` is run, to the same value as + :data:`exec_prefix`. If not running in a virtual environment, the values + will stay the same; if ``site.py`` finds that a virtual environment is in + use, the values of :data:`prefix` and :data:`exec_prefix` will be changed to + point to the virtual environment, whereas :data:`base_prefix` and + :data:`base_exec_prefix` will remain pointing to the base Python + installation (the one which the virtual environment was created from). + +.. data:: base_prefix + + Set during Python startup, before ``site.py`` is run, to the same value as + :data:`prefix`. If not running in a virtual environment, the values + will stay the same; if ``site.py`` finds that a virtual environment is in + use, the values of :data:`prefix` and :data:`exec_prefix` will be changed to + point to the virtual environment, whereas :data:`base_prefix` and + :data:`base_exec_prefix` will remain pointing to the base Python + installation (the one which the virtual environment was created from). + .. data:: byteorder An indicator of the native byte order. This will have the value ``'big'`` on @@ -199,6 +219,10 @@ installed in :file:`{exec_prefix}/lib/python{X.Y}/lib-dynload`, where *X.Y* is the version number of Python, for example ``3.2``. + .. note:: If a virtual environment is in effect, this value will be changed + in ``site.py`` to point to the virtual environment. The value for the + Python installation will still be available, via :data:`base_exec_prefix`. + .. data:: executable @@ -775,6 +799,10 @@ stored in :file:`{prefix}/include/python{X.Y}`, where *X.Y* is the version number of Python, for example ``3.2``. + .. note:: If a virtual environment is in effect, this value will be changed + in ``site.py`` to point to the virtual environment. The value for the + Python installation will still be available, via :data:`base_prefix`. + .. data:: ps1 ps2 diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst new file mode 100644 --- /dev/null +++ b/Doc/library/venv.rst @@ -0,0 +1,193 @@ +:mod:`venv` --- Creation of virtual environments +================================================ + +.. module:: venv + :synopsis: Creation of virtual environments. +.. moduleauthor:: Vinay Sajip +.. sectionauthor:: Vinay Sajip + + +.. index:: pair: Environments; virtual + +.. versionadded:: 3.3 + +**Source code:** :source:`Lib/venv.py` + +-------------- + +The :mod:`venv` module provides support for creating lightweight +"virtual environments" with their own site directories, optionally +isolated from system site directories. Each virtual environment has +its own Python binary (allowing creation of environments with various +Python versions) and can have its own independent set of installed +Python packages in its site directories. + +Creating virtual environments +----------------------------- + +Creation of virtual environments is simplest executing the ``pyvenv`` +script:: + + pyvenv /path/to/new/virtual/environment + +Running this command creates the target directory (creating any parent +directories that don't exist already) and places a ``pyvenv.cfg`` file +in it with a ``home`` key pointing to the Python installation the +command was run from. It also creates a ``bin`` (or ``Scripts`` on +Windows) subdirectory containing a copy of the ``python`` binary (or +binaries, in the case of Windows) and the ``pysetup3`` script (to +facilitate easy installation of packages from PyPI into the new virtualenv). +It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` +subdirectory (on Windows, this is ``Lib\site-packages``). + +.. highlight:: none + +On Windows, you may have to invoke the ``pyvenv`` script as follows, if you +don't have the relevant PATH and PATHEXT settings:: + + c:\Temp>c:\Python33\python c:\Python33\Tools\Scripts\pyvenv.py myenv + +or equivalently:: + + c:\Temp>c:\Python33\python -m venv myenv + +The command, if run with ``-h``, will show the available options:: + + usage: pyvenv [-h] [--system-site-packages] [--symlink] [--clear] + ENV_DIR [ENV_DIR ...] + + Creates virtual Python environments in one or more target directories. + + positional arguments: + ENV_DIR A directory to create the environment in. + + optional arguments: + -h, --help show this help message and exit + --system-site-packages Give access to the global site-packages dir to the + virtual environment. + --symlink Attempt to symlink rather than copy. + --clear Delete the environment directory if it already exists. + If not specified and the directory exists, an error is + raised. + + +If the target directory already exists an error will be raised, unless +the ``--clear`` option was provided, in which case the target +directory will be deleted and virtual environment creation will +proceed as usual. + +The created ``pyvenv.cfg`` file also includes the +``include-system-site-packages`` key, set to ``true`` if ``venv`` is +run with the ``--system-site-packages`` option, ``false`` otherwise. + +Multiple paths can be given to ``pyvenv``, in which case an identical +virtualenv will be created, according to the given options, at each +provided path. + + +API +--- + +The high-level method described above makes use of a simple API which provides +mechanisms for third-party virtual environment creators to customize +environment creation according to their needs. + +The :class:`EnvBuilder` class accepts the following keyword arguments on +instantiation: + + * ``system_site_packages`` - A Boolean value indicating that the + system Python site-packages should be available to the + environment (defaults to ``False``). + + * ``clear`` - A Boolean value which, if True, will delete any + existing target directory instead of raising an exception + (defaults to ``False``). + + * ``symlinks`` - A Boolean value indicating whether to attempt + to symlink the Python binary (and any necessary DLLs or other + binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to + ``True`` on Linux and Unix systems, but ``False`` on Windows and + Mac OS X. + +The returned env-builder is an object which has a method, ``create``, +which takes as required argument the path (absolute or relative to the current +directory) of the target directory which is to contain the virtual environment. +The ``create`` method will either create the environment in the specified +directory, or raise an appropriate exception. + +Creators of third-party virtual environment tools will be free to use +the provided ``EnvBuilder`` class as a base class. + +.. highlight:: python + +The ``venv`` module will also provide a module-level function as a +convenience:: + + def create(env_dir, + system_site_packages=False, clear=False, symlinks=False): + builder = EnvBuilder( + system_site_packages=system_site_packages, + clear=clear, + symlinks=symlinks) + builder.create(env_dir) + +The ``create`` method of the ``EnvBuilder`` class illustrates the +hooks available for subclass customization:: + + def create(self, env_dir): + """ + Create a virtualized Python environment in a directory. + + :param env_dir: The target directory to create an environment in. + + """ + env_dir = os.path.abspath(env_dir) + context = self.create_directories(env_dir) + self.create_configuration(context) + self.setup_python(context) + self.setup_scripts(context) + self.post_setup(context) + +Each of the methods ``create_directories``, ``create_configuration``, +``setup_python``, ``setup_scripts`` and ``post_setup`` can be +overridden. The functions of these methods are: + + * ``create_directories`` - creates the environment directory and + all necessary directories, and returns a context object. This is + just a holder for attributes (such as paths), for use by the + other methods. + + * ``create_configuration`` - creates the ``pyvenv.cfg`` + configuration file in the environment. + + * ``setup_python`` - creates a copy of the Python executable (and, + under Windows, DLLs) in the environment. + + * ``setup_scripts`` - Installs activation scripts appropriate to the + platform into the virtual environment. + + * ``post_setup`` - A placeholder method which can be overridden + in third party implementations to pre-install packages in the + virtual environment or perform other post-creation steps. + +In addition, ``EnvBuilder`` provides an ``install_scripts`` utility +method that can be called from ``setup_scripts`` or ``post_setup`` in +subclasses to assist in installing custom scripts into the virtual +environment. The method accepts as arguments the context object (see +above) and a path to a directory. The directory should contain +subdirectories "common", "posix", "nt", each containing scripts +destined for the bin directory in the environment. The contents of +"common" and the directory corresponding to ``os.name`` are copied +after some text replacement of placeholders: + +* ``__VENV_DIR__`` is replaced with the absolute path of the + environment directory. + +* ``__VENV_NAME__`` is replaced with the environment name (final path + segment of environment directory). + +* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory + (either ``bin`` or ``Scripts``). + +* ``__VENV_PYTHON__`` is replaced with the absolute path of the + environment's executable. diff --git a/Lib/distutils/sysconfig.py b/Lib/distutils/sysconfig.py --- a/Lib/distutils/sysconfig.py +++ b/Lib/distutils/sysconfig.py @@ -18,6 +18,8 @@ # These are needed in a couple of spots, so just compute them once. PREFIX = os.path.normpath(sys.prefix) EXEC_PREFIX = os.path.normpath(sys.exec_prefix) +BASE_PREFIX = os.path.normpath(sys.base_prefix) +BASE_EXEC_PREFIX = os.path.normpath(sys.base_exec_prefix) # Path to the base directory of the project. On Windows the binary may # live in project/PCBuild9. If we're dealing with an x64 Windows build, @@ -39,11 +41,18 @@ # different (hard-wired) directories. # Setup.local is available for Makefile builds including VPATH builds, # Setup.dist is available on Windows -def _python_build(): +def _is_python_source_dir(d): for fn in ("Setup.dist", "Setup.local"): - if os.path.isfile(os.path.join(project_base, "Modules", fn)): + if os.path.isfile(os.path.join(d, "Modules", fn)): return True return False +_sys_home = getattr(sys, '_home', None) +if _sys_home and os.name == 'nt' and _sys_home.lower().endswith('pcbuild'): + _sys_home = os.path.dirname(_sys_home) +def _python_build(): + if _sys_home: + return _is_python_source_dir(_sys_home) + return _is_python_source_dir(project_base) python_build = _python_build() # Calculate the build qualifier flags if they are defined. Adding the flags @@ -74,11 +83,11 @@ otherwise, this is the path to platform-specific header files (namely pyconfig.h). - If 'prefix' is supplied, use it instead of sys.prefix or - sys.exec_prefix -- i.e., ignore 'plat_specific'. + If 'prefix' is supplied, use it instead of sys.base_prefix or + sys.base_exec_prefix -- i.e., ignore 'plat_specific'. """ if prefix is None: - prefix = plat_specific and EXEC_PREFIX or PREFIX + prefix = plat_specific and BASE_EXEC_PREFIX or BASE_PREFIX if os.name == "posix": if python_build: # Assume the executable is in the build directory. The @@ -86,11 +95,12 @@ # the build directory may not be the source directory, we # must use "srcdir" from the makefile to find the "Include" # directory. - base = os.path.dirname(os.path.abspath(sys.executable)) + base = _sys_home or os.path.dirname(os.path.abspath(sys.executable)) if plat_specific: return base else: - incdir = os.path.join(get_config_var('srcdir'), 'Include') + incdir = os.path.join(_sys_home or get_config_var('srcdir'), + 'Include') return os.path.normpath(incdir) python_dir = 'python' + get_python_version() + build_flags return os.path.join(prefix, "include", python_dir) @@ -115,11 +125,14 @@ containing standard Python library modules; otherwise, return the directory for site-specific modules. - If 'prefix' is supplied, use it instead of sys.prefix or - sys.exec_prefix -- i.e., ignore 'plat_specific'. + If 'prefix' is supplied, use it instead of sys.base_prefix or + sys.base_exec_prefix -- i.e., ignore 'plat_specific'. """ if prefix is None: - prefix = plat_specific and EXEC_PREFIX or PREFIX + if standard_lib: + prefix = plat_specific and BASE_EXEC_PREFIX or BASE_PREFIX + else: + prefix = plat_specific and EXEC_PREFIX or PREFIX if os.name == "posix": libpython = os.path.join(prefix, @@ -232,9 +245,9 @@ """Return full pathname of installed pyconfig.h file.""" if python_build: if os.name == "nt": - inc_dir = os.path.join(project_base, "PC") + inc_dir = os.path.join(_sys_home or project_base, "PC") else: - inc_dir = project_base + inc_dir = _sys_home or project_base else: inc_dir = get_python_inc(plat_specific=1) if get_python_version() < '2.2': @@ -248,7 +261,8 @@ def get_makefile_filename(): """Return full pathname of installed Makefile from the Python build.""" if python_build: - return os.path.join(os.path.dirname(sys.executable), "Makefile") + return os.path.join(_sys_home or os.path.dirname(sys.executable), + "Makefile") lib_dir = get_python_lib(plat_specific=0, standard_lib=1) config_file = 'config-{}{}'.format(get_python_version(), build_flags) return os.path.join(lib_dir, config_file, 'Makefile') diff --git a/Lib/gettext.py b/Lib/gettext.py --- a/Lib/gettext.py +++ b/Lib/gettext.py @@ -55,7 +55,7 @@ 'dgettext', 'dngettext', 'gettext', 'ngettext', ] -_default_localedir = os.path.join(sys.prefix, 'share', 'locale') +_default_localedir = os.path.join(sys.base_prefix, 'share', 'locale') def c2py(plural): diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -120,7 +120,7 @@ def __init__(self, flist=None, filename=None, key=None, root=None): if EditorWindow.help_url is None: - dochome = os.path.join(sys.prefix, 'Doc', 'index.html') + dochome = os.path.join(sys.base_prefix, 'Doc', 'index.html') if sys.platform.count('linux'): # look for html docs in a couple of standard places pyver = 'python-docs-' + '%s.%s.%s' % sys.version_info[:3] @@ -131,13 +131,13 @@ dochome = os.path.join(basepath, pyver, 'Doc', 'index.html') elif sys.platform[:3] == 'win': - chmfile = os.path.join(sys.prefix, 'Doc', + chmfile = os.path.join(sys.base_prefix, 'Doc', 'Python%s.chm' % _sphinx_version()) if os.path.isfile(chmfile): dochome = chmfile elif macosxSupport.runningAsOSXApp(): # documentation is stored inside the python framework - dochome = os.path.join(sys.prefix, + dochome = os.path.join(sys.base_prefix, 'Resources/English.lproj/Documentation/index.html') dochome = os.path.normpath(dochome) if os.path.isfile(dochome): diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py --- a/Lib/packaging/command/build_ext.py +++ b/Lib/packaging/command/build_ext.py @@ -182,7 +182,10 @@ # the 'libs' directory is for binary installs - we assume that # must be the *native* platform. But we don't really support # cross-compiling via a binary install anyway, so we let it go. - self.library_dirs.append(os.path.join(sys.exec_prefix, 'libs')) + # Note that we must use sys.base_exec_prefix here rather than + # exec_prefix, since the Python libs are not copied to a virtual + # environment. + self.library_dirs.append(os.path.join(sys.base_exec_prefix, 'libs')) if self.debug: self.build_temp = os.path.join(self.build_temp, "Debug") else: diff --git a/Lib/pydoc.py b/Lib/pydoc.py --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -369,7 +369,7 @@ docloc = os.environ.get("PYTHONDOCS", self.PYTHONDOCS) - basedir = os.path.join(sys.exec_prefix, "lib", + basedir = os.path.join(sys.base_exec_prefix, "lib", "python%d.%d" % sys.version_info[:2]) if (isinstance(object, type(os)) and (object.__name__ in ('errno', 'exceptions', 'gc', 'imp', diff --git a/Lib/site.py b/Lib/site.py --- a/Lib/site.py +++ b/Lib/site.py @@ -13,6 +13,19 @@ resulting directories, if they exist, are appended to sys.path, and also inspected for path configuration files. +If a file named "pyvenv.cfg" exists one directory above sys.executable, +sys.prefix and sys.exec_prefix are set to that directory and +it is also checked for site-packages and site-python (sys.prefix and +sys.exec_prefix will always be the "real" prefixes of the Python +installation). If "pyvenv.cfg" (a bootstrap configuration file) contains +the key "include-system-site-packages" set to anything other than "false" +(case-insensitive), the system-level prefixes will still also be +searched for site-packages; otherwise they won't. + +All of the resulting site-specific directories, if they exist, are +appended to sys.path, and also inspected for path configuration +files. + A path configuration file is a file whose name has the form .pth; its contents are additional directories (one per line) to be added to sys.path. Non-existing directories (or @@ -54,6 +67,7 @@ import sys import os +import re import builtins # Prefixes for site-packages; add additional prefixes like /usr/local here @@ -179,6 +193,7 @@ sitedir, sitedircase = makepath(sitedir) if not sitedircase in known_paths: sys.path.append(sitedir) # Add path component + known_paths.add(sitedircase) try: names = os.listdir(sitedir) except os.error: @@ -266,18 +281,21 @@ addsitedir(user_site, known_paths) return known_paths -def getsitepackages(): +def getsitepackages(prefixes=None): """Returns a list containing all global site-packages directories (and possibly site-python). - For each directory present in the global ``PREFIXES``, this function - will find its `site-packages` subdirectory depending on the system - environment, and will return a list of full paths. + For each directory present in ``prefixes`` (or the global ``PREFIXES``), + this function will find its `site-packages` subdirectory depending on the + system environment, and will return a list of full paths. """ sitepackages = [] seen = set() - for prefix in PREFIXES: + if prefixes is None: + prefixes = PREFIXES + + for prefix in prefixes: if not prefix or prefix in seen: continue seen.add(prefix) @@ -303,9 +321,9 @@ sys.version[:3], "site-packages")) return sitepackages -def addsitepackages(known_paths): +def addsitepackages(known_paths, prefixes=None): """Add site-packages (and possibly site-python) to sys.path""" - for sitedir in getsitepackages(): + for sitedir in getsitepackages(prefixes): if os.path.isdir(sitedir): addsitedir(sitedir, known_paths) @@ -475,6 +493,61 @@ encodings.aliases.aliases[enc] = 'mbcs' +CONFIG_LINE = re.compile(r'^(?P(\w|[-_])+)\s*=\s*(?P.*)\s*$') + +def venv(known_paths): + global PREFIXES, ENABLE_USER_SITE + + env = os.environ + if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env: + executable = os.environ['__PYTHONV_LAUNCHER__'] + else: + executable = sys.executable + executable_dir, executable_name = os.path.split(executable) + site_prefix = os.path.dirname(executable_dir) + sys._home = None + if sys.platform == 'win32': + executable_name = os.path.splitext(executable_name)[0] + conf_basename = 'pyvenv.cfg' + candidate_confs = [ + conffile for conffile in ( + os.path.join(executable_dir, conf_basename), + os.path.join(site_prefix, conf_basename) + ) + if os.path.isfile(conffile) + ] + + if candidate_confs: + virtual_conf = candidate_confs[0] + system_site = "true" + with open(virtual_conf) as f: + for line in f: + line = line.strip() + m = CONFIG_LINE.match(line) + if m: + d = m.groupdict() + key, value = d['key'].lower(), d['value'] + if key == 'include-system-site-packages': + system_site = value.lower() + elif key == 'home': + sys._home = value + + sys.prefix = sys.exec_prefix = site_prefix + + # Doing this here ensures venv takes precedence over user-site + addsitepackages(known_paths, [sys.prefix]) + + # addsitepackages will process site_prefix again if its in PREFIXES, + # but that's ok; known_paths will prevent anything being added twice + if system_site == "true": + PREFIXES.insert(0, sys.prefix) + else: + PREFIXES = [sys.prefix] + ENABLE_USER_SITE = False + + return known_paths + + def execsitecustomize(): """Run custom site specific code, if available.""" try: @@ -517,6 +590,7 @@ abs_paths() known_paths = removeduppaths() + known_paths = venv(known_paths) if ENABLE_USER_SITE is None: ENABLE_USER_SITE = check_enableusersite() known_paths = addusersitepackages(known_paths) diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -1021,7 +1021,7 @@ if not os.path.exists(w9xpopen): # Eeek - file-not-found - possibly an embedding # situation - see if we can locate it in sys.exec_prefix - w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix), + w9xpopen = os.path.join(os.path.dirname(sys.base_exec_prefix), "w9xpopen.exe") if not os.path.exists(w9xpopen): raise RuntimeError("Cannot locate w9xpopen.exe, which is " diff --git a/Lib/sysconfig.cfg b/Lib/sysconfig.cfg --- a/Lib/sysconfig.cfg +++ b/Lib/sysconfig.cfg @@ -36,41 +36,41 @@ # User resource directory local = ~/.local/{distribution.name} -stdlib = {base}/lib/python{py_version_short} +stdlib = {installed_base}/lib/python{py_version_short} platstdlib = {platbase}/lib/python{py_version_short} purelib = {base}/lib/python{py_version_short}/site-packages platlib = {platbase}/lib/python{py_version_short}/site-packages -include = {base}/include/python{py_version_short}{abiflags} -platinclude = {platbase}/include/python{py_version_short}{abiflags} +include = {installed_base}/include/python{py_version_short}{abiflags} +platinclude = {installed_platbase}/include/python{py_version_short}{abiflags} data = {base} [posix_home] -stdlib = {base}/lib/python +stdlib = {installed_base}/lib/python platstdlib = {base}/lib/python purelib = {base}/lib/python platlib = {base}/lib/python -include = {base}/include/python -platinclude = {base}/include/python +include = {installed_base}/include/python +platinclude = {installed_base}/include/python scripts = {base}/bin data = {base} [nt] -stdlib = {base}/Lib +stdlib = {installed_base}/Lib platstdlib = {base}/Lib purelib = {base}/Lib/site-packages platlib = {base}/Lib/site-packages -include = {base}/Include -platinclude = {base}/Include +include = {installed_base}/Include +platinclude = {installed_base}/Include scripts = {base}/Scripts data = {base} [os2] -stdlib = {base}/Lib +stdlib = {installed_base}/Lib platstdlib = {base}/Lib purelib = {base}/Lib/site-packages platlib = {base}/Lib/site-packages -include = {base}/Include -platinclude = {base}/Include +include = {installed_base}/Include +platinclude = {installed_base}/Include scripts = {base}/Scripts data = {base} diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -3,6 +3,7 @@ import os import re import sys +import os from os.path import pardir, realpath from configparser import RawConfigParser @@ -61,13 +62,15 @@ _expand_globals(_SCHEMES) - # FIXME don't rely on sys.version here, its format is an implementatin detail + # FIXME don't rely on sys.version here, its format is an implementation detail # of CPython, use sys.version_info or sys.hexversion _PY_VERSION = sys.version.split()[0] _PY_VERSION_SHORT = sys.version[:3] _PY_VERSION_SHORT_NO_DOT = _PY_VERSION[0] + _PY_VERSION[2] _PREFIX = os.path.normpath(sys.prefix) +_BASE_PREFIX = os.path.normpath(sys.base_prefix) _EXEC_PREFIX = os.path.normpath(sys.exec_prefix) +_BASE_EXEC_PREFIX = os.path.normpath(sys.base_exec_prefix) _CONFIG_VARS = None _USER_BASE = None @@ -94,14 +97,22 @@ if os.name == "nt" and "\\pcbuild\\amd64" in _PROJECT_BASE[-14:].lower(): _PROJECT_BASE = _safe_realpath(os.path.join(_PROJECT_BASE, pardir, pardir)) - -def is_python_build(): +def _is_python_source_dir(d): for fn in ("Setup.dist", "Setup.local"): - if os.path.isfile(os.path.join(_PROJECT_BASE, "Modules", fn)): + if os.path.isfile(os.path.join(d, "Modules", fn)): return True return False -_PYTHON_BUILD = is_python_build() +_sys_home = getattr(sys, '_home', None) +if _sys_home and os.name == 'nt' and _sys_home.lower().endswith('pcbuild'): + _sys_home = os.path.dirname(_sys_home) + +def is_python_build(check_home=False): + if check_home and _sys_home: + return _is_python_source_dir(_sys_home) + return _is_python_source_dir(_PROJECT_BASE) + +_PYTHON_BUILD = is_python_build(True) if _PYTHON_BUILD: for scheme in ('posix_prefix', 'posix_home'): @@ -312,7 +323,7 @@ def get_makefile_filename(): """Return the path of the Makefile.""" if _PYTHON_BUILD: - return os.path.join(_PROJECT_BASE, "Makefile") + return os.path.join(_sys_home or _PROJECT_BASE, "Makefile") if hasattr(sys, 'abiflags'): config_dir_name = 'config-%s%s' % (_PY_VERSION_SHORT, sys.abiflags) else: @@ -412,9 +423,9 @@ """Return the path of pyconfig.h.""" if _PYTHON_BUILD: if os.name == "nt": - inc_dir = os.path.join(_PROJECT_BASE, "PC") + inc_dir = os.path.join(_sys_home or _PROJECT_BASE, "PC") else: - inc_dir = _PROJECT_BASE + inc_dir = _sys_home or _PROJECT_BASE else: inc_dir = get_path('platinclude') return os.path.join(inc_dir, 'pyconfig.h') @@ -472,7 +483,9 @@ _CONFIG_VARS['py_version'] = _PY_VERSION _CONFIG_VARS['py_version_short'] = _PY_VERSION_SHORT _CONFIG_VARS['py_version_nodot'] = _PY_VERSION[0] + _PY_VERSION[2] + _CONFIG_VARS['installed_base'] = _BASE_PREFIX _CONFIG_VARS['base'] = _PREFIX + _CONFIG_VARS['installed_platbase'] = _BASE_EXEC_PREFIX _CONFIG_VARS['platbase'] = _EXEC_PREFIX _CONFIG_VARS['projectbase'] = _PROJECT_BASE try: diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py --- a/Lib/test/regrtest.py +++ b/Lib/test/regrtest.py @@ -564,7 +564,7 @@ random.shuffle(selected) if trace: import trace, tempfile - tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix, + tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix, tempfile.gettempdir()], trace=False, count=True) diff --git a/Lib/test/test_cmd.py b/Lib/test/test_cmd.py --- a/Lib/test/test_cmd.py +++ b/Lib/test/test_cmd.py @@ -228,7 +228,7 @@ def test_coverage(coverdir): trace = support.import_module('trace') - tracer=trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], + tracer=trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix,], trace=0, count=1) tracer.run('reload(cmd);test_main()') r=tracer.results() diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -2543,7 +2543,7 @@ def test_coverage(coverdir): trace = support.import_module('trace') - tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], + tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix,], trace=0, count=1) tracer.run('test_main()') r = tracer.results() diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -189,6 +189,8 @@ p.wait() self.assertEqual(p.stderr, None) + @unittest.skipIf(sys.base_prefix != sys.prefix, + 'Test is not venv-compatible') def test_executable_with_cwd(self): python_dir = os.path.dirname(os.path.realpath(sys.executable)) p = subprocess.Popen(["somethingyoudonthave", "-c", @@ -197,6 +199,8 @@ p.wait() self.assertEqual(p.returncode, 47) + @unittest.skipIf(sys.base_prefix != sys.prefix, + 'Test is not venv-compatible') @unittest.skipIf(sysconfig.is_python_build(), "need an installed Python. See #7774") def test_executable_without_cwd(self): diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -419,6 +419,7 @@ self.assertIsInstance(sys.builtin_module_names, tuple) self.assertIsInstance(sys.copyright, str) self.assertIsInstance(sys.exec_prefix, str) + self.assertIsInstance(sys.base_exec_prefix, str) self.assertIsInstance(sys.executable, str) self.assertEqual(len(sys.float_info), 11) self.assertEqual(sys.float_info.radix, 2) @@ -450,6 +451,7 @@ self.assertEqual(sys.maxunicode, 0x10FFFF) self.assertIsInstance(sys.platform, str) self.assertIsInstance(sys.prefix, str) + self.assertIsInstance(sys.base_prefix, str) self.assertIsInstance(sys.version, str) vi = sys.version_info self.assertIsInstance(vi[:], tuple) @@ -541,6 +543,8 @@ out = p.communicate()[0].strip() self.assertEqual(out, b'?') + @unittest.skipIf(sys.base_prefix != sys.prefix, + 'Test is not venv-compatible') def test_executable(self): # sys.executable should be absolute self.assertEqual(os.path.abspath(sys.executable), sys.executable) diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py --- a/Lib/test/test_sysconfig.py +++ b/Lib/test/test_sysconfig.py @@ -260,12 +260,17 @@ # the global scheme mirrors the distinction between prefix and # exec-prefix but not the user scheme, so we have to adapt the paths # before comparing (issue #9100) - adapt = sys.prefix != sys.exec_prefix + adapt = sys.base_prefix != sys.base_exec_prefix for name in ('stdlib', 'platstdlib', 'purelib', 'platlib'): global_path = get_path(name, 'posix_prefix') if adapt: - global_path = global_path.replace(sys.exec_prefix, sys.prefix) - base = base.replace(sys.exec_prefix, sys.prefix) + global_path = global_path.replace(sys.exec_prefix, sys.base_prefix) + base = base.replace(sys.exec_prefix, sys.base_prefix) + elif sys.base_prefix != sys.prefix: + # virtual environment? Likewise, we have to adapt the paths + # before comparing + global_path = global_path.replace(sys.base_prefix, sys.prefix) + base = base.replace(sys.base_prefix, sys.prefix) user_path = get_path(name, 'posix_user') self.assertEqual(user_path, global_path.replace(base, user, 1)) diff --git a/Lib/test/test_trace.py b/Lib/test/test_trace.py --- a/Lib/test/test_trace.py +++ b/Lib/test/test_trace.py @@ -316,8 +316,8 @@ # Ignore all files, nothing should be traced nor printed libpath = os.path.normpath(os.path.dirname(os.__file__)) # sys.prefix does not work when running from a checkout - tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix, libpath], - trace=0, count=1) + tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix, + libpath], trace=0, count=1) with captured_stdout() as stdout: self._coverage(tracer) if os.path.exists(TESTFN): diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_venv.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python +# +# Copyright 2011 by Vinay Sajip. All Rights Reserved. +# +# Permission to use, copy, modify, and distribute this software and its +# documentation for any purpose and without fee is hereby granted, +# provided that the above copyright notice appear in all copies and that +# both that copyright notice and this permission notice appear in +# supporting documentation, and that the name of Vinay Sajip +# not be used in advertising or publicity pertaining to distribution +# of the software without specific, written prior permission. +# VINAY SAJIP DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING +# ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL +# VINAY SAJIP BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR +# ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER +# IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT +# OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + +"""Test harness for the venv module. Run all tests. + +Copyright (C) 2011 Vinay Sajip. All Rights Reserved. +""" + +import os +import os.path +import shutil +import sys +import tempfile +from test.support import (captured_stdout, captured_stderr, run_unittest, + can_symlink) +import unittest +import venv + +class BaseTest(unittest.TestCase): + """Base class for venv tests.""" + + def setUp(self): + self.env_dir = tempfile.mkdtemp() + if os.name == 'nt': + self.bindir = 'Scripts' + self.ps3name = 'pysetup3-script.py' + self.lib = ('Lib',) + self.include = 'Include' + self.exe = 'python.exe' + else: + self.bindir = 'bin' + self.ps3name = 'pysetup3' + self.lib = ('lib', 'python%s' % sys.version[:3]) + self.include = 'include' + self.exe = 'python' + + def tearDown(self): + shutil.rmtree(self.env_dir) + + def run_with_capture(self, func, *args, **kwargs): + with captured_stdout() as output: + with captured_stderr() as error: + func(*args, **kwargs) + return output.getvalue(), error.getvalue() + + def get_env_file(self, *args): + return os.path.join(self.env_dir, *args) + + def get_text_file_contents(self, *args): + with open(self.get_env_file(*args), 'r') as f: + result = f.read() + return result + +class BasicTest(BaseTest): + """Test venv module functionality.""" + + def test_defaults(self): + """ + Test the create function with default arguments. + """ + def isdir(*args): + fn = self.get_env_file(*args) + self.assertTrue(os.path.isdir(fn)) + + shutil.rmtree(self.env_dir) + self.run_with_capture(venv.create, self.env_dir) + isdir(self.bindir) + isdir(self.include) + isdir(*self.lib) + data = self.get_text_file_contents('pyvenv.cfg') + if sys.platform == 'darwin' and ('__PYTHONV_LAUNCHER__' + in os.environ): + executable = os.environ['__PYTHONV_LAUNCHER__'] + else: + executable = sys.executable + path = os.path.dirname(executable) + self.assertIn('home = %s' % path, data) + data = self.get_text_file_contents(self.bindir, self.ps3name) + self.assertTrue(data.startswith('#!%s%s' % (self.env_dir, os.sep))) + fn = self.get_env_file(self.bindir, self.exe) + self.assertTrue(os.path.exists(fn)) + + def test_overwrite_existing(self): + """ + Test control of overwriting an existing environment directory. + """ + self.assertRaises(ValueError, venv.create, self.env_dir) + builder = venv.EnvBuilder(clear=True) + builder.create(self.env_dir) + + def test_isolation(self): + """ + Test isolation from system site-packages + """ + for ssp, s in ((True, 'true'), (False, 'false')): + builder = venv.EnvBuilder(clear=True, system_site_packages=ssp) + builder.create(self.env_dir) + data = self.get_text_file_contents('pyvenv.cfg') + self.assertIn('include-system-site-packages = %s\n' % s, data) + + @unittest.skipUnless(can_symlink(), 'Needs symlinks') + def test_symlinking(self): + """ + Test symlinking works as expected + """ + for usl in (False, True): + builder = venv.EnvBuilder(clear=True, symlinks=usl) + if (usl and sys.platform == 'darwin' and + '__PYTHONV_LAUNCHER__' in os.environ): + self.assertRaises(ValueError, builder.create, self.env_dir) + else: + builder.create(self.env_dir) + fn = self.get_env_file(self.bindir, self.exe) + # Don't test when False, because e.g. 'python' is always + # symlinked to 'python3.3' in the env, even when symlinking in + # general isn't wanted. + if usl: + self.assertTrue(os.path.islink(fn)) + +def test_main(): + run_unittest(BasicTest) + +if __name__ == "__main__": + test_main() diff --git a/Lib/tkinter/_fix.py b/Lib/tkinter/_fix.py --- a/Lib/tkinter/_fix.py +++ b/Lib/tkinter/_fix.py @@ -46,10 +46,10 @@ s = "\\" + s[3:] return s -prefix = os.path.join(sys.prefix,"tcl") +prefix = os.path.join(sys.base_prefix,"tcl") if not os.path.exists(prefix): # devdir/../tcltk/lib - prefix = os.path.join(sys.prefix, os.path.pardir, "tcltk", "lib") + prefix = os.path.join(sys.base_prefix, os.path.pardir, "tcltk", "lib") prefix = os.path.abspath(prefix) # if this does not exist, no further search is needed if os.path.exists(prefix): diff --git a/Lib/trace.py b/Lib/trace.py --- a/Lib/trace.py +++ b/Lib/trace.py @@ -39,8 +39,8 @@ # create a Trace object, telling it what to ignore, and whether to # do tracing or line-counting or both. - tracer = trace.Trace(ignoredirs=[sys.prefix, sys.exec_prefix,], trace=0, - count=1) + tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix,], + trace=0, count=1) # run the new command using the given tracer tracer.run('main()') # make a report, placing output in /tmp @@ -749,10 +749,10 @@ # should I also call expanduser? (after all, could use $HOME) s = s.replace("$prefix", - os.path.join(sys.prefix, "lib", + os.path.join(sys.base_prefix, "lib", "python" + sys.version[:3])) s = s.replace("$exec_prefix", - os.path.join(sys.exec_prefix, "lib", + os.path.join(sys.base_exec_prefix, "lib", "python" + sys.version[:3])) s = os.path.normpath(s) ignore_dirs.append(s) diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py new file mode 100644 --- /dev/null +++ b/Lib/venv/__init__.py @@ -0,0 +1,502 @@ +# Copyright (C) 2011-2012 Vinay Sajip. +# +# Use with a Python executable built from the Python fork at +# +# https://bitbucket.org/vinay.sajip/pythonv/ as follows: +# +# python -m venv env_dir +# +# You'll need an Internet connection (needed to download distribute_setup.py). +# +# The script will change to the environment's binary directory and run +# +# ./python distribute_setup.py +# +# after which you can change to the environment's directory and do some +# installations, e.g. +# +# source bin/activate.sh +# pysetup3 install setuptools-git +# pysetup3 install Pygments +# pysetup3 install Jinja2 +# pysetup3 install SQLAlchemy +# pysetup3 install coverage +# +# Note that on Windows, distributions which include C extensions (e.g. coverage) +# may fail due to lack of a suitable C compiler. +# +import base64 +import io +import logging +import os +import os.path +import shutil +import sys +import zipfile + +logger = logging.getLogger(__name__) + +class Context: + """ + Holds information about a current virtualisation request. + """ + pass + + +class EnvBuilder: + """ + This class exists to allow virtual environment creation to be + customised. The constructor parameters determine the builder's + behaviour when called upon to create a virtual environment. + + By default, the builder makes the system (global) site-packages dir + available to the created environment. + + By default, the creation process uses symlinks wherever possible. + + :param system_site_packages: If True, the system (global) site-packages + dir is available to created environments. + :param clear: If True and the target directory exists, it is deleted. + Otherwise, if the target directory exists, an error is + raised. + :param symlinks: If True, attempt to symlink rather than copy files into + virtual environment. + :param upgrade: If True, upgrade an existing virtual environment. + """ + + def __init__(self, system_site_packages=False, clear=False, + symlinks=False, upgrade=False): + self.system_site_packages = system_site_packages + self.clear = clear + self.symlinks = symlinks + self.upgrade = upgrade + + def create(self, env_dir): + """ + Create a virtual environment in a directory. + + :param env_dir: The target directory to create an environment in. + + """ + if (self.symlinks and + sys.platform == 'darwin' and + 'Library/Framework' in sys.base_prefix): + # Symlinking the stub executable in an OSX framework build will + # result in a broken virtual environment. + raise ValueError( + "Symlinking is not supported on OSX framework Python.") + env_dir = os.path.abspath(env_dir) + context = self.ensure_directories(env_dir) + self.create_configuration(context) + self.setup_python(context) + if not self.upgrade: + self.setup_scripts(context) + self.post_setup(context) + + def ensure_directories(self, env_dir): + """ + Create the directories for the environment. + + Returns a context object which holds paths in the environment, + for use by subsequent logic. + """ + + def create_if_needed(d): + if not os.path.exists(d): + os.makedirs(d) + + if os.path.exists(env_dir) and not (self.clear or self.upgrade): + raise ValueError('Directory exists: %s' % env_dir) + if os.path.exists(env_dir) and self.clear: + shutil.rmtree(env_dir) + context = Context() + context.env_dir = env_dir + context.env_name = os.path.split(env_dir)[1] + context.prompt = '(%s) ' % context.env_name + create_if_needed(env_dir) + env = os.environ + if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env: + executable = os.environ['__PYTHONV_LAUNCHER__'] + else: + executable = sys.executable + dirname, exename = os.path.split(os.path.abspath(executable)) + context.executable = executable + context.python_dir = dirname + context.python_exe = exename + if sys.platform == 'win32': + binname = 'Scripts' + incpath = 'Include' + libpath = os.path.join(env_dir, 'Lib', 'site-packages') + else: + binname = 'bin' + incpath = 'include' + libpath = os.path.join(env_dir, 'lib', 'python%d.%d' % sys.version_info[:2], 'site-packages') + context.inc_path = path = os.path.join(env_dir, incpath) + create_if_needed(path) + create_if_needed(libpath) + context.bin_path = binpath = os.path.join(env_dir, binname) + context.bin_name = binname + context.env_exe = os.path.join(binpath, exename) + create_if_needed(binpath) + return context + + def create_configuration(self, context): + """ + Create a configuration file indicating where the environment's Python + was copied from, and whether the system site-packages should be made + available in the environment. + + :param context: The information for the environment creation request + being processed. + """ + context.cfg_path = path = os.path.join(context.env_dir, 'pyvenv.cfg') + with open(path, 'w', encoding='utf-8') as f: + f.write('home = %s\n' % context.python_dir) + if self.system_site_packages: + incl = 'true' + else: + incl = 'false' + f.write('include-system-site-packages = %s\n' % incl) + f.write('version = %d.%d.%d\n' % sys.version_info[:3]) + + if os.name == 'nt': + def include_binary(self, f): + if f.endswith(('.pyd', '.dll')): + result = True + else: + result = f.startswith('python') and f.endswith('.exe') + return result + + def symlink_or_copy(self, src, dst): + """ + Try symlinking a file, and if that fails, fall back to copying. + """ + force_copy = not self.symlinks + if not force_copy: + try: + if not os.path.islink(dst): # can't link to itself! + os.symlink(src, dst) + except Exception: # may need to use a more specific exception + logger.warning('Unable to symlink %r to %r', src, dst) + force_copy = True + if force_copy: + shutil.copyfile(src, dst) + + def setup_python(self, context): + """ + Set up a Python executable in the environment. + + :param context: The information for the environment creation request + being processed. + """ + binpath = context.bin_path + exename = context.python_exe + path = context.env_exe + copier = self.symlink_or_copy + copier(context.executable, path) + dirname = context.python_dir + if os.name != 'nt': + if not os.path.islink(path): + os.chmod(path, 0o755) + path = os.path.join(binpath, 'python') + if not os.path.exists(path): + os.symlink(exename, path) + else: + subdir = 'DLLs' + include = self.include_binary + files = [f for f in os.listdir(dirname) if include(f)] + for f in files: + src = os.path.join(dirname, f) + dst = os.path.join(binpath, f) + if dst != context.env_exe: # already done, above + copier(src, dst) + dirname = os.path.join(dirname, subdir) + if os.path.isdir(dirname): + files = [f for f in os.listdir(dirname) if include(f)] + for f in files: + src = os.path.join(dirname, f) + dst = os.path.join(binpath, f) + copier(src, dst) + # copy init.tcl over + for root, dirs, files in os.walk(context.python_dir): + if 'init.tcl' in files: + tcldir = os.path.basename(root) + tcldir = os.path.join(context.env_dir, 'Lib', tcldir) + os.makedirs(tcldir) + src = os.path.join(root, 'init.tcl') + dst = os.path.join(tcldir, 'init.tcl') + shutil.copyfile(src, dst) + break + + def setup_scripts(self, context): + """ + Set up scripts into the created environment from a directory. + + This method installs the default scripts into the environment + being created. You can prevent the default installation by overriding + this method if you really need to, or if you need to specify + a different location for the scripts to install. By default, the + 'scripts' directory in the venv package is used as the source of + scripts to install. + """ + path = os.path.abspath(os.path.dirname(__file__)) + path = os.path.join(path, 'scripts') + self.install_scripts(context, path) + + def post_setup(self, context): + """ + Hook for post-setup modification of the venv. Subclasses may install + additional packages or scripts here, add activation shell scripts, etc. + + :param context: The information for the environment creation request + being processed. + """ + pass + + def replace_variables(self, text, context): + """ + Replace variable placeholders in script text with context-specific + variables. + + Return the text passed in , but with variables replaced. + + :param text: The text in which to replace placeholder variables. + :param context: The information for the environment creation request + being processed. + """ + text = text.replace('__VENV_DIR__', context.env_dir) + text = text.replace('__VENV_NAME__', context.prompt) + text = text.replace('__VENV_BIN_NAME__', context.bin_name) + text = text.replace('__VENV_PYTHON__', context.env_exe) + return text + + def install_scripts(self, context, path): + """ + Install scripts into the created environment from a directory. + + :param context: The information for the environment creation request + being processed. + :param path: Absolute pathname of a directory containing script. + Scripts in the 'common' subdirectory of this directory, + and those in the directory named for the platform + being run on, are installed in the created environment. + Placeholder variables are replaced with environment- + specific values. + """ + binpath = context.bin_path + plen = len(path) + for root, dirs, files in os.walk(path): + if root == path: # at top-level, remove irrelevant dirs + for d in dirs[:]: + if d not in ('common', os.name): + dirs.remove(d) + continue # ignore files in top level + for f in files: + srcfile = os.path.join(root, f) + suffix = root[plen:].split(os.sep)[2:] + if not suffix: + dstdir = binpath + else: + dstdir = os.path.join(binpath, *suffix) + if not os.path.exists(dstdir): + os.makedirs(dstdir) + dstfile = os.path.join(dstdir, f) + with open(srcfile, 'rb') as f: + data = f.read() + if srcfile.endswith('.exe'): + mode = 'wb' + else: + mode = 'w' + data = data.decode('utf-8') + data = self.replace_variables(data, context) + with open(dstfile, mode) as f: + f.write(data) + os.chmod(dstfile, 0o755) + + +# This class will not be included in Python core; it's here for now to +# facilitate experimentation and testing, and as proof-of-concept of what could +# be done by external extension tools. +class DistributeEnvBuilder(EnvBuilder): + """ + By default, this builder installs Distribute so that you can pip or + easy_install other packages into the created environment. + + :param nodist: If True, Distribute is not installed into the created + environment. + :param progress: If Distribute is installed, the progress of the + installation can be monitored by passing a progress + callable. If specified, it is called with two + arguments: a string indicating some progress, and a + context indicating where the string is coming from. + The context argument can have one of three values: + 'main', indicating that it is called from virtualize() + itself, and 'stdout' and 'stderr', which are obtained + by reading lines from the output streams of a subprocess + which is used to install Distribute. + + If a callable is not specified, default progress + information is output to sys.stderr. + """ + + def __init__(self, *args, **kwargs): + self.nodist = kwargs.pop("nodist", False) + self.progress = kwargs.pop("progress", None) + super().__init__(*args, **kwargs) + + def post_setup(self, context): + """ + Set up any packages which need to be pre-installed into the + environment being created. + + :param context: The information for the environment creation request + being processed. + """ + if not self.nodist: + self.install_distribute(context) + + def reader(self, stream, context): + """ + Read lines from a subprocess' output stream and either pass to a progress + callable (if specified) or write progress information to sys.stderr. + """ + progress = self.progress + while True: + s = stream.readline() + if not s: + break + if progress is not None: + progress(s, context) + else: + sys.stderr.write('.') + #sys.stderr.write(s.decode('utf-8')) + sys.stderr.flush() + stream.close() + + def install_distribute(self, context): + """ + Install Distribute in the environment. + + :param context: The information for the environment creation request + being processed. + """ + from subprocess import Popen, PIPE + from threading import Thread + from urllib.request import urlretrieve + + url = 'http://python-distribute.org/distribute_setup.py' + binpath = context.bin_path + distpath = os.path.join(binpath, 'distribute_setup.py') + # Download Distribute in the env + urlretrieve(url, distpath) + progress = self.progress + if progress is not None: + progress('Installing distribute', 'main') + else: + sys.stderr.write('Installing distribute ') + sys.stderr.flush() + # Install Distribute in the env + args = [context.env_exe, 'distribute_setup.py'] + p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) + t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) + t1.start() + t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) + t2.start() + p.wait() + t1.join() + t2.join() + if progress is not None: + progress('done.', 'main') + else: + sys.stderr.write('done.\n') + # Clean up - no longer needed + os.unlink(distpath) + +def create(env_dir, system_site_packages=False, clear=False, symlinks=False): + """ + Create a virtual environment in a directory. + + By default, makes the system (global) site-packages dir available to + the created environment. + + :param env_dir: The target directory to create an environment in. + :param system_site_packages: If True, the system (global) site-packages + dir is available to the environment. + :param clear: If True and the target directory exists, it is deleted. + Otherwise, if the target directory exists, an error is + raised. + :param symlinks: If True, attempt to symlink rather than copy files into + virtual environment. + """ + # XXX This should be changed to EnvBuilder. + builder = DistributeEnvBuilder(system_site_packages=system_site_packages, + clear=clear, symlinks=symlinks) + builder.create(env_dir) + +def main(args=None): + compatible = True + if sys.version_info < (3, 3): + compatible = False + elif not hasattr(sys, 'base_prefix'): + compatible = False + if not compatible: + raise ValueError('This script is only for use with ' + 'Python 3.3 (pythonv variant)') + else: + import argparse + + parser = argparse.ArgumentParser(prog=__name__, + description='Creates virtual Python ' + 'environments in one or ' + 'more target ' + 'directories.') + parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', + help='A directory to create the environment in.') + # XXX This option will be removed. + parser.add_argument('--no-distribute', default=False, + action='store_true', dest='nodist', + help="Don't install Distribute in the virtual " + "environment.") + parser.add_argument('--system-site-packages', default=False, + action='store_true', dest='system_site', + help="Give the virtual environment access to the " + "system site-packages dir. ") + if os.name == 'nt' or (sys.platform == 'darwin' and + 'Library/Framework' in sys.base_prefix): + use_symlinks = False + else: + use_symlinks = True + parser.add_argument('--symlinks', default=use_symlinks, + action='store_true', dest='symlinks', + help="Attempt to symlink rather than copy.") + parser.add_argument('--clear', default=False, action='store_true', + dest='clear', help='Delete the environment ' + 'directory if it already ' + 'exists. If not specified and ' + 'the directory exists, an error' + ' is raised.') + parser.add_argument('--upgrade', default=False, action='store_true', + dest='upgrade', help='Upgrade the environment ' + 'directory to use this version ' + 'of Python, assuming it has been ' + 'upgraded in-place.') + options = parser.parse_args(args) + if options.upgrade and options.clear: + raise ValueError('you cannot supply --upgrade and --clear together.') + # XXX This will be changed to EnvBuilder + builder = DistributeEnvBuilder(system_site_packages=options.system_site, + clear=options.clear, + symlinks=options.symlinks, + upgrade=options.upgrade, + nodist=options.nodist) + for d in options.dirs: + builder.create(d) + +if __name__ == '__main__': + rc = 1 + try: + main() + rc = 0 + except Exception as e: + print('Error: %s' % e, file=sys.stderr) + sys.exit(rc) diff --git a/Lib/venv/__main__.py b/Lib/venv/__main__.py new file mode 100644 --- /dev/null +++ b/Lib/venv/__main__.py @@ -0,0 +1,10 @@ +import sys +from . import main + +rc = 1 +try: + main() + rc = 0 +except Exception as e: + print('Error: %s' % e, file=sys.stderr) +sys.exit(rc) diff --git a/Lib/venv/scripts/nt/Activate.ps1 b/Lib/venv/scripts/nt/Activate.ps1 new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/nt/Activate.ps1 @@ -0,0 +1,34 @@ +$env:VIRTUAL_ENV="__VENV_DIR__" + +# Revert to original values +if (Test-Path function:_OLD_VIRTUAL_PROMPT) { + copy-item function:_OLD_VIRTUAL_PROMPT function:prompt + remove-item function:_OLD_VIRTUAL_PROMPT +} + +if (Test-Path env:_OLD_VIRTUAL_PYTHONHOME) { + copy-item env:_OLD_VIRTUAL_PYTHONHOME env:PYTHONHOME + remove-item env:_OLD_VIRTUAL_PYTHONHOME +} + +if (Test-Path env:_OLD_VIRTUAL_PATH) { + copy-item env:_OLD_VIRTUAL_PATH env:PATH + remove-item env:_OLD_VIRTUAL_PATH +} + +# Set the prompt to include the env name +copy-item function:prompt function:_OLD_VIRTUAL_PROMPT +function prompt { + Write-Host -NoNewline -ForegroundColor Green [__VENV_NAME__] + _OLD_VIRTUAL_PROMPT +} + +# Clear PYTHONHOME +if (Test-Path env:PYTHONHOME) { + copy-item env:PYTHONHOME env:_OLD_VIRTUAL_PYTHONHOME + remove-item env:PYTHONHOME +} + +# Add the venv to the PATH +copy-item env:PATH env:_OLD_VIRTUAL_PATH +$env:PATH = "$env:VIRTUAL_ENV\__VENV_BIN_NAME__;$env:PATH" diff --git a/Lib/venv/scripts/nt/Deactivate.ps1 b/Lib/venv/scripts/nt/Deactivate.ps1 new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/nt/Deactivate.ps1 @@ -0,0 +1,19 @@ +# Revert to original values +if (Test-Path function:_OLD_VIRTUAL_PROMPT) { + copy-item function:_OLD_VIRTUAL_PROMPT function:prompt + remove-item function:_OLD_VIRTUAL_PROMPT +} + +if (Test-Path env:_OLD_VIRTUAL_PYTHONHOME) { + copy-item env:_OLD_VIRTUAL_PYTHONHOME env:PYTHONHOME + remove-item env:_OLD_VIRTUAL_PYTHONHOME +} + +if (Test-Path env:_OLD_VIRTUAL_PATH) { + copy-item env:_OLD_VIRTUAL_PATH env:PATH + remove-item env:_OLD_VIRTUAL_PATH +} + +if (Test-Path env:VIRTUAL_ENV) { + remove-item env:VIRTUAL_ENV +} diff --git a/Lib/venv/scripts/nt/activate.bat b/Lib/venv/scripts/nt/activate.bat new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/nt/activate.bat @@ -0,0 +1,31 @@ + at echo off +set VIRTUAL_ENV=__VENV_DIR__ + +if not defined PROMPT ( + set PROMPT=$P$G +) + +if defined _OLD_VIRTUAL_PROMPT ( + set PROMPT=%_OLD_VIRTUAL_PROMPT% +) + +if defined _OLD_VIRTUAL_PYTHONHOME ( + set PYTHONHOME=%_OLD_VIRTUAL_PYTHONHOME% +) + +set _OLD_VIRTUAL_PROMPT=%PROMPT% +set PROMPT=__VENV_NAME__%PROMPT% + +if defined PYTHONHOME ( + set _OLD_VIRTUAL_PYTHONHOME=%PYTHONHOME% + set PYTHONHOME= +) + +if defined _OLD_VIRTUAL_PATH set PATH=%_OLD_VIRTUAL_PATH%; goto SKIPPATH + +set _OLD_VIRTUAL_PATH=%PATH% + +:SKIPPATH +set PATH=%VIRTUAL_ENV%\__VENV_BIN_NAME__;%PATH% + +:END diff --git a/Lib/venv/scripts/nt/deactivate.bat b/Lib/venv/scripts/nt/deactivate.bat new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/nt/deactivate.bat @@ -0,0 +1,17 @@ + at echo off + +if defined _OLD_VIRTUAL_PROMPT ( + set PROMPT=%_OLD_VIRTUAL_PROMPT% +) +set _OLD_VIRTUAL_PROMPT= + +if defined _OLD_VIRTUAL_PYTHONHOME ( + set PYTHONHOME=%_OLD_VIRTUAL_PYTHONHOME% + set _OLD_VIRTUAL_PYTHONHOME= +) + +if defined _OLD_VIRTUAL_PATH set PATH=%_OLD_VIRTUAL_PATH% + +set _OLD_VIRTUAL_PATH= + +:END diff --git a/Lib/venv/scripts/nt/pysetup3-script.py b/Lib/venv/scripts/nt/pysetup3-script.py new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/nt/pysetup3-script.py @@ -0,0 +1,11 @@ +#!__VENV_PYTHON__ +if __name__ == '__main__': + rc = 1 + try: + import sys, re, packaging.run + sys.argv[0] = re.sub('-script.pyw?$', '', sys.argv[0]) + rc = packaging.run.main() # None interpreted as 0 + except Exception: + # use syntax which works with either 2.x or 3.x + sys.stderr.write('%s\n' % sys.exc_info()[1]) + sys.exit(rc) diff --git a/Lib/venv/scripts/nt/pysetup3.exe b/Lib/venv/scripts/nt/pysetup3.exe new file mode 100644 index 0000000000000000000000000000000000000000..3f3c09ebc8e55f4ac3379041753cb34daef71892 GIT binary patch [stripped] diff --git a/Lib/venv/scripts/posix/activate b/Lib/venv/scripts/posix/activate new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/posix/activate @@ -0,0 +1,76 @@ +# This file must be used with "source bin/activate" *from bash* +# you cannot run it directly + +deactivate () { + # reset old environment variables + if [ -n "$_OLD_VIRTUAL_PATH" ] ; then + PATH="$_OLD_VIRTUAL_PATH" + export PATH + unset _OLD_VIRTUAL_PATH + fi + if [ -n "$_OLD_VIRTUAL_PYTHONHOME" ] ; then + PYTHONHOME="$_OLD_VIRTUAL_PYTHONHOME" + export PYTHONHOME + unset _OLD_VIRTUAL_PYTHONHOME + fi + + # This should detect bash and zsh, which have a hash command that must + # be called to get it to forget past commands. Without forgetting + # past commands the $PATH changes we made may not be respected + if [ -n "$BASH" -o -n "$ZSH_VERSION" ] ; then + hash -r + fi + + if [ -n "$_OLD_VIRTUAL_PS1" ] ; then + PS1="$_OLD_VIRTUAL_PS1" + export PS1 + unset _OLD_VIRTUAL_PS1 + fi + + unset VIRTUAL_ENV + if [ ! "$1" = "nondestructive" ] ; then + # Self destruct! + unset -f deactivate + fi +} + +# unset irrelavent variables +deactivate nondestructive + +VIRTUAL_ENV="__VENV_DIR__" +export VIRTUAL_ENV + +_OLD_VIRTUAL_PATH="$PATH" +PATH="$VIRTUAL_ENV/__VENV_BIN_NAME__:$PATH" +export PATH + +# unset PYTHONHOME if set +# this will fail if PYTHONHOME is set to the empty string (which is bad anyway) +# could use `if (set -u; : $PYTHONHOME) ;` in bash +if [ -n "$PYTHONHOME" ] ; then + _OLD_VIRTUAL_PYTHONHOME="$PYTHONHOME" + unset PYTHONHOME +fi + +if [ -z "$VIRTUAL_ENV_DISABLE_PROMPT" ] ; then + _OLD_VIRTUAL_PS1="$PS1" + if [ "x__VENV_NAME__" != x ] ; then + PS1="__VENV_NAME__$PS1" + else + if [ "`basename \"$VIRTUAL_ENV\"`" = "__" ] ; then + # special case for Aspen magic directories + # see http://www.zetadev.com/software/aspen/ + PS1="[`basename \`dirname \"$VIRTUAL_ENV\"\``] $PS1" + else + PS1="(`basename \"$VIRTUAL_ENV\"`)$PS1" + fi + fi + export PS1 +fi + +# This should detect bash and zsh, which have a hash command that must +# be called to get it to forget past commands. Without forgetting +# past commands the $PATH changes we made may not be respected +if [ -n "$BASH" -o -n "$ZSH_VERSION" ] ; then + hash -r +fi diff --git a/Lib/venv/scripts/posix/pysetup3 b/Lib/venv/scripts/posix/pysetup3 new file mode 100644 --- /dev/null +++ b/Lib/venv/scripts/posix/pysetup3 @@ -0,0 +1,11 @@ +#!__VENV_PYTHON__ +if __name__ == '__main__': + rc = 1 + try: + import sys, re, packaging.run + sys.argv[0] = re.sub('-script.pyw?$', '', sys.argv[0]) + rc = packaging.run.main() # None interpreted as 0 + except Exception: + # use syntax which works with either 2.x or 3.x + sys.stderr.write('%s\n' % sys.exc_info()[1]) + sys.exit(rc) diff --git a/Mac/Makefile.in b/Mac/Makefile.in --- a/Mac/Makefile.in +++ b/Mac/Makefile.in @@ -72,7 +72,7 @@ for fn in python3 pythonw3 idle3 pydoc3 python3-config \ python$(VERSION) pythonw$(VERSION) idle$(VERSION) \ pydoc$(VERSION) python$(VERSION)-config 2to3 \ - 2to3-$(VERSION) ;\ + 2to3-$(VERSION) pyvenv pyvenv-$(VERSION) ;\ do \ ln -fs "$(prefix)/bin/$${fn}" "$(DESTDIR)$(FRAMEWORKUNIXTOOLSPREFIX)/bin/$${fn}" ;\ done @@ -93,7 +93,7 @@ $(INSTALL) -d -m $(DIRMODE) "$(DESTDIR)$(FRAMEWORKUNIXTOOLSPREFIX)/bin" ;\ fi for fn in python$(VERSION) pythonw$(VERSION) idle$(VERSION) \ - pydoc$(VERSION) python$(VERSION)-config 2to3-$(VERSION);\ + pydoc$(VERSION) python$(VERSION)-config 2to3-$(VERSION) pyvenv-$(VERSION) ;\ do \ ln -fs "$(prefix)/bin/$${fn}" "$(DESTDIR)$(FRAMEWORKUNIXTOOLSPREFIX)/bin/$${fn}" ;\ done diff --git a/Mac/Tools/pythonw.c b/Mac/Tools/pythonw.c --- a/Mac/Tools/pythonw.c +++ b/Mac/Tools/pythonw.c @@ -150,6 +150,18 @@ int main(int argc, char **argv) { char* exec_path = get_python_path(); + static char path[PATH_MAX * 2]; + static char real_path[PATH_MAX * 2]; + int status; + uint32_t size = PATH_MAX * 2; + + /* Set the original executable path in the environment. */ + status = _NSGetExecutablePath(path, &size); + if (status == 0) { + if (realpath(path, real_path) != NULL) { + setenv("__PYTHONV_LAUNCHER__", real_path, 1); + } + } /* * Let argv[0] refer to the new interpreter. This is needed to diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -947,6 +947,8 @@ (cd $(DESTDIR)$(BINDIR); $(LN) -s 2to3-$(VERSION) 2to3) -rm -f $(DESTDIR)$(BINDIR)/pysetup3 (cd $(DESTDIR)$(BINDIR); $(LN) -s pysetup$(VERSION) pysetup3) + -rm -f $(DESTDIR)$(BINDIR)/pyvenv + (cd $(DESTDIR)$(BINDIR); $(LN) -s pyvenv-$(VERSION) pyvenv) # Install the manual page maninstall: @@ -1038,6 +1040,7 @@ turtledemo \ multiprocessing multiprocessing/dummy \ unittest unittest/test unittest/test/testmock \ + venv venv/scripts venv/scripts/posix \ curses pydoc_data $(MACHDEPS) libinstall: build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c @for i in $(SCRIPTDIR) $(LIBDEST); \ diff --git a/Modules/getpath.c b/Modules/getpath.c --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -260,6 +260,59 @@ wcscpy(path, buffer); } +/* search for a prefix value in an environment file. If found, copy it + to the provided buffer, which is expected to be no more than MAXPATHLEN + bytes long. +*/ + +static int +find_env_config_value(FILE * env_file, const wchar_t * key, wchar_t * value) +{ + int result = 0; /* meaning not found */ + char buffer[MAXPATHLEN*2+1]; /* allow extra for key, '=', etc. */ + + fseek(env_file, 0, SEEK_SET); + while (!feof(env_file)) { + char * p = fgets(buffer, MAXPATHLEN*2, env_file); + wchar_t tmpbuffer[MAXPATHLEN*2+1]; + PyObject * decoded; + int n; + + if (p == NULL) + break; + n = strlen(p); + if (p[n - 1] != '\n') { + /* line has overflowed - bail */ + break; + } + if (p[0] == '#') /* Comment - skip */ + continue; + decoded = PyUnicode_DecodeUTF8(buffer, n, "surrogateescape"); + if (decoded != NULL) { + Py_ssize_t k; + wchar_t * state; + k = PyUnicode_AsWideChar(decoded, + tmpbuffer, MAXPATHLEN * 2); + Py_DECREF(decoded); + if (k >= 0) { + wchar_t * tok = wcstok(tmpbuffer, L" \t\r\n", &state); + if ((tok != NULL) && !wcscmp(tok, key)) { + tok = wcstok(NULL, L" \t", &state); + if ((tok != NULL) && !wcscmp(tok, L"=")) { + tok = wcstok(NULL, L"\r\n", &state); + if (tok != NULL) { + wcsncpy(value, tok, MAXPATHLEN); + result = 1; + break; + } + } + } + } + } + } + return result; +} + /* search_for_prefix requires that argv0_path be no more than MAXPATHLEN bytes long. */ @@ -565,6 +618,39 @@ MAXPATHLEN bytes long. */ + /* Search for an environment configuration file, first in the + executable's directory and then in the parent directory. + If found, open it for use when searching for prefixes. + */ + + { + wchar_t tmpbuffer[MAXPATHLEN+1]; + wchar_t *env_cfg = L"pyvenv.cfg"; + FILE * env_file = NULL; + + wcscpy(tmpbuffer, argv0_path); + joinpath(tmpbuffer, env_cfg); + env_file = _Py_wfopen(tmpbuffer, L"r"); + if (env_file == NULL) { + errno = 0; + reduce(tmpbuffer); + reduce(tmpbuffer); + joinpath(tmpbuffer, env_cfg); + env_file = _Py_wfopen(tmpbuffer, L"r"); + if (env_file == NULL) { + errno = 0; + } + } + if (env_file != NULL) { + /* Look for a 'home' variable and set argv0_path to it, if found */ + if (find_env_config_value(env_file, L"home", tmpbuffer)) { + wcscpy(argv0_path, tmpbuffer); + } + fclose(env_file); + env_file = NULL; + } + } + if (!(pfound = search_for_prefix(argv0_path, home, _prefix))) { if (!Py_FrozenFlag) fprintf(stderr, diff --git a/PC/getpathp.c b/PC/getpathp.c --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -423,6 +423,53 @@ progpath[0] = '\0'; } +static int +find_env_config_value(FILE * env_file, const wchar_t * key, wchar_t * value) +{ + int result = 0; /* meaning not found */ + char buffer[MAXPATHLEN*2+1]; /* allow extra for key, '=', etc. */ + + fseek(env_file, 0, SEEK_SET); + while (!feof(env_file)) { + char * p = fgets(buffer, MAXPATHLEN*2, env_file); + wchar_t tmpbuffer[MAXPATHLEN*2+1]; + PyObject * decoded; + int n; + + if (p == NULL) + break; + n = strlen(p); + if (p[n - 1] != '\n') { + /* line has overflowed - bail */ + break; + } + if (p[0] == '#') /* Comment - skip */ + continue; + decoded = PyUnicode_DecodeUTF8(buffer, n, "surrogateescape"); + if (decoded != NULL) { + Py_ssize_t k; + k = PyUnicode_AsWideChar(decoded, + tmpbuffer, MAXPATHLEN * 2); + Py_DECREF(decoded); + if (k >= 0) { + wchar_t * tok = wcstok(tmpbuffer, L" \t\r\n"); + if ((tok != NULL) && !wcscmp(tok, key)) { + tok = wcstok(NULL, L" \t"); + if ((tok != NULL) && !wcscmp(tok, L"=")) { + tok = wcstok(NULL, L"\r\n"); + if (tok != NULL) { + wcsncpy(value, tok, MAXPATHLEN); + result = 1; + break; + } + } + } + } + } + } + return result; +} + static void calculate_path(void) { @@ -457,6 +504,40 @@ /* progpath guaranteed \0 terminated in MAXPATH+1 bytes. */ wcscpy(argv0_path, progpath); reduce(argv0_path); + + /* Search for an environment configuration file, first in the + executable's directory and then in the parent directory. + If found, open it for use when searching for prefixes. + */ + + { + wchar_t tmpbuffer[MAXPATHLEN+1]; + wchar_t *env_cfg = L"pyvenv.cfg"; + FILE * env_file = NULL; + + wcscpy(tmpbuffer, argv0_path); + join(tmpbuffer, env_cfg); + env_file = _Py_wfopen(tmpbuffer, L"r"); + if (env_file == NULL) { + errno = 0; + reduce(tmpbuffer); + reduce(tmpbuffer); + join(tmpbuffer, env_cfg); + env_file = _Py_wfopen(tmpbuffer, L"r"); + if (env_file == NULL) { + errno = 0; + } + } + if (env_file != NULL) { + /* Look for a 'home' variable and set argv0_path to it, if found */ + if (find_env_config_value(env_file, L"home", tmpbuffer)) { + wcscpy(argv0_path, tmpbuffer); + } + fclose(env_file); + env_file = NULL; + } + } + if (pythonhome == NULL || *pythonhome == '\0') { if (search_for_prefix(argv0_path, LANDMARK)) pythonhome = prefix; diff --git a/Python/sysmodule.c b/Python/sysmodule.c --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1528,6 +1528,10 @@ PyUnicode_FromWideChar(Py_GetPrefix(), -1)); SET_SYS_FROM_STRING("exec_prefix", PyUnicode_FromWideChar(Py_GetExecPrefix(), -1)); + SET_SYS_FROM_STRING("base_prefix", + PyUnicode_FromWideChar(Py_GetPrefix(), -1)); + SET_SYS_FROM_STRING("base_exec_prefix", + PyUnicode_FromWideChar(Py_GetExecPrefix(), -1)); SET_SYS_FROM_STRING("maxsize", PyLong_FromSsize_t(PY_SSIZE_T_MAX)); SET_SYS_FROM_STRING("float_info", diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -1122,6 +1122,7 @@ lib.add_file("2to3.py", src="2to3") lib.add_file("pydoc3.py", src="pydoc3") lib.add_file("pysetup3.py", src="pysetup3") + lib.add_file("pyvenv.py", src="pyvenv") if have_tcl: lib.start_component("pydocgui.pyw", tcltk, keyfile="pydocgui.pyw") lib.add_file("pydocgui.pyw") diff --git a/Tools/scripts/pyvenv b/Tools/scripts/pyvenv new file mode 100755 --- /dev/null +++ b/Tools/scripts/pyvenv @@ -0,0 +1,11 @@ +#!/usr/bin/env python3 +if __name__ == '__main__': + import sys + rc = 1 + try: + import venv + venv.main() + rc = 0 + except Exception as e: + print('Error: %s' % e, file=sys.stderr) + sys.exit(rc) diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -431,7 +431,7 @@ for directory in reversed(options.dirs): add_dir_to_list(dir_list, directory) - if os.path.normpath(sys.prefix) != '/usr' \ + if os.path.normpath(sys.base_prefix) != '/usr' \ and not sysconfig.get_config_var('PYTHONFRAMEWORK'): # OSX note: Don't add LIBDIR and INCLUDEDIR to building a framework # (PYTHONFRAMEWORK is set) to avoid # linking problems when @@ -1978,7 +1978,7 @@ newoutfiles = [] newupdated_files = [] for filename in outfiles: - if filename.endswith('2to3'): + if filename.endswith(('2to3', 'pyvenv')): newfilename = filename + fullversion else: newfilename = filename + minoronly @@ -2046,7 +2046,8 @@ # check the PyBuildScripts command above, and change the links # created by the bininstall target in Makefile.pre.in scripts = ["Tools/scripts/pydoc3", "Tools/scripts/idle3", - "Tools/scripts/2to3", "Tools/scripts/pysetup3"] + "Tools/scripts/2to3", "Tools/scripts/pysetup3", + "Tools/scripts/pyvenv"] ) # --install-platlib -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 04:48:50 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 26 May 2012 04:48:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merged_upstream_changes=2E?= Message-ID: http://hg.python.org/cpython/rev/68c5dac69a71 changeset: 77155:68c5dac69a71 parent: 77154:294f8aeb4d4b parent: 77152:09e97829ed1e user: Vinay Sajip date: Sat May 26 03:48:27 2012 +0100 summary: Merged upstream changes. files: Doc/whatsnew/3.3.rst | 96 ++++++++++++++++++++++++++++++++ Lib/urllib/parse.py | 2 +- Misc/NEWS | 13 ++++ 3 files changed, 110 insertions(+), 1 deletions(-) diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -558,6 +558,9 @@ New Email Package Features ========================== +Policy Framework +---------------- + The email package now has a :mod:`~email.policy` framework. A :class:`~email.policy.Policy` is an object with several methods and properties that control how the email package behaves. The primary policy for Python 3.3 @@ -610,6 +613,99 @@ the ``generator``. +Provisional Policy with New Header API +-------------------------------------- + +While the policy framework is worthwhile all by itself, the main motivation for +introducing it is to allow the creation of new policies that implement new +features for the email package in a way that maintains backward compatibility +for those who do not use the new policies. Because the new policies introduce a +new API, we are releasing them in Python 3.3 as a :term:`provisional policy +`. Backwards incompatible changes (up to and including +removal of the code) may occur if deemed necessary by the core developers. + +The new policies are instances of :class:`~email.policy.EmailPolicy`, +and add the following additional controls: + + =============== ======================================================= + refold_source Controls whether or not headers parsed by a + :mod:`~email.parser` are refolded by the + :mod:`~email.generator`. It can be ``none``, ``long``, + or ``all``. The default is ``long``, which means that + source headers with a line longer than + ``max_line_length`` get refolded. ``none`` means no + line get refolded, and ``all`` means that all lines + get refolded. + + header_factory A callable that take a ``name`` and ``value`` and + produces a custom header object. + =============== ======================================================= + +The ``header_factory`` is the key to the new features provided by the new +policies. When one of the new policies is used, any header retrieved from +a ``Message`` object is an object produced by the ``header_factory``, and any +time you set a header on a ``Message`` it becomes an object produced by +``header_factory``. All such header objects have a ``name`` attribute equal +to the header name. Address and Date headers have additional attributes +that give you access to the parsed data of the header. This means you can now +do things like this:: + + >>> m = Message(policy=SMTP) + >>> m['To'] = '?ric ' + >>> m['to'] + '?ric ' + >>> m['to'].addresses + (Address(display_name='?ric', username='foo', domain='example.com'),) + >>> m['to'].addresses[0].username + 'foo' + >>> m['to'].addresses[0].display_name + '?ric' + >>> m['Date'] = email.utils.localtime() + >>> m['Date'].datetime + datetime.datetime(2012, 5, 25, 21, 39, 24, 465484, tzinfo=datetime.timezone(datetime.timedelta(-1, 72000), 'EDT')) + >>> m['Date'] + 'Fri, 25 May 2012 21:44:27 -0400' + >>> print(m) + To: =?utf-8?q?=C3=89ric?= + Date: Fri, 25 May 2012 21:44:27 -0400 + +You will note that the unicode display name is automatically encoded as +``utf-8`` when the message is serialized, but that when the header is accessed +directly, you get the unicode version. This eliminates any need to deal with +the :mod:`email.header` :meth:`~email.header.decode_header` or +:meth:`~email.header.make_header` functions. + +You can also create addresses from parts:: + + >>> m['cc'] = [Group('pals', [Address('Bob', 'bob', 'example.com'), + ... Address('Sally', 'sally', 'example.com')]), + ... Address('Bonzo', addr_spec='bonz at laugh.com')] + >>> print(m) + To: =?utf-8?q?=C3=89ric?= + Date: Fri, 25 May 2012 21:44:27 -0400 + cc: pals: Bob , Sally ;, Bonzo + +Decoding to unicode is done automatically:: + + >>> m2 = message_from_string(str(m)) + >>> m2['to'] + '?ric ' + +When you parse a message, you can use the ``addresses`` and ``groups`` +attributes of the header objects to access the groups and individual +addresses:: + + >>> m2['cc'].addresses + (Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com'), Address(display_name='Bonzo', username='bonz', domain='laugh.com')) + >>> m2['cc'].groups + (Group(display_name='pals', addresses=(Address(display_name='Bob', username='bob', domain='example.com'), Address(display_name='Sally', username='sally', domain='example.com')), Group(display_name=None, addresses=(Address(display_name='Bonzo', username='bonz', domain='laugh.com'),)) + +In summary, if you use one of the new policies, header manipulation works the +way it ought to: your application works with unicode strings, and the email +package transparently encodes and decodes the unicode to and from the RFC +standard Content Transfer Encodings. + + Other Language Changes ====================== diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -707,7 +707,7 @@ def quote_from_bytes(bs, safe='/'): """Like quote(), but accepts a bytes object rather than a str, and does not perform string-to-bytes encoding. It always returns an ASCII string. - quote_from_bytes(b'abc def\xab') -> 'abc%20def%AB' + quote_from_bytes(b'abc def\x3f') -> 'abc%20def%3f' """ if not isinstance(bs, (bytes, bytearray)): raise TypeError("quote_from_bytes() expected bytes") diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #14712 (PEP 405): Virtual environments. Implemented by Vinay Sajip. + - Issue #14660 (PEP 420): Namespace packages. Implemented by Eric Smith. - Issue #14494: Fix __future__.py and its documentation to note that @@ -44,6 +46,17 @@ Library ------- +- Issue #12586: Added new provisional policies that implement convenient + unicode support for email headers. See What's New for details. + +- Issue #14731: Refactored email Policy framework to support full backward + compatibility with Python 3.2 by default yet allow for the introduction of + new features through new policies. Note that Policy.must_be_7bit is renamed + to cte_type. + +- Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. + Have ascii characters in help. + - Issue #14548: Make multiprocessing finalizers check pid before running to cope with possibility of gc running just after fork. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 04:53:20 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 04:53:20 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2312586=3A_Fix_a_small_ove?= =?utf8?q?rsight_in_the_new_email_policy_header_setting_code=2E?= Message-ID: http://hg.python.org/cpython/rev/2b6f183091b9 changeset: 77156:2b6f183091b9 user: R David Murray date: Fri May 25 22:53:12 2012 -0400 summary: #12586: Fix a small oversight in the new email policy header setting code. This is a danger of focusing on unit tests: sometimes you forget to do the integration tests. files: Lib/email/policy.py | 2 +- Lib/test/test_email/test__headerregistry.py | 22 ++++++++++ 2 files changed, 23 insertions(+), 1 deletions(-) diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -104,7 +104,7 @@ """ if hasattr(value, 'name') and value.name.lower() == name.lower(): return (name, value) - if len(value.splitlines())>1: + if isinstance(value, str) and len(value.splitlines())>1: raise ValueError("Header values may not contain linefeed " "or carriage return characters") return (name, self.header_factory(name, value)) diff --git a/Lib/test/test_email/test__headerregistry.py b/Lib/test/test_email/test__headerregistry.py --- a/Lib/test/test_email/test__headerregistry.py +++ b/Lib/test/test_email/test__headerregistry.py @@ -3,6 +3,7 @@ import unittest from email import errors from email import policy +from email.message import Message from test.test_email import TestEmailBase from email import _headerregistry # Address and Group are public but I'm not sure where to put them yet. @@ -168,6 +169,12 @@ with self.assertRaises(AttributeError): h.datetime = 'foo' + def test_set_date_header_from_datetime(self): + m = Message(policy=policy.default) + m['Date'] = self.dt + self.assertEqual(m['Date'], self.datestring) + self.assertEqual(m['Date'].datetime, self.dt) + class TestAddressHeader(TestHeaderBase): @@ -625,6 +632,20 @@ self.assertEqual(g.addresses, tuple()) self.assertEqual(str(g), 'foo bar:;') + def test_set_message_header_from_address(self): + a = Address('foo', 'bar', 'example.com') + m = Message(policy=policy.default) + m['To'] = a + self.assertEqual(m['to'], 'foo ') + self.assertEqual(m['to'].addresses, (a,)) + + def test_set_message_header_from_group(self): + g = Group('foo bar') + m = Message(policy=policy.default) + m['To'] = g + self.assertEqual(m['to'], 'foo bar:;') + self.assertEqual(m['to'].addresses, g.addresses) + class TestFolding(TestHeaderBase): @@ -713,5 +734,6 @@ 'Date: Sat, 02 Feb 2002 17:00:06 -0800\n') + if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 05:23:32 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 05:23:32 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=23665194=3A_Add_a_localtim?= =?utf8?q?e_function_to_email=2Eutils=2E?= Message-ID: http://hg.python.org/cpython/rev/df12ce0c96eb changeset: 77157:df12ce0c96eb user: R David Murray date: Fri May 25 23:22:59 2012 -0400 summary: #665194: Add a localtime function to email.utils. Without this function people would be tempted to use the other date functions in email.utils to compute an aware localtime, and those functions are not as good for that purpose as this code. The code is Alexander Belopolsy's from his proposed patch for issue 9527, with a fix (and additional tests) by Brian K. Jones. files: Doc/library/email.util.rst | 18 ++++- Lib/email/utils.py | 53 +++++++++++++ Lib/test/test_email/test_utils.py | 73 +++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 3 + 5 files changed, 146 insertions(+), 2 deletions(-) diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -93,8 +93,6 @@ corresponding a :class:`~datetime.timezone` :class:`~datetime.tzinfo`. .. versionadded:: 3.3 - - .. function:: mktime_tz(tuple) Turn a 10-tuple as returned by :func:`parsedate_tz` into a UTC timestamp. It @@ -140,6 +138,22 @@ .. versionadded:: 3.3 +.. function:: localtime(dt=None) + + Return local time as an aware datetime object. If called without + arguments, return current time. Otherwise *dt* argument should be a + :class:`~datetime.datetime` instance, and it is converted to the local time + zone according to the system time zone database. If *dt* is naive (that + is, ``dt.tzinfo`` is ``None``), it is assumed to be in local time. In this + case, a positive or zero value for *isdst* causes ``localtime`` to presume + initially that summer time (for example, Daylight Saving Time) is or is not + (respectively) in effect for the specified time. A negative value for + *isdst* causes the ``localtime`` to attempt to divine whether summer time + is in effect for the specified time. + + .. versionadded:: 3.3 + + .. function:: make_msgid(idstring=None, domain=None) Returns a string suitable for an :rfc:`2822`\ -compliant diff --git a/Lib/email/utils.py b/Lib/email/utils.py --- a/Lib/email/utils.py +++ b/Lib/email/utils.py @@ -363,3 +363,56 @@ except LookupError: # charset is not a known codec. return unquote(text) + + +# +# datetime doesn't provide a localtime function yet, so provide one. Code +# adapted from the patch in issue 9527. This may not be perfect, but it is +# better than not having it. +# + +def localtime(dt=None, isdst=-1): + """Return local time as an aware datetime object. + + If called without arguments, return current time. Otherwise *dt* + argument should be a datetime instance, and it is converted to the + local time zone according to the system time zone database. If *dt* is + naive (that is, dt.tzinfo is None), it is assumed to be in local time. + In this case, a positive or zero value for *isdst* causes localtime to + presume initially that summer time (for example, Daylight Saving Time) + is or is not (respectively) in effect for the specified time. A + negative value for *isdst* causes the localtime() function to attempt + to divine whether summer time is in effect for the specified time. + + """ + if dt is None: + seconds = time.time() + else: + if dt.tzinfo is None: + # A naive datetime is given. Convert to a (localtime) + # timetuple and pass to system mktime together with + # the isdst hint. System mktime will return seconds + # sysce epoch. + tm = dt.timetuple()[:-1] + (isdst,) + seconds = time.mktime(tm) + else: + # An aware datetime is given. Use aware datetime + # arithmetics to find seconds since epoch. + delta = dt - datetime.datetime(1970, 1, 1, + tzinfo=datetime.timezone.utc) + seconds = delta.total_seconds() + tm = time.localtime(seconds) + + # XXX: The following logic may not work correctly if UTC + # offset has changed since time provided in dt. This will be + # corrected in C implementation for platforms that support + # tm_gmtoff. + if time.daylight and tm.tm_isdst: + offset = time.altzone + tzname = time.tzname[1] + else: + offset = time.timezone + tzname = time.tzname[0] + + tz = datetime.timezone(datetime.timedelta(seconds=-offset), tzname) + return datetime.datetime.fromtimestamp(seconds, tz) diff --git a/Lib/test/test_email/test_utils.py b/Lib/test/test_email/test_utils.py --- a/Lib/test/test_email/test_utils.py +++ b/Lib/test/test_email/test_utils.py @@ -1,5 +1,7 @@ import datetime from email import utils +import test.support +import time import unittest class DateTimeTests(unittest.TestCase): @@ -43,3 +45,74 @@ self.assertEqual( utils.parsedate_to_datetime(self.datestring + ' -0000'), self.naive_dt) + + +class LocaltimeTests(unittest.TestCase): + + def test_localtime_is_tz_aware_daylight_true(self): + test.support.patch(self, time, 'daylight', True) + t = utils.localtime() + self.assertIsNot(t.tzinfo, None) + + def test_localtime_is_tz_aware_daylight_false(self): + test.support.patch(self, time, 'daylight', False) + t = utils.localtime() + self.assertIsNot(t.tzinfo, None) + + def test_localtime_daylight_true_dst_false(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0, isdst=-1) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + def test_localtime_daylight_false_dst_false(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0, isdst=-1) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + def test_localtime_daylight_true_dst_true(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0, isdst=1) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + def test_localtime_daylight_false_dst_true(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(2012, 3, 12, 1, 1) + t1 = utils.localtime(t0, isdst=1) + t2 = utils.localtime(t1) + self.assertEqual(t1, t2) + + def test_localtime_epoch_utc_daylight_true(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(1970, 1, 1, tzinfo = datetime.timezone.utc) + t1 = utils.localtime(t0) + self.assertEqual(t0, t1) + + def test_localtime_epoch_utc_daylight_false(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(1970, 1, 1, tzinfo = datetime.timezone.utc) + t1 = utils.localtime(t0) + self.assertEqual(t0, t1) + + def test_localtime_epoch_notz_daylight_true(self): + test.support.patch(self, time, 'daylight', True) + t0 = datetime.datetime(1970, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t0.replace(tzinfo=None)) + self.assertEqual(t1, t2) + + def test_localtime_epoch_notz_daylight_false(self): + test.support.patch(self, time, 'daylight', False) + t0 = datetime.datetime(1970, 1, 1) + t1 = utils.localtime(t0) + t2 = utils.localtime(t0.replace(tzinfo=None)) + self.assertEqual(t1, t2) + + +if __name__ == '__main__': + unittest.main() diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -506,6 +506,7 @@ Matt Joiner Thomas Jollans Nicolas Joly +Brian K. Jones Evan Jones Jeremy Jones Richard Jones diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -46,6 +46,9 @@ Library ------- +- Issue #665194: Added a localtime function to email.utils to provide an + aware local datetime for use in setting Date headers. + - Issue #12586: Added new provisional policies that implement convenient unicode support for email headers. See What's New for details. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sat May 26 05:51:58 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sat, 26 May 2012 05:51:58 +0200 Subject: [Python-checkins] Daily reference leaks (0189b9d2d6bc): sum=0 Message-ID: results for 0189b9d2d6bc on branch "default" -------------------------------------------- Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKSi88r', '-x'] From python-checkins at python.org Sat May 26 09:11:19 2012 From: python-checkins at python.org (georg.brandl) Date: Sat, 26 May 2012 09:11:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_missing_versionadded=2E?= Message-ID: http://hg.python.org/cpython/rev/48b1cdda7133 changeset: 77158:48b1cdda7133 user: Georg Brandl date: Sat May 26 09:11:22 2012 +0200 summary: Add missing versionadded. files: Doc/library/sys.rst | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -39,6 +39,9 @@ :data:`base_exec_prefix` will remain pointing to the base Python installation (the one which the virtual environment was created from). + .. versionadded:: 3.3 + + .. data:: base_prefix Set during Python startup, before ``site.py`` is run, to the same value as @@ -49,6 +52,9 @@ :data:`base_exec_prefix` will remain pointing to the base Python installation (the one which the virtual environment was created from). + .. versionadded:: 3.3 + + .. data:: byteorder An indicator of the native byte order. This will have the value ``'big'`` on -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 09:14:53 2012 From: python-checkins at python.org (georg.brandl) Date: Sat, 26 May 2012 09:14:53 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_PEP_421_is_implemented=2E?= Message-ID: http://hg.python.org/peps/rev/cba34504163d changeset: 4441:cba34504163d user: Georg Brandl date: Sat May 26 09:15:01 2012 +0200 summary: PEP 421 is implemented. files: pep-0398.txt | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -69,6 +69,7 @@ * PEP 417: Including mock in the Standard Library * PEP 418: Add monotonic time, performance counter, and process time functions * PEP 420: Implicit Namespace Packages +* PEP 421: Adding sys.implementation * PEP 3118: Revising the buffer protocol (protocol semantics finalised) * PEP 3144: IP Address manipulation library * PEP 3151: Reworking the OS and IO exception hierarchy @@ -87,7 +88,6 @@ * PEP 362: Function Signature Object * PEP 397: Python launcher for Windows * PEP 405: Python Virtual Environments -* PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3154: Pickle protocol version 4 -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Sat May 26 12:13:36 2012 From: python-checkins at python.org (hynek.schlawack) Date: Sat, 26 May 2012 12:13:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314814=3A_Some_PEP8_adjus?= =?utf8?q?tments_and_dead_code_weeding?= Message-ID: http://hg.python.org/cpython/rev/63d2c9affb11 changeset: 77159:63d2c9affb11 user: Hynek Schlawack date: Sat May 26 12:04:56 2012 +0200 summary: #14814: Some PEP8 adjustments and dead code weeding files: Lib/ipaddress.py | 44 +++++++++++++---------------------- 1 files changed, 17 insertions(+), 27 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -284,7 +284,8 @@ If the version is not 4 or 6. """ - if not (isinstance(first, _BaseAddress) and isinstance(last, _BaseAddress)): + if (not (isinstance(first, _BaseAddress) and + isinstance(last, _BaseAddress))): raise TypeError('first and last must be IP addresses, not networks') if first.version != last.version: raise TypeError("%s and %s are not of the same version" % ( @@ -292,8 +293,6 @@ if first > last: raise ValueError('last IP address must be greater than first') - networks = [] - if first.version == 4: ip = IPv4Network elif first.version == 6: @@ -316,7 +315,6 @@ prefix = _get_prefix_length(first_int, current, ip_bits) net = ip('%s/%d' % (str(first), prefix)) yield net - #networks.append(net) if current == ip._ALL_ONES: break first_int = current + 1 @@ -599,7 +597,7 @@ return '%s(%r)' % (self.__class__.__name__, str(self)) def __str__(self): - return '%s' % self._string_from_ip_int(self._ip) + return str(self._string_from_ip_int(self._ip)) def __hash__(self): return hash(hex(int(self._ip))) @@ -719,8 +717,7 @@ return not eq def __str__(self): - return '%s/%s' % (str(self.ip), - str(self._prefixlen)) + return '%s/%s' % (self.ip, self._prefixlen) def __hash__(self): return hash(int(self.network_address) ^ int(self.netmask)) @@ -838,13 +835,10 @@ if not (other.network_address >= self.network_address and other.broadcast_address <= self.broadcast_address): - raise ValueError('%s not contained in %s' % (str(other), str(self))) - + raise ValueError('%s not contained in %s' % (other, self)) if other == self: raise StopIteration - ret_addrs = [] - # Make sure we're comparing the network of other. other = ip_network('%s/%s' % (str(other.network_address), str(other.prefixlen)), @@ -1013,8 +1007,8 @@ An IPv4 network object. Raises: - ValueError: If self.prefixlen - prefixlen_diff < 0. I.e., you have a - negative prefix length. + ValueError: If self.prefixlen - prefixlen_diff < 0. I.e., you have + a negative prefix length. OR If prefixlen_diff and new_prefix are both set or new_prefix is a larger number than the current prefix (larger number means a @@ -1031,7 +1025,6 @@ raise ValueError('cannot set prefixlen_diff and new_prefix') prefixlen_diff = self._prefixlen - new_prefix - if self.prefixlen - prefixlen_diff < 0: raise ValueError( 'current prefixlen is %d, cannot have a prefixlen_diff of %d' % @@ -1302,7 +1295,6 @@ self.netmask = self.network.netmask self.hostmask = self.network.hostmask - def __str__(self): return '%s/%d' % (self._string_from_ip_int(self._ip), self.network.prefixlen) @@ -1364,7 +1356,6 @@ return True return False - @property def prefixlen(self): return self._prefixlen @@ -1381,6 +1372,7 @@ def with_netmask(self): return '%s/%s' % (self._string_from_ip_int(self._ip), self.netmask) + @property def with_hostmask(self): return '%s/%s' % (self._string_from_ip_int(self._ip), @@ -1652,8 +1644,9 @@ if parts_skipped < 1: raise AddressValueError(ip_str) else: - # Otherwise, allocate the entire address to parts_hi. The endpoints - # could still be empty, but _parse_hextet() will check for that. + # Otherwise, allocate the entire address to parts_hi. The + # endpoints could still be empty, but _parse_hextet() will check + # for that. if len(parts) != self._HEXTET_COUNT: raise AddressValueError(ip_str) parts_hi = len(parts) @@ -1684,7 +1677,8 @@ The hextet as an integer. Raises: - ValueError: if the input isn't strictly a hex number from [0..FFFF]. + ValueError: if the input isn't strictly a hex number from + [0..FFFF]. """ # Whitelist the characters, since int() allows a lot of bizarre stuff. @@ -1893,7 +1887,6 @@ return (self.network_address in private_network and self.broadcast_address in private_network) - @property def ipv4_mapped(self): """Return the IPv4 mapped address. @@ -2029,7 +2022,6 @@ self._prefixlen = self.network._prefixlen self.hostmask = self.network.hostmask - def __str__(self): return '%s/%d' % (self._string_from_ip_int(self._ip), self.network.prefixlen) @@ -2047,6 +2039,7 @@ @property def prefixlen(self): return self._prefixlen + @property def ip(self): return IPv6Address(self._ip) @@ -2058,6 +2051,7 @@ @property def with_netmask(self): return self.with_prefixlen + @property def with_hostmask(self): return '%s/%s' % (self._string_from_ip_int(self._ip), @@ -2081,8 +2075,8 @@ """Instantiate a new IPv6 Network object. Args: - address: A string or integer representing the IPv6 network or the IP - and prefix/netmask. + address: A string or integer representing the IPv6 network or the + IP and prefix/netmask. '2001:db8::/128' '2001:db8:0000:0000:0000:0000:0000:0000/128' '2001:db8::' @@ -2191,10 +2185,6 @@ return 0 <= prefixlen <= self._max_prefixlen @property - def with_netmask(self): - return self.with_prefixlen - - @property def with_prefixlen(self): return '%s/%d' % (str(self.network_address), self._prefixlen) -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Sat May 26 14:40:37 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 26 May 2012 22:40:37 +1000 Subject: [Python-checkins] peps: PEP 421 is implemented. In-Reply-To: References: Message-ID: On Sat, May 26, 2012 at 5:14 PM, georg.brandl wrote: > http://hg.python.org/peps/rev/cba34504163d > changeset: ? 4441:cba34504163d > user: ? ? ? ?Georg Brandl > date: ? ? ? ?Sat May 26 09:15:01 2012 +0200 > summary: > ?PEP 421 is implemented. Did you mean to move 405 instead? 421 is accepted, but not implemented yet. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python-checkins at python.org Sat May 26 14:55:06 2012 From: python-checkins at python.org (georg.brandl) Date: Sat, 26 May 2012 14:55:06 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_405_is_implemented=2C_not_421?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/peps/rev/8ed2edc948c1 changeset: 4442:8ed2edc948c1 user: Georg Brandl date: Sat May 26 14:55:12 2012 +0200 summary: 405 is implemented, not 421. files: pep-0398.txt | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0398.txt b/pep-0398.txt --- a/pep-0398.txt +++ b/pep-0398.txt @@ -62,6 +62,7 @@ * PEP 380: Syntax for Delegating to a Subgenerator * PEP 393: Flexible String Representation * PEP 399: Pure Python/C Accelerator Module Compatibility Requirements +* PEP 405: Python Virtual Environments * PEP 409: Suppressing exception context * PEP 412: Key-Sharing Dictionary * PEP 414: Explicit Unicode Literal for Python 3.3 @@ -69,7 +70,6 @@ * PEP 417: Including mock in the Standard Library * PEP 418: Add monotonic time, performance counter, and process time functions * PEP 420: Implicit Namespace Packages -* PEP 421: Adding sys.implementation * PEP 3118: Revising the buffer protocol (protocol semantics finalised) * PEP 3144: IP Address manipulation library * PEP 3151: Reworking the OS and IO exception hierarchy @@ -87,7 +87,7 @@ * PEP 362: Function Signature Object * PEP 397: Python launcher for Windows -* PEP 405: Python Virtual Environments +* PEP 421: Adding sys.implementation * PEP 3143: Standard daemon process library * PEP 3154: Pickle protocol version 4 -- Repository URL: http://hg.python.org/peps From g.brandl at gmx.net Sat May 26 14:55:19 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 26 May 2012 14:55:19 +0200 Subject: [Python-checkins] peps: PEP 421 is implemented. In-Reply-To: References: Message-ID: Am 26.05.2012 14:40, schrieb Nick Coghlan: > On Sat, May 26, 2012 at 5:14 PM, georg.brandl > wrote: >> http://hg.python.org/peps/rev/cba34504163d >> changeset: 4441:cba34504163d >> user: Georg Brandl >> date: Sat May 26 09:15:01 2012 +0200 >> summary: >> PEP 421 is implemented. > > Did you mean to move 405 instead? 421 is accepted, but not implemented yet. Ah, yes. Thanks! Georg From python-checkins at python.org Sat May 26 16:26:21 2012 From: python-checkins at python.org (nick.coghlan) Date: Sat, 26 May 2012 16:26:21 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_In_the_sp?= =?utf8?q?irit_of_TOOWTDI=2C_ditch_the_redundant_version_parameter?= Message-ID: http://hg.python.org/cpython/rev/f70e12499d05 changeset: 77160:f70e12499d05 user: Nick Coghlan date: Sun May 27 00:25:58 2012 +1000 summary: Issue #14814: In the spirit of TOOWTDI, ditch the redundant version parameter to the factory functions by using the appropriate direct class references instead files: Lib/ipaddress.py | 122 +++++++++--------------- Lib/test/test_ipaddress.py | 20 +--- 2 files changed, 48 insertions(+), 94 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -36,34 +36,22 @@ """A Value Error related to the netmask.""" -def ip_address(address, version=None): +def ip_address(address): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP address. Either IPv4 or IPv6 addresses may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. Important for things - like ip_address(1), which could be IPv4, '192.0.2.1', or IPv6, - '2001:db8::1'. Returns: An IPv4Address or IPv6Address object. Raises: ValueError: if the *address* passed isn't either a v4 or a v6 - address, or if the version is not None, 4, or 6. + address """ - if version is not None: - if version == 4: - return IPv4Address(address) - elif version == 6: - return IPv6Address(address) - else: - raise ValueError() - try: return IPv4Address(address) except (AddressValueError, NetmaskValueError): @@ -78,35 +66,22 @@ address) -def ip_network(address, version=None, strict=True): +def ip_network(address, strict=True): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP network. Either IPv4 or IPv6 networks may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. Important for things - like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, - '2001:db8::1/128'. Returns: An IPv4Network or IPv6Network object. Raises: ValueError: if the string passed isn't either a v4 or a v6 - address. Or if the network has host bits set. Or if the version - is not None, 4, or 6. + address. Or if the network has host bits set. """ - if version is not None: - if version == 4: - return IPv4Network(address, strict) - elif version == 6: - return IPv6Network(address, strict) - else: - raise ValueError() - try: return IPv4Network(address, strict) except (AddressValueError, NetmaskValueError): @@ -121,24 +96,20 @@ address) -def ip_interface(address, version=None): +def ip_interface(address): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP address. Either IPv4 or IPv6 addresses may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. Important for things - like ip_interface(1), which could be IPv4, '192.0.2.1/32', or IPv6, - '2001:db8::1/128'. Returns: An IPv4Interface or IPv6Interface object. Raises: ValueError: if the string passed isn't either a v4 or a v6 - address. Or if the version is not None, 4, or 6. + address. Notes: The IPv?Interface classes describe an Address on a particular @@ -146,14 +117,6 @@ and Network classes. """ - if version is not None: - if version == 4: - return IPv4Interface(address) - elif version == 6: - return IPv6Interface(address) - else: - raise ValueError() - try: return IPv4Interface(address) except (AddressValueError, NetmaskValueError): @@ -281,7 +244,7 @@ If the first and last objects are not the same version. ValueError: If the last object is not greater than the first. - If the version is not 4 or 6. + If the version of the first address is not 4 or 6. """ if (not (isinstance(first, _BaseAddress) and @@ -318,7 +281,7 @@ if current == ip._ALL_ONES: break first_int = current + 1 - first = ip_address(first_int, version=first._version) + first = first.__class__(first_int) def _collapse_addresses_recursive(addresses): @@ -586,12 +549,12 @@ def __add__(self, other): if not isinstance(other, int): return NotImplemented - return ip_address(int(self) + other, version=self._version) + return self.__class__(int(self) + other) def __sub__(self, other): if not isinstance(other, int): return NotImplemented - return ip_address(int(self) - other, version=self._version) + return self.__class__(int(self) - other) def __repr__(self): return '%s(%r)' % (self.__class__.__name__, str(self)) @@ -612,13 +575,12 @@ class _BaseNetwork(_IPAddressBase): - """A generic IP object. + """A generic IP network object. This IP class contains the version independent methods which are used by networks. """ - def __init__(self, address): self._cache = {} @@ -642,14 +604,14 @@ bcast = int(self.broadcast_address) - 1 while cur <= bcast: cur += 1 - yield ip_address(cur - 1, version=self._version) + yield self._address_class(cur - 1) def __iter__(self): cur = int(self.network_address) bcast = int(self.broadcast_address) while cur <= bcast: cur += 1 - yield ip_address(cur - 1, version=self._version) + yield self._address_class(cur - 1) def __getitem__(self, n): network = int(self.network_address) @@ -657,12 +619,12 @@ if n >= 0: if network + n > broadcast: raise IndexError - return ip_address(network + n, version=self._version) + return self._address_class(network + n) else: n += 1 if broadcast + n < network: raise IndexError - return ip_address(broadcast + n, version=self._version) + return self._address_class(broadcast + n) def __lt__(self, other): if self._version != other._version: @@ -746,8 +708,8 @@ def broadcast_address(self): x = self._cache.get('broadcast_address') if x is None: - x = ip_address(int(self.network_address) | int(self.hostmask), - version=self._version) + x = self._address_class(int(self.network_address) | + int(self.hostmask)) self._cache['broadcast_address'] = x return x @@ -755,15 +717,15 @@ def hostmask(self): x = self._cache.get('hostmask') if x is None: - x = ip_address(int(self.netmask) ^ self._ALL_ONES, - version=self._version) + x = self._address_class(int(self.netmask) ^ self._ALL_ONES) self._cache['hostmask'] = x return x @property def network(self): - return ip_network('%s/%d' % (str(self.network_address), - self.prefixlen)) + # XXX (ncoghlan): This is redundant now and will likely be removed + return self.__class__('%s/%d' % (str(self.network_address), + self.prefixlen)) @property def with_prefixlen(self): @@ -787,6 +749,10 @@ raise NotImplementedError('BaseNet has no version') @property + def _address_class(self): + raise NotImplementedError('BaseNet has no associated address class') + + @property def prefixlen(self): return self._prefixlen @@ -840,9 +806,8 @@ raise StopIteration # Make sure we're comparing the network of other. - other = ip_network('%s/%s' % (str(other.network_address), - str(other.prefixlen)), - version=other._version) + other = other.__class__('%s/%s' % (str(other.network_address), + str(other.prefixlen))) s1, s2 = self.subnets() while s1 != other and s2 != other: @@ -973,9 +938,9 @@ 'prefix length diff %d is invalid for netblock %s' % ( new_prefixlen, str(self))) - first = ip_network('%s/%s' % (str(self.network_address), - str(self._prefixlen + prefixlen_diff)), - version=self._version) + first = self.__class__('%s/%s' % + (str(self.network_address), + str(self._prefixlen + prefixlen_diff))) yield first current = first @@ -983,16 +948,17 @@ broadcast = current.broadcast_address if broadcast == self.broadcast_address: return - new_addr = ip_address(int(broadcast) + 1, version=self._version) - current = ip_network('%s/%s' % (str(new_addr), str(new_prefixlen)), - version=self._version) + new_addr = self._address_class(int(broadcast) + 1) + current = self.__class__('%s/%s' % (str(new_addr), + str(new_prefixlen))) yield current def masked(self): """Return the network object with the host bits masked out.""" - return ip_network('%s/%d' % (self.network_address, self._prefixlen), - version=self._version) + # XXX (ncoghlan): This is redundant now and will likely be removed + return self.__class__('%s/%d' % (self.network_address, + self._prefixlen)) def supernet(self, prefixlen_diff=1, new_prefix=None): """The supernet containing the current network. @@ -1030,11 +996,10 @@ 'current prefixlen is %d, cannot have a prefixlen_diff of %d' % (self.prefixlen, prefixlen_diff)) # TODO (pmoody): optimize this. - t = ip_network('%s/%d' % (str(self.network_address), - self.prefixlen - prefixlen_diff), - version=self._version, strict=False) - return ip_network('%s/%d' % (str(t.network_address), t.prefixlen), - version=t._version) + t = self.__class__('%s/%d' % (str(self.network_address), + self.prefixlen - prefixlen_diff), + strict=False) + return t.__class__('%s/%d' % (str(t.network_address), t.prefixlen)) class _BaseV4(object): @@ -1391,6 +1356,9 @@ .prefixlen: 27 """ + # Class to use when creating address objects + # TODO (ncoghlan): Investigate using IPv4Interface instead + _address_class = IPv4Address # the valid octets for host and netmasks. only useful for IPv4. _valid_mask_octets = set((255, 254, 252, 248, 240, 224, 192, 128, 0)) @@ -2071,6 +2039,10 @@ """ + # Class to use when creating address objects + # TODO (ncoghlan): Investigate using IPv6Interface instead + _address_class = IPv6Address + def __init__(self, address, strict=True): """Instantiate a new IPv6 Network object. diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -780,12 +780,6 @@ self.assertEqual(self.ipv4_address.version, 4) self.assertEqual(self.ipv6_address.version, 6) - with self.assertRaises(ValueError): - ipaddress.ip_address('1', version=[]) - - with self.assertRaises(ValueError): - ipaddress.ip_address('1', version=5) - def testMaxPrefixLength(self): self.assertEqual(self.ipv4_interface.max_prefixlen, 32) self.assertEqual(self.ipv6_interface.max_prefixlen, 128) @@ -1052,12 +1046,7 @@ def testForceVersion(self): self.assertEqual(ipaddress.ip_network(1).version, 4) - self.assertEqual(ipaddress.ip_network(1, version=6).version, 6) - - with self.assertRaises(ValueError): - ipaddress.ip_network(1, version='l') - with self.assertRaises(ValueError): - ipaddress.ip_network(1, version=3) + self.assertEqual(ipaddress.IPv6Network(1).version, 6) def testWithStar(self): self.assertEqual(str(self.ipv4_interface.with_prefixlen), "1.2.3.4/24") @@ -1148,13 +1137,6 @@ sixtofouraddr.sixtofour) self.assertFalse(bad_addr.sixtofour) - def testIpInterfaceVersion(self): - with self.assertRaises(ValueError): - ipaddress.ip_interface(1, version=123) - - with self.assertRaises(ValueError): - ipaddress.ip_interface(1, version='') - if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 16:57:37 2012 From: python-checkins at python.org (nick.coghlan) Date: Sat, 26 May 2012 16:57:37 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_Cleanup_i?= =?utf8?q?paddress_header_comments?= Message-ID: http://hg.python.org/cpython/rev/462fff00c3fb changeset: 77161:462fff00c3fb user: Nick Coghlan date: Sun May 27 00:57:25 2012 +1000 summary: Issue #14814: Cleanup ipaddress header comments files: Lib/ipaddress.py | 12 ------------ Lib/test/test_ipaddress.py | 16 +--------------- 2 files changed, 1 insertions(+), 27 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1,17 +1,5 @@ # Copyright 2007 Google Inc. # Licensed to PSF under a Contributor Agreement. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or -# implied. See the License for the specific language governing -# permissions and limitations under the License. """A fast, lightweight IPv4/IPv6 manipulation library in Python. diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -1,21 +1,7 @@ -#!/usr/bin/python3 -# # Copyright 2007 Google Inc. # Licensed to PSF under a Contributor Agreement. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Unittest for ipaddressmodule.""" +"""Unittest for ipaddress module.""" import unittest -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 17:03:41 2012 From: python-checkins at python.org (nick.coghlan) Date: Sat, 26 May 2012 17:03:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_Clean_out?= =?utf8?q?_an_obsolete_property_and_method_from_ipaddress_Network?= Message-ID: http://hg.python.org/cpython/rev/0b54721bf1cc changeset: 77162:0b54721bf1cc user: Nick Coghlan date: Sun May 27 01:03:25 2012 +1000 summary: Issue #14814: Clean out an obsolete property and method from ipaddress Network objects files: Lib/ipaddress.py | 14 +------------- Lib/test/test_ipaddress.py | 2 +- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -710,12 +710,6 @@ return x @property - def network(self): - # XXX (ncoghlan): This is redundant now and will likely be removed - return self.__class__('%s/%d' % (str(self.network_address), - self.prefixlen)) - - @property def with_prefixlen(self): return '%s/%d' % (str(self.ip), self._prefixlen) @@ -942,12 +936,6 @@ yield current - def masked(self): - """Return the network object with the host bits masked out.""" - # XXX (ncoghlan): This is redundant now and will likely be removed - return self.__class__('%s/%d' % (self.network_address, - self._prefixlen)) - def supernet(self, prefixlen_diff=1, new_prefix=None): """The supernet containing the current network. @@ -1908,7 +1896,7 @@ """ if isinstance(self, IPv6Network): - return int(self.network) == 1 and getattr( + return int(self) == 1 and getattr( self, '_prefixlen', 128) == 128 elif isinstance(self, IPv6Interface): return int(self.network.network_address) == 1 and getattr( diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -390,7 +390,7 @@ self.assertRaises(ValueError, list, self.ipv4_interface.network.subnets(-1)) self.assertRaises(ValueError, list, - self.ipv4_network.network.subnets(-1)) + self.ipv4_network.subnets(-1)) self.assertRaises(ValueError, list, self.ipv6_interface.network.subnets(-1)) self.assertRaises(ValueError, list, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 17:53:45 2012 From: python-checkins at python.org (nick.coghlan) Date: Sat, 26 May 2012 17:53:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314814=3A_Add_a_bas?= =?utf8?q?ic_ipaddress_tutorial_=28thanks_to_Sandro_Tosi_for_the?= Message-ID: http://hg.python.org/cpython/rev/04b939c0c84d changeset: 77163:04b939c0c84d user: Nick Coghlan date: Sun May 27 01:53:33 2012 +1000 summary: Issue #14814: Add a basic ipaddress tutorial (thanks to Sandro Tosi for the initial conversion from Peter Moody's wiki version) files: Doc/howto/index.rst | 1 + Doc/howto/ipaddress.rst | 291 ++++++++++++++++++++++++++++ 2 files changed, 292 insertions(+), 0 deletions(-) diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -28,4 +28,5 @@ urllib2.rst webservers.rst argparse.rst + ipaddress.rst diff --git a/Doc/howto/ipaddress.rst b/Doc/howto/ipaddress.rst new file mode 100644 --- /dev/null +++ b/Doc/howto/ipaddress.rst @@ -0,0 +1,291 @@ +.. _ipaddress-howto: + +*************** +Ipaddress Howto +*************** + +:author: Peter Moody + +.. topic:: Abstract + + This document is a gentle introduction to :mod:`ipaddress` module. + + +Creating Address/Network/Interface objects +========================================== + +Since :mod:`ipaddress` is a module for inspecting and manipulating IP address, +the first thing you'll want to do is create some objects. You can use +:mod:`ipaddress` to create objects from strings and integers. + + +A Note on IP Versions +--------------------- + +For readers that aren't particularly familiar with IP addressing, it's +important to know that the Internet Protocol is currently in the process +of moving from version 4 of the protocol to version 6. This transition is +occurring largely because version 4 of the protocol doesn't provide enough +addresses to handle the needs of the whole world, especially given the +increasing number of devices with direct connections to the internet. + +Explaining the details of the differences between the two versions of the +protocol is beyond the scope of this introduction, but readers need to at +least be aware that these two versions exist, and it will sometimes be +necessary to force the use of one version or the other. + + +IP Host Addresses +----------------- + +Addresses, often referred to as "host addresses" are the most basic unit +when working with IP addressing. The simplest way to create addresses is +to use the ``ip_address`` factory function, which automatically determines +whether to create an IPv4 or IPv6 address based on the passed in value:: + + >>> ipaddress.ip_address('192.0.2.1') + IPv4Address('192.0.2.1') + >>> ipaddress.ip_address('2001:DB8::1') + IPv6Address('2001:db8::1') + +Addresses can also be created directly from integers. Values that will +fit within 32 bits are assumed to be IPv4 addresses:: + + >>> ipaddress.ip_address(3221225985) + IPv4Address('192.0.2.1') + >>> ipaddress.ip_address(42540766411282592856903984951653826561) + IPv6Address('2001:db8::1') + +To force the use of IPv4 or IPv6 addresses, the relevant classes can be +invoked directly. This is particularly useful to force creation of IPv6 +addresses for small integers:: + + >>> ipaddress.ip_address(1) + IPv4Address('0.0.0.1') + >>> ipaddress.IPv4Address(1) + IPv4Address('0.0.0.1') + >>> ipaddress.IPv6Address(1) + IPv6Address('::1') + + +Defining Networks +----------------- + +Host addresses are usually grouped together into IP networks, so +:mod:`ipaddress` provides a way to create, inspect and manipulate network +definitions. IP network objects are constructed from strings that define the +range of host addresses that are part of that network. The simplest form +for that information is a "network address/network prefix" pair, where the +prefix defines the number of leading bits that are compared to determine +whether or not an address is part of the network and the network address +defines the expected value of those bits. + +As for addresses, a factory function is provided that determines the correct +IP version automatically:: + + >>> ipaddress.ip_network('192.0.2.0/24') + IPv4Network('192.0.2.0/24') + >>> ipaddress.ip_network('2001:db8::0/96') + IPv6Network('2001:db8::/96') + +Network objects cannot have any host bits set. The practical effect of this +is that ``192.0.2.1/24`` does not describe a network. Such definitions are +referred to as interface objects since the ip-on-a-network notation is +commonly used to describe network interfaces of a computer on a given network +and are described further in the next section. + +By default, attempting to create a network object with host bits set will +result in :exc:`ValueError` being raised. To request that the +additional bits instead be coerced to zero, the flag ``strict=False`` can +be passed to the constructor:: + + >>> ipaddress.ip_network('192.0.2.1/24') + Traceback (most recent call last): + ... + ValueError: 192.0.2.1/24 has host bits set + >>> ipaddress.ip_network('192.0.2.1/24', strict=False) + IPv4Network('192.0.2.0/24') + +While the string form offers significantly more flexibility, networks can +also be defined with integers, just like host addresses. In this case, the +network is considered to contain only the single address identified by the +integer, so the network prefix includes the entire network address:: + + >>> ipaddress.ip_network(3221225984) + IPv4Network('192.0.2.0/32') + >>> ipaddress.ip_network(42540766411282592856903984951653826560L) + IPv6Network('2001:db8::/128') + +Creation of a particular kind of network can be forced by calling the +class constructor directly instead of using the factory function. + + +Host Interfaces +--------------- + +As mentioned just above, if you need to describe an address on a particular +network, neither the address nor the network classes are sufficient. +Notation like ``192.0.2.1/24`` is commonly used network engineers and the +people who write tools for firewalls and routers as shorthand for "the host +``192.0.2.1`` on the network ``192.0.2.0/24``", Accordingly, :mod:`ipaddress` +provides a set of hybrid classes that associate an address with a particular +network. The interface for creation is identical to that for defining network +objects, except that the address portion isn't constrained to being a network +address. + + >>> ipaddress.ip_interface('192.0.2.1/24') + IPv4Interface('192.0.2.1/24') + >>> ipaddress.ip_network('2001:db8::1/96') + IPv6Interface('2001:db8::1/96') + +Integer inputs are accepted (as with networks), and use of a particular IP +version can be forced by calling the relevant constructor directly. + + +Inspecting Address/Network/Interface Objects +============================================ + +You've gone to the trouble of creating an IPv(4|6)(Address|Network|Interface) +object, so you probably want to get information about it. :mod:`ipaddress` +tries to make doing this easy and intuitive. + +Extracting the IP version:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> addr6 = ipaddress.ip_address('2001:db8::1') + >>> addr6.version + 6 + >>> addr4.version + 4 + +Obtaining the network from an interface:: + + >>> host4 = ipaddress.ip_interface('192.0.2.1/24') + >>> host4.network + IPv4Network('192.0.2.0/24') + >>> host6 = ipaddress.ip_interface('2001:db8::1/96') + >>> host6.network + IPv6Network('2001:db8::/96') + +Finding out how many individual addresses are in a network:: + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> net4.numhosts + 256 + >>> net6 = ipaddress.ip_network('2001:db8::0/96') + >>> net6.numhosts + 4294967296 + +Iterating through the 'usable' addresses on a network:: + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> for x in net4.iterhosts(): + print(x) + 192.0.2.1 + 192.0.2.2 + 192.0.2.3 + 192.0.2.4 + + 192.0.2.252 + 192.0.2.253 + 192.0.2.254 + + +Obtaining the netmask (i.e. set bits corresponding to the network prefix) or +the hostmask (any bits that are not part of the netmask): + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> net4.netmask + IPv4Address('255.255.255.0') + >>> net4.hostmask + IPv4Address('0.0.0.255') + >>> net6 = ipaddress.ip_network('2001:db8::0/96') + >>> net6.netmask + IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::') + >>> net6.hostmask + IPv6Address('::ffff:ffff') + + +Exploding or compressing the address:: + + >>> net6.exploded + '2001:0000:0000:0000:0000:0000:0000:0000/96' + >>> addr6.exploded + '2001:0000:0000:0000:0000:0000:0000:0001' + + +Networks as lists of Addresses +============================== + +It's sometimes useful to treat networks as lists. This means it is possible +to index them like this:: + + >>> net4[1] + IPv4Address('192.0.2.1') + >>> net4[-1] + IPv4Address('192.0.2.255') + >>> net6[1] + IPv6Address('2001::1') + >>> net6[-1] + IPv6Address('2001::ffff:ffff') + + +It also means that network objects lend themselves to using the list +membership test syntax like this:: + + if address in network: + # do something + +Containment testing is done efficiently based on the network prefix:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> addr4 in ipaddress.ip_network('192.0.2.0/24') + True + >>> addr4 in ipaddress.ip_network('192.0.3.0/24') + False + + +Comparisons +=========== + +:mod:`ipaddress` provides some simple, hopefully intuitive ways to compare +objects, where it makes sense:: + + >>> ipaddress.ip_address('192.0.2.1') < ipaddress.ip_address('192.0.2.2') + True + +A :exc:`TypeError` exception is raised if you try to compare objects of +different versions or different types. + + +Using IP Addresses with other modules +===================================== + +Other modules that use IP addresses (such as :mod:`socket`) usually won't +accept objects from this module directly. Instead, they must be coerced to +an integer or string that the other module will accept:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> str(addr4) + '192.0.2.1' + >>> int(addr4) + 3221225985 + + +Exceptions raised by :mod:`ipaddress` +===================================== + +If you try to create an address/network/interface object with an invalid value +for either the address or netmask, :mod:`ipaddress` will raise an +:exc:`AddressValueError` or :exc:`NetmaskValueError` respectively. However, +this applies only when calling the class constructors directly. The factory +functions and other module level functions will just raise :exc:`ValueError`. + +Both of the module specific exceptions have :exc:`ValueError` as their +parent class, so if you're not concerned with the particular type of error, +you can still do the following:: + + try: + ipaddress.IPv4Address(address) + except ValueError: + print 'address/netmask is invalid: %s' % address -- Repository URL: http://hg.python.org/cpython From brett at python.org Sat May 26 20:27:21 2012 From: brett at python.org (Brett Cannon) Date: Sat, 26 May 2012 14:27:21 -0400 Subject: [Python-checkins] cpython: issue 14660: Implement PEP 420, namespace packages. In-Reply-To: <20120525104130.19b416fb@limelight.wooz.org> References: <20120525104130.19b416fb@limelight.wooz.org> Message-ID: On Fri, May 25, 2012 at 10:41 AM, Barry Warsaw wrote: > On May 25, 2012, at 10:31 AM, Brett Cannon wrote: > > >Is documentation coming in a separate commit? > > Yes. I've been reworking the import machinery documentation; it's a > work-in-progress on the pep-420 feature clone ('importdocs' branch). I > made > some good progress and then got side-tracked, but I'm planning on getting > back > to it soon. OK, great! Something to keep in the back of your head, Barry, is the naming of importlib.find_loader(). Since its return value is not the same as what the PEP introduces it might stand for a name change (it's new in Python 3.3 so it can be whatever makes sense). Also just noticed that there is no update to importlib.abc.Finder for find_loader(), which I assume is because of the hasattr() check in PathFinder. That's fine, but it would be good to update the docs for ABC so people know that is an optional interface they can implement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-checkins at python.org Sat May 26 20:28:44 2012 From: python-checkins at python.org (brett.cannon) Date: Sat, 26 May 2012 20:28:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_some_redundant_decor?= =?utf8?q?ators=2E?= Message-ID: http://hg.python.org/cpython/rev/fd7afc53ac46 changeset: 77164:fd7afc53ac46 user: Brett Cannon date: Sat May 26 14:28:21 2012 -0400 summary: Remove some redundant decorators. files: Lib/importlib/_bootstrap.py | 2 -- Python/importlib.h | Bin 2 files changed, 0 insertions(+), 2 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -949,8 +949,6 @@ def module_repr(cls, module): return "".format(module.__name__) - @set_package - @set_loader @module_for_loader def load_module(self, module): """Load a namespace module.""" diff --git a/Python/importlib.h b/Python/importlib.h index bdb3644db479da7d7f6b949be135c6b1c3d1fc33..cd8fbebb9ff7c4a671320695174aa8953db75845 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 20:36:04 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 20:36:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Recognize_=27=3C=3E=27_as_a?= =?utf8?q?_special_case_of_an_angle-addr_in_header=5Fvalue=5Fparser=2E?= Message-ID: http://hg.python.org/cpython/rev/a66bf07fca4d changeset: 77165:a66bf07fca4d user: R David Murray date: Sat May 26 14:31:12 2012 -0400 summary: Recognize '<>' as a special case of an angle-addr in header_value_parser. Although '<>' is invalid according to RFC 5322, SMTP uses it for various things, and it sometimes ends up in email headers. This patch changes get_angle_addr to recognize it and just register a Defect instead of raising a parsing error. files: Lib/email/_header_value_parser.py | 12 +++++++- Lib/test/test_email/test__header_value_parser.py | 15 +++++++++- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -791,6 +791,8 @@ for x in self: if x.token_type == 'addr-spec': return x.addr_spec + else: + return '<>' class ObsRoute(TokenList): @@ -1829,6 +1831,14 @@ "expected angle-addr but found '{}'".format(value)) angle_addr.append(ValueTerminal('<', 'angle-addr-start')) value = value[1:] + # Although it is not legal per RFC5322, SMTP uses '<>' in certain + # circumstances. + if value[0] == '>': + angle_addr.append(ValueTerminal('>', 'angle-addr-end')) + angle_addr.defects.append(errors.InvalidHeaderDefect( + "null addr-spec in angle-addr")) + value = value[1:] + return angle_addr, value try: token, value = get_addr_spec(value) except errors.HeaderParseError: @@ -1838,7 +1848,7 @@ "obsolete route specification in angle-addr")) except errors.HeaderParseError: raise errors.HeaderParseError( - "expected addr-spec or but found '{}'".format(value)) + "expected addr-spec or obs-route but found '{}'".format(value)) angle_addr.append(token) token, value = get_addr_spec(value) angle_addr.append(token) diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py --- a/Lib/test/test_email/test__header_value_parser.py +++ b/Lib/test/test_email/test__header_value_parser.py @@ -1429,6 +1429,19 @@ self.assertIsNone(angle_addr.route) self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + def test_get_angle_addr_empty(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<>', + '<>', + '<>', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertIsNone(angle_addr.local_part) + self.assertIsNone(angle_addr.domain) + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, '<>') + def test_get_angle_addr_with_cfws(self): angle_addr = self._test_get_x(parser.get_angle_addr, ' (foo) (bar)', @@ -2007,7 +2020,7 @@ self.assertEqual(group.mailboxes, group.all_mailboxes) - def test_get_troup_null_addr_spec(self): + def test_get_group_null_addr_spec(self): group = self._test_get_x(parser.get_group, 'foo: <>;', 'foo: <>;', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 20:36:05 2012 From: python-checkins at python.org (r.david.murray) Date: Sat, 26 May 2012 20:36:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=238739=3A_upgrade_smtpd_to?= =?utf8?q?_RFC_5321_and_1870=2E?= Message-ID: http://hg.python.org/cpython/rev/ec7456b3f4fe changeset: 77166:ec7456b3f4fe user: R David Murray date: Sat May 26 14:33:59 2012 -0400 summary: #8739: upgrade smtpd to RFC 5321 and 1870. smtpd now handles EHLO and has infrastructure for extended smtp command mode. The SIZE extension is also implemented. In order to support parameters on MAIL FROM, the RFC 5322 parser from the email package is used to parse the address "token". Logging subclasses things and overrides __init__, so it was necessary to update those __init__ functions in the logging tests to make the logging tests pass. The original suggestion and patch were by Alberto Trevino. Juhana Jauhiainen added the --size argument and SIZE parameter support. Michele Orr? improved the patch and added more tests. Dan Boswell conditionalized various bits of code on whether or not we are in HELO or EHLO mode, as well as some other improvements and tests. I finalized the patch and added the address parsing. files: Doc/library/smtpd.rst | 20 +- Lib/smtpd.py | 245 ++++++++++++++++++----- Lib/test/test_logging.py | 3 + Lib/test/test_smtpd.py | 280 ++++++++++++++++++++++++-- Lib/test/test_smtplib.py | 19 +- Misc/ACKS | 3 + Misc/NEWS | 3 + 7 files changed, 482 insertions(+), 91 deletions(-) diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,17 +20,24 @@ Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. + + SMTPServer Objects ------------------ -.. class:: SMTPServer(localaddr, remoteaddr) +.. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It inherits from :class:`asyncore.dispatcher`, and so will insert itself into :mod:`asyncore`'s event loop on instantiation. + *data_size_limit* specifies the maximum number of bytes that will be + accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no + limit. + .. method:: process_message(peer, mailfrom, rcpttos, data) Raise :exc:`NotImplementedError` exception. Override this in subclasses to @@ -155,11 +162,15 @@ Command Action taken ======== =================================================================== HELO Accepts the greeting from the client and stores it in - :attr:`seen_greeting`. + :attr:`seen_greeting`. Sets server to base command mode. + EHLO Accepts the greeting from the client and stores it in + :attr:`seen_greeting`. Sets server to extended command mode. NOOP Takes no action. QUIT Closes the connection cleanly. MAIL Accepts the "MAIL FROM:" syntax and stores the supplied address as - :attr:`mailfrom`. + :attr:`mailfrom`. In extended command mode, accepts the + :rfc:`1870` SIZE attribute and responds appropriately based on the + value of ``data_size_limit``. RCPT Accepts the "RCPT TO:" syntax and stores the supplied addresses in the :attr:`rcpttos` list. RSET Resets the :attr:`mailfrom`, :attr:`rcpttos`, and @@ -167,4 +178,7 @@ DATA Sets the internal state to :attr:`DATA` and stores remaining lines from the client in :attr:`received_data` until the terminator "\r\n.\r\n" is received. + HELP Returns minimal information on command syntax + VRFY Returns code 252 (the server doesn't know if the address is valid) + EXPN Reports that the command is not implemented. ======== =================================================================== diff --git a/Lib/smtpd.py b/Lib/smtpd.py --- a/Lib/smtpd.py +++ b/Lib/smtpd.py @@ -1,5 +1,5 @@ #! /usr/bin/env python3 -"""An RFC 2821 smtp proxy. +"""An RFC 5321 smtp proxy. Usage: %(program)s [options] [localhost:localport [remotehost:remoteport]] @@ -20,6 +20,11 @@ Use `classname' as the concrete SMTP proxy class. Uses `PureProxy' by default. + --size limit + -s limit + Restrict the total size of the incoming message to "limit" number of + bytes via the RFC 1870 SIZE extension. Defaults to 33554432 bytes. + --debug -d Turn on debugging prints. @@ -35,10 +40,9 @@ and if remoteport is not given, then 25 is used. """ - # Overview: # -# This file implements the minimal SMTP protocol as defined in RFC 821. It +# This file implements the minimal SMTP protocol as defined in RFC 5321. It # has a hierarchy of classes which implement the backend functionality for the # smtpd. A number of classes are provided: # @@ -66,7 +70,7 @@ # # - support mailbox delivery # - alias files -# - ESMTP +# - Handle more ESMTP extensions # - handle error codes from the backend smtpd import sys @@ -77,12 +81,14 @@ import socket import asyncore import asynchat +import collections from warnings import warn +from email._header_value_parser import get_addr_spec, get_angle_addr __all__ = ["SMTPServer","DebuggingServer","PureProxy","MailmanProxy"] program = sys.argv[0] -__version__ = 'Python SMTP proxy version 0.2' +__version__ = 'Python SMTP proxy version 0.3' class Devnull: @@ -94,9 +100,9 @@ NEWLINE = '\n' EMPTYSTRING = '' COMMASPACE = ', ' +DATA_SIZE_DEFAULT = 33554432 - def usage(code, msg=''): print(__doc__ % globals(), file=sys.stderr) if msg: @@ -104,19 +110,23 @@ sys.exit(code) - class SMTPChannel(asynchat.async_chat): COMMAND = 0 DATA = 1 - data_size_limit = 33554432 command_size_limit = 512 + command_size_limits = collections.defaultdict(lambda x=command_size_limit: x) + command_size_limits.update({ + 'MAIL': command_size_limit + 26, + }) + max_command_size_limit = max(command_size_limits.values()) - def __init__(self, server, conn, addr): + def __init__(self, server, conn, addr, data_size_limit=DATA_SIZE_DEFAULT): asynchat.async_chat.__init__(self, conn) self.smtp_server = server self.conn = conn self.addr = addr + self.data_size_limit = data_size_limit self.received_lines = [] self.smtp_state = self.COMMAND self.seen_greeting = '' @@ -137,6 +147,7 @@ print('Peer:', repr(self.peer), file=DEBUGSTREAM) self.push('220 %s %s' % (self.fqdn, __version__)) self.set_terminator(b'\r\n') + self.extended_smtp = False # properties for backwards-compatibility @property @@ -268,7 +279,7 @@ def collect_incoming_data(self, data): limit = None if self.smtp_state == self.COMMAND: - limit = self.command_size_limit + limit = self.max_command_size_limit elif self.smtp_state == self.DATA: limit = self.data_size_limit if limit and self.num_bytes > limit: @@ -283,11 +294,7 @@ print('Data:', repr(line), file=DEBUGSTREAM) self.received_lines = [] if self.smtp_state == self.COMMAND: - if self.num_bytes > self.command_size_limit: - self.push('500 Error: line too long') - self.num_bytes = 0 - return - self.num_bytes = 0 + sz, self.num_bytes = self.num_bytes, 0 if not line: self.push('500 Error: bad syntax') return @@ -299,9 +306,14 @@ else: command = line[:i].upper() arg = line[i+1:].strip() + max_sz = (self.command_size_limits[command] + if self.extended_smtp else self.command_size_limit) + if sz > max_sz: + self.push('500 Error: line too long') + return method = getattr(self, 'smtp_' + command, None) if not method: - self.push('502 Error: command "%s" not implemented' % command) + self.push('500 Error: command "%s" not recognized' % command) return method(arg) return @@ -310,12 +322,12 @@ self.push('451 Internal confusion') self.num_bytes = 0 return - if self.num_bytes > self.data_size_limit: + if self.data_size_limit and self.num_bytes > self.data_size_limit: self.push('552 Error: Too much mail data') self.num_bytes = 0 return # Remove extraneous carriage returns and de-transparency according - # to RFC 821, Section 4.5.2. + # to RFC 5321, Section 4.5.2. data = [] for text in line.split('\r\n'): if text and text[0] == '.': @@ -333,7 +345,7 @@ self.num_bytes = 0 self.set_terminator(b'\r\n') if not status: - self.push('250 Ok') + self.push('250 OK') else: self.push(status) @@ -346,66 +358,188 @@ self.push('503 Duplicate HELO/EHLO') else: self.seen_greeting = arg + self.extended_smtp = False self.push('250 %s' % self.fqdn) + def smtp_EHLO(self, arg): + if not arg: + self.push('501 Syntax: EHLO hostname') + return + if self.seen_greeting: + self.push('503 Duplicate HELO/EHLO') + else: + self.seen_greeting = arg + self.extended_smtp = True + self.push('250-%s' % self.fqdn) + if self.data_size_limit: + self.push('250-SIZE %s' % self.data_size_limit) + self.push('250 HELP') + def smtp_NOOP(self, arg): if arg: self.push('501 Syntax: NOOP') else: - self.push('250 Ok') + self.push('250 OK') def smtp_QUIT(self, arg): # args is ignored self.push('221 Bye') self.close_when_done() - # factored - def __getaddr(self, keyword, arg): - address = None + def _strip_command_keyword(self, keyword, arg): keylen = len(keyword) if arg[:keylen].upper() == keyword: - address = arg[keylen:].strip() - if not address: - pass - elif address[0] == '<' and address[-1] == '>' and address != '<>': - # Addresses can be in the form but watch out - # for null address, e.g. <> - address = address[1:-1] - return address + return arg[keylen:].strip() + return '' + + def _getaddr(self, arg): + if not arg: + return '', '' + if arg.lstrip().startswith('<'): + address, rest = get_angle_addr(arg) + else: + address, rest = get_addr_spec(arg) + if not address: + return address, rest + return address.addr_spec, rest + + def _getparams(self, params): + # Return any parameters that appear to be syntactically valid according + # to RFC 1869, ignore all others. (Postel rule: accept what we can.) + params = [param.split('=', 1) for param in params.split() + if '=' in param] + return {k: v for k, v in params if k.isalnum()} + + def smtp_HELP(self, arg): + if arg: + extended = ' [SP ') + else: + self.push('501 Supported commands: EHLO HELO MAIL RCPT ' + 'DATA RSET NOOP QUIT VRFY') + else: + self.push('250 Supported commands: EHLO HELO MAIL RCPT DATA ' + 'RSET NOOP QUIT VRFY') + + def smtp_VRFY(self, arg): + if arg: + address, params = self._getaddr(arg) + if address: + self.push('252 Cannot VRFY user, but will accept message ' + 'and attempt delivery') + else: + self.push('502 Could not VRFY %s' % arg) + else: + self.push('501 Syntax: VRFY
') def smtp_MAIL(self, arg): if not self.seen_greeting: self.push('503 Error: send HELO first'); return - print('===> MAIL', arg, file=DEBUGSTREAM) - address = self.__getaddr('FROM:', arg) if arg else None + syntaxerr = '501 Syntax: MAIL FROM:
' + if self.extended_smtp: + syntaxerr += ' [SP ]' + if arg is None: + self.push(syntaxerr) + return + arg = self._strip_command_keyword('FROM:', arg) + address, params = self._getaddr(arg) if not address: - self.push('501 Syntax: MAIL FROM:
') + self.push(syntaxerr) + return + if not self.extended_smtp and params: + self.push(syntaxerr) + return + if not address: + self.push(syntaxerr) return if self.mailfrom: self.push('503 Error: nested MAIL command') return + params = self._getparams(params.upper()) + if params is None: + self.push(syntaxerr) + return + size = params.pop('SIZE', None) + if size: + if not size.isdigit(): + self.push(syntaxerr) + return + elif self.data_size_limit and int(size) > self.data_size_limit: + self.push('552 Error: message size exceeds fixed maximum message size') + return + if len(params.keys()) > 0: + self.push('555 MAIL FROM parameters not recognized or not implemented') + return self.mailfrom = address print('sender:', self.mailfrom, file=DEBUGSTREAM) - self.push('250 Ok') + self.push('250 OK') def smtp_RCPT(self, arg): if not self.seen_greeting: self.push('503 Error: send HELO first'); return - print('===> RCPT', arg, file=DEBUGSTREAM) if not self.mailfrom: self.push('503 Error: need MAIL command') return - address = self.__getaddr('TO:', arg) if arg else None + syntaxerr = '501 Syntax: RCPT TO:
' + if self.extended_smtp: + syntaxerr += ' [SP ]' + if arg is None: + self.push(syntaxerr) + return + arg = self._strip_command_keyword('TO:', arg) + address, params = self._getaddr(arg) + if not address: + self.push(syntaxerr) + return + if params: + if self.extended_smtp: + params = self._getparams(params.upper()) + if params is None: + self.push(syntaxerr) + return + else: + self.push(syntaxerr) + return + if not address: + self.push(syntaxerr) + return + if params and len(params.keys()) > 0: + self.push('555 RCPT TO parameters not recognized or not implemented') + return if not address: self.push('501 Syntax: RCPT TO:
') return self.rcpttos.append(address) print('recips:', self.rcpttos, file=DEBUGSTREAM) - self.push('250 Ok') + self.push('250 OK') def smtp_RSET(self, arg): if arg: @@ -416,13 +550,12 @@ self.rcpttos = [] self.received_data = '' self.smtp_state = self.COMMAND - self.push('250 Ok') + self.push('250 OK') def smtp_DATA(self, arg): if not self.seen_greeting: self.push('503 Error: send HELO first'); return - if not self.rcpttos: self.push('503 Error: need RCPT command') return @@ -433,15 +566,20 @@ self.set_terminator(b'\r\n.\r\n') self.push('354 End data with .') + # Commands that have not been implemented + def smtp_EXPN(self, arg): + self.push('502 EXPN not implemented') - + class SMTPServer(asyncore.dispatcher): # SMTPChannel class to use for managing client connections channel_class = SMTPChannel - def __init__(self, localaddr, remoteaddr): + def __init__(self, localaddr, remoteaddr, + data_size_limit=DATA_SIZE_DEFAULT): self._localaddr = localaddr self._remoteaddr = remoteaddr + self.data_size_limit = data_size_limit asyncore.dispatcher.__init__(self) try: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) @@ -459,7 +597,7 @@ def handle_accepted(self, conn, addr): print('Incoming connection from %s' % repr(addr), file=DEBUGSTREAM) - channel = self.channel_class(self, conn, addr) + channel = self.channel_class(self, conn, addr, self.data_size_limit) # API for "doing something useful with the message" def process_message(self, peer, mailfrom, rcpttos, data): @@ -487,7 +625,6 @@ raise NotImplementedError - class DebuggingServer(SMTPServer): # Do something with the gathered message def process_message(self, peer, mailfrom, rcpttos, data): @@ -503,7 +640,6 @@ print('------------ END MESSAGE ------------') - class PureProxy(SMTPServer): def process_message(self, peer, mailfrom, rcpttos, data): lines = data.split('\n') @@ -544,7 +680,6 @@ return refused - class MailmanProxy(PureProxy): def process_message(self, peer, mailfrom, rcpttos, data): from io import StringIO @@ -623,19 +758,18 @@ msg.Enqueue(mlist, torequest=1) - class Options: setuid = 1 classname = 'PureProxy' + size_limit = None - def parseargs(): global DEBUGSTREAM try: opts, args = getopt.getopt( - sys.argv[1:], 'nVhc:d', - ['class=', 'nosetuid', 'version', 'help', 'debug']) + sys.argv[1:], 'nVhc:s:d', + ['class=', 'nosetuid', 'version', 'help', 'size=', 'debug']) except getopt.error as e: usage(1, e) @@ -652,6 +786,13 @@ options.classname = arg elif opt in ('-d', '--debug'): DEBUGSTREAM = sys.stderr + elif opt in ('-s', '--size'): + try: + int_size = int(arg) + options.size_limit = int_size + except: + print('Invalid size: ' + arg, file=sys.stderr) + sys.exit(1) # parse the rest of the arguments if len(args) < 1: @@ -686,7 +827,6 @@ return options - if __name__ == '__main__': options = parseargs() # Become nobody @@ -699,7 +839,8 @@ import __main__ as mod class_ = getattr(mod, classname) proxy = class_((options.localhost, options.localport), - (options.remotehost, options.remoteport)) + (options.remotehost, options.remoteport), + options.size_limit) if options.setuid: try: import pwd diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -663,6 +663,7 @@ self.smtp_server = server self.conn = conn self.addr = addr + self.data_size_limit = None self.received_lines = [] self.smtp_state = self.COMMAND self.seen_greeting = '' @@ -682,6 +683,7 @@ return self.push('220 %s %s' % (self.fqdn, smtpd.__version__)) self.set_terminator(b'\r\n') + self.extended_smtp = False class TestSMTPServer(smtpd.SMTPServer): @@ -709,6 +711,7 @@ def __init__(self, addr, handler, poll_interval, sockmap): self._localaddr = addr self._remoteaddr = None + self.data_size_limit = None self.sockmap = sockmap asyncore.dispatcher.__init__(self, map=sockmap) try: diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py --- a/Lib/test/test_smtpd.py +++ b/Lib/test/test_smtpd.py @@ -1,4 +1,4 @@ -from unittest import TestCase +import unittest from test import support, mock_socket import socket import io @@ -26,7 +26,7 @@ raise DummyDispatcherBroken() -class SMTPDServerTest(TestCase): +class SMTPDServerTest(unittest.TestCase): def setUp(self): smtpd.socket = asyncore.socket = mock_socket @@ -39,7 +39,7 @@ channel.socket.queue_recv(line) channel.handle_read() - write_line(b'HELO test.example') + write_line(b'HELO example') write_line(b'MAIL From:eggs at example') write_line(b'RCPT To:spam at example') write_line(b'DATA') @@ -50,7 +50,7 @@ asyncore.socket = smtpd.socket = socket -class SMTPDChannelTest(TestCase): +class SMTPDChannelTest(unittest.TestCase): def setUp(self): smtpd.socket = asyncore.socket = mock_socket self.old_debugstream = smtpd.DEBUGSTREAM @@ -79,36 +79,94 @@ self.assertEqual(self.channel.socket.last, b'500 Error: bad syntax\r\n') - def test_EHLO_not_implemented(self): - self.write_line(b'EHLO test.example') + def test_EHLO(self): + self.write_line(b'EHLO example') + self.assertEqual(self.channel.socket.last, b'250 HELP\r\n') + + def test_EHLO_bad_syntax(self): + self.write_line(b'EHLO') self.assertEqual(self.channel.socket.last, - b'502 Error: command "EHLO" not implemented\r\n') + b'501 Syntax: EHLO hostname\r\n') + + def test_EHLO_duplicate(self): + self.write_line(b'EHLO example') + self.write_line(b'EHLO example') + self.assertEqual(self.channel.socket.last, + b'503 Duplicate HELO/EHLO\r\n') + + def test_EHLO_HELO_duplicate(self): + self.write_line(b'EHLO example') + self.write_line(b'HELO example') + self.assertEqual(self.channel.socket.last, + b'503 Duplicate HELO/EHLO\r\n') def test_HELO(self): name = smtpd.socket.getfqdn() - self.write_line(b'HELO test.example') + self.write_line(b'HELO example') self.assertEqual(self.channel.socket.last, '250 {}\r\n'.format(name).encode('ascii')) + def test_HELO_EHLO_duplicate(self): + self.write_line(b'HELO example') + self.write_line(b'EHLO example') + self.assertEqual(self.channel.socket.last, + b'503 Duplicate HELO/EHLO\r\n') + + def test_HELP(self): + self.write_line(b'HELP') + self.assertEqual(self.channel.socket.last, + b'250 Supported commands: EHLO HELO MAIL RCPT ' + \ + b'DATA RSET NOOP QUIT VRFY\r\n') + + def test_HELP_command(self): + self.write_line(b'HELP MAIL') + self.assertEqual(self.channel.socket.last, + b'250 Syntax: MAIL FROM:
\r\n') + + def test_HELP_command_unknown(self): + self.write_line(b'HELP SPAM') + self.assertEqual(self.channel.socket.last, + b'501 Supported commands: EHLO HELO MAIL RCPT ' + \ + b'DATA RSET NOOP QUIT VRFY\r\n') + def test_HELO_bad_syntax(self): self.write_line(b'HELO') self.assertEqual(self.channel.socket.last, b'501 Syntax: HELO hostname\r\n') def test_HELO_duplicate(self): - self.write_line(b'HELO test.example') - self.write_line(b'HELO test.example') + self.write_line(b'HELO example') + self.write_line(b'HELO example') self.assertEqual(self.channel.socket.last, b'503 Duplicate HELO/EHLO\r\n') + def test_HELO_parameter_rejected_when_extensions_not_enabled(self): + self.extended_smtp = False + self.write_line(b'HELO example') + self.write_line(b'MAIL from: SIZE=1234') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: MAIL FROM:
\r\n') + + def test_MAIL_allows_space_after_colon(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL from: ') + self.assertEqual(self.channel.socket.last, + b'250 OK\r\n') + + def test_extended_MAIL_allows_space_after_colon(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: size=20') + self.assertEqual(self.channel.socket.last, + b'250 OK\r\n') + def test_NOOP(self): self.write_line(b'NOOP') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') def test_HELO_NOOP(self): self.write_line(b'HELO example') self.write_line(b'NOOP') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') def test_NOOP_bad_syntax(self): self.write_line(b'NOOP hi') @@ -136,15 +194,29 @@ def test_command_too_long(self): self.write_line(b'HELO example') - self.write_line(b'MAIL from ' + + self.write_line(b'MAIL from: ' + b'a' * self.channel.command_size_limit + b'@example') self.assertEqual(self.channel.socket.last, b'500 Error: line too long\r\n') - def test_data_too_long(self): - # Small hack. Setting limit to 2K octets here will save us some time. - self.channel.data_size_limit = 2048 + def test_MAIL_command_limit_extended_with_SIZE(self): + self.write_line(b'EHLO example') + fill_len = self.channel.command_size_limit - len('MAIL from:<@example>') + self.write_line(b'MAIL from:<' + + b'a' * fill_len + + b'@example> SIZE=1234') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + self.write_line(b'MAIL from:<' + + b'a' * (fill_len + 26) + + b'@example> SIZE=1234') + self.assertEqual(self.channel.socket.last, + b'500 Error: line too long\r\n') + + def test_data_longer_than_default_data_size_limit(self): + # Hack the default so we don't have to generate so much data. + self.channel.data_size_limit = 1048 self.write_line(b'HELO example') self.write_line(b'MAIL From:eggs at example') self.write_line(b'RCPT To:spam at example') @@ -154,28 +226,93 @@ self.assertEqual(self.channel.socket.last, b'552 Error: Too much mail data\r\n') + def test_MAIL_size_parameter(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: SIZE=512') + self.assertEqual(self.channel.socket.last, + b'250 OK\r\n') + + def test_MAIL_invalid_size_parameter(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: SIZE=invalid') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: MAIL FROM:
[SP ]\r\n') + + def test_MAIL_RCPT_unknown_parameters(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: ham=green') + self.assertEqual(self.channel.socket.last, + b'555 MAIL FROM parameters not recognized or not implemented\r\n') + + self.write_line(b'MAIL FROM:') + self.write_line(b'RCPT TO: ham=green') + self.assertEqual(self.channel.socket.last, + b'555 RCPT TO parameters not recognized or not implemented\r\n') + + def test_MAIL_size_parameter_larger_than_default_data_size_limit(self): + self.channel.data_size_limit = 1048 + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: SIZE=2096') + self.assertEqual(self.channel.socket.last, + b'552 Error: message size exceeds fixed maximum message size\r\n') + def test_need_MAIL(self): self.write_line(b'HELO example') self.write_line(b'RCPT to:spam at example') self.assertEqual(self.channel.socket.last, b'503 Error: need MAIL command\r\n') - def test_MAIL_syntax(self): + def test_MAIL_syntax_HELO(self): self.write_line(b'HELO example') self.write_line(b'MAIL from eggs at example') self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
\r\n') + b'501 Syntax: MAIL FROM:
\r\n') - def test_MAIL_missing_from(self): + def test_MAIL_syntax_EHLO(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from eggs at example') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: MAIL FROM:
[SP ]\r\n') + + def test_MAIL_missing_address(self): self.write_line(b'HELO example') self.write_line(b'MAIL from:') self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
\r\n') + b'501 Syntax: MAIL FROM:
\r\n') def test_MAIL_chevrons(self): self.write_line(b'HELO example') self.write_line(b'MAIL from:') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + def test_MAIL_empty_chevrons(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from:<>') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + def test_MAIL_quoted_localpart(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: <"Fred Blogs"@example.com>') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') + + def test_MAIL_quoted_localpart_no_angles(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: "Fred Blogs"@example.com') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') + + def test_MAIL_quoted_localpart_with_size(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: <"Fred Blogs"@example.com> SIZE=1000') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') + + def test_MAIL_quoted_localpart_with_size_no_angles(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: "Fred Blogs"@example.com SIZE=1000') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') def test_nested_MAIL(self): self.write_line(b'HELO example') @@ -184,6 +321,22 @@ self.assertEqual(self.channel.socket.last, b'503 Error: nested MAIL command\r\n') + def test_VRFY(self): + self.write_line(b'VRFY eggs at example') + self.assertEqual(self.channel.socket.last, + b'252 Cannot VRFY user, but will accept message and attempt ' + \ + b'delivery\r\n') + + def test_VRFY_syntax(self): + self.write_line(b'VRFY') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: VRFY
\r\n') + + def test_EXPN_not_implemented(self): + self.write_line(b'EXPN') + self.assertEqual(self.channel.socket.last, + b'502 EXPN not implemented\r\n') + def test_no_HELO_MAIL(self): self.write_line(b'MAIL from:') self.assertEqual(self.channel.socket.last, @@ -196,13 +349,26 @@ self.assertEqual(self.channel.socket.last, b'503 Error: need RCPT command\r\n') - def test_RCPT_syntax(self): + def test_RCPT_syntax_HELO(self): self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs at example') + self.write_line(b'MAIL From: eggs at example') self.write_line(b'RCPT to eggs at example') self.assertEqual(self.channel.socket.last, b'501 Syntax: RCPT TO:
\r\n') + def test_RCPT_syntax_EHLO(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL From: eggs at example') + self.write_line(b'RCPT to eggs at example') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: RCPT TO:
[SP ]\r\n') + + def test_RCPT_lowercase_to_OK(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL From: eggs at example') + self.write_line(b'RCPT to: ') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + def test_no_HELO_RCPT(self): self.write_line(b'RCPT to eggs at example') self.assertEqual(self.channel.socket.last, @@ -211,15 +377,15 @@ def test_data_dialog(self): self.write_line(b'HELO example') self.write_line(b'MAIL From:eggs at example') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.write_line(b'RCPT To:spam at example') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.write_line(b'DATA') self.assertEqual(self.channel.socket.last, b'354 End data with .\r\n') self.write_line(b'data\r\nmore\r\n.') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.assertEqual(self.server.messages, [('peer', 'eggs at example', ['spam at example'], 'data\nmore')]) @@ -267,7 +433,7 @@ self.write_line(b'MAIL From:eggs at example') self.write_line(b'RCPT To:spam at example') self.write_line(b'RSET') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.write_line(b'MAIL From:foo at example') self.write_line(b'RCPT To:eggs at example') self.write_line(b'DATA') @@ -278,12 +444,18 @@ def test_HELO_RSET(self): self.write_line(b'HELO example') self.write_line(b'RSET') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') def test_RSET_syntax(self): self.write_line(b'RSET hi') self.assertEqual(self.channel.socket.last, b'501 Syntax: RSET\r\n') + def test_unknown_command(self): + self.write_line(b'UNKNOWN_CMD') + self.assertEqual(self.channel.socket.last, + b'500 Error: command "UNKNOWN_CMD" not ' + \ + b'recognized\r\n') + def test_attribute_deprecations(self): with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__server @@ -330,8 +502,54 @@ with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__addr = 'spam' -def test_main(): - support.run_unittest(SMTPDServerTest, SMTPDChannelTest) + +class SMTPDChannelWithDataSizeLimitTest(unittest.TestCase): + + def setUp(self): + smtpd.socket = asyncore.socket = mock_socket + self.debug = smtpd.DEBUGSTREAM = io.StringIO() + self.server = DummyServer('a', 'b') + conn, addr = self.server.accept() + # Set DATA size limit to 32 bytes for easy testing + self.channel = smtpd.SMTPChannel(self.server, conn, addr, 32) + + def tearDown(self): + asyncore.close_all() + asyncore.socket = smtpd.socket = socket + + def write_line(self, line): + self.channel.socket.queue_recv(line) + self.channel.handle_read() + + def test_data_limit_dialog(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL From:eggs at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.write_line(b'RCPT To:spam at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + self.write_line(b'DATA') + self.assertEqual(self.channel.socket.last, + b'354 End data with .\r\n') + self.write_line(b'data\r\nmore\r\n.') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.server.messages, + [('peer', 'eggs at example', ['spam at example'], 'data\nmore')]) + + def test_data_limit_dialog_too_much_data(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL From:eggs at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.write_line(b'RCPT To:spam at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + self.write_line(b'DATA') + self.assertEqual(self.channel.socket.last, + b'354 End data with .\r\n') + self.write_line(b'This message is longer than 32 bytes\r\n.') + self.assertEqual(self.channel.socket.last, + b'552 Error: Too much mail data\r\n') + if __name__ == "__main__": - test_main() + unittest.main() diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -229,13 +229,13 @@ def testNOOP(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - expected = (250, b'Ok') + expected = (250, b'OK') self.assertEqual(smtp.noop(), expected) smtp.quit() def testRSET(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - expected = (250, b'Ok') + expected = (250, b'OK') self.assertEqual(smtp.rset(), expected) smtp.quit() @@ -246,10 +246,18 @@ self.assertEqual(smtp.ehlo(), expected) smtp.quit() + def testNotImplemented(self): + # EXPN isn't implemented in DebuggingServer + smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + expected = (502, b'EXPN not implemented') + smtp.putcmd('EXPN') + self.assertEqual(smtp.getreply(), expected) + smtp.quit() + def testVRFY(self): - # VRFY isn't implemented in DebuggingServer smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - expected = (502, b'Error: command "VRFY" not implemented') + expected = (252, b'Cannot VRFY user, but will accept message ' + \ + b'and attempt delivery') self.assertEqual(smtp.vrfy('nobody at nowhere.com'), expected) self.assertEqual(smtp.verify('nobody at nowhere.com'), expected) smtp.quit() @@ -265,7 +273,8 @@ def testHELP(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - self.assertEqual(smtp.help(), b'Error: command "HELP" not implemented') + self.assertEqual(smtp.help(), b'Supported commands: EHLO HELO MAIL ' + \ + b'RCPT DATA RSET NOOP QUIT VRFY') smtp.quit() def testSend(self): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -112,6 +112,7 @@ Matias Bordese Jurjen Bos Peter Bosch +Dan Boswell Eric Bouck Thierry Bousch Sebastian Boving @@ -494,6 +495,7 @@ Jack Jansen Bill Janssen Thomas Jarosch +Juhana Jauhiainen Zbigniew J?drzejewski-Szmek Julien Jehannet Drew Jenkins @@ -1039,6 +1041,7 @@ Richard Townsend David Townshend Laurence Tratt +Alberto Trevino Matthias Troffaes John Tromp Jason Trowbridge diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -46,6 +46,9 @@ Library ------- +- Issue #8739: Updated smtpd to support RFC 5321, and added support for the + RFC 1870 SIZE extension. + - Issue #665194: Added a localtime function to email.utils to provide an aware local datetime for use in setting Date headers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 21:39:36 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 26 May 2012 21:39:36 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Addressed_some_buildbot_err?= =?utf8?q?ors_and_comments_on_the_checkin_by_Antoine_on?= Message-ID: http://hg.python.org/cpython/rev/564caa2c61ae changeset: 77167:564caa2c61ae parent: 77159:63d2c9affb11 user: Vinay Sajip date: Sat May 26 20:36:12 2012 +0100 summary: Addressed some buildbot errors and comments on the checkin by Antoine on python-dev. files: Lib/distutils/sysconfig.py | 3 +- Lib/sysconfig.py | 3 +- Lib/test/test_venv.py | 25 +------- Lib/venv/__init__.py | 76 ++++++++++++++----------- 4 files changed, 50 insertions(+), 57 deletions(-) diff --git a/Lib/distutils/sysconfig.py b/Lib/distutils/sysconfig.py --- a/Lib/distutils/sysconfig.py +++ b/Lib/distutils/sysconfig.py @@ -47,7 +47,8 @@ return True return False _sys_home = getattr(sys, '_home', None) -if _sys_home and os.name == 'nt' and _sys_home.lower().endswith('pcbuild'): +if _sys_home and os.name == 'nt' and \ + _sys_home.lower().endswith(('pcbuild', 'pcbuild\\amd64')): _sys_home = os.path.dirname(_sys_home) def _python_build(): if _sys_home: diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -104,7 +104,8 @@ return False _sys_home = getattr(sys, '_home', None) -if _sys_home and os.name == 'nt' and _sys_home.lower().endswith('pcbuild'): +if _sys_home and os.name == 'nt' and \ + _sys_home.lower().endswith(('pcbuild', 'pcbuild\\amd64')): _sys_home = os.path.dirname(_sys_home) def is_python_build(check_home=False): diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -1,24 +1,7 @@ -#!/usr/bin/env python -# -# Copyright 2011 by Vinay Sajip. All Rights Reserved. -# -# Permission to use, copy, modify, and distribute this software and its -# documentation for any purpose and without fee is hereby granted, -# provided that the above copyright notice appear in all copies and that -# both that copyright notice and this permission notice appear in -# supporting documentation, and that the name of Vinay Sajip -# not be used in advertising or publicity pertaining to distribution -# of the software without specific, written prior permission. -# VINAY SAJIP DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING -# ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL -# VINAY SAJIP BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR -# ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER -# IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT -# OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +""" +Test harness for the venv module. -"""Test harness for the venv module. Run all tests. - -Copyright (C) 2011 Vinay Sajip. All Rights Reserved. +Copyright (C) 2011-2012 Vinay Sajip. """ import os @@ -93,7 +76,7 @@ data = self.get_text_file_contents(self.bindir, self.ps3name) self.assertTrue(data.startswith('#!%s%s' % (self.env_dir, os.sep))) fn = self.get_env_file(self.bindir, self.exe) - self.assertTrue(os.path.exists(fn)) + self.assertTrue(os.path.exists(fn), 'File %r exists' % fn) def test_overwrite_existing(self): """ diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -1,30 +1,34 @@ -# Copyright (C) 2011-2012 Vinay Sajip. -# -# Use with a Python executable built from the Python fork at -# -# https://bitbucket.org/vinay.sajip/pythonv/ as follows: -# -# python -m venv env_dir -# -# You'll need an Internet connection (needed to download distribute_setup.py). -# -# The script will change to the environment's binary directory and run -# -# ./python distribute_setup.py -# -# after which you can change to the environment's directory and do some -# installations, e.g. -# -# source bin/activate.sh -# pysetup3 install setuptools-git -# pysetup3 install Pygments -# pysetup3 install Jinja2 -# pysetup3 install SQLAlchemy -# pysetup3 install coverage -# -# Note that on Windows, distributions which include C extensions (e.g. coverage) -# may fail due to lack of a suitable C compiler. -# +""" +Virtual environment (venv) package for Python. Based on PEP 405. + +Copyright (C) 20011-2012 Vinay Sajip. All Rights Reserved. + +usage: python -m venv [-h] [--no-distribute] [--system-site-packages] + [--symlinks] [--clear] [--upgrade] + ENV_DIR [ENV_DIR ...] + +Creates virtual Python environments in one or more target directories. + +positional arguments: + ENV_DIR A directory to create the environment in. + +optional arguments: + -h, --help show this help message and exit + --no-distribute Don't install Distribute in the virtual environment.* + --system-site-packages + Give the virtual environment access to the system + site-packages dir. + --symlinks Attempt to symlink rather than copy. + --clear Delete the environment directory if it already exists. + If not specified and the directory exists, an error is + raised. + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. + +*Note: Distribute support will be available during the alpha phase to +facilitate testing third-party packages with venvs created using this package. +This support will be removed after the alpha phase. +""" import base64 import io import logging @@ -32,13 +36,17 @@ import os.path import shutil import sys +try: + import threading +except ImportError: + threading = None import zipfile logger = logging.getLogger(__name__) class Context: """ - Holds information about a current virtualisation request. + Holds information about a current venv creation/upgrade request. """ pass @@ -353,7 +361,8 @@ being processed. """ if not self.nodist: - self.install_distribute(context) + if threading: + self.install_distribute(context) def reader(self, stream, context): """ @@ -381,7 +390,6 @@ being processed. """ from subprocess import Popen, PIPE - from threading import Thread from urllib.request import urlretrieve url = 'http://python-distribute.org/distribute_setup.py' @@ -398,9 +406,9 @@ # Install Distribute in the env args = [context.env_exe, 'distribute_setup.py'] p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) - t1 = Thread(target=self.reader, args=(p.stdout, 'stdout')) + t1 = threading.Thread(target=self.reader, args=(p.stdout, 'stdout')) t1.start() - t2 = Thread(target=self.reader, args=(p.stderr, 'stderr')) + t2 = threading.Thread(target=self.reader, args=(p.stderr, 'stderr')) t2.start() p.wait() t1.join() @@ -478,8 +486,8 @@ parser.add_argument('--upgrade', default=False, action='store_true', dest='upgrade', help='Upgrade the environment ' 'directory to use this version ' - 'of Python, assuming it has been ' - 'upgraded in-place.') + 'of Python, assuming Python ' + 'has been upgraded in-place.') options = parser.parse_args(args) if options.upgrade and options.clear: raise ValueError('you cannot supply --upgrade and --clear together.') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 21:40:26 2012 From: python-checkins at python.org (vinay.sajip) Date: Sat, 26 May 2012 21:40:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merged_upstream_changes=2E?= Message-ID: http://hg.python.org/cpython/rev/1b5bdef96af0 changeset: 77168:1b5bdef96af0 parent: 77167:564caa2c61ae parent: 77166:ec7456b3f4fe user: Vinay Sajip date: Sat May 26 20:39:27 2012 +0100 summary: Merged upstream changes. files: Doc/howto/index.rst | 1 + Doc/howto/ipaddress.rst | 291 ++++++++++ Doc/library/smtpd.rst | 20 +- Lib/email/_header_value_parser.py | 12 +- Lib/importlib/_bootstrap.py | 2 - Lib/ipaddress.py | 136 +--- Lib/smtpd.py | 245 ++++++- Lib/test/test_email/test__header_value_parser.py | 15 +- Lib/test/test_ipaddress.py | 38 +- Lib/test/test_logging.py | 3 + Lib/test/test_smtpd.py | 280 ++++++++- Lib/test/test_smtplib.py | 19 +- Misc/ACKS | 3 + Misc/NEWS | 3 + Python/importlib.h | Bin 15 files changed, 844 insertions(+), 224 deletions(-) diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -28,4 +28,5 @@ urllib2.rst webservers.rst argparse.rst + ipaddress.rst diff --git a/Doc/howto/ipaddress.rst b/Doc/howto/ipaddress.rst new file mode 100644 --- /dev/null +++ b/Doc/howto/ipaddress.rst @@ -0,0 +1,291 @@ +.. _ipaddress-howto: + +*************** +Ipaddress Howto +*************** + +:author: Peter Moody + +.. topic:: Abstract + + This document is a gentle introduction to :mod:`ipaddress` module. + + +Creating Address/Network/Interface objects +========================================== + +Since :mod:`ipaddress` is a module for inspecting and manipulating IP address, +the first thing you'll want to do is create some objects. You can use +:mod:`ipaddress` to create objects from strings and integers. + + +A Note on IP Versions +--------------------- + +For readers that aren't particularly familiar with IP addressing, it's +important to know that the Internet Protocol is currently in the process +of moving from version 4 of the protocol to version 6. This transition is +occurring largely because version 4 of the protocol doesn't provide enough +addresses to handle the needs of the whole world, especially given the +increasing number of devices with direct connections to the internet. + +Explaining the details of the differences between the two versions of the +protocol is beyond the scope of this introduction, but readers need to at +least be aware that these two versions exist, and it will sometimes be +necessary to force the use of one version or the other. + + +IP Host Addresses +----------------- + +Addresses, often referred to as "host addresses" are the most basic unit +when working with IP addressing. The simplest way to create addresses is +to use the ``ip_address`` factory function, which automatically determines +whether to create an IPv4 or IPv6 address based on the passed in value:: + + >>> ipaddress.ip_address('192.0.2.1') + IPv4Address('192.0.2.1') + >>> ipaddress.ip_address('2001:DB8::1') + IPv6Address('2001:db8::1') + +Addresses can also be created directly from integers. Values that will +fit within 32 bits are assumed to be IPv4 addresses:: + + >>> ipaddress.ip_address(3221225985) + IPv4Address('192.0.2.1') + >>> ipaddress.ip_address(42540766411282592856903984951653826561) + IPv6Address('2001:db8::1') + +To force the use of IPv4 or IPv6 addresses, the relevant classes can be +invoked directly. This is particularly useful to force creation of IPv6 +addresses for small integers:: + + >>> ipaddress.ip_address(1) + IPv4Address('0.0.0.1') + >>> ipaddress.IPv4Address(1) + IPv4Address('0.0.0.1') + >>> ipaddress.IPv6Address(1) + IPv6Address('::1') + + +Defining Networks +----------------- + +Host addresses are usually grouped together into IP networks, so +:mod:`ipaddress` provides a way to create, inspect and manipulate network +definitions. IP network objects are constructed from strings that define the +range of host addresses that are part of that network. The simplest form +for that information is a "network address/network prefix" pair, where the +prefix defines the number of leading bits that are compared to determine +whether or not an address is part of the network and the network address +defines the expected value of those bits. + +As for addresses, a factory function is provided that determines the correct +IP version automatically:: + + >>> ipaddress.ip_network('192.0.2.0/24') + IPv4Network('192.0.2.0/24') + >>> ipaddress.ip_network('2001:db8::0/96') + IPv6Network('2001:db8::/96') + +Network objects cannot have any host bits set. The practical effect of this +is that ``192.0.2.1/24`` does not describe a network. Such definitions are +referred to as interface objects since the ip-on-a-network notation is +commonly used to describe network interfaces of a computer on a given network +and are described further in the next section. + +By default, attempting to create a network object with host bits set will +result in :exc:`ValueError` being raised. To request that the +additional bits instead be coerced to zero, the flag ``strict=False`` can +be passed to the constructor:: + + >>> ipaddress.ip_network('192.0.2.1/24') + Traceback (most recent call last): + ... + ValueError: 192.0.2.1/24 has host bits set + >>> ipaddress.ip_network('192.0.2.1/24', strict=False) + IPv4Network('192.0.2.0/24') + +While the string form offers significantly more flexibility, networks can +also be defined with integers, just like host addresses. In this case, the +network is considered to contain only the single address identified by the +integer, so the network prefix includes the entire network address:: + + >>> ipaddress.ip_network(3221225984) + IPv4Network('192.0.2.0/32') + >>> ipaddress.ip_network(42540766411282592856903984951653826560L) + IPv6Network('2001:db8::/128') + +Creation of a particular kind of network can be forced by calling the +class constructor directly instead of using the factory function. + + +Host Interfaces +--------------- + +As mentioned just above, if you need to describe an address on a particular +network, neither the address nor the network classes are sufficient. +Notation like ``192.0.2.1/24`` is commonly used network engineers and the +people who write tools for firewalls and routers as shorthand for "the host +``192.0.2.1`` on the network ``192.0.2.0/24``", Accordingly, :mod:`ipaddress` +provides a set of hybrid classes that associate an address with a particular +network. The interface for creation is identical to that for defining network +objects, except that the address portion isn't constrained to being a network +address. + + >>> ipaddress.ip_interface('192.0.2.1/24') + IPv4Interface('192.0.2.1/24') + >>> ipaddress.ip_network('2001:db8::1/96') + IPv6Interface('2001:db8::1/96') + +Integer inputs are accepted (as with networks), and use of a particular IP +version can be forced by calling the relevant constructor directly. + + +Inspecting Address/Network/Interface Objects +============================================ + +You've gone to the trouble of creating an IPv(4|6)(Address|Network|Interface) +object, so you probably want to get information about it. :mod:`ipaddress` +tries to make doing this easy and intuitive. + +Extracting the IP version:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> addr6 = ipaddress.ip_address('2001:db8::1') + >>> addr6.version + 6 + >>> addr4.version + 4 + +Obtaining the network from an interface:: + + >>> host4 = ipaddress.ip_interface('192.0.2.1/24') + >>> host4.network + IPv4Network('192.0.2.0/24') + >>> host6 = ipaddress.ip_interface('2001:db8::1/96') + >>> host6.network + IPv6Network('2001:db8::/96') + +Finding out how many individual addresses are in a network:: + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> net4.numhosts + 256 + >>> net6 = ipaddress.ip_network('2001:db8::0/96') + >>> net6.numhosts + 4294967296 + +Iterating through the 'usable' addresses on a network:: + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> for x in net4.iterhosts(): + print(x) + 192.0.2.1 + 192.0.2.2 + 192.0.2.3 + 192.0.2.4 + + 192.0.2.252 + 192.0.2.253 + 192.0.2.254 + + +Obtaining the netmask (i.e. set bits corresponding to the network prefix) or +the hostmask (any bits that are not part of the netmask): + + >>> net4 = ipaddress.ip_network('192.0.2.0/24') + >>> net4.netmask + IPv4Address('255.255.255.0') + >>> net4.hostmask + IPv4Address('0.0.0.255') + >>> net6 = ipaddress.ip_network('2001:db8::0/96') + >>> net6.netmask + IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::') + >>> net6.hostmask + IPv6Address('::ffff:ffff') + + +Exploding or compressing the address:: + + >>> net6.exploded + '2001:0000:0000:0000:0000:0000:0000:0000/96' + >>> addr6.exploded + '2001:0000:0000:0000:0000:0000:0000:0001' + + +Networks as lists of Addresses +============================== + +It's sometimes useful to treat networks as lists. This means it is possible +to index them like this:: + + >>> net4[1] + IPv4Address('192.0.2.1') + >>> net4[-1] + IPv4Address('192.0.2.255') + >>> net6[1] + IPv6Address('2001::1') + >>> net6[-1] + IPv6Address('2001::ffff:ffff') + + +It also means that network objects lend themselves to using the list +membership test syntax like this:: + + if address in network: + # do something + +Containment testing is done efficiently based on the network prefix:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> addr4 in ipaddress.ip_network('192.0.2.0/24') + True + >>> addr4 in ipaddress.ip_network('192.0.3.0/24') + False + + +Comparisons +=========== + +:mod:`ipaddress` provides some simple, hopefully intuitive ways to compare +objects, where it makes sense:: + + >>> ipaddress.ip_address('192.0.2.1') < ipaddress.ip_address('192.0.2.2') + True + +A :exc:`TypeError` exception is raised if you try to compare objects of +different versions or different types. + + +Using IP Addresses with other modules +===================================== + +Other modules that use IP addresses (such as :mod:`socket`) usually won't +accept objects from this module directly. Instead, they must be coerced to +an integer or string that the other module will accept:: + + >>> addr4 = ipaddress.ip_address('192.0.2.1') + >>> str(addr4) + '192.0.2.1' + >>> int(addr4) + 3221225985 + + +Exceptions raised by :mod:`ipaddress` +===================================== + +If you try to create an address/network/interface object with an invalid value +for either the address or netmask, :mod:`ipaddress` will raise an +:exc:`AddressValueError` or :exc:`NetmaskValueError` respectively. However, +this applies only when calling the class constructors directly. The factory +functions and other module level functions will just raise :exc:`ValueError`. + +Both of the module specific exceptions have :exc:`ValueError` as their +parent class, so if you're not concerned with the particular type of error, +you can still do the following:: + + try: + ipaddress.IPv4Address(address) + except ValueError: + print 'address/netmask is invalid: %s' % address diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst --- a/Doc/library/smtpd.rst +++ b/Doc/library/smtpd.rst @@ -20,17 +20,24 @@ Additionally the SMTPChannel may be extended to implement very specific interaction behaviour with SMTP clients. +The code supports :RFC:`5321`, plus the :rfc:`1870` SIZE extension. + + SMTPServer Objects ------------------ -.. class:: SMTPServer(localaddr, remoteaddr) +.. class:: SMTPServer(localaddr, remoteaddr, data_size_limit=33554432) Create a new :class:`SMTPServer` object, which binds to local address *localaddr*. It will treat *remoteaddr* as an upstream SMTP relayer. It inherits from :class:`asyncore.dispatcher`, and so will insert itself into :mod:`asyncore`'s event loop on instantiation. + *data_size_limit* specifies the maximum number of bytes that will be + accepted in a ``DATA`` command. A value of ``None`` or ``0`` means no + limit. + .. method:: process_message(peer, mailfrom, rcpttos, data) Raise :exc:`NotImplementedError` exception. Override this in subclasses to @@ -155,11 +162,15 @@ Command Action taken ======== =================================================================== HELO Accepts the greeting from the client and stores it in - :attr:`seen_greeting`. + :attr:`seen_greeting`. Sets server to base command mode. + EHLO Accepts the greeting from the client and stores it in + :attr:`seen_greeting`. Sets server to extended command mode. NOOP Takes no action. QUIT Closes the connection cleanly. MAIL Accepts the "MAIL FROM:" syntax and stores the supplied address as - :attr:`mailfrom`. + :attr:`mailfrom`. In extended command mode, accepts the + :rfc:`1870` SIZE attribute and responds appropriately based on the + value of ``data_size_limit``. RCPT Accepts the "RCPT TO:" syntax and stores the supplied addresses in the :attr:`rcpttos` list. RSET Resets the :attr:`mailfrom`, :attr:`rcpttos`, and @@ -167,4 +178,7 @@ DATA Sets the internal state to :attr:`DATA` and stores remaining lines from the client in :attr:`received_data` until the terminator "\r\n.\r\n" is received. + HELP Returns minimal information on command syntax + VRFY Returns code 252 (the server doesn't know if the address is valid) + EXPN Reports that the command is not implemented. ======== =================================================================== diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -791,6 +791,8 @@ for x in self: if x.token_type == 'addr-spec': return x.addr_spec + else: + return '<>' class ObsRoute(TokenList): @@ -1829,6 +1831,14 @@ "expected angle-addr but found '{}'".format(value)) angle_addr.append(ValueTerminal('<', 'angle-addr-start')) value = value[1:] + # Although it is not legal per RFC5322, SMTP uses '<>' in certain + # circumstances. + if value[0] == '>': + angle_addr.append(ValueTerminal('>', 'angle-addr-end')) + angle_addr.defects.append(errors.InvalidHeaderDefect( + "null addr-spec in angle-addr")) + value = value[1:] + return angle_addr, value try: token, value = get_addr_spec(value) except errors.HeaderParseError: @@ -1838,7 +1848,7 @@ "obsolete route specification in angle-addr")) except errors.HeaderParseError: raise errors.HeaderParseError( - "expected addr-spec or but found '{}'".format(value)) + "expected addr-spec or obs-route but found '{}'".format(value)) angle_addr.append(token) token, value = get_addr_spec(value) angle_addr.append(token) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -949,8 +949,6 @@ def module_repr(cls, module): return "".format(module.__name__) - @set_package - @set_loader @module_for_loader def load_module(self, module): """Load a namespace module.""" diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -1,17 +1,5 @@ # Copyright 2007 Google Inc. # Licensed to PSF under a Contributor Agreement. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or -# implied. See the License for the specific language governing -# permissions and limitations under the License. """A fast, lightweight IPv4/IPv6 manipulation library in Python. @@ -36,34 +24,22 @@ """A Value Error related to the netmask.""" -def ip_address(address, version=None): +def ip_address(address): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP address. Either IPv4 or IPv6 addresses may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. Important for things - like ip_address(1), which could be IPv4, '192.0.2.1', or IPv6, - '2001:db8::1'. Returns: An IPv4Address or IPv6Address object. Raises: ValueError: if the *address* passed isn't either a v4 or a v6 - address, or if the version is not None, 4, or 6. + address """ - if version is not None: - if version == 4: - return IPv4Address(address) - elif version == 6: - return IPv6Address(address) - else: - raise ValueError() - try: return IPv4Address(address) except (AddressValueError, NetmaskValueError): @@ -78,35 +54,22 @@ address) -def ip_network(address, version=None, strict=True): +def ip_network(address, strict=True): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP network. Either IPv4 or IPv6 networks may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. Important for things - like ip_network(1), which could be IPv4, '192.0.2.1/32', or IPv6, - '2001:db8::1/128'. Returns: An IPv4Network or IPv6Network object. Raises: ValueError: if the string passed isn't either a v4 or a v6 - address. Or if the network has host bits set. Or if the version - is not None, 4, or 6. + address. Or if the network has host bits set. """ - if version is not None: - if version == 4: - return IPv4Network(address, strict) - elif version == 6: - return IPv6Network(address, strict) - else: - raise ValueError() - try: return IPv4Network(address, strict) except (AddressValueError, NetmaskValueError): @@ -121,24 +84,20 @@ address) -def ip_interface(address, version=None): +def ip_interface(address): """Take an IP string/int and return an object of the correct type. Args: address: A string or integer, the IP address. Either IPv4 or IPv6 addresses may be supplied; integers less than 2**32 will be considered to be IPv4 by default. - version: An integer, 4 or 6. If set, don't try to automatically - determine what the IP address type is. Important for things - like ip_interface(1), which could be IPv4, '192.0.2.1/32', or IPv6, - '2001:db8::1/128'. Returns: An IPv4Interface or IPv6Interface object. Raises: ValueError: if the string passed isn't either a v4 or a v6 - address. Or if the version is not None, 4, or 6. + address. Notes: The IPv?Interface classes describe an Address on a particular @@ -146,14 +105,6 @@ and Network classes. """ - if version is not None: - if version == 4: - return IPv4Interface(address) - elif version == 6: - return IPv6Interface(address) - else: - raise ValueError() - try: return IPv4Interface(address) except (AddressValueError, NetmaskValueError): @@ -281,7 +232,7 @@ If the first and last objects are not the same version. ValueError: If the last object is not greater than the first. - If the version is not 4 or 6. + If the version of the first address is not 4 or 6. """ if (not (isinstance(first, _BaseAddress) and @@ -318,7 +269,7 @@ if current == ip._ALL_ONES: break first_int = current + 1 - first = ip_address(first_int, version=first._version) + first = first.__class__(first_int) def _collapse_addresses_recursive(addresses): @@ -586,12 +537,12 @@ def __add__(self, other): if not isinstance(other, int): return NotImplemented - return ip_address(int(self) + other, version=self._version) + return self.__class__(int(self) + other) def __sub__(self, other): if not isinstance(other, int): return NotImplemented - return ip_address(int(self) - other, version=self._version) + return self.__class__(int(self) - other) def __repr__(self): return '%s(%r)' % (self.__class__.__name__, str(self)) @@ -612,13 +563,12 @@ class _BaseNetwork(_IPAddressBase): - """A generic IP object. + """A generic IP network object. This IP class contains the version independent methods which are used by networks. """ - def __init__(self, address): self._cache = {} @@ -642,14 +592,14 @@ bcast = int(self.broadcast_address) - 1 while cur <= bcast: cur += 1 - yield ip_address(cur - 1, version=self._version) + yield self._address_class(cur - 1) def __iter__(self): cur = int(self.network_address) bcast = int(self.broadcast_address) while cur <= bcast: cur += 1 - yield ip_address(cur - 1, version=self._version) + yield self._address_class(cur - 1) def __getitem__(self, n): network = int(self.network_address) @@ -657,12 +607,12 @@ if n >= 0: if network + n > broadcast: raise IndexError - return ip_address(network + n, version=self._version) + return self._address_class(network + n) else: n += 1 if broadcast + n < network: raise IndexError - return ip_address(broadcast + n, version=self._version) + return self._address_class(broadcast + n) def __lt__(self, other): if self._version != other._version: @@ -746,8 +696,8 @@ def broadcast_address(self): x = self._cache.get('broadcast_address') if x is None: - x = ip_address(int(self.network_address) | int(self.hostmask), - version=self._version) + x = self._address_class(int(self.network_address) | + int(self.hostmask)) self._cache['broadcast_address'] = x return x @@ -755,17 +705,11 @@ def hostmask(self): x = self._cache.get('hostmask') if x is None: - x = ip_address(int(self.netmask) ^ self._ALL_ONES, - version=self._version) + x = self._address_class(int(self.netmask) ^ self._ALL_ONES) self._cache['hostmask'] = x return x @property - def network(self): - return ip_network('%s/%d' % (str(self.network_address), - self.prefixlen)) - - @property def with_prefixlen(self): return '%s/%d' % (str(self.ip), self._prefixlen) @@ -787,6 +731,10 @@ raise NotImplementedError('BaseNet has no version') @property + def _address_class(self): + raise NotImplementedError('BaseNet has no associated address class') + + @property def prefixlen(self): return self._prefixlen @@ -840,9 +788,8 @@ raise StopIteration # Make sure we're comparing the network of other. - other = ip_network('%s/%s' % (str(other.network_address), - str(other.prefixlen)), - version=other._version) + other = other.__class__('%s/%s' % (str(other.network_address), + str(other.prefixlen))) s1, s2 = self.subnets() while s1 != other and s2 != other: @@ -973,9 +920,9 @@ 'prefix length diff %d is invalid for netblock %s' % ( new_prefixlen, str(self))) - first = ip_network('%s/%s' % (str(self.network_address), - str(self._prefixlen + prefixlen_diff)), - version=self._version) + first = self.__class__('%s/%s' % + (str(self.network_address), + str(self._prefixlen + prefixlen_diff))) yield first current = first @@ -983,17 +930,12 @@ broadcast = current.broadcast_address if broadcast == self.broadcast_address: return - new_addr = ip_address(int(broadcast) + 1, version=self._version) - current = ip_network('%s/%s' % (str(new_addr), str(new_prefixlen)), - version=self._version) + new_addr = self._address_class(int(broadcast) + 1) + current = self.__class__('%s/%s' % (str(new_addr), + str(new_prefixlen))) yield current - def masked(self): - """Return the network object with the host bits masked out.""" - return ip_network('%s/%d' % (self.network_address, self._prefixlen), - version=self._version) - def supernet(self, prefixlen_diff=1, new_prefix=None): """The supernet containing the current network. @@ -1030,11 +972,10 @@ 'current prefixlen is %d, cannot have a prefixlen_diff of %d' % (self.prefixlen, prefixlen_diff)) # TODO (pmoody): optimize this. - t = ip_network('%s/%d' % (str(self.network_address), - self.prefixlen - prefixlen_diff), - version=self._version, strict=False) - return ip_network('%s/%d' % (str(t.network_address), t.prefixlen), - version=t._version) + t = self.__class__('%s/%d' % (str(self.network_address), + self.prefixlen - prefixlen_diff), + strict=False) + return t.__class__('%s/%d' % (str(t.network_address), t.prefixlen)) class _BaseV4(object): @@ -1391,6 +1332,9 @@ .prefixlen: 27 """ + # Class to use when creating address objects + # TODO (ncoghlan): Investigate using IPv4Interface instead + _address_class = IPv4Address # the valid octets for host and netmasks. only useful for IPv4. _valid_mask_octets = set((255, 254, 252, 248, 240, 224, 192, 128, 0)) @@ -1952,7 +1896,7 @@ """ if isinstance(self, IPv6Network): - return int(self.network) == 1 and getattr( + return int(self) == 1 and getattr( self, '_prefixlen', 128) == 128 elif isinstance(self, IPv6Interface): return int(self.network.network_address) == 1 and getattr( @@ -2071,6 +2015,10 @@ """ + # Class to use when creating address objects + # TODO (ncoghlan): Investigate using IPv6Interface instead + _address_class = IPv6Address + def __init__(self, address, strict=True): """Instantiate a new IPv6 Network object. diff --git a/Lib/smtpd.py b/Lib/smtpd.py --- a/Lib/smtpd.py +++ b/Lib/smtpd.py @@ -1,5 +1,5 @@ #! /usr/bin/env python3 -"""An RFC 2821 smtp proxy. +"""An RFC 5321 smtp proxy. Usage: %(program)s [options] [localhost:localport [remotehost:remoteport]] @@ -20,6 +20,11 @@ Use `classname' as the concrete SMTP proxy class. Uses `PureProxy' by default. + --size limit + -s limit + Restrict the total size of the incoming message to "limit" number of + bytes via the RFC 1870 SIZE extension. Defaults to 33554432 bytes. + --debug -d Turn on debugging prints. @@ -35,10 +40,9 @@ and if remoteport is not given, then 25 is used. """ - # Overview: # -# This file implements the minimal SMTP protocol as defined in RFC 821. It +# This file implements the minimal SMTP protocol as defined in RFC 5321. It # has a hierarchy of classes which implement the backend functionality for the # smtpd. A number of classes are provided: # @@ -66,7 +70,7 @@ # # - support mailbox delivery # - alias files -# - ESMTP +# - Handle more ESMTP extensions # - handle error codes from the backend smtpd import sys @@ -77,12 +81,14 @@ import socket import asyncore import asynchat +import collections from warnings import warn +from email._header_value_parser import get_addr_spec, get_angle_addr __all__ = ["SMTPServer","DebuggingServer","PureProxy","MailmanProxy"] program = sys.argv[0] -__version__ = 'Python SMTP proxy version 0.2' +__version__ = 'Python SMTP proxy version 0.3' class Devnull: @@ -94,9 +100,9 @@ NEWLINE = '\n' EMPTYSTRING = '' COMMASPACE = ', ' +DATA_SIZE_DEFAULT = 33554432 - def usage(code, msg=''): print(__doc__ % globals(), file=sys.stderr) if msg: @@ -104,19 +110,23 @@ sys.exit(code) - class SMTPChannel(asynchat.async_chat): COMMAND = 0 DATA = 1 - data_size_limit = 33554432 command_size_limit = 512 + command_size_limits = collections.defaultdict(lambda x=command_size_limit: x) + command_size_limits.update({ + 'MAIL': command_size_limit + 26, + }) + max_command_size_limit = max(command_size_limits.values()) - def __init__(self, server, conn, addr): + def __init__(self, server, conn, addr, data_size_limit=DATA_SIZE_DEFAULT): asynchat.async_chat.__init__(self, conn) self.smtp_server = server self.conn = conn self.addr = addr + self.data_size_limit = data_size_limit self.received_lines = [] self.smtp_state = self.COMMAND self.seen_greeting = '' @@ -137,6 +147,7 @@ print('Peer:', repr(self.peer), file=DEBUGSTREAM) self.push('220 %s %s' % (self.fqdn, __version__)) self.set_terminator(b'\r\n') + self.extended_smtp = False # properties for backwards-compatibility @property @@ -268,7 +279,7 @@ def collect_incoming_data(self, data): limit = None if self.smtp_state == self.COMMAND: - limit = self.command_size_limit + limit = self.max_command_size_limit elif self.smtp_state == self.DATA: limit = self.data_size_limit if limit and self.num_bytes > limit: @@ -283,11 +294,7 @@ print('Data:', repr(line), file=DEBUGSTREAM) self.received_lines = [] if self.smtp_state == self.COMMAND: - if self.num_bytes > self.command_size_limit: - self.push('500 Error: line too long') - self.num_bytes = 0 - return - self.num_bytes = 0 + sz, self.num_bytes = self.num_bytes, 0 if not line: self.push('500 Error: bad syntax') return @@ -299,9 +306,14 @@ else: command = line[:i].upper() arg = line[i+1:].strip() + max_sz = (self.command_size_limits[command] + if self.extended_smtp else self.command_size_limit) + if sz > max_sz: + self.push('500 Error: line too long') + return method = getattr(self, 'smtp_' + command, None) if not method: - self.push('502 Error: command "%s" not implemented' % command) + self.push('500 Error: command "%s" not recognized' % command) return method(arg) return @@ -310,12 +322,12 @@ self.push('451 Internal confusion') self.num_bytes = 0 return - if self.num_bytes > self.data_size_limit: + if self.data_size_limit and self.num_bytes > self.data_size_limit: self.push('552 Error: Too much mail data') self.num_bytes = 0 return # Remove extraneous carriage returns and de-transparency according - # to RFC 821, Section 4.5.2. + # to RFC 5321, Section 4.5.2. data = [] for text in line.split('\r\n'): if text and text[0] == '.': @@ -333,7 +345,7 @@ self.num_bytes = 0 self.set_terminator(b'\r\n') if not status: - self.push('250 Ok') + self.push('250 OK') else: self.push(status) @@ -346,66 +358,188 @@ self.push('503 Duplicate HELO/EHLO') else: self.seen_greeting = arg + self.extended_smtp = False self.push('250 %s' % self.fqdn) + def smtp_EHLO(self, arg): + if not arg: + self.push('501 Syntax: EHLO hostname') + return + if self.seen_greeting: + self.push('503 Duplicate HELO/EHLO') + else: + self.seen_greeting = arg + self.extended_smtp = True + self.push('250-%s' % self.fqdn) + if self.data_size_limit: + self.push('250-SIZE %s' % self.data_size_limit) + self.push('250 HELP') + def smtp_NOOP(self, arg): if arg: self.push('501 Syntax: NOOP') else: - self.push('250 Ok') + self.push('250 OK') def smtp_QUIT(self, arg): # args is ignored self.push('221 Bye') self.close_when_done() - # factored - def __getaddr(self, keyword, arg): - address = None + def _strip_command_keyword(self, keyword, arg): keylen = len(keyword) if arg[:keylen].upper() == keyword: - address = arg[keylen:].strip() - if not address: - pass - elif address[0] == '<' and address[-1] == '>' and address != '<>': - # Addresses can be in the form but watch out - # for null address, e.g. <> - address = address[1:-1] - return address + return arg[keylen:].strip() + return '' + + def _getaddr(self, arg): + if not arg: + return '', '' + if arg.lstrip().startswith('<'): + address, rest = get_angle_addr(arg) + else: + address, rest = get_addr_spec(arg) + if not address: + return address, rest + return address.addr_spec, rest + + def _getparams(self, params): + # Return any parameters that appear to be syntactically valid according + # to RFC 1869, ignore all others. (Postel rule: accept what we can.) + params = [param.split('=', 1) for param in params.split() + if '=' in param] + return {k: v for k, v in params if k.isalnum()} + + def smtp_HELP(self, arg): + if arg: + extended = ' [SP ') + else: + self.push('501 Supported commands: EHLO HELO MAIL RCPT ' + 'DATA RSET NOOP QUIT VRFY') + else: + self.push('250 Supported commands: EHLO HELO MAIL RCPT DATA ' + 'RSET NOOP QUIT VRFY') + + def smtp_VRFY(self, arg): + if arg: + address, params = self._getaddr(arg) + if address: + self.push('252 Cannot VRFY user, but will accept message ' + 'and attempt delivery') + else: + self.push('502 Could not VRFY %s' % arg) + else: + self.push('501 Syntax: VRFY
') def smtp_MAIL(self, arg): if not self.seen_greeting: self.push('503 Error: send HELO first'); return - print('===> MAIL', arg, file=DEBUGSTREAM) - address = self.__getaddr('FROM:', arg) if arg else None + syntaxerr = '501 Syntax: MAIL FROM:
' + if self.extended_smtp: + syntaxerr += ' [SP ]' + if arg is None: + self.push(syntaxerr) + return + arg = self._strip_command_keyword('FROM:', arg) + address, params = self._getaddr(arg) if not address: - self.push('501 Syntax: MAIL FROM:
') + self.push(syntaxerr) + return + if not self.extended_smtp and params: + self.push(syntaxerr) + return + if not address: + self.push(syntaxerr) return if self.mailfrom: self.push('503 Error: nested MAIL command') return + params = self._getparams(params.upper()) + if params is None: + self.push(syntaxerr) + return + size = params.pop('SIZE', None) + if size: + if not size.isdigit(): + self.push(syntaxerr) + return + elif self.data_size_limit and int(size) > self.data_size_limit: + self.push('552 Error: message size exceeds fixed maximum message size') + return + if len(params.keys()) > 0: + self.push('555 MAIL FROM parameters not recognized or not implemented') + return self.mailfrom = address print('sender:', self.mailfrom, file=DEBUGSTREAM) - self.push('250 Ok') + self.push('250 OK') def smtp_RCPT(self, arg): if not self.seen_greeting: self.push('503 Error: send HELO first'); return - print('===> RCPT', arg, file=DEBUGSTREAM) if not self.mailfrom: self.push('503 Error: need MAIL command') return - address = self.__getaddr('TO:', arg) if arg else None + syntaxerr = '501 Syntax: RCPT TO:
' + if self.extended_smtp: + syntaxerr += ' [SP ]' + if arg is None: + self.push(syntaxerr) + return + arg = self._strip_command_keyword('TO:', arg) + address, params = self._getaddr(arg) + if not address: + self.push(syntaxerr) + return + if params: + if self.extended_smtp: + params = self._getparams(params.upper()) + if params is None: + self.push(syntaxerr) + return + else: + self.push(syntaxerr) + return + if not address: + self.push(syntaxerr) + return + if params and len(params.keys()) > 0: + self.push('555 RCPT TO parameters not recognized or not implemented') + return if not address: self.push('501 Syntax: RCPT TO:
') return self.rcpttos.append(address) print('recips:', self.rcpttos, file=DEBUGSTREAM) - self.push('250 Ok') + self.push('250 OK') def smtp_RSET(self, arg): if arg: @@ -416,13 +550,12 @@ self.rcpttos = [] self.received_data = '' self.smtp_state = self.COMMAND - self.push('250 Ok') + self.push('250 OK') def smtp_DATA(self, arg): if not self.seen_greeting: self.push('503 Error: send HELO first'); return - if not self.rcpttos: self.push('503 Error: need RCPT command') return @@ -433,15 +566,20 @@ self.set_terminator(b'\r\n.\r\n') self.push('354 End data with .') + # Commands that have not been implemented + def smtp_EXPN(self, arg): + self.push('502 EXPN not implemented') - + class SMTPServer(asyncore.dispatcher): # SMTPChannel class to use for managing client connections channel_class = SMTPChannel - def __init__(self, localaddr, remoteaddr): + def __init__(self, localaddr, remoteaddr, + data_size_limit=DATA_SIZE_DEFAULT): self._localaddr = localaddr self._remoteaddr = remoteaddr + self.data_size_limit = data_size_limit asyncore.dispatcher.__init__(self) try: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) @@ -459,7 +597,7 @@ def handle_accepted(self, conn, addr): print('Incoming connection from %s' % repr(addr), file=DEBUGSTREAM) - channel = self.channel_class(self, conn, addr) + channel = self.channel_class(self, conn, addr, self.data_size_limit) # API for "doing something useful with the message" def process_message(self, peer, mailfrom, rcpttos, data): @@ -487,7 +625,6 @@ raise NotImplementedError - class DebuggingServer(SMTPServer): # Do something with the gathered message def process_message(self, peer, mailfrom, rcpttos, data): @@ -503,7 +640,6 @@ print('------------ END MESSAGE ------------') - class PureProxy(SMTPServer): def process_message(self, peer, mailfrom, rcpttos, data): lines = data.split('\n') @@ -544,7 +680,6 @@ return refused - class MailmanProxy(PureProxy): def process_message(self, peer, mailfrom, rcpttos, data): from io import StringIO @@ -623,19 +758,18 @@ msg.Enqueue(mlist, torequest=1) - class Options: setuid = 1 classname = 'PureProxy' + size_limit = None - def parseargs(): global DEBUGSTREAM try: opts, args = getopt.getopt( - sys.argv[1:], 'nVhc:d', - ['class=', 'nosetuid', 'version', 'help', 'debug']) + sys.argv[1:], 'nVhc:s:d', + ['class=', 'nosetuid', 'version', 'help', 'size=', 'debug']) except getopt.error as e: usage(1, e) @@ -652,6 +786,13 @@ options.classname = arg elif opt in ('-d', '--debug'): DEBUGSTREAM = sys.stderr + elif opt in ('-s', '--size'): + try: + int_size = int(arg) + options.size_limit = int_size + except: + print('Invalid size: ' + arg, file=sys.stderr) + sys.exit(1) # parse the rest of the arguments if len(args) < 1: @@ -686,7 +827,6 @@ return options - if __name__ == '__main__': options = parseargs() # Become nobody @@ -699,7 +839,8 @@ import __main__ as mod class_ = getattr(mod, classname) proxy = class_((options.localhost, options.localport), - (options.remotehost, options.remoteport)) + (options.remotehost, options.remoteport), + options.size_limit) if options.setuid: try: import pwd diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py --- a/Lib/test/test_email/test__header_value_parser.py +++ b/Lib/test/test_email/test__header_value_parser.py @@ -1429,6 +1429,19 @@ self.assertIsNone(angle_addr.route) self.assertEqual(angle_addr.addr_spec, 'dinsdale at example.com') + def test_get_angle_addr_empty(self): + angle_addr = self._test_get_x(parser.get_angle_addr, + '<>', + '<>', + '<>', + [errors.InvalidHeaderDefect], + '') + self.assertEqual(angle_addr.token_type, 'angle-addr') + self.assertIsNone(angle_addr.local_part) + self.assertIsNone(angle_addr.domain) + self.assertIsNone(angle_addr.route) + self.assertEqual(angle_addr.addr_spec, '<>') + def test_get_angle_addr_with_cfws(self): angle_addr = self._test_get_x(parser.get_angle_addr, ' (foo) (bar)', @@ -2007,7 +2020,7 @@ self.assertEqual(group.mailboxes, group.all_mailboxes) - def test_get_troup_null_addr_spec(self): + def test_get_group_null_addr_spec(self): group = self._test_get_x(parser.get_group, 'foo: <>;', 'foo: <>;', diff --git a/Lib/test/test_ipaddress.py b/Lib/test/test_ipaddress.py --- a/Lib/test/test_ipaddress.py +++ b/Lib/test/test_ipaddress.py @@ -1,21 +1,7 @@ -#!/usr/bin/python3 -# # Copyright 2007 Google Inc. # Licensed to PSF under a Contributor Agreement. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -"""Unittest for ipaddressmodule.""" +"""Unittest for ipaddress module.""" import unittest @@ -404,7 +390,7 @@ self.assertRaises(ValueError, list, self.ipv4_interface.network.subnets(-1)) self.assertRaises(ValueError, list, - self.ipv4_network.network.subnets(-1)) + self.ipv4_network.subnets(-1)) self.assertRaises(ValueError, list, self.ipv6_interface.network.subnets(-1)) self.assertRaises(ValueError, list, @@ -780,12 +766,6 @@ self.assertEqual(self.ipv4_address.version, 4) self.assertEqual(self.ipv6_address.version, 6) - with self.assertRaises(ValueError): - ipaddress.ip_address('1', version=[]) - - with self.assertRaises(ValueError): - ipaddress.ip_address('1', version=5) - def testMaxPrefixLength(self): self.assertEqual(self.ipv4_interface.max_prefixlen, 32) self.assertEqual(self.ipv6_interface.max_prefixlen, 128) @@ -1052,12 +1032,7 @@ def testForceVersion(self): self.assertEqual(ipaddress.ip_network(1).version, 4) - self.assertEqual(ipaddress.ip_network(1, version=6).version, 6) - - with self.assertRaises(ValueError): - ipaddress.ip_network(1, version='l') - with self.assertRaises(ValueError): - ipaddress.ip_network(1, version=3) + self.assertEqual(ipaddress.IPv6Network(1).version, 6) def testWithStar(self): self.assertEqual(str(self.ipv4_interface.with_prefixlen), "1.2.3.4/24") @@ -1148,13 +1123,6 @@ sixtofouraddr.sixtofour) self.assertFalse(bad_addr.sixtofour) - def testIpInterfaceVersion(self): - with self.assertRaises(ValueError): - ipaddress.ip_interface(1, version=123) - - with self.assertRaises(ValueError): - ipaddress.ip_interface(1, version='') - if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -663,6 +663,7 @@ self.smtp_server = server self.conn = conn self.addr = addr + self.data_size_limit = None self.received_lines = [] self.smtp_state = self.COMMAND self.seen_greeting = '' @@ -682,6 +683,7 @@ return self.push('220 %s %s' % (self.fqdn, smtpd.__version__)) self.set_terminator(b'\r\n') + self.extended_smtp = False class TestSMTPServer(smtpd.SMTPServer): @@ -709,6 +711,7 @@ def __init__(self, addr, handler, poll_interval, sockmap): self._localaddr = addr self._remoteaddr = None + self.data_size_limit = None self.sockmap = sockmap asyncore.dispatcher.__init__(self, map=sockmap) try: diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py --- a/Lib/test/test_smtpd.py +++ b/Lib/test/test_smtpd.py @@ -1,4 +1,4 @@ -from unittest import TestCase +import unittest from test import support, mock_socket import socket import io @@ -26,7 +26,7 @@ raise DummyDispatcherBroken() -class SMTPDServerTest(TestCase): +class SMTPDServerTest(unittest.TestCase): def setUp(self): smtpd.socket = asyncore.socket = mock_socket @@ -39,7 +39,7 @@ channel.socket.queue_recv(line) channel.handle_read() - write_line(b'HELO test.example') + write_line(b'HELO example') write_line(b'MAIL From:eggs at example') write_line(b'RCPT To:spam at example') write_line(b'DATA') @@ -50,7 +50,7 @@ asyncore.socket = smtpd.socket = socket -class SMTPDChannelTest(TestCase): +class SMTPDChannelTest(unittest.TestCase): def setUp(self): smtpd.socket = asyncore.socket = mock_socket self.old_debugstream = smtpd.DEBUGSTREAM @@ -79,36 +79,94 @@ self.assertEqual(self.channel.socket.last, b'500 Error: bad syntax\r\n') - def test_EHLO_not_implemented(self): - self.write_line(b'EHLO test.example') + def test_EHLO(self): + self.write_line(b'EHLO example') + self.assertEqual(self.channel.socket.last, b'250 HELP\r\n') + + def test_EHLO_bad_syntax(self): + self.write_line(b'EHLO') self.assertEqual(self.channel.socket.last, - b'502 Error: command "EHLO" not implemented\r\n') + b'501 Syntax: EHLO hostname\r\n') + + def test_EHLO_duplicate(self): + self.write_line(b'EHLO example') + self.write_line(b'EHLO example') + self.assertEqual(self.channel.socket.last, + b'503 Duplicate HELO/EHLO\r\n') + + def test_EHLO_HELO_duplicate(self): + self.write_line(b'EHLO example') + self.write_line(b'HELO example') + self.assertEqual(self.channel.socket.last, + b'503 Duplicate HELO/EHLO\r\n') def test_HELO(self): name = smtpd.socket.getfqdn() - self.write_line(b'HELO test.example') + self.write_line(b'HELO example') self.assertEqual(self.channel.socket.last, '250 {}\r\n'.format(name).encode('ascii')) + def test_HELO_EHLO_duplicate(self): + self.write_line(b'HELO example') + self.write_line(b'EHLO example') + self.assertEqual(self.channel.socket.last, + b'503 Duplicate HELO/EHLO\r\n') + + def test_HELP(self): + self.write_line(b'HELP') + self.assertEqual(self.channel.socket.last, + b'250 Supported commands: EHLO HELO MAIL RCPT ' + \ + b'DATA RSET NOOP QUIT VRFY\r\n') + + def test_HELP_command(self): + self.write_line(b'HELP MAIL') + self.assertEqual(self.channel.socket.last, + b'250 Syntax: MAIL FROM:
\r\n') + + def test_HELP_command_unknown(self): + self.write_line(b'HELP SPAM') + self.assertEqual(self.channel.socket.last, + b'501 Supported commands: EHLO HELO MAIL RCPT ' + \ + b'DATA RSET NOOP QUIT VRFY\r\n') + def test_HELO_bad_syntax(self): self.write_line(b'HELO') self.assertEqual(self.channel.socket.last, b'501 Syntax: HELO hostname\r\n') def test_HELO_duplicate(self): - self.write_line(b'HELO test.example') - self.write_line(b'HELO test.example') + self.write_line(b'HELO example') + self.write_line(b'HELO example') self.assertEqual(self.channel.socket.last, b'503 Duplicate HELO/EHLO\r\n') + def test_HELO_parameter_rejected_when_extensions_not_enabled(self): + self.extended_smtp = False + self.write_line(b'HELO example') + self.write_line(b'MAIL from: SIZE=1234') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: MAIL FROM:
\r\n') + + def test_MAIL_allows_space_after_colon(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL from: ') + self.assertEqual(self.channel.socket.last, + b'250 OK\r\n') + + def test_extended_MAIL_allows_space_after_colon(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: size=20') + self.assertEqual(self.channel.socket.last, + b'250 OK\r\n') + def test_NOOP(self): self.write_line(b'NOOP') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') def test_HELO_NOOP(self): self.write_line(b'HELO example') self.write_line(b'NOOP') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') def test_NOOP_bad_syntax(self): self.write_line(b'NOOP hi') @@ -136,15 +194,29 @@ def test_command_too_long(self): self.write_line(b'HELO example') - self.write_line(b'MAIL from ' + + self.write_line(b'MAIL from: ' + b'a' * self.channel.command_size_limit + b'@example') self.assertEqual(self.channel.socket.last, b'500 Error: line too long\r\n') - def test_data_too_long(self): - # Small hack. Setting limit to 2K octets here will save us some time. - self.channel.data_size_limit = 2048 + def test_MAIL_command_limit_extended_with_SIZE(self): + self.write_line(b'EHLO example') + fill_len = self.channel.command_size_limit - len('MAIL from:<@example>') + self.write_line(b'MAIL from:<' + + b'a' * fill_len + + b'@example> SIZE=1234') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + self.write_line(b'MAIL from:<' + + b'a' * (fill_len + 26) + + b'@example> SIZE=1234') + self.assertEqual(self.channel.socket.last, + b'500 Error: line too long\r\n') + + def test_data_longer_than_default_data_size_limit(self): + # Hack the default so we don't have to generate so much data. + self.channel.data_size_limit = 1048 self.write_line(b'HELO example') self.write_line(b'MAIL From:eggs at example') self.write_line(b'RCPT To:spam at example') @@ -154,28 +226,93 @@ self.assertEqual(self.channel.socket.last, b'552 Error: Too much mail data\r\n') + def test_MAIL_size_parameter(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: SIZE=512') + self.assertEqual(self.channel.socket.last, + b'250 OK\r\n') + + def test_MAIL_invalid_size_parameter(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: SIZE=invalid') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: MAIL FROM:
[SP ]\r\n') + + def test_MAIL_RCPT_unknown_parameters(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: ham=green') + self.assertEqual(self.channel.socket.last, + b'555 MAIL FROM parameters not recognized or not implemented\r\n') + + self.write_line(b'MAIL FROM:') + self.write_line(b'RCPT TO: ham=green') + self.assertEqual(self.channel.socket.last, + b'555 RCPT TO parameters not recognized or not implemented\r\n') + + def test_MAIL_size_parameter_larger_than_default_data_size_limit(self): + self.channel.data_size_limit = 1048 + self.write_line(b'EHLO example') + self.write_line(b'MAIL FROM: SIZE=2096') + self.assertEqual(self.channel.socket.last, + b'552 Error: message size exceeds fixed maximum message size\r\n') + def test_need_MAIL(self): self.write_line(b'HELO example') self.write_line(b'RCPT to:spam at example') self.assertEqual(self.channel.socket.last, b'503 Error: need MAIL command\r\n') - def test_MAIL_syntax(self): + def test_MAIL_syntax_HELO(self): self.write_line(b'HELO example') self.write_line(b'MAIL from eggs at example') self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
\r\n') + b'501 Syntax: MAIL FROM:
\r\n') - def test_MAIL_missing_from(self): + def test_MAIL_syntax_EHLO(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from eggs at example') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: MAIL FROM:
[SP ]\r\n') + + def test_MAIL_missing_address(self): self.write_line(b'HELO example') self.write_line(b'MAIL from:') self.assertEqual(self.channel.socket.last, - b'501 Syntax: MAIL FROM:
\r\n') + b'501 Syntax: MAIL FROM:
\r\n') def test_MAIL_chevrons(self): self.write_line(b'HELO example') self.write_line(b'MAIL from:') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + def test_MAIL_empty_chevrons(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from:<>') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + def test_MAIL_quoted_localpart(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: <"Fred Blogs"@example.com>') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') + + def test_MAIL_quoted_localpart_no_angles(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: "Fred Blogs"@example.com') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') + + def test_MAIL_quoted_localpart_with_size(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: <"Fred Blogs"@example.com> SIZE=1000') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') + + def test_MAIL_quoted_localpart_with_size_no_angles(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL from: "Fred Blogs"@example.com SIZE=1000') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.channel.mailfrom, '"Fred Blogs"@example.com') def test_nested_MAIL(self): self.write_line(b'HELO example') @@ -184,6 +321,22 @@ self.assertEqual(self.channel.socket.last, b'503 Error: nested MAIL command\r\n') + def test_VRFY(self): + self.write_line(b'VRFY eggs at example') + self.assertEqual(self.channel.socket.last, + b'252 Cannot VRFY user, but will accept message and attempt ' + \ + b'delivery\r\n') + + def test_VRFY_syntax(self): + self.write_line(b'VRFY') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: VRFY
\r\n') + + def test_EXPN_not_implemented(self): + self.write_line(b'EXPN') + self.assertEqual(self.channel.socket.last, + b'502 EXPN not implemented\r\n') + def test_no_HELO_MAIL(self): self.write_line(b'MAIL from:') self.assertEqual(self.channel.socket.last, @@ -196,13 +349,26 @@ self.assertEqual(self.channel.socket.last, b'503 Error: need RCPT command\r\n') - def test_RCPT_syntax(self): + def test_RCPT_syntax_HELO(self): self.write_line(b'HELO example') - self.write_line(b'MAIL From:eggs at example') + self.write_line(b'MAIL From: eggs at example') self.write_line(b'RCPT to eggs at example') self.assertEqual(self.channel.socket.last, b'501 Syntax: RCPT TO:
\r\n') + def test_RCPT_syntax_EHLO(self): + self.write_line(b'EHLO example') + self.write_line(b'MAIL From: eggs at example') + self.write_line(b'RCPT to eggs at example') + self.assertEqual(self.channel.socket.last, + b'501 Syntax: RCPT TO:
[SP ]\r\n') + + def test_RCPT_lowercase_to_OK(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL From: eggs at example') + self.write_line(b'RCPT to: ') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + def test_no_HELO_RCPT(self): self.write_line(b'RCPT to eggs at example') self.assertEqual(self.channel.socket.last, @@ -211,15 +377,15 @@ def test_data_dialog(self): self.write_line(b'HELO example') self.write_line(b'MAIL From:eggs at example') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.write_line(b'RCPT To:spam at example') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.write_line(b'DATA') self.assertEqual(self.channel.socket.last, b'354 End data with .\r\n') self.write_line(b'data\r\nmore\r\n.') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.assertEqual(self.server.messages, [('peer', 'eggs at example', ['spam at example'], 'data\nmore')]) @@ -267,7 +433,7 @@ self.write_line(b'MAIL From:eggs at example') self.write_line(b'RCPT To:spam at example') self.write_line(b'RSET') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') self.write_line(b'MAIL From:foo at example') self.write_line(b'RCPT To:eggs at example') self.write_line(b'DATA') @@ -278,12 +444,18 @@ def test_HELO_RSET(self): self.write_line(b'HELO example') self.write_line(b'RSET') - self.assertEqual(self.channel.socket.last, b'250 Ok\r\n') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') def test_RSET_syntax(self): self.write_line(b'RSET hi') self.assertEqual(self.channel.socket.last, b'501 Syntax: RSET\r\n') + def test_unknown_command(self): + self.write_line(b'UNKNOWN_CMD') + self.assertEqual(self.channel.socket.last, + b'500 Error: command "UNKNOWN_CMD" not ' + \ + b'recognized\r\n') + def test_attribute_deprecations(self): with support.check_warnings(('', DeprecationWarning)): spam = self.channel._SMTPChannel__server @@ -330,8 +502,54 @@ with support.check_warnings(('', DeprecationWarning)): self.channel._SMTPChannel__addr = 'spam' -def test_main(): - support.run_unittest(SMTPDServerTest, SMTPDChannelTest) + +class SMTPDChannelWithDataSizeLimitTest(unittest.TestCase): + + def setUp(self): + smtpd.socket = asyncore.socket = mock_socket + self.debug = smtpd.DEBUGSTREAM = io.StringIO() + self.server = DummyServer('a', 'b') + conn, addr = self.server.accept() + # Set DATA size limit to 32 bytes for easy testing + self.channel = smtpd.SMTPChannel(self.server, conn, addr, 32) + + def tearDown(self): + asyncore.close_all() + asyncore.socket = smtpd.socket = socket + + def write_line(self, line): + self.channel.socket.queue_recv(line) + self.channel.handle_read() + + def test_data_limit_dialog(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL From:eggs at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.write_line(b'RCPT To:spam at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + self.write_line(b'DATA') + self.assertEqual(self.channel.socket.last, + b'354 End data with .\r\n') + self.write_line(b'data\r\nmore\r\n.') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.assertEqual(self.server.messages, + [('peer', 'eggs at example', ['spam at example'], 'data\nmore')]) + + def test_data_limit_dialog_too_much_data(self): + self.write_line(b'HELO example') + self.write_line(b'MAIL From:eggs at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + self.write_line(b'RCPT To:spam at example') + self.assertEqual(self.channel.socket.last, b'250 OK\r\n') + + self.write_line(b'DATA') + self.assertEqual(self.channel.socket.last, + b'354 End data with .\r\n') + self.write_line(b'This message is longer than 32 bytes\r\n.') + self.assertEqual(self.channel.socket.last, + b'552 Error: Too much mail data\r\n') + if __name__ == "__main__": - test_main() + unittest.main() diff --git a/Lib/test/test_smtplib.py b/Lib/test/test_smtplib.py --- a/Lib/test/test_smtplib.py +++ b/Lib/test/test_smtplib.py @@ -229,13 +229,13 @@ def testNOOP(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - expected = (250, b'Ok') + expected = (250, b'OK') self.assertEqual(smtp.noop(), expected) smtp.quit() def testRSET(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - expected = (250, b'Ok') + expected = (250, b'OK') self.assertEqual(smtp.rset(), expected) smtp.quit() @@ -246,10 +246,18 @@ self.assertEqual(smtp.ehlo(), expected) smtp.quit() + def testNotImplemented(self): + # EXPN isn't implemented in DebuggingServer + smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) + expected = (502, b'EXPN not implemented') + smtp.putcmd('EXPN') + self.assertEqual(smtp.getreply(), expected) + smtp.quit() + def testVRFY(self): - # VRFY isn't implemented in DebuggingServer smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - expected = (502, b'Error: command "VRFY" not implemented') + expected = (252, b'Cannot VRFY user, but will accept message ' + \ + b'and attempt delivery') self.assertEqual(smtp.vrfy('nobody at nowhere.com'), expected) self.assertEqual(smtp.verify('nobody at nowhere.com'), expected) smtp.quit() @@ -265,7 +273,8 @@ def testHELP(self): smtp = smtplib.SMTP(HOST, self.port, local_hostname='localhost', timeout=3) - self.assertEqual(smtp.help(), b'Error: command "HELP" not implemented') + self.assertEqual(smtp.help(), b'Supported commands: EHLO HELO MAIL ' + \ + b'RCPT DATA RSET NOOP QUIT VRFY') smtp.quit() def testSend(self): diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -112,6 +112,7 @@ Matias Bordese Jurjen Bos Peter Bosch +Dan Boswell Eric Bouck Thierry Bousch Sebastian Boving @@ -494,6 +495,7 @@ Jack Jansen Bill Janssen Thomas Jarosch +Juhana Jauhiainen Zbigniew J?drzejewski-Szmek Julien Jehannet Drew Jenkins @@ -1039,6 +1041,7 @@ Richard Townsend David Townshend Laurence Tratt +Alberto Trevino Matthias Troffaes John Tromp Jason Trowbridge diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -46,6 +46,9 @@ Library ------- +- Issue #8739: Updated smtpd to support RFC 5321, and added support for the + RFC 1870 SIZE extension. + - Issue #665194: Added a localtime function to email.utils to provide an aware local datetime for use in setting Date headers. diff --git a/Python/importlib.h b/Python/importlib.h index bdb3644db479da7d7f6b949be135c6b1c3d1fc33..cd8fbebb9ff7c4a671320695174aa8953db75845 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 22:57:22 2012 From: python-checkins at python.org (terry.reedy) Date: Sat, 26 May 2012 22:57:22 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0ODc2?= =?utf8?q?=3A_Use_user-selected_font_for_highlight_configuration=2E?= Message-ID: http://hg.python.org/cpython/rev/e443cce4f183 changeset: 77169:e443cce4f183 branch: 3.2 parent: 77149:ea25ce432343 user: Terry Jan Reedy date: Sat May 26 16:31:00 2012 -0400 summary: Issue #14876: Use user-selected font for highlight configuration. Patch by Roger Serwy. files: Lib/idlelib/configDialog.py | 6 ++++-- Misc/NEWS | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -187,7 +187,7 @@ text=' Highlighting Theme ') #frameCustom self.textHighlightSample=Text(frameCustom,relief=SOLID,borderwidth=1, - font=('courier',12,''),cursor='hand2',width=21,height=10, + font=('courier',12,''),cursor='hand2',width=21,height=11, takefocus=FALSE,highlightthickness=0,wrap=NONE) text=self.textHighlightSample text.bind('',lambda e: 'break') @@ -821,8 +821,10 @@ fontWeight=tkFont.BOLD else: fontWeight=tkFont.NORMAL - self.editFont.config(size=self.fontSize.get(), + size=self.fontSize.get() + self.editFont.config(size=size, weight=fontWeight,family=fontName) + self.textHighlightSample.configure(font=(fontName, size, fontWeight)) def SetHighlightTarget(self): if self.highlightTarget.get()=='Cursor': #bg not possible diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,8 @@ Library ------- +- Issue #14876: Use user-selected font for highlight configuration. + - Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. Have ascii characters in help. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 22:57:23 2012 From: python-checkins at python.org (terry.reedy) Date: Sat, 26 May 2012 22:57:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_Issue_=2314876=3A_Use_user-selected_font_for_highlight?= =?utf8?q?_configuration=2E?= Message-ID: http://hg.python.org/cpython/rev/a9e9045d5546 changeset: 77170:a9e9045d5546 parent: 77168:1b5bdef96af0 parent: 77169:e443cce4f183 user: Terry Jan Reedy date: Sat May 26 16:50:30 2012 -0400 summary: Merge Issue #14876: Use user-selected font for highlight configuration. Patch by Roger Serwy. files: Lib/idlelib/configDialog.py | 6 ++++-- Misc/NEWS | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -187,7 +187,7 @@ text=' Highlighting Theme ') #frameCustom self.textHighlightSample=Text(frameCustom,relief=SOLID,borderwidth=1, - font=('courier',12,''),cursor='hand2',width=21,height=10, + font=('courier',12,''),cursor='hand2',width=21,height=11, takefocus=FALSE,highlightthickness=0,wrap=NONE) text=self.textHighlightSample text.bind('',lambda e: 'break') @@ -821,8 +821,10 @@ fontWeight=tkFont.BOLD else: fontWeight=tkFont.NORMAL - self.editFont.config(size=self.fontSize.get(), + size=self.fontSize.get() + self.editFont.config(size=size, weight=fontWeight,family=fontName) + self.textHighlightSample.configure(font=(fontName, size, fontWeight)) def SetHighlightTarget(self): if self.highlightTarget.get()=='Cursor': #bg not possible diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,8 @@ new features through new policies. Note that Policy.must_be_7bit is renamed to cte_type. +- Issue #14876: Use user-selected font for highlight configuration. + - Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. Have ascii characters in help. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 22:57:23 2012 From: python-checkins at python.org (terry.reedy) Date: Sat, 26 May 2012 22:57:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0ODc2?= =?utf8?q?=3A_Use_user-selected_font_for_highlight_configuration=2E?= Message-ID: http://hg.python.org/cpython/rev/dc9ddad40bba changeset: 77171:dc9ddad40bba branch: 2.7 parent: 77137:1f5d2642929a user: Terry Jan Reedy date: Sat May 26 16:55:43 2012 -0400 summary: Issue #14876: Use user-selected font for highlight configuration. Patch by Roger Serwy. files: Lib/idlelib/configDialog.py | 6 ++++-- Misc/NEWS | 3 +++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/Lib/idlelib/configDialog.py b/Lib/idlelib/configDialog.py --- a/Lib/idlelib/configDialog.py +++ b/Lib/idlelib/configDialog.py @@ -183,7 +183,7 @@ text=' Highlighting Theme ') #frameCustom self.textHighlightSample=Text(frameCustom,relief=SOLID,borderwidth=1, - font=('courier',12,''),cursor='hand2',width=21,height=10, + font=('courier',12,''),cursor='hand2',width=21,height=11, takefocus=FALSE,highlightthickness=0,wrap=NONE) text=self.textHighlightSample text.bind('',lambda e: 'break') @@ -832,8 +832,10 @@ fontWeight=tkFont.BOLD else: fontWeight=tkFont.NORMAL - self.editFont.config(size=self.fontSize.get(), + size=self.fontSize.get() + self.editFont.config(size=size, weight=fontWeight,family=fontName) + self.textHighlightSample.configure(font=(fontName, size, fontWeight)) def SetHighlightTarget(self): if self.highlightTarget.get()=='Cursor': #bg not possible diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue #14876: Use user-selected font for highlight configuration. + Patch by Roger Serwy. + - Issue #14036: Add an additional check to validate that port in urlparse does not go in illegal range and returns None. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sat May 26 23:17:12 2012 From: python-checkins at python.org (richard.oudkerk) Date: Sat, 26 May 2012 23:17:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_multiprocessing=27s_sh?= =?utf8?q?ared_memory_use_memoryview_instead_of_raw_pointer?= Message-ID: http://hg.python.org/cpython/rev/0685f51e9891 changeset: 77172:0685f51e9891 parent: 77170:a9e9045d5546 user: Richard Oudkerk date: Sat May 26 22:09:59 2012 +0100 summary: Make multiprocessing's shared memory use memoryview instead of raw pointer files: Lib/multiprocessing/heap.py | 11 +---- Lib/multiprocessing/sharedctypes.py | 3 +- Modules/_multiprocessing/multiprocessing.c | 17 ---------- 3 files changed, 5 insertions(+), 26 deletions(-) diff --git a/Lib/multiprocessing/heap.py b/Lib/multiprocessing/heap.py --- a/Lib/multiprocessing/heap.py +++ b/Lib/multiprocessing/heap.py @@ -205,7 +205,7 @@ self._lock.release() # -# Class representing a chunk of an mmap -- can be inherited +# Class representing a chunk of an mmap -- can be inherited by child process # class BufferWrapper(object): @@ -218,11 +218,6 @@ self._state = (block, size) Finalize(self, BufferWrapper._heap.free, args=(block,)) - def get_address(self): + def create_memoryview(self): (arena, start, stop), size = self._state - address, length = _multiprocessing.address_of_buffer(arena.buffer) - assert size <= length - return address + start - - def get_size(self): - return self._state[1] + return memoryview(arena.buffer)[start:start+size] diff --git a/Lib/multiprocessing/sharedctypes.py b/Lib/multiprocessing/sharedctypes.py --- a/Lib/multiprocessing/sharedctypes.py +++ b/Lib/multiprocessing/sharedctypes.py @@ -132,7 +132,8 @@ if length is not None: type_ = type_ * length ForkingPickler.register(type_, reduce_ctype) - obj = type_.from_address(wrapper.get_address()) + buf = wrapper.create_memoryview() + obj = type_.from_buffer(buf) obj._wrapper = wrapper return obj diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c --- a/Modules/_multiprocessing/multiprocessing.c +++ b/Modules/_multiprocessing/multiprocessing.c @@ -51,20 +51,6 @@ return NULL; } - -static PyObject* -multiprocessing_address_of_buffer(PyObject *self, PyObject *obj) -{ - void *buffer; - Py_ssize_t buffer_len; - - if (PyObject_AsWriteBuffer(obj, &buffer, &buffer_len) < 0) - return NULL; - - return Py_BuildValue("Nn", - PyLong_FromVoidPtr(buffer), buffer_len); -} - #ifdef MS_WINDOWS static PyObject * multiprocessing_closesocket(PyObject *self, PyObject *args) @@ -137,9 +123,6 @@ */ static PyMethodDef module_methods[] = { - {"address_of_buffer", multiprocessing_address_of_buffer, METH_O, - "address_of_buffer(obj) -> int\n" - "Return address of obj assuming obj supports buffer inteface"}, #ifdef MS_WINDOWS {"closesocket", multiprocessing_closesocket, METH_VARARGS, ""}, {"recv", multiprocessing_recv, METH_VARARGS, ""}, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 02:47:13 2012 From: python-checkins at python.org (terry.reedy) Date: Sun, 27 May 2012 02:47:13 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEwMzY1?= =?utf8?q?=3A_File_open_dialog_now_works_instead_of_crashing?= Message-ID: http://hg.python.org/cpython/rev/a2877fbabf95 changeset: 77173:a2877fbabf95 branch: 3.2 parent: 77169:e443cce4f183 user: Terry Jan Reedy date: Sat May 26 20:23:45 2012 -0400 summary: Issue #10365: File open dialog now works instead of crashing even when parent window is closed. Patch by Roger Serwy. files: Lib/idlelib/IOBinding.py | 17 ++++++++++++----- Lib/idlelib/PyShell.py | 3 ++- Misc/NEWS | 3 +++ 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -156,7 +156,8 @@ self.filename_change_hook() def open(self, event=None, editFile=None): - if self.editwin.flist: + flist = self.editwin.flist + if flist: if not editFile: filename = self.askopenfile() else: @@ -167,16 +168,22 @@ # we open a new window. But we won't replace the # shell window (which has an interp(reter) attribute), which # gets set to "not modified" at every new prompt. + # Also, make sure the current window has not been closed, + # since it can be closed during the Open File dialog. try: interp = self.editwin.interp except AttributeError: interp = None - if not self.filename and self.get_saved() and not interp: - self.editwin.flist.open(filename, self.loadfile) + + if self.editwin and not self.filename and \ + self.get_saved() and not interp: + flist.open(filename, self.loadfile) else: - self.editwin.flist.open(filename) + flist.open(filename) else: - self.text.focus_set() + if self.text: + self.text.focus_set() + return "break" # # Code for use outside IDLE: diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1435,7 +1435,8 @@ if tkversionwarning: shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) - root.mainloop() + while flist.inversedict: # keep IDLE running while files are open. + root.mainloop() root.destroy() if __name__ == "__main__": diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #10365: File open dialog now works instead of crashing + even when parent window is closed. Patch by Roger Serwy. + - Issue #14876: Use user-selected font for highlight configuration. - Issue #14920: Fix the help(urllib.parse) failure on locale C on terminals. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 02:47:14 2012 From: python-checkins at python.org (terry.reedy) Date: Sun, 27 May 2012 02:47:14 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_with_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/21862628a013 changeset: 77174:21862628a013 parent: 77172:0685f51e9891 parent: 77173:a2877fbabf95 user: Terry Jan Reedy date: Sat May 26 20:29:25 2012 -0400 summary: Merge with 3.2 Issue #10365: File open dialog now works instead of crashing even when parent window is closed. Patch by Roger Serwy. files: Lib/idlelib/IOBinding.py | 17 ++++++++++++----- Lib/idlelib/PyShell.py | 3 ++- Misc/NEWS | 3 +++ 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -156,7 +156,8 @@ self.filename_change_hook() def open(self, event=None, editFile=None): - if self.editwin.flist: + flist = self.editwin.flist + if flist: if not editFile: filename = self.askopenfile() else: @@ -167,16 +168,22 @@ # we open a new window. But we won't replace the # shell window (which has an interp(reter) attribute), which # gets set to "not modified" at every new prompt. + # Also, make sure the current window has not been closed, + # since it can be closed during the Open File dialog. try: interp = self.editwin.interp except AttributeError: interp = None - if not self.filename and self.get_saved() and not interp: - self.editwin.flist.open(filename, self.loadfile) + + if self.editwin and not self.filename and \ + self.get_saved() and not interp: + flist.open(filename, self.loadfile) else: - self.editwin.flist.open(filename) + flist.open(filename) else: - self.text.focus_set() + if self.text: + self.text.focus_set() + return "break" # # Code for use outside IDLE: diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1451,7 +1451,8 @@ if tkversionwarning: shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) - root.mainloop() + while flist.inversedict: # keep IDLE running while files are open. + root.mainloop() root.destroy() if __name__ == "__main__": diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -46,6 +46,9 @@ Library ------- +- Issue #10365: File open dialog now works instead of crashing + even when parent window is closed. Patch by Roger Serwy. + - Issue #8739: Updated smtpd to support RFC 5321, and added support for the RFC 1870 SIZE extension. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 02:47:14 2012 From: python-checkins at python.org (terry.reedy) Date: Sun, 27 May 2012 02:47:14 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEwMzY1?= =?utf8?q?=3A_File_open_dialog_now_works_instead_of_crashing?= Message-ID: http://hg.python.org/cpython/rev/4334964993b9 changeset: 77175:4334964993b9 branch: 2.7 parent: 77171:dc9ddad40bba user: Terry Jan Reedy date: Sat May 26 20:33:32 2012 -0400 summary: Issue #10365: File open dialog now works instead of crashing even when parent window is closed. Patch by Roger Serwy. files: Lib/idlelib/IOBinding.py | 17 ++++++++++++----- Lib/idlelib/PyShell.py | 3 ++- Misc/NEWS | 3 +++ 3 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -196,7 +196,8 @@ self.filename_change_hook() def open(self, event=None, editFile=None): - if self.editwin.flist: + flist = self.editwin.flist + if flist: if not editFile: filename = self.askopenfile() else: @@ -207,16 +208,22 @@ # we open a new window. But we won't replace the # shell window (which has an interp(reter) attribute), which # gets set to "not modified" at every new prompt. + # Also, make sure the current window has not been closed, + # since it can be closed during the Open File dialog. try: interp = self.editwin.interp except AttributeError: interp = None - if not self.filename and self.get_saved() and not interp: - self.editwin.flist.open(filename, self.loadfile) + + if self.editwin and not self.filename and \ + self.get_saved() and not interp: + flist.open(filename, self.loadfile) else: - self.editwin.flist.open(filename) + flist.open(filename) else: - self.text.focus_set() + if self.text: + self.text.focus_set() + return "break" # # Code for use outside IDLE: diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -1458,7 +1458,8 @@ if tkversionwarning: shell.interp.runcommand(''.join(("print('", tkversionwarning, "')"))) - root.mainloop() + while flist.inversedict: # keep IDLE running while files are open. + root.mainloop() root.destroy() if __name__ == "__main__": diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue #10365: File open dialog now works instead of crashing + even when parent window is closed. Patch by Roger Serwy. + - Issue #14876: Use user-selected font for highlight configuration. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 02:47:15 2012 From: python-checkins at python.org (terry.reedy) Date: Sun, 27 May 2012 02:47:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_=2310365_Trim_t?= =?utf8?q?railing_whitespace?= Message-ID: http://hg.python.org/cpython/rev/7dae83057e83 changeset: 77176:7dae83057e83 branch: 3.2 parent: 77173:a2877fbabf95 user: Terry Jan Reedy date: Sat May 26 20:43:17 2012 -0400 summary: #10365 Trim trailing whitespace files: Lib/idlelib/IOBinding.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -183,7 +183,7 @@ else: if self.text: self.text.focus_set() - + return "break" # # Code for use outside IDLE: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 02:47:16 2012 From: python-checkins at python.org (terry.reedy) Date: Sun, 27 May 2012 02:47:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_with_3=2E2_=2310635_whitespace?= Message-ID: http://hg.python.org/cpython/rev/e8d89503f061 changeset: 77177:e8d89503f061 parent: 77174:21862628a013 parent: 77176:7dae83057e83 user: Terry Jan Reedy date: Sat May 26 20:44:45 2012 -0400 summary: Merge with 3.2 #10635 whitespace files: Lib/idlelib/IOBinding.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -183,7 +183,7 @@ else: if self.text: self.text.focus_set() - + return "break" # # Code for use outside IDLE: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 02:47:16 2012 From: python-checkins at python.org (terry.reedy) Date: Sun, 27 May 2012 02:47:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_=2310365_Trim_t?= =?utf8?q?railing_whitespace?= Message-ID: http://hg.python.org/cpython/rev/630baa300173 changeset: 77178:630baa300173 branch: 2.7 parent: 77175:4334964993b9 user: Terry Jan Reedy date: Sat May 26 20:45:35 2012 -0400 summary: #10365 Trim trailing whitespace files: Lib/idlelib/IOBinding.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/IOBinding.py b/Lib/idlelib/IOBinding.py --- a/Lib/idlelib/IOBinding.py +++ b/Lib/idlelib/IOBinding.py @@ -223,7 +223,7 @@ else: if self.text: self.text.focus_set() - + return "break" # # Code for use outside IDLE: -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Sun May 27 05:54:32 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Sun, 27 May 2012 05:54:32 +0200 Subject: [Python-checkins] Daily reference leaks (e8d89503f061): sum=462 Message-ID: results for e8d89503f061 on branch "default" -------------------------------------------- test_smtplib leaked [154, 154, 154] references, sum=462 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogcC1Gun', '-x'] From python-checkins at python.org Sun May 27 09:31:06 2012 From: python-checkins at python.org (georg.brandl) Date: Sun, 27 May 2012 09:31:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_markup=2E?= Message-ID: http://hg.python.org/cpython/rev/7d98c3666700 changeset: 77179:7d98c3666700 parent: 77177:e8d89503f061 user: Georg Brandl date: Sun May 27 09:31:10 2012 +0200 summary: Fix markup. files: Doc/library/email.policy.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -415,7 +415,7 @@ The following instances of :class:`EmailPolicy` provide defaults suitable for specific application domains. Note that in the future the behavior of these -instances (in particular the ``HTTP` instance) may be adjusted to conform even +instances (in particular the ``HTTP`` instance) may be adjusted to conform even more closely to the RFCs relevant to their domains. .. data:: default -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 10:17:22 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 27 May 2012 10:17:22 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Tweak_importlib=2E=5Fbootst?= =?utf8?q?rap_to_avoid_zero-argument_super_so_I_can_work_on_issue?= Message-ID: http://hg.python.org/cpython/rev/bcb3b81853cc changeset: 77180:bcb3b81853cc user: Nick Coghlan date: Sun May 27 17:49:58 2012 +1000 summary: Tweak importlib._bootstrap to avoid zero-argument super so I can work on issue #14857 without breaking imports files: Lib/importlib/_bootstrap.py | 4 +++- Python/importlib.h | Bin 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -771,7 +771,9 @@ @_check_name def load_module(self, fullname): """Load a module from a file.""" - return super().load_module(fullname) + # Issue #14857: Avoid the zero-argument form so the implementation + # of that form can be updated without breaking the frozen module + return super(FileLoader, self).load_module(fullname) @_check_name def get_filename(self, fullname): diff --git a/Python/importlib.h b/Python/importlib.h index cd8fbebb9ff7c4a671320695174aa8953db75845..224e28c029a7dc4b51e87b6f07a9f213f8d2bcf2 GIT binary patch [stripped] -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 10:17:23 2012 From: python-checkins at python.org (nick.coghlan) Date: Sun, 27 May 2012 10:17:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314857=3A_fix_regre?= =?utf8?q?ssion_in_references_to_PEP_3135_implicit_=5F=5Fclass=5F=5F?= Message-ID: http://hg.python.org/cpython/rev/96ab78ef82a7 changeset: 77181:96ab78ef82a7 user: Nick Coghlan date: Sun May 27 18:17:07 2012 +1000 summary: Close #14857: fix regression in references to PEP 3135 implicit __class__ closure variable. Reopens issue #12370, but also updates unittest.mock to workaround that issue files: Lib/test/test_super.py | 24 ++++++++++++++++++++++++ Lib/unittest/mock.py | 9 ++++++--- Misc/NEWS | 3 +++ Objects/typeobject.c | 2 +- Python/compile.c | 2 +- Python/import.c | 3 ++- Python/symtable.c | 19 ++++++++----------- 7 files changed, 45 insertions(+), 17 deletions(-) diff --git a/Lib/test/test_super.py b/Lib/test/test_super.py --- a/Lib/test/test_super.py +++ b/Lib/test/test_super.py @@ -81,6 +81,7 @@ self.assertEqual(E().f(), 'AE') + @unittest.expectedFailure def test___class___set(self): # See issue #12370 class X(A): @@ -91,6 +92,29 @@ self.assertEqual(x.f(), 'A') self.assertEqual(x.__class__, 413) + def test___class___instancemethod(self): + # See issue #14857 + class X: + def f(self): + return __class__ + self.assertIs(X().f(), X) + + def test___class___classmethod(self): + # See issue #14857 + class X: + @classmethod + def f(cls): + return __class__ + self.assertIs(X.f(), X) + + def test___class___staticmethod(self): + # See issue #14857 + class X: + @staticmethod + def f(): + return __class__ + self.assertIs(X.f(), X) + def test_main(): support.run_unittest(TestSuper) diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -39,6 +39,9 @@ FILTER_DIR = True +# Workaround for issue #12370 +# Without this, the __class__ properties wouldn't be set correctly +_safe_super = super def _is_instance_mock(obj): # can't use isinstance on Mock objects because they override __class__ @@ -397,7 +400,7 @@ if kwargs: self.configure_mock(**kwargs) - super(NonCallableMock, self).__init__( + _safe_super(NonCallableMock, self).__init__( spec, wraps, name, spec_set, parent, _spec_state ) @@ -820,7 +823,7 @@ _spec_state=None, _new_name='', _new_parent=None, **kwargs): self.__dict__['_mock_return_value'] = return_value - super(CallableMixin, self).__init__( + _safe_super(CallableMixin, self).__init__( spec, wraps, name, spec_set, parent, _spec_state, _new_name, _new_parent, **kwargs ) @@ -1690,7 +1693,7 @@ class MagicMixin(object): def __init__(self, *args, **kw): - super(MagicMixin, self).__init__(*args, **kw) + _safe_super(MagicMixin, self).__init__(*args, **kw) self._mock_set_magics() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14857: fix regression in references to PEP 3135 implicit __class__ + closure variable (Reopens issue #12370) + - Issue #14712 (PEP 405): Virtual environments. Implemented by Vinay Sajip. - Issue #14660 (PEP 420): Namespace packages. Implemented by Eric Smith. diff --git a/Objects/typeobject.c b/Objects/typeobject.c --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -6436,7 +6436,7 @@ PyObject *name = PyTuple_GET_ITEM(co->co_freevars, i); assert(PyUnicode_Check(name)); if (!PyUnicode_CompareWithASCIIString(name, - "@__class__")) { + "__class__")) { Py_ssize_t index = co->co_nlocals + PyTuple_GET_SIZE(co->co_cellvars) + i; PyObject *cell = f->f_localsplus[index]; diff --git a/Python/compile.c b/Python/compile.c --- a/Python/compile.c +++ b/Python/compile.c @@ -1676,7 +1676,7 @@ return 0; } /* return the (empty) __class__ cell */ - str = PyUnicode_InternFromString("@__class__"); + str = PyUnicode_InternFromString("__class__"); if (str == NULL) { compiler_exit_scope(c); return 0; diff --git a/Python/import.c b/Python/import.c --- a/Python/import.c +++ b/Python/import.c @@ -106,6 +106,7 @@ Python 3.3a0 3200 (__qualname__ added) 3210 (added size modulo 2**32 to the pyc header) Python 3.3a1 3220 (changed PEP 380 implementation) + Python 3.3a4 3230 (revert changes to implicit __class__ closure) */ /* MAGIC must change whenever the bytecode emitted by the compiler may no @@ -118,7 +119,7 @@ #define STRIFY(name) QUOTE(name) #define MAJOR STRIFY(PY_MAJOR_VERSION) #define MINOR STRIFY(PY_MINOR_VERSION) -#define MAGIC (3220 | ((long)'\r'<<16) | ((long)'\n'<<24)) +#define MAGIC (3230 | ((long)'\r'<<16) | ((long)'\n'<<24)) #define TAG "cpython-" MAJOR MINOR; #define CACHEDIR "__pycache__" /* Current magic word and string tag as globals. */ diff --git a/Python/symtable.c b/Python/symtable.c --- a/Python/symtable.c +++ b/Python/symtable.c @@ -221,17 +221,10 @@ struct symtable * PySymtable_Build(mod_ty mod, const char *filename, PyFutureFeatures *future) { - struct symtable *st; + struct symtable *st = symtable_new(); asdl_seq *seq; int i; - if (__class__ == NULL) { - __class__ = PyUnicode_InternFromString("@__class__"); - if (__class__ == NULL) - return NULL; - } - - st = symtable_new(); if (st == NULL) return st; st->st_filename = filename; @@ -747,6 +740,8 @@ } else { /* Special-case __class__ */ + if (!GET_IDENTIFIER(__class__)) + goto error; assert(PySet_Contains(local, __class__) == 1); if (PySet_Add(newbound, __class__) < 0) goto error; @@ -784,7 +779,7 @@ NULL)) goto error; else if (ste->ste_type == ClassBlock && !analyze_cells(scopes, newfree, - "@__class__")) + "__class__")) goto error; /* Records the results of the analysis in the symbol table entry */ if (!update_symbols(ste->ste_symbols, scopes, bound, newfree, @@ -1111,7 +1106,8 @@ if (!symtable_enter_block(st, s->v.ClassDef.name, ClassBlock, (void *)s, s->lineno, s->col_offset)) return 0; - if (!symtable_add_def(st, __class__, DEF_LOCAL) || + if (!GET_IDENTIFIER(__class__) || + !symtable_add_def(st, __class__, DEF_LOCAL) || !GET_IDENTIFIER(__locals__) || !symtable_add_def(st, __locals__, DEF_PARAM)) { symtable_exit_block(st, s); @@ -1376,7 +1372,8 @@ if (e->v.Name.ctx == Load && st->st_cur->ste_type == FunctionBlock && !PyUnicode_CompareWithASCIIString(e->v.Name.id, "super")) { - if (!symtable_add_def(st, __class__, USE)) + if (!GET_IDENTIFIER(__class__) || + !symtable_add_def(st, __class__, USE)) return 0; } break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 18:30:15 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 27 May 2012 18:30:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fixed_=5Fsys=5Fhome_computa?= =?utf8?q?tion_and_added_diagnostics_for_Windows_buildbot_failures=2E?= Message-ID: http://hg.python.org/cpython/rev/a45c86f5ba96 changeset: 77182:a45c86f5ba96 user: Vinay Sajip date: Sun May 27 17:30:09 2012 +0100 summary: Fixed _sys_home computation and added diagnostics for Windows buildbot failures. files: Lib/distutils/sysconfig.py | 2 ++ Lib/sysconfig.py | 3 ++- Lib/test/test_venv.py | 5 ++++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Lib/distutils/sysconfig.py b/Lib/distutils/sysconfig.py --- a/Lib/distutils/sysconfig.py +++ b/Lib/distutils/sysconfig.py @@ -50,6 +50,8 @@ if _sys_home and os.name == 'nt' and \ _sys_home.lower().endswith(('pcbuild', 'pcbuild\\amd64')): _sys_home = os.path.dirname(_sys_home) + if _sys_home.endswith('pcbuild'): # must be amd64 + _sys_home = os.path.dirname(_sys_home) def _python_build(): if _sys_home: return _is_python_source_dir(_sys_home) diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -107,7 +107,8 @@ if _sys_home and os.name == 'nt' and \ _sys_home.lower().endswith(('pcbuild', 'pcbuild\\amd64')): _sys_home = os.path.dirname(_sys_home) - + if _sys_home.endswith('pcbuild'): # must be amd64 + _sys_home = os.path.dirname(_sys_home) def is_python_build(check_home=False): if check_home and _sys_home: return _is_python_source_dir(_sys_home) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -76,7 +76,10 @@ data = self.get_text_file_contents(self.bindir, self.ps3name) self.assertTrue(data.startswith('#!%s%s' % (self.env_dir, os.sep))) fn = self.get_env_file(self.bindir, self.exe) - self.assertTrue(os.path.exists(fn), 'File %r exists' % fn) + if not os.path.exists(fn): # diagnostics for Windows buildbot failures + print('Contents of %r:' % self.bindir) + print(' %r' % os.listdir(self.bindir)) + self.assertTrue(os.path.exists(fn), 'File %r should exist.' % fn) def test_overwrite_existing(self): """ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 18:40:07 2012 From: python-checkins at python.org (r.david.murray) Date: Sun, 27 May 2012 18:40:07 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbjogQWRkICdfX2FsbF9fJyB0byBf?= =?utf8?q?encoded=5Fwords_and_mark_QByteMap_as_private=2E?= Message-ID: http://hg.python.org/cpython/rev/c1eab1ef9c0b changeset: 77183:c1eab1ef9c0b user: R David Murray date: Sun May 27 12:39:54 2012 -0400 summary: Add '__all__' to _encoded_words and mark QByteMap as private. files: Lib/email/_encoded_words.py | 14 ++++++++++++-- 1 files changed, 12 insertions(+), 2 deletions(-) diff --git a/Lib/email/_encoded_words.py b/Lib/email/_encoded_words.py --- a/Lib/email/_encoded_words.py +++ b/Lib/email/_encoded_words.py @@ -46,6 +46,16 @@ from string import ascii_letters, digits from email import errors +__all__ = ['decode_q', + 'encode_q', + 'decode_b', + 'encode_b', + 'len_q', + 'len_b', + 'decode', + 'encode', + ] + # # Quoted Printable # @@ -60,7 +70,7 @@ # dict mapping bytes to their encoded form -class QByteMap(dict): +class _QByteMap(dict): safe = b'-!*+/' + ascii_letters.encode('ascii') + digits.encode('ascii') @@ -71,7 +81,7 @@ self[key] = "={:02X}".format(key) return self[key] -_q_byte_map = QByteMap() +_q_byte_map = _QByteMap() # In headers spaces are mapped to '_'. _q_byte_map[ord(' ')] = '_' -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 19:39:27 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 27 May 2012 19:39:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Refined_venv_tests=2E?= Message-ID: http://hg.python.org/cpython/rev/f9f19dfe15af changeset: 77184:f9f19dfe15af user: Vinay Sajip date: Sun May 27 18:39:22 2012 +0100 summary: Refined venv tests. files: Lib/test/test_venv.py | 35 +++++++++++++++++++++++------- 1 files changed, 26 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -52,19 +52,19 @@ class BasicTest(BaseTest): """Test venv module functionality.""" + def isdir(self, *args): + fn = self.get_env_file(*args) + self.assertTrue(os.path.isdir(fn)) + def test_defaults(self): """ Test the create function with default arguments. """ - def isdir(*args): - fn = self.get_env_file(*args) - self.assertTrue(os.path.isdir(fn)) - shutil.rmtree(self.env_dir) self.run_with_capture(venv.create, self.env_dir) - isdir(self.bindir) - isdir(self.include) - isdir(*self.lib) + self.isdir(self.bindir) + self.isdir(self.include) + self.isdir(*self.lib) data = self.get_text_file_contents('pyvenv.cfg') if sys.platform == 'darwin' and ('__PYTHONV_LAUNCHER__' in os.environ): @@ -77,8 +77,9 @@ self.assertTrue(data.startswith('#!%s%s' % (self.env_dir, os.sep))) fn = self.get_env_file(self.bindir, self.exe) if not os.path.exists(fn): # diagnostics for Windows buildbot failures - print('Contents of %r:' % self.bindir) - print(' %r' % os.listdir(self.bindir)) + bd = self.get_env_file(self.bindir) + print('Contents of %r:' % bd) + print(' %r' % os.listdir(bd)) self.assertTrue(os.path.exists(fn), 'File %r should exist.' % fn) def test_overwrite_existing(self): @@ -89,6 +90,22 @@ builder = venv.EnvBuilder(clear=True) builder.create(self.env_dir) + def test_upgrade(self): + """ + Test upgrading an existing environment directory. + """ + builder = venv.EnvBuilder(upgrade=True) + self.run_with_capture(builder.create, self.env_dir) + self.isdir(self.bindir) + self.isdir(self.include) + self.isdir(*self.lib) + fn = self.get_env_file(self.bindir, self.exe) + if not os.path.exists(fn): # diagnostics for Windows buildbot failures + bd = self.get_env_file(self.bindir) + print('Contents of %r:' % bd) + print(' %r' % os.listdir(bd)) + self.assertTrue(os.path.exists(fn), 'File %r should exist.' % fn) + def test_isolation(self): """ Test isolation from system site-packages -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 20:05:55 2012 From: python-checkins at python.org (vinay.sajip) Date: Sun, 27 May 2012 20:05:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Changed_executable_name_com?= =?utf8?q?putation_in_test=5Fvenv_to_allow_for_debug_executables=2E?= Message-ID: http://hg.python.org/cpython/rev/44b9d1bf5348 changeset: 77185:44b9d1bf5348 user: Vinay Sajip date: Sun May 27 19:05:36 2012 +0100 summary: Changed executable name computation in test_venv to allow for debug executables. files: Lib/test/test_venv.py | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -24,13 +24,12 @@ self.ps3name = 'pysetup3-script.py' self.lib = ('Lib',) self.include = 'Include' - self.exe = 'python.exe' else: self.bindir = 'bin' self.ps3name = 'pysetup3' self.lib = ('lib', 'python%s' % sys.version[:3]) self.include = 'include' - self.exe = 'python' + self.exe = os.path.split(sys.executable)[-1] def tearDown(self): shutil.rmtree(self.env_dir) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 21:04:03 2012 From: python-checkins at python.org (r.david.murray) Date: Sun, 27 May 2012 21:04:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_headerregistry_fully_p?= =?utf8?q?art_of_the_provisional_api=2E?= Message-ID: http://hg.python.org/cpython/rev/dacba47892e2 changeset: 77186:dacba47892e2 user: R David Murray date: Sun May 27 15:03:38 2012 -0400 summary: Make headerregistry fully part of the provisional api. When I made the checkin of the provisional email policy, I knew that Address and Group needed to be made accessible from somewhere. The more I looked at it, though, the more it became clear that since this is a provisional API anyway, there's no good reason to hide headerregistry as a private API. It was designed to ultimately be part of the public API, and so it should be part of the provisional API. This patch fully documents the headerregistry API, and deletes the abbreviated version of those docs I had added to the provisional policy docs. files: Doc/library/email.headerregistry.rst | 379 ++++++++++ Doc/library/email.policy.rst | 186 +---- Lib/email/_headerregistry.py | 0 Lib/email/policy.py | 6 +- Lib/test/test_email/test__headerregistry.py | 57 +- Lib/test/test_email/test_pickleable.py | 2 +- Lib/test/test_email/test_policy.py | 12 +- 7 files changed, 429 insertions(+), 213 deletions(-) diff --git a/Doc/library/email.headerregistry.rst b/Doc/library/email.headerregistry.rst new file mode 100644 --- /dev/null +++ b/Doc/library/email.headerregistry.rst @@ -0,0 +1,379 @@ +:mod:`email.headerregistry`: Custom Header Objects +-------------------------------------------------- + +.. module:: email.headerregistry + :synopsis: Automatic Parsing of headers based on the field name + +.. note:: + + The headerregistry module has been included in the standard library on a + :term:`provisional basis `. Backwards incompatible + changes (up to and including removal of the module) may occur if deemed + necessary by the core developers. + +.. versionadded:: 3.3 + as a :term:`provisional module ` + +Headers are represented by customized subclasses of :class:`str`. The +particular class used to represent a given header is determined by the +:attr:`~email.policy.EmailPolicy.header_factory` of the :mod:`~email.policy` in +effect when the headers are created. This section documents the particular +``header_factory`` implemented by the email package for handling :RFC:`5322` +compliant email messages, which not only provides customized header objects for +various header types, but also provides an extension mechanism for applications +to add their own custom header types. + +When using any of the policy objects derived from +:data:`~email.policy.EmailPolicy`, all headers are produced by +:class:`.HeaderRegistry` and have :class:`.BaseHeader` as their last base +class. Each header class has an additional base class that is determined by +the type of the header. For example, many headers have the class +:class:`.UnstructuredHeader` as their other base class. The specialized second +class for a header is determined by the name of the header, using a lookup +table stored in the :class:`.HeaderRegistry`. All of this is managed +transparently for the typical application program, but interfaces are provided +for modifying the default behavior for use by more complex applications. + +The sections below first document the header base classes and their attributes, +followed by the API for modifying the behavior of :class:`.HeaderRegistry`, and +finally the support classes used to represent the data parsed from structured +headers. + + +.. class:: BaseHeader(name, value) + + *name* and *value* are passed to ``BaseHeader`` from the + :attr:`~email.policy.EmailPolicy.header_factory` call. The string value of + any header object is the *value* fully decoded to unicode. + + This base class defines the following read-only properties: + + + .. attribute:: name + + The name of the header (the portion of the field before the ':'). This + is exactly the value passed in the :attr:`~EmailPolicy.header_factory` + call for *name*; that is, case is preserved. + + + .. attribute:: defects + + A tuple of :exc:`~email.errors.HeaderDefect` instances reporting any + RFC compliance problems found during parsing. The email package tries to + be complete about detecting compliance issues. See the :mod:`errors` + module for a discussion of the types of defects that may be reported. + + + .. attribute:: max_count + + The maximum number of headers of this type that can have the same + ``name``. A value of ``None`` means unlimited. The ``BaseHeader`` value + for this attribute is ``None``; it is expected that specialized header + classes will override this value as needed. + + ``BaseHeader`` also provides the following method, which is called by the + email library code and should not in general be called by application + programs: + + .. method:: fold(*, policy) + + Return a string containing :attr:`~email.policy.Policy.linesep` + characters as required to correctly fold the header according + to *policy*. A :attr:`~email.policy.Policy.cte_type` of + ``8bit`` will be treated as if it were ``7bit``, since strings + may not contain binary data. + + + ``BaseHeader`` by itself cannot be used to create a header object. It + defines a protocol that each specialized header cooperates with in order to + produce the header object. Specifically, ``BaseHeader`` requires that + the specialized class provide a :func:`classmethod` named ``parse``. This + method is called as follows:: + + parse(string, kwds) + + ``kwds`` is a dictionary containing one pre-initialized key, ``defects``. + ``defects`` is an empty list. The parse method should append any detected + defects to this list. On return, the ``kwds`` dictionary *must* contain + values for at least the keys ``decoded`` and ``defects``. ``decoded`` + should be the string value for the header (that is, the header value fully + decoded to unicode). The parse method should assume that *string* may + contain transport encoded parts, but should correctly handle all valid + unicode characters as well so that it can parse un-encoded header values. + + ``BaseHeader``'s ``__new__`` then creates the header instance, and calls its + ``init`` method. The specialized class only needs to provide an ``init`` + method if it wishes to set additional attributes beyond those provided by + ``BaseHeader`` itself. Such an ``init`` method should look like this:: + + def init(self, *args, **kw): + self._myattr = kw.pop('myattr') + super().init(*args, **kw) + + That is, anything extra that the specialized class puts in to the ``kwds`` + dictionary should be removed and handled, and the remaining contents of + ``kw`` (and ``args``) passed to the ``BaseHeader`` ``init`` method. + + +.. class:: UnstructuredHeader + + An "unstructured" header is the default type of header in :rfc:`5322`. + Any header that does not have a specified syntax is treated as + unstructured. The classic example of an unstructured header is the + :mailheader:`Subject` header. + + In :rfc:`5322`, an unstructured header is a run of arbitrary text in the + ASCII character set. :rfc:`2047`, however, has an :rfc:`5322` compatible + mechanism for encoding non-ASCII text as ASCII characters within a header + value. When a *value* containing encoded words is passed to the + constructor, the ``UnstructuredHeader`` parser converts such encoded words + back in to the original unicode, following the :rfc:`2047` rules for + unstructured text. The parser uses heuristics to attempt to decode certain + non-compliant encoded words. Defects are registered in such cases, as well + as defects for issues such as invalid characters within the encoded words or + the non-encoded text. + + This header type provides no additional attributes. + + +.. class:: DateHeader + + :rfc:`5322` specifies a very specific format for dates within email headers. + The ``DateHeader`` parser recognizes that date format, as well as + recognizing a number of variant forms that are sometimes found "in the + wild". + + This header type provides the following additional attributes: + + .. attribute:: datetime + + If the header value can be recognized as a valid date of one form or + another, this attribute will contain a :class:`~datetime.datetime` + instance representing that date. If the timezone of the input date is + specified as ``-0000`` (indicating it is in UTC but contains no + information about the source timezone), then :attr:`.datetime` will be a + naive :class:`~datetime.datetime`. If a specific timezone offset is + found (including `+0000`), then :attr:`.datetime` will contain an aware + ``datetime`` that uses :class:`datetime.timezone` to record the timezone + offset. + + The ``decoded`` value of the header is determined by formatting the + ``datetime`` according to the :rfc:`5322` rules; that is, it is set to:: + + email.utils.format_datetime(self.datetime) + + When creating a ``DateHeader``, *value* may be + :class:`~datetime.datetime` instance. This means, for example, that + the following code is valid and does what one would expect:: + + msg['Date'] = datetime(2011, 7, 15, 21) + + Because this is a naive ``datetime`` it will be interpreted as a UTC + timestamp, and the resulting value will have a timezone of ``-0000``. Much + more useful is to use the :func:`~email.utils.localtime` function from the + :mod:`~email.utils` module:: + + msg['Date'] = utils.localtime() + + This example sets the date header to the current time and date using + the current timezone offset. + + +.. class:: AddressHeader + + Address headers are one of the most complex structured header types. + The ``AddressHeader`` class provides a generic interface to any address + header. + + This header type provides the following additional attributes: + + + .. attribute:: groups + + A tuple of :class:`.Group` objects encoding the + addresses and groups found in the header value. Addresses that are + not part of a group are represented in this list as single-address + ``Groups`` whose :attr:`~.Group.display_name` is ``None``. + + + .. attribute:: addresses + + A tuple of :class:`.Address` objects encoding all + of the individual addresses from the header value. If the header value + contains any groups, the individual addresses from the group are included + in the list at the point where the group occurs in the value (that is, + the list of addresses is "flattened" into a one dimensional list). + + The ``decoded`` value of the header will have all encoded words decoded to + unicode. :class:`~encodings.idna` encoded domain names are also decoded to unicode. The + ``decoded`` value is set by :attr:`~str.join`\ ing the :class:`str` value of + the elements of the ``groups`` attribute with ``', '``. + + A list of :class:`.Address` and :class:`.Group` objects in any combination + may be used to set the value of an address header. ``Group`` objects whose + ``display_name`` is ``None`` will be interpreted as single addresses, which + allows an address list to be copied with groups intact by using the list + obtained ``groups`` attribute of the source header. + + +.. class:: SingleAddressHeader + + A subclass of :class:`.AddressHeader` that adds one + additional attribute: + + + .. attribute:: address + + The single address encoded by the header value. If the header value + actually contains more than one address (which would be a violation of + the RFC under the default :mod:`policy`), accessing this attribute will + result in a :exc:`ValueError`. + + +Each of the above classes also has a ``Unique`` variant (for example, +``UniqueUnstructuredHeader``). The only difference is that in the ``Unique`` +variant, :attr:`~.BaseHeader.max_count` is set to 1. + + +.. class:: HeaderRegistry(base_class=BaseHeader, \ + default_class=UnstructuredHeader, \ + use_default_map=True) + + This is the factory used by :class:`~email.policy.EmailPolicy` by default. + ``HeaderRegistry`` builds the class used to create a header instance + dynamically, using *base_class* and a specialized class retrieved from a + registry that it holds. When a given header name does not appear in the + registry, the class specified by *default_class* is used as the specialized + class. When *use_default_map* is ``True`` (the default), the standard + mapping of header names to classes is copied in to the registry during + initialization. *base_class* is always the last class in the generated + class's ``__bases__`` list. + + The default mappings are: + + :subject: UniqueUnstructuredHeader + :date: UniqueDateHeader + :resent-date: DateHeader + :orig-date: UniqueDateHeader + :sender: UniqueSingleAddressHeader + :resent-sender: SingleAddressHeader + :to: UniqueAddressHeader + :resent-to: AddressHeader + :cc: UniqueAddressHeader + :resent-cc: AddressHeader + :from: UniqueAddressHeader + :resent-from: AddressHeader + :reply-to: UniqueAddressHeader + + ``HeaderRegistry`` has the following methods: + + + .. method:: map_to_type(self, name, cls) + + *name* is the name of the header to be mapped. It will be converted to + lower case in the registry. *cls* is the specialized class to be used, + along with *base_class*, to create the class used to instantiate headers + that match *name*. + + + .. method:: __getitem__(name) + + Construct and return a class to handle creating a *name* header. + + + .. method:: __call__(name, value) + + Retrieves the specialized header associated with *name* from the + registry (using *default_class* if *name* does not appear in the + registry) and composes it with *base_class* to produce a class, + calls the constructed class's constructor, passing it the same + argument list, and finally returns the class instance created thereby. + + +The following classes are the classes used to represent data parsed from +structured headers and can, in general, be used by an application program to +construct structured values to assign to specific headers. + + +.. class:: Address(display_name='', username='', domain='', addr_spec=None) + + The class used to represent an email address. The general form of an + address is:: + + [display_name] + + or:: + + username at domain + + where each part must conform to specific syntax rules spelled out in + :rfc:`5322`. + + As a convenience *addr_spec* can be specified instead of *username* and + *domain*, in which case *username* and *domain* will be parsed from the + *addr_spec*. An *addr_spec* must be a properly RFC quoted string; if it is + not ``Address`` will raise an error. Unicode characters are allowed and + will be property encoded when serialized. However, per the RFCs, unicode is + *not* allowed in the username portion of the address. + + .. attribute:: display_name + + The display name portion of the address, if any, with all quoting + removed. If the address does not have a display name, this attribute + will be an empty string. + + .. attribute:: username + + The ``username`` portion of the address, with all quoting removed. + + .. attribute:: domain + + The ``domain`` portion of the address. + + .. attribute:: addr_spec + + The ``username at domain`` portion of the address, correctly quoted + for use as a bare address (the second form shown above). This + attribute is not mutable. + + .. method:: __str__() + + The ``str`` value of the object is the address quoted according to + :rfc:`5322` rules, but with no Content Transfer Encoding of any non-ASCII + characters. + + To support SMTP (:rfc:`5321`), ``Address`` handles one special case: if + ``username`` and ``domain`` are both the empty string (or ``None``), then + the string value of the ``Address`` is ``<>``. + + +.. class:: Group(display_name=None, addresses=None) + + The class used to represent an address group. The general form of an + address group is:: + + display_name: [address-list]; + + As a convenience for processing lists of addresses that consist of a mixture + of groups and single addresses, a ``Group`` may also be used to represent + single addresses that are not part of a group by setting *display_name* to + ``None`` and providing a list of the single address as *addresses*. + + .. attribute:: display_name + + The ``display_name`` of the group. If it is ``None`` and there is + exactly one ``Address`` in ``addresses``, then the ``Group`` represents a + single address that is not in a group. + + .. attribute:: addresses + + A possibly empty tuple of :class:`.Address` objects representing the + addresses in the group. + + .. method:: __str__() + + The ``str`` value of a ``Group`` is formatted according to :rfc:`5322`, + but with no Content Transfer Encoding of any non-ASCII characters. If + ``display_name`` is none and there is a single ``Address`` in the + ``addresses`` list, the ``str`` value will be the same as the ``str`` of + that single ``Address``. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -310,10 +310,10 @@ .. note:: - The remainder of the classes documented below are included in the standard - library on a :term:`provisional basis `. Backwards - incompatible changes (up to and including removal of the feature) may occur - if deemed necessary by the core developers. + The documentation below describes new policies that are included in the + standard library on a :term:`provisional basis `. + Backwards incompatible changes (up to and including removal of the feature) + may occur if deemed necessary by the core developers. .. class:: EmailPolicy(**kw) @@ -353,12 +353,12 @@ A callable that takes two arguments, ``name`` and ``value``, where ``name`` is a header field name and ``value`` is an unfolded header field - value, and returns a string-like object that represents that header. A - default ``header_factory`` is provided that understands some of the - :RFC:`5322` header field types. (Currently address fields and date - fields have special treatment, while all other fields are treated as - unstructured. This list will be completed before the extension is marked - stable.) + value, and returns a string subclass that represents that header. A + default ``header_factory`` (see :mod:`~email.headerregistry`) is provided + that understands some of the :RFC:`5322` header field types. (Currently + address fields and date fields have special treatment, while all other + fields are treated as unstructured. This list will be completed before + the extension is marked stable.) The class provides the following concrete implementations of the abstract methods of :class:`Policy`: @@ -465,167 +465,5 @@ created, using a unicode string, and the policy will take care of converting the unicode string into the correct RFC encoded form. -The custom header objects and their attributes are described below. All custom -header objects are string subclasses, and their string value is the fully -decoded value of the header field (the part of the field after the ``:``) - - -.. class:: BaseHeader - - This is the base class for all custom header objects. It provides the - following attributes: - - .. attribute:: name - - The header field name (the portion of the field before the ':'). - - .. attribute:: defects - - A possibly empty list of :class:`~email.errors.MessageDefect` objects - that record any RFC violations found while parsing the header field. - - .. method:: fold(*, policy) - - Return a string containing :attr:`~email.policy.Policy.linesep` - characters as required to correctly fold the header according - to *policy*. A :attr:`~email.policy.Policy.cte_type` of - ``8bit`` will be treated as if it were ``7bit``, since strings - may not contain binary data. - - -.. class:: UnstructuredHeader - - The class used for any header that does not have a more specific - type. (The :mailheader:`Subject` header is an example of an - unstructured header.) It does not have any additional attributes. - - -.. class:: DateHeader - - The value of this type of header is a single date and time value. The - primary example of this type of header is the :mailheader:`Date` header. - - .. attribute:: datetime - - A :class:`~datetime.datetime` encoding the date and time from the - header value. - - The ``datetime`` will be a naive ``datetime`` if the value either does - not have a specified timezone (which would be a violation of the RFC) or - if the timezone is specified as ``-0000``. This timezone value indicates - that the date and time is to be considered to be in UTC, but with no - indication of the local timezone in which it was generated. (This - contrasts to ``+0000``, which indicates a date and time that really is in - the UTC ``0000`` timezone.) - - If the header value contains a valid timezone that is not ``-0000``, the - ``datetime`` will be an aware ``datetime`` having a - :class:`~datetime.tzinfo` set to the :class:`~datetime.timezone` - indicated by the header value. - - A ``datetime`` may also be assigned to a :mailheader:`Date` type header. - The resulting string value will use a timezone of ``-0000`` if the - ``datetime`` is naive, and the appropriate UTC offset if the ``datetime`` is - aware. - - -.. class:: AddressHeader - - This class is used for all headers that can contain addresses, whether they - are supposed to be singleton addresses or a list. - - .. attribute:: addresses - - A list of :class:`.Address` objects listing all of the addresses that - could be parsed out of the field value. - - .. attribute:: groups - - A list of :class:`.Group` objects. Every address in :attr:`.addresses` - appears in one of the group objects in the tuple. Addresses that are not - syntactically part of a group are represented by ``Group`` objects whose - ``name`` is ``None``. - - In addition to addresses in string form, any combination of - :class:`.Address` and :class:`.Group` objects, singly or in a list, may be - assigned to an address header. - - -.. class:: Address(display_name='', username='', domain='', addr_spec=None): - - The class used to represent an email address. The general form of an - address is:: - - [display_name] - - or:: - - username at domain - - where each part must conform to specific syntax rules spelled out in - :rfc:`5322`. - - As a convenience *addr_spec* can be specified instead of *username* and - *domain*, in which case *username* and *domain* will be parsed from the - *addr_spec*. An *addr_spec* must be a properly RFC quoted string; if it is - not ``Address`` will raise an error. Unicode characters are allowed and - will be property encoded when serialized. However, per the RFCs, unicode is - *not* allowed in the username portion of the address. - - .. attribute:: display_name - - The display name portion of the address, if any, with all quoting - removed. If the address does not have a display name, this attribute - will be an empty string. - - .. attribute:: username - - The ``username`` portion of the address, with all quoting removed. - - .. attribute:: domain - - The ``domain`` portion of the address. - - .. attribute:: addr_spec - - The ``username at domain`` portion of the address, correctly quoted - for use as a bare address (the second form shown above). This - attribute is not mutable. - - .. method:: __str__() - - The ``str`` value of the object is the address quoted according to - :rfc:`5322` rules, but with no Content Transfer Encoding of any non-ASCII - characters. - - -.. class:: Group(display_name=None, addresses=None) - - The class used to represent an address group. The general form of an - address group is:: - - display_name: [address-list]; - - As a convenience for processing lists of addresses that consist of a mixture - of groups and single addresses, a ``Group`` may also be used to represent - single addresses that are not part of a group by setting *display_name* to - ``None`` and providing a list of the single address as *addresses*. - - .. attribute:: display_name - - The ``display_name`` of the group. If it is ``None`` and there is - exactly one ``Address`` in ``addresses``, then the ``Group`` represents a - single address that is not in a group. - - .. attribute:: addresses - - A possibly empty tuple of :class:`.Address` objects representing the - addresses in the group. - - .. method:: __str__() - - The ``str`` value of a ``Group`` is formatted according to :rfc:`5322`, - but with no Content Transfer Encoding of any non-ASCII characters. If - ``display_name`` is none and there is a single ``Address`` in the - ``addresses` list, the ``str`` value will be the same as the ``str`` of - that single ``Address``. +The custom header objects and their attributes are described in +:mod:`~email.headerregistry`. diff --git a/Lib/email/_headerregistry.py b/Lib/email/headerregistry.py rename from Lib/email/_headerregistry.py rename to Lib/email/headerregistry.py diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -4,7 +4,7 @@ from email._policybase import Policy, Compat32, compat32 from email.utils import _has_surrogates -from email._headerregistry import HeaderRegistry as _HeaderRegistry +from email.headerregistry import HeaderRegistry as HeaderRegistry __all__ = [ 'Compat32', @@ -60,13 +60,13 @@ """ refold_source = 'long' - header_factory = _HeaderRegistry() + header_factory = HeaderRegistry() def __init__(self, **kw): # Ensure that each new instance gets a unique header factory # (as opposed to clones, which share the factory). if 'header_factory' not in kw: - object.__setattr__(self, 'header_factory', _HeaderRegistry()) + object.__setattr__(self, 'header_factory', HeaderRegistry()) super().__init__(**kw) # The logic of the next three methods is chosen such that it is possible to diff --git a/Lib/test/test_email/test__headerregistry.py b/Lib/test/test_email/test_headerregistry.py rename from Lib/test/test_email/test__headerregistry.py rename to Lib/test/test_email/test_headerregistry.py --- a/Lib/test/test_email/test__headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -5,72 +5,71 @@ from email import policy from email.message import Message from test.test_email import TestEmailBase -from email import _headerregistry -# Address and Group are public but I'm not sure where to put them yet. -from email._headerregistry import Address, Group +from email import headerregistry +from email.headerregistry import Address, Group class TestHeaderRegistry(TestEmailBase): def test_arbitrary_name_unstructured(self): - factory = _headerregistry.HeaderRegistry() + factory = headerregistry.HeaderRegistry() h = factory('foobar', 'test') - self.assertIsInstance(h, _headerregistry.BaseHeader) - self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) def test_name_case_ignored(self): - factory = _headerregistry.HeaderRegistry() + factory = headerregistry.HeaderRegistry() # Whitebox check that test is valid self.assertNotIn('Subject', factory.registry) h = factory('Subject', 'test') - self.assertIsInstance(h, _headerregistry.BaseHeader) - self.assertIsInstance(h, _headerregistry.UniqueUnstructuredHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UniqueUnstructuredHeader) class FooBase: def __init__(self, *args, **kw): pass def test_override_default_base_class(self): - factory = _headerregistry.HeaderRegistry(base_class=self.FooBase) + factory = headerregistry.HeaderRegistry(base_class=self.FooBase) h = factory('foobar', 'test') self.assertIsInstance(h, self.FooBase) - self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) class FooDefault: - parse = _headerregistry.UnstructuredHeader.parse + parse = headerregistry.UnstructuredHeader.parse def test_override_default_class(self): - factory = _headerregistry.HeaderRegistry(default_class=self.FooDefault) + factory = headerregistry.HeaderRegistry(default_class=self.FooDefault) h = factory('foobar', 'test') - self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) self.assertIsInstance(h, self.FooDefault) def test_override_default_class_only_overrides_default(self): - factory = _headerregistry.HeaderRegistry(default_class=self.FooDefault) + factory = headerregistry.HeaderRegistry(default_class=self.FooDefault) h = factory('subject', 'test') - self.assertIsInstance(h, _headerregistry.BaseHeader) - self.assertIsInstance(h, _headerregistry.UniqueUnstructuredHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UniqueUnstructuredHeader) def test_dont_use_default_map(self): - factory = _headerregistry.HeaderRegistry(use_default_map=False) + factory = headerregistry.HeaderRegistry(use_default_map=False) h = factory('subject', 'test') - self.assertIsInstance(h, _headerregistry.BaseHeader) - self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) def test_map_to_type(self): - factory = _headerregistry.HeaderRegistry() + factory = headerregistry.HeaderRegistry() h1 = factory('foobar', 'test') - factory.map_to_type('foobar', _headerregistry.UniqueUnstructuredHeader) + factory.map_to_type('foobar', headerregistry.UniqueUnstructuredHeader) h2 = factory('foobar', 'test') - self.assertIsInstance(h1, _headerregistry.BaseHeader) - self.assertIsInstance(h1, _headerregistry.UnstructuredHeader) - self.assertIsInstance(h2, _headerregistry.BaseHeader) - self.assertIsInstance(h2, _headerregistry.UniqueUnstructuredHeader) + self.assertIsInstance(h1, headerregistry.BaseHeader) + self.assertIsInstance(h1, headerregistry.UnstructuredHeader) + self.assertIsInstance(h2, headerregistry.BaseHeader) + self.assertIsInstance(h2, headerregistry.UniqueUnstructuredHeader) class TestHeaderBase(TestEmailBase): - factory = _headerregistry.HeaderRegistry() + factory = headerregistry.HeaderRegistry() def make_header(self, name, value): return self.factory(name, value) @@ -149,13 +148,13 @@ def test_date_header_properties(self): h = self.make_header('date', self.datestring) - self.assertIsInstance(h, _headerregistry.UniqueDateHeader) + self.assertIsInstance(h, headerregistry.UniqueDateHeader) self.assertEqual(h.max_count, 1) self.assertEqual(h.defects, ()) def test_resent_date_header_properties(self): h = self.make_header('resent-date', self.datestring) - self.assertIsInstance(h, _headerregistry.DateHeader) + self.assertIsInstance(h, headerregistry.DateHeader) self.assertEqual(h.max_count, None) self.assertEqual(h.defects, ()) diff --git a/Lib/test/test_email/test_pickleable.py b/Lib/test/test_email/test_pickleable.py --- a/Lib/test/test_email/test_pickleable.py +++ b/Lib/test/test_email/test_pickleable.py @@ -4,7 +4,7 @@ import pickle from email import policy from email import message_from_string -from email._headerregistry import HeaderRegistry +from email.headerregistry import HeaderRegistry from test.test_email import TestEmailBase class TestPickleCopyHeader(TestEmailBase): diff --git a/Lib/test/test_email/test_policy.py b/Lib/test/test_email/test_policy.py --- a/Lib/test/test_email/test_policy.py +++ b/Lib/test/test_email/test_policy.py @@ -5,7 +5,7 @@ import email.policy import email.parser import email.generator -from email import _headerregistry +from email import headerregistry def make_defaults(base_defaults, differences): defaults = base_defaults.copy() @@ -185,11 +185,11 @@ def test_default_header_factory(self): h = email.policy.default.header_factory('Test', 'test') self.assertEqual(h.name, 'Test') - self.assertIsInstance(h, _headerregistry.UnstructuredHeader) - self.assertIsInstance(h, _headerregistry.BaseHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) + self.assertIsInstance(h, headerregistry.BaseHeader) class Foo: - parse = _headerregistry.UnstructuredHeader.parse + parse = headerregistry.UnstructuredHeader.parse def test_each_Policy_gets_unique_factory(self): policy1 = email.policy.EmailPolicy() @@ -197,10 +197,10 @@ policy1.header_factory.map_to_type('foo', self.Foo) h = policy1.header_factory('foo', 'test') self.assertIsInstance(h, self.Foo) - self.assertNotIsInstance(h, _headerregistry.UnstructuredHeader) + self.assertNotIsInstance(h, headerregistry.UnstructuredHeader) h = policy2.header_factory('foo', 'test') self.assertNotIsInstance(h, self.Foo) - self.assertIsInstance(h, _headerregistry.UnstructuredHeader) + self.assertIsInstance(h, headerregistry.UnstructuredHeader) def test_clone_copies_factory(self): policy1 = email.policy.EmailPolicy() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 23:10:46 2012 From: python-checkins at python.org (r.david.murray) Date: Sun, 27 May 2012 23:10:46 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2311785=3A_fix_the_=3Amod?= =?utf8?q?=3A_references_in_email_package_submodule_titles=2E?= Message-ID: http://hg.python.org/cpython/rev/64e82c4f4e10 changeset: 77187:64e82c4f4e10 user: R David Murray date: Sun May 27 17:10:36 2012 -0400 summary: #11785: fix the :mod: references in email package submodule titles. Also adds the TOC entry for headerregistry. files: Doc/library/email.charset.rst | 3 ++- Doc/library/email.encoders.rst | 3 ++- Doc/library/email.errors.rst | 3 ++- Doc/library/email.generator.rst | 3 ++- Doc/library/email.header.rst | 3 ++- Doc/library/email.headerregistry.rst | 4 ++++ Doc/library/email.iterators.rst | 3 ++- Doc/library/email.message.rst | 3 ++- Doc/library/email.mime.rst | 3 ++- Doc/library/email.parser.rst | 3 ++- Doc/library/email.policy.rst | 6 +++++- Doc/library/email.rst | 1 + Doc/library/email.util.rst | 3 ++- 13 files changed, 30 insertions(+), 11 deletions(-) diff --git a/Doc/library/email.charset.rst b/Doc/library/email.charset.rst --- a/Doc/library/email.charset.rst +++ b/Doc/library/email.charset.rst @@ -1,5 +1,5 @@ -:mod:`email`: Representing character sets ------------------------------------------ +:mod:`email.charset`: Representing character sets +------------------------------------------------- .. module:: email.charset :synopsis: Character Sets diff --git a/Doc/library/email.encoders.rst b/Doc/library/email.encoders.rst --- a/Doc/library/email.encoders.rst +++ b/Doc/library/email.encoders.rst @@ -1,5 +1,5 @@ -:mod:`email`: Encoders ----------------------- +:mod:`email.encoders`: Encoders +------------------------------- .. module:: email.encoders :synopsis: Encoders for email message payloads. diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -1,5 +1,5 @@ -:mod:`email`: Exception and Defect classes ------------------------------------------- +:mod:`email.errors`: Exception and Defect classes +------------------------------------------------- .. module:: email.errors :synopsis: The exception classes used by the email package. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -1,5 +1,5 @@ -:mod:`email`: Generating MIME documents ---------------------------------------- +:mod:`email.generator`: Generating MIME documents +------------------------------------------------- .. module:: email.generator :synopsis: Generate flat text email messages from a message structure. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -1,5 +1,5 @@ -:mod:`email`: Internationalized headers ---------------------------------------- +:mod:`email.header`: Internationalized headers +---------------------------------------------- .. module:: email.header :synopsis: Representing non-ASCII headers diff --git a/Doc/library/email.headerregistry.rst b/Doc/library/email.headerregistry.rst --- a/Doc/library/email.headerregistry.rst +++ b/Doc/library/email.headerregistry.rst @@ -4,6 +4,10 @@ .. module:: email.headerregistry :synopsis: Automatic Parsing of headers based on the field name +.. moduleauthor:: R. David Murray +.. sectionauthor:: R. David Murray + + .. note:: The headerregistry module has been included in the standard library on a diff --git a/Doc/library/email.iterators.rst b/Doc/library/email.iterators.rst --- a/Doc/library/email.iterators.rst +++ b/Doc/library/email.iterators.rst @@ -1,5 +1,5 @@ -:mod:`email`: Iterators ------------------------ +:mod:`email.iterators`: Iterators +--------------------------------- .. module:: email.iterators :synopsis: Iterate over a message object tree. diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -1,5 +1,5 @@ -:mod:`email`: Representing an email message -------------------------------------------- +:mod:`email.message`: Representing an email message +--------------------------------------------------- .. module:: email.message :synopsis: The base class representing email messages. diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -1,5 +1,5 @@ -:mod:`email`: Creating email and MIME objects from scratch ----------------------------------------------------------- +:mod:`email.mime`: Creating email and MIME objects from scratch +--------------------------------------------------------------- .. module:: email.mime :synopsis: Build MIME messages. diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -1,5 +1,5 @@ -:mod:`email`: Parsing email messages ------------------------------------- +:mod:`email.parser`: Parsing email messages +------------------------------------------- .. module:: email.parser :synopsis: Parse flat text email messages to produce a message object structure. diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -1,9 +1,12 @@ -:mod:`email`: Policy Objects ----------------------------- +:mod:`email.policy`: Policy Objects +----------------------------------- .. module:: email.policy :synopsis: Controlling the parsing and generating of messages +.. moduleauthor:: R. David Murray +.. sectionauthor:: R. David Murray + .. versionadded:: 3.3 diff --git a/Doc/library/email.rst b/Doc/library/email.rst --- a/Doc/library/email.rst +++ b/Doc/library/email.rst @@ -52,6 +52,7 @@ email.parser.rst email.generator.rst email.policy.rst + email.headerregistry.rst email.mime.rst email.header.rst email.charset.rst diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -1,5 +1,5 @@ -:mod:`email`: Miscellaneous utilities -------------------------------------- +:mod:`email.utils`: Miscellaneous utilities +------------------------------------------- .. module:: email.utils :synopsis: Miscellaneous email package utilities. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 23:18:45 2012 From: python-checkins at python.org (r.david.murray) Date: Sun, 27 May 2012 23:18:45 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzExNzg1OiBmaXgg?= =?utf8?q?the_=3Amod=3A_references_in_email_package_submodule_titles=2E?= Message-ID: http://hg.python.org/cpython/rev/df59aefdb1c8 changeset: 77188:df59aefdb1c8 branch: 2.7 parent: 77178:630baa300173 user: R David Murray date: Sun May 27 17:13:54 2012 -0400 summary: #11785: fix the :mod: references in email package submodule titles. files: Doc/library/email.charset.rst | 3 ++- Doc/library/email.encoders.rst | 3 ++- Doc/library/email.errors.rst | 3 ++- Doc/library/email.generator.rst | 3 ++- Doc/library/email.header.rst | 3 ++- Doc/library/email.iterators.rst | 3 ++- Doc/library/email.message.rst | 3 ++- Doc/library/email.mime.rst | 3 ++- Doc/library/email.parser.rst | 3 ++- Doc/library/email.util.rst | 3 ++- 10 files changed, 20 insertions(+), 10 deletions(-) diff --git a/Doc/library/email.charset.rst b/Doc/library/email.charset.rst --- a/Doc/library/email.charset.rst +++ b/Doc/library/email.charset.rst @@ -1,5 +1,5 @@ -:mod:`email`: Representing character sets ------------------------------------------ +:mod:`email.charset`: Representing character sets +------------------------------------------------- .. module:: email.charset :synopsis: Character Sets diff --git a/Doc/library/email.encoders.rst b/Doc/library/email.encoders.rst --- a/Doc/library/email.encoders.rst +++ b/Doc/library/email.encoders.rst @@ -1,5 +1,5 @@ -:mod:`email`: Encoders ----------------------- +:mod:`email.encoders`: Encoders +------------------------------- .. module:: email.encoders :synopsis: Encoders for email message payloads. diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -1,5 +1,5 @@ -:mod:`email`: Exception and Defect classes ------------------------------------------- +:mod:`email.errors`: Exception and Defect classes +------------------------------------------------- .. module:: email.errors :synopsis: The exception classes used by the email package. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -1,5 +1,5 @@ -:mod:`email`: Generating MIME documents ---------------------------------------- +:mod:`email.generator`: Generating MIME documents +------------------------------------------------- .. module:: email.generator :synopsis: Generate flat text email messages from a message structure. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -1,5 +1,5 @@ -:mod:`email`: Internationalized headers ---------------------------------------- +:mod:`email.header`: Internationalized headers +---------------------------------------------- .. module:: email.header :synopsis: Representing non-ASCII headers diff --git a/Doc/library/email.iterators.rst b/Doc/library/email.iterators.rst --- a/Doc/library/email.iterators.rst +++ b/Doc/library/email.iterators.rst @@ -1,5 +1,5 @@ -:mod:`email`: Iterators ------------------------ +:mod:`email.iterators`: Iterators +--------------------------------- .. module:: email.iterators :synopsis: Iterate over a message object tree. diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -1,5 +1,5 @@ -:mod:`email`: Representing an email message -------------------------------------------- +:mod:`email.message`: Representing an email message +--------------------------------------------------- .. module:: email.message :synopsis: The base class representing email messages. diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -1,5 +1,5 @@ -:mod:`email`: Creating email and MIME objects from scratch ----------------------------------------------------------- +:mod:`email.mime`: Creating email and MIME objects from scratch +--------------------------------------------------------------- .. module:: email.mime :synopsis: Build MIME messages. diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -1,5 +1,5 @@ -:mod:`email`: Parsing email messages ------------------------------------- +:mod:`email.parser`: Parsing email messages +------------------------------------------- .. module:: email.parser :synopsis: Parse flat text email messages to produce a message object structure. diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -1,5 +1,5 @@ -:mod:`email`: Miscellaneous utilities -------------------------------------- +:mod:`email.utils`: Miscellaneous utilities +------------------------------------------- .. module:: email.utils :synopsis: Miscellaneous email package utilities. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 23:18:46 2012 From: python-checkins at python.org (r.david.murray) Date: Sun, 27 May 2012 23:18:46 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzExNzg1OiBmaXgg?= =?utf8?q?the_=3Amod=3A_references_in_email_package_submodule_titles=2E?= Message-ID: http://hg.python.org/cpython/rev/6737c2ca98ee changeset: 77189:6737c2ca98ee branch: 3.2 parent: 77176:7dae83057e83 user: R David Murray date: Sun May 27 17:17:53 2012 -0400 summary: #11785: fix the :mod: references in email package submodule titles. files: Doc/library/email.charset.rst | 3 ++- Doc/library/email.encoders.rst | 3 ++- Doc/library/email.errors.rst | 3 ++- Doc/library/email.generator.rst | 3 ++- Doc/library/email.header.rst | 3 ++- Doc/library/email.iterators.rst | 3 ++- Doc/library/email.message.rst | 3 ++- Doc/library/email.mime.rst | 3 ++- Doc/library/email.parser.rst | 3 ++- Doc/library/email.util.rst | 3 ++- 10 files changed, 20 insertions(+), 10 deletions(-) diff --git a/Doc/library/email.charset.rst b/Doc/library/email.charset.rst --- a/Doc/library/email.charset.rst +++ b/Doc/library/email.charset.rst @@ -1,5 +1,5 @@ -:mod:`email`: Representing character sets ------------------------------------------ +:mod:`email.charset`: Representing character sets +------------------------------------------------- .. module:: email.charset :synopsis: Character Sets diff --git a/Doc/library/email.encoders.rst b/Doc/library/email.encoders.rst --- a/Doc/library/email.encoders.rst +++ b/Doc/library/email.encoders.rst @@ -1,5 +1,5 @@ -:mod:`email`: Encoders ----------------------- +:mod:`email.encoders`: Encoders +------------------------------- .. module:: email.encoders :synopsis: Encoders for email message payloads. diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -1,5 +1,5 @@ -:mod:`email`: Exception and Defect classes ------------------------------------------- +:mod:`email.errors`: Exception and Defect classes +------------------------------------------------- .. module:: email.errors :synopsis: The exception classes used by the email package. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -1,5 +1,5 @@ -:mod:`email`: Generating MIME documents ---------------------------------------- +:mod:`email.generator`: Generating MIME documents +------------------------------------------------- .. module:: email.generator :synopsis: Generate flat text email messages from a message structure. diff --git a/Doc/library/email.header.rst b/Doc/library/email.header.rst --- a/Doc/library/email.header.rst +++ b/Doc/library/email.header.rst @@ -1,5 +1,5 @@ -:mod:`email`: Internationalized headers ---------------------------------------- +:mod:`email.header`: Internationalized headers +---------------------------------------------- .. module:: email.header :synopsis: Representing non-ASCII headers diff --git a/Doc/library/email.iterators.rst b/Doc/library/email.iterators.rst --- a/Doc/library/email.iterators.rst +++ b/Doc/library/email.iterators.rst @@ -1,5 +1,5 @@ -:mod:`email`: Iterators ------------------------ +:mod:`email.iterators`: Iterators +--------------------------------- .. module:: email.iterators :synopsis: Iterate over a message object tree. diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -1,5 +1,5 @@ -:mod:`email`: Representing an email message -------------------------------------------- +:mod:`email.message`: Representing an email message +--------------------------------------------------- .. module:: email.message :synopsis: The base class representing email messages. diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -1,5 +1,5 @@ -:mod:`email`: Creating email and MIME objects from scratch ----------------------------------------------------------- +:mod:`email.mime`: Creating email and MIME objects from scratch +--------------------------------------------------------------- .. module:: email.mime :synopsis: Build MIME messages. diff --git a/Doc/library/email.parser.rst b/Doc/library/email.parser.rst --- a/Doc/library/email.parser.rst +++ b/Doc/library/email.parser.rst @@ -1,5 +1,5 @@ -:mod:`email`: Parsing email messages ------------------------------------- +:mod:`email.parser`: Parsing email messages +------------------------------------------- .. module:: email.parser :synopsis: Parse flat text email messages to produce a message object structure. diff --git a/Doc/library/email.util.rst b/Doc/library/email.util.rst --- a/Doc/library/email.util.rst +++ b/Doc/library/email.util.rst @@ -1,5 +1,5 @@ -:mod:`email`: Miscellaneous utilities -------------------------------------- +:mod:`email.utils`: Miscellaneous utilities +------------------------------------------- .. module:: email.utils :synopsis: Miscellaneous email package utilities. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Sun May 27 23:18:47 2012 From: python-checkins at python.org (r.david.murray) Date: Sun, 27 May 2012 23:18:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Null_merge_of_email_doc_patch_already_applied_to_this_branch?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/e10f71c8b684 changeset: 77190:e10f71c8b684 parent: 77187:64e82c4f4e10 parent: 77189:6737c2ca98ee user: R David Murray date: Sun May 27 17:18:28 2012 -0400 summary: Null merge of email doc patch already applied to this branch. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 02:45:16 2012 From: python-checkins at python.org (r.david.murray) Date: Mon, 28 May 2012 02:45:16 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314925=3A_email_now_regis?= =?utf8?q?ters_a_defect_for_missing_header/body_separator=2E?= Message-ID: http://hg.python.org/cpython/rev/0869f5f47608 changeset: 77191:0869f5f47608 user: R David Murray date: Sun May 27 20:45:01 2012 -0400 summary: #14925: email now registers a defect for missing header/body separator. This patch also deprecates the MalformedHeaderDefect. My best guess is that this defect was rendered obsolete by a refactoring of the parser, and the corresponding defect for the new parser (which this patch introduces) was overlooked. files: Doc/library/email.errors.rst | 9 +++++ Lib/email/errors.py | 6 ++- Lib/email/feedparser.py | 10 +++--- Lib/test/test_email/test_email.py | 22 ++++++++++--- Lib/test/test_email/test_parser.py | 28 ++++++++++++++--- Misc/NEWS | 4 ++ 6 files changed, 61 insertions(+), 18 deletions(-) diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -79,9 +79,18 @@ * :class:`MisplacedEnvelopeHeaderDefect` - A "Unix From" header was found in the middle of a header block. +* :class:`MissingHeaderBodySeparatorDefect` - A line was found while parsing + headers that had no leading white space but contained no ':'. Parsing + continues assuming that the line represents the first line of the body. + + .. versionadded: 3.3 + * :class:`MalformedHeaderDefect` -- A header was found that was missing a colon, or was otherwise malformed. + .. deprecated:: 3.3 + This defect has not been used for several Python versions. + * :class:`MultipartInvariantViolationDefect` -- A message claimed to be a :mimetype:`multipart`, but no subparts were found. Note that when a message has this defect, its :meth:`is_multipart` method may return false even though its diff --git a/Lib/email/errors.py b/Lib/email/errors.py --- a/Lib/email/errors.py +++ b/Lib/email/errors.py @@ -48,8 +48,10 @@ class MisplacedEnvelopeHeaderDefect(MessageDefect): """A 'Unix-from' header was found in the middle of a header block.""" -class MalformedHeaderDefect(MessageDefect): - """Found a header that was missing a colon, or was otherwise malformed.""" +class MissingHeaderBodySeparatorDefect(MessageDefect): + """Found line with no leading whitespace and no colon before blank line.""" +# XXX: backward compatibility, just in case (it was never emitted). +MalformedHeaderDefect = MissingHeaderBodySeparatorDefect class MultipartInvariantViolationDefect(MessageDefect): """A message claimed to be a multipart but no subparts were found.""" diff --git a/Lib/email/feedparser.py b/Lib/email/feedparser.py --- a/Lib/email/feedparser.py +++ b/Lib/email/feedparser.py @@ -219,6 +219,8 @@ # (i.e. newline), just throw it away. Otherwise the line is # part of the body so push it back. if not NLCRE.match(line): + defect = errors.MissingHeaderBodySeparatorDefect() + self.policy.handle_defect(self._cur, defect) self._input.unreadline(line) break headers.append(line) @@ -488,12 +490,10 @@ self._cur.defects.append(defect) continue # Split the line on the colon separating field name from value. + # There will always be a colon, because if there wasn't the part of + # the parser that calls us would have started parsing the body. i = line.find(':') - if i < 0: - defect = errors.MalformedHeaderDefect(line) - # XXX: fixme (defect not going through policy) - self._cur.defects.append(defect) - continue + assert i>0, "_parse_headers fed line with no : and no leading WS" lastheader = line[:i] lastvalue = [line] # Done with all the lines, so handle the last header. diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -1960,15 +1960,27 @@ # test_parser.TestMessageDefectDetectionBase def test_first_line_is_continuation_header(self): eq = self.assertEqual - m = ' Line 1\nLine 2\nLine 3' + m = ' Line 1\nSubject: test\n\nbody' msg = email.message_from_string(m) - eq(msg.keys(), []) - eq(msg.get_payload(), 'Line 2\nLine 3') + eq(msg.keys(), ['Subject']) + eq(msg.get_payload(), 'body') eq(len(msg.defects), 1) - self.assertTrue(isinstance(msg.defects[0], - errors.FirstHeaderLineIsContinuationDefect)) + self.assertDefectsEqual(msg.defects, + [errors.FirstHeaderLineIsContinuationDefect]) eq(msg.defects[0].line, ' Line 1\n') + # test_parser.TestMessageDefectDetectionBase + def test_missing_header_body_separator(self): + # Our heuristic if we see a line that doesn't look like a header (no + # leading whitespace but no ':') is to assume that the blank line that + # separates the header from the body is missing, and to stop parsing + # headers and start parsing the body. + msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') + self.assertDefectsEqual(msg.defects, + [errors.MissingHeaderBodySeparatorDefect]) + # Test RFC 2047 header encoding and decoding class TestRFC2047(TestEmailBase): diff --git a/Lib/test/test_email/test_parser.py b/Lib/test/test_email/test_parser.py --- a/Lib/test/test_email/test_parser.py +++ b/Lib/test/test_email/test_parser.py @@ -237,17 +237,33 @@ policy=self.policy.clone(raise_on_defect=True)) def test_first_line_is_continuation_header(self): - msg = self._str_msg(' Line 1\nLine 2\nLine 3') - self.assertEqual(msg.keys(), []) - self.assertEqual(msg.get_payload(), 'Line 2\nLine 3') + msg = self._str_msg(' Line 1\nSubject: test\n\nbody') + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'body') self.assertEqual(len(self.get_defects(msg)), 1) - self.assertTrue(isinstance(self.get_defects(msg)[0], - errors.FirstHeaderLineIsContinuationDefect)) + self.assertDefectsEqual(self.get_defects(msg), + [errors.FirstHeaderLineIsContinuationDefect]) self.assertEqual(self.get_defects(msg)[0].line, ' Line 1\n') def test_first_line_is_continuation_header_raise_on_defect(self): with self.assertRaises(errors.FirstHeaderLineIsContinuationDefect): - self._str_msg(' Line 1\nLine 2\nLine 3', + self._str_msg(' Line 1\nSubject: test\n\nbody\n', + policy=self.policy.clone(raise_on_defect=True)) + + def test_missing_header_body_separator(self): + # Our heuristic if we see a line that doesn't look like a header (no + # leading whitespace but no ':') is to assume that the blank line that + # separates the header from the body is missing, and to stop parsing + # headers and start parsing the body. + msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') + self.assertDefectsEqual(self.get_defects(msg), + [errors.MissingHeaderBodySeparatorDefect]) + + def test_missing_header_body_separator_raise_on_defect(self): + with self.assertRaises(errors.MissingHeaderBodySeparatorDefect): + self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n', policy=self.policy.clone(raise_on_defect=True)) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,10 @@ Library ------- +- Issue #14925: email now registers a defect when the parser decides that there + is a missing header/body separator line. MalformedHeaderDefect, which the + existing code would never actually generate, is deprecated. + - Issue #10365: File open dialog now works instead of crashing even when parent window is closed. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 03:24:23 2012 From: python-checkins at python.org (r.david.murray) Date: Mon, 28 May 2012 03:24:23 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=231672568=3A_email_now_reg?= =?utf8?q?isters_defects_for_base64_payload_format_errors=2E?= Message-ID: http://hg.python.org/cpython/rev/17341b51af4f changeset: 77192:17341b51af4f user: R David Murray date: Sun May 27 21:23:34 2012 -0400 summary: #1672568: email now registers defects for base64 payload format errors. Which also means that it is now producing *something* for any base64 payload, which is what leads to the couple of older test changes in test_email. This is a slightly backward incompatible behavior change, but the new behavior is so much more useful than the old (you can now *reliably* detect errors, and any program that was detecting errors by sniffing for a base64 return from get_payload(decode=True) and then doing its own error-recovery decode will just get the error-recovery decode right away). So this seems to me to be worth the small risk inherent in this behavior change. This patch also refactors the defect tests into a separate test file, since they are no longer just parser tests. files: Doc/library/email.errors.rst | 7 + Doc/library/email.message.rst | 8 +- Lib/email/message.py | 12 +- Lib/test/test_email/test_defect_handling.py | 304 ++++++++++ Lib/test/test_email/test_email.py | 32 +- Lib/test/test_email/test_parser.py | 256 -------- 6 files changed, 344 insertions(+), 275 deletions(-) diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -96,3 +96,10 @@ this defect, its :meth:`is_multipart` method may return false even though its content type claims to be :mimetype:`multipart`. +* :class:`InvalidBase64PaddingDefect` -- When decoding a block of base64 + enocded bytes, the padding was not correct. Enough padding is added to + perform the decode, but the resulting decoded bytes may be invalid. + +* :class:`InvalidBase64CharactersDefect` -- When decoding a block of base64 + enocded bytes, characters outside the base64 alphebet were encountered. + The characters are ignored, but the resulting decoded bytes may be invalid. diff --git a/Doc/library/email.message.rst b/Doc/library/email.message.rst --- a/Doc/library/email.message.rst +++ b/Doc/library/email.message.rst @@ -111,10 +111,14 @@ header. When ``True`` and the message is not a multipart, the payload will be decoded if this header's value is ``quoted-printable`` or ``base64``. If some other encoding is used, or :mailheader:`Content-Transfer-Encoding` - header is missing, or if the payload has bogus base64 data, the payload is + header is missing, the payload is returned as-is (undecoded). In all cases the returned value is binary data. If the message is a multipart and the *decode* flag is ``True``, - then ``None`` is returned. + then ``None`` is returned. If the payload is base64 and it was not + perfectly formed (missing padding, characters outside the base64 + alphabet), then an appropriate defect will be added to the message's + defect property (:class:`~email.errors.InvalidBase64PaddingDefect` or + :class:`~email.errors.InvalidBase64CharactersDefect`, respectively). When *decode* is ``False`` (the default) the body is returned as a string without decoding the :mailheader:`Content-Transfer-Encoding`. However, diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -17,6 +17,7 @@ from email import errors from email._policybase import compat32 from email import charset as _charset +from email._encoded_words import decode_b Charset = _charset.Charset SEMISPACE = '; ' @@ -249,11 +250,12 @@ if cte == 'quoted-printable': return utils._qdecode(bpayload) elif cte == 'base64': - try: - return base64.b64decode(bpayload) - except binascii.Error: - # Incorrect padding - return bpayload + # XXX: this is a bit of a hack; decode_b should probably be factored + # out somewhere, but I haven't figured out where yet. + value, defects = decode_b(b''.join(bpayload.splitlines())) + for defect in defects: + self.policy.handle_defect(self, defect) + return value elif cte in ('x-uuencode', 'uuencode', 'uue', 'x-uue'): in_file = BytesIO(bpayload) out_file = BytesIO() diff --git a/Lib/test/test_email/test_defect_handling.py b/Lib/test/test_email/test_defect_handling.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test_defect_handling.py @@ -0,0 +1,304 @@ +import textwrap +import unittest +from email._policybase import Compat32 +from email import errors +from test.test_email import TestEmailBase + + +class TestMessageDefectDetectionBase: + + dup_boundary_msg = textwrap.dedent("""\ + Subject: XX + From: xx at xx.dk + To: XX + Mime-version: 1.0 + Content-type: multipart/mixed; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: multipart/alternative; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/plain; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + text + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/html; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: image/gif; name="xx.gif"; + Content-disposition: attachment + Content-transfer-encoding: base64 + + Some removed base64 encoded chars. + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + """) + + def test_same_boundary_inner_outer(self): + # XXX better would be to actually detect the duplicate. + msg = self._str_msg(self.dup_boundary_msg) + inner = msg.get_payload(0) + self.assertTrue(hasattr(inner, 'defects')) + self.assertEqual(len(self.get_defects(inner)), 1) + self.assertTrue(isinstance(self.get_defects(inner)[0], + errors.StartBoundaryNotFoundDefect)) + + def test_same_boundary_inner_outer_raises_on_defect(self): + with self.assertRaises(errors.StartBoundaryNotFoundDefect): + self._str_msg(self.dup_boundary_msg, + policy=self.policy.clone(raise_on_defect=True)) + + no_boundary_msg = textwrap.dedent("""\ + Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) + From: foobar + Subject: broken mail + MIME-Version: 1.0 + Content-Type: multipart/report; report-type=delivery-status; + + --JAB03225.986577786/zinfandel.lacita.com + + One part + + --JAB03225.986577786/zinfandel.lacita.com + Content-Type: message/delivery-status + + Header: Another part + + --JAB03225.986577786/zinfandel.lacita.com-- + """) + + def test_multipart_no_boundary(self): + msg = self._str_msg(self.no_boundary_msg) + self.assertTrue(isinstance(msg.get_payload(), str)) + self.assertEqual(len(self.get_defects(msg)), 2) + self.assertTrue(isinstance(self.get_defects(msg)[0], + errors.NoBoundaryInMultipartDefect)) + self.assertTrue(isinstance(self.get_defects(msg)[1], + errors.MultipartInvariantViolationDefect)) + + def test_multipart_no_boundary_raise_on_defect(self): + with self.assertRaises(errors.NoBoundaryInMultipartDefect): + self._str_msg(self.no_boundary_msg, + policy=self.policy.clone(raise_on_defect=True)) + + multipart_msg = textwrap.dedent("""\ + Date: Wed, 14 Nov 2007 12:56:23 GMT + From: foo at bar.invalid + To: foo at bar.invalid + Subject: Content-Transfer-Encoding: base64 and multipart + MIME-Version: 1.0 + Content-Type: multipart/mixed; + boundary="===============3344438784458119861=="{} + + --===============3344438784458119861== + Content-Type: text/plain + + Test message + + --===============3344438784458119861== + Content-Type: application/octet-stream + Content-Transfer-Encoding: base64 + + YWJj + + --===============3344438784458119861==-- + """) + + def test_multipart_invalid_cte(self): + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) + self.assertEqual(len(self.get_defects(msg)), 1) + self.assertIsInstance(self.get_defects(msg)[0], + errors.InvalidMultipartContentTransferEncodingDefect) + + def test_multipart_invalid_cte_raise_on_defect(self): + with self.assertRaises( + errors.InvalidMultipartContentTransferEncodingDefect): + self._str_msg( + self.multipart_msg.format( + "\nContent-Transfer-Encoding: base64"), + policy=self.policy.clone(raise_on_defect=True)) + + def test_multipart_no_cte_no_defect(self): + msg = self._str_msg(self.multipart_msg.format('')) + self.assertEqual(len(self.get_defects(msg)), 0) + + def test_multipart_valid_cte_no_defect(self): + for cte in ('7bit', '8bit', 'BINary'): + msg = self._str_msg( + self.multipart_msg.format("\nContent-Transfer-Encoding: "+cte)) + self.assertEqual(len(self.get_defects(msg)), 0, "cte="+cte) + + lying_multipart_msg = textwrap.dedent("""\ + From: "Allison Dunlap" + To: yyy at example.com + Subject: 64423 + Date: Sun, 11 Jul 2004 16:09:27 -0300 + MIME-Version: 1.0 + Content-Type: multipart/alternative; + + Blah blah blah + """) + + def test_lying_multipart(self): + msg = self._str_msg(self.lying_multipart_msg) + self.assertTrue(hasattr(msg, 'defects')) + self.assertEqual(len(self.get_defects(msg)), 2) + self.assertTrue(isinstance(self.get_defects(msg)[0], + errors.NoBoundaryInMultipartDefect)) + self.assertTrue(isinstance(self.get_defects(msg)[1], + errors.MultipartInvariantViolationDefect)) + + def test_lying_multipart_raise_on_defect(self): + with self.assertRaises(errors.NoBoundaryInMultipartDefect): + self._str_msg(self.lying_multipart_msg, + policy=self.policy.clone(raise_on_defect=True)) + + missing_start_boundary_msg = textwrap.dedent("""\ + Content-Type: multipart/mixed; boundary="AAA" + From: Mail Delivery Subsystem + To: yyy at example.com + + --AAA + + Stuff + + --AAA + Content-Type: message/rfc822 + + From: webmaster at python.org + To: zzz at example.com + Content-Type: multipart/mixed; boundary="BBB" + + --BBB-- + + --AAA-- + + """) + + def test_missing_start_boundary(self): + # The message structure is: + # + # multipart/mixed + # text/plain + # message/rfc822 + # multipart/mixed [*] + # + # [*] This message is missing its start boundary + outer = self._str_msg(self.missing_start_boundary_msg) + bad = outer.get_payload(1).get_payload(0) + self.assertEqual(len(self.get_defects(bad)), 1) + self.assertTrue(isinstance(self.get_defects(bad)[0], + errors.StartBoundaryNotFoundDefect)) + + def test_missing_start_boundary_raise_on_defect(self): + with self.assertRaises(errors.StartBoundaryNotFoundDefect): + self._str_msg(self.missing_start_boundary_msg, + policy=self.policy.clone(raise_on_defect=True)) + + def test_first_line_is_continuation_header(self): + msg = self._str_msg(' Line 1\nSubject: test\n\nbody') + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'body') + self.assertEqual(len(self.get_defects(msg)), 1) + self.assertDefectsEqual(self.get_defects(msg), + [errors.FirstHeaderLineIsContinuationDefect]) + self.assertEqual(self.get_defects(msg)[0].line, ' Line 1\n') + + def test_first_line_is_continuation_header_raise_on_defect(self): + with self.assertRaises(errors.FirstHeaderLineIsContinuationDefect): + self._str_msg(' Line 1\nSubject: test\n\nbody\n', + policy=self.policy.clone(raise_on_defect=True)) + + def test_missing_header_body_separator(self): + # Our heuristic if we see a line that doesn't look like a header (no + # leading whitespace but no ':') is to assume that the blank line that + # separates the header from the body is missing, and to stop parsing + # headers and start parsing the body. + msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + self.assertEqual(msg.keys(), ['Subject']) + self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') + self.assertDefectsEqual(self.get_defects(msg), + [errors.MissingHeaderBodySeparatorDefect]) + + def test_missing_header_body_separator_raise_on_defect(self): + with self.assertRaises(errors.MissingHeaderBodySeparatorDefect): + self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n', + policy=self.policy.clone(raise_on_defect=True)) + + badly_padded_base64_payload = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + + dmk + """) + + def test_bad_padding_in_base64_payload(self): + msg = self._str_msg(self.badly_padded_base64_payload) + self.assertEqual(msg.get_payload(decode=True), b'vi') + self.assertDefectsEqual(self.get_defects(msg), + [errors.InvalidBase64PaddingDefect]) + + def test_bad_padding_in_base64_payload_raise_on_defect(self): + msg = self._str_msg(self.badly_padded_base64_payload, + policy=self.policy.clone(raise_on_defect=True)) + with self.assertRaises(errors.InvalidBase64PaddingDefect): + msg.get_payload(decode=True) + + invalid_chars_in_base64_payload = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 + + dm\x01k=== + """) + + def test_invalid_chars_in_base64_payload(self): + msg = self._str_msg(self.invalid_chars_in_base64_payload) + self.assertEqual(msg.get_payload(decode=True), b'vi') + self.assertDefectsEqual(self.get_defects(msg), + [errors.InvalidBase64CharactersDefect]) + + def test_invalid_chars_in_base64_payload_raise_on_defect(self): + msg = self._str_msg(self.invalid_chars_in_base64_payload, + policy=self.policy.clone(raise_on_defect=True)) + with self.assertRaises(errors.InvalidBase64CharactersDefect): + msg.get_payload(decode=True) + + +class TestMessageDefectDetection(TestMessageDefectDetectionBase, TestEmailBase): + + def get_defects(self, obj): + return obj.defects + + +class TestMessageDefectDetectionCapture(TestMessageDefectDetectionBase, + TestEmailBase): + + class CapturePolicy(Compat32): + captured = None + def register_defect(self, obj, defect): + self.captured.append(defect) + + def setUp(self): + self.policy = self.CapturePolicy(captured=list()) + + def get_defects(self, obj): + return self.policy.captured + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -513,6 +513,7 @@ eq(msg.values(), ['One Hundred', 'Twenty', 'Three', 'Eleven']) self.assertRaises(KeyError, msg.replace_header, 'Fourth', 'Missing') + # test_defect_handling:test_invalid_chars_in_base64_payload def test_broken_base64_payload(self): x = 'AwDp0P7//y6LwKEAcPa/6Q=9' msg = Message() @@ -520,7 +521,10 @@ msg['content-transfer-encoding'] = 'base64' msg.set_payload(x) self.assertEqual(msg.get_payload(decode=True), - bytes(x, 'raw-unicode-escape')) + (b'\x03\x00\xe9\xd0\xfe\xff\xff.\x8b\xc0' + b'\xa1\x00p\xf6\xbf\xe9\x0f')) + self.assertIsInstance(msg.defects[0], + errors.InvalidBase64CharactersDefect) def test_broken_unicode_payload(self): # This test improves coverage but is not a compliance test. @@ -1815,7 +1819,7 @@ eq(msg.get_content_maintype(), 'text') eq(msg.get_content_subtype(), 'plain') - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_same_boundary_inner_outer(self): unless = self.assertTrue msg = self._msgobj('msg_15.txt') @@ -1826,7 +1830,7 @@ unless(isinstance(inner.defects[0], errors.StartBoundaryNotFoundDefect)) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_multipart_no_boundary(self): unless = self.assertTrue msg = self._msgobj('msg_25.txt') @@ -1860,7 +1864,7 @@ --===============3344438784458119861==-- """) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_multipart_invalid_cte(self): msg = self._str_msg( self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) @@ -1868,12 +1872,12 @@ self.assertIsInstance(msg.defects[0], errors.InvalidMultipartContentTransferEncodingDefect) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_multipart_no_cte_no_defect(self): msg = self._str_msg(self.multipart_msg.format('')) self.assertEqual(len(msg.defects), 0) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_multipart_valid_cte_no_defect(self): for cte in ('7bit', '8bit', 'BINary'): msg = self._str_msg( @@ -1930,7 +1934,7 @@ counter to RFC 2822, there's no separating newline here """) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_lying_multipart(self): unless = self.assertTrue msg = self._msgobj('msg_41.txt') @@ -1941,7 +1945,7 @@ unless(isinstance(msg.defects[1], errors.MultipartInvariantViolationDefect)) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_missing_start_boundary(self): outer = self._msgobj('msg_42.txt') # The message structure is: @@ -1957,7 +1961,7 @@ self.assertTrue(isinstance(bad.defects[0], errors.StartBoundaryNotFoundDefect)) - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_first_line_is_continuation_header(self): eq = self.assertEqual m = ' Line 1\nSubject: test\n\nbody' @@ -3271,15 +3275,19 @@ self.assertEqual(msg.get_payload(decode=True), 'p?st?l\n'.encode('utf-8')) + # test_defect_handling:test_invalid_chars_in_base64_payload def test_8bit_in_base64_body(self): - # Sticking an 8bit byte in a base64 block makes it undecodable by - # normal means, so the block is returned undecoded, but as bytes. + # If we get 8bit bytes in a base64 body, we can just ignore them + # as being outside the base64 alphabet and decode anyway. But + # we register a defect. m = self.bodytest_msg.format(charset='utf-8', cte='base64', bodyline='cMO2c3RhbA?=').encode('utf-8') msg = email.message_from_bytes(m) self.assertEqual(msg.get_payload(decode=True), - 'cMO2c3RhbA?=\n'.encode('utf-8')) + 'p?stal'.encode('utf-8')) + self.assertIsInstance(msg.defects[0], + errors.InvalidBase64CharactersDefect) def test_8bit_in_uuencode_body(self): # Sticking an 8bit byte in a uuencode block makes it undecodable by diff --git a/Lib/test/test_email/test_parser.py b/Lib/test/test_email/test_parser.py --- a/Lib/test/test_email/test_parser.py +++ b/Lib/test/test_email/test_parser.py @@ -1,9 +1,6 @@ import io import email -import textwrap import unittest -from email._policybase import Compat32 -from email import errors from email.message import Message from test.test_email import TestEmailBase @@ -35,258 +32,5 @@ # XXX add tests for other functions that take Message arg. -class TestMessageDefectDetectionBase: - - dup_boundary_msg = textwrap.dedent("""\ - Subject: XX - From: xx at xx.dk - To: XX - Mime-version: 1.0 - Content-type: multipart/mixed; - boundary="MS_Mac_OE_3071477847_720252_MIME_Part" - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: multipart/alternative; - boundary="MS_Mac_OE_3071477847_720252_MIME_Part" - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: text/plain; charset="ISO-8859-1" - Content-transfer-encoding: quoted-printable - - text - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: text/html; charset="ISO-8859-1" - Content-transfer-encoding: quoted-printable - - - - --MS_Mac_OE_3071477847_720252_MIME_Part-- - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: image/gif; name="xx.gif"; - Content-disposition: attachment - Content-transfer-encoding: base64 - - Some removed base64 encoded chars. - - --MS_Mac_OE_3071477847_720252_MIME_Part-- - - """) - - def test_same_boundary_inner_outer(self): - # XXX better would be to actually detect the duplicate. - msg = self._str_msg(self.dup_boundary_msg) - inner = msg.get_payload(0) - self.assertTrue(hasattr(inner, 'defects')) - self.assertEqual(len(self.get_defects(inner)), 1) - self.assertTrue(isinstance(self.get_defects(inner)[0], - errors.StartBoundaryNotFoundDefect)) - - def test_same_boundary_inner_outer_raises_on_defect(self): - with self.assertRaises(errors.StartBoundaryNotFoundDefect): - self._str_msg(self.dup_boundary_msg, - policy=self.policy.clone(raise_on_defect=True)) - - no_boundary_msg = textwrap.dedent("""\ - Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) - From: foobar - Subject: broken mail - MIME-Version: 1.0 - Content-Type: multipart/report; report-type=delivery-status; - - --JAB03225.986577786/zinfandel.lacita.com - - One part - - --JAB03225.986577786/zinfandel.lacita.com - Content-Type: message/delivery-status - - Header: Another part - - --JAB03225.986577786/zinfandel.lacita.com-- - """) - - def test_multipart_no_boundary(self): - msg = self._str_msg(self.no_boundary_msg) - self.assertTrue(isinstance(msg.get_payload(), str)) - self.assertEqual(len(self.get_defects(msg)), 2) - self.assertTrue(isinstance(self.get_defects(msg)[0], - errors.NoBoundaryInMultipartDefect)) - self.assertTrue(isinstance(self.get_defects(msg)[1], - errors.MultipartInvariantViolationDefect)) - - def test_multipart_no_boundary_raise_on_defect(self): - with self.assertRaises(errors.NoBoundaryInMultipartDefect): - self._str_msg(self.no_boundary_msg, - policy=self.policy.clone(raise_on_defect=True)) - - multipart_msg = textwrap.dedent("""\ - Date: Wed, 14 Nov 2007 12:56:23 GMT - From: foo at bar.invalid - To: foo at bar.invalid - Subject: Content-Transfer-Encoding: base64 and multipart - MIME-Version: 1.0 - Content-Type: multipart/mixed; - boundary="===============3344438784458119861=="{} - - --===============3344438784458119861== - Content-Type: text/plain - - Test message - - --===============3344438784458119861== - Content-Type: application/octet-stream - Content-Transfer-Encoding: base64 - - YWJj - - --===============3344438784458119861==-- - """) - - def test_multipart_invalid_cte(self): - msg = self._str_msg( - self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) - self.assertEqual(len(self.get_defects(msg)), 1) - self.assertIsInstance(self.get_defects(msg)[0], - errors.InvalidMultipartContentTransferEncodingDefect) - - def test_multipart_invalid_cte_raise_on_defect(self): - with self.assertRaises( - errors.InvalidMultipartContentTransferEncodingDefect): - self._str_msg( - self.multipart_msg.format( - "\nContent-Transfer-Encoding: base64"), - policy=self.policy.clone(raise_on_defect=True)) - - def test_multipart_no_cte_no_defect(self): - msg = self._str_msg(self.multipart_msg.format('')) - self.assertEqual(len(self.get_defects(msg)), 0) - - def test_multipart_valid_cte_no_defect(self): - for cte in ('7bit', '8bit', 'BINary'): - msg = self._str_msg( - self.multipart_msg.format("\nContent-Transfer-Encoding: "+cte)) - self.assertEqual(len(self.get_defects(msg)), 0, "cte="+cte) - - lying_multipart_msg = textwrap.dedent("""\ - From: "Allison Dunlap" - To: yyy at example.com - Subject: 64423 - Date: Sun, 11 Jul 2004 16:09:27 -0300 - MIME-Version: 1.0 - Content-Type: multipart/alternative; - - Blah blah blah - """) - - def test_lying_multipart(self): - msg = self._str_msg(self.lying_multipart_msg) - self.assertTrue(hasattr(msg, 'defects')) - self.assertEqual(len(self.get_defects(msg)), 2) - self.assertTrue(isinstance(self.get_defects(msg)[0], - errors.NoBoundaryInMultipartDefect)) - self.assertTrue(isinstance(self.get_defects(msg)[1], - errors.MultipartInvariantViolationDefect)) - - def test_lying_multipart_raise_on_defect(self): - with self.assertRaises(errors.NoBoundaryInMultipartDefect): - self._str_msg(self.lying_multipart_msg, - policy=self.policy.clone(raise_on_defect=True)) - - missing_start_boundary_msg = textwrap.dedent("""\ - Content-Type: multipart/mixed; boundary="AAA" - From: Mail Delivery Subsystem - To: yyy at example.com - - --AAA - - Stuff - - --AAA - Content-Type: message/rfc822 - - From: webmaster at python.org - To: zzz at example.com - Content-Type: multipart/mixed; boundary="BBB" - - --BBB-- - - --AAA-- - - """) - - def test_missing_start_boundary(self): - # The message structure is: - # - # multipart/mixed - # text/plain - # message/rfc822 - # multipart/mixed [*] - # - # [*] This message is missing its start boundary - outer = self._str_msg(self.missing_start_boundary_msg) - bad = outer.get_payload(1).get_payload(0) - self.assertEqual(len(self.get_defects(bad)), 1) - self.assertTrue(isinstance(self.get_defects(bad)[0], - errors.StartBoundaryNotFoundDefect)) - - def test_missing_start_boundary_raise_on_defect(self): - with self.assertRaises(errors.StartBoundaryNotFoundDefect): - self._str_msg(self.missing_start_boundary_msg, - policy=self.policy.clone(raise_on_defect=True)) - - def test_first_line_is_continuation_header(self): - msg = self._str_msg(' Line 1\nSubject: test\n\nbody') - self.assertEqual(msg.keys(), ['Subject']) - self.assertEqual(msg.get_payload(), 'body') - self.assertEqual(len(self.get_defects(msg)), 1) - self.assertDefectsEqual(self.get_defects(msg), - [errors.FirstHeaderLineIsContinuationDefect]) - self.assertEqual(self.get_defects(msg)[0].line, ' Line 1\n') - - def test_first_line_is_continuation_header_raise_on_defect(self): - with self.assertRaises(errors.FirstHeaderLineIsContinuationDefect): - self._str_msg(' Line 1\nSubject: test\n\nbody\n', - policy=self.policy.clone(raise_on_defect=True)) - - def test_missing_header_body_separator(self): - # Our heuristic if we see a line that doesn't look like a header (no - # leading whitespace but no ':') is to assume that the blank line that - # separates the header from the body is missing, and to stop parsing - # headers and start parsing the body. - msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') - self.assertEqual(msg.keys(), ['Subject']) - self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') - self.assertDefectsEqual(self.get_defects(msg), - [errors.MissingHeaderBodySeparatorDefect]) - - def test_missing_header_body_separator_raise_on_defect(self): - with self.assertRaises(errors.MissingHeaderBodySeparatorDefect): - self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n', - policy=self.policy.clone(raise_on_defect=True)) - - -class TestMessageDefectDetection(TestMessageDefectDetectionBase, TestEmailBase): - - def get_defects(self, obj): - return obj.defects - - -class TestMessageDefectDetectionCapture(TestMessageDefectDetectionBase, - TestEmailBase): - - class CapturePolicy(Compat32): - captured = None - def register_defect(self, obj, defect): - self.captured.append(defect) - - def setUp(self): - self.policy = self.CapturePolicy(captured=list()) - - def get_defects(self, obj): - return self.policy.captured - - if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 03:28:07 2012 From: python-checkins at python.org (r.david.murray) Date: Mon, 28 May 2012 03:28:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_News_item_for_=231672568=2E?= Message-ID: http://hg.python.org/cpython/rev/d68e30be755e changeset: 77193:d68e30be755e user: R David Murray date: Sun May 27 21:27:31 2012 -0400 summary: News item for #1672568. files: Misc/NEWS | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,10 @@ Library ------- +- Issue #1672568: email now always decodes base64 payloads, adding padding and + ignoring non-base64-alphabet characters if needed, and registering defects + for any such problems. + - Issue #14925: email now registers a defect when the parser decides that there is a missing header/body separator line. MalformedHeaderDefect, which the existing code would never actually generate, is deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 03:55:03 2012 From: python-checkins at python.org (terry.reedy) Date: Mon, 28 May 2012 03:55:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Issue12510=3A_A?= =?utf8?q?ttempting_to_get_invalid_tooltip_no_longer_closes_Idle=2E?= Message-ID: http://hg.python.org/cpython/rev/938b12452af7 changeset: 77194:938b12452af7 branch: 2.7 parent: 77188:df59aefdb1c8 user: Terry Jan Reedy date: Sun May 27 21:28:42 2012 -0400 summary: Issue12510: Attempting to get invalid tooltip no longer closes Idle. Original patch by Roger Serwy. files: Lib/idlelib/CallTips.py | 9 ++++++--- Misc/NEWS | 3 +++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Lib/idlelib/CallTips.py b/Lib/idlelib/CallTips.py --- a/Lib/idlelib/CallTips.py +++ b/Lib/idlelib/CallTips.py @@ -112,7 +112,9 @@ namespace.update(__main__.__dict__) try: return eval(name, namespace) - except (NameError, AttributeError): + # any exception is possible if evalfuncs True in open_calltip + # at least Syntax, Name, Attribute, Index, and Key E. if not + except: return None def _find_constructor(class_ob): @@ -127,9 +129,10 @@ return None def get_arg_text(ob): - """Get a string describing the arguments for the given object""" + """Get a string describing the arguments for the given object, + only if it is callable.""" arg_text = "" - if ob is not None: + if ob is not None and hasattr(ob, '__call__'): arg_offset = 0 if type(ob) in (types.ClassType, types.TypeType): # Look for the highest __init__ in the class chain. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -64,6 +64,9 @@ Library ------- +- Issue12510: Attempting to get invalid tooltip no longer closes Idle. + Original patch by Roger Serwy. + - Issue #10365: File open dialog now works instead of crashing even when parent window is closed. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 03:55:03 2012 From: python-checkins at python.org (terry.reedy) Date: Mon, 28 May 2012 03:55:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Issue12510=3A_A?= =?utf8?q?ttempting_to_get_invalid_tooltip_no_longer_closes_Idle=2E?= Message-ID: http://hg.python.org/cpython/rev/4a7582866735 changeset: 77195:4a7582866735 branch: 3.2 parent: 77189:6737c2ca98ee user: Terry Jan Reedy date: Sun May 27 21:29:17 2012 -0400 summary: Issue12510: Attempting to get invalid tooltip no longer closes Idle. Original patch by Roger Serwy. files: Lib/idlelib/CallTips.py | 9 ++++++--- Misc/NEWS | 3 +++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Lib/idlelib/CallTips.py b/Lib/idlelib/CallTips.py --- a/Lib/idlelib/CallTips.py +++ b/Lib/idlelib/CallTips.py @@ -110,7 +110,9 @@ namespace.update(__main__.__dict__) try: return eval(name, namespace) - except (NameError, AttributeError): + # any exception is possible if evalfuncs True in open_calltip + # at least Syntax, Name, Attribute, Index, and Key E. if not + except: return None def _find_constructor(class_ob): @@ -125,9 +127,10 @@ return None def get_argspec(ob): - """Get a string describing the arguments for the given object.""" + """Get a string describing the arguments for the given object, + only if it is callable.""" argspec = "" - if ob is not None: + if ob is not None and hasattr(ob, '__call__'): if isinstance(ob, type): fob = _find_constructor(ob) if fob is None: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue12510: Attempting to get invalid tooltip no longer closes Idle. + Original patch by Roger Serwy. + - Issue #10365: File open dialog now works instead of crashing even when parent window is closed. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 03:55:04 2012 From: python-checkins at python.org (terry.reedy) Date: Mon, 28 May 2012 03:55:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_3=2E2_closes_=2312510?= Message-ID: http://hg.python.org/cpython/rev/0835bee19f86 changeset: 77196:0835bee19f86 parent: 77192:17341b51af4f parent: 77195:4a7582866735 user: Terry Jan Reedy date: Sun May 27 21:39:39 2012 -0400 summary: Merge 3.2 closes #12510 files: Lib/idlelib/CallTips.py | 9 ++++++--- Misc/NEWS | 3 +++ 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Lib/idlelib/CallTips.py b/Lib/idlelib/CallTips.py --- a/Lib/idlelib/CallTips.py +++ b/Lib/idlelib/CallTips.py @@ -110,7 +110,9 @@ namespace.update(__main__.__dict__) try: return eval(name, namespace) - except (NameError, AttributeError): + # any exception is possible if evalfuncs True in open_calltip + # at least Syntax, Name, Attribute, Index, and Key E. if not + except: return None def _find_constructor(class_ob): @@ -125,9 +127,10 @@ return None def get_argspec(ob): - """Get a string describing the arguments for the given object.""" + """Get a string describing the arguments for the given object, + only if it is callable.""" argspec = "" - if ob is not None: + if ob is not None and hasattr(ob, '__call__'): if isinstance(ob, type): fob = _find_constructor(ob) if fob is None: diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,9 @@ Library ------- +- Issue12510: Attempting to get invalid tooltip no longer closes Idle. + Original patch by Roger Serwy. + - Issue #14925: email now registers a defect when the parser decides that there is a missing header/body separator line. MalformedHeaderDefect, which the existing code would never actually generate, is deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 03:55:05 2012 From: python-checkins at python.org (terry.reedy) Date: Mon, 28 May 2012 03:55:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/b870875f054a changeset: 77197:b870875f054a parent: 77196:0835bee19f86 parent: 77193:d68e30be755e user: Terry Jan Reedy date: Sun May 27 21:54:28 2012 -0400 summary: merge heads files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 04:20:54 2012 From: python-checkins at python.org (r.david.murray) Date: Mon, 28 May 2012 04:20:54 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2312515=3A_email_now_regis?= =?utf8?q?ters_a_defect_if_the_MIME_end_boundary_is_missing=2E?= Message-ID: http://hg.python.org/cpython/rev/81e008f13b4f changeset: 77198:81e008f13b4f user: R David Murray date: Sun May 27 22:20:42 2012 -0400 summary: #12515: email now registers a defect if the MIME end boundary is missing. This commit also restores the news item for 167256 that it looks like Terry inadvertently deleted. (Either that, or I don't understand now merging works...which is equally possible.) files: Doc/library/email.errors.rst | 5 + Lib/email/errors.py | 3 + Lib/email/feedparser.py | 14 +++- Lib/test/test_email/test_defect_handling.py | 33 ++++++++++ Misc/NEWS | 7 ++ 5 files changed, 59 insertions(+), 3 deletions(-) diff --git a/Doc/library/email.errors.rst b/Doc/library/email.errors.rst --- a/Doc/library/email.errors.rst +++ b/Doc/library/email.errors.rst @@ -73,6 +73,11 @@ * :class:`StartBoundaryNotFoundDefect` -- The start boundary claimed in the :mailheader:`Content-Type` header was never found. +* :class:`CloseBoundaryNotFoundDefect` -- A start boundary was found, but + no corresponding close boundary was ever found. + + .. versionadded: 3.3 + * :class:`FirstHeaderLineIsContinuationDefect` -- The message had a continuation line as its first header line. diff --git a/Lib/email/errors.py b/Lib/email/errors.py --- a/Lib/email/errors.py +++ b/Lib/email/errors.py @@ -42,6 +42,9 @@ class StartBoundaryNotFoundDefect(MessageDefect): """The claimed start boundary was never found.""" +class CloseBoundaryNotFoundDefect(MessageDefect): + """A start boundary was found, but not the corresponding close boundary.""" + class FirstHeaderLineIsContinuationDefect(MessageDefect): """A message had a continuation line as its first header line.""" diff --git a/Lib/email/feedparser.py b/Lib/email/feedparser.py --- a/Lib/email/feedparser.py +++ b/Lib/email/feedparser.py @@ -324,6 +324,7 @@ capturing_preamble = True preamble = [] linesep = False + close_boundary_seen = False while True: line = self._input.readline() if line is NeedMoreData: @@ -338,6 +339,7 @@ # the closing boundary, then we need to initialize the # epilogue with the empty string (see below). if mo.group('end'): + close_boundary_seen = True linesep = mo.group('linesep') break # We saw an inter-part boundary. Were we in the preamble? @@ -406,7 +408,6 @@ # We've seen either the EOF or the end boundary. If we're still # capturing the preamble, we never saw the start boundary. Note # that as a defect and store the captured text as the payload. - # Everything from here to the EOF is epilogue. if capturing_preamble: defect = errors.StartBoundaryNotFoundDefect() self.policy.handle_defect(self._cur, defect) @@ -418,8 +419,15 @@ continue self._cur.epilogue = EMPTYSTRING.join(epilogue) return - # If the end boundary ended in a newline, we'll need to make sure - # the epilogue isn't None + # If we're not processing the preamble, then we might have seen + # EOF without seeing that end boundary...that is also a defect. + if not close_boundary_seen: + defect = errors.CloseBoundaryNotFoundDefect() + self.policy.handle_defect(self._cur, defect) + return + # Everything from here to the EOF is epilogue. If the end boundary + # ended in a newline, we'll need to make sure the epilogue isn't + # None if linesep: epilogue = [''] else: diff --git a/Lib/test/test_email/test_defect_handling.py b/Lib/test/test_email/test_defect_handling.py --- a/Lib/test/test_email/test_defect_handling.py +++ b/Lib/test/test_email/test_defect_handling.py @@ -278,6 +278,39 @@ with self.assertRaises(errors.InvalidBase64CharactersDefect): msg.get_payload(decode=True) + missing_ending_boundary = textwrap.dedent("""\ + To: 1 at harrydomain4.com + Subject: Fwd: 1 + MIME-Version: 1.0 + Content-Type: multipart/alternative; + boundary="------------000101020201080900040301" + + --------------000101020201080900040301 + Content-Type: text/plain; charset=ISO-8859-1 + Content-Transfer-Encoding: 7bit + + Alternative 1 + + --------------000101020201080900040301 + Content-Type: text/html; charset=ISO-8859-1 + Content-Transfer-Encoding: 7bit + + Alternative 2 + + """) + + def test_missing_ending_boundary(self): + msg = self._str_msg(self.missing_ending_boundary) + self.assertEqual(len(msg.get_payload()), 2) + self.assertEqual(msg.get_payload(1).get_payload(), 'Alternative 2\n') + self.assertDefectsEqual(self.get_defects(msg), + [errors.CloseBoundaryNotFoundDefect]) + + def test_missing_ending_boundary_raise_on_defect(self): + with self.assertRaises(errors.CloseBoundaryNotFoundDefect): + self._str_msg(self.missing_ending_boundary, + policy=self.policy.clone(raise_on_defect=True)) + class TestMessageDefectDetection(TestMessageDefectDetectionBase, TestEmailBase): diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,9 +49,16 @@ Library ------- +- Issue #12515: email now registers a defect if it gets to EOF while parsing + a MIME part without seeing the closing MIME boundary. + - Issue12510: Attempting to get invalid tooltip no longer closes Idle. Original patch by Roger Serwy. +- Issue #1672568: email now always decodes base64 payloads, adding padding and + ignoring non-base64-alphabet characters if needed, and registering defects + for any such problems. + - Issue #14925: email now registers a defect when the parser decides that there is a missing header/body separator line. MalformedHeaderDefect, which the existing code would never actually generate, is deprecated. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 05:06:40 2012 From: python-checkins at python.org (terry.reedy) Date: Mon, 28 May 2012 05:06:40 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Issue14929=3A_S?= =?utf8?q?top_Idle_3=2Ex_from_closing_on_Unicode_decode_errors_when_greppi?= =?utf8?b?bmcu?= Message-ID: http://hg.python.org/cpython/rev/12454f78967b changeset: 77199:12454f78967b branch: 3.2 parent: 77195:4a7582866735 user: Terry Jan Reedy date: Sun May 27 22:56:49 2012 -0400 summary: Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. Patch by Roger Serwy. files: Lib/idlelib/GrepDialog.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/GrepDialog.py b/Lib/idlelib/GrepDialog.py --- a/Lib/idlelib/GrepDialog.py +++ b/Lib/idlelib/GrepDialog.py @@ -81,7 +81,7 @@ hits = 0 for fn in list: try: - f = open(fn) + f = open(fn, errors='replace') except IOError as msg: print(msg) continue diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. + Patch by Roger Serwy. + - Issue12510: Attempting to get invalid tooltip no longer closes Idle. Original patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 05:06:41 2012 From: python-checkins at python.org (terry.reedy) Date: Mon, 28 May 2012 05:06:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_3=2E2_=2314929?= Message-ID: http://hg.python.org/cpython/rev/058c3099e82d changeset: 77200:058c3099e82d parent: 77198:81e008f13b4f parent: 77199:12454f78967b user: Terry Jan Reedy date: Sun May 27 23:06:14 2012 -0400 summary: Merge 3.2 #14929 files: Lib/idlelib/GrepDialog.py | 2 +- Misc/NEWS | 3 +++ 2 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/GrepDialog.py b/Lib/idlelib/GrepDialog.py --- a/Lib/idlelib/GrepDialog.py +++ b/Lib/idlelib/GrepDialog.py @@ -81,7 +81,7 @@ hits = 0 for fn in list: try: - f = open(fn) + f = open(fn, errors='replace') except IOError as msg: print(msg) continue diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,9 @@ Library ------- +- Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. + Patch by Roger Serwy. + - Issue #12515: email now registers a defect if it gets to EOF while parsing a MIME part without seeing the closing MIME boundary. -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Mon May 28 05:54:29 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Mon, 28 May 2012 05:54:29 +0200 Subject: [Python-checkins] Daily reference leaks (d68e30be755e): sum=467 Message-ID: results for d68e30be755e on branch "default" -------------------------------------------- test_dbm leaked [0, 0, 2] references, sum=2 test_smtplib leaked [154, 154, 154] references, sum=462 test_super leaked [1, 1, 1] references, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogioJY8a', '-x'] From python-checkins at python.org Mon May 28 07:35:46 2012 From: python-checkins at python.org (ned.deily) Date: Mon, 28 May 2012 07:35:46 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314660=3A_Install_n?= =?utf8?q?amespace=5Fpkgs_test_directories_and_files=2E?= Message-ID: http://hg.python.org/cpython/rev/f26721ab3476 changeset: 77201:f26721ab3476 user: Ned Deily date: Sun May 27 22:34:33 2012 -0700 summary: Issue #14660: Install namespace_pkgs test directories and files. files: Makefile.pre.in | 18 ++++++++++++++++++ 1 files changed, 18 insertions(+), 0 deletions(-) diff --git a/Makefile.pre.in b/Makefile.pre.in --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -974,6 +974,24 @@ test/cjkencodings test/decimaltestdata test/xmltestdata \ test/subprocessdata test/sndhdrdata \ test/tracedmodules test/encoded_modules \ + test/namespace_pkgs \ + test/namespace_pkgs/both_portions \ + test/namespace_pkgs/both_portions/foo \ + test/namespace_pkgs/not_a_namespace_pkg \ + test/namespace_pkgs/not_a_namespace_pkg/foo \ + test/namespace_pkgs/portion1 \ + test/namespace_pkgs/portion1/foo \ + test/namespace_pkgs/portion2 \ + test/namespace_pkgs/portion2/foo \ + test/namespace_pkgs/project1 \ + test/namespace_pkgs/project1/parent \ + test/namespace_pkgs/project1/parent/child \ + test/namespace_pkgs/project2 \ + test/namespace_pkgs/project2/parent \ + test/namespace_pkgs/project2/parent/child \ + test/namespace_pkgs/project3 \ + test/namespace_pkgs/project3/parent \ + test/namespace_pkgs/project3/parent/child \ collections concurrent concurrent/futures encodings \ email email/mime test/test_email test/test_email/data \ html json test/json_tests http dbm xmlrpc \ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 14:36:26 2012 From: python-checkins at python.org (nick.coghlan) Date: Mon, 28 May 2012 14:36:26 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0NDQz?= =?utf8?q?=3A_Tell_rpmbuild_to_use_the_correct_version_of_Python?= Message-ID: http://hg.python.org/cpython/rev/3d61e27cc570 changeset: 77202:3d61e27cc570 branch: 3.2 parent: 77199:12454f78967b user: Nick Coghlan date: Mon May 28 22:34:46 2012 +1000 summary: Issue #14443: Tell rpmbuild to use the correct version of Python files: Lib/distutils/command/bdist_rpm.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/command/bdist_rpm.py b/Lib/distutils/command/bdist_rpm.py --- a/Lib/distutils/command/bdist_rpm.py +++ b/Lib/distutils/command/bdist_rpm.py @@ -190,7 +190,7 @@ if self.fix_python: self.python = sys.executable else: - self.python = "python" + self.python = "python3" elif self.fix_python: raise DistutilsOptionError( "--python and --fix-python are mutually exclusive options") @@ -320,6 +320,7 @@ rpm_cmd.append('-bb') else: rpm_cmd.append('-ba') + rpm_cmd.extend(['--define', '__python %s' % self.python]) if self.rpm3_mode: rpm_cmd.extend(['--define', '_topdir %s' % os.path.abspath(self.rpm_base)]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14443: Tell rpmbuild to use the correct version of Python in + bdist_rpm. Initial patch by Ross Lagerwall. + - Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 14:36:26 2012 From: python-checkins at python.org (nick.coghlan) Date: Mon, 28 May 2012 14:36:26 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_from_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/cb716ee277cc changeset: 77203:cb716ee277cc parent: 77201:f26721ab3476 parent: 77202:3d61e27cc570 user: Nick Coghlan date: Mon May 28 22:36:07 2012 +1000 summary: Merge from 3.2 files: Lib/distutils/command/bdist_rpm.py | 3 ++- Misc/NEWS | 3 +++ 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/distutils/command/bdist_rpm.py b/Lib/distutils/command/bdist_rpm.py --- a/Lib/distutils/command/bdist_rpm.py +++ b/Lib/distutils/command/bdist_rpm.py @@ -190,7 +190,7 @@ if self.fix_python: self.python = sys.executable else: - self.python = "python" + self.python = "python3" elif self.fix_python: raise DistutilsOptionError( "--python and --fix-python are mutually exclusive options") @@ -320,6 +320,7 @@ rpm_cmd.append('-bb') else: rpm_cmd.append('-ba') + rpm_cmd.extend(['--define', '__python %s' % self.python]) if self.rpm3_mode: rpm_cmd.extend(['--define', '_topdir %s' % os.path.abspath(self.rpm_base)]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -49,6 +49,9 @@ Library ------- +- Issue #14443: Tell rpmbuild to use the correct version of Python in + bdist_rpm. Initial patch by Ross Lagerwall. + - Issue14929: Stop Idle 3.x from closing on Unicode decode errors when grepping. Patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 17:33:08 2012 From: python-checkins at python.org (vinay.sajip) Date: Mon, 28 May 2012 17:33:08 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Added_python3_symlink=3B_re?= =?utf8?q?moved_Distribute-related_code=2C_docs_and_comments=3B?= Message-ID: http://hg.python.org/cpython/rev/459821d578e1 changeset: 77204:459821d578e1 user: Vinay Sajip date: Mon May 28 16:33:01 2012 +0100 summary: Added python3 symlink; removed Distribute-related code, docs and comments; changed Mac OS X computation to determine framework builds. files: Lib/venv/__init__.py | 142 +++--------------------------- 1 files changed, 16 insertions(+), 126 deletions(-) diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -3,8 +3,8 @@ Copyright (C) 20011-2012 Vinay Sajip. All Rights Reserved. -usage: python -m venv [-h] [--no-distribute] [--system-site-packages] - [--symlinks] [--clear] [--upgrade] +usage: python -m venv [-h] [--system-site-packages] [--symlinks] [--clear] + [--upgrade] ENV_DIR [ENV_DIR ...] Creates virtual Python environments in one or more target directories. @@ -14,7 +14,6 @@ optional arguments: -h, --help show this help message and exit - --no-distribute Don't install Distribute in the virtual environment.* --system-site-packages Give the virtual environment access to the system site-packages dir. @@ -24,10 +23,6 @@ raised. --upgrade Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place. - -*Note: Distribute support will be available during the alpha phase to -facilitate testing third-party packages with venvs created using this package. -This support will be removed after the alpha phase. """ import base64 import io @@ -36,6 +31,7 @@ import os.path import shutil import sys +import sysconfig try: import threading except ImportError: @@ -88,11 +84,11 @@ """ if (self.symlinks and sys.platform == 'darwin' and - 'Library/Framework' in sys.base_prefix): + sysconfig.get_config_var('PYTHONFRAMEWORK')): # Symlinking the stub executable in an OSX framework build will # result in a broken virtual environment. raise ValueError( - "Symlinking is not supported on OSX framework Python.") + 'Symlinking is not supported on OSX framework Python.') env_dir = os.path.abspath(env_dir) context = self.ensure_directories(env_dir) self.create_configuration(context) @@ -206,9 +202,10 @@ if os.name != 'nt': if not os.path.islink(path): os.chmod(path, 0o755) - path = os.path.join(binpath, 'python') - if not os.path.exists(path): - os.symlink(exename, path) + for suffix in ('python', 'python3'): + path = os.path.join(binpath, suffix) + if not os.path.exists(path): + os.symlink(exename, path) else: subdir = 'DLLs' include = self.include_binary @@ -322,104 +319,6 @@ os.chmod(dstfile, 0o755) -# This class will not be included in Python core; it's here for now to -# facilitate experimentation and testing, and as proof-of-concept of what could -# be done by external extension tools. -class DistributeEnvBuilder(EnvBuilder): - """ - By default, this builder installs Distribute so that you can pip or - easy_install other packages into the created environment. - - :param nodist: If True, Distribute is not installed into the created - environment. - :param progress: If Distribute is installed, the progress of the - installation can be monitored by passing a progress - callable. If specified, it is called with two - arguments: a string indicating some progress, and a - context indicating where the string is coming from. - The context argument can have one of three values: - 'main', indicating that it is called from virtualize() - itself, and 'stdout' and 'stderr', which are obtained - by reading lines from the output streams of a subprocess - which is used to install Distribute. - - If a callable is not specified, default progress - information is output to sys.stderr. - """ - - def __init__(self, *args, **kwargs): - self.nodist = kwargs.pop("nodist", False) - self.progress = kwargs.pop("progress", None) - super().__init__(*args, **kwargs) - - def post_setup(self, context): - """ - Set up any packages which need to be pre-installed into the - environment being created. - - :param context: The information for the environment creation request - being processed. - """ - if not self.nodist: - if threading: - self.install_distribute(context) - - def reader(self, stream, context): - """ - Read lines from a subprocess' output stream and either pass to a progress - callable (if specified) or write progress information to sys.stderr. - """ - progress = self.progress - while True: - s = stream.readline() - if not s: - break - if progress is not None: - progress(s, context) - else: - sys.stderr.write('.') - #sys.stderr.write(s.decode('utf-8')) - sys.stderr.flush() - stream.close() - - def install_distribute(self, context): - """ - Install Distribute in the environment. - - :param context: The information for the environment creation request - being processed. - """ - from subprocess import Popen, PIPE - from urllib.request import urlretrieve - - url = 'http://python-distribute.org/distribute_setup.py' - binpath = context.bin_path - distpath = os.path.join(binpath, 'distribute_setup.py') - # Download Distribute in the env - urlretrieve(url, distpath) - progress = self.progress - if progress is not None: - progress('Installing distribute', 'main') - else: - sys.stderr.write('Installing distribute ') - sys.stderr.flush() - # Install Distribute in the env - args = [context.env_exe, 'distribute_setup.py'] - p = Popen(args, stdout=PIPE, stderr=PIPE, cwd=binpath) - t1 = threading.Thread(target=self.reader, args=(p.stdout, 'stdout')) - t1.start() - t2 = threading.Thread(target=self.reader, args=(p.stderr, 'stderr')) - t2.start() - p.wait() - t1.join() - t2.join() - if progress is not None: - progress('done.', 'main') - else: - sys.stderr.write('done.\n') - # Clean up - no longer needed - os.unlink(distpath) - def create(env_dir, system_site_packages=False, clear=False, symlinks=False): """ Create a virtual environment in a directory. @@ -436,8 +335,7 @@ :param symlinks: If True, attempt to symlink rather than copy files into virtual environment. """ - # XXX This should be changed to EnvBuilder. - builder = DistributeEnvBuilder(system_site_packages=system_site_packages, + builder = EnvBuilder(system_site_packages=system_site_packages, clear=clear, symlinks=symlinks) builder.create(env_dir) @@ -460,17 +358,12 @@ 'directories.') parser.add_argument('dirs', metavar='ENV_DIR', nargs='+', help='A directory to create the environment in.') - # XXX This option will be removed. - parser.add_argument('--no-distribute', default=False, - action='store_true', dest='nodist', - help="Don't install Distribute in the virtual " - "environment.") parser.add_argument('--system-site-packages', default=False, action='store_true', dest='system_site', - help="Give the virtual environment access to the " - "system site-packages dir. ") + help='Give the virtual environment access to the ' + 'system site-packages dir.') if os.name == 'nt' or (sys.platform == 'darwin' and - 'Library/Framework' in sys.base_prefix): + sysconfig.get_config_var('PYTHONFRAMEWORK')): use_symlinks = False else: use_symlinks = True @@ -491,12 +384,9 @@ options = parser.parse_args(args) if options.upgrade and options.clear: raise ValueError('you cannot supply --upgrade and --clear together.') - # XXX This will be changed to EnvBuilder - builder = DistributeEnvBuilder(system_site_packages=options.system_site, - clear=options.clear, - symlinks=options.symlinks, - upgrade=options.upgrade, - nodist=options.nodist) + builder = EnvBuilder(system_site_packages=options.system_site, + clear=options.clear, symlinks=options.symlinks, + upgrade=options.upgrade) for d in options.dirs: builder.create(d) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 17:34:53 2012 From: python-checkins at python.org (vinay.sajip) Date: Mon, 28 May 2012 17:34:53 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Tweaked_tests_to_use_launch?= =?utf8?q?er_executable_name_on_OS_X=2E?= Message-ID: http://hg.python.org/cpython/rev/ba24c81ac713 changeset: 77205:ba24c81ac713 user: Vinay Sajip date: Mon May 28 16:34:47 2012 +0100 summary: Tweaked tests to use launcher executable name on OS X. files: Lib/test/test_venv.py | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -29,7 +29,11 @@ self.ps3name = 'pysetup3' self.lib = ('lib', 'python%s' % sys.version[:3]) self.include = 'include' - self.exe = os.path.split(sys.executable)[-1] + if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env: + executable = os.environ['__PYTHONV_LAUNCHER__'] + else: + executable = sys.executable + self.exe = os.path.split(executable)[-1] def tearDown(self): shutil.rmtree(self.env_dir) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 18:01:39 2012 From: python-checkins at python.org (vinay.sajip) Date: Mon, 28 May 2012 18:01:39 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Fixed_typo=2E?= Message-ID: http://hg.python.org/cpython/rev/f4a7e738ad24 changeset: 77206:f4a7e738ad24 user: Vinay Sajip date: Mon May 28 17:01:17 2012 +0100 summary: Fixed typo. files: Lib/test/test_venv.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -29,7 +29,7 @@ self.ps3name = 'pysetup3' self.lib = ('lib', 'python%s' % sys.version[:3]) self.include = 'include' - if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env: + if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in os.environ: executable = os.environ['__PYTHONV_LAUNCHER__'] else: executable = sys.executable -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 21:52:25 2012 From: python-checkins at python.org (meador.inge) Date: Mon, 28 May 2012 21:52:25 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzkwNDE6?= =?utf8?q?_raised_exception_is_misleading?= Message-ID: http://hg.python.org/cpython/rev/8a65eceea56c changeset: 77207:8a65eceea56c branch: 2.7 parent: 77194:938b12452af7 user: Meador Inge date: Mon May 28 13:52:59 2012 -0500 summary: Issue #9041: raised exception is misleading An issue in ctypes.c_longdouble, ctypes.c_double, and ctypes.c_float that caused an incorrect exception to be returned in the case of overflow has been fixed. files: Lib/ctypes/test/test_numbers.py | 10 +++++++ Misc/NEWS | 4 ++ Modules/_ctypes/cfield.c | 30 +++----------------- 3 files changed, 19 insertions(+), 25 deletions(-) diff --git a/Lib/ctypes/test/test_numbers.py b/Lib/ctypes/test/test_numbers.py --- a/Lib/ctypes/test/test_numbers.py +++ b/Lib/ctypes/test/test_numbers.py @@ -216,6 +216,16 @@ # probably be changed: self.assertRaises(TypeError, c_int, c_long(42)) + def test_float_overflow(self): + import sys + big_int = int(sys.float_info.max) * 2 + for t in float_types + [c_longdouble]: + self.assertRaises(OverflowError, t, big_int) + if (hasattr(t, "__ctype_be__")): + self.assertRaises(OverflowError, t.__ctype_be__, big_int) + if (hasattr(t, "__ctype_le__")): + self.assertRaises(OverflowError, t.__ctype_le__, big_int) + ## def test_perf(self): ## check_perf() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -708,6 +708,10 @@ Extension Modules ----------------- +- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and + ctypes.c_float that caused an incorrect exception to be returned in the + case of overflow has been fixed. + - bsddb module: Erratic behaviour of "DBEnv->rep_elect()" because a typo. Possible crash. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1003,12 +1003,8 @@ long double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(long double)); _RET(value); } @@ -1027,12 +1023,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(double)); _RET(value); } @@ -1051,12 +1043,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1)) return NULL; @@ -1083,12 +1071,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(x)); _RET(value); } @@ -1107,12 +1091,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1)) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 21:52:26 2012 From: python-checkins at python.org (meador.inge) Date: Mon, 28 May 2012 21:52:26 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzkwNDE6?= =?utf8?q?_raised_exception_is_misleading?= Message-ID: http://hg.python.org/cpython/rev/a2e140b083e0 changeset: 77208:a2e140b083e0 branch: 3.2 parent: 77202:3d61e27cc570 user: Meador Inge date: Mon May 28 14:21:16 2012 -0500 summary: Issue #9041: raised exception is misleading An issue in ctypes.c_longdouble, ctypes.c_double, and ctypes.c_float that caused an incorrect exception to be returned in the case of overflow has been fixed. files: Lib/ctypes/test/test_numbers.py | 10 +++++++ Misc/NEWS | 4 ++ Modules/_ctypes/cfield.c | 30 +++----------------- 3 files changed, 19 insertions(+), 25 deletions(-) diff --git a/Lib/ctypes/test/test_numbers.py b/Lib/ctypes/test/test_numbers.py --- a/Lib/ctypes/test/test_numbers.py +++ b/Lib/ctypes/test/test_numbers.py @@ -217,6 +217,16 @@ # probably be changed: self.assertRaises(TypeError, c_int, c_long(42)) + def test_float_overflow(self): + import sys + big_int = int(sys.float_info.max) * 2 + for t in float_types + [c_longdouble]: + self.assertRaises(OverflowError, t, big_int) + if (hasattr(t, "__ctype_be__")): + self.assertRaises(OverflowError, t.__ctype_be__, big_int) + if (hasattr(t, "__ctype_le__")): + self.assertRaises(OverflowError, t.__ctype_le__, big_int) + ## def test_perf(self): ## check_perf() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -262,6 +262,10 @@ Extension Modules ----------------- +- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and + ctypes.c_float that caused an incorrect exception to be returned in the + case of overflow has been fixed. + - Issue #14212: The re module didn't retain a reference to buffers it was scanning, resulting in segfaults. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1002,12 +1002,8 @@ long double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(long double)); _RET(value); } @@ -1026,12 +1022,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(double)); _RET(value); } @@ -1050,12 +1042,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1)) return NULL; @@ -1082,12 +1070,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(x)); _RET(value); } @@ -1106,12 +1090,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1)) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 21:52:27 2012 From: python-checkins at python.org (meador.inge) Date: Mon, 28 May 2012 21:52:27 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=239041=3A_raised_exception_is_misleading?= Message-ID: http://hg.python.org/cpython/rev/7c7bbf4a70b7 changeset: 77209:7c7bbf4a70b7 parent: 77206:f4a7e738ad24 parent: 77208:a2e140b083e0 user: Meador Inge date: Mon May 28 14:47:53 2012 -0500 summary: Issue #9041: raised exception is misleading An issue in ctypes.c_longdouble, ctypes.c_double, and ctypes.c_float that caused an incorrect exception to be returned in the case of overflow has been fixed. files: Lib/ctypes/test/test_numbers.py | 10 +++++++ Misc/NEWS | 4 ++ Modules/_ctypes/cfield.c | 30 +++----------------- 3 files changed, 19 insertions(+), 25 deletions(-) diff --git a/Lib/ctypes/test/test_numbers.py b/Lib/ctypes/test/test_numbers.py --- a/Lib/ctypes/test/test_numbers.py +++ b/Lib/ctypes/test/test_numbers.py @@ -217,6 +217,16 @@ # probably be changed: self.assertRaises(TypeError, c_int, c_long(42)) + def test_float_overflow(self): + import sys + big_int = int(sys.float_info.max) * 2 + for t in float_types + [c_longdouble]: + self.assertRaises(OverflowError, t, big_int) + if (hasattr(t, "__ctype_be__")): + self.assertRaises(OverflowError, t.__ctype_be__, big_int) + if (hasattr(t, "__ctype_le__")): + self.assertRaises(OverflowError, t.__ctype_le__, big_int) + ## def test_perf(self): ## check_perf() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -714,6 +714,10 @@ Extension Modules ----------------- +- Issue #9041: An issue in ctypes.c_longdouble, ctypes.c_double, and + ctypes.c_float that caused an incorrect exception to be returned in the + case of overflow has been fixed. + - Issue #14212: The re module didn't retain a reference to buffers it was scanning, resulting in segfaults. diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -999,12 +999,8 @@ long double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(long double)); _RET(value); } @@ -1023,12 +1019,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(double)); _RET(value); } @@ -1047,12 +1039,8 @@ double x; x = PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack8(x, (unsigned char *)ptr, 1)) return NULL; @@ -1079,12 +1067,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } memcpy(ptr, &x, sizeof(x)); _RET(value); } @@ -1103,12 +1087,8 @@ float x; x = (float)PyFloat_AsDouble(value); - if (x == -1 && PyErr_Occurred()) { - PyErr_Format(PyExc_TypeError, - " float expected instead of %s instance", - value->ob_type->tp_name); + if (x == -1 && PyErr_Occurred()) return NULL; - } #ifdef WORDS_BIGENDIAN if (_PyFloat_Pack4(x, (unsigned char *)ptr, 1)) return NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 22:31:23 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 28 May 2012 22:31:23 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0Nzc1?= =?utf8?q?=3A_Fix_a_potential_quadratic_dict_build-up_due_to_the_garbage?= Message-ID: http://hg.python.org/cpython/rev/47e6217d0e84 changeset: 77210:47e6217d0e84 branch: 3.2 parent: 77208:a2e140b083e0 user: Antoine Pitrou date: Mon May 28 22:22:34 2012 +0200 summary: Issue #14775: Fix a potential quadratic dict build-up due to the garbage collector repeatedly trying to untrack dicts. Additional comments by Tim Silk. files: Misc/ACKS | 1 + Misc/NEWS | 3 + Modules/gcmodule.c | 60 ++++++++++++++++++++++++++++++++- 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -866,6 +866,7 @@ Itamar Shtull-Trauring Eric Siegerman Paul Sijben +Tim Silk Kirill Simonov Nathan Paul Simons Janne Sinkkonen diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14775: Fix a potential quadratic dict build-up due to the garbage + collector repeatedly trying to untrack dicts. + - Issue #14494: Fix __future__.py and its documentation to note that absolute imports are the default behavior in 3.0 instead of 2.7. Patch by Sven Marnach. diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -116,6 +116,46 @@ http://mail.python.org/pipermail/python-dev/2008-June/080579.html */ +/* + NOTE: about untracking of mutable objects. + + Certain types of container cannot participate in a reference cycle, and + so do not need to be tracked by the garbage collector. Untracking these + objects reduces the cost of garbage collections. However, determining + which objects may be untracked is not free, and the costs must be + weighed against the benefits for garbage collection. + + There are two possible strategies for when to untrack a container: + + i) When the container is created. + ii) When the container is examined by the garbage collector. + + Tuples containing only immutable objects (integers, strings etc, and + recursively, tuples of immutable objects) do not need to be tracked. + The interpreter creates a large number of tuples, many of which will + not survive until garbage collection. It is therefore not worthwhile + to untrack eligible tuples at creation time. + + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage + collection cycles. It may take more than one cycle to untrack a tuple. + + Dictionaries containing only immutable objects also do not need to be + tracked. Dictionaries are untracked when created. If a tracked item is + inserted into a dictionary (either as a key or value), the dictionary + becomes tracked. During a full garbage collection (all generations), + the collector will untrack any dictionaries whose contents are not + tracked. + + The module provides the python function is_tracked(obj), which returns + the CURRENT tracking status of the object. Subsequent garbage + collections may change the tracking status of the object. + + Untracking of certain containers was introduced in issue #4688, and + the algorithm was refined in response to issue #14775. +*/ /* set for debugging information */ #define DEBUG_STATS (1<<0) /* print collection statistics */ @@ -437,9 +477,6 @@ if (PyTuple_CheckExact(op)) { _PyTuple_MaybeUntrack(op); } - else if (PyDict_CheckExact(op)) { - _PyDict_MaybeUntrack(op); - } } else { /* This *may* be unreachable. To make progress, @@ -457,6 +494,20 @@ } } +/* Try to untrack all currently tracked dictionaries */ +static void +untrack_dicts(PyGC_Head *head) +{ + PyGC_Head *next, *gc = head->gc.gc_next; + while (gc != head) { + PyObject *op = FROM_GC(gc); + next = gc->gc.gc_next; + if (PyDict_CheckExact(op)) + _PyDict_MaybeUntrack(op); + gc = next; + } +} + /* Return true if object has a finalization method. */ static int has_finalizer(PyObject *op) @@ -857,6 +908,9 @@ gc_list_merge(young, old); } else { + /* We only untrack dicts in full collections, to avoid quadratic + dict build-up. See issue #14775. */ + untrack_dicts(young); long_lived_pending = 0; long_lived_total = gc_list_size(young); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 22:31:24 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 28 May 2012 22:31:24 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314775=3A_Fix_a_potential_quadratic_dict_build-up_du?= =?utf8?q?e_to_the_garbage?= Message-ID: http://hg.python.org/cpython/rev/7951900afd00 changeset: 77211:7951900afd00 parent: 77209:7c7bbf4a70b7 parent: 77210:47e6217d0e84 user: Antoine Pitrou date: Mon May 28 22:23:42 2012 +0200 summary: Issue #14775: Fix a potential quadratic dict build-up due to the garbage collector repeatedly trying to untrack dicts. Additional comments by Tim Silk. files: Misc/ACKS | 1 + Misc/NEWS | 3 + Modules/gcmodule.c | 60 ++++++++++++++++++++++++++++++++- 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -957,6 +957,7 @@ Itamar Shtull-Trauring Eric Siegerman Paul Sijben +Tim Silk Kirill Simonov Nathan Paul Simons Adam Simpkins diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14775: Fix a potential quadratic dict build-up due to the garbage + collector repeatedly trying to untrack dicts. + - Issue #14857: fix regression in references to PEP 3135 implicit __class__ closure variable (Reopens issue #12370) diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -116,6 +116,46 @@ http://mail.python.org/pipermail/python-dev/2008-June/080579.html */ +/* + NOTE: about untracking of mutable objects. + + Certain types of container cannot participate in a reference cycle, and + so do not need to be tracked by the garbage collector. Untracking these + objects reduces the cost of garbage collections. However, determining + which objects may be untracked is not free, and the costs must be + weighed against the benefits for garbage collection. + + There are two possible strategies for when to untrack a container: + + i) When the container is created. + ii) When the container is examined by the garbage collector. + + Tuples containing only immutable objects (integers, strings etc, and + recursively, tuples of immutable objects) do not need to be tracked. + The interpreter creates a large number of tuples, many of which will + not survive until garbage collection. It is therefore not worthwhile + to untrack eligible tuples at creation time. + + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage + collection cycles. It may take more than one cycle to untrack a tuple. + + Dictionaries containing only immutable objects also do not need to be + tracked. Dictionaries are untracked when created. If a tracked item is + inserted into a dictionary (either as a key or value), the dictionary + becomes tracked. During a full garbage collection (all generations), + the collector will untrack any dictionaries whose contents are not + tracked. + + The module provides the python function is_tracked(obj), which returns + the CURRENT tracking status of the object. Subsequent garbage + collections may change the tracking status of the object. + + Untracking of certain containers was introduced in issue #4688, and + the algorithm was refined in response to issue #14775. +*/ /* set for debugging information */ #define DEBUG_STATS (1<<0) /* print collection statistics */ @@ -437,9 +477,6 @@ if (PyTuple_CheckExact(op)) { _PyTuple_MaybeUntrack(op); } - else if (PyDict_CheckExact(op)) { - _PyDict_MaybeUntrack(op); - } } else { /* This *may* be unreachable. To make progress, @@ -457,6 +494,20 @@ } } +/* Try to untrack all currently tracked dictionaries */ +static void +untrack_dicts(PyGC_Head *head) +{ + PyGC_Head *next, *gc = head->gc.gc_next; + while (gc != head) { + PyObject *op = FROM_GC(gc); + next = gc->gc.gc_next; + if (PyDict_CheckExact(op)) + _PyDict_MaybeUntrack(op); + gc = next; + } +} + /* Return true if object has a finalization method. */ static int has_finalizer(PyObject *op) @@ -856,6 +907,9 @@ gc_list_merge(young, old); } else { + /* We only untrack dicts in full collections, to avoid quadratic + dict build-up. See issue #14775. */ + untrack_dicts(young); long_lived_pending = 0; long_lived_total = gc_list_size(young); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 22:31:25 2012 From: python-checkins at python.org (antoine.pitrou) Date: Mon, 28 May 2012 22:31:25 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0Nzc1?= =?utf8?q?=3A_Fix_a_potential_quadratic_dict_build-up_due_to_the_garbage?= Message-ID: http://hg.python.org/cpython/rev/6554ebbeb2f3 changeset: 77212:6554ebbeb2f3 branch: 2.7 parent: 77207:8a65eceea56c user: Antoine Pitrou date: Mon May 28 22:22:34 2012 +0200 summary: Issue #14775: Fix a potential quadratic dict build-up due to the garbage collector repeatedly trying to untrack dicts. Additional comments by Tim Silk. files: Misc/ACKS | 1 + Misc/NEWS | 3 + Modules/gcmodule.c | 60 ++++++++++++++++++++++++++++++++- 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -791,6 +791,7 @@ Itamar Shtull-Trauring Eric Siegerman Paul Sijben +Tim Silk Kirill Simonov Nathan Paul Simons Janne Sinkkonen diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -9,6 +9,9 @@ Core and Builtins ----------------- +- Issue #14775: Fix a potential quadratic dict build-up due to the garbage + collector repeatedly trying to untrack dicts. + - Issue #14494: Fix __future__.py and its documentation to note that absolute imports are the default behavior in 3.0 instead of 2.7. Patch by Sven Marnach. diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -111,6 +111,46 @@ http://mail.python.org/pipermail/python-dev/2008-June/080579.html */ +/* + NOTE: about untracking of mutable objects. + + Certain types of container cannot participate in a reference cycle, and + so do not need to be tracked by the garbage collector. Untracking these + objects reduces the cost of garbage collections. However, determining + which objects may be untracked is not free, and the costs must be + weighed against the benefits for garbage collection. + + There are two possible strategies for when to untrack a container: + + i) When the container is created. + ii) When the container is examined by the garbage collector. + + Tuples containing only immutable objects (integers, strings etc, and + recursively, tuples of immutable objects) do not need to be tracked. + The interpreter creates a large number of tuples, many of which will + not survive until garbage collection. It is therefore not worthwhile + to untrack eligible tuples at creation time. + + Instead, all tuples except the empty tuple are tracked when created. + During garbage collection it is determined whether any surviving tuples + can be untracked. A tuple can be untracked if all of its contents are + already not tracked. Tuples are examined for untracking in all garbage + collection cycles. It may take more than one cycle to untrack a tuple. + + Dictionaries containing only immutable objects also do not need to be + tracked. Dictionaries are untracked when created. If a tracked item is + inserted into a dictionary (either as a key or value), the dictionary + becomes tracked. During a full garbage collection (all generations), + the collector will untrack any dictionaries whose contents are not + tracked. + + The module provides the python function is_tracked(obj), which returns + the CURRENT tracking status of the object. Subsequent garbage + collections may change the tracking status of the object. + + Untracking of certain containers was introduced in issue #4688, and + the algorithm was refined in response to issue #14775. +*/ /* set for debugging information */ #define DEBUG_STATS (1<<0) /* print collection statistics */ @@ -436,9 +476,6 @@ if (PyTuple_CheckExact(op)) { _PyTuple_MaybeUntrack(op); } - else if (PyDict_CheckExact(op)) { - _PyDict_MaybeUntrack(op); - } } else { /* This *may* be unreachable. To make progress, @@ -478,6 +515,20 @@ return 0; } +/* Try to untrack all currently tracked dictionaries */ +static void +untrack_dicts(PyGC_Head *head) +{ + PyGC_Head *next, *gc = head->gc.gc_next; + while (gc != head) { + PyObject *op = FROM_GC(gc); + next = gc->gc.gc_next; + if (PyDict_CheckExact(op)) + _PyDict_MaybeUntrack(op); + gc = next; + } +} + /* Move the objects in unreachable with __del__ methods into `finalizers`. * Objects moved into `finalizers` have gc_refs set to GC_REACHABLE; the * objects remaining in unreachable are left at GC_TENTATIVELY_UNREACHABLE. @@ -890,6 +941,9 @@ gc_list_merge(young, old); } else { + /* We only untrack dicts in full collections, to avoid quadratic + dict build-up. See issue #14775. */ + untrack_dicts(young); long_lived_pending = 0; long_lived_total = gc_list_size(young); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Mon May 28 22:36:41 2012 From: python-checkins at python.org (richard.oudkerk) Date: Mon, 28 May 2012 22:36:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314930=3A_Make_memo?= =?utf8?q?ryview_objects_weakrefable=2E?= Message-ID: http://hg.python.org/cpython/rev/bc0281f85409 changeset: 77213:bc0281f85409 parent: 77211:7951900afd00 user: Richard Oudkerk date: Mon May 28 21:35:09 2012 +0100 summary: Issue #14930: Make memoryview objects weakrefable. files: Include/memoryobject.h | 1 + Lib/test/test_memoryview.py | 15 +++++++++++++++ Lib/test/test_sys.py | 2 +- Misc/NEWS | 2 ++ Objects/memoryobject.c | 5 ++++- 5 files changed, 23 insertions(+), 2 deletions(-) diff --git a/Include/memoryobject.h b/Include/memoryobject.h --- a/Include/memoryobject.h +++ b/Include/memoryobject.h @@ -63,6 +63,7 @@ Py_ssize_t exports; /* number of buffer re-exports */ Py_buffer view; /* private copy of the exporter's view */ char format[_Py_MEMORYVIEW_MAX_FORMAT]; /* used for casting */ + PyObject *weakreflist; Py_ssize_t ob_array[1]; /* shape, strides, suboffsets */ } PyMemoryViewObject; #endif diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py --- a/Lib/test/test_memoryview.py +++ b/Lib/test/test_memoryview.py @@ -336,6 +336,21 @@ m = self._view(b) self.assertRaises(ValueError, hash, m) + def test_weakref(self): + # Check memoryviews are weakrefable + for tp in self._types: + b = tp(self._source) + m = self._view(b) + L = [] + def callback(wr, b=b): + L.append(b) + wr = weakref.ref(m, callback) + self.assertIs(wr(), m) + del m + test.support.gc_collect() + self.assertIs(wr(), None) + self.assertIs(L[0], b) + # Variations on source objects for the buffer: bytes-like objects, then arrays # with itemsize > 1. # NOTE: support for multi-dimensional objects is unimplemented. diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -778,7 +778,7 @@ check(int(PyLong_BASE**2-1), size(vh) + 2*self.longdigit) check(int(PyLong_BASE**2), size(vh) + 3*self.longdigit) # memoryview - check(memoryview(b''), size(h + 'PPiP4P2i5P3cP')) + check(memoryview(b''), size(h + 'PPiP4P2i5P3c2P')) # module check(unittest, size(h + '3P')) # None diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,8 @@ Core and Builtins ----------------- +- Issue #14930: Make memoryview objects weakrefable. + - Issue #14775: Fix a potential quadratic dict build-up due to the garbage collector repeatedly trying to untrack dicts. diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -595,6 +595,7 @@ mv->view.shape = mv->ob_array; mv->view.strides = mv->ob_array + ndim; mv->view.suboffsets = mv->ob_array + 2 * ndim; + mv->weakreflist = NULL; _PyObject_GC_TRACK(mv); return mv; @@ -969,6 +970,8 @@ _PyObject_GC_UNTRACK(self); (void)_memory_release(self); Py_CLEAR(self->mbuf); + if (self->weakreflist != NULL) + PyObject_ClearWeakRefs((PyObject *) self); PyObject_GC_Del(self); } @@ -2608,7 +2611,7 @@ (traverseproc)memory_traverse, /* tp_traverse */ (inquiry)memory_clear, /* tp_clear */ memory_richcompare, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ + offsetof(PyMemoryViewObject, weakreflist),/* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ memory_methods, /* tp_methods */ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 02:14:29 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 29 May 2012 02:14:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Refactor_test=5Femail/test?= =?utf8?q?=5Fdefect=5Fhandling=2E?= Message-ID: http://hg.python.org/cpython/rev/fe426268eea6 changeset: 77214:fe426268eea6 user: R David Murray date: Mon May 28 20:14:10 2012 -0400 summary: Refactor test_email/test_defect_handling. files: Lib/test/test_email/test_defect_handling.py | 355 ++++----- Lib/test/test_email/test_email.py | 2 +- 2 files changed, 170 insertions(+), 187 deletions(-) diff --git a/Lib/test/test_email/test_defect_handling.py b/Lib/test/test_email/test_defect_handling.py --- a/Lib/test/test_email/test_defect_handling.py +++ b/Lib/test/test_email/test_defect_handling.py @@ -1,84 +1,89 @@ import textwrap import unittest -from email._policybase import Compat32 +import contextlib +from email import policy from email import errors from test.test_email import TestEmailBase -class TestMessageDefectDetectionBase: +class TestDefectsBase: - dup_boundary_msg = textwrap.dedent("""\ - Subject: XX - From: xx at xx.dk - To: XX - Mime-version: 1.0 - Content-type: multipart/mixed; - boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + policy = policy.default + raise_expected = False - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: multipart/alternative; - boundary="MS_Mac_OE_3071477847_720252_MIME_Part" - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: text/plain; charset="ISO-8859-1" - Content-transfer-encoding: quoted-printable - - text - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: text/html; charset="ISO-8859-1" - Content-transfer-encoding: quoted-printable - - - - --MS_Mac_OE_3071477847_720252_MIME_Part-- - - --MS_Mac_OE_3071477847_720252_MIME_Part - Content-type: image/gif; name="xx.gif"; - Content-disposition: attachment - Content-transfer-encoding: base64 - - Some removed base64 encoded chars. - - --MS_Mac_OE_3071477847_720252_MIME_Part-- - - """) + @contextlib.contextmanager + def _raise_point(self, defect): + yield def test_same_boundary_inner_outer(self): + source = textwrap.dedent("""\ + Subject: XX + From: xx at xx.dk + To: XX + Mime-version: 1.0 + Content-type: multipart/mixed; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: multipart/alternative; + boundary="MS_Mac_OE_3071477847_720252_MIME_Part" + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/plain; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + text + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: text/html; charset="ISO-8859-1" + Content-transfer-encoding: quoted-printable + + + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + --MS_Mac_OE_3071477847_720252_MIME_Part + Content-type: image/gif; name="xx.gif"; + Content-disposition: attachment + Content-transfer-encoding: base64 + + Some removed base64 encoded chars. + + --MS_Mac_OE_3071477847_720252_MIME_Part-- + + """) # XXX better would be to actually detect the duplicate. - msg = self._str_msg(self.dup_boundary_msg) + with self._raise_point(errors.StartBoundaryNotFoundDefect): + msg = self._str_msg(source) + if self.raise_expected: return inner = msg.get_payload(0) self.assertTrue(hasattr(inner, 'defects')) self.assertEqual(len(self.get_defects(inner)), 1) self.assertTrue(isinstance(self.get_defects(inner)[0], errors.StartBoundaryNotFoundDefect)) - def test_same_boundary_inner_outer_raises_on_defect(self): - with self.assertRaises(errors.StartBoundaryNotFoundDefect): - self._str_msg(self.dup_boundary_msg, - policy=self.policy.clone(raise_on_defect=True)) + def test_multipart_no_boundary(self): + source = textwrap.dedent("""\ + Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) + From: foobar + Subject: broken mail + MIME-Version: 1.0 + Content-Type: multipart/report; report-type=delivery-status; - no_boundary_msg = textwrap.dedent("""\ - Date: Fri, 6 Apr 2001 09:23:06 -0800 (GMT-0800) - From: foobar - Subject: broken mail - MIME-Version: 1.0 - Content-Type: multipart/report; report-type=delivery-status; + --JAB03225.986577786/zinfandel.lacita.com - --JAB03225.986577786/zinfandel.lacita.com + One part - One part + --JAB03225.986577786/zinfandel.lacita.com + Content-Type: message/delivery-status - --JAB03225.986577786/zinfandel.lacita.com - Content-Type: message/delivery-status + Header: Another part - Header: Another part - - --JAB03225.986577786/zinfandel.lacita.com-- - """) - - def test_multipart_no_boundary(self): - msg = self._str_msg(self.no_boundary_msg) + --JAB03225.986577786/zinfandel.lacita.com-- + """) + with self._raise_point(errors.NoBoundaryInMultipartDefect): + msg = self._str_msg(source) + if self.raise_expected: return self.assertTrue(isinstance(msg.get_payload(), str)) self.assertEqual(len(self.get_defects(msg)), 2) self.assertTrue(isinstance(self.get_defects(msg)[0], @@ -86,11 +91,6 @@ self.assertTrue(isinstance(self.get_defects(msg)[1], errors.MultipartInvariantViolationDefect)) - def test_multipart_no_boundary_raise_on_defect(self): - with self.assertRaises(errors.NoBoundaryInMultipartDefect): - self._str_msg(self.no_boundary_msg, - policy=self.policy.clone(raise_on_defect=True)) - multipart_msg = textwrap.dedent("""\ Date: Wed, 14 Nov 2007 12:56:23 GMT From: foo at bar.invalid @@ -115,43 +115,42 @@ """) def test_multipart_invalid_cte(self): - msg = self._str_msg( - self.multipart_msg.format("\nContent-Transfer-Encoding: base64")) + with self._raise_point( + errors.InvalidMultipartContentTransferEncodingDefect): + msg = self._str_msg( + self.multipart_msg.format( + "\nContent-Transfer-Encoding: base64")) + if self.raise_expected: return self.assertEqual(len(self.get_defects(msg)), 1) self.assertIsInstance(self.get_defects(msg)[0], errors.InvalidMultipartContentTransferEncodingDefect) - def test_multipart_invalid_cte_raise_on_defect(self): - with self.assertRaises( - errors.InvalidMultipartContentTransferEncodingDefect): - self._str_msg( - self.multipart_msg.format( - "\nContent-Transfer-Encoding: base64"), - policy=self.policy.clone(raise_on_defect=True)) - def test_multipart_no_cte_no_defect(self): + if self.raise_expected: return msg = self._str_msg(self.multipart_msg.format('')) self.assertEqual(len(self.get_defects(msg)), 0) def test_multipart_valid_cte_no_defect(self): + if self.raise_expected: return for cte in ('7bit', '8bit', 'BINary'): msg = self._str_msg( self.multipart_msg.format("\nContent-Transfer-Encoding: "+cte)) self.assertEqual(len(self.get_defects(msg)), 0, "cte="+cte) - lying_multipart_msg = textwrap.dedent("""\ - From: "Allison Dunlap" - To: yyy at example.com - Subject: 64423 - Date: Sun, 11 Jul 2004 16:09:27 -0300 - MIME-Version: 1.0 - Content-Type: multipart/alternative; + def test_lying_multipart(self): + source = textwrap.dedent("""\ + From: "Allison Dunlap" + To: yyy at example.com + Subject: 64423 + Date: Sun, 11 Jul 2004 16:09:27 -0300 + MIME-Version: 1.0 + Content-Type: multipart/alternative; - Blah blah blah - """) - - def test_lying_multipart(self): - msg = self._str_msg(self.lying_multipart_msg) + Blah blah blah + """) + with self._raise_point(errors.NoBoundaryInMultipartDefect): + msg = self._str_msg(source) + if self.raise_expected: return self.assertTrue(hasattr(msg, 'defects')) self.assertEqual(len(self.get_defects(msg)), 2) self.assertTrue(isinstance(self.get_defects(msg)[0], @@ -159,34 +158,28 @@ self.assertTrue(isinstance(self.get_defects(msg)[1], errors.MultipartInvariantViolationDefect)) - def test_lying_multipart_raise_on_defect(self): - with self.assertRaises(errors.NoBoundaryInMultipartDefect): - self._str_msg(self.lying_multipart_msg, - policy=self.policy.clone(raise_on_defect=True)) + def test_missing_start_boundary(self): + source = textwrap.dedent("""\ + Content-Type: multipart/mixed; boundary="AAA" + From: Mail Delivery Subsystem + To: yyy at example.com - missing_start_boundary_msg = textwrap.dedent("""\ - Content-Type: multipart/mixed; boundary="AAA" - From: Mail Delivery Subsystem - To: yyy at example.com + --AAA - --AAA + Stuff - Stuff + --AAA + Content-Type: message/rfc822 - --AAA - Content-Type: message/rfc822 + From: webmaster at python.org + To: zzz at example.com + Content-Type: multipart/mixed; boundary="BBB" - From: webmaster at python.org - To: zzz at example.com - Content-Type: multipart/mixed; boundary="BBB" + --BBB-- - --BBB-- + --AAA-- - --AAA-- - - """) - - def test_missing_start_boundary(self): + """) # The message structure is: # # multipart/mixed @@ -195,19 +188,18 @@ # multipart/mixed [*] # # [*] This message is missing its start boundary - outer = self._str_msg(self.missing_start_boundary_msg) + with self._raise_point(errors.StartBoundaryNotFoundDefect): + outer = self._str_msg(source) + if self.raise_expected: return bad = outer.get_payload(1).get_payload(0) self.assertEqual(len(self.get_defects(bad)), 1) self.assertTrue(isinstance(self.get_defects(bad)[0], errors.StartBoundaryNotFoundDefect)) - def test_missing_start_boundary_raise_on_defect(self): - with self.assertRaises(errors.StartBoundaryNotFoundDefect): - self._str_msg(self.missing_start_boundary_msg, - policy=self.policy.clone(raise_on_defect=True)) - def test_first_line_is_continuation_header(self): - msg = self._str_msg(' Line 1\nSubject: test\n\nbody') + with self._raise_point(errors.FirstHeaderLineIsContinuationDefect): + msg = self._str_msg(' Line 1\nSubject: test\n\nbody') + if self.raise_expected: return self.assertEqual(msg.keys(), ['Subject']) self.assertEqual(msg.get_payload(), 'body') self.assertEqual(len(self.get_defects(msg)), 1) @@ -215,113 +207,92 @@ [errors.FirstHeaderLineIsContinuationDefect]) self.assertEqual(self.get_defects(msg)[0].line, ' Line 1\n') - def test_first_line_is_continuation_header_raise_on_defect(self): - with self.assertRaises(errors.FirstHeaderLineIsContinuationDefect): - self._str_msg(' Line 1\nSubject: test\n\nbody\n', - policy=self.policy.clone(raise_on_defect=True)) - def test_missing_header_body_separator(self): # Our heuristic if we see a line that doesn't look like a header (no # leading whitespace but no ':') is to assume that the blank line that # separates the header from the body is missing, and to stop parsing # headers and start parsing the body. - msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + with self._raise_point(errors.MissingHeaderBodySeparatorDefect): + msg = self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n') + if self.raise_expected: return self.assertEqual(msg.keys(), ['Subject']) self.assertEqual(msg.get_payload(), 'not a header\nTo: abc\n\nb\n') self.assertDefectsEqual(self.get_defects(msg), [errors.MissingHeaderBodySeparatorDefect]) - def test_missing_header_body_separator_raise_on_defect(self): - with self.assertRaises(errors.MissingHeaderBodySeparatorDefect): - self._str_msg('Subject: test\nnot a header\nTo: abc\n\nb\n', - policy=self.policy.clone(raise_on_defect=True)) + def test_bad_padding_in_base64_payload(self): + source = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 - badly_padded_base64_payload = textwrap.dedent("""\ - Subject: test - MIME-Version: 1.0 - Content-Type: text/plain; charset="utf-8" - Content-Transfer-Encoding: base64 - - dmk - """) - - def test_bad_padding_in_base64_payload(self): - msg = self._str_msg(self.badly_padded_base64_payload) - self.assertEqual(msg.get_payload(decode=True), b'vi') + dmk + """) + msg = self._str_msg(source) + with self._raise_point(errors.InvalidBase64PaddingDefect): + payload = msg.get_payload(decode=True) + if self.raise_expected: return + self.assertEqual(payload, b'vi') self.assertDefectsEqual(self.get_defects(msg), [errors.InvalidBase64PaddingDefect]) - def test_bad_padding_in_base64_payload_raise_on_defect(self): - msg = self._str_msg(self.badly_padded_base64_payload, - policy=self.policy.clone(raise_on_defect=True)) - with self.assertRaises(errors.InvalidBase64PaddingDefect): - msg.get_payload(decode=True) + def test_invalid_chars_in_base64_payload(self): + source = textwrap.dedent("""\ + Subject: test + MIME-Version: 1.0 + Content-Type: text/plain; charset="utf-8" + Content-Transfer-Encoding: base64 - invalid_chars_in_base64_payload = textwrap.dedent("""\ - Subject: test - MIME-Version: 1.0 - Content-Type: text/plain; charset="utf-8" - Content-Transfer-Encoding: base64 - - dm\x01k=== - """) - - def test_invalid_chars_in_base64_payload(self): - msg = self._str_msg(self.invalid_chars_in_base64_payload) - self.assertEqual(msg.get_payload(decode=True), b'vi') + dm\x01k=== + """) + msg = self._str_msg(source) + with self._raise_point(errors.InvalidBase64CharactersDefect): + payload = msg.get_payload(decode=True) + if self.raise_expected: return + self.assertEqual(payload, b'vi') self.assertDefectsEqual(self.get_defects(msg), [errors.InvalidBase64CharactersDefect]) - def test_invalid_chars_in_base64_payload_raise_on_defect(self): - msg = self._str_msg(self.invalid_chars_in_base64_payload, - policy=self.policy.clone(raise_on_defect=True)) - with self.assertRaises(errors.InvalidBase64CharactersDefect): - msg.get_payload(decode=True) + def test_missing_ending_boundary(self): + source = textwrap.dedent("""\ + To: 1 at harrydomain4.com + Subject: Fwd: 1 + MIME-Version: 1.0 + Content-Type: multipart/alternative; + boundary="------------000101020201080900040301" - missing_ending_boundary = textwrap.dedent("""\ - To: 1 at harrydomain4.com - Subject: Fwd: 1 - MIME-Version: 1.0 - Content-Type: multipart/alternative; - boundary="------------000101020201080900040301" + --------------000101020201080900040301 + Content-Type: text/plain; charset=ISO-8859-1 + Content-Transfer-Encoding: 7bit - --------------000101020201080900040301 - Content-Type: text/plain; charset=ISO-8859-1 - Content-Transfer-Encoding: 7bit + Alternative 1 - Alternative 1 + --------------000101020201080900040301 + Content-Type: text/html; charset=ISO-8859-1 + Content-Transfer-Encoding: 7bit - --------------000101020201080900040301 - Content-Type: text/html; charset=ISO-8859-1 - Content-Transfer-Encoding: 7bit + Alternative 2 - Alternative 2 - - """) - - def test_missing_ending_boundary(self): - msg = self._str_msg(self.missing_ending_boundary) + """) + with self._raise_point(errors.CloseBoundaryNotFoundDefect): + msg = self._str_msg(source) + if self.raise_expected: return self.assertEqual(len(msg.get_payload()), 2) self.assertEqual(msg.get_payload(1).get_payload(), 'Alternative 2\n') self.assertDefectsEqual(self.get_defects(msg), [errors.CloseBoundaryNotFoundDefect]) - def test_missing_ending_boundary_raise_on_defect(self): - with self.assertRaises(errors.CloseBoundaryNotFoundDefect): - self._str_msg(self.missing_ending_boundary, - policy=self.policy.clone(raise_on_defect=True)) - -class TestMessageDefectDetection(TestMessageDefectDetectionBase, TestEmailBase): +class TestDefectDetection(TestDefectsBase, TestEmailBase): def get_defects(self, obj): return obj.defects -class TestMessageDefectDetectionCapture(TestMessageDefectDetectionBase, - TestEmailBase): +class TestDefectCapture(TestDefectsBase, TestEmailBase): - class CapturePolicy(Compat32): + class CapturePolicy(policy.EmailPolicy): captured = None def register_defect(self, obj, defect): self.captured.append(defect) @@ -333,5 +304,17 @@ return self.policy.captured +class TestDefectRaising(TestDefectsBase, TestEmailBase): + + policy = TestDefectsBase.policy + policy = policy.clone(raise_on_defect=True) + raise_expected = True + + @contextlib.contextmanager + def _raise_point(self, defect): + with self.assertRaises(defect): + yield + + if __name__ == '__main__': unittest.main() diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -1973,7 +1973,7 @@ [errors.FirstHeaderLineIsContinuationDefect]) eq(msg.defects[0].line, ' Line 1\n') - # test_parser.TestMessageDefectDetectionBase + # test_defect_handling def test_missing_header_body_separator(self): # Our heuristic if we see a line that doesn't look like a header (no # leading whitespace but no ':') is to assume that the blank line that -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 02:22:56 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 29 May 2012 02:22:56 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Regularize_test=5Femail/tes?= =?utf8?q?t=5Fheaderregistry=27s_references_to_policy=2E?= Message-ID: http://hg.python.org/cpython/rev/7c70eb6537b8 changeset: 77215:7c70eb6537b8 user: R David Murray date: Mon May 28 20:22:37 2012 -0400 summary: Regularize test_email/test_headerregistry's references to policy. files: Lib/test/test_email/test_headerregistry.py | 12 +++++----- 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py --- a/Lib/test/test_email/test_headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -650,14 +650,14 @@ def test_short_unstructured(self): h = self.make_header('subject', 'this is a test') - self.assertEqual(h.fold(policy=self.policy), + self.assertEqual(h.fold(policy=policy.default), 'subject: this is a test\n') def test_long_unstructured(self): h = self.make_header('Subject', 'This is a long header ' 'line that will need to be folded into two lines ' 'and will demonstrate basic folding') - self.assertEqual(h.fold(policy=self.policy), + self.assertEqual(h.fold(policy=policy.default), 'Subject: This is a long header line that will ' 'need to be folded into two lines\n' ' and will demonstrate basic folding\n') @@ -676,11 +676,11 @@ def test_fold_unstructured_single_word(self): h = self.make_header('Subject', 'test') - self.assertEqual(h.fold(policy=self.policy), 'Subject: test\n') + self.assertEqual(h.fold(policy=policy.default), 'Subject: test\n') def test_fold_unstructured_short(self): h = self.make_header('Subject', 'test test test') - self.assertEqual(h.fold(policy=self.policy), + self.assertEqual(h.fold(policy=policy.default), 'Subject: test test test\n') def test_fold_unstructured_with_overlong_word(self): @@ -721,7 +721,7 @@ h = self.make_header('To', '"Theodore H. Perfect" , ' '"My address is very long because my name is long" , ' '"Only A. Friend" ') - self.assertEqual(h.fold(policy=self.policy), textwrap.dedent("""\ + self.assertEqual(h.fold(policy=policy.default), textwrap.dedent("""\ To: "Theodore H. Perfect" , "My address is very long because my name is long" , "Only A. Friend" @@ -729,7 +729,7 @@ def test_fold_date_header(self): h = self.make_header('Date', 'Sat, 2 Feb 2002 17:00:06 -0800') - self.assertEqual(h.fold(policy=self.policy), + self.assertEqual(h.fold(policy=policy.default), 'Date: Sat, 02 Feb 2002 17:00:06 -0800\n') -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 03:09:25 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 29 May 2012 03:09:25 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Refactor_test=5Femail/test?= =?utf8?q?=5Fpickleable_and_add_tests_for_date_headers?= Message-ID: http://hg.python.org/cpython/rev/5c40627e9f11 changeset: 77216:5c40627e9f11 user: R David Murray date: Mon May 28 21:09:04 2012 -0400 summary: Refactor test_email/test_pickleable and add tests for date headers files: Lib/test/test_email/test_pickleable.py | 87 +++++++++---- 1 files changed, 59 insertions(+), 28 deletions(-) diff --git a/Lib/test/test_email/test_pickleable.py b/Lib/test/test_email/test_pickleable.py --- a/Lib/test/test_email/test_pickleable.py +++ b/Lib/test/test_email/test_pickleable.py @@ -2,55 +2,86 @@ import textwrap import copy import pickle +import email +import email.message from email import policy -from email import message_from_string from email.headerregistry import HeaderRegistry from test.test_email import TestEmailBase class TestPickleCopyHeader(TestEmailBase): - unstructured = HeaderRegistry()('subject', 'this is a test') + header_factory = HeaderRegistry() - def test_deepcopy_unstructured(self): - h = copy.deepcopy(self.unstructured) - self.assertEqual(str(h), str(self.unstructured)) + unstructured = header_factory('subject', 'this is a test') - def test_pickle_unstructured(self): - p = pickle.dumps(self.unstructured) + def _test_deepcopy(self, name, value): + header = self.header_factory(name, value) + h = copy.deepcopy(header) + self.assertEqual(str(h), str(header)) + + def _test_pickle(self, name, value): + header = self.header_factory(name, value) + p = pickle.dumps(header) h = pickle.loads(p) - self.assertEqual(str(h), str(self.unstructured)) + self.assertEqual(str(h), str(header)) - address = HeaderRegistry()('from', 'frodo at mordor.net') + headers = ( + ('subject', 'this is a test'), + ('from', 'frodo at mordor.net'), + ('to', 'a: k at b.com, y at z.com;, j at f.com'), + ('date', 'Tue, 29 May 2012 09:24:26 +1000'), + ) - def test_deepcopy_address(self): - h = copy.deepcopy(self.address) - self.assertEqual(str(h), str(self.address)) + for header in headers: + locals()['test_deepcopy_'+header[0]] = ( + lambda self, header=header: + self._test_deepcopy(*header)) - def test_pickle_address(self): - p = pickle.dumps(self.address) - h = pickle.loads(p) - self.assertEqual(str(h), str(self.address)) + for header in headers: + locals()['test_pickle_'+header[0]] = ( + lambda self, header=header: + self._test_pickle(*header)) class TestPickleCopyMessage(TestEmailBase): - testmsg = message_from_string(textwrap.dedent("""\ - From: frodo at mordor.net - To: bilbo at underhill.org - Subject: help + msgs = {} - I think I forgot the ring. - """), policy=policy.default) + # Note: there will be no custom header objects in the parsed message. + msgs['parsed'] = email.message_from_string(textwrap.dedent("""\ + Date: Tue, 29 May 2012 09:24:26 +1000 + From: frodo at mordor.net + To: bilbo at underhill.org + Subject: help - def test_deepcopy(self): - msg2 = copy.deepcopy(self.testmsg) - self.assertEqual(msg2.as_string(), self.testmsg.as_string()) + I think I forgot the ring. + """), policy=policy.default) - def test_pickle(self): - p = pickle.dumps(self.testmsg) + msgs['created'] = email.message.Message(policy=policy.default) + msgs['created']['Date'] = 'Tue, 29 May 2012 09:24:26 +1000' + msgs['created']['From'] = 'frodo at mordor.net' + msgs['created']['To'] = 'bilbo at underhill.org' + msgs['created']['Subject'] = 'help' + msgs['created'].set_payload('I think I forgot the ring.') + + def _test_deepcopy(self, msg): + msg2 = copy.deepcopy(msg) + self.assertEqual(msg2.as_string(), msg.as_string()) + + def _test_pickle(self, msg): + p = pickle.dumps(msg) msg2 = pickle.loads(p) - self.assertEqual(msg2.as_string(), self.testmsg.as_string()) + self.assertEqual(msg2.as_string(), msg.as_string()) + for name, msg in msgs.items(): + locals()['test_deepcopy_'+name] = ( + lambda self, msg=msg: + self._test_deepcopy(msg)) + + for name, msg in msgs.items(): + locals()['test_pickle_'+name] = ( + lambda self, msg=msg: + self._test_pickle(msg)) if __name__ == '__main__': -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 05:03:50 2012 From: python-checkins at python.org (eli.bendersky) Date: Tue, 29 May 2012 05:03:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_14814=3A_Add_namespac?= =?utf8?q?es_keyword_arg_to_find=28*=29_methods_in_=5Felementtree=2E?= Message-ID: http://hg.python.org/cpython/rev/7d252dbfbee3 changeset: 77217:7d252dbfbee3 user: Eli Bendersky date: Tue May 29 06:02:56 2012 +0300 summary: Issue 14814: Add namespaces keyword arg to find(*) methods in _elementtree. Add attrib keyword to Element and SubElement in _elementtree. Patch developed with Ezio Melotti. files: Doc/library/xml.etree.elementtree.rst | 29 ++- Lib/test/test_xml_etree.py | 66 ++++++++- Lib/xml/etree/ElementTree.py | 3 + Modules/_elementtree.c | 113 ++++++++++--- 4 files changed, 173 insertions(+), 38 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -476,27 +476,30 @@ .. versionadded:: 3.2 - .. method:: find(match) + .. method:: find(match, namespaces=None) Finds the first subelement matching *match*. *match* may be a tag name or a :ref:`path `. Returns an element instance - or ``None``. + or ``None``. *namespaces* is an optional mapping from namespace prefix + to full name. - .. method:: findall(match) + .. method:: findall(match, namespaces=None) Finds all matching subelements, by tag name or :ref:`path `. Returns a list containing all matching - elements in document order. + elements in document order. *namespaces* is an optional mapping from + namespace prefix to full name. - .. method:: findtext(match, default=None) + .. method:: findtext(match, default=None, namespaces=None) Finds text for the first subelement matching *match*. *match* may be a tag name or a :ref:`path `. Returns the text content of the first matching element, or *default* if no element was found. Note that if the matching element has no text content an empty string - is returned. + is returned. *namespaces* is an optional mapping from namespace prefix + to full name. .. method:: getchildren() @@ -528,11 +531,13 @@ .. versionadded:: 3.2 - .. method:: iterfind(match) + .. method:: iterfind(match, namespaces=None) Finds all matching subelements, by tag name or :ref:`path `. Returns an iterable yielding all - matching elements in document order. + matching elements in document order. *namespaces* is an optional mapping + from namespace prefix to full name. + .. versionadded:: 3.2 @@ -597,17 +602,17 @@ care. *element* is an element instance. - .. method:: find(match) + .. method:: find(match, namespaces=None) Same as :meth:`Element.find`, starting at the root of the tree. - .. method:: findall(match) + .. method:: findall(match, namespaces=None) Same as :meth:`Element.findall`, starting at the root of the tree. - .. method:: findtext(match, default=None) + .. method:: findtext(match, default=None, namespaces=None) Same as :meth:`Element.findtext`, starting at the root of the tree. @@ -630,7 +635,7 @@ to look for (default is to return all elements) - .. method:: iterfind(match) + .. method:: iterfind(match, namespaces=None) Same as :meth:`Element.iterfind`, starting at the root of the tree. diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -62,6 +62,22 @@ """ +SAMPLE_XML_NS_ELEMS = """ + + + + Apples + Bananas + + + + + African Coffee Table + 80 + 120 + + +""" def sanity(): """ @@ -1995,6 +2011,17 @@ self.assertEqual(pyET.SubElement.__module__, 'xml.etree.ElementTree') +class NamespaceParseTest(unittest.TestCase): + def test_find_with_namespace(self): + nsmap = {'h': 'hello', 'f': 'foo'} + doc = ET.fromstring(SAMPLE_XML_NS_ELEMS) + + self.assertEqual(len(doc.findall('{hello}table', nsmap)), 1) + self.assertEqual(len(doc.findall('.//{hello}td', nsmap)), 2) + self.assertEqual(len(doc.findall('.//{foo}name', nsmap)), 1) + + + class ElementSlicingTest(unittest.TestCase): def _elem_tags(self, elemlist): return [e.tag for e in elemlist] @@ -2102,6 +2129,41 @@ ERRORS.codes[ERRORS.XML_ERROR_SYNTAX]) +class KeywordArgsTest(unittest.TestCase): + # Test various issues with keyword arguments passed to ET.Element + # constructor and methods + def test_issue14818(self): + x = ET.XML("foo") + self.assertEqual(x.find('a', None), + x.find(path='a', namespaces=None)) + self.assertEqual(x.findtext('a', None, None), + x.findtext(path='a', default=None, namespaces=None)) + self.assertEqual(x.findall('a', None), + x.findall(path='a', namespaces=None)) + self.assertEqual(list(x.iterfind('a', None)), + list(x.iterfind(path='a', namespaces=None))) + + self.assertEqual(ET.Element('a').attrib, {}) + elements = [ + ET.Element('a', dict(href="#", id="foo")), + ET.Element('a', attrib=dict(href="#", id="foo")), + ET.Element('a', dict(href="#"), id="foo"), + ET.Element('a', href="#", id="foo"), + ET.Element('a', dict(href="#", id="foo"), href="#", id="foo"), + ] + for e in elements: + self.assertEqual(e.tag, 'a') + self.assertEqual(e.attrib, dict(href="#", id="foo")) + + e2 = ET.SubElement(elements[0], 'foobar', attrib={'key1': 'value1'}) + self.assertEqual(e2.attrib['key1'], 'value1') + + with self.assertRaisesRegex(TypeError, 'must be dict, not str'): + ET.Element('a', "I'm not a dict") + with self.assertRaisesRegex(TypeError, 'must be dict, not str'): + ET.Element('a', attrib="I'm not a dict") + + # -------------------------------------------------------------------- @@ -2157,7 +2219,9 @@ StringIOTest, ParseErrorTest, ElementTreeTest, - TreeBuilderTest] + NamespaceParseTest, + TreeBuilderTest, + KeywordArgsTest] if module is pyET: # Run the tests specific to the Python implementation test_classes += [NoAcceleratorTest] diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py --- a/Lib/xml/etree/ElementTree.py +++ b/Lib/xml/etree/ElementTree.py @@ -205,6 +205,9 @@ # constructor def __init__(self, tag, attrib={}, **extra): + if not isinstance(attrib, dict): + raise TypeError("attrib must be dict, not %s" % ( + attrib.__class__.__name__,)) attrib = attrib.copy() attrib.update(extra) self.tag = tag diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -347,6 +347,41 @@ return (PyObject *)e; } +/* Helper function for extracting the attrib dictionary from a keywords dict. + * This is required by some constructors/functions in this module that can + * either accept attrib as a keyword argument or all attributes splashed + * directly into *kwds. + * If there is no 'attrib' keyword, return an empty dict. + */ +static PyObject* +get_attrib_from_keywords(PyObject *kwds) +{ + PyObject *attrib_str = PyUnicode_FromString("attrib"); + PyObject *attrib = PyDict_GetItem(kwds, attrib_str); + + if (attrib) { + /* If attrib was found in kwds, copy its value and remove it from + * kwds + */ + if (!PyDict_Check(attrib)) { + Py_DECREF(attrib_str); + PyErr_Format(PyExc_TypeError, "attrib must be dict, not %.100s", + Py_TYPE(attrib)->tp_name); + return NULL; + } + attrib = PyDict_Copy(attrib); + PyDict_DelItem(kwds, attrib_str); + } else { + attrib = PyDict_New(); + } + + Py_DECREF(attrib_str); + + if (attrib) + PyDict_Update(attrib, kwds); + return attrib; +} + static int element_init(PyObject *self, PyObject *args, PyObject *kwds) { @@ -358,13 +393,23 @@ if (!PyArg_ParseTuple(args, "O|O!:Element", &tag, &PyDict_Type, &attrib)) return -1; - if (attrib || kwds) { - attrib = (attrib) ? PyDict_Copy(attrib) : PyDict_New(); + if (attrib) { + /* attrib passed as positional arg */ + attrib = PyDict_Copy(attrib); if (!attrib) return -1; - if (kwds) - PyDict_Update(attrib, kwds); + if (kwds) { + if (PyDict_Update(attrib, kwds) < 0) { + return -1; + } + } + } else if (kwds) { + /* have keywords args */ + attrib = get_attrib_from_keywords(kwds); + if (!attrib) + return -1; } else { + /* no attrib arg, no kwds, so no attributes */ Py_INCREF(Py_None); attrib = Py_None; } @@ -536,7 +581,7 @@ } static PyObject* -subelement(PyObject* self, PyObject* args, PyObject* kw) +subelement(PyObject *self, PyObject *args, PyObject *kwds) { PyObject* elem; @@ -548,13 +593,23 @@ &PyDict_Type, &attrib)) return NULL; - if (attrib || kw) { - attrib = (attrib) ? PyDict_Copy(attrib) : PyDict_New(); + if (attrib) { + /* attrib passed as positional arg */ + attrib = PyDict_Copy(attrib); if (!attrib) return NULL; - if (kw) - PyDict_Update(attrib, kw); + if (kwds) { + if (PyDict_Update(attrib, kwds) < 0) { + return NULL; + } + } + } else if (kwds) { + /* have keyword args */ + attrib = get_attrib_from_keywords(kwds); + if (!attrib) + return NULL; } else { + /* no attrib arg, no kwds, so no attribute */ Py_INCREF(Py_None); attrib = Py_None; } @@ -881,13 +936,15 @@ } static PyObject* -element_find(ElementObject* self, PyObject* args) +element_find(ElementObject *self, PyObject *args, PyObject *kwds) { int i; PyObject* tag; PyObject* namespaces = Py_None; - - if (!PyArg_ParseTuple(args, "O|O:find", &tag, &namespaces)) + static char *kwlist[] = {"path", "namespaces", 0}; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O:find", kwlist, + &tag, &namespaces)) return NULL; if (checkpath(tag) || namespaces != Py_None) { @@ -913,15 +970,17 @@ } static PyObject* -element_findtext(ElementObject* self, PyObject* args) +element_findtext(ElementObject *self, PyObject *args, PyObject *kwds) { int i; PyObject* tag; PyObject* default_value = Py_None; PyObject* namespaces = Py_None; _Py_IDENTIFIER(findtext); - - if (!PyArg_ParseTuple(args, "O|OO:findtext", &tag, &default_value, &namespaces)) + static char *kwlist[] = {"path", "default", "namespaces", 0}; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|OO:findtext", kwlist, + &tag, &default_value, &namespaces)) return NULL; if (checkpath(tag) || namespaces != Py_None) @@ -951,14 +1010,16 @@ } static PyObject* -element_findall(ElementObject* self, PyObject* args) +element_findall(ElementObject *self, PyObject *args, PyObject *kwds) { int i; PyObject* out; PyObject* tag; PyObject* namespaces = Py_None; - - if (!PyArg_ParseTuple(args, "O|O:findall", &tag, &namespaces)) + static char *kwlist[] = {"path", "namespaces", 0}; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O:findall", kwlist, + &tag, &namespaces)) return NULL; if (checkpath(tag) || namespaces != Py_None) { @@ -990,13 +1051,15 @@ } static PyObject* -element_iterfind(ElementObject* self, PyObject* args) +element_iterfind(ElementObject *self, PyObject *args, PyObject *kwds) { PyObject* tag; PyObject* namespaces = Py_None; _Py_IDENTIFIER(iterfind); - - if (!PyArg_ParseTuple(args, "O|O:iterfind", &tag, &namespaces)) + static char *kwlist[] = {"path", "namespaces", 0}; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|O:iterfind", kwlist, + &tag, &namespaces)) return NULL; return _PyObject_CallMethodId( @@ -1567,9 +1630,9 @@ {"get", (PyCFunction) element_get, METH_VARARGS}, {"set", (PyCFunction) element_set, METH_VARARGS}, - {"find", (PyCFunction) element_find, METH_VARARGS}, - {"findtext", (PyCFunction) element_findtext, METH_VARARGS}, - {"findall", (PyCFunction) element_findall, METH_VARARGS}, + {"find", (PyCFunction) element_find, METH_VARARGS | METH_KEYWORDS}, + {"findtext", (PyCFunction) element_findtext, METH_VARARGS | METH_KEYWORDS}, + {"findall", (PyCFunction) element_findall, METH_VARARGS | METH_KEYWORDS}, {"append", (PyCFunction) element_append, METH_VARARGS}, {"extend", (PyCFunction) element_extend, METH_VARARGS}, @@ -1578,7 +1641,7 @@ {"iter", (PyCFunction) element_iter, METH_VARARGS}, {"itertext", (PyCFunction) element_itertext, METH_VARARGS}, - {"iterfind", (PyCFunction) element_iterfind, METH_VARARGS}, + {"iterfind", (PyCFunction) element_iterfind, METH_VARARGS | METH_KEYWORDS}, {"getiterator", (PyCFunction) element_iter, METH_VARARGS}, {"getchildren", (PyCFunction) element_getchildren, METH_VARARGS}, -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Tue May 29 05:19:08 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 May 2012 13:19:08 +1000 Subject: [Python-checkins] cpython: Issue 14814: Add namespaces keyword arg to find(*) methods in _elementtree. In-Reply-To: References: Message-ID: On Tue, May 29, 2012 at 1:03 PM, eli.bendersky wrote: > http://hg.python.org/cpython/rev/7d252dbfbee3 > changeset: ? 77217:7d252dbfbee3 > user: ? ? ? ?Eli Bendersky > date: ? ? ? ?Tue May 29 06:02:56 2012 +0300 > summary: > ?Issue 14814: Add namespaces keyword arg to find(*) methods in _elementtree. > Add attrib keyword to Element and SubElement in _elementtree. > Patch developed with Ezio Melotti. I'm not sure which issue you intended to reference here, but I'm fairly sure the PEP 3144 integration issue wasn't it :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Tue May 29 05:54:49 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Tue, 29 May 2012 05:54:49 +0200 Subject: [Python-checkins] Daily reference leaks (5c40627e9f11): sum=465 Message-ID: results for 5c40627e9f11 on branch "default" -------------------------------------------- test_smtplib leaked [154, 154, 154] references, sum=462 test_super leaked [1, 1, 1] references, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog2cWmiD', '-x'] From python-checkins at python.org Tue May 29 12:08:15 2012 From: python-checkins at python.org (hynek.schlawack) Date: Tue, 29 May 2012 12:08:15 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314835=3A_Make_plistlib_o?= =?utf8?q?utput_empty_arrays_=26_dicts_like_OS_X?= Message-ID: http://hg.python.org/cpython/rev/9e64084f9980 changeset: 77218:9e64084f9980 user: Hynek Schlawack date: Tue May 29 12:04:54 2012 +0200 summary: #14835: Make plistlib output empty arrays & dicts like OS X Patch by Sidney San Mart?n. files: Lib/plistlib.py | 30 ++++++++++++++++---------- Lib/test/test_plistlib.py | 6 +++++ Misc/ACKS | 1 + Misc/NEWS | 3 ++ 4 files changed, 28 insertions(+), 12 deletions(-) diff --git a/Lib/plistlib.py b/Lib/plistlib.py --- a/Lib/plistlib.py +++ b/Lib/plistlib.py @@ -237,20 +237,26 @@ self.endElement("data") def writeDict(self, d): - self.beginElement("dict") - items = sorted(d.items()) - for key, value in items: - if not isinstance(key, str): - raise TypeError("keys must be strings") - self.simpleElement("key", key) - self.writeValue(value) - self.endElement("dict") + if d: + self.beginElement("dict") + items = sorted(d.items()) + for key, value in items: + if not isinstance(key, str): + raise TypeError("keys must be strings") + self.simpleElement("key", key) + self.writeValue(value) + self.endElement("dict") + else: + self.simpleElement("dict") def writeArray(self, array): - self.beginElement("array") - for value in array: - self.writeValue(value) - self.endElement("array") + if array: + self.beginElement("array") + for value in array: + self.writeValue(value) + self.endElement("array") + else: + self.simpleElement("array") class _InternalDict(dict): diff --git a/Lib/test/test_plistlib.py b/Lib/test/test_plistlib.py --- a/Lib/test/test_plistlib.py +++ b/Lib/test/test_plistlib.py @@ -55,6 +55,10 @@ aString Doodah + anEmptyDict + + anEmptyList + anInt 728 nestedData @@ -112,6 +116,8 @@ someMoreData = plistlib.Data(b"\0\1\2\3" * 10), nestedData = [plistlib.Data(b"\0\1\2\3" * 10)], aDate = datetime.datetime(2004, 10, 26, 10, 33, 33), + anEmptyDict = dict(), + anEmptyList = list() ) pl['\xc5benraa'] = "That was a unicode key." return pl diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -662,6 +662,7 @@ Anthony Martin Owen Martin S?bastien Martini +Sidney San Mart?n Roger Masse Nick Mathewson Simon Mathieu diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Core and Builtins ----------------- +- Issue #14835: Make plistlib output empty arrays & dicts like OS X. + Patch by Sidney San Mart?n. + - Issue #14930: Make memoryview objects weakrefable. - Issue #14775: Fix a potential quadratic dict build-up due to the garbage -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 12:12:48 2012 From: python-checkins at python.org (vinay.sajip) Date: Tue, 29 May 2012 12:12:48 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Removed_pysetup3=2Eexe_exec?= =?utf8?q?utable=2E?= Message-ID: http://hg.python.org/cpython/rev/01381723bc50 changeset: 77219:01381723bc50 user: Vinay Sajip date: Tue May 29 11:12:43 2012 +0100 summary: Removed pysetup3.exe executable. files: Lib/venv/scripts/nt/pysetup3.exe | Bin Lib/venv/scripts/nt/pysetup3-script.py | 0 2 files changed, 0 insertions(+), 0 deletions(-) diff --git a/Lib/venv/scripts/nt/pysetup3.exe b/Lib/venv/scripts/nt/pysetup3.exe deleted file mode 100644 Binary file Lib/venv/scripts/nt/pysetup3.exe has changed diff --git a/Lib/venv/scripts/nt/pysetup3-script.py b/Lib/venv/scripts/nt/pysetup3.py rename from Lib/venv/scripts/nt/pysetup3-script.py rename to Lib/venv/scripts/nt/pysetup3.py -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 13:03:12 2012 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 29 May 2012 13:03:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_=5F=5Fgetslice=5F=5F?= =?utf8?b?LCBfX3NldHNsaWNlX18sIF9fZGVsc2xpY2VfXyBtZXRob2RzIGZyb20gcHJveGll?= =?utf8?q?s?= Message-ID: http://hg.python.org/cpython/rev/1f7c452e797e changeset: 77220:1f7c452e797e user: Richard Oudkerk date: Tue May 29 12:01:45 2012 +0100 summary: Remove __getslice__, __setslice__, __delslice__ methods from proxies Proxy classes in multiprocessing do not need these methods in Python 3.x. files: Lib/multiprocessing/managers.py | 11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-) diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -1035,12 +1035,11 @@ BaseListProxy = MakeProxyType('BaseListProxy', ( - '__add__', '__contains__', '__delitem__', '__delslice__', - '__getitem__', '__getslice__', '__len__', '__mul__', - '__reversed__', '__rmul__', '__setitem__', '__setslice__', + '__add__', '__contains__', '__delitem__', '__getitem__', '__len__', + '__mul__', '__reversed__', '__rmul__', '__setitem__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort', '__imul__' - )) # XXX __getslice__ and __setslice__ unneeded in Py3.0 + )) class ListProxy(BaseListProxy): def __iadd__(self, value): self._callmethod('extend', (value,)) @@ -1058,8 +1057,8 @@ ArrayProxy = MakeProxyType('ArrayProxy', ( - '__len__', '__getitem__', '__setitem__', '__getslice__', '__setslice__' - )) # XXX __getslice__ and __setslice__ unneeded in Py3.0 + '__len__', '__getitem__', '__setitem__' + )) PoolProxy = MakeProxyType('PoolProxy', ( -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 13:03:13 2012 From: python-checkins at python.org (richard.oudkerk) Date: Tue, 29 May 2012 13:03:13 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_Python_3=2Ex-style_keyw?= =?utf8?q?ord_only_arg_in_Array=28=29?= Message-ID: http://hg.python.org/cpython/rev/b31ea7995214 changeset: 77221:b31ea7995214 user: Richard Oudkerk date: Tue May 29 12:01:47 2012 +0100 summary: Use Python 3.x-style keyword only arg in Array() Previously a Python 2.x compatible hack was used for multiprocessing.sharedctypes.Array(). Also the documented signature was wrong. files: Doc/library/multiprocessing.rst | 6 +++--- Lib/multiprocessing/__init__.py | 8 ++++---- Lib/multiprocessing/sharedctypes.py | 7 ++----- 3 files changed, 9 insertions(+), 12 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -953,7 +953,7 @@ It is possible to create shared objects using shared memory which can be inherited by child processes. -.. function:: Value(typecode_or_type, *args[, lock]) +.. function:: Value(typecode_or_type, *args, lock=True) Return a :mod:`ctypes` object allocated from shared memory. By default the return value is actually a synchronized wrapper for the object. @@ -1045,7 +1045,7 @@ attributes which allow one to use it to store and retrieve strings -- see documentation for :mod:`ctypes`. -.. function:: Array(typecode_or_type, size_or_initializer, *args[, lock]) +.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True) The same as :func:`RawArray` except that depending on the value of *lock* a process-safe synchronization wrapper may be returned instead of a raw ctypes @@ -1060,7 +1060,7 @@ Note that *lock* is a keyword-only argument. -.. function:: Value(typecode_or_type, *args[, lock]) +.. function:: Value(typecode_or_type, *args, lock=True) The same as :func:`RawValue` except that depending on the value of *lock* a process-safe synchronization wrapper may be returned instead of a raw ctypes diff --git a/Lib/multiprocessing/__init__.py b/Lib/multiprocessing/__init__.py --- a/Lib/multiprocessing/__init__.py +++ b/Lib/multiprocessing/__init__.py @@ -228,19 +228,19 @@ from multiprocessing.sharedctypes import RawArray return RawArray(typecode_or_type, size_or_initializer) -def Value(typecode_or_type, *args, **kwds): +def Value(typecode_or_type, *args, lock=True): ''' Returns a synchronized shared object ''' from multiprocessing.sharedctypes import Value - return Value(typecode_or_type, *args, **kwds) + return Value(typecode_or_type, *args, lock=lock) -def Array(typecode_or_type, size_or_initializer, **kwds): +def Array(typecode_or_type, size_or_initializer, *, lock=True): ''' Returns a synchronized shared array ''' from multiprocessing.sharedctypes import Array - return Array(typecode_or_type, size_or_initializer, **kwds) + return Array(typecode_or_type, size_or_initializer, lock=lock) # # diff --git a/Lib/multiprocessing/sharedctypes.py b/Lib/multiprocessing/sharedctypes.py --- a/Lib/multiprocessing/sharedctypes.py +++ b/Lib/multiprocessing/sharedctypes.py @@ -63,7 +63,7 @@ result.__init__(*size_or_initializer) return result -def Value(typecode_or_type, *args, lock=None): +def Value(typecode_or_type, *args, lock=True): ''' Return a synchronization wrapper for a Value ''' @@ -76,13 +76,10 @@ raise AttributeError("'%r' has no method 'acquire'" % lock) return synchronized(obj, lock) -def Array(typecode_or_type, size_or_initializer, **kwds): +def Array(typecode_or_type, size_or_initializer, *, lock=True): ''' Return a synchronization wrapper for a RawArray ''' - lock = kwds.pop('lock', None) - if kwds: - raise ValueError('unrecognized keyword argument(s): %s' % list(kwds.keys())) obj = RawArray(typecode_or_type, size_or_initializer) if lock is False: return obj -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 13:13:17 2012 From: python-checkins at python.org (victor.stinner) Date: Tue, 29 May 2012 13:13:17 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbjogUHlBcmdfUGFyc2UqKCJVIik6?= =?utf8?q?_ensure_that_the_Unicode_string_is_ready?= Message-ID: http://hg.python.org/cpython/rev/1700fb28d2ab changeset: 77222:1700fb28d2ab user: Victor Stinner date: Tue May 29 12:30:29 2012 +0200 summary: PyArg_Parse*("U"): ensure that the Unicode string is ready files: Python/getargs.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-) diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1167,8 +1167,11 @@ case 'U': { /* PyUnicode object */ PyObject **p = va_arg(*p_va, PyObject **); - if (PyUnicode_Check(arg)) + if (PyUnicode_Check(arg)) { + if (PyUnicode_READY(arg) == -1) + RETURN_ERR_OCCURRED; *p = arg; + } else return converterr("str", arg, msgbuf, bufsize); break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 13:13:19 2012 From: python-checkins at python.org (victor.stinner) Date: Tue, 29 May 2012 13:13:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314744=3A_Use_the_n?= =?utf8?q?ew_=5FPyUnicodeWriter_internal_API_to_speed_up_str=25args?= Message-ID: http://hg.python.org/cpython/rev/22b56b0b8619 changeset: 77223:22b56b0b8619 user: Victor Stinner date: Tue May 29 12:57:52 2012 +0200 summary: Issue #14744: Use the new _PyUnicodeWriter internal API to speed up str%args and str.format(args) * Formatting string, int, float and complex use the _PyUnicodeWriter API. It avoids a temporary buffer in most cases. * Add _PyUnicodeWriter_WriteStr() to restore the PyAccu optimization: just keep a reference to the string if the output is only composed of one string * Disable overallocation when formatting the last argument of str%args and str.format(args) * Overallocation allocates at least 100 characters: add min_length attribute to the _PyUnicodeWriter structure * Add new private functions: _PyUnicode_FastCopyCharacters(), _PyUnicode_FastFill() and _PyUnicode_FromASCII() The speed up is around 20% in average. files: Include/complexobject.h | 10 +- Include/floatobject.h | 10 +- Include/longobject.h | 18 +- Include/unicodeobject.h | 95 +++- Misc/NEWS | 3 + Objects/complexobject.c | 17 +- Objects/floatobject.c | 27 +- Objects/longobject.c | 333 +++++++++---- Objects/stringlib/asciilib.h | 2 +- Objects/stringlib/unicode_format.h | 48 +- Objects/unicodeobject.c | 372 +++++++++++---- Python/formatter_unicode.c | 412 ++++++++-------- 12 files changed, 894 insertions(+), 453 deletions(-) diff --git a/Include/complexobject.h b/Include/complexobject.h --- a/Include/complexobject.h +++ b/Include/complexobject.h @@ -63,10 +63,12 @@ /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ #ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) _PyComplex_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyComplex_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif #ifdef __cplusplus diff --git a/Include/floatobject.h b/Include/floatobject.h --- a/Include/floatobject.h +++ b/Include/floatobject.h @@ -112,10 +112,12 @@ /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ -PyAPI_FUNC(PyObject *) _PyFloat_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyFloat_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif /* Py_LIMITED_API */ #ifdef __cplusplus diff --git a/Include/longobject.h b/Include/longobject.h --- a/Include/longobject.h +++ b/Include/longobject.h @@ -151,14 +151,22 @@ /* _PyLong_Format: Convert the long to a string object with given base, appending a base prefix of 0[box] if base is 2, 8 or 16. */ -PyAPI_FUNC(PyObject *) _PyLong_Format(PyObject *aa, int base); +PyAPI_FUNC(PyObject *) _PyLong_Format(PyObject *obj, int base); + +PyAPI_FUNC(int) _PyLong_FormatWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + int base, + int alternate); /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ -PyAPI_FUNC(PyObject *) _PyLong_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyLong_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif /* Py_LIMITED_API */ /* These aren't really part of the long object, but they're handy. The diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -648,8 +648,20 @@ Py_ssize_t from_start, Py_ssize_t how_many ); + +/* Unsafe version of PyUnicode_CopyCharacters(): don't check arguments and so + may crash if parameters are invalid (e.g. if the output string + is too short). */ +PyAPI_FUNC(void) _PyUnicode_FastCopyCharacters( + PyObject *to, + Py_ssize_t to_start, + PyObject *from, + Py_ssize_t from_start, + Py_ssize_t how_many + ); #endif +#ifndef Py_LIMITED_API /* Fill a string with a character: write fill_char into unicode[start:start+length]. @@ -658,13 +670,21 @@ Return the number of written character, or return -1 and raise an exception on error. */ -#ifndef Py_LIMITED_API PyAPI_FUNC(Py_ssize_t) PyUnicode_Fill( PyObject *unicode, Py_ssize_t start, Py_ssize_t length, Py_UCS4 fill_char ); + +/* Unsafe version of PyUnicode_Fill(): don't check arguments and so may crash + if parameters are invalid (e.g. if length is longer than the string). */ +PyAPI_FUNC(void) _PyUnicode_FastFill( + PyObject *unicode, + Py_ssize_t start, + Py_ssize_t length, + Py_UCS4 fill_char + ); #endif /* Create a Unicode Object from the Py_UNICODE buffer u of the given @@ -696,13 +716,19 @@ const char *u /* UTF-8 encoded string */ ); +#ifndef Py_LIMITED_API /* Create a new string from a buffer of Py_UCS1, Py_UCS2 or Py_UCS4 characters. Scan the string to find the maximum character. */ -#ifndef Py_LIMITED_API PyAPI_FUNC(PyObject*) PyUnicode_FromKindAndData( int kind, const void *buffer, Py_ssize_t size); + +/* Create a new string from a buffer of ASCII characters. + WARNING: Don't check if the string contains any non-ASCII character. */ +PyAPI_FUNC(PyObject*) _PyUnicode_FromASCII( + const char *buffer, + Py_ssize_t size); #endif PyAPI_FUNC(PyObject*) PyUnicode_Substring( @@ -865,12 +891,69 @@ ); #ifndef Py_LIMITED_API +typedef struct { + PyObject *buffer; + void *data; + enum PyUnicode_Kind kind; + Py_UCS4 maxchar; + Py_ssize_t size; + Py_ssize_t pos; + /* minimum length of the buffer when overallocation is enabled, + see _PyUnicodeWriter_Init() */ + Py_ssize_t min_length; + struct { + unsigned char overallocate:1; + /* If readonly is 1, buffer is a shared string (cannot be modified) + and size is set to 0. */ + unsigned char readonly:1; + } flags; +} _PyUnicodeWriter ; + +/* Initialize a Unicode writer. + + If min_length is greater than zero, _PyUnicodeWriter_Prepare() + overallocates the buffer and min_length is the minimum length in characters + of the buffer. */ +PyAPI_FUNC(void) +_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, Py_ssize_t min_length); + +/* Prepare the buffer to write 'length' characters + with the specified maximum character. + + Return 0 on success, raise an exception and return -1 on error. */ +#define _PyUnicodeWriter_Prepare(WRITER, LENGTH, MAXCHAR) \ + (((MAXCHAR) <= (WRITER)->maxchar \ + && (LENGTH) <= (WRITER)->size - (WRITER)->pos) \ + ? 0 \ + : (((LENGTH) == 0) \ + ? 0 \ + : _PyUnicodeWriter_PrepareInternal((WRITER), (LENGTH), (MAXCHAR)))) + +/* Don't call this function directly, use the _PyUnicodeWriter_Prepare() macro + instead. */ +PyAPI_FUNC(int) +_PyUnicodeWriter_PrepareInternal(_PyUnicodeWriter *writer, + Py_ssize_t length, Py_UCS4 maxchar); + +PyAPI_FUNC(int) +_PyUnicodeWriter_WriteStr(_PyUnicodeWriter *writer, PyObject *str); + +PyAPI_FUNC(PyObject *) +_PyUnicodeWriter_Finish(_PyUnicodeWriter *writer); + +PyAPI_FUNC(void) +_PyUnicodeWriter_Dealloc(_PyUnicodeWriter *writer); +#endif + +#ifndef Py_LIMITED_API /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ -PyAPI_FUNC(PyObject *) _PyUnicode_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyUnicode_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif PyAPI_FUNC(void) PyUnicode_InternInPlace(PyObject **); diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,9 @@ - Issue #14835: Make plistlib output empty arrays & dicts like OS X. Patch by Sidney San Mart?n. +- Issue #14744: Use the new _PyUnicodeWriter internal API to speed up + str%args and str.format(args). + - Issue #14930: Make memoryview objects weakrefable. - Issue #14775: Fix a potential quadratic dict build-up due to the garbage diff --git a/Objects/complexobject.c b/Objects/complexobject.c --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -699,11 +699,22 @@ complex__format__(PyObject* self, PyObject* args) { PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) - return NULL; - return _PyComplex_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); + return NULL; + + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyComplex_FormatAdvancedWriter( + &writer, + self, + format_spec, 0, PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } #if 0 diff --git a/Objects/floatobject.c b/Objects/floatobject.c --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -267,13 +267,15 @@ float_repr(PyFloatObject *v) { PyObject *result; - char *buf = PyOS_double_to_string(PyFloat_AS_DOUBLE(v), - 'r', 0, - Py_DTSF_ADD_DOT_0, - NULL); + char *buf; + + buf = PyOS_double_to_string(PyFloat_AS_DOUBLE(v), + 'r', 0, + Py_DTSF_ADD_DOT_0, + NULL); if (!buf) return PyErr_NoMemory(); - result = PyUnicode_FromString(buf); + result = _PyUnicode_FromASCII(buf, strlen(buf)); PyMem_Free(buf); return result; } @@ -1703,11 +1705,22 @@ float__format__(PyObject *self, PyObject *args) { PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) return NULL; - return _PyFloat_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); + + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyFloat_FormatAdvancedWriter( + &writer, + self, + format_spec, 0, PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } PyDoc_STRVAR(float__format__doc, diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -1550,20 +1550,22 @@ string. (Return value is non-shared so that callers can modify the returned value if necessary.) */ -static PyObject * -long_to_decimal_string(PyObject *aa) +static int +long_to_decimal_string_internal(PyObject *aa, + PyObject **p_output, + _PyUnicodeWriter *writer) { PyLongObject *scratch, *a; PyObject *str; Py_ssize_t size, strlen, size_a, i, j; digit *pout, *pin, rem, tenpow; - unsigned char *p; int negative; + enum PyUnicode_Kind kind; a = (PyLongObject *)aa; if (a == NULL || !PyLong_Check(a)) { PyErr_BadInternalCall(); - return NULL; + return -1; } size_a = ABS(Py_SIZE(a)); negative = Py_SIZE(a) < 0; @@ -1580,13 +1582,13 @@ if (size_a > PY_SSIZE_T_MAX / PyLong_SHIFT) { PyErr_SetString(PyExc_OverflowError, "long is too large to format"); - return NULL; + return -1; } /* the expression size_a * PyLong_SHIFT is now safe from overflow */ size = 1 + size_a * PyLong_SHIFT / (3 * _PyLong_DECIMAL_SHIFT); scratch = _PyLong_New(size); if (scratch == NULL) - return NULL; + return -1; /* convert array of base _PyLong_BASE digits in pin to an array of base _PyLong_DECIMAL_BASE digits in pout, following Knuth (TAOCP, @@ -1609,7 +1611,7 @@ /* check for keyboard interrupt */ SIGCHECK({ Py_DECREF(scratch); - return NULL; + return -1; }); } /* pout should have at least one digit, so that the case when a = 0 @@ -1625,65 +1627,113 @@ tenpow *= 10; strlen++; } - str = PyUnicode_New(strlen, '9'); - if (str == NULL) { - Py_DECREF(scratch); + if (writer) { + if (_PyUnicodeWriter_Prepare(writer, strlen, '9') == -1) + return -1; + kind = writer->kind; + str = NULL; + } + else { + str = PyUnicode_New(strlen, '9'); + if (str == NULL) { + Py_DECREF(scratch); + return -1; + } + kind = PyUnicode_KIND(str); + } + +#define WRITE_DIGITS(TYPE) \ + do { \ + if (writer) \ + p = (TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos + strlen; \ + else \ + p = (TYPE*)PyUnicode_DATA(str) + strlen; \ + \ + *p = '\0'; \ + /* pout[0] through pout[size-2] contribute exactly \ + _PyLong_DECIMAL_SHIFT digits each */ \ + for (i=0; i < size - 1; i++) { \ + rem = pout[i]; \ + for (j = 0; j < _PyLong_DECIMAL_SHIFT; j++) { \ + *--p = '0' + rem % 10; \ + rem /= 10; \ + } \ + } \ + /* pout[size-1]: always produce at least one decimal digit */ \ + rem = pout[i]; \ + do { \ + *--p = '0' + rem % 10; \ + rem /= 10; \ + } while (rem != 0); \ + \ + /* and sign */ \ + if (negative) \ + *--p = '-'; \ + \ + /* check we've counted correctly */ \ + if (writer) \ + assert(p == ((TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos)); \ + else \ + assert(p == (TYPE*)PyUnicode_DATA(str)); \ + } while (0) + + /* fill the string right-to-left */ + if (kind == PyUnicode_1BYTE_KIND) { + Py_UCS1 *p; + WRITE_DIGITS(Py_UCS1); + } + else if (kind == PyUnicode_2BYTE_KIND) { + Py_UCS2 *p; + WRITE_DIGITS(Py_UCS2); + } + else { + assert (kind == PyUnicode_4BYTE_KIND); + Py_UCS4 *p; + WRITE_DIGITS(Py_UCS4); + } +#undef WRITE_DIGITS + + Py_DECREF(scratch); + if (writer) { + writer->pos += strlen; + } + else { + assert(_PyUnicode_CheckConsistency(str, 1)); + *p_output = (PyObject *)str; + } + return 0; +} + +static PyObject * +long_to_decimal_string(PyObject *aa) +{ + PyObject *v; + if (long_to_decimal_string_internal(aa, &v, NULL) == -1) return NULL; - } - - /* fill the string right-to-left */ - assert(PyUnicode_KIND(str) == PyUnicode_1BYTE_KIND); - p = PyUnicode_1BYTE_DATA(str) + strlen; - *p = '\0'; - /* pout[0] through pout[size-2] contribute exactly - _PyLong_DECIMAL_SHIFT digits each */ - for (i=0; i < size - 1; i++) { - rem = pout[i]; - for (j = 0; j < _PyLong_DECIMAL_SHIFT; j++) { - *--p = '0' + rem % 10; - rem /= 10; - } - } - /* pout[size-1]: always produce at least one decimal digit */ - rem = pout[i]; - do { - *--p = '0' + rem % 10; - rem /= 10; - } while (rem != 0); - - /* and sign */ - if (negative) - *--p = '-'; - - /* check we've counted correctly */ - assert(p == PyUnicode_1BYTE_DATA(str)); - assert(_PyUnicode_CheckConsistency(str, 1)); - Py_DECREF(scratch); - return (PyObject *)str; + return v; } /* Convert a long int object to a string, using a given conversion base, - which should be one of 2, 8, 10 or 16. Return a string object. - If base is 2, 8 or 16, add the proper prefix '0b', '0o' or '0x'. */ - -PyObject * -_PyLong_Format(PyObject *aa, int base) + which should be one of 2, 8 or 16. Return a string object. + If base is 2, 8 or 16, add the proper prefix '0b', '0o' or '0x' + if alternate is nonzero. */ + +static int +long_format_binary(PyObject *aa, int base, int alternate, + PyObject **p_output, _PyUnicodeWriter *writer) { register PyLongObject *a = (PyLongObject *)aa; PyObject *v; Py_ssize_t sz; Py_ssize_t size_a; - Py_UCS1 *p; + enum PyUnicode_Kind kind; int negative; int bits; - assert(base == 2 || base == 8 || base == 10 || base == 16); - if (base == 10) - return long_to_decimal_string((PyObject *)a); - + assert(base == 2 || base == 8 || base == 16); if (a == NULL || !PyLong_Check(a)) { PyErr_BadInternalCall(); - return NULL; + return -1; } size_a = ABS(Py_SIZE(a)); negative = Py_SIZE(a) < 0; @@ -1706,7 +1756,7 @@ /* Compute exact length 'sz' of output string. */ if (size_a == 0) { - sz = 3; + sz = 1; } else { Py_ssize_t size_a_in_bits; @@ -1714,56 +1764,126 @@ if (size_a > (PY_SSIZE_T_MAX - 3) / PyLong_SHIFT) { PyErr_SetString(PyExc_OverflowError, "int is too large to format"); - return NULL; + return -1; } size_a_in_bits = (size_a - 1) * PyLong_SHIFT + bits_in_digit(a->ob_digit[size_a - 1]); - /* Allow 2 characters for prefix and 1 for a '-' sign. */ - sz = 2 + negative + (size_a_in_bits + (bits - 1)) / bits; - } - - v = PyUnicode_New(sz, 'x'); - if (v == NULL) { + /* Allow 1 character for a '-' sign. */ + sz = negative + (size_a_in_bits + (bits - 1)) / bits; + } + if (alternate) { + /* 2 characters for prefix */ + sz += 2; + } + + if (writer) { + if (_PyUnicodeWriter_Prepare(writer, sz, 'x') == -1) + return -1; + kind = writer->kind; + v = NULL; + } + else { + v = PyUnicode_New(sz, 'x'); + if (v == NULL) + return -1; + kind = PyUnicode_KIND(v); + } + +#define WRITE_DIGITS(TYPE) \ + do { \ + if (writer) \ + p = (TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos + sz; \ + else \ + p = (TYPE*)PyUnicode_DATA(v) + sz; \ + \ + if (size_a == 0) { \ + *--p = '0'; \ + } \ + else { \ + /* JRH: special case for power-of-2 bases */ \ + twodigits accum = 0; \ + int accumbits = 0; /* # of bits in accum */ \ + Py_ssize_t i; \ + for (i = 0; i < size_a; ++i) { \ + accum |= (twodigits)a->ob_digit[i] << accumbits; \ + accumbits += PyLong_SHIFT; \ + assert(accumbits >= bits); \ + do { \ + char cdigit; \ + cdigit = (char)(accum & (base - 1)); \ + cdigit += (cdigit < 10) ? '0' : 'a'-10; \ + *--p = cdigit; \ + accumbits -= bits; \ + accum >>= bits; \ + } while (i < size_a-1 ? accumbits >= bits : accum > 0); \ + } \ + } \ + \ + if (alternate) { \ + if (base == 16) \ + *--p = 'x'; \ + else if (base == 8) \ + *--p = 'o'; \ + else /* (base == 2) */ \ + *--p = 'b'; \ + *--p = '0'; \ + } \ + if (negative) \ + *--p = '-'; \ + if (writer) \ + assert(p == ((TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos)); \ + else \ + assert(p == (TYPE*)PyUnicode_DATA(v)); \ + } while (0) + + if (kind == PyUnicode_1BYTE_KIND) { + Py_UCS1 *p; + WRITE_DIGITS(Py_UCS1); + } + else if (kind == PyUnicode_2BYTE_KIND) { + Py_UCS2 *p; + WRITE_DIGITS(Py_UCS2); + } + else { + assert (kind == PyUnicode_4BYTE_KIND); + Py_UCS4 *p; + WRITE_DIGITS(Py_UCS4); + } +#undef WRITE_DIGITS + + if (writer) { + writer->pos += sz; + } + else { + assert(_PyUnicode_CheckConsistency(v, 1)); + *p_output = v; + } + return 0; +} + +PyObject * +_PyLong_Format(PyObject *obj, int base) +{ + PyObject *str; + int err; + if (base == 10) + err = long_to_decimal_string_internal(obj, &str, NULL); + else + err = long_format_binary(obj, base, 1, &str, NULL); + if (err == -1) return NULL; - } - assert(PyUnicode_KIND(v) == PyUnicode_1BYTE_KIND); - - p = PyUnicode_1BYTE_DATA(v) + sz; - if (size_a == 0) { - *--p = '0'; - } - else { - /* JRH: special case for power-of-2 bases */ - twodigits accum = 0; - int accumbits = 0; /* # of bits in accum */ - Py_ssize_t i; - for (i = 0; i < size_a; ++i) { - accum |= (twodigits)a->ob_digit[i] << accumbits; - accumbits += PyLong_SHIFT; - assert(accumbits >= bits); - do { - char cdigit; - cdigit = (char)(accum & (base - 1)); - cdigit += (cdigit < 10) ? '0' : 'a'-10; - *--p = cdigit; - accumbits -= bits; - accum >>= bits; - } while (i < size_a-1 ? accumbits >= bits : accum > 0); - } - } - - if (base == 16) - *--p = 'x'; - else if (base == 8) - *--p = 'o'; - else /* (base == 2) */ - *--p = 'b'; - *--p = '0'; - if (negative) - *--p = '-'; - assert(p == PyUnicode_1BYTE_DATA(v)); - assert(_PyUnicode_CheckConsistency(v, 1)); - return v; + return str; +} + +int +_PyLong_FormatWriter(_PyUnicodeWriter *writer, + PyObject *obj, + int base, int alternate) +{ + if (base == 10) + return long_to_decimal_string_internal(obj, NULL, writer); + else + return long_format_binary(obj, base, alternate, NULL, writer); } /* Table of digit values for 8-bit string -> integer conversion. @@ -4232,11 +4352,22 @@ long__format__(PyObject *self, PyObject *args) { PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) return NULL; - return _PyLong_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); + + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyLong_FormatAdvancedWriter( + &writer, + self, + format_spec, 0, PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } /* Return a pair (q, r) such that a = b * q + r, and diff --git a/Objects/stringlib/asciilib.h b/Objects/stringlib/asciilib.h --- a/Objects/stringlib/asciilib.h +++ b/Objects/stringlib/asciilib.h @@ -18,7 +18,7 @@ #define STRINGLIB_TODECIMAL Py_UNICODE_TODECIMAL #define STRINGLIB_STR PyUnicode_1BYTE_DATA #define STRINGLIB_LEN PyUnicode_GET_LENGTH -#define STRINGLIB_NEW unicode_fromascii +#define STRINGLIB_NEW(STR,LEN) _PyUnicode_FromASCII((char*)(STR),(LEN)) #define STRINGLIB_RESIZE not_supported #define STRINGLIB_CHECK PyUnicode_Check #define STRINGLIB_CHECK_EXACT PyUnicode_CheckExact diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h --- a/Objects/stringlib/unicode_format.h +++ b/Objects/stringlib/unicode_format.h @@ -499,26 +499,26 @@ int ok = 0; PyObject *result = NULL; PyObject *format_spec_object = NULL; - PyObject *(*formatter)(PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL; - Py_ssize_t len; + int (*formatter) (_PyUnicodeWriter*, PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL; + int err; /* If we know the type exactly, skip the lookup of __format__ and just call the formatter directly. */ if (PyUnicode_CheckExact(fieldobj)) - formatter = _PyUnicode_FormatAdvanced; + formatter = _PyUnicode_FormatAdvancedWriter; else if (PyLong_CheckExact(fieldobj)) - formatter =_PyLong_FormatAdvanced; + formatter = _PyLong_FormatAdvancedWriter; else if (PyFloat_CheckExact(fieldobj)) - formatter = _PyFloat_FormatAdvanced; - - /* XXX: for 2.6, convert format_spec to the appropriate type - (unicode, str) */ + formatter = _PyFloat_FormatAdvancedWriter; + else if (PyComplex_CheckExact(fieldobj)) + formatter = _PyComplex_FormatAdvancedWriter; if (formatter) { /* we know exactly which formatter will be called when __format__ is looked up, so call it directly, instead. */ - result = formatter(fieldobj, format_spec->str, - format_spec->start, format_spec->end); + err = formatter(writer, fieldobj, format_spec->str, + format_spec->start, format_spec->end); + return (err == 0); } else { /* We need to create an object out of the pointers we have, because @@ -536,17 +536,11 @@ } if (result == NULL) goto done; - if (PyUnicode_READY(result) == -1) + + if (_PyUnicodeWriter_WriteStr(writer, result) == -1) goto done; + ok = 1; - len = PyUnicode_GET_LENGTH(result); - if (_PyUnicodeWriter_Prepare(writer, - len, PyUnicode_MAX_CHAR_VALUE(result)) == -1) - goto done; - copy_characters(writer->buffer, writer->pos, - result, 0, len); - writer->pos += len; - ok = 1; done: Py_XDECREF(format_spec_object); Py_XDECREF(result); @@ -897,16 +891,19 @@ err = _PyUnicodeWriter_Prepare(writer, sublen, maxchar); if (err == -1) return 0; - copy_characters(writer->buffer, writer->pos, - literal.str, literal.start, sublen); + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + literal.str, literal.start, sublen); writer->pos += sublen; } - if (field_present) + if (field_present) { + if (iter.str.start == iter.str.end) + writer->flags.overallocate = 0; if (!output_markup(&field_name, &format_spec, format_spec_needs_expanding, conversion, writer, args, kwargs, recursion_depth, auto_number)) return 0; + } } return result; } @@ -921,7 +918,7 @@ int recursion_depth, AutoNumber *auto_number) { _PyUnicodeWriter writer; - Py_ssize_t initlen; + Py_ssize_t minlen; /* check the recursion level */ if (recursion_depth <= 0) { @@ -930,9 +927,8 @@ return NULL; } - initlen = PyUnicode_GET_LENGTH(input->str) + 100; - if (_PyUnicodeWriter_Init(&writer, initlen, 127) == -1) - return NULL; + minlen = PyUnicode_GET_LENGTH(input->str) + 100; + _PyUnicodeWriter_Init(&writer, minlen); if (!do_markup(input, args, kwargs, &writer, recursion_depth, auto_number)) { diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -225,16 +225,10 @@ /* forward */ static PyUnicodeObject *_PyUnicode_New(Py_ssize_t length); static PyObject* get_latin1_char(unsigned char ch); -static void copy_characters( - PyObject *to, Py_ssize_t to_start, - PyObject *from, Py_ssize_t from_start, - Py_ssize_t how_many); static int unicode_modifiable(PyObject *unicode); static PyObject * -unicode_fromascii(const unsigned char *s, Py_ssize_t size); -static PyObject * _PyUnicode_FromUCS1(const unsigned char *s, Py_ssize_t size); static PyObject * _PyUnicode_FromUCS2(const Py_UCS2 *s, Py_ssize_t size); @@ -783,7 +777,7 @@ return NULL; copy_length = Py_MIN(length, PyUnicode_GET_LENGTH(unicode)); - copy_characters(copy, 0, unicode, 0, copy_length); + _PyUnicode_FastCopyCharacters(copy, 0, unicode, 0, copy_length); return copy; } else { @@ -1154,15 +1148,16 @@ assert(0 <= from_start); assert(0 <= to_start); assert(PyUnicode_Check(from)); - assert(PyUnicode_Check(to)); assert(PyUnicode_IS_READY(from)); - assert(PyUnicode_IS_READY(to)); assert(from_start + how_many <= PyUnicode_GET_LENGTH(from)); - assert(to_start + how_many <= PyUnicode_GET_LENGTH(to)); if (how_many == 0) return 0; + assert(PyUnicode_Check(to)); + assert(PyUnicode_IS_READY(to)); + assert(to_start + how_many <= PyUnicode_GET_LENGTH(to)); + from_kind = PyUnicode_KIND(from); from_data = PyUnicode_DATA(from); to_kind = PyUnicode_KIND(to); @@ -1267,10 +1262,10 @@ return 0; } -static void -copy_characters(PyObject *to, Py_ssize_t to_start, - PyObject *from, Py_ssize_t from_start, - Py_ssize_t how_many) +void +_PyUnicode_FastCopyCharacters( + PyObject *to, Py_ssize_t to_start, + PyObject *from, Py_ssize_t from_start, Py_ssize_t how_many) { (void)_copy_characters(to, to_start, from, from_start, how_many, 0); } @@ -1292,6 +1287,14 @@ if (PyUnicode_READY(to) == -1) return -1; + if (from_start < 0) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return -1; + } + if (to_start < 0) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return -1; + } how_many = Py_MIN(PyUnicode_GET_LENGTH(from), how_many); if (to_start + how_many > PyUnicode_GET_LENGTH(to)) { PyErr_Format(PyExc_SystemError, @@ -1641,7 +1644,7 @@ maxchar); if (result == NULL) return -1; - PyUnicode_CopyCharacters(result, 0, *p_unicode, 0, length); + _PyUnicode_FastCopyCharacters(result, 0, *p_unicode, 0, length); Py_DECREF(*p_unicode); *p_unicode = result; return 0; @@ -1841,9 +1844,10 @@ /* Internal function, doesn't check maximum character */ -static PyObject* -unicode_fromascii(const unsigned char* s, Py_ssize_t size) -{ +PyObject* +_PyUnicode_FromASCII(const char *buffer, Py_ssize_t size) +{ + const unsigned char *s = (const unsigned char *)buffer; PyObject *unicode; if (size == 1) { #ifdef Py_DEBUG @@ -2085,7 +2089,7 @@ return; } copy = PyUnicode_New(len, max_char); - copy_characters(copy, 0, unicode, 0, len); + _PyUnicode_FastCopyCharacters(copy, 0, unicode, 0, len); Py_DECREF(unicode); *p_unicode = copy; } @@ -2753,7 +2757,7 @@ (void) va_arg(vargs, char *); size = PyUnicode_GET_LENGTH(*callresult); assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string)); - copy_characters(string, i, *callresult, 0, size); + _PyUnicode_FastCopyCharacters(string, i, *callresult, 0, size); i += size; /* We're done with the unicode()/repr() => forget it */ Py_DECREF(*callresult); @@ -2767,7 +2771,7 @@ Py_ssize_t size; assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string)); size = PyUnicode_GET_LENGTH(obj); - copy_characters(string, i, obj, 0, size); + _PyUnicode_FastCopyCharacters(string, i, obj, 0, size); i += size; break; } @@ -2779,13 +2783,13 @@ if (obj) { size = PyUnicode_GET_LENGTH(obj); assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string)); - copy_characters(string, i, obj, 0, size); + _PyUnicode_FastCopyCharacters(string, i, obj, 0, size); i += size; } else { size = PyUnicode_GET_LENGTH(*callresult); assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string)); - copy_characters(string, i, *callresult, 0, size); + _PyUnicode_FastCopyCharacters(string, i, *callresult, 0, size); i += size; Py_DECREF(*callresult); } @@ -2800,7 +2804,7 @@ /* unused, since we already have the result */ (void) va_arg(vargs, PyObject *); assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string)); - copy_characters(string, i, *callresult, 0, size); + _PyUnicode_FastCopyCharacters(string, i, *callresult, 0, size); i += size; /* We're done with the unicode()/repr() => forget it */ Py_DECREF(*callresult); @@ -4171,7 +4175,7 @@ if (unicode_widen(output, *outpos, PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0) goto onError; - copy_characters(*output, *outpos, repunicode, 0, replen); + _PyUnicode_FastCopyCharacters(*output, *outpos, repunicode, 0, replen); *outpos += replen; } else { @@ -9216,12 +9220,14 @@ /* If the maxchar increased so that the kind changed, not all characters are representable anymore and we need to fix the string again. This only happens in very few cases. */ - copy_characters(v, 0, self, 0, PyUnicode_GET_LENGTH(self)); + _PyUnicode_FastCopyCharacters(v, 0, + self, 0, PyUnicode_GET_LENGTH(self)); maxchar_old = fixfct(v); assert(maxchar_old > 0 && maxchar_old <= maxchar_new); } else { - copy_characters(v, 0, u, 0, PyUnicode_GET_LENGTH(self)); + _PyUnicode_FastCopyCharacters(v, 0, + u, 0, PyUnicode_GET_LENGTH(self)); } Py_DECREF(u); assert(_PyUnicode_CheckConsistency(v, 1)); @@ -9603,7 +9609,7 @@ res_data += kind * seplen; } else { - copy_characters(res, res_offset, sep, 0, seplen); + _PyUnicode_FastCopyCharacters(res, res_offset, sep, 0, seplen); res_offset += seplen; } } @@ -9616,7 +9622,7 @@ res_data += kind * itemlen; } else { - copy_characters(res, res_offset, item, 0, itemlen); + _PyUnicode_FastCopyCharacters(res, res_offset, item, 0, itemlen); res_offset += itemlen; } } @@ -9663,13 +9669,25 @@ } \ } while (0) +void +_PyUnicode_FastFill(PyObject *unicode, Py_ssize_t start, Py_ssize_t length, + Py_UCS4 fill_char) +{ + const enum PyUnicode_Kind kind = PyUnicode_KIND(unicode); + const void *data = PyUnicode_DATA(unicode); + assert(PyUnicode_IS_READY(unicode)); + assert(unicode_modifiable(unicode)); + assert(fill_char <= PyUnicode_MAX_CHAR_VALUE(unicode)); + assert(start >= 0); + assert(start + length <= PyUnicode_GET_LENGTH(unicode)); + FILL(kind, data, fill_char, start, length); +} + Py_ssize_t PyUnicode_Fill(PyObject *unicode, Py_ssize_t start, Py_ssize_t length, Py_UCS4 fill_char) { Py_ssize_t maxlen; - enum PyUnicode_Kind kind; - void *data; if (!PyUnicode_Check(unicode)) { PyErr_BadInternalCall(); @@ -9680,6 +9698,10 @@ if (unicode_check_modifiable(unicode)) return -1; + if (start < 0) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return -1; + } if (fill_char > PyUnicode_MAX_CHAR_VALUE(unicode)) { PyErr_SetString(PyExc_ValueError, "fill character is bigger than " @@ -9692,9 +9714,7 @@ if (length <= 0) return 0; - kind = PyUnicode_KIND(unicode); - data = PyUnicode_DATA(unicode); - FILL(kind, data, fill_char, start, length); + _PyUnicode_FastFill(unicode, start, length, fill_char); return length; } @@ -9734,7 +9754,7 @@ FILL(kind, data, fill, 0, left); if (right) FILL(kind, data, fill, left + _PyUnicode_LENGTH(self), right); - copy_characters(u, left, self, 0, _PyUnicode_LENGTH(self)); + _PyUnicode_FastCopyCharacters(u, left, self, 0, _PyUnicode_LENGTH(self)); assert(_PyUnicode_CheckConsistency(u, 1)); return u; } @@ -10058,7 +10078,7 @@ u = PyUnicode_New(slen, maxchar); if (!u) goto error; - copy_characters(u, 0, self, 0, slen); + _PyUnicode_FastCopyCharacters(u, 0, self, 0, slen); rkind = PyUnicode_KIND(u); PyUnicode_WRITE(rkind, PyUnicode_DATA(u), pos, u2); @@ -10626,8 +10646,8 @@ w = PyUnicode_New(new_len, maxchar); if (w == NULL) goto onError; - copy_characters(w, 0, u, 0, u_len); - copy_characters(w, u_len, v, 0, v_len); + _PyUnicode_FastCopyCharacters(w, 0, u, 0, u_len); + _PyUnicode_FastCopyCharacters(w, u_len, v, 0, v_len); Py_DECREF(u); Py_DECREF(v); assert(_PyUnicode_CheckConsistency(w, 1)); @@ -10702,7 +10722,7 @@ goto error; } /* copy 'right' into the newly allocated area of 'left' */ - copy_characters(*p_left, left_len, right, 0, right_len); + _PyUnicode_FastCopyCharacters(*p_left, left_len, right, 0, right_len); } else { maxchar = PyUnicode_MAX_CHAR_VALUE(left); @@ -10713,8 +10733,8 @@ res = PyUnicode_New(new_len, maxchar); if (res == NULL) goto error; - copy_characters(res, 0, left, 0, left_len); - copy_characters(res, left_len, right, 0, right_len); + _PyUnicode_FastCopyCharacters(res, 0, left, 0, left_len); + _PyUnicode_FastCopyCharacters(res, left_len, right, 0, right_len); Py_DECREF(left); *p_left = res; } @@ -11650,7 +11670,7 @@ length = end - start; if (PyUnicode_IS_ASCII(self)) { data = PyUnicode_1BYTE_DATA(self); - return unicode_fromascii(data + start, length); + return _PyUnicode_FromASCII((char*)(data + start), length); } else { kind = PyUnicode_KIND(self); @@ -12769,60 +12789,74 @@ return PyBool_FromLong(result); } -typedef struct { - PyObject *buffer; - void *data; - enum PyUnicode_Kind kind; - Py_UCS4 maxchar; - Py_ssize_t pos; -} _PyUnicodeWriter ; - Py_LOCAL_INLINE(void) _PyUnicodeWriter_Update(_PyUnicodeWriter *writer) { + writer->size = PyUnicode_GET_LENGTH(writer->buffer); writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); writer->data = PyUnicode_DATA(writer->buffer); writer->kind = PyUnicode_KIND(writer->buffer); } -Py_LOCAL(int) -_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, - Py_ssize_t length, Py_UCS4 maxchar) -{ - writer->pos = 0; - writer->buffer = PyUnicode_New(length, maxchar); - if (writer->buffer == NULL) - return -1; - _PyUnicodeWriter_Update(writer); - return 0; -} - -Py_LOCAL_INLINE(int) -_PyUnicodeWriter_Prepare(_PyUnicodeWriter *writer, - Py_ssize_t length, Py_UCS4 maxchar) +void +_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, Py_ssize_t min_length) +{ + memset(writer, 0, sizeof(*writer)); +#ifdef Py_DEBUG + writer->kind = 5; /* invalid kind */ +#endif + writer->min_length = Py_MAX(min_length, 100); + writer->flags.overallocate = (min_length > 0); +} + +int +_PyUnicodeWriter_PrepareInternal(_PyUnicodeWriter *writer, + Py_ssize_t length, Py_UCS4 maxchar) { Py_ssize_t newlen; PyObject *newbuffer; + assert(length > 0); + if (length > PY_SSIZE_T_MAX - writer->pos) { PyErr_NoMemory(); return -1; } newlen = writer->pos + length; - if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { - /* overallocate 25% to limit the number of resize */ - if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) - newlen += newlen / 4; - - if (maxchar > writer->maxchar) { + if (writer->buffer == NULL) { + if (writer->flags.overallocate) { + /* overallocate 25% to limit the number of resize */ + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + newlen += newlen / 4; + if (newlen < writer->min_length) + newlen = writer->min_length; + } + writer->buffer = PyUnicode_New(newlen, maxchar); + if (writer->buffer == NULL) + return -1; + _PyUnicodeWriter_Update(writer); + return 0; + } + + if (newlen > writer->size) { + if (writer->flags.overallocate) { + /* overallocate 25% to limit the number of resize */ + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + newlen += newlen / 4; + if (newlen < writer->min_length) + newlen = writer->min_length; + } + + if (maxchar > writer->maxchar || writer->flags.readonly) { /* resize + widen */ newbuffer = PyUnicode_New(newlen, maxchar); if (newbuffer == NULL) return -1; - PyUnicode_CopyCharacters(newbuffer, 0, - writer->buffer, 0, writer->pos); + _PyUnicode_FastCopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); Py_DECREF(writer->buffer); + writer->flags.readonly = 0; } else { newbuffer = resize_compact(writer->buffer, newlen); @@ -12833,25 +12867,76 @@ _PyUnicodeWriter_Update(writer); } else if (maxchar > writer->maxchar) { - if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) + assert(!writer->flags.readonly); + newbuffer = PyUnicode_New(writer->size, maxchar); + if (newbuffer == NULL) return -1; + _PyUnicode_FastCopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); + Py_DECREF(writer->buffer); + writer->buffer = newbuffer; _PyUnicodeWriter_Update(writer); } return 0; } -Py_LOCAL(PyObject *) +int +_PyUnicodeWriter_WriteStr(_PyUnicodeWriter *writer, PyObject *str) +{ + Py_UCS4 maxchar; + Py_ssize_t len; + + if (PyUnicode_READY(str) == -1) + return -1; + len = PyUnicode_GET_LENGTH(str); + if (len == 0) + return 0; + maxchar = PyUnicode_MAX_CHAR_VALUE(str); + if (maxchar > writer->maxchar || len > writer->size - writer->pos) { + if (writer->buffer == NULL && !writer->flags.overallocate) { + Py_INCREF(str); + writer->buffer = str; + _PyUnicodeWriter_Update(writer); + writer->flags.readonly = 1; + writer->size = 0; + writer->pos += len; + return 0; + } + if (_PyUnicodeWriter_PrepareInternal(writer, len, maxchar) == -1) + return -1; + } + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + str, 0, len); + writer->pos += len; + return 0; +} + +PyObject * _PyUnicodeWriter_Finish(_PyUnicodeWriter *writer) { - if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { - Py_DECREF(writer->buffer); - return NULL; + if (writer->pos == 0) { + Py_XDECREF(writer->buffer); + Py_INCREF(unicode_empty); + return unicode_empty; + } + if (writer->flags.readonly) { + assert(PyUnicode_GET_LENGTH(writer->buffer) == writer->pos); + return writer->buffer; + } + if (PyUnicode_GET_LENGTH(writer->buffer) != writer->pos) { + PyObject *newbuffer; + newbuffer = resize_compact(writer->buffer, writer->pos); + if (newbuffer == NULL) { + Py_DECREF(writer->buffer); + return NULL; + } + writer->buffer = newbuffer; } assert(_PyUnicode_CheckConsistency(writer->buffer, 1)); return writer->buffer; } -Py_LOCAL(void) +void _PyUnicodeWriter_Dealloc(_PyUnicodeWriter *writer) { Py_CLEAR(writer->buffer); @@ -12874,14 +12959,24 @@ static PyObject * unicode__format__(PyObject* self, PyObject* args) { - PyObject *format_spec, *out; + PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) return NULL; - out = _PyUnicode_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); - return out; + if (PyUnicode_READY(self) == -1) + return NULL; + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyUnicode_FormatAdvancedWriter(&writer, + self, format_spec, 0, + PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } PyDoc_STRVAR(p_format__doc__, @@ -13111,16 +13206,17 @@ /* Returns a new reference to a PyUnicode object, or NULL on failure. */ -static PyObject * -formatfloat(PyObject *v, int flags, int prec, int type) +static int +formatfloat(PyObject *v, int flags, int prec, int type, + PyObject **p_output, _PyUnicodeWriter *writer) { char *p; - PyObject *result; double x; + Py_ssize_t len; x = PyFloat_AsDouble(v); if (x == -1.0 && PyErr_Occurred()) - return NULL; + return -1; if (prec < 0) prec = 6; @@ -13128,10 +13224,20 @@ p = PyOS_double_to_string(x, type, prec, (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL); if (p == NULL) - return NULL; - result = unicode_fromascii((unsigned char*)p, strlen(p)); + return -1; + len = strlen(p); + if (writer) { + if (_PyUnicodeWriter_Prepare(writer, len, 127) == -1) + return -1; + memcpy(writer->data + writer->pos * writer->kind, + p, + len); + writer->pos += len; + } + else + *p_output = _PyUnicode_FromASCII(p, len); PyMem_Free(p); - return result; + return 0; } /* formatlong() emulates the format codes d, u, o, x and X, and @@ -13267,7 +13373,7 @@ } if (!PyUnicode_Check(result) || len != PyUnicode_GET_LENGTH(result)) { PyObject *unicode; - unicode = unicode_fromascii((unsigned char *)buf, len); + unicode = _PyUnicode_FromASCII(buf, len); Py_DECREF(result); result = unicode; } @@ -13336,8 +13442,7 @@ fmtcnt = PyUnicode_GET_LENGTH(uformat); fmtpos = 0; - if (_PyUnicodeWriter_Init(&writer, fmtcnt + 100, 127) < 0) - goto onError; + _PyUnicodeWriter_Init(&writer, fmtcnt + 100); if (PyTuple_Check(args)) { arglen = PyTuple_Size(args); @@ -13368,8 +13473,8 @@ if (_PyUnicodeWriter_Prepare(&writer, sublen, maxchar) == -1) goto onError; - copy_characters(writer.buffer, writer.pos, - uformat, nonfmtpos, sublen); + _PyUnicode_FastCopyCharacters(writer.buffer, writer.pos, + uformat, nonfmtpos, sublen); writer.pos += sublen; } else { @@ -13530,6 +13635,8 @@ "incomplete format"); goto onError; } + if (fmtcnt == 0) + writer.flags.overallocate = 0; if (c == '%') { if (_PyUnicodeWriter_Prepare(&writer, 1, '%') == -1) @@ -13539,7 +13646,6 @@ continue; } - v = getnextarg(args, arglen, &argidx); if (v == NULL) goto onError; @@ -13552,6 +13658,13 @@ case 's': case 'r': case 'a': + if (PyLong_CheckExact(v) && width == -1 && prec == -1) { + /* Fast path */ + if (_PyLong_FormatWriter(&writer, v, 10, flags & F_ALT) == -1) + goto onError; + goto nextarg; + } + if (PyUnicode_CheckExact(v) && c == 's') { temp = v; Py_INCREF(temp); @@ -13572,6 +13685,32 @@ case 'o': case 'x': case 'X': + if (PyLong_CheckExact(v) + && width == -1 && prec == -1 + && !(flags & (F_SIGN | F_BLANK))) + { + /* Fast path */ + switch(c) + { + case 'd': + case 'i': + case 'u': + if (_PyLong_FormatWriter(&writer, v, 10, flags & F_ALT) == -1) + goto onError; + goto nextarg; + case 'x': + if (_PyLong_FormatWriter(&writer, v, 16, flags & F_ALT) == -1) + goto onError; + goto nextarg; + case 'o': + if (_PyLong_FormatWriter(&writer, v, 8, flags & F_ALT) == -1) + goto onError; + goto nextarg; + default: + break; + } + } + isnumok = 0; if (PyNumber_Check(v)) { PyObject *iobj=NULL; @@ -13611,10 +13750,20 @@ case 'F': case 'g': case 'G': + if (width == -1 && prec == -1 + && !(flags & (F_SIGN | F_BLANK))) + { + /* Fast path */ + if (formatfloat(v, flags, prec, c, NULL, &writer) == -1) + goto onError; + goto nextarg; + } + sign = 1; if (flags & F_ZERO) fill = '0'; - temp = formatfloat(v, flags, prec, c); + if (formatfloat(v, flags, prec, c, &temp, NULL) == -1) + temp = NULL; break; case 'c': @@ -13622,6 +13771,14 @@ Py_UCS4 ch = formatchar(v); if (ch == (Py_UCS4) -1) goto onError; + if (width == -1 && prec == -1) { + /* Fast path */ + if (_PyUnicodeWriter_Prepare(&writer, 1, ch) == -1) + goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, ch); + writer.pos += 1; + goto nextarg; + } temp = PyUnicode_FromOrdinal(ch); break; } @@ -13638,6 +13795,16 @@ if (temp == NULL) goto onError; assert (PyUnicode_Check(temp)); + + if (width == -1 && prec == -1 + && !(flags & (F_SIGN | F_BLANK))) + { + /* Fast path */ + if (_PyUnicodeWriter_WriteStr(&writer, temp) == -1) + goto onError; + goto nextarg; + } + if (PyUnicode_READY(temp) == -1) { Py_CLEAR(temp); goto onError; @@ -13676,15 +13843,15 @@ if (!(flags & F_LJUST)) { if (sign) { if ((width-1) > len) - bufmaxchar = Py_MAX(bufmaxchar, fill); + bufmaxchar = MAX_MAXCHAR(bufmaxchar, fill); } else { if (width > len) - bufmaxchar = Py_MAX(bufmaxchar, fill); + bufmaxchar = MAX_MAXCHAR(bufmaxchar, fill); } } maxchar = _PyUnicode_FindMaxChar(temp, 0, pindex+len); - bufmaxchar = Py_MAX(bufmaxchar, maxchar); + bufmaxchar = MAX_MAXCHAR(bufmaxchar, maxchar); buflen = width; if (sign && len == width) @@ -13737,8 +13904,8 @@ } } - copy_characters(writer.buffer, writer.pos, - temp, pindex, len); + _PyUnicode_FastCopyCharacters(writer.buffer, writer.pos, + temp, pindex, len); writer.pos += len; if (width > len) { sublen = width - len; @@ -13746,6 +13913,7 @@ writer.pos += sublen; } +nextarg: if (dict && (argidx < arglen) && c != '%') { PyErr_SetString(PyExc_TypeError, "not all arguments converted during string formatting"); diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c --- a/Python/formatter_unicode.c +++ b/Python/formatter_unicode.c @@ -316,21 +316,28 @@ /* Do the padding, and return a pointer to where the caller-supplied content goes. */ static Py_ssize_t -fill_padding(PyObject *s, Py_ssize_t start, Py_ssize_t nchars, +fill_padding(_PyUnicodeWriter *writer, + Py_ssize_t nchars, Py_UCS4 fill_char, Py_ssize_t n_lpadding, Py_ssize_t n_rpadding) { + Py_ssize_t pos; + /* Pad on left. */ - if (n_lpadding) - PyUnicode_Fill(s, start, start + n_lpadding, fill_char); + if (n_lpadding) { + pos = writer->pos; + _PyUnicode_FastFill(writer->buffer, pos, n_lpadding, fill_char); + } /* Pad on right. */ - if (n_rpadding) - PyUnicode_Fill(s, start + nchars + n_lpadding, - start + nchars + n_lpadding + n_rpadding, fill_char); + if (n_rpadding) { + pos = writer->pos + nchars + n_lpadding; + _PyUnicode_FastFill(writer->buffer, pos, n_rpadding, fill_char); + } /* Pointer to the user content. */ - return start + n_lpadding; + writer->pos += n_lpadding; + return 0; } /************************************************************************/ @@ -541,7 +548,7 @@ as determined in calc_number_widths(). Return -1 on error, or 0 on success. */ static int -fill_number(PyObject *out, Py_ssize_t pos, const NumberFieldWidths *spec, +fill_number(_PyUnicodeWriter *writer, const NumberFieldWidths *spec, PyObject *digits, Py_ssize_t d_start, Py_ssize_t d_end, PyObject *prefix, Py_ssize_t p_start, Py_UCS4 fill_char, @@ -549,36 +556,38 @@ { /* Used to keep track of digits, decimal, and remainder. */ Py_ssize_t d_pos = d_start; - unsigned int kind = PyUnicode_KIND(out); - void *data = PyUnicode_DATA(out); + const enum PyUnicode_Kind kind = writer->kind; + const void *data = writer->data; Py_ssize_t r; if (spec->n_lpadding) { - PyUnicode_Fill(out, pos, pos + spec->n_lpadding, fill_char); - pos += spec->n_lpadding; + _PyUnicode_FastFill(writer->buffer, + writer->pos, spec->n_lpadding, fill_char); + writer->pos += spec->n_lpadding; } if (spec->n_sign == 1) { - PyUnicode_WRITE(kind, data, pos++, spec->sign); + PyUnicode_WRITE(kind, data, writer->pos, spec->sign); + writer->pos++; } if (spec->n_prefix) { - if (PyUnicode_CopyCharacters(out, pos, - prefix, p_start, - spec->n_prefix) < 0) - return -1; + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + prefix, p_start, + spec->n_prefix); if (toupper) { Py_ssize_t t; for (t = 0; t < spec->n_prefix; t++) { - Py_UCS4 c = PyUnicode_READ(kind, data, pos + t); + Py_UCS4 c = PyUnicode_READ(kind, data, writer->pos + t); c = Py_TOUPPER(c); assert (c <= 127); - PyUnicode_WRITE(kind, data, pos + t, c); + PyUnicode_WRITE(kind, data, writer->pos + t, c); } } - pos += spec->n_prefix; + writer->pos += spec->n_prefix; } if (spec->n_spadding) { - PyUnicode_Fill(out, pos, pos + spec->n_spadding, fill_char); - pos += spec->n_spadding; + _PyUnicode_FastFill(writer->buffer, + writer->pos, spec->n_spadding, fill_char); + writer->pos += spec->n_spadding; } /* Only for type 'c' special case, it has no digits. */ @@ -594,7 +603,7 @@ return -1; } r = _PyUnicode_InsertThousandsGrouping( - out, pos, + writer->buffer, writer->pos, spec->n_grouped_digits, pdigits + kind * d_pos, spec->n_digits, spec->n_min_width, @@ -609,34 +618,38 @@ if (toupper) { Py_ssize_t t; for (t = 0; t < spec->n_grouped_digits; t++) { - Py_UCS4 c = PyUnicode_READ(kind, data, pos + t); + Py_UCS4 c = PyUnicode_READ(kind, data, writer->pos + t); c = Py_TOUPPER(c); if (c > 127) { PyErr_SetString(PyExc_SystemError, "non-ascii grouped digit"); return -1; } - PyUnicode_WRITE(kind, data, pos + t, c); + PyUnicode_WRITE(kind, data, writer->pos + t, c); } } - pos += spec->n_grouped_digits; + writer->pos += spec->n_grouped_digits; if (spec->n_decimal) { - if (PyUnicode_CopyCharacters(out, pos, locale->decimal_point, 0, spec->n_decimal) < 0) - return -1; - pos += spec->n_decimal; + _PyUnicode_FastCopyCharacters( + writer->buffer, writer->pos, + locale->decimal_point, 0, spec->n_decimal); + writer->pos += spec->n_decimal; d_pos += 1; } if (spec->n_remainder) { - if (PyUnicode_CopyCharacters(out, pos, digits, d_pos, spec->n_remainder) < 0) - return -1; - pos += spec->n_remainder; + _PyUnicode_FastCopyCharacters( + writer->buffer, writer->pos, + digits, d_pos, spec->n_remainder); + writer->pos += spec->n_remainder; d_pos += spec->n_remainder; } if (spec->n_rpadding) { - PyUnicode_Fill(out, pos, pos + spec->n_rpadding, fill_char); - pos += spec->n_rpadding; + _PyUnicode_FastFill(writer->buffer, + writer->pos, spec->n_rpadding, + fill_char); + writer->pos += spec->n_rpadding; } return 0; } @@ -707,17 +720,20 @@ /*********** string formatting ******************************************/ /************************************************************************/ -static PyObject * -format_string_internal(PyObject *value, const InternalFormatSpec *format) +static int +format_string_internal(PyObject *value, const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { Py_ssize_t lpad; Py_ssize_t rpad; Py_ssize_t total; - Py_ssize_t pos; - Py_ssize_t len = PyUnicode_GET_LENGTH(value); - PyObject *result = NULL; + Py_ssize_t len; + int result = -1; Py_UCS4 maxchar; + assert(PyUnicode_IS_READY(value)); + len = PyUnicode_GET_LENGTH(value); + /* sign is not allowed on strings */ if (format->sign != '\0') { PyErr_SetString(PyExc_ValueError, @@ -741,6 +757,11 @@ goto done; } + if (format->width == -1 && format->precision == -1) { + /* Fast path */ + return _PyUnicodeWriter_WriteStr(writer, value); + } + /* if precision is specified, output no more that format.precision characters */ if (format->precision >= 0 && len >= format->precision) { @@ -754,21 +775,23 @@ maxchar = Py_MAX(maxchar, format->fill_char); /* allocate the resulting string */ - result = PyUnicode_New(total, maxchar); - if (result == NULL) + if (_PyUnicodeWriter_Prepare(writer, total, maxchar) == -1) goto done; /* Write into that space. First the padding. */ - pos = fill_padding(result, 0, len, - format->fill_char=='\0'?' ':format->fill_char, - lpad, rpad); + result = fill_padding(writer, len, + format->fill_char=='\0'?' ':format->fill_char, + lpad, rpad); + if (result == -1) + goto done; /* Then the source string. */ - if (PyUnicode_CopyCharacters(result, pos, value, 0, len) < 0) - Py_CLEAR(result); + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + value, 0, len); + writer->pos += (len + rpad); + result = 0; done: - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } @@ -780,11 +803,11 @@ typedef PyObject* (*IntOrLongToString)(PyObject *value, int base); -static PyObject * -format_int_or_long_internal(PyObject *value, const InternalFormatSpec *format, - IntOrLongToString tostring) +static int +format_long_internal(PyObject *value, const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { - PyObject *result = NULL; + int result = -1; Py_UCS4 maxchar = 127; PyObject *tmp = NULL; Py_ssize_t inumeric_chars; @@ -798,7 +821,6 @@ Py_ssize_t prefix = 0; NumberFieldWidths spec; long x; - int err; /* Locale settings, either from the actual locale or from a hard-code pseudo-locale */ @@ -872,13 +894,23 @@ break; } + if (format->sign != '+' && format->sign != ' ' + && format->width == -1 + && format->type != 'X' && format->type != 'n' + && !format->thousands_separators + && PyLong_CheckExact(value)) + { + /* Fast path */ + return _PyLong_FormatWriter(writer, value, base, format->alternate); + } + /* The number of prefix chars is the same as the leading chars to skip */ if (format->alternate) n_prefix = leading_chars_to_skip; /* Do the hard part, converting to a string in a given base */ - tmp = tostring(value, base); + tmp = _PyLong_Format(value, base); if (tmp == NULL || PyUnicode_READY(tmp) == -1) goto done; @@ -914,23 +946,19 @@ &locale, format, &maxchar); /* Allocate the memory. */ - result = PyUnicode_New(n_total, maxchar); - if (!result) + if (_PyUnicodeWriter_Prepare(writer, n_total, maxchar) == -1) goto done; /* Populate the memory. */ - err = fill_number(result, 0, &spec, - tmp, inumeric_chars, inumeric_chars + n_digits, - tmp, prefix, - format->fill_char == '\0' ? ' ' : format->fill_char, - &locale, format->type == 'X'); - if (err) - Py_CLEAR(result); + result = fill_number(writer, &spec, + tmp, inumeric_chars, inumeric_chars + n_digits, + tmp, prefix, + format->fill_char == '\0' ? ' ' : format->fill_char, + &locale, format->type == 'X'); done: Py_XDECREF(tmp); free_locale_info(&locale); - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } @@ -938,16 +966,11 @@ /*********** float formatting *******************************************/ /************************************************************************/ -static PyObject* -strtounicode(char *charbuffer, Py_ssize_t len) -{ - return PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, charbuffer, len); -} - /* much of this is taken from unicodeobject.c */ -static PyObject * +static int format_float_internal(PyObject *value, - const InternalFormatSpec *format) + const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { char *buf = NULL; /* buffer returned from PyOS_double_to_string */ Py_ssize_t n_digits; @@ -962,12 +985,11 @@ Py_ssize_t index; NumberFieldWidths spec; int flags = 0; - PyObject *result = NULL; + int result = -1; Py_UCS4 maxchar = 127; Py_UCS4 sign_char = '\0'; int float_type; /* Used to see if we have a nan, inf, or regular float. */ PyObject *unicode_tmp = NULL; - int err; /* Locale settings, either from the actual locale or from a hard-code pseudo-locale */ @@ -1024,13 +1046,25 @@ /* Since there is no unicode version of PyOS_double_to_string, just use the 8 bit version and then convert to unicode. */ - unicode_tmp = strtounicode(buf, n_digits); + unicode_tmp = _PyUnicode_FromASCII(buf, n_digits); + PyMem_Free(buf); if (unicode_tmp == NULL) goto done; - index = 0; + + if (format->sign != '+' && format->sign != ' ' + && format->width == -1 + && format->type != 'n' + && !format->thousands_separators) + { + /* Fast path */ + result = _PyUnicodeWriter_WriteStr(writer, unicode_tmp); + Py_DECREF(unicode_tmp); + return result; + } /* Is a sign character present in the output? If so, remember it and skip it */ + index = 0; if (PyUnicode_READ_CHAR(unicode_tmp, index) == '-') { sign_char = '-'; ++index; @@ -1055,24 +1089,19 @@ &locale, format, &maxchar); /* Allocate the memory. */ - result = PyUnicode_New(n_total, maxchar); - if (result == NULL) + if (_PyUnicodeWriter_Prepare(writer, n_total, maxchar) == -1) goto done; /* Populate the memory. */ - err = fill_number(result, 0, &spec, - unicode_tmp, index, index + n_digits, - NULL, 0, - format->fill_char == '\0' ? ' ' : format->fill_char, - &locale, 0); - if (err) - Py_CLEAR(result); + result = fill_number(writer, &spec, + unicode_tmp, index, index + n_digits, + NULL, 0, + format->fill_char == '\0' ? ' ' : format->fill_char, + &locale, 0); done: - PyMem_Free(buf); Py_DECREF(unicode_tmp); free_locale_info(&locale); - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } @@ -1080,9 +1109,10 @@ /*********** complex formatting *****************************************/ /************************************************************************/ -static PyObject * +static int format_complex_internal(PyObject *value, - const InternalFormatSpec *format) + const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { double re; double im; @@ -1106,11 +1136,10 @@ NumberFieldWidths re_spec; NumberFieldWidths im_spec; int flags = 0; - PyObject *result = NULL; + int result = -1; Py_UCS4 maxchar = 127; - int rkind; + enum PyUnicode_Kind rkind; void *rdata; - Py_ssize_t index; Py_UCS4 re_sign_char = '\0'; Py_UCS4 im_sign_char = '\0'; int re_float_type; /* Used to see if we have a nan, inf, or regular float. */ @@ -1122,7 +1151,6 @@ Py_ssize_t total; PyObject *re_unicode_tmp = NULL; PyObject *im_unicode_tmp = NULL; - int err; /* Locale settings, either from the actual locale or from a hard-code pseudo-locale */ @@ -1191,12 +1219,12 @@ /* Since there is no unicode version of PyOS_double_to_string, just use the 8 bit version and then convert to unicode. */ - re_unicode_tmp = strtounicode(re_buf, n_re_digits); + re_unicode_tmp = _PyUnicode_FromASCII(re_buf, n_re_digits); if (re_unicode_tmp == NULL) goto done; i_re = 0; - im_unicode_tmp = strtounicode(im_buf, n_im_digits); + im_unicode_tmp = _PyUnicode_FromASCII(im_buf, n_im_digits); if (im_unicode_tmp == NULL) goto done; i_im = 0; @@ -1261,47 +1289,49 @@ if (lpad || rpad) maxchar = Py_MAX(maxchar, format->fill_char); - result = PyUnicode_New(total, maxchar); - if (result == NULL) + if (_PyUnicodeWriter_Prepare(writer, total, maxchar) == -1) goto done; - rkind = PyUnicode_KIND(result); - rdata = PyUnicode_DATA(result); + rkind = writer->kind; + rdata = writer->data; /* Populate the memory. First, the padding. */ - index = fill_padding(result, 0, - n_re_total + n_im_total + 1 + add_parens * 2, - format->fill_char=='\0' ? ' ' : format->fill_char, - lpad, rpad); + result = fill_padding(writer, + n_re_total + n_im_total + 1 + add_parens * 2, + format->fill_char=='\0' ? ' ' : format->fill_char, + lpad, rpad); + if (result == -1) + goto done; - if (add_parens) - PyUnicode_WRITE(rkind, rdata, index++, '('); + if (add_parens) { + PyUnicode_WRITE(rkind, rdata, writer->pos, '('); + writer->pos++; + } if (!skip_re) { - err = fill_number(result, index, &re_spec, - re_unicode_tmp, i_re, i_re + n_re_digits, - NULL, 0, - 0, - &locale, 0); - if (err) { - Py_CLEAR(result); + result = fill_number(writer, &re_spec, + re_unicode_tmp, i_re, i_re + n_re_digits, + NULL, 0, + 0, + &locale, 0); + if (result == -1) goto done; - } - index += n_re_total; } - err = fill_number(result, index, &im_spec, - im_unicode_tmp, i_im, i_im + n_im_digits, - NULL, 0, - 0, - &locale, 0); - if (err) { - Py_CLEAR(result); + result = fill_number(writer, &im_spec, + im_unicode_tmp, i_im, i_im + n_im_digits, + NULL, 0, + 0, + &locale, 0); + if (result == -1) goto done; + PyUnicode_WRITE(rkind, rdata, writer->pos, 'j'); + writer->pos++; + + if (add_parens) { + PyUnicode_WRITE(rkind, rdata, writer->pos, ')'); + writer->pos++; } - index += n_im_total; - PyUnicode_WRITE(rkind, rdata, index++, 'j'); - if (add_parens) - PyUnicode_WRITE(rkind, rdata, index++, ')'); + writer->pos += rpad; done: PyMem_Free(re_buf); @@ -1309,61 +1339,79 @@ Py_XDECREF(re_unicode_tmp); Py_XDECREF(im_unicode_tmp); free_locale_info(&locale); - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } /************************************************************************/ /*********** built in formatters ****************************************/ /************************************************************************/ -PyObject * -_PyUnicode_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) +int +format_obj(PyObject *obj, _PyUnicodeWriter *writer) +{ + PyObject *str; + int err; + + str = PyObject_Str(obj); + if (str == NULL) + return -1; + err = _PyUnicodeWriter_WriteStr(writer, str); + Py_DECREF(str); + return err; +} + +int +_PyUnicode_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { InternalFormatSpec format; - PyObject *result; + + assert(PyUnicode_Check(obj)); /* check for the special case of zero length format spec, make it equivalent to str(obj) */ - if (start == end) - return PyObject_Str(obj); + if (start == end) { + if (PyUnicode_CheckExact(obj)) + return _PyUnicodeWriter_WriteStr(writer, obj); + else + return format_obj(obj, writer); + } /* parse the format_spec */ if (!parse_internal_render_format_spec(format_spec, start, end, &format, 's', '<')) - return NULL; + return -1; /* type conversion? */ switch (format.type) { case 's': /* no type conversion needed, already a string. do the formatting */ - result = format_string_internal(obj, &format); - if (result != NULL) - assert(_PyUnicode_CheckConsistency(result, 1)); - break; + return format_string_internal(obj, &format, writer); default: /* unknown */ unknown_presentation_type(format.type, obj->ob_type->tp_name); - result = NULL; + return -1; } - return result; } -static PyObject* -format_int_or_long(PyObject* obj, PyObject* format_spec, - Py_ssize_t start, Py_ssize_t end, - IntOrLongToString tostring) +int +_PyLong_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { - PyObject *result = NULL; - PyObject *tmp = NULL; + PyObject *tmp = NULL, *str = NULL; InternalFormatSpec format; + int result = -1; /* check for the special case of zero length format spec, make it equivalent to str(obj) */ if (start == end) { - result = PyObject_Str(obj); - goto done; + if (PyLong_CheckExact(obj)) + return _PyLong_FormatWriter(writer, obj, 10, 0); + else + return format_obj(obj, writer); } /* parse the format_spec */ @@ -1382,7 +1430,7 @@ case 'n': /* no type conversion needed, already an int (or long). do the formatting */ - result = format_int_or_long_internal(obj, &format, tostring); + result = format_long_internal(obj, &format, writer); break; case 'e': @@ -1396,7 +1444,7 @@ tmp = PyNumber_Float(obj); if (tmp == NULL) goto done; - result = format_float_internal(tmp, &format); + result = format_float_internal(tmp, &format, writer); break; default: @@ -1407,41 +1455,27 @@ done: Py_XDECREF(tmp); + Py_XDECREF(str); return result; } -/* Need to define long_format as a function that will convert a long - to a string. In 3.0, _PyLong_Format has the correct signature. */ -#define long_format _PyLong_Format - -PyObject * -_PyLong_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) +int +_PyFloat_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { - return format_int_or_long(obj, format_spec, start, end, - long_format); -} - -PyObject * -_PyFloat_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) -{ - PyObject *result = NULL; InternalFormatSpec format; /* check for the special case of zero length format spec, make it equivalent to str(obj) */ - if (start == end) { - result = PyObject_Str(obj); - goto done; - } + if (start == end) + return format_obj(obj, writer); /* parse the format_spec */ if (!parse_internal_render_format_spec(format_spec, start, end, &format, '\0', '>')) - goto done; + return -1; /* type conversion? */ switch (format.type) { @@ -1455,38 +1489,32 @@ case 'n': case '%': /* no conversion, already a float. do the formatting */ - result = format_float_internal(obj, &format); - break; + return format_float_internal(obj, &format, writer); default: /* unknown */ unknown_presentation_type(format.type, obj->ob_type->tp_name); - goto done; + return -1; } - -done: - return result; } -PyObject * -_PyComplex_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) +int +_PyComplex_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { - PyObject *result = NULL; InternalFormatSpec format; /* check for the special case of zero length format spec, make it equivalent to str(obj) */ - if (start == end) { - result = PyObject_Str(obj); - goto done; - } + if (start == end) + return format_obj(obj, writer); /* parse the format_spec */ if (!parse_internal_render_format_spec(format_spec, start, end, &format, '\0', '>')) - goto done; + return -1; /* type conversion? */ switch (format.type) { @@ -1499,15 +1527,11 @@ case 'G': case 'n': /* no conversion, already a complex. do the formatting */ - result = format_complex_internal(obj, &format); - break; + return format_complex_internal(obj, &format, writer); default: /* unknown */ unknown_presentation_type(format.type, obj->ob_type->tp_name); - goto done; + return -1; } - -done: - return result; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 13:53:06 2012 From: python-checkins at python.org (vinay.sajip) Date: Tue, 29 May 2012 13:53:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Added_pyvenv_docs_to_=27Pyt?= =?utf8?q?hon_Setup_and_Usage=27_section=2E?= Message-ID: http://hg.python.org/cpython/rev/ace7c340d95d changeset: 77224:ace7c340d95d parent: 77219:01381723bc50 user: Vinay Sajip date: Tue May 29 12:52:14 2012 +0100 summary: Added pyvenv docs to 'Python Setup and Usage' section. files: Doc/library/venv.rst | 9 +-- Doc/using/index.rst | 2 +- Doc/using/scripts.rst | 66 +++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 6 deletions(-) diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -54,7 +54,7 @@ The command, if run with ``-h``, will show the available options:: usage: pyvenv [-h] [--system-site-packages] [--symlink] [--clear] - ENV_DIR [ENV_DIR ...] + [--upgrade] ENV_DIR [ENV_DIR ...] Creates virtual Python environments in one or more target directories. @@ -69,12 +69,11 @@ --clear Delete the environment directory if it already exists. If not specified and the directory exists, an error is raised. - + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. If the target directory already exists an error will be raised, unless -the ``--clear`` option was provided, in which case the target -directory will be deleted and virtual environment creation will -proceed as usual. +the ``--clear`` or ``--upgrade`` option was provided. The created ``pyvenv.cfg`` file also includes the ``include-system-site-packages`` key, set to ``true`` if ``venv`` is diff --git a/Doc/using/index.rst b/Doc/using/index.rst --- a/Doc/using/index.rst +++ b/Doc/using/index.rst @@ -17,4 +17,4 @@ unix.rst windows.rst mac.rst - + scripts.rst diff --git a/Doc/using/scripts.rst b/Doc/using/scripts.rst new file mode 100644 --- /dev/null +++ b/Doc/using/scripts.rst @@ -0,0 +1,66 @@ +.. _tools-and-scripts: + +Additional Tools and Scripts +============================ + +pyvenv - Creating virtual environments +-------------------------------------- + +Creation of virtual environments is done by executing the ``pyvenv`` +script:: + + pyvenv /path/to/new/virtual/environment + +Running this command creates the target directory (creating any parent +directories that don't exist already) and places a ``pyvenv.cfg`` file +in it with a ``home`` key pointing to the Python installation the +command was run from. It also creates a ``bin`` (or ``Scripts`` on +Windows) subdirectory containing a copy of the ``python`` binary (or +binaries, in the case of Windows) and the ``pysetup3`` script (to +facilitate easy installation of packages from PyPI into the new virtualenv). +It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` +subdirectory (on Windows, this is ``Lib\site-packages``). + +.. highlight:: none + +On Windows, you may have to invoke the ``pyvenv`` script as follows, if you +don't have the relevant PATH and PATHEXT settings:: + + c:\Temp>c:\Python33\python c:\Python33\Tools\Scripts\pyvenv.py myenv + +or equivalently:: + + c:\Temp>c:\Python33\python -m venv myenv + +The command, if run with ``-h``, will show the available options:: + + usage: pyvenv [-h] [--system-site-packages] [--symlink] [--clear] + [--upgrade] ENV_DIR [ENV_DIR ...] + + Creates virtual Python environments in one or more target directories. + + positional arguments: + ENV_DIR A directory to create the environment in. + + optional arguments: + -h, --help show this help message and exit + --system-site-packages Give access to the global site-packages dir to the + virtual environment. + --symlink Attempt to symlink rather than copy. + --clear Delete the environment directory if it already exists. + If not specified and the directory exists, an error is + raised. + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. + +If the target directory already exists an error will be raised, unless +the ``--clear`` or ``--upgrade`` option was provided. + +The created ``pyvenv.cfg`` file also includes the +``include-system-site-packages`` key, set to ``true`` if ``venv`` is +run with the ``--system-site-packages`` option, ``false`` otherwise. + +Multiple paths can be given to ``pyvenv``, in which case an identical +virtualenv will be created, according to the given options, at each +provided path. + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 13:53:07 2012 From: python-checkins at python.org (vinay.sajip) Date: Tue, 29 May 2012 13:53:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_Merged_upstream_changes=2E?= Message-ID: http://hg.python.org/cpython/rev/c4560d003e36 changeset: 77225:c4560d003e36 parent: 77224:ace7c340d95d parent: 77223:22b56b0b8619 user: Vinay Sajip date: Tue May 29 12:53:00 2012 +0100 summary: Merged upstream changes. files: Doc/library/multiprocessing.rst | 6 +- Include/complexobject.h | 10 +- Include/floatobject.h | 10 +- Include/longobject.h | 18 +- Include/unicodeobject.h | 95 +++- Lib/multiprocessing/__init__.py | 8 +- Lib/multiprocessing/managers.py | 11 +- Lib/multiprocessing/sharedctypes.py | 7 +- Misc/NEWS | 3 + Objects/complexobject.c | 17 +- Objects/floatobject.c | 27 +- Objects/longobject.c | 333 +++++++++--- Objects/stringlib/asciilib.h | 2 +- Objects/stringlib/unicode_format.h | 48 +- Objects/unicodeobject.c | 372 ++++++++++--- Python/formatter_unicode.c | 412 ++++++++------- Python/getargs.c | 5 +- 17 files changed, 912 insertions(+), 472 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -953,7 +953,7 @@ It is possible to create shared objects using shared memory which can be inherited by child processes. -.. function:: Value(typecode_or_type, *args[, lock]) +.. function:: Value(typecode_or_type, *args, lock=True) Return a :mod:`ctypes` object allocated from shared memory. By default the return value is actually a synchronized wrapper for the object. @@ -1045,7 +1045,7 @@ attributes which allow one to use it to store and retrieve strings -- see documentation for :mod:`ctypes`. -.. function:: Array(typecode_or_type, size_or_initializer, *args[, lock]) +.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True) The same as :func:`RawArray` except that depending on the value of *lock* a process-safe synchronization wrapper may be returned instead of a raw ctypes @@ -1060,7 +1060,7 @@ Note that *lock* is a keyword-only argument. -.. function:: Value(typecode_or_type, *args[, lock]) +.. function:: Value(typecode_or_type, *args, lock=True) The same as :func:`RawValue` except that depending on the value of *lock* a process-safe synchronization wrapper may be returned instead of a raw ctypes diff --git a/Include/complexobject.h b/Include/complexobject.h --- a/Include/complexobject.h +++ b/Include/complexobject.h @@ -63,10 +63,12 @@ /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ #ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) _PyComplex_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyComplex_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif #ifdef __cplusplus diff --git a/Include/floatobject.h b/Include/floatobject.h --- a/Include/floatobject.h +++ b/Include/floatobject.h @@ -112,10 +112,12 @@ /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ -PyAPI_FUNC(PyObject *) _PyFloat_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyFloat_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif /* Py_LIMITED_API */ #ifdef __cplusplus diff --git a/Include/longobject.h b/Include/longobject.h --- a/Include/longobject.h +++ b/Include/longobject.h @@ -151,14 +151,22 @@ /* _PyLong_Format: Convert the long to a string object with given base, appending a base prefix of 0[box] if base is 2, 8 or 16. */ -PyAPI_FUNC(PyObject *) _PyLong_Format(PyObject *aa, int base); +PyAPI_FUNC(PyObject *) _PyLong_Format(PyObject *obj, int base); + +PyAPI_FUNC(int) _PyLong_FormatWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + int base, + int alternate); /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ -PyAPI_FUNC(PyObject *) _PyLong_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyLong_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif /* Py_LIMITED_API */ /* These aren't really part of the long object, but they're handy. The diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -648,8 +648,20 @@ Py_ssize_t from_start, Py_ssize_t how_many ); + +/* Unsafe version of PyUnicode_CopyCharacters(): don't check arguments and so + may crash if parameters are invalid (e.g. if the output string + is too short). */ +PyAPI_FUNC(void) _PyUnicode_FastCopyCharacters( + PyObject *to, + Py_ssize_t to_start, + PyObject *from, + Py_ssize_t from_start, + Py_ssize_t how_many + ); #endif +#ifndef Py_LIMITED_API /* Fill a string with a character: write fill_char into unicode[start:start+length]. @@ -658,13 +670,21 @@ Return the number of written character, or return -1 and raise an exception on error. */ -#ifndef Py_LIMITED_API PyAPI_FUNC(Py_ssize_t) PyUnicode_Fill( PyObject *unicode, Py_ssize_t start, Py_ssize_t length, Py_UCS4 fill_char ); + +/* Unsafe version of PyUnicode_Fill(): don't check arguments and so may crash + if parameters are invalid (e.g. if length is longer than the string). */ +PyAPI_FUNC(void) _PyUnicode_FastFill( + PyObject *unicode, + Py_ssize_t start, + Py_ssize_t length, + Py_UCS4 fill_char + ); #endif /* Create a Unicode Object from the Py_UNICODE buffer u of the given @@ -696,13 +716,19 @@ const char *u /* UTF-8 encoded string */ ); +#ifndef Py_LIMITED_API /* Create a new string from a buffer of Py_UCS1, Py_UCS2 or Py_UCS4 characters. Scan the string to find the maximum character. */ -#ifndef Py_LIMITED_API PyAPI_FUNC(PyObject*) PyUnicode_FromKindAndData( int kind, const void *buffer, Py_ssize_t size); + +/* Create a new string from a buffer of ASCII characters. + WARNING: Don't check if the string contains any non-ASCII character. */ +PyAPI_FUNC(PyObject*) _PyUnicode_FromASCII( + const char *buffer, + Py_ssize_t size); #endif PyAPI_FUNC(PyObject*) PyUnicode_Substring( @@ -865,12 +891,69 @@ ); #ifndef Py_LIMITED_API +typedef struct { + PyObject *buffer; + void *data; + enum PyUnicode_Kind kind; + Py_UCS4 maxchar; + Py_ssize_t size; + Py_ssize_t pos; + /* minimum length of the buffer when overallocation is enabled, + see _PyUnicodeWriter_Init() */ + Py_ssize_t min_length; + struct { + unsigned char overallocate:1; + /* If readonly is 1, buffer is a shared string (cannot be modified) + and size is set to 0. */ + unsigned char readonly:1; + } flags; +} _PyUnicodeWriter ; + +/* Initialize a Unicode writer. + + If min_length is greater than zero, _PyUnicodeWriter_Prepare() + overallocates the buffer and min_length is the minimum length in characters + of the buffer. */ +PyAPI_FUNC(void) +_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, Py_ssize_t min_length); + +/* Prepare the buffer to write 'length' characters + with the specified maximum character. + + Return 0 on success, raise an exception and return -1 on error. */ +#define _PyUnicodeWriter_Prepare(WRITER, LENGTH, MAXCHAR) \ + (((MAXCHAR) <= (WRITER)->maxchar \ + && (LENGTH) <= (WRITER)->size - (WRITER)->pos) \ + ? 0 \ + : (((LENGTH) == 0) \ + ? 0 \ + : _PyUnicodeWriter_PrepareInternal((WRITER), (LENGTH), (MAXCHAR)))) + +/* Don't call this function directly, use the _PyUnicodeWriter_Prepare() macro + instead. */ +PyAPI_FUNC(int) +_PyUnicodeWriter_PrepareInternal(_PyUnicodeWriter *writer, + Py_ssize_t length, Py_UCS4 maxchar); + +PyAPI_FUNC(int) +_PyUnicodeWriter_WriteStr(_PyUnicodeWriter *writer, PyObject *str); + +PyAPI_FUNC(PyObject *) +_PyUnicodeWriter_Finish(_PyUnicodeWriter *writer); + +PyAPI_FUNC(void) +_PyUnicodeWriter_Dealloc(_PyUnicodeWriter *writer); +#endif + +#ifndef Py_LIMITED_API /* Format the object based on the format_spec, as defined in PEP 3101 (Advanced String Formatting). */ -PyAPI_FUNC(PyObject *) _PyUnicode_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +PyAPI_FUNC(int) _PyUnicode_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); #endif PyAPI_FUNC(void) PyUnicode_InternInPlace(PyObject **); diff --git a/Lib/multiprocessing/__init__.py b/Lib/multiprocessing/__init__.py --- a/Lib/multiprocessing/__init__.py +++ b/Lib/multiprocessing/__init__.py @@ -228,19 +228,19 @@ from multiprocessing.sharedctypes import RawArray return RawArray(typecode_or_type, size_or_initializer) -def Value(typecode_or_type, *args, **kwds): +def Value(typecode_or_type, *args, lock=True): ''' Returns a synchronized shared object ''' from multiprocessing.sharedctypes import Value - return Value(typecode_or_type, *args, **kwds) + return Value(typecode_or_type, *args, lock=lock) -def Array(typecode_or_type, size_or_initializer, **kwds): +def Array(typecode_or_type, size_or_initializer, *, lock=True): ''' Returns a synchronized shared array ''' from multiprocessing.sharedctypes import Array - return Array(typecode_or_type, size_or_initializer, **kwds) + return Array(typecode_or_type, size_or_initializer, lock=lock) # # diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -1035,12 +1035,11 @@ BaseListProxy = MakeProxyType('BaseListProxy', ( - '__add__', '__contains__', '__delitem__', '__delslice__', - '__getitem__', '__getslice__', '__len__', '__mul__', - '__reversed__', '__rmul__', '__setitem__', '__setslice__', + '__add__', '__contains__', '__delitem__', '__getitem__', '__len__', + '__mul__', '__reversed__', '__rmul__', '__setitem__', 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort', '__imul__' - )) # XXX __getslice__ and __setslice__ unneeded in Py3.0 + )) class ListProxy(BaseListProxy): def __iadd__(self, value): self._callmethod('extend', (value,)) @@ -1058,8 +1057,8 @@ ArrayProxy = MakeProxyType('ArrayProxy', ( - '__len__', '__getitem__', '__setitem__', '__getslice__', '__setslice__' - )) # XXX __getslice__ and __setslice__ unneeded in Py3.0 + '__len__', '__getitem__', '__setitem__' + )) PoolProxy = MakeProxyType('PoolProxy', ( diff --git a/Lib/multiprocessing/sharedctypes.py b/Lib/multiprocessing/sharedctypes.py --- a/Lib/multiprocessing/sharedctypes.py +++ b/Lib/multiprocessing/sharedctypes.py @@ -63,7 +63,7 @@ result.__init__(*size_or_initializer) return result -def Value(typecode_or_type, *args, lock=None): +def Value(typecode_or_type, *args, lock=True): ''' Return a synchronization wrapper for a Value ''' @@ -76,13 +76,10 @@ raise AttributeError("'%r' has no method 'acquire'" % lock) return synchronized(obj, lock) -def Array(typecode_or_type, size_or_initializer, **kwds): +def Array(typecode_or_type, size_or_initializer, *, lock=True): ''' Return a synchronization wrapper for a RawArray ''' - lock = kwds.pop('lock', None) - if kwds: - raise ValueError('unrecognized keyword argument(s): %s' % list(kwds.keys())) obj = RawArray(typecode_or_type, size_or_initializer) if lock is False: return obj diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -13,6 +13,9 @@ - Issue #14835: Make plistlib output empty arrays & dicts like OS X. Patch by Sidney San Mart?n. +- Issue #14744: Use the new _PyUnicodeWriter internal API to speed up + str%args and str.format(args). + - Issue #14930: Make memoryview objects weakrefable. - Issue #14775: Fix a potential quadratic dict build-up due to the garbage diff --git a/Objects/complexobject.c b/Objects/complexobject.c --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -699,11 +699,22 @@ complex__format__(PyObject* self, PyObject* args) { PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) - return NULL; - return _PyComplex_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); + return NULL; + + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyComplex_FormatAdvancedWriter( + &writer, + self, + format_spec, 0, PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } #if 0 diff --git a/Objects/floatobject.c b/Objects/floatobject.c --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -267,13 +267,15 @@ float_repr(PyFloatObject *v) { PyObject *result; - char *buf = PyOS_double_to_string(PyFloat_AS_DOUBLE(v), - 'r', 0, - Py_DTSF_ADD_DOT_0, - NULL); + char *buf; + + buf = PyOS_double_to_string(PyFloat_AS_DOUBLE(v), + 'r', 0, + Py_DTSF_ADD_DOT_0, + NULL); if (!buf) return PyErr_NoMemory(); - result = PyUnicode_FromString(buf); + result = _PyUnicode_FromASCII(buf, strlen(buf)); PyMem_Free(buf); return result; } @@ -1703,11 +1705,22 @@ float__format__(PyObject *self, PyObject *args) { PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) return NULL; - return _PyFloat_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); + + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyFloat_FormatAdvancedWriter( + &writer, + self, + format_spec, 0, PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } PyDoc_STRVAR(float__format__doc, diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -1550,20 +1550,22 @@ string. (Return value is non-shared so that callers can modify the returned value if necessary.) */ -static PyObject * -long_to_decimal_string(PyObject *aa) +static int +long_to_decimal_string_internal(PyObject *aa, + PyObject **p_output, + _PyUnicodeWriter *writer) { PyLongObject *scratch, *a; PyObject *str; Py_ssize_t size, strlen, size_a, i, j; digit *pout, *pin, rem, tenpow; - unsigned char *p; int negative; + enum PyUnicode_Kind kind; a = (PyLongObject *)aa; if (a == NULL || !PyLong_Check(a)) { PyErr_BadInternalCall(); - return NULL; + return -1; } size_a = ABS(Py_SIZE(a)); negative = Py_SIZE(a) < 0; @@ -1580,13 +1582,13 @@ if (size_a > PY_SSIZE_T_MAX / PyLong_SHIFT) { PyErr_SetString(PyExc_OverflowError, "long is too large to format"); - return NULL; + return -1; } /* the expression size_a * PyLong_SHIFT is now safe from overflow */ size = 1 + size_a * PyLong_SHIFT / (3 * _PyLong_DECIMAL_SHIFT); scratch = _PyLong_New(size); if (scratch == NULL) - return NULL; + return -1; /* convert array of base _PyLong_BASE digits in pin to an array of base _PyLong_DECIMAL_BASE digits in pout, following Knuth (TAOCP, @@ -1609,7 +1611,7 @@ /* check for keyboard interrupt */ SIGCHECK({ Py_DECREF(scratch); - return NULL; + return -1; }); } /* pout should have at least one digit, so that the case when a = 0 @@ -1625,65 +1627,113 @@ tenpow *= 10; strlen++; } - str = PyUnicode_New(strlen, '9'); - if (str == NULL) { - Py_DECREF(scratch); + if (writer) { + if (_PyUnicodeWriter_Prepare(writer, strlen, '9') == -1) + return -1; + kind = writer->kind; + str = NULL; + } + else { + str = PyUnicode_New(strlen, '9'); + if (str == NULL) { + Py_DECREF(scratch); + return -1; + } + kind = PyUnicode_KIND(str); + } + +#define WRITE_DIGITS(TYPE) \ + do { \ + if (writer) \ + p = (TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos + strlen; \ + else \ + p = (TYPE*)PyUnicode_DATA(str) + strlen; \ + \ + *p = '\0'; \ + /* pout[0] through pout[size-2] contribute exactly \ + _PyLong_DECIMAL_SHIFT digits each */ \ + for (i=0; i < size - 1; i++) { \ + rem = pout[i]; \ + for (j = 0; j < _PyLong_DECIMAL_SHIFT; j++) { \ + *--p = '0' + rem % 10; \ + rem /= 10; \ + } \ + } \ + /* pout[size-1]: always produce at least one decimal digit */ \ + rem = pout[i]; \ + do { \ + *--p = '0' + rem % 10; \ + rem /= 10; \ + } while (rem != 0); \ + \ + /* and sign */ \ + if (negative) \ + *--p = '-'; \ + \ + /* check we've counted correctly */ \ + if (writer) \ + assert(p == ((TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos)); \ + else \ + assert(p == (TYPE*)PyUnicode_DATA(str)); \ + } while (0) + + /* fill the string right-to-left */ + if (kind == PyUnicode_1BYTE_KIND) { + Py_UCS1 *p; + WRITE_DIGITS(Py_UCS1); + } + else if (kind == PyUnicode_2BYTE_KIND) { + Py_UCS2 *p; + WRITE_DIGITS(Py_UCS2); + } + else { + assert (kind == PyUnicode_4BYTE_KIND); + Py_UCS4 *p; + WRITE_DIGITS(Py_UCS4); + } +#undef WRITE_DIGITS + + Py_DECREF(scratch); + if (writer) { + writer->pos += strlen; + } + else { + assert(_PyUnicode_CheckConsistency(str, 1)); + *p_output = (PyObject *)str; + } + return 0; +} + +static PyObject * +long_to_decimal_string(PyObject *aa) +{ + PyObject *v; + if (long_to_decimal_string_internal(aa, &v, NULL) == -1) return NULL; - } - - /* fill the string right-to-left */ - assert(PyUnicode_KIND(str) == PyUnicode_1BYTE_KIND); - p = PyUnicode_1BYTE_DATA(str) + strlen; - *p = '\0'; - /* pout[0] through pout[size-2] contribute exactly - _PyLong_DECIMAL_SHIFT digits each */ - for (i=0; i < size - 1; i++) { - rem = pout[i]; - for (j = 0; j < _PyLong_DECIMAL_SHIFT; j++) { - *--p = '0' + rem % 10; - rem /= 10; - } - } - /* pout[size-1]: always produce at least one decimal digit */ - rem = pout[i]; - do { - *--p = '0' + rem % 10; - rem /= 10; - } while (rem != 0); - - /* and sign */ - if (negative) - *--p = '-'; - - /* check we've counted correctly */ - assert(p == PyUnicode_1BYTE_DATA(str)); - assert(_PyUnicode_CheckConsistency(str, 1)); - Py_DECREF(scratch); - return (PyObject *)str; + return v; } /* Convert a long int object to a string, using a given conversion base, - which should be one of 2, 8, 10 or 16. Return a string object. - If base is 2, 8 or 16, add the proper prefix '0b', '0o' or '0x'. */ - -PyObject * -_PyLong_Format(PyObject *aa, int base) + which should be one of 2, 8 or 16. Return a string object. + If base is 2, 8 or 16, add the proper prefix '0b', '0o' or '0x' + if alternate is nonzero. */ + +static int +long_format_binary(PyObject *aa, int base, int alternate, + PyObject **p_output, _PyUnicodeWriter *writer) { register PyLongObject *a = (PyLongObject *)aa; PyObject *v; Py_ssize_t sz; Py_ssize_t size_a; - Py_UCS1 *p; + enum PyUnicode_Kind kind; int negative; int bits; - assert(base == 2 || base == 8 || base == 10 || base == 16); - if (base == 10) - return long_to_decimal_string((PyObject *)a); - + assert(base == 2 || base == 8 || base == 16); if (a == NULL || !PyLong_Check(a)) { PyErr_BadInternalCall(); - return NULL; + return -1; } size_a = ABS(Py_SIZE(a)); negative = Py_SIZE(a) < 0; @@ -1706,7 +1756,7 @@ /* Compute exact length 'sz' of output string. */ if (size_a == 0) { - sz = 3; + sz = 1; } else { Py_ssize_t size_a_in_bits; @@ -1714,56 +1764,126 @@ if (size_a > (PY_SSIZE_T_MAX - 3) / PyLong_SHIFT) { PyErr_SetString(PyExc_OverflowError, "int is too large to format"); - return NULL; + return -1; } size_a_in_bits = (size_a - 1) * PyLong_SHIFT + bits_in_digit(a->ob_digit[size_a - 1]); - /* Allow 2 characters for prefix and 1 for a '-' sign. */ - sz = 2 + negative + (size_a_in_bits + (bits - 1)) / bits; - } - - v = PyUnicode_New(sz, 'x'); - if (v == NULL) { + /* Allow 1 character for a '-' sign. */ + sz = negative + (size_a_in_bits + (bits - 1)) / bits; + } + if (alternate) { + /* 2 characters for prefix */ + sz += 2; + } + + if (writer) { + if (_PyUnicodeWriter_Prepare(writer, sz, 'x') == -1) + return -1; + kind = writer->kind; + v = NULL; + } + else { + v = PyUnicode_New(sz, 'x'); + if (v == NULL) + return -1; + kind = PyUnicode_KIND(v); + } + +#define WRITE_DIGITS(TYPE) \ + do { \ + if (writer) \ + p = (TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos + sz; \ + else \ + p = (TYPE*)PyUnicode_DATA(v) + sz; \ + \ + if (size_a == 0) { \ + *--p = '0'; \ + } \ + else { \ + /* JRH: special case for power-of-2 bases */ \ + twodigits accum = 0; \ + int accumbits = 0; /* # of bits in accum */ \ + Py_ssize_t i; \ + for (i = 0; i < size_a; ++i) { \ + accum |= (twodigits)a->ob_digit[i] << accumbits; \ + accumbits += PyLong_SHIFT; \ + assert(accumbits >= bits); \ + do { \ + char cdigit; \ + cdigit = (char)(accum & (base - 1)); \ + cdigit += (cdigit < 10) ? '0' : 'a'-10; \ + *--p = cdigit; \ + accumbits -= bits; \ + accum >>= bits; \ + } while (i < size_a-1 ? accumbits >= bits : accum > 0); \ + } \ + } \ + \ + if (alternate) { \ + if (base == 16) \ + *--p = 'x'; \ + else if (base == 8) \ + *--p = 'o'; \ + else /* (base == 2) */ \ + *--p = 'b'; \ + *--p = '0'; \ + } \ + if (negative) \ + *--p = '-'; \ + if (writer) \ + assert(p == ((TYPE*)PyUnicode_DATA(writer->buffer) + writer->pos)); \ + else \ + assert(p == (TYPE*)PyUnicode_DATA(v)); \ + } while (0) + + if (kind == PyUnicode_1BYTE_KIND) { + Py_UCS1 *p; + WRITE_DIGITS(Py_UCS1); + } + else if (kind == PyUnicode_2BYTE_KIND) { + Py_UCS2 *p; + WRITE_DIGITS(Py_UCS2); + } + else { + assert (kind == PyUnicode_4BYTE_KIND); + Py_UCS4 *p; + WRITE_DIGITS(Py_UCS4); + } +#undef WRITE_DIGITS + + if (writer) { + writer->pos += sz; + } + else { + assert(_PyUnicode_CheckConsistency(v, 1)); + *p_output = v; + } + return 0; +} + +PyObject * +_PyLong_Format(PyObject *obj, int base) +{ + PyObject *str; + int err; + if (base == 10) + err = long_to_decimal_string_internal(obj, &str, NULL); + else + err = long_format_binary(obj, base, 1, &str, NULL); + if (err == -1) return NULL; - } - assert(PyUnicode_KIND(v) == PyUnicode_1BYTE_KIND); - - p = PyUnicode_1BYTE_DATA(v) + sz; - if (size_a == 0) { - *--p = '0'; - } - else { - /* JRH: special case for power-of-2 bases */ - twodigits accum = 0; - int accumbits = 0; /* # of bits in accum */ - Py_ssize_t i; - for (i = 0; i < size_a; ++i) { - accum |= (twodigits)a->ob_digit[i] << accumbits; - accumbits += PyLong_SHIFT; - assert(accumbits >= bits); - do { - char cdigit; - cdigit = (char)(accum & (base - 1)); - cdigit += (cdigit < 10) ? '0' : 'a'-10; - *--p = cdigit; - accumbits -= bits; - accum >>= bits; - } while (i < size_a-1 ? accumbits >= bits : accum > 0); - } - } - - if (base == 16) - *--p = 'x'; - else if (base == 8) - *--p = 'o'; - else /* (base == 2) */ - *--p = 'b'; - *--p = '0'; - if (negative) - *--p = '-'; - assert(p == PyUnicode_1BYTE_DATA(v)); - assert(_PyUnicode_CheckConsistency(v, 1)); - return v; + return str; +} + +int +_PyLong_FormatWriter(_PyUnicodeWriter *writer, + PyObject *obj, + int base, int alternate) +{ + if (base == 10) + return long_to_decimal_string_internal(obj, NULL, writer); + else + return long_format_binary(obj, base, alternate, NULL, writer); } /* Table of digit values for 8-bit string -> integer conversion. @@ -4232,11 +4352,22 @@ long__format__(PyObject *self, PyObject *args) { PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) return NULL; - return _PyLong_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); + + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyLong_FormatAdvancedWriter( + &writer, + self, + format_spec, 0, PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } /* Return a pair (q, r) such that a = b * q + r, and diff --git a/Objects/stringlib/asciilib.h b/Objects/stringlib/asciilib.h --- a/Objects/stringlib/asciilib.h +++ b/Objects/stringlib/asciilib.h @@ -18,7 +18,7 @@ #define STRINGLIB_TODECIMAL Py_UNICODE_TODECIMAL #define STRINGLIB_STR PyUnicode_1BYTE_DATA #define STRINGLIB_LEN PyUnicode_GET_LENGTH -#define STRINGLIB_NEW unicode_fromascii +#define STRINGLIB_NEW(STR,LEN) _PyUnicode_FromASCII((char*)(STR),(LEN)) #define STRINGLIB_RESIZE not_supported #define STRINGLIB_CHECK PyUnicode_Check #define STRINGLIB_CHECK_EXACT PyUnicode_CheckExact diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h --- a/Objects/stringlib/unicode_format.h +++ b/Objects/stringlib/unicode_format.h @@ -499,26 +499,26 @@ int ok = 0; PyObject *result = NULL; PyObject *format_spec_object = NULL; - PyObject *(*formatter)(PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL; - Py_ssize_t len; + int (*formatter) (_PyUnicodeWriter*, PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL; + int err; /* If we know the type exactly, skip the lookup of __format__ and just call the formatter directly. */ if (PyUnicode_CheckExact(fieldobj)) - formatter = _PyUnicode_FormatAdvanced; + formatter = _PyUnicode_FormatAdvancedWriter; else if (PyLong_CheckExact(fieldobj)) - formatter =_PyLong_FormatAdvanced; + formatter = _PyLong_FormatAdvancedWriter; else if (PyFloat_CheckExact(fieldobj)) - formatter = _PyFloat_FormatAdvanced; - - /* XXX: for 2.6, convert format_spec to the appropriate type - (unicode, str) */ + formatter = _PyFloat_FormatAdvancedWriter; + else if (PyComplex_CheckExact(fieldobj)) + formatter = _PyComplex_FormatAdvancedWriter; if (formatter) { /* we know exactly which formatter will be called when __format__ is looked up, so call it directly, instead. */ - result = formatter(fieldobj, format_spec->str, - format_spec->start, format_spec->end); + err = formatter(writer, fieldobj, format_spec->str, + format_spec->start, format_spec->end); + return (err == 0); } else { /* We need to create an object out of the pointers we have, because @@ -536,17 +536,11 @@ } if (result == NULL) goto done; - if (PyUnicode_READY(result) == -1) + + if (_PyUnicodeWriter_WriteStr(writer, result) == -1) goto done; + ok = 1; - len = PyUnicode_GET_LENGTH(result); - if (_PyUnicodeWriter_Prepare(writer, - len, PyUnicode_MAX_CHAR_VALUE(result)) == -1) - goto done; - copy_characters(writer->buffer, writer->pos, - result, 0, len); - writer->pos += len; - ok = 1; done: Py_XDECREF(format_spec_object); Py_XDECREF(result); @@ -897,16 +891,19 @@ err = _PyUnicodeWriter_Prepare(writer, sublen, maxchar); if (err == -1) return 0; - copy_characters(writer->buffer, writer->pos, - literal.str, literal.start, sublen); + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + literal.str, literal.start, sublen); writer->pos += sublen; } - if (field_present) + if (field_present) { + if (iter.str.start == iter.str.end) + writer->flags.overallocate = 0; if (!output_markup(&field_name, &format_spec, format_spec_needs_expanding, conversion, writer, args, kwargs, recursion_depth, auto_number)) return 0; + } } return result; } @@ -921,7 +918,7 @@ int recursion_depth, AutoNumber *auto_number) { _PyUnicodeWriter writer; - Py_ssize_t initlen; + Py_ssize_t minlen; /* check the recursion level */ if (recursion_depth <= 0) { @@ -930,9 +927,8 @@ return NULL; } - initlen = PyUnicode_GET_LENGTH(input->str) + 100; - if (_PyUnicodeWriter_Init(&writer, initlen, 127) == -1) - return NULL; + minlen = PyUnicode_GET_LENGTH(input->str) + 100; + _PyUnicodeWriter_Init(&writer, minlen); if (!do_markup(input, args, kwargs, &writer, recursion_depth, auto_number)) { diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -225,16 +225,10 @@ /* forward */ static PyUnicodeObject *_PyUnicode_New(Py_ssize_t length); static PyObject* get_latin1_char(unsigned char ch); -static void copy_characters( - PyObject *to, Py_ssize_t to_start, - PyObject *from, Py_ssize_t from_start, - Py_ssize_t how_many); static int unicode_modifiable(PyObject *unicode); static PyObject * -unicode_fromascii(const unsigned char *s, Py_ssize_t size); -static PyObject * _PyUnicode_FromUCS1(const unsigned char *s, Py_ssize_t size); static PyObject * _PyUnicode_FromUCS2(const Py_UCS2 *s, Py_ssize_t size); @@ -783,7 +777,7 @@ return NULL; copy_length = Py_MIN(length, PyUnicode_GET_LENGTH(unicode)); - copy_characters(copy, 0, unicode, 0, copy_length); + _PyUnicode_FastCopyCharacters(copy, 0, unicode, 0, copy_length); return copy; } else { @@ -1154,15 +1148,16 @@ assert(0 <= from_start); assert(0 <= to_start); assert(PyUnicode_Check(from)); - assert(PyUnicode_Check(to)); assert(PyUnicode_IS_READY(from)); - assert(PyUnicode_IS_READY(to)); assert(from_start + how_many <= PyUnicode_GET_LENGTH(from)); - assert(to_start + how_many <= PyUnicode_GET_LENGTH(to)); if (how_many == 0) return 0; + assert(PyUnicode_Check(to)); + assert(PyUnicode_IS_READY(to)); + assert(to_start + how_many <= PyUnicode_GET_LENGTH(to)); + from_kind = PyUnicode_KIND(from); from_data = PyUnicode_DATA(from); to_kind = PyUnicode_KIND(to); @@ -1267,10 +1262,10 @@ return 0; } -static void -copy_characters(PyObject *to, Py_ssize_t to_start, - PyObject *from, Py_ssize_t from_start, - Py_ssize_t how_many) +void +_PyUnicode_FastCopyCharacters( + PyObject *to, Py_ssize_t to_start, + PyObject *from, Py_ssize_t from_start, Py_ssize_t how_many) { (void)_copy_characters(to, to_start, from, from_start, how_many, 0); } @@ -1292,6 +1287,14 @@ if (PyUnicode_READY(to) == -1) return -1; + if (from_start < 0) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return -1; + } + if (to_start < 0) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return -1; + } how_many = Py_MIN(PyUnicode_GET_LENGTH(from), how_many); if (to_start + how_many > PyUnicode_GET_LENGTH(to)) { PyErr_Format(PyExc_SystemError, @@ -1641,7 +1644,7 @@ maxchar); if (result == NULL) return -1; - PyUnicode_CopyCharacters(result, 0, *p_unicode, 0, length); + _PyUnicode_FastCopyCharacters(result, 0, *p_unicode, 0, length); Py_DECREF(*p_unicode); *p_unicode = result; return 0; @@ -1841,9 +1844,10 @@ /* Internal function, doesn't check maximum character */ -static PyObject* -unicode_fromascii(const unsigned char* s, Py_ssize_t size) -{ +PyObject* +_PyUnicode_FromASCII(const char *buffer, Py_ssize_t size) +{ + const unsigned char *s = (const unsigned char *)buffer; PyObject *unicode; if (size == 1) { #ifdef Py_DEBUG @@ -2085,7 +2089,7 @@ return; } copy = PyUnicode_New(len, max_char); - copy_characters(copy, 0, unicode, 0, len); + _PyUnicode_FastCopyCharacters(copy, 0, unicode, 0, len); Py_DECREF(unicode); *p_unicode = copy; } @@ -2753,7 +2757,7 @@ (void) va_arg(vargs, char *); size = PyUnicode_GET_LENGTH(*callresult); assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string)); - copy_characters(string, i, *callresult, 0, size); + _PyUnicode_FastCopyCharacters(string, i, *callresult, 0, size); i += size; /* We're done with the unicode()/repr() => forget it */ Py_DECREF(*callresult); @@ -2767,7 +2771,7 @@ Py_ssize_t size; assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string)); size = PyUnicode_GET_LENGTH(obj); - copy_characters(string, i, obj, 0, size); + _PyUnicode_FastCopyCharacters(string, i, obj, 0, size); i += size; break; } @@ -2779,13 +2783,13 @@ if (obj) { size = PyUnicode_GET_LENGTH(obj); assert(PyUnicode_KIND(obj) <= PyUnicode_KIND(string)); - copy_characters(string, i, obj, 0, size); + _PyUnicode_FastCopyCharacters(string, i, obj, 0, size); i += size; } else { size = PyUnicode_GET_LENGTH(*callresult); assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string)); - copy_characters(string, i, *callresult, 0, size); + _PyUnicode_FastCopyCharacters(string, i, *callresult, 0, size); i += size; Py_DECREF(*callresult); } @@ -2800,7 +2804,7 @@ /* unused, since we already have the result */ (void) va_arg(vargs, PyObject *); assert(PyUnicode_KIND(*callresult) <= PyUnicode_KIND(string)); - copy_characters(string, i, *callresult, 0, size); + _PyUnicode_FastCopyCharacters(string, i, *callresult, 0, size); i += size; /* We're done with the unicode()/repr() => forget it */ Py_DECREF(*callresult); @@ -4171,7 +4175,7 @@ if (unicode_widen(output, *outpos, PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0) goto onError; - copy_characters(*output, *outpos, repunicode, 0, replen); + _PyUnicode_FastCopyCharacters(*output, *outpos, repunicode, 0, replen); *outpos += replen; } else { @@ -9216,12 +9220,14 @@ /* If the maxchar increased so that the kind changed, not all characters are representable anymore and we need to fix the string again. This only happens in very few cases. */ - copy_characters(v, 0, self, 0, PyUnicode_GET_LENGTH(self)); + _PyUnicode_FastCopyCharacters(v, 0, + self, 0, PyUnicode_GET_LENGTH(self)); maxchar_old = fixfct(v); assert(maxchar_old > 0 && maxchar_old <= maxchar_new); } else { - copy_characters(v, 0, u, 0, PyUnicode_GET_LENGTH(self)); + _PyUnicode_FastCopyCharacters(v, 0, + u, 0, PyUnicode_GET_LENGTH(self)); } Py_DECREF(u); assert(_PyUnicode_CheckConsistency(v, 1)); @@ -9603,7 +9609,7 @@ res_data += kind * seplen; } else { - copy_characters(res, res_offset, sep, 0, seplen); + _PyUnicode_FastCopyCharacters(res, res_offset, sep, 0, seplen); res_offset += seplen; } } @@ -9616,7 +9622,7 @@ res_data += kind * itemlen; } else { - copy_characters(res, res_offset, item, 0, itemlen); + _PyUnicode_FastCopyCharacters(res, res_offset, item, 0, itemlen); res_offset += itemlen; } } @@ -9663,13 +9669,25 @@ } \ } while (0) +void +_PyUnicode_FastFill(PyObject *unicode, Py_ssize_t start, Py_ssize_t length, + Py_UCS4 fill_char) +{ + const enum PyUnicode_Kind kind = PyUnicode_KIND(unicode); + const void *data = PyUnicode_DATA(unicode); + assert(PyUnicode_IS_READY(unicode)); + assert(unicode_modifiable(unicode)); + assert(fill_char <= PyUnicode_MAX_CHAR_VALUE(unicode)); + assert(start >= 0); + assert(start + length <= PyUnicode_GET_LENGTH(unicode)); + FILL(kind, data, fill_char, start, length); +} + Py_ssize_t PyUnicode_Fill(PyObject *unicode, Py_ssize_t start, Py_ssize_t length, Py_UCS4 fill_char) { Py_ssize_t maxlen; - enum PyUnicode_Kind kind; - void *data; if (!PyUnicode_Check(unicode)) { PyErr_BadInternalCall(); @@ -9680,6 +9698,10 @@ if (unicode_check_modifiable(unicode)) return -1; + if (start < 0) { + PyErr_SetString(PyExc_IndexError, "string index out of range"); + return -1; + } if (fill_char > PyUnicode_MAX_CHAR_VALUE(unicode)) { PyErr_SetString(PyExc_ValueError, "fill character is bigger than " @@ -9692,9 +9714,7 @@ if (length <= 0) return 0; - kind = PyUnicode_KIND(unicode); - data = PyUnicode_DATA(unicode); - FILL(kind, data, fill_char, start, length); + _PyUnicode_FastFill(unicode, start, length, fill_char); return length; } @@ -9734,7 +9754,7 @@ FILL(kind, data, fill, 0, left); if (right) FILL(kind, data, fill, left + _PyUnicode_LENGTH(self), right); - copy_characters(u, left, self, 0, _PyUnicode_LENGTH(self)); + _PyUnicode_FastCopyCharacters(u, left, self, 0, _PyUnicode_LENGTH(self)); assert(_PyUnicode_CheckConsistency(u, 1)); return u; } @@ -10058,7 +10078,7 @@ u = PyUnicode_New(slen, maxchar); if (!u) goto error; - copy_characters(u, 0, self, 0, slen); + _PyUnicode_FastCopyCharacters(u, 0, self, 0, slen); rkind = PyUnicode_KIND(u); PyUnicode_WRITE(rkind, PyUnicode_DATA(u), pos, u2); @@ -10626,8 +10646,8 @@ w = PyUnicode_New(new_len, maxchar); if (w == NULL) goto onError; - copy_characters(w, 0, u, 0, u_len); - copy_characters(w, u_len, v, 0, v_len); + _PyUnicode_FastCopyCharacters(w, 0, u, 0, u_len); + _PyUnicode_FastCopyCharacters(w, u_len, v, 0, v_len); Py_DECREF(u); Py_DECREF(v); assert(_PyUnicode_CheckConsistency(w, 1)); @@ -10702,7 +10722,7 @@ goto error; } /* copy 'right' into the newly allocated area of 'left' */ - copy_characters(*p_left, left_len, right, 0, right_len); + _PyUnicode_FastCopyCharacters(*p_left, left_len, right, 0, right_len); } else { maxchar = PyUnicode_MAX_CHAR_VALUE(left); @@ -10713,8 +10733,8 @@ res = PyUnicode_New(new_len, maxchar); if (res == NULL) goto error; - copy_characters(res, 0, left, 0, left_len); - copy_characters(res, left_len, right, 0, right_len); + _PyUnicode_FastCopyCharacters(res, 0, left, 0, left_len); + _PyUnicode_FastCopyCharacters(res, left_len, right, 0, right_len); Py_DECREF(left); *p_left = res; } @@ -11650,7 +11670,7 @@ length = end - start; if (PyUnicode_IS_ASCII(self)) { data = PyUnicode_1BYTE_DATA(self); - return unicode_fromascii(data + start, length); + return _PyUnicode_FromASCII((char*)(data + start), length); } else { kind = PyUnicode_KIND(self); @@ -12769,60 +12789,74 @@ return PyBool_FromLong(result); } -typedef struct { - PyObject *buffer; - void *data; - enum PyUnicode_Kind kind; - Py_UCS4 maxchar; - Py_ssize_t pos; -} _PyUnicodeWriter ; - Py_LOCAL_INLINE(void) _PyUnicodeWriter_Update(_PyUnicodeWriter *writer) { + writer->size = PyUnicode_GET_LENGTH(writer->buffer); writer->maxchar = PyUnicode_MAX_CHAR_VALUE(writer->buffer); writer->data = PyUnicode_DATA(writer->buffer); writer->kind = PyUnicode_KIND(writer->buffer); } -Py_LOCAL(int) -_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, - Py_ssize_t length, Py_UCS4 maxchar) -{ - writer->pos = 0; - writer->buffer = PyUnicode_New(length, maxchar); - if (writer->buffer == NULL) - return -1; - _PyUnicodeWriter_Update(writer); - return 0; -} - -Py_LOCAL_INLINE(int) -_PyUnicodeWriter_Prepare(_PyUnicodeWriter *writer, - Py_ssize_t length, Py_UCS4 maxchar) +void +_PyUnicodeWriter_Init(_PyUnicodeWriter *writer, Py_ssize_t min_length) +{ + memset(writer, 0, sizeof(*writer)); +#ifdef Py_DEBUG + writer->kind = 5; /* invalid kind */ +#endif + writer->min_length = Py_MAX(min_length, 100); + writer->flags.overallocate = (min_length > 0); +} + +int +_PyUnicodeWriter_PrepareInternal(_PyUnicodeWriter *writer, + Py_ssize_t length, Py_UCS4 maxchar) { Py_ssize_t newlen; PyObject *newbuffer; + assert(length > 0); + if (length > PY_SSIZE_T_MAX - writer->pos) { PyErr_NoMemory(); return -1; } newlen = writer->pos + length; - if (newlen > PyUnicode_GET_LENGTH(writer->buffer)) { - /* overallocate 25% to limit the number of resize */ - if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) - newlen += newlen / 4; - - if (maxchar > writer->maxchar) { + if (writer->buffer == NULL) { + if (writer->flags.overallocate) { + /* overallocate 25% to limit the number of resize */ + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + newlen += newlen / 4; + if (newlen < writer->min_length) + newlen = writer->min_length; + } + writer->buffer = PyUnicode_New(newlen, maxchar); + if (writer->buffer == NULL) + return -1; + _PyUnicodeWriter_Update(writer); + return 0; + } + + if (newlen > writer->size) { + if (writer->flags.overallocate) { + /* overallocate 25% to limit the number of resize */ + if (newlen <= (PY_SSIZE_T_MAX - newlen / 4)) + newlen += newlen / 4; + if (newlen < writer->min_length) + newlen = writer->min_length; + } + + if (maxchar > writer->maxchar || writer->flags.readonly) { /* resize + widen */ newbuffer = PyUnicode_New(newlen, maxchar); if (newbuffer == NULL) return -1; - PyUnicode_CopyCharacters(newbuffer, 0, - writer->buffer, 0, writer->pos); + _PyUnicode_FastCopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); Py_DECREF(writer->buffer); + writer->flags.readonly = 0; } else { newbuffer = resize_compact(writer->buffer, newlen); @@ -12833,25 +12867,76 @@ _PyUnicodeWriter_Update(writer); } else if (maxchar > writer->maxchar) { - if (unicode_widen(&writer->buffer, writer->pos, maxchar) < 0) + assert(!writer->flags.readonly); + newbuffer = PyUnicode_New(writer->size, maxchar); + if (newbuffer == NULL) return -1; + _PyUnicode_FastCopyCharacters(newbuffer, 0, + writer->buffer, 0, writer->pos); + Py_DECREF(writer->buffer); + writer->buffer = newbuffer; _PyUnicodeWriter_Update(writer); } return 0; } -Py_LOCAL(PyObject *) +int +_PyUnicodeWriter_WriteStr(_PyUnicodeWriter *writer, PyObject *str) +{ + Py_UCS4 maxchar; + Py_ssize_t len; + + if (PyUnicode_READY(str) == -1) + return -1; + len = PyUnicode_GET_LENGTH(str); + if (len == 0) + return 0; + maxchar = PyUnicode_MAX_CHAR_VALUE(str); + if (maxchar > writer->maxchar || len > writer->size - writer->pos) { + if (writer->buffer == NULL && !writer->flags.overallocate) { + Py_INCREF(str); + writer->buffer = str; + _PyUnicodeWriter_Update(writer); + writer->flags.readonly = 1; + writer->size = 0; + writer->pos += len; + return 0; + } + if (_PyUnicodeWriter_PrepareInternal(writer, len, maxchar) == -1) + return -1; + } + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + str, 0, len); + writer->pos += len; + return 0; +} + +PyObject * _PyUnicodeWriter_Finish(_PyUnicodeWriter *writer) { - if (PyUnicode_Resize(&writer->buffer, writer->pos) < 0) { - Py_DECREF(writer->buffer); - return NULL; + if (writer->pos == 0) { + Py_XDECREF(writer->buffer); + Py_INCREF(unicode_empty); + return unicode_empty; + } + if (writer->flags.readonly) { + assert(PyUnicode_GET_LENGTH(writer->buffer) == writer->pos); + return writer->buffer; + } + if (PyUnicode_GET_LENGTH(writer->buffer) != writer->pos) { + PyObject *newbuffer; + newbuffer = resize_compact(writer->buffer, writer->pos); + if (newbuffer == NULL) { + Py_DECREF(writer->buffer); + return NULL; + } + writer->buffer = newbuffer; } assert(_PyUnicode_CheckConsistency(writer->buffer, 1)); return writer->buffer; } -Py_LOCAL(void) +void _PyUnicodeWriter_Dealloc(_PyUnicodeWriter *writer) { Py_CLEAR(writer->buffer); @@ -12874,14 +12959,24 @@ static PyObject * unicode__format__(PyObject* self, PyObject* args) { - PyObject *format_spec, *out; + PyObject *format_spec; + _PyUnicodeWriter writer; + int ret; if (!PyArg_ParseTuple(args, "U:__format__", &format_spec)) return NULL; - out = _PyUnicode_FormatAdvanced(self, format_spec, 0, - PyUnicode_GET_LENGTH(format_spec)); - return out; + if (PyUnicode_READY(self) == -1) + return NULL; + _PyUnicodeWriter_Init(&writer, 0); + ret = _PyUnicode_FormatAdvancedWriter(&writer, + self, format_spec, 0, + PyUnicode_GET_LENGTH(format_spec)); + if (ret == -1) { + _PyUnicodeWriter_Dealloc(&writer); + return NULL; + } + return _PyUnicodeWriter_Finish(&writer); } PyDoc_STRVAR(p_format__doc__, @@ -13111,16 +13206,17 @@ /* Returns a new reference to a PyUnicode object, or NULL on failure. */ -static PyObject * -formatfloat(PyObject *v, int flags, int prec, int type) +static int +formatfloat(PyObject *v, int flags, int prec, int type, + PyObject **p_output, _PyUnicodeWriter *writer) { char *p; - PyObject *result; double x; + Py_ssize_t len; x = PyFloat_AsDouble(v); if (x == -1.0 && PyErr_Occurred()) - return NULL; + return -1; if (prec < 0) prec = 6; @@ -13128,10 +13224,20 @@ p = PyOS_double_to_string(x, type, prec, (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL); if (p == NULL) - return NULL; - result = unicode_fromascii((unsigned char*)p, strlen(p)); + return -1; + len = strlen(p); + if (writer) { + if (_PyUnicodeWriter_Prepare(writer, len, 127) == -1) + return -1; + memcpy(writer->data + writer->pos * writer->kind, + p, + len); + writer->pos += len; + } + else + *p_output = _PyUnicode_FromASCII(p, len); PyMem_Free(p); - return result; + return 0; } /* formatlong() emulates the format codes d, u, o, x and X, and @@ -13267,7 +13373,7 @@ } if (!PyUnicode_Check(result) || len != PyUnicode_GET_LENGTH(result)) { PyObject *unicode; - unicode = unicode_fromascii((unsigned char *)buf, len); + unicode = _PyUnicode_FromASCII(buf, len); Py_DECREF(result); result = unicode; } @@ -13336,8 +13442,7 @@ fmtcnt = PyUnicode_GET_LENGTH(uformat); fmtpos = 0; - if (_PyUnicodeWriter_Init(&writer, fmtcnt + 100, 127) < 0) - goto onError; + _PyUnicodeWriter_Init(&writer, fmtcnt + 100); if (PyTuple_Check(args)) { arglen = PyTuple_Size(args); @@ -13368,8 +13473,8 @@ if (_PyUnicodeWriter_Prepare(&writer, sublen, maxchar) == -1) goto onError; - copy_characters(writer.buffer, writer.pos, - uformat, nonfmtpos, sublen); + _PyUnicode_FastCopyCharacters(writer.buffer, writer.pos, + uformat, nonfmtpos, sublen); writer.pos += sublen; } else { @@ -13530,6 +13635,8 @@ "incomplete format"); goto onError; } + if (fmtcnt == 0) + writer.flags.overallocate = 0; if (c == '%') { if (_PyUnicodeWriter_Prepare(&writer, 1, '%') == -1) @@ -13539,7 +13646,6 @@ continue; } - v = getnextarg(args, arglen, &argidx); if (v == NULL) goto onError; @@ -13552,6 +13658,13 @@ case 's': case 'r': case 'a': + if (PyLong_CheckExact(v) && width == -1 && prec == -1) { + /* Fast path */ + if (_PyLong_FormatWriter(&writer, v, 10, flags & F_ALT) == -1) + goto onError; + goto nextarg; + } + if (PyUnicode_CheckExact(v) && c == 's') { temp = v; Py_INCREF(temp); @@ -13572,6 +13685,32 @@ case 'o': case 'x': case 'X': + if (PyLong_CheckExact(v) + && width == -1 && prec == -1 + && !(flags & (F_SIGN | F_BLANK))) + { + /* Fast path */ + switch(c) + { + case 'd': + case 'i': + case 'u': + if (_PyLong_FormatWriter(&writer, v, 10, flags & F_ALT) == -1) + goto onError; + goto nextarg; + case 'x': + if (_PyLong_FormatWriter(&writer, v, 16, flags & F_ALT) == -1) + goto onError; + goto nextarg; + case 'o': + if (_PyLong_FormatWriter(&writer, v, 8, flags & F_ALT) == -1) + goto onError; + goto nextarg; + default: + break; + } + } + isnumok = 0; if (PyNumber_Check(v)) { PyObject *iobj=NULL; @@ -13611,10 +13750,20 @@ case 'F': case 'g': case 'G': + if (width == -1 && prec == -1 + && !(flags & (F_SIGN | F_BLANK))) + { + /* Fast path */ + if (formatfloat(v, flags, prec, c, NULL, &writer) == -1) + goto onError; + goto nextarg; + } + sign = 1; if (flags & F_ZERO) fill = '0'; - temp = formatfloat(v, flags, prec, c); + if (formatfloat(v, flags, prec, c, &temp, NULL) == -1) + temp = NULL; break; case 'c': @@ -13622,6 +13771,14 @@ Py_UCS4 ch = formatchar(v); if (ch == (Py_UCS4) -1) goto onError; + if (width == -1 && prec == -1) { + /* Fast path */ + if (_PyUnicodeWriter_Prepare(&writer, 1, ch) == -1) + goto onError; + PyUnicode_WRITE(writer.kind, writer.data, writer.pos, ch); + writer.pos += 1; + goto nextarg; + } temp = PyUnicode_FromOrdinal(ch); break; } @@ -13638,6 +13795,16 @@ if (temp == NULL) goto onError; assert (PyUnicode_Check(temp)); + + if (width == -1 && prec == -1 + && !(flags & (F_SIGN | F_BLANK))) + { + /* Fast path */ + if (_PyUnicodeWriter_WriteStr(&writer, temp) == -1) + goto onError; + goto nextarg; + } + if (PyUnicode_READY(temp) == -1) { Py_CLEAR(temp); goto onError; @@ -13676,15 +13843,15 @@ if (!(flags & F_LJUST)) { if (sign) { if ((width-1) > len) - bufmaxchar = Py_MAX(bufmaxchar, fill); + bufmaxchar = MAX_MAXCHAR(bufmaxchar, fill); } else { if (width > len) - bufmaxchar = Py_MAX(bufmaxchar, fill); + bufmaxchar = MAX_MAXCHAR(bufmaxchar, fill); } } maxchar = _PyUnicode_FindMaxChar(temp, 0, pindex+len); - bufmaxchar = Py_MAX(bufmaxchar, maxchar); + bufmaxchar = MAX_MAXCHAR(bufmaxchar, maxchar); buflen = width; if (sign && len == width) @@ -13737,8 +13904,8 @@ } } - copy_characters(writer.buffer, writer.pos, - temp, pindex, len); + _PyUnicode_FastCopyCharacters(writer.buffer, writer.pos, + temp, pindex, len); writer.pos += len; if (width > len) { sublen = width - len; @@ -13746,6 +13913,7 @@ writer.pos += sublen; } +nextarg: if (dict && (argidx < arglen) && c != '%') { PyErr_SetString(PyExc_TypeError, "not all arguments converted during string formatting"); diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c --- a/Python/formatter_unicode.c +++ b/Python/formatter_unicode.c @@ -316,21 +316,28 @@ /* Do the padding, and return a pointer to where the caller-supplied content goes. */ static Py_ssize_t -fill_padding(PyObject *s, Py_ssize_t start, Py_ssize_t nchars, +fill_padding(_PyUnicodeWriter *writer, + Py_ssize_t nchars, Py_UCS4 fill_char, Py_ssize_t n_lpadding, Py_ssize_t n_rpadding) { + Py_ssize_t pos; + /* Pad on left. */ - if (n_lpadding) - PyUnicode_Fill(s, start, start + n_lpadding, fill_char); + if (n_lpadding) { + pos = writer->pos; + _PyUnicode_FastFill(writer->buffer, pos, n_lpadding, fill_char); + } /* Pad on right. */ - if (n_rpadding) - PyUnicode_Fill(s, start + nchars + n_lpadding, - start + nchars + n_lpadding + n_rpadding, fill_char); + if (n_rpadding) { + pos = writer->pos + nchars + n_lpadding; + _PyUnicode_FastFill(writer->buffer, pos, n_rpadding, fill_char); + } /* Pointer to the user content. */ - return start + n_lpadding; + writer->pos += n_lpadding; + return 0; } /************************************************************************/ @@ -541,7 +548,7 @@ as determined in calc_number_widths(). Return -1 on error, or 0 on success. */ static int -fill_number(PyObject *out, Py_ssize_t pos, const NumberFieldWidths *spec, +fill_number(_PyUnicodeWriter *writer, const NumberFieldWidths *spec, PyObject *digits, Py_ssize_t d_start, Py_ssize_t d_end, PyObject *prefix, Py_ssize_t p_start, Py_UCS4 fill_char, @@ -549,36 +556,38 @@ { /* Used to keep track of digits, decimal, and remainder. */ Py_ssize_t d_pos = d_start; - unsigned int kind = PyUnicode_KIND(out); - void *data = PyUnicode_DATA(out); + const enum PyUnicode_Kind kind = writer->kind; + const void *data = writer->data; Py_ssize_t r; if (spec->n_lpadding) { - PyUnicode_Fill(out, pos, pos + spec->n_lpadding, fill_char); - pos += spec->n_lpadding; + _PyUnicode_FastFill(writer->buffer, + writer->pos, spec->n_lpadding, fill_char); + writer->pos += spec->n_lpadding; } if (spec->n_sign == 1) { - PyUnicode_WRITE(kind, data, pos++, spec->sign); + PyUnicode_WRITE(kind, data, writer->pos, spec->sign); + writer->pos++; } if (spec->n_prefix) { - if (PyUnicode_CopyCharacters(out, pos, - prefix, p_start, - spec->n_prefix) < 0) - return -1; + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + prefix, p_start, + spec->n_prefix); if (toupper) { Py_ssize_t t; for (t = 0; t < spec->n_prefix; t++) { - Py_UCS4 c = PyUnicode_READ(kind, data, pos + t); + Py_UCS4 c = PyUnicode_READ(kind, data, writer->pos + t); c = Py_TOUPPER(c); assert (c <= 127); - PyUnicode_WRITE(kind, data, pos + t, c); + PyUnicode_WRITE(kind, data, writer->pos + t, c); } } - pos += spec->n_prefix; + writer->pos += spec->n_prefix; } if (spec->n_spadding) { - PyUnicode_Fill(out, pos, pos + spec->n_spadding, fill_char); - pos += spec->n_spadding; + _PyUnicode_FastFill(writer->buffer, + writer->pos, spec->n_spadding, fill_char); + writer->pos += spec->n_spadding; } /* Only for type 'c' special case, it has no digits. */ @@ -594,7 +603,7 @@ return -1; } r = _PyUnicode_InsertThousandsGrouping( - out, pos, + writer->buffer, writer->pos, spec->n_grouped_digits, pdigits + kind * d_pos, spec->n_digits, spec->n_min_width, @@ -609,34 +618,38 @@ if (toupper) { Py_ssize_t t; for (t = 0; t < spec->n_grouped_digits; t++) { - Py_UCS4 c = PyUnicode_READ(kind, data, pos + t); + Py_UCS4 c = PyUnicode_READ(kind, data, writer->pos + t); c = Py_TOUPPER(c); if (c > 127) { PyErr_SetString(PyExc_SystemError, "non-ascii grouped digit"); return -1; } - PyUnicode_WRITE(kind, data, pos + t, c); + PyUnicode_WRITE(kind, data, writer->pos + t, c); } } - pos += spec->n_grouped_digits; + writer->pos += spec->n_grouped_digits; if (spec->n_decimal) { - if (PyUnicode_CopyCharacters(out, pos, locale->decimal_point, 0, spec->n_decimal) < 0) - return -1; - pos += spec->n_decimal; + _PyUnicode_FastCopyCharacters( + writer->buffer, writer->pos, + locale->decimal_point, 0, spec->n_decimal); + writer->pos += spec->n_decimal; d_pos += 1; } if (spec->n_remainder) { - if (PyUnicode_CopyCharacters(out, pos, digits, d_pos, spec->n_remainder) < 0) - return -1; - pos += spec->n_remainder; + _PyUnicode_FastCopyCharacters( + writer->buffer, writer->pos, + digits, d_pos, spec->n_remainder); + writer->pos += spec->n_remainder; d_pos += spec->n_remainder; } if (spec->n_rpadding) { - PyUnicode_Fill(out, pos, pos + spec->n_rpadding, fill_char); - pos += spec->n_rpadding; + _PyUnicode_FastFill(writer->buffer, + writer->pos, spec->n_rpadding, + fill_char); + writer->pos += spec->n_rpadding; } return 0; } @@ -707,17 +720,20 @@ /*********** string formatting ******************************************/ /************************************************************************/ -static PyObject * -format_string_internal(PyObject *value, const InternalFormatSpec *format) +static int +format_string_internal(PyObject *value, const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { Py_ssize_t lpad; Py_ssize_t rpad; Py_ssize_t total; - Py_ssize_t pos; - Py_ssize_t len = PyUnicode_GET_LENGTH(value); - PyObject *result = NULL; + Py_ssize_t len; + int result = -1; Py_UCS4 maxchar; + assert(PyUnicode_IS_READY(value)); + len = PyUnicode_GET_LENGTH(value); + /* sign is not allowed on strings */ if (format->sign != '\0') { PyErr_SetString(PyExc_ValueError, @@ -741,6 +757,11 @@ goto done; } + if (format->width == -1 && format->precision == -1) { + /* Fast path */ + return _PyUnicodeWriter_WriteStr(writer, value); + } + /* if precision is specified, output no more that format.precision characters */ if (format->precision >= 0 && len >= format->precision) { @@ -754,21 +775,23 @@ maxchar = Py_MAX(maxchar, format->fill_char); /* allocate the resulting string */ - result = PyUnicode_New(total, maxchar); - if (result == NULL) + if (_PyUnicodeWriter_Prepare(writer, total, maxchar) == -1) goto done; /* Write into that space. First the padding. */ - pos = fill_padding(result, 0, len, - format->fill_char=='\0'?' ':format->fill_char, - lpad, rpad); + result = fill_padding(writer, len, + format->fill_char=='\0'?' ':format->fill_char, + lpad, rpad); + if (result == -1) + goto done; /* Then the source string. */ - if (PyUnicode_CopyCharacters(result, pos, value, 0, len) < 0) - Py_CLEAR(result); + _PyUnicode_FastCopyCharacters(writer->buffer, writer->pos, + value, 0, len); + writer->pos += (len + rpad); + result = 0; done: - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } @@ -780,11 +803,11 @@ typedef PyObject* (*IntOrLongToString)(PyObject *value, int base); -static PyObject * -format_int_or_long_internal(PyObject *value, const InternalFormatSpec *format, - IntOrLongToString tostring) +static int +format_long_internal(PyObject *value, const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { - PyObject *result = NULL; + int result = -1; Py_UCS4 maxchar = 127; PyObject *tmp = NULL; Py_ssize_t inumeric_chars; @@ -798,7 +821,6 @@ Py_ssize_t prefix = 0; NumberFieldWidths spec; long x; - int err; /* Locale settings, either from the actual locale or from a hard-code pseudo-locale */ @@ -872,13 +894,23 @@ break; } + if (format->sign != '+' && format->sign != ' ' + && format->width == -1 + && format->type != 'X' && format->type != 'n' + && !format->thousands_separators + && PyLong_CheckExact(value)) + { + /* Fast path */ + return _PyLong_FormatWriter(writer, value, base, format->alternate); + } + /* The number of prefix chars is the same as the leading chars to skip */ if (format->alternate) n_prefix = leading_chars_to_skip; /* Do the hard part, converting to a string in a given base */ - tmp = tostring(value, base); + tmp = _PyLong_Format(value, base); if (tmp == NULL || PyUnicode_READY(tmp) == -1) goto done; @@ -914,23 +946,19 @@ &locale, format, &maxchar); /* Allocate the memory. */ - result = PyUnicode_New(n_total, maxchar); - if (!result) + if (_PyUnicodeWriter_Prepare(writer, n_total, maxchar) == -1) goto done; /* Populate the memory. */ - err = fill_number(result, 0, &spec, - tmp, inumeric_chars, inumeric_chars + n_digits, - tmp, prefix, - format->fill_char == '\0' ? ' ' : format->fill_char, - &locale, format->type == 'X'); - if (err) - Py_CLEAR(result); + result = fill_number(writer, &spec, + tmp, inumeric_chars, inumeric_chars + n_digits, + tmp, prefix, + format->fill_char == '\0' ? ' ' : format->fill_char, + &locale, format->type == 'X'); done: Py_XDECREF(tmp); free_locale_info(&locale); - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } @@ -938,16 +966,11 @@ /*********** float formatting *******************************************/ /************************************************************************/ -static PyObject* -strtounicode(char *charbuffer, Py_ssize_t len) -{ - return PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, charbuffer, len); -} - /* much of this is taken from unicodeobject.c */ -static PyObject * +static int format_float_internal(PyObject *value, - const InternalFormatSpec *format) + const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { char *buf = NULL; /* buffer returned from PyOS_double_to_string */ Py_ssize_t n_digits; @@ -962,12 +985,11 @@ Py_ssize_t index; NumberFieldWidths spec; int flags = 0; - PyObject *result = NULL; + int result = -1; Py_UCS4 maxchar = 127; Py_UCS4 sign_char = '\0'; int float_type; /* Used to see if we have a nan, inf, or regular float. */ PyObject *unicode_tmp = NULL; - int err; /* Locale settings, either from the actual locale or from a hard-code pseudo-locale */ @@ -1024,13 +1046,25 @@ /* Since there is no unicode version of PyOS_double_to_string, just use the 8 bit version and then convert to unicode. */ - unicode_tmp = strtounicode(buf, n_digits); + unicode_tmp = _PyUnicode_FromASCII(buf, n_digits); + PyMem_Free(buf); if (unicode_tmp == NULL) goto done; - index = 0; + + if (format->sign != '+' && format->sign != ' ' + && format->width == -1 + && format->type != 'n' + && !format->thousands_separators) + { + /* Fast path */ + result = _PyUnicodeWriter_WriteStr(writer, unicode_tmp); + Py_DECREF(unicode_tmp); + return result; + } /* Is a sign character present in the output? If so, remember it and skip it */ + index = 0; if (PyUnicode_READ_CHAR(unicode_tmp, index) == '-') { sign_char = '-'; ++index; @@ -1055,24 +1089,19 @@ &locale, format, &maxchar); /* Allocate the memory. */ - result = PyUnicode_New(n_total, maxchar); - if (result == NULL) + if (_PyUnicodeWriter_Prepare(writer, n_total, maxchar) == -1) goto done; /* Populate the memory. */ - err = fill_number(result, 0, &spec, - unicode_tmp, index, index + n_digits, - NULL, 0, - format->fill_char == '\0' ? ' ' : format->fill_char, - &locale, 0); - if (err) - Py_CLEAR(result); + result = fill_number(writer, &spec, + unicode_tmp, index, index + n_digits, + NULL, 0, + format->fill_char == '\0' ? ' ' : format->fill_char, + &locale, 0); done: - PyMem_Free(buf); Py_DECREF(unicode_tmp); free_locale_info(&locale); - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } @@ -1080,9 +1109,10 @@ /*********** complex formatting *****************************************/ /************************************************************************/ -static PyObject * +static int format_complex_internal(PyObject *value, - const InternalFormatSpec *format) + const InternalFormatSpec *format, + _PyUnicodeWriter *writer) { double re; double im; @@ -1106,11 +1136,10 @@ NumberFieldWidths re_spec; NumberFieldWidths im_spec; int flags = 0; - PyObject *result = NULL; + int result = -1; Py_UCS4 maxchar = 127; - int rkind; + enum PyUnicode_Kind rkind; void *rdata; - Py_ssize_t index; Py_UCS4 re_sign_char = '\0'; Py_UCS4 im_sign_char = '\0'; int re_float_type; /* Used to see if we have a nan, inf, or regular float. */ @@ -1122,7 +1151,6 @@ Py_ssize_t total; PyObject *re_unicode_tmp = NULL; PyObject *im_unicode_tmp = NULL; - int err; /* Locale settings, either from the actual locale or from a hard-code pseudo-locale */ @@ -1191,12 +1219,12 @@ /* Since there is no unicode version of PyOS_double_to_string, just use the 8 bit version and then convert to unicode. */ - re_unicode_tmp = strtounicode(re_buf, n_re_digits); + re_unicode_tmp = _PyUnicode_FromASCII(re_buf, n_re_digits); if (re_unicode_tmp == NULL) goto done; i_re = 0; - im_unicode_tmp = strtounicode(im_buf, n_im_digits); + im_unicode_tmp = _PyUnicode_FromASCII(im_buf, n_im_digits); if (im_unicode_tmp == NULL) goto done; i_im = 0; @@ -1261,47 +1289,49 @@ if (lpad || rpad) maxchar = Py_MAX(maxchar, format->fill_char); - result = PyUnicode_New(total, maxchar); - if (result == NULL) + if (_PyUnicodeWriter_Prepare(writer, total, maxchar) == -1) goto done; - rkind = PyUnicode_KIND(result); - rdata = PyUnicode_DATA(result); + rkind = writer->kind; + rdata = writer->data; /* Populate the memory. First, the padding. */ - index = fill_padding(result, 0, - n_re_total + n_im_total + 1 + add_parens * 2, - format->fill_char=='\0' ? ' ' : format->fill_char, - lpad, rpad); + result = fill_padding(writer, + n_re_total + n_im_total + 1 + add_parens * 2, + format->fill_char=='\0' ? ' ' : format->fill_char, + lpad, rpad); + if (result == -1) + goto done; - if (add_parens) - PyUnicode_WRITE(rkind, rdata, index++, '('); + if (add_parens) { + PyUnicode_WRITE(rkind, rdata, writer->pos, '('); + writer->pos++; + } if (!skip_re) { - err = fill_number(result, index, &re_spec, - re_unicode_tmp, i_re, i_re + n_re_digits, - NULL, 0, - 0, - &locale, 0); - if (err) { - Py_CLEAR(result); + result = fill_number(writer, &re_spec, + re_unicode_tmp, i_re, i_re + n_re_digits, + NULL, 0, + 0, + &locale, 0); + if (result == -1) goto done; - } - index += n_re_total; } - err = fill_number(result, index, &im_spec, - im_unicode_tmp, i_im, i_im + n_im_digits, - NULL, 0, - 0, - &locale, 0); - if (err) { - Py_CLEAR(result); + result = fill_number(writer, &im_spec, + im_unicode_tmp, i_im, i_im + n_im_digits, + NULL, 0, + 0, + &locale, 0); + if (result == -1) goto done; + PyUnicode_WRITE(rkind, rdata, writer->pos, 'j'); + writer->pos++; + + if (add_parens) { + PyUnicode_WRITE(rkind, rdata, writer->pos, ')'); + writer->pos++; } - index += n_im_total; - PyUnicode_WRITE(rkind, rdata, index++, 'j'); - if (add_parens) - PyUnicode_WRITE(rkind, rdata, index++, ')'); + writer->pos += rpad; done: PyMem_Free(re_buf); @@ -1309,61 +1339,79 @@ Py_XDECREF(re_unicode_tmp); Py_XDECREF(im_unicode_tmp); free_locale_info(&locale); - assert(!result || _PyUnicode_CheckConsistency(result, 1)); return result; } /************************************************************************/ /*********** built in formatters ****************************************/ /************************************************************************/ -PyObject * -_PyUnicode_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) +int +format_obj(PyObject *obj, _PyUnicodeWriter *writer) +{ + PyObject *str; + int err; + + str = PyObject_Str(obj); + if (str == NULL) + return -1; + err = _PyUnicodeWriter_WriteStr(writer, str); + Py_DECREF(str); + return err; +} + +int +_PyUnicode_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { InternalFormatSpec format; - PyObject *result; + + assert(PyUnicode_Check(obj)); /* check for the special case of zero length format spec, make it equivalent to str(obj) */ - if (start == end) - return PyObject_Str(obj); + if (start == end) { + if (PyUnicode_CheckExact(obj)) + return _PyUnicodeWriter_WriteStr(writer, obj); + else + return format_obj(obj, writer); + } /* parse the format_spec */ if (!parse_internal_render_format_spec(format_spec, start, end, &format, 's', '<')) - return NULL; + return -1; /* type conversion? */ switch (format.type) { case 's': /* no type conversion needed, already a string. do the formatting */ - result = format_string_internal(obj, &format); - if (result != NULL) - assert(_PyUnicode_CheckConsistency(result, 1)); - break; + return format_string_internal(obj, &format, writer); default: /* unknown */ unknown_presentation_type(format.type, obj->ob_type->tp_name); - result = NULL; + return -1; } - return result; } -static PyObject* -format_int_or_long(PyObject* obj, PyObject* format_spec, - Py_ssize_t start, Py_ssize_t end, - IntOrLongToString tostring) +int +_PyLong_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { - PyObject *result = NULL; - PyObject *tmp = NULL; + PyObject *tmp = NULL, *str = NULL; InternalFormatSpec format; + int result = -1; /* check for the special case of zero length format spec, make it equivalent to str(obj) */ if (start == end) { - result = PyObject_Str(obj); - goto done; + if (PyLong_CheckExact(obj)) + return _PyLong_FormatWriter(writer, obj, 10, 0); + else + return format_obj(obj, writer); } /* parse the format_spec */ @@ -1382,7 +1430,7 @@ case 'n': /* no type conversion needed, already an int (or long). do the formatting */ - result = format_int_or_long_internal(obj, &format, tostring); + result = format_long_internal(obj, &format, writer); break; case 'e': @@ -1396,7 +1444,7 @@ tmp = PyNumber_Float(obj); if (tmp == NULL) goto done; - result = format_float_internal(tmp, &format); + result = format_float_internal(tmp, &format, writer); break; default: @@ -1407,41 +1455,27 @@ done: Py_XDECREF(tmp); + Py_XDECREF(str); return result; } -/* Need to define long_format as a function that will convert a long - to a string. In 3.0, _PyLong_Format has the correct signature. */ -#define long_format _PyLong_Format - -PyObject * -_PyLong_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) +int +_PyFloat_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { - return format_int_or_long(obj, format_spec, start, end, - long_format); -} - -PyObject * -_PyFloat_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) -{ - PyObject *result = NULL; InternalFormatSpec format; /* check for the special case of zero length format spec, make it equivalent to str(obj) */ - if (start == end) { - result = PyObject_Str(obj); - goto done; - } + if (start == end) + return format_obj(obj, writer); /* parse the format_spec */ if (!parse_internal_render_format_spec(format_spec, start, end, &format, '\0', '>')) - goto done; + return -1; /* type conversion? */ switch (format.type) { @@ -1455,38 +1489,32 @@ case 'n': case '%': /* no conversion, already a float. do the formatting */ - result = format_float_internal(obj, &format); - break; + return format_float_internal(obj, &format, writer); default: /* unknown */ unknown_presentation_type(format.type, obj->ob_type->tp_name); - goto done; + return -1; } - -done: - return result; } -PyObject * -_PyComplex_FormatAdvanced(PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, Py_ssize_t end) +int +_PyComplex_FormatAdvancedWriter(_PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, Py_ssize_t end) { - PyObject *result = NULL; InternalFormatSpec format; /* check for the special case of zero length format spec, make it equivalent to str(obj) */ - if (start == end) { - result = PyObject_Str(obj); - goto done; - } + if (start == end) + return format_obj(obj, writer); /* parse the format_spec */ if (!parse_internal_render_format_spec(format_spec, start, end, &format, '\0', '>')) - goto done; + return -1; /* type conversion? */ switch (format.type) { @@ -1499,15 +1527,11 @@ case 'G': case 'n': /* no conversion, already a complex. do the formatting */ - result = format_complex_internal(obj, &format); - break; + return format_complex_internal(obj, &format, writer); default: /* unknown */ unknown_presentation_type(format.type, obj->ob_type->tp_name); - goto done; + return -1; } - -done: - return result; } diff --git a/Python/getargs.c b/Python/getargs.c --- a/Python/getargs.c +++ b/Python/getargs.c @@ -1167,8 +1167,11 @@ case 'U': { /* PyUnicode object */ PyObject **p = va_arg(*p_va, PyObject **); - if (PyUnicode_Check(arg)) + if (PyUnicode_Check(arg)) { + if (PyUnicode_READY(arg) == -1) + RETURN_ERR_OCCURRED; *p = arg; + } else return converterr("str", arg, msgbuf, bufsize); break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 14:46:52 2012 From: python-checkins at python.org (eli.bendersky) Date: Tue, 29 May 2012 14:46:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314007=3A_make_Tree?= =?utf8?q?Builder_an_actual_type_exposed_from_=5Felementtree=2C_and?= Message-ID: http://hg.python.org/cpython/rev/717632ae7b3f changeset: 77226:717632ae7b3f parent: 77223:22b56b0b8619 user: Eli Bendersky date: Tue May 29 15:45:16 2012 +0300 summary: Issue #14007: make TreeBuilder an actual type exposed from _elementtree, and subclassable. files: Lib/test/test_xml_etree.py | 14 ++ Modules/_elementtree.c | 163 ++++++++++++------------ 2 files changed, 97 insertions(+), 80 deletions(-) diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1979,6 +1979,20 @@ parser.feed(self.sample1) self.assertIsNone(parser.close()) + def test_subclass(self): + class MyTreeBuilder(ET.TreeBuilder): + def foobar(self, x): + return x * 2 + + tb = MyTreeBuilder() + self.assertEqual(tb.foobar(10), 20) + + parser = ET.XMLParser(target=tb) + parser.feed(self.sample1) + + e = parser.close() + self.assertEqual(e.tag, 'html') + # XXX in _elementtree, the constructor of TreeBuilder expects no # arguments @unittest.expectedFailure diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -1803,23 +1803,22 @@ typedef struct { PyObject_HEAD - PyObject* root; /* root node (first created node) */ - - ElementObject* this; /* current node */ - ElementObject* last; /* most recently created node */ - - PyObject* data; /* data collector (string or list), or NULL */ - - PyObject* stack; /* element stack */ - Py_ssize_t index; /* current stack size (0=empty) */ + PyObject *root; /* root node (first created node) */ + + ElementObject *this; /* current node */ + ElementObject *last; /* most recently created node */ + + PyObject *data; /* data collector (string or list), or NULL */ + + PyObject *stack; /* element stack */ + Py_ssize_t index; /* current stack size (0 means empty) */ /* element tracing */ - PyObject* events; /* list of events, or NULL if not collecting */ - PyObject* start_event_obj; /* event objects (NULL to ignore) */ - PyObject* end_event_obj; - PyObject* start_ns_event_obj; - PyObject* end_ns_event_obj; - + PyObject *events; /* list of events, or NULL if not collecting */ + PyObject *start_event_obj; /* event objects (NULL to ignore) */ + PyObject *end_event_obj; + PyObject *start_ns_event_obj; + PyObject *end_ns_event_obj; } TreeBuilderObject; static PyTypeObject TreeBuilder_Type; @@ -1829,48 +1828,42 @@ /* -------------------------------------------------------------------- */ /* constructor and destructor */ -LOCAL(PyObject*) -treebuilder_new(void) +static PyObject * +treebuilder_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { - TreeBuilderObject* self; - - self = PyObject_New(TreeBuilderObject, &TreeBuilder_Type); - if (self == NULL) - return NULL; - - self->root = NULL; - - Py_INCREF(Py_None); - self->this = (ElementObject*) Py_None; - - Py_INCREF(Py_None); - self->last = (ElementObject*) Py_None; - - self->data = NULL; - - self->stack = PyList_New(20); - self->index = 0; - - self->events = NULL; - self->start_event_obj = self->end_event_obj = NULL; - self->start_ns_event_obj = self->end_ns_event_obj = NULL; - - ALLOC(sizeof(TreeBuilderObject), "create treebuilder"); - - return (PyObject*) self; + TreeBuilderObject *t = (TreeBuilderObject *)type->tp_alloc(type, 0); + if (t != NULL) { + t->root = NULL; + + Py_INCREF(Py_None); + t->this = (ElementObject *)Py_None; + Py_INCREF(Py_None); + t->last = (ElementObject *)Py_None; + + t->data = NULL; + t->stack = PyList_New(20); + if (!t->stack) { + Py_DECREF(t->this); + Py_DECREF(t->last); + return NULL; + } + t->index = 0; + + t->events = NULL; + t->start_event_obj = t->end_event_obj = NULL; + t->start_ns_event_obj = t->end_ns_event_obj = NULL; + } + return (PyObject *)t; } -static PyObject* -treebuilder(PyObject* self_, PyObject* args) +static int +treebuilder_init(PyObject *self, PyObject *args, PyObject *kwds) { - if (!PyArg_ParseTuple(args, ":TreeBuilder")) - return NULL; - - return treebuilder_new(); + return 0; } static void -treebuilder_dealloc(TreeBuilderObject* self) +treebuilder_dealloc(TreeBuilderObject *self) { Py_XDECREF(self->end_ns_event_obj); Py_XDECREF(self->start_ns_event_obj); @@ -1883,9 +1876,7 @@ Py_DECREF(self->this); Py_XDECREF(self->root); - RELEASE(sizeof(TreeBuilderObject), "destroy treebuilder"); - - PyObject_Del(self); + Py_TYPE(self)->tp_free((PyObject *)self); } /* -------------------------------------------------------------------- */ @@ -2174,31 +2165,41 @@ PyVarObject_HEAD_INIT(NULL, 0) "TreeBuilder", sizeof(TreeBuilderObject), 0, /* methods */ - (destructor)treebuilder_dealloc, /* tp_dealloc */ - 0, /* tp_print */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_reserved */ - 0, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - 0, /* tp_call */ - 0, /* tp_str */ - 0, /* tp_getattro */ - 0, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT, /* tp_flags */ - 0, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ - 0, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - treebuilder_methods, /* tp_methods */ - 0, /* tp_members */ + (destructor)treebuilder_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ + 0, /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + 0, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + treebuilder_methods, /* tp_methods */ + 0, /* tp_members */ + 0, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + (initproc)treebuilder_init, /* tp_init */ + PyType_GenericAlloc, /* tp_alloc */ + treebuilder_new, /* tp_new */ + 0, /* tp_free */ }; /* ==================================================================== */ @@ -2684,7 +2685,7 @@ /* setup target handlers */ if (!target) { - target = treebuilder_new(); + target = treebuilder_new(&TreeBuilder_Type, NULL, NULL); if (!target) { EXPAT(ParserFree)(self->parser); PyObject_Del(self->names); @@ -3073,7 +3074,6 @@ static PyMethodDef _functions[] = { {"SubElement", (PyCFunction) subelement, METH_VARARGS|METH_KEYWORDS}, - {"TreeBuilder", (PyCFunction) treebuilder, METH_VARARGS}, #if defined(USE_EXPAT) {"XMLParser", (PyCFunction) xmlparser, METH_VARARGS|METH_KEYWORDS}, #endif @@ -3185,5 +3185,8 @@ Py_INCREF((PyObject *)&Element_Type); PyModule_AddObject(m, "Element", (PyObject *)&Element_Type); + Py_INCREF((PyObject *)&TreeBuilder_Type); + PyModule_AddObject(m, "TreeBuilder", (PyObject *)&TreeBuilder_Type); + return m; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 14:46:52 2012 From: python-checkins at python.org (eli.bendersky) Date: Tue, 29 May 2012 14:46:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?q?=29=3A_merge_heads?= Message-ID: http://hg.python.org/cpython/rev/a3aa056fe8c2 changeset: 77227:a3aa056fe8c2 parent: 77226:717632ae7b3f parent: 77225:c4560d003e36 user: Eli Bendersky date: Tue May 29 15:45:40 2012 +0300 summary: merge heads files: Doc/library/venv.rst | 9 +-- Doc/using/index.rst | 2 +- Doc/using/scripts.rst | 66 +++++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 6 deletions(-) diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -54,7 +54,7 @@ The command, if run with ``-h``, will show the available options:: usage: pyvenv [-h] [--system-site-packages] [--symlink] [--clear] - ENV_DIR [ENV_DIR ...] + [--upgrade] ENV_DIR [ENV_DIR ...] Creates virtual Python environments in one or more target directories. @@ -69,12 +69,11 @@ --clear Delete the environment directory if it already exists. If not specified and the directory exists, an error is raised. - + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. If the target directory already exists an error will be raised, unless -the ``--clear`` option was provided, in which case the target -directory will be deleted and virtual environment creation will -proceed as usual. +the ``--clear`` or ``--upgrade`` option was provided. The created ``pyvenv.cfg`` file also includes the ``include-system-site-packages`` key, set to ``true`` if ``venv`` is diff --git a/Doc/using/index.rst b/Doc/using/index.rst --- a/Doc/using/index.rst +++ b/Doc/using/index.rst @@ -17,4 +17,4 @@ unix.rst windows.rst mac.rst - + scripts.rst diff --git a/Doc/using/scripts.rst b/Doc/using/scripts.rst new file mode 100644 --- /dev/null +++ b/Doc/using/scripts.rst @@ -0,0 +1,66 @@ +.. _tools-and-scripts: + +Additional Tools and Scripts +============================ + +pyvenv - Creating virtual environments +-------------------------------------- + +Creation of virtual environments is done by executing the ``pyvenv`` +script:: + + pyvenv /path/to/new/virtual/environment + +Running this command creates the target directory (creating any parent +directories that don't exist already) and places a ``pyvenv.cfg`` file +in it with a ``home`` key pointing to the Python installation the +command was run from. It also creates a ``bin`` (or ``Scripts`` on +Windows) subdirectory containing a copy of the ``python`` binary (or +binaries, in the case of Windows) and the ``pysetup3`` script (to +facilitate easy installation of packages from PyPI into the new virtualenv). +It also creates an (initially empty) ``lib/pythonX.Y/site-packages`` +subdirectory (on Windows, this is ``Lib\site-packages``). + +.. highlight:: none + +On Windows, you may have to invoke the ``pyvenv`` script as follows, if you +don't have the relevant PATH and PATHEXT settings:: + + c:\Temp>c:\Python33\python c:\Python33\Tools\Scripts\pyvenv.py myenv + +or equivalently:: + + c:\Temp>c:\Python33\python -m venv myenv + +The command, if run with ``-h``, will show the available options:: + + usage: pyvenv [-h] [--system-site-packages] [--symlink] [--clear] + [--upgrade] ENV_DIR [ENV_DIR ...] + + Creates virtual Python environments in one or more target directories. + + positional arguments: + ENV_DIR A directory to create the environment in. + + optional arguments: + -h, --help show this help message and exit + --system-site-packages Give access to the global site-packages dir to the + virtual environment. + --symlink Attempt to symlink rather than copy. + --clear Delete the environment directory if it already exists. + If not specified and the directory exists, an error is + raised. + --upgrade Upgrade the environment directory to use this version + of Python, assuming Python has been upgraded in-place. + +If the target directory already exists an error will be raised, unless +the ``--clear`` or ``--upgrade`` option was provided. + +The created ``pyvenv.cfg`` file also includes the +``include-system-site-packages`` key, set to ``true`` if ``venv`` is +run with the ``--system-site-packages`` option, ``false`` otherwise. + +Multiple paths can be given to ``pyvenv``, in which case an identical +virtualenv will be created, according to the given options, at each +provided path. + -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 15:14:55 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 29 May 2012 15:14:55 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2310839=3A_raise_an_error_?= =?utf8?q?on_add_of_duplicate_unique_headers_in_new_email_policies?= Message-ID: http://hg.python.org/cpython/rev/d7881a371c41 changeset: 77228:d7881a371c41 user: R David Murray date: Tue May 29 09:14:44 2012 -0400 summary: #10839: raise an error on add of duplicate unique headers in new email policies This feature was supposed to be part of the initial email6 checkin, but it got lost in my big refactoring. In this patch I'm not providing an easy way to turn off the errors, but they only happen when a header is added programmatically, and it is almost never the right thing to do to allow the duplicate to be added. An application that needs to add duplicates of unique headers can create a policy subclass to allow it. files: Doc/library/email.policy.rst | 25 ++++++++++++++++++++++++ Lib/email/_policybase.py | 19 ++++++++++++++++++ Lib/email/message.py | 10 +++++++++ Lib/email/policy.py | 8 +++++++ 4 files changed, 62 insertions(+), 0 deletions(-) diff --git a/Doc/library/email.policy.rst b/Doc/library/email.policy.rst --- a/Doc/library/email.policy.rst +++ b/Doc/library/email.policy.rst @@ -196,6 +196,25 @@ custom ``Message`` objects) should also provide such an attribute, otherwise defects in parsed messages will raise unexpected errors. + .. method:: header_max_count(name) + + Return the maximum allowed number of headers named *name*. + + Called when a header is added to a :class:`~email.message.Message` + object. If the returned value is not ``0`` or ``None``, and there are + already a number of headers with the name *name* equal to the value + returned, a :exc:`ValueError` is raised. + + Because the default behavior of ``Message.__setitem__`` is to append the + value to the list of headers, it is easy to create duplicate headers + without realizing it. This method allows certain headers to be limited + in the number of instances of that header that may be added to a + ``Message`` programmatically. (The limit is not observed by the parser, + which will faithfully produce as many headers as exist in the message + being parsed.) + + The default implementation returns ``None`` for all header names. + .. method:: header_source_parse(sourcelines) The email package calls this method with a list of strings, each string @@ -366,6 +385,12 @@ The class provides the following concrete implementations of the abstract methods of :class:`Policy`: + .. method:: header_max_count(name) + + Returns the value of the + :attr:`~email.headerregistry.BaseHeader.max_count` attribute of the + specialized class used to represent the header with the given name. + .. method:: header_source_parse(sourcelines) The implementation of this method is the same as that for the diff --git a/Lib/email/_policybase.py b/Lib/email/_policybase.py --- a/Lib/email/_policybase.py +++ b/Lib/email/_policybase.py @@ -194,6 +194,25 @@ """ obj.defects.append(defect) + def header_max_count(self, name): + """Return the maximum allowed number of headers named 'name'. + + Called when a header is added to a Message object. If the returned + value is not 0 or None, and there are already a number of headers with + the name 'name' equal to the value returned, a ValueError is raised. + + Because the default behavior of Message's __setitem__ is to append the + value to the list of headers, it is easy to create duplicate headers + without realizing it. This method allows certain headers to be limited + in the number of instances of that header that may be added to a + Message programmatically. (The limit is not observed by the parser, + which will faithfully produce as many headers as exist in the message + being parsed.) + + The default implementation returns None for all header names. + """ + return None + @abc.abstractmethod def header_source_parse(self, sourcelines): """Given a list of linesep terminated strings constituting the lines of diff --git a/Lib/email/message.py b/Lib/email/message.py --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -346,6 +346,16 @@ Note: this does not overwrite an existing header with the same field name. Use __delitem__() first to delete any existing headers. """ + max_count = self.policy.header_max_count(name) + if max_count: + lname = name.lower() + found = 0 + for k, v in self._headers: + if k.lower() == lname: + found += 1 + if found >= max_count: + raise ValueError("There may be at most {} {} headers " + "in a message".format(max_count, name)) self._headers.append(self.policy.header_store_parse(name, val)) def __delitem__(self, name): diff --git a/Lib/email/policy.py b/Lib/email/policy.py --- a/Lib/email/policy.py +++ b/Lib/email/policy.py @@ -69,6 +69,14 @@ object.__setattr__(self, 'header_factory', HeaderRegistry()) super().__init__(**kw) + def header_max_count(self, name): + """+ + The implementation for this class returns the max_count attribute from + the specialized header class that would be used to construct a header + of type 'name'. + """ + return self.header_factory[name].max_count + # The logic of the next three methods is chosen such that it is possible to # switch a Message object between a Compat32 policy and a policy derived # from this class and have the results stay consistent. This allows a -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 18:31:38 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 29 May 2012 18:31:38 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2310839=3A_add_new_test_fi?= =?utf8?q?le_that_was_omitted_from_checkin?= Message-ID: http://hg.python.org/cpython/rev/08857f4e9709 changeset: 77229:08857f4e9709 user: R David Murray date: Tue May 29 12:31:11 2012 -0400 summary: #10839: add new test file that was omitted from checkin files: Lib/test/test_email/test_message.py | 18 +++++++++++++++++ 1 files changed, 18 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_email/test_message.py b/Lib/test/test_email/test_message.py new file mode 100644 --- /dev/null +++ b/Lib/test/test_email/test_message.py @@ -0,0 +1,18 @@ +import unittest +from email import policy +from test.test_email import TestEmailBase + + +class Test(TestEmailBase): + + policy = policy.default + + def test_error_on_setitem_if_max_count_exceeded(self): + m = self._str_msg("") + m['To'] = 'abc at xyz' + with self.assertRaises(ValueError): + m['To'] = 'xyz at abc' + + +if __name__ == '__main__': + unittest.main() -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 18:52:11 2012 From: python-checkins at python.org (victor.stinner) Date: Tue, 29 May 2012 18:52:11 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314744=3A_Fix_compi?= =?utf8?q?lation_on_Windows?= Message-ID: http://hg.python.org/cpython/rev/6abab1a103a6 changeset: 77230:6abab1a103a6 user: Victor Stinner date: Tue May 29 18:51:10 2012 +0200 summary: Issue #14744: Fix compilation on Windows files: Objects/longobject.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Objects/longobject.c b/Objects/longobject.c --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -1687,8 +1687,8 @@ WRITE_DIGITS(Py_UCS2); } else { + Py_UCS4 *p; assert (kind == PyUnicode_4BYTE_KIND); - Py_UCS4 *p; WRITE_DIGITS(Py_UCS4); } #undef WRITE_DIGITS @@ -1845,8 +1845,8 @@ WRITE_DIGITS(Py_UCS2); } else { + Py_UCS4 *p; assert (kind == PyUnicode_4BYTE_KIND); - Py_UCS4 *p; WRITE_DIGITS(Py_UCS4); } #undef WRITE_DIGITS -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 18:54:01 2012 From: python-checkins at python.org (victor.stinner) Date: Tue, 29 May 2012 18:54:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314744=3A_Fix_compi?= =?utf8?q?lation_on_Windows_=28part_2=29?= Message-ID: http://hg.python.org/cpython/rev/df0144f68d76 changeset: 77231:df0144f68d76 user: Victor Stinner date: Tue May 29 18:53:56 2012 +0200 summary: Issue #14744: Fix compilation on Windows (part 2) files: Objects/unicodeobject.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13229,7 +13229,7 @@ if (writer) { if (_PyUnicodeWriter_Prepare(writer, len, 127) == -1) return -1; - memcpy(writer->data + writer->pos * writer->kind, + memcpy((char*)writer->data + writer->pos * writer->kind, p, len); writer->pos += len; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 18:55:29 2012 From: python-checkins at python.org (r.david.murray) Date: Tue, 29 May 2012 18:55:29 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314796=3A_improve_calenda?= =?utf8?q?r_test_coverage=2E?= Message-ID: http://hg.python.org/cpython/rev/98bc9e357f74 changeset: 77232:98bc9e357f74 user: R David Murray date: Tue May 29 12:55:05 2012 -0400 summary: #14796: improve calendar test coverage. Patch by Oleg Plakhotnyuk. files: Lib/test/test_calendar.py | 220 ++++++++++++++++++++++++- 1 files changed, 211 insertions(+), 9 deletions(-) diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -5,6 +5,17 @@ from test.script_helper import assert_python_ok import time import locale +import sys + +result_2004_01_text = """ + January 2004 +Mo Tu We Th Fr Sa Su + 1 2 3 4 + 5 6 7 8 9 10 11 +12 13 14 15 16 17 18 +19 20 21 22 23 24 25 +26 27 28 29 30 31 +""" result_2004_text = """ 2004 @@ -45,11 +56,11 @@ """ result_2004_html = """ - + - + Calendar for 2004 @@ -169,6 +180,135 @@ """ +result_2004_days = [ + [[[0, 0, 0, 1, 2, 3, 4], + [5, 6, 7, 8, 9, 10, 11], + [12, 13, 14, 15, 16, 17, 18], + [19, 20, 21, 22, 23, 24, 25], + [26, 27, 28, 29, 30, 31, 0]], + [[0, 0, 0, 0, 0, 0, 1], + [2, 3, 4, 5, 6, 7, 8], + [9, 10, 11, 12, 13, 14, 15], + [16, 17, 18, 19, 20, 21, 22], + [23, 24, 25, 26, 27, 28, 29]], + [[1, 2, 3, 4, 5, 6, 7], + [8, 9, 10, 11, 12, 13, 14], + [15, 16, 17, 18, 19, 20, 21], + [22, 23, 24, 25, 26, 27, 28], + [29, 30, 31, 0, 0, 0, 0]]], + [[[0, 0, 0, 1, 2, 3, 4], + [5, 6, 7, 8, 9, 10, 11], + [12, 13, 14, 15, 16, 17, 18], + [19, 20, 21, 22, 23, 24, 25], + [26, 27, 28, 29, 30, 0, 0]], + [[0, 0, 0, 0, 0, 1, 2], + [3, 4, 5, 6, 7, 8, 9], + [10, 11, 12, 13, 14, 15, 16], + [17, 18, 19, 20, 21, 22, 23], + [24, 25, 26, 27, 28, 29, 30], + [31, 0, 0, 0, 0, 0, 0]], + [[0, 1, 2, 3, 4, 5, 6], + [7, 8, 9, 10, 11, 12, 13], + [14, 15, 16, 17, 18, 19, 20], + [21, 22, 23, 24, 25, 26, 27], + [28, 29, 30, 0, 0, 0, 0]]], + [[[0, 0, 0, 1, 2, 3, 4], + [5, 6, 7, 8, 9, 10, 11], + [12, 13, 14, 15, 16, 17, 18], + [19, 20, 21, 22, 23, 24, 25], + [26, 27, 28, 29, 30, 31, 0]], + [[0, 0, 0, 0, 0, 0, 1], + [2, 3, 4, 5, 6, 7, 8], + [9, 10, 11, 12, 13, 14, 15], + [16, 17, 18, 19, 20, 21, 22], + [23, 24, 25, 26, 27, 28, 29], + [30, 31, 0, 0, 0, 0, 0]], + [[0, 0, 1, 2, 3, 4, 5], + [6, 7, 8, 9, 10, 11, 12], + [13, 14, 15, 16, 17, 18, 19], + [20, 21, 22, 23, 24, 25, 26], + [27, 28, 29, 30, 0, 0, 0]]], + [[[0, 0, 0, 0, 1, 2, 3], + [4, 5, 6, 7, 8, 9, 10], + [11, 12, 13, 14, 15, 16, 17], + [18, 19, 20, 21, 22, 23, 24], + [25, 26, 27, 28, 29, 30, 31]], + [[1, 2, 3, 4, 5, 6, 7], + [8, 9, 10, 11, 12, 13, 14], + [15, 16, 17, 18, 19, 20, 21], + [22, 23, 24, 25, 26, 27, 28], + [29, 30, 0, 0, 0, 0, 0]], + [[0, 0, 1, 2, 3, 4, 5], + [6, 7, 8, 9, 10, 11, 12], + [13, 14, 15, 16, 17, 18, 19], + [20, 21, 22, 23, 24, 25, 26], + [27, 28, 29, 30, 31, 0, 0]]] +] + +result_2004_dates = \ + [[['12/29/03 12/30/03 12/31/03 01/01/04 01/02/04 01/03/04 01/04/04', + '01/05/04 01/06/04 01/07/04 01/08/04 01/09/04 01/10/04 01/11/04', + '01/12/04 01/13/04 01/14/04 01/15/04 01/16/04 01/17/04 01/18/04', + '01/19/04 01/20/04 01/21/04 01/22/04 01/23/04 01/24/04 01/25/04', + '01/26/04 01/27/04 01/28/04 01/29/04 01/30/04 01/31/04 02/01/04'], + ['01/26/04 01/27/04 01/28/04 01/29/04 01/30/04 01/31/04 02/01/04', + '02/02/04 02/03/04 02/04/04 02/05/04 02/06/04 02/07/04 02/08/04', + '02/09/04 02/10/04 02/11/04 02/12/04 02/13/04 02/14/04 02/15/04', + '02/16/04 02/17/04 02/18/04 02/19/04 02/20/04 02/21/04 02/22/04', + '02/23/04 02/24/04 02/25/04 02/26/04 02/27/04 02/28/04 02/29/04'], + ['03/01/04 03/02/04 03/03/04 03/04/04 03/05/04 03/06/04 03/07/04', + '03/08/04 03/09/04 03/10/04 03/11/04 03/12/04 03/13/04 03/14/04', + '03/15/04 03/16/04 03/17/04 03/18/04 03/19/04 03/20/04 03/21/04', + '03/22/04 03/23/04 03/24/04 03/25/04 03/26/04 03/27/04 03/28/04', + '03/29/04 03/30/04 03/31/04 04/01/04 04/02/04 04/03/04 04/04/04']], + [['03/29/04 03/30/04 03/31/04 04/01/04 04/02/04 04/03/04 04/04/04', + '04/05/04 04/06/04 04/07/04 04/08/04 04/09/04 04/10/04 04/11/04', + '04/12/04 04/13/04 04/14/04 04/15/04 04/16/04 04/17/04 04/18/04', + '04/19/04 04/20/04 04/21/04 04/22/04 04/23/04 04/24/04 04/25/04', + '04/26/04 04/27/04 04/28/04 04/29/04 04/30/04 05/01/04 05/02/04'], + ['04/26/04 04/27/04 04/28/04 04/29/04 04/30/04 05/01/04 05/02/04', + '05/03/04 05/04/04 05/05/04 05/06/04 05/07/04 05/08/04 05/09/04', + '05/10/04 05/11/04 05/12/04 05/13/04 05/14/04 05/15/04 05/16/04', + '05/17/04 05/18/04 05/19/04 05/20/04 05/21/04 05/22/04 05/23/04', + '05/24/04 05/25/04 05/26/04 05/27/04 05/28/04 05/29/04 05/30/04', + '05/31/04 06/01/04 06/02/04 06/03/04 06/04/04 06/05/04 06/06/04'], + ['05/31/04 06/01/04 06/02/04 06/03/04 06/04/04 06/05/04 06/06/04', + '06/07/04 06/08/04 06/09/04 06/10/04 06/11/04 06/12/04 06/13/04', + '06/14/04 06/15/04 06/16/04 06/17/04 06/18/04 06/19/04 06/20/04', + '06/21/04 06/22/04 06/23/04 06/24/04 06/25/04 06/26/04 06/27/04', + '06/28/04 06/29/04 06/30/04 07/01/04 07/02/04 07/03/04 07/04/04']], + [['06/28/04 06/29/04 06/30/04 07/01/04 07/02/04 07/03/04 07/04/04', + '07/05/04 07/06/04 07/07/04 07/08/04 07/09/04 07/10/04 07/11/04', + '07/12/04 07/13/04 07/14/04 07/15/04 07/16/04 07/17/04 07/18/04', + '07/19/04 07/20/04 07/21/04 07/22/04 07/23/04 07/24/04 07/25/04', + '07/26/04 07/27/04 07/28/04 07/29/04 07/30/04 07/31/04 08/01/04'], + ['07/26/04 07/27/04 07/28/04 07/29/04 07/30/04 07/31/04 08/01/04', + '08/02/04 08/03/04 08/04/04 08/05/04 08/06/04 08/07/04 08/08/04', + '08/09/04 08/10/04 08/11/04 08/12/04 08/13/04 08/14/04 08/15/04', + '08/16/04 08/17/04 08/18/04 08/19/04 08/20/04 08/21/04 08/22/04', + '08/23/04 08/24/04 08/25/04 08/26/04 08/27/04 08/28/04 08/29/04', + '08/30/04 08/31/04 09/01/04 09/02/04 09/03/04 09/04/04 09/05/04'], + ['08/30/04 08/31/04 09/01/04 09/02/04 09/03/04 09/04/04 09/05/04', + '09/06/04 09/07/04 09/08/04 09/09/04 09/10/04 09/11/04 09/12/04', + '09/13/04 09/14/04 09/15/04 09/16/04 09/17/04 09/18/04 09/19/04', + '09/20/04 09/21/04 09/22/04 09/23/04 09/24/04 09/25/04 09/26/04', + '09/27/04 09/28/04 09/29/04 09/30/04 10/01/04 10/02/04 10/03/04']], + [['09/27/04 09/28/04 09/29/04 09/30/04 10/01/04 10/02/04 10/03/04', + '10/04/04 10/05/04 10/06/04 10/07/04 10/08/04 10/09/04 10/10/04', + '10/11/04 10/12/04 10/13/04 10/14/04 10/15/04 10/16/04 10/17/04', + '10/18/04 10/19/04 10/20/04 10/21/04 10/22/04 10/23/04 10/24/04', + '10/25/04 10/26/04 10/27/04 10/28/04 10/29/04 10/30/04 10/31/04'], + ['11/01/04 11/02/04 11/03/04 11/04/04 11/05/04 11/06/04 11/07/04', + '11/08/04 11/09/04 11/10/04 11/11/04 11/12/04 11/13/04 11/14/04', + '11/15/04 11/16/04 11/17/04 11/18/04 11/19/04 11/20/04 11/21/04', + '11/22/04 11/23/04 11/24/04 11/25/04 11/26/04 11/27/04 11/28/04', + '11/29/04 11/30/04 12/01/04 12/02/04 12/03/04 12/04/04 12/05/04'], + ['11/29/04 11/30/04 12/01/04 12/02/04 12/03/04 12/04/04 12/05/04', + '12/06/04 12/07/04 12/08/04 12/09/04 12/10/04 12/11/04 12/12/04', + '12/13/04 12/14/04 12/15/04 12/16/04 12/17/04 12/18/04 12/19/04', + '12/20/04 12/21/04 12/22/04 12/23/04 12/24/04 12/25/04 12/26/04', + '12/27/04 12/28/04 12/29/04 12/30/04 12/31/04 01/01/05 01/02/05']]] + class OutputTestCase(unittest.TestCase): def normalize_calendar(self, s): @@ -183,6 +323,13 @@ lines.append(line) return lines + def check_htmlcalendar_encoding(self, req, res): + cal = calendar.HTMLCalendar() + self.assertEqual( + cal.formatyearpage(2004, encoding=req).strip(b' \t\n'), + (result_2004_html % {'e': res}).strip(' \t\n').encode(res) + ) + def test_output(self): self.assertEqual( self.normalize_calendar(calendar.calendar(2004)), @@ -195,12 +342,59 @@ result_2004_text.strip() ) - def test_output_htmlcalendar(self): - encoding = 'ascii' - cal = calendar.HTMLCalendar() + def test_output_htmlcalendar_encoding_ascii(self): + self.check_htmlcalendar_encoding('ascii', 'ascii') + + def test_output_htmlcalendar_encoding_utf8(self): + self.check_htmlcalendar_encoding('utf-8', 'utf-8') + + def test_output_htmlcalendar_encoding_default(self): + self.check_htmlcalendar_encoding(None, sys.getdefaultencoding()) + + def test_yeardatescalendar(self): + def shrink(cal): + return [[[' '.join((d.strftime('%D') + for d in z)) for z in y] for y in x] for x in cal] self.assertEqual( - cal.formatyearpage(2004, encoding=encoding).strip(b' \t\n'), - result_2004_html.strip(' \t\n').encode(encoding) + shrink(calendar.Calendar().yeardatescalendar(2004)), + result_2004_dates + ) + + def test_yeardayscalendar(self): + self.assertEqual( + calendar.Calendar().yeardayscalendar(2004), + result_2004_days + ) + + def test_formatweekheader_short(self): + self.assertEqual( + calendar.TextCalendar().formatweekheader(2), + 'Mo Tu We Th Fr Sa Su' + ) + + def test_formatweekheader_long(self): + self.assertEqual( + calendar.TextCalendar().formatweekheader(9), + ' Monday Tuesday Wednesday Thursday ' + ' Friday Saturday Sunday ' + ) + + def test_formatmonth(self): + self.assertEqual( + calendar.TextCalendar().formatmonth(2004, 1).strip(), + result_2004_01_text.strip() + ) + + def test_formatmonthname_with_year(self): + self.assertEqual( + calendar.HTMLCalendar().formatmonthname(2004, 1, withyear=True), + 'January 2004' + ) + + def test_formatmonthname_without_year(self): + self.assertEqual( + calendar.HTMLCalendar().formatmonthname(2004, 1, withyear=False), + 'January' ) @@ -226,7 +420,11 @@ self.assertEqual(calendar.firstweekday(), calendar.MONDAY) calendar.setfirstweekday(orig) - def test_enumerateweekdays(self): + def test_illegal_weekday_reported(self): + with self.assertRaisesRegex(calendar.IllegalWeekdayError, '123'): + calendar.setfirstweekday(123) + + def test_enumerate_weekdays(self): self.assertRaises(IndexError, calendar.day_abbr.__getitem__, -10) self.assertRaises(IndexError, calendar.day_name.__getitem__, 10) self.assertEqual(len([d for d in calendar.day_abbr]), 7) @@ -252,7 +450,7 @@ # verify it "acts like a sequence" in two forms of iteration self.assertEqual(value[::-1], list(reversed(value))) - def test_localecalendars(self): + def test_locale_calendars(self): # ensure that Locale{Text,HTML}Calendar resets the locale properly # (it is still not thread-safe though) old_october = calendar.TextCalendar().formatmonthname(2010, 10, 10) @@ -431,6 +629,10 @@ with self.assertRaises(calendar.IllegalMonthError): calendar.monthrange(2004, 13) + def test_illegal_month_reported(self): + with self.assertRaisesRegex(calendar.IllegalMonthError, '65'): + calendar.monthrange(2004, 65) + class LeapdaysTestCase(unittest.TestCase): def test_no_range(self): # test when no range i.e. two identical years as args -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 19:52:17 2012 From: python-checkins at python.org (ned.deily) Date: Tue, 29 May 2012 19:52:17 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEwOTk3?= =?utf8?q?=3A_Prevent_a_duplicate_entry_in_IDLE=27s_=22Recent_Files=22_men?= =?utf8?q?u=2E?= Message-ID: http://hg.python.org/cpython/rev/3108331c88ec changeset: 77233:3108331c88ec branch: 2.7 parent: 77212:6554ebbeb2f3 user: Ned Deily date: Tue May 29 10:42:34 2012 -0700 summary: Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. files: Lib/idlelib/EditorWindow.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -856,7 +856,7 @@ # for each edit window instance, construct the recent files menu for instance in self.top.instance_dict.keys(): menu = instance.recent_files_menu - menu.delete(1, END) # clear, and rebuild: + menu.delete(0, END) # clear, and rebuild: for i, file_name in enumerate(rf_list): file_name = file_name.rstrip() # zap \n # make unicode string to display non-ASCII chars correctly diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,8 @@ Library ------- +- Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. + - Issue12510: Attempting to get invalid tooltip no longer closes Idle. Original patch by Roger Serwy. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 19:52:18 2012 From: python-checkins at python.org (ned.deily) Date: Tue, 29 May 2012 19:52:18 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEwOTk3?= =?utf8?q?=3A_Prevent_a_duplicate_entry_in_IDLE=27s_=22Recent_Files=22_men?= =?utf8?q?u=2E?= Message-ID: http://hg.python.org/cpython/rev/64a7fae544a6 changeset: 77234:64a7fae544a6 branch: 3.2 parent: 77210:47e6217d0e84 user: Ned Deily date: Tue May 29 10:43:36 2012 -0700 summary: Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. files: Lib/idlelib/EditorWindow.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -860,7 +860,7 @@ # for each edit window instance, construct the recent files menu for instance in self.top.instance_dict: menu = instance.recent_files_menu - menu.delete(1, END) # clear, and rebuild: + menu.delete(0, END) # clear, and rebuild: for i, file_name in enumerate(rf_list): file_name = file_name.rstrip() # zap \n # make unicode string to display non-ASCII chars correctly diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -70,6 +70,8 @@ Library ------- +- Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. + - Issue #14443: Tell rpmbuild to use the correct version of Python in bdist_rpm. Initial patch by Ross Lagerwall. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 19:52:19 2012 From: python-checkins at python.org (ned.deily) Date: Tue, 29 May 2012 19:52:19 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2310997=3A_merge_from_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/cb7421cdaec4 changeset: 77235:cb7421cdaec4 parent: 77232:98bc9e357f74 parent: 77234:64a7fae544a6 user: Ned Deily date: Tue May 29 10:51:38 2012 -0700 summary: Issue #10997: merge from 3.2 files: Lib/idlelib/EditorWindow.py | 2 +- Misc/NEWS | 2 ++ 2 files changed, 3 insertions(+), 1 deletions(-) diff --git a/Lib/idlelib/EditorWindow.py b/Lib/idlelib/EditorWindow.py --- a/Lib/idlelib/EditorWindow.py +++ b/Lib/idlelib/EditorWindow.py @@ -860,7 +860,7 @@ # for each edit window instance, construct the recent files menu for instance in self.top.instance_dict: menu = instance.recent_files_menu - menu.delete(1, END) # clear, and rebuild: + menu.delete(0, END) # clear, and rebuild: for i, file_name in enumerate(rf_list): file_name = file_name.rstrip() # zap \n # make unicode string to display non-ASCII chars correctly diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,8 @@ Library ------- +- Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. + - Issue #14443: Tell rpmbuild to use the correct version of Python in bdist_rpm. Initial patch by Ross Lagerwall. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 23:30:01 2012 From: python-checkins at python.org (vinay.sajip) Date: Tue, 29 May 2012 23:30:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Updated_test_to_reflect_ren?= =?utf8?q?amed_file=2E?= Message-ID: http://hg.python.org/cpython/rev/029307ea07f6 changeset: 77236:029307ea07f6 user: Vinay Sajip date: Tue May 29 22:29:50 2012 +0100 summary: Updated test to reflect renamed file. files: Lib/test/test_venv.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -21,7 +21,7 @@ self.env_dir = tempfile.mkdtemp() if os.name == 'nt': self.bindir = 'Scripts' - self.ps3name = 'pysetup3-script.py' + self.ps3name = 'pysetup3.py' self.lib = ('Lib',) self.include = 'Include' else: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Tue May 29 23:48:20 2012 From: python-checkins at python.org (vinay.sajip) Date: Tue, 29 May 2012 23:48:20 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Reapplied_skip_logic_for_te?= =?utf8?q?st_on_Windows=2C_which_appears_to_have_been_lost?= Message-ID: http://hg.python.org/cpython/rev/4dd878993b88 changeset: 77237:4dd878993b88 user: Vinay Sajip date: Tue May 29 22:48:10 2012 +0100 summary: Reapplied skip logic for test on Windows, which appears to have been lost during a merge. files: Lib/test/test_logging.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -582,6 +582,7 @@ self.assertFalse(h.shouldFlush(r)) h.close() + @unittest.skipIf(os.name == 'nt', 'WatchedFileHandler not appropriate for Windows.') @unittest.skipUnless(threading, 'Threading required for this test.') def test_race(self): # Issue #14632 refers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 01:44:40 2012 From: python-checkins at python.org (brian.curtin) Date: Wed, 30 May 2012 01:44:40 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogRml4ICMxNDk0My4g?= =?utf8?q?Update_the_proper_default_value_and_list_the_proper_argument_nam?= =?utf8?q?es?= Message-ID: http://hg.python.org/cpython/rev/fcb6c4a4ac0e changeset: 77238:fcb6c4a4ac0e branch: 3.2 parent: 77234:64a7fae544a6 user: Brian Curtin date: Tue May 29 18:34:45 2012 -0500 summary: Fix #14943. Update the proper default value and list the proper argument names in the explanation. files: Doc/library/winreg.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -243,7 +243,7 @@ specified in *file_name* is relative to the remote computer. -.. function:: OpenKey(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) +.. function:: OpenKey(key, sub_key, reserved=0, access=KEY_READ) Opens the specified key, returning a :ref:`handle object `. @@ -252,9 +252,9 @@ *sub_key* is a string that identifies the sub_key to open. - *res* is a reserved integer, and must be zero. The default is zero. + *reserved* is a reserved integer, and must be zero. The default is zero. - *sam* is an integer that specifies an access mask that describes the desired + *access* is an integer that specifies an access mask that describes the desired security access for the key. Default is :const:`KEY_READ`. See :ref:`Access Rights ` for other allowed values. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 01:44:41 2012 From: python-checkins at python.org (brian.curtin) Date: Wed, 30 May 2012 01:44:41 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Fix_=2314943=2E_Merge_3=2E2?= Message-ID: http://hg.python.org/cpython/rev/29e0f08ef065 changeset: 77239:29e0f08ef065 parent: 77237:4dd878993b88 parent: 77238:fcb6c4a4ac0e user: Brian Curtin date: Tue May 29 18:36:40 2012 -0500 summary: Fix #14943. Merge 3.2 files: Doc/library/winreg.rst | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst --- a/Doc/library/winreg.rst +++ b/Doc/library/winreg.rst @@ -271,7 +271,7 @@ specified in *file_name* is relative to the remote computer. -.. function:: OpenKey(key, sub_key, reserved=0, access=KEY_ALL_ACCESS) +.. function:: OpenKey(key, sub_key, reserved=0, access=KEY_READ) Opens the specified key, returning a :ref:`handle object `. @@ -280,9 +280,9 @@ *sub_key* is a string that identifies the sub_key to open. - *res* is a reserved integer, and must be zero. The default is zero. + *reserved* is a reserved integer, and must be zero. The default is zero. - *sam* is an integer that specifies an access mask that describes the desired + *access* is an integer that specifies an access mask that describes the desired security access for the key. Default is :const:`KEY_READ`. See :ref:`Access Rights ` for other allowed values. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 01:44:46 2012 From: python-checkins at python.org (brian.curtin) Date: Wed, 30 May 2012 01:44:46 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_news_item_f?= =?utf8?q?or_=2314943?= Message-ID: http://hg.python.org/cpython/rev/8ec62c9eea34 changeset: 77240:8ec62c9eea34 branch: 3.2 parent: 77238:fcb6c4a4ac0e user: Brian Curtin date: Tue May 29 18:41:30 2012 -0500 summary: Add news item for #14943 files: Misc/NEWS | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -307,6 +307,9 @@ - Issue #8799: Fix and improve the threading.Condition documentation. +- Issue #14943: Correct a default argument value for winreg.OpenKey + and correctly list the argument names in the function's explanation. + Documentation ------------- -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 01:44:47 2012 From: python-checkins at python.org (brian.curtin) Date: Wed, 30 May 2012 01:44:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Merge_3=2E2_news_item?= Message-ID: http://hg.python.org/cpython/rev/f1b5fc67524a changeset: 77241:f1b5fc67524a parent: 77239:29e0f08ef065 parent: 77240:8ec62c9eea34 user: Brian Curtin date: Tue May 29 18:44:17 2012 -0500 summary: Merge 3.2 news item files: Misc/NEWS | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -260,6 +260,8 @@ - Issue #14588: The language reference now accurately documents the Python 3 class definition process. Patch by Nick Coghlan. +- Issue #14943: Correct a default argument value for winreg.OpenKey + and correctly list the argument names in the function's explanation. What's New in Python 3.3.0 Alpha 3? -- Repository URL: http://hg.python.org/cpython From solipsis at pitrou.net Wed May 30 05:52:52 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Wed, 30 May 2012 05:52:52 +0200 Subject: [Python-checkins] Daily reference leaks (f1b5fc67524a): sum=465 Message-ID: results for f1b5fc67524a on branch "default" -------------------------------------------- test_smtplib leaked [154, 154, 154] references, sum=462 test_super leaked [1, 1, 1] references, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogB2sPDT', '-x'] From python-checkins at python.org Wed May 30 07:56:52 2012 From: python-checkins at python.org (ned.deily) Date: Wed, 30 May 2012 07:56:52 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314958=3A_Change_ID?= =?utf8?q?LE_systax_highlighting_to_recognize_all_string_and?= Message-ID: http://hg.python.org/cpython/rev/94a5bf416e50 changeset: 77242:94a5bf416e50 user: Ned Deily date: Tue May 29 22:55:43 2012 -0700 summary: Issue #14958: Change IDLE systax highlighting to recognize all string and byte literals supported in Python 3.3. files: Lib/idlelib/ColorDelegator.py | 9 +++++---- Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/Lib/idlelib/ColorDelegator.py b/Lib/idlelib/ColorDelegator.py --- a/Lib/idlelib/ColorDelegator.py +++ b/Lib/idlelib/ColorDelegator.py @@ -21,10 +21,11 @@ # 1st 'file' colorized normal, 2nd as builtin, 3rd as string builtin = r"([^.'\"\\#]\b|^)" + any("BUILTIN", builtinlist) + r"\b" comment = any("COMMENT", [r"#[^\n]*"]) - sqstring = r"(\b[rRbB])?'[^'\\\n]*(\\.[^'\\\n]*)*'?" - dqstring = r'(\b[rRbB])?"[^"\\\n]*(\\.[^"\\\n]*)*"?' - sq3string = r"(\b[rRbB])?'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" - dq3string = r'(\b[rRbB])?"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' + stringprefix = r"(\br|u|ur|R|U|UR|Ur|uR|b|B|br|Br|bR|BR|rb|rB|Rb|RB)?" + sqstring = stringprefix + r"'[^'\\\n]*(\\.[^'\\\n]*)*'?" + dqstring = stringprefix + r'"[^"\\\n]*(\\.[^"\\\n]*)*"?' + sq3string = stringprefix + r"'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?" + dq3string = stringprefix + r'"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?' string = any("STRING", [sq3string, dq3string, sqstring, dqstring]) return kw + "|" + builtin + "|" + comment + "|" + string +\ "|" + any("SYNC", [r"\n"]) diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #14958: Change IDLE systax highlighting to recognize all string and + byte literals supported in Python 3.3. + - Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. - Issue #14443: Tell rpmbuild to use the correct version of Python in -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 12:44:01 2012 From: python-checkins at python.org (nick.coghlan) Date: Wed, 30 May 2012 12:44:01 +0200 Subject: [Python-checkins] =?utf8?q?peps=3A_Flag_PEP_2_as_obsolete_with_a_?= =?utf8?q?reference_to_the_relevant_section_of_the?= Message-ID: http://hg.python.org/peps/rev/26c61cd277ed changeset: 4443:26c61cd277ed user: Nick Coghlan date: Wed May 30 20:43:49 2012 +1000 summary: Flag PEP 2 as obsolete with a reference to the relevant section of the developer's guide files: pep-0002.txt | 13 +++++++++++-- 1 files changed, 11 insertions(+), 2 deletions(-) diff --git a/pep-0002.txt b/pep-0002.txt --- a/pep-0002.txt +++ b/pep-0002.txt @@ -2,13 +2,18 @@ Title: Procedure for Adding New Modules Version: $Revision$ Last-Modified: $Date$ -Author: faassen at infrae.com (Martijn Faassen) -Status: Active +Author: Martijn Faassen +Status: Final Type: Process Created: 07-Jul-2001 Post-History: 07-Jul-2001, 09-Mar-2002 +PEP Replacement + + This PEP has been superseded by the updated material in the Python + Developer's Guide [1]. + Introduction The Python Standard Library contributes significantly to Python's @@ -185,6 +190,10 @@ Relationship to the mythical Catalog? +References + + [1] Adding to the Stdlib + http://docs.python.org/devguide/stdlibchanges.html Copyright -- Repository URL: http://hg.python.org/peps From python-checkins at python.org Wed May 30 13:31:45 2012 From: python-checkins at python.org (victor.stinner) Date: Wed, 30 May 2012 13:31:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314690=3A_Use_monot?= =?utf8?q?onic_clock_instead_of_system_clock_in_the_sched=2C?= Message-ID: http://hg.python.org/cpython/rev/1345cf58738d changeset: 77243:1345cf58738d user: Victor Stinner date: Wed May 30 13:30:32 2012 +0200 summary: Close #14690: Use monotonic clock instead of system clock in the sched, subprocess and trace modules. files: Lib/sched.py | 6 +++++- Lib/subprocess.py | 12 ++++++++---- Lib/trace.py | 10 +++++++--- Misc/NEWS | 3 +++ 4 files changed, 23 insertions(+), 8 deletions(-) diff --git a/Lib/sched.py b/Lib/sched.py --- a/Lib/sched.py +++ b/Lib/sched.py @@ -35,6 +35,10 @@ import threading except ImportError: import dummy_threading as threading +try: + from time import monotonic as _time +except ImportError: + from time import time as _time __all__ = ["scheduler"] @@ -48,7 +52,7 @@ class scheduler: - def __init__(self, timefunc=time.time, delayfunc=time.sleep): + def __init__(self, timefunc=_time, delayfunc=time.sleep): """Initialize a new instance, passing the time and delay functions""" self._queue = [] diff --git a/Lib/subprocess.py b/Lib/subprocess.py --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -349,6 +349,10 @@ import builtins import warnings import errno +try: + from time import monotonic as _time +except ImportError: + from time import time as _time # Exception classes used by this module. class SubprocessError(Exception): pass @@ -894,7 +898,7 @@ self.wait() else: if timeout is not None: - endtime = time.time() + timeout + endtime = _time() + timeout else: endtime = None @@ -917,14 +921,14 @@ if endtime is None: return None else: - return endtime - time.time() + return endtime - _time() def _check_timeout(self, endtime, orig_timeout): """Convenience for checking if a timeout has expired.""" if endtime is None: return - if time.time() > endtime: + if _time() > endtime: raise TimeoutExpired(self.args, orig_timeout) @@ -1471,7 +1475,7 @@ # printing. if endtime is not None or timeout is not None: if endtime is None: - endtime = time.time() + timeout + endtime = _time() + timeout elif timeout is None: timeout = self._remaining_time(endtime) diff --git a/Lib/trace.py b/Lib/trace.py --- a/Lib/trace.py +++ b/Lib/trace.py @@ -61,6 +61,10 @@ import dis import pickle from warnings import warn as _warn +try: + from time import monotonic as _time +except ImportError: + from time import time as _time try: import threading @@ -476,7 +480,7 @@ self._caller_cache = {} self.start_time = None if timing: - self.start_time = time.time() + self.start_time = _time() if countcallers: self.globaltrace = self.globaltrace_trackcallers elif countfuncs: @@ -614,7 +618,7 @@ self.counts[key] = self.counts.get(key, 0) + 1 if self.start_time: - print('%.2f' % (time.time() - self.start_time), end=' ') + print('%.2f' % (_time() - self.start_time), end=' ') bname = os.path.basename(filename) print("%s(%d): %s" % (bname, lineno, linecache.getline(filename, lineno)), end='') @@ -627,7 +631,7 @@ lineno = frame.f_lineno if self.start_time: - print('%.2f' % (time.time() - self.start_time), end=' ') + print('%.2f' % (_time() - self.start_time), end=' ') bname = os.path.basename(filename) print("%s(%d): %s" % (bname, lineno, linecache.getline(filename, lineno)), end='') diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -60,6 +60,9 @@ Library ------- +- Issue #14690: Use monotonic clock instead of system clock in the sched, + subprocess and trace modules. + - Issue #14958: Change IDLE systax highlighting to recognize all string and byte literals supported in Python 3.3. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 14:11:01 2012 From: python-checkins at python.org (r.david.murray) Date: Wed, 30 May 2012 14:11:01 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_=2314796=3A_fix_failure_of_?= =?utf8?q?new_calendar_test_on_windows=2E?= Message-ID: http://hg.python.org/cpython/rev/d3321c010af5 changeset: 77244:d3321c010af5 user: R David Murray date: Wed May 30 08:10:54 2012 -0400 summary: #14796: fix failure of new calendar test on windows. files: Lib/test/test_calendar.py | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -353,8 +353,9 @@ def test_yeardatescalendar(self): def shrink(cal): - return [[[' '.join((d.strftime('%D') - for d in z)) for z in y] for y in x] for x in cal] + return [[[' '.join('{:02d}/{:02d}/{}'.format( + d.month, d.day, str(d.year)[-2:]) for d in z) + for z in y] for y in x] for x in cal] self.assertEqual( shrink(calendar.Calendar().yeardatescalendar(2004)), result_2004_dates -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 14:17:44 2012 From: python-checkins at python.org (nick.coghlan) Date: Wed, 30 May 2012 14:17:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314947=3A_add_missi?= =?utf8?q?ng_cross-reference_to_Language_Definition_from_the_new?= Message-ID: http://hg.python.org/cpython/rev/a5e621c8dd44 changeset: 77245:a5e621c8dd44 user: Nick Coghlan date: Wed May 30 22:17:30 2012 +1000 summary: Close #14947: add missing cross-reference to Language Definition from the new dynamic type creation functions. Also cleaned up the general wording of the docs files: Doc/library/types.rst | 31 ++++++++++++++++++------------- 1 files changed, 18 insertions(+), 13 deletions(-) diff --git a/Doc/library/types.rst b/Doc/library/types.rst --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -23,12 +23,15 @@ Creates a class object dynamically using the appropriate metaclass. - The arguments are the components that make up a class definition: the - class name, the base classes (in order), the keyword arguments (such as - ``metaclass``) and the callback function to populate the class namespace. + The first three arguments are the components that make up a class + definition header: the class name, the base classes (in order), the + keyword arguments (such as ``metaclass``). - The *exec_body* callback should accept the class namespace as its sole - argument and update the namespace directly with the class contents. + The *exec_body* argument is a callback that is used to populate the + freshly created class namespace. It should accept the class namespace + as its sole argument and update the namespace directly with the class + contents. If no callback is provided, it has the same effect as passing + in ``lambda ns: ns``. .. versionadded:: 3.3 @@ -36,22 +39,24 @@ Calculates the appropriate metaclass and creates the class namespace. - The arguments are the components that make up a class definition: the - class name, the base classes (in order) and the keyword arguments (such as - ``metaclass``). + The arguments are the components that make up a class definition header: + the class name, the base classes (in order) and the keyword arguments + (such as ``metaclass``). The return value is a 3-tuple: ``metaclass, namespace, kwds`` - *metaclass* is the appropriate metaclass - *namespace* is the prepared class namespace - *kwds* is an updated copy of the passed in *kwds* argument with any - ``'metaclass'`` entry removed. If no *kwds* argument is passed in, this - will be an empty dict. + *metaclass* is the appropriate metaclass, *namespace* is the + prepared class namespace and *kwds* is an updated copy of the passed + in *kwds* argument with any ``'metaclass'`` entry removed. If no *kwds* + argument is passed in, this will be an empty dict. .. versionadded:: 3.3 .. seealso:: + :ref:`metaclasses` + Full details of the class creation process supported by these functions + :pep:`3115` - Metaclasses in Python 3000 Introduced the ``__prepare__`` namespace hook -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 16:59:10 2012 From: python-checkins at python.org (eli.bendersky) Date: Wed, 30 May 2012 16:59:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314007=3A_implement?= =?utf8?q?ed_the_=27element=5Ffactory=27_feature_of_TreeBuilder_in?= Message-ID: http://hg.python.org/cpython/rev/20b8f0ee3d64 changeset: 77246:20b8f0ee3d64 user: Eli Bendersky date: Wed May 30 17:57:50 2012 +0300 summary: Issue #14007: implemented the 'element_factory' feature of TreeBuilder in _elementtree, with a test. files: Doc/library/xml.etree.elementtree.rst | 6 +- Lib/test/test_xml_etree.py | 18 +- Modules/_elementtree.c | 105 +++++++++---- 3 files changed, 91 insertions(+), 38 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -716,9 +716,9 @@ Generic element structure builder. This builder converts a sequence of start, data, and end method calls to a well-formed element structure. You can use this class to build an element structure using a custom XML parser, - or a parser for some other XML-like format. The *element_factory* is called - to create new :class:`Element` instances when given. - + or a parser for some other XML-like format. *element_factory*, when given, + must be a callable accepting two positional arguments: a tag and + a dict of attributes. It is expected to return a new element instance. .. method:: close() diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1959,6 +1959,8 @@ ' "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">' 'text') + sample2 = '''sometext''' + def test_dummy_builder(self): class BaseDummyBuilder: def close(self): @@ -1993,11 +1995,19 @@ e = parser.close() self.assertEqual(e.tag, 'html') - # XXX in _elementtree, the constructor of TreeBuilder expects no - # arguments - @unittest.expectedFailure def test_element_factory(self): - tb = ET.TreeBuilder(element_factory=lambda: ET.Element()) + lst = [] + def myfactory(tag, attrib): + nonlocal lst + lst.append(tag) + return ET.Element(tag, attrib) + + tb = ET.TreeBuilder(element_factory=myfactory) + parser = ET.XMLParser(target=tb) + parser.feed(self.sample2) + parser.close() + + self.assertEqual(lst, ['toplevel']) @unittest.expectedFailure # XXX issue 14007 with C ElementTree def test_doctype(self): diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -191,6 +191,15 @@ return result; } +/* Is the given object an empty dictionary? +*/ +static int +is_empty_dict(PyObject *obj) +{ + return PyDict_CheckExact(obj) && PyDict_Size(obj) == 0; +} + + /* -------------------------------------------------------------------- */ /* the Element type */ @@ -297,14 +306,9 @@ self = PyObject_GC_New(ElementObject, &Element_Type); if (self == NULL) return NULL; - - /* use None for empty dictionaries */ - if (PyDict_CheckExact(attrib) && !PyDict_Size(attrib)) - attrib = Py_None; - self->extra = NULL; - if (attrib != Py_None) { + if (attrib != Py_None && !is_empty_dict(attrib)) { if (create_extra(self, attrib) < 0) { PyObject_Del(self); return NULL; @@ -416,22 +420,14 @@ self_elem = (ElementObject *)self; - /* Use None for empty dictionaries */ - if (PyDict_CheckExact(attrib) && PyDict_Size(attrib) == 0) { - Py_INCREF(Py_None); - attrib = Py_None; - } - - if (attrib != Py_None) { + if (attrib != Py_None && !is_empty_dict(attrib)) { if (create_extra(self_elem, attrib) < 0) { PyObject_Del(self_elem); return -1; } } - /* If create_extra needed attrib, it took a reference to it, so we can - * release ours anyway. - */ + /* We own a reference to attrib here and it's no longer needed. */ Py_DECREF(attrib); /* Replace the objects already pointed to by tag, text and tail. */ @@ -1813,6 +1809,8 @@ PyObject *stack; /* element stack */ Py_ssize_t index; /* current stack size (0 means empty) */ + PyObject *element_factory; + /* element tracing */ PyObject *events; /* list of events, or NULL if not collecting */ PyObject *start_event_obj; /* event objects (NULL to ignore) */ @@ -1841,6 +1839,7 @@ t->last = (ElementObject *)Py_None; t->data = NULL; + t->element_factory = NULL; t->stack = PyList_New(20); if (!t->stack) { Py_DECREF(t->this); @@ -1859,11 +1858,38 @@ static int treebuilder_init(PyObject *self, PyObject *args, PyObject *kwds) { + static char *kwlist[] = {"element_factory", NULL}; + PyObject *element_factory = NULL; + TreeBuilderObject *self_tb = (TreeBuilderObject *)self; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O:TreeBuilder", kwlist, + &element_factory)) { + return -1; + } + + if (element_factory) { + Py_INCREF(element_factory); + Py_XDECREF(self_tb->element_factory); + self_tb->element_factory = element_factory; + } + return 0; } -static void -treebuilder_dealloc(TreeBuilderObject *self) +static int +treebuilder_gc_traverse(TreeBuilderObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->root); + Py_VISIT(self->this); + Py_VISIT(self->last); + Py_VISIT(self->data); + Py_VISIT(self->stack); + Py_VISIT(self->element_factory); + return 0; +} + +static int +treebuilder_gc_clear(TreeBuilderObject *self) { Py_XDECREF(self->end_ns_event_obj); Py_XDECREF(self->start_ns_event_obj); @@ -1874,8 +1900,16 @@ Py_XDECREF(self->data); Py_DECREF(self->last); Py_DECREF(self->this); + Py_CLEAR(self->element_factory); Py_XDECREF(self->root); - + return 0; +} + +static void +treebuilder_dealloc(TreeBuilderObject *self) +{ + PyObject_GC_UnTrack(self); + treebuilder_gc_clear(self); Py_TYPE(self)->tp_free((PyObject *)self); } @@ -1904,9 +1938,14 @@ self->data = NULL; } - node = create_new_element(tag, attrib); - if (!node) + if (self->element_factory) { + node = PyObject_CallFunction(self->element_factory, "OO", tag, attrib); + } else { + node = create_new_element(tag, attrib); + } + if (!node) { return NULL; + } this = (PyObject*) self->this; @@ -2180,10 +2219,11 @@ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, + /* tp_flags */ 0, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ + (traverseproc)treebuilder_gc_traverse, /* tp_traverse */ + (inquiry)treebuilder_gc_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ @@ -2443,17 +2483,20 @@ attrib = Py_None; } - if (TreeBuilder_CheckExact(self->target)) + /* If we get None, pass an empty dictionary on */ + if (attrib == Py_None) { + Py_DECREF(attrib); + attrib = PyDict_New(); + if (!attrib) + return; + } + + if (TreeBuilder_CheckExact(self->target)) { /* shortcut */ res = treebuilder_handle_start((TreeBuilderObject*) self->target, tag, attrib); + } else if (self->handle_start) { - if (attrib == Py_None) { - Py_DECREF(attrib); - attrib = PyDict_New(); - if (!attrib) - return; - } res = PyObject_CallFunction(self->handle_start, "OO", tag, attrib); } else res = NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 19:31:50 2012 From: python-checkins at python.org (martin.v.loewis) Date: Wed, 30 May 2012 19:31:50 +0200 (CEST) Subject: [Python-checkins] r88970 - tracker/instances/python-dev/scripts/addpatchsets Message-ID: <3W2f4Q3h4MzMyB@mail.python.org> Author: martin.v.loewis Date: Wed May 30 19:31:50 2012 New Revision: 88970 Log: Port to new Rietveld API. Modified: tracker/instances/python-dev/scripts/addpatchsets Modified: tracker/instances/python-dev/scripts/addpatchsets ============================================================================== --- tracker/instances/python-dev/scripts/addpatchsets (original) +++ tracker/instances/python-dev/scripts/addpatchsets Wed May 30 19:31:50 2012 @@ -16,7 +16,7 @@ from codereview.models import (Repository, Branch, Patch, PatchSet, Issue, Content) from django.contrib.auth.models import User -from codereview import engine, patching +from codereview import patching, utils from roundup_helper.models import File, RoundupIssue from django.db import connection, transaction from google.appengine.ext import db as gae_db @@ -44,7 +44,7 @@ # been a non-empty base, patching will fail. base = "" else: - base = engine.UnifyLinebreaks(r.read()) + base = utils.unify_linebreaks(r.read()) lines = base.splitlines(True) for (start, end), newrange, oldlines, newlines in chunks: if lines[start:end] != oldlines: @@ -148,7 +148,7 @@ issue = issue[0] if verbose: print "Doing", f.id - data = engine.UnifyLinebreaks(data) + data = utils.unify_linebreaks(data) branch, bases = find_bases(data) if not branch: if f.id < 15000: @@ -159,7 +159,6 @@ blob = gae_db.Blob(data) patchset = PatchSet(issue=issue, data=blob, parent=issue, - owner=User.objects.get(id=f._creator), created=f._creation, modified=f._creation) patchset.put() issue.patchset=patchset @@ -168,10 +167,10 @@ f._patchset = str(patchset.id) f.save() for filename, data, chunks, base in bases: - patch = Patch(patchset=patchset, text=engine.ToText(data), + patch = Patch(patchset=patchset, text=utils.to_dbtext(data), filename=filename, parent=patchset) patch.put() - content = Content(text=engine.ToText(base), parent=patch) + content = Content(text=utils.to_dbtext(base), parent=patch) content.put() patch.content = content patch.put() From python-checkins at python.org Wed May 30 19:38:28 2012 From: python-checkins at python.org (martin.v.loewis) Date: Wed, 30 May 2012 19:38:28 +0200 (CEST) Subject: [Python-checkins] r88971 - tracker/instances/python-dev/extensions/openid_login.py Message-ID: <3W2fD44RQ3zMvf@mail.python.org> Author: martin.v.loewis Date: Wed May 30 19:38:28 2012 New Revision: 88971 Log: Drop duplicate get_session call. Modified: tracker/instances/python-dev/extensions/openid_login.py Modified: tracker/instances/python-dev/extensions/openid_login.py ============================================================================== --- tracker/instances/python-dev/extensions/openid_login.py (original) +++ tracker/instances/python-dev/extensions/openid_login.py Wed May 30 19:38:28 2012 @@ -166,11 +166,6 @@ return services, op_endpoint, op_local = result session = self.get_session(op_endpoint, services) - try: - session = self.get_session(op_endpoint, services) - except NoCertificate: - self.client.error_message.append(self._('Peer did not return certificate')) - return realm = self.base+"?@action=openid_return" return_to = realm + "&__came_from=%s" % urllib.quote(self.client.path) url = openid2rp.request_authentication(services, op_endpoint, From python-checkins at python.org Wed May 30 22:04:57 2012 From: python-checkins at python.org (georg.brandl) Date: Wed, 30 May 2012 22:04:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_pydoc_topics_and_fix?= =?utf8?q?_new_suspicious_markup=2E?= Message-ID: http://hg.python.org/cpython/rev/ab0950575fc1 changeset: 77247:ab0950575fc1 parent: 77245:a5e621c8dd44 user: Georg Brandl date: Wed May 30 22:03:20 2012 +0200 summary: Update pydoc topics and fix new suspicious markup. files: Doc/faq/library.rst | 2 +- Doc/library/email.generator.rst | 2 +- Doc/library/venv.rst | 4 +- Doc/tools/sphinxext/susp-ignored.csv | 34 ++++++++++++++++ Doc/whatsnew/3.3.rst | 2 +- Lib/pydoc_data/topics.py | 18 ++++---- 6 files changed, 47 insertions(+), 15 deletions(-) diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst --- a/Doc/faq/library.rst +++ b/Doc/faq/library.rst @@ -351,7 +351,7 @@ Worker running with argument 5 ... -Consult the module's documentation for more details; the :class:`~queue.Queue`` +Consult the module's documentation for more details; the :class:`~queue.Queue` class provides a featureful interface. diff --git a/Doc/library/email.generator.rst b/Doc/library/email.generator.rst --- a/Doc/library/email.generator.rst +++ b/Doc/library/email.generator.rst @@ -55,7 +55,7 @@ The *policy* keyword specifies a :mod:`~email.policy` object that controls a number of aspects of the generator's operation. If no *policy* is specified, - then the *policy* attached to the message object passed to :attr:``flatten`` + then the *policy* attached to the message object passed to :attr:`flatten` is used. .. versionchanged:: 3.3 Added the *policy* keyword. diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -136,9 +136,7 @@ def create(self, env_dir): """ Create a virtualized Python environment in a directory. - - :param env_dir: The target directory to create an environment in. - + env_dir is the target directory to create an environment in. """ env_dir = os.path.abspath(env_dir) context = self.create_directories(env_dir) diff --git a/Doc/tools/sphinxext/susp-ignored.csv b/Doc/tools/sphinxext/susp-ignored.csv --- a/Doc/tools/sphinxext/susp-ignored.csv +++ b/Doc/tools/sphinxext/susp-ignored.csv @@ -30,6 +30,40 @@ howto/curses,,:red,"They are: 0:black, 1:red, 2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and" howto/curses,,:white,"7:white." howto/curses,,:yellow,"They are: 0:black, 1:red, 2:green, 3:yellow, 4:blue, 5:magenta, 6:cyan, and" +howto/ipaddress,,:DB8,>>> ipaddress.ip_address('2001:DB8::1') +howto/ipaddress,,::,>>> ipaddress.ip_address('2001:DB8::1') +howto/ipaddress,,:db8,IPv6Address('2001:db8::1') +howto/ipaddress,,::,IPv6Address('2001:db8::1') +howto/ipaddress,,:db8,IPv6Address('2001:db8::1') +howto/ipaddress,,::,IPv6Address('2001:db8::1') +howto/ipaddress,,::,IPv6Address('::1') +howto/ipaddress,,:db8,>>> ipaddress.ip_network('2001:db8::0/96') +howto/ipaddress,,::,>>> ipaddress.ip_network('2001:db8::0/96') +howto/ipaddress,,:db8,IPv6Network('2001:db8::/96') +howto/ipaddress,,::,IPv6Network('2001:db8::/96') +howto/ipaddress,,:db8,IPv6Network('2001:db8::/128') +howto/ipaddress,,::,IPv6Network('2001:db8::/128') +howto/ipaddress,,:db8,>>> ipaddress.ip_network('2001:db8::1/96') +howto/ipaddress,,::,>>> ipaddress.ip_network('2001:db8::1/96') +howto/ipaddress,,:db8,IPv6Interface('2001:db8::1/96') +howto/ipaddress,,::,IPv6Interface('2001:db8::1/96') +howto/ipaddress,,:db8,>>> addr6 = ipaddress.ip_address('2001:db8::1') +howto/ipaddress,,::,>>> addr6 = ipaddress.ip_address('2001:db8::1') +howto/ipaddress,,:db8,>>> host6 = ipaddress.ip_interface('2001:db8::1/96') +howto/ipaddress,,::,>>> host6 = ipaddress.ip_interface('2001:db8::1/96') +howto/ipaddress,,:db8,IPv6Network('2001:db8::/96') +howto/ipaddress,,::,IPv6Network('2001:db8::/96') +howto/ipaddress,,:db8,>>> net6 = ipaddress.ip_network('2001:db8::0/96') +howto/ipaddress,,::,>>> net6 = ipaddress.ip_network('2001:db8::0/96') +howto/ipaddress,,:db8,>>> net6 = ipaddress.ip_network('2001:db8::0/96') +howto/ipaddress,,::,>>> net6 = ipaddress.ip_network('2001:db8::0/96') +howto/ipaddress,,:ffff,IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::') +howto/ipaddress,,::,IPv6Address('ffff:ffff:ffff:ffff:ffff:ffff::') +howto/ipaddress,,::,IPv6Address('::ffff:ffff') +howto/ipaddress,,:ffff,IPv6Address('::ffff:ffff') +howto/ipaddress,,::,IPv6Address('2001::1') +howto/ipaddress,,::,IPv6Address('2001::ffff:ffff') +howto/ipaddress,,:ffff,IPv6Address('2001::ffff:ffff') howto/logging,,:And,"WARNING:And this, too" howto/logging,,:And,"WARNING:root:And this, too" howto/logging,,:Doing,INFO:root:Doing something diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst --- a/Doc/whatsnew/3.3.rst +++ b/Doc/whatsnew/3.3.rst @@ -600,7 +600,7 @@ the call retains its default value. Thus you can create a policy that uses ``\r\n`` linesep characters like this:: - mypolicy = compat32.clone(linesep=`\r\n`) + mypolicy = compat32.clone(linesep='\r\n') Policies can be used to make the generation of messages in the format needed by your application simpler. Instead of having to remember to specify diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Tue May 1 09:25:14 2012 +# Autogenerated by Sphinx on Wed May 30 21:54:26 2012 topics = {'assert': '\nThe ``assert`` statement\n************************\n\nAssert statements are a convenient way to insert debugging assertions\ninto a program:\n\n assert_stmt ::= "assert" expression ["," expression]\n\nThe simple form, ``assert expression``, is equivalent to\n\n if __debug__:\n if not expression: raise AssertionError\n\nThe extended form, ``assert expression1, expression2``, is equivalent\nto\n\n if __debug__:\n if not expression1: raise AssertionError(expression2)\n\nThese equivalences assume that ``__debug__`` and ``AssertionError``\nrefer to the built-in variables with those names. In the current\nimplementation, the built-in variable ``__debug__`` is ``True`` under\nnormal circumstances, ``False`` when optimization is requested\n(command line option -O). The current code generator emits no code\nfor an assert statement when optimization is requested at compile\ntime. Note that it is unnecessary to include the source code for the\nexpression that failed in the error message; it will be displayed as\npart of the stack trace.\n\nAssignments to ``__debug__`` are illegal. The value for the built-in\nvariable is determined when the interpreter starts.\n', 'assignment': '\nAssignment statements\n*********************\n\nAssignment statements are used to (re)bind names to values and to\nmodify attributes or items of mutable objects:\n\n assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)\n target_list ::= target ("," target)* [","]\n target ::= identifier\n | "(" target_list ")"\n | "[" target_list "]"\n | attributeref\n | subscription\n | slicing\n | "*" target\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn assignment statement evaluates the expression list (remember that\nthis can be a single expression or a comma-separated list, the latter\nyielding a tuple) and assigns the single resulting object to each of\nthe target lists, from left to right.\n\nAssignment is defined recursively depending on the form of the target\n(list). When a target is part of a mutable object (an attribute\nreference, subscription or slicing), the mutable object must\nultimately perform the assignment and decide about its validity, and\nmay raise an exception if the assignment is unacceptable. The rules\nobserved by various types and the exceptions raised are given with the\ndefinition of the object types (see section *The standard type\nhierarchy*).\n\nAssignment of an object to a target list, optionally enclosed in\nparentheses or square brackets, is recursively defined as follows.\n\n* If the target list is a single target: The object is assigned to\n that target.\n\n* If the target list is a comma-separated list of targets: The object\n must be an iterable with the same number of items as there are\n targets in the target list, and the items are assigned, from left to\n right, to the corresponding targets.\n\n * If the target list contains one target prefixed with an asterisk,\n called a "starred" target: The object must be a sequence with at\n least as many items as there are targets in the target list, minus\n one. The first items of the sequence are assigned, from left to\n right, to the targets before the starred target. The final items\n of the sequence are assigned to the targets after the starred\n target. A list of the remaining items in the sequence is then\n assigned to the starred target (the list can be empty).\n\n * Else: The object must be a sequence with the same number of items\n as there are targets in the target list, and the items are\n assigned, from left to right, to the corresponding targets.\n\nAssignment of an object to a single target is recursively defined as\nfollows.\n\n* If the target is an identifier (name):\n\n * If the name does not occur in a ``global`` or ``nonlocal``\n statement in the current code block: the name is bound to the\n object in the current local namespace.\n\n * Otherwise: the name is bound to the object in the global namespace\n or the outer namespace determined by ``nonlocal``, respectively.\n\n The name is rebound if it was already bound. This may cause the\n reference count for the object previously bound to the name to reach\n zero, causing the object to be deallocated and its destructor (if it\n has one) to be called.\n\n* If the target is a target list enclosed in parentheses or in square\n brackets: The object must be an iterable with the same number of\n items as there are targets in the target list, and its items are\n assigned, from left to right, to the corresponding targets.\n\n* If the target is an attribute reference: The primary expression in\n the reference is evaluated. It should yield an object with\n assignable attributes; if this is not the case, ``TypeError`` is\n raised. That object is then asked to assign the assigned object to\n the given attribute; if it cannot perform the assignment, it raises\n an exception (usually but not necessarily ``AttributeError``).\n\n Note: If the object is a class instance and the attribute reference\n occurs on both sides of the assignment operator, the RHS expression,\n ``a.x`` can access either an instance attribute or (if no instance\n attribute exists) a class attribute. The LHS target ``a.x`` is\n always set as an instance attribute, creating it if necessary.\n Thus, the two occurrences of ``a.x`` do not necessarily refer to the\n same attribute: if the RHS expression refers to a class attribute,\n the LHS creates a new instance attribute as the target of the\n assignment:\n\n class Cls:\n x = 3 # class variable\n inst = Cls()\n inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3\n\n This description does not necessarily apply to descriptor\n attributes, such as properties created with ``property()``.\n\n* If the target is a subscription: The primary expression in the\n reference is evaluated. It should yield either a mutable sequence\n object (such as a list) or a mapping object (such as a dictionary).\n Next, the subscript expression is evaluated.\n\n If the primary is a mutable sequence object (such as a list), the\n subscript must yield an integer. If it is negative, the sequence\'s\n length is added to it. The resulting value must be a nonnegative\n integer less than the sequence\'s length, and the sequence is asked\n to assign the assigned object to its item with that index. If the\n index is out of range, ``IndexError`` is raised (assignment to a\n subscripted sequence cannot add new items to a list).\n\n If the primary is a mapping object (such as a dictionary), the\n subscript must have a type compatible with the mapping\'s key type,\n and the mapping is then asked to create a key/datum pair which maps\n the subscript to the assigned object. This can either replace an\n existing key/value pair with the same key value, or insert a new\n key/value pair (if no key with the same value existed).\n\n For user-defined objects, the ``__setitem__()`` method is called\n with appropriate arguments.\n\n* If the target is a slicing: The primary expression in the reference\n is evaluated. It should yield a mutable sequence object (such as a\n list). The assigned object should be a sequence object of the same\n type. Next, the lower and upper bound expressions are evaluated,\n insofar they are present; defaults are zero and the sequence\'s\n length. The bounds should evaluate to integers. If either bound is\n negative, the sequence\'s length is added to it. The resulting\n bounds are clipped to lie between zero and the sequence\'s length,\n inclusive. Finally, the sequence object is asked to replace the\n slice with the items of the assigned sequence. The length of the\n slice may be different from the length of the assigned sequence,\n thus changing the length of the target sequence, if the object\n allows it.\n\n**CPython implementation detail:** In the current implementation, the\nsyntax for targets is taken to be the same as for expressions, and\ninvalid syntax is rejected during the code generation phase, causing\nless detailed error messages.\n\nWARNING: Although the definition of assignment implies that overlaps\nbetween the left-hand side and the right-hand side are \'safe\' (for\nexample ``a, b = b, a`` swaps two variables), overlaps *within* the\ncollection of assigned-to variables are not safe! For instance, the\nfollowing program prints ``[0, 2]``:\n\n x = [0, 1]\n i = 0\n i, x[i] = 1, 2\n print(x)\n\nSee also:\n\n **PEP 3132** - Extended Iterable Unpacking\n The specification for the ``*target`` feature.\n\n\nAugmented assignment statements\n===============================\n\nAugmented assignment is the combination, in a single statement, of a\nbinary operation and an assignment statement:\n\n augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)\n augtarget ::= identifier | attributeref | subscription | slicing\n augop ::= "+=" | "-=" | "*=" | "/=" | "//=" | "%=" | "**="\n | ">>=" | "<<=" | "&=" | "^=" | "|="\n\n(See section *Primaries* for the syntax definitions for the last three\nsymbols.)\n\nAn augmented assignment evaluates the target (which, unlike normal\nassignment statements, cannot be an unpacking) and the expression\nlist, performs the binary operation specific to the type of assignment\non the two operands, and assigns the result to the original target.\nThe target is only evaluated once.\n\nAn augmented assignment expression like ``x += 1`` can be rewritten as\n``x = x + 1`` to achieve a similar, but not exactly equal effect. In\nthe augmented version, ``x`` is only evaluated once. Also, when\npossible, the actual operation is performed *in-place*, meaning that\nrather than creating a new object and assigning that to the target,\nthe old object is modified instead.\n\nWith the exception of assigning to tuples and multiple targets in a\nsingle statement, the assignment done by augmented assignment\nstatements is handled the same way as normal assignments. Similarly,\nwith the exception of the possible *in-place* behavior, the binary\noperation performed by augmented assignment is the same as the normal\nbinary operations.\n\nFor targets which are attribute references, the same *caveat about\nclass and instance attributes* applies as for regular assignments.\n', 'atom-identifiers': '\nIdentifiers (Names)\n*******************\n\nAn identifier occurring as an atom is a name. See section\n*Identifiers and keywords* for lexical definition and section *Naming\nand binding* for documentation of naming and binding.\n\nWhen the name is bound to an object, evaluation of the atom yields\nthat object. When a name is not bound, an attempt to evaluate it\nraises a ``NameError`` exception.\n\n**Private name mangling:** When an identifier that textually occurs in\na class definition begins with two or more underscore characters and\ndoes not end in two or more underscores, it is considered a *private\nname* of that class. Private names are transformed to a longer form\nbefore code is generated for them. The transformation inserts the\nclass name in front of the name, with leading underscores removed, and\na single underscore inserted in front of the class name. For example,\nthe identifier ``__spam`` occurring in a class named ``Ham`` will be\ntransformed to ``_Ham__spam``. This transformation is independent of\nthe syntactical context in which the identifier is used. If the\ntransformed name is extremely long (longer than 255 characters),\nimplementation defined truncation may happen. If the class name\nconsists only of underscores, no transformation is done.\n', @@ -10,7 +10,7 @@ 'binary': '\nBinary arithmetic operations\n****************************\n\nThe binary arithmetic operations have the conventional priority\nlevels. Note that some of these operations also apply to certain non-\nnumeric types. Apart from the power operator, there are only two\nlevels, one for multiplicative operators and one for additive\noperators:\n\n m_expr ::= u_expr | m_expr "*" u_expr | m_expr "//" u_expr | m_expr "/" u_expr\n | m_expr "%" u_expr\n a_expr ::= m_expr | a_expr "+" m_expr | a_expr "-" m_expr\n\nThe ``*`` (multiplication) operator yields the product of its\narguments. The arguments must either both be numbers, or one argument\nmust be an integer and the other must be a sequence. In the former\ncase, the numbers are converted to a common type and then multiplied\ntogether. In the latter case, sequence repetition is performed; a\nnegative repetition factor yields an empty sequence.\n\nThe ``/`` (division) and ``//`` (floor division) operators yield the\nquotient of their arguments. The numeric arguments are first\nconverted to a common type. Integer division yields a float, while\nfloor division of integers results in an integer; the result is that\nof mathematical division with the \'floor\' function applied to the\nresult. Division by zero raises the ``ZeroDivisionError`` exception.\n\nThe ``%`` (modulo) operator yields the remainder from the division of\nthe first argument by the second. The numeric arguments are first\nconverted to a common type. A zero right argument raises the\n``ZeroDivisionError`` exception. The arguments may be floating point\nnumbers, e.g., ``3.14%0.7`` equals ``0.34`` (since ``3.14`` equals\n``4*0.7 + 0.34``.) The modulo operator always yields a result with\nthe same sign as its second operand (or zero); the absolute value of\nthe result is strictly smaller than the absolute value of the second\noperand [1].\n\nThe floor division and modulo operators are connected by the following\nidentity: ``x == (x//y)*y + (x%y)``. Floor division and modulo are\nalso connected with the built-in function ``divmod()``: ``divmod(x, y)\n== (x//y, x%y)``. [2].\n\nIn addition to performing the modulo operation on numbers, the ``%``\noperator is also overloaded by string objects to perform old-style\nstring formatting (also known as interpolation). The syntax for\nstring formatting is described in the Python Library Reference,\nsection *Old String Formatting Operations*.\n\nThe floor division operator, the modulo operator, and the ``divmod()``\nfunction are not defined for complex numbers. Instead, convert to a\nfloating point number using the ``abs()`` function if appropriate.\n\nThe ``+`` (addition) operator yields the sum of its arguments. The\narguments must either both be numbers or both sequences of the same\ntype. In the former case, the numbers are converted to a common type\nand then added together. In the latter case, the sequences are\nconcatenated.\n\nThe ``-`` (subtraction) operator yields the difference of its\narguments. The numeric arguments are first converted to a common\ntype.\n', 'bitwise': '\nBinary bitwise operations\n*************************\n\nEach of the three bitwise operations has a different priority level:\n\n and_expr ::= shift_expr | and_expr "&" shift_expr\n xor_expr ::= and_expr | xor_expr "^" and_expr\n or_expr ::= xor_expr | or_expr "|" xor_expr\n\nThe ``&`` operator yields the bitwise AND of its arguments, which must\nbe integers.\n\nThe ``^`` operator yields the bitwise XOR (exclusive OR) of its\narguments, which must be integers.\n\nThe ``|`` operator yields the bitwise (inclusive) OR of its arguments,\nwhich must be integers.\n', 'bltin-code-objects': '\nCode Objects\n************\n\nCode objects are used by the implementation to represent "pseudo-\ncompiled" executable Python code such as a function body. They differ\nfrom function objects because they don\'t contain a reference to their\nglobal execution environment. Code objects are returned by the built-\nin ``compile()`` function and can be extracted from function objects\nthrough their ``__code__`` attribute. See also the ``code`` module.\n\nA code object can be executed or evaluated by passing it (instead of a\nsource string) to the ``exec()`` or ``eval()`` built-in functions.\n\nSee *The standard type hierarchy* for more information.\n', - 'bltin-ellipsis-object': '\nThe Ellipsis Object\n*******************\n\nThis object is commonly used by slicing (see *Slicings*), but may also\nbe used in other situations where a sentinel value other than ``None``\nis needed. It supports no special operations. There is exactly one\nellipsis object, named ``Ellipsis`` (a built-in name).\n``type(Ellipsis)()`` produces the ``Ellipsis`` singleton.\n\nIt is written as ``Ellipsis`` or ``...``.\n', + 'bltin-ellipsis-object': '\nThe Ellipsis Object\n*******************\n\nThis object is commonly used by slicing (see *Slicings*). It supports\nno special operations. There is exactly one ellipsis object, named\n``Ellipsis`` (a built-in name). ``type(Ellipsis)()`` produces the\n``Ellipsis`` singleton.\n\nIt is written as ``Ellipsis`` or ``...``.\n', 'bltin-null-object': "\nThe Null Object\n***************\n\nThis object is returned by functions that don't explicitly return a\nvalue. It supports no special operations. There is exactly one null\nobject, named ``None`` (a built-in name). ``type(None)()`` produces\nthe same singleton.\n\nIt is written as ``None``.\n", 'bltin-type-objects': "\nType Objects\n************\n\nType objects represent the various object types. An object's type is\naccessed by the built-in function ``type()``. There are no special\noperations on types. The standard module ``types`` defines names for\nall standard built-in types.\n\nTypes are written like this: ````.\n", 'booleans': '\nBoolean operations\n******************\n\n or_test ::= and_test | or_test "or" and_test\n and_test ::= not_test | and_test "and" not_test\n not_test ::= comparison | "not" not_test\n\nIn the context of Boolean operations, and also when expressions are\nused by control flow statements, the following values are interpreted\nas false: ``False``, ``None``, numeric zero of all types, and empty\nstrings and containers (including strings, tuples, lists,\ndictionaries, sets and frozensets). All other values are interpreted\nas true. User-defined objects can customize their truth value by\nproviding a ``__bool__()`` method.\n\nThe operator ``not`` yields ``True`` if its argument is false,\n``False`` otherwise.\n\nThe expression ``x and y`` first evaluates *x*; if *x* is false, its\nvalue is returned; otherwise, *y* is evaluated and the resulting value\nis returned.\n\nThe expression ``x or y`` first evaluates *x*; if *x* is true, its\nvalue is returned; otherwise, *y* is evaluated and the resulting value\nis returned.\n\n(Note that neither ``and`` nor ``or`` restrict the value and type they\nreturn to ``False`` and ``True``, but rather return the last evaluated\nargument. This is sometimes useful, e.g., if ``s`` is a string that\nshould be replaced by a default value if it is empty, the expression\n``s or \'foo\'`` yields the desired value. Because ``not`` has to\ninvent a value anyway, it does not bother to return a value of the\nsame type as its argument, so e.g., ``not \'foo\'`` yields ``False``,\nnot ``\'\'``.)\n', @@ -23,7 +23,7 @@ 'context-managers': '\nWith Statement Context Managers\n*******************************\n\nA *context manager* is an object that defines the runtime context to\nbe established when executing a ``with`` statement. The context\nmanager handles the entry into, and the exit from, the desired runtime\ncontext for the execution of the block of code. Context managers are\nnormally invoked using the ``with`` statement (described in section\n*The with statement*), but can also be used by directly invoking their\nmethods.\n\nTypical uses of context managers include saving and restoring various\nkinds of global state, locking and unlocking resources, closing opened\nfiles, etc.\n\nFor more information on context managers, see *Context Manager Types*.\n\nobject.__enter__(self)\n\n Enter the runtime context related to this object. The ``with``\n statement will bind this method\'s return value to the target(s)\n specified in the ``as`` clause of the statement, if any.\n\nobject.__exit__(self, exc_type, exc_value, traceback)\n\n Exit the runtime context related to this object. The parameters\n describe the exception that caused the context to be exited. If the\n context was exited without an exception, all three arguments will\n be ``None``.\n\n If an exception is supplied, and the method wishes to suppress the\n exception (i.e., prevent it from being propagated), it should\n return a true value. Otherwise, the exception will be processed\n normally upon exit from this method.\n\n Note that ``__exit__()`` methods should not reraise the passed-in\n exception; this is the caller\'s responsibility.\n\nSee also:\n\n **PEP 0343** - The "with" statement\n The specification, background, and examples for the Python\n ``with`` statement.\n', 'continue': '\nThe ``continue`` statement\n**************************\n\n continue_stmt ::= "continue"\n\n``continue`` may only occur syntactically nested in a ``for`` or\n``while`` loop, but not nested in a function or class definition or\n``finally`` clause within that loop. It continues with the next cycle\nof the nearest enclosing loop.\n\nWhen ``continue`` passes control out of a ``try`` statement with a\n``finally`` clause, that ``finally`` clause is executed before really\nstarting the next loop cycle.\n', 'conversions': '\nArithmetic conversions\n**********************\n\nWhen a description of an arithmetic operator below uses the phrase\n"the numeric arguments are converted to a common type," this means\nthat the operator implementation for built-in types works that way:\n\n* If either argument is a complex number, the other is converted to\n complex;\n\n* otherwise, if either argument is a floating point number, the other\n is converted to floating point;\n\n* otherwise, both must be integers and no conversion is necessary.\n\nSome additional rules apply for certain operators (e.g., a string left\nargument to the \'%\' operator). Extensions must define their own\nconversion behavior.\n', - 'customization': '\nBasic customization\n*******************\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. ``__new__()`` is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of ``__new__()`` should be the new object instance (usually\n an instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s ``__new__()`` method using\n ``super(currentclass, cls).__new__(cls[, ...])`` with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If ``__new__()`` returns an instance of *cls*, then the new\n instance\'s ``__init__()`` method will be invoked like\n ``__init__(self[, ...])``, where *self* is the new instance and the\n remaining arguments are the same as were passed to ``__new__()``.\n\n If ``__new__()`` does not return an instance of *cls*, then the new\n instance\'s ``__init__()`` method will not be invoked.\n\n ``__new__()`` is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n ``__init__()`` method, the derived class\'s ``__init__()`` method,\n if any, must explicitly call it to ensure proper initialization of\n the base class part of the instance; for example:\n ``BaseClass.__init__(self, [args...])``. As a special constraint\n on constructors, no value may be returned; doing so will cause a\n ``TypeError`` to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a ``__del__()`` method,\n the derived class\'s ``__del__()`` method, if any, must explicitly\n call it to ensure proper deletion of the base class part of the\n instance. Note that it is possible (though not recommended!) for\n the ``__del__()`` method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n ``__del__()`` methods are called for objects that still exist when\n the interpreter exits.\n\n Note: ``del x`` doesn\'t directly call ``x.__del__()`` --- the former\n decrements the reference count for ``x`` by one, and the latter\n is only called when ``x``\'s reference count reaches zero. Some\n common situations that may prevent the reference count of an\n object from going to zero include: circular references between\n objects (e.g., a doubly-linked list or a tree data structure with\n parent and child pointers); a reference to the object on the\n stack frame of a function that caught an exception (the traceback\n stored in ``sys.exc_info()[2]`` keeps the stack frame alive); or\n a reference to the object on the stack frame that raised an\n unhandled exception in interactive mode (the traceback stored in\n ``sys.last_traceback`` keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing ``None`` in\n ``sys.last_traceback``. Circular references which are garbage are\n detected when the option cycle detector is enabled (it\'s on by\n default), but can only be cleaned up if there are no Python-\n level ``__del__()`` methods involved. Refer to the documentation\n for the ``gc`` module for more information about how\n ``__del__()`` methods are handled by the cycle detector,\n particularly the description of the ``garbage`` value.\n\n Warning: Due to the precarious circumstances under which ``__del__()``\n methods are invoked, exceptions that occur during their execution\n are ignored, and a warning is printed to ``sys.stderr`` instead.\n Also, when ``__del__()`` is invoked in response to a module being\n deleted (e.g., when execution of the program is done), other\n globals referenced by the ``__del__()`` method may already have\n been deleted or in the process of being torn down (e.g. the\n import machinery shutting down). For this reason, ``__del__()``\n methods should do the absolute minimum needed to maintain\n external invariants. Starting with version 1.5, Python\n guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the ``__del__()`` method is called.\n\nobject.__repr__(self)\n\n Called by the ``repr()`` built-in function to compute the\n "official" string representation of an object. If at all possible,\n this should look like a valid Python expression that could be used\n to recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n ``<...some useful description...>`` should be returned. The return\n value must be a string object. If a class defines ``__repr__()``\n but not ``__str__()``, then ``__repr__()`` is also used when an\n "informal" string representation of instances of that class is\n required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by the ``str()`` built-in function and by the ``print()``\n function to compute the "informal" string representation of an\n object. This differs from ``__repr__()`` in that it does not have\n to be a valid Python expression: a more convenient or concise\n representation may be used instead. The return value must be a\n string object.\n\nobject.__bytes__(self)\n\n Called by ``bytes()`` to compute a byte-string representation of an\n object. This should return a ``bytes`` object.\n\nobject.__format__(self, format_spec)\n\n Called by the ``format()`` built-in function (and by extension, the\n ``format()`` method of class ``str``) to produce a "formatted"\n string representation of an object. The ``format_spec`` argument is\n a string that contains a description of the formatting options\n desired. The interpretation of the ``format_spec`` argument is up\n to the type implementing ``__format__()``, however most classes\n will either delegate formatting to one of the built-in types, or\n use a similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: ``xy`` calls ``x.__gt__(y)``, and ``x>=y`` calls\n ``x.__ge__(y)``.\n\n A rich comparison method may return the singleton\n ``NotImplemented`` if it does not implement the operation for a\n given pair of arguments. By convention, ``False`` and ``True`` are\n returned for a successful comparison. However, these methods can\n return any value, so if the comparison operator is used in a\n Boolean context (e.g., in the condition of an ``if`` statement),\n Python will call ``bool()`` on the value to determine if the result\n is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of ``x==y`` does not imply that ``x!=y`` is false.\n Accordingly, when defining ``__eq__()``, one should also define\n ``__ne__()`` so that the operators will behave as expected. See\n the paragraph on ``__hash__()`` for some important notes on\n creating *hashable* objects which support custom comparison\n operations and are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, ``__lt__()`` and ``__gt__()`` are each\n other\'s reflection, ``__le__()`` and ``__ge__()`` are each other\'s\n reflection, and ``__eq__()`` and ``__ne__()`` are their own\n reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see ``functools.total_ordering()``.\n\nobject.__hash__(self)\n\n Called by built-in function ``hash()`` and for operations on\n members of hashed collections including ``set``, ``frozenset``, and\n ``dict``. ``__hash__()`` should return an integer. The only\n required property is that objects which compare equal have the same\n hash value; it is advised to somehow mix together (e.g. using\n exclusive or) the hash values for the components of the object that\n also play a part in comparison of objects.\n\n If a class does not define an ``__eq__()`` method it should not\n define a ``__hash__()`` operation either; if it defines\n ``__eq__()`` but not ``__hash__()``, its instances will not be\n usable as items in hashable collections. If a class defines\n mutable objects and implements an ``__eq__()`` method, it should\n not implement ``__hash__()``, since the implementation of hashable\n collections requires that a key\'s hash value is immutable (if the\n object\'s hash value changes, it will be in the wrong hash bucket).\n\n User-defined classes have ``__eq__()`` and ``__hash__()`` methods\n by default; with them, all objects compare unequal (except with\n themselves) and ``x.__hash__()`` returns ``id(x)``.\n\n Classes which inherit a ``__hash__()`` method from a parent class\n but change the meaning of ``__eq__()`` such that the hash value\n returned is no longer appropriate (e.g. by switching to a value-\n based concept of equality instead of the default identity based\n equality) can explicitly flag themselves as being unhashable by\n setting ``__hash__ = None`` in the class definition. Doing so means\n that not only will instances of the class raise an appropriate\n ``TypeError`` when a program attempts to retrieve their hash value,\n but they will also be correctly identified as unhashable when\n checking ``isinstance(obj, collections.Hashable)`` (unlike classes\n which define their own ``__hash__()`` to explicitly raise\n ``TypeError``).\n\n If a class that overrides ``__eq__()`` needs to retain the\n implementation of ``__hash__()`` from a parent class, the\n interpreter must be told this explicitly by setting ``__hash__ =\n .__hash__``. Otherwise the inheritance of\n ``__hash__()`` will be blocked, just as if ``__hash__`` had been\n explicitly set to ``None``.\n\n Note: Note by default the ``__hash__()`` values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the order in which keys are\n retrieved from a dict. Note Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also ``PYTHONHASHSEED``.\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n ``bool()``; should return ``False`` or ``True``. When this method\n is not defined, ``__len__()`` is called, if it is defined, and the\n object is considered true if its result is nonzero. If a class\n defines neither ``__len__()`` nor ``__bool__()``, all its instances\n are considered true.\n', + 'customization': '\nBasic customization\n*******************\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. ``__new__()`` is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of ``__new__()`` should be the new object instance (usually\n an instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s ``__new__()`` method using\n ``super(currentclass, cls).__new__(cls[, ...])`` with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If ``__new__()`` returns an instance of *cls*, then the new\n instance\'s ``__init__()`` method will be invoked like\n ``__init__(self[, ...])``, where *self* is the new instance and the\n remaining arguments are the same as were passed to ``__new__()``.\n\n If ``__new__()`` does not return an instance of *cls*, then the new\n instance\'s ``__init__()`` method will not be invoked.\n\n ``__new__()`` is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n ``__init__()`` method, the derived class\'s ``__init__()`` method,\n if any, must explicitly call it to ensure proper initialization of\n the base class part of the instance; for example:\n ``BaseClass.__init__(self, [args...])``. As a special constraint\n on constructors, no value may be returned; doing so will cause a\n ``TypeError`` to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a ``__del__()`` method,\n the derived class\'s ``__del__()`` method, if any, must explicitly\n call it to ensure proper deletion of the base class part of the\n instance. Note that it is possible (though not recommended!) for\n the ``__del__()`` method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n ``__del__()`` methods are called for objects that still exist when\n the interpreter exits.\n\n Note: ``del x`` doesn\'t directly call ``x.__del__()`` --- the former\n decrements the reference count for ``x`` by one, and the latter\n is only called when ``x``\'s reference count reaches zero. Some\n common situations that may prevent the reference count of an\n object from going to zero include: circular references between\n objects (e.g., a doubly-linked list or a tree data structure with\n parent and child pointers); a reference to the object on the\n stack frame of a function that caught an exception (the traceback\n stored in ``sys.exc_info()[2]`` keeps the stack frame alive); or\n a reference to the object on the stack frame that raised an\n unhandled exception in interactive mode (the traceback stored in\n ``sys.last_traceback`` keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing ``None`` in\n ``sys.last_traceback``. Circular references which are garbage are\n detected when the option cycle detector is enabled (it\'s on by\n default), but can only be cleaned up if there are no Python-\n level ``__del__()`` methods involved. Refer to the documentation\n for the ``gc`` module for more information about how\n ``__del__()`` methods are handled by the cycle detector,\n particularly the description of the ``garbage`` value.\n\n Warning: Due to the precarious circumstances under which ``__del__()``\n methods are invoked, exceptions that occur during their execution\n are ignored, and a warning is printed to ``sys.stderr`` instead.\n Also, when ``__del__()`` is invoked in response to a module being\n deleted (e.g., when execution of the program is done), other\n globals referenced by the ``__del__()`` method may already have\n been deleted or in the process of being torn down (e.g. the\n import machinery shutting down). For this reason, ``__del__()``\n methods should do the absolute minimum needed to maintain\n external invariants. Starting with version 1.5, Python\n guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the ``__del__()`` method is called.\n\nobject.__repr__(self)\n\n Called by the ``repr()`` built-in function to compute the\n "official" string representation of an object. If at all possible,\n this should look like a valid Python expression that could be used\n to recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n ``<...some useful description...>`` should be returned. The return\n value must be a string object. If a class defines ``__repr__()``\n but not ``__str__()``, then ``__repr__()`` is also used when an\n "informal" string representation of instances of that class is\n required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by the ``str()`` built-in function and by the ``print()``\n function to compute the "informal" string representation of an\n object. This differs from ``__repr__()`` in that it does not have\n to be a valid Python expression: a more convenient or concise\n representation may be used instead. The return value must be a\n string object.\n\nobject.__bytes__(self)\n\n Called by ``bytes()`` to compute a byte-string representation of an\n object. This should return a ``bytes`` object.\n\nobject.__format__(self, format_spec)\n\n Called by the ``format()`` built-in function (and by extension, the\n ``format()`` method of class ``str``) to produce a "formatted"\n string representation of an object. The ``format_spec`` argument is\n a string that contains a description of the formatting options\n desired. The interpretation of the ``format_spec`` argument is up\n to the type implementing ``__format__()``, however most classes\n will either delegate formatting to one of the built-in types, or\n use a similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: ``xy`` calls ``x.__gt__(y)``, and ``x>=y`` calls\n ``x.__ge__(y)``.\n\n A rich comparison method may return the singleton\n ``NotImplemented`` if it does not implement the operation for a\n given pair of arguments. By convention, ``False`` and ``True`` are\n returned for a successful comparison. However, these methods can\n return any value, so if the comparison operator is used in a\n Boolean context (e.g., in the condition of an ``if`` statement),\n Python will call ``bool()`` on the value to determine if the result\n is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of ``x==y`` does not imply that ``x!=y`` is false.\n Accordingly, when defining ``__eq__()``, one should also define\n ``__ne__()`` so that the operators will behave as expected. See\n the paragraph on ``__hash__()`` for some important notes on\n creating *hashable* objects which support custom comparison\n operations and are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, ``__lt__()`` and ``__gt__()`` are each\n other\'s reflection, ``__le__()`` and ``__ge__()`` are each other\'s\n reflection, and ``__eq__()`` and ``__ne__()`` are their own\n reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see ``functools.total_ordering()``.\n\nobject.__hash__(self)\n\n Called by built-in function ``hash()`` and for operations on\n members of hashed collections including ``set``, ``frozenset``, and\n ``dict``. ``__hash__()`` should return an integer. The only\n required property is that objects which compare equal have the same\n hash value; it is advised to somehow mix together (e.g. using\n exclusive or) the hash values for the components of the object that\n also play a part in comparison of objects.\n\n If a class does not define an ``__eq__()`` method it should not\n define a ``__hash__()`` operation either; if it defines\n ``__eq__()`` but not ``__hash__()``, its instances will not be\n usable as items in hashable collections. If a class defines\n mutable objects and implements an ``__eq__()`` method, it should\n not implement ``__hash__()``, since the implementation of hashable\n collections requires that a key\'s hash value is immutable (if the\n object\'s hash value changes, it will be in the wrong hash bucket).\n\n User-defined classes have ``__eq__()`` and ``__hash__()`` methods\n by default; with them, all objects compare unequal (except with\n themselves) and ``x.__hash__()`` returns an appropriate value such\n that ``x == y`` implies both that ``x is y`` and ``hash(x) ==\n hash(y)``.\n\n Classes which inherit a ``__hash__()`` method from a parent class\n but change the meaning of ``__eq__()`` such that the hash value\n returned is no longer appropriate (e.g. by switching to a value-\n based concept of equality instead of the default identity based\n equality) can explicitly flag themselves as being unhashable by\n setting ``__hash__ = None`` in the class definition. Doing so means\n that not only will instances of the class raise an appropriate\n ``TypeError`` when a program attempts to retrieve their hash value,\n but they will also be correctly identified as unhashable when\n checking ``isinstance(obj, collections.Hashable)`` (unlike classes\n which define their own ``__hash__()`` to explicitly raise\n ``TypeError``).\n\n If a class that overrides ``__eq__()`` needs to retain the\n implementation of ``__hash__()`` from a parent class, the\n interpreter must be told this explicitly by setting ``__hash__ =\n .__hash__``. Otherwise the inheritance of\n ``__hash__()`` will be blocked, just as if ``__hash__`` had been\n explicitly set to ``None``.\n\n Note: Note by default the ``__hash__()`` values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the order in which keys are\n retrieved from a dict. Note Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also ``PYTHONHASHSEED``.\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n ``bool()``; should return ``False`` or ``True``. When this method\n is not defined, ``__len__()`` is called, if it is defined, and the\n object is considered true if its result is nonzero. If a class\n defines neither ``__len__()`` nor ``__bool__()``, all its instances\n are considered true.\n', 'debugger': '\n``pdb`` --- The Python Debugger\n*******************************\n\nThe module ``pdb`` defines an interactive source code debugger for\nPython programs. It supports setting (conditional) breakpoints and\nsingle stepping at the source line level, inspection of stack frames,\nsource code listing, and evaluation of arbitrary Python code in the\ncontext of any stack frame. It also supports post-mortem debugging\nand can be called under program control.\n\nThe debugger is extensible -- it is actually defined as the class\n``Pdb``. This is currently undocumented but easily understood by\nreading the source. The extension interface uses the modules ``bdb``\nand ``cmd``.\n\nThe debugger\'s prompt is ``(Pdb)``. Typical usage to run a program\nunder control of the debugger is:\n\n >>> import pdb\n >>> import mymodule\n >>> pdb.run(\'mymodule.test()\')\n > (0)?()\n (Pdb) continue\n > (1)?()\n (Pdb) continue\n NameError: \'spam\'\n > (1)?()\n (Pdb)\n\nChanged in version 3.3: Tab-completion via the ``readline`` module is\navailable for commands and command arguments, e.g. the current global\nand local names are offered as arguments of the ``print`` command.\n\n``pdb.py`` can also be invoked as a script to debug other scripts.\nFor example:\n\n python3 -m pdb myscript.py\n\nWhen invoked as a script, pdb will automatically enter post-mortem\ndebugging if the program being debugged exits abnormally. After post-\nmortem debugging (or after normal exit of the program), pdb will\nrestart the program. Automatic restarting preserves pdb\'s state (such\nas breakpoints) and in most cases is more useful than quitting the\ndebugger upon program\'s exit.\n\nNew in version 3.2: ``pdb.py`` now accepts a ``-c`` option that\nexecutes commands as if given in a ``.pdbrc`` file, see *Debugger\nCommands*.\n\nThe typical usage to break into the debugger from a running program is\nto insert\n\n import pdb; pdb.set_trace()\n\nat the location you want to break into the debugger. You can then\nstep through the code following this statement, and continue running\nwithout the debugger using the ``continue`` command.\n\nThe typical usage to inspect a crashed program is:\n\n >>> import pdb\n >>> import mymodule\n >>> mymodule.test()\n Traceback (most recent call last):\n File "", line 1, in ?\n File "./mymodule.py", line 4, in test\n test2()\n File "./mymodule.py", line 3, in test2\n print(spam)\n NameError: spam\n >>> pdb.pm()\n > ./mymodule.py(3)test2()\n -> print(spam)\n (Pdb)\n\nThe module defines the following functions; each enters the debugger\nin a slightly different way:\n\npdb.run(statement, globals=None, locals=None)\n\n Execute the *statement* (given as a string or a code object) under\n debugger control. The debugger prompt appears before any code is\n executed; you can set breakpoints and type ``continue``, or you can\n step through the statement using ``step`` or ``next`` (all these\n commands are explained below). The optional *globals* and *locals*\n arguments specify the environment in which the code is executed; by\n default the dictionary of the module ``__main__`` is used. (See\n the explanation of the built-in ``exec()`` or ``eval()``\n functions.)\n\npdb.runeval(expression, globals=None, locals=None)\n\n Evaluate the *expression* (given as a string or a code object)\n under debugger control. When ``runeval()`` returns, it returns the\n value of the expression. Otherwise this function is similar to\n ``run()``.\n\npdb.runcall(function, *args, **kwds)\n\n Call the *function* (a function or method object, not a string)\n with the given arguments. When ``runcall()`` returns, it returns\n whatever the function call returned. The debugger prompt appears\n as soon as the function is entered.\n\npdb.set_trace()\n\n Enter the debugger at the calling stack frame. This is useful to\n hard-code a breakpoint at a given point in a program, even if the\n code is not otherwise being debugged (e.g. when an assertion\n fails).\n\npdb.post_mortem(traceback=None)\n\n Enter post-mortem debugging of the given *traceback* object. If no\n *traceback* is given, it uses the one of the exception that is\n currently being handled (an exception must be being handled if the\n default is to be used).\n\npdb.pm()\n\n Enter post-mortem debugging of the traceback found in\n ``sys.last_traceback``.\n\nThe ``run*`` functions and ``set_trace()`` are aliases for\ninstantiating the ``Pdb`` class and calling the method of the same\nname. If you want to access further features, you have to do this\nyourself:\n\nclass class pdb.Pdb(completekey=\'tab\', stdin=None, stdout=None, skip=None, nosigint=False)\n\n ``Pdb`` is the debugger class.\n\n The *completekey*, *stdin* and *stdout* arguments are passed to the\n underlying ``cmd.Cmd`` class; see the description there.\n\n The *skip* argument, if given, must be an iterable of glob-style\n module name patterns. The debugger will not step into frames that\n originate in a module that matches one of these patterns. [1]\n\n By default, Pdb sets a handler for the SIGINT signal (which is sent\n when the user presses Ctrl-C on the console) when you give a\n ``continue`` command. This allows you to break into the debugger\n again by pressing Ctrl-C. If you want Pdb not to touch the SIGINT\n handler, set *nosigint* tot true.\n\n Example call to enable tracing with *skip*:\n\n import pdb; pdb.Pdb(skip=[\'django.*\']).set_trace()\n\n New in version 3.1: The *skip* argument.\n\n New in version 3.2: The *nosigint* argument. Previously, a SIGINT\n handler was never set by Pdb.\n\n run(statement, globals=None, locals=None)\n runeval(expression, globals=None, locals=None)\n runcall(function, *args, **kwds)\n set_trace()\n\n See the documentation for the functions explained above.\n\n\nDebugger Commands\n=================\n\nThe commands recognized by the debugger are listed below. Most\ncommands can be abbreviated to one or two letters as indicated; e.g.\n``h(elp)`` means that either ``h`` or ``help`` can be used to enter\nthe help command (but not ``he`` or ``hel``, nor ``H`` or ``Help`` or\n``HELP``). Arguments to commands must be separated by whitespace\n(spaces or tabs). Optional arguments are enclosed in square brackets\n(``[]``) in the command syntax; the square brackets must not be typed.\nAlternatives in the command syntax are separated by a vertical bar\n(``|``).\n\nEntering a blank line repeats the last command entered. Exception: if\nthe last command was a ``list`` command, the next 11 lines are listed.\n\nCommands that the debugger doesn\'t recognize are assumed to be Python\nstatements and are executed in the context of the program being\ndebugged. Python statements can also be prefixed with an exclamation\npoint (``!``). This is a powerful way to inspect the program being\ndebugged; it is even possible to change a variable or call a function.\nWhen an exception occurs in such a statement, the exception name is\nprinted but the debugger\'s state is not changed.\n\nThe debugger supports *aliases*. Aliases can have parameters which\nallows one a certain level of adaptability to the context under\nexamination.\n\nMultiple commands may be entered on a single line, separated by\n``;;``. (A single ``;`` is not used as it is the separator for\nmultiple commands in a line that is passed to the Python parser.) No\nintelligence is applied to separating the commands; the input is split\nat the first ``;;`` pair, even if it is in the middle of a quoted\nstring.\n\nIf a file ``.pdbrc`` exists in the user\'s home directory or in the\ncurrent directory, it is read in and executed as if it had been typed\nat the debugger prompt. This is particularly useful for aliases. If\nboth files exist, the one in the home directory is read first and\naliases defined there can be overridden by the local file.\n\nChanged in version 3.2: ``.pdbrc`` can now contain commands that\ncontinue debugging, such as ``continue`` or ``next``. Previously,\nthese commands had no effect.\n\nh(elp) [command]\n\n Without argument, print the list of available commands. With a\n *command* as argument, print help about that command. ``help pdb``\n displays the full documentation (the docstring of the ``pdb``\n module). Since the *command* argument must be an identifier,\n ``help exec`` must be entered to get help on the ``!`` command.\n\nw(here)\n\n Print a stack trace, with the most recent frame at the bottom. An\n arrow indicates the current frame, which determines the context of\n most commands.\n\nd(own) [count]\n\n Move the current frame *count* (default one) levels down in the\n stack trace (to a newer frame).\n\nu(p) [count]\n\n Move the current frame *count* (default one) levels up in the stack\n trace (to an older frame).\n\nb(reak) [([filename:]lineno | function) [, condition]]\n\n With a *lineno* argument, set a break there in the current file.\n With a *function* argument, set a break at the first executable\n statement within that function. The line number may be prefixed\n with a filename and a colon, to specify a breakpoint in another\n file (probably one that hasn\'t been loaded yet). The file is\n searched on ``sys.path``. Note that each breakpoint is assigned a\n number to which all the other breakpoint commands refer.\n\n If a second argument is present, it is an expression which must\n evaluate to true before the breakpoint is honored.\n\n Without argument, list all breaks, including for each breakpoint,\n the number of times that breakpoint has been hit, the current\n ignore count, and the associated condition if any.\n\ntbreak [([filename:]lineno | function) [, condition]]\n\n Temporary breakpoint, which is removed automatically when it is\n first hit. The arguments are the same as for ``break``.\n\ncl(ear) [filename:lineno | bpnumber [bpnumber ...]]\n\n With a *filename:lineno* argument, clear all the breakpoints at\n this line. With a space separated list of breakpoint numbers, clear\n those breakpoints. Without argument, clear all breaks (but first\n ask confirmation).\n\ndisable [bpnumber [bpnumber ...]]\n\n Disable the breakpoints given as a space separated list of\n breakpoint numbers. Disabling a breakpoint means it cannot cause\n the program to stop execution, but unlike clearing a breakpoint, it\n remains in the list of breakpoints and can be (re-)enabled.\n\nenable [bpnumber [bpnumber ...]]\n\n Enable the breakpoints specified.\n\nignore bpnumber [count]\n\n Set the ignore count for the given breakpoint number. If count is\n omitted, the ignore count is set to 0. A breakpoint becomes active\n when the ignore count is zero. When non-zero, the count is\n decremented each time the breakpoint is reached and the breakpoint\n is not disabled and any associated condition evaluates to true.\n\ncondition bpnumber [condition]\n\n Set a new *condition* for the breakpoint, an expression which must\n evaluate to true before the breakpoint is honored. If *condition*\n is absent, any existing condition is removed; i.e., the breakpoint\n is made unconditional.\n\ncommands [bpnumber]\n\n Specify a list of commands for breakpoint number *bpnumber*. The\n commands themselves appear on the following lines. Type a line\n containing just ``end`` to terminate the commands. An example:\n\n (Pdb) commands 1\n (com) print some_variable\n (com) end\n (Pdb)\n\n To remove all commands from a breakpoint, type commands and follow\n it immediately with ``end``; that is, give no commands.\n\n With no *bpnumber* argument, commands refers to the last breakpoint\n set.\n\n You can use breakpoint commands to start your program up again.\n Simply use the continue command, or step, or any other command that\n resumes execution.\n\n Specifying any command resuming execution (currently continue,\n step, next, return, jump, quit and their abbreviations) terminates\n the command list (as if that command was immediately followed by\n end). This is because any time you resume execution (even with a\n simple next or step), you may encounter another breakpoint--which\n could have its own command list, leading to ambiguities about which\n list to execute.\n\n If you use the \'silent\' command in the command list, the usual\n message about stopping at a breakpoint is not printed. This may be\n desirable for breakpoints that are to print a specific message and\n then continue. If none of the other commands print anything, you\n see no sign that the breakpoint was reached.\n\ns(tep)\n\n Execute the current line, stop at the first possible occasion\n (either in a function that is called or on the next line in the\n current function).\n\nn(ext)\n\n Continue execution until the next line in the current function is\n reached or it returns. (The difference between ``next`` and\n ``step`` is that ``step`` stops inside a called function, while\n ``next`` executes called functions at (nearly) full speed, only\n stopping at the next line in the current function.)\n\nunt(il) [lineno]\n\n Without argument, continue execution until the line with a number\n greater than the current one is reached.\n\n With a line number, continue execution until a line with a number\n greater or equal to that is reached. In both cases, also stop when\n the current frame returns.\n\n Changed in version 3.2: Allow giving an explicit line number.\n\nr(eturn)\n\n Continue execution until the current function returns.\n\nc(ont(inue))\n\n Continue execution, only stop when a breakpoint is encountered.\n\nj(ump) lineno\n\n Set the next line that will be executed. Only available in the\n bottom-most frame. This lets you jump back and execute code again,\n or jump forward to skip code that you don\'t want to run.\n\n It should be noted that not all jumps are allowed -- for instance\n it is not possible to jump into the middle of a ``for`` loop or out\n of a ``finally`` clause.\n\nl(ist) [first[, last]]\n\n List source code for the current file. Without arguments, list 11\n lines around the current line or continue the previous listing.\n With ``.`` as argument, list 11 lines around the current line.\n With one argument, list 11 lines around at that line. With two\n arguments, list the given range; if the second argument is less\n than the first, it is interpreted as a count.\n\n The current line in the current frame is indicated by ``->``. If\n an exception is being debugged, the line where the exception was\n originally raised or propagated is indicated by ``>>``, if it\n differs from the current line.\n\n New in version 3.2: The ``>>`` marker.\n\nll | longlist\n\n List all source code for the current function or frame.\n Interesting lines are marked as for ``list``.\n\n New in version 3.2.\n\na(rgs)\n\n Print the argument list of the current function.\n\np(rint) expression\n\n Evaluate the *expression* in the current context and print its\n value.\n\npp expression\n\n Like the ``print`` command, except the value of the expression is\n pretty-printed using the ``pprint`` module.\n\nwhatis expression\n\n Print the type of the *expression*.\n\nsource expression\n\n Try to get source code for the given object and display it.\n\n New in version 3.2.\n\ndisplay [expression]\n\n Display the value of the expression if it changed, each time\n execution stops in the current frame.\n\n Without expression, list all display expressions for the current\n frame.\n\n New in version 3.2.\n\nundisplay [expression]\n\n Do not display the expression any more in the current frame.\n Without expression, clear all display expressions for the current\n frame.\n\n New in version 3.2.\n\ninteract\n\n Start an interative interpreter (using the ``code`` module) whose\n global namespace contains all the (global and local) names found in\n the current scope.\n\n New in version 3.2.\n\nalias [name [command]]\n\n Create an alias called *name* that executes *command*. The command\n must *not* be enclosed in quotes. Replaceable parameters can be\n indicated by ``%1``, ``%2``, and so on, while ``%*`` is replaced by\n all the parameters. If no command is given, the current alias for\n *name* is shown. If no arguments are given, all aliases are listed.\n\n Aliases may be nested and can contain anything that can be legally\n typed at the pdb prompt. Note that internal pdb commands *can* be\n overridden by aliases. Such a command is then hidden until the\n alias is removed. Aliasing is recursively applied to the first\n word of the command line; all other words in the line are left\n alone.\n\n As an example, here are two useful aliases (especially when placed\n in the ``.pdbrc`` file):\n\n # Print instance variables (usage "pi classInst")\n alias pi for k in %1.__dict__.keys(): print("%1.",k,"=",%1.__dict__[k])\n # Print instance variables in self\n alias ps pi self\n\nunalias name\n\n Delete the specified alias.\n\n! statement\n\n Execute the (one-line) *statement* in the context of the current\n stack frame. The exclamation point can be omitted unless the first\n word of the statement resembles a debugger command. To set a\n global variable, you can prefix the assignment command with a\n ``global`` statement on the same line, e.g.:\n\n (Pdb) global list_options; list_options = [\'-l\']\n (Pdb)\n\nrun [args ...]\nrestart [args ...]\n\n Restart the debugged Python program. If an argument is supplied,\n it is split with ``shlex`` and the result is used as the new\n ``sys.argv``. History, breakpoints, actions and debugger options\n are preserved. ``restart`` is an alias for ``run``.\n\nq(uit)\n\n Quit from the debugger. The program being executed is aborted.\n\n-[ Footnotes ]-\n\n[1] Whether a frame is considered to originate in a certain module is\n determined by the ``__name__`` in the frame globals.\n', 'del': '\nThe ``del`` statement\n*********************\n\n del_stmt ::= "del" target_list\n\nDeletion is recursively defined very similar to the way assignment is\ndefined. Rather than spelling it out in full details, here are some\nhints.\n\nDeletion of a target list recursively deletes each target, from left\nto right.\n\nDeletion of a name removes the binding of that name from the local or\nglobal namespace, depending on whether the name occurs in a ``global``\nstatement in the same code block. If the name is unbound, a\n``NameError`` exception will be raised.\n\nDeletion of attribute references, subscriptions and slicings is passed\nto the primary object involved; deletion of a slicing is in general\nequivalent to assignment of an empty slice of the right type (but even\nthis is determined by the sliced object).\n\nChanged in version 3.2.\n', 'dict': '\nDictionary displays\n*******************\n\nA dictionary display is a possibly empty series of key/datum pairs\nenclosed in curly braces:\n\n dict_display ::= "{" [key_datum_list | dict_comprehension] "}"\n key_datum_list ::= key_datum ("," key_datum)* [","]\n key_datum ::= expression ":" expression\n dict_comprehension ::= expression ":" expression comp_for\n\nA dictionary display yields a new dictionary object.\n\nIf a comma-separated sequence of key/datum pairs is given, they are\nevaluated from left to right to define the entries of the dictionary:\neach key object is used as a key into the dictionary to store the\ncorresponding datum. This means that you can specify the same key\nmultiple times in the key/datum list, and the final dictionary\'s value\nfor that key will be the last one given.\n\nA dict comprehension, in contrast to list and set comprehensions,\nneeds two expressions separated with a colon followed by the usual\n"for" and "if" clauses. When the comprehension is run, the resulting\nkey and value elements are inserted in the new dictionary in the order\nthey are produced.\n\nRestrictions on the types of the key values are listed earlier in\nsection *The standard type hierarchy*. (To summarize, the key type\nshould be *hashable*, which excludes all mutable objects.) Clashes\nbetween duplicate keys are not detected; the last datum (textually\nrightmost in the display) stored for a given key value prevails.\n', @@ -34,7 +34,7 @@ 'exprlists': '\nExpression lists\n****************\n\n expression_list ::= expression ( "," expression )* [","]\n\nAn expression list containing at least one comma yields a tuple. The\nlength of the tuple is the number of expressions in the list. The\nexpressions are evaluated from left to right.\n\nThe trailing comma is required only to create a single tuple (a.k.a. a\n*singleton*); it is optional in all other cases. A single expression\nwithout a trailing comma doesn\'t create a tuple, but rather yields the\nvalue of that expression. (To create an empty tuple, use an empty pair\nof parentheses: ``()``.)\n', 'floating': '\nFloating point literals\n***********************\n\nFloating point literals are described by the following lexical\ndefinitions:\n\n floatnumber ::= pointfloat | exponentfloat\n pointfloat ::= [intpart] fraction | intpart "."\n exponentfloat ::= (intpart | pointfloat) exponent\n intpart ::= digit+\n fraction ::= "." digit+\n exponent ::= ("e" | "E") ["+" | "-"] digit+\n\nNote that the integer and exponent parts are always interpreted using\nradix 10. For example, ``077e010`` is legal, and denotes the same\nnumber as ``77e10``. The allowed range of floating point literals is\nimplementation-dependent. Some examples of floating point literals:\n\n 3.14 10. .001 1e100 3.14e-10 0e0\n\nNote that numeric literals do not include a sign; a phrase like ``-1``\nis actually an expression composed of the unary operator ``-`` and the\nliteral ``1``.\n', 'for': '\nThe ``for`` statement\n*********************\n\nThe ``for`` statement is used to iterate over the elements of a\nsequence (such as a string, tuple or list) or other iterable object:\n\n for_stmt ::= "for" target_list "in" expression_list ":" suite\n ["else" ":" suite]\n\nThe expression list is evaluated once; it should yield an iterable\nobject. An iterator is created for the result of the\n``expression_list``. The suite is then executed once for each item\nprovided by the iterator, in the order of ascending indices. Each\nitem in turn is assigned to the target list using the standard rules\nfor assignments (see *Assignment statements*), and then the suite is\nexecuted. When the items are exhausted (which is immediately when the\nsequence is empty or an iterator raises a ``StopIteration``\nexception), the suite in the ``else`` clause, if present, is executed,\nand the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite. A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ncontinues with the next item, or with the ``else`` clause if there was\nno next item.\n\nThe suite may assign to the variable(s) in the target list; this does\nnot affect the next item assigned to it.\n\nNames in the target list are not deleted when the loop is finished,\nbut if the sequence is empty, it will not have been assigned to at all\nby the loop. Hint: the built-in function ``range()`` returns an\niterator of integers suitable to emulate the effect of Pascal\'s ``for\ni := a to b do``; e.g., ``list(range(3))`` returns the list ``[0, 1,\n2]``.\n\nNote: There is a subtlety when the sequence is being modified by the loop\n (this can only occur for mutable sequences, i.e. lists). An\n internal counter is used to keep track of which item is used next,\n and this is incremented on each iteration. When this counter has\n reached the length of the sequence the loop terminates. This means\n that if the suite deletes the current (or a previous) item from the\n sequence, the next item will be skipped (since it gets the index of\n the current item which has already been treated). Likewise, if the\n suite inserts an item in the sequence before the current item, the\n current item will be treated again the next time through the loop.\n This can lead to nasty bugs that can be avoided by making a\n temporary copy using a slice of the whole sequence, e.g.,\n\n for x in a[:]:\n if x < 0: a.remove(x)\n', - 'formatstrings': '\nFormat String Syntax\n********************\n\nThe ``str.format()`` method and the ``Formatter`` class share the same\nsyntax for format strings (although in the case of ``Formatter``,\nsubclasses can define their own format string syntax).\n\nFormat strings contain "replacement fields" surrounded by curly braces\n``{}``. Anything that is not contained in braces is considered literal\ntext, which is copied unchanged to the output. If you need to include\na brace character in the literal text, it can be escaped by doubling:\n``{{`` and ``}}``.\n\nThe grammar for a replacement field is as follows:\n\n replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"\n field_name ::= arg_name ("." attribute_name | "[" element_index "]")*\n arg_name ::= [identifier | integer]\n attribute_name ::= identifier\n element_index ::= integer | index_string\n index_string ::= +\n conversion ::= "r" | "s" | "a"\n format_spec ::= \n\nIn less formal terms, the replacement field can start with a\n*field_name* that specifies the object whose value is to be formatted\nand inserted into the output instead of the replacement field. The\n*field_name* is optionally followed by a *conversion* field, which is\npreceded by an exclamation point ``\'!\'``, and a *format_spec*, which\nis preceded by a colon ``\':\'``. These specify a non-default format\nfor the replacement value.\n\nSee also the *Format Specification Mini-Language* section.\n\nThe *field_name* itself begins with an *arg_name* that is either a\nnumber or a keyword. If it\'s a number, it refers to a positional\nargument, and if it\'s a keyword, it refers to a named keyword\nargument. If the numerical arg_names in a format string are 0, 1, 2,\n... in sequence, they can all be omitted (not just some) and the\nnumbers 0, 1, 2, ... will be automatically inserted in that order.\nBecause *arg_name* is not quote-delimited, it is not possible to\nspecify arbitrary dictionary keys (e.g., the strings ``\'10\'`` or\n``\':-]\'``) within a format string. The *arg_name* can be followed by\nany number of index or attribute expressions. An expression of the\nform ``\'.name\'`` selects the named attribute using ``getattr()``,\nwhile an expression of the form ``\'[index]\'`` does an index lookup\nusing ``__getitem__()``.\n\nChanged in version 3.1: The positional argument specifiers can be\nomitted, so ``\'{} {}\'`` is equivalent to ``\'{0} {1}\'``.\n\nSome simple format string examples:\n\n "First, thou shalt count to {0}" # References first positional argument\n "Bring me a {}" # Implicitly references the first positional argument\n "From {} to {}" # Same as "From {0} to {1}"\n "My quest is {name}" # References keyword argument \'name\'\n "Weight in tons {0.weight}" # \'weight\' attribute of first positional arg\n "Units destroyed: {players[0]}" # First element of keyword argument \'players\'.\n\nThe *conversion* field causes a type coercion before formatting.\nNormally, the job of formatting a value is done by the\n``__format__()`` method of the value itself. However, in some cases\nit is desirable to force a type to be formatted as a string,\noverriding its own definition of formatting. By converting the value\nto a string before calling ``__format__()``, the normal formatting\nlogic is bypassed.\n\nThree conversion flags are currently supported: ``\'!s\'`` which calls\n``str()`` on the value, ``\'!r\'`` which calls ``repr()`` and ``\'!a\'``\nwhich calls ``ascii()``.\n\nSome examples:\n\n "Harold\'s a clever {0!s}" # Calls str() on the argument first\n "Bring out the holy {name!r}" # Calls repr() on the argument first\n "More {!a}" # Calls ascii() on the argument first\n\nThe *format_spec* field contains a specification of how the value\nshould be presented, including such details as field width, alignment,\npadding, decimal precision and so on. Each value type can define its\nown "formatting mini-language" or interpretation of the *format_spec*.\n\nMost built-in types support a common formatting mini-language, which\nis described in the next section.\n\nA *format_spec* field can also include nested replacement fields\nwithin it. These nested replacement fields can contain only a field\nname; conversion flags and format specifications are not allowed. The\nreplacement fields within the format_spec are substituted before the\n*format_spec* string is interpreted. This allows the formatting of a\nvalue to be dynamically specified.\n\nSee the *Format examples* section for some examples.\n\n\nFormat Specification Mini-Language\n==================================\n\n"Format specifications" are used within replacement fields contained\nwithin a format string to define how individual values are presented\n(see *Format String Syntax*). They can also be passed directly to the\nbuilt-in ``format()`` function. Each formattable type may define how\nthe format specification is to be interpreted.\n\nMost built-in types implement the following options for format\nspecifications, although some of the formatting options are only\nsupported by the numeric types.\n\nA general convention is that an empty format string (``""``) produces\nthe same result as if you had called ``str()`` on the value. A non-\nempty format string typically modifies the result.\n\nThe general form of a *standard format specifier* is:\n\n format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]\n fill ::= \n align ::= "<" | ">" | "=" | "^"\n sign ::= "+" | "-" | " "\n width ::= integer\n precision ::= integer\n type ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n\nThe *fill* character can be any character other than \'{\' or \'}\'. The\npresence of a fill character is signaled by the character following\nit, which must be one of the alignment options. If the second\ncharacter of *format_spec* is not a valid alignment option, then it is\nassumed that both the fill character and the alignment option are\nabsent.\n\nThe meaning of the various alignment options is as follows:\n\n +-----------+------------------------------------------------------------+\n | Option | Meaning |\n +===========+============================================================+\n | ``\'<\'`` | Forces the field to be left-aligned within the available |\n | | space (this is the default for most objects). |\n +-----------+------------------------------------------------------------+\n | ``\'>\'`` | Forces the field to be right-aligned within the available |\n | | space (this is the default for numbers). |\n +-----------+------------------------------------------------------------+\n | ``\'=\'`` | Forces the padding to be placed after the sign (if any) |\n | | but before the digits. This is used for printing fields |\n | | in the form \'+000000120\'. This alignment option is only |\n | | valid for numeric types. |\n +-----------+------------------------------------------------------------+\n | ``\'^\'`` | Forces the field to be centered within the available |\n | | space. |\n +-----------+------------------------------------------------------------+\n\nNote that unless a minimum field width is defined, the field width\nwill always be the same size as the data to fill it, so that the\nalignment option has no meaning in this case.\n\nThe *sign* option is only valid for number types, and can be one of\nthe following:\n\n +-----------+------------------------------------------------------------+\n | Option | Meaning |\n +===========+============================================================+\n | ``\'+\'`` | indicates that a sign should be used for both positive as |\n | | well as negative numbers. |\n +-----------+------------------------------------------------------------+\n | ``\'-\'`` | indicates that a sign should be used only for negative |\n | | numbers (this is the default behavior). |\n +-----------+------------------------------------------------------------+\n | space | indicates that a leading space should be used on positive |\n | | numbers, and a minus sign on negative numbers. |\n +-----------+------------------------------------------------------------+\n\nThe ``\'#\'`` option causes the "alternate form" to be used for the\nconversion. The alternate form is defined differently for different\ntypes. This option is only valid for integer, float, complex and\nDecimal types. For integers, when binary, octal, or hexadecimal output\nis used, this option adds the prefix respective ``\'0b\'``, ``\'0o\'``, or\n``\'0x\'`` to the output value. For floats, complex and Decimal the\nalternate form causes the result of the conversion to always contain a\ndecimal-point character, even if no digits follow it. Normally, a\ndecimal-point character appears in the result of these conversions\nonly if a digit follows it. In addition, for ``\'g\'`` and ``\'G\'``\nconversions, trailing zeros are not removed from the result.\n\nThe ``\',\'`` option signals the use of a comma for a thousands\nseparator. For a locale aware separator, use the ``\'n\'`` integer\npresentation type instead.\n\nChanged in version 3.1: Added the ``\',\'`` option (see also **PEP\n378**).\n\n*width* is a decimal integer defining the minimum field width. If not\nspecified, then the field width will be determined by the content.\n\nIf the *width* field is preceded by a zero (``\'0\'``) character, this\nenables zero-padding. This is equivalent to an *alignment* type of\n``\'=\'`` and a *fill* character of ``\'0\'``.\n\nThe *precision* is a decimal number indicating how many digits should\nbe displayed after the decimal point for a floating point value\nformatted with ``\'f\'`` and ``\'F\'``, or before and after the decimal\npoint for a floating point value formatted with ``\'g\'`` or ``\'G\'``.\nFor non-number types the field indicates the maximum field size - in\nother words, how many characters will be used from the field content.\nThe *precision* is not allowed for integer values.\n\nFinally, the *type* determines how the data should be presented.\n\nThe available string presentation types are:\n\n +-----------+------------------------------------------------------------+\n | Type | Meaning |\n +===========+============================================================+\n | ``\'s\'`` | String format. This is the default type for strings and |\n | | may be omitted. |\n +-----------+------------------------------------------------------------+\n | None | The same as ``\'s\'``. |\n +-----------+------------------------------------------------------------+\n\nThe available integer presentation types are:\n\n +-----------+------------------------------------------------------------+\n | Type | Meaning |\n +===========+============================================================+\n | ``\'b\'`` | Binary format. Outputs the number in base 2. |\n +-----------+------------------------------------------------------------+\n | ``\'c\'`` | Character. Converts the integer to the corresponding |\n | | unicode character before printing. |\n +-----------+------------------------------------------------------------+\n | ``\'d\'`` | Decimal Integer. Outputs the number in base 10. |\n +-----------+------------------------------------------------------------+\n | ``\'o\'`` | Octal format. Outputs the number in base 8. |\n +-----------+------------------------------------------------------------+\n | ``\'x\'`` | Hex format. Outputs the number in base 16, using lower- |\n | | case letters for the digits above 9. |\n +-----------+------------------------------------------------------------+\n | ``\'X\'`` | Hex format. Outputs the number in base 16, using upper- |\n | | case letters for the digits above 9. |\n +-----------+------------------------------------------------------------+\n | ``\'n\'`` | Number. This is the same as ``\'d\'``, except that it uses |\n | | the current locale setting to insert the appropriate |\n | | number separator characters. |\n +-----------+------------------------------------------------------------+\n | None | The same as ``\'d\'``. |\n +-----------+------------------------------------------------------------+\n\nIn addition to the above presentation types, integers can be formatted\nwith the floating point presentation types listed below (except\n``\'n\'`` and None). When doing so, ``float()`` is used to convert the\ninteger to a floating point number before formatting.\n\nThe available presentation types for floating point and decimal values\nare:\n\n +-----------+------------------------------------------------------------+\n | Type | Meaning |\n +===========+============================================================+\n | ``\'e\'`` | Exponent notation. Prints the number in scientific |\n | | notation using the letter \'e\' to indicate the exponent. |\n +-----------+------------------------------------------------------------+\n | ``\'E\'`` | Exponent notation. Same as ``\'e\'`` except it uses an upper |\n | | case \'E\' as the separator character. |\n +-----------+------------------------------------------------------------+\n | ``\'f\'`` | Fixed point. Displays the number as a fixed-point number. |\n +-----------+------------------------------------------------------------+\n | ``\'F\'`` | Fixed point. Same as ``\'f\'``, but converts ``nan`` to |\n | | ``NAN`` and ``inf`` to ``INF``. |\n +-----------+------------------------------------------------------------+\n | ``\'g\'`` | General format. For a given precision ``p >= 1``, this |\n | | rounds the number to ``p`` significant digits and then |\n | | formats the result in either fixed-point format or in |\n | | scientific notation, depending on its magnitude. The |\n | | precise rules are as follows: suppose that the result |\n | | formatted with presentation type ``\'e\'`` and precision |\n | | ``p-1`` would have exponent ``exp``. Then if ``-4 <= exp |\n | | < p``, the number is formatted with presentation type |\n | | ``\'f\'`` and precision ``p-1-exp``. Otherwise, the number |\n | | is formatted with presentation type ``\'e\'`` and precision |\n | | ``p-1``. In both cases insignificant trailing zeros are |\n | | removed from the significand, and the decimal point is |\n | | also removed if there are no remaining digits following |\n | | it. Positive and negative infinity, positive and negative |\n | | zero, and nans, are formatted as ``inf``, ``-inf``, ``0``, |\n | | ``-0`` and ``nan`` respectively, regardless of the |\n | | precision. A precision of ``0`` is treated as equivalent |\n | | to a precision of ``1``. |\n +-----------+------------------------------------------------------------+\n | ``\'G\'`` | General format. Same as ``\'g\'`` except switches to ``\'E\'`` |\n | | if the number gets too large. The representations of |\n | | infinity and NaN are uppercased, too. |\n +-----------+------------------------------------------------------------+\n | ``\'n\'`` | Number. This is the same as ``\'g\'``, except that it uses |\n | | the current locale setting to insert the appropriate |\n | | number separator characters. |\n +-----------+------------------------------------------------------------+\n | ``\'%\'`` | Percentage. Multiplies the number by 100 and displays in |\n | | fixed (``\'f\'``) format, followed by a percent sign. |\n +-----------+------------------------------------------------------------+\n | None | Similar to ``\'g\'``, except with at least one digit past |\n | | the decimal point and a default precision of 12. This is |\n | | intended to match ``str()``, except you can add the other |\n | | format modifiers. |\n +-----------+------------------------------------------------------------+\n\n\nFormat examples\n===============\n\nThis section contains examples of the new format syntax and comparison\nwith the old ``%``-formatting.\n\nIn most of the cases the syntax is similar to the old\n``%``-formatting, with the addition of the ``{}`` and with ``:`` used\ninstead of ``%``. For example, ``\'%03.2f\'`` can be translated to\n``\'{:03.2f}\'``.\n\nThe new format syntax also supports new and different options, shown\nin the follow examples.\n\nAccessing arguments by position:\n\n >>> \'{0}, {1}, {2}\'.format(\'a\', \'b\', \'c\')\n \'a, b, c\'\n >>> \'{}, {}, {}\'.format(\'a\', \'b\', \'c\') # 3.1+ only\n \'a, b, c\'\n >>> \'{2}, {1}, {0}\'.format(\'a\', \'b\', \'c\')\n \'c, b, a\'\n >>> \'{2}, {1}, {0}\'.format(*\'abc\') # unpacking argument sequence\n \'c, b, a\'\n >>> \'{0}{1}{0}\'.format(\'abra\', \'cad\') # arguments\' indices can be repeated\n \'abracadabra\'\n\nAccessing arguments by name:\n\n >>> \'Coordinates: {latitude}, {longitude}\'.format(latitude=\'37.24N\', longitude=\'-115.81W\')\n \'Coordinates: 37.24N, -115.81W\'\n >>> coord = {\'latitude\': \'37.24N\', \'longitude\': \'-115.81W\'}\n >>> \'Coordinates: {latitude}, {longitude}\'.format(**coord)\n \'Coordinates: 37.24N, -115.81W\'\n\nAccessing arguments\' attributes:\n\n >>> c = 3-5j\n >>> (\'The complex number {0} is formed from the real part {0.real} \'\n ... \'and the imaginary part {0.imag}.\').format(c)\n \'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.\'\n >>> class Point:\n ... def __init__(self, x, y):\n ... self.x, self.y = x, y\n ... def __str__(self):\n ... return \'Point({self.x}, {self.y})\'.format(self=self)\n ...\n >>> str(Point(4, 2))\n \'Point(4, 2)\'\n\nAccessing arguments\' items:\n\n >>> coord = (3, 5)\n >>> \'X: {0[0]}; Y: {0[1]}\'.format(coord)\n \'X: 3; Y: 5\'\n\nReplacing ``%s`` and ``%r``:\n\n >>> "repr() shows quotes: {!r}; str() doesn\'t: {!s}".format(\'test1\', \'test2\')\n "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n\nAligning the text and specifying a width:\n\n >>> \'{:<30}\'.format(\'left aligned\')\n \'left aligned \'\n >>> \'{:>30}\'.format(\'right aligned\')\n \' right aligned\'\n >>> \'{:^30}\'.format(\'centered\')\n \' centered \'\n >>> \'{:*^30}\'.format(\'centered\') # use \'*\' as a fill char\n \'***********centered***********\'\n\nReplacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:\n\n >>> \'{:+f}; {:+f}\'.format(3.14, -3.14) # show it always\n \'+3.140000; -3.140000\'\n >>> \'{: f}; {: f}\'.format(3.14, -3.14) # show a space for positive numbers\n \' 3.140000; -3.140000\'\n >>> \'{:-f}; {:-f}\'.format(3.14, -3.14) # show only the minus -- same as \'{:f}; {:f}\'\n \'3.140000; -3.140000\'\n\nReplacing ``%x`` and ``%o`` and converting the value to different\nbases:\n\n >>> # format also supports binary numbers\n >>> "int: {0:d}; hex: {0:x}; oct: {0:o}; bin: {0:b}".format(42)\n \'int: 42; hex: 2a; oct: 52; bin: 101010\'\n >>> # with 0x, 0o, or 0b as prefix:\n >>> "int: {0:d}; hex: {0:#x}; oct: {0:#o}; bin: {0:#b}".format(42)\n \'int: 42; hex: 0x2a; oct: 0o52; bin: 0b101010\'\n\nUsing the comma as a thousands separator:\n\n >>> \'{:,}\'.format(1234567890)\n \'1,234,567,890\'\n\nExpressing a percentage:\n\n >>> points = 19\n >>> total = 22\n >>> \'Correct answers: {:.2%}\'.format(points/total)\n \'Correct answers: 86.36%\'\n\nUsing type-specific formatting:\n\n >>> import datetime\n >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n >>> \'{:%Y-%m-%d %H:%M:%S}\'.format(d)\n \'2010-07-04 12:15:58\'\n\nNesting arguments and more complex examples:\n\n >>> for align, text in zip(\'<^>\', [\'left\', \'center\', \'right\']):\n ... \'{0:{fill}{align}16}\'.format(text, fill=align, align=align)\n ...\n \'left<<<<<<<<<<<<\'\n \'^^^^^center^^^^^\'\n \'>>>>>>>>>>>right\'\n >>>\n >>> octets = [192, 168, 0, 1]\n >>> \'{:02X}{:02X}{:02X}{:02X}\'.format(*octets)\n \'C0A80001\'\n >>> int(_, 16)\n 3232235521\n >>>\n >>> width = 5\n >>> for num in range(5,12):\n ... for base in \'dXob\':\n ... print(\'{0:{width}{base}}\'.format(num, base=base, width=width), end=\' \')\n ... print()\n ...\n 5 5 5 101\n 6 6 6 110\n 7 7 7 111\n 8 8 10 1000\n 9 9 11 1001\n 10 A 12 1010\n 11 B 13 1011\n', + 'formatstrings': '\nFormat String Syntax\n********************\n\nThe ``str.format()`` method and the ``Formatter`` class share the same\nsyntax for format strings (although in the case of ``Formatter``,\nsubclasses can define their own format string syntax).\n\nFormat strings contain "replacement fields" surrounded by curly braces\n``{}``. Anything that is not contained in braces is considered literal\ntext, which is copied unchanged to the output. If you need to include\na brace character in the literal text, it can be escaped by doubling:\n``{{`` and ``}}``.\n\nThe grammar for a replacement field is as follows:\n\n replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}"\n field_name ::= arg_name ("." attribute_name | "[" element_index "]")*\n arg_name ::= [identifier | integer]\n attribute_name ::= identifier\n element_index ::= integer | index_string\n index_string ::= +\n conversion ::= "r" | "s" | "a"\n format_spec ::= \n\nIn less formal terms, the replacement field can start with a\n*field_name* that specifies the object whose value is to be formatted\nand inserted into the output instead of the replacement field. The\n*field_name* is optionally followed by a *conversion* field, which is\npreceded by an exclamation point ``\'!\'``, and a *format_spec*, which\nis preceded by a colon ``\':\'``. These specify a non-default format\nfor the replacement value.\n\nSee also the *Format Specification Mini-Language* section.\n\nThe *field_name* itself begins with an *arg_name* that is either a\nnumber or a keyword. If it\'s a number, it refers to a positional\nargument, and if it\'s a keyword, it refers to a named keyword\nargument. If the numerical arg_names in a format string are 0, 1, 2,\n... in sequence, they can all be omitted (not just some) and the\nnumbers 0, 1, 2, ... will be automatically inserted in that order.\nBecause *arg_name* is not quote-delimited, it is not possible to\nspecify arbitrary dictionary keys (e.g., the strings ``\'10\'`` or\n``\':-]\'``) within a format string. The *arg_name* can be followed by\nany number of index or attribute expressions. An expression of the\nform ``\'.name\'`` selects the named attribute using ``getattr()``,\nwhile an expression of the form ``\'[index]\'`` does an index lookup\nusing ``__getitem__()``.\n\nChanged in version 3.1: The positional argument specifiers can be\nomitted, so ``\'{} {}\'`` is equivalent to ``\'{0} {1}\'``.\n\nSome simple format string examples:\n\n "First, thou shalt count to {0}" # References first positional argument\n "Bring me a {}" # Implicitly references the first positional argument\n "From {} to {}" # Same as "From {0} to {1}"\n "My quest is {name}" # References keyword argument \'name\'\n "Weight in tons {0.weight}" # \'weight\' attribute of first positional arg\n "Units destroyed: {players[0]}" # First element of keyword argument \'players\'.\n\nThe *conversion* field causes a type coercion before formatting.\nNormally, the job of formatting a value is done by the\n``__format__()`` method of the value itself. However, in some cases\nit is desirable to force a type to be formatted as a string,\noverriding its own definition of formatting. By converting the value\nto a string before calling ``__format__()``, the normal formatting\nlogic is bypassed.\n\nThree conversion flags are currently supported: ``\'!s\'`` which calls\n``str()`` on the value, ``\'!r\'`` which calls ``repr()`` and ``\'!a\'``\nwhich calls ``ascii()``.\n\nSome examples:\n\n "Harold\'s a clever {0!s}" # Calls str() on the argument first\n "Bring out the holy {name!r}" # Calls repr() on the argument first\n "More {!a}" # Calls ascii() on the argument first\n\nThe *format_spec* field contains a specification of how the value\nshould be presented, including such details as field width, alignment,\npadding, decimal precision and so on. Each value type can define its\nown "formatting mini-language" or interpretation of the *format_spec*.\n\nMost built-in types support a common formatting mini-language, which\nis described in the next section.\n\nA *format_spec* field can also include nested replacement fields\nwithin it. These nested replacement fields can contain only a field\nname; conversion flags and format specifications are not allowed. The\nreplacement fields within the format_spec are substituted before the\n*format_spec* string is interpreted. This allows the formatting of a\nvalue to be dynamically specified.\n\nSee the *Format examples* section for some examples.\n\n\nFormat Specification Mini-Language\n==================================\n\n"Format specifications" are used within replacement fields contained\nwithin a format string to define how individual values are presented\n(see *Format String Syntax*). They can also be passed directly to the\nbuilt-in ``format()`` function. Each formattable type may define how\nthe format specification is to be interpreted.\n\nMost built-in types implement the following options for format\nspecifications, although some of the formatting options are only\nsupported by the numeric types.\n\nA general convention is that an empty format string (``""``) produces\nthe same result as if you had called ``str()`` on the value. A non-\nempty format string typically modifies the result.\n\nThe general form of a *standard format specifier* is:\n\n format_spec ::= [[fill]align][sign][#][0][width][,][.precision][type]\n fill ::= \n align ::= "<" | ">" | "=" | "^"\n sign ::= "+" | "-" | " "\n width ::= integer\n precision ::= integer\n type ::= "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"\n\nThe *fill* character can be any character other than \'{\' or \'}\'. The\npresence of a fill character is signaled by the character following\nit, which must be one of the alignment options. If the second\ncharacter of *format_spec* is not a valid alignment option, then it is\nassumed that both the fill character and the alignment option are\nabsent.\n\nThe meaning of the various alignment options is as follows:\n\n +-----------+------------------------------------------------------------+\n | Option | Meaning |\n +===========+============================================================+\n | ``\'<\'`` | Forces the field to be left-aligned within the available |\n | | space (this is the default for most objects). |\n +-----------+------------------------------------------------------------+\n | ``\'>\'`` | Forces the field to be right-aligned within the available |\n | | space (this is the default for numbers). |\n +-----------+------------------------------------------------------------+\n | ``\'=\'`` | Forces the padding to be placed after the sign (if any) |\n | | but before the digits. This is used for printing fields |\n | | in the form \'+000000120\'. This alignment option is only |\n | | valid for numeric types. |\n +-----------+------------------------------------------------------------+\n | ``\'^\'`` | Forces the field to be centered within the available |\n | | space. |\n +-----------+------------------------------------------------------------+\n\nNote that unless a minimum field width is defined, the field width\nwill always be the same size as the data to fill it, so that the\nalignment option has no meaning in this case.\n\nThe *sign* option is only valid for number types, and can be one of\nthe following:\n\n +-----------+------------------------------------------------------------+\n | Option | Meaning |\n +===========+============================================================+\n | ``\'+\'`` | indicates that a sign should be used for both positive as |\n | | well as negative numbers. |\n +-----------+------------------------------------------------------------+\n | ``\'-\'`` | indicates that a sign should be used only for negative |\n | | numbers (this is the default behavior). |\n +-----------+------------------------------------------------------------+\n | space | indicates that a leading space should be used on positive |\n | | numbers, and a minus sign on negative numbers. |\n +-----------+------------------------------------------------------------+\n\nThe ``\'#\'`` option causes the "alternate form" to be used for the\nconversion. The alternate form is defined differently for different\ntypes. This option is only valid for integer, float, complex and\nDecimal types. For integers, when binary, octal, or hexadecimal output\nis used, this option adds the prefix respective ``\'0b\'``, ``\'0o\'``, or\n``\'0x\'`` to the output value. For floats, complex and Decimal the\nalternate form causes the result of the conversion to always contain a\ndecimal-point character, even if no digits follow it. Normally, a\ndecimal-point character appears in the result of these conversions\nonly if a digit follows it. In addition, for ``\'g\'`` and ``\'G\'``\nconversions, trailing zeros are not removed from the result.\n\nThe ``\',\'`` option signals the use of a comma for a thousands\nseparator. For a locale aware separator, use the ``\'n\'`` integer\npresentation type instead.\n\nChanged in version 3.1: Added the ``\',\'`` option (see also **PEP\n378**).\n\n*width* is a decimal integer defining the minimum field width. If not\nspecified, then the field width will be determined by the content.\n\nIf the *width* field is preceded by a zero (``\'0\'``) character, this\nenables zero-padding. This is equivalent to an *alignment* type of\n``\'=\'`` and a *fill* character of ``\'0\'``.\n\nThe *precision* is a decimal number indicating how many digits should\nbe displayed after the decimal point for a floating point value\nformatted with ``\'f\'`` and ``\'F\'``, or before and after the decimal\npoint for a floating point value formatted with ``\'g\'`` or ``\'G\'``.\nFor non-number types the field indicates the maximum field size - in\nother words, how many characters will be used from the field content.\nThe *precision* is not allowed for integer values.\n\nFinally, the *type* determines how the data should be presented.\n\nThe available string presentation types are:\n\n +-----------+------------------------------------------------------------+\n | Type | Meaning |\n +===========+============================================================+\n | ``\'s\'`` | String format. This is the default type for strings and |\n | | may be omitted. |\n +-----------+------------------------------------------------------------+\n | None | The same as ``\'s\'``. |\n +-----------+------------------------------------------------------------+\n\nThe available integer presentation types are:\n\n +-----------+------------------------------------------------------------+\n | Type | Meaning |\n +===========+============================================================+\n | ``\'b\'`` | Binary format. Outputs the number in base 2. |\n +-----------+------------------------------------------------------------+\n | ``\'c\'`` | Character. Converts the integer to the corresponding |\n | | unicode character before printing. |\n +-----------+------------------------------------------------------------+\n | ``\'d\'`` | Decimal Integer. Outputs the number in base 10. |\n +-----------+------------------------------------------------------------+\n | ``\'o\'`` | Octal format. Outputs the number in base 8. |\n +-----------+------------------------------------------------------------+\n | ``\'x\'`` | Hex format. Outputs the number in base 16, using lower- |\n | | case letters for the digits above 9. |\n +-----------+------------------------------------------------------------+\n | ``\'X\'`` | Hex format. Outputs the number in base 16, using upper- |\n | | case letters for the digits above 9. |\n +-----------+------------------------------------------------------------+\n | ``\'n\'`` | Number. This is the same as ``\'d\'``, except that it uses |\n | | the current locale setting to insert the appropriate |\n | | number separator characters. |\n +-----------+------------------------------------------------------------+\n | None | The same as ``\'d\'``. |\n +-----------+------------------------------------------------------------+\n\nIn addition to the above presentation types, integers can be formatted\nwith the floating point presentation types listed below (except\n``\'n\'`` and None). When doing so, ``float()`` is used to convert the\ninteger to a floating point number before formatting.\n\nThe available presentation types for floating point and decimal values\nare:\n\n +-----------+------------------------------------------------------------+\n | Type | Meaning |\n +===========+============================================================+\n | ``\'e\'`` | Exponent notation. Prints the number in scientific |\n | | notation using the letter \'e\' to indicate the exponent. |\n +-----------+------------------------------------------------------------+\n | ``\'E\'`` | Exponent notation. Same as ``\'e\'`` except it uses an upper |\n | | case \'E\' as the separator character. |\n +-----------+------------------------------------------------------------+\n | ``\'f\'`` | Fixed point. Displays the number as a fixed-point number. |\n +-----------+------------------------------------------------------------+\n | ``\'F\'`` | Fixed point. Same as ``\'f\'``, but converts ``nan`` to |\n | | ``NAN`` and ``inf`` to ``INF``. |\n +-----------+------------------------------------------------------------+\n | ``\'g\'`` | General format. For a given precision ``p >= 1``, this |\n | | rounds the number to ``p`` significant digits and then |\n | | formats the result in either fixed-point format or in |\n | | scientific notation, depending on its magnitude. The |\n | | precise rules are as follows: suppose that the result |\n | | formatted with presentation type ``\'e\'`` and precision |\n | | ``p-1`` would have exponent ``exp``. Then if ``-4 <= exp |\n | | < p``, the number is formatted with presentation type |\n | | ``\'f\'`` and precision ``p-1-exp``. Otherwise, the number |\n | | is formatted with presentation type ``\'e\'`` and precision |\n | | ``p-1``. In both cases insignificant trailing zeros are |\n | | removed from the significand, and the decimal point is |\n | | also removed if there are no remaining digits following |\n | | it. Positive and negative infinity, positive and negative |\n | | zero, and nans, are formatted as ``inf``, ``-inf``, ``0``, |\n | | ``-0`` and ``nan`` respectively, regardless of the |\n | | precision. A precision of ``0`` is treated as equivalent |\n | | to a precision of ``1``. |\n +-----------+------------------------------------------------------------+\n | ``\'G\'`` | General format. Same as ``\'g\'`` except switches to ``\'E\'`` |\n | | if the number gets too large. The representations of |\n | | infinity and NaN are uppercased, too. |\n +-----------+------------------------------------------------------------+\n | ``\'n\'`` | Number. This is the same as ``\'g\'``, except that it uses |\n | | the current locale setting to insert the appropriate |\n | | number separator characters. |\n +-----------+------------------------------------------------------------+\n | ``\'%\'`` | Percentage. Multiplies the number by 100 and displays in |\n | | fixed (``\'f\'``) format, followed by a percent sign. |\n +-----------+------------------------------------------------------------+\n | None | Similar to ``\'g\'``, except with at least one digit past |\n | | the decimal point and a default precision of 12. This is |\n | | intended to match ``str()``, except you can add the other |\n | | format modifiers. |\n +-----------+------------------------------------------------------------+\n\n\nFormat examples\n===============\n\nThis section contains examples of the new format syntax and comparison\nwith the old ``%``-formatting.\n\nIn most of the cases the syntax is similar to the old\n``%``-formatting, with the addition of the ``{}`` and with ``:`` used\ninstead of ``%``. For example, ``\'%03.2f\'`` can be translated to\n``\'{:03.2f}\'``.\n\nThe new format syntax also supports new and different options, shown\nin the follow examples.\n\nAccessing arguments by position:\n\n >>> \'{0}, {1}, {2}\'.format(\'a\', \'b\', \'c\')\n \'a, b, c\'\n >>> \'{}, {}, {}\'.format(\'a\', \'b\', \'c\') # 3.1+ only\n \'a, b, c\'\n >>> \'{2}, {1}, {0}\'.format(\'a\', \'b\', \'c\')\n \'c, b, a\'\n >>> \'{2}, {1}, {0}\'.format(*\'abc\') # unpacking argument sequence\n \'c, b, a\'\n >>> \'{0}{1}{0}\'.format(\'abra\', \'cad\') # arguments\' indices can be repeated\n \'abracadabra\'\n\nAccessing arguments by name:\n\n >>> \'Coordinates: {latitude}, {longitude}\'.format(latitude=\'37.24N\', longitude=\'-115.81W\')\n \'Coordinates: 37.24N, -115.81W\'\n >>> coord = {\'latitude\': \'37.24N\', \'longitude\': \'-115.81W\'}\n >>> \'Coordinates: {latitude}, {longitude}\'.format(**coord)\n \'Coordinates: 37.24N, -115.81W\'\n\nAccessing arguments\' attributes:\n\n >>> c = 3-5j\n >>> (\'The complex number {0} is formed from the real part {0.real} \'\n ... \'and the imaginary part {0.imag}.\').format(c)\n \'The complex number (3-5j) is formed from the real part 3.0 and the imaginary part -5.0.\'\n >>> class Point:\n ... def __init__(self, x, y):\n ... self.x, self.y = x, y\n ... def __str__(self):\n ... return \'Point({self.x}, {self.y})\'.format(self=self)\n ...\n >>> str(Point(4, 2))\n \'Point(4, 2)\'\n\nAccessing arguments\' items:\n\n >>> coord = (3, 5)\n >>> \'X: {0[0]}; Y: {0[1]}\'.format(coord)\n \'X: 3; Y: 5\'\n\nReplacing ``%s`` and ``%r``:\n\n >>> "repr() shows quotes: {!r}; str() doesn\'t: {!s}".format(\'test1\', \'test2\')\n "repr() shows quotes: \'test1\'; str() doesn\'t: test2"\n\nAligning the text and specifying a width:\n\n >>> \'{:<30}\'.format(\'left aligned\')\n \'left aligned \'\n >>> \'{:>30}\'.format(\'right aligned\')\n \' right aligned\'\n >>> \'{:^30}\'.format(\'centered\')\n \' centered \'\n >>> \'{:*^30}\'.format(\'centered\') # use \'*\' as a fill char\n \'***********centered***********\'\n\nReplacing ``%+f``, ``%-f``, and ``% f`` and specifying a sign:\n\n >>> \'{:+f}; {:+f}\'.format(3.14, -3.14) # show it always\n \'+3.140000; -3.140000\'\n >>> \'{: f}; {: f}\'.format(3.14, -3.14) # show a space for positive numbers\n \' 3.140000; -3.140000\'\n >>> \'{:-f}; {:-f}\'.format(3.14, -3.14) # show only the minus -- same as \'{:f}; {:f}\'\n \'3.140000; -3.140000\'\n\nReplacing ``%x`` and ``%o`` and converting the value to different\nbases:\n\n >>> # format also supports binary numbers\n >>> "int: {0:d}; hex: {0:x}; oct: {0:o}; bin: {0:b}".format(42)\n \'int: 42; hex: 2a; oct: 52; bin: 101010\'\n >>> # with 0x, 0o, or 0b as prefix:\n >>> "int: {0:d}; hex: {0:#x}; oct: {0:#o}; bin: {0:#b}".format(42)\n \'int: 42; hex: 0x2a; oct: 0o52; bin: 0b101010\'\n\nUsing the comma as a thousands separator:\n\n >>> \'{:,}\'.format(1234567890)\n \'1,234,567,890\'\n\nExpressing a percentage:\n\n >>> points = 19\n >>> total = 22\n >>> \'Correct answers: {:.2%}\'.format(points/total)\n \'Correct answers: 86.36%\'\n\nUsing type-specific formatting:\n\n >>> import datetime\n >>> d = datetime.datetime(2010, 7, 4, 12, 15, 58)\n >>> \'{:%Y-%m-%d %H:%M:%S}\'.format(d)\n \'2010-07-04 12:15:58\'\n\nNesting arguments and more complex examples:\n\n >>> for align, text in zip(\'<^>\', [\'left\', \'center\', \'right\']):\n ... \'{0:{fill}{align}16}\'.format(text, fill=align, align=align)\n ...\n \'left<<<<<<<<<<<<\'\n \'^^^^^center^^^^^\'\n \'>>>>>>>>>>>right\'\n >>>\n >>> octets = [192, 168, 0, 1]\n >>> \'{:02X}{:02X}{:02X}{:02X}\'.format(*octets)\n \'C0A80001\'\n >>> int(_, 16)\n 3232235521\n >>>\n >>> width = 5\n >>> for num in range(5,12):\n ... for base in \'dXob\':\n ... print(\'{0:{width}{base}}\'.format(num, base=base, width=width), end=\' \')\n ... print()\n ...\n 5 5 5 101\n 6 6 6 110\n 7 7 7 111\n 8 8 10 1000\n 9 9 11 1001\n 10 A 12 1010\n 11 B 13 1011\n', 'function': '\nFunction definitions\n********************\n\nA function definition defines a user-defined function object (see\nsection *The standard type hierarchy*):\n\n funcdef ::= [decorators] "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite\n decorators ::= decorator+\n decorator ::= "@" dotted_name ["(" [parameter_list [","]] ")"] NEWLINE\n dotted_name ::= identifier ("." identifier)*\n parameter_list ::= (defparameter ",")*\n ( "*" [parameter] ("," defparameter)*\n [, "**" parameter]\n | "**" parameter\n | defparameter [","] )\n parameter ::= identifier [":" expression]\n defparameter ::= parameter ["=" expression]\n funcname ::= identifier\n\nA function definition is an executable statement. Its execution binds\nthe function name in the current local namespace to a function object\n(a wrapper around the executable code for the function). This\nfunction object contains a reference to the current global namespace\nas the global namespace to be used when the function is called.\n\nThe function definition does not execute the function body; this gets\nexecuted only when the function is called. [3]\n\nA function definition may be wrapped by one or more *decorator*\nexpressions. Decorator expressions are evaluated when the function is\ndefined, in the scope that contains the function definition. The\nresult must be a callable, which is invoked with the function object\nas the only argument. The returned value is bound to the function name\ninstead of the function object. Multiple decorators are applied in\nnested fashion. For example, the following code\n\n @f1(arg)\n @f2\n def func(): pass\n\nis equivalent to\n\n def func(): pass\n func = f1(arg)(f2(func))\n\nWhen one or more parameters have the form *parameter* ``=``\n*expression*, the function is said to have "default parameter values."\nFor a parameter with a default value, the corresponding argument may\nbe omitted from a call, in which case the parameter\'s default value is\nsubstituted. If a parameter has a default value, all following\nparameters up until the "``*``" must also have a default value ---\nthis is a syntactic restriction that is not expressed by the grammar.\n\n**Default parameter values are evaluated when the function definition\nis executed.** This means that the expression is evaluated once, when\nthe function is defined, and that the same "pre-computed" value is\nused for each call. This is especially important to understand when a\ndefault parameter is a mutable object, such as a list or a dictionary:\nif the function modifies the object (e.g. by appending an item to a\nlist), the default value is in effect modified. This is generally not\nwhat was intended. A way around this is to use ``None`` as the\ndefault, and explicitly test for it in the body of the function, e.g.:\n\n def whats_on_the_telly(penguin=None):\n if penguin is None:\n penguin = []\n penguin.append("property of the zoo")\n return penguin\n\nFunction call semantics are described in more detail in section\n*Calls*. A function call always assigns values to all parameters\nmentioned in the parameter list, either from position arguments, from\nkeyword arguments, or from default values. If the form\n"``*identifier``" is present, it is initialized to a tuple receiving\nany excess positional parameters, defaulting to the empty tuple. If\nthe form "``**identifier``" is present, it is initialized to a new\ndictionary receiving any excess keyword arguments, defaulting to a new\nempty dictionary. Parameters after "``*``" or "``*identifier``" are\nkeyword-only parameters and may only be passed used keyword arguments.\n\nParameters may have annotations of the form "``: expression``"\nfollowing the parameter name. Any parameter may have an annotation\neven those of the form ``*identifier`` or ``**identifier``. Functions\nmay have "return" annotation of the form "``-> expression``" after the\nparameter list. These annotations can be any valid Python expression\nand are evaluated when the function definition is executed.\nAnnotations may be evaluated in a different order than they appear in\nthe source code. The presence of annotations does not change the\nsemantics of a function. The annotation values are available as\nvalues of a dictionary keyed by the parameters\' names in the\n``__annotations__`` attribute of the function object.\n\nIt is also possible to create anonymous functions (functions not bound\nto a name), for immediate use in expressions. This uses lambda forms,\ndescribed in section *Lambdas*. Note that the lambda form is merely a\nshorthand for a simplified function definition; a function defined in\na "``def``" statement can be passed around or assigned to another name\njust like a function defined by a lambda form. The "``def``" form is\nactually more powerful since it allows the execution of multiple\nstatements and annotations.\n\n**Programmer\'s note:** Functions are first-class objects. A "``def``"\nform executed inside a function definition defines a local function\nthat can be returned or passed around. Free variables used in the\nnested function can access the local variables of the function\ncontaining the def. See section *Naming and binding* for details.\n\nSee also:\n\n **PEP 3107** - Function Annotations\n The original specification for function annotations.\n', 'global': '\nThe ``global`` statement\n************************\n\n global_stmt ::= "global" identifier ("," identifier)*\n\nThe ``global`` statement is a declaration which holds for the entire\ncurrent code block. It means that the listed identifiers are to be\ninterpreted as globals. It would be impossible to assign to a global\nvariable without ``global``, although free variables may refer to\nglobals without being declared global.\n\nNames listed in a ``global`` statement must not be used in the same\ncode block textually preceding that ``global`` statement.\n\nNames listed in a ``global`` statement must not be defined as formal\nparameters or in a ``for`` loop control target, ``class`` definition,\nfunction definition, or ``import`` statement.\n\n**CPython implementation detail:** The current implementation does not\nenforce the latter two restrictions, but programs should not abuse\nthis freedom, as future implementations may enforce them or silently\nchange the meaning of the program.\n\n**Programmer\'s note:** the ``global`` is a directive to the parser.\nIt applies only to code parsed at the same time as the ``global``\nstatement. In particular, a ``global`` statement contained in a string\nor code object supplied to the built-in ``exec()`` function does not\naffect the code block *containing* the function call, and code\ncontained in such a string is unaffected by ``global`` statements in\nthe code containing the function call. The same applies to the\n``eval()`` and ``compile()`` functions.\n', 'id-classes': '\nReserved classes of identifiers\n*******************************\n\nCertain classes of identifiers (besides keywords) have special\nmeanings. These classes are identified by the patterns of leading and\ntrailing underscore characters:\n\n``_*``\n Not imported by ``from module import *``. The special identifier\n ``_`` is used in the interactive interpreter to store the result of\n the last evaluation; it is stored in the ``builtins`` module. When\n not in interactive mode, ``_`` has no special meaning and is not\n defined. See section *The import statement*.\n\n Note: The name ``_`` is often used in conjunction with\n internationalization; refer to the documentation for the\n ``gettext`` module for more information on this convention.\n\n``__*__``\n System-defined names. These names are defined by the interpreter\n and its implementation (including the standard library). Current\n system names are discussed in the *Special method names* section\n and elsewhere. More will likely be defined in future versions of\n Python. *Any* use of ``__*__`` names, in any context, that does\n not follow explicitly documented use, is subject to breakage\n without warning.\n\n``__*``\n Class-private names. Names in this category, when used within the\n context of a class definition, are re-written to use a mangled form\n to help avoid name clashes between "private" attributes of base and\n derived classes. See section *Identifiers (Names)*.\n', @@ -50,7 +50,7 @@ 'nonlocal': '\nThe ``nonlocal`` statement\n**************************\n\n nonlocal_stmt ::= "nonlocal" identifier ("," identifier)*\n\nThe ``nonlocal`` statement causes the listed identifiers to refer to\npreviously bound variables in the nearest enclosing scope. This is\nimportant because the default behavior for binding is to search the\nlocal namespace first. The statement allows encapsulated code to\nrebind variables outside of the local scope besides the global\n(module) scope.\n\nNames listed in a ``nonlocal`` statement, unlike to those listed in a\n``global`` statement, must refer to pre-existing bindings in an\nenclosing scope (the scope in which a new binding should be created\ncannot be determined unambiguously).\n\nNames listed in a ``nonlocal`` statement must not collide with pre-\nexisting bindings in the local scope.\n\nSee also:\n\n **PEP 3104** - Access to Names in Outer Scopes\n The specification for the ``nonlocal`` statement.\n', 'numbers': "\nNumeric literals\n****************\n\nThere are three types of numeric literals: integers, floating point\nnumbers, and imaginary numbers. There are no complex literals\n(complex numbers can be formed by adding a real number and an\nimaginary number).\n\nNote that numeric literals do not include a sign; a phrase like ``-1``\nis actually an expression composed of the unary operator '``-``' and\nthe literal ``1``.\n", 'numeric-types': "\nEmulating numeric types\n***********************\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations (``+``, ``-``, ``*``, ``/``, ``//``, ``%``,\n ``divmod()``, ``pow()``, ``**``, ``<<``, ``>>``, ``&``, ``^``,\n ``|``). For instance, to evaluate the expression ``x + y``, where\n *x* is an instance of a class that has an ``__add__()`` method,\n ``x.__add__(y)`` is called. The ``__divmod__()`` method should be\n the equivalent to using ``__floordiv__()`` and ``__mod__()``; it\n should not be related to ``__truediv__()``. Note that\n ``__pow__()`` should be defined to accept an optional third\n argument if the ternary version of the built-in ``pow()`` function\n is to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return ``NotImplemented``.\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations (``+``, ``-``, ``*``, ``/``, ``//``, ``%``,\n ``divmod()``, ``pow()``, ``**``, ``<<``, ``>>``, ``&``, ``^``,\n ``|``) with reflected (swapped) operands. These functions are only\n called if the left operand does not support the corresponding\n operation and the operands are of different types. [2] For\n instance, to evaluate the expression ``x - y``, where *y* is an\n instance of a class that has an ``__rsub__()`` method,\n ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns\n *NotImplemented*.\n\n Note that ternary ``pow()`` will not try calling ``__rpow__()``\n (the coercion rules would become too complicated).\n\n Note: If the right operand's type is a subclass of the left operand's\n type and that subclass provides the reflected method for the\n operation, this method will be called before the left operand's\n non-reflected method. This behavior allows subclasses to\n override their ancestors' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``,\n ``**=``, ``<<=``, ``>>=``, ``&=``, ``^=``, ``|=``). These methods\n should attempt to do the operation in-place (modifying *self*) and\n return the result (which could be, but does not have to be,\n *self*). If a specific method is not defined, the augmented\n assignment falls back to the normal methods. For instance, to\n execute the statement ``x += y``, where *x* is an instance of a\n class that has an ``__iadd__()`` method, ``x.__iadd__(y)`` is\n called. If *x* is an instance of a class that does not define a\n ``__iadd__()`` method, ``x.__add__(y)`` and ``y.__radd__(x)`` are\n considered, as with the evaluation of ``x + y``.\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations (``-``, ``+``,\n ``abs()`` and ``~``).\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions ``complex()``,\n ``int()``, ``float()`` and ``round()``. Should return a value of\n the appropriate type.\n\nobject.__index__(self)\n\n Called to implement ``operator.index()``. Also called whenever\n Python needs an integer object (such as in slicing, or in the\n built-in ``bin()``, ``hex()`` and ``oct()`` functions). Must return\n an integer.\n", - 'objects': '\nObjects, values and types\n*************************\n\n*Objects* are Python\'s abstraction for data. All data in a Python\nprogram is represented by objects or by relations between objects. (In\na sense, and in conformance to Von Neumann\'s model of a "stored\nprogram computer," code is also represented by objects.)\n\nEvery object has an identity, a type and a value. An object\'s\n*identity* never changes once it has been created; you may think of it\nas the object\'s address in memory. The \'``is``\' operator compares the\nidentity of two objects; the ``id()`` function returns an integer\nrepresenting its identity (currently implemented as its address). An\nobject\'s *type* is also unchangeable. [1] An object\'s type determines\nthe operations that the object supports (e.g., "does it have a\nlength?") and also defines the possible values for objects of that\ntype. The ``type()`` function returns an object\'s type (which is an\nobject itself). The *value* of some objects can change. Objects\nwhose value can change are said to be *mutable*; objects whose value\nis unchangeable once they are created are called *immutable*. (The\nvalue of an immutable container object that contains a reference to a\nmutable object can change when the latter\'s value is changed; however\nthe container is still considered immutable, because the collection of\nobjects it contains cannot be changed. So, immutability is not\nstrictly the same as having an unchangeable value, it is more subtle.)\nAn object\'s mutability is determined by its type; for instance,\nnumbers, strings and tuples are immutable, while dictionaries and\nlists are mutable.\n\nObjects are never explicitly destroyed; however, when they become\nunreachable they may be garbage-collected. An implementation is\nallowed to postpone garbage collection or omit it altogether --- it is\na matter of implementation quality how garbage collection is\nimplemented, as long as no objects are collected that are still\nreachable.\n\n**CPython implementation detail:** CPython currently uses a reference-\ncounting scheme with (optional) delayed detection of cyclically linked\ngarbage, which collects most objects as soon as they become\nunreachable, but is not guaranteed to collect garbage containing\ncircular references. See the documentation of the ``gc`` module for\ninformation on controlling the collection of cyclic garbage. Other\nimplementations act differently and CPython may change. Do not depend\non immediate finalization of objects when they become unreachable (ex:\nalways close files).\n\nNote that the use of the implementation\'s tracing or debugging\nfacilities may keep objects alive that would normally be collectable.\nAlso note that catching an exception with a \'``try``...``except``\'\nstatement may keep objects alive.\n\nSome objects contain references to "external" resources such as open\nfiles or windows. It is understood that these resources are freed\nwhen the object is garbage-collected, but since garbage collection is\nnot guaranteed to happen, such objects also provide an explicit way to\nrelease the external resource, usually a ``close()`` method. Programs\nare strongly recommended to explicitly close such objects. The\n\'``try``...``finally``\' statement and the \'``with``\' statement provide\nconvenient ways to do this.\n\nSome objects contain references to other objects; these are called\n*containers*. Examples of containers are tuples, lists and\ndictionaries. The references are part of a container\'s value. In\nmost cases, when we talk about the value of a container, we imply the\nvalues, not the identities of the contained objects; however, when we\ntalk about the mutability of a container, only the identities of the\nimmediately contained objects are implied. So, if an immutable\ncontainer (like a tuple) contains a reference to a mutable object, its\nvalue changes if that mutable object is changed.\n\nTypes affect almost all aspects of object behavior. Even the\nimportance of object identity is affected in some sense: for immutable\ntypes, operations that compute new values may actually return a\nreference to any existing object with the same type and value, while\nfor mutable objects this is not allowed. E.g., after ``a = 1; b =\n1``, ``a`` and ``b`` may or may not refer to the same object with the\nvalue one, depending on the implementation, but after ``c = []; d =\n[]``, ``c`` and ``d`` are guaranteed to refer to two different,\nunique, newly created empty lists. (Note that ``c = d = []`` assigns\nthe same object to both ``c`` and ``d``.)\n', + 'objects': '\nObjects, values and types\n*************************\n\n*Objects* are Python\'s abstraction for data. All data in a Python\nprogram is represented by objects or by relations between objects. (In\na sense, and in conformance to Von Neumann\'s model of a "stored\nprogram computer," code is also represented by objects.)\n\nEvery object has an identity, a type and a value. An object\'s\n*identity* never changes once it has been created; you may think of it\nas the object\'s address in memory. The \'``is``\' operator compares the\nidentity of two objects; the ``id()`` function returns an integer\nrepresenting its identity.\n\n**CPython implementation detail:** For CPython, ``id(x)`` is the\nmemory address where ``x`` is stored.\n\nAn object\'s type determines the operations that the object supports\n(e.g., "does it have a length?") and also defines the possible values\nfor objects of that type. The ``type()`` function returns an object\'s\ntype (which is an object itself). Like its identity, an object\'s\n*type* is also unchangeable. [1]\n\nThe *value* of some objects can change. Objects whose value can\nchange are said to be *mutable*; objects whose value is unchangeable\nonce they are created are called *immutable*. (The value of an\nimmutable container object that contains a reference to a mutable\nobject can change when the latter\'s value is changed; however the\ncontainer is still considered immutable, because the collection of\nobjects it contains cannot be changed. So, immutability is not\nstrictly the same as having an unchangeable value, it is more subtle.)\nAn object\'s mutability is determined by its type; for instance,\nnumbers, strings and tuples are immutable, while dictionaries and\nlists are mutable.\n\nObjects are never explicitly destroyed; however, when they become\nunreachable they may be garbage-collected. An implementation is\nallowed to postpone garbage collection or omit it altogether --- it is\na matter of implementation quality how garbage collection is\nimplemented, as long as no objects are collected that are still\nreachable.\n\n**CPython implementation detail:** CPython currently uses a reference-\ncounting scheme with (optional) delayed detection of cyclically linked\ngarbage, which collects most objects as soon as they become\nunreachable, but is not guaranteed to collect garbage containing\ncircular references. See the documentation of the ``gc`` module for\ninformation on controlling the collection of cyclic garbage. Other\nimplementations act differently and CPython may change. Do not depend\non immediate finalization of objects when they become unreachable (ex:\nalways close files).\n\nNote that the use of the implementation\'s tracing or debugging\nfacilities may keep objects alive that would normally be collectable.\nAlso note that catching an exception with a \'``try``...``except``\'\nstatement may keep objects alive.\n\nSome objects contain references to "external" resources such as open\nfiles or windows. It is understood that these resources are freed\nwhen the object is garbage-collected, but since garbage collection is\nnot guaranteed to happen, such objects also provide an explicit way to\nrelease the external resource, usually a ``close()`` method. Programs\nare strongly recommended to explicitly close such objects. The\n\'``try``...``finally``\' statement and the \'``with``\' statement provide\nconvenient ways to do this.\n\nSome objects contain references to other objects; these are called\n*containers*. Examples of containers are tuples, lists and\ndictionaries. The references are part of a container\'s value. In\nmost cases, when we talk about the value of a container, we imply the\nvalues, not the identities of the contained objects; however, when we\ntalk about the mutability of a container, only the identities of the\nimmediately contained objects are implied. So, if an immutable\ncontainer (like a tuple) contains a reference to a mutable object, its\nvalue changes if that mutable object is changed.\n\nTypes affect almost all aspects of object behavior. Even the\nimportance of object identity is affected in some sense: for immutable\ntypes, operations that compute new values may actually return a\nreference to any existing object with the same type and value, while\nfor mutable objects this is not allowed. E.g., after ``a = 1; b =\n1``, ``a`` and ``b`` may or may not refer to the same object with the\nvalue one, depending on the implementation, but after ``c = []; d =\n[]``, ``c`` and ``d`` are guaranteed to refer to two different,\nunique, newly created empty lists. (Note that ``c = d = []`` assigns\nthe same object to both ``c`` and ``d``.)\n', 'operator-summary': '\nSummary\n*******\n\nThe following table summarizes the operator precedences in Python,\nfrom lowest precedence (least binding) to highest precedence (most\nbinding). Operators in the same box have the same precedence. Unless\nthe syntax is explicitly given, operators are binary. Operators in\nthe same box group left to right (except for comparisons, including\ntests, which all have the same precedence and chain from left to right\n--- see section *Comparisons* --- and exponentiation, which groups\nfrom right to left).\n\n+-------------------------------------------------+---------------------------------------+\n| Operator | Description |\n+=================================================+=======================================+\n| ``lambda`` | Lambda expression |\n+-------------------------------------------------+---------------------------------------+\n| ``if`` -- ``else`` | Conditional expression |\n+-------------------------------------------------+---------------------------------------+\n| ``or`` | Boolean OR |\n+-------------------------------------------------+---------------------------------------+\n| ``and`` | Boolean AND |\n+-------------------------------------------------+---------------------------------------+\n| ``not`` *x* | Boolean NOT |\n+-------------------------------------------------+---------------------------------------+\n| ``in``, ``not`` ``in``, ``is``, ``is not``, | Comparisons, including membership |\n| ``<``, ``<=``, ``>``, ``>=``, ``!=``, ``==`` | tests and identity tests, |\n+-------------------------------------------------+---------------------------------------+\n| ``|`` | Bitwise OR |\n+-------------------------------------------------+---------------------------------------+\n| ``^`` | Bitwise XOR |\n+-------------------------------------------------+---------------------------------------+\n| ``&`` | Bitwise AND |\n+-------------------------------------------------+---------------------------------------+\n| ``<<``, ``>>`` | Shifts |\n+-------------------------------------------------+---------------------------------------+\n| ``+``, ``-`` | Addition and subtraction |\n+-------------------------------------------------+---------------------------------------+\n| ``*``, ``/``, ``//``, ``%`` | Multiplication, division, remainder |\n| | [5] |\n+-------------------------------------------------+---------------------------------------+\n| ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT |\n+-------------------------------------------------+---------------------------------------+\n| ``**`` | Exponentiation [6] |\n+-------------------------------------------------+---------------------------------------+\n| ``x[index]``, ``x[index:index]``, | Subscription, slicing, call, |\n| ``x(arguments...)``, ``x.attribute`` | attribute reference |\n+-------------------------------------------------+---------------------------------------+\n| ``(expressions...)``, ``[expressions...]``, | Binding or tuple display, list |\n| ``{key:datum...}``, ``{expressions...}`` | display, dictionary display, set |\n| | display |\n+-------------------------------------------------+---------------------------------------+\n\n-[ Footnotes ]-\n\n[1] While ``abs(x%y) < abs(y)`` is true mathematically, for floats it\n may not be true numerically due to roundoff. For example, and\n assuming a platform on which a Python float is an IEEE 754 double-\n precision number, in order that ``-1e-100 % 1e100`` have the same\n sign as ``1e100``, the computed result is ``-1e-100 + 1e100``,\n which is numerically exactly equal to ``1e100``. The function\n ``math.fmod()`` returns a result whose sign matches the sign of\n the first argument instead, and so returns ``-1e-100`` in this\n case. Which approach is more appropriate depends on the\n application.\n\n[2] If x is very close to an exact integer multiple of y, it\'s\n possible for ``x//y`` to be one larger than ``(x-x%y)//y`` due to\n rounding. In such cases, Python returns the latter result, in\n order to preserve that ``divmod(x,y)[0] * y + x % y`` be very\n close to ``x``.\n\n[3] While comparisons between strings make sense at the byte level,\n they may be counter-intuitive to users. For example, the strings\n ``"\\u00C7"`` and ``"\\u0327\\u0043"`` compare differently, even\n though they both represent the same unicode character (LATIN\n CAPITAL LETTER C WITH CEDILLA). To compare strings in a human\n recognizable way, compare using ``unicodedata.normalize()``.\n\n[4] Due to automatic garbage-collection, free lists, and the dynamic\n nature of descriptors, you may notice seemingly unusual behaviour\n in certain uses of the ``is`` operator, like those involving\n comparisons between instance methods, or constants. Check their\n documentation for more info.\n\n[5] The ``%`` operator is also used for string formatting; the same\n precedence applies.\n\n[6] The power operator ``**`` binds less tightly than an arithmetic or\n bitwise unary operator on its right, that is, ``2**-1`` is\n ``0.5``.\n', 'pass': '\nThe ``pass`` statement\n**********************\n\n pass_stmt ::= "pass"\n\n``pass`` is a null operation --- when it is executed, nothing happens.\nIt is useful as a placeholder when a statement is required\nsyntactically, but no code needs to be executed, for example:\n\n def f(arg): pass # a function that does nothing (yet)\n\n class C: pass # a class with no methods (yet)\n', 'power': '\nThe power operator\n******************\n\nThe power operator binds more tightly than unary operators on its\nleft; it binds less tightly than unary operators on its right. The\nsyntax is:\n\n power ::= primary ["**" u_expr]\n\nThus, in an unparenthesized sequence of power and unary operators, the\noperators are evaluated from right to left (this does not constrain\nthe evaluation order for the operands): ``-1**2`` results in ``-1``.\n\nThe power operator has the same semantics as the built-in ``pow()``\nfunction, when called with two arguments: it yields its left argument\nraised to the power of its right argument. The numeric arguments are\nfirst converted to a common type, and the result is of that type.\n\nFor int operands, the result has the same type as the operands unless\nthe second argument is negative; in that case, all arguments are\nconverted to float and a float result is delivered. For example,\n``10**2`` returns ``100``, but ``10**-2`` returns ``0.01``.\n\nRaising ``0.0`` to a negative power results in a\n``ZeroDivisionError``. Raising a negative number to a fractional power\nresults in a ``complex`` number. (In earlier versions it raised a\n``ValueError``.)\n', @@ -60,18 +60,18 @@ 'shifting': '\nShifting operations\n*******************\n\nThe shifting operations have lower priority than the arithmetic\noperations:\n\n shift_expr ::= a_expr | shift_expr ( "<<" | ">>" ) a_expr\n\nThese operators accept integers as arguments. They shift the first\nargument to the left or right by the number of bits given by the\nsecond argument.\n\nA right shift by *n* bits is defined as division by ``pow(2,n)``. A\nleft shift by *n* bits is defined as multiplication with ``pow(2,n)``.\n\nNote: In the current implementation, the right-hand operand is required to\n be at most ``sys.maxsize``. If the right-hand operand is larger\n than ``sys.maxsize`` an ``OverflowError`` exception is raised.\n', 'slicings': '\nSlicings\n********\n\nA slicing selects a range of items in a sequence object (e.g., a\nstring, tuple or list). Slicings may be used as expressions or as\ntargets in assignment or ``del`` statements. The syntax for a\nslicing:\n\n slicing ::= primary "[" slice_list "]"\n slice_list ::= slice_item ("," slice_item)* [","]\n slice_item ::= expression | proper_slice\n proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" [stride] ]\n lower_bound ::= expression\n upper_bound ::= expression\n stride ::= expression\n\nThere is ambiguity in the formal syntax here: anything that looks like\nan expression list also looks like a slice list, so any subscription\ncan be interpreted as a slicing. Rather than further complicating the\nsyntax, this is disambiguated by defining that in this case the\ninterpretation as a subscription takes priority over the\ninterpretation as a slicing (this is the case if the slice list\ncontains no proper slice).\n\nThe semantics for a slicing are as follows. The primary must evaluate\nto a mapping object, and it is indexed (using the same\n``__getitem__()`` method as normal subscription) with a key that is\nconstructed from the slice list, as follows. If the slice list\ncontains at least one comma, the key is a tuple containing the\nconversion of the slice items; otherwise, the conversion of the lone\nslice item is the key. The conversion of a slice item that is an\nexpression is that expression. The conversion of a proper slice is a\nslice object (see section *The standard type hierarchy*) whose\n``start``, ``stop`` and ``step`` attributes are the values of the\nexpressions given as lower bound, upper bound and stride,\nrespectively, substituting ``None`` for missing expressions.\n', 'specialattrs': '\nSpecial Attributes\n******************\n\nThe implementation adds a few special read-only attributes to several\nobject types, where they are relevant. Some of these are not reported\nby the ``dir()`` built-in function.\n\nobject.__dict__\n\n A dictionary or other mapping object used to store an object\'s\n (writable) attributes.\n\ninstance.__class__\n\n The class to which a class instance belongs.\n\nclass.__bases__\n\n The tuple of base classes of a class object.\n\nclass.__name__\n\n The name of the class or type.\n\nclass.__qualname__\n\n The *qualified name* of the class or type.\n\n New in version 3.3.\n\nclass.__mro__\n\n This attribute is a tuple of classes that are considered when\n looking for base classes during method resolution.\n\nclass.mro()\n\n This method can be overridden by a metaclass to customize the\n method resolution order for its instances. It is called at class\n instantiation, and its result is stored in ``__mro__``.\n\nclass.__subclasses__()\n\n Each class keeps a list of weak references to its immediate\n subclasses. This method returns a list of all those references\n still alive. Example:\n\n >>> int.__subclasses__()\n []\n\n-[ Footnotes ]-\n\n[1] Additional information on these special methods may be found in\n the Python Reference Manual (*Basic customization*).\n\n[2] As a consequence, the list ``[1, 2]`` is considered equal to\n ``[1.0, 2.0]``, and similarly for tuples.\n\n[3] They must have since the parser can\'t tell the type of the\n operands.\n\n[4] Cased characters are those with general category property being\n one of "Lu" (Letter, uppercase), "Ll" (Letter, lowercase), or "Lt"\n (Letter, titlecase).\n\n[5] To format only a tuple you should therefore provide a singleton\n tuple whose only element is the tuple to be formatted.\n', - 'specialnames': '\nSpecial method names\n********************\n\nA class can implement certain operations that are invoked by special\nsyntax (such as arithmetic operations or subscripting and slicing) by\ndefining methods with special names. This is Python\'s approach to\n*operator overloading*, allowing classes to define their own behavior\nwith respect to language operators. For instance, if a class defines\na method named ``__getitem__()``, and ``x`` is an instance of this\nclass, then ``x[i]`` is roughly equivalent to ``type(x).__getitem__(x,\ni)``. Except where mentioned, attempts to execute an operation raise\nan exception when no appropriate method is defined (typically\n``AttributeError`` or ``TypeError``).\n\nWhen implementing a class that emulates any built-in type, it is\nimportant that the emulation only be implemented to the degree that it\nmakes sense for the object being modelled. For example, some\nsequences may work well with retrieval of individual elements, but\nextracting a slice may not make sense. (One example of this is the\n``NodeList`` interface in the W3C\'s Document Object Model.)\n\n\nBasic customization\n===================\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. ``__new__()`` is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of ``__new__()`` should be the new object instance (usually\n an instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s ``__new__()`` method using\n ``super(currentclass, cls).__new__(cls[, ...])`` with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If ``__new__()`` returns an instance of *cls*, then the new\n instance\'s ``__init__()`` method will be invoked like\n ``__init__(self[, ...])``, where *self* is the new instance and the\n remaining arguments are the same as were passed to ``__new__()``.\n\n If ``__new__()`` does not return an instance of *cls*, then the new\n instance\'s ``__init__()`` method will not be invoked.\n\n ``__new__()`` is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n ``__init__()`` method, the derived class\'s ``__init__()`` method,\n if any, must explicitly call it to ensure proper initialization of\n the base class part of the instance; for example:\n ``BaseClass.__init__(self, [args...])``. As a special constraint\n on constructors, no value may be returned; doing so will cause a\n ``TypeError`` to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a ``__del__()`` method,\n the derived class\'s ``__del__()`` method, if any, must explicitly\n call it to ensure proper deletion of the base class part of the\n instance. Note that it is possible (though not recommended!) for\n the ``__del__()`` method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n ``__del__()`` methods are called for objects that still exist when\n the interpreter exits.\n\n Note: ``del x`` doesn\'t directly call ``x.__del__()`` --- the former\n decrements the reference count for ``x`` by one, and the latter\n is only called when ``x``\'s reference count reaches zero. Some\n common situations that may prevent the reference count of an\n object from going to zero include: circular references between\n objects (e.g., a doubly-linked list or a tree data structure with\n parent and child pointers); a reference to the object on the\n stack frame of a function that caught an exception (the traceback\n stored in ``sys.exc_info()[2]`` keeps the stack frame alive); or\n a reference to the object on the stack frame that raised an\n unhandled exception in interactive mode (the traceback stored in\n ``sys.last_traceback`` keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing ``None`` in\n ``sys.last_traceback``. Circular references which are garbage are\n detected when the option cycle detector is enabled (it\'s on by\n default), but can only be cleaned up if there are no Python-\n level ``__del__()`` methods involved. Refer to the documentation\n for the ``gc`` module for more information about how\n ``__del__()`` methods are handled by the cycle detector,\n particularly the description of the ``garbage`` value.\n\n Warning: Due to the precarious circumstances under which ``__del__()``\n methods are invoked, exceptions that occur during their execution\n are ignored, and a warning is printed to ``sys.stderr`` instead.\n Also, when ``__del__()`` is invoked in response to a module being\n deleted (e.g., when execution of the program is done), other\n globals referenced by the ``__del__()`` method may already have\n been deleted or in the process of being torn down (e.g. the\n import machinery shutting down). For this reason, ``__del__()``\n methods should do the absolute minimum needed to maintain\n external invariants. Starting with version 1.5, Python\n guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the ``__del__()`` method is called.\n\nobject.__repr__(self)\n\n Called by the ``repr()`` built-in function to compute the\n "official" string representation of an object. If at all possible,\n this should look like a valid Python expression that could be used\n to recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n ``<...some useful description...>`` should be returned. The return\n value must be a string object. If a class defines ``__repr__()``\n but not ``__str__()``, then ``__repr__()`` is also used when an\n "informal" string representation of instances of that class is\n required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by the ``str()`` built-in function and by the ``print()``\n function to compute the "informal" string representation of an\n object. This differs from ``__repr__()`` in that it does not have\n to be a valid Python expression: a more convenient or concise\n representation may be used instead. The return value must be a\n string object.\n\nobject.__bytes__(self)\n\n Called by ``bytes()`` to compute a byte-string representation of an\n object. This should return a ``bytes`` object.\n\nobject.__format__(self, format_spec)\n\n Called by the ``format()`` built-in function (and by extension, the\n ``format()`` method of class ``str``) to produce a "formatted"\n string representation of an object. The ``format_spec`` argument is\n a string that contains a description of the formatting options\n desired. The interpretation of the ``format_spec`` argument is up\n to the type implementing ``__format__()``, however most classes\n will either delegate formatting to one of the built-in types, or\n use a similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: ``xy`` calls ``x.__gt__(y)``, and ``x>=y`` calls\n ``x.__ge__(y)``.\n\n A rich comparison method may return the singleton\n ``NotImplemented`` if it does not implement the operation for a\n given pair of arguments. By convention, ``False`` and ``True`` are\n returned for a successful comparison. However, these methods can\n return any value, so if the comparison operator is used in a\n Boolean context (e.g., in the condition of an ``if`` statement),\n Python will call ``bool()`` on the value to determine if the result\n is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of ``x==y`` does not imply that ``x!=y`` is false.\n Accordingly, when defining ``__eq__()``, one should also define\n ``__ne__()`` so that the operators will behave as expected. See\n the paragraph on ``__hash__()`` for some important notes on\n creating *hashable* objects which support custom comparison\n operations and are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, ``__lt__()`` and ``__gt__()`` are each\n other\'s reflection, ``__le__()`` and ``__ge__()`` are each other\'s\n reflection, and ``__eq__()`` and ``__ne__()`` are their own\n reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see ``functools.total_ordering()``.\n\nobject.__hash__(self)\n\n Called by built-in function ``hash()`` and for operations on\n members of hashed collections including ``set``, ``frozenset``, and\n ``dict``. ``__hash__()`` should return an integer. The only\n required property is that objects which compare equal have the same\n hash value; it is advised to somehow mix together (e.g. using\n exclusive or) the hash values for the components of the object that\n also play a part in comparison of objects.\n\n If a class does not define an ``__eq__()`` method it should not\n define a ``__hash__()`` operation either; if it defines\n ``__eq__()`` but not ``__hash__()``, its instances will not be\n usable as items in hashable collections. If a class defines\n mutable objects and implements an ``__eq__()`` method, it should\n not implement ``__hash__()``, since the implementation of hashable\n collections requires that a key\'s hash value is immutable (if the\n object\'s hash value changes, it will be in the wrong hash bucket).\n\n User-defined classes have ``__eq__()`` and ``__hash__()`` methods\n by default; with them, all objects compare unequal (except with\n themselves) and ``x.__hash__()`` returns ``id(x)``.\n\n Classes which inherit a ``__hash__()`` method from a parent class\n but change the meaning of ``__eq__()`` such that the hash value\n returned is no longer appropriate (e.g. by switching to a value-\n based concept of equality instead of the default identity based\n equality) can explicitly flag themselves as being unhashable by\n setting ``__hash__ = None`` in the class definition. Doing so means\n that not only will instances of the class raise an appropriate\n ``TypeError`` when a program attempts to retrieve their hash value,\n but they will also be correctly identified as unhashable when\n checking ``isinstance(obj, collections.Hashable)`` (unlike classes\n which define their own ``__hash__()`` to explicitly raise\n ``TypeError``).\n\n If a class that overrides ``__eq__()`` needs to retain the\n implementation of ``__hash__()`` from a parent class, the\n interpreter must be told this explicitly by setting ``__hash__ =\n .__hash__``. Otherwise the inheritance of\n ``__hash__()`` will be blocked, just as if ``__hash__`` had been\n explicitly set to ``None``.\n\n Note: Note by default the ``__hash__()`` values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the order in which keys are\n retrieved from a dict. Note Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also ``PYTHONHASHSEED``.\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n ``bool()``; should return ``False`` or ``True``. When this method\n is not defined, ``__len__()`` is called, if it is defined, and the\n object is considered true if its result is nonzero. If a class\n defines neither ``__len__()`` nor ``__bool__()``, all its instances\n are considered true.\n\n\nCustomizing attribute access\n============================\n\nThe following methods can be defined to customize the meaning of\nattribute access (use of, assignment to, or deletion of ``x.name``)\nfor class instances.\n\nobject.__getattr__(self, name)\n\n Called when an attribute lookup has not found the attribute in the\n usual places (i.e. it is not an instance attribute nor is it found\n in the class tree for ``self``). ``name`` is the attribute name.\n This method should return the (computed) attribute value or raise\n an ``AttributeError`` exception.\n\n Note that if the attribute is found through the normal mechanism,\n ``__getattr__()`` is not called. (This is an intentional asymmetry\n between ``__getattr__()`` and ``__setattr__()``.) This is done both\n for efficiency reasons and because otherwise ``__getattr__()``\n would have no way to access other attributes of the instance. Note\n that at least for instance variables, you can fake total control by\n not inserting any values in the instance attribute dictionary (but\n instead inserting them in another object). See the\n ``__getattribute__()`` method below for a way to actually get total\n control over attribute access.\n\nobject.__getattribute__(self, name)\n\n Called unconditionally to implement attribute accesses for\n instances of the class. If the class also defines\n ``__getattr__()``, the latter will not be called unless\n ``__getattribute__()`` either calls it explicitly or raises an\n ``AttributeError``. This method should return the (computed)\n attribute value or raise an ``AttributeError`` exception. In order\n to avoid infinite recursion in this method, its implementation\n should always call the base class method with the same name to\n access any attributes it needs, for example,\n ``object.__getattribute__(self, name)``.\n\n Note: This method may still be bypassed when looking up special methods\n as the result of implicit invocation via language syntax or\n built-in functions. See *Special method lookup*.\n\nobject.__setattr__(self, name, value)\n\n Called when an attribute assignment is attempted. This is called\n instead of the normal mechanism (i.e. store the value in the\n instance dictionary). *name* is the attribute name, *value* is the\n value to be assigned to it.\n\n If ``__setattr__()`` wants to assign to an instance attribute, it\n should call the base class method with the same name, for example,\n ``object.__setattr__(self, name, value)``.\n\nobject.__delattr__(self, name)\n\n Like ``__setattr__()`` but for attribute deletion instead of\n assignment. This should only be implemented if ``del obj.name`` is\n meaningful for the object.\n\nobject.__dir__(self)\n\n Called when ``dir()`` is called on the object. A sequence must be\n returned. ``dir()`` converts the returned sequence to a list and\n sorts it.\n\n\nImplementing Descriptors\n------------------------\n\nThe following methods only apply when an instance of the class\ncontaining the method (a so-called *descriptor* class) appears in an\n*owner* class (the descriptor must be in either the owner\'s class\ndictionary or in the class dictionary for one of its parents). In the\nexamples below, "the attribute" refers to the attribute whose name is\nthe key of the property in the owner class\' ``__dict__``.\n\nobject.__get__(self, instance, owner)\n\n Called to get the attribute of the owner class (class attribute\n access) or of an instance of that class (instance attribute\n access). *owner* is always the owner class, while *instance* is the\n instance that the attribute was accessed through, or ``None`` when\n the attribute is accessed through the *owner*. This method should\n return the (computed) attribute value or raise an\n ``AttributeError`` exception.\n\nobject.__set__(self, instance, value)\n\n Called to set the attribute on an instance *instance* of the owner\n class to a new value, *value*.\n\nobject.__delete__(self, instance)\n\n Called to delete the attribute on an instance *instance* of the\n owner class.\n\n\nInvoking Descriptors\n--------------------\n\nIn general, a descriptor is an object attribute with "binding\nbehavior", one whose attribute access has been overridden by methods\nin the descriptor protocol: ``__get__()``, ``__set__()``, and\n``__delete__()``. If any of those methods are defined for an object,\nit is said to be a descriptor.\n\nThe default behavior for attribute access is to get, set, or delete\nthe attribute from an object\'s dictionary. For instance, ``a.x`` has a\nlookup chain starting with ``a.__dict__[\'x\']``, then\n``type(a).__dict__[\'x\']``, and continuing through the base classes of\n``type(a)`` excluding metaclasses.\n\nHowever, if the looked-up value is an object defining one of the\ndescriptor methods, then Python may override the default behavior and\ninvoke the descriptor method instead. Where this occurs in the\nprecedence chain depends on which descriptor methods were defined and\nhow they were called.\n\nThe starting point for descriptor invocation is a binding, ``a.x``.\nHow the arguments are assembled depends on ``a``:\n\nDirect Call\n The simplest and least common call is when user code directly\n invokes a descriptor method: ``x.__get__(a)``.\n\nInstance Binding\n If binding to an object instance, ``a.x`` is transformed into the\n call: ``type(a).__dict__[\'x\'].__get__(a, type(a))``.\n\nClass Binding\n If binding to a class, ``A.x`` is transformed into the call:\n ``A.__dict__[\'x\'].__get__(None, A)``.\n\nSuper Binding\n If ``a`` is an instance of ``super``, then the binding ``super(B,\n obj).m()`` searches ``obj.__class__.__mro__`` for the base class\n ``A`` immediately preceding ``B`` and then invokes the descriptor\n with the call: ``A.__dict__[\'m\'].__get__(obj, obj.__class__)``.\n\nFor instance bindings, the precedence of descriptor invocation depends\non the which descriptor methods are defined. A descriptor can define\nany combination of ``__get__()``, ``__set__()`` and ``__delete__()``.\nIf it does not define ``__get__()``, then accessing the attribute will\nreturn the descriptor object itself unless there is a value in the\nobject\'s instance dictionary. If the descriptor defines ``__set__()``\nand/or ``__delete__()``, it is a data descriptor; if it defines\nneither, it is a non-data descriptor. Normally, data descriptors\ndefine both ``__get__()`` and ``__set__()``, while non-data\ndescriptors have just the ``__get__()`` method. Data descriptors with\n``__set__()`` and ``__get__()`` defined always override a redefinition\nin an instance dictionary. In contrast, non-data descriptors can be\noverridden by instances.\n\nPython methods (including ``staticmethod()`` and ``classmethod()``)\nare implemented as non-data descriptors. Accordingly, instances can\nredefine and override methods. This allows individual instances to\nacquire behaviors that differ from other instances of the same class.\n\nThe ``property()`` function is implemented as a data descriptor.\nAccordingly, instances cannot override the behavior of a property.\n\n\n__slots__\n---------\n\nBy default, instances of classes have a dictionary for attribute\nstorage. This wastes space for objects having very few instance\nvariables. The space consumption can become acute when creating large\nnumbers of instances.\n\nThe default can be overridden by defining *__slots__* in a class\ndefinition. The *__slots__* declaration takes a sequence of instance\nvariables and reserves just enough space in each instance to hold a\nvalue for each variable. Space is saved because *__dict__* is not\ncreated for each instance.\n\nobject.__slots__\n\n This class variable can be assigned a string, iterable, or sequence\n of strings with variable names used by instances. If defined in a\n class, *__slots__* reserves space for the declared variables and\n prevents the automatic creation of *__dict__* and *__weakref__* for\n each instance.\n\n\nNotes on using *__slots__*\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n* When inheriting from a class without *__slots__*, the *__dict__*\n attribute of that class will always be accessible, so a *__slots__*\n definition in the subclass is meaningless.\n\n* Without a *__dict__* variable, instances cannot be assigned new\n variables not listed in the *__slots__* definition. Attempts to\n assign to an unlisted variable name raises ``AttributeError``. If\n dynamic assignment of new variables is desired, then add\n ``\'__dict__\'`` to the sequence of strings in the *__slots__*\n declaration.\n\n* Without a *__weakref__* variable for each instance, classes defining\n *__slots__* do not support weak references to its instances. If weak\n reference support is needed, then add ``\'__weakref__\'`` to the\n sequence of strings in the *__slots__* declaration.\n\n* *__slots__* are implemented at the class level by creating\n descriptors (*Implementing Descriptors*) for each variable name. As\n a result, class attributes cannot be used to set default values for\n instance variables defined by *__slots__*; otherwise, the class\n attribute would overwrite the descriptor assignment.\n\n* The action of a *__slots__* declaration is limited to the class\n where it is defined. As a result, subclasses will have a *__dict__*\n unless they also define *__slots__* (which must only contain names\n of any *additional* slots).\n\n* If a class defines a slot also defined in a base class, the instance\n variable defined by the base class slot is inaccessible (except by\n retrieving its descriptor directly from the base class). This\n renders the meaning of the program undefined. In the future, a\n check may be added to prevent this.\n\n* Nonempty *__slots__* does not work for classes derived from\n "variable-length" built-in types such as ``int``, ``str`` and\n ``tuple``.\n\n* Any non-string iterable may be assigned to *__slots__*. Mappings may\n also be used; however, in the future, special meaning may be\n assigned to the values corresponding to each key.\n\n* *__class__* assignment works only if both classes have the same\n *__slots__*.\n\n\nCustomizing class creation\n==========================\n\nBy default, classes are constructed using ``type()``. A class\ndefinition is read into a separate namespace and the value of class\nname is bound to the result of ``type(name, bases, dict)``.\n\nWhen the class definition is read, if a callable ``metaclass`` keyword\nargument is passed after the bases in the class definition, the\ncallable given will be called instead of ``type()``. If other keyword\narguments are passed, they will also be passed to the metaclass. This\nallows classes or functions to be written which monitor or alter the\nclass creation process:\n\n* Modifying the class dictionary prior to the class being created.\n\n* Returning an instance of another class -- essentially performing the\n role of a factory function.\n\nThese steps will have to be performed in the metaclass\'s ``__new__()``\nmethod -- ``type.__new__()`` can then be called from this method to\ncreate a class with different properties. This example adds a new\nelement to the class dictionary before creating the class:\n\n class metacls(type):\n def __new__(mcs, name, bases, dict):\n dict[\'foo\'] = \'metacls was here\'\n return type.__new__(mcs, name, bases, dict)\n\nYou can of course also override other class methods (or add new\nmethods); for example defining a custom ``__call__()`` method in the\nmetaclass allows custom behavior when the class is called, e.g. not\nalways creating a new instance.\n\nIf the metaclass has a ``__prepare__()`` attribute (usually\nimplemented as a class or static method), it is called before the\nclass body is evaluated with the name of the class and a tuple of its\nbases for arguments. It should return an object that supports the\nmapping interface that will be used to store the namespace of the\nclass. The default is a plain dictionary. This could be used, for\nexample, to keep track of the order that class attributes are declared\nin by returning an ordered dictionary.\n\nThe appropriate metaclass is determined by the following precedence\nrules:\n\n* If the ``metaclass`` keyword argument is passed with the bases, it\n is used.\n\n* Otherwise, if there is at least one base class, its metaclass is\n used.\n\n* Otherwise, the default metaclass (``type``) is used.\n\nThe potential uses for metaclasses are boundless. Some ideas that have\nbeen explored including logging, interface checking, automatic\ndelegation, automatic property creation, proxies, frameworks, and\nautomatic resource locking/synchronization.\n\nHere is an example of a metaclass that uses an\n``collections.OrderedDict`` to remember the order that class members\nwere defined:\n\n class OrderedClass(type):\n\n @classmethod\n def __prepare__(metacls, name, bases, **kwds):\n return collections.OrderedDict()\n\n def __new__(cls, name, bases, classdict):\n result = type.__new__(cls, name, bases, dict(classdict))\n result.members = tuple(classdict)\n return result\n\n class A(metaclass=OrderedClass):\n def one(self): pass\n def two(self): pass\n def three(self): pass\n def four(self): pass\n\n >>> A.members\n (\'__module__\', \'one\', \'two\', \'three\', \'four\')\n\nWhen the class definition for *A* gets executed, the process begins\nwith calling the metaclass\'s ``__prepare__()`` method which returns an\nempty ``collections.OrderedDict``. That mapping records the methods\nand attributes of *A* as they are defined within the body of the class\nstatement. Once those definitions are executed, the ordered dictionary\nis fully populated and the metaclass\'s ``__new__()`` method gets\ninvoked. That method builds the new type and it saves the ordered\ndictionary keys in an attribute called ``members``.\n\n\nCustomizing instance and subclass checks\n========================================\n\nThe following methods are used to override the default behavior of the\n``isinstance()`` and ``issubclass()`` built-in functions.\n\nIn particular, the metaclass ``abc.ABCMeta`` implements these methods\nin order to allow the addition of Abstract Base Classes (ABCs) as\n"virtual base classes" to any class or type (including built-in\ntypes), including other ABCs.\n\nclass.__instancecheck__(self, instance)\n\n Return true if *instance* should be considered a (direct or\n indirect) instance of *class*. If defined, called to implement\n ``isinstance(instance, class)``.\n\nclass.__subclasscheck__(self, subclass)\n\n Return true if *subclass* should be considered a (direct or\n indirect) subclass of *class*. If defined, called to implement\n ``issubclass(subclass, class)``.\n\nNote that these methods are looked up on the type (metaclass) of a\nclass. They cannot be defined as class methods in the actual class.\nThis is consistent with the lookup of special methods that are called\non instances, only in this case the instance is itself a class.\n\nSee also:\n\n **PEP 3119** - Introducing Abstract Base Classes\n Includes the specification for customizing ``isinstance()`` and\n ``issubclass()`` behavior through ``__instancecheck__()`` and\n ``__subclasscheck__()``, with motivation for this functionality\n in the context of adding Abstract Base Classes (see the ``abc``\n module) to the language.\n\n\nEmulating callable objects\n==========================\n\nobject.__call__(self[, args...])\n\n Called when the instance is "called" as a function; if this method\n is defined, ``x(arg1, arg2, ...)`` is a shorthand for\n ``x.__call__(arg1, arg2, ...)``.\n\n\nEmulating container types\n=========================\n\nThe following methods can be defined to implement container objects.\nContainers usually are sequences (such as lists or tuples) or mappings\n(like dictionaries), but can represent other containers as well. The\nfirst set of methods is used either to emulate a sequence or to\nemulate a mapping; the difference is that for a sequence, the\nallowable keys should be the integers *k* for which ``0 <= k < N``\nwhere *N* is the length of the sequence, or slice objects, which\ndefine a range of items. It is also recommended that mappings provide\nthe methods ``keys()``, ``values()``, ``items()``, ``get()``,\n``clear()``, ``setdefault()``, ``pop()``, ``popitem()``, ``copy()``,\nand ``update()`` behaving similar to those for Python\'s standard\ndictionary objects. The ``collections`` module provides a\n``MutableMapping`` abstract base class to help create those methods\nfrom a base set of ``__getitem__()``, ``__setitem__()``,\n``__delitem__()``, and ``keys()``. Mutable sequences should provide\nmethods ``append()``, ``count()``, ``index()``, ``extend()``,\n``insert()``, ``pop()``, ``remove()``, ``reverse()`` and ``sort()``,\nlike Python standard list objects. Finally, sequence types should\nimplement addition (meaning concatenation) and multiplication (meaning\nrepetition) by defining the methods ``__add__()``, ``__radd__()``,\n``__iadd__()``, ``__mul__()``, ``__rmul__()`` and ``__imul__()``\ndescribed below; they should not define other numerical operators. It\nis recommended that both mappings and sequences implement the\n``__contains__()`` method to allow efficient use of the ``in``\noperator; for mappings, ``in`` should search the mapping\'s keys; for\nsequences, it should search through the values. It is further\nrecommended that both mappings and sequences implement the\n``__iter__()`` method to allow efficient iteration through the\ncontainer; for mappings, ``__iter__()`` should be the same as\n``keys()``; for sequences, it should iterate through the values.\n\nobject.__len__(self)\n\n Called to implement the built-in function ``len()``. Should return\n the length of the object, an integer ``>=`` 0. Also, an object\n that doesn\'t define a ``__bool__()`` method and whose ``__len__()``\n method returns zero is considered to be false in a Boolean context.\n\nNote: Slicing is done exclusively with the following three methods. A\n call like\n\n a[1:2] = b\n\n is translated to\n\n a[slice(1, 2, None)] = b\n\n and so forth. Missing slice items are always filled in with\n ``None``.\n\nobject.__getitem__(self, key)\n\n Called to implement evaluation of ``self[key]``. For sequence\n types, the accepted keys should be integers and slice objects.\n Note that the special interpretation of negative indexes (if the\n class wishes to emulate a sequence type) is up to the\n ``__getitem__()`` method. If *key* is of an inappropriate type,\n ``TypeError`` may be raised; if of a value outside the set of\n indexes for the sequence (after any special interpretation of\n negative values), ``IndexError`` should be raised. For mapping\n types, if *key* is missing (not in the container), ``KeyError``\n should be raised.\n\n Note: ``for`` loops expect that an ``IndexError`` will be raised for\n illegal indexes to allow proper detection of the end of the\n sequence.\n\nobject.__setitem__(self, key, value)\n\n Called to implement assignment to ``self[key]``. Same note as for\n ``__getitem__()``. This should only be implemented for mappings if\n the objects support changes to the values for keys, or if new keys\n can be added, or for sequences if elements can be replaced. The\n same exceptions should be raised for improper *key* values as for\n the ``__getitem__()`` method.\n\nobject.__delitem__(self, key)\n\n Called to implement deletion of ``self[key]``. Same note as for\n ``__getitem__()``. This should only be implemented for mappings if\n the objects support removal of keys, or for sequences if elements\n can be removed from the sequence. The same exceptions should be\n raised for improper *key* values as for the ``__getitem__()``\n method.\n\nobject.__iter__(self)\n\n This method is called when an iterator is required for a container.\n This method should return a new iterator object that can iterate\n over all the objects in the container. For mappings, it should\n iterate over the keys of the container, and should also be made\n available as the method ``keys()``.\n\n Iterator objects also need to implement this method; they are\n required to return themselves. For more information on iterator\n objects, see *Iterator Types*.\n\nobject.__reversed__(self)\n\n Called (if present) by the ``reversed()`` built-in to implement\n reverse iteration. It should return a new iterator object that\n iterates over all the objects in the container in reverse order.\n\n If the ``__reversed__()`` method is not provided, the\n ``reversed()`` built-in will fall back to using the sequence\n protocol (``__len__()`` and ``__getitem__()``). Objects that\n support the sequence protocol should only provide\n ``__reversed__()`` if they can provide an implementation that is\n more efficient than the one provided by ``reversed()``.\n\nThe membership test operators (``in`` and ``not in``) are normally\nimplemented as an iteration through a sequence. However, container\nobjects can supply the following special method with a more efficient\nimplementation, which also does not require the object be a sequence.\n\nobject.__contains__(self, item)\n\n Called to implement membership test operators. Should return true\n if *item* is in *self*, false otherwise. For mapping objects, this\n should consider the keys of the mapping rather than the values or\n the key-item pairs.\n\n For objects that don\'t define ``__contains__()``, the membership\n test first tries iteration via ``__iter__()``, then the old\n sequence iteration protocol via ``__getitem__()``, see *this\n section in the language reference*.\n\n\nEmulating numeric types\n=======================\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations (``+``, ``-``, ``*``, ``/``, ``//``, ``%``,\n ``divmod()``, ``pow()``, ``**``, ``<<``, ``>>``, ``&``, ``^``,\n ``|``). For instance, to evaluate the expression ``x + y``, where\n *x* is an instance of a class that has an ``__add__()`` method,\n ``x.__add__(y)`` is called. The ``__divmod__()`` method should be\n the equivalent to using ``__floordiv__()`` and ``__mod__()``; it\n should not be related to ``__truediv__()``. Note that\n ``__pow__()`` should be defined to accept an optional third\n argument if the ternary version of the built-in ``pow()`` function\n is to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return ``NotImplemented``.\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations (``+``, ``-``, ``*``, ``/``, ``//``, ``%``,\n ``divmod()``, ``pow()``, ``**``, ``<<``, ``>>``, ``&``, ``^``,\n ``|``) with reflected (swapped) operands. These functions are only\n called if the left operand does not support the corresponding\n operation and the operands are of different types. [2] For\n instance, to evaluate the expression ``x - y``, where *y* is an\n instance of a class that has an ``__rsub__()`` method,\n ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns\n *NotImplemented*.\n\n Note that ternary ``pow()`` will not try calling ``__rpow__()``\n (the coercion rules would become too complicated).\n\n Note: If the right operand\'s type is a subclass of the left operand\'s\n type and that subclass provides the reflected method for the\n operation, this method will be called before the left operand\'s\n non-reflected method. This behavior allows subclasses to\n override their ancestors\' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``,\n ``**=``, ``<<=``, ``>>=``, ``&=``, ``^=``, ``|=``). These methods\n should attempt to do the operation in-place (modifying *self*) and\n return the result (which could be, but does not have to be,\n *self*). If a specific method is not defined, the augmented\n assignment falls back to the normal methods. For instance, to\n execute the statement ``x += y``, where *x* is an instance of a\n class that has an ``__iadd__()`` method, ``x.__iadd__(y)`` is\n called. If *x* is an instance of a class that does not define a\n ``__iadd__()`` method, ``x.__add__(y)`` and ``y.__radd__(x)`` are\n considered, as with the evaluation of ``x + y``.\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations (``-``, ``+``,\n ``abs()`` and ``~``).\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions ``complex()``,\n ``int()``, ``float()`` and ``round()``. Should return a value of\n the appropriate type.\n\nobject.__index__(self)\n\n Called to implement ``operator.index()``. Also called whenever\n Python needs an integer object (such as in slicing, or in the\n built-in ``bin()``, ``hex()`` and ``oct()`` functions). Must return\n an integer.\n\n\nWith Statement Context Managers\n===============================\n\nA *context manager* is an object that defines the runtime context to\nbe established when executing a ``with`` statement. The context\nmanager handles the entry into, and the exit from, the desired runtime\ncontext for the execution of the block of code. Context managers are\nnormally invoked using the ``with`` statement (described in section\n*The with statement*), but can also be used by directly invoking their\nmethods.\n\nTypical uses of context managers include saving and restoring various\nkinds of global state, locking and unlocking resources, closing opened\nfiles, etc.\n\nFor more information on context managers, see *Context Manager Types*.\n\nobject.__enter__(self)\n\n Enter the runtime context related to this object. The ``with``\n statement will bind this method\'s return value to the target(s)\n specified in the ``as`` clause of the statement, if any.\n\nobject.__exit__(self, exc_type, exc_value, traceback)\n\n Exit the runtime context related to this object. The parameters\n describe the exception that caused the context to be exited. If the\n context was exited without an exception, all three arguments will\n be ``None``.\n\n If an exception is supplied, and the method wishes to suppress the\n exception (i.e., prevent it from being propagated), it should\n return a true value. Otherwise, the exception will be processed\n normally upon exit from this method.\n\n Note that ``__exit__()`` methods should not reraise the passed-in\n exception; this is the caller\'s responsibility.\n\nSee also:\n\n **PEP 0343** - The "with" statement\n The specification, background, and examples for the Python\n ``with`` statement.\n\n\nSpecial method lookup\n=====================\n\nFor custom classes, implicit invocations of special methods are only\nguaranteed to work correctly if defined on an object\'s type, not in\nthe object\'s instance dictionary. That behaviour is the reason why\nthe following code raises an exception:\n\n >>> class C:\n ... pass\n ...\n >>> c = C()\n >>> c.__len__ = lambda: 5\n >>> len(c)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: object of type \'C\' has no len()\n\nThe rationale behind this behaviour lies with a number of special\nmethods such as ``__hash__()`` and ``__repr__()`` that are implemented\nby all objects, including type objects. If the implicit lookup of\nthese methods used the conventional lookup process, they would fail\nwhen invoked on the type object itself:\n\n >>> 1 .__hash__() == hash(1)\n True\n >>> int.__hash__() == hash(int)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: descriptor \'__hash__\' of \'int\' object needs an argument\n\nIncorrectly attempting to invoke an unbound method of a class in this\nway is sometimes referred to as \'metaclass confusion\', and is avoided\nby bypassing the instance when looking up special methods:\n\n >>> type(1).__hash__(1) == hash(1)\n True\n >>> type(int).__hash__(int) == hash(int)\n True\n\nIn addition to bypassing any instance attributes in the interest of\ncorrectness, implicit special method lookup generally also bypasses\nthe ``__getattribute__()`` method even of the object\'s metaclass:\n\n >>> class Meta(type):\n ... def __getattribute__(*args):\n ... print("Metaclass getattribute invoked")\n ... return type.__getattribute__(*args)\n ...\n >>> class C(object, metaclass=Meta):\n ... def __len__(self):\n ... return 10\n ... def __getattribute__(*args):\n ... print("Class getattribute invoked")\n ... return object.__getattribute__(*args)\n ...\n >>> c = C()\n >>> c.__len__() # Explicit lookup via instance\n Class getattribute invoked\n 10\n >>> type(c).__len__(c) # Explicit lookup via type\n Metaclass getattribute invoked\n 10\n >>> len(c) # Implicit lookup\n 10\n\nBypassing the ``__getattribute__()`` machinery in this fashion\nprovides significant scope for speed optimisations within the\ninterpreter, at the cost of some flexibility in the handling of\nspecial methods (the special method *must* be set on the class object\nitself in order to be consistently invoked by the interpreter).\n\n-[ Footnotes ]-\n\n[1] It *is* possible in some cases to change an object\'s type, under\n certain controlled conditions. It generally isn\'t a good idea\n though, since it can lead to some very strange behaviour if it is\n handled incorrectly.\n\n[2] For operands of the same type, it is assumed that if the non-\n reflected method (such as ``__add__()``) fails the operation is\n not supported, which is why the reflected method is not called.\n', - 'string-methods': '\nString Methods\n**************\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n Return a copy of the string with its first character capitalized\n and the rest lowercased.\n\nstr.casefold()\n\n Return a casefolded copy of the string. Casefolded strings may be\n used for caseless matching.\n\n Casefolding is similar to lowercasing but more aggressive because\n it is intended to remove all case distinctions in a string. For\n example, the German lowercase letter ``\'\xc3\x9f\'`` is equivalent to\n ``"ss"``. Since it is already lowercase, ``lower()`` would do\n nothing to ``\'\xc3\x9f\'``; ``casefold()`` converts it to ``"ss"``.\n\n The casefolding algorithm is described in section 3.13 of the\n Unicode Standard.\n\n New in version 3.3.\n\nstr.center(width[, fillchar])\n\n Return centered in a string of length *width*. Padding is done\n using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n Return the number of non-overlapping occurrences of substring *sub*\n in the range [*start*, *end*]. Optional arguments *start* and\n *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n Return an encoded version of the string as a bytes object. Default\n encoding is ``\'utf-8\'``. *errors* may be given to set a different\n error handling scheme. The default for *errors* is ``\'strict\'``,\n meaning that encoding errors raise a ``UnicodeError``. Other\n possible values are ``\'ignore\'``, ``\'replace\'``,\n ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n Return ``True`` if the string ends with the specified *suffix*,\n otherwise return ``False``. *suffix* can also be a tuple of\n suffixes to look for. With optional *start*, test beginning at\n that position. With optional *end*, stop comparing at that\n position.\n\nstr.expandtabs([tabsize])\n\n Return a copy of the string where all tab characters are replaced\n by zero or more spaces, depending on the current column and the\n given tab size. The column number is reset to zero after each\n newline occurring in the string. If *tabsize* is not given, a tab\n size of ``8`` characters is assumed. This doesn\'t understand other\n non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n Return the lowest index in the string where substring *sub* is\n found, such that *sub* is contained in the slice ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` if *sub* is not found.\n\n Note: The ``find()`` method should be used only if you need to know the\n position of *sub*. To check if *sub* is a substring or not, use\n the ``in`` operator:\n\n >>> \'Py\' in \'Python\'\n True\n\nstr.format(*args, **kwargs)\n\n Perform a string formatting operation. The string on which this\n method is called can contain literal text or replacement fields\n delimited by braces ``{}``. Each replacement field contains either\n the numeric index of a positional argument, or the name of a\n keyword argument. Returns a copy of the string where each\n replacement field is replaced with the string value of the\n corresponding argument.\n\n >>> "The sum of 1 + 2 is {0}".format(1+2)\n \'The sum of 1 + 2 is 3\'\n\n See *Format String Syntax* for a description of the various\n formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n Similar to ``str.format(**mapping)``, except that ``mapping`` is\n used directly and not copied to a ``dict`` . This is useful if for\n example ``mapping`` is a dict subclass:\n\n >>> class Default(dict):\n ... def __missing__(self, key):\n ... return key\n ...\n >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n \'Guido was born in country\'\n\n New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n Like ``find()``, but raise ``ValueError`` when the substring is not\n found.\n\nstr.isalnum()\n\n Return true if all characters in the string are alphanumeric and\n there is at least one character, false otherwise. A character\n ``c`` is alphanumeric if one of the following returns ``True``:\n ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n ``c.isnumeric()``.\n\nstr.isalpha()\n\n Return true if all characters in the string are alphabetic and\n there is at least one character, false otherwise. Alphabetic\n characters are those characters defined in the Unicode character\n database as "Letter", i.e., those with general category property\n being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is\n different from the "Alphabetic" property defined in the Unicode\n Standard.\n\nstr.isdecimal()\n\n Return true if all characters in the string are decimal characters\n and there is at least one character, false otherwise. Decimal\n characters are those from general category "Nd". This category\n includes digit characters, and all characters that can be used to\n form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.\n\nstr.isdigit()\n\n Return true if all characters in the string are digits and there is\n at least one character, false otherwise. Digits include decimal\n characters and digits that need special handling, such as the\n compatibility superscript digits. Formally, a digit is a character\n that has the property value Numeric_Type=Digit or\n Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n Return true if the string is a valid identifier according to the\n language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n Return true if all cased characters [4] in the string are lowercase\n and there is at least one cased character, false otherwise.\n\nstr.isnumeric()\n\n Return true if all characters in the string are numeric characters,\n and there is at least one character, false otherwise. Numeric\n characters include digit characters, and all characters that have\n the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n ONE FIFTH. Formally, numeric characters are those with the\n property value Numeric_Type=Digit, Numeric_Type=Decimal or\n Numeric_Type=Numeric.\n\nstr.isprintable()\n\n Return true if all characters in the string are printable or the\n string is empty, false otherwise. Nonprintable characters are\n those characters defined in the Unicode character database as\n "Other" or "Separator", excepting the ASCII space (0x20) which is\n considered printable. (Note that printable characters in this\n context are those which should not be escaped when ``repr()`` is\n invoked on a string. It has no bearing on the handling of strings\n written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n Return true if there are only whitespace characters in the string\n and there is at least one character, false otherwise. Whitespace\n characters are those characters defined in the Unicode character\n database as "Other" or "Separator" and those with bidirectional\n property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n Return true if the string is a titlecased string and there is at\n least one character, for example uppercase characters may only\n follow uncased characters and lowercase characters only cased ones.\n Return false otherwise.\n\nstr.isupper()\n\n Return true if all cased characters [4] in the string are uppercase\n and there is at least one cased character, false otherwise.\n\nstr.join(iterable)\n\n Return a string which is the concatenation of the strings in the\n *iterable* *iterable*. A ``TypeError`` will be raised if there are\n any non-string values in *iterable*, including ``bytes`` objects.\n The separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n Return the string left justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.lower()\n\n Return a copy of the string with all the cased characters [4]\n converted to lowercase.\n\n The lowercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.lstrip([chars])\n\n Return a copy of the string with leading characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a prefix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.lstrip()\n \'spacious \'\n >>> \'www.example.com\'.lstrip(\'cmowz.\')\n \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n This static method returns a translation table usable for\n ``str.translate()``.\n\n If there is only one argument, it must be a dictionary mapping\n Unicode ordinals (integers) or characters (strings of length 1) to\n Unicode ordinals, strings (of arbitrary lengths) or None.\n Character keys will then be converted to ordinals.\n\n If there are two arguments, they must be strings of equal length,\n and in the resulting dictionary, each character in x will be mapped\n to the character at the same position in y. If there is a third\n argument, it must be a string, whose characters will be mapped to\n None in the result.\n\nstr.partition(sep)\n\n Split the string at the first occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing the string itself, followed by\n two empty strings.\n\nstr.replace(old, new[, count])\n\n Return a copy of the string with all occurrences of substring *old*\n replaced by *new*. If the optional argument *count* is given, only\n the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n Return the highest index in the string where substring *sub* is\n found, such that *sub* is contained within ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n is not found.\n\nstr.rjust(width[, fillchar])\n\n Return the string right justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.rpartition(sep)\n\n Split the string at the last occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing two empty strings, followed by\n the string itself.\n\nstr.rsplit(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n are done, the *rightmost* ones. If *sep* is not specified or\n ``None``, any whitespace string is a separator. Except for\n splitting from the right, ``rsplit()`` behaves like ``split()``\n which is described in detail below.\n\nstr.rstrip([chars])\n\n Return a copy of the string with trailing characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a suffix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.rstrip()\n \' spacious\'\n >>> \'mississippi\'.rstrip(\'ipz\')\n \'mississ\'\n\nstr.split(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit*\n splits are done (thus, the list will have at most ``maxsplit+1``\n elements). If *maxsplit* is not specified, then there is no limit\n on the number of splits (all possible splits are made).\n\n If *sep* is given, consecutive delimiters are not grouped together\n and are deemed to delimit empty strings (for example,\n ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``). The *sep*\n argument may consist of multiple characters (for example,\n ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n an empty string with a specified separator returns ``[\'\']``.\n\n If *sep* is not specified or is ``None``, a different splitting\n algorithm is applied: runs of consecutive whitespace are regarded\n as a single separator, and the result will contain no empty strings\n at the start or end if the string has leading or trailing\n whitespace. Consequently, splitting an empty string or a string\n consisting of just whitespace with a ``None`` separator returns\n ``[]``.\n\n For example, ``\' 1 2 3 \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n and ``\' 1 2 3 \'.split(None, 1)`` returns ``[\'1\', \'2 3 \']``.\n\nstr.splitlines([keepends])\n\n Return a list of the lines in the string, breaking at line\n boundaries. Line breaks are not included in the resulting list\n unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n Return ``True`` if string starts with the *prefix*, otherwise\n return ``False``. *prefix* can also be a tuple of prefixes to look\n for. With optional *start*, test string beginning at that\n position. With optional *end*, stop comparing string at that\n position.\n\nstr.strip([chars])\n\n Return a copy of the string with the leading and trailing\n characters removed. The *chars* argument is a string specifying the\n set of characters to be removed. If omitted or ``None``, the\n *chars* argument defaults to removing whitespace. The *chars*\n argument is not a prefix or suffix; rather, all combinations of its\n values are stripped:\n\n >>> \' spacious \'.strip()\n \'spacious\'\n >>> \'www.example.com\'.strip(\'cmowz.\')\n \'example\'\n\nstr.swapcase()\n\n Return a copy of the string with uppercase characters converted to\n lowercase and vice versa. Note that it is not necessarily true that\n ``s.swapcase().swapcase() == s``.\n\nstr.title()\n\n Return a titlecased version of the string where words start with an\n uppercase character and the remaining characters are lowercase.\n\n The algorithm uses a simple language-independent definition of a\n word as groups of consecutive letters. The definition works in\n many contexts but it means that apostrophes in contractions and\n possessives form word boundaries, which may not be the desired\n result:\n\n >>> "they\'re bill\'s friends from the UK".title()\n "They\'Re Bill\'S Friends From The Uk"\n\n A workaround for apostrophes can be constructed using regular\n expressions:\n\n >>> import re\n >>> def titlecase(s):\n return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n lambda mo: mo.group(0)[0].upper() +\n mo.group(0)[1:].lower(),\n s)\n\n >>> titlecase("they\'re bill\'s friends.")\n "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n Return a copy of the *s* where all characters have been mapped\n through the *map* which must be a dictionary of Unicode ordinals\n (integers) to Unicode ordinals, strings or ``None``. Unmapped\n characters are left untouched. Characters mapped to ``None`` are\n deleted.\n\n You can use ``str.maketrans()`` to create a translation map from\n character-to-character mappings in different formats.\n\n Note: An even more flexible approach is to create a custom character\n mapping codec using the ``codecs`` module (see\n ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n Return a copy of the string with all the cased characters [4]\n converted to uppercase. Note that ``str.upper().isupper()`` might\n be ``False`` if ``s`` contains uncased characters or if the Unicode\n category of the resulting character(s) is not "Lu" (Letter,\n uppercase), but e.g. "Lt" (Letter, titlecase).\n\n The uppercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.zfill(width)\n\n Return the numeric string left filled with zeros in a string of\n length *width*. A sign prefix is handled correctly. The original\n string is returned if *width* is less than or equal to ``len(s)``.\n', + 'specialnames': '\nSpecial method names\n********************\n\nA class can implement certain operations that are invoked by special\nsyntax (such as arithmetic operations or subscripting and slicing) by\ndefining methods with special names. This is Python\'s approach to\n*operator overloading*, allowing classes to define their own behavior\nwith respect to language operators. For instance, if a class defines\na method named ``__getitem__()``, and ``x`` is an instance of this\nclass, then ``x[i]`` is roughly equivalent to ``type(x).__getitem__(x,\ni)``. Except where mentioned, attempts to execute an operation raise\nan exception when no appropriate method is defined (typically\n``AttributeError`` or ``TypeError``).\n\nWhen implementing a class that emulates any built-in type, it is\nimportant that the emulation only be implemented to the degree that it\nmakes sense for the object being modelled. For example, some\nsequences may work well with retrieval of individual elements, but\nextracting a slice may not make sense. (One example of this is the\n``NodeList`` interface in the W3C\'s Document Object Model.)\n\n\nBasic customization\n===================\n\nobject.__new__(cls[, ...])\n\n Called to create a new instance of class *cls*. ``__new__()`` is a\n static method (special-cased so you need not declare it as such)\n that takes the class of which an instance was requested as its\n first argument. The remaining arguments are those passed to the\n object constructor expression (the call to the class). The return\n value of ``__new__()`` should be the new object instance (usually\n an instance of *cls*).\n\n Typical implementations create a new instance of the class by\n invoking the superclass\'s ``__new__()`` method using\n ``super(currentclass, cls).__new__(cls[, ...])`` with appropriate\n arguments and then modifying the newly-created instance as\n necessary before returning it.\n\n If ``__new__()`` returns an instance of *cls*, then the new\n instance\'s ``__init__()`` method will be invoked like\n ``__init__(self[, ...])``, where *self* is the new instance and the\n remaining arguments are the same as were passed to ``__new__()``.\n\n If ``__new__()`` does not return an instance of *cls*, then the new\n instance\'s ``__init__()`` method will not be invoked.\n\n ``__new__()`` is intended mainly to allow subclasses of immutable\n types (like int, str, or tuple) to customize instance creation. It\n is also commonly overridden in custom metaclasses in order to\n customize class creation.\n\nobject.__init__(self[, ...])\n\n Called when the instance is created. The arguments are those\n passed to the class constructor expression. If a base class has an\n ``__init__()`` method, the derived class\'s ``__init__()`` method,\n if any, must explicitly call it to ensure proper initialization of\n the base class part of the instance; for example:\n ``BaseClass.__init__(self, [args...])``. As a special constraint\n on constructors, no value may be returned; doing so will cause a\n ``TypeError`` to be raised at runtime.\n\nobject.__del__(self)\n\n Called when the instance is about to be destroyed. This is also\n called a destructor. If a base class has a ``__del__()`` method,\n the derived class\'s ``__del__()`` method, if any, must explicitly\n call it to ensure proper deletion of the base class part of the\n instance. Note that it is possible (though not recommended!) for\n the ``__del__()`` method to postpone destruction of the instance by\n creating a new reference to it. It may then be called at a later\n time when this new reference is deleted. It is not guaranteed that\n ``__del__()`` methods are called for objects that still exist when\n the interpreter exits.\n\n Note: ``del x`` doesn\'t directly call ``x.__del__()`` --- the former\n decrements the reference count for ``x`` by one, and the latter\n is only called when ``x``\'s reference count reaches zero. Some\n common situations that may prevent the reference count of an\n object from going to zero include: circular references between\n objects (e.g., a doubly-linked list or a tree data structure with\n parent and child pointers); a reference to the object on the\n stack frame of a function that caught an exception (the traceback\n stored in ``sys.exc_info()[2]`` keeps the stack frame alive); or\n a reference to the object on the stack frame that raised an\n unhandled exception in interactive mode (the traceback stored in\n ``sys.last_traceback`` keeps the stack frame alive). The first\n situation can only be remedied by explicitly breaking the cycles;\n the latter two situations can be resolved by storing ``None`` in\n ``sys.last_traceback``. Circular references which are garbage are\n detected when the option cycle detector is enabled (it\'s on by\n default), but can only be cleaned up if there are no Python-\n level ``__del__()`` methods involved. Refer to the documentation\n for the ``gc`` module for more information about how\n ``__del__()`` methods are handled by the cycle detector,\n particularly the description of the ``garbage`` value.\n\n Warning: Due to the precarious circumstances under which ``__del__()``\n methods are invoked, exceptions that occur during their execution\n are ignored, and a warning is printed to ``sys.stderr`` instead.\n Also, when ``__del__()`` is invoked in response to a module being\n deleted (e.g., when execution of the program is done), other\n globals referenced by the ``__del__()`` method may already have\n been deleted or in the process of being torn down (e.g. the\n import machinery shutting down). For this reason, ``__del__()``\n methods should do the absolute minimum needed to maintain\n external invariants. Starting with version 1.5, Python\n guarantees that globals whose name begins with a single\n underscore are deleted from their module before other globals are\n deleted; if no other references to such globals exist, this may\n help in assuring that imported modules are still available at the\n time when the ``__del__()`` method is called.\n\nobject.__repr__(self)\n\n Called by the ``repr()`` built-in function to compute the\n "official" string representation of an object. If at all possible,\n this should look like a valid Python expression that could be used\n to recreate an object with the same value (given an appropriate\n environment). If this is not possible, a string of the form\n ``<...some useful description...>`` should be returned. The return\n value must be a string object. If a class defines ``__repr__()``\n but not ``__str__()``, then ``__repr__()`` is also used when an\n "informal" string representation of instances of that class is\n required.\n\n This is typically used for debugging, so it is important that the\n representation is information-rich and unambiguous.\n\nobject.__str__(self)\n\n Called by the ``str()`` built-in function and by the ``print()``\n function to compute the "informal" string representation of an\n object. This differs from ``__repr__()`` in that it does not have\n to be a valid Python expression: a more convenient or concise\n representation may be used instead. The return value must be a\n string object.\n\nobject.__bytes__(self)\n\n Called by ``bytes()`` to compute a byte-string representation of an\n object. This should return a ``bytes`` object.\n\nobject.__format__(self, format_spec)\n\n Called by the ``format()`` built-in function (and by extension, the\n ``format()`` method of class ``str``) to produce a "formatted"\n string representation of an object. The ``format_spec`` argument is\n a string that contains a description of the formatting options\n desired. The interpretation of the ``format_spec`` argument is up\n to the type implementing ``__format__()``, however most classes\n will either delegate formatting to one of the built-in types, or\n use a similar formatting option syntax.\n\n See *Format Specification Mini-Language* for a description of the\n standard formatting syntax.\n\n The return value must be a string object.\n\nobject.__lt__(self, other)\nobject.__le__(self, other)\nobject.__eq__(self, other)\nobject.__ne__(self, other)\nobject.__gt__(self, other)\nobject.__ge__(self, other)\n\n These are the so-called "rich comparison" methods. The\n correspondence between operator symbols and method names is as\n follows: ``xy`` calls ``x.__gt__(y)``, and ``x>=y`` calls\n ``x.__ge__(y)``.\n\n A rich comparison method may return the singleton\n ``NotImplemented`` if it does not implement the operation for a\n given pair of arguments. By convention, ``False`` and ``True`` are\n returned for a successful comparison. However, these methods can\n return any value, so if the comparison operator is used in a\n Boolean context (e.g., in the condition of an ``if`` statement),\n Python will call ``bool()`` on the value to determine if the result\n is true or false.\n\n There are no implied relationships among the comparison operators.\n The truth of ``x==y`` does not imply that ``x!=y`` is false.\n Accordingly, when defining ``__eq__()``, one should also define\n ``__ne__()`` so that the operators will behave as expected. See\n the paragraph on ``__hash__()`` for some important notes on\n creating *hashable* objects which support custom comparison\n operations and are usable as dictionary keys.\n\n There are no swapped-argument versions of these methods (to be used\n when the left argument does not support the operation but the right\n argument does); rather, ``__lt__()`` and ``__gt__()`` are each\n other\'s reflection, ``__le__()`` and ``__ge__()`` are each other\'s\n reflection, and ``__eq__()`` and ``__ne__()`` are their own\n reflection.\n\n Arguments to rich comparison methods are never coerced.\n\n To automatically generate ordering operations from a single root\n operation, see ``functools.total_ordering()``.\n\nobject.__hash__(self)\n\n Called by built-in function ``hash()`` and for operations on\n members of hashed collections including ``set``, ``frozenset``, and\n ``dict``. ``__hash__()`` should return an integer. The only\n required property is that objects which compare equal have the same\n hash value; it is advised to somehow mix together (e.g. using\n exclusive or) the hash values for the components of the object that\n also play a part in comparison of objects.\n\n If a class does not define an ``__eq__()`` method it should not\n define a ``__hash__()`` operation either; if it defines\n ``__eq__()`` but not ``__hash__()``, its instances will not be\n usable as items in hashable collections. If a class defines\n mutable objects and implements an ``__eq__()`` method, it should\n not implement ``__hash__()``, since the implementation of hashable\n collections requires that a key\'s hash value is immutable (if the\n object\'s hash value changes, it will be in the wrong hash bucket).\n\n User-defined classes have ``__eq__()`` and ``__hash__()`` methods\n by default; with them, all objects compare unequal (except with\n themselves) and ``x.__hash__()`` returns an appropriate value such\n that ``x == y`` implies both that ``x is y`` and ``hash(x) ==\n hash(y)``.\n\n Classes which inherit a ``__hash__()`` method from a parent class\n but change the meaning of ``__eq__()`` such that the hash value\n returned is no longer appropriate (e.g. by switching to a value-\n based concept of equality instead of the default identity based\n equality) can explicitly flag themselves as being unhashable by\n setting ``__hash__ = None`` in the class definition. Doing so means\n that not only will instances of the class raise an appropriate\n ``TypeError`` when a program attempts to retrieve their hash value,\n but they will also be correctly identified as unhashable when\n checking ``isinstance(obj, collections.Hashable)`` (unlike classes\n which define their own ``__hash__()`` to explicitly raise\n ``TypeError``).\n\n If a class that overrides ``__eq__()`` needs to retain the\n implementation of ``__hash__()`` from a parent class, the\n interpreter must be told this explicitly by setting ``__hash__ =\n .__hash__``. Otherwise the inheritance of\n ``__hash__()`` will be blocked, just as if ``__hash__`` had been\n explicitly set to ``None``.\n\n Note: Note by default the ``__hash__()`` values of str, bytes and\n datetime objects are "salted" with an unpredictable random value.\n Although they remain constant within an individual Python\n process, they are not predictable between repeated invocations of\n Python.This is intended to provide protection against a denial-\n of-service caused by carefully-chosen inputs that exploit the\n worst case performance of a dict insertion, O(n^2) complexity.\n See http://www.ocert.org/advisories/ocert-2011-003.html for\n details.Changing hash values affects the order in which keys are\n retrieved from a dict. Note Python has never made guarantees\n about this ordering (and it typically varies between 32-bit and\n 64-bit builds).See also ``PYTHONHASHSEED``.\n\n Changed in version 3.3: Hash randomization is enabled by default.\n\nobject.__bool__(self)\n\n Called to implement truth value testing and the built-in operation\n ``bool()``; should return ``False`` or ``True``. When this method\n is not defined, ``__len__()`` is called, if it is defined, and the\n object is considered true if its result is nonzero. If a class\n defines neither ``__len__()`` nor ``__bool__()``, all its instances\n are considered true.\n\n\nCustomizing attribute access\n============================\n\nThe following methods can be defined to customize the meaning of\nattribute access (use of, assignment to, or deletion of ``x.name``)\nfor class instances.\n\nobject.__getattr__(self, name)\n\n Called when an attribute lookup has not found the attribute in the\n usual places (i.e. it is not an instance attribute nor is it found\n in the class tree for ``self``). ``name`` is the attribute name.\n This method should return the (computed) attribute value or raise\n an ``AttributeError`` exception.\n\n Note that if the attribute is found through the normal mechanism,\n ``__getattr__()`` is not called. (This is an intentional asymmetry\n between ``__getattr__()`` and ``__setattr__()``.) This is done both\n for efficiency reasons and because otherwise ``__getattr__()``\n would have no way to access other attributes of the instance. Note\n that at least for instance variables, you can fake total control by\n not inserting any values in the instance attribute dictionary (but\n instead inserting them in another object). See the\n ``__getattribute__()`` method below for a way to actually get total\n control over attribute access.\n\nobject.__getattribute__(self, name)\n\n Called unconditionally to implement attribute accesses for\n instances of the class. If the class also defines\n ``__getattr__()``, the latter will not be called unless\n ``__getattribute__()`` either calls it explicitly or raises an\n ``AttributeError``. This method should return the (computed)\n attribute value or raise an ``AttributeError`` exception. In order\n to avoid infinite recursion in this method, its implementation\n should always call the base class method with the same name to\n access any attributes it needs, for example,\n ``object.__getattribute__(self, name)``.\n\n Note: This method may still be bypassed when looking up special methods\n as the result of implicit invocation via language syntax or\n built-in functions. See *Special method lookup*.\n\nobject.__setattr__(self, name, value)\n\n Called when an attribute assignment is attempted. This is called\n instead of the normal mechanism (i.e. store the value in the\n instance dictionary). *name* is the attribute name, *value* is the\n value to be assigned to it.\n\n If ``__setattr__()`` wants to assign to an instance attribute, it\n should call the base class method with the same name, for example,\n ``object.__setattr__(self, name, value)``.\n\nobject.__delattr__(self, name)\n\n Like ``__setattr__()`` but for attribute deletion instead of\n assignment. This should only be implemented if ``del obj.name`` is\n meaningful for the object.\n\nobject.__dir__(self)\n\n Called when ``dir()`` is called on the object. A sequence must be\n returned. ``dir()`` converts the returned sequence to a list and\n sorts it.\n\n\nImplementing Descriptors\n------------------------\n\nThe following methods only apply when an instance of the class\ncontaining the method (a so-called *descriptor* class) appears in an\n*owner* class (the descriptor must be in either the owner\'s class\ndictionary or in the class dictionary for one of its parents). In the\nexamples below, "the attribute" refers to the attribute whose name is\nthe key of the property in the owner class\' ``__dict__``.\n\nobject.__get__(self, instance, owner)\n\n Called to get the attribute of the owner class (class attribute\n access) or of an instance of that class (instance attribute\n access). *owner* is always the owner class, while *instance* is the\n instance that the attribute was accessed through, or ``None`` when\n the attribute is accessed through the *owner*. This method should\n return the (computed) attribute value or raise an\n ``AttributeError`` exception.\n\nobject.__set__(self, instance, value)\n\n Called to set the attribute on an instance *instance* of the owner\n class to a new value, *value*.\n\nobject.__delete__(self, instance)\n\n Called to delete the attribute on an instance *instance* of the\n owner class.\n\n\nInvoking Descriptors\n--------------------\n\nIn general, a descriptor is an object attribute with "binding\nbehavior", one whose attribute access has been overridden by methods\nin the descriptor protocol: ``__get__()``, ``__set__()``, and\n``__delete__()``. If any of those methods are defined for an object,\nit is said to be a descriptor.\n\nThe default behavior for attribute access is to get, set, or delete\nthe attribute from an object\'s dictionary. For instance, ``a.x`` has a\nlookup chain starting with ``a.__dict__[\'x\']``, then\n``type(a).__dict__[\'x\']``, and continuing through the base classes of\n``type(a)`` excluding metaclasses.\n\nHowever, if the looked-up value is an object defining one of the\ndescriptor methods, then Python may override the default behavior and\ninvoke the descriptor method instead. Where this occurs in the\nprecedence chain depends on which descriptor methods were defined and\nhow they were called.\n\nThe starting point for descriptor invocation is a binding, ``a.x``.\nHow the arguments are assembled depends on ``a``:\n\nDirect Call\n The simplest and least common call is when user code directly\n invokes a descriptor method: ``x.__get__(a)``.\n\nInstance Binding\n If binding to an object instance, ``a.x`` is transformed into the\n call: ``type(a).__dict__[\'x\'].__get__(a, type(a))``.\n\nClass Binding\n If binding to a class, ``A.x`` is transformed into the call:\n ``A.__dict__[\'x\'].__get__(None, A)``.\n\nSuper Binding\n If ``a`` is an instance of ``super``, then the binding ``super(B,\n obj).m()`` searches ``obj.__class__.__mro__`` for the base class\n ``A`` immediately preceding ``B`` and then invokes the descriptor\n with the call: ``A.__dict__[\'m\'].__get__(obj, obj.__class__)``.\n\nFor instance bindings, the precedence of descriptor invocation depends\non the which descriptor methods are defined. A descriptor can define\nany combination of ``__get__()``, ``__set__()`` and ``__delete__()``.\nIf it does not define ``__get__()``, then accessing the attribute will\nreturn the descriptor object itself unless there is a value in the\nobject\'s instance dictionary. If the descriptor defines ``__set__()``\nand/or ``__delete__()``, it is a data descriptor; if it defines\nneither, it is a non-data descriptor. Normally, data descriptors\ndefine both ``__get__()`` and ``__set__()``, while non-data\ndescriptors have just the ``__get__()`` method. Data descriptors with\n``__set__()`` and ``__get__()`` defined always override a redefinition\nin an instance dictionary. In contrast, non-data descriptors can be\noverridden by instances.\n\nPython methods (including ``staticmethod()`` and ``classmethod()``)\nare implemented as non-data descriptors. Accordingly, instances can\nredefine and override methods. This allows individual instances to\nacquire behaviors that differ from other instances of the same class.\n\nThe ``property()`` function is implemented as a data descriptor.\nAccordingly, instances cannot override the behavior of a property.\n\n\n__slots__\n---------\n\nBy default, instances of classes have a dictionary for attribute\nstorage. This wastes space for objects having very few instance\nvariables. The space consumption can become acute when creating large\nnumbers of instances.\n\nThe default can be overridden by defining *__slots__* in a class\ndefinition. The *__slots__* declaration takes a sequence of instance\nvariables and reserves just enough space in each instance to hold a\nvalue for each variable. Space is saved because *__dict__* is not\ncreated for each instance.\n\nobject.__slots__\n\n This class variable can be assigned a string, iterable, or sequence\n of strings with variable names used by instances. If defined in a\n class, *__slots__* reserves space for the declared variables and\n prevents the automatic creation of *__dict__* and *__weakref__* for\n each instance.\n\n\nNotes on using *__slots__*\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n* When inheriting from a class without *__slots__*, the *__dict__*\n attribute of that class will always be accessible, so a *__slots__*\n definition in the subclass is meaningless.\n\n* Without a *__dict__* variable, instances cannot be assigned new\n variables not listed in the *__slots__* definition. Attempts to\n assign to an unlisted variable name raises ``AttributeError``. If\n dynamic assignment of new variables is desired, then add\n ``\'__dict__\'`` to the sequence of strings in the *__slots__*\n declaration.\n\n* Without a *__weakref__* variable for each instance, classes defining\n *__slots__* do not support weak references to its instances. If weak\n reference support is needed, then add ``\'__weakref__\'`` to the\n sequence of strings in the *__slots__* declaration.\n\n* *__slots__* are implemented at the class level by creating\n descriptors (*Implementing Descriptors*) for each variable name. As\n a result, class attributes cannot be used to set default values for\n instance variables defined by *__slots__*; otherwise, the class\n attribute would overwrite the descriptor assignment.\n\n* The action of a *__slots__* declaration is limited to the class\n where it is defined. As a result, subclasses will have a *__dict__*\n unless they also define *__slots__* (which must only contain names\n of any *additional* slots).\n\n* If a class defines a slot also defined in a base class, the instance\n variable defined by the base class slot is inaccessible (except by\n retrieving its descriptor directly from the base class). This\n renders the meaning of the program undefined. In the future, a\n check may be added to prevent this.\n\n* Nonempty *__slots__* does not work for classes derived from\n "variable-length" built-in types such as ``int``, ``str`` and\n ``tuple``.\n\n* Any non-string iterable may be assigned to *__slots__*. Mappings may\n also be used; however, in the future, special meaning may be\n assigned to the values corresponding to each key.\n\n* *__class__* assignment works only if both classes have the same\n *__slots__*.\n\n\nCustomizing class creation\n==========================\n\nBy default, classes are constructed using ``type()``. The class body\nis executed in a new namespace and the class name is bound locally to\nthe result of ``type(name, bases, namespace)``.\n\nThe class creation process can be customised by passing the\n``metaclass`` keyword argument in the class definition line, or by\ninheriting from an existing class that included such an argument. In\nthe following example, both ``MyClass`` and ``MySubclass`` are\ninstances of ``Meta``:\n\n class Meta(type):\n pass\n\n class MyClass(metaclass=Meta):\n pass\n\n class MySubclass(MyClass):\n pass\n\nAny other keyword arguments that are specified in the class definition\nare passed through to all metaclass operations described below.\n\nWhen a class definition is executed, the following steps occur:\n\n* the appropriate metaclass is determined\n\n* the class namespace is prepared\n\n* the class body is executed\n\n* the class object is created\n\n\nDetermining the appropriate metaclass\n-------------------------------------\n\nThe appropriate metaclass for a class definition is determined as\nfollows:\n\n* if no bases and no explicit metaclass are given, then ``type()`` is\n used\n\n* if an explicit metaclass is given and it is *not* an instance of\n ``type()``, then it is used directly as the metaclass\n\n* if an instance of ``type()`` is given as the explicit metaclass, or\n bases are defined, then the most derived metaclass is used\n\nThe most derived metaclass is selected from the explicitly specified\nmetaclass (if any) and the metaclasses (i.e. ``type(cls)``) of all\nspecified base classes. The most derived metaclass is one which is a\nsubtype of *all* of these candidate metaclasses. If none of the\ncandidate metaclasses meets that criterion, then the class definition\nwill fail with ``TypeError``.\n\n\nPreparing the class namespace\n-----------------------------\n\nOnce the appropriate metaclass has been identified, then the class\nnamespace is prepared. If the metaclass has a ``__prepare__``\nattribute, it is called as ``namespace = metaclass.__prepare__(name,\nbases, **kwds)`` (where the additional keyword arguments, if any, come\nfrom the class definition).\n\nIf the metaclass has no ``__prepare__`` attribute, then the class\nnamespace is initialised as an empty ``dict()`` instance.\n\nSee also:\n\n **PEP 3115** - Metaclasses in Python 3000\n Introduced the ``__prepare__`` namespace hook\n\n\nExecuting the class body\n------------------------\n\nThe class body is executed (approximately) as ``exec(body, globals(),\nnamespace)``. The key difference from a normal call to ``exec()`` is\nthat lexical scoping allows the class body (including any methods) to\nreference names from the current and outer scopes when the class\ndefinition occurs inside a function.\n\nHowever, even when the class definition occurs inside the function,\nmethods defined inside the class still cannot see names defined at the\nclass scope. Class variables must be accessed through the first\nparameter of instance or class methods, and cannot be accessed at all\nfrom static methods.\n\n\nCreating the class object\n-------------------------\n\nOnce the class namespace has been populated by executing the class\nbody, the class object is created by calling ``metaclass(name, bases,\nnamespace, **kwds)`` (the additional keywords passed here are the same\nas those passed to ``__prepare__``).\n\nThis class object is the one that will be referenced by the zero-\nargument form of ``super()``. ``__class__`` is an implicit closure\nreference created by the compiler if any methods in a class body refer\nto either ``__class__`` or ``super``. This allows the zero argument\nform of ``super()`` to correctly identify the class being defined\nbased on lexical scoping, while the class or instance that was used to\nmake the current call is identified based on the first argument passed\nto the method.\n\nAfter the class object is created, it is passed to the class\ndecorators included in the class definition (if any) and the resulting\nobject is bound in the local namespace as the defined class.\n\nSee also:\n\n **PEP 3135** - New super\n Describes the implicit ``__class__`` closure reference\n\n\nMetaclass example\n-----------------\n\nThe potential uses for metaclasses are boundless. Some ideas that have\nbeen explored include logging, interface checking, automatic\ndelegation, automatic property creation, proxies, frameworks, and\nautomatic resource locking/synchronization.\n\nHere is an example of a metaclass that uses an\n``collections.OrderedDict`` to remember the order that class members\nwere defined:\n\n class OrderedClass(type):\n\n @classmethod\n def __prepare__(metacls, name, bases, **kwds):\n return collections.OrderedDict()\n\n def __new__(cls, name, bases, namespace, **kwds):\n result = type.__new__(cls, name, bases, dict(namespace))\n result.members = tuple(namespace)\n return result\n\n class A(metaclass=OrderedClass):\n def one(self): pass\n def two(self): pass\n def three(self): pass\n def four(self): pass\n\n >>> A.members\n (\'__module__\', \'one\', \'two\', \'three\', \'four\')\n\nWhen the class definition for *A* gets executed, the process begins\nwith calling the metaclass\'s ``__prepare__()`` method which returns an\nempty ``collections.OrderedDict``. That mapping records the methods\nand attributes of *A* as they are defined within the body of the class\nstatement. Once those definitions are executed, the ordered dictionary\nis fully populated and the metaclass\'s ``__new__()`` method gets\ninvoked. That method builds the new type and it saves the ordered\ndictionary keys in an attribute called ``members``.\n\n\nCustomizing instance and subclass checks\n========================================\n\nThe following methods are used to override the default behavior of the\n``isinstance()`` and ``issubclass()`` built-in functions.\n\nIn particular, the metaclass ``abc.ABCMeta`` implements these methods\nin order to allow the addition of Abstract Base Classes (ABCs) as\n"virtual base classes" to any class or type (including built-in\ntypes), including other ABCs.\n\nclass.__instancecheck__(self, instance)\n\n Return true if *instance* should be considered a (direct or\n indirect) instance of *class*. If defined, called to implement\n ``isinstance(instance, class)``.\n\nclass.__subclasscheck__(self, subclass)\n\n Return true if *subclass* should be considered a (direct or\n indirect) subclass of *class*. If defined, called to implement\n ``issubclass(subclass, class)``.\n\nNote that these methods are looked up on the type (metaclass) of a\nclass. They cannot be defined as class methods in the actual class.\nThis is consistent with the lookup of special methods that are called\non instances, only in this case the instance is itself a class.\n\nSee also:\n\n **PEP 3119** - Introducing Abstract Base Classes\n Includes the specification for customizing ``isinstance()`` and\n ``issubclass()`` behavior through ``__instancecheck__()`` and\n ``__subclasscheck__()``, with motivation for this functionality\n in the context of adding Abstract Base Classes (see the ``abc``\n module) to the language.\n\n\nEmulating callable objects\n==========================\n\nobject.__call__(self[, args...])\n\n Called when the instance is "called" as a function; if this method\n is defined, ``x(arg1, arg2, ...)`` is a shorthand for\n ``x.__call__(arg1, arg2, ...)``.\n\n\nEmulating container types\n=========================\n\nThe following methods can be defined to implement container objects.\nContainers usually are sequences (such as lists or tuples) or mappings\n(like dictionaries), but can represent other containers as well. The\nfirst set of methods is used either to emulate a sequence or to\nemulate a mapping; the difference is that for a sequence, the\nallowable keys should be the integers *k* for which ``0 <= k < N``\nwhere *N* is the length of the sequence, or slice objects, which\ndefine a range of items. It is also recommended that mappings provide\nthe methods ``keys()``, ``values()``, ``items()``, ``get()``,\n``clear()``, ``setdefault()``, ``pop()``, ``popitem()``, ``copy()``,\nand ``update()`` behaving similar to those for Python\'s standard\ndictionary objects. The ``collections`` module provides a\n``MutableMapping`` abstract base class to help create those methods\nfrom a base set of ``__getitem__()``, ``__setitem__()``,\n``__delitem__()``, and ``keys()``. Mutable sequences should provide\nmethods ``append()``, ``count()``, ``index()``, ``extend()``,\n``insert()``, ``pop()``, ``remove()``, ``reverse()`` and ``sort()``,\nlike Python standard list objects. Finally, sequence types should\nimplement addition (meaning concatenation) and multiplication (meaning\nrepetition) by defining the methods ``__add__()``, ``__radd__()``,\n``__iadd__()``, ``__mul__()``, ``__rmul__()`` and ``__imul__()``\ndescribed below; they should not define other numerical operators. It\nis recommended that both mappings and sequences implement the\n``__contains__()`` method to allow efficient use of the ``in``\noperator; for mappings, ``in`` should search the mapping\'s keys; for\nsequences, it should search through the values. It is further\nrecommended that both mappings and sequences implement the\n``__iter__()`` method to allow efficient iteration through the\ncontainer; for mappings, ``__iter__()`` should be the same as\n``keys()``; for sequences, it should iterate through the values.\n\nobject.__len__(self)\n\n Called to implement the built-in function ``len()``. Should return\n the length of the object, an integer ``>=`` 0. Also, an object\n that doesn\'t define a ``__bool__()`` method and whose ``__len__()``\n method returns zero is considered to be false in a Boolean context.\n\nNote: Slicing is done exclusively with the following three methods. A\n call like\n\n a[1:2] = b\n\n is translated to\n\n a[slice(1, 2, None)] = b\n\n and so forth. Missing slice items are always filled in with\n ``None``.\n\nobject.__getitem__(self, key)\n\n Called to implement evaluation of ``self[key]``. For sequence\n types, the accepted keys should be integers and slice objects.\n Note that the special interpretation of negative indexes (if the\n class wishes to emulate a sequence type) is up to the\n ``__getitem__()`` method. If *key* is of an inappropriate type,\n ``TypeError`` may be raised; if of a value outside the set of\n indexes for the sequence (after any special interpretation of\n negative values), ``IndexError`` should be raised. For mapping\n types, if *key* is missing (not in the container), ``KeyError``\n should be raised.\n\n Note: ``for`` loops expect that an ``IndexError`` will be raised for\n illegal indexes to allow proper detection of the end of the\n sequence.\n\nobject.__setitem__(self, key, value)\n\n Called to implement assignment to ``self[key]``. Same note as for\n ``__getitem__()``. This should only be implemented for mappings if\n the objects support changes to the values for keys, or if new keys\n can be added, or for sequences if elements can be replaced. The\n same exceptions should be raised for improper *key* values as for\n the ``__getitem__()`` method.\n\nobject.__delitem__(self, key)\n\n Called to implement deletion of ``self[key]``. Same note as for\n ``__getitem__()``. This should only be implemented for mappings if\n the objects support removal of keys, or for sequences if elements\n can be removed from the sequence. The same exceptions should be\n raised for improper *key* values as for the ``__getitem__()``\n method.\n\nobject.__iter__(self)\n\n This method is called when an iterator is required for a container.\n This method should return a new iterator object that can iterate\n over all the objects in the container. For mappings, it should\n iterate over the keys of the container, and should also be made\n available as the method ``keys()``.\n\n Iterator objects also need to implement this method; they are\n required to return themselves. For more information on iterator\n objects, see *Iterator Types*.\n\nobject.__reversed__(self)\n\n Called (if present) by the ``reversed()`` built-in to implement\n reverse iteration. It should return a new iterator object that\n iterates over all the objects in the container in reverse order.\n\n If the ``__reversed__()`` method is not provided, the\n ``reversed()`` built-in will fall back to using the sequence\n protocol (``__len__()`` and ``__getitem__()``). Objects that\n support the sequence protocol should only provide\n ``__reversed__()`` if they can provide an implementation that is\n more efficient than the one provided by ``reversed()``.\n\nThe membership test operators (``in`` and ``not in``) are normally\nimplemented as an iteration through a sequence. However, container\nobjects can supply the following special method with a more efficient\nimplementation, which also does not require the object be a sequence.\n\nobject.__contains__(self, item)\n\n Called to implement membership test operators. Should return true\n if *item* is in *self*, false otherwise. For mapping objects, this\n should consider the keys of the mapping rather than the values or\n the key-item pairs.\n\n For objects that don\'t define ``__contains__()``, the membership\n test first tries iteration via ``__iter__()``, then the old\n sequence iteration protocol via ``__getitem__()``, see *this\n section in the language reference*.\n\n\nEmulating numeric types\n=======================\n\nThe following methods can be defined to emulate numeric objects.\nMethods corresponding to operations that are not supported by the\nparticular kind of number implemented (e.g., bitwise operations for\nnon-integral numbers) should be left undefined.\n\nobject.__add__(self, other)\nobject.__sub__(self, other)\nobject.__mul__(self, other)\nobject.__truediv__(self, other)\nobject.__floordiv__(self, other)\nobject.__mod__(self, other)\nobject.__divmod__(self, other)\nobject.__pow__(self, other[, modulo])\nobject.__lshift__(self, other)\nobject.__rshift__(self, other)\nobject.__and__(self, other)\nobject.__xor__(self, other)\nobject.__or__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations (``+``, ``-``, ``*``, ``/``, ``//``, ``%``,\n ``divmod()``, ``pow()``, ``**``, ``<<``, ``>>``, ``&``, ``^``,\n ``|``). For instance, to evaluate the expression ``x + y``, where\n *x* is an instance of a class that has an ``__add__()`` method,\n ``x.__add__(y)`` is called. The ``__divmod__()`` method should be\n the equivalent to using ``__floordiv__()`` and ``__mod__()``; it\n should not be related to ``__truediv__()``. Note that\n ``__pow__()`` should be defined to accept an optional third\n argument if the ternary version of the built-in ``pow()`` function\n is to be supported.\n\n If one of those methods does not support the operation with the\n supplied arguments, it should return ``NotImplemented``.\n\nobject.__radd__(self, other)\nobject.__rsub__(self, other)\nobject.__rmul__(self, other)\nobject.__rtruediv__(self, other)\nobject.__rfloordiv__(self, other)\nobject.__rmod__(self, other)\nobject.__rdivmod__(self, other)\nobject.__rpow__(self, other)\nobject.__rlshift__(self, other)\nobject.__rrshift__(self, other)\nobject.__rand__(self, other)\nobject.__rxor__(self, other)\nobject.__ror__(self, other)\n\n These methods are called to implement the binary arithmetic\n operations (``+``, ``-``, ``*``, ``/``, ``//``, ``%``,\n ``divmod()``, ``pow()``, ``**``, ``<<``, ``>>``, ``&``, ``^``,\n ``|``) with reflected (swapped) operands. These functions are only\n called if the left operand does not support the corresponding\n operation and the operands are of different types. [2] For\n instance, to evaluate the expression ``x - y``, where *y* is an\n instance of a class that has an ``__rsub__()`` method,\n ``y.__rsub__(x)`` is called if ``x.__sub__(y)`` returns\n *NotImplemented*.\n\n Note that ternary ``pow()`` will not try calling ``__rpow__()``\n (the coercion rules would become too complicated).\n\n Note: If the right operand\'s type is a subclass of the left operand\'s\n type and that subclass provides the reflected method for the\n operation, this method will be called before the left operand\'s\n non-reflected method. This behavior allows subclasses to\n override their ancestors\' operations.\n\nobject.__iadd__(self, other)\nobject.__isub__(self, other)\nobject.__imul__(self, other)\nobject.__itruediv__(self, other)\nobject.__ifloordiv__(self, other)\nobject.__imod__(self, other)\nobject.__ipow__(self, other[, modulo])\nobject.__ilshift__(self, other)\nobject.__irshift__(self, other)\nobject.__iand__(self, other)\nobject.__ixor__(self, other)\nobject.__ior__(self, other)\n\n These methods are called to implement the augmented arithmetic\n assignments (``+=``, ``-=``, ``*=``, ``/=``, ``//=``, ``%=``,\n ``**=``, ``<<=``, ``>>=``, ``&=``, ``^=``, ``|=``). These methods\n should attempt to do the operation in-place (modifying *self*) and\n return the result (which could be, but does not have to be,\n *self*). If a specific method is not defined, the augmented\n assignment falls back to the normal methods. For instance, to\n execute the statement ``x += y``, where *x* is an instance of a\n class that has an ``__iadd__()`` method, ``x.__iadd__(y)`` is\n called. If *x* is an instance of a class that does not define a\n ``__iadd__()`` method, ``x.__add__(y)`` and ``y.__radd__(x)`` are\n considered, as with the evaluation of ``x + y``.\n\nobject.__neg__(self)\nobject.__pos__(self)\nobject.__abs__(self)\nobject.__invert__(self)\n\n Called to implement the unary arithmetic operations (``-``, ``+``,\n ``abs()`` and ``~``).\n\nobject.__complex__(self)\nobject.__int__(self)\nobject.__float__(self)\nobject.__round__(self[, n])\n\n Called to implement the built-in functions ``complex()``,\n ``int()``, ``float()`` and ``round()``. Should return a value of\n the appropriate type.\n\nobject.__index__(self)\n\n Called to implement ``operator.index()``. Also called whenever\n Python needs an integer object (such as in slicing, or in the\n built-in ``bin()``, ``hex()`` and ``oct()`` functions). Must return\n an integer.\n\n\nWith Statement Context Managers\n===============================\n\nA *context manager* is an object that defines the runtime context to\nbe established when executing a ``with`` statement. The context\nmanager handles the entry into, and the exit from, the desired runtime\ncontext for the execution of the block of code. Context managers are\nnormally invoked using the ``with`` statement (described in section\n*The with statement*), but can also be used by directly invoking their\nmethods.\n\nTypical uses of context managers include saving and restoring various\nkinds of global state, locking and unlocking resources, closing opened\nfiles, etc.\n\nFor more information on context managers, see *Context Manager Types*.\n\nobject.__enter__(self)\n\n Enter the runtime context related to this object. The ``with``\n statement will bind this method\'s return value to the target(s)\n specified in the ``as`` clause of the statement, if any.\n\nobject.__exit__(self, exc_type, exc_value, traceback)\n\n Exit the runtime context related to this object. The parameters\n describe the exception that caused the context to be exited. If the\n context was exited without an exception, all three arguments will\n be ``None``.\n\n If an exception is supplied, and the method wishes to suppress the\n exception (i.e., prevent it from being propagated), it should\n return a true value. Otherwise, the exception will be processed\n normally upon exit from this method.\n\n Note that ``__exit__()`` methods should not reraise the passed-in\n exception; this is the caller\'s responsibility.\n\nSee also:\n\n **PEP 0343** - The "with" statement\n The specification, background, and examples for the Python\n ``with`` statement.\n\n\nSpecial method lookup\n=====================\n\nFor custom classes, implicit invocations of special methods are only\nguaranteed to work correctly if defined on an object\'s type, not in\nthe object\'s instance dictionary. That behaviour is the reason why\nthe following code raises an exception:\n\n >>> class C:\n ... pass\n ...\n >>> c = C()\n >>> c.__len__ = lambda: 5\n >>> len(c)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: object of type \'C\' has no len()\n\nThe rationale behind this behaviour lies with a number of special\nmethods such as ``__hash__()`` and ``__repr__()`` that are implemented\nby all objects, including type objects. If the implicit lookup of\nthese methods used the conventional lookup process, they would fail\nwhen invoked on the type object itself:\n\n >>> 1 .__hash__() == hash(1)\n True\n >>> int.__hash__() == hash(int)\n Traceback (most recent call last):\n File "", line 1, in \n TypeError: descriptor \'__hash__\' of \'int\' object needs an argument\n\nIncorrectly attempting to invoke an unbound method of a class in this\nway is sometimes referred to as \'metaclass confusion\', and is avoided\nby bypassing the instance when looking up special methods:\n\n >>> type(1).__hash__(1) == hash(1)\n True\n >>> type(int).__hash__(int) == hash(int)\n True\n\nIn addition to bypassing any instance attributes in the interest of\ncorrectness, implicit special method lookup generally also bypasses\nthe ``__getattribute__()`` method even of the object\'s metaclass:\n\n >>> class Meta(type):\n ... def __getattribute__(*args):\n ... print("Metaclass getattribute invoked")\n ... return type.__getattribute__(*args)\n ...\n >>> class C(object, metaclass=Meta):\n ... def __len__(self):\n ... return 10\n ... def __getattribute__(*args):\n ... print("Class getattribute invoked")\n ... return object.__getattribute__(*args)\n ...\n >>> c = C()\n >>> c.__len__() # Explicit lookup via instance\n Class getattribute invoked\n 10\n >>> type(c).__len__(c) # Explicit lookup via type\n Metaclass getattribute invoked\n 10\n >>> len(c) # Implicit lookup\n 10\n\nBypassing the ``__getattribute__()`` machinery in this fashion\nprovides significant scope for speed optimisations within the\ninterpreter, at the cost of some flexibility in the handling of\nspecial methods (the special method *must* be set on the class object\nitself in order to be consistently invoked by the interpreter).\n\n-[ Footnotes ]-\n\n[1] It *is* possible in some cases to change an object\'s type, under\n certain controlled conditions. It generally isn\'t a good idea\n though, since it can lead to some very strange behaviour if it is\n handled incorrectly.\n\n[2] For operands of the same type, it is assumed that if the non-\n reflected method (such as ``__add__()``) fails the operation is\n not supported, which is why the reflected method is not called.\n', + 'string-methods': '\nString Methods\n**************\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n Return a copy of the string with its first character capitalized\n and the rest lowercased.\n\nstr.casefold()\n\n Return a casefolded copy of the string. Casefolded strings may be\n used for caseless matching.\n\n Casefolding is similar to lowercasing but more aggressive because\n it is intended to remove all case distinctions in a string. For\n example, the German lowercase letter ``\'\xc3\x9f\'`` is equivalent to\n ``"ss"``. Since it is already lowercase, ``lower()`` would do\n nothing to ``\'\xc3\x9f\'``; ``casefold()`` converts it to ``"ss"``.\n\n The casefolding algorithm is described in section 3.13 of the\n Unicode Standard.\n\n New in version 3.3.\n\nstr.center(width[, fillchar])\n\n Return centered in a string of length *width*. Padding is done\n using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n Return the number of non-overlapping occurrences of substring *sub*\n in the range [*start*, *end*]. Optional arguments *start* and\n *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n Return an encoded version of the string as a bytes object. Default\n encoding is ``\'utf-8\'``. *errors* may be given to set a different\n error handling scheme. The default for *errors* is ``\'strict\'``,\n meaning that encoding errors raise a ``UnicodeError``. Other\n possible values are ``\'ignore\'``, ``\'replace\'``,\n ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n Return ``True`` if the string ends with the specified *suffix*,\n otherwise return ``False``. *suffix* can also be a tuple of\n suffixes to look for. With optional *start*, test beginning at\n that position. With optional *end*, stop comparing at that\n position.\n\nstr.expandtabs([tabsize])\n\n Return a copy of the string where all tab characters are replaced\n by zero or more spaces, depending on the current column and the\n given tab size. The column number is reset to zero after each\n newline occurring in the string. If *tabsize* is not given, a tab\n size of ``8`` characters is assumed. This doesn\'t understand other\n non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n Return the lowest index in the string where substring *sub* is\n found, such that *sub* is contained in the slice ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` if *sub* is not found.\n\n Note: The ``find()`` method should be used only if you need to know the\n position of *sub*. To check if *sub* is a substring or not, use\n the ``in`` operator:\n\n >>> \'Py\' in \'Python\'\n True\n\nstr.format(*args, **kwargs)\n\n Perform a string formatting operation. The string on which this\n method is called can contain literal text or replacement fields\n delimited by braces ``{}``. Each replacement field contains either\n the numeric index of a positional argument, or the name of a\n keyword argument. Returns a copy of the string where each\n replacement field is replaced with the string value of the\n corresponding argument.\n\n >>> "The sum of 1 + 2 is {0}".format(1+2)\n \'The sum of 1 + 2 is 3\'\n\n See *Format String Syntax* for a description of the various\n formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n Similar to ``str.format(**mapping)``, except that ``mapping`` is\n used directly and not copied to a ``dict`` . This is useful if for\n example ``mapping`` is a dict subclass:\n\n >>> class Default(dict):\n ... def __missing__(self, key):\n ... return key\n ...\n >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n \'Guido was born in country\'\n\n New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n Like ``find()``, but raise ``ValueError`` when the substring is not\n found.\n\nstr.isalnum()\n\n Return true if all characters in the string are alphanumeric and\n there is at least one character, false otherwise. A character\n ``c`` is alphanumeric if one of the following returns ``True``:\n ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n ``c.isnumeric()``.\n\nstr.isalpha()\n\n Return true if all characters in the string are alphabetic and\n there is at least one character, false otherwise. Alphabetic\n characters are those characters defined in the Unicode character\n database as "Letter", i.e., those with general category property\n being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is\n different from the "Alphabetic" property defined in the Unicode\n Standard.\n\nstr.isdecimal()\n\n Return true if all characters in the string are decimal characters\n and there is at least one character, false otherwise. Decimal\n characters are those from general category "Nd". This category\n includes digit characters, and all characters that can be used to\n form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.\n\nstr.isdigit()\n\n Return true if all characters in the string are digits and there is\n at least one character, false otherwise. Digits include decimal\n characters and digits that need special handling, such as the\n compatibility superscript digits. Formally, a digit is a character\n that has the property value Numeric_Type=Digit or\n Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n Return true if the string is a valid identifier according to the\n language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n Return true if all cased characters [4] in the string are lowercase\n and there is at least one cased character, false otherwise.\n\nstr.isnumeric()\n\n Return true if all characters in the string are numeric characters,\n and there is at least one character, false otherwise. Numeric\n characters include digit characters, and all characters that have\n the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n ONE FIFTH. Formally, numeric characters are those with the\n property value Numeric_Type=Digit, Numeric_Type=Decimal or\n Numeric_Type=Numeric.\n\nstr.isprintable()\n\n Return true if all characters in the string are printable or the\n string is empty, false otherwise. Nonprintable characters are\n those characters defined in the Unicode character database as\n "Other" or "Separator", excepting the ASCII space (0x20) which is\n considered printable. (Note that printable characters in this\n context are those which should not be escaped when ``repr()`` is\n invoked on a string. It has no bearing on the handling of strings\n written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n Return true if there are only whitespace characters in the string\n and there is at least one character, false otherwise. Whitespace\n characters are those characters defined in the Unicode character\n database as "Other" or "Separator" and those with bidirectional\n property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n Return true if the string is a titlecased string and there is at\n least one character, for example uppercase characters may only\n follow uncased characters and lowercase characters only cased ones.\n Return false otherwise.\n\nstr.isupper()\n\n Return true if all cased characters [4] in the string are uppercase\n and there is at least one cased character, false otherwise.\n\nstr.join(iterable)\n\n Return a string which is the concatenation of the strings in the\n *iterable* *iterable*. A ``TypeError`` will be raised if there are\n any non-string values in *iterable*, including ``bytes`` objects.\n The separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n Return the string left justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.lower()\n\n Return a copy of the string with all the cased characters [4]\n converted to lowercase.\n\n The lowercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.lstrip([chars])\n\n Return a copy of the string with leading characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a prefix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.lstrip()\n \'spacious \'\n >>> \'www.example.com\'.lstrip(\'cmowz.\')\n \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n This static method returns a translation table usable for\n ``str.translate()``.\n\n If there is only one argument, it must be a dictionary mapping\n Unicode ordinals (integers) or characters (strings of length 1) to\n Unicode ordinals, strings (of arbitrary lengths) or None.\n Character keys will then be converted to ordinals.\n\n If there are two arguments, they must be strings of equal length,\n and in the resulting dictionary, each character in x will be mapped\n to the character at the same position in y. If there is a third\n argument, it must be a string, whose characters will be mapped to\n None in the result.\n\nstr.partition(sep)\n\n Split the string at the first occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing the string itself, followed by\n two empty strings.\n\nstr.replace(old, new[, count])\n\n Return a copy of the string with all occurrences of substring *old*\n replaced by *new*. If the optional argument *count* is given, only\n the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n Return the highest index in the string where substring *sub* is\n found, such that *sub* is contained within ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n is not found.\n\nstr.rjust(width[, fillchar])\n\n Return the string right justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.rpartition(sep)\n\n Split the string at the last occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing two empty strings, followed by\n the string itself.\n\nstr.rsplit(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n are done, the *rightmost* ones. If *sep* is not specified or\n ``None``, any whitespace string is a separator. Except for\n splitting from the right, ``rsplit()`` behaves like ``split()``\n which is described in detail below.\n\nstr.rstrip([chars])\n\n Return a copy of the string with trailing characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a suffix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.rstrip()\n \' spacious\'\n >>> \'mississippi\'.rstrip(\'ipz\')\n \'mississ\'\n\nstr.split(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit*\n splits are done (thus, the list will have at most ``maxsplit+1``\n elements). If *maxsplit* is not specified or ``-1``, then there is\n no limit on the number of splits (all possible splits are made).\n\n If *sep* is given, consecutive delimiters are not grouped together\n and are deemed to delimit empty strings (for example,\n ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``). The *sep*\n argument may consist of multiple characters (for example,\n ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n an empty string with a specified separator returns ``[\'\']``.\n\n If *sep* is not specified or is ``None``, a different splitting\n algorithm is applied: runs of consecutive whitespace are regarded\n as a single separator, and the result will contain no empty strings\n at the start or end if the string has leading or trailing\n whitespace. Consequently, splitting an empty string or a string\n consisting of just whitespace with a ``None`` separator returns\n ``[]``.\n\n For example, ``\' 1 2 3 \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n and ``\' 1 2 3 \'.split(None, 1)`` returns ``[\'1\', \'2 3 \']``.\n\nstr.splitlines([keepends])\n\n Return a list of the lines in the string, breaking at line\n boundaries. Line breaks are not included in the resulting list\n unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n Return ``True`` if string starts with the *prefix*, otherwise\n return ``False``. *prefix* can also be a tuple of prefixes to look\n for. With optional *start*, test string beginning at that\n position. With optional *end*, stop comparing string at that\n position.\n\nstr.strip([chars])\n\n Return a copy of the string with the leading and trailing\n characters removed. The *chars* argument is a string specifying the\n set of characters to be removed. If omitted or ``None``, the\n *chars* argument defaults to removing whitespace. The *chars*\n argument is not a prefix or suffix; rather, all combinations of its\n values are stripped:\n\n >>> \' spacious \'.strip()\n \'spacious\'\n >>> \'www.example.com\'.strip(\'cmowz.\')\n \'example\'\n\nstr.swapcase()\n\n Return a copy of the string with uppercase characters converted to\n lowercase and vice versa. Note that it is not necessarily true that\n ``s.swapcase().swapcase() == s``.\n\nstr.title()\n\n Return a titlecased version of the string where words start with an\n uppercase character and the remaining characters are lowercase.\n\n The algorithm uses a simple language-independent definition of a\n word as groups of consecutive letters. The definition works in\n many contexts but it means that apostrophes in contractions and\n possessives form word boundaries, which may not be the desired\n result:\n\n >>> "they\'re bill\'s friends from the UK".title()\n "They\'Re Bill\'S Friends From The Uk"\n\n A workaround for apostrophes can be constructed using regular\n expressions:\n\n >>> import re\n >>> def titlecase(s):\n return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n lambda mo: mo.group(0)[0].upper() +\n mo.group(0)[1:].lower(),\n s)\n\n >>> titlecase("they\'re bill\'s friends.")\n "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n Return a copy of the *s* where all characters have been mapped\n through the *map* which must be a dictionary of Unicode ordinals\n (integers) to Unicode ordinals, strings or ``None``. Unmapped\n characters are left untouched. Characters mapped to ``None`` are\n deleted.\n\n You can use ``str.maketrans()`` to create a translation map from\n character-to-character mappings in different formats.\n\n Note: An even more flexible approach is to create a custom character\n mapping codec using the ``codecs`` module (see\n ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n Return a copy of the string with all the cased characters [4]\n converted to uppercase. Note that ``str.upper().isupper()`` might\n be ``False`` if ``s`` contains uncased characters or if the Unicode\n category of the resulting character(s) is not "Lu" (Letter,\n uppercase), but e.g. "Lt" (Letter, titlecase).\n\n The uppercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.zfill(width)\n\n Return the numeric string left filled with zeros in a string of\n length *width*. A sign prefix is handled correctly. The original\n string is returned if *width* is less than or equal to ``len(s)``.\n', 'strings': '\nString and Bytes literals\n*************************\n\nString literals are described by the following lexical definitions:\n\n stringliteral ::= [stringprefix](shortstring | longstring)\n stringprefix ::= "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR"\n shortstring ::= "\'" shortstringitem* "\'" | \'"\' shortstringitem* \'"\'\n longstring ::= "\'\'\'" longstringitem* "\'\'\'" | \'"""\' longstringitem* \'"""\'\n shortstringitem ::= shortstringchar | stringescapeseq\n longstringitem ::= longstringchar | stringescapeseq\n shortstringchar ::= \n longstringchar ::= \n stringescapeseq ::= "\\" \n\n bytesliteral ::= bytesprefix(shortbytes | longbytes)\n bytesprefix ::= "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB"\n shortbytes ::= "\'" shortbytesitem* "\'" | \'"\' shortbytesitem* \'"\'\n longbytes ::= "\'\'\'" longbytesitem* "\'\'\'" | \'"""\' longbytesitem* \'"""\'\n shortbytesitem ::= shortbyteschar | bytesescapeseq\n longbytesitem ::= longbyteschar | bytesescapeseq\n shortbyteschar ::= \n longbyteschar ::= \n bytesescapeseq ::= "\\" \n\nOne syntactic restriction not indicated by these productions is that\nwhitespace is not allowed between the ``stringprefix`` or\n``bytesprefix`` and the rest of the literal. The source character set\nis defined by the encoding declaration; it is UTF-8 if no encoding\ndeclaration is given in the source file; see section *Encoding\ndeclarations*.\n\nIn plain English: Both types of literals can be enclosed in matching\nsingle quotes (``\'``) or double quotes (``"``). They can also be\nenclosed in matching groups of three single or double quotes (these\nare generally referred to as *triple-quoted strings*). The backslash\n(``\\``) character is used to escape characters that otherwise have a\nspecial meaning, such as newline, backslash itself, or the quote\ncharacter.\n\nBytes literals are always prefixed with ``\'b\'`` or ``\'B\'``; they\nproduce an instance of the ``bytes`` type instead of the ``str`` type.\nThey may only contain ASCII characters; bytes with a numeric value of\n128 or greater must be expressed with escapes.\n\nAs of Python 3.3 it is possible again to prefix unicode strings with a\n``u`` prefix to simplify maintenance of dual 2.x and 3.x codebases.\n\nBoth string and bytes literals may optionally be prefixed with a\nletter ``\'r\'`` or ``\'R\'``; such strings are called *raw strings* and\ntreat backslashes as literal characters. As a result, in string\nliterals, ``\'\\U\'`` and ``\'\\u\'`` escapes in raw strings are not treated\nspecially.\n\n New in version 3.3: The ``\'rb\'`` prefix of raw bytes literals has\n been added as a synonym of ``\'br\'``.\n\n New in version 3.3: Support for the unicode legacy literal\n (``u\'value\'``) and other versions were reintroduced to simplify the\n maintenance of dual Python 2.x and 3.x codebases. See **PEP 414**\n for more information.\n\nIn triple-quoted strings, unescaped newlines and quotes are allowed\n(and are retained), except that three unescaped quotes in a row\nterminate the string. (A "quote" is the character used to open the\nstring, i.e. either ``\'`` or ``"``.)\n\nUnless an ``\'r\'`` or ``\'R\'`` prefix is present, escape sequences in\nstrings are interpreted according to rules similar to those used by\nStandard C. The recognized escape sequences are:\n\n+-------------------+-----------------------------------+---------+\n| Escape Sequence | Meaning | Notes |\n+===================+===================================+=========+\n| ``\\newline`` | Backslash and newline ignored | |\n+-------------------+-----------------------------------+---------+\n| ``\\\\`` | Backslash (``\\``) | |\n+-------------------+-----------------------------------+---------+\n| ``\\\'`` | Single quote (``\'``) | |\n+-------------------+-----------------------------------+---------+\n| ``\\"`` | Double quote (``"``) | |\n+-------------------+-----------------------------------+---------+\n| ``\\a`` | ASCII Bell (BEL) | |\n+-------------------+-----------------------------------+---------+\n| ``\\b`` | ASCII Backspace (BS) | |\n+-------------------+-----------------------------------+---------+\n| ``\\f`` | ASCII Formfeed (FF) | |\n+-------------------+-----------------------------------+---------+\n| ``\\n`` | ASCII Linefeed (LF) | |\n+-------------------+-----------------------------------+---------+\n| ``\\r`` | ASCII Carriage Return (CR) | |\n+-------------------+-----------------------------------+---------+\n| ``\\t`` | ASCII Horizontal Tab (TAB) | |\n+-------------------+-----------------------------------+---------+\n| ``\\v`` | ASCII Vertical Tab (VT) | |\n+-------------------+-----------------------------------+---------+\n| ``\\ooo`` | Character with octal value *ooo* | (1,3) |\n+-------------------+-----------------------------------+---------+\n| ``\\xhh`` | Character with hex value *hh* | (2,3) |\n+-------------------+-----------------------------------+---------+\n\nEscape sequences only recognized in string literals are:\n\n+-------------------+-----------------------------------+---------+\n| Escape Sequence | Meaning | Notes |\n+===================+===================================+=========+\n| ``\\N{name}`` | Character named *name* in the | (4) |\n| | Unicode database | |\n+-------------------+-----------------------------------+---------+\n| ``\\uxxxx`` | Character with 16-bit hex value | (5) |\n| | *xxxx* | |\n+-------------------+-----------------------------------+---------+\n| ``\\Uxxxxxxxx`` | Character with 32-bit hex value | (6) |\n| | *xxxxxxxx* | |\n+-------------------+-----------------------------------+---------+\n\nNotes:\n\n1. As in Standard C, up to three octal digits are accepted.\n\n2. Unlike in Standard C, exactly two hex digits are required.\n\n3. In a bytes literal, hexadecimal and octal escapes denote the byte\n with the given value. In a string literal, these escapes denote a\n Unicode character with the given value.\n\n4. Changed in version 3.3: Support for name aliases [1] has been\n added.\n\n5. Individual code units which form parts of a surrogate pair can be\n encoded using this escape sequence. Exactly four hex digits are\n required.\n\n6. Any Unicode character can be encoded this way, but characters\n outside the Basic Multilingual Plane (BMP) will be encoded using a\n surrogate pair if Python is compiled to use 16-bit code units (the\n default). Exactly eight hex digits are required.\n\nUnlike Standard C, all unrecognized escape sequences are left in the\nstring unchanged, i.e., *the backslash is left in the string*. (This\nbehavior is useful when debugging: if an escape sequence is mistyped,\nthe resulting output is more easily recognized as broken.) It is also\nimportant to note that the escape sequences only recognized in string\nliterals fall into the category of unrecognized escapes for bytes\nliterals.\n\nEven in a raw string, string quotes can be escaped with a backslash,\nbut the backslash remains in the string; for example, ``r"\\""`` is a\nvalid string literal consisting of two characters: a backslash and a\ndouble quote; ``r"\\"`` is not a valid string literal (even a raw\nstring cannot end in an odd number of backslashes). Specifically, *a\nraw string cannot end in a single backslash* (since the backslash\nwould escape the following quote character). Note also that a single\nbackslash followed by a newline is interpreted as those two characters\nas part of the string, *not* as a line continuation.\n', 'subscriptions': '\nSubscriptions\n*************\n\nA subscription selects an item of a sequence (string, tuple or list)\nor mapping (dictionary) object:\n\n subscription ::= primary "[" expression_list "]"\n\nThe primary must evaluate to an object that supports subscription,\ne.g. a list or dictionary. User-defined objects can support\nsubscription by defining a ``__getitem__()`` method.\n\nFor built-in objects, there are two types of objects that support\nsubscription:\n\nIf the primary is a mapping, the expression list must evaluate to an\nobject whose value is one of the keys of the mapping, and the\nsubscription selects the value in the mapping that corresponds to that\nkey. (The expression list is a tuple except if it has exactly one\nitem.)\n\nIf the primary is a sequence, the expression (list) must evaluate to\nan integer or a slice (as discussed in the following section).\n\nThe formal syntax makes no special provision for negative indices in\nsequences; however, built-in sequences all provide a ``__getitem__()``\nmethod that interprets negative indices by adding the length of the\nsequence to the index (so that ``x[-1]`` selects the last item of\n``x``). The resulting value must be a nonnegative integer less than\nthe number of items in the sequence, and the subscription selects the\nitem whose index is that value (counting from zero). Since the support\nfor negative indices and slicing occurs in the object\'s\n``__getitem__()`` method, subclasses overriding this method will need\nto explicitly add that support.\n\nA string\'s items are characters. A character is not a separate data\ntype but a string of exactly one character.\n', 'truth': "\nTruth Value Testing\n*******************\n\nAny object can be tested for truth value, for use in an ``if`` or\n``while`` condition or as operand of the Boolean operations below. The\nfollowing values are considered false:\n\n* ``None``\n\n* ``False``\n\n* zero of any numeric type, for example, ``0``, ``0.0``, ``0j``.\n\n* any empty sequence, for example, ``''``, ``()``, ``[]``.\n\n* any empty mapping, for example, ``{}``.\n\n* instances of user-defined classes, if the class defines a\n ``__bool__()`` or ``__len__()`` method, when that method returns the\n integer zero or ``bool`` value ``False``. [1]\n\nAll other values are considered true --- so objects of many types are\nalways true.\n\nOperations and built-in functions that have a Boolean result always\nreturn ``0`` or ``False`` for false and ``1`` or ``True`` for true,\nunless otherwise stated. (Important exception: the Boolean operations\n``or`` and ``and`` always return one of their operands.)\n", 'try': '\nThe ``try`` statement\n*********************\n\nThe ``try`` statement specifies exception handlers and/or cleanup code\nfor a group of statements:\n\n try_stmt ::= try1_stmt | try2_stmt\n try1_stmt ::= "try" ":" suite\n ("except" [expression ["as" target]] ":" suite)+\n ["else" ":" suite]\n ["finally" ":" suite]\n try2_stmt ::= "try" ":" suite\n "finally" ":" suite\n\nThe ``except`` clause(s) specify one or more exception handlers. When\nno exception occurs in the ``try`` clause, no exception handler is\nexecuted. When an exception occurs in the ``try`` suite, a search for\nan exception handler is started. This search inspects the except\nclauses in turn until one is found that matches the exception. An\nexpression-less except clause, if present, must be last; it matches\nany exception. For an except clause with an expression, that\nexpression is evaluated, and the clause matches the exception if the\nresulting object is "compatible" with the exception. An object is\ncompatible with an exception if it is the class or a base class of the\nexception object or a tuple containing an item compatible with the\nexception.\n\nIf no except clause matches the exception, the search for an exception\nhandler continues in the surrounding code and on the invocation stack.\n[1]\n\nIf the evaluation of an expression in the header of an except clause\nraises an exception, the original search for a handler is canceled and\na search starts for the new exception in the surrounding code and on\nthe call stack (it is treated as if the entire ``try`` statement\nraised the exception).\n\nWhen a matching except clause is found, the exception is assigned to\nthe target specified after the ``as`` keyword in that except clause,\nif present, and the except clause\'s suite is executed. All except\nclauses must have an executable block. When the end of this block is\nreached, execution continues normally after the entire try statement.\n(This means that if two nested handlers exist for the same exception,\nand the exception occurs in the try clause of the inner handler, the\nouter handler will not handle the exception.)\n\nWhen an exception has been assigned using ``as target``, it is cleared\nat the end of the except clause. This is as if\n\n except E as N:\n foo\n\nwas translated to\n\n except E as N:\n try:\n foo\n finally:\n del N\n\nThis means the exception must be assigned to a different name to be\nable to refer to it after the except clause. Exceptions are cleared\nbecause with the traceback attached to them, they form a reference\ncycle with the stack frame, keeping all locals in that frame alive\nuntil the next garbage collection occurs.\n\nBefore an except clause\'s suite is executed, details about the\nexception are stored in the ``sys`` module and can be access via\n``sys.exc_info()``. ``sys.exc_info()`` returns a 3-tuple consisting of\nthe exception class, the exception instance and a traceback object\n(see section *The standard type hierarchy*) identifying the point in\nthe program where the exception occurred. ``sys.exc_info()`` values\nare restored to their previous values (before the call) when returning\nfrom a function that handled an exception.\n\nThe optional ``else`` clause is executed if and when control flows off\nthe end of the ``try`` clause. [2] Exceptions in the ``else`` clause\nare not handled by the preceding ``except`` clauses.\n\nIf ``finally`` is present, it specifies a \'cleanup\' handler. The\n``try`` clause is executed, including any ``except`` and ``else``\nclauses. If an exception occurs in any of the clauses and is not\nhandled, the exception is temporarily saved. The ``finally`` clause is\nexecuted. If there is a saved exception, it is re-raised at the end\nof the ``finally`` clause. If the ``finally`` clause raises another\nexception or executes a ``return`` or ``break`` statement, the saved\nexception is set as the context of the new exception. The exception\ninformation is not available to the program during execution of the\n``finally`` clause.\n\nWhen a ``return``, ``break`` or ``continue`` statement is executed in\nthe ``try`` suite of a ``try``...``finally`` statement, the\n``finally`` clause is also executed \'on the way out.\' A ``continue``\nstatement is illegal in the ``finally`` clause. (The reason is a\nproblem with the current implementation --- this restriction may be\nlifted in the future).\n\nAdditional information on exceptions can be found in section\n*Exceptions*, and information on using the ``raise`` statement to\ngenerate exceptions may be found in section *The raise statement*.\n', 'types': '\nThe standard type hierarchy\n***************************\n\nBelow is a list of the types that are built into Python. Extension\nmodules (written in C, Java, or other languages, depending on the\nimplementation) can define additional types. Future versions of\nPython may add types to the type hierarchy (e.g., rational numbers,\nefficiently stored arrays of integers, etc.), although such additions\nwill often be provided via the standard library instead.\n\nSome of the type descriptions below contain a paragraph listing\n\'special attributes.\' These are attributes that provide access to the\nimplementation and are not intended for general use. Their definition\nmay change in the future.\n\nNone\n This type has a single value. There is a single object with this\n value. This object is accessed through the built-in name ``None``.\n It is used to signify the absence of a value in many situations,\n e.g., it is returned from functions that don\'t explicitly return\n anything. Its truth value is false.\n\nNotImplemented\n This type has a single value. There is a single object with this\n value. This object is accessed through the built-in name\n ``NotImplemented``. Numeric methods and rich comparison methods may\n return this value if they do not implement the operation for the\n operands provided. (The interpreter will then try the reflected\n operation, or some other fallback, depending on the operator.) Its\n truth value is true.\n\nEllipsis\n This type has a single value. There is a single object with this\n value. This object is accessed through the literal ``...`` or the\n built-in name ``Ellipsis``. Its truth value is true.\n\n``numbers.Number``\n These are created by numeric literals and returned as results by\n arithmetic operators and arithmetic built-in functions. Numeric\n objects are immutable; once created their value never changes.\n Python numbers are of course strongly related to mathematical\n numbers, but subject to the limitations of numerical representation\n in computers.\n\n Python distinguishes between integers, floating point numbers, and\n complex numbers:\n\n ``numbers.Integral``\n These represent elements from the mathematical set of integers\n (positive and negative).\n\n There are two types of integers:\n\n Integers (``int``)\n\n These represent numbers in an unlimited range, subject to\n available (virtual) memory only. For the purpose of shift\n and mask operations, a binary representation is assumed, and\n negative numbers are represented in a variant of 2\'s\n complement which gives the illusion of an infinite string of\n sign bits extending to the left.\n\n Booleans (``bool``)\n These represent the truth values False and True. The two\n objects representing the values False and True are the only\n Boolean objects. The Boolean type is a subtype of the integer\n type, and Boolean values behave like the values 0 and 1,\n respectively, in almost all contexts, the exception being\n that when converted to a string, the strings ``"False"`` or\n ``"True"`` are returned, respectively.\n\n The rules for integer representation are intended to give the\n most meaningful interpretation of shift and mask operations\n involving negative integers.\n\n ``numbers.Real`` (``float``)\n These represent machine-level double precision floating point\n numbers. You are at the mercy of the underlying machine\n architecture (and C or Java implementation) for the accepted\n range and handling of overflow. Python does not support single-\n precision floating point numbers; the savings in processor and\n memory usage that are usually the reason for using these is\n dwarfed by the overhead of using objects in Python, so there is\n no reason to complicate the language with two kinds of floating\n point numbers.\n\n ``numbers.Complex`` (``complex``)\n These represent complex numbers as a pair of machine-level\n double precision floating point numbers. The same caveats apply\n as for floating point numbers. The real and imaginary parts of a\n complex number ``z`` can be retrieved through the read-only\n attributes ``z.real`` and ``z.imag``.\n\nSequences\n These represent finite ordered sets indexed by non-negative\n numbers. The built-in function ``len()`` returns the number of\n items of a sequence. When the length of a sequence is *n*, the\n index set contains the numbers 0, 1, ..., *n*-1. Item *i* of\n sequence *a* is selected by ``a[i]``.\n\n Sequences also support slicing: ``a[i:j]`` selects all items with\n index *k* such that *i* ``<=`` *k* ``<`` *j*. When used as an\n expression, a slice is a sequence of the same type. This implies\n that the index set is renumbered so that it starts at 0.\n\n Some sequences also support "extended slicing" with a third "step"\n parameter: ``a[i:j:k]`` selects all items of *a* with index *x*\n where ``x = i + n*k``, *n* ``>=`` ``0`` and *i* ``<=`` *x* ``<``\n *j*.\n\n Sequences are distinguished according to their mutability:\n\n Immutable sequences\n An object of an immutable sequence type cannot change once it is\n created. (If the object contains references to other objects,\n these other objects may be mutable and may be changed; however,\n the collection of objects directly referenced by an immutable\n object cannot change.)\n\n The following types are immutable sequences:\n\n Strings\n A string is a sequence of values that represent Unicode\n codepoints. All the codepoints in range ``U+0000 - U+10FFFF``\n can be represented in a string. Python doesn\'t have a\n ``chr`` type, and every character in the string is\n represented as a string object with length ``1``. The built-\n in function ``ord()`` converts a character to its codepoint\n (as an integer); ``chr()`` converts an integer in range ``0 -\n 10FFFF`` to the corresponding character. ``str.encode()`` can\n be used to convert a ``str`` to ``bytes`` using the given\n encoding, and ``bytes.decode()`` can be used to achieve the\n opposite.\n\n Tuples\n The items of a tuple are arbitrary Python objects. Tuples of\n two or more items are formed by comma-separated lists of\n expressions. A tuple of one item (a \'singleton\') can be\n formed by affixing a comma to an expression (an expression by\n itself does not create a tuple, since parentheses must be\n usable for grouping of expressions). An empty tuple can be\n formed by an empty pair of parentheses.\n\n Bytes\n A bytes object is an immutable array. The items are 8-bit\n bytes, represented by integers in the range 0 <= x < 256.\n Bytes literals (like ``b\'abc\'`` and the built-in function\n ``bytes()`` can be used to construct bytes objects. Also,\n bytes objects can be decoded to strings via the ``decode()``\n method.\n\n Mutable sequences\n Mutable sequences can be changed after they are created. The\n subscription and slicing notations can be used as the target of\n assignment and ``del`` (delete) statements.\n\n There are currently two intrinsic mutable sequence types:\n\n Lists\n The items of a list are arbitrary Python objects. Lists are\n formed by placing a comma-separated list of expressions in\n square brackets. (Note that there are no special cases needed\n to form lists of length 0 or 1.)\n\n Byte Arrays\n A bytearray object is a mutable array. They are created by\n the built-in ``bytearray()`` constructor. Aside from being\n mutable (and hence unhashable), byte arrays otherwise provide\n the same interface and functionality as immutable bytes\n objects.\n\n The extension module ``array`` provides an additional example of\n a mutable sequence type, as does the ``collections`` module.\n\nSet types\n These represent unordered, finite sets of unique, immutable\n objects. As such, they cannot be indexed by any subscript. However,\n they can be iterated over, and the built-in function ``len()``\n returns the number of items in a set. Common uses for sets are fast\n membership testing, removing duplicates from a sequence, and\n computing mathematical operations such as intersection, union,\n difference, and symmetric difference.\n\n For set elements, the same immutability rules apply as for\n dictionary keys. Note that numeric types obey the normal rules for\n numeric comparison: if two numbers compare equal (e.g., ``1`` and\n ``1.0``), only one of them can be contained in a set.\n\n There are currently two intrinsic set types:\n\n Sets\n These represent a mutable set. They are created by the built-in\n ``set()`` constructor and can be modified afterwards by several\n methods, such as ``add()``.\n\n Frozen sets\n These represent an immutable set. They are created by the\n built-in ``frozenset()`` constructor. As a frozenset is\n immutable and *hashable*, it can be used again as an element of\n another set, or as a dictionary key.\n\nMappings\n These represent finite sets of objects indexed by arbitrary index\n sets. The subscript notation ``a[k]`` selects the item indexed by\n ``k`` from the mapping ``a``; this can be used in expressions and\n as the target of assignments or ``del`` statements. The built-in\n function ``len()`` returns the number of items in a mapping.\n\n There is currently a single intrinsic mapping type:\n\n Dictionaries\n These represent finite sets of objects indexed by nearly\n arbitrary values. The only types of values not acceptable as\n keys are values containing lists or dictionaries or other\n mutable types that are compared by value rather than by object\n identity, the reason being that the efficient implementation of\n dictionaries requires a key\'s hash value to remain constant.\n Numeric types used for keys obey the normal rules for numeric\n comparison: if two numbers compare equal (e.g., ``1`` and\n ``1.0``) then they can be used interchangeably to index the same\n dictionary entry.\n\n Dictionaries are mutable; they can be created by the ``{...}``\n notation (see section *Dictionary displays*).\n\n The extension modules ``dbm.ndbm`` and ``dbm.gnu`` provide\n additional examples of mapping types, as does the\n ``collections`` module.\n\nCallable types\n These are the types to which the function call operation (see\n section *Calls*) can be applied:\n\n User-defined functions\n A user-defined function object is created by a function\n definition (see section *Function definitions*). It should be\n called with an argument list containing the same number of items\n as the function\'s formal parameter list.\n\n Special attributes:\n\n +---------------------------+---------------------------------+-------------+\n | Attribute | Meaning | |\n +===========================+=================================+=============+\n | ``__doc__`` | The function\'s documentation | Writable |\n | | string, or ``None`` if | |\n | | unavailable | |\n +---------------------------+---------------------------------+-------------+\n | ``__name__`` | The function\'s name | Writable |\n +---------------------------+---------------------------------+-------------+\n | ``__qualname__`` | The function\'s *qualified name* | Writable |\n | | New in version 3.3. | |\n +---------------------------+---------------------------------+-------------+\n | ``__module__`` | The name of the module the | Writable |\n | | function was defined in, or | |\n | | ``None`` if unavailable. | |\n +---------------------------+---------------------------------+-------------+\n | ``__defaults__`` | A tuple containing default | Writable |\n | | argument values for those | |\n | | arguments that have defaults, | |\n | | or ``None`` if no arguments | |\n | | have a default value | |\n +---------------------------+---------------------------------+-------------+\n | ``__code__`` | The code object representing | Writable |\n | | the compiled function body. | |\n +---------------------------+---------------------------------+-------------+\n | ``__globals__`` | A reference to the dictionary | Read-only |\n | | that holds the function\'s | |\n | | global variables --- the global | |\n | | namespace of the module in | |\n | | which the function was defined. | |\n +---------------------------+---------------------------------+-------------+\n | ``__dict__`` | The namespace supporting | Writable |\n | | arbitrary function attributes. | |\n +---------------------------+---------------------------------+-------------+\n | ``__closure__`` | ``None`` or a tuple of cells | Read-only |\n | | that contain bindings for the | |\n | | function\'s free variables. | |\n +---------------------------+---------------------------------+-------------+\n | ``__annotations__`` | A dict containing annotations | Writable |\n | | of parameters. The keys of the | |\n | | dict are the parameter names, | |\n | | or ``\'return\'`` for the return | |\n | | annotation, if provided. | |\n +---------------------------+---------------------------------+-------------+\n | ``__kwdefaults__`` | A dict containing defaults for | Writable |\n | | keyword-only parameters. | |\n +---------------------------+---------------------------------+-------------+\n\n Most of the attributes labelled "Writable" check the type of the\n assigned value.\n\n Function objects also support getting and setting arbitrary\n attributes, which can be used, for example, to attach metadata\n to functions. Regular attribute dot-notation is used to get and\n set such attributes. *Note that the current implementation only\n supports function attributes on user-defined functions. Function\n attributes on built-in functions may be supported in the\n future.*\n\n Additional information about a function\'s definition can be\n retrieved from its code object; see the description of internal\n types below.\n\n Instance methods\n An instance method object combines a class, a class instance and\n any callable object (normally a user-defined function).\n\n Special read-only attributes: ``__self__`` is the class instance\n object, ``__func__`` is the function object; ``__doc__`` is the\n method\'s documentation (same as ``__func__.__doc__``);\n ``__name__`` is the method name (same as ``__func__.__name__``);\n ``__module__`` is the name of the module the method was defined\n in, or ``None`` if unavailable.\n\n Methods also support accessing (but not setting) the arbitrary\n function attributes on the underlying function object.\n\n User-defined method objects may be created when getting an\n attribute of a class (perhaps via an instance of that class), if\n that attribute is a user-defined function object or a class\n method object.\n\n When an instance method object is created by retrieving a user-\n defined function object from a class via one of its instances,\n its ``__self__`` attribute is the instance, and the method\n object is said to be bound. The new method\'s ``__func__``\n attribute is the original function object.\n\n When a user-defined method object is created by retrieving\n another method object from a class or instance, the behaviour is\n the same as for a function object, except that the ``__func__``\n attribute of the new instance is not the original method object\n but its ``__func__`` attribute.\n\n When an instance method object is created by retrieving a class\n method object from a class or instance, its ``__self__``\n attribute is the class itself, and its ``__func__`` attribute is\n the function object underlying the class method.\n\n When an instance method object is called, the underlying\n function (``__func__``) is called, inserting the class instance\n (``__self__``) in front of the argument list. For instance,\n when ``C`` is a class which contains a definition for a function\n ``f()``, and ``x`` is an instance of ``C``, calling ``x.f(1)``\n is equivalent to calling ``C.f(x, 1)``.\n\n When an instance method object is derived from a class method\n object, the "class instance" stored in ``__self__`` will\n actually be the class itself, so that calling either ``x.f(1)``\n or ``C.f(1)`` is equivalent to calling ``f(C,1)`` where ``f`` is\n the underlying function.\n\n Note that the transformation from function object to instance\n method object happens each time the attribute is retrieved from\n the instance. In some cases, a fruitful optimization is to\n assign the attribute to a local variable and call that local\n variable. Also notice that this transformation only happens for\n user-defined functions; other callable objects (and all non-\n callable objects) are retrieved without transformation. It is\n also important to note that user-defined functions which are\n attributes of a class instance are not converted to bound\n methods; this *only* happens when the function is an attribute\n of the class.\n\n Generator functions\n A function or method which uses the ``yield`` statement (see\n section *The yield statement*) is called a *generator function*.\n Such a function, when called, always returns an iterator object\n which can be used to execute the body of the function: calling\n the iterator\'s ``__next__()`` method will cause the function to\n execute until it provides a value using the ``yield`` statement.\n When the function executes a ``return`` statement or falls off\n the end, a ``StopIteration`` exception is raised and the\n iterator will have reached the end of the set of values to be\n returned.\n\n Built-in functions\n A built-in function object is a wrapper around a C function.\n Examples of built-in functions are ``len()`` and ``math.sin()``\n (``math`` is a standard built-in module). The number and type of\n the arguments are determined by the C function. Special read-\n only attributes: ``__doc__`` is the function\'s documentation\n string, or ``None`` if unavailable; ``__name__`` is the\n function\'s name; ``__self__`` is set to ``None`` (but see the\n next item); ``__module__`` is the name of the module the\n function was defined in or ``None`` if unavailable.\n\n Built-in methods\n This is really a different disguise of a built-in function, this\n time containing an object passed to the C function as an\n implicit extra argument. An example of a built-in method is\n ``alist.append()``, assuming *alist* is a list object. In this\n case, the special read-only attribute ``__self__`` is set to the\n object denoted by *alist*.\n\n Classes\n Classes are callable. These objects normally act as factories\n for new instances of themselves, but variations are possible for\n class types that override ``__new__()``. The arguments of the\n call are passed to ``__new__()`` and, in the typical case, to\n ``__init__()`` to initialize the new instance.\n\n Class Instances\n Instances of arbitrary classes can be made callable by defining\n a ``__call__()`` method in their class.\n\nModules\n Modules are imported by the ``import`` statement (see section *The\n import statement*). A module object has a namespace implemented by\n a dictionary object (this is the dictionary referenced by the\n __globals__ attribute of functions defined in the module).\n Attribute references are translated to lookups in this dictionary,\n e.g., ``m.x`` is equivalent to ``m.__dict__["x"]``. A module object\n does not contain the code object used to initialize the module\n (since it isn\'t needed once the initialization is done).\n\n Attribute assignment updates the module\'s namespace dictionary,\n e.g., ``m.x = 1`` is equivalent to ``m.__dict__["x"] = 1``.\n\n Special read-only attribute: ``__dict__`` is the module\'s namespace\n as a dictionary object.\n\n **CPython implementation detail:** Because of the way CPython\n clears module dictionaries, the module dictionary will be cleared\n when the module falls out of scope even if the dictionary still has\n live references. To avoid this, copy the dictionary or keep the\n module around while using its dictionary directly.\n\n Predefined (writable) attributes: ``__name__`` is the module\'s\n name; ``__doc__`` is the module\'s documentation string, or ``None``\n if unavailable; ``__file__`` is the pathname of the file from which\n the module was loaded, if it was loaded from a file. The\n ``__file__`` attribute is not present for C modules that are\n statically linked into the interpreter; for extension modules\n loaded dynamically from a shared library, it is the pathname of the\n shared library file.\n\nCustom classes\n Custom class types are typically created by class definitions (see\n section *Class definitions*). A class has a namespace implemented\n by a dictionary object. Class attribute references are translated\n to lookups in this dictionary, e.g., ``C.x`` is translated to\n ``C.__dict__["x"]`` (although there are a number of hooks which\n allow for other means of locating attributes). When the attribute\n name is not found there, the attribute search continues in the base\n classes. This search of the base classes uses the C3 method\n resolution order which behaves correctly even in the presence of\n \'diamond\' inheritance structures where there are multiple\n inheritance paths leading back to a common ancestor. Additional\n details on the C3 MRO used by Python can be found in the\n documentation accompanying the 2.3 release at\n http://www.python.org/download/releases/2.3/mro/.\n\n When a class attribute reference (for class ``C``, say) would yield\n a class method object, it is transformed into an instance method\n object whose ``__self__`` attributes is ``C``. When it would yield\n a static method object, it is transformed into the object wrapped\n by the static method object. See section *Implementing Descriptors*\n for another way in which attributes retrieved from a class may\n differ from those actually contained in its ``__dict__``.\n\n Class attribute assignments update the class\'s dictionary, never\n the dictionary of a base class.\n\n A class object can be called (see above) to yield a class instance\n (see below).\n\n Special attributes: ``__name__`` is the class name; ``__module__``\n is the module name in which the class was defined; ``__dict__`` is\n the dictionary containing the class\'s namespace; ``__bases__`` is a\n tuple (possibly empty or a singleton) containing the base classes,\n in the order of their occurrence in the base class list;\n ``__doc__`` is the class\'s documentation string, or None if\n undefined.\n\nClass instances\n A class instance is created by calling a class object (see above).\n A class instance has a namespace implemented as a dictionary which\n is the first place in which attribute references are searched.\n When an attribute is not found there, and the instance\'s class has\n an attribute by that name, the search continues with the class\n attributes. If a class attribute is found that is a user-defined\n function object, it is transformed into an instance method object\n whose ``__self__`` attribute is the instance. Static method and\n class method objects are also transformed; see above under\n "Classes". See section *Implementing Descriptors* for another way\n in which attributes of a class retrieved via its instances may\n differ from the objects actually stored in the class\'s\n ``__dict__``. If no class attribute is found, and the object\'s\n class has a ``__getattr__()`` method, that is called to satisfy the\n lookup.\n\n Attribute assignments and deletions update the instance\'s\n dictionary, never a class\'s dictionary. If the class has a\n ``__setattr__()`` or ``__delattr__()`` method, this is called\n instead of updating the instance dictionary directly.\n\n Class instances can pretend to be numbers, sequences, or mappings\n if they have methods with certain special names. See section\n *Special method names*.\n\n Special attributes: ``__dict__`` is the attribute dictionary;\n ``__class__`` is the instance\'s class.\n\nI/O objects (also known as file objects)\n A *file object* represents an open file. Various shortcuts are\n available to create file objects: the ``open()`` built-in function,\n and also ``os.popen()``, ``os.fdopen()``, and the ``makefile()``\n method of socket objects (and perhaps by other functions or methods\n provided by extension modules).\n\n The objects ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` are\n initialized to file objects corresponding to the interpreter\'s\n standard input, output and error streams; they are all open in text\n mode and therefore follow the interface defined by the\n ``io.TextIOBase`` abstract class.\n\nInternal types\n A few types used internally by the interpreter are exposed to the\n user. Their definitions may change with future versions of the\n interpreter, but they are mentioned here for completeness.\n\n Code objects\n Code objects represent *byte-compiled* executable Python code,\n or *bytecode*. The difference between a code object and a\n function object is that the function object contains an explicit\n reference to the function\'s globals (the module in which it was\n defined), while a code object contains no context; also the\n default argument values are stored in the function object, not\n in the code object (because they represent values calculated at\n run-time). Unlike function objects, code objects are immutable\n and contain no references (directly or indirectly) to mutable\n objects.\n\n Special read-only attributes: ``co_name`` gives the function\n name; ``co_argcount`` is the number of positional arguments\n (including arguments with default values); ``co_nlocals`` is the\n number of local variables used by the function (including\n arguments); ``co_varnames`` is a tuple containing the names of\n the local variables (starting with the argument names);\n ``co_cellvars`` is a tuple containing the names of local\n variables that are referenced by nested functions;\n ``co_freevars`` is a tuple containing the names of free\n variables; ``co_code`` is a string representing the sequence of\n bytecode instructions; ``co_consts`` is a tuple containing the\n literals used by the bytecode; ``co_names`` is a tuple\n containing the names used by the bytecode; ``co_filename`` is\n the filename from which the code was compiled;\n ``co_firstlineno`` is the first line number of the function;\n ``co_lnotab`` is a string encoding the mapping from bytecode\n offsets to line numbers (for details see the source code of the\n interpreter); ``co_stacksize`` is the required stack size\n (including local variables); ``co_flags`` is an integer encoding\n a number of flags for the interpreter.\n\n The following flag bits are defined for ``co_flags``: bit\n ``0x04`` is set if the function uses the ``*arguments`` syntax\n to accept an arbitrary number of positional arguments; bit\n ``0x08`` is set if the function uses the ``**keywords`` syntax\n to accept arbitrary keyword arguments; bit ``0x20`` is set if\n the function is a generator.\n\n Future feature declarations (``from __future__ import\n division``) also use bits in ``co_flags`` to indicate whether a\n code object was compiled with a particular feature enabled: bit\n ``0x2000`` is set if the function was compiled with future\n division enabled; bits ``0x10`` and ``0x1000`` were used in\n earlier versions of Python.\n\n Other bits in ``co_flags`` are reserved for internal use.\n\n If a code object represents a function, the first item in\n ``co_consts`` is the documentation string of the function, or\n ``None`` if undefined.\n\n Frame objects\n Frame objects represent execution frames. They may occur in\n traceback objects (see below).\n\n Special read-only attributes: ``f_back`` is to the previous\n stack frame (towards the caller), or ``None`` if this is the\n bottom stack frame; ``f_code`` is the code object being executed\n in this frame; ``f_locals`` is the dictionary used to look up\n local variables; ``f_globals`` is used for global variables;\n ``f_builtins`` is used for built-in (intrinsic) names;\n ``f_lasti`` gives the precise instruction (this is an index into\n the bytecode string of the code object).\n\n Special writable attributes: ``f_trace``, if not ``None``, is a\n function called at the start of each source code line (this is\n used by the debugger); ``f_lineno`` is the current line number\n of the frame --- writing to this from within a trace function\n jumps to the given line (only for the bottom-most frame). A\n debugger can implement a Jump command (aka Set Next Statement)\n by writing to f_lineno.\n\n Traceback objects\n Traceback objects represent a stack trace of an exception. A\n traceback object is created when an exception occurs. When the\n search for an exception handler unwinds the execution stack, at\n each unwound level a traceback object is inserted in front of\n the current traceback. When an exception handler is entered,\n the stack trace is made available to the program. (See section\n *The try statement*.) It is accessible as the third item of the\n tuple returned by ``sys.exc_info()``. When the program contains\n no suitable handler, the stack trace is written (nicely\n formatted) to the standard error stream; if the interpreter is\n interactive, it is also made available to the user as\n ``sys.last_traceback``.\n\n Special read-only attributes: ``tb_next`` is the next level in\n the stack trace (towards the frame where the exception\n occurred), or ``None`` if there is no next level; ``tb_frame``\n points to the execution frame of the current level;\n ``tb_lineno`` gives the line number where the exception\n occurred; ``tb_lasti`` indicates the precise instruction. The\n line number and last instruction in the traceback may differ\n from the line number of its frame object if the exception\n occurred in a ``try`` statement with no matching except clause\n or with a finally clause.\n\n Slice objects\n Slice objects are used to represent slices for ``__getitem__()``\n methods. They are also created by the built-in ``slice()``\n function.\n\n Special read-only attributes: ``start`` is the lower bound;\n ``stop`` is the upper bound; ``step`` is the step value; each is\n ``None`` if omitted. These attributes can have any type.\n\n Slice objects support one method:\n\n slice.indices(self, length)\n\n This method takes a single integer argument *length* and\n computes information about the slice that the slice object\n would describe if applied to a sequence of *length* items.\n It returns a tuple of three integers; respectively these are\n the *start* and *stop* indices and the *step* or stride\n length of the slice. Missing or out-of-bounds indices are\n handled in a manner consistent with regular slices.\n\n Static method objects\n Static method objects provide a way of defeating the\n transformation of function objects to method objects described\n above. A static method object is a wrapper around any other\n object, usually a user-defined method object. When a static\n method object is retrieved from a class or a class instance, the\n object actually returned is the wrapped object, which is not\n subject to any further transformation. Static method objects are\n not themselves callable, although the objects they wrap usually\n are. Static method objects are created by the built-in\n ``staticmethod()`` constructor.\n\n Class method objects\n A class method object, like a static method object, is a wrapper\n around another object that alters the way in which that object\n is retrieved from classes and class instances. The behaviour of\n class method objects upon such retrieval is described above,\n under "User-defined methods". Class method objects are created\n by the built-in ``classmethod()`` constructor.\n', 'typesfunctions': '\nFunctions\n*********\n\nFunction objects are created by function definitions. The only\noperation on a function object is to call it: ``func(argument-list)``.\n\nThere are really two flavors of function objects: built-in functions\nand user-defined functions. Both support the same operation (to call\nthe function), but the implementation is different, hence the\ndifferent object types.\n\nSee *Function definitions* for more information.\n', - 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects. There is currently only one standard\nmapping type, the *dictionary*. (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values. Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys. Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry. (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n Return a new dictionary initialized from an optional positional\n argument or from a set of keyword arguments. If no arguments are\n given, return a new empty dictionary. If the positional argument\n *arg* is a mapping object, return a dictionary mapping the same\n keys to the same values as does the mapping object. Otherwise the\n positional argument must be a sequence, a container that supports\n iteration, or an iterator object. The elements of the argument\n must each also be of one of those kinds, and each must in turn\n contain exactly two objects. The first is used as a key in the new\n dictionary, and the second as the key\'s value. If a given key is\n seen more than once, the last value associated with it is retained\n in the new dictionary.\n\n If keyword arguments are given, the keywords themselves with their\n associated values are added as items to the dictionary. If a key\n is specified both in the positional argument and as a keyword\n argument, the value associated with the keyword is retained in the\n dictionary. For example, these all return a dictionary equal to\n ``{"one": 1, "two": 2}``:\n\n * ``dict(one=1, two=2)``\n\n * ``dict({\'one\': 1, \'two\': 2})``\n\n * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n The first example only works for keys that are valid Python\n identifiers; the others work with any valid keys.\n\n These are the operations that dictionaries support (and therefore,\n custom mapping types should support too):\n\n len(d)\n\n Return the number of items in the dictionary *d*.\n\n d[key]\n\n Return the item of *d* with key *key*. Raises a ``KeyError`` if\n *key* is not in the map.\n\n If a subclass of dict defines a method ``__missing__()``, if the\n key *key* is not present, the ``d[key]`` operation calls that\n method with the key *key* as argument. The ``d[key]`` operation\n then returns or raises whatever is returned or raised by the\n ``__missing__(key)`` call if the key is not present. No other\n operations or methods invoke ``__missing__()``. If\n ``__missing__()`` is not defined, ``KeyError`` is raised.\n ``__missing__()`` must be a method; it cannot be an instance\n variable:\n\n >>> class Counter(dict):\n ... def __missing__(self, key):\n ... return 0\n >>> c = Counter()\n >>> c[\'red\']\n 0\n >>> c[\'red\'] += 1\n >>> c[\'red\']\n 1\n\n See ``collections.Counter`` for a complete implementation\n including other methods helpful for accumulating and managing\n tallies.\n\n Changed in version 3.3: If the dict is modified during the\n lookup, a ``RuntimeError`` exception is now raised.\n\n d[key] = value\n\n Set ``d[key]`` to *value*.\n\n del d[key]\n\n Remove ``d[key]`` from *d*. Raises a ``KeyError`` if *key* is\n not in the map.\n\n key in d\n\n Return ``True`` if *d* has a key *key*, else ``False``.\n\n key not in d\n\n Equivalent to ``not key in d``.\n\n iter(d)\n\n Return an iterator over the keys of the dictionary. This is a\n shortcut for ``iter(d.keys())``.\n\n clear()\n\n Remove all items from the dictionary.\n\n copy()\n\n Return a shallow copy of the dictionary.\n\n classmethod fromkeys(seq[, value])\n\n Create a new dictionary with keys from *seq* and values set to\n *value*.\n\n ``fromkeys()`` is a class method that returns a new dictionary.\n *value* defaults to ``None``.\n\n get(key[, default])\n\n Return the value for *key* if *key* is in the dictionary, else\n *default*. If *default* is not given, it defaults to ``None``,\n so that this method never raises a ``KeyError``.\n\n items()\n\n Return a new view of the dictionary\'s items (``(key, value)``\n pairs). See the *documentation of view objects*.\n\n keys()\n\n Return a new view of the dictionary\'s keys. See the\n *documentation of view objects*.\n\n pop(key[, default])\n\n If *key* is in the dictionary, remove it and return its value,\n else return *default*. If *default* is not given and *key* is\n not in the dictionary, a ``KeyError`` is raised.\n\n popitem()\n\n Remove and return an arbitrary ``(key, value)`` pair from the\n dictionary.\n\n ``popitem()`` is useful to destructively iterate over a\n dictionary, as often used in set algorithms. If the dictionary\n is empty, calling ``popitem()`` raises a ``KeyError``.\n\n setdefault(key[, default])\n\n If *key* is in the dictionary, return its value. If not, insert\n *key* with a value of *default* and return *default*. *default*\n defaults to ``None``.\n\n update([other])\n\n Update the dictionary with the key/value pairs from *other*,\n overwriting existing keys. Return ``None``.\n\n ``update()`` accepts either another dictionary object or an\n iterable of key/value pairs (as tuples or other iterables of\n length two). If keyword arguments are specified, the dictionary\n is then updated with those key/value pairs: ``d.update(red=1,\n blue=2)``.\n\n values()\n\n Return a new view of the dictionary\'s values. See the\n *documentation of view objects*.\n\nSee also:\n\n ``types.MappingProxyType`` can be used to create a read-only view\n of a ``dict``.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*. They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n Return the number of entries in the dictionary.\n\niter(dictview)\n\n Return an iterator over the keys, values or items (represented as\n tuples of ``(key, value)``) in the dictionary.\n\n Keys and values are iterated over in an arbitrary order which is\n non-random, varies across Python implementations, and depends on\n the dictionary\'s history of insertions and deletions. If keys,\n values and items views are iterated over with no intervening\n modifications to the dictionary, the order of items will directly\n correspond. This allows the creation of ``(value, key)`` pairs\n using ``zip()``: ``pairs = zip(d.values(), d.keys())``. Another\n way to create the same list is ``pairs = [(v, k) for (k, v) in\n d.items()]``.\n\n Iterating views while adding or deleting entries in the dictionary\n may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n Return ``True`` if *x* is in the underlying dictionary\'s keys,\n values or items (in the latter case, *x* should be a ``(key,\n value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like. (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n >>> keys = dishes.keys()\n >>> values = dishes.values()\n\n >>> # iteration\n >>> n = 0\n >>> for val in values:\n ... n += val\n >>> print(n)\n 504\n\n >>> # keys and values are iterated over in the same order\n >>> list(keys)\n [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n >>> list(values)\n [2, 1, 1, 500]\n\n >>> # view objects are dynamic and reflect dict changes\n >>> del dishes[\'eggs\']\n >>> del dishes[\'sausage\']\n >>> list(keys)\n [\'spam\', \'bacon\']\n\n >>> # set operations\n >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n {\'bacon\'}\n >>> keys ^ {\'sausage\', \'juice\'}\n {\'juice\', \'sausage\', \'bacon\', \'spam\'}\n', + 'typesmapping': '\nMapping Types --- ``dict``\n**************************\n\nA *mapping* object maps *hashable* values to arbitrary objects.\nMappings are mutable objects. There is currently only one standard\nmapping type, the *dictionary*. (For other containers see the built\nin ``list``, ``set``, and ``tuple`` classes, and the ``collections``\nmodule.)\n\nA dictionary\'s keys are *almost* arbitrary values. Values that are\nnot *hashable*, that is, values containing lists, dictionaries or\nother mutable types (that are compared by value rather than by object\nidentity) may not be used as keys. Numeric types used for keys obey\nthe normal rules for numeric comparison: if two numbers compare equal\n(such as ``1`` and ``1.0``) then they can be used interchangeably to\nindex the same dictionary entry. (Note however, that since computers\nstore floating-point numbers as approximations it is usually unwise to\nuse them as dictionary keys.)\n\nDictionaries can be created by placing a comma-separated list of\n``key: value`` pairs within braces, for example: ``{\'jack\': 4098,\n\'sjoerd\': 4127}`` or ``{4098: \'jack\', 4127: \'sjoerd\'}``, or by the\n``dict`` constructor.\n\nclass class dict([arg])\n\n Return a new dictionary initialized from an optional positional\n argument or from a set of keyword arguments. If no arguments are\n given, return a new empty dictionary. If the positional argument\n *arg* is a mapping object, return a dictionary mapping the same\n keys to the same values as does the mapping object. Otherwise the\n positional argument must be a sequence, a container that supports\n iteration, or an iterator object. The elements of the argument\n must each also be of one of those kinds, and each must in turn\n contain exactly two objects. The first is used as a key in the new\n dictionary, and the second as the key\'s value. If a given key is\n seen more than once, the last value associated with it is retained\n in the new dictionary.\n\n If keyword arguments are given, the keywords themselves with their\n associated values are added as items to the dictionary. If a key\n is specified both in the positional argument and as a keyword\n argument, the value associated with the keyword is retained in the\n dictionary. For example, these all return a dictionary equal to\n ``{"one": 1, "two": 2}``:\n\n * ``dict(one=1, two=2)``\n\n * ``dict({\'one\': 1, \'two\': 2})``\n\n * ``dict(zip((\'one\', \'two\'), (1, 2)))``\n\n * ``dict([[\'two\', 2], [\'one\', 1]])``\n\n The first example only works for keys that are valid Python\n identifiers; the others work with any valid keys.\n\n These are the operations that dictionaries support (and therefore,\n custom mapping types should support too):\n\n len(d)\n\n Return the number of items in the dictionary *d*.\n\n d[key]\n\n Return the item of *d* with key *key*. Raises a ``KeyError`` if\n *key* is not in the map.\n\n If a subclass of dict defines a method ``__missing__()``, if the\n key *key* is not present, the ``d[key]`` operation calls that\n method with the key *key* as argument. The ``d[key]`` operation\n then returns or raises whatever is returned or raised by the\n ``__missing__(key)`` call if the key is not present. No other\n operations or methods invoke ``__missing__()``. If\n ``__missing__()`` is not defined, ``KeyError`` is raised.\n ``__missing__()`` must be a method; it cannot be an instance\n variable:\n\n >>> class Counter(dict):\n ... def __missing__(self, key):\n ... return 0\n >>> c = Counter()\n >>> c[\'red\']\n 0\n >>> c[\'red\'] += 1\n >>> c[\'red\']\n 1\n\n See ``collections.Counter`` for a complete implementation\n including other methods helpful for accumulating and managing\n tallies.\n\n d[key] = value\n\n Set ``d[key]`` to *value*.\n\n del d[key]\n\n Remove ``d[key]`` from *d*. Raises a ``KeyError`` if *key* is\n not in the map.\n\n key in d\n\n Return ``True`` if *d* has a key *key*, else ``False``.\n\n key not in d\n\n Equivalent to ``not key in d``.\n\n iter(d)\n\n Return an iterator over the keys of the dictionary. This is a\n shortcut for ``iter(d.keys())``.\n\n clear()\n\n Remove all items from the dictionary.\n\n copy()\n\n Return a shallow copy of the dictionary.\n\n classmethod fromkeys(seq[, value])\n\n Create a new dictionary with keys from *seq* and values set to\n *value*.\n\n ``fromkeys()`` is a class method that returns a new dictionary.\n *value* defaults to ``None``.\n\n get(key[, default])\n\n Return the value for *key* if *key* is in the dictionary, else\n *default*. If *default* is not given, it defaults to ``None``,\n so that this method never raises a ``KeyError``.\n\n items()\n\n Return a new view of the dictionary\'s items (``(key, value)``\n pairs). See the *documentation of view objects*.\n\n keys()\n\n Return a new view of the dictionary\'s keys. See the\n *documentation of view objects*.\n\n pop(key[, default])\n\n If *key* is in the dictionary, remove it and return its value,\n else return *default*. If *default* is not given and *key* is\n not in the dictionary, a ``KeyError`` is raised.\n\n popitem()\n\n Remove and return an arbitrary ``(key, value)`` pair from the\n dictionary.\n\n ``popitem()`` is useful to destructively iterate over a\n dictionary, as often used in set algorithms. If the dictionary\n is empty, calling ``popitem()`` raises a ``KeyError``.\n\n setdefault(key[, default])\n\n If *key* is in the dictionary, return its value. If not, insert\n *key* with a value of *default* and return *default*. *default*\n defaults to ``None``.\n\n update([other])\n\n Update the dictionary with the key/value pairs from *other*,\n overwriting existing keys. Return ``None``.\n\n ``update()`` accepts either another dictionary object or an\n iterable of key/value pairs (as tuples or other iterables of\n length two). If keyword arguments are specified, the dictionary\n is then updated with those key/value pairs: ``d.update(red=1,\n blue=2)``.\n\n values()\n\n Return a new view of the dictionary\'s values. See the\n *documentation of view objects*.\n\nSee also:\n\n ``types.MappingProxyType`` can be used to create a read-only view\n of a ``dict``.\n\n\nDictionary view objects\n=======================\n\nThe objects returned by ``dict.keys()``, ``dict.values()`` and\n``dict.items()`` are *view objects*. They provide a dynamic view on\nthe dictionary\'s entries, which means that when the dictionary\nchanges, the view reflects these changes.\n\nDictionary views can be iterated over to yield their respective data,\nand support membership tests:\n\nlen(dictview)\n\n Return the number of entries in the dictionary.\n\niter(dictview)\n\n Return an iterator over the keys, values or items (represented as\n tuples of ``(key, value)``) in the dictionary.\n\n Keys and values are iterated over in an arbitrary order which is\n non-random, varies across Python implementations, and depends on\n the dictionary\'s history of insertions and deletions. If keys,\n values and items views are iterated over with no intervening\n modifications to the dictionary, the order of items will directly\n correspond. This allows the creation of ``(value, key)`` pairs\n using ``zip()``: ``pairs = zip(d.values(), d.keys())``. Another\n way to create the same list is ``pairs = [(v, k) for (k, v) in\n d.items()]``.\n\n Iterating views while adding or deleting entries in the dictionary\n may raise a ``RuntimeError`` or fail to iterate over all entries.\n\nx in dictview\n\n Return ``True`` if *x* is in the underlying dictionary\'s keys,\n values or items (in the latter case, *x* should be a ``(key,\n value)`` tuple).\n\nKeys views are set-like since their entries are unique and hashable.\nIf all values are hashable, so that ``(key, value)`` pairs are unique\nand hashable, then the items view is also set-like. (Values views are\nnot treated as set-like since the entries are generally not unique.)\nFor set-like views, all of the operations defined for the abstract\nbase class ``collections.Set`` are available (for example, ``==``,\n``<``, or ``^``).\n\nAn example of dictionary view usage:\n\n >>> dishes = {\'eggs\': 2, \'sausage\': 1, \'bacon\': 1, \'spam\': 500}\n >>> keys = dishes.keys()\n >>> values = dishes.values()\n\n >>> # iteration\n >>> n = 0\n >>> for val in values:\n ... n += val\n >>> print(n)\n 504\n\n >>> # keys and values are iterated over in the same order\n >>> list(keys)\n [\'eggs\', \'bacon\', \'sausage\', \'spam\']\n >>> list(values)\n [2, 1, 1, 500]\n\n >>> # view objects are dynamic and reflect dict changes\n >>> del dishes[\'eggs\']\n >>> del dishes[\'sausage\']\n >>> list(keys)\n [\'spam\', \'bacon\']\n\n >>> # set operations\n >>> keys & {\'eggs\', \'bacon\', \'salad\'}\n {\'bacon\'}\n >>> keys ^ {\'sausage\', \'juice\'}\n {\'juice\', \'sausage\', \'bacon\', \'spam\'}\n', 'typesmethods': "\nMethods\n*******\n\nMethods are functions that are called using the attribute notation.\nThere are two flavors: built-in methods (such as ``append()`` on\nlists) and class instance methods. Built-in methods are described\nwith the types that support them.\n\nIf you access a method (a function defined in a class namespace)\nthrough an instance, you get a special object: a *bound method* (also\ncalled *instance method*) object. When called, it will add the\n``self`` argument to the argument list. Bound methods have two\nspecial read-only attributes: ``m.__self__`` is the object on which\nthe method operates, and ``m.__func__`` is the function implementing\nthe method. Calling ``m(arg-1, arg-2, ..., arg-n)`` is completely\nequivalent to calling ``m.__func__(m.__self__, arg-1, arg-2, ...,\narg-n)``.\n\nLike function objects, bound method objects support getting arbitrary\nattributes. However, since method attributes are actually stored on\nthe underlying function object (``meth.__func__``), setting method\nattributes on bound methods is disallowed. Attempting to set a method\nattribute results in a ``TypeError`` being raised. In order to set a\nmethod attribute, you need to explicitly set it on the underlying\nfunction object:\n\n class C:\n def method(self):\n pass\n\n c = C()\n c.method.__func__.whoami = 'my name is c'\n\nSee *The standard type hierarchy* for more information.\n", 'typesmodules': "\nModules\n*******\n\nThe only special operation on a module is attribute access:\n``m.name``, where *m* is a module and *name* accesses a name defined\nin *m*'s symbol table. Module attributes can be assigned to. (Note\nthat the ``import`` statement is not, strictly speaking, an operation\non a module object; ``import foo`` does not require a module object\nnamed *foo* to exist, rather it requires an (external) *definition*\nfor a module named *foo* somewhere.)\n\nA special attribute of every module is ``__dict__``. This is the\ndictionary containing the module's symbol table. Modifying this\ndictionary will actually change the module's symbol table, but direct\nassignment to the ``__dict__`` attribute is not possible (you can\nwrite ``m.__dict__['a'] = 1``, which defines ``m.a`` to be ``1``, but\nyou can't write ``m.__dict__ = {}``). Modifying ``__dict__`` directly\nis not recommended.\n\nModules built into the interpreter are written like this: ````. If loaded from a file, they are written as\n````.\n", - 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects. For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters. Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``. See *String and\nBytes literals* for more about string literals. In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence. Bytes objects can\nbe constructed the constructor, ``bytes()``, and from literals; use a\n``b`` prefix with normal string syntax: ``b\'xyzzy\'``. To construct\nbyte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes. That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1. The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``. You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts. There\'s no implicit en-/decoding if you pass an object of\nthe wrong type. A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``. Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``. A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations. The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations. The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority). In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation | Result | Notes |\n+====================+==================================+============+\n| ``x in s`` | ``True`` if an item of *s* is | (1) |\n| | equal to *x*, else ``False`` | |\n+--------------------+----------------------------------+------------+\n| ``x not in s`` | ``False`` if an item of *s* is | (1) |\n| | equal to *x*, else ``True`` | |\n+--------------------+----------------------------------+------------+\n| ``s + t`` | the concatenation of *s* and *t* | (6) |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s`` | *n* shallow copies of *s* | (2) |\n| | concatenated | |\n+--------------------+----------------------------------+------------+\n| ``s[i]`` | *i*th item of *s*, origin 0 | (3) |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]`` | slice of *s* from *i* to *j* | (3)(4) |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]`` | slice of *s* from *i* to *j* | (3)(5) |\n| | with step *k* | |\n+--------------------+----------------------------------+------------+\n| ``len(s)`` | length of *s* | |\n+--------------------+----------------------------------+------------+\n| ``min(s)`` | smallest item of *s* | |\n+--------------------+----------------------------------+------------+\n| ``max(s)`` | largest item of *s* | |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)`` | index of the first occurence of | |\n| | *i* in *s* | |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)`` | total number of occurences of | |\n| | *i* in *s* | |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons. In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements. This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length. (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n empty sequence of the same type as *s*). Note also that the copies\n are shallow; nested structures are not copied. This often haunts\n new Python programmers; consider:\n\n >>> lists = [[]] * 3\n >>> lists\n [[], [], []]\n >>> lists[0].append(3)\n >>> lists\n [[3], [3], [3]]\n\n What has happened is that ``[[]]`` is a one-element list containing\n an empty list, so all three elements of ``[[]] * 3`` are (pointers\n to) this single empty list. Modifying any of the elements of\n ``lists`` modifies this single list. You can create a list of\n different lists this way:\n\n >>> lists = [[] for i in range(3)]\n >>> lists[0].append(3)\n >>> lists[1].append(5)\n >>> lists[2].append(7)\n >>> lists\n [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n string: ``len(s) + i`` or ``len(s) + j`` is substituted. But note\n that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n items with index *k* such that ``i <= k < j``. If *i* or *j* is\n greater than ``len(s)``, use ``len(s)``. If *i* is omitted or\n ``None``, use ``0``. If *j* is omitted or ``None``, use\n ``len(s)``. If *i* is greater than or equal to *j*, the slice is\n empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n sequence of items with index ``x = i + n*k`` such that ``0 <= n <\n (j-i)/k``. In other words, the indices are ``i``, ``i+k``,\n ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n never including *j*). If *i* or *j* is greater than ``len(s)``,\n use ``len(s)``. If *i* or *j* are omitted or ``None``, they become\n "end" values (which end depends on the sign of *k*). Note, *k*\n cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. Concatenating immutable strings always results in a new object.\n This means that building up a string by repeated concatenation will\n have a quadratic runtime cost in the total string length. To get a\n linear runtime cost, you must switch to one of the alternatives\n below:\n\n * if concatenating ``str`` objects, you can build a list and use\n ``str.join()`` at the end;\n\n * if concatenating ``bytes`` objects, you can similarly use\n ``bytes.join()``, or you can do in-place concatenation with a\n ``bytearray`` object. ``bytearray`` objects are mutable and have\n an efficient overallocation mechanism.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n Return a copy of the string with its first character capitalized\n and the rest lowercased.\n\nstr.casefold()\n\n Return a casefolded copy of the string. Casefolded strings may be\n used for caseless matching.\n\n Casefolding is similar to lowercasing but more aggressive because\n it is intended to remove all case distinctions in a string. For\n example, the German lowercase letter ``\'\xc3\x9f\'`` is equivalent to\n ``"ss"``. Since it is already lowercase, ``lower()`` would do\n nothing to ``\'\xc3\x9f\'``; ``casefold()`` converts it to ``"ss"``.\n\n The casefolding algorithm is described in section 3.13 of the\n Unicode Standard.\n\n New in version 3.3.\n\nstr.center(width[, fillchar])\n\n Return centered in a string of length *width*. Padding is done\n using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n Return the number of non-overlapping occurrences of substring *sub*\n in the range [*start*, *end*]. Optional arguments *start* and\n *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n Return an encoded version of the string as a bytes object. Default\n encoding is ``\'utf-8\'``. *errors* may be given to set a different\n error handling scheme. The default for *errors* is ``\'strict\'``,\n meaning that encoding errors raise a ``UnicodeError``. Other\n possible values are ``\'ignore\'``, ``\'replace\'``,\n ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n Return ``True`` if the string ends with the specified *suffix*,\n otherwise return ``False``. *suffix* can also be a tuple of\n suffixes to look for. With optional *start*, test beginning at\n that position. With optional *end*, stop comparing at that\n position.\n\nstr.expandtabs([tabsize])\n\n Return a copy of the string where all tab characters are replaced\n by zero or more spaces, depending on the current column and the\n given tab size. The column number is reset to zero after each\n newline occurring in the string. If *tabsize* is not given, a tab\n size of ``8`` characters is assumed. This doesn\'t understand other\n non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n Return the lowest index in the string where substring *sub* is\n found, such that *sub* is contained in the slice ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` if *sub* is not found.\n\n Note: The ``find()`` method should be used only if you need to know the\n position of *sub*. To check if *sub* is a substring or not, use\n the ``in`` operator:\n\n >>> \'Py\' in \'Python\'\n True\n\nstr.format(*args, **kwargs)\n\n Perform a string formatting operation. The string on which this\n method is called can contain literal text or replacement fields\n delimited by braces ``{}``. Each replacement field contains either\n the numeric index of a positional argument, or the name of a\n keyword argument. Returns a copy of the string where each\n replacement field is replaced with the string value of the\n corresponding argument.\n\n >>> "The sum of 1 + 2 is {0}".format(1+2)\n \'The sum of 1 + 2 is 3\'\n\n See *Format String Syntax* for a description of the various\n formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n Similar to ``str.format(**mapping)``, except that ``mapping`` is\n used directly and not copied to a ``dict`` . This is useful if for\n example ``mapping`` is a dict subclass:\n\n >>> class Default(dict):\n ... def __missing__(self, key):\n ... return key\n ...\n >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n \'Guido was born in country\'\n\n New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n Like ``find()``, but raise ``ValueError`` when the substring is not\n found.\n\nstr.isalnum()\n\n Return true if all characters in the string are alphanumeric and\n there is at least one character, false otherwise. A character\n ``c`` is alphanumeric if one of the following returns ``True``:\n ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n ``c.isnumeric()``.\n\nstr.isalpha()\n\n Return true if all characters in the string are alphabetic and\n there is at least one character, false otherwise. Alphabetic\n characters are those characters defined in the Unicode character\n database as "Letter", i.e., those with general category property\n being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is\n different from the "Alphabetic" property defined in the Unicode\n Standard.\n\nstr.isdecimal()\n\n Return true if all characters in the string are decimal characters\n and there is at least one character, false otherwise. Decimal\n characters are those from general category "Nd". This category\n includes digit characters, and all characters that can be used to\n form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.\n\nstr.isdigit()\n\n Return true if all characters in the string are digits and there is\n at least one character, false otherwise. Digits include decimal\n characters and digits that need special handling, such as the\n compatibility superscript digits. Formally, a digit is a character\n that has the property value Numeric_Type=Digit or\n Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n Return true if the string is a valid identifier according to the\n language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n Return true if all cased characters [4] in the string are lowercase\n and there is at least one cased character, false otherwise.\n\nstr.isnumeric()\n\n Return true if all characters in the string are numeric characters,\n and there is at least one character, false otherwise. Numeric\n characters include digit characters, and all characters that have\n the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n ONE FIFTH. Formally, numeric characters are those with the\n property value Numeric_Type=Digit, Numeric_Type=Decimal or\n Numeric_Type=Numeric.\n\nstr.isprintable()\n\n Return true if all characters in the string are printable or the\n string is empty, false otherwise. Nonprintable characters are\n those characters defined in the Unicode character database as\n "Other" or "Separator", excepting the ASCII space (0x20) which is\n considered printable. (Note that printable characters in this\n context are those which should not be escaped when ``repr()`` is\n invoked on a string. It has no bearing on the handling of strings\n written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n Return true if there are only whitespace characters in the string\n and there is at least one character, false otherwise. Whitespace\n characters are those characters defined in the Unicode character\n database as "Other" or "Separator" and those with bidirectional\n property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n Return true if the string is a titlecased string and there is at\n least one character, for example uppercase characters may only\n follow uncased characters and lowercase characters only cased ones.\n Return false otherwise.\n\nstr.isupper()\n\n Return true if all cased characters [4] in the string are uppercase\n and there is at least one cased character, false otherwise.\n\nstr.join(iterable)\n\n Return a string which is the concatenation of the strings in the\n *iterable* *iterable*. A ``TypeError`` will be raised if there are\n any non-string values in *iterable*, including ``bytes`` objects.\n The separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n Return the string left justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.lower()\n\n Return a copy of the string with all the cased characters [4]\n converted to lowercase.\n\n The lowercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.lstrip([chars])\n\n Return a copy of the string with leading characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a prefix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.lstrip()\n \'spacious \'\n >>> \'www.example.com\'.lstrip(\'cmowz.\')\n \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n This static method returns a translation table usable for\n ``str.translate()``.\n\n If there is only one argument, it must be a dictionary mapping\n Unicode ordinals (integers) or characters (strings of length 1) to\n Unicode ordinals, strings (of arbitrary lengths) or None.\n Character keys will then be converted to ordinals.\n\n If there are two arguments, they must be strings of equal length,\n and in the resulting dictionary, each character in x will be mapped\n to the character at the same position in y. If there is a third\n argument, it must be a string, whose characters will be mapped to\n None in the result.\n\nstr.partition(sep)\n\n Split the string at the first occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing the string itself, followed by\n two empty strings.\n\nstr.replace(old, new[, count])\n\n Return a copy of the string with all occurrences of substring *old*\n replaced by *new*. If the optional argument *count* is given, only\n the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n Return the highest index in the string where substring *sub* is\n found, such that *sub* is contained within ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n is not found.\n\nstr.rjust(width[, fillchar])\n\n Return the string right justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.rpartition(sep)\n\n Split the string at the last occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing two empty strings, followed by\n the string itself.\n\nstr.rsplit(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n are done, the *rightmost* ones. If *sep* is not specified or\n ``None``, any whitespace string is a separator. Except for\n splitting from the right, ``rsplit()`` behaves like ``split()``\n which is described in detail below.\n\nstr.rstrip([chars])\n\n Return a copy of the string with trailing characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a suffix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.rstrip()\n \' spacious\'\n >>> \'mississippi\'.rstrip(\'ipz\')\n \'mississ\'\n\nstr.split(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit*\n splits are done (thus, the list will have at most ``maxsplit+1``\n elements). If *maxsplit* is not specified, then there is no limit\n on the number of splits (all possible splits are made).\n\n If *sep* is given, consecutive delimiters are not grouped together\n and are deemed to delimit empty strings (for example,\n ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``). The *sep*\n argument may consist of multiple characters (for example,\n ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n an empty string with a specified separator returns ``[\'\']``.\n\n If *sep* is not specified or is ``None``, a different splitting\n algorithm is applied: runs of consecutive whitespace are regarded\n as a single separator, and the result will contain no empty strings\n at the start or end if the string has leading or trailing\n whitespace. Consequently, splitting an empty string or a string\n consisting of just whitespace with a ``None`` separator returns\n ``[]``.\n\n For example, ``\' 1 2 3 \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n and ``\' 1 2 3 \'.split(None, 1)`` returns ``[\'1\', \'2 3 \']``.\n\nstr.splitlines([keepends])\n\n Return a list of the lines in the string, breaking at line\n boundaries. Line breaks are not included in the resulting list\n unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n Return ``True`` if string starts with the *prefix*, otherwise\n return ``False``. *prefix* can also be a tuple of prefixes to look\n for. With optional *start*, test string beginning at that\n position. With optional *end*, stop comparing string at that\n position.\n\nstr.strip([chars])\n\n Return a copy of the string with the leading and trailing\n characters removed. The *chars* argument is a string specifying the\n set of characters to be removed. If omitted or ``None``, the\n *chars* argument defaults to removing whitespace. The *chars*\n argument is not a prefix or suffix; rather, all combinations of its\n values are stripped:\n\n >>> \' spacious \'.strip()\n \'spacious\'\n >>> \'www.example.com\'.strip(\'cmowz.\')\n \'example\'\n\nstr.swapcase()\n\n Return a copy of the string with uppercase characters converted to\n lowercase and vice versa. Note that it is not necessarily true that\n ``s.swapcase().swapcase() == s``.\n\nstr.title()\n\n Return a titlecased version of the string where words start with an\n uppercase character and the remaining characters are lowercase.\n\n The algorithm uses a simple language-independent definition of a\n word as groups of consecutive letters. The definition works in\n many contexts but it means that apostrophes in contractions and\n possessives form word boundaries, which may not be the desired\n result:\n\n >>> "they\'re bill\'s friends from the UK".title()\n "They\'Re Bill\'S Friends From The Uk"\n\n A workaround for apostrophes can be constructed using regular\n expressions:\n\n >>> import re\n >>> def titlecase(s):\n return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n lambda mo: mo.group(0)[0].upper() +\n mo.group(0)[1:].lower(),\n s)\n\n >>> titlecase("they\'re bill\'s friends.")\n "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n Return a copy of the *s* where all characters have been mapped\n through the *map* which must be a dictionary of Unicode ordinals\n (integers) to Unicode ordinals, strings or ``None``. Unmapped\n characters are left untouched. Characters mapped to ``None`` are\n deleted.\n\n You can use ``str.maketrans()`` to create a translation map from\n character-to-character mappings in different formats.\n\n Note: An even more flexible approach is to create a custom character\n mapping codec using the ``codecs`` module (see\n ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n Return a copy of the string with all the cased characters [4]\n converted to uppercase. Note that ``str.upper().isupper()`` might\n be ``False`` if ``s`` contains uncased characters or if the Unicode\n category of the resulting character(s) is not "Lu" (Letter,\n uppercase), but e.g. "Lt" (Letter, titlecase).\n\n The uppercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.zfill(width)\n\n Return the numeric string left filled with zeros in a string of\n length *width*. A sign prefix is handled correctly. The original\n string is returned if *width* is less than or equal to ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are modelled on C\'s\n printf() syntax. They only support formatting of certain builtin\n types. The use of a binary operator means that care may be needed\n in order to format tuples and dictionaries correctly. As the new\n *String Formatting* syntax is more flexible and handles tuples and\n dictionaries naturally, it is recommended for new code. However,\n there are no current plans to deprecate printf-style formatting.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [5] Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n conversion types.\n\n4. Minimum field width (optional). If specified as an ``\'*\'``\n (asterisk), the actual width is read from the next element of the\n tuple in *values*, and the object to convert comes after the\n minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n precision. If specified as ``\'*\'`` (an asterisk), the actual\n precision is read from the next element of the tuple in *values*,\n and the value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n... {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag | Meaning |\n+===========+=======================================================================+\n| ``\'#\'`` | The value conversion will use the "alternate form" (where defined |\n| | below). |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'`` | The conversion will be zero padded for numeric values. |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'`` | The converted value is left adjusted (overrides the ``\'0\'`` |\n| | conversion if both are given). |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'`` | (a space) A blank should be left before a positive number (or empty |\n| | string) produced by a signed conversion. |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'`` | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion |\n| | (overrides a "space" flag). |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion | Meaning | Notes |\n+==============+=======================================================+=========+\n| ``\'d\'`` | Signed integer decimal. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'`` | Signed integer decimal. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'`` | Signed octal value. | (1) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'`` | Obsolete type -- it is identical to ``\'d\'``. | (7) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'`` | Signed hexadecimal (lowercase). | (2) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'`` | Signed hexadecimal (uppercase). | (2) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'`` | Floating point exponential format (lowercase). | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'`` | Floating point exponential format (uppercase). | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'`` | Floating point decimal format. | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'`` | Floating point decimal format. | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'`` | Floating point format. Uses lowercase exponential | (4) |\n| | format if exponent is less than -4 or not less than | |\n| | precision, decimal format otherwise. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'`` | Floating point format. Uses uppercase exponential | (4) |\n| | format if exponent is less than -4 or not less than | |\n| | precision, decimal format otherwise. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'`` | Single character (accepts integer or single character | |\n| | string). | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'`` | String (converts any Python object using ``repr()``). | (5) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'`` | String (converts any Python object using ``str()``). | (5) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'a\'`` | String (converts any Python object using | (5) |\n| | ``ascii()``). | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'`` | No argument is converted, results in a ``\'%\'`` | |\n| | character in the result. | |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n between left-hand padding and the formatting of the number if the\n leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n between left-hand padding and the formatting of the number if the\n leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n point, even if no digits follow it.\n\n The precision determines the number of digits after the decimal\n point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n point, and trailing zeroes are not removed as they would otherwise\n be.\n\n The precision determines the number of significant digits before\n and after the decimal point and defaults to 6.\n\n5. If precision is ``N``, the output is truncated to ``N`` characters.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping. The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n Return the number of *i*\'s for which ``s[i] == x``.\n\n New in version 3.2.\n\nrange.index(x)\n\n Return the smallest *i* such that ``s[i] == x``. Raises\n ``ValueError`` when *x* is not in the range.\n\n New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object. Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation | Result | Notes |\n+================================+==================================+=======================+\n| ``s[i] = x`` | item *i* of *s* is replaced by | |\n| | *x* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t`` | slice of *s* from *i* to *j* is | |\n| | replaced by the contents of the | |\n| | iterable *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]`` | same as ``s[i:j] = []`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t`` | the elements of ``s[i:j:k]`` are | (1) |\n| | replaced by those of *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]`` | removes the elements of | |\n| | ``s[i:j:k]`` from the list | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)`` | same as ``s[len(s):len(s)] = | |\n| | [x]`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)`` | same as ``s[len(s):len(s)] = x`` | (2) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.clear()`` | remove all items from ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.copy()`` | return a shallow copy of ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)`` | return number of *i*\'s for which | |\n| | ``s[i] == x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])`` | return smallest *k* such that | (3) |\n| | ``s[k] == x`` and ``i <= k < j`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)`` | same as ``s[i:i] = [x]`` | (4) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])`` | same as ``x = s[i]; del s[i]; | (5) |\n| | return x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)`` | same as ``del s[s.index(x)]`` | (3) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()`` | reverses the items of *s* in | (6) |\n| | place | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])`` | sort the items of *s* in place | (6), (7), (8) |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n index is passed as the second or third parameter to the ``index()``\n method, the sequence length is added, as for slice indices. If it\n is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n ``insert()`` method, the sequence length is added, as for slice\n indices. If it is still negative, it is truncated to zero, as for\n slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n place for economy of space when sorting or reversing a large\n sequence. To remind you that they operate by side effect, they\n don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n comparisons. Each must be specified as a keyword argument.\n\n *key* specifies a function of one argument that is used to extract\n a comparison key from each list element: ``key=str.lower``. The\n default value is ``None``. Use ``functools.cmp_to_key()`` to\n convert an old-style *cmp* function to a *key* function.\n\n *reverse* is a boolean value. If set to ``True``, then the list\n elements are sorted as if each comparison were reversed.\n\n The ``sort()`` method is guaranteed to be stable. A sort is stable\n if it guarantees not to change the relative order of elements that\n compare equal --- this is helpful for sorting in multiple passes\n (for example, sort by department, then by salary grade).\n\n **CPython implementation detail:** While a list is being sorted,\n the effect of attempting to mutate, or even inspect, the list is\n undefined. The C implementation of Python makes the list appear\n empty for the duration, and raises ``ValueError`` if it can detect\n that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n New in version 3.3: ``clear()`` and ``copy()`` methods.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types. For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNew in version 3.3: The functions ``count()``, ``find()``,\n``index()``, ``rfind()`` and ``rindex()`` have additional semantics\ncompared to the corresponding string functions: They also accept an\ninteger in range 0 to 255 (a byte) as their first argument.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n their arguments, just as the methods on strings don\'t accept bytes\n as their arguments. For example, you have to write\n\n a = "abc"\n b = a.replace("a", "f")\n\n and\n\n a = b"abc"\n b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n Return a string decoded from the given bytes. Default encoding is\n ``\'utf-8\'``. *errors* may be given to set a different error\n handling scheme. The default for *errors* is ``\'strict\'``, meaning\n that encoding errors raise a ``UnicodeError``. Other possible\n values are ``\'ignore\'``, ``\'replace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n This ``bytes`` class method returns a bytes or bytearray object,\n decoding the given string object. The string must contain two\n hexadecimal digits per byte, spaces are ignored.\n\n >>> bytes.fromhex(\'f0 f1f2 \')\n b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n Return a copy of the bytes or bytearray object where all bytes\n occurring in the optional argument *delete* are removed, and the\n remaining bytes have been mapped through the given translation\n table, which must be a bytes object of length 256.\n\n You can use the ``bytes.maketrans()`` method to create a\n translation table.\n\n Set the *table* argument to ``None`` for translations that only\n delete characters:\n\n >>> b\'read this short text\'.translate(None, b\'aeiou\')\n b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n This static method returns a translation table usable for\n ``bytes.translate()`` that will map each character in *from* into\n the character at the same position in *to*; *from* and *to* must be\n bytes objects and have the same length.\n\n New in version 3.1.\n', + 'typesseq': '\nSequence Types --- ``str``, ``bytes``, ``bytearray``, ``list``, ``tuple``, ``range``\n************************************************************************************\n\nThere are six sequence types: strings, byte sequences (``bytes``\nobjects), byte arrays (``bytearray`` objects), lists, tuples, and\nrange objects. For other containers see the built in ``dict`` and\n``set`` classes, and the ``collections`` module.\n\nStrings contain Unicode characters. Their literals are written in\nsingle or double quotes: ``\'xyzzy\'``, ``"frobozz"``. See *String and\nBytes literals* for more about string literals. In addition to the\nfunctionality described here, there are also string-specific methods\ndescribed in the *String Methods* section.\n\nBytes and bytearray objects contain single bytes -- the former is\nimmutable while the latter is a mutable sequence. Bytes objects can be\nconstructed by using the constructor, ``bytes()``, and from literals;\nuse a ``b`` prefix with normal string syntax: ``b\'xyzzy\'``. To\nconstruct byte arrays, use the ``bytearray()`` function.\n\nWhile string objects are sequences of characters (represented by\nstrings of length 1), bytes and bytearray objects are sequences of\n*integers* (between 0 and 255), representing the ASCII value of single\nbytes. That means that for a bytes or bytearray object *b*, ``b[0]``\nwill be an integer, while ``b[0:1]`` will be a bytes or bytearray\nobject of length 1. The representation of bytes objects uses the\nliteral format (``b\'...\'``) since it is generally more useful than\ne.g. ``bytes([50, 19, 100])``. You can always convert a bytes object\ninto a list of integers using ``list(b)``.\n\nAlso, while in previous Python versions, byte strings and Unicode\nstrings could be exchanged for each other rather freely (barring\nencoding issues), strings and bytes are now completely separate\nconcepts. There\'s no implicit en-/decoding if you pass an object of\nthe wrong type. A string always compares unequal to a bytes or\nbytearray object.\n\nLists are constructed with square brackets, separating items with\ncommas: ``[a, b, c]``. Tuples are constructed by the comma operator\n(not within square brackets), with or without enclosing parentheses,\nbut an empty tuple must have the enclosing parentheses, such as ``a,\nb, c`` or ``()``. A single item tuple must have a trailing comma,\nsuch as ``(d,)``.\n\nObjects of type range are created using the ``range()`` function.\nThey don\'t support concatenation or repetition, and using ``min()`` or\n``max()`` on them is inefficient.\n\nMost sequence types support the following operations. The ``in`` and\n``not in`` operations have the same priorities as the comparison\noperations. The ``+`` and ``*`` operations have the same priority as\nthe corresponding numeric operations. [3] Additional methods are\nprovided for *Mutable Sequence Types*.\n\nThis table lists the sequence operations sorted in ascending priority\n(operations in the same box have the same priority). In the table,\n*s* and *t* are sequences of the same type; *n*, *i*, *j* and *k* are\nintegers.\n\n+--------------------+----------------------------------+------------+\n| Operation | Result | Notes |\n+====================+==================================+============+\n| ``x in s`` | ``True`` if an item of *s* is | (1) |\n| | equal to *x*, else ``False`` | |\n+--------------------+----------------------------------+------------+\n| ``x not in s`` | ``False`` if an item of *s* is | (1) |\n| | equal to *x*, else ``True`` | |\n+--------------------+----------------------------------+------------+\n| ``s + t`` | the concatenation of *s* and *t* | (6) |\n+--------------------+----------------------------------+------------+\n| ``s * n, n * s`` | *n* shallow copies of *s* | (2) |\n| | concatenated | |\n+--------------------+----------------------------------+------------+\n| ``s[i]`` | *i*th item of *s*, origin 0 | (3) |\n+--------------------+----------------------------------+------------+\n| ``s[i:j]`` | slice of *s* from *i* to *j* | (3)(4) |\n+--------------------+----------------------------------+------------+\n| ``s[i:j:k]`` | slice of *s* from *i* to *j* | (3)(5) |\n| | with step *k* | |\n+--------------------+----------------------------------+------------+\n| ``len(s)`` | length of *s* | |\n+--------------------+----------------------------------+------------+\n| ``min(s)`` | smallest item of *s* | |\n+--------------------+----------------------------------+------------+\n| ``max(s)`` | largest item of *s* | |\n+--------------------+----------------------------------+------------+\n| ``s.index(i)`` | index of the first occurence of | |\n| | *i* in *s* | |\n+--------------------+----------------------------------+------------+\n| ``s.count(i)`` | total number of occurences of | |\n| | *i* in *s* | |\n+--------------------+----------------------------------+------------+\n\nSequence types also support comparisons. In particular, tuples and\nlists are compared lexicographically by comparing corresponding\nelements. This means that to compare equal, every element must\ncompare equal and the two sequences must be of the same type and have\nthe same length. (For full details see *Comparisons* in the language\nreference.)\n\nNotes:\n\n1. When *s* is a string object, the ``in`` and ``not in`` operations\n act like a substring test.\n\n2. Values of *n* less than ``0`` are treated as ``0`` (which yields an\n empty sequence of the same type as *s*). Note also that the copies\n are shallow; nested structures are not copied. This often haunts\n new Python programmers; consider:\n\n >>> lists = [[]] * 3\n >>> lists\n [[], [], []]\n >>> lists[0].append(3)\n >>> lists\n [[3], [3], [3]]\n\n What has happened is that ``[[]]`` is a one-element list containing\n an empty list, so all three elements of ``[[]] * 3`` are (pointers\n to) this single empty list. Modifying any of the elements of\n ``lists`` modifies this single list. You can create a list of\n different lists this way:\n\n >>> lists = [[] for i in range(3)]\n >>> lists[0].append(3)\n >>> lists[1].append(5)\n >>> lists[2].append(7)\n >>> lists\n [[3], [5], [7]]\n\n3. If *i* or *j* is negative, the index is relative to the end of the\n string: ``len(s) + i`` or ``len(s) + j`` is substituted. But note\n that ``-0`` is still ``0``.\n\n4. The slice of *s* from *i* to *j* is defined as the sequence of\n items with index *k* such that ``i <= k < j``. If *i* or *j* is\n greater than ``len(s)``, use ``len(s)``. If *i* is omitted or\n ``None``, use ``0``. If *j* is omitted or ``None``, use\n ``len(s)``. If *i* is greater than or equal to *j*, the slice is\n empty.\n\n5. The slice of *s* from *i* to *j* with step *k* is defined as the\n sequence of items with index ``x = i + n*k`` such that ``0 <= n <\n (j-i)/k``. In other words, the indices are ``i``, ``i+k``,\n ``i+2*k``, ``i+3*k`` and so on, stopping when *j* is reached (but\n never including *j*). If *i* or *j* is greater than ``len(s)``,\n use ``len(s)``. If *i* or *j* are omitted or ``None``, they become\n "end" values (which end depends on the sign of *k*). Note, *k*\n cannot be zero. If *k* is ``None``, it is treated like ``1``.\n\n6. Concatenating immutable strings always results in a new object.\n This means that building up a string by repeated concatenation will\n have a quadratic runtime cost in the total string length. To get a\n linear runtime cost, you must switch to one of the alternatives\n below:\n\n * if concatenating ``str`` objects, you can build a list and use\n ``str.join()`` at the end;\n\n * if concatenating ``bytes`` objects, you can similarly use\n ``bytes.join()``, or you can do in-place concatenation with a\n ``bytearray`` object. ``bytearray`` objects are mutable and have\n an efficient overallocation mechanism.\n\n\nString Methods\n==============\n\nString objects support the methods listed below.\n\nIn addition, Python\'s strings support the sequence type methods\ndescribed in the *Sequence Types --- str, bytes, bytearray, list,\ntuple, range* section. To output formatted strings, see the *String\nFormatting* section. Also, see the ``re`` module for string functions\nbased on regular expressions.\n\nstr.capitalize()\n\n Return a copy of the string with its first character capitalized\n and the rest lowercased.\n\nstr.casefold()\n\n Return a casefolded copy of the string. Casefolded strings may be\n used for caseless matching.\n\n Casefolding is similar to lowercasing but more aggressive because\n it is intended to remove all case distinctions in a string. For\n example, the German lowercase letter ``\'\xc3\x9f\'`` is equivalent to\n ``"ss"``. Since it is already lowercase, ``lower()`` would do\n nothing to ``\'\xc3\x9f\'``; ``casefold()`` converts it to ``"ss"``.\n\n The casefolding algorithm is described in section 3.13 of the\n Unicode Standard.\n\n New in version 3.3.\n\nstr.center(width[, fillchar])\n\n Return centered in a string of length *width*. Padding is done\n using the specified *fillchar* (default is a space).\n\nstr.count(sub[, start[, end]])\n\n Return the number of non-overlapping occurrences of substring *sub*\n in the range [*start*, *end*]. Optional arguments *start* and\n *end* are interpreted as in slice notation.\n\nstr.encode(encoding="utf-8", errors="strict")\n\n Return an encoded version of the string as a bytes object. Default\n encoding is ``\'utf-8\'``. *errors* may be given to set a different\n error handling scheme. The default for *errors* is ``\'strict\'``,\n meaning that encoding errors raise a ``UnicodeError``. Other\n possible values are ``\'ignore\'``, ``\'replace\'``,\n ``\'xmlcharrefreplace\'``, ``\'backslashreplace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Support for keyword arguments added.\n\nstr.endswith(suffix[, start[, end]])\n\n Return ``True`` if the string ends with the specified *suffix*,\n otherwise return ``False``. *suffix* can also be a tuple of\n suffixes to look for. With optional *start*, test beginning at\n that position. With optional *end*, stop comparing at that\n position.\n\nstr.expandtabs([tabsize])\n\n Return a copy of the string where all tab characters are replaced\n by zero or more spaces, depending on the current column and the\n given tab size. The column number is reset to zero after each\n newline occurring in the string. If *tabsize* is not given, a tab\n size of ``8`` characters is assumed. This doesn\'t understand other\n non-printing characters or escape sequences.\n\nstr.find(sub[, start[, end]])\n\n Return the lowest index in the string where substring *sub* is\n found, such that *sub* is contained in the slice ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` if *sub* is not found.\n\n Note: The ``find()`` method should be used only if you need to know the\n position of *sub*. To check if *sub* is a substring or not, use\n the ``in`` operator:\n\n >>> \'Py\' in \'Python\'\n True\n\nstr.format(*args, **kwargs)\n\n Perform a string formatting operation. The string on which this\n method is called can contain literal text or replacement fields\n delimited by braces ``{}``. Each replacement field contains either\n the numeric index of a positional argument, or the name of a\n keyword argument. Returns a copy of the string where each\n replacement field is replaced with the string value of the\n corresponding argument.\n\n >>> "The sum of 1 + 2 is {0}".format(1+2)\n \'The sum of 1 + 2 is 3\'\n\n See *Format String Syntax* for a description of the various\n formatting options that can be specified in format strings.\n\nstr.format_map(mapping)\n\n Similar to ``str.format(**mapping)``, except that ``mapping`` is\n used directly and not copied to a ``dict`` . This is useful if for\n example ``mapping`` is a dict subclass:\n\n >>> class Default(dict):\n ... def __missing__(self, key):\n ... return key\n ...\n >>> \'{name} was born in {country}\'.format_map(Default(name=\'Guido\'))\n \'Guido was born in country\'\n\n New in version 3.2.\n\nstr.index(sub[, start[, end]])\n\n Like ``find()``, but raise ``ValueError`` when the substring is not\n found.\n\nstr.isalnum()\n\n Return true if all characters in the string are alphanumeric and\n there is at least one character, false otherwise. A character\n ``c`` is alphanumeric if one of the following returns ``True``:\n ``c.isalpha()``, ``c.isdecimal()``, ``c.isdigit()``, or\n ``c.isnumeric()``.\n\nstr.isalpha()\n\n Return true if all characters in the string are alphabetic and\n there is at least one character, false otherwise. Alphabetic\n characters are those characters defined in the Unicode character\n database as "Letter", i.e., those with general category property\n being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is\n different from the "Alphabetic" property defined in the Unicode\n Standard.\n\nstr.isdecimal()\n\n Return true if all characters in the string are decimal characters\n and there is at least one character, false otherwise. Decimal\n characters are those from general category "Nd". This category\n includes digit characters, and all characters that can be used to\n form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO.\n\nstr.isdigit()\n\n Return true if all characters in the string are digits and there is\n at least one character, false otherwise. Digits include decimal\n characters and digits that need special handling, such as the\n compatibility superscript digits. Formally, a digit is a character\n that has the property value Numeric_Type=Digit or\n Numeric_Type=Decimal.\n\nstr.isidentifier()\n\n Return true if the string is a valid identifier according to the\n language definition, section *Identifiers and keywords*.\n\nstr.islower()\n\n Return true if all cased characters [4] in the string are lowercase\n and there is at least one cased character, false otherwise.\n\nstr.isnumeric()\n\n Return true if all characters in the string are numeric characters,\n and there is at least one character, false otherwise. Numeric\n characters include digit characters, and all characters that have\n the Unicode numeric value property, e.g. U+2155, VULGAR FRACTION\n ONE FIFTH. Formally, numeric characters are those with the\n property value Numeric_Type=Digit, Numeric_Type=Decimal or\n Numeric_Type=Numeric.\n\nstr.isprintable()\n\n Return true if all characters in the string are printable or the\n string is empty, false otherwise. Nonprintable characters are\n those characters defined in the Unicode character database as\n "Other" or "Separator", excepting the ASCII space (0x20) which is\n considered printable. (Note that printable characters in this\n context are those which should not be escaped when ``repr()`` is\n invoked on a string. It has no bearing on the handling of strings\n written to ``sys.stdout`` or ``sys.stderr``.)\n\nstr.isspace()\n\n Return true if there are only whitespace characters in the string\n and there is at least one character, false otherwise. Whitespace\n characters are those characters defined in the Unicode character\n database as "Other" or "Separator" and those with bidirectional\n property being one of "WS", "B", or "S".\n\nstr.istitle()\n\n Return true if the string is a titlecased string and there is at\n least one character, for example uppercase characters may only\n follow uncased characters and lowercase characters only cased ones.\n Return false otherwise.\n\nstr.isupper()\n\n Return true if all cased characters [4] in the string are uppercase\n and there is at least one cased character, false otherwise.\n\nstr.join(iterable)\n\n Return a string which is the concatenation of the strings in the\n *iterable* *iterable*. A ``TypeError`` will be raised if there are\n any non-string values in *iterable*, including ``bytes`` objects.\n The separator between elements is the string providing this method.\n\nstr.ljust(width[, fillchar])\n\n Return the string left justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.lower()\n\n Return a copy of the string with all the cased characters [4]\n converted to lowercase.\n\n The lowercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.lstrip([chars])\n\n Return a copy of the string with leading characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a prefix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.lstrip()\n \'spacious \'\n >>> \'www.example.com\'.lstrip(\'cmowz.\')\n \'example.com\'\n\nstatic str.maketrans(x[, y[, z]])\n\n This static method returns a translation table usable for\n ``str.translate()``.\n\n If there is only one argument, it must be a dictionary mapping\n Unicode ordinals (integers) or characters (strings of length 1) to\n Unicode ordinals, strings (of arbitrary lengths) or None.\n Character keys will then be converted to ordinals.\n\n If there are two arguments, they must be strings of equal length,\n and in the resulting dictionary, each character in x will be mapped\n to the character at the same position in y. If there is a third\n argument, it must be a string, whose characters will be mapped to\n None in the result.\n\nstr.partition(sep)\n\n Split the string at the first occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing the string itself, followed by\n two empty strings.\n\nstr.replace(old, new[, count])\n\n Return a copy of the string with all occurrences of substring *old*\n replaced by *new*. If the optional argument *count* is given, only\n the first *count* occurrences are replaced.\n\nstr.rfind(sub[, start[, end]])\n\n Return the highest index in the string where substring *sub* is\n found, such that *sub* is contained within ``s[start:end]``.\n Optional arguments *start* and *end* are interpreted as in slice\n notation. Return ``-1`` on failure.\n\nstr.rindex(sub[, start[, end]])\n\n Like ``rfind()`` but raises ``ValueError`` when the substring *sub*\n is not found.\n\nstr.rjust(width[, fillchar])\n\n Return the string right justified in a string of length *width*.\n Padding is done using the specified *fillchar* (default is a\n space). The original string is returned if *width* is less than or\n equal to ``len(s)``.\n\nstr.rpartition(sep)\n\n Split the string at the last occurrence of *sep*, and return a\n 3-tuple containing the part before the separator, the separator\n itself, and the part after the separator. If the separator is not\n found, return a 3-tuple containing two empty strings, followed by\n the string itself.\n\nstr.rsplit(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit* splits\n are done, the *rightmost* ones. If *sep* is not specified or\n ``None``, any whitespace string is a separator. Except for\n splitting from the right, ``rsplit()`` behaves like ``split()``\n which is described in detail below.\n\nstr.rstrip([chars])\n\n Return a copy of the string with trailing characters removed. The\n *chars* argument is a string specifying the set of characters to be\n removed. If omitted or ``None``, the *chars* argument defaults to\n removing whitespace. The *chars* argument is not a suffix; rather,\n all combinations of its values are stripped:\n\n >>> \' spacious \'.rstrip()\n \' spacious\'\n >>> \'mississippi\'.rstrip(\'ipz\')\n \'mississ\'\n\nstr.split(sep=None, maxsplit=-1)\n\n Return a list of the words in the string, using *sep* as the\n delimiter string. If *maxsplit* is given, at most *maxsplit*\n splits are done (thus, the list will have at most ``maxsplit+1``\n elements). If *maxsplit* is not specified or ``-1``, then there is\n no limit on the number of splits (all possible splits are made).\n\n If *sep* is given, consecutive delimiters are not grouped together\n and are deemed to delimit empty strings (for example,\n ``\'1,,2\'.split(\',\')`` returns ``[\'1\', \'\', \'2\']``). The *sep*\n argument may consist of multiple characters (for example,\n ``\'1<>2<>3\'.split(\'<>\')`` returns ``[\'1\', \'2\', \'3\']``). Splitting\n an empty string with a specified separator returns ``[\'\']``.\n\n If *sep* is not specified or is ``None``, a different splitting\n algorithm is applied: runs of consecutive whitespace are regarded\n as a single separator, and the result will contain no empty strings\n at the start or end if the string has leading or trailing\n whitespace. Consequently, splitting an empty string or a string\n consisting of just whitespace with a ``None`` separator returns\n ``[]``.\n\n For example, ``\' 1 2 3 \'.split()`` returns ``[\'1\', \'2\', \'3\']``,\n and ``\' 1 2 3 \'.split(None, 1)`` returns ``[\'1\', \'2 3 \']``.\n\nstr.splitlines([keepends])\n\n Return a list of the lines in the string, breaking at line\n boundaries. Line breaks are not included in the resulting list\n unless *keepends* is given and true.\n\nstr.startswith(prefix[, start[, end]])\n\n Return ``True`` if string starts with the *prefix*, otherwise\n return ``False``. *prefix* can also be a tuple of prefixes to look\n for. With optional *start*, test string beginning at that\n position. With optional *end*, stop comparing string at that\n position.\n\nstr.strip([chars])\n\n Return a copy of the string with the leading and trailing\n characters removed. The *chars* argument is a string specifying the\n set of characters to be removed. If omitted or ``None``, the\n *chars* argument defaults to removing whitespace. The *chars*\n argument is not a prefix or suffix; rather, all combinations of its\n values are stripped:\n\n >>> \' spacious \'.strip()\n \'spacious\'\n >>> \'www.example.com\'.strip(\'cmowz.\')\n \'example\'\n\nstr.swapcase()\n\n Return a copy of the string with uppercase characters converted to\n lowercase and vice versa. Note that it is not necessarily true that\n ``s.swapcase().swapcase() == s``.\n\nstr.title()\n\n Return a titlecased version of the string where words start with an\n uppercase character and the remaining characters are lowercase.\n\n The algorithm uses a simple language-independent definition of a\n word as groups of consecutive letters. The definition works in\n many contexts but it means that apostrophes in contractions and\n possessives form word boundaries, which may not be the desired\n result:\n\n >>> "they\'re bill\'s friends from the UK".title()\n "They\'Re Bill\'S Friends From The Uk"\n\n A workaround for apostrophes can be constructed using regular\n expressions:\n\n >>> import re\n >>> def titlecase(s):\n return re.sub(r"[A-Za-z]+(\'[A-Za-z]+)?",\n lambda mo: mo.group(0)[0].upper() +\n mo.group(0)[1:].lower(),\n s)\n\n >>> titlecase("they\'re bill\'s friends.")\n "They\'re Bill\'s Friends."\n\nstr.translate(map)\n\n Return a copy of the *s* where all characters have been mapped\n through the *map* which must be a dictionary of Unicode ordinals\n (integers) to Unicode ordinals, strings or ``None``. Unmapped\n characters are left untouched. Characters mapped to ``None`` are\n deleted.\n\n You can use ``str.maketrans()`` to create a translation map from\n character-to-character mappings in different formats.\n\n Note: An even more flexible approach is to create a custom character\n mapping codec using the ``codecs`` module (see\n ``encodings.cp1251`` for an example).\n\nstr.upper()\n\n Return a copy of the string with all the cased characters [4]\n converted to uppercase. Note that ``str.upper().isupper()`` might\n be ``False`` if ``s`` contains uncased characters or if the Unicode\n category of the resulting character(s) is not "Lu" (Letter,\n uppercase), but e.g. "Lt" (Letter, titlecase).\n\n The uppercasing algorithm used is described in section 3.13 of the\n Unicode Standard.\n\nstr.zfill(width)\n\n Return the numeric string left filled with zeros in a string of\n length *width*. A sign prefix is handled correctly. The original\n string is returned if *width* is less than or equal to ``len(s)``.\n\n\nOld String Formatting Operations\n================================\n\nNote: The formatting operations described here are modelled on C\'s\n printf() syntax. They only support formatting of certain builtin\n types. The use of a binary operator means that care may be needed\n in order to format tuples and dictionaries correctly. As the new\n *String Formatting* syntax is more flexible and handles tuples and\n dictionaries naturally, it is recommended for new code. However,\n there are no current plans to deprecate printf-style formatting.\n\nString objects have one unique built-in operation: the ``%`` operator\n(modulo). This is also known as the string *formatting* or\n*interpolation* operator. Given ``format % values`` (where *format* is\na string), ``%`` conversion specifications in *format* are replaced\nwith zero or more elements of *values*. The effect is similar to the\nusing ``sprintf()`` in the C language.\n\nIf *format* requires a single argument, *values* may be a single non-\ntuple object. [5] Otherwise, *values* must be a tuple with exactly\nthe number of items specified by the format string, or a single\nmapping object (for example, a dictionary).\n\nA conversion specifier contains two or more characters and has the\nfollowing components, which must occur in this order:\n\n1. The ``\'%\'`` character, which marks the start of the specifier.\n\n2. Mapping key (optional), consisting of a parenthesised sequence of\n characters (for example, ``(somename)``).\n\n3. Conversion flags (optional), which affect the result of some\n conversion types.\n\n4. Minimum field width (optional). If specified as an ``\'*\'``\n (asterisk), the actual width is read from the next element of the\n tuple in *values*, and the object to convert comes after the\n minimum field width and optional precision.\n\n5. Precision (optional), given as a ``\'.\'`` (dot) followed by the\n precision. If specified as ``\'*\'`` (an asterisk), the actual\n precision is read from the next element of the tuple in *values*,\n and the value to convert comes after the precision.\n\n6. Length modifier (optional).\n\n7. Conversion type.\n\nWhen the right argument is a dictionary (or other mapping type), then\nthe formats in the string *must* include a parenthesised mapping key\ninto that dictionary inserted immediately after the ``\'%\'`` character.\nThe mapping key selects the value to be formatted from the mapping.\nFor example:\n\n>>> print(\'%(language)s has %(number)03d quote types.\' %\n... {\'language\': "Python", "number": 2})\nPython has 002 quote types.\n\nIn this case no ``*`` specifiers may occur in a format (since they\nrequire a sequential parameter list).\n\nThe conversion flag characters are:\n\n+-----------+-----------------------------------------------------------------------+\n| Flag | Meaning |\n+===========+=======================================================================+\n| ``\'#\'`` | The value conversion will use the "alternate form" (where defined |\n| | below). |\n+-----------+-----------------------------------------------------------------------+\n| ``\'0\'`` | The conversion will be zero padded for numeric values. |\n+-----------+-----------------------------------------------------------------------+\n| ``\'-\'`` | The converted value is left adjusted (overrides the ``\'0\'`` |\n| | conversion if both are given). |\n+-----------+-----------------------------------------------------------------------+\n| ``\' \'`` | (a space) A blank should be left before a positive number (or empty |\n| | string) produced by a signed conversion. |\n+-----------+-----------------------------------------------------------------------+\n| ``\'+\'`` | A sign character (``\'+\'`` or ``\'-\'``) will precede the conversion |\n| | (overrides a "space" flag). |\n+-----------+-----------------------------------------------------------------------+\n\nA length modifier (``h``, ``l``, or ``L``) may be present, but is\nignored as it is not necessary for Python -- so e.g. ``%ld`` is\nidentical to ``%d``.\n\nThe conversion types are:\n\n+--------------+-------------------------------------------------------+---------+\n| Conversion | Meaning | Notes |\n+==============+=======================================================+=========+\n| ``\'d\'`` | Signed integer decimal. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'i\'`` | Signed integer decimal. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'o\'`` | Signed octal value. | (1) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'u\'`` | Obsolete type -- it is identical to ``\'d\'``. | (7) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'x\'`` | Signed hexadecimal (lowercase). | (2) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'X\'`` | Signed hexadecimal (uppercase). | (2) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'e\'`` | Floating point exponential format (lowercase). | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'E\'`` | Floating point exponential format (uppercase). | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'f\'`` | Floating point decimal format. | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'F\'`` | Floating point decimal format. | (3) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'g\'`` | Floating point format. Uses lowercase exponential | (4) |\n| | format if exponent is less than -4 or not less than | |\n| | precision, decimal format otherwise. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'G\'`` | Floating point format. Uses uppercase exponential | (4) |\n| | format if exponent is less than -4 or not less than | |\n| | precision, decimal format otherwise. | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'c\'`` | Single character (accepts integer or single character | |\n| | string). | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'r\'`` | String (converts any Python object using ``repr()``). | (5) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'s\'`` | String (converts any Python object using ``str()``). | (5) |\n+--------------+-------------------------------------------------------+---------+\n| ``\'a\'`` | String (converts any Python object using | (5) |\n| | ``ascii()``). | |\n+--------------+-------------------------------------------------------+---------+\n| ``\'%\'`` | No argument is converted, results in a ``\'%\'`` | |\n| | character in the result. | |\n+--------------+-------------------------------------------------------+---------+\n\nNotes:\n\n1. The alternate form causes a leading zero (``\'0\'``) to be inserted\n between left-hand padding and the formatting of the number if the\n leading character of the result is not already a zero.\n\n2. The alternate form causes a leading ``\'0x\'`` or ``\'0X\'`` (depending\n on whether the ``\'x\'`` or ``\'X\'`` format was used) to be inserted\n between left-hand padding and the formatting of the number if the\n leading character of the result is not already a zero.\n\n3. The alternate form causes the result to always contain a decimal\n point, even if no digits follow it.\n\n The precision determines the number of digits after the decimal\n point and defaults to 6.\n\n4. The alternate form causes the result to always contain a decimal\n point, and trailing zeroes are not removed as they would otherwise\n be.\n\n The precision determines the number of significant digits before\n and after the decimal point and defaults to 6.\n\n5. If precision is ``N``, the output is truncated to ``N`` characters.\n\n1. See **PEP 237**.\n\nSince Python strings have an explicit length, ``%s`` conversions do\nnot assume that ``\'\\0\'`` is the end of the string.\n\nChanged in version 3.1: ``%f`` conversions for numbers whose absolute\nvalue is over 1e50 are no longer replaced by ``%g`` conversions.\n\nAdditional string operations are defined in standard modules\n``string`` and ``re``.\n\n\nRange Type\n==========\n\nThe ``range`` type is an immutable sequence which is commonly used for\nlooping. The advantage of the ``range`` type is that an ``range``\nobject will always take the same amount of memory, no matter the size\nof the range it represents.\n\nRange objects have relatively little behavior: they support indexing,\ncontains, iteration, the ``len()`` function, and the following\nmethods:\n\nrange.count(x)\n\n Return the number of *i*\'s for which ``s[i] == x``.\n\n New in version 3.2.\n\nrange.index(x)\n\n Return the smallest *i* such that ``s[i] == x``. Raises\n ``ValueError`` when *x* is not in the range.\n\n New in version 3.2.\n\n\nMutable Sequence Types\n======================\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object. Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation | Result | Notes |\n+================================+==================================+=======================+\n| ``s[i] = x`` | item *i* of *s* is replaced by | |\n| | *x* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t`` | slice of *s* from *i* to *j* is | |\n| | replaced by the contents of the | |\n| | iterable *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]`` | same as ``s[i:j] = []`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t`` | the elements of ``s[i:j:k]`` are | (1) |\n| | replaced by those of *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]`` | removes the elements of | |\n| | ``s[i:j:k]`` from the list | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)`` | same as ``s[len(s):len(s)] = | |\n| | [x]`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)`` | same as ``s[len(s):len(s)] = x`` | (2) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.clear()`` | remove all items from ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.copy()`` | return a shallow copy of ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)`` | return number of *i*\'s for which | |\n| | ``s[i] == x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])`` | return smallest *k* such that | (3) |\n| | ``s[k] == x`` and ``i <= k < j`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)`` | same as ``s[i:i] = [x]`` | (4) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])`` | same as ``x = s[i]; del s[i]; | (5) |\n| | return x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)`` | same as ``del s[s.index(x)]`` | (3) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()`` | reverses the items of *s* in | (6) |\n| | place | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])`` | sort the items of *s* in place | (6), (7), (8) |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n index is passed as the second or third parameter to the ``index()``\n method, the sequence length is added, as for slice indices. If it\n is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n ``insert()`` method, the sequence length is added, as for slice\n indices. If it is still negative, it is truncated to zero, as for\n slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n place for economy of space when sorting or reversing a large\n sequence. To remind you that they operate by side effect, they\n don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n comparisons. Each must be specified as a keyword argument.\n\n *key* specifies a function of one argument that is used to extract\n a comparison key from each list element: ``key=str.lower``. The\n default value is ``None``. Use ``functools.cmp_to_key()`` to\n convert an old-style *cmp* function to a *key* function.\n\n *reverse* is a boolean value. If set to ``True``, then the list\n elements are sorted as if each comparison were reversed.\n\n The ``sort()`` method is guaranteed to be stable. A sort is stable\n if it guarantees not to change the relative order of elements that\n compare equal --- this is helpful for sorting in multiple passes\n (for example, sort by department, then by salary grade).\n\n **CPython implementation detail:** While a list is being sorted,\n the effect of attempting to mutate, or even inspect, the list is\n undefined. The C implementation of Python makes the list appear\n empty for the duration, and raises ``ValueError`` if it can detect\n that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n New in version 3.3: ``clear()`` and ``copy()`` methods.\n\n\nBytes and Byte Array Methods\n============================\n\nBytes and bytearray objects, being "strings of bytes", have all\nmethods found on strings, with the exception of ``encode()``,\n``format()`` and ``isidentifier()``, which do not make sense with\nthese types. For converting the objects to strings, they have a\n``decode()`` method.\n\nWherever one of these methods needs to interpret the bytes as\ncharacters (e.g. the ``is...()`` methods), the ASCII character set is\nassumed.\n\nNew in version 3.3: The functions ``count()``, ``find()``,\n``index()``, ``rfind()`` and ``rindex()`` have additional semantics\ncompared to the corresponding string functions: They also accept an\ninteger in range 0 to 255 (a byte) as their first argument.\n\nNote: The methods on bytes and bytearray objects don\'t accept strings as\n their arguments, just as the methods on strings don\'t accept bytes\n as their arguments. For example, you have to write\n\n a = "abc"\n b = a.replace("a", "f")\n\n and\n\n a = b"abc"\n b = a.replace(b"a", b"f")\n\nbytes.decode(encoding="utf-8", errors="strict")\nbytearray.decode(encoding="utf-8", errors="strict")\n\n Return a string decoded from the given bytes. Default encoding is\n ``\'utf-8\'``. *errors* may be given to set a different error\n handling scheme. The default for *errors* is ``\'strict\'``, meaning\n that encoding errors raise a ``UnicodeError``. Other possible\n values are ``\'ignore\'``, ``\'replace\'`` and any other name\n registered via ``codecs.register_error()``, see section *Codec Base\n Classes*. For a list of possible encodings, see section *Standard\n Encodings*.\n\n Changed in version 3.1: Added support for keyword arguments.\n\nThe bytes and bytearray types have an additional class method:\n\nclassmethod bytes.fromhex(string)\nclassmethod bytearray.fromhex(string)\n\n This ``bytes`` class method returns a bytes or bytearray object,\n decoding the given string object. The string must contain two\n hexadecimal digits per byte, spaces are ignored.\n\n >>> bytes.fromhex(\'f0 f1f2 \')\n b\'\\xf0\\xf1\\xf2\'\n\nThe maketrans and translate methods differ in semantics from the\nversions available on strings:\n\nbytes.translate(table[, delete])\nbytearray.translate(table[, delete])\n\n Return a copy of the bytes or bytearray object where all bytes\n occurring in the optional argument *delete* are removed, and the\n remaining bytes have been mapped through the given translation\n table, which must be a bytes object of length 256.\n\n You can use the ``bytes.maketrans()`` method to create a\n translation table.\n\n Set the *table* argument to ``None`` for translations that only\n delete characters:\n\n >>> b\'read this short text\'.translate(None, b\'aeiou\')\n b\'rd ths shrt txt\'\n\nstatic bytes.maketrans(from, to)\nstatic bytearray.maketrans(from, to)\n\n This static method returns a translation table usable for\n ``bytes.translate()`` that will map each character in *from* into\n the character at the same position in *to*; *from* and *to* must be\n bytes objects and have the same length.\n\n New in version 3.1.\n', 'typesseq-mutable': '\nMutable Sequence Types\n**********************\n\nList and bytearray objects support additional operations that allow\nin-place modification of the object. Other mutable sequence types\n(when added to the language) should also support these operations.\nStrings and tuples are immutable sequence types: such objects cannot\nbe modified once created. The following operations are defined on\nmutable sequence types (where *x* is an arbitrary object).\n\nNote that while lists allow their items to be of any type, bytearray\nobject "items" are all integers in the range 0 <= x < 256.\n\n+--------------------------------+----------------------------------+-----------------------+\n| Operation | Result | Notes |\n+================================+==================================+=======================+\n| ``s[i] = x`` | item *i* of *s* is replaced by | |\n| | *x* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j] = t`` | slice of *s* from *i* to *j* is | |\n| | replaced by the contents of the | |\n| | iterable *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j]`` | same as ``s[i:j] = []`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s[i:j:k] = t`` | the elements of ``s[i:j:k]`` are | (1) |\n| | replaced by those of *t* | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``del s[i:j:k]`` | removes the elements of | |\n| | ``s[i:j:k]`` from the list | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.append(x)`` | same as ``s[len(s):len(s)] = | |\n| | [x]`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.extend(x)`` | same as ``s[len(s):len(s)] = x`` | (2) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.clear()`` | remove all items from ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.copy()`` | return a shallow copy of ``s`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.count(x)`` | return number of *i*\'s for which | |\n| | ``s[i] == x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.index(x[, i[, j]])`` | return smallest *k* such that | (3) |\n| | ``s[k] == x`` and ``i <= k < j`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.insert(i, x)`` | same as ``s[i:i] = [x]`` | (4) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.pop([i])`` | same as ``x = s[i]; del s[i]; | (5) |\n| | return x`` | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.remove(x)`` | same as ``del s[s.index(x)]`` | (3) |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.reverse()`` | reverses the items of *s* in | (6) |\n| | place | |\n+--------------------------------+----------------------------------+-----------------------+\n| ``s.sort([key[, reverse]])`` | sort the items of *s* in place | (6), (7), (8) |\n+--------------------------------+----------------------------------+-----------------------+\n\nNotes:\n\n1. *t* must have the same length as the slice it is replacing.\n\n2. *x* can be any iterable object.\n\n3. Raises ``ValueError`` when *x* is not found in *s*. When a negative\n index is passed as the second or third parameter to the ``index()``\n method, the sequence length is added, as for slice indices. If it\n is still negative, it is truncated to zero, as for slice indices.\n\n4. When a negative index is passed as the first parameter to the\n ``insert()`` method, the sequence length is added, as for slice\n indices. If it is still negative, it is truncated to zero, as for\n slice indices.\n\n5. The optional argument *i* defaults to ``-1``, so that by default\n the last item is removed and returned.\n\n6. The ``sort()`` and ``reverse()`` methods modify the sequence in\n place for economy of space when sorting or reversing a large\n sequence. To remind you that they operate by side effect, they\n don\'t return the sorted or reversed sequence.\n\n7. The ``sort()`` method takes optional arguments for controlling the\n comparisons. Each must be specified as a keyword argument.\n\n *key* specifies a function of one argument that is used to extract\n a comparison key from each list element: ``key=str.lower``. The\n default value is ``None``. Use ``functools.cmp_to_key()`` to\n convert an old-style *cmp* function to a *key* function.\n\n *reverse* is a boolean value. If set to ``True``, then the list\n elements are sorted as if each comparison were reversed.\n\n The ``sort()`` method is guaranteed to be stable. A sort is stable\n if it guarantees not to change the relative order of elements that\n compare equal --- this is helpful for sorting in multiple passes\n (for example, sort by department, then by salary grade).\n\n **CPython implementation detail:** While a list is being sorted,\n the effect of attempting to mutate, or even inspect, the list is\n undefined. The C implementation of Python makes the list appear\n empty for the duration, and raises ``ValueError`` if it can detect\n that the list has been mutated during a sort.\n\n8. ``sort()`` is not supported by ``bytearray`` objects.\n\n New in version 3.3: ``clear()`` and ``copy()`` methods.\n', 'unary': '\nUnary arithmetic and bitwise operations\n***************************************\n\nAll unary arithmetic and bitwise operations have the same priority:\n\n u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr\n\nThe unary ``-`` (minus) operator yields the negation of its numeric\nargument.\n\nThe unary ``+`` (plus) operator yields its numeric argument unchanged.\n\nThe unary ``~`` (invert) operator yields the bitwise inversion of its\ninteger argument. The bitwise inversion of ``x`` is defined as\n``-(x+1)``. It only applies to integral numbers.\n\nIn all three cases, if the argument does not have the proper type, a\n``TypeError`` exception is raised.\n', 'while': '\nThe ``while`` statement\n***********************\n\nThe ``while`` statement is used for repeated execution as long as an\nexpression is true:\n\n while_stmt ::= "while" expression ":" suite\n ["else" ":" suite]\n\nThis repeatedly tests the expression and, if it is true, executes the\nfirst suite; if the expression is false (which may be the first time\nit is tested) the suite of the ``else`` clause, if present, is\nexecuted and the loop terminates.\n\nA ``break`` statement executed in the first suite terminates the loop\nwithout executing the ``else`` clause\'s suite. A ``continue``\nstatement executed in the first suite skips the rest of the suite and\ngoes back to testing the expression.\n', -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 22:04:58 2012 From: python-checkins at python.org (georg.brandl) Date: Wed, 30 May 2012 22:04:58 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Bump_version_to_3=2E3=2E0a4?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/7c51388a3aa7 changeset: 77248:7c51388a3aa7 tag: v3.3.0a4 user: Georg Brandl date: Wed May 30 22:04:31 2012 +0200 summary: Bump version to 3.3.0a4. files: Include/patchlevel.h | 4 ++-- Lib/distutils/__init__.py | 2 +- Lib/idlelib/idlever.py | 2 +- Misc/NEWS | 2 +- Misc/RPM/python-3.3.spec | 2 +- README | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 3 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA -#define PY_RELEASE_SERIAL 3 +#define PY_RELEASE_SERIAL 4 /* Version as a string */ -#define PY_VERSION "3.3.0a3+" +#define PY_VERSION "3.3.0a4" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/distutils/__init__.py b/Lib/distutils/__init__.py --- a/Lib/distutils/__init__.py +++ b/Lib/distutils/__init__.py @@ -13,5 +13,5 @@ # Updated automatically by the Python release process. # #--start constants-- -__version__ = "3.3.0a3" +__version__ = "3.3.0a4" #--end constants-- diff --git a/Lib/idlelib/idlever.py b/Lib/idlelib/idlever.py --- a/Lib/idlelib/idlever.py +++ b/Lib/idlelib/idlever.py @@ -1,1 +1,1 @@ -IDLE_VERSION = "3.3.0a3" +IDLE_VERSION = "3.3.0a4" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -5,7 +5,7 @@ What's New in Python 3.3.0 Alpha 4? =================================== -*Release date: XX-XXX-2012* +*Release date: 31-May-2012* Core and Builtins ----------------- diff --git a/Misc/RPM/python-3.3.spec b/Misc/RPM/python-3.3.spec --- a/Misc/RPM/python-3.3.spec +++ b/Misc/RPM/python-3.3.spec @@ -39,7 +39,7 @@ %define name python #--start constants-- -%define version 3.3.0a3 +%define version 3.3.0a4 %define libvers 3.3 #--end constants-- %define release 1pydotorg diff --git a/README b/README --- a/README +++ b/README @@ -1,4 +1,4 @@ -This is Python version 3.3.0 alpha 3 +This is Python version 3.3.0 alpha 4 ==================================== Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 22:05:06 2012 From: python-checkins at python.org (georg.brandl) Date: Wed, 30 May 2012 22:05:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Added_tag_v3=2E3=2E0a4_for_?= =?utf8?q?changeset_7c51388a3aa7?= Message-ID: http://hg.python.org/cpython/rev/377e376fa802 changeset: 77249:377e376fa802 user: Georg Brandl date: Wed May 30 22:04:40 2012 +0200 summary: Added tag v3.3.0a4 for changeset 7c51388a3aa7 files: .hgtags | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/.hgtags b/.hgtags --- a/.hgtags +++ b/.hgtags @@ -102,3 +102,4 @@ f1a9a6505731714f0e157453ff850e3b71615c45 v3.3.0a1 2f69db52d6de306cdaef0a0cc00cc823fb350b01 v3.3.0a2 0b53b70a40a00013505eb35e3660057b62be77be v3.3.0a3 +7c51388a3aa7ce76a8541bbbdfc05d2d259a162c v3.3.0a4 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Wed May 30 22:05:07 2012 From: python-checkins at python.org (georg.brandl) Date: Wed, 30 May 2012 22:05:07 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?b?KTogTWVyZ2UgaGVhZHMu?= Message-ID: http://hg.python.org/cpython/rev/f1dd70bfb4c5 changeset: 77250:f1dd70bfb4c5 parent: 77249:377e376fa802 parent: 77246:20b8f0ee3d64 user: Georg Brandl date: Wed May 30 22:04:57 2012 +0200 summary: Merge heads. files: Doc/library/xml.etree.elementtree.rst | 6 +- Lib/test/test_xml_etree.py | 18 +- Modules/_elementtree.c | 105 +++++++++---- 3 files changed, 91 insertions(+), 38 deletions(-) diff --git a/Doc/library/xml.etree.elementtree.rst b/Doc/library/xml.etree.elementtree.rst --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@ -716,9 +716,9 @@ Generic element structure builder. This builder converts a sequence of start, data, and end method calls to a well-formed element structure. You can use this class to build an element structure using a custom XML parser, - or a parser for some other XML-like format. The *element_factory* is called - to create new :class:`Element` instances when given. - + or a parser for some other XML-like format. *element_factory*, when given, + must be a callable accepting two positional arguments: a tag and + a dict of attributes. It is expected to return a new element instance. .. method:: close() diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -1959,6 +1959,8 @@ ' "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">' 'text') + sample2 = '''sometext''' + def test_dummy_builder(self): class BaseDummyBuilder: def close(self): @@ -1993,11 +1995,19 @@ e = parser.close() self.assertEqual(e.tag, 'html') - # XXX in _elementtree, the constructor of TreeBuilder expects no - # arguments - @unittest.expectedFailure def test_element_factory(self): - tb = ET.TreeBuilder(element_factory=lambda: ET.Element()) + lst = [] + def myfactory(tag, attrib): + nonlocal lst + lst.append(tag) + return ET.Element(tag, attrib) + + tb = ET.TreeBuilder(element_factory=myfactory) + parser = ET.XMLParser(target=tb) + parser.feed(self.sample2) + parser.close() + + self.assertEqual(lst, ['toplevel']) @unittest.expectedFailure # XXX issue 14007 with C ElementTree def test_doctype(self): diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c --- a/Modules/_elementtree.c +++ b/Modules/_elementtree.c @@ -191,6 +191,15 @@ return result; } +/* Is the given object an empty dictionary? +*/ +static int +is_empty_dict(PyObject *obj) +{ + return PyDict_CheckExact(obj) && PyDict_Size(obj) == 0; +} + + /* -------------------------------------------------------------------- */ /* the Element type */ @@ -297,14 +306,9 @@ self = PyObject_GC_New(ElementObject, &Element_Type); if (self == NULL) return NULL; - - /* use None for empty dictionaries */ - if (PyDict_CheckExact(attrib) && !PyDict_Size(attrib)) - attrib = Py_None; - self->extra = NULL; - if (attrib != Py_None) { + if (attrib != Py_None && !is_empty_dict(attrib)) { if (create_extra(self, attrib) < 0) { PyObject_Del(self); return NULL; @@ -416,22 +420,14 @@ self_elem = (ElementObject *)self; - /* Use None for empty dictionaries */ - if (PyDict_CheckExact(attrib) && PyDict_Size(attrib) == 0) { - Py_INCREF(Py_None); - attrib = Py_None; - } - - if (attrib != Py_None) { + if (attrib != Py_None && !is_empty_dict(attrib)) { if (create_extra(self_elem, attrib) < 0) { PyObject_Del(self_elem); return -1; } } - /* If create_extra needed attrib, it took a reference to it, so we can - * release ours anyway. - */ + /* We own a reference to attrib here and it's no longer needed. */ Py_DECREF(attrib); /* Replace the objects already pointed to by tag, text and tail. */ @@ -1813,6 +1809,8 @@ PyObject *stack; /* element stack */ Py_ssize_t index; /* current stack size (0 means empty) */ + PyObject *element_factory; + /* element tracing */ PyObject *events; /* list of events, or NULL if not collecting */ PyObject *start_event_obj; /* event objects (NULL to ignore) */ @@ -1841,6 +1839,7 @@ t->last = (ElementObject *)Py_None; t->data = NULL; + t->element_factory = NULL; t->stack = PyList_New(20); if (!t->stack) { Py_DECREF(t->this); @@ -1859,11 +1858,38 @@ static int treebuilder_init(PyObject *self, PyObject *args, PyObject *kwds) { + static char *kwlist[] = {"element_factory", NULL}; + PyObject *element_factory = NULL; + TreeBuilderObject *self_tb = (TreeBuilderObject *)self; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O:TreeBuilder", kwlist, + &element_factory)) { + return -1; + } + + if (element_factory) { + Py_INCREF(element_factory); + Py_XDECREF(self_tb->element_factory); + self_tb->element_factory = element_factory; + } + return 0; } -static void -treebuilder_dealloc(TreeBuilderObject *self) +static int +treebuilder_gc_traverse(TreeBuilderObject *self, visitproc visit, void *arg) +{ + Py_VISIT(self->root); + Py_VISIT(self->this); + Py_VISIT(self->last); + Py_VISIT(self->data); + Py_VISIT(self->stack); + Py_VISIT(self->element_factory); + return 0; +} + +static int +treebuilder_gc_clear(TreeBuilderObject *self) { Py_XDECREF(self->end_ns_event_obj); Py_XDECREF(self->start_ns_event_obj); @@ -1874,8 +1900,16 @@ Py_XDECREF(self->data); Py_DECREF(self->last); Py_DECREF(self->this); + Py_CLEAR(self->element_factory); Py_XDECREF(self->root); - + return 0; +} + +static void +treebuilder_dealloc(TreeBuilderObject *self) +{ + PyObject_GC_UnTrack(self); + treebuilder_gc_clear(self); Py_TYPE(self)->tp_free((PyObject *)self); } @@ -1904,9 +1938,14 @@ self->data = NULL; } - node = create_new_element(tag, attrib); - if (!node) + if (self->element_factory) { + node = PyObject_CallFunction(self->element_factory, "OO", tag, attrib); + } else { + node = create_new_element(tag, attrib); + } + if (!node) { return NULL; + } this = (PyObject*) self->this; @@ -2180,10 +2219,11 @@ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, + /* tp_flags */ 0, /* tp_doc */ - 0, /* tp_traverse */ - 0, /* tp_clear */ + (traverseproc)treebuilder_gc_traverse, /* tp_traverse */ + (inquiry)treebuilder_gc_clear, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ @@ -2443,17 +2483,20 @@ attrib = Py_None; } - if (TreeBuilder_CheckExact(self->target)) + /* If we get None, pass an empty dictionary on */ + if (attrib == Py_None) { + Py_DECREF(attrib); + attrib = PyDict_New(); + if (!attrib) + return; + } + + if (TreeBuilder_CheckExact(self->target)) { /* shortcut */ res = treebuilder_handle_start((TreeBuilderObject*) self->target, tag, attrib); + } else if (self->handle_start) { - if (attrib == Py_None) { - Py_DECREF(attrib); - attrib = PyDict_New(); - if (!attrib) - return; - } res = PyObject_CallFunction(self->handle_start, "OO", tag, attrib); } else res = NULL; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 03:54:12 2012 From: python-checkins at python.org (r.david.murray) Date: Thu, 31 May 2012 03:54:12 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_parameterized_tests_in?= =?utf8?q?_email_less_hackish=2E?= Message-ID: http://hg.python.org/cpython/rev/e6a33938b03f changeset: 77251:e6a33938b03f user: R David Murray date: Wed May 30 21:53:40 2012 -0400 summary: Make parameterized tests in email less hackish. Or perhaps more hackish, depending on your perspective. But at least this way it is now possible to run the individual tests using the unittest CLI. files: Lib/test/test_email/__init__.py | 79 ++++++++++ Lib/test/test_email/test_generator.py | 41 +---- Lib/test/test_email/test_headerregistry.py | 26 +-- Lib/test/test_email/test_pickleable.py | 69 +++----- 4 files changed, 122 insertions(+), 93 deletions(-) diff --git a/Lib/test/test_email/__init__.py b/Lib/test/test_email/__init__.py --- a/Lib/test/test_email/__init__.py +++ b/Lib/test/test_email/__init__.py @@ -71,3 +71,82 @@ for i in range(len(actual)): self.assertIsInstance(actual[i], expected[i], 'item {}'.format(i)) + + +# Metaclass to allow for parameterized tests +class Parameterized(type): + + """Provide a test method parameterization facility. + + Parameters are specified as the value of a class attribute that ends with + the string '_params'. Call the portion before '_params' the prefix. Then + a method to be parameterized must have the same prefix, the string + '_as_', and an arbitrary suffix. + + The value of the _params attribute may be either a dictionary or a list. + The values in the dictionary and the elements of the list may either be + single values, or a list. If single values, they are turned into single + element tuples. However derived, the resulting sequence is passed via + *args to the parameterized test function. + + In a _params dictioanry, the keys become part of the name of the generated + tests. In a _params list, the values in the list are converted into a + string by joining the string values of the elements of the tuple by '_' and + converting any blanks into '_'s, and this become part of the name. The + full name of a generated test is the portion of the _params name before the + '_params' portion, plus an '_', plus the name derived as explained above. + + For example, if we have: + + count_params = range(2) + + def count_as_foo_arg(self, foo): + self.assertEqual(foo+1, myfunc(foo)) + + we will get parameterized test methods named: + test_foo_arg_0 + test_foo_arg_1 + test_foo_arg_2 + + Or we could have: + + example_params = {'foo': ('bar', 1), 'bing': ('bang', 2)} + + def example_as_myfunc_input(self, name, count): + self.assertEqual(name+str(count), myfunc(name, count)) + + and get: + test_myfunc_input_foo + test_myfunc_input_bing + + Note: if and only if the generated test name is a valid identifier can it + be used to select the test individually from the unittest command line. + + """ + + def __new__(meta, classname, bases, classdict): + paramdicts = {} + for name, attr in classdict.items(): + if name.endswith('_params'): + if not hasattr(attr, 'keys'): + d = {} + for x in attr: + if not hasattr(x, '__iter__'): + x = (x,) + n = '_'.join(str(v) for v in x).replace(' ', '_') + d[n] = x + attr = d + paramdicts[name[:-7] + '_as_'] = attr + testfuncs = {} + for name, attr in classdict.items(): + for paramsname, paramsdict in paramdicts.items(): + if name.startswith(paramsname): + testnameroot = 'test_' + name[len(paramsname):] + for paramname, params in paramsdict.items(): + test = (lambda self, name=name, params=params: + getattr(self, name)(*params)) + testname = testnameroot + '_' + paramname + test.__name__ = testname + testfuncs[testname] = test + classdict.update(testfuncs) + return super().__new__(meta, classname, bases, classdict) diff --git a/Lib/test/test_email/test_generator.py b/Lib/test/test_email/test_generator.py --- a/Lib/test/test_email/test_generator.py +++ b/Lib/test/test_email/test_generator.py @@ -4,10 +4,10 @@ from email import message_from_string, message_from_bytes from email.generator import Generator, BytesGenerator from email import policy -from test.test_email import TestEmailBase +from test.test_email import TestEmailBase, Parameterized -class TestGeneratorBase: +class TestGeneratorBase(metaclass=Parameterized): policy = policy.default @@ -80,31 +80,23 @@ "\n" "None\n") - def _test_maxheaderlen_parameter(self, n): + length_params = [n for n in refold_long_expected] + + def length_as_maxheaderlen_parameter(self, n): msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() g = self.genclass(s, maxheaderlen=n, policy=self.policy) g.flatten(msg) self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) - for n in refold_long_expected: - locals()['test_maxheaderlen_parameter_' + str(n)] = ( - lambda self, n=n: - self._test_maxheaderlen_parameter(n)) - - def _test_max_line_length_policy(self, n): + def length_as_max_line_length_policy(self, n): msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() g = self.genclass(s, policy=self.policy.clone(max_line_length=n)) g.flatten(msg) self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) - for n in refold_long_expected: - locals()['test_max_line_length_policy' + str(n)] = ( - lambda self, n=n: - self._test_max_line_length_policy(n)) - - def _test_maxheaderlen_parm_overrides_policy(self, n): + def length_as_maxheaderlen_parm_overrides_policy(self, n): msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() g = self.genclass(s, maxheaderlen=n, @@ -112,12 +104,7 @@ g.flatten(msg) self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[n])) - for n in refold_long_expected: - locals()['test_maxheaderlen_parm_overrides_policy' + str(n)] = ( - lambda self, n=n: - self._test_maxheaderlen_parm_overrides_policy(n)) - - def _test_refold_none_does_not_fold(self, n): + def length_as_max_line_length_with_refold_none_does_not_fold(self, n): msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() g = self.genclass(s, policy=self.policy.clone(refold_source='none', @@ -125,12 +112,7 @@ g.flatten(msg) self.assertEqual(s.getvalue(), self.typ(self.refold_long_expected[0])) - for n in refold_long_expected: - locals()['test_refold_none_does_not_fold' + str(n)] = ( - lambda self, n=n: - self._test_refold_none_does_not_fold(n)) - - def _test_refold_all(self, n): + def length_as_max_line_length_with_refold_all_folds(self, n): msg = self.msgmaker(self.typ(self.refold_long_expected[0])) s = self.ioclass() g = self.genclass(s, policy=self.policy.clone(refold_source='all', @@ -138,11 +120,6 @@ g.flatten(msg) self.assertEqual(s.getvalue(), self.typ(self.refold_all_expected[n])) - for n in refold_long_expected: - locals()['test_refold_all' + str(n)] = ( - lambda self, n=n: - self._test_refold_all(n)) - def test_crlf_control_via_policy(self): source = "Subject: test\r\n\r\ntest body\r\n" expected = source diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py --- a/Lib/test/test_email/test_headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -4,7 +4,7 @@ from email import errors from email import policy from email.message import Message -from test.test_email import TestEmailBase +from test.test_email import TestEmailBase, Parameterized from email import headerregistry from email.headerregistry import Address, Group @@ -175,9 +175,9 @@ self.assertEqual(m['Date'].datetime, self.dt) -class TestAddressHeader(TestHeaderBase): +class TestAddressHeader(TestHeaderBase, metaclass=Parameterized): - examples = { + example_params = { 'empty': ('<>', @@ -305,8 +305,8 @@ # trailing comments, which aren't currently handled. comments in # general are not handled yet. - def _test_single_addr(self, source, defects, decoded, display_name, - addr_spec, username, domain, comment): + def example_as_address(self, source, defects, decoded, display_name, + addr_spec, username, domain, comment): h = self.make_header('sender', source) self.assertEqual(h, decoded) self.assertDefectsEqual(h.defects, defects) @@ -322,13 +322,8 @@ # XXX: we have no comment support yet. #self.assertEqual(a.comment, comment) - for name in examples: - locals()['test_'+name] = ( - lambda self, name=name: - self._test_single_addr(*self.examples[name])) - - def _test_group_single_addr(self, source, defects, decoded, display_name, - addr_spec, username, domain, comment): + def example_as_group(self, source, defects, decoded, display_name, + addr_spec, username, domain, comment): source = 'foo: {};'.format(source) gdecoded = 'foo: {};'.format(decoded) if decoded else 'foo:;' h = self.make_header('to', source) @@ -344,11 +339,6 @@ self.assertEqual(a.username, username) self.assertEqual(a.domain, domain) - for name in examples: - locals()['test_group_'+name] = ( - lambda self, name=name: - self._test_group_single_addr(*self.examples[name])) - def test_simple_address_list(self): value = ('Fred , foo at example.com, ' '"Harry W. Hastings" ') @@ -366,7 +356,7 @@ 'Harry W. Hastings') def test_complex_address_list(self): - examples = list(self.examples.values()) + examples = list(self.example_params.values()) source = ('dummy list:;, another: (empty);,' + ', '.join([x[0] for x in examples[:4]]) + ', ' + r'"A \"list\"": ' + diff --git a/Lib/test/test_email/test_pickleable.py b/Lib/test/test_email/test_pickleable.py --- a/Lib/test/test_email/test_pickleable.py +++ b/Lib/test/test_email/test_pickleable.py @@ -6,83 +6,66 @@ import email.message from email import policy from email.headerregistry import HeaderRegistry -from test.test_email import TestEmailBase +from test.test_email import TestEmailBase, Parameterized -class TestPickleCopyHeader(TestEmailBase): +class TestPickleCopyHeader(TestEmailBase, metaclass=Parameterized): header_factory = HeaderRegistry() unstructured = header_factory('subject', 'this is a test') - def _test_deepcopy(self, name, value): + header_params = { + 'subject': ('subject', 'this is a test'), + 'from': ('from', 'frodo at mordor.net'), + 'to': ('to', 'a: k at b.com, y at z.com;, j at f.com'), + 'date': ('date', 'Tue, 29 May 2012 09:24:26 +1000'), + } + + def header_as_deepcopy(self, name, value): header = self.header_factory(name, value) h = copy.deepcopy(header) self.assertEqual(str(h), str(header)) - def _test_pickle(self, name, value): + def header_as_pickle(self, name, value): header = self.header_factory(name, value) p = pickle.dumps(header) h = pickle.loads(p) self.assertEqual(str(h), str(header)) - headers = ( - ('subject', 'this is a test'), - ('from', 'frodo at mordor.net'), - ('to', 'a: k at b.com, y at z.com;, j at f.com'), - ('date', 'Tue, 29 May 2012 09:24:26 +1000'), - ) - for header in headers: - locals()['test_deepcopy_'+header[0]] = ( - lambda self, header=header: - self._test_deepcopy(*header)) +class TestPickleCopyMessage(TestEmailBase, metaclass=Parameterized): - for header in headers: - locals()['test_pickle_'+header[0]] = ( - lambda self, header=header: - self._test_pickle(*header)) - - -class TestPickleCopyMessage(TestEmailBase): - - msgs = {} + # Message objects are a sequence, so we have to make them a one-tuple in + # msg_params so they get passed to the parameterized test method as a + # single argument instead of as a list of headers. + msg_params = {} # Note: there will be no custom header objects in the parsed message. - msgs['parsed'] = email.message_from_string(textwrap.dedent("""\ + msg_params['parsed'] = (email.message_from_string(textwrap.dedent("""\ Date: Tue, 29 May 2012 09:24:26 +1000 From: frodo at mordor.net To: bilbo at underhill.org Subject: help I think I forgot the ring. - """), policy=policy.default) + """), policy=policy.default),) - msgs['created'] = email.message.Message(policy=policy.default) - msgs['created']['Date'] = 'Tue, 29 May 2012 09:24:26 +1000' - msgs['created']['From'] = 'frodo at mordor.net' - msgs['created']['To'] = 'bilbo at underhill.org' - msgs['created']['Subject'] = 'help' - msgs['created'].set_payload('I think I forgot the ring.') + msg_params['created'] = (email.message.Message(policy=policy.default),) + msg_params['created'][0]['Date'] = 'Tue, 29 May 2012 09:24:26 +1000' + msg_params['created'][0]['From'] = 'frodo at mordor.net' + msg_params['created'][0]['To'] = 'bilbo at underhill.org' + msg_params['created'][0]['Subject'] = 'help' + msg_params['created'][0].set_payload('I think I forgot the ring.') - def _test_deepcopy(self, msg): + def msg_as_deepcopy(self, msg): msg2 = copy.deepcopy(msg) self.assertEqual(msg2.as_string(), msg.as_string()) - def _test_pickle(self, msg): + def msg_as_pickle(self, msg): p = pickle.dumps(msg) msg2 = pickle.loads(p) self.assertEqual(msg2.as_string(), msg.as_string()) - for name, msg in msgs.items(): - locals()['test_deepcopy_'+name] = ( - lambda self, msg=msg: - self._test_deepcopy(msg)) - - for name, msg in msgs.items(): - locals()['test_pickle_'+name] = ( - lambda self, msg=msg: - self._test_pickle(msg)) - if __name__ == '__main__': unittest.main() -- Repository URL: http://hg.python.org/cpython From ncoghlan at gmail.com Thu May 31 05:11:14 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 31 May 2012 13:11:14 +1000 Subject: [Python-checkins] cpython: Make parameterized tests in email less hackish. In-Reply-To: References: Message-ID: I'm not clear on why this is a metaclass rather than a simple class decorator. On Thu, May 31, 2012 at 11:54 AM, r.david.murray wrote: > + ? ?In a _params dictioanry, the keys become part of the name of the generated > + ? ?tests. ?In a _params list, the values in the list are converted into a > + ? ?string by joining the string values of the elements of the tuple by '_' and > + ? ?converting any blanks into '_'s, and this become part of the name. ?The > + ? ?full name of a generated test is the portion of the _params name before the > + ? ?'_params' portion, plus an '_', plus the name derived as explained above. Your description doesn't match your examples or implementation. Assuming the example and implementation are correct (and they look more sensible than the currently described approach), I believe that last sentence should be: "The full name of a generated test is a 'test_' prefix, the portion of the test function name after the '_as_' separator, plus an '_', plus the name derived as explained above." Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Thu May 31 05:53:24 2012 From: solipsis at pitrou.net (solipsis at pitrou.net) Date: Thu, 31 May 2012 05:53:24 +0200 Subject: [Python-checkins] Daily reference leaks (f1dd70bfb4c5): sum=487 Message-ID: results for f1dd70bfb4c5 on branch "default" -------------------------------------------- test_multiprocessing leaked [22, 0, 0] references, sum=22 test_smtplib leaked [154, 154, 154] references, sum=462 test_super leaked [1, 1, 1] references, sum=3 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogrAsVFR', '-x'] From python-checkins at python.org Thu May 31 11:54:59 2012 From: python-checkins at python.org (kristjan.jonsson) Date: Thu, 31 May 2012 11:54:59 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314909=3A_A_number_?= =?utf8?q?of_places_were_using_PyMem=5FRealloc=28=29_apis_and?= Message-ID: http://hg.python.org/cpython/rev/588ea940e5e3 changeset: 77252:588ea940e5e3 user: Kristjan Valur Jonsson date: Thu May 31 09:37:31 2012 +0000 summary: Issue #14909: A number of places were using PyMem_Realloc() apis and PyObject_GC_Resize() with incorrect error handling. In case of errors, the original object would be leaked. This checkin fixes those cases. files: Modules/_localemodule.c | 5 +++-- Modules/_randommodule.c | 7 ++++--- Modules/unicodedata.c | 7 +++++-- Objects/frameobject.c | 6 ++++-- Objects/unicodeobject.c | 6 ++++-- 5 files changed, 20 insertions(+), 11 deletions(-) diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c --- a/Modules/_localemodule.c +++ b/Modules/_localemodule.c @@ -257,11 +257,12 @@ n2 = wcsxfrm(buf, s, n1); if (n2 >= (size_t)n1) { /* more space needed */ - buf = PyMem_Realloc(buf, (n2+1)*sizeof(wchar_t)); - if (!buf) { + wchar_t * new_buf = PyMem_Realloc(buf, (n2+1)*sizeof(wchar_t)); + if (!new_buf) { PyErr_NoMemory(); goto exit; } + buf = new_buf; n2 = wcsxfrm(buf, s, n2+1); } result = PyUnicode_FromWideChar(buf, n2); diff --git a/Modules/_randommodule.c b/Modules/_randommodule.c --- a/Modules/_randommodule.c +++ b/Modules/_randommodule.c @@ -210,7 +210,7 @@ PyObject *masklower = NULL; PyObject *thirtytwo = NULL; PyObject *n = NULL; - unsigned long *key = NULL; + unsigned long *new_key, *key = NULL; unsigned long keymax; /* # of allocated slots in key */ unsigned long keyused; /* # of used slots in key */ int err; @@ -287,10 +287,11 @@ PyErr_NoMemory(); goto Done; } - key = (unsigned long *)PyMem_Realloc(key, + new_key = (unsigned long *)PyMem_Realloc(key, bigger * sizeof(*key)); - if (key == NULL) + if (new_key == NULL) goto Done; + key = new_key; keymax = bigger; } assert(keyused < keymax); diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c --- a/Modules/unicodedata.c +++ b/Modules/unicodedata.c @@ -526,13 +526,16 @@ /* Hangul Decomposition adds three characters in a single step, so we need atleast that much room. */ if (space < 3) { + Py_UCS4 *new_output; osize += 10; space += 10; - output = PyMem_Realloc(output, osize*sizeof(Py_UCS4)); - if (output == NULL) { + new_output = PyMem_Realloc(output, osize*sizeof(Py_UCS4)); + if (new_output == NULL) { + PyMem_Free(output); PyErr_NoMemory(); return NULL; } + output = new_output; } /* Hangul Decomposition. */ if (SBase <= code && code < (SBase+SCount)) { diff --git a/Objects/frameobject.c b/Objects/frameobject.c --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -663,11 +663,13 @@ f = free_list; free_list = free_list->f_back; if (Py_SIZE(f) < extras) { - f = PyObject_GC_Resize(PyFrameObject, f, extras); - if (f == NULL) { + PyFrameObject *new_f = PyObject_GC_Resize(PyFrameObject, f, extras); + if (new_f == NULL) { + PyObject_GC_Del(f); Py_DECREF(builtins); return NULL; } + f = new_f; } _Py_NewReference((PyObject *)f); } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -8343,13 +8343,15 @@ Py_ssize_t requiredsize) { Py_ssize_t oldsize = *psize; + Py_UCS4 *new_outobj; if (requiredsize > oldsize) { /* exponentially overallocate to minimize reallocations */ if (requiredsize < 2 * oldsize) requiredsize = 2 * oldsize; - *outobj = PyMem_Realloc(*outobj, requiredsize * sizeof(Py_UCS4)); - if (*outobj == 0) + new_outobj = PyMem_Realloc(*outobj, requiredsize * sizeof(Py_UCS4)); + if (new_outobj == 0) return -1; + *outobj = new_outobj; *psize = requiredsize; } return 0; -- Repository URL: http://hg.python.org/cpython From rdmurray at bitdance.com Thu May 31 12:58:46 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 31 May 2012 06:58:46 -0400 Subject: [Python-checkins] [Python-Dev] cpython: Make parameterized tests in email less hackish. In-Reply-To: References: Message-ID: <20120531105847.05274250072@webabinitio.net> On Thu, 31 May 2012 13:11:14 +1000, Nick Coghlan wrote: > I'm not clear on why this is a metaclass rather than a simple class decorator. Because I didn't think of it. I don't (yet) think of "class" and "decorator" in the same sentence :) > On Thu, May 31, 2012 at 11:54 AM, r.david.murray > wrote: > > + ?? ??In a _params dictioanry, the keys become part of the name of the generated > > + ?? ??tests. ??In a _params list, the values in the list are converted into a > > + ?? ??string by joining the string values of the elements of the tuple by '_' and > > + ?? ??converting any blanks into '_'s, and this become part of the name. ??The > > + ?? ??full name of a generated test is the portion of the _params name before the > > + ?? ??'_params' portion, plus an '_', plus the name derived as explained above. > > Your description doesn't match your examples or implementation. > Assuming the example and implementation are correct (and they look > more sensible than the currently described approach), I believe that > last sentence should be: > > "The full name of a generated test is a 'test_' prefix, the portion > of the test function name after the '_as_' separator, plus an '_', > plus the name derived as explained above." Oops, yes. Thanks for the catch. --David From python-checkins at python.org Thu May 31 13:39:04 2012 From: python-checkins at python.org (vinay.sajip) Date: Thu, 31 May 2012 13:39:04 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Changed_comment?= =?utf8?q?_on_test_skip=2E?= Message-ID: http://hg.python.org/cpython/rev/8817c295a8ee changeset: 77253:8817c295a8ee branch: 2.7 parent: 77233:3108331c88ec user: Vinay Sajip date: Thu May 31 12:35:13 2012 +0100 summary: Changed comment on test skip. files: Lib/test/test_logging.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -1877,7 +1877,7 @@ class HandlerTest(BaseTest): - @unittest.skipIf(os.name == 'nt', 'Temporarily disabled for Windows') + @unittest.skipIf(os.name == 'nt', 'WatchedFileHandler not appropriate for Windows.') @unittest.skipUnless(threading, 'Threading required for this test.') def test_race(self): # Issue #14632 refers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 13:39:05 2012 From: python-checkins at python.org (vinay.sajip) Date: Thu, 31 May 2012 13:39:05 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Added_test_skip?= =?utf8?q?_under_Windows=2C_as_not_applicable_there=2E?= Message-ID: http://hg.python.org/cpython/rev/a0023e779bce changeset: 77254:a0023e779bce branch: 3.2 parent: 77240:8ec62c9eea34 user: Vinay Sajip date: Thu May 31 12:37:04 2012 +0100 summary: Added test skip under Windows, as not applicable there. files: Lib/test/test_logging.py | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -2331,6 +2331,7 @@ class HandlerTest(BaseTest): + @unittest.skipIf(os.name == 'nt', 'WatchedFileHandler not appropriate for Windows.') @unittest.skipUnless(threading, 'Threading required for this test.') def test_race(self): # Issue #14632 refers. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 13:39:06 2012 From: python-checkins at python.org (vinay.sajip) Date: Thu, 31 May 2012 13:39:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Null_merge_from_3=2E2=2E?= Message-ID: http://hg.python.org/cpython/rev/73d9f608b5c7 changeset: 77255:73d9f608b5c7 parent: 77252:588ea940e5e3 parent: 77254:a0023e779bce user: Vinay Sajip date: Thu May 31 12:38:55 2012 +0100 summary: Null merge from 3.2. files: -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 14:17:20 2012 From: python-checkins at python.org (nick.coghlan) Date: Thu, 31 May 2012 14:17:20 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Additional_ExitStack_exampl?= =?utf8?q?es=2C_and_a_few_other_cleanups_for_the_ExitStack_docs?= Message-ID: http://hg.python.org/cpython/rev/5d91d87b5969 changeset: 77256:5d91d87b5969 user: Nick Coghlan date: Thu May 31 22:17:08 2012 +1000 summary: Additional ExitStack examples, and a few other cleanups for the ExitStack docs files: Doc/library/contextlib.rst | 82 +++++++++++++++++++++++-- 1 files changed, 75 insertions(+), 7 deletions(-) diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -188,15 +188,15 @@ Each instance maintains a stack of registered callbacks that are called in reverse order when the instance is closed (either explicitly or implicitly - at the end of a ``with`` statement). Note that callbacks are *not* invoked - implicitly when the context stack instance is garbage collected. + at the end of a :keyword:`with` statement). Note that callbacks are *not* + invoked implicitly when the context stack instance is garbage collected. This stack model is used so that context managers that acquire their resources in their ``__init__`` method (such as file objects) can be handled correctly. Since registered callbacks are invoked in the reverse order of - registration, this ends up behaving as if multiple nested ``with`` + registration, this ends up behaving as if multiple nested :keyword:`with` statements had been used with the registered set of callbacks. This even extends to exception handling - if an inner callback suppresses or replaces an exception, then outer callbacks will be passed arguments based on that @@ -216,7 +216,7 @@ manager's own :meth:`__enter__` method. These context managers may suppress exceptions just as they normally - would if used directly as part of a ``with`` statement. + would if used directly as part of a :keyword:`with` statement. .. method:: push(exit) @@ -234,7 +234,7 @@ same way context manager :meth:`__exit__` methods can. The passed in object is returned from the function, allowing this - method to be used is a function decorator. + method to be used as a function decorator. .. method:: callback(callback, *args, **kwds) @@ -245,14 +245,14 @@ exceptions (as they are never passed the exception details). The passed in callback is returned from the function, allowing this - method to be used is a function decorator. + method to be used as a function decorator. .. method:: pop_all() Transfers the callback stack to a fresh :class:`ExitStack` instance and returns it. No callbacks are invoked by this operation - instead, they will now be invoked when the new stack is closed (either - explicitly or implicitly). + explicitly or implicitly at the end of a :keyword:`with` statement). For example, a group of files can be opened as an "all or nothing" operation as follows:: @@ -280,6 +280,74 @@ the tools provided by :mod:`contextlib`. +Supporting a variable number of context managers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The primary use case for :class:`ExitStack` is the one given in the class +documentation: supporting a variable number of context managers and other +cleanup operations in a single :keyword:`with` statement. The variability +may come from the number of context managers needed being driven by user +input (such as opening a user specified collection of files), or from +some of the context managers being optional:: + + with ExitStack() as stack: + for resource in resources: + stack.enter_context(resource) + if need_special resource: + special = acquire_special_resource() + stack.callback(release_special_resource, special) + # Perform operations that use the acquired resources + +As shown, :class:`ExitStack` also makes it quite easy to use :keyword:`with` +statements to manage arbitrary resources that don't natively support the +context management protocol. + + +Simplifying support for single optional context managers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In the specific case of a single optional context manager, :class:`ExitStack` +instances can be used as a "do nothing" context manager, allowing a context +managers to easily be omitted without affecting the overall structure of +the source code:: + + def debug_trace(details): + if __debug__: + return TraceContext(details) + # Don't do anything special with the context in release mode + return ExitStack() + + with debug_trace(): + # Suite is traced in debug mode, but runs normally otherwise + + +Catching exceptions from ``__enter__`` methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is occasionally desirable to catch exceptions from an ``__enter__`` +method implementation, *without* inadvertently catching exceptions from +the :keyword:`with` statement body or the context manager's ``__exit__`` +method. By using :class:`ExitStack` the steps in the context management +protocol can be separated slightly in order to allow this:: + + stack = ExitStack() + try: + x = stack.enter_context(cm) + except Exception: + # handle __enter__ exception + else: + with stack: + # Handle normal case + +Actually needing to do this is likely to indicate that the underlying API +should be providing a direct resource management interface for use with +:keyword:`try`/:keyword:`except`/:keyword:`finally` statements, but not +all APIs are well designed in that regard. When a context manager is the +only resource management API provided, then :class:`ExitStack` can make it +easier to handle various situations that can't be handled directly in a +:keyword:`with` statement. + + Cleaning up in an ``__enter__`` implementation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 14:35:10 2012 From: python-checkins at python.org (nick.coghlan) Date: Thu, 31 May 2012 14:35:10 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Typo_fix?= Message-ID: http://hg.python.org/cpython/rev/e2998e66fab6 changeset: 77257:e2998e66fab6 user: Nick Coghlan date: Thu May 31 22:34:59 2012 +1000 summary: Typo fix files: Doc/library/contextlib.rst | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -308,7 +308,7 @@ In the specific case of a single optional context manager, :class:`ExitStack` instances can be used as a "do nothing" context manager, allowing a context -managers to easily be omitted without affecting the overall structure of +manager to easily be omitted without affecting the overall structure of the source code:: def debug_trace(details): -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 15:22:17 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 31 May 2012 15:22:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Do_not_clobber_existing_fla?= =?utf8?b?Z3Mu?= Message-ID: http://hg.python.org/cpython/rev/05097240ffb0 changeset: 77258:05097240ffb0 user: Stefan Krah date: Thu May 31 15:09:27 2012 +0200 summary: Do not clobber existing flags. files: Modules/_decimal/libmpdec/mpdecimal.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3904,7 +3904,7 @@ if (_mpd_cmp(&aa, &lim) <= 0) { _settriple(result, 0, 1, 0); _mpd_zeropad(result, ctx, status); - *status = MPD_Rounded|MPD_Inexact; + *status |= MPD_Rounded|MPD_Inexact; return 1; } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 15:49:44 2012 From: python-checkins at python.org (nick.coghlan) Date: Thu, 31 May 2012 15:49:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314963=3A_Added_tes?= =?utf8?q?t_cases_for_contextlib=2EExitStack_exception_handling?= Message-ID: http://hg.python.org/cpython/rev/fc73e6ea9e73 changeset: 77259:fc73e6ea9e73 user: Nick Coghlan date: Thu May 31 23:49:26 2012 +1000 summary: Issue #14963: Added test cases for contextlib.ExitStack exception handling behaviour (Initial patch by Alon Horev) files: Lib/test/test_contextlib.py | 89 +++++++++++++++++++++++++ Misc/ACKS | 1 + Misc/NEWS | 11 +++ 3 files changed, 101 insertions(+), 0 deletions(-) diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -483,6 +483,95 @@ new_stack.close() self.assertEqual(result, [1, 2, 3]) + def test_exit_raise(self): + with self.assertRaises(ZeroDivisionError): + with ExitStack() as stack: + stack.push(lambda *exc: False) + 1/0 + + def test_exit_suppress(self): + with ExitStack() as stack: + stack.push(lambda *exc: True) + 1/0 + + def test_exit_exception_chaining_reference(self): + # Sanity check to make sure that ExitStack chaining matches + # actual nested with statements + class RaiseExc: + def __init__(self, exc): + self.exc = exc + def __enter__(self): + return self + def __exit__(self, *exc_details): + raise self.exc + + class SuppressExc: + def __enter__(self): + return self + def __exit__(self, *exc_details): + type(self).saved_details = exc_details + return True + + try: + with RaiseExc(IndexError): + with RaiseExc(KeyError): + with RaiseExc(AttributeError): + with SuppressExc(): + with RaiseExc(ValueError): + 1 / 0 + except IndexError as exc: + self.assertIsInstance(exc.__context__, KeyError) + self.assertIsInstance(exc.__context__.__context__, AttributeError) + # Inner exceptions were suppressed + self.assertIsNone(exc.__context__.__context__.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + # Check the inner exceptions + inner_exc = SuppressExc.saved_details[1] + self.assertIsInstance(inner_exc, ValueError) + self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + + def test_exit_exception_chaining(self): + # Ensure exception chaining matches the reference behaviour + def raise_exc(exc): + raise exc + + saved_details = None + def suppress_exc(*exc_details): + nonlocal saved_details + saved_details = exc_details + return True + + try: + with ExitStack() as stack: + stack.callback(raise_exc, IndexError) + stack.callback(raise_exc, KeyError) + stack.callback(raise_exc, AttributeError) + stack.push(suppress_exc) + stack.callback(raise_exc, ValueError) + 1 / 0 + except IndexError as exc: + self.assertIsInstance(exc.__context__, KeyError) + self.assertIsInstance(exc.__context__.__context__, AttributeError) + # Inner exceptions were suppressed, but the with statement + # cleanup code adds the one from the body back in as the + # context of the exception raised by the outer callbacks + # See http://bugs.python.org/issue14969 + suite_exc = exc.__context__.__context__.__context__ + self.assertIsInstance(suite_exc, ZeroDivisionError) + else: + self.fail("Expected IndexError, but no exception was raised") + # Check the inner exceptions + inner_exc = saved_details[1] + self.assertIsInstance(inner_exc, ValueError) + self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + + def test_exit_exception_chaining_suppress(self): + with ExitStack() as stack: + stack.push(lambda *exc: True) + stack.push(lambda *exc: 1/0) + stack.push(lambda *exc: {}[1]) + def test_instance_bypass(self): class Example(object): pass cm = Example() diff --git a/Misc/ACKS b/Misc/ACKS --- a/Misc/ACKS +++ b/Misc/ACKS @@ -460,6 +460,7 @@ Brian Hooper Randall Hopper Nadav Horesh +Alon Horev Jan Hosang Ken Howard Brad Howes diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -2,6 +2,17 @@ Python News +++++++++++ +What's New in Python 3.3.0 Beta 1? +=================================== + +*Release date: TBD* + +Tests +----- + +- Issue #14963 (partial): Add test cases for exception handling behaviour + in contextlib.ContextStack (Initial patch by Alon Horev) + What's New in Python 3.3.0 Alpha 4? =================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 16:00:57 2012 From: python-checkins at python.org (nick.coghlan) Date: Thu, 31 May 2012 16:00:57 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2314963=3A_Use_an_it?= =?utf8?q?erative_algorithm_in_contextlib=2EExitStack=2E=5F=5Fexit=5F=5F?= Message-ID: http://hg.python.org/cpython/rev/c0c7618762e5 changeset: 77260:c0c7618762e5 user: Nick Coghlan date: Fri Jun 01 00:00:38 2012 +1000 summary: Close #14963: Use an iterative algorithm in contextlib.ExitStack.__exit__ (Patch by Alon Horev) files: Lib/contextlib.py | 41 +++++++++--------------- Lib/test/test_contextlib.py | 6 +++ Misc/NEWS | 8 ++++- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/Lib/contextlib.py b/Lib/contextlib.py --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -225,32 +225,21 @@ return self def __exit__(self, *exc_details): - if not self._exit_callbacks: - return - # This looks complicated, but it is really just - # setting up a chain of try-expect statements to ensure - # that outer callbacks still get invoked even if an - # inner one throws an exception - def _invoke_next_callback(exc_details): - # Callbacks are removed from the list in FIFO order - # but the recursion means they're invoked in LIFO order - cb = self._exit_callbacks.popleft() - if not self._exit_callbacks: - # Innermost callback is invoked directly - return cb(*exc_details) - # More callbacks left, so descend another level in the stack + # Callbacks are invoked in LIFO order to match the behaviour of + # nested context managers + suppressed_exc = False + while self._exit_callbacks: + cb = self._exit_callbacks.pop() try: - suppress_exc = _invoke_next_callback(exc_details) + if cb(*exc_details): + suppressed_exc = True + exc_details = (None, None, None) except: - suppress_exc = cb(*sys.exc_info()) - # Check if this cb suppressed the inner exception - if not suppress_exc: + new_exc_details = sys.exc_info() + if exc_details != (None, None, None): + # simulate the stack of exceptions by setting the context + new_exc_details[1].__context__ = exc_details[1] + if not self._exit_callbacks: raise - else: - # Check if inner cb suppressed the original exception - if suppress_exc: - exc_details = (None, None, None) - suppress_exc = cb(*exc_details) or suppress_exc - return suppress_exc - # Kick off the recursive chain - return _invoke_next_callback(exc_details) + exc_details = new_exc_details + return suppressed_exc diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -572,6 +572,12 @@ stack.push(lambda *exc: 1/0) stack.push(lambda *exc: {}[1]) + def test_excessive_nesting(self): + # The original implementation would die with RecursionError here + with ExitStack() as stack: + for i in range(10000): + stack.callback(int) + def test_instance_bypass(self): class Example(object): pass cm = Example() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -7,11 +7,17 @@ *Release date: TBD* +Library +------- + +- Issue #14963: Convert contextlib.ExitStack.__exit__ to use an iterative + algorithm (Patch by Alon Horev) + Tests ----- - Issue #14963 (partial): Add test cases for exception handling behaviour - in contextlib.ContextStack (Initial patch by Alon Horev) + in contextlib.ExitStack (Initial patch by Alon Horev) What's New in Python 3.3.0 Alpha 4? =================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 16:05:03 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 31 May 2012 16:05:03 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Pad_the_result_with_zeros_j?= =?utf8?q?ust_before_the_final_rounding=2E?= Message-ID: http://hg.python.org/cpython/rev/dff6dd248059 changeset: 77261:dff6dd248059 parent: 77259:fc73e6ea9e73 user: Stefan Krah date: Thu May 31 16:00:21 2012 +0200 summary: Pad the result with zeros just before the final rounding. files: Modules/_decimal/libmpdec/mpdecimal.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3903,7 +3903,6 @@ /* abs(a) <= 9 * 10**(-prec-1) */ if (_mpd_cmp(&aa, &lim) <= 0) { _settriple(result, 0, 1, 0); - _mpd_zeropad(result, ctx, status); *status |= MPD_Rounded|MPD_Inexact; return 1; } @@ -4074,8 +4073,6 @@ } #endif - _mpd_zeropad(result, ctx, status); - mpd_del(&tmp); mpd_del(&sum); *status |= (workctx.status&MPD_Errors); @@ -4148,6 +4145,7 @@ if (mpd_isspecial(result) || mpd_iszerocoeff(result) || mpd_qcmp(&t1, &t2, status) == 0) { workctx.clamp = ctx->clamp; + _mpd_zeropad(result, ctx, status); mpd_check_underflow(result, &workctx, status); mpd_qfinalize(result, &workctx, status); break; -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 16:05:06 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 31 May 2012 16:05:06 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?= =?utf8?b?KTogTWVyZ2Uu?= Message-ID: http://hg.python.org/cpython/rev/c623f681ad1a changeset: 77262:c623f681ad1a parent: 77261:dff6dd248059 parent: 77260:c0c7618762e5 user: Stefan Krah date: Thu May 31 16:03:49 2012 +0200 summary: Merge. files: Lib/contextlib.py | 41 +++++++++--------------- Lib/test/test_contextlib.py | 6 +++ Misc/NEWS | 8 ++++- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/Lib/contextlib.py b/Lib/contextlib.py --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -225,32 +225,21 @@ return self def __exit__(self, *exc_details): - if not self._exit_callbacks: - return - # This looks complicated, but it is really just - # setting up a chain of try-expect statements to ensure - # that outer callbacks still get invoked even if an - # inner one throws an exception - def _invoke_next_callback(exc_details): - # Callbacks are removed from the list in FIFO order - # but the recursion means they're invoked in LIFO order - cb = self._exit_callbacks.popleft() - if not self._exit_callbacks: - # Innermost callback is invoked directly - return cb(*exc_details) - # More callbacks left, so descend another level in the stack + # Callbacks are invoked in LIFO order to match the behaviour of + # nested context managers + suppressed_exc = False + while self._exit_callbacks: + cb = self._exit_callbacks.pop() try: - suppress_exc = _invoke_next_callback(exc_details) + if cb(*exc_details): + suppressed_exc = True + exc_details = (None, None, None) except: - suppress_exc = cb(*sys.exc_info()) - # Check if this cb suppressed the inner exception - if not suppress_exc: + new_exc_details = sys.exc_info() + if exc_details != (None, None, None): + # simulate the stack of exceptions by setting the context + new_exc_details[1].__context__ = exc_details[1] + if not self._exit_callbacks: raise - else: - # Check if inner cb suppressed the original exception - if suppress_exc: - exc_details = (None, None, None) - suppress_exc = cb(*exc_details) or suppress_exc - return suppress_exc - # Kick off the recursive chain - return _invoke_next_callback(exc_details) + exc_details = new_exc_details + return suppressed_exc diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -572,6 +572,12 @@ stack.push(lambda *exc: 1/0) stack.push(lambda *exc: {}[1]) + def test_excessive_nesting(self): + # The original implementation would die with RecursionError here + with ExitStack() as stack: + for i in range(10000): + stack.callback(int) + def test_instance_bypass(self): class Example(object): pass cm = Example() diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -7,11 +7,17 @@ *Release date: TBD* +Library +------- + +- Issue #14963: Convert contextlib.ExitStack.__exit__ to use an iterative + algorithm (Patch by Alon Horev) + Tests ----- - Issue #14963 (partial): Add test cases for exception handling behaviour - in contextlib.ContextStack (Initial patch by Alon Horev) + in contextlib.ExitStack (Initial patch by Alon Horev) What's New in Python 3.3.0 Alpha 4? =================================== -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 16:35:09 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 31 May 2012 16:35:09 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Improve_comments=2E?= Message-ID: http://hg.python.org/cpython/rev/8c7bfd7ab995 changeset: 77263:8c7bfd7ab995 user: Stefan Krah date: Thu May 31 16:21:34 2012 +0200 summary: Improve comments. files: Modules/_decimal/libmpdec/mpdecimal.c | 18 +++++++++++--- 1 files changed, 14 insertions(+), 4 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -3953,8 +3953,18 @@ } /* - * Internal function, specials have been dealt with. The result has a - * relative error of less than 0.5 * 10**(-ctx->prec). + * Internal function, specials have been dealt with. Apart from Overflow + * and Underflow, two cases must be considered for the error of the result: + * + * 1) abs(a) <= 9 * 10**(-prec-1) ==> result == 1 + * + * Absolute error: abs(1 - e**x) < 10**(-prec) + * ------------------------------------------- + * + * 2) abs(a) > 9 * 10**(-prec-1) + * + * Relative error: abs(result - e**x) < 0.5 * 10**(-prec) * e**x + * ------------------------------------------------------------- * * The algorithm is from Hull&Abrham, Variable Precision Exponential Function, * ACM Transactions on Mathematical Software, Vol. 12, No. 2, June 1986. @@ -3998,9 +4008,9 @@ * * MAX-EMAX+1 < log10(e^(0.1*10*t)) <= log10(e^(r*10^t)) < adjexp(e^(r*10^t))+1 * - * (2) -1 < r <= -0.1, so e^r <= e^-0.1. It t > MAX_T, underflow occurs: + * (2) -1 < r <= -0.1, so e^r <= e^-0.1. If t > MAX_T, underflow occurs: * - * adjexp(e^(r*10^t)) <= log10(e^(r*10^t)) <= log10(e^(-0.1*10^t) < MIN-ETINY + * adjexp(e^(r*10^t)) <= log10(e^(r*10^t)) <= log10(e^(-0.1*10^t)) < MIN-ETINY */ #if defined(CONFIG_64) #define MPD_EXP_MAX_T 19 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 17:11:17 2012 From: python-checkins at python.org (kristjan.jonsson) Date: Thu, 31 May 2012 17:11:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2314952=3A_Fix_incor?= =?utf8?q?rect_output_dll_names_for_win64/debug_builds=2C_causing?= Message-ID: http://hg.python.org/cpython/rev/c7b16e2be71a changeset: 77264:c7b16e2be71a user: Kristjan Valur Jonsson date: Thu May 31 15:09:21 2012 +0000 summary: Issue #14952: Fix incorrect output dll names for win64/debug builds, causing the dll importer on windows to fail. files: PCbuild/pythoncore.vcxproj | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -203,7 +203,6 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) - $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) 0x1e000000 @@ -257,7 +256,6 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) - $(OutDir)$(PyDllName)_d.dll libc;%(IgnoreSpecificDefaultLibraries) 0x1e000000 @@ -306,7 +304,6 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) - $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) 0x1e000000 MachineX64 @@ -356,7 +353,6 @@ $(IntDir)getbuildinfo.o;%(AdditionalDependencies) - $(OutDir)$(PyDllName).dll libc;%(IgnoreSpecificDefaultLibraries) 0x1e000000 MachineX64 -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 18:58:48 2012 From: python-checkins at python.org (ned.deily) Date: Thu, 31 May 2012 18:58:48 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE0OTYy?= =?utf8?q?=3A_Update_text_coloring_in_IDLE_shell_window_after_changing?= Message-ID: http://hg.python.org/cpython/rev/1a4e99460438 changeset: 77265:1a4e99460438 branch: 2.7 parent: 77253:8817c295a8ee user: Ned Deily date: Thu May 31 09:17:29 2012 -0700 summary: Issue #14962: Update text coloring in IDLE shell window after changing options. Patch by Roger Serwy. files: Lib/idlelib/PyShell.py | 5 +++++ Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -313,6 +313,11 @@ "console": idleConf.GetHighlight(theme, "console"), }) + def removecolors(self): + # Don't remove shell color tags before "iomark" + for tag in self.tagdefs: + self.tag_remove(tag, "iomark", "end") + class ModifiedUndoDelegator(UndoDelegator): "Extend base class: forbid insert/delete before the I/O mark" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -67,6 +67,9 @@ Library ------- +- Issue #14962: Update text coloring in IDLE shell window after changing + options. Patch by Roger Serwy. + - Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. - Issue12510: Attempting to get invalid tooltip no longer closes Idle. -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 18:58:49 2012 From: python-checkins at python.org (ned.deily) Date: Thu, 31 May 2012 18:58:49 +0200 Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzE0OTYy?= =?utf8?q?=3A_Update_text_coloring_in_IDLE_shell_window_after_changing?= Message-ID: http://hg.python.org/cpython/rev/9d0c3a835bfe changeset: 77266:9d0c3a835bfe branch: 3.2 parent: 77254:a0023e779bce user: Ned Deily date: Thu May 31 09:17:29 2012 -0700 summary: Issue #14962: Update text coloring in IDLE shell window after changing options. Patch by Roger Serwy. files: Lib/idlelib/PyShell.py | 5 +++++ Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -309,6 +309,11 @@ "console": idleConf.GetHighlight(theme, "console"), }) + def removecolors(self): + # Don't remove shell color tags before "iomark" + for tag in self.tagdefs: + self.tag_remove(tag, "iomark", "end") + class ModifiedUndoDelegator(UndoDelegator): "Extend base class: forbid insert/delete before the I/O mark" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -70,6 +70,9 @@ Library ------- +- Issue #14962: Update text coloring in IDLE shell window after changing + options. Patch by Roger Serwy. + - Issue #10997: Prevent a duplicate entry in IDLE's "Recent Files" menu. - Issue #14443: Tell rpmbuild to use the correct version of Python in -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 18:58:50 2012 From: python-checkins at python.org (ned.deily) Date: Thu, 31 May 2012 18:58:50 +0200 Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?= =?utf8?q?=3A_Issue_=2314962=3A_merge?= Message-ID: http://hg.python.org/cpython/rev/86f62adb09cf changeset: 77267:86f62adb09cf parent: 77264:c7b16e2be71a parent: 77266:9d0c3a835bfe user: Ned Deily date: Thu May 31 09:58:08 2012 -0700 summary: Issue #14962: merge files: Lib/idlelib/PyShell.py | 5 +++++ Misc/NEWS | 3 +++ 2 files changed, 8 insertions(+), 0 deletions(-) diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py --- a/Lib/idlelib/PyShell.py +++ b/Lib/idlelib/PyShell.py @@ -309,6 +309,11 @@ "console": idleConf.GetHighlight(theme, "console"), }) + def removecolors(self): + # Don't remove shell color tags before "iomark" + for tag in self.tagdefs: + self.tag_remove(tag, "iomark", "end") + class ModifiedUndoDelegator(UndoDelegator): "Extend base class: forbid insert/delete before the I/O mark" diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -10,6 +10,9 @@ Library ------- +- Issue #14962: Update text coloring in IDLE shell window after changing + options. Patch by Roger Serwy. + - Issue #14963: Convert contextlib.ExitStack.__exit__ to use an iterative algorithm (Patch by Alon Horev) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 20:03:45 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 31 May 2012 20:03:45 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Improve_Underflow_handling_?= =?utf8?q?in_the_correct-rounding_loop=2E_The_case_for?= Message-ID: http://hg.python.org/cpython/rev/fabc593a5822 changeset: 77268:fabc593a5822 user: Stefan Krah date: Thu May 31 20:01:05 2012 +0200 summary: Improve Underflow handling in the correct-rounding loop. The case for Underflow to zero hasn't changed: _mpd_qexp() internally uses MIN_EMIN, so the result would also underflow to zero for all emin > MIN_EMIN. In case digits are left, the informal argument is as follows: Underflow can occur only once in the last multiplication of the power stage (in the Horner stage Underflow provably cannot occur, and if Underflow occurred twice in the power stage, the result would underflow to zero on the second occasion). Since there is no double rounding during Underflow, the effective work precision is now 1 <= result->digits < prec. It can be shown by a somewhat tedious argument that abs(result - e**x) < ulp(result, result->digits). Therefore the correct rounding loop now uses ulp(result, result->digits) to generate the bounds for e**x in case of Underflow. files: Modules/_decimal/libmpdec/mpdecimal.c | 21 ++++++++++++-- 1 files changed, 17 insertions(+), 4 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -4122,6 +4122,8 @@ MPD_NEW_STATIC(ulp, 0,0,0,0); MPD_NEW_STATIC(aa, 0,0,0,0); mpd_ssize_t prec; + mpd_ssize_t ulpexp; + uint32_t workstatus; if (result == a) { if (!mpd_qcopy(&aa, a, status)) { @@ -4135,12 +4137,20 @@ prec = ctx->prec + 3; while (1) { workctx.prec = prec; - _mpd_qexp(result, a, &workctx, status); - _ssettriple(&ulp, MPD_POS, 1, - result->exp + result->digits-workctx.prec); + workstatus = 0; + + _mpd_qexp(result, a, &workctx, &workstatus); + *status |= workstatus; + + ulpexp = result->exp + result->digits - workctx.prec; + if (workstatus & MPD_Underflow) { + /* The effective work precision is result->digits. */ + ulpexp = result->exp; + } + _ssettriple(&ulp, MPD_POS, 1, ulpexp); /* - * At this point: + * At this point [1]: * 1) abs(result - e**x) < 0.5 * 10**(-prec) * e**x * 2) result - ulp < e**x < result + ulp * 3) result - ulp < result < result + ulp @@ -4148,6 +4158,9 @@ * If round(result-ulp)==round(result+ulp), then * round(result)==round(e**x). Therefore the result * is correctly rounded. + * + * [1] If abs(a) <= 9 * 10**(-prec-1), use the absolute + * error for a similar argument. */ workctx.prec = ctx->prec; mpd_qadd(&t1, result, &ulp, &workctx, &workctx.status); -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 20:50:44 2012 From: python-checkins at python.org (stefan.krah) Date: Thu, 31 May 2012 20:50:44 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_workctx_instead_of_ctx_?= =?utf8?q?for_cosmetic_reasons=2E_Also_zero-pad_the_result?= Message-ID: http://hg.python.org/cpython/rev/fb79cf1d21c9 changeset: 77269:fb79cf1d21c9 user: Stefan Krah date: Thu May 31 20:49:24 2012 +0200 summary: Use workctx instead of ctx for cosmetic reasons. Also zero-pad the result in the simple path (not correctly rounded but faster). files: Modules/_decimal/libmpdec/mpdecimal.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/Modules/_decimal/libmpdec/mpdecimal.c b/Modules/_decimal/libmpdec/mpdecimal.c --- a/Modules/_decimal/libmpdec/mpdecimal.c +++ b/Modules/_decimal/libmpdec/mpdecimal.c @@ -4168,7 +4168,7 @@ if (mpd_isspecial(result) || mpd_iszerocoeff(result) || mpd_qcmp(&t1, &t2, status) == 0) { workctx.clamp = ctx->clamp; - _mpd_zeropad(result, ctx, status); + _mpd_zeropad(result, &workctx, status); mpd_check_underflow(result, &workctx, status); mpd_qfinalize(result, &workctx, status); break; @@ -4182,6 +4182,7 @@ } else { _mpd_qexp(result, a, &workctx, status); + _mpd_zeropad(result, &workctx, status); mpd_check_underflow(result, &workctx, status); mpd_qfinalize(result, &workctx, status); } -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 21:55:17 2012 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 31 May 2012 21:55:17 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_to_Tcl/Tk_8=2E5=2E11?= =?utf8?q?=2E?= Message-ID: http://hg.python.org/cpython/rev/99786479dafb changeset: 77270:99786479dafb user: Martin v. L?wis date: Thu May 31 21:53:36 2012 +0200 summary: Update to Tcl/Tk 8.5.11. files: PCbuild/build_tkinter.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/PCbuild/build_tkinter.py b/PCbuild/build_tkinter.py --- a/PCbuild/build_tkinter.py +++ b/PCbuild/build_tkinter.py @@ -11,8 +11,8 @@ here = os.path.abspath(os.path.dirname(__file__)) par = os.path.pardir -TCL = "tcl8.5.9" -TK = "tk8.5.9" +TCL = "tcl8.5.11" +TK = "tk8.5.11" TIX = "tix-8.4.3.x" ROOT = os.path.abspath(os.path.join(here, par, par)) @@ -32,7 +32,7 @@ def build(platform, clean): if platform == "Win32": dest = os.path.join(ROOT, "tcltk") - machine = "X86" + machine = "IX86" elif platform == "AMD64": dest = os.path.join(ROOT, "tcltk64") machine = "AMD64" -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 21:59:28 2012 From: python-checkins at python.org (martin.v.loewis) Date: Thu, 31 May 2012 21:59:28 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_VS_2010_has_the_AMD64_redis?= =?utf8?q?t_files_in_VC=5Credist=5Cx64=2E?= Message-ID: http://hg.python.org/cpython/rev/7528b178692e changeset: 77271:7528b178692e user: Martin v. L?wis date: Thu May 31 21:58:21 2012 +0200 summary: VS 2010 has the AMD64 redist files in VC\redist\x64. files: Tools/msi/msi.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py --- a/Tools/msi/msi.py +++ b/Tools/msi/msi.py @@ -875,7 +875,7 @@ def extract_msvcr100(): # Find the redistributable files if msilib.Win64: - arch = "amd64" + arch = "x64" else: arch = "x86" dir = os.path.join(os.environ['VS100COMNTOOLS'], r"..\..\VC\redist\%s\Microsoft.VC100.CRT" % arch) -- Repository URL: http://hg.python.org/cpython From python-checkins at python.org Thu May 31 22:41:47 2012 From: python-checkins at python.org (georg.brandl) Date: Thu, 31 May 2012 22:41:47 +0200 Subject: [Python-checkins] =?utf8?q?cpython=3A_Post-release_update=2E?= Message-ID: http://hg.python.org/cpython/rev/56cc9de5c024 changeset: 77272:56cc9de5c024 user: Georg Brandl date: Thu May 31 22:41:51 2012 +0200 summary: Post-release update. files: Include/patchlevel.h | 2 +- Misc/NEWS | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/Include/patchlevel.h b/Include/patchlevel.h --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -23,7 +23,7 @@ #define PY_RELEASE_SERIAL 4 /* Version as a string */ -#define PY_VERSION "3.3.0a4" +#define PY_VERSION "3.3.0a4+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Misc/NEWS b/Misc/NEWS --- a/Misc/NEWS +++ b/Misc/NEWS @@ -3,9 +3,9 @@ +++++++++++ What's New in Python 3.3.0 Beta 1? -=================================== - -*Release date: TBD* +================================== + +*Release date: XX-XXX-2012* Library ------- @@ -22,6 +22,7 @@ - Issue #14963 (partial): Add test cases for exception handling behaviour in contextlib.ExitStack (Initial patch by Alon Horev) + What's New in Python 3.3.0 Alpha 4? =================================== -- Repository URL: http://hg.python.org/cpython From amauryfa at gmail.com Sat May 12 15:19:27 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sat, 12 May 2012 13:19:27 -0000 Subject: [Python-checkins] cpython: Remove_uninitialized_compiler_warning Message-ID: Hi, The previous code looked wrong, but the new "break" will change behaviour. >From a comment above, the LZMA1 filter needs a "dict_size" as well. The "fallthrough" was probably intentional. > diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c > --- a/Modules/_lzmamodule.c > +++ b/Modules/_lzmamodule.c > @@ -412,7 +412,11 @@ > ADD_FIELD(options, lc); > ADD_FIELD(options, lp); > ADD_FIELD(options, pb); > - case LZMA_FILTER_LZMA2: > + ADD_FIELD(options, dict_size); > + break; > + } > + case LZMA_FILTER_LZMA2: { > + lzma_options_lzma *options = f->options; > ADD_FIELD(options, dict_size); > break; > } -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: