From python-checkins at python.org  Tue Nov  1 00:07:13 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 00:07:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_regression_due_to_chang?=
 =?utf8?q?eset_2096158376e5_=28issue_=2313305=29=2E?=
Message-ID: <E1RL0wj-0001Mu-1H@dinsdale.python.org>

http://hg.python.org/cpython/rev/3f025427f02b
changeset:   73265:3f025427f02b
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 00:06:58 2011 +0100
summary:
  Fix regression due to changeset 2096158376e5 (issue #13305).

files:
  Lib/xmlrpc/client.py |  27 +++++++++++++++++++++------
  1 files changed, 21 insertions(+), 6 deletions(-)


diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py
--- a/Lib/xmlrpc/client.py
+++ b/Lib/xmlrpc/client.py
@@ -250,18 +250,33 @@
 # Wrapper for XML-RPC DateTime values.  This converts a time value to
 # the format used by XML-RPC.
 # <p>
-# The value can be given as a string in the format
-# "yyyymmddThh:mm:ss", as a 9-item time tuple (as returned by
+# The value can be given as a datetime object, as a string in the
+# format "yyyymmddThh:mm:ss", as a 9-item time tuple (as returned by
 # time.localtime()), or an integer value (as returned by time.time()).
 # The wrapper uses time.localtime() to convert an integer to a time
 # tuple.
 #
-# @param value The time, given as an ISO 8601 string, a time
-#              tuple, or a integer time value.
+# @param value The time, given as a datetime object, an ISO 8601 string,
+#              a time tuple, or an integer time value.
+
+
+# Issue #13305: different format codes across platforms
+_day0 = datetime(1, 1, 1)
+if _day0.strftime('%Y') == '0001':      # Mac OS X
+    def _iso8601_format(value):
+        return value.strftime("%Y%m%dT%H:%M:%S")
+elif _day0.strftime('%4Y') == '0001':   # Linux
+    def _iso8601_format(value):
+        return value.strftime("%4Y%m%dT%H:%M:%S")
+else:
+    def _iso8601_format(value):
+        return value.strftime("%Y%m%dT%H:%M:%S").zfill(17)
+del _day0
+
 
 def _strftime(value):
     if isinstance(value, datetime):
-        return value.strftime("%Y%m%dT%H:%M:%S")
+        return _iso8601_format(value)
 
     if not isinstance(value, (tuple, time.struct_time)):
         if value == 0:
@@ -288,7 +303,7 @@
             o = other.value
         elif isinstance(other, datetime):
             s = self.value
-            o = other.strftime("%Y%m%dT%H:%M:%S")
+            o = _iso8601_format(other)
         elif isinstance(other, str):
             s = self.value
             o = other

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 00:23:49 2011
From: python-checkins at python.org (ned.deily)
Date: Tue, 01 Nov 2011 00:23:49 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMzA0?=
 =?utf8?q?=3A_Skip_test_case_if_user_site-packages_disabled_=28-s_or?=
Message-ID: <E1RL1Cn-0002Th-Lh@dinsdale.python.org>

http://hg.python.org/cpython/rev/1689b9cf6b1c
changeset:   73266:1689b9cf6b1c
branch:      2.7
parent:      73263:848210b179ab
user:        Ned Deily <nad at acm.org>
date:        Mon Oct 31 16:14:52 2011 -0700
summary:
  Issue #13304: Skip test case if user site-packages disabled (-s or
PYTHONNOUSERSITE).  (Patch by Carl Meyer)

files:
  Lib/test/test_site.py |  4 +++-
  Misc/ACKS             |  1 +
  Misc/NEWS             |  3 +++
  3 files changed, 7 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -24,7 +24,7 @@
 else:
     raise unittest.SkipTest("importation of site.py suppressed")
 
-if not os.path.isdir(site.USER_SITE):
+if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE):
     # need to add user site directory for tests
     os.makedirs(site.USER_SITE)
     site.addsitedir(site.USER_SITE)
@@ -161,6 +161,8 @@
         finally:
             pth_file.cleanup()
 
+    @unittest.skipUnless(site.ENABLE_USER_SITE, "requires access to PEP 370 "
+                          "user-site (site.ENABLE_USER_SITE)")
     def test_s_option(self):
         usersite = site.USER_SITE
         self.assertIn(usersite, sys.path)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -556,6 +556,7 @@
 Ezio Melotti
 Brian Merrell
 Luke Mewburn
+Carl Meyer
 Mike Meyer
 Steven Miale
 Trent Mick
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -328,6 +328,9 @@
 Tests
 -----
 
+- Issue #13304: Skip test case if user site-packages disabled (-s or
+  PYTHONNOUSERSITE).  (Patch by Carl Meyer)
+
 - Issue #13218: Fix test_ssl failures on Debian/Ubuntu.
 
 - Issue #12821: Fix test_fcntl failures on OpenBSD 5.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 00:23:50 2011
From: python-checkins at python.org (ned.deily)
Date: Tue, 01 Nov 2011 00:23:50 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzA0?=
 =?utf8?q?=3A_Skip_test_case_if_user_site-packages_disabled_=28-s_or?=
Message-ID: <E1RL1Co-0002Tl-FI@dinsdale.python.org>

http://hg.python.org/cpython/rev/c497011a4769
changeset:   73267:c497011a4769
branch:      3.2
parent:      73256:2ca415cbf2ac
user:        Ned Deily <nad at acm.org>
date:        Mon Oct 31 16:16:35 2011 -0700
summary:
  Issue #13304: Skip test case if user site-packages disabled (-s or
PYTHONNOUSERSITE).  (Patch by Carl Meyer)

files:
  Lib/test/test_site.py |  4 +++-
  Misc/ACKS             |  1 +
  Misc/NEWS             |  3 +++
  3 files changed, 7 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -24,7 +24,7 @@
 else:
     raise unittest.SkipTest("importation of site.py suppressed")
 
-if not os.path.isdir(site.USER_SITE):
+if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE):
     # need to add user site directory for tests
     os.makedirs(site.USER_SITE)
     site.addsitedir(site.USER_SITE)
@@ -157,6 +157,8 @@
         finally:
             pth_file.cleanup()
 
+    @unittest.skipUnless(site.ENABLE_USER_SITE, "requires access to PEP 370 "
+                          "user-site (site.ENABLE_USER_SITE)")
     def test_s_option(self):
         usersite = site.USER_SITE
         self.assertIn(usersite, sys.path)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -601,6 +601,7 @@
 Ezio Melotti
 Brian Merrell
 Luke Mewburn
+Carl Meyer
 Mike Meyer
 Steven Miale
 Trent Mick
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -187,6 +187,9 @@
 Tests
 -----
 
+- Issue #13304: Skip test case if user site-packages disabled (-s or
+  PYTHONNOUSERSITE).  (Patch by Carl Meyer)
+
 - Issue #13218: Fix test_ssl failures on Debian/Ubuntu.
 
 - Issue #12821: Fix test_fcntl failures on OpenBSD 5.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 00:23:51 2011
From: python-checkins at python.org (ned.deily)
Date: Tue, 01 Nov 2011 00:23:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313304=3A_Skip_test_case_if_user_site-packages_disab?=
 =?utf8?q?led_=28-s_or?=
Message-ID: <E1RL1Cp-0002Tp-3G@dinsdale.python.org>

http://hg.python.org/cpython/rev/c343c095d08b
changeset:   73268:c343c095d08b
parent:      73265:3f025427f02b
parent:      73267:c497011a4769
user:        Ned Deily <nad at acm.org>
date:        Mon Oct 31 16:22:53 2011 -0700
summary:
  Issue #13304: Skip test case if user site-packages disabled (-s or
PYTHONNOUSERSITE).  (Patch by Carl Meyer)

files:
  Lib/test/test_site.py |  4 +++-
  Misc/NEWS             |  3 +++
  2 files changed, 6 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -24,7 +24,7 @@
 else:
     raise unittest.SkipTest("importation of site.py suppressed")
 
-if not os.path.isdir(site.USER_SITE):
+if site.ENABLE_USER_SITE and not os.path.isdir(site.USER_SITE):
     # need to add user site directory for tests
     os.makedirs(site.USER_SITE)
     site.addsitedir(site.USER_SITE)
@@ -157,6 +157,8 @@
         finally:
             pth_file.cleanup()
 
+    @unittest.skipUnless(site.ENABLE_USER_SITE, "requires access to PEP 370 "
+                          "user-site (site.ENABLE_USER_SITE)")
     def test_s_option(self):
         usersite = site.USER_SITE
         self.assertIn(usersite, sys.path)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1523,6 +1523,9 @@
 Tests
 -----
 
+- Issue #13304: Skip test case if user site-packages disabled (-s or
+  PYTHONNOUSERSITE).  (Patch by Carl Meyer)
+
 - Issue #5661: Add a test for ECONNRESET/EPIPE handling to test_asyncore. Patch
   by Xavier de Gaye.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue Nov  1 05:29:08 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 01 Nov 2011 05:29:08 +0100
Subject: [Python-checkins] Daily reference leaks (c343c095d08b): sum=0
Message-ID: <E1RL5yG-0003TD-9L@ap.vmr.nerim.net>

results for c343c095d08b on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogVLAypM', '-x']

From python-checkins at python.org  Tue Nov  1 10:34:37 2011
From: python-checkins at python.org (sandro.tosi)
Date: Tue, 01 Nov 2011 10:34:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_correct_signatu?=
 =?utf8?q?re_for_tzinfo=2Edst=28=29_in_examples=3B_thanks_to_Daniil_Shved_?=
 =?utf8?q?from?=
Message-ID: <E1RLAjt-0004Th-OJ@dinsdale.python.org>

http://hg.python.org/cpython/rev/a38dd36a44fa
changeset:   73269:a38dd36a44fa
branch:      2.7
parent:      73266:1689b9cf6b1c
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Tue Nov 01 10:31:26 2011 +0100
summary:
  correct signature for tzinfo.dst() in examples; thanks to Daniil Shved from docs@

files:
  Doc/library/datetime.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst
--- a/Doc/library/datetime.rst
+++ b/Doc/library/datetime.rst
@@ -1385,13 +1385,13 @@
 
    Most implementations of :meth:`dst` will probably look like one of these two::
 
-      def dst(self):
+      def dst(self, dt):
           # a fixed-offset class:  doesn't account for DST
           return timedelta(0)
 
    or ::
 
-      def dst(self):
+      def dst(self, dt):
           # Code to set dston and dstoff to the time zone's DST
           # transition times based on the input dt.year, and expressed
           # in standard local time.  Then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 10:34:38 2011
From: python-checkins at python.org (sandro.tosi)
Date: Tue, 01 Nov 2011 10:34:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_correct_signatu?=
 =?utf8?q?re_for_tzinfo=2Edst=28=29_in_examples=3B_thanks_to_Daniil_Shved_?=
 =?utf8?q?from?=
Message-ID: <E1RLAju-0004Tr-IA@dinsdale.python.org>

http://hg.python.org/cpython/rev/1d47d5fdb633
changeset:   73270:1d47d5fdb633
branch:      3.2
parent:      73267:c497011a4769
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Tue Nov 01 10:32:05 2011 +0100
summary:
  correct signature for tzinfo.dst() in examples; thanks to Daniil Shved from docs@

files:
  Doc/library/datetime.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst
--- a/Doc/library/datetime.rst
+++ b/Doc/library/datetime.rst
@@ -1422,13 +1422,13 @@
 
    Most implementations of :meth:`dst` will probably look like one of these two::
 
-      def dst(self):
+      def dst(self, dt):
           # a fixed-offset class:  doesn't account for DST
           return timedelta(0)
 
    or ::
 
-      def dst(self):
+      def dst(self, dt):
           # Code to set dston and dstoff to the time zone's DST
           # transition times based on the input dt.year, and expressed
           # in standard local time.  Then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 10:34:39 2011
From: python-checkins at python.org (sandro.tosi)
Date: Tue, 01 Nov 2011 10:34:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_with_3=2E2?=
Message-ID: <E1RLAjv-0004Tv-DM@dinsdale.python.org>

http://hg.python.org/cpython/rev/4e8d67c67959
changeset:   73271:4e8d67c67959
parent:      73268:c343c095d08b
parent:      73270:1d47d5fdb633
user:        Sandro Tosi <sandro.tosi at gmail.com>
date:        Tue Nov 01 10:32:22 2011 +0100
summary:
  merge with 3.2

files:
  Doc/library/datetime.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst
--- a/Doc/library/datetime.rst
+++ b/Doc/library/datetime.rst
@@ -1438,13 +1438,13 @@
 
    Most implementations of :meth:`dst` will probably look like one of these two::
 
-      def dst(self):
+      def dst(self, dt):
           # a fixed-offset class:  doesn't account for DST
           return timedelta(0)
 
    or ::
 
-      def dst(self):
+      def dst(self, dt):
           # Code to set dston and dstoff to the time zone's DST
           # transition times based on the input dt.year, and expressed
           # in standard local time.  Then

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 12:56:43 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 12:56:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Strengthen_the_tests_for_fo?=
 =?utf8?q?rmat_=27=25Y=27=2C_in_relation_with_issue_=2313305=2E?=
Message-ID: <E1RLCxP-0000SP-6z@dinsdale.python.org>

http://hg.python.org/cpython/rev/230f0956aaa3
changeset:   73272:230f0956aaa3
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 12:56:14 2011 +0100
summary:
  Strengthen the tests for format '%Y', in relation with issue #13305.

files:
  Lib/test/datetimetester.py |  12 +++-
  Lib/test/test_time.py      |  66 ++++++++++++++++++++-----
  2 files changed, 62 insertions(+), 16 deletions(-)


diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py
--- a/Lib/test/datetimetester.py
+++ b/Lib/test/datetimetester.py
@@ -1291,8 +1291,16 @@
 
     def test_strftime_y2k(self):
         for y in (1, 49, 70, 99, 100, 999, 1000, 1970):
-            self.assertIn(self.theclass(y, 1, 1).strftime("%Y"),
-                          [str(y),'%04d' % y])
+            d = self.theclass(y, 1, 1)
+            # Issue 13305:  For years < 1000, the value is not always
+            # padded to 4 digits across platforms.  The C standard
+            # assumes year >= 1900, so it does not specify the number
+            # of digits.
+            if d.strftime("%Y") != '%04d' % y:
+                # Year 42 returns '42', not padded
+                self.assertEqual(d.strftime("%Y"), '%d' % y)
+                # '0042' is obtained anyway
+                self.assertEqual(d.strftime("%4Y"), '%04d' % y)
 
     def test_replace(self):
         cls = self.theclass
diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -70,28 +70,35 @@
             with self.assertRaises(ValueError):
                 time.strftime('%f')
 
-    def _bounds_checking(self, func=time.strftime):
+    def _bounds_checking(self, func):
         # Make sure that strftime() checks the bounds of the various parts
-        #of the time tuple (0 is valid for *all* values).
+        # of the time tuple (0 is valid for *all* values).
 
         # The year field is tested by other test cases above
 
         # Check month [1, 12] + zero support
+        func((1900, 0, 1, 0, 0, 0, 0, 1, -1))
+        func((1900, 12, 1, 0, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, -1, 1, 0, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 13, 1, 0, 0, 0, 0, 1, -1))
         # Check day of month [1, 31] + zero support
+        func((1900, 1, 0, 0, 0, 0, 0, 1, -1))
+        func((1900, 1, 31, 0, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, -1, 0, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 32, 0, 0, 0, 0, 1, -1))
         # Check hour [0, 23]
+        func((1900, 1, 1, 0, 0, 0, 0, 1, -1))
+        func((1900, 1, 1, 23, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, -1, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, 24, 0, 0, 0, 1, -1))
         # Check minute [0, 59]
+        func((1900, 1, 1, 0, 59, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, 0, -1, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
@@ -101,15 +108,21 @@
                             (1900, 1, 1, 0, 0, -1, 0, 1, -1))
         # C99 only requires allowing for one leap second, but Python's docs say
         # allow two leap seconds (0..61)
+        func((1900, 1, 1, 0, 0, 60, 0, 1, -1))
+        func((1900, 1, 1, 0, 0, 61, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, 0, 0, 62, 0, 1, -1))
         # No check for upper-bound day of week;
         #  value forced into range by a ``% 7`` calculation.
         # Start check at -2 since gettmarg() increments value before taking
         #  modulo.
+        self.assertEqual(func((1900, 1, 1, 0, 0, 0, -1, 1, -1)),
+                         func((1900, 1, 1, 0, 0, 0, +6, 1, -1)))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, 0, 0, 0, -2, 1, -1))
         # Check day of the year [1, 366] + zero support
+        func((1900, 1, 1, 0, 0, 0, 0, 0, -1))
+        func((1900, 1, 1, 0, 0, 0, 0, 366, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, 0, 0, 0, 0, -1, -1))
         self.assertRaises(ValueError, func,
@@ -299,6 +312,8 @@
         raise NotImplementedError()
 
 class _TestAsctimeYear:
+    _format = '%d'
+
     def yearstr(self, y):
         return time.asctime((y,) + (0,) * 8).split()[-1]
 
@@ -308,8 +323,41 @@
         self.assertEqual(self.yearstr(123456789), '123456789')
 
 class _TestStrftimeYear:
+
+    # Issue 13305:  For years < 1000, the value is not always
+    # padded to 4 digits across platforms.  The C standard
+    # assumes year >= 1900, so it does not specify the number
+    # of digits.
+
+    if time.strftime('%Y', (1,) + (0,) * 8) == '0001':
+        _format = '%04d'
+    else:
+        _format = '%d'
+
     def yearstr(self, y):
-        return time.strftime('%Y', (y,) + (0,) * 8).split()[-1]
+        return time.strftime('%Y', (y,) + (0,) * 8)
+
+    def test_4dyear(self):
+        # Check that we can return the zero padded value.
+        if self._format == '%04d':
+            self.test_year('%04d')
+        else:
+            def year4d(y):
+                return time.strftime('%4Y', (y,) + (0,) * 8)
+            self.test_year('%04d', func=year4d)
+
+class _Test4dYear(_BaseYearTest):
+    _format = '%d'
+
+    def test_year(self, fmt=None, func=None):
+        fmt = fmt or self._format
+        func = func or self.yearstr
+        self.assertEqual(func(1),    fmt % 1)
+        self.assertEqual(func(68),   fmt % 68)
+        self.assertEqual(func(69),   fmt % 69)
+        self.assertEqual(func(99),   fmt % 99)
+        self.assertEqual(func(999),  fmt % 999)
+        self.assertEqual(func(9999), fmt % 9999)
 
     def test_large_year(self):
         # Check that it doesn't crash for year > 9999
@@ -321,23 +369,13 @@
         self.assertEqual(text, '12345')
         self.assertEqual(self.yearstr(123456789), '123456789')
 
-class _Test4dYear(_BaseYearTest):
-
-    def test_year(self):
-        self.assertIn(self.yearstr(1),     ('1', '0001'))
-        self.assertIn(self.yearstr(68),   ('68', '0068'))
-        self.assertIn(self.yearstr(69),   ('69', '0069'))
-        self.assertIn(self.yearstr(99),   ('99', '0099'))
-        self.assertIn(self.yearstr(999), ('999', '0999'))
-        self.assertEqual(self.yearstr(9999), '9999')
-
     def test_negative(self):
         try:
             text = self.yearstr(-1)
         except ValueError:
             # strftime() is limited to [1; 9999] with Visual Studio
             return
-        self.assertIn(text, ('-1', '-001'))
+        self.assertEqual(text, self._format % -1)
 
         self.assertEqual(self.yearstr(-1234), '-1234')
         self.assertEqual(self.yearstr(-123456), '-123456')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 13:14:28 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 13:14:28 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzY3MDY2NDogRml4?=
 =?utf8?q?_HTMLParser_to_correctly_handle_the_content_of?=
Message-ID: <E1RLDEa-0002hw-Tf@dinsdale.python.org>

http://hg.python.org/cpython/rev/0a5eb57d5876
changeset:   73273:0a5eb57d5876
branch:      2.7
parent:      73269:a38dd36a44fa
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 14:09:56 2011 +0200
summary:
  #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``.

files:
  Doc/library/htmlparser.rst  |   3 +-
  Lib/HTMLParser.py           |  22 ++++++++++--
  Lib/test/test_htmlparser.py |  42 +++++++++++++++++-------
  Misc/NEWS                   |   7 ++-
  4 files changed, 55 insertions(+), 19 deletions(-)


diff --git a/Doc/library/htmlparser.rst b/Doc/library/htmlparser.rst
--- a/Doc/library/htmlparser.rst
+++ b/Doc/library/htmlparser.rst
@@ -123,7 +123,8 @@
 
 .. method:: HTMLParser.handle_data(data)
 
-   This method is called to process arbitrary data.  It is intended to be
+   This method is called to process arbitrary data (e.g. the content of
+   ``<script>...</script>`` and ``<style>...</style>``).  It is intended to be
    overridden by a derived class; the base class implementation does nothing.
 
 
diff --git a/Lib/HTMLParser.py b/Lib/HTMLParser.py
--- a/Lib/HTMLParser.py
+++ b/Lib/HTMLParser.py
@@ -43,6 +43,8 @@
   \s*                                # trailing whitespace
 """, re.VERBOSE)
 endendtag = re.compile('>')
+# the HTML 5 spec, section 8.1.2.2, doesn't allow spaces between
+# </ and the tag name, so maybe this should be fixed
 endtagfind = re.compile('</\s*([a-zA-Z][-.a-zA-Z0-9:_]*)\s*>')
 
 
@@ -96,6 +98,7 @@
         self.rawdata = ''
         self.lasttag = '???'
         self.interesting = interesting_normal
+        self.cdata_elem = None
         markupbase.ParserBase.reset(self)
 
     def feed(self, data):
@@ -120,11 +123,13 @@
         """Return full source of start tag: '<...>'."""
         return self.__starttag_text
 
-    def set_cdata_mode(self):
+    def set_cdata_mode(self, elem):
         self.interesting = interesting_cdata
+        self.cdata_elem = elem.lower()
 
     def clear_cdata_mode(self):
         self.interesting = interesting_normal
+        self.cdata_elem = None
 
     # Internal -- handle data as far as reasonable.  May leave state
     # and data to be processed by a subsequent call.  If 'end' is
@@ -270,7 +275,7 @@
         else:
             self.handle_starttag(tag, attrs)
             if tag in self.CDATA_CONTENT_ELEMENTS:
-                self.set_cdata_mode()
+                self.set_cdata_mode(tag)
         return endpos
 
     # Internal -- check to see if we have a complete starttag; return end
@@ -314,9 +319,18 @@
         j = match.end()
         match = endtagfind.match(rawdata, i) # </ + tag + >
         if not match:
+            if self.cdata_elem is not None:
+                self.handle_data(rawdata[i:j])
+                return j
             self.error("bad end tag: %r" % (rawdata[i:j],))
-        tag = match.group(1)
-        self.handle_endtag(tag.lower())
+
+        elem = match.group(1).lower() # script or style
+        if self.cdata_elem is not None:
+            if elem != self.cdata_elem:
+                self.handle_data(rawdata[i:j])
+                return j
+
+        self.handle_endtag(elem)
         self.clear_cdata_mode()
         return j
 
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -312,18 +312,36 @@
             ("starttag_text", s)])
 
     def test_cdata_content(self):
-        s = """<script> <!-- not a comment --> &not-an-entity-ref; </script>"""
-        self._run_check(s, [
-            ("starttag", "script", []),
-            ("data", " <!-- not a comment --> &not-an-entity-ref; "),
-            ("endtag", "script"),
-            ])
-        s = """<script> <not a='start tag'> </script>"""
-        self._run_check(s, [
-            ("starttag", "script", []),
-            ("data", " <not a='start tag'> "),
-            ("endtag", "script"),
-            ])
+        contents = [
+            '<!-- not a comment --> &not-an-entity-ref;',
+            "<not a='start tag'>",
+            '<a href="" /> <p> <span></span>',
+            'foo = "</scr" + "ipt>";',
+            'foo = "</SCRIPT" + ">";',
+            'foo = <\n/script> ',
+            '<!-- document.write("</scr" + "ipt>"); -->',
+            ('\n//<![CDATA[\n'
+             'document.write(\'<s\'+\'cript type="text/javascript" '
+             'src="http://www.example.org/r=\'+new '
+             'Date().getTime()+\'"><\\/s\'+\'cript>\');\n//]]>'),
+            '\n<!-- //\nvar foo = 3.14;\n// -->\n',
+            'foo = "</sty" + "le>";',
+            u'<!-- \u2603 -->',
+            # these two should be invalid according to the HTML 5 spec,
+            # section 8.1.2.2
+            #'foo = </\nscript>',
+            #'foo = </ script>',
+        ]
+        elements = ['script', 'style', 'SCRIPT', 'STYLE', 'Script', 'Style']
+        for content in contents:
+            for element in elements:
+                element_lower = element.lower()
+                s = u'<{element}>{content}</{element}>'.format(element=element,
+                                                               content=content)
+                self._run_check(s, [("starttag", element_lower, []),
+                                    ("data", content),
+                                    ("endtag", element_lower)])
+
 
     def test_entityrefs_in_attributes(self):
         self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>", [
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -74,10 +74,13 @@
 Library
 -------
 
-- Issue 10817: Fix urlretrieve function to raise ContentTooShortError even
+- Issue #670664: Fix HTMLParser to correctly handle the content of
+  ``<script>...</script>`` and ``<style>...</style>``.
+
+- Issue #10817: Fix urlretrieve function to raise ContentTooShortError even
   when reporthook is None. Patch by Jyrki Pulliainen.
 
-- Issue 13296: Fix IDLE to clear compile __future__ flags on shell restart.
+- Issue #13296: Fix IDLE to clear compile __future__ flags on shell restart.
   (Patch by Roger Serwy)
 
 - Issue #7334: close source files on ElementTree.parse and iterparse.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 13:14:33 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 13:14:33 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzY3MDY2NDogRml4?=
 =?utf8?q?_HTMLParser_to_correctly_handle_the_content_of?=
Message-ID: <E1RLDEf-0002i1-6V@dinsdale.python.org>

http://hg.python.org/cpython/rev/a6f2244b251f
changeset:   73274:a6f2244b251f
branch:      3.2
parent:      73270:1d47d5fdb633
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 14:12:22 2011 +0200
summary:
  #670664: Fix HTMLParser to correctly handle the content of ``<script>...</script>`` and ``<style>...</style>``.

files:
  Doc/library/html.parser.rst |   3 +-
  Lib/html/parser.py          |  22 ++++++++++--
  Lib/test/test_htmlparser.py |  42 +++++++++++++++++-------
  Misc/NEWS                   |   7 ++-
  4 files changed, 55 insertions(+), 19 deletions(-)


diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst
--- a/Doc/library/html.parser.rst
+++ b/Doc/library/html.parser.rst
@@ -115,7 +115,8 @@
 
 .. method:: HTMLParser.handle_data(data)
 
-   This method is called to process arbitrary data.  It is intended to be
+   This method is called to process arbitrary data (e.g. the content of
+   ``<script>...</script>`` and ``<style>...</style>``).  It is intended to be
    overridden by a derived class; the base class implementation does nothing.
 
 
diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -62,6 +62,8 @@
   \s*                                # trailing whitespace
 """, re.VERBOSE)
 endendtag = re.compile('>')
+# the HTML 5 spec, section 8.1.2.2, doesn't allow spaces between
+# </ and the tag name, so maybe this should be fixed
 endtagfind = re.compile('</\s*([a-zA-Z][-.a-zA-Z0-9:_]*)\s*>')
 
 
@@ -121,6 +123,7 @@
         self.rawdata = ''
         self.lasttag = '???'
         self.interesting = interesting_normal
+        self.cdata_elem = None
         _markupbase.ParserBase.reset(self)
 
     def feed(self, data):
@@ -145,11 +148,13 @@
         """Return full source of start tag: '<...>'."""
         return self.__starttag_text
 
-    def set_cdata_mode(self):
+    def set_cdata_mode(self, elem):
         self.interesting = interesting_cdata
+        self.cdata_elem = elem.lower()
 
     def clear_cdata_mode(self):
         self.interesting = interesting_normal
+        self.cdata_elem = None
 
     # Internal -- handle data as far as reasonable.  May leave state
     # and data to be processed by a subsequent call.  If 'end' is
@@ -314,7 +319,7 @@
         else:
             self.handle_starttag(tag, attrs)
             if tag in self.CDATA_CONTENT_ELEMENTS:
-                self.set_cdata_mode()
+                self.set_cdata_mode(tag)
         return endpos
 
     # Internal -- check to see if we have a complete starttag; return end
@@ -371,6 +376,9 @@
         j = match.end()
         match = endtagfind.match(rawdata, i) # </ + tag + >
         if not match:
+            if self.cdata_elem is not None:
+                self.handle_data(rawdata[i:j])
+                return j
             if self.strict:
                 self.error("bad end tag: %r" % (rawdata[i:j],))
             k = rawdata.find('<', i + 1, j)
@@ -380,8 +388,14 @@
                 j = i + 1
             self.handle_data(rawdata[i:j])
             return j
-        tag = match.group(1)
-        self.handle_endtag(tag.lower())
+
+        elem = match.group(1).lower() # script or style
+        if self.cdata_elem is not None:
+            if elem != self.cdata_elem:
+                self.handle_data(rawdata[i:j])
+                return j
+
+        self.handle_endtag(elem.lower())
         self.clear_cdata_mode()
         return j
 
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -321,18 +321,36 @@
             ("starttag_text", s)])
 
     def test_cdata_content(self):
-        s = """<script> <!-- not a comment --> &not-an-entity-ref; </script>"""
-        self._run_check(s, [
-            ("starttag", "script", []),
-            ("data", " <!-- not a comment --> &not-an-entity-ref; "),
-            ("endtag", "script"),
-            ])
-        s = """<script> <not a='start tag'> </script>"""
-        self._run_check(s, [
-            ("starttag", "script", []),
-            ("data", " <not a='start tag'> "),
-            ("endtag", "script"),
-            ])
+        contents = [
+            '<!-- not a comment --> &not-an-entity-ref;',
+            "<not a='start tag'>",
+            '<a href="" /> <p> <span></span>',
+            'foo = "</scr" + "ipt>";',
+            'foo = "</SCRIPT" + ">";',
+            'foo = <\n/script> ',
+            '<!-- document.write("</scr" + "ipt>"); -->',
+            ('\n//<![CDATA[\n'
+             'document.write(\'<s\'+\'cript type="text/javascript" '
+             'src="http://www.example.org/r=\'+new '
+             'Date().getTime()+\'"><\\/s\'+\'cript>\');\n//]]>'),
+            '\n<!-- //\nvar foo = 3.14;\n// -->\n',
+            'foo = "</sty" + "le>";',
+            '<!-- \u2603 -->',
+            # these two should be invalid according to the HTML 5 spec,
+            # section 8.1.2.2
+            #'foo = </\nscript>',
+            #'foo = </ script>',
+        ]
+        elements = ['script', 'style', 'SCRIPT', 'STYLE', 'Script', 'Style']
+        for content in contents:
+            for element in elements:
+                element_lower = element.lower()
+                s = '<{element}>{content}</{element}>'.format(element=element,
+                                                               content=content)
+                self._run_check(s, [("starttag", element_lower, []),
+                                    ("data", content),
+                                    ("endtag", element_lower)])
+
 
     def test_entityrefs_in_attributes(self):
         self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>", [
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,10 +66,13 @@
 Library
 -------
 
-- Issue 10817: Fix urlretrieve function to raise ContentTooShortError even
+- Issue #670664: Fix HTMLParser to correctly handle the content of
+  ``<script>...</script>`` and ``<style>...</style>``.
+
+- Issue #10817: Fix urlretrieve function to raise ContentTooShortError even
   when reporthook is None. Patch by Jyrki Pulliainen.
 
-- Issue 13296: Fix IDLE to clear compile __future__ flags on shell restart.
+- Issue #13296: Fix IDLE to clear compile __future__ flags on shell restart.
   (Patch by Roger Serwy)
 
 - Issue #13293: Better error message when trying to marshal bytes using

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 13:14:33 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 13:14:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=23670664=3A_merge_with_3=2E2=2E?=
Message-ID: <E1RLDEf-0002iY-VQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/b40752e227fa
changeset:   73275:b40752e227fa
parent:      73272:230f0956aaa3
parent:      73274:a6f2244b251f
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 14:14:15 2011 +0200
summary:
  #670664: merge with 3.2.

files:
  Doc/library/html.parser.rst |   3 +-
  Lib/html/parser.py          |  22 ++++++++++--
  Lib/test/test_htmlparser.py |  42 +++++++++++++++++-------
  Misc/NEWS                   |   7 ++-
  4 files changed, 55 insertions(+), 19 deletions(-)


diff --git a/Doc/library/html.parser.rst b/Doc/library/html.parser.rst
--- a/Doc/library/html.parser.rst
+++ b/Doc/library/html.parser.rst
@@ -115,7 +115,8 @@
 
 .. method:: HTMLParser.handle_data(data)
 
-   This method is called to process arbitrary data.  It is intended to be
+   This method is called to process arbitrary data (e.g. the content of
+   ``<script>...</script>`` and ``<style>...</style>``).  It is intended to be
    overridden by a derived class; the base class implementation does nothing.
 
 
diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -62,6 +62,8 @@
   \s*                                # trailing whitespace
 """, re.VERBOSE)
 endendtag = re.compile('>')
+# the HTML 5 spec, section 8.1.2.2, doesn't allow spaces between
+# </ and the tag name, so maybe this should be fixed
 endtagfind = re.compile('</\s*([a-zA-Z][-.a-zA-Z0-9:_]*)\s*>')
 
 
@@ -121,6 +123,7 @@
         self.rawdata = ''
         self.lasttag = '???'
         self.interesting = interesting_normal
+        self.cdata_elem = None
         _markupbase.ParserBase.reset(self)
 
     def feed(self, data):
@@ -145,11 +148,13 @@
         """Return full source of start tag: '<...>'."""
         return self.__starttag_text
 
-    def set_cdata_mode(self):
+    def set_cdata_mode(self, elem):
         self.interesting = interesting_cdata
+        self.cdata_elem = elem.lower()
 
     def clear_cdata_mode(self):
         self.interesting = interesting_normal
+        self.cdata_elem = None
 
     # Internal -- handle data as far as reasonable.  May leave state
     # and data to be processed by a subsequent call.  If 'end' is
@@ -314,7 +319,7 @@
         else:
             self.handle_starttag(tag, attrs)
             if tag in self.CDATA_CONTENT_ELEMENTS:
-                self.set_cdata_mode()
+                self.set_cdata_mode(tag)
         return endpos
 
     # Internal -- check to see if we have a complete starttag; return end
@@ -371,6 +376,9 @@
         j = match.end()
         match = endtagfind.match(rawdata, i) # </ + tag + >
         if not match:
+            if self.cdata_elem is not None:
+                self.handle_data(rawdata[i:j])
+                return j
             if self.strict:
                 self.error("bad end tag: %r" % (rawdata[i:j],))
             k = rawdata.find('<', i + 1, j)
@@ -380,8 +388,14 @@
                 j = i + 1
             self.handle_data(rawdata[i:j])
             return j
-        tag = match.group(1)
-        self.handle_endtag(tag.lower())
+
+        elem = match.group(1).lower() # script or style
+        if self.cdata_elem is not None:
+            if elem != self.cdata_elem:
+                self.handle_data(rawdata[i:j])
+                return j
+
+        self.handle_endtag(elem.lower())
         self.clear_cdata_mode()
         return j
 
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -321,18 +321,36 @@
             ("starttag_text", s)])
 
     def test_cdata_content(self):
-        s = """<script> <!-- not a comment --> &not-an-entity-ref; </script>"""
-        self._run_check(s, [
-            ("starttag", "script", []),
-            ("data", " <!-- not a comment --> &not-an-entity-ref; "),
-            ("endtag", "script"),
-            ])
-        s = """<script> <not a='start tag'> </script>"""
-        self._run_check(s, [
-            ("starttag", "script", []),
-            ("data", " <not a='start tag'> "),
-            ("endtag", "script"),
-            ])
+        contents = [
+            '<!-- not a comment --> &not-an-entity-ref;',
+            "<not a='start tag'>",
+            '<a href="" /> <p> <span></span>',
+            'foo = "</scr" + "ipt>";',
+            'foo = "</SCRIPT" + ">";',
+            'foo = <\n/script> ',
+            '<!-- document.write("</scr" + "ipt>"); -->',
+            ('\n//<![CDATA[\n'
+             'document.write(\'<s\'+\'cript type="text/javascript" '
+             'src="http://www.example.org/r=\'+new '
+             'Date().getTime()+\'"><\\/s\'+\'cript>\');\n//]]>'),
+            '\n<!-- //\nvar foo = 3.14;\n// -->\n',
+            'foo = "</sty" + "le>";',
+            '<!-- \u2603 -->',
+            # these two should be invalid according to the HTML 5 spec,
+            # section 8.1.2.2
+            #'foo = </\nscript>',
+            #'foo = </ script>',
+        ]
+        elements = ['script', 'style', 'SCRIPT', 'STYLE', 'Script', 'Style']
+        for content in contents:
+            for element in elements:
+                element_lower = element.lower()
+                s = '<{element}>{content}</{element}>'.format(element=element,
+                                                               content=content)
+                self._run_check(s, [("starttag", element_lower, []),
+                                    ("data", content),
+                                    ("endtag", element_lower)])
+
 
     def test_entityrefs_in_attributes(self):
         self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>", [
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,10 +350,13 @@
 Library
 -------
 
-- Issue 10817: Fix urlretrieve function to raise ContentTooShortError even
+- Issue #670664: Fix HTMLParser to correctly handle the content of
+  ``<script>...</script>`` and ``<style>...</style>``.
+
+- Issue #10817: Fix urlretrieve function to raise ContentTooShortError even
   when reporthook is None. Patch by Jyrki Pulliainen.
 
-- Issue 13296: Fix IDLE to clear compile __future__ flags on shell restart.
+- Issue #13296: Fix IDLE to clear compile __future__ flags on shell restart.
   (Patch by Roger Serwy)
 
 - Fix the xmlrpc.client user agent to return something similar to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 13:16:36 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 13:16:36 +0100
Subject: [Python-checkins] =?utf8?q?devguide=3A_All_your_tags_are_belong_t?=
	=?utf8?q?o_me=2E?=
Message-ID: <E1RLDGe-0002w9-FC@dinsdale.python.org>

http://hg.python.org/devguide/rev/eca36de46929
changeset:   461:eca36de46929
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 14:16:22 2011 +0200
summary:
  All your tags are belong to me.

files:
  experts.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -121,7 +121,7 @@
 hashlib
 heapq                 rhettinger, stutzbach
 hmac
-html
+html                  ezio.melotti
 http
 idlelib               kbk
 imaplib

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Tue Nov  1 13:28:43 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Tue, 01 Nov 2011 13:28:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_=7BGet=2CSet=2CHas?=
 =?utf8?q?=7DAttrString_with_*AttrId=2E?=
Message-ID: <E1RLDSN-00042T-NC@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a2947376d2e
changeset:   73276:5a2947376d2e
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Fri Oct 14 15:35:36 2011 +0200
summary:
  Replace {Get,Set,Has}AttrString with *AttrId.

files:
  Modules/_io/_iomodule.c  |   3 ++-
  Modules/_io/bufferedio.c |   8 +++++---
  Modules/_io/fileio.c     |   3 ++-
  Modules/_io/iobase.c     |   9 ++++++---
  Modules/_io/textio.c     |  16 ++++++++++------
  5 files changed, 25 insertions(+), 14 deletions(-)


diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c
--- a/Modules/_io/_iomodule.c
+++ b/Modules/_io/_iomodule.c
@@ -233,6 +233,7 @@
 
     _Py_IDENTIFIER(isatty);
     _Py_IDENTIFIER(fileno);
+    _Py_IDENTIFIER(mode);
 
     if (!PyArg_ParseTupleAndKeywords(args, kwds, "O|sizzziO:open", kwlist,
                                      &file, &mode, &buffering,
@@ -440,7 +441,7 @@
     if (wrapper == NULL)
         goto error;
 
-    if (PyObject_SetAttrString(wrapper, "mode", modeobj) < 0)
+    if (_PyObject_SetAttrId(wrapper, &PyId_mode, modeobj) < 0)
         goto error;
     Py_DECREF(modeobj);
     return wrapper;
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -17,6 +17,8 @@
 _Py_IDENTIFIER(_dealloc_warn);
 _Py_IDENTIFIER(flush);
 _Py_IDENTIFIER(isatty);
+_Py_IDENTIFIER(mode);
+_Py_IDENTIFIER(name);
 _Py_IDENTIFIER(peek);
 _Py_IDENTIFIER(read);
 _Py_IDENTIFIER(read1);
@@ -556,14 +558,14 @@
 buffered_name_get(buffered *self, void *context)
 {
     CHECK_INITIALIZED(self)
-    return PyObject_GetAttrString(self->raw, "name");
+    return _PyObject_GetAttrId(self->raw, &PyId_name);
 }
 
 static PyObject *
 buffered_mode_get(buffered *self, void *context)
 {
     CHECK_INITIALIZED(self)
-    return PyObject_GetAttrString(self->raw, "mode");
+    return _PyObject_GetAttrId(self->raw, &PyId_mode);
 }
 
 /* Lower-level APIs */
@@ -1301,7 +1303,7 @@
 {
     PyObject *nameobj, *res;
 
-    nameobj = PyObject_GetAttrString((PyObject *) self, "name");
+    nameobj = _PyObject_GetAttrId((PyObject *) self, &PyId_name);
     if (nameobj == NULL) {
         if (PyErr_ExceptionMatches(PyExc_AttributeError))
             PyErr_Clear();
diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c
--- a/Modules/_io/fileio.c
+++ b/Modules/_io/fileio.c
@@ -993,12 +993,13 @@
 static PyObject *
 fileio_repr(fileio *self)
 {
+    _Py_IDENTIFIER(name);
     PyObject *nameobj, *res;
 
     if (self->fd < 0)
         return PyUnicode_FromFormat("<_io.FileIO [closed]>");
 
-    nameobj = PyObject_GetAttrString((PyObject *) self, "name");
+    nameobj = _PyObject_GetAttrId((PyObject *) self, &PyId_name);
     if (nameobj == NULL) {
         if (PyErr_ExceptionMatches(PyExc_AttributeError))
             PyErr_Clear();
diff --git a/Modules/_io/iobase.c b/Modules/_io/iobase.c
--- a/Modules/_io/iobase.c
+++ b/Modules/_io/iobase.c
@@ -59,8 +59,9 @@
    of the IOBase object rather than the virtual `closed` attribute as returned
    by whatever subclass. */
 
+_Py_IDENTIFIER(__IOBase_closed);
 #define IS_CLOSED(self) \
-    PyObject_HasAttrString(self, "__IOBase_closed")
+    _PyObject_HasAttrId(self, &PyId___IOBase_closed)
 
 /* Internal methods */
 static PyObject *
@@ -192,12 +193,13 @@
 iobase_close(PyObject *self, PyObject *args)
 {
     PyObject *res;
+    _Py_IDENTIFIER(__IOBase_closed);
 
     if (IS_CLOSED(self))
         Py_RETURN_NONE;
 
     res = PyObject_CallMethodObjArgs(self, _PyIO_str_flush, NULL);
-    PyObject_SetAttrString(self, "__IOBase_closed", Py_True);
+    _PyObject_SetAttrId(self, &PyId___IOBase_closed, Py_True);
     if (res == NULL) {
         return NULL;
     }
@@ -467,12 +469,13 @@
     PyObject *buffer, *result;
     Py_ssize_t old_size = -1;
     _Py_IDENTIFIER(read);
+    _Py_IDENTIFIER(peek);
 
     if (!PyArg_ParseTuple(args, "|O&:readline", &_PyIO_ConvertSsize_t, &limit)) {
         return NULL;
     }
 
-    if (PyObject_HasAttrString(self, "peek"))
+    if (_PyObject_HasAttrId(self, &PyId_peek))
         has_peek = 1;
 
     buffer = PyByteArray_FromStringAndSize(NULL, 0);
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -19,7 +19,11 @@
 _Py_IDENTIFIER(flush);
 _Py_IDENTIFIER(getpreferredencoding);
 _Py_IDENTIFIER(isatty);
+_Py_IDENTIFIER(mode);
+_Py_IDENTIFIER(name);
+_Py_IDENTIFIER(raw);
 _Py_IDENTIFIER(read);
+_Py_IDENTIFIER(read1);
 _Py_IDENTIFIER(readable);
 _Py_IDENTIFIER(replace);
 _Py_IDENTIFIER(reset);
@@ -999,7 +1003,7 @@
         ci = _PyCodec_Lookup(encoding);
         if (ci == NULL)
             goto error;
-        res = PyObject_GetAttrString(ci, "name");
+        res = _PyObject_GetAttrId(ci, &PyId_name);
         Py_DECREF(ci);
         if (res == NULL) {
             if (PyErr_ExceptionMatches(PyExc_AttributeError))
@@ -1026,7 +1030,7 @@
     if (Py_TYPE(buffer) == &PyBufferedReader_Type ||
         Py_TYPE(buffer) == &PyBufferedWriter_Type ||
         Py_TYPE(buffer) == &PyBufferedRandom_Type) {
-        raw = PyObject_GetAttrString(buffer, "raw");
+        raw = _PyObject_GetAttrId(buffer, &PyId_raw);
         /* Cache the raw FileIO object to speed up 'closed' checks */
         if (raw == NULL) {
             if (PyErr_ExceptionMatches(PyExc_AttributeError))
@@ -1046,7 +1050,7 @@
     self->seekable = self->telling = PyObject_IsTrue(res);
     Py_DECREF(res);
 
-    self->has_read1 = PyObject_HasAttrString(buffer, "read1");
+    self->has_read1 = _PyObject_HasAttrId(buffer, &PyId_read1);
 
     self->encoding_start_of_stream = 0;
     if (self->seekable && self->encoder) {
@@ -2401,7 +2405,7 @@
     res = PyUnicode_FromString("<_io.TextIOWrapper");
     if (res == NULL)
         return NULL;
-    nameobj = PyObject_GetAttrString((PyObject *) self, "name");
+    nameobj = _PyObject_GetAttrId((PyObject *) self, &PyId_name);
     if (nameobj == NULL) {
         if (PyErr_ExceptionMatches(PyExc_AttributeError))
             PyErr_Clear();
@@ -2417,7 +2421,7 @@
         if (res == NULL)
             return NULL;
     }
-    modeobj = PyObject_GetAttrString((PyObject *) self, "mode");
+    modeobj = _PyObject_GetAttrId((PyObject *) self, &PyId_mode);
     if (modeobj == NULL) {
         if (PyErr_ExceptionMatches(PyExc_AttributeError))
             PyErr_Clear();
@@ -2578,7 +2582,7 @@
 textiowrapper_name_get(textio *self, void *context)
 {
     CHECK_INITIALIZED(self);
-    return PyObject_GetAttrString(self->buffer, "name");
+    return _PyObject_GetAttrId(self->buffer, &PyId_name);
 }
 
 static PyObject *

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 13:44:05 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 13:44:05 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEyMDA4OiBhZGQg?=
	=?utf8?q?a_test=2E?=
Message-ID: <E1RLDhF-0005Py-9q@dinsdale.python.org>

http://hg.python.org/cpython/rev/6107a84e3c44
changeset:   73277:6107a84e3c44
branch:      3.2
parent:      73274:a6f2244b251f
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 14:42:54 2011 +0200
summary:
  #12008: add a test.

files:
  Lib/test/test_htmlparser.py |  24 ++++++++++++++++++++++++
  1 files changed, 24 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -377,6 +377,30 @@
                              ('endtag', 'html')],
                         collector = self.collector)
 
+    def test_with_unquoted_attributes(self):
+        html = ("<html><body bgcolor=d0ca90 text='181008'>"
+                "<table cellspacing=0 cellpadding=1 width=100% ><tr>"
+                "<td align=left><font size=-1>"
+                "- <a href=/rabota/><span class=en> software-and-i</span></a>"
+                "- <a href='/1/'><span class=en> library</span></a></table>")
+        expected = [
+            ('starttag', 'html', []),
+            ('starttag', 'body', [('bgcolor', 'd0ca90'), ('text', '181008')]),
+            ('starttag', 'table',
+                [('cellspacing', '0'), ('cellpadding', '1'), ('width', '100%')]),
+            ('starttag', 'tr', []),
+            ('starttag', 'td', [('align', 'left')]),
+            ('starttag', 'font', [('size', '-1')]),
+            ('data', '- '), ('starttag', 'a', [('href', '/rabota/')]),
+            ('starttag', 'span', [('class', 'en')]), ('data', ' software-and-i'),
+            ('endtag', 'span'), ('endtag', 'a'),
+            ('data', '- '), ('starttag', 'a', [('href', '/1/')]),
+            ('starttag', 'span', [('class', 'en')]), ('data', ' library'),
+            ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table')
+        ]
+
+        self._run_check(html, expected, collector=self.collector)
+
     def test_comma_between_attributes(self):
         self._run_check('<form action="/xxx.php?a=1&amp;b=2&amp", '
                         'method="post">', [

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 13:44:06 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 13:44:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2312008=3A_merge_with_3=2E2=2E?=
Message-ID: <E1RLDhG-0005Q2-29@dinsdale.python.org>

http://hg.python.org/cpython/rev/495b31a8b280
changeset:   73278:495b31a8b280
parent:      73276:5a2947376d2e
parent:      73277:6107a84e3c44
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 14:43:51 2011 +0200
summary:
  #12008: merge with 3.2.

files:
  Lib/test/test_htmlparser.py |  24 ++++++++++++++++++++++++
  1 files changed, 24 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -377,6 +377,30 @@
                              ('endtag', 'html')],
                         collector = self.collector)
 
+    def test_with_unquoted_attributes(self):
+        html = ("<html><body bgcolor=d0ca90 text='181008'>"
+                "<table cellspacing=0 cellpadding=1 width=100% ><tr>"
+                "<td align=left><font size=-1>"
+                "- <a href=/rabota/><span class=en> software-and-i</span></a>"
+                "- <a href='/1/'><span class=en> library</span></a></table>")
+        expected = [
+            ('starttag', 'html', []),
+            ('starttag', 'body', [('bgcolor', 'd0ca90'), ('text', '181008')]),
+            ('starttag', 'table',
+                [('cellspacing', '0'), ('cellpadding', '1'), ('width', '100%')]),
+            ('starttag', 'tr', []),
+            ('starttag', 'td', [('align', 'left')]),
+            ('starttag', 'font', [('size', '-1')]),
+            ('data', '- '), ('starttag', 'a', [('href', '/rabota/')]),
+            ('starttag', 'span', [('class', 'en')]), ('data', ' software-and-i'),
+            ('endtag', 'span'), ('endtag', 'a'),
+            ('data', '- '), ('starttag', 'a', [('href', '/1/')]),
+            ('starttag', 'span', [('class', 'en')]), ('data', ' library'),
+            ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table')
+        ]
+
+        self._run_check(html, expected, collector=self.collector)
+
     def test_comma_between_attributes(self):
         self._run_check('<form action="/xxx.php?a=1&amp;b=2&amp", '
                         'method="post">', [

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 14:02:30 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 14:02:30 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Avoid_reusing_t?=
 =?utf8?q?he_same_collector_in_the_tests=2E?=
Message-ID: <E1RLDz4-000719-0Z@dinsdale.python.org>

http://hg.python.org/cpython/rev/4404e784918c
changeset:   73279:4404e784918c
branch:      3.2
parent:      73277:6107a84e3c44
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 15:00:59 2011 +0200
summary:
  Avoid reusing the same collector in the tests.

files:
  Lib/test/test_htmlparser.py |  21 +++++++++++----------
  1 files changed, 11 insertions(+), 10 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -360,8 +360,8 @@
 
 class HTMLParserTolerantTestCase(TestCaseBase):
 
-    def setUp(self):
-        self.collector = EventCollector(strict=False)
+    def get_collector(self):
+        return EventCollector(strict=False)
 
     def test_tolerant_parsing(self):
         self._run_check('<html <html>te>>xt&a<<bc</a></html>\n'
@@ -375,9 +375,10 @@
                              ('endtag', 'html'),
                              ('data', '\n<img src="URL><//img></html'),
                              ('endtag', 'html')],
-                        collector = self.collector)
+                        collector=self.get_collector())
 
     def test_with_unquoted_attributes(self):
+        # see #12008
         html = ("<html><body bgcolor=d0ca90 text='181008'>"
                 "<table cellspacing=0 cellpadding=1 width=100% ><tr>"
                 "<td align=left><font size=-1>"
@@ -399,7 +400,7 @@
             ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table')
         ]
 
-        self._run_check(html, expected, collector=self.collector)
+        self._run_check(html, expected, collector=self.get_collector())
 
     def test_comma_between_attributes(self):
         self._run_check('<form action="/xxx.php?a=1&amp;b=2&amp", '
@@ -407,15 +408,16 @@
                             ('starttag', 'form',
                                 [('action', '/xxx.php?a=1&b=2&amp'),
                                  ('method', 'post')])],
-                        collector = self.collector)
+                        collector=self.get_collector())
 
     def test_weird_chars_in_unquoted_attribute_values(self):
         self._run_check('<form action=bogus|&#()value>', [
                             ('starttag', 'form',
                                 [('action', 'bogus|&#()value')])],
-                        collector = self.collector)
+                        collector=self.get_collector())
 
-    def test_issue13273(self):
+    def test_correct_detection_of_start_tags(self):
+        # see #13273
         html = ('<div style=""    ><b>The <a href="some_url">rain</a> '
                 '<br /> in <span>Spain</span></b></div>')
         expected = [
@@ -434,9 +436,8 @@
             ('endtag', 'b'),
             ('endtag', 'div')
         ]
-        self._run_check(html, expected, collector=self.collector)
+        self._run_check(html, expected, collector=self.get_collector())
 
-    def test_issue13273_2(self):
         html = '<div style="", foo = "bar" ><b>The <a href="some_url">rain</a>'
         expected = [
             ('starttag', 'div', [('style', ''), ('foo', 'bar')]),
@@ -446,7 +447,7 @@
             ('data', 'rain'),
             ('endtag', 'a'),
         ]
-        self._run_check(html, expected, collector=self.collector)
+        self._run_check(html, expected, collector=self.get_collector())
 
     def test_unescape_function(self):
         p = html.parser.HTMLParser()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 14:02:30 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 14:02:30 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_test_fixes_from_3=2E2=2E?=
Message-ID: <E1RLDz4-00071J-Op@dinsdale.python.org>

http://hg.python.org/cpython/rev/136832dc69e4
changeset:   73280:136832dc69e4
parent:      73278:495b31a8b280
parent:      73279:4404e784918c
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 15:02:16 2011 +0200
summary:
  Merge test fixes from 3.2.

files:
  Lib/test/test_htmlparser.py |  21 +++++++++++----------
  1 files changed, 11 insertions(+), 10 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -360,8 +360,8 @@
 
 class HTMLParserTolerantTestCase(TestCaseBase):
 
-    def setUp(self):
-        self.collector = EventCollector(strict=False)
+    def get_collector(self):
+        return EventCollector(strict=False)
 
     def test_tolerant_parsing(self):
         self._run_check('<html <html>te>>xt&a<<bc</a></html>\n'
@@ -375,9 +375,10 @@
                              ('endtag', 'html'),
                              ('data', '\n<img src="URL><//img></html'),
                              ('endtag', 'html')],
-                        collector = self.collector)
+                        collector=self.get_collector())
 
     def test_with_unquoted_attributes(self):
+        # see #12008
         html = ("<html><body bgcolor=d0ca90 text='181008'>"
                 "<table cellspacing=0 cellpadding=1 width=100% ><tr>"
                 "<td align=left><font size=-1>"
@@ -399,7 +400,7 @@
             ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table')
         ]
 
-        self._run_check(html, expected, collector=self.collector)
+        self._run_check(html, expected, collector=self.get_collector())
 
     def test_comma_between_attributes(self):
         self._run_check('<form action="/xxx.php?a=1&amp;b=2&amp", '
@@ -407,15 +408,16 @@
                             ('starttag', 'form',
                                 [('action', '/xxx.php?a=1&b=2&amp'),
                                  ('method', 'post')])],
-                        collector = self.collector)
+                        collector=self.get_collector())
 
     def test_weird_chars_in_unquoted_attribute_values(self):
         self._run_check('<form action=bogus|&#()value>', [
                             ('starttag', 'form',
                                 [('action', 'bogus|&#()value')])],
-                        collector = self.collector)
+                        collector=self.get_collector())
 
-    def test_issue13273(self):
+    def test_correct_detection_of_start_tags(self):
+        # see #13273
         html = ('<div style=""    ><b>The <a href="some_url">rain</a> '
                 '<br /> in <span>Spain</span></b></div>')
         expected = [
@@ -434,9 +436,8 @@
             ('endtag', 'b'),
             ('endtag', 'div')
         ]
-        self._run_check(html, expected, collector=self.collector)
+        self._run_check(html, expected, collector=self.get_collector())
 
-    def test_issue13273_2(self):
         html = '<div style="", foo = "bar" ><b>The <a href="some_url">rain</a>'
         expected = [
             ('starttag', 'div', [('style', ''), ('foo', 'bar')]),
@@ -446,7 +447,7 @@
             ('data', 'rain'),
             ('endtag', 'a'),
         ]
-        self._run_check(html, expected, collector=self.collector)
+        self._run_check(html, expected, collector=self.get_collector())
 
     def test_unescape_function(self):
         p = html.parser.HTMLParser()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 14:12:12 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 14:12:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Test_the_year_range_support?=
 =?utf8?q?ed_by_time=2Estrftime=28=29_and_time=2Easctime=28=29=2E?=
Message-ID: <E1RLE8S-0007uU-TT@dinsdale.python.org>

http://hg.python.org/cpython/rev/55a3b563f0db
changeset:   73281:55a3b563f0db
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 14:11:34 2011 +0100
summary:
  Test the year range supported by time.strftime() and time.asctime().

files:
  Lib/test/test_time.py |  96 +++++++++++++++++++-----------
  1 files changed, 59 insertions(+), 37 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -6,6 +6,12 @@
 import sys
 import warnings
 
+# Max year is only limited by the size of C int.
+SIZEOF_INT = sysconfig.get_config_var('SIZEOF_INT') or 4
+TIME_MAXYEAR = (1 << 8 * SIZEOF_INT - 1) - 1
+TIME_MINYEAR = -TIME_MAXYEAR - 1
+
+
 class TimeTestCase(unittest.TestCase):
 
     def setUp(self):
@@ -91,7 +97,6 @@
         self.assertRaises(ValueError, func,
                             (1900, 1, 32, 0, 0, 0, 0, 1, -1))
         # Check hour [0, 23]
-        func((1900, 1, 1, 0, 0, 0, 0, 1, -1))
         func((1900, 1, 1, 23, 0, 0, 0, 1, -1))
         self.assertRaises(ValueError, func,
                             (1900, 1, 1, -1, 0, 0, 0, 1, -1))
@@ -165,11 +170,13 @@
         time.asctime(time.gmtime(self.t))
 
         # Max year is only limited by the size of C int.
-        sizeof_int = sysconfig.get_config_var('SIZEOF_INT') or 4
-        bigyear = (1 << 8 * sizeof_int - 1) - 1
-        asc = time.asctime((bigyear, 6, 1) + (0,)*6)
-        self.assertEqual(asc[-len(str(bigyear)):], str(bigyear))
-        self.assertRaises(OverflowError, time.asctime, (bigyear + 1,) + (0,)*8)
+        for bigyear in TIME_MAXYEAR, TIME_MINYEAR:
+            asc = time.asctime((bigyear, 6, 1) + (0,) * 6)
+            self.assertEqual(asc[-len(str(bigyear)):], str(bigyear))
+        self.assertRaises(OverflowError, time.asctime,
+                          (TIME_MAXYEAR + 1,) + (0,) * 8)
+        self.assertRaises(OverflowError, time.asctime,
+                          (TIME_MINYEAR - 1,) + (0,) * 8)
         self.assertRaises(TypeError, time.asctime, 0)
         self.assertRaises(TypeError, time.asctime, ())
         self.assertRaises(TypeError, time.asctime, (0,) * 10)
@@ -290,6 +297,24 @@
         t1 = time.mktime(lt1)
         self.assertAlmostEqual(t1, t0, delta=0.2)
 
+    def test_mktime(self):
+        # Issue #1726687
+        for t in (-2, -1, 0, 1):
+            try:
+                tt = time.localtime(t)
+            except (OverflowError, ValueError):
+                pass
+            else:
+                self.assertEqual(time.mktime(tt), t)
+        # It may not be possible to reliably make mktime return error
+        # on all platfom.  This will make sure that no other exception
+        # than OverflowError is raised for an extreme value.
+        try:
+            time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))
+        except OverflowError:
+            pass
+
+
 class TestLocale(unittest.TestCase):
     def setUp(self):
         self.oldloc = locale.setlocale(locale.LC_ALL)
@@ -346,6 +371,28 @@
                 return time.strftime('%4Y', (y,) + (0,) * 8)
             self.test_year('%04d', func=year4d)
 
+    def skip_if_not_supported(y):
+        msg = "strftime() is limited to [1; 9999] with Visual Studio"
+        # Check that it doesn't crash for year > 9999
+        try:
+            time.strftime('%Y', (y,) + (0,) * 8)
+        except ValueError:
+            cond = False
+        else:
+            cond = True
+        return unittest.skipUnless(cond, msg)
+
+    @skip_if_not_supported(10000)
+    def test_large_year(self):
+        return super().test_large_year()
+
+    @skip_if_not_supported(0)
+    def test_negative(self):
+        return super().test_negative()
+
+    del skip_if_not_supported
+
+
 class _Test4dYear(_BaseYearTest):
     _format = '%d'
 
@@ -360,44 +407,19 @@
         self.assertEqual(func(9999), fmt % 9999)
 
     def test_large_year(self):
-        # Check that it doesn't crash for year > 9999
-        try:
-            text = self.yearstr(12345)
-        except ValueError:
-            # strftime() is limited to [1; 9999] with Visual Studio
-            return
-        self.assertEqual(text, '12345')
+        self.assertEqual(self.yearstr(12345), '12345')
         self.assertEqual(self.yearstr(123456789), '123456789')
+        self.assertEqual(self.yearstr(TIME_MAXYEAR), str(TIME_MAXYEAR))
+        self.assertRaises(OverflowError, self.yearstr, TIME_MAXYEAR + 1)
 
     def test_negative(self):
-        try:
-            text = self.yearstr(-1)
-        except ValueError:
-            # strftime() is limited to [1; 9999] with Visual Studio
-            return
-        self.assertEqual(text, self._format % -1)
-
+        self.assertEqual(self.yearstr(-1), self._format % -1)
         self.assertEqual(self.yearstr(-1234), '-1234')
         self.assertEqual(self.yearstr(-123456), '-123456')
+        self.assertEqual(self.yearstr(TIME_MINYEAR), str(TIME_MINYEAR))
+        self.assertRaises(OverflowError, self.yearstr, TIME_MINYEAR - 1)
 
 
-    def test_mktime(self):
-        # Issue #1726687
-        for t in (-2, -1, 0, 1):
-            try:
-                tt = time.localtime(t)
-            except (OverflowError, ValueError):
-                pass
-            else:
-                self.assertEqual(time.mktime(tt), t)
-        # It may not be possible to reliably make mktime return error
-        # on all platfom.  This will make sure that no other exception
-        # than OverflowError is raised for an extreme value.
-        try:
-            time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))
-        except OverflowError:
-            pass
-
 class TestAsctime4dyear(_TestAsctimeYear, _Test4dYear):
     pass
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 15:12:44 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 15:12:44 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_temporary_tests_to_trou?=
 =?utf8?q?bleshoot_issue_=2313309_on_Gentoo_buildbot=2E?=
Message-ID: <E1RLF52-0005SU-9v@dinsdale.python.org>

http://hg.python.org/cpython/rev/2771f7e96a52
changeset:   73282:2771f7e96a52
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 15:12:34 2011 +0100
summary:
  Add temporary tests to troubleshoot issue #13309 on Gentoo buildbot.

files:
  Lib/test/test_time.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -161,6 +161,10 @@
                 self.fail("conversion specifier %r failed with '%s' input." %
                           (format, strf_output))
 
+    # XXX Temporary tests to troubleshoot issue #13309 on buildbots
+    test_aaa_strptime = test_strptime
+    test_bbb_strptime = test_strptime
+
     def test_strptime_bytes(self):
         # Make sure only strings are accepted as arguments to strptime.
         self.assertRaises(TypeError, time.strptime, b'2009', "%Y")

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 16:07:34 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 16:07:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Troubleshoot_issue_=2313309?=
	=?utf8?q?_on_Gentoo_buildbot=2E?=
Message-ID: <E1RLFw6-0002Pm-JF@dinsdale.python.org>

http://hg.python.org/cpython/rev/bb0ae7df08f8
changeset:   73283:bb0ae7df08f8
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 16:07:23 2011 +0100
summary:
  Troubleshoot issue #13309 on Gentoo buildbot.

files:
  Lib/test/test_time.py |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -162,8 +162,8 @@
                           (format, strf_output))
 
     # XXX Temporary tests to troubleshoot issue #13309 on buildbots
-    test_aaa_strptime = test_strptime
-    test_bbb_strptime = test_strptime
+    test_maa_strptime = test_strptime
+    test_mzz_strptime = test_strptime
 
     def test_strptime_bytes(self):
         # Make sure only strings are accepted as arguments to strptime.
@@ -310,6 +310,8 @@
                 pass
             else:
                 self.assertEqual(time.mktime(tt), t)
+            self.assertNotEqual(time.strftime('%Z', time.gmtime(self.t)), 'LMT',
+                                "strftime bug after processing t = %s" % t)
         # It may not be possible to reliably make mktime return error
         # on all platfom.  This will make sure that no other exception
         # than OverflowError is raised for an extreme value.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 16:20:44 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 01 Nov 2011 16:20:44 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_issue13287_-_Define_=5F=5Fa?=
 =?utf8?q?ll=5F=5F_for_urllib=2Erequest_and_urllib=2Eerror_and_expose_only?=
Message-ID: <E1RLG8q-0003sd-AU@dinsdale.python.org>

http://hg.python.org/cpython/rev/70dedd8ce8f3
changeset:   73284:70dedd8ce8f3
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Nov 01 23:20:31 2011 +0800
summary:
  issue13287 - Define __all__ for urllib.request and urllib.error and expose only
the relevant module. Other cleanup improvements. Patch by flox.

files:
  Lib/test/test_urllib2.py |  12 ++++++++
  Lib/urllib/error.py      |   3 ++
  Lib/urllib/request.py    |  40 +++++++++++++++------------
  Misc/NEWS                |   3 ++
  4 files changed, 40 insertions(+), 18 deletions(-)


diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py
--- a/Lib/test/test_urllib2.py
+++ b/Lib/test/test_urllib2.py
@@ -19,6 +19,18 @@
 # parse_keqv_list, parse_http_list, HTTPDigestAuthHandler
 
 class TrivialTests(unittest.TestCase):
+
+    def test___all__(self):
+        # Verify which names are exposed
+        for module in 'request', 'response', 'parse', 'error', 'robotparser':
+            context = {}
+            exec('from urllib.%s import *' % module, context)
+            del context['__builtins__']
+            for k, v in context.items():
+                self.assertEqual(v.__module__, 'urllib.%s' % module,
+                    "%r is exposed in 'urllib.%s' but defined in %r" %
+                    (k, module, v.__module__))
+
     def test_trivial(self):
         # A couple trivial tests
 
diff --git a/Lib/urllib/error.py b/Lib/urllib/error.py
--- a/Lib/urllib/error.py
+++ b/Lib/urllib/error.py
@@ -13,6 +13,9 @@
 
 import urllib.response
 
+__all__ = ['URLError', 'HTTPError', 'ContentTooShortError']
+
+
 # do these error classes make sense?
 # make sure all of the IOError stuff is overridden.  we just want to be
 # subtypes.
diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py
--- a/Lib/urllib/request.py
+++ b/Lib/urllib/request.py
@@ -89,7 +89,6 @@
 import io
 import os
 import posixpath
-import random
 import re
 import socket
 import sys
@@ -111,6 +110,22 @@
 else:
     _have_ssl = True
 
+__all__ = [
+    # Classes
+    'Request', 'OpenerDirector', 'BaseHandler', 'HTTPDefaultErrorHandler',
+    'HTTPRedirectHandler', 'HTTPCookieProcessor', 'ProxyHandler',
+    'HTTPPasswordMgr', 'HTTPPasswordMgrWithDefaultRealm',
+    'AbstractBasicAuthHandler', 'HTTPBasicAuthHandler', 'ProxyBasicAuthHandler',
+    'AbstractDigestAuthHandler', 'HTTPDigestAuthHandler', 'ProxyDigestAuthHandler',
+    'HTTPHandler', 'FileHandler', 'FTPHandler', 'CacheFTPHandler',
+    'UnknownHandler', 'HTTPErrorProcessor',
+    # Functions
+    'urlopen', 'install_opener', 'build_opener',
+    'pathname2url', 'url2pathname', 'getproxies',
+    # Legacy interface
+    'urlretrieve', 'urlcleanup', 'URLopener', 'FancyURLopener',
+]
+
 # used in User-Agent header sent
 __version__ = sys.version[:3]
 
@@ -885,9 +900,9 @@
         return response
 
 
-def randombytes(n):
-    """Return n random bytes."""
-    return os.urandom(n)
+# Return n random bytes.
+_randombytes = os.urandom
+
 
 class AbstractDigestAuthHandler:
     # Digest authentication is specified in RFC 2617.
@@ -951,7 +966,7 @@
         # authentication, and to provide some message integrity protection.
         # This isn't a fabulous effort, but it's probably Good Enough.
         s = "%s:%s:%s:" % (self.nonce_count, nonce, time.ctime())
-        b = s.encode("ascii") + randombytes(8)
+        b = s.encode("ascii") + _randombytes(8)
         dig = hashlib.sha1(b).hexdigest()
         return dig[:16]
 
@@ -1171,7 +1186,6 @@
     http_request = AbstractHTTPHandler.do_request_
 
 if hasattr(http.client, 'HTTPSConnection'):
-    import ssl
 
     class HTTPSHandler(AbstractHTTPHandler):
 
@@ -1677,13 +1691,11 @@
         if not host: raise IOError('http error', 'no host given')
 
         if proxy_passwd:
-            import base64
             proxy_auth = base64.b64encode(proxy_passwd.encode()).decode('ascii')
         else:
             proxy_auth = None
 
         if user_passwd:
-            import base64
             auth = base64.b64encode(user_passwd.encode()).decode('ascii')
         else:
             auth = None
@@ -1773,8 +1785,8 @@
 
     def open_local_file(self, url):
         """Use local file."""
-        import mimetypes, email.utils
-        from io import StringIO
+        import email.utils
+        import mimetypes
         host, file = splithost(url)
         localname = url2pathname(file)
         try:
@@ -1806,7 +1818,6 @@
         if not isinstance(url, str):
             raise URLError('ftp error', 'proxy support for ftp protocol currently not implemented')
         import mimetypes
-        from io import StringIO
         host, path = splithost(url)
         if not host: raise URLError('ftp error', 'no host given')
         host, port = splitport(host)
@@ -1888,7 +1899,6 @@
                                             time.gmtime(time.time())))
         msg.append('Content-type: %s' % type)
         if encoding == 'base64':
-            import base64
             # XXX is this encoding/decoding ok?
             data = base64.decodebytes(data.encode('ascii')).decode('latin-1')
         else:
@@ -1984,7 +1994,6 @@
             URLopener.http_error_default(self, url, fp,
                                          errcode, errmsg, headers)
         stuff = headers['www-authenticate']
-        import re
         match = re.match('[ \t]*([^ \t]+)[ \t]+realm="([^"]*)"', stuff)
         if not match:
             URLopener.http_error_default(self, url, fp,
@@ -2010,7 +2019,6 @@
             URLopener.http_error_default(self, url, fp,
                                          errcode, errmsg, headers)
         stuff = headers['proxy-authenticate']
-        import re
         match = re.match('[ \t]*([^ \t]+)[ \t]+realm="([^"]*)"', stuff)
         if not match:
             URLopener.http_error_default(self, url, fp,
@@ -2302,8 +2310,6 @@
       'exceptions': ['foo.bar', '*.bar.com', '127.0.0.1', '10.1', '10.0/16']
     }
     """
-    import re
-    import socket
     from fnmatch import fnmatch
 
     hostonly, port = splitport(host)
@@ -2406,7 +2412,6 @@
                     for p in proxyServer.split(';'):
                         protocol, address = p.split('=', 1)
                         # See if address has a type:// prefix
-                        import re
                         if not re.match('^([^/:]+)://', address):
                             address = '%s://%s' % (protocol, address)
                         proxies[protocol] = address
@@ -2438,7 +2443,6 @@
     def proxy_bypass_registry(host):
         try:
             import winreg
-            import re
         except ImportError:
             # Std modules, so should be around - but you never know!
             return 0
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,9 @@
 Library
 -------
 
+- Issue #13287: urllib.request and urllib.error now contains a __all__ and
+  exposes only relevant Classes, Functions. Patch by Florent Xicluna.
+
 - Issue #670664: Fix HTMLParser to correctly handle the content of
   ``<script>...</script>`` and ``<style>...</style>``.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 16:49:57 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 01 Nov 2011 16:49:57 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Append_HTTPSHandler_to_=5F?=
 =?utf8?q?=5Fall=5F=5F_when_it_is_available=2E?=
Message-ID: <E1RLGb7-0006qH-3F@dinsdale.python.org>

http://hg.python.org/cpython/rev/e763999592a1
changeset:   73285:e763999592a1
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Nov 01 23:49:46 2011 +0800
summary:
  Append HTTPSHandler to __all__ when it is available.

files:
  Lib/urllib/request.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py
--- a/Lib/urllib/request.py
+++ b/Lib/urllib/request.py
@@ -1200,6 +1200,8 @@
 
         https_request = AbstractHTTPHandler.do_request_
 
+    __all__.append(HTTPSHandler)
+
 class HTTPCookieProcessor(BaseHandler):
     def __init__(self, cookiejar=None):
         import http.cookiejar

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 16:58:08 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Tue, 01 Nov 2011 16:58:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=27HTTPSHandler=27_and_not_?=
	=?utf8?q?the_type=2E?=
Message-ID: <E1RLGj2-0007p2-KH@dinsdale.python.org>

http://hg.python.org/cpython/rev/b388bee313a8
changeset:   73286:b388bee313a8
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Tue Nov 01 23:57:57 2011 +0800
summary:
  'HTTPSHandler' and not the type.

files:
  Lib/urllib/request.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py
--- a/Lib/urllib/request.py
+++ b/Lib/urllib/request.py
@@ -1200,7 +1200,7 @@
 
         https_request = AbstractHTTPHandler.do_request_
 
-    __all__.append(HTTPSHandler)
+    __all__.append('HTTPSHandler')
 
 class HTTPCookieProcessor(BaseHandler):
     def __init__(self, cookiejar=None):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 17:06:47 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 17:06:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_temporary_tests_wit?=
 =?utf8?q?h_the_real_test_case_for_issue_=2313309_on_Gentoo=2E?=
Message-ID: <E1RLGrP-0000Nw-3Z@dinsdale.python.org>

http://hg.python.org/cpython/rev/5b1e1967ea9d
changeset:   73287:5b1e1967ea9d
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 16:58:54 2011 +0100
summary:
  Replace temporary tests with the real test case for issue #13309 on Gentoo.

files:
  Lib/test/test_time.py |  16 +++++++++-------
  1 files changed, 9 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -161,10 +161,6 @@
                 self.fail("conversion specifier %r failed with '%s' input." %
                           (format, strf_output))
 
-    # XXX Temporary tests to troubleshoot issue #13309 on buildbots
-    test_maa_strptime = test_strptime
-    test_mzz_strptime = test_strptime
-
     def test_strptime_bytes(self):
         # Make sure only strings are accepted as arguments to strptime.
         self.assertRaises(TypeError, time.strptime, b'2009', "%Y")
@@ -301,7 +297,8 @@
         t1 = time.mktime(lt1)
         self.assertAlmostEqual(t1, t0, delta=0.2)
 
-    def test_mktime(self):
+    # XXX run last to work around issue #13309 on Gentoo
+    def test_ZZZ_mktime(self):
         # Issue #1726687
         for t in (-2, -1, 0, 1):
             try:
@@ -310,8 +307,9 @@
                 pass
             else:
                 self.assertEqual(time.mktime(tt), t)
-            self.assertNotEqual(time.strftime('%Z', time.gmtime(self.t)), 'LMT',
-                                "strftime bug after processing t = %s" % t)
+        tt = time.gmtime(self.t)
+        tzname = time.strftime('%Z', tt)
+        self.assertNotEqual(tzname, 'LMT')
         # It may not be possible to reliably make mktime return error
         # on all platfom.  This will make sure that no other exception
         # than OverflowError is raised for an extreme value.
@@ -319,6 +317,10 @@
             time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))
         except OverflowError:
             pass
+        msg = "Issue #13309: the '%Z' specifier reports wrong timezone"
+        self.assertEqual(time.strftime('%Z', tt), tzname, msg)
+        tt = time.gmtime(self.t)
+        self.assertEqual(time.strftime('%Z', tt), tzname, msg)
 
 
 class TestLocale(unittest.TestCase):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 17:42:33 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 17:42:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Lowercase_the_test_name=2C_?=
	=?utf8?q?to_run_last=2E?=
Message-ID: <E1RLHQ1-0004Nl-Au@dinsdale.python.org>

http://hg.python.org/cpython/rev/a994b0cbe49d
changeset:   73288:a994b0cbe49d
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 17:42:24 2011 +0100
summary:
  Lowercase the test name, to run last.

files:
  Lib/test/test_time.py |  7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -298,7 +298,7 @@
         self.assertAlmostEqual(t1, t0, delta=0.2)
 
     # XXX run last to work around issue #13309 on Gentoo
-    def test_ZZZ_mktime(self):
+    def test_zzz_mktime(self):
         # Issue #1726687
         for t in (-2, -1, 0, 1):
             try:
@@ -317,9 +317,8 @@
             time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))
         except OverflowError:
             pass
-        msg = "Issue #13309: the '%Z' specifier reports wrong timezone"
-        self.assertEqual(time.strftime('%Z', tt), tzname, msg)
-        tt = time.gmtime(self.t)
+        msg = "Issue #13309: the '%Z' specifier reports erroneous timezone"
+        msg += " after time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))."
         self.assertEqual(time.strftime('%Z', tt), tzname, msg)
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 17:58:31 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 17:58:31 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Make_sure_that_?=
 =?utf8?q?the_tolerant_parser_still_parses_valid_HTML_correctly=2E?=
Message-ID: <E1RLHfT-00068u-BW@dinsdale.python.org>

http://hg.python.org/cpython/rev/8f9f490b4777
changeset:   73289:8f9f490b4777
branch:      3.2
parent:      73279:4404e784918c
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 18:57:15 2011 +0200
summary:
  Make sure that the tolerant parser still parses valid HTML correctly.

files:
  Lib/test/test_htmlparser.py |  36 +++++++++++++-----------
  1 files changed, 19 insertions(+), 17 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -72,9 +72,12 @@
 
 class TestCaseBase(unittest.TestCase):
 
+    def get_collector(self):
+        raise NotImplementedError
+
     def _run_check(self, source, expected_events, collector=None):
         if collector is None:
-            collector = EventCollector()
+            collector = self.get_collector()
         parser = collector
         for s in source:
             parser.feed(s)
@@ -96,7 +99,10 @@
         self.assertRaises(html.parser.HTMLParseError, parse)
 
 
-class HTMLParserTestCase(TestCaseBase):
+class HTMLParserStrictTestCase(TestCaseBase):
+
+    def get_collector(self):
+        return EventCollector(strict=True)
 
     def test_processing_instruction_only(self):
         self._run_check("<?processing instruction>", [
@@ -353,12 +359,11 @@
 
 
     def test_entityrefs_in_attributes(self):
-        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>", [
-                ("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])
-                ])
+        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
+                        [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
 
 
-class HTMLParserTolerantTestCase(TestCaseBase):
+class HTMLParserTolerantTestCase(HTMLParserStrictTestCase):
 
     def get_collector(self):
         return EventCollector(strict=False)
@@ -374,8 +379,7 @@
                              ('endtag', 'a'),
                              ('endtag', 'html'),
                              ('data', '\n<img src="URL><//img></html'),
-                             ('endtag', 'html')],
-                        collector=self.get_collector())
+                             ('endtag', 'html')])
 
     def test_with_unquoted_attributes(self):
         # see #12008
@@ -399,22 +403,19 @@
             ('starttag', 'span', [('class', 'en')]), ('data', ' library'),
             ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table')
         ]
-
-        self._run_check(html, expected, collector=self.get_collector())
+        self._run_check(html, expected)
 
     def test_comma_between_attributes(self):
         self._run_check('<form action="/xxx.php?a=1&amp;b=2&amp", '
                         'method="post">', [
                             ('starttag', 'form',
                                 [('action', '/xxx.php?a=1&b=2&amp'),
-                                 ('method', 'post')])],
-                        collector=self.get_collector())
+                                 ('method', 'post')])])
 
     def test_weird_chars_in_unquoted_attribute_values(self):
         self._run_check('<form action=bogus|&#()value>', [
                             ('starttag', 'form',
-                                [('action', 'bogus|&#()value')])],
-                        collector=self.get_collector())
+                                [('action', 'bogus|&#()value')])])
 
     def test_correct_detection_of_start_tags(self):
         # see #13273
@@ -436,7 +437,7 @@
             ('endtag', 'b'),
             ('endtag', 'div')
         ]
-        self._run_check(html, expected, collector=self.get_collector())
+        self._run_check(html, expected)
 
         html = '<div style="", foo = "bar" ><b>The <a href="some_url">rain</a>'
         expected = [
@@ -447,7 +448,7 @@
             ('data', 'rain'),
             ('endtag', 'a'),
         ]
-        self._run_check(html, expected, collector=self.get_collector())
+        self._run_check(html, expected)
 
     def test_unescape_function(self):
         p = html.parser.HTMLParser()
@@ -456,8 +457,9 @@
         # see #12888
         self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
+
 def test_main():
-    support.run_unittest(HTMLParserTestCase, HTMLParserTolerantTestCase)
+    support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase)
 
 
 if __name__ == "__main__":

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 17:58:33 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 01 Nov 2011 17:58:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_test_improvements_from_3=2E2=2E?=
Message-ID: <E1RLHfV-000695-Nh@dinsdale.python.org>

http://hg.python.org/cpython/rev/30a20f69a919
changeset:   73290:30a20f69a919
parent:      73288:a994b0cbe49d
parent:      73289:8f9f490b4777
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 01 18:58:19 2011 +0200
summary:
  Merge test improvements from 3.2.

files:
  Lib/test/test_htmlparser.py |  36 +++++++++++++-----------
  1 files changed, 19 insertions(+), 17 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -72,9 +72,12 @@
 
 class TestCaseBase(unittest.TestCase):
 
+    def get_collector(self):
+        raise NotImplementedError
+
     def _run_check(self, source, expected_events, collector=None):
         if collector is None:
-            collector = EventCollector()
+            collector = self.get_collector()
         parser = collector
         for s in source:
             parser.feed(s)
@@ -96,7 +99,10 @@
         self.assertRaises(html.parser.HTMLParseError, parse)
 
 
-class HTMLParserTestCase(TestCaseBase):
+class HTMLParserStrictTestCase(TestCaseBase):
+
+    def get_collector(self):
+        return EventCollector(strict=True)
 
     def test_processing_instruction_only(self):
         self._run_check("<?processing instruction>", [
@@ -353,12 +359,11 @@
 
 
     def test_entityrefs_in_attributes(self):
-        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>", [
-                ("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])
-                ])
+        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
+                        [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
 
 
-class HTMLParserTolerantTestCase(TestCaseBase):
+class HTMLParserTolerantTestCase(HTMLParserStrictTestCase):
 
     def get_collector(self):
         return EventCollector(strict=False)
@@ -374,8 +379,7 @@
                              ('endtag', 'a'),
                              ('endtag', 'html'),
                              ('data', '\n<img src="URL><//img></html'),
-                             ('endtag', 'html')],
-                        collector=self.get_collector())
+                             ('endtag', 'html')])
 
     def test_with_unquoted_attributes(self):
         # see #12008
@@ -399,22 +403,19 @@
             ('starttag', 'span', [('class', 'en')]), ('data', ' library'),
             ('endtag', 'span'), ('endtag', 'a'), ('endtag', 'table')
         ]
-
-        self._run_check(html, expected, collector=self.get_collector())
+        self._run_check(html, expected)
 
     def test_comma_between_attributes(self):
         self._run_check('<form action="/xxx.php?a=1&amp;b=2&amp", '
                         'method="post">', [
                             ('starttag', 'form',
                                 [('action', '/xxx.php?a=1&b=2&amp'),
-                                 ('method', 'post')])],
-                        collector=self.get_collector())
+                                 ('method', 'post')])])
 
     def test_weird_chars_in_unquoted_attribute_values(self):
         self._run_check('<form action=bogus|&#()value>', [
                             ('starttag', 'form',
-                                [('action', 'bogus|&#()value')])],
-                        collector=self.get_collector())
+                                [('action', 'bogus|&#()value')])])
 
     def test_correct_detection_of_start_tags(self):
         # see #13273
@@ -436,7 +437,7 @@
             ('endtag', 'b'),
             ('endtag', 'div')
         ]
-        self._run_check(html, expected, collector=self.get_collector())
+        self._run_check(html, expected)
 
         html = '<div style="", foo = "bar" ><b>The <a href="some_url">rain</a>'
         expected = [
@@ -447,7 +448,7 @@
             ('data', 'rain'),
             ('endtag', 'a'),
         ]
-        self._run_check(html, expected, collector=self.get_collector())
+        self._run_check(html, expected)
 
     def test_unescape_function(self):
         p = html.parser.HTMLParser()
@@ -456,8 +457,9 @@
         # see #12888
         self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
+
 def test_main():
-    support.run_unittest(HTMLParserTestCase, HTMLParserTolerantTestCase)
+    support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase)
 
 
 if __name__ == "__main__":

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 23:35:17 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 23:35:17 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2VzICMyODky?=
 =?utf8?q?=3A_preserve_iterparse_events_in_case_of_SyntaxError=2E?=
Message-ID: <E1RLMvN-0006W4-Gl@dinsdale.python.org>

http://hg.python.org/cpython/rev/23ffaf975267
changeset:   73291:23ffaf975267
branch:      3.2
parent:      73289:8f9f490b4777
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 23:31:09 2011 +0100
summary:
  Closes #2892: preserve iterparse events in case of SyntaxError.

files:
  Lib/test/test_xml_etree.py   |   1 +
  Lib/xml/etree/ElementTree.py |  38 ++++++++++++++---------
  Misc/NEWS                    |   2 +
  Modules/_elementtree.c       |  38 ++++++++++++++---------
  4 files changed, 49 insertions(+), 30 deletions(-)


diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py
--- a/Lib/test/test_xml_etree.py
+++ b/Lib/test/test_xml_etree.py
@@ -754,6 +754,7 @@
     ...     print(action, elem.tag)
     ... except ET.ParseError as v:
     ...   print(v)
+    end document
     junk after document element: line 1, column 12
     """
 
diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py
--- a/Lib/xml/etree/ElementTree.py
+++ b/Lib/xml/etree/ElementTree.py
@@ -1250,6 +1250,7 @@
         self._close_file = close_source
         self._events = []
         self._index = 0
+        self._error = None
         self.root = self._root = None
         self._parser = parser
         # wire up the parser for event reporting
@@ -1291,24 +1292,31 @@
         while 1:
             try:
                 item = self._events[self._index]
+                self._index += 1
+                return item
             except IndexError:
-                if self._parser is None:
-                    self.root = self._root
-                    if self._close_file:
-                        self._file.close()
-                    raise StopIteration
-                # load event buffer
-                del self._events[:]
-                self._index = 0
-                data = self._file.read(16384)
-                if data:
+                pass
+            if self._error:
+                e = self._error
+                self._error = None
+                raise e
+            if self._parser is None:
+                self.root = self._root
+                if self._close_file:
+                    self._file.close()
+                raise StopIteration
+            # load event buffer
+            del self._events[:]
+            self._index = 0
+            data = self._file.read(16384)
+            if data:
+                try:
                     self._parser.feed(data)
-                else:
-                    self._root = self._parser.close()
-                    self._parser = None
+                except SyntaxError as exc:
+                    self._error = exc
             else:
-                self._index = self._index + 1
-                return item
+                self._root = self._parser.close()
+                self._parser = None
 
     def __iter__(self):
         return self
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,6 +66,8 @@
 Library
 -------
 
+- Issue #2892: preserve iterparse events in case of SyntaxError.
+
 - Issue #670664: Fix HTMLParser to correctly handle the content of
   ``<script>...</script>`` and ``<style>...</style>``.
 
diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
--- a/Modules/_elementtree.c
+++ b/Modules/_elementtree.c
@@ -3000,6 +3000,7 @@
         "  self._file = file\n"
         "  self._events = []\n"
         "  self._index = 0\n"
+        "  self._error = None\n"
         "  self.root = self._root = None\n"
         "  b = cElementTree.TreeBuilder()\n"
         "  self._parser = cElementTree.XMLParser(b)\n"
@@ -3008,24 +3009,31 @@
         "  while 1:\n"
         "    try:\n"
         "      item = self._events[self._index]\n"
+        "      self._index += 1\n"
+        "      return item\n"
         "    except IndexError:\n"
-        "      if self._parser is None:\n"
-        "        self.root = self._root\n"
-        "        if self._close_file:\n"
-        "          self._file.close()\n"
-        "        raise StopIteration\n"
-        "      # load event buffer\n"
-        "      del self._events[:]\n"
-        "      self._index = 0\n"
-        "      data = self._file.read(16384)\n"
-        "      if data:\n"
+        "      pass\n"
+        "    if self._error:\n"
+        "      e = self._error\n"
+        "      self._error = None\n"
+        "      raise e\n"
+        "    if self._parser is None:\n"
+        "      self.root = self._root\n"
+        "      if self._close_file:\n"
+        "        self._file.close()\n"
+        "      raise StopIteration\n"
+        "    # load event buffer\n"
+        "    del self._events[:]\n"
+        "    self._index = 0\n"
+        "    data = self._file.read(16384)\n"
+        "    if data:\n"
+        "      try:\n"
         "        self._parser.feed(data)\n"
-        "      else:\n"
-        "        self._root = self._parser.close()\n"
-        "        self._parser = None\n"
+        "      except SyntaxError as exc:\n"
+        "        self._error = exc\n"
         "    else:\n"
-        "      self._index = self._index + 1\n"
-        "      return item\n"
+        "      self._root = self._parser.close()\n"
+        "      self._parser = None\n"
         " def __iter__(self):\n"
         "  return self\n"
         "cElementTree.iterparse = iterparse\n"

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 23:35:18 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 23:35:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_issue_=232892?=
Message-ID: <E1RLMvO-0006W8-Dt@dinsdale.python.org>

http://hg.python.org/cpython/rev/ca1e2cf2947b
changeset:   73292:ca1e2cf2947b
parent:      73290:30a20f69a919
parent:      73291:23ffaf975267
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 23:33:14 2011 +0100
summary:
  Merge 3.2: issue #2892

files:
  Lib/test/test_xml_etree.py   |   1 +
  Lib/xml/etree/ElementTree.py |  38 ++++++++++++++---------
  Misc/NEWS                    |   2 +
  Modules/_elementtree.c       |  38 ++++++++++++++---------
  4 files changed, 49 insertions(+), 30 deletions(-)


diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py
--- a/Lib/test/test_xml_etree.py
+++ b/Lib/test/test_xml_etree.py
@@ -754,6 +754,7 @@
     ...     print(action, elem.tag)
     ... except ET.ParseError as v:
     ...   print(v)
+    end document
     junk after document element: line 1, column 12
     """
 
diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py
--- a/Lib/xml/etree/ElementTree.py
+++ b/Lib/xml/etree/ElementTree.py
@@ -1250,6 +1250,7 @@
         self._close_file = close_source
         self._events = []
         self._index = 0
+        self._error = None
         self.root = self._root = None
         self._parser = parser
         # wire up the parser for event reporting
@@ -1291,24 +1292,31 @@
         while 1:
             try:
                 item = self._events[self._index]
+                self._index += 1
+                return item
             except IndexError:
-                if self._parser is None:
-                    self.root = self._root
-                    if self._close_file:
-                        self._file.close()
-                    raise StopIteration
-                # load event buffer
-                del self._events[:]
-                self._index = 0
-                data = self._file.read(16384)
-                if data:
+                pass
+            if self._error:
+                e = self._error
+                self._error = None
+                raise e
+            if self._parser is None:
+                self.root = self._root
+                if self._close_file:
+                    self._file.close()
+                raise StopIteration
+            # load event buffer
+            del self._events[:]
+            self._index = 0
+            data = self._file.read(16384)
+            if data:
+                try:
                     self._parser.feed(data)
-                else:
-                    self._root = self._parser.close()
-                    self._parser = None
+                except SyntaxError as exc:
+                    self._error = exc
             else:
-                self._index = self._index + 1
-                return item
+                self._root = self._parser.close()
+                self._parser = None
 
     def __iter__(self):
         return self
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,8 @@
 Library
 -------
 
+- Issue #2892: preserve iterparse events in case of SyntaxError.
+
 - Issue #13287: urllib.request and urllib.error now contains a __all__ and
   exposes only relevant Classes, Functions. Patch by Florent Xicluna.
 
diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
--- a/Modules/_elementtree.c
+++ b/Modules/_elementtree.c
@@ -3016,6 +3016,7 @@
         "  self._file = file\n"
         "  self._events = []\n"
         "  self._index = 0\n"
+        "  self._error = None\n"
         "  self.root = self._root = None\n"
         "  b = cElementTree.TreeBuilder()\n"
         "  self._parser = cElementTree.XMLParser(b)\n"
@@ -3024,24 +3025,31 @@
         "  while 1:\n"
         "    try:\n"
         "      item = self._events[self._index]\n"
+        "      self._index += 1\n"
+        "      return item\n"
         "    except IndexError:\n"
-        "      if self._parser is None:\n"
-        "        self.root = self._root\n"
-        "        if self._close_file:\n"
-        "          self._file.close()\n"
-        "        raise StopIteration\n"
-        "      # load event buffer\n"
-        "      del self._events[:]\n"
-        "      self._index = 0\n"
-        "      data = self._file.read(16384)\n"
-        "      if data:\n"
+        "      pass\n"
+        "    if self._error:\n"
+        "      e = self._error\n"
+        "      self._error = None\n"
+        "      raise e\n"
+        "    if self._parser is None:\n"
+        "      self.root = self._root\n"
+        "      if self._close_file:\n"
+        "        self._file.close()\n"
+        "      raise StopIteration\n"
+        "    # load event buffer\n"
+        "    del self._events[:]\n"
+        "    self._index = 0\n"
+        "    data = self._file.read(16384)\n"
+        "    if data:\n"
+        "      try:\n"
         "        self._parser.feed(data)\n"
-        "      else:\n"
-        "        self._root = self._parser.close()\n"
-        "        self._parser = None\n"
+        "      except SyntaxError as exc:\n"
+        "        self._error = exc\n"
         "    else:\n"
-        "      self._index = self._index + 1\n"
-        "      return item\n"
+        "      self._root = self._parser.close()\n"
+        "      self._parser = None\n"
         " def __iter__(self):\n"
         "  return self\n"
         "cElementTree.iterparse = iterparse\n"

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 23:35:19 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 23:35:19 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzI4OTI6?=
 =?utf8?q?_preserve_iterparse_events_in_case_of_SyntaxError?=
Message-ID: <E1RLMvP-0006WD-6x@dinsdale.python.org>

http://hg.python.org/cpython/rev/e1dde980a92c
changeset:   73293:e1dde980a92c
branch:      2.7
parent:      73273:0a5eb57d5876
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 23:34:41 2011 +0100
summary:
  Issue #2892: preserve iterparse events in case of SyntaxError

files:
  Lib/test/test_xml_etree.py   |   1 +
  Lib/xml/etree/ElementTree.py |  38 ++++++++++++++---------
  Modules/_elementtree.c       |  38 ++++++++++++++---------
  3 files changed, 47 insertions(+), 30 deletions(-)


diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py
--- a/Lib/test/test_xml_etree.py
+++ b/Lib/test/test_xml_etree.py
@@ -739,6 +739,7 @@
     ... except ET.ParseError, v:
     ...   print v
     junk after document element: line 1, column 12
+    end document
     """
 
 def writefile():
diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py
--- a/Lib/xml/etree/ElementTree.py
+++ b/Lib/xml/etree/ElementTree.py
@@ -1210,6 +1210,7 @@
         self._close_file = close_source
         self._events = []
         self._index = 0
+        self._error = None
         self.root = self._root = None
         self._parser = parser
         # wire up the parser for event reporting
@@ -1255,24 +1256,31 @@
         while 1:
             try:
                 item = self._events[self._index]
+                self._index += 1
+                return item
             except IndexError:
-                if self._parser is None:
-                    self.root = self._root
-                    if self._close_file:
-                        self._file.close()
-                    raise StopIteration
-                # load event buffer
-                del self._events[:]
-                self._index = 0
-                data = self._file.read(16384)
-                if data:
+                pass
+            if self._error:
+                e = self._error
+                self._error = None
+                raise e
+            if self._parser is None:
+                self.root = self._root
+                if self._close_file:
+                    self._file.close()
+                raise StopIteration
+            # load event buffer
+            del self._events[:]
+            self._index = 0
+            data = self._file.read(16384)
+            if data:
+                try:
                     self._parser.feed(data)
-                else:
-                    self._root = self._parser.close()
-                    self._parser = None
+                except SyntaxError as exc:
+                    self._error = exc
             else:
-                self._index = self._index + 1
-                return item
+                self._root = self._parser.close()
+                self._parser = None
 
     def __iter__(self):
         return self
diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
--- a/Modules/_elementtree.c
+++ b/Modules/_elementtree.c
@@ -2970,6 +2970,7 @@
         "  self._file = file\n"
         "  self._events = []\n"
         "  self._index = 0\n"
+        "  self._error = None\n"
         "  self.root = self._root = None\n"
         "  b = cElementTree.TreeBuilder()\n"
         "  self._parser = cElementTree.XMLParser(b)\n"
@@ -2978,24 +2979,31 @@
         "  while 1:\n"
         "    try:\n"
         "      item = self._events[self._index]\n"
+        "      self._index += 1\n"
+        "      return item\n"
         "    except IndexError:\n"
-        "      if self._parser is None:\n"
-        "        self.root = self._root\n"
-        "        if self._close_file:\n"
-        "          self._file.close()\n"
-        "        raise StopIteration\n"
-        "      # load event buffer\n"
-        "      del self._events[:]\n"
-        "      self._index = 0\n"
-        "      data = self._file.read(16384)\n"
-        "      if data:\n"
+        "      pass\n"
+        "    if self._error:\n"
+        "      e = self._error\n"
+        "      self._error = None\n"
+        "      raise e\n"
+        "    if self._parser is None:\n"
+        "      self.root = self._root\n"
+        "      if self._close_file:\n"
+        "        self._file.close()\n"
+        "      raise StopIteration\n"
+        "    # load event buffer\n"
+        "    del self._events[:]\n"
+        "    self._index = 0\n"
+        "    data = self._file.read(16384)\n"
+        "    if data:\n"
+        "      try:\n"
         "        self._parser.feed(data)\n"
-        "      else:\n"
-        "        self._root = self._parser.close()\n"
-        "        self._parser = None\n"
+        "      except SyntaxError as exc:\n"
+        "        self._error = exc\n"
         "    else:\n"
-        "      self._index = self._index + 1\n"
-        "      return item\n"
+        "      self._root = self._parser.close()\n"
+        "      self._parser = None\n"
         " def __iter__(self):\n"
         "  return self\n"
         "cElementTree.iterparse = iterparse\n"

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 23:42:24 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 23:42:24 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Additional_tests_for_negati?=
	=?utf8?q?ve_years=2E?=
Message-ID: <E1RLN2G-000767-UV@dinsdale.python.org>

http://hg.python.org/cpython/rev/f666ecd5c2c5
changeset:   73294:f666ecd5c2c5
parent:      73292:ca1e2cf2947b
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 23:42:05 2011 +0100
summary:
  Additional tests for negative years.

files:
  Lib/test/test_time.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -423,6 +423,8 @@
         self.assertEqual(self.yearstr(-1), self._format % -1)
         self.assertEqual(self.yearstr(-1234), '-1234')
         self.assertEqual(self.yearstr(-123456), '-123456')
+        self.assertEqual(self.yearstr(-123456789), str(-123456789))
+        self.assertEqual(self.yearstr(-1234567890), str(-1234567890))
         self.assertEqual(self.yearstr(TIME_MINYEAR), str(TIME_MINYEAR))
         self.assertRaises(OverflowError, self.yearstr, TIME_MINYEAR - 1)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 23:48:04 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 23:48:04 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Missing_NEWS_en?=
 =?utf8?q?try_in_changeset_e1dde980a92c?=
Message-ID: <E1RLN7k-0007q2-Kl@dinsdale.python.org>

http://hg.python.org/cpython/rev/2ea38bff99f8
changeset:   73295:2ea38bff99f8
branch:      2.7
parent:      73293:e1dde980a92c
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 23:47:46 2011 +0100
summary:
  Missing NEWS entry in changeset e1dde980a92c

files:
  Misc/NEWS |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -74,6 +74,8 @@
 Library
 -------
 
+- Issue #2892: preserve iterparse events in case of SyntaxError.
+
 - Issue #670664: Fix HTMLParser to correctly handle the content of
   ``<script>...</script>`` and ``<style>...</style>``.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  1 23:54:32 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 01 Nov 2011 23:54:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_merge_in_ch?=
	=?utf8?q?angeset_e1dde980a92c=2E?=
Message-ID: <E1RLNE0-0008HW-Pm@dinsdale.python.org>

http://hg.python.org/cpython/rev/fd4236c787cb
changeset:   73296:fd4236c787cb
branch:      2.7
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 01 23:54:16 2011 +0100
summary:
  Fix merge in changeset e1dde980a92c.

files:
  Lib/test/test_xml_etree.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py
--- a/Lib/test/test_xml_etree.py
+++ b/Lib/test/test_xml_etree.py
@@ -738,8 +738,8 @@
     ...     print action, elem.tag
     ... except ET.ParseError, v:
     ...   print v
+    end document
     junk after document element: line 1, column 12
-    end document
     """
 
 def writefile():

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 00:26:29 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 02 Nov 2011 00:26:29 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Update_PEP_302_integration=2E_?=
	=?utf8?q?Drop_2=2E7_discussion=2E?=
Message-ID: <E1RLNiv-0002Va-HV@dinsdale.python.org>

http://hg.python.org/peps/rev/396a9ba620a1
changeset:   3979:396a9ba620a1
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Tue Nov 01 18:09:31 2011 +0100
summary:
  Update PEP 302 integration. Drop 2.7 discussion.

files:
  pep-0382.txt |  54 ++++++++++++++++-----------------------
  1 files changed, 22 insertions(+), 32 deletions(-)


diff --git a/pep-0382.txt b/pep-0382.txt
--- a/pep-0382.txt
+++ b/pep-0382.txt
@@ -94,11 +94,15 @@
 with ``.pyp`` (for Python package) contains a portion of a package.
 
 The import statement is extended so that computes the package's
-``__path__`` attribute for a package named ``P`` as consisting of all
-directories named ``P.pyp``, plus optionally a single directory named
-``P`` containing a file ``__init__.py``. If either of these are
-found on the path of the parent package (or sys.path for a top-level
-package), search for additional portions of the package continues.
+``__path__`` attribute for a package named ``P`` as consisting of
+optionally a single directory name ``P`` containing a file
+``__init__.py``, plus all directories named ``P.pyp``, in the order in
+which they are found in the parent's package ``__path__`` (or
+``sys.path``). If either of these are found, search for additional
+portions of the package continues.
+
+A directory may contain both a package in the ``P/__init__.py`` and
+the ``P.pyp`` form.
 
 No other change to the importing mechanism is made; searching modules
 (including __init__.py) will continue to stop at the first module
@@ -129,27 +133,22 @@
 algorithm. To do so, a finder used as a path hook must support a
 method:
 
-   finder.is_package_portion(fullname)
+   finder.find_package_portion(fullname)
 
 This method will be called in the same manner as find_module, and it
-must return True if there is a package portion with that name; False
-if fullname indicates a subpackage/submodule that is not a package
-portion; otherwise, it shall raise an ImportError.
+must return an string to be added to the package's ``__path__``.
+If the finder doesn't find a portion of the package, it shall return
+``None``. Raising ``AttributeError`` from above call will be treated
+as non-conformance with this PEP, and the exception will be ignored.
+All other exceptions are reported.
 
-If any \*.pyp directories are found, but no loader was returned from
-find_module, a package is created and initialized with the path.
+A finder may report both success from ``find_module`` and from
+``find_package_portion``, allowing for both a package containing
+an ``__init__.py`` and a portion of the same package.
 
-If a loader was return, but no \*.pyp directories, load_module is
-called as defined in PEP 302.
-
-If both \*.pyp directories where found, and a loader was returned, a
-new method is called on the loader:
-
-   loader.load_module_with_path(load_module, path)
-
-where the path parameter is the list that will become the __path__
-attribute of the new package.
-
+All strings returned from ``find_package_portion``, along with all
+path names of ``.pyp`` directories are added to the new package's
+``__path__``.
 
 Discussion
 ==========
@@ -181,7 +180,7 @@
 package as being a Python package, if there is no need to declare
 Python packages.
 
-packages can stop filling out the namespace package's __init__.py.  As
+Packages can stop filling out the namespace package's __init__.py.  As
 a consequence, extend_path and declare_namespace become obsolete.
 
 Namespace packages can start providing non-trivial __init__.py
@@ -195,15 +194,6 @@
 any other mechanism might cause portions to get added twice to
 __path__.
 
-It has been proposed to also add this feature to Python 2.7. Given
-that 2.x reaches its end-of-life, it is questionable whether the
-addition of the feature would really do more good than harm (in having
-users and tools starting to special-case 2.7). Prospective users of
-this feature are encouraged to comment on this particular question. In
-addition, Python 2.7 is in bug-fix mode now, so adding new features to
-it would be a violation of the established policies.
-
-
 Copyright
 =========
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed Nov  2 00:27:59 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 02 Nov 2011 00:27:59 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_Py=5FUNICODE=5Fstrc?=
 =?utf8?q?hr_with_PyUnicode=5FFindChar=2E?=
Message-ID: <E1RLNkN-0002du-Ee@dinsdale.python.org>

http://hg.python.org/cpython/rev/d133601583d5
changeset:   73297:d133601583d5
parent:      73294:f666ecd5c2c5
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Tue Nov 01 18:42:23 2011 +0100
summary:
  Replace Py_UNICODE_strchr with PyUnicode_FindChar.

files:
  Modules/_csv.c |  9 +++------
  1 files changed, 3 insertions(+), 6 deletions(-)


diff --git a/Modules/_csv.c b/Modules/_csv.c
--- a/Modules/_csv.c
+++ b/Modules/_csv.c
@@ -950,7 +950,6 @@
     DialectObj *dialect = self->dialect;
     int i;
     Py_ssize_t rec_len;
-    Py_UNICODE *lineterm;
 
 #define ADDCH(c) \
     do {\
@@ -959,10 +958,6 @@
         rec_len++;\
     } while(0)
 
-    lineterm = PyUnicode_AsUnicode(dialect->lineterminator);
-    if (lineterm == NULL)
-        return -1;
-
     rec_len = self->rec_len;
 
     /* If this is not the first field we need a field separator */
@@ -982,7 +977,9 @@
         if (c == dialect->delimiter ||
             c == dialect->escapechar ||
             c == dialect->quotechar  ||
-            Py_UNICODE_strchr(lineterm, c)) {
+            PyUnicode_FindChar(
+                dialect->lineterminator, c, 0,
+                PyUnicode_GET_LENGTH(dialect->lineterminator), 1) >= 0) {
             if (dialect->quoting == QUOTE_NONE)
                 want_escape = 1;
             else {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 01:28:28 2011
From: python-checkins at python.org (florent.xicluna)
Date: Wed, 02 Nov 2011 01:28:28 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313312=3A_skip_the_?=
 =?utf8?q?single_failing_value_for_now=2E?=
Message-ID: <E1RLOgu-0007Nf-Tr@dinsdale.python.org>

http://hg.python.org/cpython/rev/9cb1b85237a9
changeset:   73298:9cb1b85237a9
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Wed Nov 02 01:28:17 2011 +0100
summary:
  Issue #13312: skip the single failing value for now.

files:
  Lib/test/test_time.py |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -425,7 +425,9 @@
         self.assertEqual(self.yearstr(-123456), '-123456')
         self.assertEqual(self.yearstr(-123456789), str(-123456789))
         self.assertEqual(self.yearstr(-1234567890), str(-1234567890))
-        self.assertEqual(self.yearstr(TIME_MINYEAR), str(TIME_MINYEAR))
+        self.assertEqual(self.yearstr(TIME_MINYEAR + 1), str(TIME_MINYEAR + 1))
+        # On some platforms it loses the sign (issue #13312)
+        # self.assertEqual(self.yearstr(TIME_MINYEAR), str(TIME_MINYEAR))
         self.assertRaises(OverflowError, self.yearstr, TIME_MINYEAR - 1)
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 03:22:42 2011
From: python-checkins at python.org (florent.xicluna)
Date: Wed, 02 Nov 2011 03:22:42 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Actually=2C_there=27s_more_?=
 =?utf8?q?than_one_failing_value=2E_=28changeset_9cb1b85237a9=2C_issue?=
Message-ID: <E1RLQTS-0007CZ-Sl@dinsdale.python.org>

http://hg.python.org/cpython/rev/d877d7f3b679
changeset:   73299:d877d7f3b679
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Wed Nov 02 03:22:15 2011 +0100
summary:
  Actually, there's more than one failing value. (changeset 9cb1b85237a9, issue #13312).

files:
  Lib/test/test_time.py |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -425,8 +425,9 @@
         self.assertEqual(self.yearstr(-123456), '-123456')
         self.assertEqual(self.yearstr(-123456789), str(-123456789))
         self.assertEqual(self.yearstr(-1234567890), str(-1234567890))
-        self.assertEqual(self.yearstr(TIME_MINYEAR + 1), str(TIME_MINYEAR + 1))
-        # On some platforms it loses the sign (issue #13312)
+        # On some platforms it gives weird result (issue #13312)
+        for y in range(TIME_MINYEAR + 10000, TIME_MINYEAR, -1):
+            self.assertEqual(self.yearstr(y), str(y))
         # self.assertEqual(self.yearstr(TIME_MINYEAR), str(TIME_MINYEAR))
         self.assertRaises(OverflowError, self.yearstr, TIME_MINYEAR - 1)
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Nov  2 05:28:52 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 02 Nov 2011 05:28:52 +0100
Subject: [Python-checkins] Daily reference leaks (d877d7f3b679): sum=0
Message-ID: <E1RLSRY-0005va-DF@ap.vmr.nerim.net>

results for d877d7f3b679 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog8N0bXk', '-x']

From python-checkins at python.org  Wed Nov  2 08:14:08 2011
From: python-checkins at python.org (florent.xicluna)
Date: Wed, 02 Nov 2011 08:14:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313312=3A_skip_the_?=
 =?utf8?q?failing_negative_years_for_now=2E?=
Message-ID: <E1RLV1U-000456-1Z@dinsdale.python.org>

http://hg.python.org/cpython/rev/1a0bfc26af57
changeset:   73300:1a0bfc26af57
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Wed Nov 02 08:13:43 2011 +0100
summary:
  Issue #13312: skip the failing negative years for now.

files:
  Lib/test/test_time.py |  7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -425,9 +425,10 @@
         self.assertEqual(self.yearstr(-123456), '-123456')
         self.assertEqual(self.yearstr(-123456789), str(-123456789))
         self.assertEqual(self.yearstr(-1234567890), str(-1234567890))
-        # On some platforms it gives weird result (issue #13312)
-        for y in range(TIME_MINYEAR + 10000, TIME_MINYEAR, -1):
-            self.assertEqual(self.yearstr(y), str(y))
+        self.assertEqual(self.yearstr(TIME_MINYEAR + 1900), str(TIME_MINYEAR + 1900))
+        # Issue #13312: it may return wrong value for year < TIME_MINYEAR + 1900
+        # Skip the value test, but check that no error is raised
+        self.yearstr(TIME_MINYEAR)
         # self.assertEqual(self.yearstr(TIME_MINYEAR), str(TIME_MINYEAR))
         self.assertRaises(OverflowError, self.yearstr, TIME_MINYEAR - 1)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 11:08:08 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 02 Nov 2011 11:08:08 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Put_implementation_link_into_r?=
	=?utf8?q?eferences=2E?=
Message-ID: <E1RLXjs-0001Zq-HI@dinsdale.python.org>

http://hg.python.org/peps/rev/16715ea42fb3
changeset:   3980:16715ea42fb3
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Nov 02 11:08:06 2011 +0100
summary:
  Put implementation link into references.

files:
  pep-0382.txt |  8 +++++++-
  1 files changed, 7 insertions(+), 1 deletions(-)


diff --git a/pep-0382.txt b/pep-0382.txt
--- a/pep-0382.txt
+++ b/pep-0382.txt
@@ -18,7 +18,7 @@
 versions, an algorithm to compute the packages __path__ must be
 formulated. With the enhancement proposed here, the import machinery
 itself will construct the list of directories that make up the
-package.
+package.   An implementation of this PEP is available at [1]_.
 
 Terminology
 ===========
@@ -194,6 +194,12 @@
 any other mechanism might cause portions to get added twice to
 __path__.
 
+References
+==========
+
+.. [1] PEP 382 branch
+       (http://hg.python.org/features/pep-382-2#pep-382)
+
 Copyright
 =========
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed Nov  2 13:29:15 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 02 Nov 2011 13:29:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Introduce_PyObject*_API_for?=
	=?utf8?q?_raising_encode_errors=2E?=
Message-ID: <E1RLZwR-0006Wv-9V@dinsdale.python.org>

http://hg.python.org/cpython/rev/38e961ddeb9d
changeset:   73301:38e961ddeb9d
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Nov 02 12:45:42 2011 +0100
summary:
  Introduce PyObject* API for raising encode errors.

files:
  Objects/unicodeobject.c |  52 +++++++++++++++++++++++++++-
  1 files changed, 49 insertions(+), 3 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -257,6 +257,12 @@
                        const Py_UNICODE *unicode, Py_ssize_t size,
                        Py_ssize_t startpos, Py_ssize_t endpos,
                        const char *reason);
+static void
+raise_encode_exception_obj(PyObject **exceptionObject,
+                       const char *encoding,
+                       PyObject *unicode,
+                       Py_ssize_t startpos, Py_ssize_t endpos,
+                       const char *reason);
 
 /* Same for linebreaks */
 static unsigned char ascii_linebreak[] = {
@@ -4786,9 +4792,9 @@
                 for(k=0; k<repsize; k++) {
                     c = prep[k];
                     if (0x80 <= c) {
-                        raise_encode_exception(&exc, "utf-8",
-                                               PyUnicode_AS_UNICODE(unicode),
-                                               size, i-1, i,
+                        raise_encode_exception_obj(&exc, "utf-8",
+                                               (PyObject*)unicode,
+                                               i-1, i,
                                                "surrogates not allowed");
                         goto error;
                     }
@@ -6434,6 +6440,33 @@
     }
 }
 
+/* This is ultimately going t replace above function. */
+static void
+make_encode_exception_obj(PyObject **exceptionObject,
+                      const char *encoding,
+                      PyObject *unicode,
+                      Py_ssize_t startpos, Py_ssize_t endpos,
+                      const char *reason)
+{
+    if (*exceptionObject == NULL) {
+        *exceptionObject = PyObject_CallFunction(
+            PyExc_UnicodeEncodeError, "sUnns",
+            encoding, unicode, startpos, endpos, reason);
+    }
+    else {
+        if (PyUnicodeEncodeError_SetStart(*exceptionObject, startpos))
+            goto onError;
+        if (PyUnicodeEncodeError_SetEnd(*exceptionObject, endpos))
+            goto onError;
+        if (PyUnicodeEncodeError_SetReason(*exceptionObject, reason))
+            goto onError;
+        return;
+      onError:
+        Py_DECREF(*exceptionObject);
+        *exceptionObject = NULL;
+    }
+}
+
 /* raises a UnicodeEncodeError */
 static void
 raise_encode_exception(PyObject **exceptionObject,
@@ -6447,6 +6480,19 @@
     if (*exceptionObject != NULL)
         PyCodec_StrictErrors(*exceptionObject);
 }
+/* This is ultimately going to replace above function. */
+static void
+raise_encode_exception_obj(PyObject **exceptionObject,
+                       const char *encoding,
+                       PyObject *unicode,
+                       Py_ssize_t startpos, Py_ssize_t endpos,
+                       const char *reason)
+{
+    make_encode_exception_obj(exceptionObject,
+                          encoding, unicode, startpos, endpos, reason);
+    if (*exceptionObject != NULL)
+        PyCodec_StrictErrors(*exceptionObject);
+}
 
 /* error handling callback helper:
    build arguments, call the callback and check the arguments,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 18:02:55 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Wed, 02 Nov 2011 18:02:55 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Port_UCS1_and_charmap_codec?=
	=?utf8?q?s_to_new_API=2E?=
Message-ID: <E1RLeDH-0002Kk-QS@dinsdale.python.org>

http://hg.python.org/cpython/rev/295fdfd4f422
changeset:   73302:295fdfd4f422
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Wed Nov 02 18:02:51 2011 +0100
summary:
  Port UCS1 and charmap codecs to new API.

files:
  Include/unicodeobject.h |    6 +
  Modules/_codecsmodule.c |    6 +-
  Objects/unicodeobject.c |  215 ++++++++++++++++-----------
  3 files changed, 136 insertions(+), 91 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1425,6 +1425,12 @@
                                    (unicode ordinal -> char ordinal) */
     const char *errors          /* error handling */
     );
+PyAPI_FUNC(PyObject*) _PyUnicode_EncodeCharmap(
+    PyObject *unicode,          /* Unicode object */
+    PyObject *mapping,          /* character mapping
+                                   (unicode ordinal -> char ordinal) */
+    const char *errors          /* error handling */
+    );
 #endif
 
 /* Translate a Py_UNICODE buffer of the given length by applying a
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -992,11 +992,7 @@
     str = PyUnicode_FromObject(str);
     if (str == NULL)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeCharmap(
-                               PyUnicode_AS_UNICODE(str),
-                               PyUnicode_GET_SIZE(str),
-                               mapping,
-                               errors),
+    v = codec_tuple(_PyUnicode_EncodeCharmap(str, mapping, errors),
                     PyUnicode_GET_SIZE(str));
     Py_DECREF(str);
     return v;
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -248,7 +248,7 @@
 static PyObject *
 unicode_encode_call_errorhandler(const char *errors,
        PyObject **errorHandler,const char *encoding, const char *reason,
-       const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
+       PyObject *unicode, PyObject **exceptionObject,
        Py_ssize_t startpos, Py_ssize_t endpos, Py_ssize_t *newpos);
 
 static void
@@ -4745,8 +4745,7 @@
 #endif
             rep = unicode_encode_call_errorhandler(
                   errors, &errorHandler, "utf-8", "surrogates not allowed",
-                  PyUnicode_AS_UNICODE(unicode), PyUnicode_GET_SIZE(unicode),
-                  &exc, startpos, startpos+1, &newpos);
+                  obj, &exc, startpos, startpos+1, &newpos);
             if (!rep)
                 goto error;
 
@@ -6450,7 +6449,7 @@
 {
     if (*exceptionObject == NULL) {
         *exceptionObject = PyObject_CallFunction(
-            PyExc_UnicodeEncodeError, "sUnns",
+            PyExc_UnicodeEncodeError, "sOnns",
             encoding, unicode, startpos, endpos, reason);
     }
     else {
@@ -6502,12 +6501,12 @@
 unicode_encode_call_errorhandler(const char *errors,
                                  PyObject **errorHandler,
                                  const char *encoding, const char *reason,
-                                 const Py_UNICODE *unicode, Py_ssize_t size, PyObject **exceptionObject,
+                                 PyObject *unicode, PyObject **exceptionObject,
                                  Py_ssize_t startpos, Py_ssize_t endpos,
                                  Py_ssize_t *newpos)
 {
     static char *argparse = "On;encoding error handler must return (str/bytes, int) tuple";
-
+    Py_ssize_t len;
     PyObject *restuple;
     PyObject *resunicode;
 
@@ -6517,8 +6516,12 @@
             return NULL;
     }
 
-    make_encode_exception(exceptionObject,
-                          encoding, unicode, size, startpos, endpos, reason);
+    if (PyUnicode_READY(unicode) < 0)
+        return NULL;
+    len = PyUnicode_GET_LENGTH(unicode);
+
+    make_encode_exception_obj(exceptionObject,
+                          encoding, unicode, startpos, endpos, reason);
     if (*exceptionObject == NULL)
         return NULL;
 
@@ -6542,8 +6545,8 @@
         return NULL;
     }
     if (*newpos<0)
-        *newpos = size+*newpos;
-    if (*newpos<0 || *newpos>size) {
+        *newpos = len + *newpos;
+    if (*newpos<0 || *newpos>len) {
         PyErr_Format(PyExc_IndexError, "position %zd from error handler out of bounds", *newpos);
         Py_DECREF(restuple);
         return NULL;
@@ -6554,18 +6557,16 @@
 }
 
 static PyObject *
-unicode_encode_ucs1(const Py_UNICODE *p,
-                    Py_ssize_t size,
+unicode_encode_ucs1(PyObject *unicode,
                     const char *errors,
                     int limit)
 {
+    /* input state */
+    Py_ssize_t pos=0, size;
+    int kind;
+    void *data;
     /* output object */
     PyObject *res;
-    /* pointers to the beginning and end+1 of input */
-    const Py_UNICODE *startp = p;
-    const Py_UNICODE *endp = p + size;
-    /* pointer to the beginning of the unencodable characters */
-    /* const Py_UNICODE *badp = NULL; */
     /* pointer into the output */
     char *str;
     /* current output position */
@@ -6578,6 +6579,11 @@
      * -1=not initialized, 0=unknown, 1=strict, 2=replace, 3=ignore, 4=xmlcharrefreplace */
     int known_errorHandler = -1;
 
+    if (PyUnicode_READY(unicode) < 0)
+        return NULL;
+    size = PyUnicode_GET_LENGTH(unicode);
+    kind = PyUnicode_KIND(unicode);
+    data = PyUnicode_DATA(unicode);
     /* allocate enough for a simple encoding without
        replacements, if we need more, we'll resize */
     if (size == 0)
@@ -6588,28 +6594,24 @@
     str = PyBytes_AS_STRING(res);
     ressize = size;
 
-    while (p<endp) {
-        Py_UNICODE c = *p;
+    while (pos < size) {
+        Py_UCS4 c = PyUnicode_READ(kind, data, pos);
 
         /* can we encode this? */
         if (c<limit) {
             /* no overflow check, because we know that the space is enough */
             *str++ = (char)c;
-            ++p;
+            ++pos;
         }
         else {
-            Py_ssize_t unicodepos = p-startp;
             Py_ssize_t requiredsize;
             PyObject *repunicode;
-            Py_ssize_t repsize;
-            Py_ssize_t newpos;
-            Py_ssize_t respos;
-            Py_UNICODE *uni2;
+            Py_ssize_t repsize, newpos, respos, i;
             /* startpos for collecting unencodable chars */
-            const Py_UNICODE *collstart = p;
-            const Py_UNICODE *collend = p;
+            Py_ssize_t collstart = pos;
+            Py_ssize_t collend = pos;
             /* find all unecodable characters */
-            while ((collend < endp) && ((*collend)>=limit))
+            while ((collend < size) && (PyUnicode_READ(kind, data, collend)>=limit))
                 ++collend;
             /* cache callback name lookup (if not done yet, i.e. it's the first error) */
             if (known_errorHandler==-1) {
@@ -6626,39 +6628,40 @@
             }
             switch (known_errorHandler) {
             case 1: /* strict */
-                raise_encode_exception(&exc, encoding, startp, size, collstart-startp, collend-startp, reason);
+                raise_encode_exception_obj(&exc, encoding, unicode, collstart, collend, reason);
                 goto onError;
             case 2: /* replace */
                 while (collstart++<collend)
                     *str++ = '?'; /* fall through */
             case 3: /* ignore */
-                p = collend;
+                pos = collend;
                 break;
             case 4: /* xmlcharrefreplace */
                 respos = str - PyBytes_AS_STRING(res);
-                /* determine replacement size (temporarily (mis)uses p) */
-                for (p = collstart, repsize = 0; p < collend; ++p) {
-                    if (*p<10)
+                /* determine replacement size */
+                for (i = collstart, repsize = 0; i < collend; ++i) {
+                    Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+                    if (ch < 10)
                         repsize += 2+1+1;
-                    else if (*p<100)
+                    else if (ch < 100)
                         repsize += 2+2+1;
-                    else if (*p<1000)
+                    else if (ch < 1000)
                         repsize += 2+3+1;
-                    else if (*p<10000)
+                    else if (ch < 10000)
                         repsize += 2+4+1;
 #ifndef Py_UNICODE_WIDE
                     else
                         repsize += 2+5+1;
 #else
-                    else if (*p<100000)
+                    else if (ch < 100000)
                         repsize += 2+5+1;
-                    else if (*p<1000000)
+                    else if (ch < 1000000)
                         repsize += 2+6+1;
                     else
                         repsize += 2+7+1;
 #endif
                 }
-                requiredsize = respos+repsize+(endp-collend);
+                requiredsize = respos+repsize+(size-collend);
                 if (requiredsize > ressize) {
                     if (requiredsize<2*ressize)
                         requiredsize = 2*ressize;
@@ -6667,17 +6670,18 @@
                     str = PyBytes_AS_STRING(res) + respos;
                     ressize = requiredsize;
                 }
-                /* generate replacement (temporarily (mis)uses p) */
-                for (p = collstart; p < collend; ++p) {
-                    str += sprintf(str, "&#%d;", (int)*p);
-                }
-                p = collend;
+                /* generate replacement */
+                for (i = collstart; i < collend; ++i) {
+                    str += sprintf(str, "&#%d;", PyUnicode_READ(kind, data, i));
+                }
+                pos = collend;
                 break;
             default:
                 repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
-                                                              encoding, reason, startp, size, &exc,
-                                                              collstart-startp, collend-startp, &newpos);
-                if (repunicode == NULL)
+                                                              encoding, reason, unicode, &exc,
+                                                              collstart, collend, &newpos);
+                if (repunicode == NULL || (PyUnicode_Check(repunicode) &&
+                                           PyUnicode_READY(repunicode) < 0))
                     goto onError;
                 if (PyBytes_Check(repunicode)) {
                     /* Directly copy bytes result to output. */
@@ -6694,7 +6698,7 @@
                     }
                     memcpy(str, PyBytes_AsString(repunicode), repsize);
                     str += repsize;
-                    p = startp + newpos;
+                    pos = newpos;
                     Py_DECREF(repunicode);
                     break;
                 }
@@ -6702,8 +6706,8 @@
                    have+the replacement+the rest of the string, so
                    we won't have to check space for encodable characters) */
                 respos = str - PyBytes_AS_STRING(res);
-                repsize = PyUnicode_GET_SIZE(repunicode);
-                requiredsize = respos+repsize+(endp-collend);
+                repsize = PyUnicode_GET_LENGTH(repunicode);
+                requiredsize = respos+repsize+(size-collend);
                 if (requiredsize > ressize) {
                     if (requiredsize<2*ressize)
                         requiredsize = 2*ressize;
@@ -6716,17 +6720,17 @@
                 }
                 /* check if there is anything unencodable in the replacement
                    and copy it to the output */
-                for (uni2 = PyUnicode_AS_UNICODE(repunicode);repsize-->0; ++uni2, ++str) {
-                    c = *uni2;
+                for (i = 0; repsize-->0; ++i, ++str) {
+                    c = PyUnicode_READ_CHAR(repunicode, i);
                     if (c >= limit) {
-                        raise_encode_exception(&exc, encoding, startp, size,
-                                               unicodepos, unicodepos+1, reason);
+                        raise_encode_exception_obj(&exc, encoding, unicode,
+                                               pos, pos+1, reason);
                         Py_DECREF(repunicode);
                         goto onError;
                     }
                     *str = (char)c;
                 }
-                p = startp + newpos;
+                pos = newpos;
                 Py_DECREF(repunicode);
             }
         }
@@ -6750,12 +6754,19 @@
     return NULL;
 }
 
+/* Deprecated */
 PyObject *
 PyUnicode_EncodeLatin1(const Py_UNICODE *p,
                        Py_ssize_t size,
                        const char *errors)
 {
-    return unicode_encode_ucs1(p, size, errors, 256);
+    PyObject *result;
+    PyObject *unicode = PyUnicode_FromUnicode(p, size);
+    if (unicode == NULL)
+        return NULL;
+    result = unicode_encode_ucs1(unicode, errors, 256);
+    Py_DECREF(unicode);
+    return result;
 }
 
 PyObject *
@@ -6774,9 +6785,7 @@
                                          PyUnicode_GET_LENGTH(unicode));
     /* Non-Latin-1 characters present. Defer to above function to
        raise the exception. */
-    return PyUnicode_EncodeLatin1(PyUnicode_AS_UNICODE(unicode),
-                                  PyUnicode_GET_SIZE(unicode),
-                                  errors);
+    return unicode_encode_ucs1(unicode, errors, 256);
 }
 
 PyObject*
@@ -6888,12 +6897,19 @@
     return NULL;
 }
 
+/* Deprecated */
 PyObject *
 PyUnicode_EncodeASCII(const Py_UNICODE *p,
                       Py_ssize_t size,
                       const char *errors)
 {
-    return unicode_encode_ucs1(p, size, errors, 128);
+    PyObject *result;
+    PyObject *unicode = PyUnicode_FromUnicode(p, size);
+    if (unicode == NULL)
+        return NULL;
+    result = unicode_encode_ucs1(unicode, errors, 128);
+    Py_DECREF(unicode);
+    return result;
 }
 
 PyObject *
@@ -6910,9 +6926,7 @@
     if (PyUnicode_MAX_CHAR_VALUE(unicode) < 128)
         return PyBytes_FromStringAndSize(PyUnicode_DATA(unicode),
                                          PyUnicode_GET_LENGTH(unicode));
-    return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(unicode),
-                                 PyUnicode_GET_SIZE(unicode),
-                                 errors);
+    return unicode_encode_ucs1(unicode, errors, 128);
 }
 
 PyObject *
@@ -8182,13 +8196,13 @@
    Return 0 on success, -1 on error */
 static int
 charmap_encoding_error(
-    const Py_UNICODE *p, Py_ssize_t size, Py_ssize_t *inpos, PyObject *mapping,
+    PyObject *unicode, Py_ssize_t *inpos, PyObject *mapping,
     PyObject **exceptionObject,
     int *known_errorHandler, PyObject **errorHandler, const char *errors,
     PyObject **res, Py_ssize_t *respos)
 {
     PyObject *repunicode = NULL; /* initialize to prevent gcc warning */
-    Py_ssize_t repsize;
+    Py_ssize_t size, repsize;
     Py_ssize_t newpos;
     Py_UNICODE *uni2;
     /* startpos for collecting unencodable chars */
@@ -8198,19 +8212,25 @@
     char *encoding = "charmap";
     char *reason = "character maps to <undefined>";
     charmapencode_result x;
-
+    Py_UCS4 ch;
+
+    if (PyUnicode_READY(unicode) < 0)
+        return -1;
+    size = PyUnicode_GET_LENGTH(unicode);
     /* find all unencodable characters */
     while (collendpos < size) {
         PyObject *rep;
         if (Py_TYPE(mapping) == &EncodingMapType) {
-            int res = encoding_map_lookup(p[collendpos], mapping);
+            ch = PyUnicode_READ_CHAR(unicode, collendpos);
+            int res = encoding_map_lookup(ch, mapping);
             if (res != -1)
                 break;
             ++collendpos;
             continue;
         }
 
-        rep = charmapencode_lookup(p[collendpos], mapping);
+        ch = PyUnicode_READ_CHAR(unicode, collendpos);
+        rep = charmapencode_lookup(ch, mapping);
         if (rep==NULL)
             return -1;
         else if (rep!=Py_None) {
@@ -8236,7 +8256,7 @@
     }
     switch (*known_errorHandler) {
     case 1: /* strict */
-        raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
+        raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
         return -1;
     case 2: /* replace */
         for (collpos = collstartpos; collpos<collendpos; ++collpos) {
@@ -8245,7 +8265,7 @@
                 return -1;
             }
             else if (x==enc_FAILED) {
-                raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
+                raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
                 return -1;
             }
         }
@@ -8258,13 +8278,13 @@
         for (collpos = collstartpos; collpos < collendpos; ++collpos) {
             char buffer[2+29+1+1];
             char *cp;
-            sprintf(buffer, "&#%d;", (int)p[collpos]);
+            sprintf(buffer, "&#%d;", (int)PyUnicode_READ_CHAR(unicode, collpos));
             for (cp = buffer; *cp; ++cp) {
                 x = charmapencode_output(*cp, mapping, res, respos);
                 if (x==enc_EXCEPTION)
                     return -1;
                 else if (x==enc_FAILED) {
-                    raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
+                    raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
                     return -1;
                 }
             }
@@ -8273,7 +8293,7 @@
         break;
     default:
         repunicode = unicode_encode_call_errorhandler(errors, errorHandler,
-                                                      encoding, reason, p, size, exceptionObject,
+                                                      encoding, reason, unicode, exceptionObject,
                                                       collstartpos, collendpos, &newpos);
         if (repunicode == NULL)
             return -1;
@@ -8305,7 +8325,7 @@
             }
             else if (x==enc_FAILED) {
                 Py_DECREF(repunicode);
-                raise_encode_exception(exceptionObject, encoding, p, size, collstartpos, collendpos, reason);
+                raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
                 return -1;
             }
         }
@@ -8316,15 +8336,15 @@
 }
 
 PyObject *
-PyUnicode_EncodeCharmap(const Py_UNICODE *p,
-                        Py_ssize_t size,
-                        PyObject *mapping,
-                        const char *errors)
+_PyUnicode_EncodeCharmap(PyObject *unicode,
+                         PyObject *mapping,
+                         const char *errors)
 {
     /* output object */
     PyObject *res = NULL;
     /* current input position */
     Py_ssize_t inpos = 0;
+    Py_ssize_t size;
     /* current output position */
     Py_ssize_t respos = 0;
     PyObject *errorHandler = NULL;
@@ -8334,9 +8354,13 @@
      * 3=ignore, 4=xmlcharrefreplace */
     int known_errorHandler = -1;
 
+    if (PyUnicode_READY(unicode) < 0)
+        return NULL;
+    size = PyUnicode_GET_LENGTH(unicode);
+
     /* Default to Latin-1 */
     if (mapping == NULL)
-        return PyUnicode_EncodeLatin1(p, size, errors);
+        return unicode_encode_ucs1(unicode, errors, 256);
 
     /* allocate enough for a simple encoding without
        replacements, if we need more, we'll resize */
@@ -8347,12 +8371,13 @@
         return res;
 
     while (inpos<size) {
+        Py_UCS4 ch = PyUnicode_READ_CHAR(unicode, inpos);
         /* try to encode it */
-        charmapencode_result x = charmapencode_output(p[inpos], mapping, &res, &respos);
+        charmapencode_result x = charmapencode_output(ch, mapping, &res, &respos);
         if (x==enc_EXCEPTION) /* error */
             goto onError;
         if (x==enc_FAILED) { /* unencodable character */
-            if (charmap_encoding_error(p, size, &inpos, mapping,
+            if (charmap_encoding_error(unicode, &inpos, mapping,
                                        &exc,
                                        &known_errorHandler, &errorHandler, errors,
                                        &res, &respos)) {
@@ -8380,6 +8405,22 @@
     return NULL;
 }
 
+/* Deprecated */
+PyObject *
+PyUnicode_EncodeCharmap(const Py_UNICODE *p,
+                        Py_ssize_t size,
+                        PyObject *mapping,
+                        const char *errors)
+{
+    PyObject *result;
+    PyObject *unicode = PyUnicode_FromUnicode(p, size);
+    if (unicode == NULL)
+        return NULL;
+    result = _PyUnicode_EncodeCharmap(unicode, mapping, errors);
+    Py_DECREF(unicode);
+    return NULL;
+}
+
 PyObject *
 PyUnicode_AsCharmapString(PyObject *unicode,
                           PyObject *mapping)
@@ -8388,10 +8429,7 @@
         PyErr_BadArgument();
         return NULL;
     }
-    return PyUnicode_EncodeCharmap(PyUnicode_AS_UNICODE(unicode),
-                                   PyUnicode_GET_SIZE(unicode),
-                                   mapping,
-                                   NULL);
+    return _PyUnicode_EncodeCharmap(unicode, mapping, NULL);
 }
 
 /* create or adjust a UnicodeTranslateError */
@@ -8893,6 +8931,7 @@
     Py_UNICODE *p, *end;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
+    PyObject *unicode;
     const char *encoding = "decimal";
     const char *reason = "invalid decimal Unicode string";
     /* the following variable is used for caching string comparisons
@@ -8973,9 +9012,13 @@
             p = collend;
             break;
         default:
+            unicode = PyUnicode_FromUnicode(s, length);
+            if (unicode == NULL)
+                goto onError;
             repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
-                                                          encoding, reason, s, length, &exc,
+                                                          encoding, reason, unicode, &exc,
                                                           collstart-s, collend-s, &newpos);
+            Py_DECREF(unicode);
             if (repunicode == NULL)
                 goto onError;
             if (!PyUnicode_Check(repunicode)) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 18:33:48 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 02 Nov 2011 18:33:48 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313295=3A_http=2Eserver_n?=
 =?utf8?q?ow_produces_valid_HTML_4=2E01_strict=2E?=
Message-ID: <E1RLehA-0005nk-Gk@dinsdale.python.org>

http://hg.python.org/cpython/rev/10823e7be085
changeset:   73303:10823e7be085
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Wed Nov 02 19:33:29 2011 +0200
summary:
  #13295: http.server now produces valid HTML 4.01 strict.

files:
  Lib/http/server.py |  22 ++++++++++++++--------
  Misc/NEWS          |   2 ++
  2 files changed, 16 insertions(+), 8 deletions(-)


diff --git a/Lib/http/server.py b/Lib/http/server.py
--- a/Lib/http/server.py
+++ b/Lib/http/server.py
@@ -105,6 +105,7 @@
 DEFAULT_ERROR_MESSAGE = """\
 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
         "http://www.w3.org/TR/html4/strict.dtd">
+<html>
     <head>
         <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
         <title>Error response</title>
@@ -734,10 +735,16 @@
         list.sort(key=lambda a: a.lower())
         r = []
         displaypath = html.escape(urllib.parse.unquote(self.path))
-        r.append('<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">')
-        r.append("<html>\n<title>Directory listing for %s</title>\n" % displaypath)
-        r.append("<body>\n<h2>Directory listing for %s</h2>\n" % displaypath)
-        r.append("<hr>\n<ul>\n")
+        enc = sys.getfilesystemencoding()
+        title = 'Directory listing for %s' % displaypath
+        r.append('<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" '
+                 '"http://www.w3.org/TR/html4/strict.dtd">')
+        r.append('<html>\n<head>')
+        r.append('<meta http-equiv="Content-Type" '
+                 'content="text/html; charset=%s">' % enc)
+        r.append('<title>%s</title>\n</head>' % title)
+        r.append('<body>\n<h1>%s</h1>' % title)
+        r.append('<hr>\n<ul>')
         for name in list:
             fullname = os.path.join(path, name)
             displayname = linkname = name
@@ -748,11 +755,10 @@
             if os.path.islink(fullname):
                 displayname = name + "@"
                 # Note: a link to a directory displays with @ and links with /
-            r.append('<li><a href="%s">%s</a>\n'
+            r.append('<li><a href="%s">%s</a></li>'
                     % (urllib.parse.quote(linkname), html.escape(displayname)))
-        r.append("</ul>\n<hr>\n</body>\n</html>\n")
-        enc = sys.getfilesystemencoding()
-        encoded = ''.join(r).encode(enc)
+        r.append('</ul>\n<hr>\n</body>\n</html>\n')
+        encoded = '\n'.join(r).encode(enc)
         f = io.BytesIO()
         f.write(encoded)
         f.seek(0)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,8 @@
 Library
 -------
 
+- Issue #13295: http.server now produces valid HTML 4.01 strict.
+
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
 - Issue #13287: urllib.request and urllib.error now contains a __all__ and

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 18:59:18 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 02 Nov 2011 18:59:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313324=3A_fcntlmodu?=
 =?utf8?q?le=3A_Add_the_F=5FNOCACHE_flag=2E_Patch_by_Alex_Stewart=2E?=
Message-ID: <E1RLf5q-0008Nd-Vf@dinsdale.python.org>

http://hg.python.org/cpython/rev/cee6fdd6436d
changeset:   73304:cee6fdd6436d
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Nov 02 18:58:25 2011 +0100
summary:
  Issue #13324: fcntlmodule: Add the F_NOCACHE flag. Patch by Alex Stewart.

files:
  Misc/ACKS             |  1 +
  Modules/fcntlmodule.c |  5 ++++-
  2 files changed, 5 insertions(+), 1 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -919,6 +919,7 @@
 Oliver Steele
 Greg Stein
 Chris Stern
+Alex Stewart
 Victor Stinner
 Richard Stoakley
 Peter Stoehr
diff --git a/Modules/fcntlmodule.c b/Modules/fcntlmodule.c
--- a/Modules/fcntlmodule.c
+++ b/Modules/fcntlmodule.c
@@ -540,10 +540,13 @@
     if (ins(d, "F_SHLCK", (long)F_SHLCK)) return -1;
 #endif
 
-/* OS X (and maybe others) let you tell the storage device to flush to physical media */
+/* OS X specifics */
 #ifdef F_FULLFSYNC
     if (ins(d, "F_FULLFSYNC", (long)F_FULLFSYNC)) return -1;
 #endif
+#ifdef F_NOCACHE
+    if (ins(d, "F_NOCACHE", (long)F_NOCACHE)) return -1;
+#endif
 
 /* For F_{GET|SET}FL */
 #ifdef FD_CLOEXEC

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 19:03:37 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Wed, 02 Nov 2011 19:03:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_Issue13147_?=
 =?utf8?q?-_Correct_the_Multiprocessing_Pool=2Emap=5Fasync_method_signatur?=
 =?utf8?q?e=2E?=
Message-ID: <E1RLfA1-0000cO-3v@dinsdale.python.org>

http://hg.python.org/cpython/rev/37e34a983d6d
changeset:   73305:37e34a983d6d
branch:      2.7
parent:      73296:fd4236c787cb
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Thu Nov 03 02:02:38 2011 +0800
summary:
  Fix Issue13147 - Correct the Multiprocessing Pool.map_async method signature.
Remove the error_callback which is not present in 2.7. Patch by Jyrki
Pulliainen.

files:
  Doc/library/multiprocessing.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst
--- a/Doc/library/multiprocessing.rst
+++ b/Doc/library/multiprocessing.rst
@@ -1602,7 +1602,7 @@
       the process pool as separate tasks.  The (approximate) size of these
       chunks can be specified by setting *chunksize* to a positive integer.
 
-   .. method:: map_async(func, iterable[, chunksize[, callback[, error_callback]]])
+   .. method:: map_async(func, iterable[, chunksize[, callback]])
 
       A variant of the :meth:`.map` method which returns a result object.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 19:36:36 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 02 Nov 2011 19:36:36 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMzA4?=
 =?utf8?q?=3A_Fix_test=5Fhttpservers_failures_when_run_as_root=2E?=
Message-ID: <E1RLffw-0003wZ-DS@dinsdale.python.org>

http://hg.python.org/cpython/rev/d1cde7081bf5
changeset:   73306:d1cde7081bf5
branch:      2.7
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Nov 02 19:32:54 2011 +0100
summary:
  Issue #13308: Fix test_httpservers failures when run as root.

files:
  Lib/test/test_httpservers.py |  9 +++++++--
  1 files changed, 7 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py
--- a/Lib/test/test_httpservers.py
+++ b/Lib/test/test_httpservers.py
@@ -324,8 +324,10 @@
         f = open(os.path.join(self.tempdir_name, 'index.html'), 'w')
         response = self.request('/' + self.tempdir_name + '/')
         self.check_status_and_reason(response, 200)
-        if os.name == 'posix':
-            # chmod won't work as expected on Windows platforms
+
+        # chmod() doesn't work as expected on Windows, and filesystem
+        # permissions are ignored by root on Unix.
+        if os.name == 'posix' and os.geteuid() != 0:
             os.chmod(self.tempdir, 0)
             response = self.request(self.tempdir_name + '/')
             self.check_status_and_reason(response, 404)
@@ -370,6 +372,9 @@
                           form.getfirst("bacon"))
 """
 
+
+ at unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+        "This test can't be run reliably as root (issue #13308).")
 class CGIHTTPServerTestCase(BaseTestCase):
     class request_handler(NoLogRequestHandler, CGIHTTPRequestHandler):
         pass

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 19:36:37 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 02 Nov 2011 19:36:37 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzA4?=
 =?utf8?q?=3A_Fix_test=5Fhttpservers_failures_when_run_as_root=2E?=
Message-ID: <E1RLffx-0003we-AM@dinsdale.python.org>

http://hg.python.org/cpython/rev/4dc5590dbd0b
changeset:   73307:4dc5590dbd0b
branch:      3.2
parent:      73291:23ffaf975267
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Nov 02 19:35:14 2011 +0100
summary:
  Issue #13308: Fix test_httpservers failures when run as root.

files:
  Lib/test/test_httpservers.py |  8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py
--- a/Lib/test/test_httpservers.py
+++ b/Lib/test/test_httpservers.py
@@ -259,8 +259,9 @@
         with open(os.path.join(self.tempdir_name, 'index.html'), 'w') as f:
             response = self.request('/' + self.tempdir_name + '/')
             self.check_status_and_reason(response, 200)
-            if os.name == 'posix':
-                # chmod won't work as expected on Windows platforms
+            # chmod() doesn't work as expected on Windows, and filesystem
+            # permissions are ignored by root on Unix.
+            if os.name == 'posix' and os.geteuid() != 0:
                 os.chmod(self.tempdir, 0)
                 response = self.request(self.tempdir_name + '/')
                 self.check_status_and_reason(response, 404)
@@ -305,6 +306,9 @@
                           form.getfirst("bacon")))
 """
 
+
+ at unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+        "This test can't be run reliably as root (issue #13308).")
 class CGIHTTPServerTestCase(BaseTestCase):
     class request_handler(NoLogRequestHandler, CGIHTTPRequestHandler):
         pass

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 19:36:38 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 02 Nov 2011 19:36:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313308=3A_Fix_test=5Fhttpservers_failures_when_run_a?=
 =?utf8?q?s_root=2E?=
Message-ID: <E1RLffy-0003wi-9o@dinsdale.python.org>

http://hg.python.org/cpython/rev/f2d3927a1165
changeset:   73308:f2d3927a1165
parent:      73304:cee6fdd6436d
parent:      73307:4dc5590dbd0b
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Nov 02 19:36:02 2011 +0100
summary:
  Issue #13308: Fix test_httpservers failures when run as root.

files:
  Lib/test/test_httpservers.py |  8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_httpservers.py b/Lib/test/test_httpservers.py
--- a/Lib/test/test_httpservers.py
+++ b/Lib/test/test_httpservers.py
@@ -259,8 +259,9 @@
         with open(os.path.join(self.tempdir_name, 'index.html'), 'w') as f:
             response = self.request('/' + self.tempdir_name + '/')
             self.check_status_and_reason(response, 200)
-            if os.name == 'posix':
-                # chmod won't work as expected on Windows platforms
+            # chmod() doesn't work as expected on Windows, and filesystem
+            # permissions are ignored by root on Unix.
+            if os.name == 'posix' and os.geteuid() != 0:
                 os.chmod(self.tempdir, 0)
                 response = self.request(self.tempdir_name + '/')
                 self.check_status_and_reason(response, 404)
@@ -305,6 +306,9 @@
                           form.getfirst("bacon")))
 """
 
+
+ at unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+        "This test can't be run reliably as root (issue #13308).")
 class CGIHTTPServerTestCase(BaseTestCase):
     class request_handler(NoLogRequestHandler, CGIHTTPRequestHandler):
         pass

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 20:34:27 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Wed, 02 Nov 2011 20:34:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_test=5Fasyncore=3A_Actually?=
 =?utf8?q?_try_to_received_OOB_data=2E?=
Message-ID: <E1RLgZv-0001D3-6t@dinsdale.python.org>

http://hg.python.org/cpython/rev/b14b6a6a6180
changeset:   73309:b14b6a6a6180
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Wed Nov 02 20:30:59 2011 +0100
summary:
  test_asyncore: Actually try to received OOB data.

files:
  Lib/test/test_asyncore.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_asyncore.py b/Lib/test/test_asyncore.py
--- a/Lib/test/test_asyncore.py
+++ b/Lib/test/test_asyncore.py
@@ -675,6 +675,7 @@
 
         class TestClient(BaseClient):
             def handle_expt(self):
+                self.socket.recv(1024, socket.MSG_OOB)
                 self.flag = True
 
         class TestHandler(BaseTestHandler):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 21:09:45 2011
From: python-checkins at python.org (brian.curtin)
Date: Wed, 02 Nov 2011 21:09:45 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_compile_error_=28appa?=
 =?utf8?q?rently_Windows_only=29_introduced_in_295fdfd4f422?=
Message-ID: <E1RLh85-0004hP-9p@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a0839d412b9
changeset:   73310:5a0839d412b9
user:        Brian Curtin <brian at python.org>
date:        Wed Nov 02 15:09:37 2011 -0500
summary:
  Fix a compile error (apparently Windows only) introduced in 295fdfd4f422

files:
  Objects/unicodeobject.c |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8213,6 +8213,7 @@
     char *reason = "character maps to <undefined>";
     charmapencode_result x;
     Py_UCS4 ch;
+    int val;
 
     if (PyUnicode_READY(unicode) < 0)
         return -1;
@@ -8222,8 +8223,8 @@
         PyObject *rep;
         if (Py_TYPE(mapping) == &EncodingMapType) {
             ch = PyUnicode_READ_CHAR(unicode, collendpos);
-            int res = encoding_map_lookup(ch, mapping);
-            if (res != -1)
+            val = encoding_map_lookup(ch, mapping);
+            if (val != -1)
                 break;
             ++collendpos;
             continue;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 23:46:32 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 02 Nov 2011 23:46:32 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEwNTcw?=
 =?utf8?q?=3A_curses=2Etigetstr=28=29_is_now_expecting_a_byte_string=2C_in?=
 =?utf8?q?stead_of_a?=
Message-ID: <E1RLjZo-0002hf-IC@dinsdale.python.org>

http://hg.python.org/cpython/rev/e41663970ca5
changeset:   73311:e41663970ca5
branch:      3.2
parent:      73307:4dc5590dbd0b
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 02 23:45:29 2011 +0100
summary:
  Issue #10570: curses.tigetstr() is now expecting a byte string, instead of a
Unicode string.

This is an incompatible change, but the previous behaviour was completly wrong.

files:
  Doc/library/curses.rst  |  2 +-
  Lib/test/test_curses.py |  7 ++++++-
  Misc/NEWS               |  3 +++
  Modules/_cursesmodule.c |  2 +-
  4 files changed, 11 insertions(+), 3 deletions(-)


diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -566,7 +566,7 @@
 
    Instantiate the string *str* with the supplied parameters, where *str* should
    be a parameterized string obtained from the terminfo database.  E.g.
-   ``tparm(tigetstr("cup"), 5, 3)`` could result in ``'\033[6;4H'``, the exact
+   ``tparm(tigetstr("cup"), 5, 3)`` could result in ``b'\033[6;4H'``, the exact
    result depending on terminal type.
 
 
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -190,7 +190,7 @@
     curses.tigetflag('hc')
     curses.tigetnum('co')
     curses.tigetstr('cr')
-    curses.tparm('cr')
+    curses.tparm(b'cr')
     curses.typeahead(sys.__stdin__.fileno())
     curses.unctrl('a')
     curses.ungetch('a')
@@ -264,6 +264,10 @@
     curses.ungetch(1025)
     stdscr.getkey()
 
+def test_issue10570():
+    b = curses.tparm(curses.tigetstr("cup"), 5, 3)
+    assert type(b) is bytes
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -272,6 +276,7 @@
         test_userptr_without_set(stdscr)
         test_resize_term(stdscr)
         test_issue6243(stdscr)
+        test_issue10570()
     finally:
         curses.resetty()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,6 +66,9 @@
 Library
 -------
 
+- Issue #10570: curses.tigetstr() is now expecting a byte string, instead of
+  a Unicode string.
+
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
 - Issue #670664: Fix HTMLParser to correctly handle the content of
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2600,7 +2600,7 @@
 
     PyCursesSetupTermCalled;
 
-    if (!PyArg_ParseTuple(args, "s|iiiiiiiii:tparm",
+    if (!PyArg_ParseTuple(args, "y|iiiiiiiii:tparm",
                           &fmt, &i1, &i2, &i3, &i4,
                           &i5, &i6, &i7, &i8, &i9)) {
         return NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  2 23:46:33 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 02 Nov 2011 23:46:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiAoTWVyZ2UgMy4yKSBJc3N1ZSAjMTA1NzA6IGN1cnNlcy50aWdldHN0cigpIGlz?=
 =?utf8?q?_now_expecting_a_byte_string=2C?=
Message-ID: <E1RLjZp-0002hk-I6@dinsdale.python.org>

http://hg.python.org/cpython/rev/ab11a6a73683
changeset:   73312:ab11a6a73683
parent:      73310:5a0839d412b9
parent:      73311:e41663970ca5
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 02 23:47:58 2011 +0100
summary:
  (Merge 3.2) Issue #10570: curses.tigetstr() is now expecting a byte string,
instead of a Unicode string.

This is an incompatible change, but the previous behaviour was completly wrong.

files:
  Doc/library/curses.rst  |  2 +-
  Lib/test/test_curses.py |  7 ++++++-
  Misc/NEWS               |  3 +++
  Modules/_cursesmodule.c |  2 +-
  4 files changed, 11 insertions(+), 3 deletions(-)


diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -566,7 +566,7 @@
 
    Instantiate the string *str* with the supplied parameters, where *str* should
    be a parameterized string obtained from the terminfo database.  E.g.
-   ``tparm(tigetstr("cup"), 5, 3)`` could result in ``'\033[6;4H'``, the exact
+   ``tparm(tigetstr("cup"), 5, 3)`` could result in ``b'\033[6;4H'``, the exact
    result depending on terminal type.
 
 
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -190,7 +190,7 @@
     curses.tigetflag('hc')
     curses.tigetnum('co')
     curses.tigetstr('cr')
-    curses.tparm('cr')
+    curses.tparm(b'cr')
     curses.typeahead(sys.__stdin__.fileno())
     curses.unctrl('a')
     curses.ungetch('a')
@@ -280,6 +280,10 @@
     if read != ch:
         raise AssertionError("%r != %r" % (read, ch))
 
+def test_issue10570():
+    b = curses.tparm(curses.tigetstr("cup"), 5, 3)
+    assert type(b) is bytes
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -289,6 +293,7 @@
         test_resize_term(stdscr)
         test_issue6243(stdscr)
         test_unget_wch(stdscr)
+        test_issue10570()
     finally:
         curses.resetty()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,9 @@
 Library
 -------
 
+- Issue #10570: curses.tigetstr() is now expecting a byte string, instead of
+  a Unicode string.
+
 - Issue #13295: http.server now produces valid HTML 4.01 strict.
 
 - Issue #2892: preserve iterparse events in case of SyntaxError.
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2642,7 +2642,7 @@
 
     PyCursesSetupTermCalled;
 
-    if (!PyArg_ParseTuple(args, "s|iiiiiiiii:tparm",
+    if (!PyArg_ParseTuple(args, "y|iiiiiiiii:tparm",
                           &fmt, &i1, &i2, &i3, &i4,
                           &i5, &i6, &i7, &i8, &i9)) {
         return NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 02:51:11 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 03 Nov 2011 02:51:11 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzA3?=
 =?utf8?q?=3A_fix_bdist=5Frpm_test_failures?=
Message-ID: <E1RLmSV-0008Ja-R1@dinsdale.python.org>

http://hg.python.org/cpython/rev/2c0253d4d9ba
changeset:   73313:2c0253d4d9ba
branch:      3.2
parent:      73311:e41663970ca5
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 03 02:45:46 2011 +0100
summary:
  Issue #13307: fix bdist_rpm test failures

files:
  Lib/distutils/command/build_py.py    |  6 +++---
  Lib/distutils/command/install_lib.py |  5 +++--
  2 files changed, 6 insertions(+), 5 deletions(-)


diff --git a/Lib/distutils/command/build_py.py b/Lib/distutils/command/build_py.py
--- a/Lib/distutils/command/build_py.py
+++ b/Lib/distutils/command/build_py.py
@@ -3,7 +3,7 @@
 Implements the Distutils 'build_py' command."""
 
 import sys, os
-import sys
+import imp
 from glob import glob
 
 from distutils.core import Command
@@ -311,9 +311,9 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(filename + "c")
+                    outputs.append(imp.cache_from_source(filename, True))
                 if self.optimize > 0:
-                    outputs.append(filename + "o")
+                    outputs.append(imp.cache_from_source(filename, False))
 
         outputs += [
             os.path.join(build_dir, filename)
diff --git a/Lib/distutils/command/install_lib.py b/Lib/distutils/command/install_lib.py
--- a/Lib/distutils/command/install_lib.py
+++ b/Lib/distutils/command/install_lib.py
@@ -4,6 +4,7 @@
 (install all Python modules)."""
 
 import os
+import imp
 import sys
 
 from distutils.core import Command
@@ -164,9 +165,9 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(py_file + "c")
+                bytecode_files.append(imp.cache_from_source(py_file, True))
             if self.optimize > 0:
-                bytecode_files.append(py_file + "o")
+                bytecode_files.append(imp.cache_from_source(py_file, False))
 
         return bytecode_files
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 02:51:12 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 03 Nov 2011 02:51:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313307=3A_fix_bdist=5Frpm_test_failures?=
Message-ID: <E1RLmSW-0008Je-J3@dinsdale.python.org>

http://hg.python.org/cpython/rev/eb2991f7cdc8
changeset:   73314:eb2991f7cdc8
parent:      73312:ab11a6a73683
parent:      73313:2c0253d4d9ba
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 03 02:46:52 2011 +0100
summary:
  Issue #13307: fix bdist_rpm test failures

files:
  Lib/distutils/command/build_py.py    |  6 +++---
  Lib/distutils/command/install_lib.py |  5 +++--
  2 files changed, 6 insertions(+), 5 deletions(-)


diff --git a/Lib/distutils/command/build_py.py b/Lib/distutils/command/build_py.py
--- a/Lib/distutils/command/build_py.py
+++ b/Lib/distutils/command/build_py.py
@@ -3,7 +3,7 @@
 Implements the Distutils 'build_py' command."""
 
 import sys, os
-import sys
+import imp
 from glob import glob
 
 from distutils.core import Command
@@ -311,9 +311,9 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(filename + "c")
+                    outputs.append(imp.cache_from_source(filename, True))
                 if self.optimize > 0:
-                    outputs.append(filename + "o")
+                    outputs.append(imp.cache_from_source(filename, False))
 
         outputs += [
             os.path.join(build_dir, filename)
diff --git a/Lib/distutils/command/install_lib.py b/Lib/distutils/command/install_lib.py
--- a/Lib/distutils/command/install_lib.py
+++ b/Lib/distutils/command/install_lib.py
@@ -4,6 +4,7 @@
 (install all Python modules)."""
 
 import os
+import imp
 import sys
 
 from distutils.core import Command
@@ -164,9 +165,9 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(py_file + "c")
+                bytecode_files.append(imp.cache_from_source(py_file, True))
             if self.optimize > 0:
-                bytecode_files.append(py_file + "o")
+                bytecode_files.append(imp.cache_from_source(py_file, False))
 
         return bytecode_files
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Nov  3 05:28:32 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 03 Nov 2011 05:28:32 +0100
Subject: [Python-checkins] Daily reference leaks (eb2991f7cdc8): sum=0
Message-ID: <E1RLoum-0006CI-Oz@ap.vmr.nerim.net>

results for eb2991f7cdc8 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog8E5loy', '-x']

From python-checkins at python.org  Thu Nov  3 16:45:44 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:44 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typos_in_recent_NEWS_en?=
	=?utf8?q?tries?=
Message-ID: <E1RLzU8-000310-R2@dinsdale.python.org>

http://hg.python.org/cpython/rev/5623378faa1c
changeset:   73315:5623378faa1c
parent:      73301:38e961ddeb9d
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 00:12:39 2011 +0100
summary:
  Fix typos in recent NEWS entries

files:
  Misc/NEWS |  9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -97,7 +97,7 @@
 
 - Issue #12773: Make __doc__ mutable on user-defined classes.
 
-- Issue #12766: Raise an ValueError when creating a class with a class variable
+- Issue #12766: Raise a ValueError when creating a class with a class variable
   that conflicts with a name in __slots__.
 
 - Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
@@ -158,7 +158,7 @@
   given, produce a informative error message which includes the name(s) of the
   missing arguments.
 
-- Issue #12370: Fix super with not arguments when __class__ is overriden in the
+- Issue #12370: Fix super with no arguments when __class__ is overriden in the
   class body.
 
 - Issue #12084: os.stat on Windows now works properly with relative symbolic
@@ -352,8 +352,9 @@
 
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
-- Issue #13287: urllib.request and urllib.error now contains a __all__ and
-  exposes only relevant Classes, Functions. Patch by Florent Xicluna.
+- Issue #13287: urllib.request and urllib.error now contains an __all__
+  attribute to expose only public classes and functions.  Patch by Florent
+  Xicluna.
 
 - Issue #670664: Fix HTMLParser to correctly handle the content of
   ``<script>...</script>`` and ``<style>...</style>``.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:45 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:45 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_typo_=E2=80=9Cseperate?=
	=?utf8?b?4oCd?=
Message-ID: <E1RLzU9-000314-Jf@dinsdale.python.org>

http://hg.python.org/cpython/rev/0eb11baa8a26
changeset:   73316:0eb11baa8a26
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 00:13:05 2011 +0100
summary:
  Fix typo ?seperate?

files:
  Lib/packaging/command/build.py    |  2 +-
  Lib/packaging/command/build_py.py |  2 +-
  Lib/packaging/dist.py             |  2 +-
  3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/packaging/command/build.py b/Lib/packaging/command/build.py
--- a/Lib/packaging/command/build.py
+++ b/Lib/packaging/command/build.py
@@ -41,7 +41,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -29,7 +29,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -69,7 +69,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ]
     display_option_names = [x[0].replace('-', '_') for x in display_options]
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:46 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:46 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_code_reorganization_i?=
	=?utf8?q?n_one_packaging_test_file?=
Message-ID: <E1RLzUA-000318-Bh@dinsdale.python.org>

http://hg.python.org/cpython/rev/2cdb0063c854
changeset:   73317:2cdb0063c854
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 00:20:03 2011 +0100
summary:
  Minor code reorganization in one packaging test file

files:
  Lib/packaging/tests/test_command_sdist.py |  54 +++++-----
  1 files changed, 26 insertions(+), 28 deletions(-)


diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -1,9 +1,7 @@
 """Tests for packaging.command.sdist."""
 import os
+import tarfile
 import zipfile
-import tarfile
-
-from packaging.tests.support import requires_zlib
 
 try:
     import grp
@@ -12,16 +10,16 @@
 except ImportError:
     UID_GID_SUPPORT = False
 
+from shutil import get_archive_formats
 from os.path import join
+from packaging.dist import Distribution
+from packaging.util import find_executable
+from packaging.errors import PackagingOptionError
+from packaging.command.sdist import sdist, show_formats
+
+from packaging.tests import support, unittest
 from packaging.tests import captured_stdout
-from packaging.command.sdist import sdist
-from packaging.command.sdist import show_formats
-from packaging.dist import Distribution
-from packaging.tests import unittest
-from packaging.errors import PackagingOptionError
-from packaging.util import find_executable
-from packaging.tests import support
-from shutil import get_archive_formats
+from packaging.tests.support import requires_zlib
 
 
 MANIFEST = """\
@@ -88,7 +86,6 @@
 
         # creating VCS directories with some files in them
         os.mkdir(join(self.tmp_dir, 'somecode', '.svn'))
-
         self.write_file((self.tmp_dir, 'somecode', '.svn', 'ok.py'), 'xxx')
 
         os.mkdir(join(self.tmp_dir, 'somecode', '.hg'))
@@ -216,12 +213,14 @@
         # testing the `check-metadata` option
         dist, cmd = self.get_cmd(metadata={'name': 'xxx', 'version': 'xxx'})
 
-        # this should raise some warnings
-        # with the check subcommand
+        # this should cause the check subcommand to log two warnings:
+        # version is invalid, home-page and author are missing
         cmd.ensure_finalized()
         cmd.run()
         warnings = self.get_logs()
-        self.assertEqual(len(warnings), 4)
+        check_warnings = [msg for msg in warnings if
+                          not msg.startswith('sdist:')]
+        self.assertEqual(len(check_warnings), 2, warnings)
 
         # trying with a complete set of metadata
         self.loghandler.flush()
@@ -244,7 +243,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -264,6 +262,18 @@
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     @requires_zlib
+    def test_template(self):
+        dist, cmd = self.get_cmd()
+        dist.extra_files = ['include yeah']
+        cmd.ensure_finalized()
+        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
+        cmd.run()
+        with open(cmd.manifest) as f:
+            content = f.read()
+
+        self.assertIn('yeah', content)
+
+    @requires_zlib
     @unittest.skipUnless(UID_GID_SUPPORT, "requires grp and pwd support")
     @unittest.skipIf(find_executable('tar') is None or
                      find_executable('gzip') is None,
@@ -368,18 +378,6 @@
         self.assertEqual(manifest, ['README.manual'])
 
     @requires_zlib
-    def test_template(self):
-        dist, cmd = self.get_cmd()
-        dist.extra_files = ['include yeah']
-        cmd.ensure_finalized()
-        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
-        cmd.run()
-        with open(cmd.manifest) as f:
-            content = f.read()
-
-        self.assertIn('yeah', content)
-
-    @requires_zlib
     def test_manifest_builder(self):
         dist, cmd = self.get_cmd()
         cmd.manifest_builders = 'packaging.tests.test_command_sdist.builder'

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:47 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Cleanups_in_dis?=
	=?utf8?q?tutils_tests=2E?=
Message-ID: <E1RLzUB-00031C-3w@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f606f4ecf0b
changeset:   73318:0f606f4ecf0b
branch:      3.2
parent:      73291:23ffaf975267
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Nov 02 18:05:41 2011 +0100
summary:
  Cleanups in distutils tests.

- Actually check the contents of the file created by bdist_dumb.
- Don?t use ?RECORD? as filename for non-PEP 376 record file
- Don?t start method name with ?_test?, it smells like a disabled test
  method instead of an helper method
- Fix some idioms (assertIn, addCleanup)

files:
  Lib/distutils/tests/test_bdist_dumb.py |  23 +++++++++---
  Lib/distutils/tests/test_install.py    |  26 +++++--------
  Lib/distutils/tests/test_sdist.py      |   8 ++--
  3 files changed, 32 insertions(+), 25 deletions(-)


diff --git a/Lib/distutils/tests/test_bdist_dumb.py b/Lib/distutils/tests/test_bdist_dumb.py
--- a/Lib/distutils/tests/test_bdist_dumb.py
+++ b/Lib/distutils/tests/test_bdist_dumb.py
@@ -1,8 +1,10 @@
 """Tests for distutils.command.bdist_dumb."""
 
+import os
+import imp
+import sys
+import zipfile
 import unittest
-import sys
-import os
 from test.support import run_unittest
 
 from distutils.core import Distribution
@@ -72,15 +74,24 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        fp = zipfile.ZipFile(os.path.join('dist', base))
+        try:
+            contents = fp.namelist()
+        finally:
+            fp.close()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo-0.1-py%s.%s.egg-info' % sys.version_info[:2],
+                  'foo.%s.pyc' % imp.get_tag(),
+                  'foo.py']
+        self.assertEqual(contents, sorted(wanted))
 
 def test_suite():
     return unittest.makeSuite(BuildDumbTestCase)
diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -87,19 +87,17 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        try:
-            # this is the actual test
-            self._test_user_site()
-        finally:
+        def cleanup():
             site.USER_BASE = self.old_user_base
             site.USER_SITE = self.old_user_site
             install_module.USER_BASE = self.old_user_base
             install_module.USER_SITE = self.old_user_site
             os.path.expanduser = self.old_expand
 
-    def _test_user_site(self):
+        self.addCleanup(cleanup)
+
         for key in ('nt_user', 'unix_user', 'os2_home'):
-            self.assertTrue(key in INSTALL_SCHEMES)
+            self.assertIn(key, INSTALL_SCHEMES)
 
         dist = Distribution({'name': 'xx'})
         cmd = install(dist)
@@ -107,14 +105,14 @@
         # making sure the user option is there
         options = [name for name, short, lable in
                    cmd.user_options]
-        self.assertTrue('user' in options)
+        self.assertIn('user', options)
 
         # setting a value
         cmd.user = 1
 
         # user base and site shouldn't be created yet
-        self.assertTrue(not os.path.exists(self.user_base))
-        self.assertTrue(not os.path.exists(self.user_site))
+        self.assertFalse(os.path.exists(self.user_base))
+        self.assertFalse(os.path.exists(self.user_site))
 
         # let's run finalize
         cmd.ensure_finalized()
@@ -123,8 +121,8 @@
         self.assertTrue(os.path.exists(self.user_base))
         self.assertTrue(os.path.exists(self.user_site))
 
-        self.assertTrue('userbase' in cmd.config_vars)
-        self.assertTrue('usersite' in cmd.config_vars)
+        self.assertIn('userbase', cmd.config_vars)
+        self.assertIn('usersite', cmd.config_vars)
 
     def test_handle_extra_path(self):
         dist = Distribution({'name': 'xx', 'extra_path': 'path,dirs'})
@@ -178,14 +176,13 @@
     def test_record(self):
         install_dir = self.mkdtemp()
         project_dir, dist = self.create_dist(scripts=['hello'])
-        self.addCleanup(os.chdir, os.getcwd())
         os.chdir(project_dir)
         self.write_file('hello', "print('o hai')")
 
         cmd = install(dist)
         dist.command_obj['install'] = cmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -204,7 +201,6 @@
         install_dir = self.mkdtemp()
         project_dir, dist = self.create_dist(ext_modules=[
             Extension('xx', ['xxmodule.c'])])
-        self.addCleanup(os.chdir, os.getcwd())
         os.chdir(project_dir)
         support.copy_xxmodule_c(project_dir)
 
@@ -216,7 +212,7 @@
         dist.command_obj['install'] = cmd
         dist.command_obj['build_ext'] = buildextcmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -288,7 +288,7 @@
     # the following tests make sure there is a nice error message instead
     # of a traceback when parsing an invalid manifest template
 
-    def _test_template(self, content):
+    def _check_template(self, content):
         dist, cmd = self.get_cmd()
         os.chdir(self.tmp_dir)
         self.write_file('MANIFEST.in', content)
@@ -299,17 +299,17 @@
         self.assertEqual(len(warnings), 1)
 
     def test_invalid_template_unknown_command(self):
-        self._test_template('taunt knights *')
+        self._check_template('taunt knights *')
 
     def test_invalid_template_wrong_arguments(self):
         # this manifest command takes one argument
-        self._test_template('prune')
+        self._check_template('prune')
 
     @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
     def test_invalid_template_wrong_path(self):
         # on Windows, trailing slashes are not allowed
         # this used to crash instead of raising a warning: #8286
-        self._test_template('include examples/')
+        self._check_template('include examples/')
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:47 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_typo?=
Message-ID: <E1RLzUB-00031G-V5@dinsdale.python.org>

http://hg.python.org/cpython/rev/ca4c7543c823
changeset:   73319:ca4c7543c823
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 00:08:48 2011 +0100
summary:
  Fix typo

files:
  Lib/_pyio.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -1460,7 +1460,7 @@
     enabled.  With this enabled, on input, the lines endings '\n', '\r',
     or '\r\n' are translated to '\n' before being returned to the
     caller. Conversely, on output, '\n' is translated to the system
-    default line seperator, os.linesep. If newline is any other of its
+    default line separator, os.linesep. If newline is any other of its
     legal values, that newline becomes the newline when the file is read
     and it is returned untranslated. On output, '\n' is converted to the
     newline.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:48 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:48 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_State_explicite?=
 =?utf8?q?ly_that_PYTHONDONTWRITEBYTECODE_is_equivalent_to_-B?=
Message-ID: <E1RLzUC-00031K-Mi@dinsdale.python.org>

http://hg.python.org/cpython/rev/64ac030fd5d5
changeset:   73320:64ac030fd5d5
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 03:20:43 2011 +0100
summary:
  State explicitely that PYTHONDONTWRITEBYTECODE is equivalent to -B

files:
  Doc/using/cmdline.rst |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -453,7 +453,8 @@
 .. envvar:: PYTHONDONTWRITEBYTECODE
 
    If this is set, Python won't try to write ``.pyc`` or ``.pyo`` files on the
-   import of source modules.
+   import of source modules.  This is equivalent to specifying the :option:`-B`
+   option.
 
 
 .. envvar:: PYTHONIOENCODING

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:50 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_More_fixes_for_?=
 =?utf8?q?PEP_3147_compliance_in_distutils_=28=2311254=29?=
Message-ID: <E1RLzUE-00031S-4v@dinsdale.python.org>

http://hg.python.org/cpython/rev/ea926dff958f
changeset:   73322:ea926dff958f
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 03:45:33 2011 +0100
summary:
  More fixes for PEP 3147 compliance in distutils (#11254)

files:
  Lib/distutils/command/build_py.py       |   9 +-
  Lib/distutils/command/install_lib.py    |   7 +-
  Lib/distutils/tests/test_build_py.py    |  70 ++++++++----
  Lib/distutils/tests/test_install.py     |  15 +-
  Lib/distutils/tests/test_install_lib.py |  52 +++++----
  5 files changed, 94 insertions(+), 59 deletions(-)


diff --git a/Lib/distutils/command/build_py.py b/Lib/distutils/command/build_py.py
--- a/Lib/distutils/command/build_py.py
+++ b/Lib/distutils/command/build_py.py
@@ -2,7 +2,8 @@
 
 Implements the Distutils 'build_py' command."""
 
-import sys, os
+import os
+import imp
 import sys
 from glob import glob
 
@@ -311,9 +312,11 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(filename + "c")
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=True))
                 if self.optimize > 0:
-                    outputs.append(filename + "o")
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=False))
 
         outputs += [
             os.path.join(build_dir, filename)
diff --git a/Lib/distutils/command/install_lib.py b/Lib/distutils/command/install_lib.py
--- a/Lib/distutils/command/install_lib.py
+++ b/Lib/distutils/command/install_lib.py
@@ -4,6 +4,7 @@
 (install all Python modules)."""
 
 import os
+import imp
 import sys
 
 from distutils.core import Command
@@ -164,9 +165,11 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(py_file + "c")
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=True))
             if self.optimize > 0:
-                bytecode_files.append(py_file + "o")
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=False))
 
         return bytecode_files
 
diff --git a/Lib/distutils/tests/test_build_py.py b/Lib/distutils/tests/test_build_py.py
--- a/Lib/distutils/tests/test_build_py.py
+++ b/Lib/distutils/tests/test_build_py.py
@@ -2,7 +2,6 @@
 
 import os
 import sys
-import io
 import imp
 import unittest
 
@@ -54,7 +53,6 @@
         # This makes sure the list of outputs includes byte-compiled
         # files for Python modules but not for package data files
         # (there shouldn't *be* byte-code files for those!).
-        #
         self.assertEqual(len(cmd.get_outputs()), 3)
         pkgdest = os.path.join(destination, "pkg")
         files = os.listdir(pkgdest)
@@ -64,15 +62,11 @@
         if sys.dont_write_bytecode:
             self.assertFalse(os.path.exists(pycache_dir))
         else:
-            # XXX even with -O, distutils writes pyc, not pyo; bug?
             pyc_files = os.listdir(pycache_dir)
             self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
-        # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
-        # create the distribution files.
+        # See bugs #1668596/#1720897
         sources = self.mkdtemp()
         open(os.path.join(sources, "__init__.py"), "w").close()
 
@@ -81,30 +75,55 @@
         open(os.path.join(testdir, "testfile"), "w").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        sys.stdout = io.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": {"pkg": ""},
+                             "package_data": {"pkg": ["doc/*"]}})
+        # script_name need not exist, it just need to be initialized
+        dist.script_name = os.path.join(sources, "setup.py")
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": {"pkg": ""},
-                                 "package_data": {"pkg": ["doc/*"]}})
-            # script_name need not exist, it just need to be initialized
-            dist.script_name = os.path.join(sources, "setup.py")
-            dist.script_args = ["build"]
-            dist.parse_command_line()
+            dist.run_commands()
+        except DistutilsFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
-            try:
-                dist.run_commands()
-            except DistutilsFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
+
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile_optimized(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 0
+        cmd.optimize = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(sorted(found), ['boiledeggs.%s.pyo' % imp.get_tag()])
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = build_py(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -118,6 +137,7 @@
 
         self.assertIn('byte-compiling is disabled', self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(BuildPyTestCase)
 
diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -1,6 +1,7 @@
 """Tests for distutils.command.install."""
 
 import os
+import imp
 import sys
 import unittest
 import site
@@ -67,10 +68,7 @@
         check_path(cmd.install_data, destination)
 
     def test_user_site(self):
-        # site.USER_SITE was introduced in 2.6
-        if sys.version < '2.6':
-            return
-
+        # test install with --user
         # preparing the environment for the test
         self.old_user_base = site.USER_BASE
         self.old_user_site = site.USER_SITE
@@ -175,9 +173,11 @@
 
     def test_record(self):
         install_dir = self.mkdtemp()
-        project_dir, dist = self.create_dist(scripts=['hello'])
+        project_dir, dist = self.create_dist(py_modules=['hello'],
+                                             scripts=['sayhi'])
         os.chdir(project_dir)
-        self.write_file('hello', "print('o hai')")
+        self.write_file('hello.py', "def main(): print('o hai')")
+        self.write_file('sayhi', 'from hello import main; main()')
 
         cmd = install(dist)
         dist.command_obj['install'] = cmd
@@ -193,7 +193,7 @@
             f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['hello',
+        expected = ['hello.py', 'hello.%s.pyc' % imp.get_tag(), 'sayhi',
                     'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
         self.assertEqual(found, expected)
 
@@ -238,6 +238,7 @@
             install_module.DEBUG = False
         self.assertTrue(len(self.logs) > old_logs_len)
 
+
 def test_suite():
     return unittest.makeSuite(InstallTestCase)
 
diff --git a/Lib/distutils/tests/test_install_lib.py b/Lib/distutils/tests/test_install_lib.py
--- a/Lib/distutils/tests/test_install_lib.py
+++ b/Lib/distutils/tests/test_install_lib.py
@@ -10,13 +10,14 @@
 from distutils.errors import DistutilsOptionError
 from test.support import run_unittest
 
+
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingSilencer,
                          support.EnvironGuard,
                          unittest.TestCase):
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -35,56 +36,62 @@
 
     @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py')
+        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
         pyo_file = imp.cache_from_source('foo.py', debug_override=False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_output should return 4 elements
-        self.assertTrue(len(cmd.get_outputs()) >= 2)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.import-tag-abiflags.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.import-tag-abiflags.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -98,6 +105,7 @@
 
         self.assertTrue('byte-compiling is disabled' in self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(InstallLibTestCase)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:51 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Improve_byte-compilation_in?=
 =?utf8?q?_packaging_to_be_independent_of_-O_or_-B=2E?=
Message-ID: <E1RLzUF-00031y-2N@dinsdale.python.org>

http://hg.python.org/cpython/rev/dad02a080bbc
changeset:   73323:dad02a080bbc
parent:      73317:2cdb0063c854
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 05:08:28 2011 +0100
summary:
  Improve byte-compilation in packaging to be independent of -O or -B.

The code I fixed to comply with PEP 3147 still had one bug: When run
under python -O, some paths for pyc files would be pyo, because I called
imp.cache_from_source without explicit debug_override argument in some
places, and under -O that would return .pyo (this is well explained in
the imp docs).  Now all code (util.byte_compile, build_py, install_lib)
can create .pyo files according to options given by users,
without interference from the calling Python?s own optimize mode.

On a related topic, I also removed the code that prevented byte
compilation under python -B.  The rationale is that packaging gives
control over the creation of pyc files to the user with its own explicit
option, and the behavior should not be changed if the calling Python
happens to run with -B for whatever reason.  I will argue that this is a
bug fix and ask to be allowed to backport this change to distutils.

Finally, I moved one nugget of information about the --compile and
--optimize options from the source into the doc.  It clears up a
misunderstanding that I (and maybe other people) had.

files:
  Doc/library/packaging.util.rst                  |   6 +-
  Doc/packaging/builtdist.rst                     |   4 +-
  Doc/packaging/commandref.rst                    |  28 +++-
  Lib/packaging/command/build_py.py               |  47 +++---
  Lib/packaging/command/install_lib.py            |  40 +----
  Lib/packaging/errors.py                         |   4 -
  Lib/packaging/tests/support.py                  |   6 +-
  Lib/packaging/tests/test_command_build_py.py    |  30 +---
  Lib/packaging/tests/test_command_install_lib.py |  70 +++++----
  Lib/packaging/tests/test_mixin2to3.py           |   1 +
  Lib/packaging/tests/test_util.py                |  54 +++----
  Lib/packaging/util.py                           |  20 +-
  Misc/NEWS                                       |   3 +
  13 files changed, 151 insertions(+), 162 deletions(-)


diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -116,7 +116,6 @@
    values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``.  Raises
    :exc:`ValueError` if *val* is anything else.
 
-.. TODO Add :term: markup to bytecode when merging into the stdlib
 
 .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])
 
@@ -134,7 +133,7 @@
 
    If *force* is true, all files are recompiled regardless of timestamps.
 
-   The source filename encoded in each bytecode file defaults to the filenames
+   The source filename encoded in each :term:`bytecode` file defaults to the filenames
    listed in *py_files*; you can modify these with *prefix* and *basedir*.
    *prefix* is a string that will be stripped off of each source filename, and
    *base_dir* is a directory name that will be prepended (after *prefix* is
@@ -150,3 +149,6 @@
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
+
+   This function is independent from the running Python's :option:`-O` or
+   :option:`-B` options; it is fully controlled by the parameters passed in.
diff --git a/Doc/packaging/builtdist.rst b/Doc/packaging/builtdist.rst
--- a/Doc/packaging/builtdist.rst
+++ b/Doc/packaging/builtdist.rst
@@ -164,9 +164,7 @@
 have to create a separate installer for every Python version you want to
 support.
 
-.. TODO Add :term: markup to bytecode when merging into the stdlib
-
-The installer will try to compile pure modules into bytecode after installation
+The installer will try to compile pure modules into :term:`bytecode` after installation
 on the target system in normal and optimizing mode.  If you don't want this to
 happen for some reason, you can run the :command:`bdist_wininst` command with
 the :option:`--no-target-compile` and/or the :option:`--no-target-optimize`
diff --git a/Doc/packaging/commandref.rst b/Doc/packaging/commandref.rst
--- a/Doc/packaging/commandref.rst
+++ b/Doc/packaging/commandref.rst
@@ -115,7 +115,24 @@
 -------------------
 
 Build the Python modules (just copy them to the build directory) and
-byte-compile them to .pyc files.
+:term:`byte-compile <bytecode>` them to :file:`.pyc` and/or :file:`.pyo` files.
+
+The byte compilation is controlled by two sets of options:
+
+- ``--compile`` and ``--no-compile`` are used to control the creation of
+  :file:`.pyc` files; the default is ``--no-compile``.
+
+- ``--optimize N`` (or ``-ON``) is used to control the creation of :file:`.pyo`
+  files: ``-O1`` turns on basic optimizations, ``-O2`` also discards docstrings,
+  ``-O0`` does not create :file:`.pyo` files; the default is ``-O0``.
+
+You can mix and match these options: for example, ``--no-compile --optimize 2``
+will create :file:`.pyo` files but no :file:`.pyc` files.
+
+.. XXX these option roles do not work
+
+Calling Python with :option:`-O` or :option:`-B` does not control the creation
+of bytecode files, only the options described above do.
 
 
 :command:`build_scripts`
@@ -341,7 +358,14 @@
 :command:`install_lib`
 ----------------------
 
-Install C library files.
+Install all modules (extensions and pure Python).
+
+.. XXX what about C libraries created with build_clib?
+
+Similarly to ``build_py``, there are options to control the compilation of
+Python code to :term:`bytecode` files (see above).  By default, :file:`.pyc`
+files will be created (``--compile``) and :file:`.pyo` files will not
+(``--optimize 0``).
 
 
 :command:`install_scripts`
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -2,7 +2,6 @@
 
 import os
 import imp
-import sys
 from glob import glob
 
 from packaging import logger
@@ -14,10 +13,14 @@
 # marking public APIs
 __all__ = ['build_py']
 
+
 class build_py(Command, Mixin2to3):
 
     description = "build pure Python modules (copy to build directory)"
 
+    # The options for controlling byte compilations are two independent sets;
+    # more info in install_lib or the reST docs
+
     user_options = [
         ('build-lib=', 'd', "directory to build (copy) to"),
         ('compile', 'c', "compile .py to .pyc"),
@@ -35,7 +38,8 @@
         ]
 
     boolean_options = ['compile', 'force']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         self.build_lib = None
@@ -116,7 +120,7 @@
     def get_data_files(self):
         """Generate list of '(package,src_dir,build_dir,filenames)' tuples.
 
-        Helper function for `finalize_options()`.
+        Helper function for finalize_options.
         """
         data = []
         if not self.packages:
@@ -131,7 +135,7 @@
             # Length of path to strip from found files
             plen = 0
             if src_dir:
-                plen = len(src_dir)+1
+                plen = len(src_dir) + 1
 
             # Strip directory from globbed filenames
             filenames = [
@@ -143,7 +147,7 @@
     def find_data_files(self, package, src_dir):
         """Return filenames for package's data files in 'src_dir'.
 
-        Helper function for `get_data_files()`.
+        Helper function for get_data_files.
         """
         globs = (self.package_data.get('', [])
                  + self.package_data.get(package, []))
@@ -158,7 +162,7 @@
     def build_package_data(self):
         """Copy data files into build directory.
 
-        Helper function for `run()`.
+        Helper function for run.
         """
         # FIXME add tests for this method
         for package, src_dir, build_dir, filenames in self.data_files:
@@ -168,16 +172,17 @@
                 self.mkpath(os.path.dirname(target))
                 outf, copied = self.copy_file(srcfile,
                                target, preserve_mode=False)
-                if copied and srcfile in self.distribution.convert_2to3.doctests:
+                doctests = self.distribution.convert_2to3_doctests
+                if copied and srcfile in doctests:
                     self._doctests_2to3.append(outf)
 
     # XXX - this should be moved to the Distribution class as it is not
     # only needed for build_py. It also has no dependencies on this class.
     def get_package_dir(self, package):
         """Return the directory, relative to the top of the source
-           distribution, where package 'package' should be found
-           (at least according to the 'package_dir' option, if any)."""
-
+        distribution, where package 'package' should be found
+        (at least according to the 'package_dir' option, if any).
+        """
         path = package.split('.')
         if self.package_dir is not None:
             path.insert(0, self.package_dir)
@@ -188,8 +193,7 @@
         return ''
 
     def check_package(self, package, package_dir):
-        """Helper function for `find_package_modules()` and `find_modules()'.
-        """
+        """Helper function for find_package_modules and find_modules."""
         # Empty dir name means current directory, which we can probably
         # assume exists.  Also, os.path.exists and isdir don't know about
         # my "empty string means current dir" convention, so we have to
@@ -209,8 +213,8 @@
             if os.path.isfile(init_py):
                 return init_py
             else:
-                logger.warning(("package init file '%s' not found " +
-                                "(or not a regular file)"), init_py)
+                logger.warning("package init file %r not found "
+                               "(or not a regular file)", init_py)
 
         # Either not in a package at all (__init__.py not expected), or
         # __init__.py doesn't exist -- so don't return the filename.
@@ -218,7 +222,7 @@
 
     def check_module(self, module, module_file):
         if not os.path.isfile(module_file):
-            logger.warning("file %s (for module %s) not found",
+            logger.warning("file %r (for module %r) not found",
                            module_file, module)
             return False
         else:
@@ -239,7 +243,7 @@
                 module = os.path.splitext(os.path.basename(f))[0]
                 modules.append((package, module, f))
             else:
-                logger.debug("excluding %s", setup_script)
+                logger.debug("excluding %r", setup_script)
         return modules
 
     def find_modules(self):
@@ -331,7 +335,8 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(imp.cache_from_source(filename))
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=True))
                 if self.optimize > 0:
                     outputs.append(imp.cache_from_source(filename,
                                                          debug_override=False))
@@ -361,7 +366,6 @@
     def build_modules(self):
         modules = self.find_modules()
         for package, module, module_file in modules:
-
             # Now "build" the module -- ie. copy the source file to
             # self.build_lib (the build directory for Python source).
             # (Actually, it gets copied to the directory for this package
@@ -370,7 +374,6 @@
 
     def build_packages(self):
         for package in self.packages:
-
             # Get list of (package, module, module_file) tuples based on
             # scanning the package directory.  'package' is only included
             # in the tuple so that 'find_modules()' and
@@ -390,11 +393,6 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if sys.dont_write_bytecode:
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
         from packaging.util import byte_compile  # FIXME use compileall
         prefix = self.build_lib
         if prefix[-1] != os.sep:
@@ -403,7 +401,6 @@
         # XXX this code is essentially the same as the 'byte_compile()
         # method of the "install_lib" command, except for the determination
         # of the 'prefix' string.  Hmmm.
-
         if self.compile:
             byte_compile(files, optimize=0,
                          force=self.force, prefix=prefix, dry_run=self.dry_run)
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -2,7 +2,6 @@
 
 import os
 import imp
-import sys
 import logging
 
 from packaging import logger
@@ -11,25 +10,18 @@
 
 
 # Extension for Python source files.
+# XXX dead code?  most of the codebase checks for literal '.py'
 if hasattr(os, 'extsep'):
     PYTHON_SOURCE_EXTENSION = os.extsep + "py"
 else:
     PYTHON_SOURCE_EXTENSION = ".py"
 
+
 class install_lib(Command):
 
     description = "install all modules (extensions and pure Python)"
 
-    # The byte-compilation options are a tad confusing.  Here are the
-    # possible scenarios:
-    #   1) no compilation at all (--no-compile --no-optimize)
-    #   2) compile .pyc only (--compile --no-optimize; default)
-    #   3) compile .pyc and "level 1" .pyo (--compile --optimize)
-    #   4) compile "level 1" .pyo only (--no-compile --optimize)
-    #   5) compile .pyc and "level 2" .pyo (--compile --optimize-more)
-    #   6) compile "level 2" .pyo only (--no-compile --optimize-more)
-    #
-    # The UI for this is two option, 'compile' and 'optimize'.
+    # The options for controlling byte compilations are two independent sets:
     # 'compile' is strictly boolean, and only decides whether to
     # generate .pyc files.  'optimize' is three-way (0, 1, or 2), and
     # decides both whether to generate .pyo files and what level of
@@ -37,7 +29,7 @@
 
     user_options = [
         ('install-dir=', 'd', "directory to install to"),
-        ('build-dir=','b', "build directory (where to install from)"),
+        ('build-dir=', 'b', "build directory (where to install from)"),
         ('force', 'f', "force installation (overwrite existing files)"),
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
@@ -48,7 +40,8 @@
         ]
 
     boolean_options = ['force', 'compile', 'skip-build']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         # let the 'install_dist' command dictate our installation directory
@@ -66,7 +59,8 @@
         self.set_undefined_options('install_dist',
                                    ('build_lib', 'build_dir'),
                                    ('install_lib', 'install_dir'),
-                                   'force', 'compile', 'optimize', 'skip_build')
+                                   'force', 'compile', 'optimize',
+                                   'skip_build')
 
         if self.compile is None:
             self.compile = True
@@ -115,14 +109,6 @@
         return outfiles
 
     def byte_compile(self, files):
-        if sys.dont_write_bytecode:
-            # XXX do we want this?  because a Python runs without bytecode
-            # doesn't mean that the *dists should not contain bytecode
-            #--or does it?
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
         from packaging.util import byte_compile  # FIXME use compileall
 
         # Get the "--root" directory supplied to the "install_dist" command,
@@ -138,13 +124,11 @@
         if self.compile:
             byte_compile(files, optimize=0,
                          force=self.force, prefix=install_root,
-                         dry_run=self.dry_run)
+                         verbose=verbose, dry_run=self.dry_run)
         if self.optimize > 0:
             byte_compile(files, optimize=self.optimize,
                          force=self.force, prefix=install_root,
-                         verbose=verbose,
-                         dry_run=self.dry_run)
-
+                         verbose=verbose, dry_run=self.dry_run)
 
     # -- Utility methods -----------------------------------------------
 
@@ -173,14 +157,14 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(imp.cache_from_source(py_file))
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=True))
             if self.optimize > 0:
                 bytecode_files.append(imp.cache_from_source(
                     py_file, debug_override=False))
 
         return bytecode_files
 
-
     # -- External interface --------------------------------------------
     # (called by outsiders)
 
diff --git a/Lib/packaging/errors.py b/Lib/packaging/errors.py
--- a/Lib/packaging/errors.py
+++ b/Lib/packaging/errors.py
@@ -72,10 +72,6 @@
     """Syntax error in a file list template."""
 
 
-class PackagingByteCompileError(PackagingError):
-    """Byte compile error."""
-
-
 class PackagingPyPIError(PackagingError):
     """Any problem occuring during using the indexes."""
 
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -53,7 +53,7 @@
     # misc. functions and decorators
     'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
-    'unittest', 'requires_zlib', 'skip_unless_symlink',
+    'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
 
 
@@ -357,3 +357,7 @@
 except ImportError:
     skip_unless_symlink = unittest.skip(
         'requires test.support.skip_unless_symlink')
+
+
+skip_2to3_optimize = unittest.skipIf(sys.flags.optimize,
+                                     "2to3 doesn't work under -O")
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -62,12 +62,8 @@
         pycache_dir = os.path.join(pkgdest, "__pycache__")
         self.assertIn("__init__.py", files)
         self.assertIn("README.txt", files)
-        if sys.dont_write_bytecode:
-            self.assertFalse(os.path.exists(pycache_dir))
-        else:
-            # XXX even with -O, packaging writes pyc, not pyo; bug?
-            pyc_files = os.listdir(pycache_dir)
-            self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
+        pyc_files = os.listdir(pycache_dir)
+        self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
@@ -102,7 +98,6 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
         os.chdir(project_dir)
@@ -118,7 +113,6 @@
         found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
         self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile_optimized(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
         os.chdir(project_dir)
@@ -136,21 +130,13 @@
         self.assertEqual(sorted(found), ['boiledeggs.%s.pyc' % imp.get_tag(),
                                          'boiledeggs.%s.pyo' % imp.get_tag()])
 
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
-        cmd = build_py(dist)
-        cmd.compile = True
-        cmd.optimize = 1
-
-        old_dont_write_bytecode = sys.dont_write_bytecode
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            cmd.byte_compile([])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
-
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+        self.test_byte_compile()
+        self.test_byte_compile_optimized()
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -17,7 +17,7 @@
     restore_environ = ['PYTHONPATH']
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -34,71 +34,73 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = True
         cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py')
+        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
         pyo_file = imp.cache_from_source('foo.py', debug_override=False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
+        sys.dont_write_bytecode = True
+        self.test_byte_compile()
+
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # make sure the build_lib is set the temp dir
-        build_dir = os.path.split(pkg_dir)[0]
+        # make sure the build_lib is set the temp dir  # XXX what?  this is not
+        # needed in the same distutils test and should work without manual
+        # intervention
+        build_dir = os.path.split(project_dir)[0]
         cmd.get_finalized_command('build_py').build_lib = build_dir
 
-        # get_output should return 4 elements
-        self.assertEqual(len(cmd.get_outputs()), 4)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.import-tag-abiflags.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
-
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
-        cmd = install_lib(dist)
-        cmd.compile = True
-        cmd.optimize = 1
-
-        self.addCleanup(setattr, sys, 'dont_write_bytecode',
-                        sys.dont_write_bytecode)
-        sys.dont_write_bytecode = True
-        cmd.byte_compile([])
-
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.import-tag-abiflags.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_mixin2to3.py b/Lib/packaging/tests/test_mixin2to3.py
--- a/Lib/packaging/tests/test_mixin2to3.py
+++ b/Lib/packaging/tests/test_mixin2to3.py
@@ -8,6 +8,7 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
+    @support.skip_2to3_optimize
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -10,7 +10,7 @@
 from io import StringIO
 
 from packaging.errors import (
-    PackagingPlatformError, PackagingByteCompileError, PackagingFileError,
+    PackagingPlatformError, PackagingFileError,
     PackagingExecError, InstallationException)
 from packaging import util
 from packaging.dist import Distribution
@@ -138,15 +138,8 @@
             self._uname = None
         os.uname = self._get_uname
 
-        # patching POpen
-        self.old_find_executable = util.find_executable
-        util.find_executable = self._find_executable
-        self._exes = {}
-        self.old_popen = subprocess.Popen
-        self.old_stdout = sys.stdout
-        self.old_stderr = sys.stderr
-        FakePopen.test_class = self
-        subprocess.Popen = FakePopen
+    def _get_uname(self):
+        return self._uname
 
     def tearDown(self):
         # getting back the environment
@@ -161,17 +154,24 @@
             os.uname = self.uname
         else:
             del os.uname
+        super(UtilTestCase, self).tearDown()
+
+    def mock_popen(self):
+        self.old_find_executable = util.find_executable
+        util.find_executable = self._find_executable
+        self._exes = {}
+        self.old_popen = subprocess.Popen
+        self.old_stdout = sys.stdout
+        self.old_stderr = sys.stderr
+        FakePopen.test_class = self
+        subprocess.Popen = FakePopen
+        self.addCleanup(self.unmock_popen)
+
+    def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
         sys.old_stdout = self.old_stdout
         sys.old_stderr = self.old_stderr
-        super(UtilTestCase, self).tearDown()
-
-    def _set_uname(self, uname):
-        self._uname = uname
-
-    def _get_uname(self):
-        return self._uname
 
     def test_convert_path(self):
         # linux/mac
@@ -283,6 +283,7 @@
         return None
 
     def test_get_compiler_versions(self):
+        self.mock_popen()
         # get_versions calls distutils.spawn.find_executable on
         # 'gcc', 'ld' and 'dllwrap'
         self.assertEqual(get_compiler_versions(), (None, None, None))
@@ -323,15 +324,12 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile raise a PackagingError
-        # if sys.dont_write_bytecode is True
-        old_dont_write_bytecode = sys.dont_write_bytecode
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            self.assertRaises(PackagingByteCompileError, byte_compile, [])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
+        byte_compile([])
 
     def test_newer(self):
         self.assertRaises(PackagingFileError, util.newer, 'xxx', 'xxx')
@@ -418,6 +416,7 @@
         self.assertRaises(ImportError, resolve_name, 'a.b.Spam')
         self.assertRaises(ImportError, resolve_name, 'a.b.c.Spam')
 
+    @support.skip_2to3_optimize
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -431,6 +430,7 @@
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
+    @support.skip_2to3_optimize
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
@@ -448,8 +448,6 @@
     @unittest.skipUnless(os.name in ('nt', 'posix'),
                          'runs only under posix or nt')
     def test_spawn(self):
-        # no patching of Popen here
-        subprocess.Popen = self.old_popen
         tmpdir = self.mkdtemp()
 
         # creating something executable
@@ -545,8 +543,6 @@
         self.assertEqual(args['py_modules'], dist.py_modules)
 
     def test_generate_setup_py(self):
-        # undo subprocess.Popen monkey-patching before using assert_python_*
-        subprocess.Popen = self.old_popen
         os.chdir(self.mkdtemp())
         self.write_file('setup.cfg', textwrap.dedent("""\
             [metadata]
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -20,8 +20,8 @@
 
 from packaging import logger
 from packaging.errors import (PackagingPlatformError, PackagingFileError,
-                              PackagingByteCompileError, PackagingExecError,
-                              InstallationException, PackagingInternalError)
+                              PackagingExecError, InstallationException,
+                              PackagingInternalError)
 
 __all__ = [
     # file dependencies
@@ -325,12 +325,10 @@
     the source for details).  The 'direct' flag is used by the script
     generated in indirect mode; unless you know what you're doing, leave
     it set to None.
+
+    This function is independent from the running Python's -O or -B options;
+    it is fully controlled by the parameters passed in.
     """
-    # nothing is done if sys.dont_write_bytecode is True
-    # FIXME this should not raise an error
-    if sys.dont_write_bytecode:
-        raise PackagingByteCompileError('byte-compiling is disabled.')
-
     # First, if the caller didn't force us into direct or indirect mode,
     # figure out which mode we should be in.  We take a conservative
     # approach: choose direct mode *only* if the current interpreter is
@@ -417,12 +415,10 @@
             # Terminology from the py_compile module:
             #   cfile - byte-compiled file
             #   dfile - purported source filename (same as 'file' by default)
-            if optimize >= 0:
-                cfile = imp.cache_from_source(file,
-                                              debug_override=not optimize)
-            else:
-                cfile = imp.cache_from_source(file)
+            debug_override = not optimize
+            cfile = imp.cache_from_source(file, debug_override)
             dfile = file
+
             if prefix:
                 if file[:len(prefix)] != prefix:
                     raise ValueError("invalid prefix: filename %r doesn't "
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,9 @@
 Library
 -------
 
+- Byte compilation in packaging is now isolated from the calling Python -B or
+  -O options, instead of being disallowed under -B or buggy under -O.
+
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
 - Issue #13287: urllib.request and urllib.error now contains an __all__

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:51 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_follow-up_for_=2311254_and_other_changes_from_3=2E2?=
Message-ID: <E1RLzUF-000322-Ml@dinsdale.python.org>

http://hg.python.org/cpython/rev/60ede940089f
changeset:   73324:60ede940089f
parent:      73323:dad02a080bbc
parent:      73322:ea926dff958f
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 05:17:11 2011 +0100
summary:
  Merge follow-up for #11254 and other changes from 3.2

files:
  Doc/using/cmdline.rst                   |   3 +-
  Lib/_pyio.py                            |   2 +-
  Lib/distutils/command/build_py.py       |   9 +-
  Lib/distutils/command/install_lib.py    |   7 +-
  Lib/distutils/tests/test_bdist_dumb.py  |  23 +++-
  Lib/distutils/tests/test_build_py.py    |  70 ++++++++----
  Lib/distutils/tests/test_install.py     |  41 +++----
  Lib/distutils/tests/test_install_lib.py |  52 +++++----
  Lib/distutils/tests/test_sdist.py       |   8 +-
  Python/import.c                         |   6 +-
  10 files changed, 133 insertions(+), 88 deletions(-)


diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -455,7 +455,8 @@
 .. envvar:: PYTHONDONTWRITEBYTECODE
 
    If this is set, Python won't try to write ``.pyc`` or ``.pyo`` files on the
-   import of source modules.
+   import of source modules.  This is equivalent to specifying the :option:`-B`
+   option.
 
 
 .. envvar:: PYTHONIOENCODING
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -1455,7 +1455,7 @@
     enabled.  With this enabled, on input, the lines endings '\n', '\r',
     or '\r\n' are translated to '\n' before being returned to the
     caller. Conversely, on output, '\n' is translated to the system
-    default line seperator, os.linesep. If newline is any other of its
+    default line separator, os.linesep. If newline is any other of its
     legal values, that newline becomes the newline when the file is read
     and it is returned untranslated. On output, '\n' is converted to the
     newline.
diff --git a/Lib/distutils/command/build_py.py b/Lib/distutils/command/build_py.py
--- a/Lib/distutils/command/build_py.py
+++ b/Lib/distutils/command/build_py.py
@@ -2,7 +2,8 @@
 
 Implements the Distutils 'build_py' command."""
 
-import sys, os
+import os
+import imp
 import sys
 from glob import glob
 
@@ -311,9 +312,11 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(filename + "c")
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=True))
                 if self.optimize > 0:
-                    outputs.append(filename + "o")
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=False))
 
         outputs += [
             os.path.join(build_dir, filename)
diff --git a/Lib/distutils/command/install_lib.py b/Lib/distutils/command/install_lib.py
--- a/Lib/distutils/command/install_lib.py
+++ b/Lib/distutils/command/install_lib.py
@@ -4,6 +4,7 @@
 (install all Python modules)."""
 
 import os
+import imp
 import sys
 
 from distutils.core import Command
@@ -164,9 +165,11 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(py_file + "c")
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=True))
             if self.optimize > 0:
-                bytecode_files.append(py_file + "o")
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=False))
 
         return bytecode_files
 
diff --git a/Lib/distutils/tests/test_bdist_dumb.py b/Lib/distutils/tests/test_bdist_dumb.py
--- a/Lib/distutils/tests/test_bdist_dumb.py
+++ b/Lib/distutils/tests/test_bdist_dumb.py
@@ -1,8 +1,10 @@
 """Tests for distutils.command.bdist_dumb."""
 
+import os
+import imp
+import sys
+import zipfile
 import unittest
-import sys
-import os
 from test.support import run_unittest
 
 from distutils.core import Distribution
@@ -72,15 +74,24 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        fp = zipfile.ZipFile(os.path.join('dist', base))
+        try:
+            contents = fp.namelist()
+        finally:
+            fp.close()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo-0.1-py%s.%s.egg-info' % sys.version_info[:2],
+                  'foo.%s.pyc' % imp.get_tag(),
+                  'foo.py']
+        self.assertEqual(contents, sorted(wanted))
 
 def test_suite():
     return unittest.makeSuite(BuildDumbTestCase)
diff --git a/Lib/distutils/tests/test_build_py.py b/Lib/distutils/tests/test_build_py.py
--- a/Lib/distutils/tests/test_build_py.py
+++ b/Lib/distutils/tests/test_build_py.py
@@ -2,7 +2,6 @@
 
 import os
 import sys
-import io
 import imp
 import unittest
 
@@ -54,7 +53,6 @@
         # This makes sure the list of outputs includes byte-compiled
         # files for Python modules but not for package data files
         # (there shouldn't *be* byte-code files for those!).
-        #
         self.assertEqual(len(cmd.get_outputs()), 3)
         pkgdest = os.path.join(destination, "pkg")
         files = os.listdir(pkgdest)
@@ -64,15 +62,11 @@
         if sys.dont_write_bytecode:
             self.assertFalse(os.path.exists(pycache_dir))
         else:
-            # XXX even with -O, distutils writes pyc, not pyo; bug?
             pyc_files = os.listdir(pycache_dir)
             self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
-        # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
-        # create the distribution files.
+        # See bugs #1668596/#1720897
         sources = self.mkdtemp()
         open(os.path.join(sources, "__init__.py"), "w").close()
 
@@ -81,30 +75,55 @@
         open(os.path.join(testdir, "testfile"), "w").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        sys.stdout = io.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": {"pkg": ""},
+                             "package_data": {"pkg": ["doc/*"]}})
+        # script_name need not exist, it just need to be initialized
+        dist.script_name = os.path.join(sources, "setup.py")
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": {"pkg": ""},
-                                 "package_data": {"pkg": ["doc/*"]}})
-            # script_name need not exist, it just need to be initialized
-            dist.script_name = os.path.join(sources, "setup.py")
-            dist.script_args = ["build"]
-            dist.parse_command_line()
+            dist.run_commands()
+        except DistutilsFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
-            try:
-                dist.run_commands()
-            except DistutilsFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
+
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile_optimized(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 0
+        cmd.optimize = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(sorted(found), ['boiledeggs.%s.pyo' % imp.get_tag()])
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = build_py(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -118,6 +137,7 @@
 
         self.assertIn('byte-compiling is disabled', self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(BuildPyTestCase)
 
diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -1,6 +1,7 @@
 """Tests for distutils.command.install."""
 
 import os
+import imp
 import sys
 import unittest
 import site
@@ -67,10 +68,7 @@
         check_path(cmd.install_data, destination)
 
     def test_user_site(self):
-        # site.USER_SITE was introduced in 2.6
-        if sys.version < '2.6':
-            return
-
+        # test install with --user
         # preparing the environment for the test
         self.old_user_base = site.USER_BASE
         self.old_user_site = site.USER_SITE
@@ -87,19 +85,17 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        try:
-            # this is the actual test
-            self._test_user_site()
-        finally:
+        def cleanup():
             site.USER_BASE = self.old_user_base
             site.USER_SITE = self.old_user_site
             install_module.USER_BASE = self.old_user_base
             install_module.USER_SITE = self.old_user_site
             os.path.expanduser = self.old_expand
 
-    def _test_user_site(self):
+        self.addCleanup(cleanup)
+
         for key in ('nt_user', 'unix_user', 'os2_home'):
-            self.assertTrue(key in INSTALL_SCHEMES)
+            self.assertIn(key, INSTALL_SCHEMES)
 
         dist = Distribution({'name': 'xx'})
         cmd = install(dist)
@@ -107,14 +103,14 @@
         # making sure the user option is there
         options = [name for name, short, lable in
                    cmd.user_options]
-        self.assertTrue('user' in options)
+        self.assertIn('user', options)
 
         # setting a value
         cmd.user = 1
 
         # user base and site shouldn't be created yet
-        self.assertTrue(not os.path.exists(self.user_base))
-        self.assertTrue(not os.path.exists(self.user_site))
+        self.assertFalse(os.path.exists(self.user_base))
+        self.assertFalse(os.path.exists(self.user_site))
 
         # let's run finalize
         cmd.ensure_finalized()
@@ -123,8 +119,8 @@
         self.assertTrue(os.path.exists(self.user_base))
         self.assertTrue(os.path.exists(self.user_site))
 
-        self.assertTrue('userbase' in cmd.config_vars)
-        self.assertTrue('usersite' in cmd.config_vars)
+        self.assertIn('userbase', cmd.config_vars)
+        self.assertIn('usersite', cmd.config_vars)
 
     def test_handle_extra_path(self):
         dist = Distribution({'name': 'xx', 'extra_path': 'path,dirs'})
@@ -177,15 +173,16 @@
 
     def test_record(self):
         install_dir = self.mkdtemp()
-        project_dir, dist = self.create_dist(scripts=['hello'])
-        self.addCleanup(os.chdir, os.getcwd())
+        project_dir, dist = self.create_dist(py_modules=['hello'],
+                                             scripts=['sayhi'])
         os.chdir(project_dir)
-        self.write_file('hello', "print('o hai')")
+        self.write_file('hello.py', "def main(): print('o hai')")
+        self.write_file('sayhi', 'from hello import main; main()')
 
         cmd = install(dist)
         dist.command_obj['install'] = cmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -196,7 +193,7 @@
             f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['hello',
+        expected = ['hello.py', 'hello.%s.pyc' % imp.get_tag(), 'sayhi',
                     'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
         self.assertEqual(found, expected)
 
@@ -204,7 +201,6 @@
         install_dir = self.mkdtemp()
         project_dir, dist = self.create_dist(ext_modules=[
             Extension('xx', ['xxmodule.c'])])
-        self.addCleanup(os.chdir, os.getcwd())
         os.chdir(project_dir)
         support.copy_xxmodule_c(project_dir)
 
@@ -216,7 +212,7 @@
         dist.command_obj['install'] = cmd
         dist.command_obj['build_ext'] = buildextcmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -242,6 +238,7 @@
             install_module.DEBUG = False
         self.assertTrue(len(self.logs) > old_logs_len)
 
+
 def test_suite():
     return unittest.makeSuite(InstallTestCase)
 
diff --git a/Lib/distutils/tests/test_install_lib.py b/Lib/distutils/tests/test_install_lib.py
--- a/Lib/distutils/tests/test_install_lib.py
+++ b/Lib/distutils/tests/test_install_lib.py
@@ -10,13 +10,14 @@
 from distutils.errors import DistutilsOptionError
 from test.support import run_unittest
 
+
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingSilencer,
                          support.EnvironGuard,
                          unittest.TestCase):
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -35,56 +36,62 @@
 
     @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py')
+        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
         pyo_file = imp.cache_from_source('foo.py', debug_override=False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_output should return 4 elements
-        self.assertTrue(len(cmd.get_outputs()) >= 2)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.import-tag-abiflags.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.import-tag-abiflags.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -98,6 +105,7 @@
 
         self.assertTrue('byte-compiling is disabled' in self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(InstallLibTestCase)
 
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -288,7 +288,7 @@
     # the following tests make sure there is a nice error message instead
     # of a traceback when parsing an invalid manifest template
 
-    def _test_template(self, content):
+    def _check_template(self, content):
         dist, cmd = self.get_cmd()
         os.chdir(self.tmp_dir)
         self.write_file('MANIFEST.in', content)
@@ -299,17 +299,17 @@
         self.assertEqual(len(warnings), 1)
 
     def test_invalid_template_unknown_command(self):
-        self._test_template('taunt knights *')
+        self._check_template('taunt knights *')
 
     def test_invalid_template_wrong_arguments(self):
         # this manifest command takes one argument
-        self._test_template('prune')
+        self._check_template('prune')
 
     @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
     def test_invalid_template_wrong_path(self):
         # on Windows, trailing slashes are not allowed
         # this used to crash instead of raising a warning: #8286
-        self._test_template('include examples/')
+        self._check_template('include examples/')
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -3908,7 +3908,8 @@
 }
 
 PyDoc_STRVAR(doc_cache_from_source,
-"Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
+"cache_from_source(path, [debug_override]) -> path\n\
+Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
 \n\
 The .py file does not need to exist; this simply returns the path to the\n\
 .pyc/.pyo file calculated as if the .py file were imported.  The extension\n\
@@ -3940,7 +3941,8 @@
 }
 
 PyDoc_STRVAR(doc_source_from_cache,
-"Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
+"source_from_cache(path) -> path\n\
+Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
 \n\
 The .pyc/.pyo file does not need to exist; this simply returns the path to\n\
 the .py file calculated to correspond to the .pyc/.pyo file.  If path\n\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:52 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Actually_check_the_contents?=
 =?utf8?q?_of_the_file_created_by_packaging=E2=80=99s_bdist=5Fdumb?=
Message-ID: <E1RLzUG-000327-FH@dinsdale.python.org>

http://hg.python.org/cpython/rev/5420316921a0
changeset:   73325:5420316921a0
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 06:00:02 2011 +0100
summary:
  Actually check the contents of the file created by packaging?s bdist_dumb

files:
  Lib/packaging/tests/test_command_bdist_dumb.py |  17 +++++++--
  1 files changed, 13 insertions(+), 4 deletions(-)


diff --git a/Lib/packaging/tests/test_command_bdist_dumb.py b/Lib/packaging/tests/test_command_bdist_dumb.py
--- a/Lib/packaging/tests/test_command_bdist_dumb.py
+++ b/Lib/packaging/tests/test_command_bdist_dumb.py
@@ -1,6 +1,9 @@
 """Tests for distutils.command.bdist_dumb."""
 
 import os
+import imp
+import sys
+import zipfile
 import packaging.util
 
 from packaging.dist import Distribution
@@ -49,15 +52,21 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        with zipfile.ZipFile(os.path.join('dist', base)) as fp:
+            contents = fp.namelist()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo.py',
+                  'foo.%s.pyc' % imp.get_tag(),
+                  'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(contents, sorted(wanted))
 
     def test_finalize_options(self):
         pkg_dir, dist = self.create_dist()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:53 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1RLzUH-00032B-As@dinsdale.python.org>

http://hg.python.org/cpython/rev/a1c17a5a460f
changeset:   73326:a1c17a5a460f
parent:      73314:eb2991f7cdc8
parent:      73325:5420316921a0
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 16:44:13 2011 +0100
summary:
  Branch merge

files:
  Doc/library/packaging.util.rst                  |   6 +-
  Doc/packaging/builtdist.rst                     |   4 +-
  Doc/packaging/commandref.rst                    |  28 +++-
  Doc/using/cmdline.rst                           |   3 +-
  Lib/_pyio.py                                    |   2 +-
  Lib/distutils/command/build_py.py               |   9 +-
  Lib/distutils/command/install_lib.py            |   6 +-
  Lib/distutils/tests/test_bdist_dumb.py          |  23 ++-
  Lib/distutils/tests/test_build_py.py            |  70 ++++++---
  Lib/distutils/tests/test_install.py             |  41 ++---
  Lib/distutils/tests/test_install_lib.py         |  52 ++++---
  Lib/distutils/tests/test_sdist.py               |   8 +-
  Lib/packaging/command/build.py                  |   2 +-
  Lib/packaging/command/build_py.py               |  49 +++---
  Lib/packaging/command/install_lib.py            |  40 +----
  Lib/packaging/dist.py                           |   2 +-
  Lib/packaging/errors.py                         |   4 -
  Lib/packaging/tests/support.py                  |   6 +-
  Lib/packaging/tests/test_command_bdist_dumb.py  |  17 +-
  Lib/packaging/tests/test_command_build_py.py    |  30 +---
  Lib/packaging/tests/test_command_install_lib.py |  70 +++++----
  Lib/packaging/tests/test_command_sdist.py       |  54 +++----
  Lib/packaging/tests/test_mixin2to3.py           |   1 +
  Lib/packaging/tests/test_util.py                |  54 +++----
  Lib/packaging/util.py                           |  20 +-
  Misc/NEWS                                       |  12 +-
  Python/import.c                                 |   6 +-
  27 files changed, 330 insertions(+), 289 deletions(-)


diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -116,7 +116,6 @@
    values are ``n``, ``no``, ``f``, ``false``, ``off`` and ``0``.  Raises
    :exc:`ValueError` if *val* is anything else.
 
-.. TODO Add :term: markup to bytecode when merging into the stdlib
 
 .. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])
 
@@ -134,7 +133,7 @@
 
    If *force* is true, all files are recompiled regardless of timestamps.
 
-   The source filename encoded in each bytecode file defaults to the filenames
+   The source filename encoded in each :term:`bytecode` file defaults to the filenames
    listed in *py_files*; you can modify these with *prefix* and *basedir*.
    *prefix* is a string that will be stripped off of each source filename, and
    *base_dir* is a directory name that will be prepended (after *prefix* is
@@ -150,3 +149,6 @@
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
+
+   This function is independent from the running Python's :option:`-O` or
+   :option:`-B` options; it is fully controlled by the parameters passed in.
diff --git a/Doc/packaging/builtdist.rst b/Doc/packaging/builtdist.rst
--- a/Doc/packaging/builtdist.rst
+++ b/Doc/packaging/builtdist.rst
@@ -164,9 +164,7 @@
 have to create a separate installer for every Python version you want to
 support.
 
-.. TODO Add :term: markup to bytecode when merging into the stdlib
-
-The installer will try to compile pure modules into bytecode after installation
+The installer will try to compile pure modules into :term:`bytecode` after installation
 on the target system in normal and optimizing mode.  If you don't want this to
 happen for some reason, you can run the :command:`bdist_wininst` command with
 the :option:`--no-target-compile` and/or the :option:`--no-target-optimize`
diff --git a/Doc/packaging/commandref.rst b/Doc/packaging/commandref.rst
--- a/Doc/packaging/commandref.rst
+++ b/Doc/packaging/commandref.rst
@@ -115,7 +115,24 @@
 -------------------
 
 Build the Python modules (just copy them to the build directory) and
-byte-compile them to .pyc files.
+:term:`byte-compile <bytecode>` them to :file:`.pyc` and/or :file:`.pyo` files.
+
+The byte compilation is controlled by two sets of options:
+
+- ``--compile`` and ``--no-compile`` are used to control the creation of
+  :file:`.pyc` files; the default is ``--no-compile``.
+
+- ``--optimize N`` (or ``-ON``) is used to control the creation of :file:`.pyo`
+  files: ``-O1`` turns on basic optimizations, ``-O2`` also discards docstrings,
+  ``-O0`` does not create :file:`.pyo` files; the default is ``-O0``.
+
+You can mix and match these options: for example, ``--no-compile --optimize 2``
+will create :file:`.pyo` files but no :file:`.pyc` files.
+
+.. XXX these option roles do not work
+
+Calling Python with :option:`-O` or :option:`-B` does not control the creation
+of bytecode files, only the options described above do.
 
 
 :command:`build_scripts`
@@ -341,7 +358,14 @@
 :command:`install_lib`
 ----------------------
 
-Install C library files.
+Install all modules (extensions and pure Python).
+
+.. XXX what about C libraries created with build_clib?
+
+Similarly to ``build_py``, there are options to control the compilation of
+Python code to :term:`bytecode` files (see above).  By default, :file:`.pyc`
+files will be created (``--compile``) and :file:`.pyo` files will not
+(``--optimize 0``).
 
 
 :command:`install_scripts`
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -455,7 +455,8 @@
 .. envvar:: PYTHONDONTWRITEBYTECODE
 
    If this is set, Python won't try to write ``.pyc`` or ``.pyo`` files on the
-   import of source modules.
+   import of source modules.  This is equivalent to specifying the :option:`-B`
+   option.
 
 
 .. envvar:: PYTHONIOENCODING
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -1455,7 +1455,7 @@
     enabled.  With this enabled, on input, the lines endings '\n', '\r',
     or '\r\n' are translated to '\n' before being returned to the
     caller. Conversely, on output, '\n' is translated to the system
-    default line seperator, os.linesep. If newline is any other of its
+    default line separator, os.linesep. If newline is any other of its
     legal values, that newline becomes the newline when the file is read
     and it is returned untranslated. On output, '\n' is converted to the
     newline.
diff --git a/Lib/distutils/command/build_py.py b/Lib/distutils/command/build_py.py
--- a/Lib/distutils/command/build_py.py
+++ b/Lib/distutils/command/build_py.py
@@ -2,8 +2,9 @@
 
 Implements the Distutils 'build_py' command."""
 
-import sys, os
+import os
 import imp
+import sys
 from glob import glob
 
 from distutils.core import Command
@@ -311,9 +312,11 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(imp.cache_from_source(filename, True))
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=True))
                 if self.optimize > 0:
-                    outputs.append(imp.cache_from_source(filename, False))
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=False))
 
         outputs += [
             os.path.join(build_dir, filename)
diff --git a/Lib/distutils/command/install_lib.py b/Lib/distutils/command/install_lib.py
--- a/Lib/distutils/command/install_lib.py
+++ b/Lib/distutils/command/install_lib.py
@@ -165,9 +165,11 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(imp.cache_from_source(py_file, True))
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=True))
             if self.optimize > 0:
-                bytecode_files.append(imp.cache_from_source(py_file, False))
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=False))
 
         return bytecode_files
 
diff --git a/Lib/distutils/tests/test_bdist_dumb.py b/Lib/distutils/tests/test_bdist_dumb.py
--- a/Lib/distutils/tests/test_bdist_dumb.py
+++ b/Lib/distutils/tests/test_bdist_dumb.py
@@ -1,8 +1,10 @@
 """Tests for distutils.command.bdist_dumb."""
 
+import os
+import imp
+import sys
+import zipfile
 import unittest
-import sys
-import os
 from test.support import run_unittest
 
 from distutils.core import Distribution
@@ -72,15 +74,24 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        fp = zipfile.ZipFile(os.path.join('dist', base))
+        try:
+            contents = fp.namelist()
+        finally:
+            fp.close()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo-0.1-py%s.%s.egg-info' % sys.version_info[:2],
+                  'foo.%s.pyc' % imp.get_tag(),
+                  'foo.py']
+        self.assertEqual(contents, sorted(wanted))
 
 def test_suite():
     return unittest.makeSuite(BuildDumbTestCase)
diff --git a/Lib/distutils/tests/test_build_py.py b/Lib/distutils/tests/test_build_py.py
--- a/Lib/distutils/tests/test_build_py.py
+++ b/Lib/distutils/tests/test_build_py.py
@@ -2,7 +2,6 @@
 
 import os
 import sys
-import io
 import imp
 import unittest
 
@@ -54,7 +53,6 @@
         # This makes sure the list of outputs includes byte-compiled
         # files for Python modules but not for package data files
         # (there shouldn't *be* byte-code files for those!).
-        #
         self.assertEqual(len(cmd.get_outputs()), 3)
         pkgdest = os.path.join(destination, "pkg")
         files = os.listdir(pkgdest)
@@ -64,15 +62,11 @@
         if sys.dont_write_bytecode:
             self.assertFalse(os.path.exists(pycache_dir))
         else:
-            # XXX even with -O, distutils writes pyc, not pyo; bug?
             pyc_files = os.listdir(pycache_dir)
             self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
-        # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
-        # create the distribution files.
+        # See bugs #1668596/#1720897
         sources = self.mkdtemp()
         open(os.path.join(sources, "__init__.py"), "w").close()
 
@@ -81,30 +75,55 @@
         open(os.path.join(testdir, "testfile"), "w").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        sys.stdout = io.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": {"pkg": ""},
+                             "package_data": {"pkg": ["doc/*"]}})
+        # script_name need not exist, it just need to be initialized
+        dist.script_name = os.path.join(sources, "setup.py")
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": {"pkg": ""},
-                                 "package_data": {"pkg": ["doc/*"]}})
-            # script_name need not exist, it just need to be initialized
-            dist.script_name = os.path.join(sources, "setup.py")
-            dist.script_args = ["build"]
-            dist.parse_command_line()
+            dist.run_commands()
+        except DistutilsFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
-            try:
-                dist.run_commands()
-            except DistutilsFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
+
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile_optimized(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 0
+        cmd.optimize = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(sorted(found), ['boiledeggs.%s.pyo' % imp.get_tag()])
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = build_py(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -118,6 +137,7 @@
 
         self.assertIn('byte-compiling is disabled', self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(BuildPyTestCase)
 
diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -1,6 +1,7 @@
 """Tests for distutils.command.install."""
 
 import os
+import imp
 import sys
 import unittest
 import site
@@ -67,10 +68,7 @@
         check_path(cmd.install_data, destination)
 
     def test_user_site(self):
-        # site.USER_SITE was introduced in 2.6
-        if sys.version < '2.6':
-            return
-
+        # test install with --user
         # preparing the environment for the test
         self.old_user_base = site.USER_BASE
         self.old_user_site = site.USER_SITE
@@ -87,19 +85,17 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        try:
-            # this is the actual test
-            self._test_user_site()
-        finally:
+        def cleanup():
             site.USER_BASE = self.old_user_base
             site.USER_SITE = self.old_user_site
             install_module.USER_BASE = self.old_user_base
             install_module.USER_SITE = self.old_user_site
             os.path.expanduser = self.old_expand
 
-    def _test_user_site(self):
+        self.addCleanup(cleanup)
+
         for key in ('nt_user', 'unix_user', 'os2_home'):
-            self.assertTrue(key in INSTALL_SCHEMES)
+            self.assertIn(key, INSTALL_SCHEMES)
 
         dist = Distribution({'name': 'xx'})
         cmd = install(dist)
@@ -107,14 +103,14 @@
         # making sure the user option is there
         options = [name for name, short, lable in
                    cmd.user_options]
-        self.assertTrue('user' in options)
+        self.assertIn('user', options)
 
         # setting a value
         cmd.user = 1
 
         # user base and site shouldn't be created yet
-        self.assertTrue(not os.path.exists(self.user_base))
-        self.assertTrue(not os.path.exists(self.user_site))
+        self.assertFalse(os.path.exists(self.user_base))
+        self.assertFalse(os.path.exists(self.user_site))
 
         # let's run finalize
         cmd.ensure_finalized()
@@ -123,8 +119,8 @@
         self.assertTrue(os.path.exists(self.user_base))
         self.assertTrue(os.path.exists(self.user_site))
 
-        self.assertTrue('userbase' in cmd.config_vars)
-        self.assertTrue('usersite' in cmd.config_vars)
+        self.assertIn('userbase', cmd.config_vars)
+        self.assertIn('usersite', cmd.config_vars)
 
     def test_handle_extra_path(self):
         dist = Distribution({'name': 'xx', 'extra_path': 'path,dirs'})
@@ -177,15 +173,16 @@
 
     def test_record(self):
         install_dir = self.mkdtemp()
-        project_dir, dist = self.create_dist(scripts=['hello'])
-        self.addCleanup(os.chdir, os.getcwd())
+        project_dir, dist = self.create_dist(py_modules=['hello'],
+                                             scripts=['sayhi'])
         os.chdir(project_dir)
-        self.write_file('hello', "print('o hai')")
+        self.write_file('hello.py', "def main(): print('o hai')")
+        self.write_file('sayhi', 'from hello import main; main()')
 
         cmd = install(dist)
         dist.command_obj['install'] = cmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -196,7 +193,7 @@
             f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['hello',
+        expected = ['hello.py', 'hello.%s.pyc' % imp.get_tag(), 'sayhi',
                     'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
         self.assertEqual(found, expected)
 
@@ -204,7 +201,6 @@
         install_dir = self.mkdtemp()
         project_dir, dist = self.create_dist(ext_modules=[
             Extension('xx', ['xxmodule.c'])])
-        self.addCleanup(os.chdir, os.getcwd())
         os.chdir(project_dir)
         support.copy_xxmodule_c(project_dir)
 
@@ -216,7 +212,7 @@
         dist.command_obj['install'] = cmd
         dist.command_obj['build_ext'] = buildextcmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -242,6 +238,7 @@
             install_module.DEBUG = False
         self.assertTrue(len(self.logs) > old_logs_len)
 
+
 def test_suite():
     return unittest.makeSuite(InstallTestCase)
 
diff --git a/Lib/distutils/tests/test_install_lib.py b/Lib/distutils/tests/test_install_lib.py
--- a/Lib/distutils/tests/test_install_lib.py
+++ b/Lib/distutils/tests/test_install_lib.py
@@ -10,13 +10,14 @@
 from distutils.errors import DistutilsOptionError
 from test.support import run_unittest
 
+
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingSilencer,
                          support.EnvironGuard,
                          unittest.TestCase):
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -35,56 +36,62 @@
 
     @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py')
+        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
         pyo_file = imp.cache_from_source('foo.py', debug_override=False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_output should return 4 elements
-        self.assertTrue(len(cmd.get_outputs()) >= 2)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.import-tag-abiflags.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.import-tag-abiflags.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -98,6 +105,7 @@
 
         self.assertTrue('byte-compiling is disabled' in self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(InstallLibTestCase)
 
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -288,7 +288,7 @@
     # the following tests make sure there is a nice error message instead
     # of a traceback when parsing an invalid manifest template
 
-    def _test_template(self, content):
+    def _check_template(self, content):
         dist, cmd = self.get_cmd()
         os.chdir(self.tmp_dir)
         self.write_file('MANIFEST.in', content)
@@ -299,17 +299,17 @@
         self.assertEqual(len(warnings), 1)
 
     def test_invalid_template_unknown_command(self):
-        self._test_template('taunt knights *')
+        self._check_template('taunt knights *')
 
     def test_invalid_template_wrong_arguments(self):
         # this manifest command takes one argument
-        self._test_template('prune')
+        self._check_template('prune')
 
     @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
     def test_invalid_template_wrong_path(self):
         # on Windows, trailing slashes are not allowed
         # this used to crash instead of raising a warning: #8286
-        self._test_template('include examples/')
+        self._check_template('include examples/')
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
diff --git a/Lib/packaging/command/build.py b/Lib/packaging/command/build.py
--- a/Lib/packaging/command/build.py
+++ b/Lib/packaging/command/build.py
@@ -41,7 +41,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -2,7 +2,6 @@
 
 import os
 import imp
-import sys
 from glob import glob
 
 from packaging import logger
@@ -14,10 +13,14 @@
 # marking public APIs
 __all__ = ['build_py']
 
+
 class build_py(Command, Mixin2to3):
 
     description = "build pure Python modules (copy to build directory)"
 
+    # The options for controlling byte compilations are two independent sets;
+    # more info in install_lib or the reST docs
+
     user_options = [
         ('build-lib=', 'd', "directory to build (copy) to"),
         ('compile', 'c', "compile .py to .pyc"),
@@ -29,13 +32,14 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
 
     boolean_options = ['compile', 'force']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         self.build_lib = None
@@ -116,7 +120,7 @@
     def get_data_files(self):
         """Generate list of '(package,src_dir,build_dir,filenames)' tuples.
 
-        Helper function for `finalize_options()`.
+        Helper function for finalize_options.
         """
         data = []
         if not self.packages:
@@ -131,7 +135,7 @@
             # Length of path to strip from found files
             plen = 0
             if src_dir:
-                plen = len(src_dir)+1
+                plen = len(src_dir) + 1
 
             # Strip directory from globbed filenames
             filenames = [
@@ -143,7 +147,7 @@
     def find_data_files(self, package, src_dir):
         """Return filenames for package's data files in 'src_dir'.
 
-        Helper function for `get_data_files()`.
+        Helper function for get_data_files.
         """
         globs = (self.package_data.get('', [])
                  + self.package_data.get(package, []))
@@ -158,7 +162,7 @@
     def build_package_data(self):
         """Copy data files into build directory.
 
-        Helper function for `run()`.
+        Helper function for run.
         """
         # FIXME add tests for this method
         for package, src_dir, build_dir, filenames in self.data_files:
@@ -168,16 +172,17 @@
                 self.mkpath(os.path.dirname(target))
                 outf, copied = self.copy_file(srcfile,
                                target, preserve_mode=False)
-                if copied and srcfile in self.distribution.convert_2to3.doctests:
+                doctests = self.distribution.convert_2to3_doctests
+                if copied and srcfile in doctests:
                     self._doctests_2to3.append(outf)
 
     # XXX - this should be moved to the Distribution class as it is not
     # only needed for build_py. It also has no dependencies on this class.
     def get_package_dir(self, package):
         """Return the directory, relative to the top of the source
-           distribution, where package 'package' should be found
-           (at least according to the 'package_dir' option, if any)."""
-
+        distribution, where package 'package' should be found
+        (at least according to the 'package_dir' option, if any).
+        """
         path = package.split('.')
         if self.package_dir is not None:
             path.insert(0, self.package_dir)
@@ -188,8 +193,7 @@
         return ''
 
     def check_package(self, package, package_dir):
-        """Helper function for `find_package_modules()` and `find_modules()'.
-        """
+        """Helper function for find_package_modules and find_modules."""
         # Empty dir name means current directory, which we can probably
         # assume exists.  Also, os.path.exists and isdir don't know about
         # my "empty string means current dir" convention, so we have to
@@ -209,8 +213,8 @@
             if os.path.isfile(init_py):
                 return init_py
             else:
-                logger.warning(("package init file '%s' not found " +
-                                "(or not a regular file)"), init_py)
+                logger.warning("package init file %r not found "
+                               "(or not a regular file)", init_py)
 
         # Either not in a package at all (__init__.py not expected), or
         # __init__.py doesn't exist -- so don't return the filename.
@@ -218,7 +222,7 @@
 
     def check_module(self, module, module_file):
         if not os.path.isfile(module_file):
-            logger.warning("file %s (for module %s) not found",
+            logger.warning("file %r (for module %r) not found",
                            module_file, module)
             return False
         else:
@@ -239,7 +243,7 @@
                 module = os.path.splitext(os.path.basename(f))[0]
                 modules.append((package, module, f))
             else:
-                logger.debug("excluding %s", setup_script)
+                logger.debug("excluding %r", setup_script)
         return modules
 
     def find_modules(self):
@@ -331,7 +335,8 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(imp.cache_from_source(filename))
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=True))
                 if self.optimize > 0:
                     outputs.append(imp.cache_from_source(filename,
                                                          debug_override=False))
@@ -361,7 +366,6 @@
     def build_modules(self):
         modules = self.find_modules()
         for package, module, module_file in modules:
-
             # Now "build" the module -- ie. copy the source file to
             # self.build_lib (the build directory for Python source).
             # (Actually, it gets copied to the directory for this package
@@ -370,7 +374,6 @@
 
     def build_packages(self):
         for package in self.packages:
-
             # Get list of (package, module, module_file) tuples based on
             # scanning the package directory.  'package' is only included
             # in the tuple so that 'find_modules()' and
@@ -390,11 +393,6 @@
                 self.build_module(module, module_file, package)
 
     def byte_compile(self, files):
-        if sys.dont_write_bytecode:
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
         from packaging.util import byte_compile  # FIXME use compileall
         prefix = self.build_lib
         if prefix[-1] != os.sep:
@@ -403,7 +401,6 @@
         # XXX this code is essentially the same as the 'byte_compile()
         # method of the "install_lib" command, except for the determination
         # of the 'prefix' string.  Hmmm.
-
         if self.compile:
             byte_compile(files, optimize=0,
                          force=self.force, prefix=prefix, dry_run=self.dry_run)
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -2,7 +2,6 @@
 
 import os
 import imp
-import sys
 import logging
 
 from packaging import logger
@@ -11,25 +10,18 @@
 
 
 # Extension for Python source files.
+# XXX dead code?  most of the codebase checks for literal '.py'
 if hasattr(os, 'extsep'):
     PYTHON_SOURCE_EXTENSION = os.extsep + "py"
 else:
     PYTHON_SOURCE_EXTENSION = ".py"
 
+
 class install_lib(Command):
 
     description = "install all modules (extensions and pure Python)"
 
-    # The byte-compilation options are a tad confusing.  Here are the
-    # possible scenarios:
-    #   1) no compilation at all (--no-compile --no-optimize)
-    #   2) compile .pyc only (--compile --no-optimize; default)
-    #   3) compile .pyc and "level 1" .pyo (--compile --optimize)
-    #   4) compile "level 1" .pyo only (--no-compile --optimize)
-    #   5) compile .pyc and "level 2" .pyo (--compile --optimize-more)
-    #   6) compile "level 2" .pyo only (--no-compile --optimize-more)
-    #
-    # The UI for this is two option, 'compile' and 'optimize'.
+    # The options for controlling byte compilations are two independent sets:
     # 'compile' is strictly boolean, and only decides whether to
     # generate .pyc files.  'optimize' is three-way (0, 1, or 2), and
     # decides both whether to generate .pyo files and what level of
@@ -37,7 +29,7 @@
 
     user_options = [
         ('install-dir=', 'd', "directory to install to"),
-        ('build-dir=','b', "build directory (where to install from)"),
+        ('build-dir=', 'b', "build directory (where to install from)"),
         ('force', 'f', "force installation (overwrite existing files)"),
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
@@ -48,7 +40,8 @@
         ]
 
     boolean_options = ['force', 'compile', 'skip-build']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         # let the 'install_dist' command dictate our installation directory
@@ -66,7 +59,8 @@
         self.set_undefined_options('install_dist',
                                    ('build_lib', 'build_dir'),
                                    ('install_lib', 'install_dir'),
-                                   'force', 'compile', 'optimize', 'skip_build')
+                                   'force', 'compile', 'optimize',
+                                   'skip_build')
 
         if self.compile is None:
             self.compile = True
@@ -115,14 +109,6 @@
         return outfiles
 
     def byte_compile(self, files):
-        if sys.dont_write_bytecode:
-            # XXX do we want this?  because a Python runs without bytecode
-            # doesn't mean that the *dists should not contain bytecode
-            #--or does it?
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
         from packaging.util import byte_compile  # FIXME use compileall
 
         # Get the "--root" directory supplied to the "install_dist" command,
@@ -138,13 +124,11 @@
         if self.compile:
             byte_compile(files, optimize=0,
                          force=self.force, prefix=install_root,
-                         dry_run=self.dry_run)
+                         verbose=verbose, dry_run=self.dry_run)
         if self.optimize > 0:
             byte_compile(files, optimize=self.optimize,
                          force=self.force, prefix=install_root,
-                         verbose=verbose,
-                         dry_run=self.dry_run)
-
+                         verbose=verbose, dry_run=self.dry_run)
 
     # -- Utility methods -----------------------------------------------
 
@@ -173,14 +157,14 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(imp.cache_from_source(py_file))
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=True))
             if self.optimize > 0:
                 bytecode_files.append(imp.cache_from_source(
                     py_file, debug_override=False))
 
         return bytecode_files
 
-
     # -- External interface --------------------------------------------
     # (called by outsiders)
 
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -69,7 +69,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ]
     display_option_names = [x[0].replace('-', '_') for x in display_options]
 
diff --git a/Lib/packaging/errors.py b/Lib/packaging/errors.py
--- a/Lib/packaging/errors.py
+++ b/Lib/packaging/errors.py
@@ -72,10 +72,6 @@
     """Syntax error in a file list template."""
 
 
-class PackagingByteCompileError(PackagingError):
-    """Byte compile error."""
-
-
 class PackagingPyPIError(PackagingError):
     """Any problem occuring during using the indexes."""
 
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -53,7 +53,7 @@
     # misc. functions and decorators
     'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
-    'unittest', 'requires_zlib', 'skip_unless_symlink',
+    'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
 
 
@@ -357,3 +357,7 @@
 except ImportError:
     skip_unless_symlink = unittest.skip(
         'requires test.support.skip_unless_symlink')
+
+
+skip_2to3_optimize = unittest.skipIf(sys.flags.optimize,
+                                     "2to3 doesn't work under -O")
diff --git a/Lib/packaging/tests/test_command_bdist_dumb.py b/Lib/packaging/tests/test_command_bdist_dumb.py
--- a/Lib/packaging/tests/test_command_bdist_dumb.py
+++ b/Lib/packaging/tests/test_command_bdist_dumb.py
@@ -1,6 +1,9 @@
 """Tests for distutils.command.bdist_dumb."""
 
 import os
+import imp
+import sys
+import zipfile
 import packaging.util
 
 from packaging.dist import Distribution
@@ -49,15 +52,21 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        with zipfile.ZipFile(os.path.join('dist', base)) as fp:
+            contents = fp.namelist()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo.py',
+                  'foo.%s.pyc' % imp.get_tag(),
+                  'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(contents, sorted(wanted))
 
     def test_finalize_options(self):
         pkg_dir, dist = self.create_dist()
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -62,12 +62,8 @@
         pycache_dir = os.path.join(pkgdest, "__pycache__")
         self.assertIn("__init__.py", files)
         self.assertIn("README.txt", files)
-        if sys.dont_write_bytecode:
-            self.assertFalse(os.path.exists(pycache_dir))
-        else:
-            # XXX even with -O, packaging writes pyc, not pyo; bug?
-            pyc_files = os.listdir(pycache_dir)
-            self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
+        pyc_files = os.listdir(pycache_dir)
+        self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
@@ -102,7 +98,6 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
         os.chdir(project_dir)
@@ -118,7 +113,6 @@
         found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
         self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile_optimized(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
         os.chdir(project_dir)
@@ -136,21 +130,13 @@
         self.assertEqual(sorted(found), ['boiledeggs.%s.pyc' % imp.get_tag(),
                                          'boiledeggs.%s.pyo' % imp.get_tag()])
 
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
-        cmd = build_py(dist)
-        cmd.compile = True
-        cmd.optimize = 1
-
-        old_dont_write_bytecode = sys.dont_write_bytecode
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            cmd.byte_compile([])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
-
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+        self.test_byte_compile()
+        self.test_byte_compile_optimized()
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -17,7 +17,7 @@
     restore_environ = ['PYTHONPATH']
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -34,71 +34,73 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = True
         cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py')
+        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
         pyo_file = imp.cache_from_source('foo.py', debug_override=False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
+        sys.dont_write_bytecode = True
+        self.test_byte_compile()
+
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # make sure the build_lib is set the temp dir
-        build_dir = os.path.split(pkg_dir)[0]
+        # make sure the build_lib is set the temp dir  # XXX what?  this is not
+        # needed in the same distutils test and should work without manual
+        # intervention
+        build_dir = os.path.split(project_dir)[0]
         cmd.get_finalized_command('build_py').build_lib = build_dir
 
-        # get_output should return 4 elements
-        self.assertEqual(len(cmd.get_outputs()), 4)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.import-tag-abiflags.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
-
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
-        cmd = install_lib(dist)
-        cmd.compile = True
-        cmd.optimize = 1
-
-        self.addCleanup(setattr, sys, 'dont_write_bytecode',
-                        sys.dont_write_bytecode)
-        sys.dont_write_bytecode = True
-        cmd.byte_compile([])
-
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.import-tag-abiflags.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -1,9 +1,7 @@
 """Tests for packaging.command.sdist."""
 import os
+import tarfile
 import zipfile
-import tarfile
-
-from packaging.tests.support import requires_zlib
 
 try:
     import grp
@@ -12,16 +10,16 @@
 except ImportError:
     UID_GID_SUPPORT = False
 
+from shutil import get_archive_formats
 from os.path import join
+from packaging.dist import Distribution
+from packaging.util import find_executable
+from packaging.errors import PackagingOptionError
+from packaging.command.sdist import sdist, show_formats
+
+from packaging.tests import support, unittest
 from packaging.tests import captured_stdout
-from packaging.command.sdist import sdist
-from packaging.command.sdist import show_formats
-from packaging.dist import Distribution
-from packaging.tests import unittest
-from packaging.errors import PackagingOptionError
-from packaging.util import find_executable
-from packaging.tests import support
-from shutil import get_archive_formats
+from packaging.tests.support import requires_zlib
 
 
 MANIFEST = """\
@@ -88,7 +86,6 @@
 
         # creating VCS directories with some files in them
         os.mkdir(join(self.tmp_dir, 'somecode', '.svn'))
-
         self.write_file((self.tmp_dir, 'somecode', '.svn', 'ok.py'), 'xxx')
 
         os.mkdir(join(self.tmp_dir, 'somecode', '.hg'))
@@ -216,12 +213,14 @@
         # testing the `check-metadata` option
         dist, cmd = self.get_cmd(metadata={'name': 'xxx', 'version': 'xxx'})
 
-        # this should raise some warnings
-        # with the check subcommand
+        # this should cause the check subcommand to log two warnings:
+        # version is invalid, home-page and author are missing
         cmd.ensure_finalized()
         cmd.run()
         warnings = self.get_logs()
-        self.assertEqual(len(warnings), 4)
+        check_warnings = [msg for msg in warnings if
+                          not msg.startswith('sdist:')]
+        self.assertEqual(len(check_warnings), 2, warnings)
 
         # trying with a complete set of metadata
         self.loghandler.flush()
@@ -244,7 +243,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -264,6 +262,18 @@
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     @requires_zlib
+    def test_template(self):
+        dist, cmd = self.get_cmd()
+        dist.extra_files = ['include yeah']
+        cmd.ensure_finalized()
+        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
+        cmd.run()
+        with open(cmd.manifest) as f:
+            content = f.read()
+
+        self.assertIn('yeah', content)
+
+    @requires_zlib
     @unittest.skipUnless(UID_GID_SUPPORT, "requires grp and pwd support")
     @unittest.skipIf(find_executable('tar') is None or
                      find_executable('gzip') is None,
@@ -368,18 +378,6 @@
         self.assertEqual(manifest, ['README.manual'])
 
     @requires_zlib
-    def test_template(self):
-        dist, cmd = self.get_cmd()
-        dist.extra_files = ['include yeah']
-        cmd.ensure_finalized()
-        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
-        cmd.run()
-        with open(cmd.manifest) as f:
-            content = f.read()
-
-        self.assertIn('yeah', content)
-
-    @requires_zlib
     def test_manifest_builder(self):
         dist, cmd = self.get_cmd()
         cmd.manifest_builders = 'packaging.tests.test_command_sdist.builder'
diff --git a/Lib/packaging/tests/test_mixin2to3.py b/Lib/packaging/tests/test_mixin2to3.py
--- a/Lib/packaging/tests/test_mixin2to3.py
+++ b/Lib/packaging/tests/test_mixin2to3.py
@@ -8,6 +8,7 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
+    @support.skip_2to3_optimize
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -10,7 +10,7 @@
 from io import StringIO
 
 from packaging.errors import (
-    PackagingPlatformError, PackagingByteCompileError, PackagingFileError,
+    PackagingPlatformError, PackagingFileError,
     PackagingExecError, InstallationException)
 from packaging import util
 from packaging.dist import Distribution
@@ -138,15 +138,8 @@
             self._uname = None
         os.uname = self._get_uname
 
-        # patching POpen
-        self.old_find_executable = util.find_executable
-        util.find_executable = self._find_executable
-        self._exes = {}
-        self.old_popen = subprocess.Popen
-        self.old_stdout = sys.stdout
-        self.old_stderr = sys.stderr
-        FakePopen.test_class = self
-        subprocess.Popen = FakePopen
+    def _get_uname(self):
+        return self._uname
 
     def tearDown(self):
         # getting back the environment
@@ -161,17 +154,24 @@
             os.uname = self.uname
         else:
             del os.uname
+        super(UtilTestCase, self).tearDown()
+
+    def mock_popen(self):
+        self.old_find_executable = util.find_executable
+        util.find_executable = self._find_executable
+        self._exes = {}
+        self.old_popen = subprocess.Popen
+        self.old_stdout = sys.stdout
+        self.old_stderr = sys.stderr
+        FakePopen.test_class = self
+        subprocess.Popen = FakePopen
+        self.addCleanup(self.unmock_popen)
+
+    def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
         sys.old_stdout = self.old_stdout
         sys.old_stderr = self.old_stderr
-        super(UtilTestCase, self).tearDown()
-
-    def _set_uname(self, uname):
-        self._uname = uname
-
-    def _get_uname(self):
-        return self._uname
 
     def test_convert_path(self):
         # linux/mac
@@ -283,6 +283,7 @@
         return None
 
     def test_get_compiler_versions(self):
+        self.mock_popen()
         # get_versions calls distutils.spawn.find_executable on
         # 'gcc', 'ld' and 'dllwrap'
         self.assertEqual(get_compiler_versions(), (None, None, None))
@@ -323,15 +324,12 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile raise a PackagingError
-        # if sys.dont_write_bytecode is True
-        old_dont_write_bytecode = sys.dont_write_bytecode
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            self.assertRaises(PackagingByteCompileError, byte_compile, [])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
+        byte_compile([])
 
     def test_newer(self):
         self.assertRaises(PackagingFileError, util.newer, 'xxx', 'xxx')
@@ -418,6 +416,7 @@
         self.assertRaises(ImportError, resolve_name, 'a.b.Spam')
         self.assertRaises(ImportError, resolve_name, 'a.b.c.Spam')
 
+    @support.skip_2to3_optimize
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -431,6 +430,7 @@
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
+    @support.skip_2to3_optimize
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
@@ -448,8 +448,6 @@
     @unittest.skipUnless(os.name in ('nt', 'posix'),
                          'runs only under posix or nt')
     def test_spawn(self):
-        # no patching of Popen here
-        subprocess.Popen = self.old_popen
         tmpdir = self.mkdtemp()
 
         # creating something executable
@@ -545,8 +543,6 @@
         self.assertEqual(args['py_modules'], dist.py_modules)
 
     def test_generate_setup_py(self):
-        # undo subprocess.Popen monkey-patching before using assert_python_*
-        subprocess.Popen = self.old_popen
         os.chdir(self.mkdtemp())
         self.write_file('setup.cfg', textwrap.dedent("""\
             [metadata]
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -20,8 +20,8 @@
 
 from packaging import logger
 from packaging.errors import (PackagingPlatformError, PackagingFileError,
-                              PackagingByteCompileError, PackagingExecError,
-                              InstallationException, PackagingInternalError)
+                              PackagingExecError, InstallationException,
+                              PackagingInternalError)
 
 __all__ = [
     # file dependencies
@@ -325,12 +325,10 @@
     the source for details).  The 'direct' flag is used by the script
     generated in indirect mode; unless you know what you're doing, leave
     it set to None.
+
+    This function is independent from the running Python's -O or -B options;
+    it is fully controlled by the parameters passed in.
     """
-    # nothing is done if sys.dont_write_bytecode is True
-    # FIXME this should not raise an error
-    if sys.dont_write_bytecode:
-        raise PackagingByteCompileError('byte-compiling is disabled.')
-
     # First, if the caller didn't force us into direct or indirect mode,
     # figure out which mode we should be in.  We take a conservative
     # approach: choose direct mode *only* if the current interpreter is
@@ -417,12 +415,10 @@
             # Terminology from the py_compile module:
             #   cfile - byte-compiled file
             #   dfile - purported source filename (same as 'file' by default)
-            if optimize >= 0:
-                cfile = imp.cache_from_source(file,
-                                              debug_override=not optimize)
-            else:
-                cfile = imp.cache_from_source(file)
+            debug_override = not optimize
+            cfile = imp.cache_from_source(file, debug_override)
             dfile = file
+
             if prefix:
                 if file[:len(prefix)] != prefix:
                     raise ValueError("invalid prefix: filename %r doesn't "
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -97,7 +97,7 @@
 
 - Issue #12773: Make __doc__ mutable on user-defined classes.
 
-- Issue #12766: Raise an ValueError when creating a class with a class variable
+- Issue #12766: Raise a ValueError when creating a class with a class variable
   that conflicts with a name in __slots__.
 
 - Issue #12266: Fix str.capitalize() to correctly uppercase/lowercase
@@ -158,7 +158,7 @@
   given, produce a informative error message which includes the name(s) of the
   missing arguments.
 
-- Issue #12370: Fix super with not arguments when __class__ is overriden in the
+- Issue #12370: Fix super with no arguments when __class__ is overriden in the
   class body.
 
 - Issue #12084: os.stat on Windows now works properly with relative symbolic
@@ -350,6 +350,9 @@
 Library
 -------
 
+- Byte compilation in packaging is now isolated from the calling Python -B or
+  -O options, instead of being disallowed under -B or buggy under -O.
+
 - Issue #10570: curses.tigetstr() is now expecting a byte string, instead of
   a Unicode string.
 
@@ -357,8 +360,9 @@
 
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
-- Issue #13287: urllib.request and urllib.error now contains a __all__ and
-  exposes only relevant Classes, Functions. Patch by Florent Xicluna.
+- Issue #13287: urllib.request and urllib.error now contains an __all__
+  attribute to expose only relevant classes and functions.  Patch by Florent
+  Xicluna.
 
 - Issue #670664: Fix HTMLParser to correctly handle the content of
   ``<script>...</script>`` and ``<style>...</style>``.
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -3908,7 +3908,8 @@
 }
 
 PyDoc_STRVAR(doc_cache_from_source,
-"Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
+"cache_from_source(path, [debug_override]) -> path\n\
+Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
 \n\
 The .py file does not need to exist; this simply returns the path to the\n\
 .pyc/.pyo file calculated as if the .py file were imported.  The extension\n\
@@ -3940,7 +3941,8 @@
 }
 
 PyDoc_STRVAR(doc_source_from_cache,
-"Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
+"source_from_cache(path) -> path\n\
+Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
 \n\
 The .pyc/.pyo file does not need to exist; this simply returns the path to\n\
 the .py file calculated to correspond to the .pyc/.pyo file.  If path\n\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:53 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_incorrect_d?=
 =?utf8?q?ocstring_I_changed_a_while_back=2E?=
Message-ID: <E1RLzUH-00032F-UR@dinsdale.python.org>

http://hg.python.org/cpython/rev/65e5b68b24b1
changeset:   73327:65e5b68b24b1
branch:      3.2
parent:      73322:ea926dff958f
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 04:34:09 2011 +0100
summary:
  Fix incorrect docstring I changed a while back.

New wording is taken straight from the PEP, so this time should be good
:)

files:
  Lib/numbers.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/numbers.py b/Lib/numbers.py
--- a/Lib/numbers.py
+++ b/Lib/numbers.py
@@ -303,7 +303,7 @@
         raise NotImplementedError
 
     def __index__(self):
-        """someobject[self]"""
+        """Called whenever an index is needed, such as in slicing"""
         return int(self)
 
     @abstractmethod

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_signatures_?=
 =?utf8?q?to_the_docstring_of_functions_added_to_imp_by_PEP_3147?=
Message-ID: <E1RLzUD-00031O-GV@dinsdale.python.org>

http://hg.python.org/cpython/rev/425591f6c2a5
changeset:   73321:425591f6c2a5
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 03:38:44 2011 +0100
summary:
  Add signatures to the docstring of functions added to imp by PEP 3147

files:
  Python/import.c |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -3534,7 +3534,8 @@
 }
 
 PyDoc_STRVAR(doc_cache_from_source,
-"Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
+"cache_from_source(path, [debug_override]) -> path\n\
+Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
 \n\
 The .py file does not need to exist; this simply returns the path to the\n\
 .pyc/.pyo file calculated as if the .py file were imported.  The extension\n\
@@ -3569,7 +3570,8 @@
 }
 
 PyDoc_STRVAR(doc_source_from_cache,
-"Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
+"source_from_cache(path) -> path\n\
+Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
 \n\
 The .pyc/.pyo file does not need to exist; this simply returns the path to\n\
 the .py file calculated to correspond to the .pyc/.pyo file.  If path\n\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:54 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Avoid_relying_o?=
 =?utf8?q?n_the_default_reST_role_in_logging_library_doc?=
Message-ID: <E1RLzUI-00032m-Ly@dinsdale.python.org>

http://hg.python.org/cpython/rev/695e4d701c37
changeset:   73328:695e4d701c37
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 04:35:20 2011 +0100
summary:
  Avoid relying on the default reST role in logging library doc

files:
  Doc/library/logging.rst |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -137,7 +137,7 @@
 
        Stack (most recent call last):
 
-   This mimics the `Traceback (most recent call last):` which is used when
+   This mimics the ``Traceback (most recent call last):`` which is used when
    displaying exception frames.
 
    The third keyword argument is *extra* which can be used to pass a
@@ -820,7 +820,7 @@
 
        Stack (most recent call last):
 
-   This mimics the `Traceback (most recent call last):` which is used when
+   This mimics the ``Traceback (most recent call last):`` which is used when
    displaying exception frames.
 
    The third optional keyword argument is *extra* which can be used to pass a
@@ -1059,11 +1059,11 @@
    If *capture* is ``True``, warnings issued by the :mod:`warnings` module will
    be redirected to the logging system. Specifically, a warning will be
    formatted using :func:`warnings.formatwarning` and the resulting string
-   logged to a logger named 'py.warnings' with a severity of `WARNING`.
+   logged to a logger named ``'py.warnings'`` with a severity of ``'WARNING'``.
 
    If *capture* is ``False``, the redirection of warnings to the logging system
    will stop, and warnings will be redirected to their original destinations
-   (i.e. those in effect before `captureWarnings(True)` was called).
+   (i.e. those in effect before ``captureWarnings(True)`` was called).
 
 
 .. seealso::

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:55 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:55 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1RLzUJ-00032s-Fj@dinsdale.python.org>

http://hg.python.org/cpython/rev/3e4c0caf56a8
changeset:   73329:3e4c0caf56a8
branch:      3.2
parent:      73313:2c0253d4d9ba
parent:      73328:695e4d701c37
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 16:27:57 2011 +0100
summary:
  Branch merge

files:
  Doc/library/logging.rst                 |   8 +-
  Doc/using/cmdline.rst                   |   3 +-
  Lib/_pyio.py                            |   2 +-
  Lib/distutils/command/build_py.py       |   9 +-
  Lib/distutils/command/install_lib.py    |   6 +-
  Lib/distutils/tests/test_bdist_dumb.py  |  23 +++-
  Lib/distutils/tests/test_build_py.py    |  70 ++++++++----
  Lib/distutils/tests/test_install.py     |  41 +++----
  Lib/distutils/tests/test_install_lib.py |  52 +++++----
  Lib/distutils/tests/test_sdist.py       |   8 +-
  Lib/numbers.py                          |   2 +-
  Python/import.c                         |   6 +-
  12 files changed, 137 insertions(+), 93 deletions(-)


diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -137,7 +137,7 @@
 
        Stack (most recent call last):
 
-   This mimics the `Traceback (most recent call last):` which is used when
+   This mimics the ``Traceback (most recent call last):`` which is used when
    displaying exception frames.
 
    The third keyword argument is *extra* which can be used to pass a
@@ -820,7 +820,7 @@
 
        Stack (most recent call last):
 
-   This mimics the `Traceback (most recent call last):` which is used when
+   This mimics the ``Traceback (most recent call last):`` which is used when
    displaying exception frames.
 
    The third optional keyword argument is *extra* which can be used to pass a
@@ -1059,11 +1059,11 @@
    If *capture* is ``True``, warnings issued by the :mod:`warnings` module will
    be redirected to the logging system. Specifically, a warning will be
    formatted using :func:`warnings.formatwarning` and the resulting string
-   logged to a logger named 'py.warnings' with a severity of `WARNING`.
+   logged to a logger named ``'py.warnings'`` with a severity of ``'WARNING'``.
 
    If *capture* is ``False``, the redirection of warnings to the logging system
    will stop, and warnings will be redirected to their original destinations
-   (i.e. those in effect before `captureWarnings(True)` was called).
+   (i.e. those in effect before ``captureWarnings(True)`` was called).
 
 
 .. seealso::
diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst
--- a/Doc/using/cmdline.rst
+++ b/Doc/using/cmdline.rst
@@ -453,7 +453,8 @@
 .. envvar:: PYTHONDONTWRITEBYTECODE
 
    If this is set, Python won't try to write ``.pyc`` or ``.pyo`` files on the
-   import of source modules.
+   import of source modules.  This is equivalent to specifying the :option:`-B`
+   option.
 
 
 .. envvar:: PYTHONIOENCODING
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -1460,7 +1460,7 @@
     enabled.  With this enabled, on input, the lines endings '\n', '\r',
     or '\r\n' are translated to '\n' before being returned to the
     caller. Conversely, on output, '\n' is translated to the system
-    default line seperator, os.linesep. If newline is any other of its
+    default line separator, os.linesep. If newline is any other of its
     legal values, that newline becomes the newline when the file is read
     and it is returned untranslated. On output, '\n' is converted to the
     newline.
diff --git a/Lib/distutils/command/build_py.py b/Lib/distutils/command/build_py.py
--- a/Lib/distutils/command/build_py.py
+++ b/Lib/distutils/command/build_py.py
@@ -2,8 +2,9 @@
 
 Implements the Distutils 'build_py' command."""
 
-import sys, os
+import os
 import imp
+import sys
 from glob import glob
 
 from distutils.core import Command
@@ -311,9 +312,11 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(imp.cache_from_source(filename, True))
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=True))
                 if self.optimize > 0:
-                    outputs.append(imp.cache_from_source(filename, False))
+                    outputs.append(imp.cache_from_source(filename,
+                                                         debug_override=False))
 
         outputs += [
             os.path.join(build_dir, filename)
diff --git a/Lib/distutils/command/install_lib.py b/Lib/distutils/command/install_lib.py
--- a/Lib/distutils/command/install_lib.py
+++ b/Lib/distutils/command/install_lib.py
@@ -165,9 +165,11 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(imp.cache_from_source(py_file, True))
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=True))
             if self.optimize > 0:
-                bytecode_files.append(imp.cache_from_source(py_file, False))
+                bytecode_files.append(imp.cache_from_source(
+                    py_file, debug_override=False))
 
         return bytecode_files
 
diff --git a/Lib/distutils/tests/test_bdist_dumb.py b/Lib/distutils/tests/test_bdist_dumb.py
--- a/Lib/distutils/tests/test_bdist_dumb.py
+++ b/Lib/distutils/tests/test_bdist_dumb.py
@@ -1,8 +1,10 @@
 """Tests for distutils.command.bdist_dumb."""
 
+import os
+import imp
+import sys
+import zipfile
 import unittest
-import sys
-import os
 from test.support import run_unittest
 
 from distutils.core import Distribution
@@ -72,15 +74,24 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        fp = zipfile.ZipFile(os.path.join('dist', base))
+        try:
+            contents = fp.namelist()
+        finally:
+            fp.close()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo-0.1-py%s.%s.egg-info' % sys.version_info[:2],
+                  'foo.%s.pyc' % imp.get_tag(),
+                  'foo.py']
+        self.assertEqual(contents, sorted(wanted))
 
 def test_suite():
     return unittest.makeSuite(BuildDumbTestCase)
diff --git a/Lib/distutils/tests/test_build_py.py b/Lib/distutils/tests/test_build_py.py
--- a/Lib/distutils/tests/test_build_py.py
+++ b/Lib/distutils/tests/test_build_py.py
@@ -2,7 +2,6 @@
 
 import os
 import sys
-import io
 import imp
 import unittest
 
@@ -54,7 +53,6 @@
         # This makes sure the list of outputs includes byte-compiled
         # files for Python modules but not for package data files
         # (there shouldn't *be* byte-code files for those!).
-        #
         self.assertEqual(len(cmd.get_outputs()), 3)
         pkgdest = os.path.join(destination, "pkg")
         files = os.listdir(pkgdest)
@@ -64,15 +62,11 @@
         if sys.dont_write_bytecode:
             self.assertFalse(os.path.exists(pycache_dir))
         else:
-            # XXX even with -O, distutils writes pyc, not pyo; bug?
             pyc_files = os.listdir(pycache_dir)
             self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
-        # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
-        # create the distribution files.
+        # See bugs #1668596/#1720897
         sources = self.mkdtemp()
         open(os.path.join(sources, "__init__.py"), "w").close()
 
@@ -81,30 +75,55 @@
         open(os.path.join(testdir, "testfile"), "w").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        sys.stdout = io.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": {"pkg": ""},
+                             "package_data": {"pkg": ["doc/*"]}})
+        # script_name need not exist, it just need to be initialized
+        dist.script_name = os.path.join(sources, "setup.py")
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": {"pkg": ""},
-                                 "package_data": {"pkg": ["doc/*"]}})
-            # script_name need not exist, it just need to be initialized
-            dist.script_name = os.path.join(sources, "setup.py")
-            dist.script_args = ["build"]
-            dist.parse_command_line()
+            dist.run_commands()
+        except DistutilsFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
-            try:
-                dist.run_commands()
-            except DistutilsFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
+
+    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
+    def test_byte_compile_optimized(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = 0
+        cmd.optimize = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+        found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+        self.assertEqual(sorted(found), ['boiledeggs.%s.pyo' % imp.get_tag()])
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = build_py(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -118,6 +137,7 @@
 
         self.assertIn('byte-compiling is disabled', self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(BuildPyTestCase)
 
diff --git a/Lib/distutils/tests/test_install.py b/Lib/distutils/tests/test_install.py
--- a/Lib/distutils/tests/test_install.py
+++ b/Lib/distutils/tests/test_install.py
@@ -1,6 +1,7 @@
 """Tests for distutils.command.install."""
 
 import os
+import imp
 import sys
 import unittest
 import site
@@ -67,10 +68,7 @@
         check_path(cmd.install_data, destination)
 
     def test_user_site(self):
-        # site.USER_SITE was introduced in 2.6
-        if sys.version < '2.6':
-            return
-
+        # test install with --user
         # preparing the environment for the test
         self.old_user_base = site.USER_BASE
         self.old_user_site = site.USER_SITE
@@ -87,19 +85,17 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        try:
-            # this is the actual test
-            self._test_user_site()
-        finally:
+        def cleanup():
             site.USER_BASE = self.old_user_base
             site.USER_SITE = self.old_user_site
             install_module.USER_BASE = self.old_user_base
             install_module.USER_SITE = self.old_user_site
             os.path.expanduser = self.old_expand
 
-    def _test_user_site(self):
+        self.addCleanup(cleanup)
+
         for key in ('nt_user', 'unix_user', 'os2_home'):
-            self.assertTrue(key in INSTALL_SCHEMES)
+            self.assertIn(key, INSTALL_SCHEMES)
 
         dist = Distribution({'name': 'xx'})
         cmd = install(dist)
@@ -107,14 +103,14 @@
         # making sure the user option is there
         options = [name for name, short, lable in
                    cmd.user_options]
-        self.assertTrue('user' in options)
+        self.assertIn('user', options)
 
         # setting a value
         cmd.user = 1
 
         # user base and site shouldn't be created yet
-        self.assertTrue(not os.path.exists(self.user_base))
-        self.assertTrue(not os.path.exists(self.user_site))
+        self.assertFalse(os.path.exists(self.user_base))
+        self.assertFalse(os.path.exists(self.user_site))
 
         # let's run finalize
         cmd.ensure_finalized()
@@ -123,8 +119,8 @@
         self.assertTrue(os.path.exists(self.user_base))
         self.assertTrue(os.path.exists(self.user_site))
 
-        self.assertTrue('userbase' in cmd.config_vars)
-        self.assertTrue('usersite' in cmd.config_vars)
+        self.assertIn('userbase', cmd.config_vars)
+        self.assertIn('usersite', cmd.config_vars)
 
     def test_handle_extra_path(self):
         dist = Distribution({'name': 'xx', 'extra_path': 'path,dirs'})
@@ -177,15 +173,16 @@
 
     def test_record(self):
         install_dir = self.mkdtemp()
-        project_dir, dist = self.create_dist(scripts=['hello'])
-        self.addCleanup(os.chdir, os.getcwd())
+        project_dir, dist = self.create_dist(py_modules=['hello'],
+                                             scripts=['sayhi'])
         os.chdir(project_dir)
-        self.write_file('hello', "print('o hai')")
+        self.write_file('hello.py', "def main(): print('o hai')")
+        self.write_file('sayhi', 'from hello import main; main()')
 
         cmd = install(dist)
         dist.command_obj['install'] = cmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -196,7 +193,7 @@
             f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['hello',
+        expected = ['hello.py', 'hello.%s.pyc' % imp.get_tag(), 'sayhi',
                     'UNKNOWN-0.0.0-py%s.%s.egg-info' % sys.version_info[:2]]
         self.assertEqual(found, expected)
 
@@ -204,7 +201,6 @@
         install_dir = self.mkdtemp()
         project_dir, dist = self.create_dist(ext_modules=[
             Extension('xx', ['xxmodule.c'])])
-        self.addCleanup(os.chdir, os.getcwd())
         os.chdir(project_dir)
         support.copy_xxmodule_c(project_dir)
 
@@ -216,7 +212,7 @@
         dist.command_obj['install'] = cmd
         dist.command_obj['build_ext'] = buildextcmd
         cmd.root = install_dir
-        cmd.record = os.path.join(project_dir, 'RECORD')
+        cmd.record = os.path.join(project_dir, 'filelist')
         cmd.ensure_finalized()
         cmd.run()
 
@@ -242,6 +238,7 @@
             install_module.DEBUG = False
         self.assertTrue(len(self.logs) > old_logs_len)
 
+
 def test_suite():
     return unittest.makeSuite(InstallTestCase)
 
diff --git a/Lib/distutils/tests/test_install_lib.py b/Lib/distutils/tests/test_install_lib.py
--- a/Lib/distutils/tests/test_install_lib.py
+++ b/Lib/distutils/tests/test_install_lib.py
@@ -10,13 +10,14 @@
 from distutils.errors import DistutilsOptionError
 from test.support import run_unittest
 
+
 class InstallLibTestCase(support.TempdirManager,
                          support.LoggingSilencer,
                          support.EnvironGuard,
                          unittest.TestCase):
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -35,56 +36,62 @@
 
     @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py')
+        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
         pyo_file = imp.cache_from_source('foo.py', debug_override=False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_output should return 4 elements
-        self.assertTrue(len(cmd.get_outputs()) >= 2)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.import-tag-abiflags.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, 'foo.py')
-        self.write_file(f, '# python file')
-        cmd.distribution.py_modules = [pkg_dir]
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
+        self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
         cmd.distribution.script_name = 'setup.py'
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.import-tag-abiflags.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
     def test_dont_write_bytecode(self):
         # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
         cmd.compile = 1
         cmd.optimize = 1
@@ -98,6 +105,7 @@
 
         self.assertTrue('byte-compiling is disabled' in self.logs[0][1])
 
+
 def test_suite():
     return unittest.makeSuite(InstallLibTestCase)
 
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -288,7 +288,7 @@
     # the following tests make sure there is a nice error message instead
     # of a traceback when parsing an invalid manifest template
 
-    def _test_template(self, content):
+    def _check_template(self, content):
         dist, cmd = self.get_cmd()
         os.chdir(self.tmp_dir)
         self.write_file('MANIFEST.in', content)
@@ -299,17 +299,17 @@
         self.assertEqual(len(warnings), 1)
 
     def test_invalid_template_unknown_command(self):
-        self._test_template('taunt knights *')
+        self._check_template('taunt knights *')
 
     def test_invalid_template_wrong_arguments(self):
         # this manifest command takes one argument
-        self._test_template('prune')
+        self._check_template('prune')
 
     @unittest.skipIf(os.name != 'nt', 'test relevant for Windows only')
     def test_invalid_template_wrong_path(self):
         # on Windows, trailing slashes are not allowed
         # this used to crash instead of raising a warning: #8286
-        self._test_template('include examples/')
+        self._check_template('include examples/')
 
     @unittest.skipUnless(ZLIB_SUPPORT, 'Need zlib support to run')
     def test_get_file_list(self):
diff --git a/Lib/numbers.py b/Lib/numbers.py
--- a/Lib/numbers.py
+++ b/Lib/numbers.py
@@ -303,7 +303,7 @@
         raise NotImplementedError
 
     def __index__(self):
-        """someobject[self]"""
+        """Called whenever an index is needed, such as in slicing"""
         return int(self)
 
     @abstractmethod
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -3534,7 +3534,8 @@
 }
 
 PyDoc_STRVAR(doc_cache_from_source,
-"Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
+"cache_from_source(path, [debug_override]) -> path\n\
+Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
 \n\
 The .py file does not need to exist; this simply returns the path to the\n\
 .pyc/.pyo file calculated as if the .py file were imported.  The extension\n\
@@ -3569,7 +3570,8 @@
 }
 
 PyDoc_STRVAR(doc_source_from_cache,
-"Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
+"source_from_cache(path) -> path\n\
+Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
 \n\
 The .pyc/.pyo file does not need to exist; this simply returns the path to\n\
 the .py file calculated to correspond to the .pyc/.pyo file.  If path\n\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 16:45:56 2011
From: python-checkins at python.org (eric.araujo)
Date: Thu, 03 Nov 2011 16:45:56 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1RLzUK-00032w-33@dinsdale.python.org>

http://hg.python.org/cpython/rev/6df109464e9d
changeset:   73330:6df109464e9d
parent:      73326:a1c17a5a460f
parent:      73329:3e4c0caf56a8
user:        ?ric Araujo <merwok at netwok.org>
date:        Thu Nov 03 16:45:33 2011 +0100
summary:
  Merge 3.2

files:
  Lib/numbers.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/numbers.py b/Lib/numbers.py
--- a/Lib/numbers.py
+++ b/Lib/numbers.py
@@ -303,7 +303,7 @@
         raise NotImplementedError
 
     def __index__(self):
-        """someobject[self]"""
+        """Called whenever an index is needed, such as in slicing"""
         return int(self)
 
     @abstractmethod

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 20:35:30 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 03 Nov 2011 20:35:30 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEwNTcw?=
 =?utf8?q?=3A_curses=2Eputp=28=29_is_now_expecting_a_byte_string=2C_instea?=
 =?utf8?q?d_of_a?=
Message-ID: <E1RM34U-0001Vk-GX@dinsdale.python.org>

http://hg.python.org/cpython/rev/38f4a251608f
changeset:   73331:38f4a251608f
branch:      3.2
parent:      73329:3e4c0caf56a8
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 03 20:35:40 2011 +0100
summary:
  Issue #10570: curses.putp() is now expecting a byte string, instead of a
Unicode string.

This is an incompatible change, but putp() is used to emit terminfo commands,
which are bytes strings, not Unicode strings.

files:
  Lib/test/test_curses.py |  3 ++-
  Misc/NEWS               |  4 ++--
  Modules/_cursesmodule.c |  3 ++-
  3 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -183,7 +183,7 @@
     win = curses.newwin(5,5)
     win = curses.newwin(5,5, 1,1)
     curses.nl() ; curses.nl(1)
-    curses.putp('abc')
+    curses.putp(b'abc')
     curses.qiflush()
     curses.raw() ; curses.raw(1)
     curses.setsyx(5,5)
@@ -267,6 +267,7 @@
 def test_issue10570():
     b = curses.tparm(curses.tigetstr("cup"), 5, 3)
     assert type(b) is bytes
+    curses.putp(b)
 
 def main(stdscr):
     curses.savetty()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,8 +66,8 @@
 Library
 -------
 
-- Issue #10570: curses.tigetstr() is now expecting a byte string, instead of
-  a Unicode string.
+- Issue #10570: curses.putp() and curses.tigetstr() are now expecting a byte
+  string, instead of a Unicode string.
 
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2379,7 +2379,8 @@
 {
     char *str;
 
-    if (!PyArg_ParseTuple(args,"s;str", &str)) return NULL;
+    if (!PyArg_ParseTuple(args,"y;str", &str))
+        return NULL;
     return PyCursesCheckERR(putp(str), "putp");
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 20:35:31 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 03 Nov 2011 20:35:31 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiAoTWVyZ2UgMy4yKSBJc3N1ZSAjMTA1NzA6IGN1cnNlcy5wdXRwKCkgaXMgbm93?=
 =?utf8?q?_expecting_a_byte_string=2C_instead?=
Message-ID: <E1RM34V-0001Vo-DA@dinsdale.python.org>

http://hg.python.org/cpython/rev/08f44eb760a6
changeset:   73332:08f44eb760a6
parent:      73330:6df109464e9d
parent:      73331:38f4a251608f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 03 20:36:55 2011 +0100
summary:
  (Merge 3.2) Issue #10570: curses.putp() is now expecting a byte string, instead
of a Unicode string.

This is an incompatible change, but putp() is used to emit terminfo commands,
which are bytes strings, not Unicode strings.

files:
  Lib/test/test_curses.py |  3 ++-
  Misc/NEWS               |  4 ++--
  Modules/_cursesmodule.c |  3 ++-
  3 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -183,7 +183,7 @@
     win = curses.newwin(5,5)
     win = curses.newwin(5,5, 1,1)
     curses.nl() ; curses.nl(1)
-    curses.putp('abc')
+    curses.putp(b'abc')
     curses.qiflush()
     curses.raw() ; curses.raw(1)
     curses.setsyx(5,5)
@@ -283,6 +283,7 @@
 def test_issue10570():
     b = curses.tparm(curses.tigetstr("cup"), 5, 3)
     assert type(b) is bytes
+    curses.putp(b)
 
 def main(stdscr):
     curses.savetty()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -353,8 +353,8 @@
 - Byte compilation in packaging is now isolated from the calling Python -B or
   -O options, instead of being disallowed under -B or buggy under -O.
 
-- Issue #10570: curses.tigetstr() is now expecting a byte string, instead of
-  a Unicode string.
+- Issue #10570: curses.putp() and curses.tigetstr() are now expecting a byte
+  string, instead of a Unicode string.
 
 - Issue #13295: http.server now produces valid HTML 4.01 strict.
 
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -2418,7 +2418,8 @@
 {
     char *str;
 
-    if (!PyArg_ParseTuple(args,"s;str", &str)) return NULL;
+    if (!PyArg_ParseTuple(args,"y;str", &str))
+        return NULL;
     return PyCursesCheckERR(putp(str), "putp");
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 22:31:06 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 03 Nov 2011 22:31:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Adapt_the_code_page_encoder?=
 =?utf8?q?_to_the_new_unicode=5Fencode=5Fcall=5Ferrorhandler=28=29?=
Message-ID: <E1RM4sM-0004La-Ni@dinsdale.python.org>

http://hg.python.org/cpython/rev/09884be552f8
changeset:   73333:09884be552f8
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 03 22:32:33 2011 +0100
summary:
  Adapt the code page encoder to the new unicode_encode_call_errorhandler()

The code is not correct, but at least it doesn't crash anymore.

files:
  Objects/unicodeobject.c |  88 +++++++++++++++++-----------
  1 files changed, 52 insertions(+), 36 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7405,6 +7405,7 @@
  */
 static int
 encode_code_page_errors(UINT code_page, PyObject **outbytes,
+                        PyObject *unicode, Py_ssize_t unicode_offset,
                         const Py_UNICODE *in, const int insize,
                         const char* errors)
 {
@@ -7505,14 +7506,14 @@
         }
 
         charsize = Py_MAX(charsize - 1, 1);
-        startpos = in - startin;
+        startpos = unicode_offset + in - startin;
         rep = unicode_encode_call_errorhandler(
                   errors, &errorHandler, encoding, reason,
-                  startin, insize, &exc,
+                  unicode, &exc,
                   startpos, startpos + charsize, &newpos);
         if (rep == NULL)
             goto error;
-        in = startin + newpos;
+        in += (newpos - startpos);
 
         if (PyBytes_Check(rep)) {
             outsize = PyBytes_GET_SIZE(rep);
@@ -7590,6 +7591,7 @@
  */
 static int
 encode_code_page_chunk(UINT code_page, PyObject **outbytes,
+                       PyObject *unicode, Py_ssize_t unicode_offset,
                        const Py_UNICODE *p, int size,
                        const char* errors)
 {
@@ -7604,45 +7606,60 @@
         return 0;
     }
 
-    done = encode_code_page_strict(code_page, outbytes, p, size, errors);
+    done = encode_code_page_strict(code_page, outbytes,
+                                   p, size,
+                                   errors);
     if (done == -2)
-        done = encode_code_page_errors(code_page, outbytes, p, size, errors);
+        done = encode_code_page_errors(code_page, outbytes,
+                                       unicode, unicode_offset,
+                                       p, size,
+                                       errors);
     return done;
 }
 
 static PyObject *
 encode_code_page(int code_page,
-                 const Py_UNICODE *p, Py_ssize_t size,
+                 PyObject *unicode,
                  const char *errors)
 {
+    const Py_UNICODE *p;
+    Py_ssize_t size;
     PyObject *outbytes = NULL;
-    int ret;
+    Py_ssize_t offset;
+    int chunk_len, ret;
+
+    p = PyUnicode_AsUnicodeAndSize(unicode, &size);
+    if (p == NULL)
+        return NULL;
 
     if (code_page < 0) {
         PyErr_SetString(PyExc_ValueError, "invalid code page number");
         return NULL;
     }
 
+    offset = 0;
+    do
+    {
 #ifdef NEED_RETRY
-  retry:
-    if (size > INT_MAX)
-        ret = encode_code_page_chunk(code_page, &outbytes, p, INT_MAX, errors);
-    else
-#endif
-        ret = encode_code_page_chunk(code_page, &outbytes, p, (int)size, errors);
-
-    if (ret < 0) {
-        Py_XDECREF(outbytes);
-        return NULL;
-    }
-
-#ifdef NEED_RETRY
-    if (size > INT_MAX) {
-        p += INT_MAX;
-        size -= INT_MAX;
-        goto retry;
-    }
-#endif
+        if (size > INT_MAX)
+            chunk_len = INT_MAX;
+        else
+#endif
+            chunk_len = (int)size;
+        ret = encode_code_page_chunk(code_page, &outbytes,
+                                     unicode, offset,
+                                     p, chunk_len,
+                                     errors);
+
+        if (ret < 0) {
+            Py_XDECREF(outbytes);
+            return NULL;
+        }
+
+        p += chunk_len;
+        offset += chunk_len;
+        size -= chunk_len;
+    } while (size != 0);
 
     return outbytes;
 }
@@ -7652,7 +7669,13 @@
                      Py_ssize_t size,
                      const char *errors)
 {
-    return encode_code_page(CP_ACP, p, size, errors);
+    PyObject *unicode, *res;
+    unicode = PyUnicode_FromUnicode(p, size);
+    if (unicode == NULL)
+        return NULL;
+    res = encode_code_page(CP_ACP, unicode, errors);
+    Py_DECREF(unicode);
+    return res;
 }
 
 PyObject *
@@ -7660,12 +7683,7 @@
                          PyObject *unicode,
                          const char *errors)
 {
-    const Py_UNICODE *p;
-    Py_ssize_t size;
-    p = PyUnicode_AsUnicodeAndSize(unicode, &size);
-    if (p == NULL)
-        return NULL;
-    return encode_code_page(code_page, p, size, errors);
+    return encode_code_page(code_page, unicode, errors);
 }
 
 PyObject *
@@ -7675,9 +7693,7 @@
         PyErr_BadArgument();
         return NULL;
     }
-    return PyUnicode_EncodeMBCS(PyUnicode_AS_UNICODE(unicode),
-                                PyUnicode_GET_SIZE(unicode),
-                                NULL);
+    return PyUnicode_EncodeCodePage(CP_ACP, unicode, NULL);
 }
 
 #undef NEED_RETRY

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 23:12:00 2011
From: python-checkins at python.org (florent.xicluna)
Date: Thu, 03 Nov 2011 23:12:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_PEP-3151_exceptions_for?=
	=?utf8?q?_test=5Fpep277=2E?=
Message-ID: <E1RM5Vw-00007t-EN@dinsdale.python.org>

http://hg.python.org/cpython/rev/e71de9396e45
changeset:   73334:e71de9396e45
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Thu Nov 03 23:11:14 2011 +0100
summary:
  Use PEP-3151 exceptions for test_pep277.

files:
  Lib/test/test_pep277.py |  42 +++++++++++++++-------------
  1 files changed, 22 insertions(+), 20 deletions(-)


diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -38,8 +38,8 @@
         '17_\u2001\u2001\u2001A',
         '18_\u2003\u2003\u2003A',  # == NFC('\u2001\u2001\u2001A')
         '19_\u0020\u0020\u0020A',  # '\u0020' == ' ' == NFKC('\u2000') ==
-                                #  NFKC('\u2001') == NFKC('\u2003')
-])
+                                   #  NFKC('\u2001') == NFKC('\u2003')
+    ])
 
 
 # Is it Unicode-friendly?
@@ -71,7 +71,7 @@
     def setUp(self):
         try:
             os.mkdir(support.TESTFN)
-        except OSError:
+        except FileExistsError:
             pass
         files = set()
         for name in self.files:
@@ -90,15 +90,16 @@
             return normalize(self.normal_form, s)
         return s
 
-    def _apply_failure(self, fn, filename, expected_exception,
-                       check_fn_in_exception = True):
+    def _apply_failure(self, fn, filename,
+                       expected_exception=FileNotFoundError,
+                       check_filename=True):
         with self.assertRaises(expected_exception) as c:
             fn(filename)
         exc_filename = c.exception.filename
         # the "filename" exception attribute may be encoded
         if isinstance(exc_filename, bytes):
             filename = filename.encode(sys.getfilesystemencoding())
-        if check_fn_in_exception:
+        if check_filename:
             self.assertEqual(exc_filename, filename, "Function '%s(%a) failed "
                              "with bad filename in the exception: %a" %
                              (fn.__name__, filename, exc_filename))
@@ -107,13 +108,13 @@
         # Pass non-existing Unicode filenames all over the place.
         for name in self.files:
             name = "not_" + name
-            self._apply_failure(open, name, IOError)
-            self._apply_failure(os.stat, name, OSError)
-            self._apply_failure(os.chdir, name, OSError)
-            self._apply_failure(os.rmdir, name, OSError)
-            self._apply_failure(os.remove, name, OSError)
+            self._apply_failure(open, name)
+            self._apply_failure(os.stat, name)
+            self._apply_failure(os.chdir, name)
+            self._apply_failure(os.rmdir, name)
+            self._apply_failure(os.remove, name)
             # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, OSError, False)
+            self._apply_failure(os.listdir, name, check_filename=False)
 
     def test_open(self):
         for name in self.files:
@@ -121,12 +122,13 @@
             f.write((name+'\n').encode("utf-8"))
             f.close()
             os.stat(name)
+            self._apply_failure(os.listdir, name, NotADirectoryError)
 
     # Skip the test on darwin, because darwin does normalize the filename to
     # NFD (a variant of Unicode NFD form). Normalize the filename to NFC, NFKC,
     # NFKD in Python is useless, because darwin will normalize it later and so
     # open(), os.stat(), etc. don't raise any exception.
-    @unittest.skipIf(sys.platform == 'darwin', 'irrevelant test on Mac OS X')
+    @unittest.skipIf(sys.platform == 'darwin', 'irrelevant test on Mac OS X')
     def test_normalize(self):
         files = set(self.files)
         others = set()
@@ -134,18 +136,18 @@
             others |= set(normalize(nf, file) for file in files)
         others -= files
         for name in others:
-            self._apply_failure(open, name, IOError)
-            self._apply_failure(os.stat, name, OSError)
-            self._apply_failure(os.chdir, name, OSError)
-            self._apply_failure(os.rmdir, name, OSError)
-            self._apply_failure(os.remove, name, OSError)
+            self._apply_failure(open, name)
+            self._apply_failure(os.stat, name)
+            self._apply_failure(os.chdir, name)
+            self._apply_failure(os.rmdir, name)
+            self._apply_failure(os.remove, name)
             # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, OSError, False)
+            self._apply_failure(os.listdir, name, False)
 
     # Skip the test on darwin, because darwin uses a normalization different
     # than Python NFD normalization: filenames are different even if we use
     # Python NFD normalization.
-    @unittest.skipIf(sys.platform == 'darwin', 'irrevelant test on Mac OS X')
+    @unittest.skipIf(sys.platform == 'darwin', 'irrelevant test on Mac OS X')
     def test_listdir(self):
         sf0 = set(self.files)
         f1 = os.listdir(support.TESTFN.encode(sys.getfilesystemencoding()))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov  3 23:44:24 2011
From: python-checkins at python.org (florent.xicluna)
Date: Thu, 03 Nov 2011 23:44:24 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Oversight_in_previous_commi?=
	=?utf8?q?t_for_test=5Fpep277=2E?=
Message-ID: <E1RM61I-0002bC-PG@dinsdale.python.org>

http://hg.python.org/cpython/rev/7d5ac0fc8a35
changeset:   73335:7d5ac0fc8a35
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Thu Nov 03 23:44:15 2011 +0100
summary:
  Oversight in previous commit for test_pep277.

files:
  Lib/test/test_pep277.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -142,7 +142,7 @@
             self._apply_failure(os.rmdir, name)
             self._apply_failure(os.remove, name)
             # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, False)
+            self._apply_failure(os.listdir, name, check_filename=False)
 
     # Skip the test on darwin, because darwin uses a normalization different
     # than Python NFD normalization: filenames are different even if we use

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 00:04:10 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 00:04:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Cleanup_decode=5Fcode=5Fpag?=
 =?utf8?b?ZV9zdGF0ZWZ1bCgpIGFuZCBlbmNvZGVfY29kZV9wYWdlKCk=?=
Message-ID: <E1RM6KQ-0004OL-4Z@dinsdale.python.org>

http://hg.python.org/cpython/rev/17341b93871b
changeset:   73336:17341b93871b
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 00:05:13 2011 +0100
summary:
  Cleanup decode_code_page_stateful() and encode_code_page()

 * Fix decode_code_page_errors() result
 * Inline decode_code_page() and encode_code_page_chunk()
 * Replace the PyUnicodeObject type by PyObject

files:
  Lib/test/test_codecs.py |    3 +
  Objects/unicodeobject.c |  198 +++++++++++----------------
  2 files changed, 84 insertions(+), 117 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1980,6 +1980,9 @@
             ))
 
     def test_incremental(self):
+        decoded = codecs.code_page_decode(932, b'\x82', 'strict', False)
+        self.assertEqual(decoded, ('', 0))
+
         decoded = codecs.code_page_decode(932,
                                           b'\xe9\x80\xe9', 'strict',
                                           False)
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7006,7 +7006,7 @@
  */
 static int
 decode_code_page_strict(UINT code_page,
-                        PyUnicodeObject **v,
+                        PyObject **v,
                         const char *in,
                         int insize)
 {
@@ -7022,7 +7022,7 @@
 
     if (*v == NULL) {
         /* Create unicode object */
-        *v = _PyUnicode_New(outsize);
+        *v = (PyObject*)_PyUnicode_New(outsize);
         if (*v == NULL)
             return -1;
         out = PyUnicode_AS_UNICODE(*v);
@@ -7030,7 +7030,7 @@
     else {
         /* Extend unicode object */
         Py_ssize_t n = PyUnicode_GET_SIZE(*v);
-        if (PyUnicode_Resize((PyObject**)v, n + outsize) < 0)
+        if (PyUnicode_Resize(v, n + outsize) < 0)
             return -1;
         out = PyUnicode_AS_UNICODE(*v) + n;
     }
@@ -7057,9 +7057,8 @@
  */
 static int
 decode_code_page_errors(UINT code_page,
-                        PyUnicodeObject **v,
-                        const char *in,
-                        int size,
+                        PyObject **v,
+                        const char *in, const int size,
                         const char *errors)
 {
     const char *startin = in;
@@ -7103,7 +7102,7 @@
             PyErr_NoMemory();
             goto error;
         }
-        *v = _PyUnicode_New(size * Py_ARRAY_LENGTH(buffer));
+        *v = (PyObject*)_PyUnicode_New(size * Py_ARRAY_LENGTH(buffer));
         if (*v == NULL)
             goto error;
         startout = PyUnicode_AS_UNICODE(*v);
@@ -7115,7 +7114,7 @@
             PyErr_NoMemory();
             goto error;
         }
-        if (PyUnicode_Resize((PyObject**)v, n + size * Py_ARRAY_LENGTH(buffer)) < 0)
+        if (PyUnicode_Resize(v, n + size * Py_ARRAY_LENGTH(buffer)) < 0)
             goto error;
         startout = PyUnicode_AS_UNICODE(*v) + n;
     }
@@ -7173,9 +7172,9 @@
     /* Extend unicode object */
     outsize = out - startout;
     assert(outsize <= PyUnicode_WSTR_LENGTH(*v));
-    if (PyUnicode_Resize((PyObject**)v, outsize) < 0)
+    if (PyUnicode_Resize(v, outsize) < 0)
         goto error;
-    ret = 0;
+    ret = size;
 
 error:
     Py_XDECREF(encoding_obj);
@@ -7184,50 +7183,13 @@
     return ret;
 }
 
-/*
- * Decode a byte string from a Windows code page into unicode object. If
- * 'final' is set, converts trailing lead-byte too.
- *
- * Returns consumed size if succeed, or raise a WindowsError or
- * UnicodeDecodeError exception and returns -1 on error.
- */
-static int
-decode_code_page(UINT code_page,
-                 PyUnicodeObject **v,
-                 const char *s,  int size,
-                 int final, const char *errors)
-{
-    int done;
-
-    /* Skip trailing lead-byte unless 'final' is set */
-    if (size == 0) {
-        if (*v == NULL) {
-            Py_INCREF(unicode_empty);
-            *v = (PyUnicodeObject*)unicode_empty;
-            if (*v == NULL)
-                return -1;
-        }
-        return 0;
-    }
-
-    if (!final && is_dbcs_lead_byte(code_page, s, size - 1))
-        --size;
-
-    done = decode_code_page_strict(code_page, v, s, size);
-    if (done == -2)
-        done = decode_code_page_errors(code_page, v, s, size, errors);
-    return done;
-}
-
 static PyObject *
 decode_code_page_stateful(int code_page,
-                          const char *s,
-                          Py_ssize_t size,
-                          const char *errors,
-                          Py_ssize_t *consumed)
-{
-    PyUnicodeObject *v = NULL;
-    int done;
+                          const char *s, Py_ssize_t size,
+                          const char *errors, Py_ssize_t *consumed)
+{
+    PyObject *v = NULL;
+    int chunk_size, final, converted, done;
 
     if (code_page < 0) {
         PyErr_SetString(PyExc_ValueError, "invalid code page number");
@@ -7237,29 +7199,53 @@
     if (consumed)
         *consumed = 0;
 
+    do
+    {
 #ifdef NEED_RETRY
-  retry:
-    if (size > INT_MAX)
-        done = decode_code_page(code_page, &v, s, INT_MAX, 0, errors);
-    else
-#endif
-        done = decode_code_page(code_page, &v, s, (int)size, !consumed, errors);
-
-    if (done < 0) {
-        Py_XDECREF(v);
-        return NULL;
-    }
-
-    if (consumed)
-        *consumed += done;
-
-#ifdef NEED_RETRY
-    if (size > INT_MAX) {
-        s += done;
-        size -= done;
-        goto retry;
-    }
-#endif
+        if (size > INT_MAX) {
+            chunk_size = INT_MAX;
+            final = 0;
+            done = 0;
+        }
+        else
+#endif
+        {
+            chunk_size = (int)size;
+            final = (consumed == NULL);
+            done = 1;
+        }
+
+        /* Skip trailing lead-byte unless 'final' is set */
+        if (!final && is_dbcs_lead_byte(code_page, s, chunk_size - 1))
+            --chunk_size;
+
+        if (chunk_size == 0 && done) {
+            if (v != NULL)
+                break;
+            Py_INCREF(unicode_empty);
+            return unicode_empty;
+        }
+
+
+        converted = decode_code_page_strict(code_page, &v,
+                                            s, chunk_size);
+        if (converted == -2)
+            converted = decode_code_page_errors(code_page, &v,
+                                                s, chunk_size,
+                                                errors);
+        assert(converted != 0);
+
+        if (converted < 0) {
+            Py_XDECREF(v);
+            return NULL;
+        }
+
+        if (consumed)
+            *consumed += converted;
+
+        s += converted;
+        size -= converted;
+    } while (!done);
 
 #ifndef DONT_MAKE_RESULT_READY
     if (_PyUnicode_READY_REPLACE(&v)) {
@@ -7268,7 +7254,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(v, 1));
-    return (PyObject *)v;
+    return v;
 }
 
 PyObject *
@@ -7583,40 +7569,6 @@
     return ret;
 }
 
-/*
- * Encode a Unicode string to a Windows code page into a byte string.
- *
- * Returns consumed characters if succeed, or raise a WindowsError and returns
- * -1 on other error.
- */
-static int
-encode_code_page_chunk(UINT code_page, PyObject **outbytes,
-                       PyObject *unicode, Py_ssize_t unicode_offset,
-                       const Py_UNICODE *p, int size,
-                       const char* errors)
-{
-    int done;
-
-    if (size == 0) {
-        if (*outbytes == NULL) {
-            *outbytes = PyBytes_FromStringAndSize(NULL, 0);
-            if (*outbytes == NULL)
-                return -1;
-        }
-        return 0;
-    }
-
-    done = encode_code_page_strict(code_page, outbytes,
-                                   p, size,
-                                   errors);
-    if (done == -2)
-        done = encode_code_page_errors(code_page, outbytes,
-                                       unicode, unicode_offset,
-                                       p, size,
-                                       errors);
-    return done;
-}
-
 static PyObject *
 encode_code_page(int code_page,
                  PyObject *unicode,
@@ -7626,7 +7578,7 @@
     Py_ssize_t size;
     PyObject *outbytes = NULL;
     Py_ssize_t offset;
-    int chunk_len, ret;
+    int chunk_len, ret, done;
 
     p = PyUnicode_AsUnicodeAndSize(unicode, &size);
     if (p == NULL)
@@ -7637,20 +7589,32 @@
         return NULL;
     }
 
+    if (size == 0)
+        return PyBytes_FromStringAndSize(NULL, 0);
+
     offset = 0;
     do
     {
 #ifdef NEED_RETRY
-        if (size > INT_MAX)
+        if (size > INT_MAX) {
             chunk_len = INT_MAX;
+            done = 0;
+        }
         else
 #endif
+        {
             chunk_len = (int)size;
-        ret = encode_code_page_chunk(code_page, &outbytes,
-                                     unicode, offset,
-                                     p, chunk_len,
-                                     errors);
-
+            done = 1;
+        }
+
+        ret = encode_code_page_strict(code_page, &outbytes,
+                                      p, chunk_len,
+                                      errors);
+        if (ret == -2)
+            ret = encode_code_page_errors(code_page, &outbytes,
+                                          unicode, offset,
+                                          p, chunk_len,
+                                          errors);
         if (ret < 0) {
             Py_XDECREF(outbytes);
             return NULL;
@@ -7659,7 +7623,7 @@
         p += chunk_len;
         offset += chunk_len;
         size -= chunk_len;
-    } while (size != 0);
+    } while (!done);
 
     return outbytes;
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 00:22:31 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 00:22:31 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_PyUnicodeObject_typ?=
	=?utf8?q?e_by_PyObject?=
Message-ID: <E1RM6cB-0005ox-F4@dinsdale.python.org>

http://hg.python.org/cpython/rev/783b4b9dfa5e
changeset:   73337:783b4b9dfa5e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 00:22:48 2011 +0100
summary:
  Replace PyUnicodeObject type by PyObject

 * _PyUnicode_CheckConsistency() now takes a PyObject* instead of void*
 * Remove now useless casts to PyObject*

files:
  Include/unicodeobject.h            |    3 +-
  Objects/stringlib/unicode_format.h |   20 +-
  Objects/unicodeobject.c            |  266 ++++++++--------
  3 files changed, 137 insertions(+), 152 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -2000,9 +2000,8 @@
 #endif /* Py_LIMITED_API */
 
 #if defined(Py_DEBUG) && !defined(Py_LIMITED_API)
-/* FIXME: use PyObject* type for op */
 PyAPI_FUNC(int) _PyUnicode_CheckConsistency(
-    void *op,
+    PyObject *op,
     int check_content);
 #endif
 
diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h
--- a/Objects/stringlib/unicode_format.h
+++ b/Objects/stringlib/unicode_format.h
@@ -528,7 +528,7 @@
     PyObject *result = NULL;
     PyObject *format_spec_object = NULL;
     PyObject *(*formatter)(PyObject *, PyObject *, Py_ssize_t, Py_ssize_t) = NULL;
-    
+
     /* If we know the type exactly, skip the lookup of __format__ and just
        call the formatter directly. */
     if (PyUnicode_CheckExact(fieldobj))
@@ -654,7 +654,7 @@
 } MarkupIterator;
 
 static int
-MarkupIterator_init(MarkupIterator *self, PyObject *str, 
+MarkupIterator_init(MarkupIterator *self, PyObject *str,
                     Py_ssize_t start, Py_ssize_t end)
 {
     SubString_init(&self->str, str, start, end);
@@ -713,8 +713,8 @@
     at_end = self->str.start >= self->str.end;
     len = self->str.start - start;
 
-    if ((c == '}') && (at_end || 
-                       (c != PyUnicode_READ_CHAR(self->str.str, 
+    if ((c == '}') && (at_end ||
+                       (c != PyUnicode_READ_CHAR(self->str.str,
                                                  self->str.start)))) {
         PyErr_SetString(PyExc_ValueError, "Single '}' encountered "
                         "in format string");
@@ -992,9 +992,7 @@
 
 typedef struct {
     PyObject_HEAD
-
-    PyUnicodeObject *str;
-
+    PyObject *str;
     MarkupIterator it_markup;
 } formatteriterobject;
 
@@ -1121,7 +1119,7 @@
    describing the parsed elements.  It's a wrapper around
    stringlib/string_format.h's MarkupIterator */
 static PyObject *
-formatter_parser(PyObject *ignored, PyUnicodeObject *self)
+formatter_parser(PyObject *ignored, PyObject *self)
 {
     formatteriterobject *it;
 
@@ -1158,9 +1156,7 @@
 
 typedef struct {
     PyObject_HEAD
-
-    PyUnicodeObject *str;
-
+    PyObject *str;
     FieldNameIterator it_field;
 } fieldnameiterobject;
 
@@ -1261,7 +1257,7 @@
    field_name_split.  The iterator it returns is a
    FieldNameIterator */
 static PyObject *
-formatter_field_name_split(PyObject *ignored, PyUnicodeObject *self)
+formatter_field_name_split(PyObject *ignored, PyObject *self)
 {
     SubString first;
     Py_ssize_t first_idx;
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -116,7 +116,7 @@
     (assert(_PyUnicode_CHECK(op)),                      \
      (PyUnicode_IS_READY(op) ?                          \
       0 :                                               \
-      _PyUnicode_Ready((PyObject *)(op))))
+      _PyUnicode_Ready(op)))
 
 #define _PyUnicode_READY_REPLACE(p_obj)                 \
     (assert(_PyUnicode_CHECK(*p_obj)),                  \
@@ -308,8 +308,7 @@
 
 #ifdef Py_DEBUG
 int
-/* FIXME: use PyObject* type for op */
-_PyUnicode_CheckConsistency(void *op, int check_content)
+_PyUnicode_CheckConsistency(PyObject *op, int check_content)
 {
     PyASCIIObject *ascii;
     unsigned int kind;
@@ -406,7 +405,7 @@
         else
             assert(maxchar >= 0x10000);
     }
-    if (check_content && !unicode_is_singleton((PyObject*)ascii))
+    if (check_content && !unicode_is_singleton(op))
         assert(ascii->hash == -1);
     return 1;
 }
@@ -764,7 +763,7 @@
     _PyUnicode_LENGTH(unicode) = 0;
     _PyUnicode_UTF8(unicode) = NULL;
     _PyUnicode_UTF8_LENGTH(unicode) = 0;
-    assert(_PyUnicode_CheckConsistency(unicode, 0));
+    assert(_PyUnicode_CheckConsistency((PyObject *)unicode, 0));
     return unicode;
 
   onError:
@@ -976,7 +975,7 @@
             _PyUnicode_WSTR(unicode) = NULL;
         }
     }
-    assert(_PyUnicode_CheckConsistency(unicode, 0));
+    assert(_PyUnicode_CheckConsistency((PyObject*)unicode, 0));
     return obj;
 }
 
@@ -1438,7 +1437,7 @@
     case SSTATE_INTERNED_MORTAL:
         /* revive dead object temporarily for DelItem */
         Py_REFCNT(unicode) = 3;
-        if (PyDict_DelItem(interned, (PyObject *)unicode) != 0)
+        if (PyDict_DelItem(interned, unicode) != 0)
             Py_FatalError(
                 "deletion of interned string failed");
         break;
@@ -1456,12 +1455,12 @@
         PyObject_DEL(_PyUnicode_UTF8(unicode));
 
     if (PyUnicode_IS_COMPACT(unicode)) {
-        Py_TYPE(unicode)->tp_free((PyObject *)unicode);
+        Py_TYPE(unicode)->tp_free(unicode);
     }
     else {
         if (_PyUnicode_DATA_ANY(unicode))
             PyObject_DEL(_PyUnicode_DATA_ANY(unicode));
-        Py_TYPE(unicode)->tp_free((PyObject *)unicode);
+        Py_TYPE(unicode)->tp_free(unicode);
     }
 }
 
@@ -2607,7 +2606,7 @@
     if (numberresults)
         PyObject_Free(numberresults);
     assert(_PyUnicode_CheckConsistency(string, 1));
-    return (PyObject *)string;
+    return string;
   fail:
     if (callresults) {
         PyObject **callresult2 = callresults;
@@ -3586,7 +3585,7 @@
                                  const char *encoding, const char *reason,
                                  const char **input, const char **inend, Py_ssize_t *startinpos,
                                  Py_ssize_t *endinpos, PyObject **exceptionObject, const char **inptr,
-                                 PyUnicodeObject **output, Py_ssize_t *outpos, Py_UNICODE **outptr)
+                                 PyObject **output, Py_ssize_t *outpos, Py_UNICODE **outptr)
 {
     static char *argparse = "O!n;decoding error handler must return (str, int) tuple";
 
@@ -3657,7 +3656,7 @@
     if (requiredsize > outsize) {
         if (requiredsize<2*outsize)
             requiredsize = 2*outsize;
-        if (PyUnicode_Resize((PyObject**)output, requiredsize) < 0)
+        if (PyUnicode_Resize(output, requiredsize) < 0)
             goto onError;
         *outptr = PyUnicode_AS_UNICODE(*output) + *outpos;
     }
@@ -3782,7 +3781,7 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     const char *e;
-    PyUnicodeObject *unicode;
+    PyObject *unicode;
     Py_UNICODE *p;
     const char *errmsg = "";
     int inShift = 0;
@@ -3793,13 +3792,13 @@
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
 
-    unicode = _PyUnicode_New(size);
+    unicode = (PyObject*)_PyUnicode_New(size);
     if (!unicode)
         return NULL;
     if (size == 0) {
         if (consumed)
             *consumed = 0;
-        return (PyObject *)unicode;
+        return unicode;
     }
 
     p = PyUnicode_AS_UNICODE(unicode);
@@ -3947,7 +3946,7 @@
         }
     }
 
-    if (PyUnicode_Resize((PyObject**)&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
+    if (PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
@@ -3959,7 +3958,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return (PyObject *)unicode;
+    return unicode;
 
   onError:
     Py_XDECREF(errorHandler);
@@ -4261,7 +4260,7 @@
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     const char *e, *aligned_end;
-    PyUnicodeObject *unicode;
+    PyObject *unicode;
     const char *errmsg = "";
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
@@ -4284,7 +4283,7 @@
     maxchar = utf8_max_char_size_and_has_errors(s, size, &unicode_size,
                                                 consumed, &has_errors);
     if (has_errors) {
-        unicode = _PyUnicode_New(size);
+        unicode = (PyObject*)_PyUnicode_New(size);
         if (!unicode)
             return NULL;
         kind = PyUnicode_WCHAR_KIND;
@@ -4292,7 +4291,7 @@
         assert(data != NULL);
     }
     else {
-        unicode = (PyUnicodeObject *)PyUnicode_New(unicode_size, maxchar);
+        unicode = PyUnicode_New(unicode_size, maxchar);
         if (!unicode)
             return NULL;
         /* When the string is ASCII only, just use memcpy and return.
@@ -4300,7 +4299,7 @@
            sequence at the end of the ASCII block.  */
         if (maxchar < 128 && size == unicode_size) {
             Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
-            return (PyObject *)unicode;
+            return unicode;
         }
         kind = PyUnicode_KIND(unicode);
         data = PyUnicode_DATA(unicode);
@@ -4482,10 +4481,10 @@
         /* If this is not yet a resizable string, make it one.. */
         if (kind != PyUnicode_WCHAR_KIND) {
             const Py_UNICODE *u;
-            PyUnicodeObject *new_unicode = _PyUnicode_New(size);
+            PyObject *new_unicode = (PyObject*)_PyUnicode_New(size);
             if (!new_unicode)
                 goto onError;
-            u = PyUnicode_AsUnicode((PyObject *)unicode);
+            u = PyUnicode_AsUnicode(unicode);
             if (!u)
                 goto onError;
 #if SIZEOF_WCHAR_T == 2
@@ -4519,7 +4518,7 @@
     /* Adjust length and ready string when it contained errors and
        is of the old resizable kind. */
     if (kind == PyUnicode_WCHAR_KIND) {
-        if (PyUnicode_Resize((PyObject**)&unicode, i) < 0)
+        if (PyUnicode_Resize(&unicode, i) < 0)
             goto onError;
     }
 
@@ -4532,7 +4531,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return (PyObject *)unicode;
+    return unicode;
 
   onError:
     Py_XDECREF(errorHandler);
@@ -4666,7 +4665,7 @@
    the excess memory at the end.
 */
 PyObject *
-_PyUnicode_AsUTF8String(PyObject *obj, const char *errors)
+_PyUnicode_AsUTF8String(PyObject *unicode, const char *errors)
 {
 #define MAX_SHORT_UNICHARS 300  /* largest size we'll do on the stack */
 
@@ -4681,7 +4680,6 @@
     int kind;
     void *data;
     Py_ssize_t size;
-    PyUnicodeObject *unicode = (PyUnicodeObject *)obj;
 #if SIZEOF_WCHAR_T == 2
     Py_ssize_t wchar_offset = 0;
 #endif
@@ -4745,7 +4743,7 @@
 #endif
             rep = unicode_encode_call_errorhandler(
                   errors, &errorHandler, "utf-8", "surrogates not allowed",
-                  obj, &exc, startpos, startpos+1, &newpos);
+                  unicode, &exc, startpos, startpos+1, &newpos);
             if (!rep)
                 goto error;
 
@@ -4792,7 +4790,7 @@
                     c = prep[k];
                     if (0x80 <= c) {
                         raise_encode_exception_obj(&exc, "utf-8",
-                                               (PyObject*)unicode,
+                                               unicode,
                                                i-1, i,
                                                "surrogates not allowed");
                         goto error;
@@ -4885,7 +4883,7 @@
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
-    PyUnicodeObject *unicode;
+    PyObject *unicode;
     Py_UNICODE *p;
 #ifndef Py_UNICODE_WIDE
     int pairs = 0;
@@ -4965,11 +4963,11 @@
 #endif
 
     /* This might be one to much, because of a BOM */
-    unicode = _PyUnicode_New((size+3)/4+pairs);
+    unicode = (PyObject*)_PyUnicode_New((size+3)/4+pairs);
     if (!unicode)
         return NULL;
     if (size == 0)
-        return (PyObject *)unicode;
+        return unicode;
 
     /* Unpack UTF-32 encoded data */
     p = PyUnicode_AS_UNICODE(unicode);
@@ -5025,7 +5023,7 @@
         *consumed = (const char *)q-starts;
 
     /* Adjust length */
-    if (PyUnicode_Resize((PyObject**)&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
+    if (PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
@@ -5037,7 +5035,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return (PyObject *)unicode;
+    return unicode;
 
   onError:
     Py_DECREF(unicode);
@@ -5185,7 +5183,7 @@
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
-    PyUnicodeObject *unicode;
+    PyObject *unicode;
     Py_UNICODE *p;
     const unsigned char *q, *e, *aligned_end;
     int bo = 0;       /* assume native ordering by default */
@@ -5202,11 +5200,11 @@
 
     /* Note: size will always be longer than the resulting Unicode
        character count */
-    unicode = _PyUnicode_New(size);
+    unicode = (PyObject*)_PyUnicode_New(size);
     if (!unicode)
         return NULL;
     if (size == 0)
-        return (PyObject *)unicode;
+        return unicode;
 
     /* Unpack UTF-16 encoded data */
     p = PyUnicode_AS_UNICODE(unicode);
@@ -5426,7 +5424,7 @@
         *consumed = (const char *)q-starts;
 
     /* Adjust length */
-    if (PyUnicode_Resize((PyObject**)&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
+    if (PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
@@ -5438,7 +5436,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return (PyObject *)unicode;
+    return unicode;
 
   onError:
     Py_DECREF(unicode);
@@ -5627,7 +5625,7 @@
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     int j;
-    PyUnicodeObject *v;
+    PyObject *v;
     Py_UNICODE *p;
     const char *end;
     char* message;
@@ -5647,7 +5645,7 @@
        or it contains \x, \u, ... escape sequences.  then we create a
        legacy wchar string and resize it at the end of this function. */
     if (ascii_length >= 0) {
-        v = (PyUnicodeObject *)PyUnicode_New(ascii_length, 127);
+        v = PyUnicode_New(ascii_length, 127);
         if (!v)
             goto onError;
         assert(PyUnicode_KIND(v) == PyUnicode_1BYTE_KIND);
@@ -5660,7 +5658,7 @@
            length after conversion to the true value.
            (but if the error callback returns a long replacement string
            we'll have to allocate more space) */
-        v = _PyUnicode_New(size);
+        v = (PyObject*)_PyUnicode_New(size);
         if (!v)
             goto onError;
         kind = PyUnicode_WCHAR_KIND;
@@ -5668,7 +5666,7 @@
     }
 
     if (size == 0)
-        return (PyObject *)v;
+        return v;
     i = 0;
     end = s + size;
 
@@ -5888,7 +5886,7 @@
 
     if (kind == PyUnicode_WCHAR_KIND)
     {
-        if (PyUnicode_Resize((PyObject**)&v, i) < 0)
+        if (PyUnicode_Resize(&v, i) < 0)
             goto onError;
     }
     Py_XDECREF(errorHandler);
@@ -5900,7 +5898,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(v, 1));
-    return (PyObject *)v;
+    return v;
 
   ucnhashError:
     PyErr_SetString(
@@ -6095,7 +6093,7 @@
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
-    PyUnicodeObject *v;
+    PyObject *v;
     Py_UNICODE *p;
     const char *end;
     const char *bs;
@@ -6106,11 +6104,11 @@
        Unicode string, so we start with size here and then reduce the
        length after conversion to the true value. (But decoding error
        handler might have to resize the string) */
-    v = _PyUnicode_New(size);
+    v = (PyObject*)_PyUnicode_New(size);
     if (v == NULL)
         goto onError;
     if (size == 0)
-        return (PyObject *)v;
+        return v;
     p = PyUnicode_AS_UNICODE(v);
     end = s + size;
     while (s < end) {
@@ -6191,7 +6189,7 @@
       nextByte:
         ;
     }
-    if (PyUnicode_Resize((PyObject**)&v, p - PyUnicode_AS_UNICODE(v)) < 0)
+    if (PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
@@ -6202,7 +6200,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(v, 1));
-    return (PyObject *)v;
+    return v;
 
   onError:
     Py_XDECREF(v);
@@ -6325,7 +6323,7 @@
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
-    PyUnicodeObject *v;
+    PyObject *v;
     Py_UNICODE *p;
     const char *end;
     const char *reason;
@@ -6337,13 +6335,13 @@
 #endif
 
     /* XXX overflow detection missing */
-    v = _PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE);
+    v = (PyObject*)_PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE);
     if (v == NULL)
         goto onError;
     /* Intentionally PyUnicode_GET_SIZE instead of PyUnicode_GET_LENGTH
        as string was created with the old API. */
     if (PyUnicode_GET_SIZE(v) == 0)
-        return (PyObject *)v;
+        return v;
     p = PyUnicode_AS_UNICODE(v);
     end = s + size;
 
@@ -6382,7 +6380,7 @@
         }
     }
 
-    if (PyUnicode_Resize((PyObject**)&v, p - PyUnicode_AS_UNICODE(v)) < 0)
+    if (PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
@@ -6393,7 +6391,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(v, 1));
-    return (PyObject *)v;
+    return v;
 
   onError:
     Py_XDECREF(v);
@@ -6802,7 +6800,7 @@
                       const char *errors)
 {
     const char *starts = s;
-    PyUnicodeObject *v;
+    PyObject *v;
     Py_UNICODE *u;
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
@@ -6851,11 +6849,11 @@
     if (!has_error)
         return unicode_fromascii((const unsigned char *)s, size);
 
-    v = _PyUnicode_New(size);
+    v = (PyObject*)_PyUnicode_New(size);
     if (v == NULL)
         goto onError;
     if (size == 0)
-        return (PyObject *)v;
+        return v;
     u = PyUnicode_AS_UNICODE(v);
     e = s + size;
     while (s < e) {
@@ -6877,7 +6875,7 @@
         }
     }
     if (u - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v))
-        if (PyUnicode_Resize((PyObject**)&v, u - PyUnicode_AS_UNICODE(v)) < 0)
+        if (PyUnicode_Resize(&v, u - PyUnicode_AS_UNICODE(v)) < 0)
             goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
@@ -6888,7 +6886,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(v, 1));
-    return (PyObject *)v;
+    return v;
 
   onError:
     Py_XDECREF(v);
@@ -7677,7 +7675,7 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     const char *e;
-    PyUnicodeObject *v;
+    PyObject *v;
     Py_UNICODE *p;
     Py_ssize_t extrachars = 0;
     PyObject *errorHandler = NULL;
@@ -7689,11 +7687,11 @@
     if (mapping == NULL)
         return PyUnicode_DecodeLatin1(s, size, errors);
 
-    v = _PyUnicode_New(size);
+    v = (PyObject*)_PyUnicode_New(size);
     if (v == NULL)
         goto onError;
     if (size == 0)
-        return (PyObject *)v;
+        return v;
     p = PyUnicode_AS_UNICODE(v);
     e = s + size;
     if (PyUnicode_CheckExact(mapping)) {
@@ -7788,8 +7786,8 @@
                             (targetsize << 2);
                         extrachars += needed;
                         /* XXX overflow detection missing */
-                        if (PyUnicode_Resize((PyObject**)&v,
-                                              PyUnicode_GET_SIZE(v) + needed) < 0) {
+                        if (PyUnicode_Resize(&v,
+                                             PyUnicode_GET_SIZE(v) + needed) < 0) {
                             Py_DECREF(x);
                             goto onError;
                         }
@@ -7815,7 +7813,7 @@
         }
     }
     if (p - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v))
-        if (PyUnicode_Resize((PyObject**)&v, p - PyUnicode_AS_UNICODE(v)) < 0)
+        if (PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
             goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
@@ -7826,7 +7824,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(v, 1));
-    return (PyObject *)v;
+    return v;
 
   onError:
     Py_XDECREF(errorHandler);
@@ -9192,12 +9190,12 @@
     kind = kind1 > kind2 ? kind1 : kind2;
     buf1 = PyUnicode_DATA(str_obj);
     if (kind1 != kind)
-        buf1 = _PyUnicode_AsKind((PyObject*)str_obj, kind);
+        buf1 = _PyUnicode_AsKind(str_obj, kind);
     if (!buf1)
         goto onError;
     buf2 = PyUnicode_DATA(sub_obj);
     if (kind2 != kind)
-        buf2 = _PyUnicode_AsKind((PyObject*)sub_obj, kind);
+        buf2 = _PyUnicode_AsKind(sub_obj, kind);
     if (!buf2)
         goto onError;
     len1 = PyUnicode_GET_LENGTH(str_obj);
@@ -9440,7 +9438,7 @@
            (to save space, not time) */
         Py_INCREF(self);
         Py_DECREF(u);
-        return (PyObject*) self;
+        return self;
     }
     else if (maxchar_new == maxchar_old) {
         return u;
@@ -9918,21 +9916,21 @@
     case PyUnicode_1BYTE_KIND:
         if (PyUnicode_IS_ASCII(string))
             list = asciilib_splitlines(
-                (PyObject*) string, PyUnicode_1BYTE_DATA(string),
+                string, PyUnicode_1BYTE_DATA(string),
                 PyUnicode_GET_LENGTH(string), keepends);
         else
             list = ucs1lib_splitlines(
-                (PyObject*) string, PyUnicode_1BYTE_DATA(string),
+                string, PyUnicode_1BYTE_DATA(string),
                 PyUnicode_GET_LENGTH(string), keepends);
         break;
     case PyUnicode_2BYTE_KIND:
         list = ucs2lib_splitlines(
-            (PyObject*) string, PyUnicode_2BYTE_DATA(string),
+            string, PyUnicode_2BYTE_DATA(string),
             PyUnicode_GET_LENGTH(string), keepends);
         break;
     case PyUnicode_4BYTE_KIND:
         list = ucs4lib_splitlines(
-            (PyObject*) string, PyUnicode_4BYTE_DATA(string),
+            string, PyUnicode_4BYTE_DATA(string),
             PyUnicode_GET_LENGTH(string), keepends);
         break;
     default:
@@ -9964,22 +9962,22 @@
         case PyUnicode_1BYTE_KIND:
             if (PyUnicode_IS_ASCII(self))
                 return asciilib_split_whitespace(
-                    (PyObject*) self,  PyUnicode_1BYTE_DATA(self),
+                    self,  PyUnicode_1BYTE_DATA(self),
                     PyUnicode_GET_LENGTH(self), maxcount
                     );
             else
                 return ucs1lib_split_whitespace(
-                    (PyObject*) self,  PyUnicode_1BYTE_DATA(self),
+                    self,  PyUnicode_1BYTE_DATA(self),
                     PyUnicode_GET_LENGTH(self), maxcount
                     );
         case PyUnicode_2BYTE_KIND:
             return ucs2lib_split_whitespace(
-                (PyObject*) self,  PyUnicode_2BYTE_DATA(self),
+                self,  PyUnicode_2BYTE_DATA(self),
                 PyUnicode_GET_LENGTH(self), maxcount
                 );
         case PyUnicode_4BYTE_KIND:
             return ucs4lib_split_whitespace(
-                (PyObject*) self,  PyUnicode_4BYTE_DATA(self),
+                self,  PyUnicode_4BYTE_DATA(self),
                 PyUnicode_GET_LENGTH(self), maxcount
                 );
         default:
@@ -9996,11 +9994,11 @@
     buf1 = PyUnicode_DATA(self);
     buf2 = PyUnicode_DATA(substring);
     if (kind1 != kind)
-        buf1 = _PyUnicode_AsKind((PyObject*)self, kind);
+        buf1 = _PyUnicode_AsKind(self, kind);
     if (!buf1)
         return NULL;
     if (kind2 != kind)
-        buf2 = _PyUnicode_AsKind((PyObject*)substring, kind);
+        buf2 = _PyUnicode_AsKind(substring, kind);
     if (!buf2) {
         if (kind1 != kind) PyMem_Free(buf1);
         return NULL;
@@ -10012,18 +10010,18 @@
     case PyUnicode_1BYTE_KIND:
         if (PyUnicode_IS_ASCII(self) && PyUnicode_IS_ASCII(substring))
             out = asciilib_split(
-                (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+                self,  buf1, len1, buf2, len2, maxcount);
         else
             out = ucs1lib_split(
-                (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+                self,  buf1, len1, buf2, len2, maxcount);
         break;
     case PyUnicode_2BYTE_KIND:
         out = ucs2lib_split(
-            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+            self,  buf1, len1, buf2, len2, maxcount);
         break;
     case PyUnicode_4BYTE_KIND:
         out = ucs4lib_split(
-            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+            self,  buf1, len1, buf2, len2, maxcount);
         break;
     default:
         out = NULL;
@@ -10056,22 +10054,22 @@
         case PyUnicode_1BYTE_KIND:
             if (PyUnicode_IS_ASCII(self))
                 return asciilib_rsplit_whitespace(
-                    (PyObject*) self,  PyUnicode_1BYTE_DATA(self),
+                    self,  PyUnicode_1BYTE_DATA(self),
                     PyUnicode_GET_LENGTH(self), maxcount
                     );
             else
                 return ucs1lib_rsplit_whitespace(
-                    (PyObject*) self,  PyUnicode_1BYTE_DATA(self),
+                    self,  PyUnicode_1BYTE_DATA(self),
                     PyUnicode_GET_LENGTH(self), maxcount
                     );
         case PyUnicode_2BYTE_KIND:
             return ucs2lib_rsplit_whitespace(
-                (PyObject*) self,  PyUnicode_2BYTE_DATA(self),
+                self,  PyUnicode_2BYTE_DATA(self),
                 PyUnicode_GET_LENGTH(self), maxcount
                 );
         case PyUnicode_4BYTE_KIND:
             return ucs4lib_rsplit_whitespace(
-                (PyObject*) self,  PyUnicode_4BYTE_DATA(self),
+                self,  PyUnicode_4BYTE_DATA(self),
                 PyUnicode_GET_LENGTH(self), maxcount
                 );
         default:
@@ -10088,11 +10086,11 @@
     buf1 = PyUnicode_DATA(self);
     buf2 = PyUnicode_DATA(substring);
     if (kind1 != kind)
-        buf1 = _PyUnicode_AsKind((PyObject*)self, kind);
+        buf1 = _PyUnicode_AsKind(self, kind);
     if (!buf1)
         return NULL;
     if (kind2 != kind)
-        buf2 = _PyUnicode_AsKind((PyObject*)substring, kind);
+        buf2 = _PyUnicode_AsKind(substring, kind);
     if (!buf2) {
         if (kind1 != kind) PyMem_Free(buf1);
         return NULL;
@@ -10104,18 +10102,18 @@
     case PyUnicode_1BYTE_KIND:
         if (PyUnicode_IS_ASCII(self) && PyUnicode_IS_ASCII(substring))
             out = asciilib_rsplit(
-                (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+                self,  buf1, len1, buf2, len2, maxcount);
         else
             out = ucs1lib_rsplit(
-                (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+                self,  buf1, len1, buf2, len2, maxcount);
         break;
     case PyUnicode_2BYTE_KIND:
         out = ucs2lib_rsplit(
-            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+            self,  buf1, len1, buf2, len2, maxcount);
         break;
     case PyUnicode_4BYTE_KIND:
         out = ucs4lib_rsplit(
-            (PyObject*) self,  buf1, len1, buf2, len2, maxcount);
+            self,  buf1, len1, buf2, len2, maxcount);
         break;
     default:
         out = NULL;
@@ -10417,7 +10415,7 @@
         PyMem_FREE(buf2);
     if (PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
-        return (PyObject *) self;
+        return self;
     }
     return PyUnicode_Copy(self);
   error:
@@ -10490,7 +10488,7 @@
 
   onError:
     Py_DECREF(list);
-    return (PyObject *)item;
+    return item;
 }
 #endif
 
@@ -10540,7 +10538,7 @@
 
     if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
-        return (PyObject*) self;
+        return self;
     }
 
     marg = width - _PyUnicode_LENGTH(self);
@@ -10712,13 +10710,13 @@
     buf1 = PyUnicode_DATA(str);
     buf2 = PyUnicode_DATA(sub);
     if (kind1 != kind)
-        buf1 = _PyUnicode_AsKind((PyObject*)str, kind);
+        buf1 = _PyUnicode_AsKind(str, kind);
     if (!buf1) {
         Py_DECREF(sub);
         return -1;
     }
     if (kind2 != kind)
-        buf2 = _PyUnicode_AsKind((PyObject*)sub, kind);
+        buf2 = _PyUnicode_AsKind(sub, kind);
     if (!buf2) {
         Py_DECREF(sub);
         if (kind1 != kind) PyMem_Free(buf1);
@@ -10929,13 +10927,13 @@
     buf1 = PyUnicode_DATA(self);
     buf2 = PyUnicode_DATA(substring);
     if (kind1 != kind)
-        buf1 = _PyUnicode_AsKind((PyObject*)self, kind);
+        buf1 = _PyUnicode_AsKind(self, kind);
     if (!buf1) {
         Py_DECREF(substring);
         return NULL;
     }
     if (kind2 != kind)
-        buf2 = _PyUnicode_AsKind((PyObject*)substring, kind);
+        buf2 = _PyUnicode_AsKind(substring, kind);
     if (!buf2) {
         Py_DECREF(substring);
         if (kind1 != kind) PyMem_Free(buf1);
@@ -11054,8 +11052,8 @@
         }
     }
     if (!found && PyUnicode_CheckExact(self)) {
-        Py_INCREF((PyObject *) self);
-        return (PyObject *) self;
+        Py_INCREF(self);
+        return self;
     }
 
     /* Second pass: create output string and fill it */
@@ -11094,7 +11092,7 @@
     }
 #endif
     assert(_PyUnicode_CheckConsistency(u, 1));
-    return (PyObject*) u;
+    return u;
 
   overflow:
     PyErr_SetString(PyExc_OverflowError, "new string is too long");
@@ -11127,9 +11125,7 @@
     if (PyUnicode_READY(substring) == -1)
         return NULL;
 
-    result = any_find_slice(1,
-        self, (PyObject*)substring, start, end
-        );
+    result = any_find_slice(1, self, substring, start, end);
 
     Py_DECREF(substring);
 
@@ -11219,9 +11215,7 @@
     if (PyUnicode_READY(substring) == -1)
         return NULL;
 
-    result = any_find_slice(1,
-        self, (PyObject*)substring, start, end
-        );
+    result = any_find_slice(1, self, substring, start, end);
 
     Py_DECREF(substring);
 
@@ -11710,10 +11704,10 @@
 
     if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
-        return (PyObject*) self;
-    }
-
-    return (PyObject*) pad(self, 0, width - _PyUnicode_LENGTH(self), fillchar);
+        return self;
+    }
+
+    return pad(self, 0, width - _PyUnicode_LENGTH(self), fillchar);
 }
 
 PyDoc_STRVAR(lower__doc__,
@@ -11772,7 +11766,7 @@
         j++;
     }
 
-    return PyUnicode_Substring((PyObject*)self, i, j);
+    return PyUnicode_Substring(self, i, j);
 }
 
 PyObject*
@@ -11849,7 +11843,7 @@
         j++;
     }
 
-    return PyUnicode_Substring((PyObject*)self, i, j);
+    return PyUnicode_Substring(self, i, j);
 }
 
 
@@ -11939,7 +11933,7 @@
     if (len == 1 && PyUnicode_CheckExact(str)) {
         /* no repeat, return original string */
         Py_INCREF(str);
-        return (PyObject*) str;
+        return str;
     }
 
     if (PyUnicode_READY(str) == -1)
@@ -12230,9 +12224,7 @@
     if (PyUnicode_READY(substring) == -1)
         return NULL;
 
-    result = any_find_slice(-1,
-        self, (PyObject*)substring, start, end
-        );
+    result = any_find_slice(-1, self, substring, start, end);
 
     Py_DECREF(substring);
 
@@ -12264,9 +12256,7 @@
     if (PyUnicode_READY(substring) == -1)
         return NULL;
 
-    result = any_find_slice(-1,
-        self, (PyObject*)substring, start, end
-        );
+    result = any_find_slice(-1, self, substring, start, end);
 
     Py_DECREF(substring);
 
@@ -12301,10 +12291,10 @@
 
     if (_PyUnicode_LENGTH(self) >= width && PyUnicode_CheckExact(self)) {
         Py_INCREF(self);
-        return (PyObject*) self;
-    }
-
-    return (PyObject*) pad(self, width - _PyUnicode_LENGTH(self), 0, fillchar);
+        return self;
+    }
+
+    return pad(self, width - _PyUnicode_LENGTH(self), 0, fillchar);
 }
 
 PyObject *
@@ -12353,7 +12343,7 @@
     else if (PyUnicode_Check(substring))
         return split(self, substring, maxcount);
     else
-        return PyUnicode_Split((PyObject *)self, substring, maxcount);
+        return PyUnicode_Split(self, substring, maxcount);
 }
 
 PyObject *
@@ -12784,10 +12774,10 @@
     if (PyUnicode_GET_LENGTH(self) >= width) {
         if (PyUnicode_CheckExact(self)) {
             Py_INCREF(self);
-            return (PyObject*) self;
+            return self;
         }
         else
-            return PyUnicode_Copy((PyObject*)self);
+            return PyUnicode_Copy(self);
     }
 
     fill = width - _PyUnicode_LENGTH(self);
@@ -12808,7 +12798,7 @@
     }
 
     assert(_PyUnicode_CheckConsistency(u, 1));
-    return (PyObject*) u;
+    return u;
 }
 
 #if 0
@@ -13334,7 +13324,7 @@
                 fmtpos++;
                 fmtcnt--;
             }
-            nonfmt = PyUnicode_Substring((PyObject *) uformat, nonfmtpos, fmtpos);
+            nonfmt = PyUnicode_Substring(uformat, nonfmtpos, fmtpos);
             if (nonfmt == NULL)
                 goto onError;
             r = _PyAccu_Accumulate(&acc, nonfmt);
@@ -13384,7 +13374,7 @@
                                     "incomplete format key");
                     goto onError;
                 }
-                key = PyUnicode_Substring((PyObject*)uformat,
+                key = PyUnicode_Substring(uformat,
                                           keystart, keystart + keylen);
                 if (key == NULL)
                     goto onError;
@@ -13751,7 +13741,7 @@
     Py_DECREF(uformat);
     Py_XDECREF(temp);
     Py_XDECREF(second);
-    return (PyObject *)result;
+    return result;
 
   onError:
     Py_DECREF(uformat);
@@ -13781,7 +13771,7 @@
                                      kwlist, &x, &encoding, &errors))
         return NULL;
     if (x == NULL)
-        return (PyObject *)PyUnicode_New(0, 0);
+        return PyUnicode_New(0, 0);
     if (encoding == NULL && errors == NULL)
         return PyObject_Str(x);
     else
@@ -13878,7 +13868,7 @@
     _PyUnicode_HASH(self) = _PyUnicode_HASH(unicode);
 #endif
     Py_DECREF(unicode);
-    return (PyObject *)self;
+    return self;
 
 onError:
     Py_DECREF(unicode);
@@ -14045,7 +14035,7 @@
        though the key is present in the dictionary,
        namely when this happens during a stack overflow. */
     Py_ALLOW_RECURSION
-        t = PyDict_GetItem(interned, (PyObject *)s);
+    t = PyDict_GetItem(interned, s);
     Py_END_ALLOW_RECURSION
 
         if (t) {
@@ -14056,7 +14046,7 @@
         }
 
     PyThreadState_GET()->recursion_critical = 1;
-    if (PyDict_SetItem(interned, (PyObject *)s, (PyObject *)s) < 0) {
+    if (PyDict_SetItem(interned, s, s) < 0) {
         PyErr_Clear();
         PyThreadState_GET()->recursion_critical = 0;
         return;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 00:23:25 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 00:23:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_PyUnicode=5FEncodeCharm?=
	=?utf8?q?ap=28=29?=
Message-ID: <E1RM6d3-0005sy-2r@dinsdale.python.org>

http://hg.python.org/cpython/rev/bb9a19a74277
changeset:   73338:bb9a19a74277
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 00:24:51 2011 +0100
summary:
  Fix PyUnicode_EncodeCharmap()

files:
  Objects/unicodeobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8397,7 +8397,7 @@
         return NULL;
     result = _PyUnicode_EncodeCharmap(unicode, mapping, errors);
     Py_DECREF(unicode);
-    return NULL;
+    return result;
 }
 
 PyObject *

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 00:27:20 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 00:27:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_compiler_warning_in_u?=
 =?utf8?q?nicode=5Fencode=5Fucs1=28=29?=
Message-ID: <E1RM6gq-0006BT-EC@dinsdale.python.org>

http://hg.python.org/cpython/rev/cc0ff80a834b
changeset:   73339:cc0ff80a834b
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 00:28:50 2011 +0100
summary:
  Fix a compiler warning in unicode_encode_ucs1()

files:
  Objects/unicodeobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6557,7 +6557,7 @@
 static PyObject *
 unicode_encode_ucs1(PyObject *unicode,
                     const char *errors,
-                    int limit)
+                    unsigned int limit)
 {
     /* input state */
     Py_ssize_t pos=0, size;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 00:42:17 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 00:42:17 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMzQy?=
 =?utf8?q?=3A_Improve_=5Ftkinter_error_message_on_unencodable_character?=
Message-ID: <E1RM6vJ-0007GA-S0@dinsdale.python.org>

http://hg.python.org/cpython/rev/9a07b73abdb1
changeset:   73340:9a07b73abdb1
branch:      3.2
parent:      73331:38f4a251608f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 00:36:46 2011 +0100
summary:
  Issue #12342: Improve _tkinter error message on unencodable character

files:
  Modules/_tkinter.c |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -993,8 +993,10 @@
         for (i = 0; i < size; i++) {
             if (inbuf[i] >= 0x10000) {
                 /* Tcl doesn't do UTF-16, yet. */
-                PyErr_SetString(PyExc_ValueError,
-                                "unsupported character");
+                PyErr_Format(PyExc_ValueError,
+                             "character U+%x is above the range "
+                             "(U+0000-U+FFFF) allowed by Tcl",
+                             inbuf[i]);
                 ckfree(FREECAST outbuf);
                 return NULL;
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 00:42:18 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 00:42:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=28Merge_3=2E2=29_Issue_=2312342=3A_Improve_=5Ftkinter_error?=
 =?utf8?q?_message_on_unencodable?=
Message-ID: <E1RM6vK-0007GE-PB@dinsdale.python.org>

http://hg.python.org/cpython/rev/5aea95d41ad2
changeset:   73341:5aea95d41ad2
parent:      73339:cc0ff80a834b
parent:      73340:9a07b73abdb1
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 00:43:35 2011 +0100
summary:
  (Merge 3.2) Issue #12342: Improve _tkinter error message on unencodable character

files:
  Modules/_tkinter.c |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -990,8 +990,10 @@
 #if TCL_UTF_MAX == 3
             if (ch >= 0x10000) {
                 /* Tcl doesn't do UTF-16, yet. */
-                PyErr_SetString(PyExc_ValueError,
-                                "unsupported character");
+                PyErr_Format(PyExc_ValueError,
+                             "character U+%x is above the range "
+                             "(U+0000-U+FFFF) allowed by Tcl",
+                             inbuf[i]);
                 ckfree(FREECAST outbuf);
                 return NULL;
 #endif

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Nov  4 05:27:25 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 04 Nov 2011 05:27:25 +0100
Subject: [Python-checkins] Daily reference leaks (5aea95d41ad2): sum=0
Message-ID: <E1RMBNF-0006la-6z@ap.vmr.nerim.net>

results for 5aea95d41ad2 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogNU95F9', '-x']

From python-checkins at python.org  Fri Nov  4 06:15:26 2011
From: python-checkins at python.org (ross.lagerwall)
Date: Fri, 04 Nov 2011 06:15:26 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzM5?=
 =?utf8?q?=3A_Fix_compile_error_in_posixmodule=2Ec_due_to_missing_semicolo?=
 =?utf8?q?n=2E?=
Message-ID: <E1RMC7i-0006g4-27@dinsdale.python.org>

http://hg.python.org/cpython/rev/fa9c69dcf31b
changeset:   73342:fa9c69dcf31b
branch:      3.2
parent:      73340:9a07b73abdb1
user:        Ross Lagerwall <rosslagerwall at gmail.com>
date:        Fri Nov 04 07:09:14 2011 +0200
summary:
  Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.
Thanks to Robert Xiao.

files:
  Misc/NEWS             |  3 +++
  Modules/posixmodule.c |  2 +-
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,6 +66,9 @@
 Library
 -------
 
+- Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.
+  Thanks to Robert Xiao.
+
 - Issue #10570: curses.putp() and curses.tigetstr() are now expecting a byte
   string, instead of a Unicode string.
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4001,7 +4001,7 @@
 static PyObject *
 posix_spawnvpe(PyObject *self, PyObject *args)
 {
-    PyObject *opath
+    PyObject *opath;
     char *path;
     PyObject *argv, *env;
     char **argvlist;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 06:15:26 2011
From: python-checkins at python.org (ross.lagerwall)
Date: Fri, 04 Nov 2011 06:15:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiAoTWVyZ2UgMy4yKSBJc3N1ZSAjMTMzMzku?=
Message-ID: <E1RMC7i-0006g8-VY@dinsdale.python.org>

http://hg.python.org/cpython/rev/9e5d14aadca4
changeset:   73343:9e5d14aadca4
parent:      73341:5aea95d41ad2
parent:      73342:fa9c69dcf31b
user:        Ross Lagerwall <rosslagerwall at gmail.com>
date:        Fri Nov 04 07:15:35 2011 +0200
summary:
  (Merge 3.2) Issue #13339.

files:
  Misc/NEWS             |  3 +++
  Modules/posixmodule.c |  2 +-
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,9 @@
 Library
 -------
 
+- Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.
+  Thanks to Robert Xiao.
+
 - Byte compilation in packaging is now isolated from the calling Python -B or
   -O options, instead of being disallowed under -B or buggy under -O.
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4509,7 +4509,7 @@
 static PyObject *
 posix_spawnvpe(PyObject *self, PyObject *args)
 {
-    PyObject *opath
+    PyObject *opath;
     char *path;
     PyObject *argv, *env;
     char **argvlist;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 08:30:19 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 04 Nov 2011 08:30:19 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Drop_unused_import_in_unitt?=
	=?utf8?q?est_package=2E?=
Message-ID: <E1RMEEF-0007vw-2g@dinsdale.python.org>

http://hg.python.org/cpython/rev/bd674826050e
changeset:   73344:bd674826050e
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 04 08:25:54 2011 +0100
summary:
  Drop unused import in unittest package.

files:
  Lib/unittest/case.py                |  3 +--
  Lib/unittest/main.py                |  1 -
  Lib/unittest/result.py              |  1 -
  Lib/unittest/test/_test_warnings.py |  1 -
  4 files changed, 1 insertions(+), 5 deletions(-)


diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -9,8 +9,7 @@
 import collections
 
 from . import result
-from .util import (strclass, safe_repr, sorted_list_difference,
-                   unorderable_list_difference, _count_diff_all_purpose,
+from .util import (strclass, safe_repr, _count_diff_all_purpose,
                    _count_diff_hashable)
 
 __unittest = True
diff --git a/Lib/unittest/main.py b/Lib/unittest/main.py
--- a/Lib/unittest/main.py
+++ b/Lib/unittest/main.py
@@ -2,7 +2,6 @@
 
 import sys
 import os
-import types
 
 from . import loader, runner
 from .signals import installHandler
diff --git a/Lib/unittest/result.py b/Lib/unittest/result.py
--- a/Lib/unittest/result.py
+++ b/Lib/unittest/result.py
@@ -1,6 +1,5 @@
 """Test result object"""
 
-import os
 import io
 import sys
 import traceback
diff --git a/Lib/unittest/test/_test_warnings.py b/Lib/unittest/test/_test_warnings.py
--- a/Lib/unittest/test/_test_warnings.py
+++ b/Lib/unittest/test/_test_warnings.py
@@ -10,7 +10,6 @@
 See #10535.
 """
 
-import io
 import sys
 import unittest
 import warnings

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 08:30:19 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 04 Nov 2011 08:30:19 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_redundant_imports=2E?=
Message-ID: <E1RMEEF-0007w6-Rn@dinsdale.python.org>

http://hg.python.org/cpython/rev/fa82071bb7e1
changeset:   73345:fa82071bb7e1
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 04 08:29:17 2011 +0100
summary:
  Remove redundant imports.

files:
  Lib/getpass.py            |  4 +---
  Lib/pickle.py             |  2 +-
  Lib/pstats.py             |  1 -
  Lib/shutil.py             |  1 +
  Lib/site.py               |  3 +--
  Lib/tkinter/__init__.py   |  5 ++---
  Lib/tkinter/filedialog.py |  2 --
  7 files changed, 6 insertions(+), 12 deletions(-)


diff --git a/Lib/getpass.py b/Lib/getpass.py
--- a/Lib/getpass.py
+++ b/Lib/getpass.py
@@ -72,7 +72,7 @@
             finally:
                 termios.tcsetattr(fd, tcsetattr_flags, old)
                 stream.flush()  # issue7208
-        except termios.error as e:
+        except termios.error:
             if passwd is not None:
                 # _raw_input succeeded.  The final tcsetattr failed.  Reraise
                 # instead of leaving the terminal in an unknown state.
@@ -145,8 +145,6 @@
 
     """
 
-    import os
-
     for name in ('LOGNAME', 'USER', 'LNAME', 'USERNAME'):
         user = os.environ.get(name)
         if user:
diff --git a/Lib/pickle.py b/Lib/pickle.py
--- a/Lib/pickle.py
+++ b/Lib/pickle.py
@@ -1326,7 +1326,7 @@
     return doctest.testmod()
 
 if __name__ == "__main__":
-    import sys, argparse
+    import argparse
     parser = argparse.ArgumentParser(
         description='display contents of the pickle files')
     parser.add_argument(
diff --git a/Lib/pstats.py b/Lib/pstats.py
--- a/Lib/pstats.py
+++ b/Lib/pstats.py
@@ -674,7 +674,6 @@
                 return stop
             return None
 
-    import sys
     if len(sys.argv) > 1:
         initprofile = sys.argv[1]
     else:
diff --git a/Lib/shutil.py b/Lib/shutil.py
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -15,6 +15,7 @@
 
 try:
     import bz2
+    del bz2
     _BZ2_SUPPORTED = True
 except ImportError:
     _BZ2_SUPPORTED = False
diff --git a/Lib/site.py b/Lib/site.py
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -154,7 +154,7 @@
                 if not dircase in known_paths and os.path.exists(dir):
                     sys.path.append(dir)
                     known_paths.add(dircase)
-            except Exception as err:
+            except Exception:
                 print("Error processing line {:d} of {}:\n".format(n+1, fullname),
                       file=sys.stderr)
                 for record in traceback.format_exception(*sys.exc_info()):
@@ -241,7 +241,6 @@
         return USER_SITE
 
     from sysconfig import get_path
-    import os
 
     if sys.platform == 'darwin':
         from sysconfig import get_config_var
diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py
--- a/Lib/tkinter/__init__.py
+++ b/Lib/tkinter/__init__.py
@@ -1159,7 +1159,6 @@
         return (e,)
     def _report_exception(self):
         """Internal function."""
-        import sys
         exc, val, tb = sys.exc_info()
         root = self._root()
         root.report_callback_exception(exc, val, tb)
@@ -1663,7 +1662,7 @@
         # ensure that self.tk is always _something_.
         self.tk = None
         if baseName is None:
-            import sys, os
+            import os
             baseName = os.path.basename(sys.argv[0])
             baseName, ext = os.path.splitext(baseName)
             if ext not in ('.py', '.pyc', '.pyo'):
@@ -1737,7 +1736,7 @@
             exec(open(base_py).read(), dir)
     def report_callback_exception(self, exc, val, tb):
         """Internal function. It reports exception on sys.stderr."""
-        import traceback, sys
+        import traceback
         sys.stderr.write("Exception in Tkinter callback\n")
         sys.last_type = exc
         sys.last_value = val
diff --git a/Lib/tkinter/filedialog.py b/Lib/tkinter/filedialog.py
--- a/Lib/tkinter/filedialog.py
+++ b/Lib/tkinter/filedialog.py
@@ -306,7 +306,6 @@
     def _fixresult(self, widget, result):
         if result:
             # keep directory and filename until next time
-            import os
             # convert Tcl path objects to strings
             try:
                 result = result.string
@@ -333,7 +332,6 @@
             # multiple results:
             result = tuple([getattr(r, "string", r) for r in result])
             if result:
-                import os
                 path, file = os.path.split(result[0])
                 self.options["initialdir"] = path
                 # don't set initialfile or filename, as we have multiple of these

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 09:49:22 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 09:49:22 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312342=3A_Fix_compi?=
	=?utf8?q?lation_on_Mac_OS_X?=
Message-ID: <E1RMFSk-0005Zw-JI@dinsdale.python.org>

http://hg.python.org/cpython/rev/5f49b496d161
changeset:   73346:5f49b496d161
user:        Victor Stinner <vstinner at wyplay.com>
date:        Fri Nov 04 09:49:24 2011 +0100
summary:
  Issue #12342: Fix compilation on Mac OS X

files:
  Modules/_tkinter.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -993,7 +993,7 @@
                 PyErr_Format(PyExc_ValueError,
                              "character U+%x is above the range "
                              "(U+0000-U+FFFF) allowed by Tcl",
-                             inbuf[i]);
+                             ch);
                 ckfree(FREECAST outbuf);
                 return NULL;
 #endif

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 10:23:29 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 04 Nov 2011 10:23:29 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMTQw?=
 =?utf8?q?=3A_Fix_the_daemon=5Fthreads_attribute_of_ThreadingMixIn=2E?=
Message-ID: <E1RMFzl-0000GA-1I@dinsdale.python.org>

http://hg.python.org/cpython/rev/f09e3b1603ee
changeset:   73347:f09e3b1603ee
branch:      2.7
parent:      73306:d1cde7081bf5
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 04 10:15:57 2011 +0100
summary:
  Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.

files:
  Lib/SocketServer.py           |  3 +--
  Lib/test/test_socketserver.py |  1 -
  Misc/NEWS                     |  2 ++
  3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/SocketServer.py b/Lib/SocketServer.py
--- a/Lib/SocketServer.py
+++ b/Lib/SocketServer.py
@@ -589,8 +589,7 @@
         """Start a new thread to process the request."""
         t = threading.Thread(target = self.process_request_thread,
                              args = (request, client_address))
-        if self.daemon_threads:
-            t.setDaemon (1)
+        t.daemon = self.daemon_threads
         t.start()
 
 
diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py
--- a/Lib/test/test_socketserver.py
+++ b/Lib/test/test_socketserver.py
@@ -123,7 +123,6 @@
         self.assertEqual(server.server_address, server.socket.getsockname())
         return server
 
-    @unittest.skipUnless(threading, 'Threading required for this test.')
     @reap_threads
     def run_server(self, svrcls, hdlrbase, testfunc):
         server = self.make_server(self.pickaddr(svrcls.address_family),
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -74,6 +74,8 @@
 Library
 -------
 
+- Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
+
 - Issue #2892: preserve iterparse events in case of SyntaxError.
 
 - Issue #670664: Fix HTMLParser to correctly handle the content of

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 10:23:29 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 04 Nov 2011 10:23:29 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_=2313140?=
 =?utf8?q?=3A_Fix_the_daemon=5Fthreads_attribute_of_ThreadingMixIn=2E?=
Message-ID: <E1RMFzl-0000GE-RP@dinsdale.python.org>

http://hg.python.org/cpython/rev/94017ce9304d
changeset:   73348:94017ce9304d
branch:      3.2
parent:      73342:fa9c69dcf31b
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 04 10:16:28 2011 +0100
summary:
  Closes #13140: Fix the daemon_threads attribute of ThreadingMixIn.

files:
  Lib/socketserver.py           |  3 +--
  Lib/test/test_socketserver.py |  1 -
  Misc/NEWS                     |  2 ++
  3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/socketserver.py b/Lib/socketserver.py
--- a/Lib/socketserver.py
+++ b/Lib/socketserver.py
@@ -588,8 +588,7 @@
         """Start a new thread to process the request."""
         t = threading.Thread(target = self.process_request_thread,
                              args = (request, client_address))
-        if self.daemon_threads:
-            t.daemon = True
+        t.daemon = self.daemon_threads
         t.start()
 
 
diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py
--- a/Lib/test/test_socketserver.py
+++ b/Lib/test/test_socketserver.py
@@ -123,7 +123,6 @@
         self.assertEqual(server.server_address, server.socket.getsockname())
         return server
 
-    @unittest.skipUnless(threading, 'Threading required for this test.')
     @reap_threads
     def run_server(self, svrcls, hdlrbase, testfunc):
         server = self.make_server(self.pickaddr(svrcls.address_family),
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,6 +66,8 @@
 Library
 -------
 
+- Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
+
 - Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.
   Thanks to Robert Xiao.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 10:23:30 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 04 Nov 2011 10:23:30 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=3A_issue_=2313140?=
Message-ID: <E1RMFzm-0000GI-Kd@dinsdale.python.org>

http://hg.python.org/cpython/rev/6fe6769e54a5
changeset:   73349:6fe6769e54a5
parent:      73346:5f49b496d161
parent:      73348:94017ce9304d
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 04 10:22:57 2011 +0100
summary:
  Merge 3.2: issue #13140

files:
  Lib/socketserver.py           |  3 +--
  Lib/test/test_socketserver.py |  1 -
  Misc/NEWS                     |  2 ++
  3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/socketserver.py b/Lib/socketserver.py
--- a/Lib/socketserver.py
+++ b/Lib/socketserver.py
@@ -606,8 +606,7 @@
         """Start a new thread to process the request."""
         t = threading.Thread(target = self.process_request_thread,
                              args = (request, client_address))
-        if self.daemon_threads:
-            t.daemon = True
+        t.daemon = self.daemon_threads
         t.start()
 
 
diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py
--- a/Lib/test/test_socketserver.py
+++ b/Lib/test/test_socketserver.py
@@ -123,7 +123,6 @@
         self.assertEqual(server.server_address, server.socket.getsockname())
         return server
 
-    @unittest.skipUnless(threading, 'Threading required for this test.')
     @reap_threads
     def run_server(self, svrcls, hdlrbase, testfunc):
         server = self.make_server(self.pickaddr(svrcls.address_family),
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,8 @@
 Library
 -------
 
+- Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
+
 - Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.
   Thanks to Robert Xiao.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 11:17:39 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 04 Nov 2011 11:17:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Port_error_handlers_from_Py?=
 =?utf8?q?=5FUNICODE_indexing_to_code_point_indexing=2E?=
Message-ID: <E1RMGqB-0005C8-HH@dinsdale.python.org>

http://hg.python.org/cpython/rev/0883d6a103a0
changeset:   73350:0883d6a103a0
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Fri Nov 04 11:16:41 2011 +0100
summary:
  Port error handlers from Py_UNICODE indexing to code point indexing.

files:
  Objects/exceptions.c |    5 +
  Python/codecs.c      |  123 +++++++++++-------------------
  2 files changed, 51 insertions(+), 77 deletions(-)


diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -1513,6 +1513,11 @@
           return -1;
     }
 
+    if (PyUnicode_READY(err->object) < -1) {
+        err->encoding = NULL;
+        return -1;
+    }
+
     Py_INCREF(err->encoding);
     Py_INCREF(err->object);
     Py_INCREF(err->reason);
diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -573,82 +573,72 @@
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
         PyObject *restuple;
         PyObject *object;
+        Py_ssize_t i, o;
         Py_ssize_t start;
         Py_ssize_t end;
         PyObject *res;
-        Py_UNICODE *p;
-        Py_UNICODE *startp;
-        Py_UNICODE *outp;
+        unsigned char *outp;
         int ressize;
+        Py_UCS4 ch;
         if (PyUnicodeEncodeError_GetStart(exc, &start))
             return NULL;
         if (PyUnicodeEncodeError_GetEnd(exc, &end))
             return NULL;
         if (!(object = PyUnicodeEncodeError_GetObject(exc)))
             return NULL;
-        startp = PyUnicode_AS_UNICODE(object);
-        for (p = startp+start, ressize = 0; p < startp+end; ++p) {
-            if (*p<10)
+        for (i = start, ressize = 0; i < end; ++i) {
+            /* object is guaranteed to be "ready" */
+            ch = PyUnicode_READ_CHAR(object, i);
+            if (ch<10)
                 ressize += 2+1+1;
-            else if (*p<100)
+            else if (ch<100)
                 ressize += 2+2+1;
-            else if (*p<1000)
+            else if (ch<1000)
                 ressize += 2+3+1;
-            else if (*p<10000)
+            else if (ch<10000)
                 ressize += 2+4+1;
-#ifndef Py_UNICODE_WIDE
-            else
+            else if (ch<100000)
                 ressize += 2+5+1;
-#else
-            else if (*p<100000)
-                ressize += 2+5+1;
-            else if (*p<1000000)
+            else if (ch<1000000)
                 ressize += 2+6+1;
             else
                 ressize += 2+7+1;
-#endif
         }
         /* allocate replacement */
-        res = PyUnicode_FromUnicode(NULL, ressize);
+        res = PyUnicode_New(ressize, 127);
         if (res == NULL) {
             Py_DECREF(object);
             return NULL;
         }
+        outp = PyUnicode_1BYTE_DATA(res);
         /* generate replacement */
-        for (p = startp+start, outp = PyUnicode_AS_UNICODE(res);
-            p < startp+end; ++p) {
-            Py_UNICODE c = *p;
+        for (i = start, o = 0; i < end; ++i) {
+            ch = PyUnicode_READ_CHAR(object, i);
             int digits;
             int base;
             *outp++ = '&';
             *outp++ = '#';
-            if (*p<10) {
+            if (ch<10) {
                 digits = 1;
                 base = 1;
             }
-            else if (*p<100) {
+            else if (ch<100) {
                 digits = 2;
                 base = 10;
             }
-            else if (*p<1000) {
+            else if (ch<1000) {
                 digits = 3;
                 base = 100;
             }
-            else if (*p<10000) {
+            else if (ch<10000) {
                 digits = 4;
                 base = 1000;
             }
-#ifndef Py_UNICODE_WIDE
-            else {
+            else if (ch<100000) {
                 digits = 5;
                 base = 10000;
             }
-#else
-            else if (*p<100000) {
-                digits = 5;
-                base = 10000;
-            }
-            else if (*p<1000000) {
+            else if (ch<1000000) {
                 digits = 6;
                 base = 100000;
             }
@@ -656,10 +646,9 @@
                 digits = 7;
                 base = 1000000;
             }
-#endif
             while (digits-->0) {
-                *outp++ = '0' + c/base;
-                c %= base;
+                *outp++ = '0' + ch/base;
+                ch %= base;
                 base /= 10;
             }
             *outp++ = ';';
@@ -677,58 +666,41 @@
 
 PyObject *PyCodec_BackslashReplaceErrors(PyObject *exc)
 {
-#ifndef Py_UNICODE_WIDE
-#define IS_SURROGATE_PAIR(p, end) \
-    (*p >= 0xD800 && *p <= 0xDBFF && (p + 1) < end && \
-     *(p + 1) >= 0xDC00 && *(p + 1) <= 0xDFFF)
-#else
-#define IS_SURROGATE_PAIR(p, end) 0
-#endif
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
         PyObject *restuple;
         PyObject *object;
+        Py_ssize_t i;
         Py_ssize_t start;
         Py_ssize_t end;
         PyObject *res;
-        Py_UNICODE *p;
-        Py_UNICODE *startp;
-        Py_UNICODE *outp;
+        unsigned char *outp;
         int ressize;
+        Py_UCS4 c;
         if (PyUnicodeEncodeError_GetStart(exc, &start))
             return NULL;
         if (PyUnicodeEncodeError_GetEnd(exc, &end))
             return NULL;
         if (!(object = PyUnicodeEncodeError_GetObject(exc)))
             return NULL;
-        startp = PyUnicode_AS_UNICODE(object);
-        for (p = startp+start, ressize = 0; p < startp+end; ++p) {
-#ifdef Py_UNICODE_WIDE
-            if (*p >= 0x00010000)
+        for (i = start, ressize = 0; i < end; ++i) {
+            /* object is guaranteed to be "ready" */
+            c = PyUnicode_READ_CHAR(object, i);
+            if (c >= 0x10000) {
                 ressize += 1+1+8;
-            else
-#endif
-            if (*p >= 0x100) {
-                if (IS_SURROGATE_PAIR(p, startp+end)) {
-                    ressize += 1+1+8;
-                    ++p;
-                }
-                else
-                    ressize += 1+1+4;
+            }
+            else if (c >= 0x100) {
+                ressize += 1+1+4;
             }
             else
                 ressize += 1+1+2;
         }
-        res = PyUnicode_FromUnicode(NULL, ressize);
+        res = PyUnicode_New(ressize, 127);
         if (res==NULL)
             return NULL;
-        for (p = startp+start, outp = PyUnicode_AS_UNICODE(res);
-            p < startp+end; ++p) {
-            Py_UCS4 c = (Py_UCS4) *p;
+        for (i = start, outp = PyUnicode_1BYTE_DATA(res);
+            i < end; ++i) {
+            c = PyUnicode_READ_CHAR(object, i);
             *outp++ = '\\';
-            if (IS_SURROGATE_PAIR(p, startp+end)) {
-                c = ((*p & 0x3FF) << 10) + (*(p + 1) & 0x3FF) + 0x10000;
-                ++p;
-            }
             if (c >= 0x00010000) {
                 *outp++ = 'U';
                 *outp++ = Py_hexdigits[(c>>28)&0xf];
@@ -758,7 +730,6 @@
         wrong_exception_type(exc);
         return NULL;
     }
-#undef IS_SURROGATE_PAIR
 }
 
 /* This handler is declared static until someone demonstrates
@@ -768,12 +739,11 @@
 {
     PyObject *restuple;
     PyObject *object;
+    Py_ssize_t i;
     Py_ssize_t start;
     Py_ssize_t end;
     PyObject *res;
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
-        Py_UNICODE *p;
-        Py_UNICODE *startp;
         char *outp;
         if (PyUnicodeEncodeError_GetStart(exc, &start))
             return NULL;
@@ -781,15 +751,15 @@
             return NULL;
         if (!(object = PyUnicodeEncodeError_GetObject(exc)))
             return NULL;
-        startp = PyUnicode_AS_UNICODE(object);
         res = PyBytes_FromStringAndSize(NULL, 3*(end-start));
         if (!res) {
             Py_DECREF(object);
             return NULL;
         }
         outp = PyBytes_AsString(res);
-        for (p = startp+start; p < startp+end; p++) {
-            Py_UNICODE ch = *p;
+        for (i = start; i < end; i++) {
+            /* object is guaranteed to be "ready" */
+            Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
             if (ch < 0xd800 || ch > 0xdfff) {
                 /* Not a surrogate, fail with original exception */
                 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
@@ -847,12 +817,11 @@
 {
     PyObject *restuple;
     PyObject *object;
+    Py_ssize_t i;
     Py_ssize_t start;
     Py_ssize_t end;
     PyObject *res;
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
-        Py_UNICODE *p;
-        Py_UNICODE *startp;
         char *outp;
         if (PyUnicodeEncodeError_GetStart(exc, &start))
             return NULL;
@@ -860,15 +829,15 @@
             return NULL;
         if (!(object = PyUnicodeEncodeError_GetObject(exc)))
             return NULL;
-        startp = PyUnicode_AS_UNICODE(object);
         res = PyBytes_FromStringAndSize(NULL, end-start);
         if (!res) {
             Py_DECREF(object);
             return NULL;
         }
         outp = PyBytes_AsString(res);
-        for (p = startp+start; p < startp+end; p++) {
-            Py_UNICODE ch = *p;
+        for (i = start; i < end; i++) {
+            /* object is guaranteed to be "ready" */
+            Py_UCS4 ch = PyUnicode_READ_CHAR(object, i);
             if (ch < 0xdc80 || ch > 0xdcff) {
                 /* Not a UTF-8b surrogate, fail with original exception */
                 PyErr_SetObject(PyExceptionInstance_Class(exc), exc);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 12:27:20 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 04 Nov 2011 12:27:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_C89_incompatibility=2E?=
Message-ID: <E1RMHvc-000331-Rc@dinsdale.python.org>

http://hg.python.org/cpython/rev/2bec7c452b39
changeset:   73351:2bec7c452b39
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Fri Nov 04 12:26:49 2011 +0100
summary:
  Fix C89 incompatibility.

files:
  Python/codecs.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -613,9 +613,9 @@
         outp = PyUnicode_1BYTE_DATA(res);
         /* generate replacement */
         for (i = start, o = 0; i < end; ++i) {
-            ch = PyUnicode_READ_CHAR(object, i);
             int digits;
             int base;
+            ch = PyUnicode_READ_CHAR(object, i);
             *outp++ = '&';
             *outp++ = '#';
             if (ch<10) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 18:15:54 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 04 Nov 2011 18:15:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Revert_commit_t?=
 =?utf8?q?hat_was_not_a_bugfix_=28=235301=29=2E?=
Message-ID: <E1RMNMw-0003WQ-N7@dinsdale.python.org>

http://hg.python.org/cpython/rev/33680de042e7
changeset:   73352:33680de042e7
branch:      2.7
parent:      73347:f09e3b1603ee
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 04 18:15:45 2011 +0100
summary:
  Revert commit that was not a bugfix (#5301).

I won?t revert in 3.2, since Georg pulled this change in 3.2.2 and
released it.

files:
  Lib/mimetypes.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -432,7 +432,6 @@
         '.hdf'    : 'application/x-hdf',
         '.htm'    : 'text/html',
         '.html'   : 'text/html',
-        '.ico'    : 'image/vnd.microsoft.icon',
         '.ief'    : 'image/ief',
         '.jpe'    : 'image/jpeg',
         '.jpeg'   : 'image/jpeg',

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 18:23:26 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 04 Nov 2011 18:23:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Port_code_page_codec_to_Uni?=
	=?utf8?q?code_API=2E?=
Message-ID: <E1RMNUE-0004Bd-4R@dinsdale.python.org>

http://hg.python.org/cpython/rev/9191f804d376
changeset:   73353:9191f804d376
parent:      73351:2bec7c452b39
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Fri Nov 04 18:23:06 2011 +0100
summary:
  Port code page codec to Unicode API.

files:
  Lib/test/test_codeccallbacks.py |   12 +-
  Objects/unicodeobject.c         |  125 +++++++++++--------
  2 files changed, 74 insertions(+), 63 deletions(-)


diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -577,22 +577,18 @@
                 UnicodeEncodeError("ascii", "\uffff", 0, 1, "ouch")),
             ("\\uffff", 1)
         )
-        if SIZEOF_WCHAR_T == 2:
-            len_wide = 2
-        else:
-            len_wide = 1
         if SIZEOF_WCHAR_T > 0:
             self.assertEqual(
                 codecs.backslashreplace_errors(
                     UnicodeEncodeError("ascii", "\U00010000",
-                                       0, len_wide, "ouch")),
-                ("\\U00010000", len_wide)
+                                       0, 1, "ouch")),
+                ("\\U00010000", 1)
             )
             self.assertEqual(
                 codecs.backslashreplace_errors(
                     UnicodeEncodeError("ascii", "\U0010ffff",
-                                       0, len_wide, "ouch")),
-                ("\\U0010ffff", len_wide)
+                                       0, 1, "ouch")),
+                ("\\U0010ffff", 1)
             )
             # Lone surrogates (regardless of unicode width)
             self.assertEqual(
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4680,9 +4680,6 @@
     int kind;
     void *data;
     Py_ssize_t size;
-#if SIZEOF_WCHAR_T == 2
-    Py_ssize_t wchar_offset = 0;
-#endif
 
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadArgument();
@@ -4738,9 +4735,6 @@
             PyObject *rep;
             Py_ssize_t repsize, k, startpos;
             startpos = i-1;
-#if SIZEOF_WCHAR_T == 2
-            startpos += wchar_offset;
-#endif
             rep = unicode_encode_call_errorhandler(
                   errors, &errorHandler, "utf-8", "surrogates not allowed",
                   unicode, &exc, startpos, startpos+1, &newpos);
@@ -4809,9 +4803,6 @@
             *p++ = (char)(0x80 | ((ch >> 12) & 0x3f));
             *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
             *p++ = (char)(0x80 | (ch & 0x3f));
-#if SIZEOF_WCHAR_T == 2
-            wchar_offset++;
-#endif
         }
     }
 
@@ -7315,23 +7306,37 @@
  */
 static int
 encode_code_page_strict(UINT code_page, PyObject **outbytes,
-                        const Py_UNICODE *p, const int size,
+                        PyObject *unicode, Py_ssize_t offset, int len,
                         const char* errors)
 {
     BOOL usedDefaultChar = FALSE;
     BOOL *pusedDefaultChar = &usedDefaultChar;
     int outsize;
     PyObject *exc = NULL;
+	Py_UNICODE *p;
+	Py_ssize_t size;
     const DWORD flags = encode_code_page_flags(code_page, NULL);
     char *out;
-
-    assert(size > 0);
+	/* Create a substring so that we can get the UTF-16 representation
+	   of just the slice under consideration. */
+	PyObject *substring;
+
+    assert(len > 0);
 
     if (code_page != CP_UTF8 && code_page != CP_UTF7)
         pusedDefaultChar = &usedDefaultChar;
     else
         pusedDefaultChar = NULL;
 
+	substring = PyUnicode_Substring(unicode, offset, offset+len);
+	if (substring == NULL)
+		return -1;
+	p = PyUnicode_AsUnicodeAndSize(substring, &size);
+	if (p == NULL) {
+		Py_DECREF(substring);
+		return -1;
+	}
+
     /* First get the size of the result */
     outsize = WideCharToMultiByte(code_page, flags,
                                   p, size,
@@ -7340,14 +7345,18 @@
     if (outsize <= 0)
         goto error;
     /* If we used a default char, then we failed! */
-    if (pusedDefaultChar && *pusedDefaultChar)
+	if (pusedDefaultChar && *pusedDefaultChar) {
+		Py_DECREF(substring);
         return -2;
+	}
 
     if (*outbytes == NULL) {
         /* Create string object */
         *outbytes = PyBytes_FromStringAndSize(NULL, outsize);
-        if (*outbytes == NULL)
+		if (*outbytes == NULL) {
+			Py_DECREF(substring);
             return -1;
+		}
         out = PyBytes_AS_STRING(*outbytes);
     }
     else {
@@ -7355,10 +7364,13 @@
         const Py_ssize_t n = PyBytes_Size(*outbytes);
         if (outsize > PY_SSIZE_T_MAX - n) {
             PyErr_NoMemory();
+			Py_DECREF(substring);
             return -1;
         }
-        if (_PyBytes_Resize(outbytes, n + outsize) < 0)
+		if (_PyBytes_Resize(outbytes, n + outsize) < 0) {
+			Py_DECREF(substring);
             return -1;
+		}
         out = PyBytes_AS_STRING(*outbytes) + n;
     }
 
@@ -7367,6 +7379,7 @@
                                   p, size,
                                   out, outsize,
                                   NULL, pusedDefaultChar);
+	Py_CLEAR(substring);
     if (outsize <= 0)
         goto error;
     if (pusedDefaultChar && *pusedDefaultChar)
@@ -7374,6 +7387,7 @@
     return 0;
 
 error:
+	Py_XDECREF(substring);
     if (GetLastError() == ERROR_NO_UNICODE_TRANSLATION)
         return -2;
     PyErr_SetFromWindowsErr(0);
@@ -7390,12 +7404,11 @@
 static int
 encode_code_page_errors(UINT code_page, PyObject **outbytes,
                         PyObject *unicode, Py_ssize_t unicode_offset,
-                        const Py_UNICODE *in, const int insize,
-                        const char* errors)
+                        Py_ssize_t insize, const char* errors)
 {
     const DWORD flags = encode_code_page_flags(code_page, errors);
-    const Py_UNICODE *startin = in;
-    const Py_UNICODE *endin = in + insize;
+	Py_ssize_t pos = unicode_offset;
+	Py_ssize_t endin = unicode_offset + insize;
     /* Ideally, we should get reason from FormatMessage. This is the Windows
        2000 English version of the message. */
     const char *reason = "invalid character";
@@ -7404,12 +7417,11 @@
     BOOL usedDefaultChar = FALSE, *pusedDefaultChar;
     Py_ssize_t outsize;
     char *out;
-    int charsize;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
     PyObject *encoding_obj = NULL;
     char *encoding;
-    Py_ssize_t startpos, newpos, newoutsize;
+    Py_ssize_t newpos, newoutsize;
     PyObject *rep;
     int ret = -1;
 
@@ -7422,7 +7434,7 @@
     if (errors == NULL || strcmp(errors, "strict") == 0) {
         /* The last error was ERROR_NO_UNICODE_TRANSLATION,
            then we raise a UnicodeEncodeError. */
-        make_encode_exception(&exc, encoding, in, insize, 0, 0, reason);
+        make_encode_exception_obj(&exc, encoding, unicode, 0, 0, reason);
         if (exc != NULL) {
             PyCodec_StrictErrors(exc);
             Py_DECREF(exc);
@@ -7462,23 +7474,30 @@
     }
 
     /* Encode the string character per character */
-    while (in < endin)
+    while (pos < endin)
     {
-        if ((in + 2) <= endin
-            && 0xD800 <= in[0] && in[0] <= 0xDBFF
-            && 0xDC00 <= in[1] && in[1] <= 0xDFFF)
-            charsize = 2;
-        else
-            charsize = 1;
-
+		Py_UCS4 ch = PyUnicode_READ_CHAR(unicode, pos);
+		wchar_t chars[2];
+		int charsize;
+		if (ch < 0x10000) {
+			chars[0] = (wchar_t)ch;
+			charsize = 1;
+		}
+		else {
+			ch -= 0x10000;
+			chars[0] = 0xd800 + (ch >> 10);
+			chars[1] = 0xdc00 + (ch & 0x3ff);
+			charsize = 2;
+		}
+		
         outsize = WideCharToMultiByte(code_page, flags,
-                                      in, charsize,
+                                      chars, charsize,
                                       buffer, Py_ARRAY_LENGTH(buffer),
                                       NULL, pusedDefaultChar);
         if (outsize > 0) {
             if (pusedDefaultChar == NULL || !(*pusedDefaultChar))
             {
-                in += charsize;
+                pos++;
                 memcpy(out, buffer, outsize);
                 out += outsize;
                 continue;
@@ -7489,15 +7508,13 @@
             goto error;
         }
 
-        charsize = Py_MAX(charsize - 1, 1);
-        startpos = unicode_offset + in - startin;
         rep = unicode_encode_call_errorhandler(
                   errors, &errorHandler, encoding, reason,
                   unicode, &exc,
-                  startpos, startpos + charsize, &newpos);
+                  pos, pos + 1, &newpos);
         if (rep == NULL)
             goto error;
-        in += (newpos - startpos);
+        pos = newpos;
 
         if (PyBytes_Check(rep)) {
             outsize = PyBytes_GET_SIZE(rep);
@@ -7538,10 +7555,9 @@
             for (i=0; i < outsize; i++) {
                 Py_UCS4 ch = PyUnicode_READ(kind, data, i);
                 if (ch > 127) {
-                    raise_encode_exception(&exc,
-                        encoding,
-                        startin, insize,
-                        startpos, startpos + charsize,
+                    raise_encode_exception_obj(&exc,
+                        encoding, unicode,
+                        pos, pos + 1,
                         "unable to encode error handler result to ASCII");
                     Py_DECREF(rep);
                     goto error;
@@ -7572,55 +7588,54 @@
                  PyObject *unicode,
                  const char *errors)
 {
-    const Py_UNICODE *p;
-    Py_ssize_t size;
+    Py_ssize_t len;
     PyObject *outbytes = NULL;
     Py_ssize_t offset;
     int chunk_len, ret, done;
 
-    p = PyUnicode_AsUnicodeAndSize(unicode, &size);
-    if (p == NULL)
-        return NULL;
+	if (PyUnicode_READY(unicode) < 0)
+		return NULL;
+	len = PyUnicode_GET_LENGTH(unicode);
 
     if (code_page < 0) {
         PyErr_SetString(PyExc_ValueError, "invalid code page number");
         return NULL;
     }
 
-    if (size == 0)
+    if (len == 0)
         return PyBytes_FromStringAndSize(NULL, 0);
 
     offset = 0;
     do
     {
 #ifdef NEED_RETRY
-        if (size > INT_MAX) {
-            chunk_len = INT_MAX;
+		/* UTF-16 encoding may double the size, so use only INT_MAX/2
+           chunks. */
+        if (len > INT_MAX/2) {
+            chunk_len = INT_MAX/2;
             done = 0;
         }
         else
 #endif
         {
-            chunk_len = (int)size;
+            chunk_len = (int)len;
             done = 1;
         }
-
+	
         ret = encode_code_page_strict(code_page, &outbytes,
-                                      p, chunk_len,
+                                      unicode, offset, chunk_len,
                                       errors);
         if (ret == -2)
             ret = encode_code_page_errors(code_page, &outbytes,
                                           unicode, offset,
-                                          p, chunk_len,
-                                          errors);
+                                          chunk_len, errors);
         if (ret < 0) {
             Py_XDECREF(outbytes);
             return NULL;
         }
 
-        p += chunk_len;
         offset += chunk_len;
-        size -= chunk_len;
+        len -= chunk_len;
     } while (!done);
 
     return outbytes;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 19:04:27 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 04 Nov 2011 19:04:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Drop_Py=5FUNICODE_based_enc?=
	=?utf8?q?ode_exceptions=2E?=
Message-ID: <E1RMO7v-0008Tq-Lf@dinsdale.python.org>

http://hg.python.org/cpython/rev/73662e13ddae
changeset:   73354:73662e13ddae
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Fri Nov 04 19:04:15 2011 +0100
summary:
  Drop Py_UNICODE based encode exceptions.

files:
  Objects/unicodeobject.c |  90 ++++++++--------------------
  1 files changed, 26 insertions(+), 64 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -254,12 +254,6 @@
 static void
 raise_encode_exception(PyObject **exceptionObject,
                        const char *encoding,
-                       const Py_UNICODE *unicode, Py_ssize_t size,
-                       Py_ssize_t startpos, Py_ssize_t endpos,
-                       const char *reason);
-static void
-raise_encode_exception_obj(PyObject **exceptionObject,
-                       const char *encoding,
                        PyObject *unicode,
                        Py_ssize_t startpos, Py_ssize_t endpos,
                        const char *reason);
@@ -3058,8 +3052,7 @@
                 if (errmsg == NULL)
                     errmsg = "Py_wchar2char() failed";
                 raise_encode_exception(&exc,
-                    "filesystemencoding",
-                    PyUnicode_AS_UNICODE(unicode), PyUnicode_GET_SIZE(unicode),
+                    "filesystemencoding", unicode,
                     error_pos, error_pos+1,
                     errmsg);
                 Py_XDECREF(exc);
@@ -4783,7 +4776,7 @@
                 for(k=0; k<repsize; k++) {
                     c = prep[k];
                     if (0x80 <= c) {
-                        raise_encode_exception_obj(&exc, "utf-8",
+                        raise_encode_exception(&exc, "utf-8",
                                                unicode,
                                                i-1, i,
                                                "surrogates not allowed");
@@ -6406,13 +6399,14 @@
 static void
 make_encode_exception(PyObject **exceptionObject,
                       const char *encoding,
-                      const Py_UNICODE *unicode, Py_ssize_t size,
+                      PyObject *unicode,
                       Py_ssize_t startpos, Py_ssize_t endpos,
                       const char *reason)
 {
     if (*exceptionObject == NULL) {
-        *exceptionObject = PyUnicodeEncodeError_Create(
-            encoding, unicode, size, startpos, endpos, reason);
+        *exceptionObject = PyObject_CallFunction(
+            PyExc_UnicodeEncodeError, "sOnns",
+            encoding, unicode, startpos, endpos, reason);
     }
     else {
         if (PyUnicodeEncodeError_SetStart(*exceptionObject, startpos))
@@ -6428,55 +6422,15 @@
     }
 }
 
-/* This is ultimately going t replace above function. */
-static void
-make_encode_exception_obj(PyObject **exceptionObject,
-                      const char *encoding,
-                      PyObject *unicode,
-                      Py_ssize_t startpos, Py_ssize_t endpos,
-                      const char *reason)
-{
-    if (*exceptionObject == NULL) {
-        *exceptionObject = PyObject_CallFunction(
-            PyExc_UnicodeEncodeError, "sOnns",
-            encoding, unicode, startpos, endpos, reason);
-    }
-    else {
-        if (PyUnicodeEncodeError_SetStart(*exceptionObject, startpos))
-            goto onError;
-        if (PyUnicodeEncodeError_SetEnd(*exceptionObject, endpos))
-            goto onError;
-        if (PyUnicodeEncodeError_SetReason(*exceptionObject, reason))
-            goto onError;
-        return;
-      onError:
-        Py_DECREF(*exceptionObject);
-        *exceptionObject = NULL;
-    }
-}
-
 /* raises a UnicodeEncodeError */
 static void
 raise_encode_exception(PyObject **exceptionObject,
                        const char *encoding,
-                       const Py_UNICODE *unicode, Py_ssize_t size,
-                       Py_ssize_t startpos, Py_ssize_t endpos,
-                       const char *reason)
-{
-    make_encode_exception(exceptionObject,
-                          encoding, unicode, size, startpos, endpos, reason);
-    if (*exceptionObject != NULL)
-        PyCodec_StrictErrors(*exceptionObject);
-}
-/* This is ultimately going to replace above function. */
-static void
-raise_encode_exception_obj(PyObject **exceptionObject,
-                       const char *encoding,
                        PyObject *unicode,
                        Py_ssize_t startpos, Py_ssize_t endpos,
                        const char *reason)
 {
-    make_encode_exception_obj(exceptionObject,
+    make_encode_exception(exceptionObject,
                           encoding, unicode, startpos, endpos, reason);
     if (*exceptionObject != NULL)
         PyCodec_StrictErrors(*exceptionObject);
@@ -6509,7 +6463,7 @@
         return NULL;
     len = PyUnicode_GET_LENGTH(unicode);
 
-    make_encode_exception_obj(exceptionObject,
+    make_encode_exception(exceptionObject,
                           encoding, unicode, startpos, endpos, reason);
     if (*exceptionObject == NULL)
         return NULL;
@@ -6617,7 +6571,7 @@
             }
             switch (known_errorHandler) {
             case 1: /* strict */
-                raise_encode_exception_obj(&exc, encoding, unicode, collstart, collend, reason);
+                raise_encode_exception(&exc, encoding, unicode, collstart, collend, reason);
                 goto onError;
             case 2: /* replace */
                 while (collstart++<collend)
@@ -6712,7 +6666,7 @@
                 for (i = 0; repsize-->0; ++i, ++str) {
                     c = PyUnicode_READ_CHAR(repunicode, i);
                     if (c >= limit) {
-                        raise_encode_exception_obj(&exc, encoding, unicode,
+                        raise_encode_exception(&exc, encoding, unicode,
                                                pos, pos+1, reason);
                         Py_DECREF(repunicode);
                         goto onError;
@@ -7434,7 +7388,7 @@
     if (errors == NULL || strcmp(errors, "strict") == 0) {
         /* The last error was ERROR_NO_UNICODE_TRANSLATION,
            then we raise a UnicodeEncodeError. */
-        make_encode_exception_obj(&exc, encoding, unicode, 0, 0, reason);
+        make_encode_exception(&exc, encoding, unicode, 0, 0, reason);
         if (exc != NULL) {
             PyCodec_StrictErrors(exc);
             Py_DECREF(exc);
@@ -7555,7 +7509,7 @@
             for (i=0; i < outsize; i++) {
                 Py_UCS4 ch = PyUnicode_READ(kind, data, i);
                 if (ch > 127) {
-                    raise_encode_exception_obj(&exc,
+                    raise_encode_exception(&exc,
                         encoding, unicode,
                         pos, pos + 1,
                         "unable to encode error handler result to ASCII");
@@ -8250,7 +8204,7 @@
     }
     switch (*known_errorHandler) {
     case 1: /* strict */
-        raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
+        raise_encode_exception(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
         return -1;
     case 2: /* replace */
         for (collpos = collstartpos; collpos<collendpos; ++collpos) {
@@ -8259,7 +8213,7 @@
                 return -1;
             }
             else if (x==enc_FAILED) {
-                raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
+                raise_encode_exception(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
                 return -1;
             }
         }
@@ -8278,7 +8232,7 @@
                 if (x==enc_EXCEPTION)
                     return -1;
                 else if (x==enc_FAILED) {
-                    raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
+                    raise_encode_exception(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
                     return -1;
                 }
             }
@@ -8319,7 +8273,7 @@
             }
             else if (x==enc_FAILED) {
                 Py_DECREF(repunicode);
-                raise_encode_exception_obj(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
+                raise_encode_exception(exceptionObject, encoding, unicode, collstartpos, collendpos, reason);
                 return -1;
             }
         }
@@ -8990,7 +8944,11 @@
         }
         switch (known_errorHandler) {
         case 1: /* strict */
-            raise_encode_exception(&exc, encoding, s, length, collstart-s, collend-s, reason);
+            unicode = PyUnicode_FromUnicode(s, length);
+            if (unicode == NULL)
+                goto onError;
+            raise_encode_exception(&exc, encoding, unicode, collstart-s, collend-s, reason);
+            Py_DECREF(unicode);
             goto onError;
         case 2: /* replace */
             for (p = collstart; p < collend; ++p)
@@ -9035,8 +8993,12 @@
                         *output++ = (char)ch;
                     else {
                         Py_DECREF(repunicode);
+                        unicode = PyUnicode_FromUnicode(s, length);
+                        if (unicode == NULL)
+                            goto onError;
                         raise_encode_exception(&exc, encoding,
-                                               s, length, collstart-s, collend-s, reason);
+                                               unicode, collstart-s, collend-s, reason);
+                        Py_DECREF(unicode);
                         goto onError;
                     }
                 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 20:05:09 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 20:05:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_tabs_by_spaces?=
Message-ID: <E1RMP4f-0006MK-Vd@dinsdale.python.org>

http://hg.python.org/cpython/rev/6aa9a151d064
changeset:   73355:6aa9a151d064
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 20:06:39 2011 +0100
summary:
  Replace tabs by spaces

files:
  Objects/unicodeobject.c |  92 ++++++++++++++--------------
  1 files changed, 46 insertions(+), 46 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7267,13 +7267,13 @@
     BOOL *pusedDefaultChar = &usedDefaultChar;
     int outsize;
     PyObject *exc = NULL;
-	Py_UNICODE *p;
-	Py_ssize_t size;
+    Py_UNICODE *p;
+    Py_ssize_t size;
     const DWORD flags = encode_code_page_flags(code_page, NULL);
     char *out;
-	/* Create a substring so that we can get the UTF-16 representation
-	   of just the slice under consideration. */
-	PyObject *substring;
+    /* Create a substring so that we can get the UTF-16 representation
+       of just the slice under consideration. */
+    PyObject *substring;
 
     assert(len > 0);
 
@@ -7282,14 +7282,14 @@
     else
         pusedDefaultChar = NULL;
 
-	substring = PyUnicode_Substring(unicode, offset, offset+len);
-	if (substring == NULL)
-		return -1;
-	p = PyUnicode_AsUnicodeAndSize(substring, &size);
-	if (p == NULL) {
-		Py_DECREF(substring);
-		return -1;
-	}
+    substring = PyUnicode_Substring(unicode, offset, offset+len);
+    if (substring == NULL)
+        return -1;
+    p = PyUnicode_AsUnicodeAndSize(substring, &size);
+    if (p == NULL) {
+        Py_DECREF(substring);
+        return -1;
+    }
 
     /* First get the size of the result */
     outsize = WideCharToMultiByte(code_page, flags,
@@ -7299,18 +7299,18 @@
     if (outsize <= 0)
         goto error;
     /* If we used a default char, then we failed! */
-	if (pusedDefaultChar && *pusedDefaultChar) {
-		Py_DECREF(substring);
+    if (pusedDefaultChar && *pusedDefaultChar) {
+        Py_DECREF(substring);
         return -2;
-	}
+    }
 
     if (*outbytes == NULL) {
         /* Create string object */
         *outbytes = PyBytes_FromStringAndSize(NULL, outsize);
-		if (*outbytes == NULL) {
-			Py_DECREF(substring);
+        if (*outbytes == NULL) {
+            Py_DECREF(substring);
             return -1;
-		}
+        }
         out = PyBytes_AS_STRING(*outbytes);
     }
     else {
@@ -7318,13 +7318,13 @@
         const Py_ssize_t n = PyBytes_Size(*outbytes);
         if (outsize > PY_SSIZE_T_MAX - n) {
             PyErr_NoMemory();
-			Py_DECREF(substring);
+            Py_DECREF(substring);
             return -1;
         }
-		if (_PyBytes_Resize(outbytes, n + outsize) < 0) {
-			Py_DECREF(substring);
+        if (_PyBytes_Resize(outbytes, n + outsize) < 0) {
+            Py_DECREF(substring);
             return -1;
-		}
+        }
         out = PyBytes_AS_STRING(*outbytes) + n;
     }
 
@@ -7333,7 +7333,7 @@
                                   p, size,
                                   out, outsize,
                                   NULL, pusedDefaultChar);
-	Py_CLEAR(substring);
+    Py_CLEAR(substring);
     if (outsize <= 0)
         goto error;
     if (pusedDefaultChar && *pusedDefaultChar)
@@ -7341,7 +7341,7 @@
     return 0;
 
 error:
-	Py_XDECREF(substring);
+    Py_XDECREF(substring);
     if (GetLastError() == ERROR_NO_UNICODE_TRANSLATION)
         return -2;
     PyErr_SetFromWindowsErr(0);
@@ -7361,8 +7361,8 @@
                         Py_ssize_t insize, const char* errors)
 {
     const DWORD flags = encode_code_page_flags(code_page, errors);
-	Py_ssize_t pos = unicode_offset;
-	Py_ssize_t endin = unicode_offset + insize;
+    Py_ssize_t pos = unicode_offset;
+    Py_ssize_t endin = unicode_offset + insize;
     /* Ideally, we should get reason from FormatMessage. This is the Windows
        2000 English version of the message. */
     const char *reason = "invalid character";
@@ -7430,20 +7430,20 @@
     /* Encode the string character per character */
     while (pos < endin)
     {
-		Py_UCS4 ch = PyUnicode_READ_CHAR(unicode, pos);
-		wchar_t chars[2];
-		int charsize;
-		if (ch < 0x10000) {
-			chars[0] = (wchar_t)ch;
-			charsize = 1;
-		}
-		else {
-			ch -= 0x10000;
-			chars[0] = 0xd800 + (ch >> 10);
-			chars[1] = 0xdc00 + (ch & 0x3ff);
-			charsize = 2;
-		}
-		
+        Py_UCS4 ch = PyUnicode_READ_CHAR(unicode, pos);
+        wchar_t chars[2];
+        int charsize;
+        if (ch < 0x10000) {
+            chars[0] = (wchar_t)ch;
+            charsize = 1;
+        }
+        else {
+            ch -= 0x10000;
+            chars[0] = 0xd800 + (ch >> 10);
+            chars[1] = 0xdc00 + (ch & 0x3ff);
+            charsize = 2;
+        }
+
         outsize = WideCharToMultiByte(code_page, flags,
                                       chars, charsize,
                                       buffer, Py_ARRAY_LENGTH(buffer),
@@ -7547,9 +7547,9 @@
     Py_ssize_t offset;
     int chunk_len, ret, done;
 
-	if (PyUnicode_READY(unicode) < 0)
-		return NULL;
-	len = PyUnicode_GET_LENGTH(unicode);
+    if (PyUnicode_READY(unicode) < 0)
+        return NULL;
+    len = PyUnicode_GET_LENGTH(unicode);
 
     if (code_page < 0) {
         PyErr_SetString(PyExc_ValueError, "invalid code page number");
@@ -7563,7 +7563,7 @@
     do
     {
 #ifdef NEED_RETRY
-		/* UTF-16 encoding may double the size, so use only INT_MAX/2
+        /* UTF-16 encoding may double the size, so use only INT_MAX/2
            chunks. */
         if (len > INT_MAX/2) {
             chunk_len = INT_MAX/2;
@@ -7575,7 +7575,7 @@
             chunk_len = (int)len;
             done = 1;
         }
-	
+
         ret = encode_code_page_strict(code_page, &outbytes,
                                       unicode, offset, chunk_len,
                                       errors);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 20:52:33 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 20:52:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_gdb/libpython=2Epy_for_?=
	=?utf8?q?not_ready_Unicode_strings?=
Message-ID: <E1RMPoX-0002R1-Ac@dinsdale.python.org>

http://hg.python.org/cpython/rev/b3543ef9b445
changeset:   73356:b3543ef9b445
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 20:54:05 2011 +0100
summary:
  Fix gdb/libpython.py for not ready Unicode strings

_PyUnicode_CheckConsistency() checks also hash and length value for not ready
Unicode strings.

files:
  Include/unicodeobject.h |  8 +++++---
  Objects/unicodeobject.c |  7 +++++--
  Tools/gdb/libpython.py  |  5 -----
  3 files changed, 10 insertions(+), 10 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -231,22 +231,24 @@
          * utf8_length = 0 if utf8 is NULL
          * wstr is shared with data and wstr_length=length
            if kind=PyUnicode_2BYTE_KIND and sizeof(wchar_t)=2
-           or if kind=PyUnicode_4BYTE_KIND and sizeof(wchar_4)=4
+           or if kind=PyUnicode_4BYTE_KIND and sizeof(wchar_t)=4
          * wstr_length = 0 if wstr is NULL
          * (data starts just after the structure)
 
        - legacy string, not ready:
 
          * structure = PyUnicodeObject
+         * length = 0 (use wstr_length)
+         * hash = -1
          * kind = PyUnicode_WCHAR_KIND
          * compact = 0
          * ascii = 0
          * ready = 0
+         * interned = SSTATE_NOT_INTERNED
          * wstr is not NULL
          * data.any is NULL
          * utf8 is NULL
          * utf8_length = 0
-         * interned = SSTATE_NOT_INTERNED
 
        - legacy string, ready:
 
@@ -258,7 +260,7 @@
          * data.any is not NULL
          * utf8 is shared and utf8_length = length with data.any if ascii = 1
          * utf8_length = 0 if utf8 is NULL
-         * wstr is shared and wstr_length = length with data.any
+         * wstr is shared with data.any and wstr_length = length
            if kind=PyUnicode_2BYTE_KIND and sizeof(wchar_t)=2
            or if kind=PyUnicode_4BYTE_KIND and sizeof(wchar_4)=4
          * wstr_length = 0 if wstr is NULL
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -328,18 +328,21 @@
             assert(ascii->state.ascii == 0);
             assert(ascii->state.ready == 1);
             assert (compact->utf8 != data);
-        } else {
+        }
+        else {
             PyUnicodeObject *unicode = (PyUnicodeObject *)op;
 
             data = unicode->data.any;
             if (kind == PyUnicode_WCHAR_KIND) {
+                assert(ascii->length == 0);
+                assert(ascii->hash == -1);
                 assert(ascii->state.compact == 0);
                 assert(ascii->state.ascii == 0);
                 assert(ascii->state.ready == 0);
+                assert(ascii->state.interned == SSTATE_NOT_INTERNED);
                 assert(ascii->wstr != NULL);
                 assert(data == NULL);
                 assert(compact->utf8 == NULL);
-                assert(ascii->state.interned == SSTATE_NOT_INTERNED);
             }
             else {
                 assert(kind == PyUnicode_1BYTE_KIND
diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py
--- a/Tools/gdb/libpython.py
+++ b/Tools/gdb/libpython.py
@@ -1123,9 +1123,6 @@
         return _type_Py_UNICODE.sizeof
 
     def proxyval(self, visited):
-        # From unicodeobject.h:
-        #     Py_ssize_t length;  /* Length of raw Unicode data in buffer */
-        #     Py_UNICODE *str;    /* Raw Unicode buffer */
         if _is_pep393:
             # Python 3.3 and newer
             may_have_surrogates = False
@@ -1138,8 +1135,6 @@
                 # string is not ready
                 may_have_surrogates = True
                 field_str = ascii['wstr']
-                if not is_compact_ascii:
-                    field_length = compact('wstr_length')
             else:
                 if is_compact_ascii:
                     field_str = ascii.address + 1

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:08:01 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Fri, 04 Nov 2011 21:08:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Inline_the_advi?=
 =?utf8?q?sory_text_on_how_to_use_the_shelve_module=2E?=
Message-ID: <E1RMQ3V-0003il-Ku@dinsdale.python.org>

http://hg.python.org/cpython/rev/e34d127f91a9
changeset:   73357:e34d127f91a9
branch:      2.7
parent:      73352:33680de042e7
user:        Raymond Hettinger <python at rcn.com>
date:        Fri Nov 04 13:07:52 2011 -0700
summary:
  Inline the advisory text on how to use the shelve module.

files:
  Doc/library/shelve.rst |  15 +++++----------
  1 files changed, 5 insertions(+), 10 deletions(-)


diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst
--- a/Doc/library/shelve.rst
+++ b/Doc/library/shelve.rst
@@ -44,17 +44,12 @@
    determine which accessed entries are mutable, nor which ones were actually
    mutated).
 
-   .. note::
+   Like file objects, shelve objects should closed explicitly to assure
+   that the peristent data is flushed to disk.
 
-      Do not rely on the shelf being closed automatically; always call
-      :meth:`close` explicitly when you don't need it any more, or use a
-      :keyword:`with` statement with :func:`contextlib.closing`.
-
-.. warning::
-
-   Because the :mod:`shelve` module is backed by :mod:`pickle`, it is insecure
-   to load a shelf from an untrusted source.  Like with pickle, loading a shelf
-   can execute arbitrary code.
+   Since the :mod:`shelve` module stores objects using :mod:`pickle`, the same
+   security precautions apply.  Accordingly, you should avoid loading a shelf
+   from an untrusted source.
 
 Shelf objects support all methods supported by dictionaries.  This eases the
 transition from dictionary based scripts to those requiring persistent storage.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:35:03 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 21:35:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyCodec=5FXMLCharRefReplace?=
 =?utf8?q?Error=28=29=3A_Remove_unused_variable?=
Message-ID: <E1RMQTf-0006WC-Cs@dinsdale.python.org>

http://hg.python.org/cpython/rev/2df7d749ee3a
changeset:   73358:2df7d749ee3a
parent:      73356:b3543ef9b445
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 21:29:10 2011 +0100
summary:
  PyCodec_XMLCharRefReplaceError(): Remove unused variable

files:
  Python/codecs.c |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -573,7 +573,7 @@
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
         PyObject *restuple;
         PyObject *object;
-        Py_ssize_t i, o;
+        Py_ssize_t i;
         Py_ssize_t start;
         Py_ssize_t end;
         PyObject *res;
@@ -612,7 +612,7 @@
         }
         outp = PyUnicode_1BYTE_DATA(res);
         /* generate replacement */
-        for (i = start, o = 0; i < end; ++i) {
+        for (i = start; i < end; ++i) {
             int digits;
             int base;
             ch = PyUnicode_READ_CHAR(object, i);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:35:04 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 21:35:04 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_the_Py=5FUNICODE_type?=
	=?utf8?q?_in_codecs=2Ec?=
Message-ID: <E1RMQTg-0006XX-5m@dinsdale.python.org>

http://hg.python.org/cpython/rev/e3b6458056c7
changeset:   73359:e3b6458056c7
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 21:36:35 2011 +0100
summary:
  Avoid the Py_UNICODE type in codecs.c

files:
  Python/codecs.c |  15 +++++++++++----
  1 files changed, 11 insertions(+), 4 deletions(-)


diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -778,7 +778,7 @@
     }
     else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
         unsigned char *p;
-        Py_UNICODE ch = 0;
+        Py_UCS4 ch = 0;
         if (PyUnicodeDecodeError_GetStart(exc, &start))
             return NULL;
         if (!(object = PyUnicodeDecodeError_GetObject(exc)))
@@ -804,7 +804,10 @@
             PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
             return NULL;
         }
-        return Py_BuildValue("(u#n)", &ch, 1, start+3);
+        res = PyUnicode_FromOrdinal(ch);
+        if (res == NULL)
+            return NULL;
+        return Py_BuildValue("(Nn)", res, start+3);
     }
     else {
         wrong_exception_type(exc);
@@ -853,8 +856,9 @@
         return restuple;
     }
     else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
+        PyObject *str;
         unsigned char *p;
-        Py_UNICODE ch[4]; /* decode up to 4 bad bytes. */
+        Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
         int consumed = 0;
         if (PyUnicodeDecodeError_GetStart(exc, &start))
             return NULL;
@@ -879,7 +883,10 @@
             PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
             return NULL;
         }
-        return Py_BuildValue("(u#n)", ch, consumed, start+consumed);
+        str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
+        if (str == NULL)
+            return NULL;
+        return Py_BuildValue("(Nn)", str, start+consumed);
     }
     else {
         wrong_exception_type(exc);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:41:47 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Fri, 04 Nov 2011 21:41:47 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzMwNjc6?=
 =?utf8?q?_Fix_the_error_raised_by_locale=2Esetlocale=28=29?=
Message-ID: <E1RMQaB-0007HK-Ro@dinsdale.python.org>

http://hg.python.org/cpython/rev/931ae170e51c
changeset:   73360:931ae170e51c
branch:      3.2
parent:      73348:94017ce9304d
user:        Petri Lehtinen <petri at digip.org>
date:        Fri Nov 04 21:35:07 2011 +0200
summary:
  Issue #3067: Fix the error raised by locale.setlocale()

files:
  Lib/locale.py           |  18 +++++++++++-------
  Lib/test/test_locale.py |   8 ++++++++
  Misc/ACKS               |   1 +
  Misc/NEWS               |   3 +++
  4 files changed, 23 insertions(+), 7 deletions(-)


diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -440,13 +440,17 @@
         No aliasing or normalizing takes place.
 
     """
-    language, encoding = localetuple
-    if language is None:
-        language = 'C'
-    if encoding is None:
-        return language
-    else:
-        return language + '.' + encoding
+    try:
+        language, encoding = localetuple
+
+        if language is None:
+            language = 'C'
+        if encoding is None:
+            return language
+        else:
+            return language + '.' + encoding
+    except (TypeError, ValueError):
+        raise TypeError('Locale must be None, a string, or an iterable of two strings -- language code, encoding.')
 
 def getdefaultlocale(envvars=('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')):
 
diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py
--- a/Lib/test/test_locale.py
+++ b/Lib/test/test_locale.py
@@ -407,6 +407,14 @@
         locale.setlocale(locale.LC_CTYPE, loc)
         self.assertEqual(loc, locale.getlocale(locale.LC_CTYPE))
 
+    def test_invalid_locale_format_in_localetuple(self):
+        with self.assertRaises(TypeError):
+            locale.setlocale(locale.LC_ALL, b'fi_FI')
+
+    def test_invalid_iterable_in_localetuple(self):
+        with self.assertRaises(TypeError):
+            locale.setlocale(locale.LC_ALL, (b'not', b'valid'))
+
 
 def test_main():
     tests = [
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -717,6 +717,7 @@
 Amrit Prem
 Paul Prescod
 Donovan Preston
+Jyrki Pulliainen
 Steve Purcell
 Fernando P?rez
 Eduardo P?rez
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -66,6 +66,9 @@
 Library
 -------
 
+- Issue #3067: locale.setlocale() now raises TypeError if the second
+  argument is an invalid iterable. Initial patch by Jyrki Pulliainen.
+
 - Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
 
 - Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:41:49 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Fri, 04 Nov 2011 21:41:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=233067=3A_Fix_the_error_raised_by_locale=2Esetlocale?=
 =?utf8?b?KCk=?=
Message-ID: <E1RMQaD-0007HU-JY@dinsdale.python.org>

http://hg.python.org/cpython/rev/d90d88380aca
changeset:   73361:d90d88380aca
parent:      73356:b3543ef9b445
parent:      73360:931ae170e51c
user:        Petri Lehtinen <petri at digip.org>
date:        Fri Nov 04 22:21:52 2011 +0200
summary:
  Issue #3067: Fix the error raised by locale.setlocale()

files:
  Lib/locale.py           |  18 +++++++++++-------
  Lib/test/test_locale.py |   8 ++++++++
  Misc/ACKS               |   1 +
  Misc/NEWS               |   3 +++
  4 files changed, 23 insertions(+), 7 deletions(-)


diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -440,13 +440,17 @@
         No aliasing or normalizing takes place.
 
     """
-    language, encoding = localetuple
-    if language is None:
-        language = 'C'
-    if encoding is None:
-        return language
-    else:
-        return language + '.' + encoding
+    try:
+        language, encoding = localetuple
+
+        if language is None:
+            language = 'C'
+        if encoding is None:
+            return language
+        else:
+            return language + '.' + encoding
+    except (TypeError, ValueError):
+        raise TypeError('Locale must be None, a string, or an iterable of two strings -- language code, encoding.')
 
 def getdefaultlocale(envvars=('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')):
 
diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py
--- a/Lib/test/test_locale.py
+++ b/Lib/test/test_locale.py
@@ -409,6 +409,14 @@
         locale.setlocale(locale.LC_CTYPE, loc)
         self.assertEqual(loc, locale.getlocale(locale.LC_CTYPE))
 
+    def test_invalid_locale_format_in_localetuple(self):
+        with self.assertRaises(TypeError):
+            locale.setlocale(locale.LC_ALL, b'fi_FI')
+
+    def test_invalid_iterable_in_localetuple(self):
+        with self.assertRaises(TypeError):
+            locale.setlocale(locale.LC_ALL, (b'not', b'valid'))
+
 
 def test_main():
     tests = [
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -768,6 +768,7 @@
 Amrit Prem
 Paul Prescod
 Donovan Preston
+Jyrki Pulliainen
 Steve Purcell
 Fernando P?rez
 Eduardo P?rez
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -350,6 +350,9 @@
 Library
 -------
 
+- Issue #3067: locale.setlocale() now raises TypeError if the second
+  argument is an invalid iterable. Initial patch by Jyrki Pulliainen.
+
 - Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
 
 - Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:41:50 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Fri, 04 Nov 2011 21:41:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge_heads?=
Message-ID: <E1RMQaE-0007I2-I9@dinsdale.python.org>

http://hg.python.org/cpython/rev/cade039a7be4
changeset:   73362:cade039a7be4
parent:      73361:d90d88380aca
parent:      73359:e3b6458056c7
user:        Petri Lehtinen <petri at digip.org>
date:        Fri Nov 04 22:36:54 2011 +0200
summary:
  Merge heads

files:
  Python/codecs.c |  19 +++++++++++++------
  1 files changed, 13 insertions(+), 6 deletions(-)


diff --git a/Python/codecs.c b/Python/codecs.c
--- a/Python/codecs.c
+++ b/Python/codecs.c
@@ -573,7 +573,7 @@
     if (PyObject_IsInstance(exc, PyExc_UnicodeEncodeError)) {
         PyObject *restuple;
         PyObject *object;
-        Py_ssize_t i, o;
+        Py_ssize_t i;
         Py_ssize_t start;
         Py_ssize_t end;
         PyObject *res;
@@ -612,7 +612,7 @@
         }
         outp = PyUnicode_1BYTE_DATA(res);
         /* generate replacement */
-        for (i = start, o = 0; i < end; ++i) {
+        for (i = start; i < end; ++i) {
             int digits;
             int base;
             ch = PyUnicode_READ_CHAR(object, i);
@@ -778,7 +778,7 @@
     }
     else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
         unsigned char *p;
-        Py_UNICODE ch = 0;
+        Py_UCS4 ch = 0;
         if (PyUnicodeDecodeError_GetStart(exc, &start))
             return NULL;
         if (!(object = PyUnicodeDecodeError_GetObject(exc)))
@@ -804,7 +804,10 @@
             PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
             return NULL;
         }
-        return Py_BuildValue("(u#n)", &ch, 1, start+3);
+        res = PyUnicode_FromOrdinal(ch);
+        if (res == NULL)
+            return NULL;
+        return Py_BuildValue("(Nn)", res, start+3);
     }
     else {
         wrong_exception_type(exc);
@@ -853,8 +856,9 @@
         return restuple;
     }
     else if (PyObject_IsInstance(exc, PyExc_UnicodeDecodeError)) {
+        PyObject *str;
         unsigned char *p;
-        Py_UNICODE ch[4]; /* decode up to 4 bad bytes. */
+        Py_UCS2 ch[4]; /* decode up to 4 bad bytes. */
         int consumed = 0;
         if (PyUnicodeDecodeError_GetStart(exc, &start))
             return NULL;
@@ -879,7 +883,10 @@
             PyErr_SetObject(PyExceptionInstance_Class(exc), exc);
             return NULL;
         }
-        return Py_BuildValue("(u#n)", ch, consumed, start+consumed);
+        str = PyUnicode_FromKindAndData(PyUnicode_2BYTE_KIND, ch, consumed);
+        if (str == NULL)
+            return NULL;
+        return Py_BuildValue("(Nn)", str, start+consumed);
     }
     else {
         wrong_exception_type(exc);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 21:41:51 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Fri, 04 Nov 2011 21:41:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Add_=2Egitignor?=
	=?utf8?q?e?=
Message-ID: <E1RMQaF-0007IK-AO@dinsdale.python.org>

http://hg.python.org/cpython/rev/9373c14cade9
changeset:   73363:9373c14cade9
branch:      2.7
parent:      73357:e34d127f91a9
user:        Petri Lehtinen <petri at digip.org>
date:        Fri Nov 04 21:25:46 2011 +0200
summary:
  Add .gitignore

files:
  .gitignore |  46 ++++++++++++++++++++++++++++++++++++++++++
  1 files changed, 46 insertions(+), 0 deletions(-)


diff --git a/.gitignore b/.gitignore
new file mode 100644
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,46 @@
+*.cover
+*.o
+*.orig
+*.pyc
+*.pyd
+*.pyo
+*.rej
+*~
+Doc/build/
+Doc/tools/docutils/
+Doc/tools/jinja2/
+Doc/tools/pygments/
+Doc/tools/sphinx/
+Lib/lib2to3/*.pickle
+Makefile
+Makefile.pre
+Misc/python.pc
+Modules/Setup
+Modules/Setup.config
+Modules/Setup.local
+Modules/config.c
+Modules/ld_so_aix
+PCbuild/*.bsc
+PCbuild/*.dll
+PCbuild/*.exe
+PCbuild/*.exp
+PCbuild/*.lib
+PCbuild/*.ncb
+PCbuild/*.o
+PCbuild/*.pdb
+PCbuild/Win32-temp-*
+Parser/pgen
+Parser/pgen.stamp
+autom4te.cache
+build/
+config.log
+config.status
+libpython*.a
+libpython*.so*
+pyconfig.h
+python
+python-gdb.py
+tags
+.coverage
+coverage/
+htmlcov/

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 22:06:07 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Fri, 04 Nov 2011 22:06:07 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_typos?=
Message-ID: <E1RMQxj-0001EE-Ng@dinsdale.python.org>

http://hg.python.org/cpython/rev/f069d923f586
changeset:   73364:f069d923f586
branch:      2.7
user:        Raymond Hettinger <python at rcn.com>
date:        Fri Nov 04 14:05:52 2011 -0700
summary:
  Fix typos

files:
  Doc/library/shelve.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/shelve.rst b/Doc/library/shelve.rst
--- a/Doc/library/shelve.rst
+++ b/Doc/library/shelve.rst
@@ -44,8 +44,8 @@
    determine which accessed entries are mutable, nor which ones were actually
    mutated).
 
-   Like file objects, shelve objects should closed explicitly to assure
-   that the peristent data is flushed to disk.
+   Like file objects, shelve objects should be closed explicitly to ensure
+   that the persistent data is flushed to disk.
 
    Since the :mod:`shelve` module stores objects using :mod:`pickle`, the same
    security precautions apply.  Accordingly, you should avoid loading a shelf

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 22:15:47 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 04 Nov 2011 22:15:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_test=5Furllib2_error_on?=
 =?utf8?q?_Windows_in_relation_with_issue_=2313287=2E?=
Message-ID: <E1RMR75-00025P-G3@dinsdale.python.org>

http://hg.python.org/cpython/rev/ca78ed7393bf
changeset:   73365:ca78ed7393bf
parent:      73362:cade039a7be4
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 04 22:15:37 2011 +0100
summary:
  Fix test_urllib2 error on Windows in relation with issue #13287.

files:
  Lib/test/test_urllib2.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py
--- a/Lib/test/test_urllib2.py
+++ b/Lib/test/test_urllib2.py
@@ -26,6 +26,10 @@
             context = {}
             exec('from urllib.%s import *' % module, context)
             del context['__builtins__']
+            if module == 'request' and os.name == 'nt':
+                u, p = context.pop('url2pathname'), context.pop('pathname2url')
+                self.assertEqual(u.__module__, 'nturl2path')
+                self.assertEqual(p.__module__, 'nturl2path')
             for k, v in context.items():
                 self.assertEqual(v.__module__, 'urllib.%s' % module,
                     "%r is exposed in 'urllib.%s' but defined in %r" %

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 22:29:48 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Fri, 04 Nov 2011 22:29:48 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzQz?=
 =?utf8?q?=3A_Fix_a_SystemError_when_a_lambda_expression_uses_a_global?=
Message-ID: <E1RMRKe-0003R8-BG@dinsdale.python.org>

http://hg.python.org/cpython/rev/1e0e821d2626
changeset:   73366:1e0e821d2626
branch:      3.2
parent:      73360:931ae170e51c
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Fri Nov 04 22:17:45 2011 +0100
summary:
  Issue #13343: Fix a SystemError when a lambda expression uses a global
variable in the default value of a keyword-only argument:
(lambda *, arg=GLOBAL_NAME: None)

files:
  Lib/test/test_keywordonlyarg.py |  8 ++++++++
  Misc/NEWS                       |  4 ++++
  Python/symtable.c               |  3 +++
  3 files changed, 15 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_keywordonlyarg.py b/Lib/test/test_keywordonlyarg.py
--- a/Lib/test/test_keywordonlyarg.py
+++ b/Lib/test/test_keywordonlyarg.py
@@ -162,6 +162,14 @@
         self.assertEqual(Example.f(Example(), k1=1, k2=2), (1, 2))
         self.assertRaises(TypeError, Example.f, k1=1, k2=2)
 
+    def test_issue13343(self):
+        # The Python compiler must scan all symbols of a function to
+        # determine their scope: global, local, cell...
+        # This was not done for the default values of keyword
+        # arguments in a lambda definition, and the following line
+        # used to fail with a SystemError.
+        lambda *, k1=unittest: None
+
 def test_main():
     run_unittest(KeywordOnlyArgTestCase)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13343: Fix a SystemError when a lambda expression uses a global
+  variable in the default value of a keyword-only argument:
+  (lambda *, arg=GLOBAL_NAME: None)
+
 - Issue #10519: Avoid unnecessary recursive function calls in
   setobject.c.
 
diff --git a/Python/symtable.c b/Python/symtable.c
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1334,6 +1334,9 @@
             return 0;
         if (e->v.Lambda.args->defaults)
             VISIT_SEQ(st, expr, e->v.Lambda.args->defaults);
+        if (e->v.Lambda.args->kw_defaults)
+            VISIT_KWONLYDEFAULTS(st,
+                                 e->v.Lambda.args->kw_defaults);
         if (!symtable_enter_block(st, lambda,
                                   FunctionBlock, (void *)e, e->lineno,
                                   e->col_offset))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 22:29:49 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Fri, 04 Nov 2011 22:29:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313343=3A_Fix_a_SystemError_when_a_lambda_expression?=
 =?utf8?q?_uses_a_global?=
Message-ID: <E1RMRKf-0003RF-7z@dinsdale.python.org>

http://hg.python.org/cpython/rev/bddb455439d0
changeset:   73367:bddb455439d0
parent:      73365:ca78ed7393bf
parent:      73366:1e0e821d2626
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Fri Nov 04 22:29:24 2011 +0100
summary:
  Issue #13343: Fix a SystemError when a lambda expression uses a global
variable in the default value of a keyword-only argument:
(lambda *, arg=GLOBAL_NAME: None)

files:
  Lib/test/test_keywordonlyarg.py |  8 ++++++++
  Misc/NEWS                       |  4 ++++
  Python/symtable.c               |  3 +++
  3 files changed, 15 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_keywordonlyarg.py b/Lib/test/test_keywordonlyarg.py
--- a/Lib/test/test_keywordonlyarg.py
+++ b/Lib/test/test_keywordonlyarg.py
@@ -162,6 +162,14 @@
         self.assertEqual(Example.f(Example(), k1=1, k2=2), (1, 2))
         self.assertRaises(TypeError, Example.f, k1=1, k2=2)
 
+    def test_issue13343(self):
+        # The Python compiler must scan all symbols of a function to
+        # determine their scope: global, local, cell...
+        # This was not done for the default values of keyword
+        # arguments in a lambda definition, and the following line
+        # used to fail with a SystemError.
+        lambda *, k1=unittest: None
+
 def test_main():
     run_unittest(KeywordOnlyArgTestCase)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13343: Fix a SystemError when a lambda expression uses a global
+  variable in the default value of a keyword-only argument:
+  (lambda *, arg=GLOBAL_NAME: None)
+
 - Issue #12797: Added custom opener parameter to builtin open() and
   FileIO.open().
 
diff --git a/Python/symtable.c b/Python/symtable.c
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1300,6 +1300,9 @@
             return 0;
         if (e->v.Lambda.args->defaults)
             VISIT_SEQ(st, expr, e->v.Lambda.args->defaults);
+        if (e->v.Lambda.args->kw_defaults)
+            VISIT_KWONLYDEFAULTS(st,
+                                 e->v.Lambda.args->kw_defaults);
         if (!symtable_enter_block(st, lambda,
                                   FunctionBlock, (void *)e, e->lineno,
                                   e->col_offset))

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov  4 22:34:35 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 04 Nov 2011 22:34:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Oops=2C_really_fix_gdb/libp?=
 =?utf8?q?ython=2Epy_for_not_ready_Unicode_strings?=
Message-ID: <E1RMRPH-0003tH-De@dinsdale.python.org>

http://hg.python.org/cpython/rev/694f70ea1419
changeset:   73368:694f70ea1419
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 04 22:34:01 2011 +0100
summary:
  Oops, really fix gdb/libpython.py for not ready Unicode strings

files:
  Tools/gdb/libpython.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py
--- a/Tools/gdb/libpython.py
+++ b/Tools/gdb/libpython.py
@@ -1130,12 +1130,13 @@
             ascii = compact['_base']
             state = ascii['state']
             is_compact_ascii = (int(state['ascii']) and int(state['compact']))
-            field_length = long(ascii['length'])
             if not int(state['ready']):
                 # string is not ready
+                field_length = long(compact['wstr_length'])
                 may_have_surrogates = True
                 field_str = ascii['wstr']
             else:
+                field_length = long(ascii['length'])
                 if is_compact_ascii:
                     field_str = ascii.address + 1
                 elif int(state['compact']):

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat Nov  5 05:31:01 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 05 Nov 2011 05:31:01 +0100
Subject: [Python-checkins] Daily reference leaks (694f70ea1419): sum=0
Message-ID: <E1RMXuH-0007Su-Kf@ap.vmr.nerim.net>

results for 694f70ea1419 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogE1cuK3', '-x']

From python-checkins at python.org  Sat Nov  5 08:53:37 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 08:53:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_Maildir_ini?=
 =?utf8?q?tialization_so_that_maildir_contents_are_read_correctly=2E?=
Message-ID: <E1RMb4L-0002xM-50@dinsdale.python.org>

http://hg.python.org/cpython/rev/5f27a9f67a34
changeset:   73369:5f27a9f67a34
branch:      3.2
parent:      73366:1e0e821d2626
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 09:44:59 2011 +0200
summary:
  Fix Maildir initialization so that maildir contents are read correctly.

Closes #13254.

files:
  Lib/mailbox.py           |   8 +++-----
  Lib/test/test_mailbox.py |  21 +++++++++++++++++++++
  Misc/NEWS                |   3 +++
  3 files changed, 27 insertions(+), 5 deletions(-)


diff --git a/Lib/mailbox.py b/Lib/mailbox.py
--- a/Lib/mailbox.py
+++ b/Lib/mailbox.py
@@ -273,11 +273,9 @@
             else:
                 raise NoSuchMailboxError(self._path)
         self._toc = {}
-        self._toc_mtimes = {}
-        for subdir in ('cur', 'new'):
-            self._toc_mtimes[subdir] = os.path.getmtime(self._paths[subdir])
-        self._last_read = time.time()  # Records last time we read cur/new
-        self._skewfactor = 0.1         # Adjust if os/fs clocks are skewing
+        self._toc_mtimes = {'cur': 0, 'new': 0}
+        self._last_read = 0         # Records last time we read cur/new
+        self._skewfactor = 0.1      # Adjust if os/fs clocks are skewing
 
     def add(self, message):
         """Add message and return assigned key."""
diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -801,6 +801,25 @@
                                           key1: os.path.join('new', key1),
                                           key2: os.path.join('new', key2)})
 
+    def test_refresh_after_safety_period(self):
+        # Issue #13254: Call _refresh after the "file system safety
+        # period" of 2 seconds has passed; _toc should still be
+        # updated because this is the first call to _refresh.
+        key0 = self._box.add(self._template % 0)
+        key1 = self._box.add(self._template % 1)
+
+        self._box = self._factory(self._path)
+        self.assertEqual(self._box._toc, {})
+
+        # Emulate sleeping. Instead of sleeping for 2 seconds, use the
+        # skew factor to make _refresh think that the filesystem
+        # safety period has passed and re-reading the _toc is only
+        # required if mtimes differ.
+        self._box._skewfactor = -2
+
+        self._box._refresh()
+        self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
+
     def test_lookup(self):
         # Look up message subpaths in the TOC
         self.assertRaises(KeyError, lambda: self._box._lookup('foo'))
@@ -876,6 +895,8 @@
         self.assertFalse((perms & 0o111)) # Execute bits should all be off.
 
     def test_reread(self):
+        # Do an initial unconditional refresh
+        self._box._refresh()
 
         # Put the last modified times more than two seconds into the past
         # (because mtime may have a two second granularity)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -70,6 +70,9 @@
 Library
 -------
 
+- Issue #13254: Fix Maildir initialization so that maildir contents
+  are read correctly.
+
 - Issue #3067: locale.setlocale() now raises TypeError if the second
   argument is an invalid iterable. Initial patch by Jyrki Pulliainen.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 08:53:37 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 08:53:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Fix_Maildir_initialization_so_that_maildir_contents_are_read?=
 =?utf8?q?_correctly=2E?=
Message-ID: <E1RMb4L-0002xQ-V3@dinsdale.python.org>

http://hg.python.org/cpython/rev/91a0f6879173
changeset:   73370:91a0f6879173
parent:      73368:694f70ea1419
parent:      73369:5f27a9f67a34
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 09:45:53 2011 +0200
summary:
  Fix Maildir initialization so that maildir contents are read correctly.

Closes #13254.

files:
  Lib/mailbox.py           |   8 +++-----
  Lib/test/test_mailbox.py |  21 +++++++++++++++++++++
  Misc/NEWS                |   3 +++
  3 files changed, 27 insertions(+), 5 deletions(-)


diff --git a/Lib/mailbox.py b/Lib/mailbox.py
--- a/Lib/mailbox.py
+++ b/Lib/mailbox.py
@@ -273,11 +273,9 @@
             else:
                 raise NoSuchMailboxError(self._path)
         self._toc = {}
-        self._toc_mtimes = {}
-        for subdir in ('cur', 'new'):
-            self._toc_mtimes[subdir] = os.path.getmtime(self._paths[subdir])
-        self._last_read = time.time()  # Records last time we read cur/new
-        self._skewfactor = 0.1         # Adjust if os/fs clocks are skewing
+        self._toc_mtimes = {'cur': 0, 'new': 0}
+        self._last_read = 0         # Records last time we read cur/new
+        self._skewfactor = 0.1      # Adjust if os/fs clocks are skewing
 
     def add(self, message):
         """Add message and return assigned key."""
diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -801,6 +801,25 @@
                                           key1: os.path.join('new', key1),
                                           key2: os.path.join('new', key2)})
 
+    def test_refresh_after_safety_period(self):
+        # Issue #13254: Call _refresh after the "file system safety
+        # period" of 2 seconds has passed; _toc should still be
+        # updated because this is the first call to _refresh.
+        key0 = self._box.add(self._template % 0)
+        key1 = self._box.add(self._template % 1)
+
+        self._box = self._factory(self._path)
+        self.assertEqual(self._box._toc, {})
+
+        # Emulate sleeping. Instead of sleeping for 2 seconds, use the
+        # skew factor to make _refresh think that the filesystem
+        # safety period has passed and re-reading the _toc is only
+        # required if mtimes differ.
+        self._box._skewfactor = -2
+
+        self._box._refresh()
+        self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
+
     def test_lookup(self):
         # Look up message subpaths in the TOC
         self.assertRaises(KeyError, lambda: self._box._lookup('foo'))
@@ -876,6 +895,8 @@
         self.assertFalse((perms & 0o111)) # Execute bits should all be off.
 
     def test_reread(self):
+        # Do an initial unconditional refresh
+        self._box._refresh()
 
         # Put the last modified times more than two seconds into the past
         # (because mtime may have a two second granularity)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -354,6 +354,9 @@
 Library
 -------
 
+- Issue #13254: Fix Maildir initialization so that maildir contents
+  are read correctly.
+
 - Issue #3067: locale.setlocale() now raises TypeError if the second
   argument is an invalid iterable. Initial patch by Jyrki Pulliainen.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 08:53:38 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 08:53:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_Maildir_ini?=
 =?utf8?q?tialization_so_that_maildir_contents__are_read_correctly=2E?=
Message-ID: <E1RMb4M-0002xU-OI@dinsdale.python.org>

http://hg.python.org/cpython/rev/0b754ee12dbd
changeset:   73371:0b754ee12dbd
branch:      2.7
parent:      73364:f069d923f586
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 09:50:37 2011 +0200
summary:
  Fix Maildir initialization so that maildir contents  are read correctly.

Closes #13254.

files:
  Lib/mailbox.py           |   8 +++-----
  Lib/test/test_mailbox.py |  21 +++++++++++++++++++++
  Misc/NEWS                |   3 +++
  3 files changed, 27 insertions(+), 5 deletions(-)


diff --git a/Lib/mailbox.py b/Lib/mailbox.py
--- a/Lib/mailbox.py
+++ b/Lib/mailbox.py
@@ -247,11 +247,9 @@
             else:
                 raise NoSuchMailboxError(self._path)
         self._toc = {}
-        self._toc_mtimes = {}
-        for subdir in ('cur', 'new'):
-            self._toc_mtimes[subdir] = os.path.getmtime(self._paths[subdir])
-        self._last_read = time.time()  # Records last time we read cur/new
-        self._skewfactor = 0.1         # Adjust if os/fs clocks are skewing
+        self._toc_mtimes = {'cur': 0, 'new': 0}
+        self._last_read = 0         # Records last time we read cur/new
+        self._skewfactor = 0.1      # Adjust if os/fs clocks are skewing
 
     def add(self, message):
         """Add message and return assigned key."""
diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -683,6 +683,25 @@
                                           key1: os.path.join('new', key1),
                                           key2: os.path.join('new', key2)})
 
+    def test_refresh_after_safety_period(self):
+        # Issue #13254: Call _refresh after the "file system safety
+        # period" of 2 seconds has passed; _toc should still be
+        # updated because this is the first call to _refresh.
+        key0 = self._box.add(self._template % 0)
+        key1 = self._box.add(self._template % 1)
+
+        self._box = self._factory(self._path)
+        self.assertEqual(self._box._toc, {})
+
+        # Emulate sleeping. Instead of sleeping for 2 seconds, use the
+        # skew factor to make _refresh think that the filesystem
+        # safety period has passed and re-reading the _toc is only
+        # required if mtimes differ.
+        self._box._skewfactor = -2
+
+        self._box._refresh()
+        self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
+
     def test_lookup(self):
         # Look up message subpaths in the TOC
         self.assertRaises(KeyError, lambda: self._box._lookup('foo'))
@@ -758,6 +777,8 @@
         self.assertFalse((perms & 0111)) # Execute bits should all be off.
 
     def test_reread(self):
+        # Do an initial unconditional refresh
+        self._box._refresh()
 
         # Put the last modified times more than two seconds into the past
         # (because mtime may have only a two second granularity).
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -74,6 +74,9 @@
 Library
 -------
 
+- Issue #13254: Fix Maildir initialization so that maildir contents
+  are read correctly.
+
 - Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
 
 - Issue #2892: preserve iterparse events in case of SyntaxError.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 09:25:06 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 09:25:06 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzMwNjc6?=
 =?utf8?q?_Enhance_the_documentation_and_docstring_of_locale=2Esetlocale?=
 =?utf8?b?KCk=?=
Message-ID: <E1RMbYo-0004xp-M9@dinsdale.python.org>

http://hg.python.org/cpython/rev/34c9465f5023
changeset:   73372:34c9465f5023
branch:      2.7
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 10:18:50 2011 +0200
summary:
  Issue #3067: Enhance the documentation and docstring of locale.setlocale()

files:
  Doc/library/locale.rst |  20 ++++++++++----------
  Lib/locale.py          |   5 +++--
  Misc/NEWS              |   3 +++
  3 files changed, 16 insertions(+), 12 deletions(-)


diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst
--- a/Doc/library/locale.rst
+++ b/Doc/library/locale.rst
@@ -23,19 +23,19 @@
 
 .. exception:: Error
 
-   Exception raised when :func:`setlocale` fails.
+   Exception raised when the locale passed to :func:`setlocale` is not
+   recognized.
 
 
 .. function:: setlocale(category[, locale])
 
-   If *locale* is specified, it may be a string, a tuple of the form ``(language
-   code, encoding)``, or ``None``. If it is a tuple, it is converted to a string
-   using the locale aliasing engine.  If *locale* is given and not ``None``,
-   :func:`setlocale` modifies the locale setting for the *category*.  The available
-   categories are listed in the data description below.  The value is the name of a
-   locale.  An empty string specifies the user's default settings. If the
-   modification of the locale fails, the exception :exc:`Error` is raised.  If
-   successful, the new locale setting is returned.
+   If *locale* is given and not ``None``, :func:`setlocale` modifies the locale
+   setting for the *category*. The available categories are listed in the data
+   description below. *locale* may be a string, or an iterable of two strings
+   (language code and encoding). If it's an iterable, it's converted to a locale
+   name using the locale aliasing engine. An empty string specifies the user's
+   default settings. If the modification of the locale fails, the exception
+   :exc:`Error` is raised. If successful, the new locale setting is returned.
 
    If *locale* is omitted or ``None``, the current setting for *category* is
    returned.
@@ -51,7 +51,7 @@
    changed thereafter, using multithreading should not cause problems.
 
    .. versionchanged:: 2.0
-      Added support for tuple values of the *locale* parameter.
+      Added support for iterable values of the *locale* parameter.
 
 
 .. function:: localeconv()
diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -524,9 +524,10 @@
 def setlocale(category, locale=None):
 
     """ Set the locale for the given category.  The locale can be
-        a string, a locale tuple (language code, encoding), or None.
+        a string, an iterable of two strings (language code and encoding),
+        or None.
 
-        Locale tuples are converted to strings the locale aliasing
+        Iterables are converted to strings using the locale aliasing
         engine.  Locale strings are passed directly to the C lib.
 
         category may be given as one of the LC_* values.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -74,6 +74,9 @@
 Library
 -------
 
+- Issue #3067: Enhance the documentation and docstring of
+  locale.setlocale().
+
 - Issue #13254: Fix Maildir initialization so that maildir contents
   are read correctly.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 09:25:07 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 09:25:07 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzMwNjc6?=
 =?utf8?q?_Enhance_the_documentation_and_docstring_of_locale=2Esetlocale?=
 =?utf8?b?KCk=?=
Message-ID: <E1RMbYp-0004yP-FZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/98806dd03506
changeset:   73373:98806dd03506
branch:      3.2
parent:      73369:5f27a9f67a34
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 10:18:50 2011 +0200
summary:
  Issue #3067: Enhance the documentation and docstring of locale.setlocale()

files:
  Doc/library/locale.rst |  18 +++++++++---------
  Lib/locale.py          |   5 +++--
  Misc/NEWS              |   3 ++-
  3 files changed, 14 insertions(+), 12 deletions(-)


diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst
--- a/Doc/library/locale.rst
+++ b/Doc/library/locale.rst
@@ -22,19 +22,19 @@
 
 .. exception:: Error
 
-   Exception raised when :func:`setlocale` fails.
+   Exception raised when the locale passed to :func:`setlocale` is not
+   recognized.
 
 
 .. function:: setlocale(category, locale=None)
 
-   If *locale* is specified, it may be a string, a tuple of the form ``(language
-   code, encoding)``, or ``None``. If it is a tuple, it is converted to a string
-   using the locale aliasing engine.  If *locale* is given and not ``None``,
-   :func:`setlocale` modifies the locale setting for the *category*.  The available
-   categories are listed in the data description below.  The value is the name of a
-   locale.  An empty string specifies the user's default settings. If the
-   modification of the locale fails, the exception :exc:`Error` is raised.  If
-   successful, the new locale setting is returned.
+   If *locale* is given and not ``None``, :func:`setlocale` modifies the locale
+   setting for the *category*. The available categories are listed in the data
+   description below. *locale* may be a string, or an iterable of two strings
+   (language code and encoding). If it's an iterable, it's converted to a locale
+   name using the locale aliasing engine. An empty string specifies the user's
+   default settings. If the modification of the locale fails, the exception
+   :exc:`Error` is raised. If successful, the new locale setting is returned.
 
    If *locale* is omitted or ``None``, the current setting for *category* is
    returned.
diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -526,9 +526,10 @@
 def setlocale(category, locale=None):
 
     """ Set the locale for the given category.  The locale can be
-        a string, a locale tuple (language code, encoding), or None.
+        a string, an iterable of two strings (language code and encoding),
+        or None.
 
-        Locale tuples are converted to strings the locale aliasing
+        Iterables are converted to strings using the locale aliasing
         engine.  Locale strings are passed directly to the C lib.
 
         category may be given as one of the LC_* values.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -74,7 +74,8 @@
   are read correctly.
 
 - Issue #3067: locale.setlocale() now raises TypeError if the second
-  argument is an invalid iterable. Initial patch by Jyrki Pulliainen.
+  argument is an invalid iterable. Its documentation and docstring
+  were also updated. Initial patch by Jyrki Pulliainen.
 
 - Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 09:25:08 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 09:25:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=233067=3A_Enhance_the_documentation_and_docstring_of_?=
 =?utf8?q?locale=2Esetlocale=28=29?=
Message-ID: <E1RMbYq-0004yT-9c@dinsdale.python.org>

http://hg.python.org/cpython/rev/8a27920efffe
changeset:   73374:8a27920efffe
parent:      73370:91a0f6879173
parent:      73373:98806dd03506
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 10:21:49 2011 +0200
summary:
  Issue #3067: Enhance the documentation and docstring of locale.setlocale()

files:
  Doc/library/locale.rst |  18 +++++++++---------
  Lib/locale.py          |   5 +++--
  Misc/NEWS              |   3 ++-
  3 files changed, 14 insertions(+), 12 deletions(-)


diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst
--- a/Doc/library/locale.rst
+++ b/Doc/library/locale.rst
@@ -22,19 +22,19 @@
 
 .. exception:: Error
 
-   Exception raised when :func:`setlocale` fails.
+   Exception raised when the locale passed to :func:`setlocale` is not
+   recognized.
 
 
 .. function:: setlocale(category, locale=None)
 
-   If *locale* is specified, it may be a string, a tuple of the form ``(language
-   code, encoding)``, or ``None``. If it is a tuple, it is converted to a string
-   using the locale aliasing engine.  If *locale* is given and not ``None``,
-   :func:`setlocale` modifies the locale setting for the *category*.  The available
-   categories are listed in the data description below.  The value is the name of a
-   locale.  An empty string specifies the user's default settings. If the
-   modification of the locale fails, the exception :exc:`Error` is raised.  If
-   successful, the new locale setting is returned.
+   If *locale* is given and not ``None``, :func:`setlocale` modifies the locale
+   setting for the *category*. The available categories are listed in the data
+   description below. *locale* may be a string, or an iterable of two strings
+   (language code and encoding). If it's an iterable, it's converted to a locale
+   name using the locale aliasing engine. An empty string specifies the user's
+   default settings. If the modification of the locale fails, the exception
+   :exc:`Error` is raised. If successful, the new locale setting is returned.
 
    If *locale* is omitted or ``None``, the current setting for *category* is
    returned.
diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -526,9 +526,10 @@
 def setlocale(category, locale=None):
 
     """ Set the locale for the given category.  The locale can be
-        a string, a locale tuple (language code, encoding), or None.
+        a string, an iterable of two strings (language code and encoding),
+        or None.
 
-        Locale tuples are converted to strings the locale aliasing
+        Iterables are converted to strings using the locale aliasing
         engine.  Locale strings are passed directly to the C lib.
 
         category may be given as one of the LC_* values.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -358,7 +358,8 @@
   are read correctly.
 
 - Issue #3067: locale.setlocale() now raises TypeError if the second
-  argument is an invalid iterable. Initial patch by Jyrki Pulliainen.
+  argument is an invalid iterable. Its documentation and docstring
+  were also updated. Initial patch by Jyrki Pulliainen.
 
 - Issue #13140: Fix the daemon_threads attribute of ThreadingMixIn.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 14:16:49 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sat, 05 Nov 2011 14:16:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_asyncore=3A_POLLERR=2C_POLL?=
 =?utf8?q?HUP_and_POLLNVAL_are_ignored_when_passed_as_input_flag?=
Message-ID: <E1RMg77-0006Dg-TC@dinsdale.python.org>

http://hg.python.org/cpython/rev/5b6d7d90410e
changeset:   73375:5b6d7d90410e
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sat Nov 05 14:16:01 2011 +0100
summary:
  asyncore: POLLERR, POLLHUP and POLLNVAL are ignored when passed as input flag
to poll(2): don't set them.

files:
  Lib/asyncore.py |  3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)


diff --git a/Lib/asyncore.py b/Lib/asyncore.py
--- a/Lib/asyncore.py
+++ b/Lib/asyncore.py
@@ -181,9 +181,6 @@
             if obj.writable() and not obj.accepting:
                 flags |= select.POLLOUT
             if flags:
-                # Only check for exceptions if object was either readable
-                # or writable.
-                flags |= select.POLLERR | select.POLLHUP | select.POLLNVAL
                 pollster.register(fd, flags)
         try:
             r = pollster.poll(timeout)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 17:52:03 2011
From: python-checkins at python.org (eric.araujo)
Date: Sat, 05 Nov 2011 17:52:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Try_to_fix_buildbot_failure?=
	=?utf8?q?s_from_=2313193?=
Message-ID: <E1RMjTP-0004ha-DQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/4eee9dd61147
changeset:   73376:4eee9dd61147
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 05 17:51:52 2011 +0100
summary:
  Try to fix buildbot failures from #13193

files:
  Lib/packaging/tests/test_command_install_data.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/packaging/tests/test_command_install_data.py b/Lib/packaging/tests/test_command_install_data.py
--- a/Lib/packaging/tests/test_command_install_data.py
+++ b/Lib/packaging/tests/test_command_install_data.py
@@ -99,7 +99,9 @@
         self.write_file('spamd', '# Python script')
         sysconfig._SCHEMES.set(_get_default_scheme(), 'scripts', scripts_dir)
         sys.path.insert(0, install_dir)
+        packaging.database.disable_cache()
         self.addCleanup(sys.path.remove, install_dir)
+        self.addCleanup(packaging.database.enable_cache)
 
         cmd = install_dist(dist)
         cmd.outputs = ['spamd']

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 17:55:11 2011
From: python-checkins at python.org (eric.araujo)
Date: Sat, 05 Nov 2011 17:55:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Add_missing_ver?=
 =?utf8?q?sionadded_=28fixes_=2312392=29?=
Message-ID: <E1RMjWR-0004xg-Py@dinsdale.python.org>

http://hg.python.org/cpython/rev/8ea34a74f118
changeset:   73377:8ea34a74f118
branch:      2.7
parent:      73372:34c9465f5023
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 05 17:55:03 2011 +0100
summary:
  Add missing versionadded (fixes #12392)

files:
  Doc/library/functions.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -123,6 +123,8 @@
 
    Without an argument, an array of size 0 is created.
 
+   .. versionadded:: 2.6
+
 
 .. function:: callable(object)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 20:19:01 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Sat, 05 Nov 2011 20:19:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_add_introspection_to_range_?=
 =?utf8?q?objects_=28closes_=239896=29?=
Message-ID: <E1RMlld-0003Rs-BZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/4643be424293
changeset:   73378:4643be424293
parent:      73376:4eee9dd61147
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat Nov 05 15:17:52 2011 -0400
summary:
  add introspection to range objects (closes #9896)

Patch by Daniel Urban.

files:
  Doc/library/functions.rst |   7 +++++-
  Lib/test/test_range.py    |  29 +++++++++++++++++++++++++++
  Objects/rangeobject.c     |  10 ++++++++-
  3 files changed, 44 insertions(+), 2 deletions(-)


diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1042,7 +1042,9 @@
    ...]``.  If *step* is positive, the last element is the largest ``start + i *
    step`` less than *stop*; if *step* is negative, the last element is the
    smallest ``start + i * step`` greater than *stop*.  *step* must not be zero
-   (or else :exc:`ValueError` is raised).  Example:
+   (or else :exc:`ValueError` is raised).  Range objects have read-only data
+   attributes :attr:`start`, :attr:`stop` and :attr:`step` which return the
+   argument values (or their default).  Example:
 
       >>> list(range(10))
       [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
@@ -1100,6 +1102,9 @@
       sequence of values they define (instead of comparing based on
       object identity).
 
+   .. versionadded:: 3.3
+      The :attr:`start`, :attr:`stop` and :attr:`step` attributes.
+
 
 .. function:: repr(object)
 
diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py
--- a/Lib/test/test_range.py
+++ b/Lib/test/test_range.py
@@ -560,6 +560,35 @@
             range(0) >= range(0)
 
 
+    def test_attributes(self):
+        # test the start, stop and step attributes of range objects
+        self.assert_attrs(range(0), 0, 0, 1)
+        self.assert_attrs(range(10), 0, 10, 1)
+        self.assert_attrs(range(-10), 0, -10, 1)
+        self.assert_attrs(range(0, 10, 1), 0, 10, 1)
+        self.assert_attrs(range(0, 10, 3), 0, 10, 3)
+        self.assert_attrs(range(10, 0, -1), 10, 0, -1)
+        self.assert_attrs(range(10, 0, -3), 10, 0, -3)
+
+    def assert_attrs(self, rangeobj, start, stop, step):
+        self.assertEqual(rangeobj.start, start)
+        self.assertEqual(rangeobj.stop, stop)
+        self.assertEqual(rangeobj.step, step)
+
+        with self.assertRaises(AttributeError):
+            rangeobj.start = 0
+        with self.assertRaises(AttributeError):
+            rangeobj.stop = 10
+        with self.assertRaises(AttributeError):
+            rangeobj.step = 1
+
+        with self.assertRaises(AttributeError):
+            del rangeobj.start
+        with self.assertRaises(AttributeError):
+            del rangeobj.stop
+        with self.assertRaises(AttributeError):
+            del rangeobj.step
+
 def test_main():
     test.support.run_unittest(RangeTest)
 
diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c
--- a/Objects/rangeobject.c
+++ b/Objects/rangeobject.c
@@ -1,6 +1,7 @@
 /* Range object implementation */
 
 #include "Python.h"
+#include "structmember.h"
 
 /* Support objects whose length is > PY_SSIZE_T_MAX.
 
@@ -880,6 +881,13 @@
     {NULL,              NULL}           /* sentinel */
 };
 
+static PyMemberDef range_members[] = {
+    {"start",   T_OBJECT_EX,    offsetof(rangeobject, start),   READONLY},
+    {"stop",    T_OBJECT_EX,    offsetof(rangeobject, stop),    READONLY},
+    {"step",    T_OBJECT_EX,    offsetof(rangeobject, step),    READONLY},
+    {0}
+};
+
 PyTypeObject PyRange_Type = {
         PyVarObject_HEAD_INIT(&PyType_Type, 0)
         "range",                /* Name of this type */
@@ -909,7 +917,7 @@
         range_iter,             /* tp_iter */
         0,                      /* tp_iternext */
         range_methods,          /* tp_methods */
-        0,                      /* tp_members */
+        range_members,          /* tp_members */
         0,                      /* tp_getset */
         0,                      /* tp_base */
         0,                      /* tp_dict */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 20:19:02 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Sat, 05 Nov 2011 20:19:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_news_note_about_range_intro?=
	=?utf8?q?spection?=
Message-ID: <E1RMlle-0003Rz-5G@dinsdale.python.org>

http://hg.python.org/cpython/rev/91d59ccf1e4d
changeset:   73379:91d59ccf1e4d
user:        Benjamin Peterson <benjamin at python.org>
date:        Sat Nov 05 15:18:51 2011 -0400
summary:
  news note about range introspection

files:
  Misc/NEWS |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,8 @@
 Core and Builtins
 -----------------
 
+- Issue #9896: Add start, stop, and step attributes to range objects.
+
 - Issue #13343: Fix a SystemError when a lambda expression uses a global
   variable in the default value of a keyword-only argument:
   (lambda *, arg=GLOBAL_NAME: None)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:02:31 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 21:02:31 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogUmVtb3ZlIF9fcHlj?=
 =?utf8?q?ache=5F=5F_directories_correctly_on_OpenBSD?=
Message-ID: <E1RMmRj-0006nP-Tl@dinsdale.python.org>

http://hg.python.org/cpython/rev/41ab1dfaf1d4
changeset:   73380:41ab1dfaf1d4
branch:      3.2
parent:      73373:98806dd03506
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 21:04:24 2011 +0200
summary:
  Remove __pycache__ directories correctly on OpenBSD

The find utility of OpenBSD doesn't support the "-exec cmd {} +"
action. Change it to use "-depth -exec cmd {} ;" instead.

Also, remove __pycache__ before *.py[co], as most bytecode files are
in __pycache__ directories. Only those generated by Python 2 under
Doc/tools are not.

Closes #13326.

files:
  Makefile.pre.in |  2 +-
  Misc/NEWS       |  5 +++++
  2 files changed, 6 insertions(+), 1 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1231,8 +1231,8 @@
 # Sanitation targets -- clean leaves libraries, executables and tags
 # files, which clobber removes as well
 pycremoval:
+	-find $(srcdir) -depth -name '__pycache__' -exec rm -rf {} ';'
 	-find $(srcdir) -name '*.py[co]' -exec rm -f {} ';'
-	-find $(srcdir) -name '__pycache__' -exec rmdir {} '+'
 
 rmtestturds:
 	-rm -f *BAD *GOOD *SKIPPED
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -208,6 +208,11 @@
 - Issue #10860: http.client now correctly handles an empty port after port
   delimiter in URLs.
 
+Build
+-----
+
+- Issue #13326: Clean __pycache__ directories correctly on OpenBSD.
+
 Tests
 -----
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:02:32 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 21:02:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Remove_=5F=5Fpycache=5F=5F_directories_correctly_on_OpenBSD?=
Message-ID: <E1RMmRk-0006nT-My@dinsdale.python.org>

http://hg.python.org/cpython/rev/f853a2cbd68b
changeset:   73381:f853a2cbd68b
parent:      73376:4eee9dd61147
parent:      73380:41ab1dfaf1d4
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 21:23:48 2011 +0200
summary:
  Remove __pycache__ directories correctly on OpenBSD

Closes #13326.

files:
  Makefile.pre.in |  2 +-
  Misc/NEWS       |  2 ++
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -1285,8 +1285,8 @@
 # Sanitation targets -- clean leaves libraries, executables and tags
 # files, which clobber removes as well
 pycremoval:
+	-find $(srcdir) -depth -name '__pycache__' -exec rm -rf {} ';'
 	-find $(srcdir) -name '*.py[co]' -exec rm -f {} ';'
-	-find $(srcdir) -name '__pycache__' -exec rmdir {} '+'
 
 rmtestturds:
 	-rm -f *BAD *GOOD *SKIPPED
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1445,6 +1445,8 @@
 Build
 -----
 
+- Issue #13326: Clean __pycache__ directories correctly on OpenBSD.
+
 - PEP 393: the configure option --with-wide-unicode is removed.
 
 - Issue #12852: Set _XOPEN_SOURCE to 700, instead of 600, to get POSIX 2008

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:02:36 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 21:02:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge_heads?=
Message-ID: <E1RMmRo-0006na-1F@dinsdale.python.org>

http://hg.python.org/cpython/rev/3f22e4f4c761
changeset:   73382:3f22e4f4c761
parent:      73381:f853a2cbd68b
parent:      73379:91d59ccf1e4d
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 22:00:14 2011 +0200
summary:
  Merge heads

files:
  Doc/library/functions.rst |   7 +++++-
  Lib/test/test_range.py    |  29 +++++++++++++++++++++++++++
  Misc/NEWS                 |   2 +
  Objects/rangeobject.c     |  10 ++++++++-
  4 files changed, 46 insertions(+), 2 deletions(-)


diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1042,7 +1042,9 @@
    ...]``.  If *step* is positive, the last element is the largest ``start + i *
    step`` less than *stop*; if *step* is negative, the last element is the
    smallest ``start + i * step`` greater than *stop*.  *step* must not be zero
-   (or else :exc:`ValueError` is raised).  Example:
+   (or else :exc:`ValueError` is raised).  Range objects have read-only data
+   attributes :attr:`start`, :attr:`stop` and :attr:`step` which return the
+   argument values (or their default).  Example:
 
       >>> list(range(10))
       [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
@@ -1100,6 +1102,9 @@
       sequence of values they define (instead of comparing based on
       object identity).
 
+   .. versionadded:: 3.3
+      The :attr:`start`, :attr:`stop` and :attr:`step` attributes.
+
 
 .. function:: repr(object)
 
diff --git a/Lib/test/test_range.py b/Lib/test/test_range.py
--- a/Lib/test/test_range.py
+++ b/Lib/test/test_range.py
@@ -560,6 +560,35 @@
             range(0) >= range(0)
 
 
+    def test_attributes(self):
+        # test the start, stop and step attributes of range objects
+        self.assert_attrs(range(0), 0, 0, 1)
+        self.assert_attrs(range(10), 0, 10, 1)
+        self.assert_attrs(range(-10), 0, -10, 1)
+        self.assert_attrs(range(0, 10, 1), 0, 10, 1)
+        self.assert_attrs(range(0, 10, 3), 0, 10, 3)
+        self.assert_attrs(range(10, 0, -1), 10, 0, -1)
+        self.assert_attrs(range(10, 0, -3), 10, 0, -3)
+
+    def assert_attrs(self, rangeobj, start, stop, step):
+        self.assertEqual(rangeobj.start, start)
+        self.assertEqual(rangeobj.stop, stop)
+        self.assertEqual(rangeobj.step, step)
+
+        with self.assertRaises(AttributeError):
+            rangeobj.start = 0
+        with self.assertRaises(AttributeError):
+            rangeobj.stop = 10
+        with self.assertRaises(AttributeError):
+            rangeobj.step = 1
+
+        with self.assertRaises(AttributeError):
+            del rangeobj.start
+        with self.assertRaises(AttributeError):
+            del rangeobj.stop
+        with self.assertRaises(AttributeError):
+            del rangeobj.step
+
 def test_main():
     test.support.run_unittest(RangeTest)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,8 @@
 Core and Builtins
 -----------------
 
+- Issue #9896: Add start, stop, and step attributes to range objects.
+
 - Issue #13343: Fix a SystemError when a lambda expression uses a global
   variable in the default value of a keyword-only argument:
   (lambda *, arg=GLOBAL_NAME: None)
diff --git a/Objects/rangeobject.c b/Objects/rangeobject.c
--- a/Objects/rangeobject.c
+++ b/Objects/rangeobject.c
@@ -1,6 +1,7 @@
 /* Range object implementation */
 
 #include "Python.h"
+#include "structmember.h"
 
 /* Support objects whose length is > PY_SSIZE_T_MAX.
 
@@ -880,6 +881,13 @@
     {NULL,              NULL}           /* sentinel */
 };
 
+static PyMemberDef range_members[] = {
+    {"start",   T_OBJECT_EX,    offsetof(rangeobject, start),   READONLY},
+    {"stop",    T_OBJECT_EX,    offsetof(rangeobject, stop),    READONLY},
+    {"step",    T_OBJECT_EX,    offsetof(rangeobject, step),    READONLY},
+    {0}
+};
+
 PyTypeObject PyRange_Type = {
         PyVarObject_HEAD_INIT(&PyType_Type, 0)
         "range",                /* Name of this type */
@@ -909,7 +917,7 @@
         range_iter,             /* tp_iter */
         0,                      /* tp_iternext */
         range_methods,          /* tp_methods */
-        0,                      /* tp_members */
+        range_members,          /* tp_members */
         0,                      /* tp_getset */
         0,                      /* tp_base */
         0,                      /* tp_dict */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:13:02 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 21:13:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Update_=2Egitig?=
	=?utf8?q?nore?=
Message-ID: <E1RMmbu-0007Zi-00@dinsdale.python.org>

http://hg.python.org/cpython/rev/dc839da60f13
changeset:   73383:dc839da60f13
branch:      3.2
parent:      73380:41ab1dfaf1d4
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 22:06:30 2011 +0200
summary:
  Update .gitignore

files:
  .gitignore |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -20,6 +20,7 @@
 Modules/Setup.local
 Modules/config.c
 Modules/ld_so_aix
+Modules/_testembed
 PCbuild/*.bsc
 PCbuild/*.dll
 PCbuild/*.exe

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:13:02 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 21:13:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1RMmbu-0007Zs-TN@dinsdale.python.org>

http://hg.python.org/cpython/rev/bf8a0cf32305
changeset:   73384:bf8a0cf32305
parent:      73382:3f22e4f4c761
parent:      73383:dc839da60f13
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 22:06:49 2011 +0200
summary:
  Merge 3.2

files:
  .gitignore |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -20,6 +20,7 @@
 Modules/Setup.local
 Modules/config.c
 Modules/ld_so_aix
+Modules/_testembed
 PCbuild/*.bsc
 PCbuild/*.dll
 PCbuild/*.exe

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:13:03 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 21:13:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_=2Egitignore?=
Message-ID: <E1RMmbv-0007Zw-Gi@dinsdale.python.org>

http://hg.python.org/cpython/rev/733e1074e9cd
changeset:   73385:733e1074e9cd
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 22:07:16 2011 +0200
summary:
  Update .gitignore

files:
  .gitignore |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -12,6 +12,7 @@
 Doc/tools/pygments/
 Doc/tools/sphinx/
 Lib/lib2to3/*.pickle
+Lib/_sysconfigdata.py
 Makefile
 Makefile.pre
 Misc/python.pc

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:43:36 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Sat, 05 Nov 2011 21:43:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Improve_Counter?=
 =?utf8?q?=2E=5F=5Frepr=5F=5F=28=29_to_not_fail_with_unorderable_values?=
Message-ID: <E1RMn5U-0001Jc-36@dinsdale.python.org>

http://hg.python.org/cpython/rev/970ec22ab368
changeset:   73386:970ec22ab368
branch:      3.2
parent:      73383:dc839da60f13
user:        Raymond Hettinger <python at rcn.com>
date:        Sat Nov 05 13:35:26 2011 -0700
summary:
  Improve Counter.__repr__() to not fail with unorderable values

files:
  Lib/collections.py           |  8 ++++++--
  Lib/test/test_collections.py |  6 ++++++
  2 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Lib/collections.py b/Lib/collections.py
--- a/Lib/collections.py
+++ b/Lib/collections.py
@@ -583,8 +583,12 @@
     def __repr__(self):
         if not self:
             return '%s()' % self.__class__.__name__
-        items = ', '.join(map('%r: %r'.__mod__, self.most_common()))
-        return '%s({%s})' % (self.__class__.__name__, items)
+        try:
+            items = ', '.join(map('%r: %r'.__mod__, self.most_common()))
+            return '%s({%s})' % (self.__class__.__name__, items)
+        except TypeError:
+            # handle case where values are not orderable
+            return '{0}({1!r})'.format(self.__class__.__name__, dict(self))
 
     # Multiset-style mathematical operations discussed in:
     #       Knuth TAOCP Volume II section 4.6.3 exercise 19
diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -893,6 +893,12 @@
         c.subtract('aaaabbcce')
         self.assertEqual(c, Counter(a=-1, b=0, c=-1, d=1, e=-1))
 
+    def test_repr_nonsortable(self):
+        c = Counter(a=2, b=None)
+        r = repr(c)
+        self.assertIn("'a': 2", r)
+        self.assertIn("'b': None", r)
+
     def test_helper_function(self):
         # two paths, one for real dicts and one for other mappings
         elems = list('abracadabra')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:43:36 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Sat, 05 Nov 2011 21:43:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge?=
Message-ID: <E1RMn5U-0001Jm-SN@dinsdale.python.org>

http://hg.python.org/cpython/rev/7144da4c2d26
changeset:   73387:7144da4c2d26
parent:      73385:733e1074e9cd
parent:      73386:970ec22ab368
user:        Raymond Hettinger <python at rcn.com>
date:        Sat Nov 05 13:39:57 2011 -0700
summary:
  Merge

files:
  Lib/collections/__init__.py  |  8 ++++++--
  Lib/test/test_collections.py |  6 ++++++
  2 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -585,8 +585,12 @@
     def __repr__(self):
         if not self:
             return '%s()' % self.__class__.__name__
-        items = ', '.join(map('%r: %r'.__mod__, self.most_common()))
-        return '%s({%s})' % (self.__class__.__name__, items)
+        try:
+            items = ', '.join(map('%r: %r'.__mod__, self.most_common()))
+            return '%s({%s})' % (self.__class__.__name__, items)
+        except TypeError:
+            # handle case where values are not orderable
+            return '{0}({1!r})'.format(self.__class__.__name__, dict(self))
 
     # Multiset-style mathematical operations discussed in:
     #       Knuth TAOCP Volume II section 4.6.3 exercise 19
diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -969,6 +969,12 @@
         self.assertEqual(dict(+c), dict(c=5, d=10, e=15, g=40))
         self.assertEqual(dict(-c), dict(a=5))
 
+    def test_repr_nonsortable(self):
+        c = Counter(a=2, b=None)
+        r = repr(c)
+        self.assertIn("'a': 2", r)
+        self.assertIn("'b': None", r)
+    
     def test_helper_function(self):
         # two paths, one for real dicts and one for other mappings
         elems = list('abracadabra')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 21:43:42 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Sat, 05 Nov 2011 21:43:42 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_whitespace?=
Message-ID: <E1RMn5a-0001Jw-G8@dinsdale.python.org>

http://hg.python.org/cpython/rev/89c7ba01a3d2
changeset:   73388:89c7ba01a3d2
user:        Raymond Hettinger <python at rcn.com>
date:        Sat Nov 05 13:43:01 2011 -0700
summary:
  Fix whitespace

files:
  Lib/test/test_collections.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py
--- a/Lib/test/test_collections.py
+++ b/Lib/test/test_collections.py
@@ -974,7 +974,7 @@
         r = repr(c)
         self.assertIn("'a': 2", r)
         self.assertIn("'b': None", r)
-    
+
     def test_helper_function(self):
         # two paths, one for real dicts and one for other mappings
         elems = list('abracadabra')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 22:30:11 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 22:30:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Accept_None_as_?=
 =?utf8?q?start_and_stop_parameters_for_list=2Eindex=28=29_and_tuple=2Eind?=
 =?utf8?q?ex=28=29?=
Message-ID: <E1RMnoZ-0004uC-RE@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f0eda4daac7
changeset:   73389:0f0eda4daac7
branch:      2.7
parent:      73377:8ea34a74f118
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 23:18:06 2011 +0200
summary:
  Accept None as start and stop parameters for list.index() and tuple.index()

Closes #13340.

files:
  Lib/test/list_tests.py |   7 +++++++
  Lib/test/seq_tests.py  |   7 +++++++
  Misc/NEWS              |   3 +++
  Objects/listobject.c   |  14 +++++++++++---
  Objects/tupleobject.c  |  15 +++++++++++----
  5 files changed, 39 insertions(+), 7 deletions(-)


diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py
--- a/Lib/test/list_tests.py
+++ b/Lib/test/list_tests.py
@@ -365,6 +365,13 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
+        self.assertEqual(u.index(1, None), 4)
+        self.assertEqual(u.index(1, None, None), 4)
+        self.assertEqual(u.index(1, 0, None), 4)
+        self.assertEqual(u.index(1, None, 6), 4)
+        self.assertRaises(ValueError, u.index, -1, 3)
+        self.assertRaises(ValueError, u.index, -1, 3, None)
+        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py
--- a/Lib/test/seq_tests.py
+++ b/Lib/test/seq_tests.py
@@ -363,6 +363,13 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
+        self.assertEqual(u.index(1, None), 4)
+        self.assertEqual(u.index(1, None, None), 4)
+        self.assertEqual(u.index(1, 0, None), 4)
+        self.assertEqual(u.index(1, None, 6), 4)
+        self.assertRaises(ValueError, u.index, -1, 3)
+        self.assertRaises(ValueError, u.index, -1, 3, None)
+        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -9,6 +9,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13340: Accept None as start and stop parameters for
+  list.index() and tuple.index().
+
 - Issue #10519: Avoid unnecessary recursive function calls in
   setobject.c.
 
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2277,12 +2277,20 @@
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
     PyObject *v, *format_tuple, *err_string;
+    PyObject *start_obj = NULL, *stop_obj = NULL;
     static PyObject *err_format = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
-                                _PyEval_SliceIndex, &start,
-                                _PyEval_SliceIndex, &stop))
+    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
+
+    if (start_obj != Py_None)
+        if (!_PyEval_SliceIndex(start_obj, &start))
+            return NULL;
+
+    if (stop_obj != Py_None)
+        if (!_PyEval_SliceIndex(stop_obj, &stop))
+            return NULL;
+
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -510,12 +510,19 @@
 tupleindex(PyTupleObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v;
+    PyObject *v, *start_obj = NULL, *stop_obj = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
-                                _PyEval_SliceIndex, &start,
-                                _PyEval_SliceIndex, &stop))
+    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
+
+    if (start_obj != Py_None)
+        if (!_PyEval_SliceIndex(start_obj, &start))
+            return NULL;
+
+    if (stop_obj != Py_None)
+        if (!_PyEval_SliceIndex(stop_obj, &stop))
+            return NULL;
+
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 22:30:12 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 22:30:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Accept_None_as_?=
 =?utf8?q?start_and_stop_parameters_for_list=2Eindex=28=29_and_tuple=2Eind?=
 =?utf8?q?ex=28=29?=
Message-ID: <E1RMnoa-0004uJ-LN@dinsdale.python.org>

http://hg.python.org/cpython/rev/5c1fcaf3cf1c
changeset:   73390:5c1fcaf3cf1c
branch:      3.2
parent:      73386:970ec22ab368
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 23:20:57 2011 +0200
summary:
  Accept None as start and stop parameters for list.index() and tuple.index()

Closes #13340.

files:
  Lib/test/list_tests.py |   7 +++++++
  Lib/test/seq_tests.py  |   7 +++++++
  Misc/NEWS              |   3 +++
  Objects/listobject.c   |  14 +++++++++++---
  Objects/tupleobject.c  |  15 +++++++++++----
  5 files changed, 39 insertions(+), 7 deletions(-)


diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py
--- a/Lib/test/list_tests.py
+++ b/Lib/test/list_tests.py
@@ -365,6 +365,13 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
+        self.assertEqual(u.index(1, None), 4)
+        self.assertEqual(u.index(1, None, None), 4)
+        self.assertEqual(u.index(1, 0, None), 4)
+        self.assertEqual(u.index(1, None, 6), 4)
+        self.assertRaises(ValueError, u.index, -1, 3)
+        self.assertRaises(ValueError, u.index, -1, 3, None)
+        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py
--- a/Lib/test/seq_tests.py
+++ b/Lib/test/seq_tests.py
@@ -361,6 +361,13 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
+        self.assertEqual(u.index(1, None), 4)
+        self.assertEqual(u.index(1, None, None), 4)
+        self.assertEqual(u.index(1, 0, None), 4)
+        self.assertEqual(u.index(1, None, 6), 4)
+        self.assertRaises(ValueError, u.index, -1, 3)
+        self.assertRaises(ValueError, u.index, -1, 3, None)
+        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13340: Accept None as start and stop parameters for
+  list.index() and tuple.index().
+
 - Issue #13343: Fix a SystemError when a lambda expression uses a global
   variable in the default value of a keyword-only argument:
   (lambda *, arg=GLOBAL_NAME: None)
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2109,12 +2109,20 @@
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
     PyObject *v, *format_tuple, *err_string;
+    PyObject *start_obj = NULL, *stop_obj = NULL;
     static PyObject *err_format = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
-                                _PyEval_SliceIndex, &start,
-                                _PyEval_SliceIndex, &stop))
+    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
+
+    if (start_obj != Py_None)
+        if (!_PyEval_SliceIndex(start_obj, &start))
+            return NULL;
+
+    if (stop_obj != Py_None)
+        if (!_PyEval_SliceIndex(stop_obj, &stop))
+            return NULL;
+
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -483,12 +483,19 @@
 tupleindex(PyTupleObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v;
+    PyObject *v, *start_obj = NULL, *stop_obj = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
-                                _PyEval_SliceIndex, &start,
-                                _PyEval_SliceIndex, &stop))
+    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
+
+    if (start_obj != Py_None)
+        if (!_PyEval_SliceIndex(start_obj, &start))
+            return NULL;
+
+    if (stop_obj != Py_None)
+        if (!_PyEval_SliceIndex(stop_obj, &stop))
+            return NULL;
+
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov  5 22:30:13 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 05 Nov 2011 22:30:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Accept_None_as_start_and_stop_parameters_for_list=2Eindex=28?=
 =?utf8?b?KSBhbmQgdHVwbGUuaW5kZXgoKS4=?=
Message-ID: <E1RMnob-0004uO-FL@dinsdale.python.org>

http://hg.python.org/cpython/rev/c33aa14f4edb
changeset:   73391:c33aa14f4edb
parent:      73388:89c7ba01a3d2
parent:      73390:5c1fcaf3cf1c
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 05 23:23:17 2011 +0200
summary:
  Accept None as start and stop parameters for list.index() and tuple.index().

Closes #13340.

files:
  Lib/test/list_tests.py |   7 +++++++
  Lib/test/seq_tests.py  |   7 +++++++
  Misc/NEWS              |   3 +++
  Objects/listobject.c   |  14 +++++++++++---
  Objects/tupleobject.c  |  15 +++++++++++----
  5 files changed, 39 insertions(+), 7 deletions(-)


diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py
--- a/Lib/test/list_tests.py
+++ b/Lib/test/list_tests.py
@@ -365,6 +365,13 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
+        self.assertEqual(u.index(1, None), 4)
+        self.assertEqual(u.index(1, None, None), 4)
+        self.assertEqual(u.index(1, 0, None), 4)
+        self.assertEqual(u.index(1, None, 6), 4)
+        self.assertRaises(ValueError, u.index, -1, 3)
+        self.assertRaises(ValueError, u.index, -1, 3, None)
+        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py
--- a/Lib/test/seq_tests.py
+++ b/Lib/test/seq_tests.py
@@ -361,6 +361,13 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
+        self.assertEqual(u.index(1, None), 4)
+        self.assertEqual(u.index(1, None, None), 4)
+        self.assertEqual(u.index(1, 0, None), 4)
+        self.assertEqual(u.index(1, None, 6), 4)
+        self.assertRaises(ValueError, u.index, -1, 3)
+        self.assertRaises(ValueError, u.index, -1, 3, None)
+        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13340: Accept None as start and stop parameters for
+  list.index() and tuple.index().
+
 - Issue #9896: Add start, stop, and step attributes to range objects.
 
 - Issue #13343: Fix a SystemError when a lambda expression uses a global
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2122,12 +2122,20 @@
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
     PyObject *v, *format_tuple, *err_string;
+    PyObject *start_obj = NULL, *stop_obj = NULL;
     static PyObject *err_format = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
-                                _PyEval_SliceIndex, &start,
-                                _PyEval_SliceIndex, &stop))
+    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
+
+    if (start_obj != Py_None)
+        if (!_PyEval_SliceIndex(start_obj, &start))
+            return NULL;
+
+    if (stop_obj != Py_None)
+        if (!_PyEval_SliceIndex(stop_obj, &stop))
+            return NULL;
+
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -484,12 +484,19 @@
 tupleindex(PyTupleObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v;
+    PyObject *v, *start_obj = NULL, *stop_obj = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
-                                _PyEval_SliceIndex, &start,
-                                _PyEval_SliceIndex, &stop))
+    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
+
+    if (start_obj != Py_None)
+        if (!_PyEval_SliceIndex(start_obj, &start))
+            return NULL;
+
+    if (stop_obj != Py_None)
+        if (!_PyEval_SliceIndex(stop_obj, &stop))
+            return NULL;
+
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 00:25:16 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 00:25:16 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_A_function=27s_local_namespace?=
 =?utf8?q?_is_signalled_by_the_=22=3Clocals=3E=22_component?=
Message-ID: <E1RMpbw-0006rz-HL@dinsdale.python.org>

http://hg.python.org/peps/rev/70ccbc8a46bc
changeset:   3981:70ccbc8a46bc
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 00:20:53 2011 +0100
summary:
  A function's local namespace is signalled by the "<locals>" component

files:
  pep-3155.txt |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/pep-3155.txt b/pep-3155.txt
--- a/pep-3155.txt
+++ b/pep-3155.txt
@@ -64,7 +64,8 @@
 ``__qname__`` attribute is equal to the ``__name__`` attribute.  For
 nested classed, methods, and nested functions, the ``__qname__``
 attribute contains a dotted path leading to the object from the module
-top-level.
+top-level.  A function's local namespace is represented in that dotted
+path by a component named ``<locals>``.
 
 The repr() and str() of functions and classes is modified to use
 ``__qname__`` rather than ``__name__``.
@@ -96,7 +97,7 @@
 >>> f.__qname__
 'f'
 >>> f().__qname__
-'f.g'
+'f.<locals>.g'
 
 
 Limitations

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sun Nov  6 00:46:50 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 00:46:50 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzQy?=
 =?utf8?q?=3A_input=28=29_used_to_ignore_sys=2Estdin=27s_and_sys=2Estdout?=
 =?utf8?q?=27s_unicode?=
Message-ID: <E1RMpwo-0008Up-5D@dinsdale.python.org>

http://hg.python.org/cpython/rev/421c8e291221
changeset:   73392:421c8e291221
branch:      3.2
parent:      73390:5c1fcaf3cf1c
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 00:34:26 2011 +0100
summary:
  Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
error handler in interactive mode (when calling into PyOS_Readline()).

files:
  Lib/test/test_builtin.py |  70 +++++++++++++++++++++++
  Misc/NEWS                |   3 +
  Python/bltinmodule.c     |  83 +++++++++++++--------------
  3 files changed, 114 insertions(+), 42 deletions(-)


diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -6,12 +6,17 @@
 import warnings
 import collections
 import io
+import os
 import ast
 import types
 import builtins
 import random
 from test.support import fcmp, TESTFN, unlink,  run_unittest, check_warnings
 from operator import neg
+try:
+    import pty
+except ImportError:
+    pty = None
 
 
 class Squares:
@@ -988,6 +993,71 @@
             fp.close()
             unlink(TESTFN)
 
+    @unittest.skipUnless(pty, "the pty module isn't available")
+    def check_input_tty(self, prompt, terminal_input, stdio_encoding=None):
+        r, w = os.pipe()
+        try:
+            pid, fd = pty.fork()
+        except (OSError, AttributeError) as e:
+            os.close(r)
+            os.close(w)
+            self.skipTest("pty.fork() raised {}".format(e))
+        if pid == 0:
+            # Child
+            os.close(r)
+            # Check the error handlers are accounted for
+            if stdio_encoding:
+                sys.stdin = io.TextIOWrapper(sys.stdin.detach(),
+                                             encoding=stdio_encoding,
+                                             errors='surrogateescape')
+                sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
+                                              encoding=stdio_encoding,
+                                              errors='replace')
+            with open(w, "w") as wpipe:
+                try:
+                    print("tty =", sys.stdin.isatty() and sys.stdout.isatty(), file=wpipe)
+                    print(ascii(input(prompt)), file=wpipe)
+                finally:
+                    print(";EOF", file=wpipe)
+            # We don't want to return to unittest...
+            os._exit(0)
+        # Parent
+        os.close(w)
+        os.write(fd, terminal_input + b"\r\n")
+        # Get results from the pipe
+        with open(r, "r") as rpipe:
+            lines = []
+            while True:
+                line = rpipe.readline().strip()
+                if line == ";EOF":
+                    break
+                lines.append(line)
+        # Check we did exercise the GNU readline path
+        self.assertIn(lines[0], {'tty = True', 'tty = False'})
+        if lines[0] != 'tty = True':
+            self.skipTest("standard IO in should have been a tty")
+        # Check the result was got and corresponds to the user's terminal input
+        self.assertEqual(len(lines), 2)
+        input_result = eval(lines[1])   # ascii() -> eval() roundtrip
+        if stdio_encoding:
+            expected = terminal_input.decode(stdio_encoding, 'surrogateescape')
+        else:
+            expected = terminal_input.decode(sys.stdin.encoding)  # what else?
+        self.assertEqual(input_result, expected)
+
+    def test_input_tty(self):
+        # Test input() functionality when wired to a tty (the code path
+        # is different and invokes GNU readline if available).
+        self.check_input_tty("prompt", b"quux")
+
+    def test_input_tty_non_ascii(self):
+        # Check stdin/stdout encoding is used when invoking GNU readline
+        self.check_input_tty("prompt?", b"quux\xe9", "utf-8")
+
+    def test_input_tty_non_ascii_unicode_errors(self):
+        # Check stdin/stdout error handler is used when invoking GNU readline
+        self.check_input_tty("prompt?", b"quux\xe9", "ascii")
+
     def test_repr(self):
         self.assertEqual(repr(''), '\'\'')
         self.assertEqual(repr(0), '0')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
+  error handler in interactive mode (when calling into PyOS_Readline()).
+
 - Issue #13340: Accept None as start and stop parameters for
   list.index() and tuple.index().
 
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -1640,76 +1640,65 @@
 
     /* If we're interactive, use (GNU) readline */
     if (tty) {
-        PyObject *po;
+        PyObject *po = NULL;
         char *prompt;
-        char *s;
-        PyObject *stdin_encoding;
-        char *stdin_encoding_str;
+        char *s = NULL;
+        PyObject *stdin_encoding = NULL, *stdin_errors = NULL;
+        PyObject *stdout_encoding = NULL, *stdout_errors = NULL;
+        char *stdin_encoding_str, *stdin_errors_str;
         PyObject *result;
         size_t len;
 
         stdin_encoding = PyObject_GetAttrString(fin, "encoding");
-        if (!stdin_encoding)
+        stdin_errors = PyObject_GetAttrString(fin, "errors");
+        if (!stdin_encoding || !stdin_errors)
             /* stdin is a text stream, so it must have an
                encoding. */
-            return NULL;
+            goto _readline_errors;
         stdin_encoding_str = _PyUnicode_AsString(stdin_encoding);
-        if (stdin_encoding_str  == NULL) {
-            Py_DECREF(stdin_encoding);
-            return NULL;
-        }
+        stdin_errors_str = _PyUnicode_AsString(stdin_errors);
+        if (!stdin_encoding_str || !stdin_errors_str)
+            goto _readline_errors;
         tmp = PyObject_CallMethod(fout, "flush", "");
         if (tmp == NULL)
             PyErr_Clear();
         else
             Py_DECREF(tmp);
         if (promptarg != NULL) {
+            /* We have a prompt, encode it as stdout would */
+            char *stdout_encoding_str, *stdout_errors_str;
             PyObject *stringpo;
-            PyObject *stdout_encoding;
-            char *stdout_encoding_str;
             stdout_encoding = PyObject_GetAttrString(fout, "encoding");
-            if (stdout_encoding == NULL) {
-                Py_DECREF(stdin_encoding);
-                return NULL;
-            }
+            stdout_errors = PyObject_GetAttrString(fout, "errors");
+            if (!stdout_encoding || !stdout_errors)
+                goto _readline_errors;
             stdout_encoding_str = _PyUnicode_AsString(stdout_encoding);
-            if (stdout_encoding_str == NULL) {
-                Py_DECREF(stdin_encoding);
-                Py_DECREF(stdout_encoding);
-                return NULL;
-            }
+            stdout_errors_str = _PyUnicode_AsString(stdout_errors);
+            if (!stdout_encoding_str || !stdout_errors_str)
+                goto _readline_errors;
             stringpo = PyObject_Str(promptarg);
-            if (stringpo == NULL) {
-                Py_DECREF(stdin_encoding);
-                Py_DECREF(stdout_encoding);
-                return NULL;
-            }
+            if (stringpo == NULL)
+                goto _readline_errors;
             po = PyUnicode_AsEncodedString(stringpo,
-                stdout_encoding_str, NULL);
-            Py_DECREF(stdout_encoding);
-            Py_DECREF(stringpo);
-            if (po == NULL) {
-                Py_DECREF(stdin_encoding);
-                return NULL;
-            }
+                stdout_encoding_str, stdout_errors_str);
+            Py_CLEAR(stdout_encoding);
+            Py_CLEAR(stdout_errors);
+            Py_CLEAR(stringpo);
+            if (po == NULL)
+                goto _readline_errors;
             prompt = PyBytes_AsString(po);
-            if (prompt == NULL) {
-                Py_DECREF(stdin_encoding);
-                Py_DECREF(po);
-                return NULL;
-            }
+            if (prompt == NULL)
+                goto _readline_errors;
         }
         else {
             po = NULL;
             prompt = "";
         }
         s = PyOS_Readline(stdin, stdout, prompt);
-        Py_XDECREF(po);
         if (s == NULL) {
             if (!PyErr_Occurred())
                 PyErr_SetNone(PyExc_KeyboardInterrupt);
-            Py_DECREF(stdin_encoding);
-            return NULL;
+            goto _readline_errors;
         }
 
         len = strlen(s);
@@ -1727,12 +1716,22 @@
                 len--;   /* strip trailing '\n' */
                 if (len != 0 && s[len-1] == '\r')
                     len--;   /* strip trailing '\r' */
-                result = PyUnicode_Decode(s, len, stdin_encoding_str, NULL);
+                result = PyUnicode_Decode(s, len, stdin_encoding_str,
+                                                  stdin_errors_str);
             }
         }
         Py_DECREF(stdin_encoding);
+        Py_DECREF(stdin_errors);
+        Py_XDECREF(po);
         PyMem_FREE(s);
         return result;
+    _readline_errors:
+        Py_XDECREF(stdin_encoding);
+        Py_XDECREF(stdout_encoding);
+        Py_XDECREF(stdin_errors);
+        Py_XDECREF(stdout_errors);
+        Py_XDECREF(po);
+        return NULL;
     }
 
     /* Fallback if we're not interactive */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 00:46:51 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 00:46:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313342=3A_input=28=29_used_to_ignore_sys=2Estdin=27s?=
 =?utf8?q?_and_sys=2Estdout=27s_unicode?=
Message-ID: <E1RMpwp-0008Ut-1o@dinsdale.python.org>

http://hg.python.org/cpython/rev/992ba03d60a8
changeset:   73393:992ba03d60a8
parent:      73391:c33aa14f4edb
parent:      73392:421c8e291221
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 00:38:45 2011 +0100
summary:
  Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
error handler in interactive mode (when calling into PyOS_Readline()).

files:
  Lib/test/test_builtin.py |  70 +++++++++++++++++++++++
  Misc/NEWS                |   3 +
  Python/bltinmodule.c     |  84 ++++++++++++++--------------
  3 files changed, 115 insertions(+), 42 deletions(-)


diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -6,12 +6,17 @@
 import warnings
 import collections
 import io
+import os
 import ast
 import types
 import builtins
 import random
 from test.support import TESTFN, unlink,  run_unittest, check_warnings
 from operator import neg
+try:
+    import pty
+except ImportError:
+    pty = None
 
 
 class Squares:
@@ -1000,6 +1005,71 @@
             fp.close()
             unlink(TESTFN)
 
+    @unittest.skipUnless(pty, "the pty module isn't available")
+    def check_input_tty(self, prompt, terminal_input, stdio_encoding=None):
+        r, w = os.pipe()
+        try:
+            pid, fd = pty.fork()
+        except (OSError, AttributeError) as e:
+            os.close(r)
+            os.close(w)
+            self.skipTest("pty.fork() raised {}".format(e))
+        if pid == 0:
+            # Child
+            os.close(r)
+            # Check the error handlers are accounted for
+            if stdio_encoding:
+                sys.stdin = io.TextIOWrapper(sys.stdin.detach(),
+                                             encoding=stdio_encoding,
+                                             errors='surrogateescape')
+                sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
+                                              encoding=stdio_encoding,
+                                              errors='replace')
+            with open(w, "w") as wpipe:
+                try:
+                    print("tty =", sys.stdin.isatty() and sys.stdout.isatty(), file=wpipe)
+                    print(ascii(input(prompt)), file=wpipe)
+                finally:
+                    print(";EOF", file=wpipe)
+            # We don't want to return to unittest...
+            os._exit(0)
+        # Parent
+        os.close(w)
+        os.write(fd, terminal_input + b"\r\n")
+        # Get results from the pipe
+        with open(r, "r") as rpipe:
+            lines = []
+            while True:
+                line = rpipe.readline().strip()
+                if line == ";EOF":
+                    break
+                lines.append(line)
+        # Check we did exercise the GNU readline path
+        self.assertIn(lines[0], {'tty = True', 'tty = False'})
+        if lines[0] != 'tty = True':
+            self.skipTest("standard IO in should have been a tty")
+        # Check the result was got and corresponds to the user's terminal input
+        self.assertEqual(len(lines), 2)
+        input_result = eval(lines[1])   # ascii() -> eval() roundtrip
+        if stdio_encoding:
+            expected = terminal_input.decode(stdio_encoding, 'surrogateescape')
+        else:
+            expected = terminal_input.decode(sys.stdin.encoding)  # what else?
+        self.assertEqual(input_result, expected)
+
+    def test_input_tty(self):
+        # Test input() functionality when wired to a tty (the code path
+        # is different and invokes GNU readline if available).
+        self.check_input_tty("prompt", b"quux")
+
+    def test_input_tty_non_ascii(self):
+        # Check stdin/stdout encoding is used when invoking GNU readline
+        self.check_input_tty("prompt?", b"quux\xe9", "utf-8")
+
+    def test_input_tty_non_ascii_unicode_errors(self):
+        # Check stdin/stdout error handler is used when invoking GNU readline
+        self.check_input_tty("prompt?", b"quux\xe9", "ascii")
+
     def test_repr(self):
         self.assertEqual(repr(''), '\'\'')
         self.assertEqual(repr(0), '0')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
+  error handler in interactive mode (when calling into PyOS_Readline()).
+
 - Issue #13340: Accept None as start and stop parameters for
   list.index() and tuple.index().
 
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -1634,77 +1634,67 @@
 
     /* If we're interactive, use (GNU) readline */
     if (tty) {
-        PyObject *po;
+        PyObject *po = NULL;
         char *prompt;
-        char *s;
-        PyObject *stdin_encoding;
-        char *stdin_encoding_str;
+        char *s = NULL;
+        PyObject *stdin_encoding = NULL, *stdin_errors = NULL;
+        PyObject *stdout_encoding = NULL, *stdout_errors = NULL;
+        char *stdin_encoding_str, *stdin_errors_str;
         PyObject *result;
         size_t len;
         _Py_IDENTIFIER(encoding);
+        _Py_IDENTIFIER(errors);
 
         stdin_encoding = _PyObject_GetAttrId(fin, &PyId_encoding);
-        if (!stdin_encoding)
+        stdin_errors = _PyObject_GetAttrId(fin, &PyId_errors);
+        if (!stdin_encoding || !stdin_errors)
             /* stdin is a text stream, so it must have an
                encoding. */
-            return NULL;
+            goto _readline_errors;
         stdin_encoding_str = _PyUnicode_AsString(stdin_encoding);
-        if (stdin_encoding_str  == NULL) {
-            Py_DECREF(stdin_encoding);
-            return NULL;
-        }
+        stdin_errors_str = _PyUnicode_AsString(stdin_errors);
+        if (!stdin_encoding_str || !stdin_errors_str)
+            goto _readline_errors;
         tmp = _PyObject_CallMethodId(fout, &PyId_flush, "");
         if (tmp == NULL)
             PyErr_Clear();
         else
             Py_DECREF(tmp);
         if (promptarg != NULL) {
+            /* We have a prompt, encode it as stdout would */
+            char *stdout_encoding_str, *stdout_errors_str;
             PyObject *stringpo;
-            PyObject *stdout_encoding;
-            char *stdout_encoding_str;
             stdout_encoding = _PyObject_GetAttrId(fout, &PyId_encoding);
-            if (stdout_encoding == NULL) {
-                Py_DECREF(stdin_encoding);
-                return NULL;
-            }
+            stdout_errors = _PyObject_GetAttrId(fout, &PyId_errors);
+            if (!stdout_encoding || !stdout_errors)
+                goto _readline_errors;
             stdout_encoding_str = _PyUnicode_AsString(stdout_encoding);
-            if (stdout_encoding_str == NULL) {
-                Py_DECREF(stdin_encoding);
-                Py_DECREF(stdout_encoding);
-                return NULL;
-            }
+            stdout_errors_str = _PyUnicode_AsString(stdout_errors);
+            if (!stdout_encoding_str || !stdout_errors_str)
+                goto _readline_errors;
             stringpo = PyObject_Str(promptarg);
-            if (stringpo == NULL) {
-                Py_DECREF(stdin_encoding);
-                Py_DECREF(stdout_encoding);
-                return NULL;
-            }
+            if (stringpo == NULL)
+                goto _readline_errors;
             po = PyUnicode_AsEncodedString(stringpo,
-                stdout_encoding_str, NULL);
-            Py_DECREF(stdout_encoding);
-            Py_DECREF(stringpo);
-            if (po == NULL) {
-                Py_DECREF(stdin_encoding);
-                return NULL;
-            }
+                stdout_encoding_str, stdout_errors_str);
+            Py_CLEAR(stdout_encoding);
+            Py_CLEAR(stdout_errors);
+            Py_CLEAR(stringpo);
+            if (po == NULL)
+                goto _readline_errors;
             prompt = PyBytes_AsString(po);
-            if (prompt == NULL) {
-                Py_DECREF(stdin_encoding);
-                Py_DECREF(po);
-                return NULL;
-            }
+            if (prompt == NULL)
+                goto _readline_errors;
         }
         else {
             po = NULL;
             prompt = "";
         }
         s = PyOS_Readline(stdin, stdout, prompt);
-        Py_XDECREF(po);
         if (s == NULL) {
             if (!PyErr_Occurred())
                 PyErr_SetNone(PyExc_KeyboardInterrupt);
-            Py_DECREF(stdin_encoding);
-            return NULL;
+            goto _readline_errors;
         }
 
         len = strlen(s);
@@ -1722,12 +1712,22 @@
                 len--;   /* strip trailing '\n' */
                 if (len != 0 && s[len-1] == '\r')
                     len--;   /* strip trailing '\r' */
-                result = PyUnicode_Decode(s, len, stdin_encoding_str, NULL);
+                result = PyUnicode_Decode(s, len, stdin_encoding_str,
+                                                  stdin_errors_str);
             }
         }
         Py_DECREF(stdin_encoding);
+        Py_DECREF(stdin_errors);
+        Py_XDECREF(po);
         PyMem_FREE(s);
         return result;
+    _readline_errors:
+        Py_XDECREF(stdin_encoding);
+        Py_XDECREF(stdout_encoding);
+        Py_XDECREF(stdin_errors);
+        Py_XDECREF(stdout_errors);
+        Py_XDECREF(po);
+        return NULL;
     }
 
     /* Fallback if we're not interactive */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 02:42:06 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 02:42:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Try_to_make_the_tty_input?=
	=?utf8?q?=28=29_tests_more_robust?=
Message-ID: <E1RMrkM-0007LL-8h@dinsdale.python.org>

http://hg.python.org/cpython/rev/89a4d0b1d193
changeset:   73394:89a4d0b1d193
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 02:37:42 2011 +0100
summary:
  Try to make the tty input() tests more robust

files:
  Lib/test/test_builtin.py |  52 ++++++++++++++++-----------
  1 files changed, 31 insertions(+), 21 deletions(-)


diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -11,12 +11,13 @@
 import types
 import builtins
 import random
+import traceback
 from test.support import TESTFN, unlink,  run_unittest, check_warnings
 from operator import neg
 try:
-    import pty
+    import pty, signal
 except ImportError:
-    pty = None
+    pty = signal = None
 
 
 class Squares:
@@ -1005,7 +1006,7 @@
             fp.close()
             unlink(TESTFN)
 
-    @unittest.skipUnless(pty, "the pty module isn't available")
+    @unittest.skipUnless(pty, "the pty and signal modules must be available")
     def check_input_tty(self, prompt, terminal_input, stdio_encoding=None):
         r, w = os.pipe()
         try:
@@ -1016,23 +1017,26 @@
             self.skipTest("pty.fork() raised {}".format(e))
         if pid == 0:
             # Child
-            os.close(r)
-            # Check the error handlers are accounted for
-            if stdio_encoding:
-                sys.stdin = io.TextIOWrapper(sys.stdin.detach(),
-                                             encoding=stdio_encoding,
-                                             errors='surrogateescape')
-                sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
-                                              encoding=stdio_encoding,
-                                              errors='replace')
-            with open(w, "w") as wpipe:
-                try:
+            try:
+                # Make sure we don't get stuck if there's a problem
+                signal.alarm(2)
+                os.close(r)
+                # Check the error handlers are accounted for
+                if stdio_encoding:
+                    sys.stdin = io.TextIOWrapper(sys.stdin.detach(),
+                                                 encoding=stdio_encoding,
+                                                 errors='surrogateescape')
+                    sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
+                                                  encoding=stdio_encoding,
+                                                  errors='replace')
+                with open(w, "w") as wpipe:
                     print("tty =", sys.stdin.isatty() and sys.stdout.isatty(), file=wpipe)
                     print(ascii(input(prompt)), file=wpipe)
-                finally:
-                    print(";EOF", file=wpipe)
-            # We don't want to return to unittest...
-            os._exit(0)
+            except:
+                traceback.print_exc()
+            finally:
+                # We don't want to return to unittest...
+                os._exit(0)
         # Parent
         os.close(w)
         os.write(fd, terminal_input + b"\r\n")
@@ -1041,15 +1045,21 @@
             lines = []
             while True:
                 line = rpipe.readline().strip()
-                if line == ";EOF":
+                if line == "":
+                    # The other end was closed => the child exited
                     break
                 lines.append(line)
+        # Check the result was got and corresponds to the user's terminal input
+        if len(lines) != 2:
+            # Something went wrong, try to get at stderr
+            with open(fd, "r", encoding="ascii", errors="ignore") as child_output:
+                self.fail("got %d lines in pipe but expected 2, child output was:\n%s"
+                          % (len(lines), child_output.read()))
+        os.close(fd)
         # Check we did exercise the GNU readline path
         self.assertIn(lines[0], {'tty = True', 'tty = False'})
         if lines[0] != 'tty = True':
             self.skipTest("standard IO in should have been a tty")
-        # Check the result was got and corresponds to the user's terminal input
-        self.assertEqual(len(lines), 2)
         input_result = eval(lines[1])   # ascii() -> eval() roundtrip
         if stdio_encoding:
             expected = terminal_input.decode(stdio_encoding, 'surrogateescape')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 02:56:20 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 02:56:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Backport_robust?=
 =?utf8?q?ness_fix_for_test=5Fbuiltin?=
Message-ID: <E1RMry8-0000BF-Lr@dinsdale.python.org>

http://hg.python.org/cpython/rev/da46c53bb410
changeset:   73395:da46c53bb410
branch:      3.2
parent:      73392:421c8e291221
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 02:51:25 2011 +0100
summary:
  Backport robustness fix for test_builtin

files:
  Lib/test/test_builtin.py |  52 ++++++++++++++++-----------
  1 files changed, 31 insertions(+), 21 deletions(-)


diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -11,12 +11,13 @@
 import types
 import builtins
 import random
+import traceback
 from test.support import fcmp, TESTFN, unlink,  run_unittest, check_warnings
 from operator import neg
 try:
-    import pty
+    import pty, signal
 except ImportError:
-    pty = None
+    pty = signal = None
 
 
 class Squares:
@@ -993,7 +994,7 @@
             fp.close()
             unlink(TESTFN)
 
-    @unittest.skipUnless(pty, "the pty module isn't available")
+    @unittest.skipUnless(pty, "the pty and signal modules must be available")
     def check_input_tty(self, prompt, terminal_input, stdio_encoding=None):
         r, w = os.pipe()
         try:
@@ -1004,23 +1005,26 @@
             self.skipTest("pty.fork() raised {}".format(e))
         if pid == 0:
             # Child
-            os.close(r)
-            # Check the error handlers are accounted for
-            if stdio_encoding:
-                sys.stdin = io.TextIOWrapper(sys.stdin.detach(),
-                                             encoding=stdio_encoding,
-                                             errors='surrogateescape')
-                sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
-                                              encoding=stdio_encoding,
-                                              errors='replace')
-            with open(w, "w") as wpipe:
-                try:
+            try:
+                # Make sure we don't get stuck if there's a problem
+                signal.alarm(2)
+                os.close(r)
+                # Check the error handlers are accounted for
+                if stdio_encoding:
+                    sys.stdin = io.TextIOWrapper(sys.stdin.detach(),
+                                                 encoding=stdio_encoding,
+                                                 errors='surrogateescape')
+                    sys.stdout = io.TextIOWrapper(sys.stdout.detach(),
+                                                  encoding=stdio_encoding,
+                                                  errors='replace')
+                with open(w, "w") as wpipe:
                     print("tty =", sys.stdin.isatty() and sys.stdout.isatty(), file=wpipe)
                     print(ascii(input(prompt)), file=wpipe)
-                finally:
-                    print(";EOF", file=wpipe)
-            # We don't want to return to unittest...
-            os._exit(0)
+            except:
+                traceback.print_exc()
+            finally:
+                # We don't want to return to unittest...
+                os._exit(0)
         # Parent
         os.close(w)
         os.write(fd, terminal_input + b"\r\n")
@@ -1029,15 +1033,21 @@
             lines = []
             while True:
                 line = rpipe.readline().strip()
-                if line == ";EOF":
+                if line == "":
+                    # The other end was closed => the child exited
                     break
                 lines.append(line)
+        # Check the result was got and corresponds to the user's terminal input
+        if len(lines) != 2:
+            # Something went wrong, try to get at stderr
+            with open(fd, "r", encoding="ascii", errors="ignore") as child_output:
+                self.fail("got %d lines in pipe but expected 2, child output was:\n%s"
+                          % (len(lines), child_output.read()))
+        os.close(fd)
         # Check we did exercise the GNU readline path
         self.assertIn(lines[0], {'tty = True', 'tty = False'})
         if lines[0] != 'tty = True':
             self.skipTest("standard IO in should have been a tty")
-        # Check the result was got and corresponds to the user's terminal input
-        self.assertEqual(len(lines), 2)
         input_result = eval(lines[1])   # ascii() -> eval() roundtrip
         if stdio_encoding:
             expected = terminal_input.decode(stdio_encoding, 'surrogateescape')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 02:56:21 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 02:56:21 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Null_merge?=
Message-ID: <E1RMry9-0000BP-Ez@dinsdale.python.org>

http://hg.python.org/cpython/rev/d4d00652e980
changeset:   73396:d4d00652e980
parent:      73394:89a4d0b1d193
parent:      73395:da46c53bb410
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 02:51:52 2011 +0100
summary:
  Null merge

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 03:10:50 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 03:10:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Skip_early_if_s?=
 =?utf8?q?tdin_and_stdout_are_not_ttys?=
Message-ID: <E1RMsCA-00013p-HV@dinsdale.python.org>

http://hg.python.org/cpython/rev/a09677cb4831
changeset:   73397:a09677cb4831
branch:      3.2
parent:      73395:da46c53bb410
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 03:03:18 2011 +0100
summary:
  Skip early if stdin and stdout are not ttys

files:
  Lib/test/test_builtin.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -996,6 +996,8 @@
 
     @unittest.skipUnless(pty, "the pty and signal modules must be available")
     def check_input_tty(self, prompt, terminal_input, stdio_encoding=None):
+        if not sys.stdin.isatty() or not sys.stdout.isatty():
+            self.skipTest("stdin and stdout must be ttys")
         r, w = os.pipe()
         try:
             pid, fd = pty.fork()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 03:10:51 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 06 Nov 2011 03:10:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Skip_early_if_stdin_and_stdout_are_not_ttys?=
Message-ID: <E1RMsCB-00013z-AJ@dinsdale.python.org>

http://hg.python.org/cpython/rev/7790ad507972
changeset:   73398:7790ad507972
parent:      73396:d4d00652e980
parent:      73397:a09677cb4831
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 06 03:04:12 2011 +0100
summary:
  Skip early if stdin and stdout are not ttys

files:
  Lib/test/test_builtin.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -1008,6 +1008,8 @@
 
     @unittest.skipUnless(pty, "the pty and signal modules must be available")
     def check_input_tty(self, prompt, terminal_input, stdio_encoding=None):
+        if not sys.stdin.isatty() or not sys.stdout.isatty():
+            self.skipTest("stdin and stdout must be ttys")
         r, w = os.pipe()
         try:
             pid, fd = pty.fork()

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Nov  6 05:34:19 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 06 Nov 2011 05:34:19 +0100
Subject: [Python-checkins] Daily reference leaks (7790ad507972): sum=0
Message-ID: <E1RMuR1-0002cZ-7d@ap.vmr.nerim.net>

results for 7790ad507972 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogq1zsT1', '-x']

From python-checkins at python.org  Sun Nov  6 06:37:56 2011
From: python-checkins at python.org (ned.deily)
Date: Sun, 06 Nov 2011 06:37:56 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313300=3A_Fix_IDLE_?=
 =?utf8?q?Restart_Shell_command_failure_introduced_by?=
Message-ID: <E1RMvQa-0006HZ-SG@dinsdale.python.org>

http://hg.python.org/cpython/rev/ce483d696c06
changeset:   73399:ce483d696c06
user:        Ned Deily <nad at acm.org>
date:        Sat Nov 05 22:36:44 2011 -0700
summary:
  Issue #13300: Fix IDLE Restart Shell command failure introduced by
3a5a0943b201.  Do not close listening socket on subprocess restart.

files:
  Lib/idlelib/PyShell.py |  4 ++++
  Lib/idlelib/rpc.py     |  4 ----
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Lib/idlelib/PyShell.py b/Lib/idlelib/PyShell.py
--- a/Lib/idlelib/PyShell.py
+++ b/Lib/idlelib/PyShell.py
@@ -460,6 +460,10 @@
 
     def kill_subprocess(self):
         try:
+            self.rpcclt.listening_sock.close()
+        except AttributeError:  # no socket
+            pass
+        try:
             self.rpcclt.close()
         except AttributeError:  # no socket
             pass
diff --git a/Lib/idlelib/rpc.py b/Lib/idlelib/rpc.py
--- a/Lib/idlelib/rpc.py
+++ b/Lib/idlelib/rpc.py
@@ -534,10 +534,6 @@
     def get_remote_proxy(self, oid):
         return RPCProxy(self, oid)
 
-    def close(self):
-        self.listening_sock.close()
-        SocketIO.close(self)
-
 class RPCProxy(object):
 
     __methods = None

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 08:31:00 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sun, 06 Nov 2011 08:31:00 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogY3Vyc2VzLnRwYXJt?=
 =?utf8?q?=28=29_is_expecting_a_byte_string=2C_not_curses=2Etigetstr=28=29?=
Message-ID: <E1RMxC0-0004EF-Ep@dinsdale.python.org>

http://hg.python.org/cpython/rev/3a0a94797ac5
changeset:   73400:3a0a94797ac5
branch:      3.2
parent:      73397:a09677cb4831
user:        Petri Lehtinen <petri at digip.org>
date:        Sun Nov 06 09:24:19 2011 +0200
summary:
  curses.tparm() is expecting a byte string, not curses.tigetstr()

Issue #10570

files:
  Misc/NEWS |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -88,7 +88,7 @@
 - Issue #13339: Fix compile error in posixmodule.c due to missing semicolon.
   Thanks to Robert Xiao.
 
-- Issue #10570: curses.putp() and curses.tigetstr() are now expecting a byte
+- Issue #10570: curses.putp() and curses.tparm() are now expecting a byte
   string, instead of a Unicode string.
 
 - Issue #2892: preserve iterparse events in case of SyntaxError.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 08:31:01 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sun, 06 Nov 2011 08:31:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_curses=2Etparm=28=29_is_expecting_a_byte_string=2C_not_curse?=
 =?utf8?b?cy50aWdldHN0cigp?=
Message-ID: <E1RMxC1-0004EJ-Az@dinsdale.python.org>

http://hg.python.org/cpython/rev/626c6c7f3af6
changeset:   73401:626c6c7f3af6
parent:      73399:ce483d696c06
parent:      73400:3a0a94797ac5
user:        Petri Lehtinen <petri at digip.org>
date:        Sun Nov 06 09:26:17 2011 +0200
summary:
  curses.tparm() is expecting a byte string, not curses.tigetstr()

Issue #10570

files:
  Misc/NEWS |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -377,7 +377,7 @@
 - Byte compilation in packaging is now isolated from the calling Python -B or
   -O options, instead of being disallowed under -B or buggy under -O.
 
-- Issue #10570: curses.putp() and curses.tigetstr() are now expecting a byte
+- Issue #10570: curses.putp() and curses.tparm() are now expecting a byte
   string, instead of a Unicode string.
 
 - Issue #13295: http.server now produces valid HTML 4.01 strict.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 14:20:20 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 06 Nov 2011 14:20:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_remove_py3k_war?=
	=?utf8?q?ning_for_callable?=
Message-ID: <E1RN2e4-00084b-MQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/39573be48b4a
changeset:   73402:39573be48b4a
branch:      2.7
parent:      73389:0f0eda4daac7
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Nov 06 08:20:12 2011 -0500
summary:
  remove py3k warning for callable

files:
  Misc/NEWS            |  2 ++
  Python/bltinmodule.c |  3 ---
  2 files changed, 2 insertions(+), 3 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -9,6 +9,8 @@
 Core and Builtins
 -----------------
 
+- Remove Py3k warning for callable.
+
 - Issue #13340: Accept None as start and stop parameters for
   list.index() and tuple.index().
 
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -224,9 +224,6 @@
 static PyObject *
 builtin_callable(PyObject *self, PyObject *v)
 {
-    if (PyErr_WarnPy3k("callable() not supported in 3.1; "
-                       "use isinstance(x, collections.Callable)", 1) < 0)
-        return NULL;
     return PyBool_FromLong((long)PyCallable_Check(v));
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 15:15:25 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Sun, 06 Nov 2011 15:15:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313350=3A_Replace_m?=
 =?utf8?q?ost_usages_of_PyUnicode=5FFormat_by_PyUnicode=5FFromFormat=2E?=
Message-ID: <E1RN3VN-0003vB-M5@dinsdale.python.org>

http://hg.python.org/cpython/rev/386a319b1825
changeset:   73403:386a319b1825
parent:      73401:626c6c7f3af6
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Sun Nov 06 15:10:48 2011 +0100
summary:
  Issue #13350: Replace most usages of PyUnicode_Format by PyUnicode_FromFormat.

files:
  Doc/includes/noddy2.c     |  18 +-----------------
  Doc/includes/noddy3.c     |  18 +-----------------
  Doc/includes/noddy4.c     |  18 +-----------------
  Misc/NEWS                 |   3 +++
  Modules/_ctypes/_ctypes.c |  24 +++---------------------
  Modules/_sqlite/cache.c   |  21 ++-------------------
  Objects/listobject.c      |  18 ++----------------
  7 files changed, 13 insertions(+), 107 deletions(-)


diff --git a/Doc/includes/noddy2.c b/Doc/includes/noddy2.c
--- a/Doc/includes/noddy2.c
+++ b/Doc/includes/noddy2.c
@@ -84,15 +84,6 @@
 static PyObject *
 Noddy_name(Noddy* self)
 {
-    static PyObject *format = NULL;
-    PyObject *args, *result;
-
-    if (format == NULL) {
-        format = PyUnicode_FromString("%s %s");
-        if (format == NULL)
-            return NULL;
-    }
-
     if (self->first == NULL) {
         PyErr_SetString(PyExc_AttributeError, "first");
         return NULL;
@@ -103,14 +94,7 @@
         return NULL;
     }
 
-    args = Py_BuildValue("OO", self->first, self->last);
-    if (args == NULL)
-        return NULL;
-
-    result = PyUnicode_Format(format, args);
-    Py_DECREF(args);
-
-    return result;
+    return PyUnicode_FromFormat("%S %S", self->first, self->last);
 }
 
 static PyMethodDef Noddy_methods[] = {
diff --git a/Doc/includes/noddy3.c b/Doc/includes/noddy3.c
--- a/Doc/includes/noddy3.c
+++ b/Doc/includes/noddy3.c
@@ -147,23 +147,7 @@
 static PyObject *
 Noddy_name(Noddy* self)
 {
-    static PyObject *format = NULL;
-    PyObject *args, *result;
-
-    if (format == NULL) {
-        format = PyUnicode_FromString("%s %s");
-        if (format == NULL)
-            return NULL;
-    }
-
-    args = Py_BuildValue("OO", self->first, self->last);
-    if (args == NULL)
-        return NULL;
-
-    result = PyUnicode_Format(format, args);
-    Py_DECREF(args);
-
-    return result;
+    return PyUnicode_FromFormat("%S %S", self->first, self->last);
 }
 
 static PyMethodDef Noddy_methods[] = {
diff --git a/Doc/includes/noddy4.c b/Doc/includes/noddy4.c
--- a/Doc/includes/noddy4.c
+++ b/Doc/includes/noddy4.c
@@ -118,15 +118,6 @@
 static PyObject *
 Noddy_name(Noddy* self)
 {
-    static PyObject *format = NULL;
-    PyObject *args, *result;
-
-    if (format == NULL) {
-        format = PyUnicode_FromString("%s %s");
-        if (format == NULL)
-            return NULL;
-    }
-
     if (self->first == NULL) {
         PyErr_SetString(PyExc_AttributeError, "first");
         return NULL;
@@ -137,14 +128,7 @@
         return NULL;
     }
 
-    args = Py_BuildValue("OO", self->first, self->last);
-    if (args == NULL)
-        return NULL;
-
-    result = PyUnicode_Format(format, args);
-    Py_DECREF(args);
-
-    return result;
+    return PyUnicode_FromFormat("%S %S", self->first, self->last);
 }
 
 static PyMethodDef Noddy_methods[] = {
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13350: Simplify some C code by replacing most usages of
+  PyUnicode_Format by PyUnicode_FromFormat.
+
 - Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
   error handler in interactive mode (when calling into PyOS_Readline()).
 
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -4599,38 +4599,20 @@
 static PyObject *
 Simple_repr(CDataObject *self)
 {
-    PyObject *val, *name, *args, *result;
-    static PyObject *format;
+    PyObject *val, *result;
 
     if (Py_TYPE(self)->tp_base != &Simple_Type) {
         return PyUnicode_FromFormat("<%s object at %p>",
                                    Py_TYPE(self)->tp_name, self);
     }
 
-    if (format == NULL) {
-        format = PyUnicode_InternFromString("%s(%r)");
-        if (format == NULL)
-            return NULL;
-    }
-
     val = Simple_get_value(self);
     if (val == NULL)
         return NULL;
 
-    name = PyUnicode_FromString(Py_TYPE(self)->tp_name);
-    if (name == NULL) {
-        Py_DECREF(val);
-        return NULL;
-    }
-
-    args = PyTuple_Pack(2, name, val);
-    Py_DECREF(name);
+    result = PyUnicode_FromFormat("%s(%R)",
+                                  Py_TYPE(self)->tp_name, val);
     Py_DECREF(val);
-    if (args == NULL)
-        return NULL;
-
-    result = PyUnicode_Format(format, args);
-    Py_DECREF(args);
     return result;
 }
 
diff --git a/Modules/_sqlite/cache.c b/Modules/_sqlite/cache.c
--- a/Modules/_sqlite/cache.c
+++ b/Modules/_sqlite/cache.c
@@ -217,8 +217,6 @@
     pysqlite_Node* ptr;
     PyObject* prevkey;
     PyObject* nextkey;
-    PyObject* fmt_args;
-    PyObject* template;
     PyObject* display_str;
 
     ptr = self->first;
@@ -229,36 +227,21 @@
         } else {
             prevkey = Py_None;
         }
-        Py_INCREF(prevkey);
 
         if (ptr->next) {
             nextkey = ptr->next->key;
         } else {
             nextkey = Py_None;
         }
-        Py_INCREF(nextkey);
 
-        fmt_args = Py_BuildValue("OOO", prevkey, ptr->key, nextkey);
-        if (!fmt_args) {
-            return NULL;
-        }
-        template = PyUnicode_FromString("%s <- %s ->%s\n");
-        if (!template) {
-            Py_DECREF(fmt_args);
-            return NULL;
-        }
-        display_str = PyUnicode_Format(template, fmt_args);
-        Py_DECREF(template);
-        Py_DECREF(fmt_args);
+        display_str = PyUnicode_FromFormat("%S <- %S -> %S\n",
+                                           prevkey, ptr->key, nextkey);
         if (!display_str) {
             return NULL;
         }
         PyObject_Print(display_str, stdout, Py_PRINT_RAW);
         Py_DECREF(display_str);
 
-        Py_DECREF(prevkey);
-        Py_DECREF(nextkey);
-
         ptr = ptr->next;
     }
 
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2121,9 +2121,8 @@
 listindex(PyListObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v, *format_tuple, *err_string;
+    PyObject *v;
     PyObject *start_obj = NULL, *stop_obj = NULL;
-    static PyObject *err_format = NULL;
 
     if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
         return NULL;
@@ -2153,20 +2152,7 @@
         else if (cmp < 0)
             return NULL;
     }
-    if (err_format == NULL) {
-        err_format = PyUnicode_FromString("%r is not in list");
-        if (err_format == NULL)
-            return NULL;
-    }
-    format_tuple = PyTuple_Pack(1, v);
-    if (format_tuple == NULL)
-        return NULL;
-    err_string = PyUnicode_Format(err_format, format_tuple);
-    Py_DECREF(format_tuple);
-    if (err_string == NULL)
-        return NULL;
-    PyErr_SetObject(PyExc_ValueError, err_string);
-    Py_DECREF(err_string);
+    PyErr_Format(PyExc_ValueError, "%R is not in list", v);
     return NULL;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 16:18:38 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sun, 06 Nov 2011 16:18:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_tests_now_t?=
 =?utf8?q?hat_the_callable_warning_is_gone=2E?=
Message-ID: <E1RN4UY-0004KH-Qm@dinsdale.python.org>

http://hg.python.org/cpython/rev/0ff37060913d
changeset:   73404:0ff37060913d
branch:      2.7
parent:      73402:39573be48b4a
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sun Nov 06 17:17:52 2011 +0200
summary:
  Fix tests now that the callable warning is gone.

files:
  Lib/test/test_bool.py    |  5 ++---
  Lib/test/test_builtin.py |  1 -
  2 files changed, 2 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_bool.py b/Lib/test/test_bool.py
--- a/Lib/test/test_bool.py
+++ b/Lib/test/test_bool.py
@@ -180,9 +180,8 @@
         self.assertIs(hasattr([], "wobble"), False)
 
     def test_callable(self):
-        with test_support.check_py3k_warnings():
-            self.assertIs(callable(len), True)
-            self.assertIs(callable(1), False)
+        self.assertIs(callable(len), True)
+        self.assertIs(callable(1), False)
 
     def test_isinstance(self):
         self.assertIs(isinstance(True, bool), True)
diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py
--- a/Lib/test/test_builtin.py
+++ b/Lib/test/test_builtin.py
@@ -1683,7 +1683,6 @@
 
 def _run_unittest(*args):
     with check_py3k_warnings(
-            (".+ not supported in 3.1", DeprecationWarning),
             (".+ not supported in 3.x", DeprecationWarning),
             (".+ is renamed to imp.reload", DeprecationWarning),
             ("classic int division", DeprecationWarning)):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 16:51:27 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sun, 06 Nov 2011 16:51:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Fix_another_cal?=
	=?utf8?q?lable_warning=2E?=
Message-ID: <E1RN50J-0006oE-MQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/8a0e7eb18f16
changeset:   73405:8a0e7eb18f16
branch:      2.7
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sun Nov 06 17:50:52 2011 +0200
summary:
  Fix another callable warning.

files:
  Lib/test/test_socket.py |  5 ++---
  1 files changed, 2 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -147,9 +147,8 @@
         self.server_ready.wait()
         self.clientSetUp()
         self.client_ready.set()
-        with test_support.check_py3k_warnings():
-            if not callable(test_func):
-                raise TypeError("test_func must be a callable function.")
+        if not callable(test_func):
+            raise TypeError("test_func must be a callable function.")
         try:
             test_func()
         except Exception, strerror:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 17:51:00 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sun, 06 Nov 2011 17:51:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Silence_a_coupl?=
	=?utf8?q?e_of_warnings=2E?=
Message-ID: <E1RN5vw-0002kP-P2@dinsdale.python.org>

http://hg.python.org/cpython/rev/a6d70861bf6d
changeset:   73406:a6d70861bf6d
branch:      2.7
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sun Nov 06 18:50:32 2011 +0200
summary:
  Silence a couple of warnings.

files:
  Lib/test/pickletester.py   |  2 ++
  Lib/test/test_cfgparser.py |  5 +++--
  2 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py
--- a/Lib/test/pickletester.py
+++ b/Lib/test/pickletester.py
@@ -132,6 +132,8 @@
     def __reduce__(self):
         return (create_dynamic_class, self.reduce_args)
 
+    __hash__ = None
+
 def create_dynamic_class(name, bases):
     result = pickling_metaclass(name, bases, dict())
     result.reduce_args = (name, bases)
diff --git a/Lib/test/test_cfgparser.py b/Lib/test/test_cfgparser.py
--- a/Lib/test/test_cfgparser.py
+++ b/Lib/test/test_cfgparser.py
@@ -548,8 +548,9 @@
                          [dcomb.get(k, 10) for k in klist])      # get()
         self.assertEqual([k in cm for k in klist],
                          [k in dcomb for k in klist])            # __contains__()
-        self.assertEqual([cm.has_key(k) for k in klist],
-                         [dcomb.has_key(k) for k in klist])      # has_key()
+        with test_support.check_py3k_warnings():
+            self.assertEqual([cm.has_key(k) for k in klist],
+                             [dcomb.has_key(k) for k in klist])  # has_key()
 
 class Issue7005TestCase(unittest.TestCase):
     """Test output when None is set() as a value and allow_no_value == False.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 20:15:13 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sun, 06 Nov 2011 20:15:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Revert_=22Accep?=
 =?utf8?q?t_None_as_start_and_stop_parameters_for_list=2Eindex=28=29_and?=
Message-ID: <E1RN8BV-0004kj-OB@dinsdale.python.org>

http://hg.python.org/cpython/rev/19ffa12ffdd4
changeset:   73407:19ffa12ffdd4
branch:      2.7
user:        Petri Lehtinen <petri at digip.org>
date:        Sun Nov 06 20:58:50 2011 +0200
summary:
  Revert "Accept None as start and stop parameters for list.index() and tuple.index()"

Issue #13340.

files:
  Lib/test/list_tests.py |   7 -------
  Lib/test/seq_tests.py  |   7 -------
  Misc/NEWS              |   3 ---
  Objects/listobject.c   |  14 +++-----------
  Objects/tupleobject.c  |  15 ++++-----------
  5 files changed, 7 insertions(+), 39 deletions(-)


diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py
--- a/Lib/test/list_tests.py
+++ b/Lib/test/list_tests.py
@@ -365,13 +365,6 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
-        self.assertEqual(u.index(1, None), 4)
-        self.assertEqual(u.index(1, None, None), 4)
-        self.assertEqual(u.index(1, 0, None), 4)
-        self.assertEqual(u.index(1, None, 6), 4)
-        self.assertRaises(ValueError, u.index, -1, 3)
-        self.assertRaises(ValueError, u.index, -1, 3, None)
-        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py
--- a/Lib/test/seq_tests.py
+++ b/Lib/test/seq_tests.py
@@ -363,13 +363,6 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
-        self.assertEqual(u.index(1, None), 4)
-        self.assertEqual(u.index(1, None, None), 4)
-        self.assertEqual(u.index(1, 0, None), 4)
-        self.assertEqual(u.index(1, None, 6), 4)
-        self.assertRaises(ValueError, u.index, -1, 3)
-        self.assertRaises(ValueError, u.index, -1, 3, None)
-        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -11,9 +11,6 @@
 
 - Remove Py3k warning for callable.
 
-- Issue #13340: Accept None as start and stop parameters for
-  list.index() and tuple.index().
-
 - Issue #10519: Avoid unnecessary recursive function calls in
   setobject.c.
 
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2277,20 +2277,12 @@
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
     PyObject *v, *format_tuple, *err_string;
-    PyObject *start_obj = NULL, *stop_obj = NULL;
     static PyObject *err_format = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
+    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
+                                _PyEval_SliceIndex, &start,
+                                _PyEval_SliceIndex, &stop))
         return NULL;
-
-    if (start_obj != Py_None)
-        if (!_PyEval_SliceIndex(start_obj, &start))
-            return NULL;
-
-    if (stop_obj != Py_None)
-        if (!_PyEval_SliceIndex(stop_obj, &stop))
-            return NULL;
-
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -510,19 +510,12 @@
 tupleindex(PyTupleObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v, *start_obj = NULL, *stop_obj = NULL;
+    PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
+    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
+                                _PyEval_SliceIndex, &start,
+                                _PyEval_SliceIndex, &stop))
         return NULL;
-
-    if (start_obj != Py_None)
-        if (!_PyEval_SliceIndex(start_obj, &start))
-            return NULL;
-
-    if (stop_obj != Py_None)
-        if (!_PyEval_SliceIndex(stop_obj, &stop))
-            return NULL;
-
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 20:15:14 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sun, 06 Nov 2011 20:15:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Revert_=22Accep?=
 =?utf8?q?t_None_as_start_and_stop_parameters_for_list=2Eindex=28=29_and?=
Message-ID: <E1RN8BW-0004kn-JN@dinsdale.python.org>

http://hg.python.org/cpython/rev/ed0e85efac47
changeset:   73408:ed0e85efac47
branch:      3.2
parent:      73400:3a0a94797ac5
user:        Petri Lehtinen <petri at digip.org>
date:        Sun Nov 06 21:02:39 2011 +0200
summary:
  Revert "Accept None as start and stop parameters for list.index() and tuple.index()"

Issue #13340.

files:
  Lib/test/list_tests.py |   7 -------
  Lib/test/seq_tests.py  |   7 -------
  Misc/NEWS              |   3 ---
  Objects/listobject.c   |  14 +++-----------
  Objects/tupleobject.c  |  15 ++++-----------
  5 files changed, 7 insertions(+), 39 deletions(-)


diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py
--- a/Lib/test/list_tests.py
+++ b/Lib/test/list_tests.py
@@ -365,13 +365,6 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
-        self.assertEqual(u.index(1, None), 4)
-        self.assertEqual(u.index(1, None, None), 4)
-        self.assertEqual(u.index(1, 0, None), 4)
-        self.assertEqual(u.index(1, None, 6), 4)
-        self.assertRaises(ValueError, u.index, -1, 3)
-        self.assertRaises(ValueError, u.index, -1, 3, None)
-        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py
--- a/Lib/test/seq_tests.py
+++ b/Lib/test/seq_tests.py
@@ -361,13 +361,6 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
-        self.assertEqual(u.index(1, None), 4)
-        self.assertEqual(u.index(1, None, None), 4)
-        self.assertEqual(u.index(1, 0, None), 4)
-        self.assertEqual(u.index(1, None, 6), 4)
-        self.assertRaises(ValueError, u.index, -1, 3)
-        self.assertRaises(ValueError, u.index, -1, 3, None)
-        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -13,9 +13,6 @@
 - Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
   error handler in interactive mode (when calling into PyOS_Readline()).
 
-- Issue #13340: Accept None as start and stop parameters for
-  list.index() and tuple.index().
-
 - Issue #13343: Fix a SystemError when a lambda expression uses a global
   variable in the default value of a keyword-only argument:
   (lambda *, arg=GLOBAL_NAME: None)
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2109,20 +2109,12 @@
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
     PyObject *v, *format_tuple, *err_string;
-    PyObject *start_obj = NULL, *stop_obj = NULL;
     static PyObject *err_format = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
+    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
+                                _PyEval_SliceIndex, &start,
+                                _PyEval_SliceIndex, &stop))
         return NULL;
-
-    if (start_obj != Py_None)
-        if (!_PyEval_SliceIndex(start_obj, &start))
-            return NULL;
-
-    if (stop_obj != Py_None)
-        if (!_PyEval_SliceIndex(stop_obj, &stop))
-            return NULL;
-
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -483,19 +483,12 @@
 tupleindex(PyTupleObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v, *start_obj = NULL, *stop_obj = NULL;
+    PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
+    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
+                                _PyEval_SliceIndex, &start,
+                                _PyEval_SliceIndex, &stop))
         return NULL;
-
-    if (start_obj != Py_None)
-        if (!_PyEval_SliceIndex(start_obj, &start))
-            return NULL;
-
-    if (stop_obj != Py_None)
-        if (!_PyEval_SliceIndex(stop_obj, &stop))
-            return NULL;
-
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 20:15:15 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sun, 06 Nov 2011 20:15:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Revert_=22Accept_None_as_start_and_stop_parameters_for_list?=
 =?utf8?b?LmluZGV4KCkgYW5k?=
Message-ID: <E1RN8BX-0004kr-CY@dinsdale.python.org>

http://hg.python.org/cpython/rev/106f9e1ad7ab
changeset:   73409:106f9e1ad7ab
parent:      73403:386a319b1825
parent:      73408:ed0e85efac47
user:        Petri Lehtinen <petri at digip.org>
date:        Sun Nov 06 21:05:41 2011 +0200
summary:
  Revert "Accept None as start and stop parameters for list.index() and tuple.index()"

Issue #13340.

files:
  Lib/test/list_tests.py |   7 -------
  Lib/test/seq_tests.py  |   7 -------
  Misc/NEWS              |   3 ---
  Objects/listobject.c   |  14 +++-----------
  Objects/tupleobject.c  |  15 ++++-----------
  5 files changed, 7 insertions(+), 39 deletions(-)


diff --git a/Lib/test/list_tests.py b/Lib/test/list_tests.py
--- a/Lib/test/list_tests.py
+++ b/Lib/test/list_tests.py
@@ -365,13 +365,6 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
-        self.assertEqual(u.index(1, None), 4)
-        self.assertEqual(u.index(1, None, None), 4)
-        self.assertEqual(u.index(1, 0, None), 4)
-        self.assertEqual(u.index(1, None, 6), 4)
-        self.assertRaises(ValueError, u.index, -1, 3)
-        self.assertRaises(ValueError, u.index, -1, 3, None)
-        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Lib/test/seq_tests.py b/Lib/test/seq_tests.py
--- a/Lib/test/seq_tests.py
+++ b/Lib/test/seq_tests.py
@@ -361,13 +361,6 @@
         self.assertEqual(u.index(0, 3), 3)
         self.assertEqual(u.index(0, 3, 4), 3)
         self.assertRaises(ValueError, u.index, 2, 0, -10)
-        self.assertEqual(u.index(1, None), 4)
-        self.assertEqual(u.index(1, None, None), 4)
-        self.assertEqual(u.index(1, 0, None), 4)
-        self.assertEqual(u.index(1, None, 6), 4)
-        self.assertRaises(ValueError, u.index, -1, 3)
-        self.assertRaises(ValueError, u.index, -1, 3, None)
-        self.assertRaises(ValueError, u.index, 1, None, 4)
 
         self.assertRaises(TypeError, u.index)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -16,9 +16,6 @@
 - Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
   error handler in interactive mode (when calling into PyOS_Readline()).
 
-- Issue #13340: Accept None as start and stop parameters for
-  list.index() and tuple.index().
-
 - Issue #9896: Add start, stop, and step attributes to range objects.
 
 - Issue #13343: Fix a SystemError when a lambda expression uses a global
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -2122,19 +2122,11 @@
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
     PyObject *v;
-    PyObject *start_obj = NULL, *stop_obj = NULL;
 
-    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
+    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
+                                _PyEval_SliceIndex, &start,
+                                _PyEval_SliceIndex, &stop))
         return NULL;
-
-    if (start_obj != Py_None)
-        if (!_PyEval_SliceIndex(start_obj, &start))
-            return NULL;
-
-    if (stop_obj != Py_None)
-        if (!_PyEval_SliceIndex(stop_obj, &stop))
-            return NULL;
-
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)
diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c
--- a/Objects/tupleobject.c
+++ b/Objects/tupleobject.c
@@ -484,19 +484,12 @@
 tupleindex(PyTupleObject *self, PyObject *args)
 {
     Py_ssize_t i, start=0, stop=Py_SIZE(self);
-    PyObject *v, *start_obj = NULL, *stop_obj = NULL;
+    PyObject *v;
 
-    if (!PyArg_ParseTuple(args, "O|OO:index", &v, &start_obj, &stop_obj))
+    if (!PyArg_ParseTuple(args, "O|O&O&:index", &v,
+                                _PyEval_SliceIndex, &start,
+                                _PyEval_SliceIndex, &stop))
         return NULL;
-
-    if (start_obj != Py_None)
-        if (!_PyEval_SliceIndex(start_obj, &start))
-            return NULL;
-
-    if (stop_obj != Py_None)
-        if (!_PyEval_SliceIndex(stop_obj, &stop))
-            return NULL;
-
     if (start < 0) {
         start += Py_SIZE(self);
         if (start < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov  6 20:41:36 2011
From: python-checkins at python.org (brian.curtin)
Date: Sun, 06 Nov 2011 20:41:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_=2313327=2E_Remove_the_?=
 =?utf8?q?need_for_an_explicit_None_as_the_second_argument_to?=
Message-ID: <E1RN8b2-0006kw-71@dinsdale.python.org>

http://hg.python.org/cpython/rev/99e118951a80
changeset:   73410:99e118951a80
user:        Brian Curtin <brian at python.org>
date:        Sun Nov 06 13:41:17 2011 -0600
summary:
  Fix #13327. Remove the need for an explicit None as the second argument to
os.utime in order to update to the current time. The second argument is now
optional.

files:
  Doc/library/os.rst    |  21 +++++++++++----------
  Lib/test/test_os.py   |  15 +++++++++++++++
  Misc/NEWS             |   3 +++
  Modules/posixmodule.c |  16 ++++++++--------
  4 files changed, 37 insertions(+), 18 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -2134,18 +2134,19 @@
    Availability: Unix, Windows.
 
 
-.. function:: utime(path, times)
+.. function:: utime(path[, times])
 
    Set the access and modified times of the file specified by *path*. If *times*
-   is ``None``, then the file's access and modified times are set to the current
-   time. (The effect is similar to running the Unix program :program:`touch` on
-   the path.)  Otherwise, *times* must be a 2-tuple of numbers, of the form
-   ``(atime, mtime)`` which is used to set the access and modified times,
-   respectively. Whether a directory can be given for *path* depends on whether
-   the operating system implements directories as files (for example, Windows
-   does not).  Note that the exact times you set here may not be returned by a
-   subsequent :func:`~os.stat` call, depending on the resolution with which your
-   operating system records access and modification times; see :func:`~os.stat`.
+   is ``None`` or not specified, then the file's access and modified times are
+   set to the current time. (The effect is similar to running the Unix program
+   :program:`touch` on the path.)  Otherwise, *times* must be a 2-tuple of
+   numbers, of the form ``(atime, mtime)`` which is used to set the access and
+   modified times, respectively. Whether a directory can be given for *path*
+   depends on whether the operating system implements directories as files
+   (for example, Windows does not).  Note that the exact times you set here may
+   not be returned by a subsequent :func:`~os.stat` call, depending on the
+   resolution with which your operating system records access and modification
+   times; see :func:`~os.stat`.
 
    Availability: Unix, Windows.
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -270,6 +270,21 @@
         st2 = os.stat(support.TESTFN)
         self.assertEqual(st2.st_mtime, int(st.st_mtime-delta))
 
+    def test_utime_noargs(self):
+        # (insert issue#) removed the requirement to pass None as the
+        # second argument. Check that the previous methods of passing
+        # a time tuple or None work in addition to no argument.
+        st = os.stat(support.TESTFN)
+        # Doesn't set anything new, but sets the time tuple way
+        os.utime(support.TESTFN, (st.st_atime, st.st_mtime))
+        # Set to the current time in the old explicit way.
+        os.utime(support.TESTFN, None)
+        st1 = os.stat(support.TESTFN)
+        # Set to the current time in the new way
+        os.utime(support.TESTFN)
+        st2 = os.stat(support.TESTFN)
+        self.assertAlmostEqual(st1.st_mtime, st2.st_mtime, delta=10)
+
     # Restrict test to Win32, since there is no guarantee other
     # systems support centiseconds
     if sys.platform == 'win32':
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13327: Remove the need for an explicit None as the second argument
+  to os.utime in order to update to the current time.
+
 - Issue #13350: Simplify some C code by replacing most usages of
   PyUnicode_Format by PyUnicode_FromFormat.
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -3543,7 +3543,7 @@
 posix_utime(PyObject *self, PyObject *args)
 {
 #ifdef MS_WINDOWS
-    PyObject *arg;
+    PyObject *arg = NULL;
     PyObject *obwpath;
     wchar_t *wpath = NULL;
     PyObject *oapath;
@@ -3554,7 +3554,7 @@
     FILETIME atime, mtime;
     PyObject *result = NULL;
 
-    if (PyArg_ParseTuple(args, "UO|:utime", &obwpath, &arg)) {
+    if (PyArg_ParseTuple(args, "U|O:utime", &obwpath, &arg)) {
         wpath = PyUnicode_AsUnicode(obwpath);
         if (wpath == NULL)
             return NULL;
@@ -3571,7 +3571,7 @@
            are also valid. */
         PyErr_Clear();
 
-        if (!PyArg_ParseTuple(args, "O&O:utime",
+        if (!PyArg_ParseTuple(args, "O&|O:utime",
                         PyUnicode_FSConverter, &oapath, &arg))
             return NULL;
 
@@ -3589,7 +3589,7 @@
         Py_DECREF(oapath);
     }
 
-    if (arg == Py_None) {
+    if (!arg || (arg == Py_None)) {
         SYSTEMTIME now;
         GetSystemTime(&now);
         if (!SystemTimeToFileTime(&now, &mtime) ||
@@ -3633,13 +3633,13 @@
     time_t atime, mtime;
     long ausec, musec;
     int res;
-    PyObject* arg;
-
-    if (!PyArg_ParseTuple(args, "O&O:utime",
+    PyObject* arg = NULL;
+
+    if (!PyArg_ParseTuple(args, "O&|O:utime",
                           PyUnicode_FSConverter, &opath, &arg))
         return NULL;
     path = PyBytes_AsString(opath);
-    if (arg == Py_None) {
+    if (!arg || (arg == Py_None)) {
         /* optional time values not given */
         Py_BEGIN_ALLOW_THREADS
         res = utime(path, NULL);

-- 
Repository URL: http://hg.python.org/cpython

From benjamin at python.org  Sun Nov  6 20:46:33 2011
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 6 Nov 2011 14:46:33 -0500
Subject: [Python-checkins] cpython: Fix #13327. Remove the need for an
 explicit None as the second argument to
In-Reply-To: <E1RN8b2-0006kw-71@dinsdale.python.org>
References: <E1RN8b2-0006kw-71@dinsdale.python.org>
Message-ID: <CAPZV6o-ehPDSw3SnasPdqgVN4X26+x9AjJ_2fYPpJF5ZcdXWcw@mail.gmail.com>

2011/11/6 brian.curtin <python-checkins at python.org>:
> -
> - ? ?if (!PyArg_ParseTuple(args, "O&O:utime",
> + ? ?PyObject* arg = NULL;

You could set arg = Py_None here.
> +
> + ? ?if (!PyArg_ParseTuple(args, "O&|O:utime",
> ? ? ? ? ? ? ? ? ? ? ? ? ? PyUnicode_FSConverter, &opath, &arg))
> ? ? ? ? return NULL;
> ? ? path = PyBytes_AsString(opath);
> - ? ?if (arg == Py_None) {
> + ? ?if (!arg || (arg == Py_None)) {

And then not have to change this.



-- 
Regards,
Benjamin

From python-checkins at python.org  Sun Nov  6 20:50:25 2011
From: python-checkins at python.org (brian.curtin)
Date: Sun, 06 Nov 2011 20:50:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Oops=2E_Update_a_placeholde?=
 =?utf8?q?r_comment_with_the_issue_number=2E?=
Message-ID: <E1RN8jZ-0007Qr-HE@dinsdale.python.org>

http://hg.python.org/cpython/rev/00d4d6b85a01
changeset:   73411:00d4d6b85a01
user:        Brian Curtin <brian at python.org>
date:        Sun Nov 06 13:50:15 2011 -0600
summary:
  Oops. Update a placeholder comment with the issue number.

files:
  Lib/test/test_os.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -271,7 +271,7 @@
         self.assertEqual(st2.st_mtime, int(st.st_mtime-delta))
 
     def test_utime_noargs(self):
-        # (insert issue#) removed the requirement to pass None as the
+        # Issue #13327 removed the requirement to pass None as the
         # second argument. Check that the previous methods of passing
         # a time tuple or None work in addition to no argument.
         st = os.stat(support.TESTFN)

-- 
Repository URL: http://hg.python.org/cpython

From brian.curtin at gmail.com  Sun Nov  6 20:54:11 2011
From: brian.curtin at gmail.com (Brian Curtin)
Date: Sun, 6 Nov 2011 13:54:11 -0600
Subject: [Python-checkins] cpython: Fix #13327. Remove the need for an
 explicit None as the second argument to
In-Reply-To: <CAPZV6o-ehPDSw3SnasPdqgVN4X26+x9AjJ_2fYPpJF5ZcdXWcw@mail.gmail.com>
References: <E1RN8b2-0006kw-71@dinsdale.python.org>
	<CAPZV6o-ehPDSw3SnasPdqgVN4X26+x9AjJ_2fYPpJF5ZcdXWcw@mail.gmail.com>
Message-ID: <CAD+XWwrj2j7MkGCmziSwgFL154CCj8eryYXz7v3dPr1ztTTa-g@mail.gmail.com>

On Sun, Nov 6, 2011 at 13:46, Benjamin Peterson <benjamin at python.org> wrote:
> 2011/11/6 brian.curtin <python-checkins at python.org>:
>> -
>> - ? ?if (!PyArg_ParseTuple(args, "O&O:utime",
>> + ? ?PyObject* arg = NULL;
>
> You could set arg = Py_None here.
>> +
>> + ? ?if (!PyArg_ParseTuple(args, "O&|O:utime",
>> ? ? ? ? ? ? ? ? ? ? ? ? ? PyUnicode_FSConverter, &opath, &arg))
>> ? ? ? ? return NULL;
>> ? ? path = PyBytes_AsString(opath);
>> - ? ?if (arg == Py_None) {
>> + ? ?if (!arg || (arg == Py_None)) {
>
> And then not have to change this.

Ah, good point. I'm going to be making this same change to the other
functions in utime family, so I'll look at updating this one and
change the others accordingly.

From python-checkins at python.org  Sun Nov  6 23:42:01 2011
From: python-checkins at python.org (vinay.sajip)
Date: Sun, 06 Nov 2011 23:42:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_issue_?=
 =?utf8?q?=2313353=3A_version_doumentation_about_utc_parameter_corrected?=
 =?utf8?q?=2E?=
Message-ID: <E1RNBPd-0002je-Er@dinsdale.python.org>

http://hg.python.org/cpython/rev/0b779988e8b7
changeset:   73412:0b779988e8b7
branch:      2.7
parent:      73407:19ffa12ffdd4
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Sun Nov 06 22:37:17 2011 +0000
summary:
  Closes issue #13353: version doumentation about utc parameter corrected.

files:
  Doc/library/logging.handlers.rst |  5 +----
  1 files changed, 1 insertions(+), 4 deletions(-)


diff --git a/Doc/library/logging.handlers.rst b/Doc/library/logging.handlers.rst
--- a/Doc/library/logging.handlers.rst
+++ b/Doc/library/logging.handlers.rst
@@ -262,10 +262,7 @@
    :meth:`emit`.
 
    .. versionchanged:: 2.6
-      *delay* was added.
-
-   .. versionchanged:: 2.7
-      *utc* was added.
+      *delay* and *utc* were added.
 
 
    .. method:: doRollover()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 01:38:12 2011
From: python-checkins at python.org (philip.jenvey)
Date: Mon, 07 Nov 2011 01:38:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_quote_the_type_name_for_imp?=
	=?utf8?q?roved_readability?=
Message-ID: <E1RNDE4-0001nt-40@dinsdale.python.org>

http://hg.python.org/cpython/rev/bbc929bc2224
changeset:   73413:bbc929bc2224
parent:      73411:00d4d6b85a01
user:        Philip Jenvey <pjenvey at underboss.org>
date:        Sun Nov 06 16:37:52 2011 -0800
summary:
  quote the type name for improved readability

files:
  Python/bltinmodule.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -1121,7 +1121,7 @@
         return NULL;
     if (!PyIter_Check(it)) {
         PyErr_Format(PyExc_TypeError,
-            "%.200s object is not an iterator",
+            "'%.200s' object is not an iterator",
             it->ob_type->tp_name);
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Mon Nov  7 05:33:43 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 07 Nov 2011 05:33:43 +0100
Subject: [Python-checkins] Daily reference leaks (bbc929bc2224): sum=0
Message-ID: <E1RNGtz-0003Aw-Vu@ap.vmr.nerim.net>

results for bbc929bc2224 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogrkRG0w', '-x']

From python-checkins at python.org  Mon Nov  7 09:55:02 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 09:55:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_=2313356?=
 =?utf8?q?=2E_Thanks_to_Florent_Xicluna_for_the_patch=2E?=
Message-ID: <E1RNKys-0002nx-Jl@dinsdale.python.org>

http://hg.python.org/cpython/rev/8726ad774cf0
changeset:   73414:8726ad774cf0
branch:      2.7
parent:      73412:0b779988e8b7
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 08:43:51 2011 +0000
summary:
  Closes #13356. Thanks to Florent Xicluna for the patch.

files:
  Lib/logging/__init__.py |  2 ++
  Lib/logging/config.py   |  5 +++--
  2 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1007,6 +1007,8 @@
         placeholder to now point to the logger.
         """
         rv = None
+        if isinstance(name, unicode):
+            name = name.encode('utf-8')
         _acquireLock()
         try:
             if name in self.loggerDict:
diff --git a/Lib/logging/config.py b/Lib/logging/config.py
--- a/Lib/logging/config.py
+++ b/Lib/logging/config.py
@@ -211,7 +211,7 @@
     #avoid disabling child loggers of explicitly
     #named loggers. With a sorted list it is easier
     #to find the child loggers.
-    existing.sort(key=_encoded)
+    existing.sort()
     #We'll keep the list of existing loggers
     #which are children of named loggers here...
     child_loggers = []
@@ -589,13 +589,14 @@
                 #avoid disabling child loggers of explicitly
                 #named loggers. With a sorted list it is easier
                 #to find the child loggers.
-                existing.sort(key=_encoded)
+                existing.sort()
                 #We'll keep the list of existing loggers
                 #which are children of named loggers here...
                 child_loggers = []
                 #now set up the new ones...
                 loggers = config.get('loggers', EMPTY_DICT)
                 for name in loggers:
+                    name = _encoded(name)
                     if name in existing:
                         i = existing.index(name)
                         prefixed = name + "."

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 09:55:04 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 09:55:04 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_=2313661?=
 =?utf8?q?=3A_Check_added_for_type_of_logger_name=2E?=
Message-ID: <E1RNKyu-0002q0-5p@dinsdale.python.org>

http://hg.python.org/cpython/rev/a3ba905447ba
changeset:   73416:a3ba905447ba
branch:      3.2
parent:      73408:ed0e85efac47
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 08:53:03 2011 +0000
summary:
  Closes #13661: Check added for type of logger name.

files:
  Lib/logging/__init__.py  |  2 ++
  Lib/test/test_logging.py |  2 ++
  2 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1094,6 +1094,8 @@
         placeholder to now point to the logger.
         """
         rv = None
+        if not isinstance(name, str):
+            raise ValueError('A logger name must be a string')
         _acquireLock()
         try:
             if name in self.loggerDict:
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -293,6 +293,8 @@
             ('INF.BADPARENT', 'INFO', '4'),
         ])
 
+    def test_invalid_name(self):
+        self.assertRaises(ValueError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 09:55:03 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 09:55:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_=2313661?=
 =?utf8?q?=3A_Check_added_for_type_of_logger_name=2E?=
Message-ID: <E1RNKyt-0002pI-DV@dinsdale.python.org>

http://hg.python.org/cpython/rev/5f3b7528b144
changeset:   73415:5f3b7528b144
branch:      2.7
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 08:49:16 2011 +0000
summary:
  Closes #13661: Check added for type of logger name.

files:
  Lib/logging/__init__.py  |  2 ++
  Lib/test/test_logging.py |  2 ++
  2 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1007,6 +1007,8 @@
         placeholder to now point to the logger.
         """
         rv = None
+        if not isinstance(name, basestring):
+            raise ValueError('A logger name must be string or Unicode')
         if isinstance(name, unicode):
             name = name.encode('utf-8')
         _acquireLock()
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -272,6 +272,8 @@
             ('INF.BADPARENT', 'INFO', '4'),
         ])
 
+    def test_invalid_name(self):
+        self.assertRaises(ValueError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 09:55:05 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 09:55:05 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merged_fix_for_=2313361_from_3=2E2=2E?=
Message-ID: <E1RNKyv-0002r7-2f@dinsdale.python.org>

http://hg.python.org/cpython/rev/8c719e106694
changeset:   73417:8c719e106694
parent:      73413:bbc929bc2224
parent:      73416:a3ba905447ba
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 08:53:58 2011 +0000
summary:
  Merged fix for #13361 from 3.2.

files:
  Lib/logging/__init__.py  |  2 ++
  Lib/test/test_logging.py |  2 ++
  2 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1096,6 +1096,8 @@
         placeholder to now point to the logger.
         """
         rv = None
+        if not isinstance(name, str):
+            raise ValueError('A logger name must be a string')
         _acquireLock()
         try:
             if name in self.loggerDict:
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -310,6 +310,8 @@
             ('INF.BADPARENT', 'INFO', '4'),
         ])
 
+    def test_invalid_name(self):
+        self.assertRaises(ValueError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 11:16:12 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 11:16:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_=2313361?=
 =?utf8?q?=3A_Raise_correct_exception_type=2E?=
Message-ID: <E1RNMFQ-0001BF-4w@dinsdale.python.org>

http://hg.python.org/cpython/rev/60dd1568bbd1
changeset:   73418:60dd1568bbd1
branch:      2.7
parent:      73415:5f3b7528b144
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 10:13:18 2011 +0000
summary:
  Closes #13361: Raise correct exception type.

files:
  Lib/logging/__init__.py  |  2 +-
  Lib/test/test_logging.py |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1008,7 +1008,7 @@
         """
         rv = None
         if not isinstance(name, basestring):
-            raise ValueError('A logger name must be string or Unicode')
+            raise TypeError('A logger name must be string or Unicode')
         if isinstance(name, unicode):
             name = name.encode('utf-8')
         _acquireLock()
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -273,7 +273,7 @@
         ])
 
     def test_invalid_name(self):
-        self.assertRaises(ValueError, logging.getLogger, any)
+        self.assertRaises(TypeError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 11:16:12 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 11:16:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_=2313361?=
 =?utf8?q?=3A_Raise_correct_exception_type=2E?=
Message-ID: <E1RNMFQ-0001BJ-V8@dinsdale.python.org>

http://hg.python.org/cpython/rev/bc05c11b340e
changeset:   73419:bc05c11b340e
branch:      3.2
parent:      73416:a3ba905447ba
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 10:15:08 2011 +0000
summary:
  Closes #13361: Raise correct exception type.

files:
  Lib/logging/__init__.py  |  2 +-
  Lib/test/test_logging.py |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1095,7 +1095,7 @@
         """
         rv = None
         if not isinstance(name, str):
-            raise ValueError('A logger name must be a string')
+            raise TypeError('A logger name must be a string')
         _acquireLock()
         try:
             if name in self.loggerDict:
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -294,7 +294,7 @@
         ])
 
     def test_invalid_name(self):
-        self.assertRaises(ValueError, logging.getLogger, any)
+        self.assertRaises(TypeError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 11:16:13 2011
From: python-checkins at python.org (vinay.sajip)
Date: Mon, 07 Nov 2011 11:16:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Closes_=2313361=3A_Merge_fix_from_3=2E2=2E?=
Message-ID: <E1RNMFR-0001BN-Ua@dinsdale.python.org>

http://hg.python.org/cpython/rev/fb73fe5d0ab1
changeset:   73420:fb73fe5d0ab1
parent:      73417:8c719e106694
parent:      73419:bc05c11b340e
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Mon Nov 07 10:15:55 2011 +0000
summary:
  Closes #13361: Merge fix from 3.2.

files:
  Lib/logging/__init__.py  |  2 +-
  Lib/test/test_logging.py |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -1097,7 +1097,7 @@
         """
         rv = None
         if not isinstance(name, str):
-            raise ValueError('A logger name must be a string')
+            raise TypeError('A logger name must be a string')
         _acquireLock()
         try:
             if name in self.loggerDict:
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -311,7 +311,7 @@
         ])
 
     def test_invalid_name(self):
-        self.assertRaises(ValueError, logging.getLogger, any)
+        self.assertRaises(TypeError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:38 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_obsolete/duplicate_e?=
	=?utf8?q?ntries?=
Message-ID: <E1RNSjS-0007Nc-OD@dinsdale.python.org>

http://hg.python.org/cpython/rev/ff7d6fd53132
changeset:   73421:ff7d6fd53132
parent:      73376:4eee9dd61147
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 05:35:32 2011 +0100
summary:
  Remove obsolete/duplicate entries

files:
  Misc/NEWS |  4 ----
  1 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -283,8 +283,6 @@
 - Issue #11320: fix bogus memory management in Modules/getpath.c, leading to
   a possible crash when calling Py_SetPath().
 
-- _ast.__version__ is now a Mercurial hex revision.
-
 - Issue #11432: A bug was introduced in subprocess.Popen on posix systems with
   3.2.0 where the stdout or stderr file descriptor being the same as the stdin
   file descriptor would raise an exception. webbrowser.open would fail. fixed.
@@ -1542,8 +1540,6 @@
   signature.  Without this, architectures where sizeof void* != sizeof int are
   broken.  Patch given by Hallvard B Furuseth.
 
-- Issue #12221: Replace pyexpat.__version__ with the Python version.
-
 - Issue #12051: Fix segfault in json.dumps() while encoding highly-nested
   objects using the C accelerations.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:39 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_sure_packaging_tests_t?=
 =?utf8?q?hat_register_custom_commands_also_clear_them?=
Message-ID: <E1RNSjT-0007Nn-IN@dinsdale.python.org>

http://hg.python.org/cpython/rev/7145ef38ca9d
changeset:   73422:7145ef38ca9d
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 07:01:18 2011 +0100
summary:
  Make sure packaging tests that register custom commands also clear them

files:
  Lib/packaging/tests/support.py     |  15 +++++++++-
  Lib/packaging/tests/test_config.py |  11 ++++--
  Lib/packaging/tests/test_dist.py   |  27 ++++++++---------
  Lib/test/regrtest.py               |  19 ++++++++++++-
  4 files changed, 52 insertions(+), 20 deletions(-)


diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -41,6 +41,9 @@
 import sysconfig
 
 from packaging.dist import Distribution
+from packaging.util import resolve_name
+from packaging.command import set_command, _COMMANDS
+
 from packaging.tests import unittest
 from test.support import requires_zlib, unlink
 
@@ -51,7 +54,8 @@
     # mocks
     'DummyCommand', 'TestDistribution',
     # misc. functions and decorators
-    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    'fake_dec', 'create_distribution', 'use_command',
+    'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
     'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
@@ -280,6 +284,15 @@
     return d
 
 
+def use_command(testcase, fullname):
+    """Register command at *fullname* for the duration of a test."""
+    set_command(fullname)
+    # XXX maybe set_command should return the class object
+    name = resolve_name(fullname).get_command_name()
+    # XXX maybe we need a public API to remove commands
+    testcase.addCleanup(_COMMANDS.__delitem__, name)
+
+
 def fake_dec(*args, **kw):
     """Fake decorator"""
     def _wrap(func):
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -183,13 +183,14 @@
 
     def __init__(self, dist):
         self.distribution = dist
+        self._record = []
 
     @classmethod
     def get_command_name(cls):
         return 'foo'
 
     def run(self):
-        self.distribution.foo_was_here = True
+        self._record.append('foo has run')
 
     def nothing(self):
         pass
@@ -491,10 +492,12 @@
             self.write_file((pkg, '__init__.py'), '#')
 
         # try to run the install command to see if foo is called
+        self.addCleanup(command._COMMANDS.__delitem__, 'foo')
         dist = self.get_dist()
-        self.assertIn('foo', command.get_command_names())
-        self.assertEqual('FooBarBazTest',
-                         dist.get_command_obj('foo').__class__.__name__)
+        dist.run_command('install_dist')
+        cmd = dist.get_command_obj('foo')
+        self.assertEqual(cmd.__class__.__name__, 'FooBarBazTest')
+        self.assertEqual(cmd._record, ['foo has run'])
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -6,30 +6,32 @@
 import packaging.dist
 
 from packaging.dist import Distribution
-from packaging.command import set_command, _COMMANDS
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
 from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
-from packaging.tests.support import create_distribution
+from packaging.tests.support import create_distribution, use_command
 from test.support import unload
 
 
 class test_dist(Command):
-    """Sample packaging extension command."""
+    """Custom command used for testing."""
 
     user_options = [
-        ("sample-option=", "S", "help text"),
+        ('sample-option=', 'S',
+         "help text"),
         ]
 
     def initialize_options(self):
         self.sample_option = None
+        self._record = []
 
     def finalize_options(self):
-        pass
+        if self.sample_option is None:
+            self.sample_option = 'default value'
 
     def run(self):
-        pass
+        self._record.append('test_dist has run')
 
 
 class DistributionTestCase(support.TempdirManager,
@@ -45,14 +47,10 @@
         # (defaulting to sys.argv)
         self.argv = sys.argv, sys.argv[:]
         del sys.argv[1:]
-        self._commands = _COMMANDS.copy()
 
     def tearDown(self):
         sys.argv = self.argv[0]
         sys.argv[:] = self.argv[1]
-        # XXX maybe we need a public API to remove commands
-        _COMMANDS.clear()
-        _COMMANDS.update(self._commands)
         super(DistributionTestCase, self).tearDown()
 
     @unittest.skip('needs to be updated')
@@ -181,7 +179,8 @@
         self.write_file((temp_home, "config2.cfg"),
                         '[test_dist]\npre-hook.b = type')
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
+
         dist = create_distribution(config_files)
         cmd = dist.get_command_obj("test_dist")
         self.assertEqual(cmd.pre_hook, {"a": 'type', "b": 'type'})
@@ -209,7 +208,7 @@
                 record.append('post-%s' % cmd.get_command_name())
             '''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
 
@@ -236,7 +235,7 @@
             [test_dist]
             pre-hook.test = nonexistent.dotted.name'''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
@@ -251,7 +250,7 @@
             [test_dist]
             pre-hook.test = packaging.tests.test_dist.__doc__'''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -172,6 +172,7 @@
 import json
 import logging
 import os
+import packaging.command
 import packaging.database
 import platform
 import random
@@ -967,7 +968,7 @@
                  'sys.warnoptions', 'threading._dangling',
                  'multiprocessing.process._dangling',
                  'sysconfig._CONFIG_VARS', 'sysconfig._SCHEMES',
-                 'packaging.database_caches',
+                 'packaging.command._COMMANDS', 'packaging.database_caches',
                 )
 
     def get_sys_argv(self):
@@ -1055,6 +1056,22 @@
         # Can't easily revert the logging state
         pass
 
+    def get_packaging_command__COMMANDS(self):
+        # registry mapping command names to full dotted path or to the actual
+        # class (resolved on demand); this check only looks at the names, not
+        # the types of the values (IOW, if a value changes from a string
+        # (dotted path) to a class it's okay but if a key (i.e. command class)
+        # is added we complain)
+        id_ = id(packaging.command._COMMANDS)
+        keys = set(packaging.command._COMMANDS)
+        return id_, keys
+    def restore_packaging_command__COMMANDS(self, saved):
+        # if command._COMMANDS was bound to another dict obhect, we can't
+        # restore the previous object and contents, because the get_ method
+        # above does not return the dict object (to ignore changes in values)
+        for key in packaging.command._COMMANDS.keys() - saved[1]:
+            del packaging.command._COMMANDS[key]
+
     def get_packaging_database_caches(self):
         # caching system used by the PEP 376 implementation
         # we have one boolean and four dictionaries, initially empty

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:40 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Undo_potentially_confusing_?=
 =?utf8?q?name_change_in_packaging=2E?=
Message-ID: <E1RNSjU-0007Ns-Bp@dinsdale.python.org>

http://hg.python.org/cpython/rev/540e278ab646
changeset:   73423:540e278ab646
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 06:54:05 2011 +0100
summary:
  Undo potentially confusing name change in packaging.

This method was named reinitialize_command in distutils and accompanied
by a comment suggesting to change it to get_reinitialized_command.
Following that, I did the change for distutils2, but it proved
confusing: The Distribution object has an internal cache of command
objects, to make sure only one instance is ever used, and the name
get_reinitialized_command could suggest that the object returned was
independent of that cache, which it was not.  I?m reverting the name
change to make code clearer.

files:
  Lib/packaging/command/bdist.py         |   2 +-
  Lib/packaging/command/bdist_dumb.py    |   4 ++--
  Lib/packaging/command/bdist_msi.py     |   6 +++---
  Lib/packaging/command/bdist_wininst.py |   5 ++---
  Lib/packaging/command/cmd.py           |   4 ++--
  Lib/packaging/command/test.py          |   2 +-
  Lib/packaging/dist.py                  |  13 +++++++------
  Lib/packaging/tests/support.py         |   2 +-
  8 files changed, 19 insertions(+), 19 deletions(-)


diff --git a/Lib/packaging/command/bdist.py b/Lib/packaging/command/bdist.py
--- a/Lib/packaging/command/bdist.py
+++ b/Lib/packaging/command/bdist.py
@@ -126,7 +126,7 @@
         # Reinitialize and run each command.
         for i in range(len(self.formats)):
             cmd_name = commands[i]
-            sub_cmd = self.get_reinitialized_command(cmd_name)
+            sub_cmd = self.reinitialize_command(cmd_name)
             sub_cmd.format = self.formats[i]
 
             # passing the owner and group names for tar archiving
diff --git a/Lib/packaging/command/bdist_dumb.py b/Lib/packaging/command/bdist_dumb.py
--- a/Lib/packaging/command/bdist_dumb.py
+++ b/Lib/packaging/command/bdist_dumb.py
@@ -80,8 +80,8 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
diff --git a/Lib/packaging/command/bdist_msi.py b/Lib/packaging/command/bdist_msi.py
--- a/Lib/packaging/command/bdist_msi.py
+++ b/Lib/packaging/command/bdist_msi.py
@@ -183,13 +183,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.prefix = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/Lib/packaging/command/bdist_wininst.py b/Lib/packaging/command/bdist_wininst.py
--- a/Lib/packaging/command/bdist_wininst.py
+++ b/Lib/packaging/command/bdist_wininst.py
@@ -115,14 +115,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install', reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
         install.plat_name = self.plat_name
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/Lib/packaging/command/cmd.py b/Lib/packaging/command/cmd.py
--- a/Lib/packaging/command/cmd.py
+++ b/Lib/packaging/command/cmd.py
@@ -318,8 +318,8 @@
         cmd_obj.ensure_finalized()
         return cmd_obj
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
-        return self.distribution.get_reinitialized_command(
+    def reinitialize_command(self, command, reinit_subcommands=False):
+        return self.distribution.reinitialize_command(
             command, reinit_subcommands)
 
     def run_command(self, command):
diff --git a/Lib/packaging/command/test.py b/Lib/packaging/command/test.py
--- a/Lib/packaging/command/test.py
+++ b/Lib/packaging/command/test.py
@@ -56,7 +56,7 @@
         prev_syspath = sys.path[:]
         try:
             # build release
-            build = self.get_reinitialized_command('build')
+            build = self.reinitialize_command('build')
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -636,9 +636,9 @@
             except ValueError as msg:
                 raise PackagingOptionError(msg)
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
+    def reinitialize_command(self, command, reinit_subcommands=False):
         """Reinitializes a command to the state it was in when first
-        returned by 'get_command_obj()': ie., initialized but not yet
+        returned by 'get_command_obj()': i.e., initialized but not yet
         finalized.  This provides the opportunity to sneak option
         values in programmatically, overriding or supplementing
         user-supplied values from the config files and command line.
@@ -650,10 +650,11 @@
         'reinit_subcommands' is true, also reinitializes the command's
         sub-commands, as declared by the 'sub_commands' class attribute (if
         it has one).  See the "install_dist" command for an example.  Only
-        reinitializes the sub-commands that actually matter, ie. those
-        whose test predicates return true.
+        reinitializes the sub-commands that actually matter, i.e. those
+        whose test predicate return true.
 
-        Returns the reinitialized command object.
+        Returns the reinitialized command object.  It will be the same
+        object as the one stored in the self.command_obj attribute.
         """
         if not isinstance(command, Command):
             command_name = command
@@ -671,7 +672,7 @@
 
         if reinit_subcommands:
             for sub in command.get_sub_commands():
-                self.get_reinitialized_command(sub, reinit_subcommands)
+                self.reinitialize_command(sub, reinit_subcommands)
 
         return command
 
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -251,7 +251,7 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
-    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # XXX does not work with dist.reinitialize_command, which typechecks
     # and wants a finalized attribute
 
     def __init__(self, **kwargs):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:41 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:41 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Minor_tweak_to_packaging_te?=
	=?utf8?q?sts=2E?=
Message-ID: <E1RNSjV-0007Nw-4v@dinsdale.python.org>

http://hg.python.org/cpython/rev/01f3cd004860
changeset:   73424:01f3cd004860
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 10:48:55 2011 +0100
summary:
  Minor tweak to packaging tests.

When an option is changed on a command object, calling ensure_finalized
for a second time will not run finalize_options again, because
ensure_finalized is a no-op the second time.  By resetting the finalized
attribute, we can be sure that whatever computation takes place in
finalize_options will happen again.

(In test_command_clean, I removed two lines that were a no-op.)

files:
  Lib/packaging/tests/test_command_clean.py        |  2 --
  Lib/packaging/tests/test_command_install_data.py |  2 ++
  Lib/packaging/tests/test_command_register.py     |  1 +
  Lib/packaging/tests/test_command_sdist.py        |  2 +-
  4 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/Lib/packaging/tests/test_command_clean.py b/Lib/packaging/tests/test_command_clean.py
--- a/Lib/packaging/tests/test_command_clean.py
+++ b/Lib/packaging/tests/test_command_clean.py
@@ -36,8 +36,6 @@
                              '%r was not removed' % path)
 
         # let's run the command again (should spit warnings but succeed)
-        cmd.all = True
-        cmd.ensure_finalized()
         cmd.run()
 
 
diff --git a/Lib/packaging/tests/test_command_install_data.py b/Lib/packaging/tests/test_command_install_data.py
--- a/Lib/packaging/tests/test_command_install_data.py
+++ b/Lib/packaging/tests/test_command_install_data.py
@@ -62,6 +62,7 @@
 
         # let's try with warn_dir one
         cmd.warn_dir = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -80,6 +81,7 @@
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_register.py b/Lib/packaging/tests/test_command_register.py
--- a/Lib/packaging/tests/test_command_register.py
+++ b/Lib/packaging/tests/test_command_register.py
@@ -143,6 +143,7 @@
 
         register_module.input = _no_way
         cmd.show_response = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -140,7 +140,7 @@
 
         # now trying a tar then a gztar
         cmd.formats = ['tar', 'gztar']
-
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:41 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:41 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Clean_up_mocking_of_stdout_?=
 =?utf8?q?and_stdin_in_packaging_tests=2E?=
Message-ID: <E1RNSjV-0007O0-OG@dinsdale.python.org>

http://hg.python.org/cpython/rev/547756d9a013
changeset:   73425:547756d9a013
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 11:32:47 2011 +0100
summary:
  Clean up mocking of stdout and stdin in packaging tests.

Running with regrtest does not show spurious output or unrestored
sys.std* objects; sometimes running with make test is different, I?ll
watch the buildbots.

In addition, update the create module to use logging.

files:
  Lib/packaging/create.py                       |  29 ++++----
  Lib/packaging/tests/support.py                |  18 +++++-
  Lib/packaging/tests/test_command_build_ext.py |  32 ++-------
  Lib/packaging/tests/test_command_build_py.py  |  26 ++-----
  Lib/packaging/tests/test_command_register.py  |  14 +----
  Lib/packaging/tests/test_config.py            |  13 +---
  Lib/packaging/tests/test_create.py            |  31 +++------
  Lib/packaging/tests/test_run.py               |   2 -
  Lib/packaging/tests/test_uninstall.py         |   5 -
  Lib/packaging/tests/test_util.py              |   4 +-
  10 files changed, 61 insertions(+), 113 deletions(-)


diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -30,6 +30,7 @@
 from tokenize import detect_encoding
 from configparser import RawConfigParser
 
+from packaging import logger
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from packaging._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -124,7 +125,7 @@
         if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
-        print('\nERROR: You must select "Y" or "N".\n')
+        logger.error('You must select "Y" or "N".')
 
 
 # XXX use util.ask
@@ -147,10 +148,7 @@
     helptext = helptext.strip("\n")
 
     while True:
-        sys.stdout.write(prompt)
-        sys.stdout.flush()
-
-        line = sys.stdin.readline().strip()
+        line = input(prompt).strip()
         if line == '?':
             print('=' * 70)
             print(helptext)
@@ -271,9 +269,10 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print("ERROR: %(name)s.old backup exists, please check that "
-                      "current %(name)s is correct and remove %(name)s.old" %
-                      {'name': _FILENAME})
+                message = ("ERROR: %(name)s.old backup exists, please check "
+                           "that current %(name)s is correct and remove "
+                           "%(name)s.old" % {'name': _FILENAME})
+                logger.error(message)
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -320,7 +319,7 @@
             fp.write('\n')
 
         os.chmod(_FILENAME, 0o644)
-        print('Wrote "%s".' % _FILENAME)
+        logger.info('Wrote "%s".' % _FILENAME)
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -614,8 +613,8 @@
                         break
 
             if len(found_list) == 0:
-                print('ERROR: Could not find a matching license for "%s"' %
-                      license)
+                logger.error('Could not find a matching license for "%s"' %
+                             license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -636,8 +635,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print("ERROR: Invalid selection, type a number from the list "
-                      "above.")
+                logger.error(
+                    "Invalid selection, type a number from the list above.")
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -660,8 +659,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print("ERROR: Invalid selection, type a single digit "
-                          "number.")
+                    logger.error(
+                        "Invalid selection, type a single digit number.")
 
 
 def main():
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -52,7 +52,7 @@
     # TestCase mixins
     'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
     # mocks
-    'DummyCommand', 'TestDistribution',
+    'DummyCommand', 'TestDistribution', 'Inputs',
     # misc. functions and decorators
     'fake_dec', 'create_distribution', 'use_command',
     'copy_xxmodule_c', 'fixup_build_ext',
@@ -274,6 +274,22 @@
         return self._config_files
 
 
+class Inputs:
+    """Fakes user inputs."""
+    # TODO document usage
+    # TODO use context manager or something for auto cleanup
+
+    def __init__(self, *answers):
+        self.answers = answers
+        self.index = 0
+
+    def __call__(self, prompt=''):
+        try:
+            return self.answers[self.index]
+        finally:
+            self.index += 1
+
+
 def create_distribution(configfiles=()):
     """Prepares a distribution with given config files parsed."""
     d = TestDistribution()
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -3,7 +3,6 @@
 import site
 import sysconfig
 import textwrap
-from io import StringIO
 from packaging.dist import Distribution
 from packaging.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
@@ -11,7 +10,7 @@
 from packaging.compiler.extension import Extension
 
 from test.script_helper import assert_python_ok
-from packaging.tests import support, unittest, verbose
+from packaging.tests import support, unittest
 
 
 class BuildExtTestCase(support.TempdirManager,
@@ -37,18 +36,10 @@
         support.fixup_build_ext(cmd)
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
+        cmd.ensure_finalized()
+        cmd.run()
 
-        old_stdout = sys.stdout
-        if not verbose:
-            # silence compiler output
-            sys.stdout = StringIO()
-        try:
-            cmd.ensure_finalized()
-            cmd.run()
-        finally:
-            sys.stdout = old_stdout
-
-        code = """if 1:
+        code = textwrap.dedent("""\
             import sys
             sys.path.insert(0, %r)
 
@@ -63,7 +54,8 @@
             doc = 'This is a template module just for instruction.'
             assert xx.__doc__ == doc
             assert isinstance(xx.Null(), xx.Null)
-            assert isinstance(xx.Str(), xx.Str)"""
+            assert isinstance(xx.Str(), xx.Str)
+            """)
         code = code % self.tmp_dir
         assert_python_ok('-c', code)
 
@@ -388,16 +380,8 @@
         cmd.build_temp = self.tmp_dir
 
         try:
-            old_stdout = sys.stdout
-            if not verbose:
-                # silence compiler output
-                sys.stdout = StringIO()
-            try:
-                cmd.ensure_finalized()
-                cmd.run()
-            finally:
-                sys.stdout = old_stdout
-
+            cmd.ensure_finalized()
+            cmd.run()
         except CompileError:
             self.fail("Wrong deployment target during compilation")
 
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -67,8 +67,6 @@
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
         # create the distribution files.
         sources = self.mkdtemp()
         pkg = os.path.join(sources, 'pkg')
@@ -79,24 +77,16 @@
         open(os.path.join(testdir, "testfile"), "wb").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        #sys.stdout = StringIO.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": sources,
+                             "package_data": {"pkg": ["doc/*"]}})
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": sources,
-                                 "package_data": {"pkg": ["doc/*"]}})
-            dist.script_args = ["build"]
-            dist.parse_command_line()
-
-            try:
-                dist.run_commands()
-            except PackagingFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+            dist.run_commands()
+        except PackagingFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
     def test_byte_compile(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
diff --git a/Lib/packaging/tests/test_command_register.py b/Lib/packaging/tests/test_command_register.py
--- a/Lib/packaging/tests/test_command_register.py
+++ b/Lib/packaging/tests/test_command_register.py
@@ -12,6 +12,7 @@
     DOCUTILS_SUPPORT = False
 
 from packaging.tests import unittest, support
+from packaging.tests.support import Inputs
 from packaging.command import register as register_module
 from packaging.command.register import register
 from packaging.errors import PackagingSetupError
@@ -38,19 +39,6 @@
 """
 
 
-class Inputs:
-    """Fakes user inputs."""
-    def __init__(self, *answers):
-        self.answers = answers
-        self.index = 0
-
-    def __call__(self, prompt=''):
-        try:
-            return self.answers[self.index]
-        finally:
-            self.index += 1
-
-
 class FakeOpener:
     """Fakes a PyPI server"""
     def __init__(self):
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -1,7 +1,6 @@
 """Tests for packaging.config."""
 import os
 import sys
-from io import StringIO
 
 from packaging import command
 from packaging.dist import Distribution
@@ -210,21 +209,11 @@
 
     def setUp(self):
         super(ConfigTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        sys.stdout = StringIO()
-        sys.stderr = StringIO()
-
-        self.addCleanup(os.chdir, os.getcwd())
         tempdir = self.mkdtemp()
         self.working_dir = os.getcwd()
         os.chdir(tempdir)
         self.tempdir = tempdir
 
-    def tearDown(self):
-        os.chdir(self.working_dir)
-        super(ConfigTestCase, self).tearDown()
-
     def write_setup(self, kwargs=None):
         opts = {'description-file': 'README', 'extra-files': '',
                 'setup-hooks': 'packaging.tests.test_config.version_hook'}
@@ -379,7 +368,7 @@
         self.assertIn('hooks', sys.modules)
 
     def test_missing_setup_hook_warns(self):
-        self.write_setup({'setup-hooks': 'this.does._not.exist'})
+        self.write_setup({'setup-hooks': 'does._not.exist'})
         self.write_file('README', 'yeah')
         self.get_dist()
         logs = self.get_logs()
diff --git a/Lib/packaging/tests/test_create.py b/Lib/packaging/tests/test_create.py
--- a/Lib/packaging/tests/test_create.py
+++ b/Lib/packaging/tests/test_create.py
@@ -2,15 +2,17 @@
 import os
 import sys
 import sysconfig
-from io import StringIO
 from textwrap import dedent
+from packaging import create
 from packaging.create import MainProgram, ask_yn, ask, main
 
 from packaging.tests import support, unittest
+from packaging.tests.support import Inputs
 
 
 class CreateTestCase(support.TempdirManager,
                      support.EnvironRestorer,
+                     support.LoggingCatcher,
                      unittest.TestCase):
 
     maxDiff = None
@@ -18,11 +20,6 @@
 
     def setUp(self):
         super(CreateTestCase, self).setUp()
-        self._stdin = sys.stdin  # TODO use Inputs
-        self._stdout = sys.stdout
-        sys.stdin = StringIO()
-        sys.stdout = StringIO()
-        self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
         # patch sysconfig
@@ -32,29 +29,24 @@
             'doc': sys.prefix + '/share/doc/pyxfoil', }
 
     def tearDown(self):
-        sys.stdin = self._stdin
-        sys.stdout = self._stdout
-        os.chdir(self._cwd)
         sysconfig.get_paths = self._old_get_paths
+        if hasattr(create, 'input'):
+            del create.input
         super(CreateTestCase, self).tearDown()
 
     def test_ask_yn(self):
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         self.assertEqual('y', ask_yn('is this a test'))
 
     def test_ask(self):
-        sys.stdin.write('a\n')
-        sys.stdin.write('b\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('a', 'b')
         self.assertEqual('a', ask('is this a test'))
         self.assertEqual('b', ask(str(list(range(0, 70))), default='c',
                                   lengthy=True))
 
     def test_set_multi(self):
         mainprogram = MainProgram()
-        sys.stdin.write('aaaaa\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('aaaaa')
         mainprogram.data['author'] = []
         mainprogram._set_multi('_set_multi test', 'author')
         self.assertEqual(['aaaaa'], mainprogram.data['author'])
@@ -130,8 +122,7 @@
               scripts=['my_script', 'bin/run'],
               )
         """), encoding='utf-8')
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
@@ -206,9 +197,7 @@
 barbar is now in the public domain,
 ho, baby!
                         '''))
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
-        # FIXME Out of memory error.
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -33,11 +33,9 @@
 
     def setUp(self):
         super(RunTestCase, self).setUp()
-        self.old_stdout = sys.stdout
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
-        sys.stdout = self.old_stdout
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
diff --git a/Lib/packaging/tests/test_uninstall.py b/Lib/packaging/tests/test_uninstall.py
--- a/Lib/packaging/tests/test_uninstall.py
+++ b/Lib/packaging/tests/test_uninstall.py
@@ -1,6 +1,5 @@
 """Tests for the packaging.uninstall module."""
 import os
-import sys
 import logging
 import packaging.util
 
@@ -31,16 +30,12 @@
 
     def setUp(self):
         super(UninstallTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        self.addCleanup(os.chdir, os.getcwd())
         self.addCleanup(enable_cache)
         self.root_dir = self.mkdtemp()
         self.cwd = os.getcwd()
         disable_cache()
 
     def tearDown(self):
-        os.chdir(self.cwd)
         packaging.util._path_created.clear()
         super(UninstallTestCase, self).tearDown()
 
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -170,8 +170,8 @@
     def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
-        sys.old_stdout = self.old_stdout
-        sys.old_stderr = self.old_stderr
+        sys.stdout = self.old_stdout
+        sys.stderr = self.old_stderr
 
     def test_convert_path(self):
         # linux/mac

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:42 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:42 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_redundant_=5F=5Fmain?=
 =?utf8?q?=5F=5F_blocks_from_packaging_modules=2E?=
Message-ID: <E1RNSjW-0007O4-Fe@dinsdale.python.org>

http://hg.python.org/cpython/rev/31ab61957d78
changeset:   73426:31ab61957d78
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 11:38:58 2011 +0100
summary:
  Remove redundant __main__ blocks from packaging modules.

The one interface we commit to maintain is the run module
(a.k.a. the pysetup script).

files:
  Lib/packaging/create.py   |  4 ----
  Lib/packaging/depgraph.py |  5 +----
  Lib/packaging/install.py  |  9 ---------
  3 files changed, 1 insertions(+), 17 deletions(-)


diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -674,7 +674,3 @@
     # program.write_setup_script()
     # packaging.util.cfg_to_args()
     program()
-
-
-if __name__ == '__main__':
-    main()
diff --git a/Lib/packaging/depgraph.py b/Lib/packaging/depgraph.py
--- a/Lib/packaging/depgraph.py
+++ b/Lib/packaging/depgraph.py
@@ -224,6 +224,7 @@
 
 
 def main():
+    # XXX move to run._graph
     from packaging.database import get_distributions
     tempout = StringIO()
     try:
@@ -267,7 +268,3 @@
     else:
         print('Supported option: -d [filename]')
         sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()
diff --git a/Lib/packaging/install.py b/Lib/packaging/install.py
--- a/Lib/packaging/install.py
+++ b/Lib/packaging/install.py
@@ -527,12 +527,3 @@
             logger.info('%r conflicts with %s', project, ','.join(projects))
 
     return True
-
-
-def _main(**attrs):
-    if 'script_args' not in attrs:
-        attrs['requirements'] = sys.argv[1]
-    get_infos(**attrs)
-
-if __name__ == '__main__':
-    _main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:43 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_more_standard_name_for_?=
 =?utf8?q?one_option_of_packaging=E2=80=99s_install=5Fdistinfo?=
Message-ID: <E1RNSjX-0007Oa-2N@dinsdale.python.org>

http://hg.python.org/cpython/rev/bc2eca77acbc
changeset:   73427:bc2eca77acbc
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 06 11:52:30 2011 +0100
summary:
  Use more standard name for one option of packaging?s install_distinfo

files:
  Lib/packaging/command/install_dist.py                |   4 +-
  Lib/packaging/command/install_distinfo.py            |  25 ++++-----
  Lib/packaging/tests/test_command_install_distinfo.py |  10 ++--
  3 files changed, 18 insertions(+), 21 deletions(-)


diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -55,9 +55,7 @@
         ('install-data=', None,
          "installation directory for data files"),
 
-        # Byte-compilation options -- see install_lib.py for details, as
-        # these are duplicated from there (but only install_lib does
-        # anything with them).
+        # Byte-compilation options -- see install_lib for details
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
         ('optimize=', 'O',
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -16,8 +16,8 @@
     description = 'create a .dist-info directory for the distribution'
 
     user_options = [
-        ('distinfo-dir=', None,
-         "directory where the the .dist-info directory will be installed"),
+        ('install-dir=', None,
+         "directory where the the .dist-info directory will be created"),
         ('installer=', None,
          "the name of the installer"),
         ('requested', None,
@@ -35,7 +35,7 @@
     negative_opt = {'no-requested': 'requested'}
 
     def initialize_options(self):
-        self.distinfo_dir = None
+        self.install_dir = None
         self.installer = None
         self.requested = None
         self.no_record = None
@@ -46,8 +46,7 @@
         self.set_undefined_options('install_dist',
                                    'installer', 'requested', 'no_record')
 
-        self.set_undefined_options('install_lib',
-                                   ('install_dir', 'distinfo_dir'))
+        self.set_undefined_options('install_lib', 'install_dir')
 
         if self.installer is None:
             # FIXME distutils or packaging?
@@ -64,26 +63,26 @@
 
         basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
-        self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
+        self.install_dir = os.path.join(self.install_dir, basename)
 
     def run(self):
-        target = self.distinfo_dir
+        target = self.install_dir
 
         if os.path.isdir(target) and not os.path.islink(target):
             if not self.dry_run:
                 rmtree(target)
         elif os.path.exists(target):
-            self.execute(os.unlink, (self.distinfo_dir,),
+            self.execute(os.unlink, (self.install_dir,),
                          "removing " + target)
 
         self.execute(os.makedirs, (target,), "creating " + target)
 
-        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        metadata_path = os.path.join(self.install_dir, 'METADATA')
         self.execute(self.distribution.metadata.write, (metadata_path,),
                      "creating " + metadata_path)
         self.outfiles.append(metadata_path)
 
-        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        installer_path = os.path.join(self.install_dir, 'INSTALLER')
         logger.info('creating %s', installer_path)
         if not self.dry_run:
             with open(installer_path, 'w') as f:
@@ -91,7 +90,7 @@
         self.outfiles.append(installer_path)
 
         if self.requested:
-            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            requested_path = os.path.join(self.install_dir, 'REQUESTED')
             logger.info('creating %s', requested_path)
             if not self.dry_run:
                 open(requested_path, 'wb').close()
@@ -100,7 +99,7 @@
         if not self.no_resources:
             install_data = self.get_finalized_command('install_data')
             if install_data.get_resources_out() != []:
-                resources_path = os.path.join(self.distinfo_dir,
+                resources_path = os.path.join(self.install_dir,
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
@@ -114,7 +113,7 @@
                 self.outfiles.append(resources_path)
 
         if not self.no_record:
-            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            record_path = os.path.join(self.install_dir, 'RECORD')
             logger.info('creating %s', record_path)
             if not self.dry_run:
                 with open(record_path, 'w', encoding='utf-8') as f:
diff --git a/Lib/packaging/tests/test_command_install_distinfo.py b/Lib/packaging/tests/test_command_install_distinfo.py
--- a/Lib/packaging/tests/test_command_install_distinfo.py
+++ b/Lib/packaging/tests/test_command_install_distinfo.py
@@ -49,7 +49,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
@@ -76,7 +76,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.installer = 'bacon-python'
         cmd.ensure_finalized()
         cmd.run()
@@ -96,7 +96,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.requested = False
         cmd.ensure_finalized()
         cmd.run()
@@ -116,7 +116,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.no_record = True
         cmd.ensure_finalized()
         cmd.run()
@@ -214,7 +214,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:43 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_unnecessary_version_?=
	=?utf8?q?check?=
Message-ID: <E1RNSjX-0007Oe-Qu@dinsdale.python.org>

http://hg.python.org/cpython/rev/4891e319899f
changeset:   73428:4891e319899f
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 07 09:18:30 2011 +0100
summary:
  Remove unnecessary version check

files:
  Lib/sysconfig.py |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -488,8 +488,7 @@
         # Setting 'userbase' is done below the call to the
         # init function to enable using 'get_config_var' in
         # the init-function.
-        if sys.version >= '2.6':
-            _CONFIG_VARS['userbase'] = _getuserbase()
+        _CONFIG_VARS['userbase'] = _getuserbase()
 
         if 'srcdir' not in _CONFIG_VARS:
             _CONFIG_VARS['srcdir'] = _PROJECT_BASE

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:44 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:44 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Document_that_s?=
 =?utf8?q?hutil=2Emake=5Farchive_does_not_typecheck_its_logger_argument?=
Message-ID: <E1RNSjY-0007Oi-Vj@dinsdale.python.org>

http://hg.python.org/cpython/rev/0e5d82c86cd7
changeset:   73429:0e5d82c86cd7
branch:      3.2
parent:      73419:bc05c11b340e
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 07 17:31:07 2011 +0100
summary:
  Document that shutil.make_archive does not typecheck its logger argument

files:
  Doc/library/shutil.rst |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -269,7 +269,8 @@
    *owner* and *group* are used when creating a tar archive. By default,
    uses the current owner and group.
 
-   *logger* is an instance of :class:`logging.Logger`.
+   *logger* must be an object compatible with :pep:`282`, usually an instance of
+   :class:`logging.Logger`.
 
    .. versionadded:: 3.2
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:46 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:46 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogRG9u4oCZdCBpbnRl?=
 =?utf8?q?rpret_backslashes_in_ASCII_diagram_in_a_docstring?=
Message-ID: <E1RNSja-0007PE-FN@dinsdale.python.org>

http://hg.python.org/cpython/rev/4facbfdc7700
changeset:   73430:4facbfdc7700
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 07 17:52:48 2011 +0100
summary:
  Don?t interpret backslashes in ASCII diagram in a docstring

files:
  Lib/http/cookiejar.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py
--- a/Lib/http/cookiejar.py
+++ b/Lib/http/cookiejar.py
@@ -1,4 +1,4 @@
-"""HTTP cookie handling for web clients.
+r"""HTTP cookie handling for web clients.
 
 This module has (now fairly distant) origins in Gisle Aas' Perl module
 HTTP::Cookies, from the libwww-perl library.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:47 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1RNSjb-0007PI-DD@dinsdale.python.org>

http://hg.python.org/cpython/rev/78532ab28271
changeset:   73431:78532ab28271
parent:      73420:fb73fe5d0ab1
parent:      73428:4891e319899f
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 07 18:11:12 2011 +0100
summary:
  Branch merge

files:
  Lib/packaging/command/bdist.py                       |   2 +-
  Lib/packaging/command/bdist_dumb.py                  |   4 +-
  Lib/packaging/command/bdist_msi.py                   |   6 +-
  Lib/packaging/command/bdist_wininst.py               |   5 +-
  Lib/packaging/command/cmd.py                         |   4 +-
  Lib/packaging/command/install_dist.py                |   4 +-
  Lib/packaging/command/install_distinfo.py            |  25 +++---
  Lib/packaging/command/test.py                        |   2 +-
  Lib/packaging/create.py                              |  33 ++++-----
  Lib/packaging/depgraph.py                            |   5 +-
  Lib/packaging/dist.py                                |  13 ++-
  Lib/packaging/install.py                             |   9 --
  Lib/packaging/tests/support.py                       |  35 +++++++++-
  Lib/packaging/tests/test_command_build_ext.py        |  32 ++------
  Lib/packaging/tests/test_command_build_py.py         |  26 ++-----
  Lib/packaging/tests/test_command_clean.py            |   2 -
  Lib/packaging/tests/test_command_install_data.py     |   2 +
  Lib/packaging/tests/test_command_install_distinfo.py |  10 +-
  Lib/packaging/tests/test_command_register.py         |  15 +---
  Lib/packaging/tests/test_command_sdist.py            |   2 +-
  Lib/packaging/tests/test_config.py                   |  24 ++----
  Lib/packaging/tests/test_create.py                   |  31 ++------
  Lib/packaging/tests/test_dist.py                     |  27 +++----
  Lib/packaging/tests/test_run.py                      |   2 -
  Lib/packaging/tests/test_uninstall.py                |   5 -
  Lib/packaging/tests/test_util.py                     |   4 +-
  Lib/sysconfig.py                                     |   3 +-
  Lib/test/regrtest.py                                 |  19 +++++-
  Misc/NEWS                                            |   4 -
  29 files changed, 156 insertions(+), 199 deletions(-)


diff --git a/Lib/packaging/command/bdist.py b/Lib/packaging/command/bdist.py
--- a/Lib/packaging/command/bdist.py
+++ b/Lib/packaging/command/bdist.py
@@ -126,7 +126,7 @@
         # Reinitialize and run each command.
         for i in range(len(self.formats)):
             cmd_name = commands[i]
-            sub_cmd = self.get_reinitialized_command(cmd_name)
+            sub_cmd = self.reinitialize_command(cmd_name)
             sub_cmd.format = self.formats[i]
 
             # passing the owner and group names for tar archiving
diff --git a/Lib/packaging/command/bdist_dumb.py b/Lib/packaging/command/bdist_dumb.py
--- a/Lib/packaging/command/bdist_dumb.py
+++ b/Lib/packaging/command/bdist_dumb.py
@@ -80,8 +80,8 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
diff --git a/Lib/packaging/command/bdist_msi.py b/Lib/packaging/command/bdist_msi.py
--- a/Lib/packaging/command/bdist_msi.py
+++ b/Lib/packaging/command/bdist_msi.py
@@ -183,13 +183,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.prefix = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/Lib/packaging/command/bdist_wininst.py b/Lib/packaging/command/bdist_wininst.py
--- a/Lib/packaging/command/bdist_wininst.py
+++ b/Lib/packaging/command/bdist_wininst.py
@@ -115,14 +115,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install', reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
         install.plat_name = self.plat_name
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/Lib/packaging/command/cmd.py b/Lib/packaging/command/cmd.py
--- a/Lib/packaging/command/cmd.py
+++ b/Lib/packaging/command/cmd.py
@@ -318,8 +318,8 @@
         cmd_obj.ensure_finalized()
         return cmd_obj
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
-        return self.distribution.get_reinitialized_command(
+    def reinitialize_command(self, command, reinit_subcommands=False):
+        return self.distribution.reinitialize_command(
             command, reinit_subcommands)
 
     def run_command(self, command):
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -55,9 +55,7 @@
         ('install-data=', None,
          "installation directory for data files"),
 
-        # Byte-compilation options -- see install_lib.py for details, as
-        # these are duplicated from there (but only install_lib does
-        # anything with them).
+        # Byte-compilation options -- see install_lib for details
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
         ('optimize=', 'O',
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -16,8 +16,8 @@
     description = 'create a .dist-info directory for the distribution'
 
     user_options = [
-        ('distinfo-dir=', None,
-         "directory where the the .dist-info directory will be installed"),
+        ('install-dir=', None,
+         "directory where the the .dist-info directory will be created"),
         ('installer=', None,
          "the name of the installer"),
         ('requested', None,
@@ -35,7 +35,7 @@
     negative_opt = {'no-requested': 'requested'}
 
     def initialize_options(self):
-        self.distinfo_dir = None
+        self.install_dir = None
         self.installer = None
         self.requested = None
         self.no_record = None
@@ -46,8 +46,7 @@
         self.set_undefined_options('install_dist',
                                    'installer', 'requested', 'no_record')
 
-        self.set_undefined_options('install_lib',
-                                   ('install_dir', 'distinfo_dir'))
+        self.set_undefined_options('install_lib', 'install_dir')
 
         if self.installer is None:
             # FIXME distutils or packaging?
@@ -64,26 +63,26 @@
 
         basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
-        self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
+        self.install_dir = os.path.join(self.install_dir, basename)
 
     def run(self):
-        target = self.distinfo_dir
+        target = self.install_dir
 
         if os.path.isdir(target) and not os.path.islink(target):
             if not self.dry_run:
                 rmtree(target)
         elif os.path.exists(target):
-            self.execute(os.unlink, (self.distinfo_dir,),
+            self.execute(os.unlink, (self.install_dir,),
                          "removing " + target)
 
         self.execute(os.makedirs, (target,), "creating " + target)
 
-        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        metadata_path = os.path.join(self.install_dir, 'METADATA')
         self.execute(self.distribution.metadata.write, (metadata_path,),
                      "creating " + metadata_path)
         self.outfiles.append(metadata_path)
 
-        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        installer_path = os.path.join(self.install_dir, 'INSTALLER')
         logger.info('creating %s', installer_path)
         if not self.dry_run:
             with open(installer_path, 'w') as f:
@@ -91,7 +90,7 @@
         self.outfiles.append(installer_path)
 
         if self.requested:
-            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            requested_path = os.path.join(self.install_dir, 'REQUESTED')
             logger.info('creating %s', requested_path)
             if not self.dry_run:
                 open(requested_path, 'wb').close()
@@ -100,7 +99,7 @@
         if not self.no_resources:
             install_data = self.get_finalized_command('install_data')
             if install_data.get_resources_out() != []:
-                resources_path = os.path.join(self.distinfo_dir,
+                resources_path = os.path.join(self.install_dir,
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
@@ -114,7 +113,7 @@
                 self.outfiles.append(resources_path)
 
         if not self.no_record:
-            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            record_path = os.path.join(self.install_dir, 'RECORD')
             logger.info('creating %s', record_path)
             if not self.dry_run:
                 with open(record_path, 'w', encoding='utf-8') as f:
diff --git a/Lib/packaging/command/test.py b/Lib/packaging/command/test.py
--- a/Lib/packaging/command/test.py
+++ b/Lib/packaging/command/test.py
@@ -56,7 +56,7 @@
         prev_syspath = sys.path[:]
         try:
             # build release
-            build = self.get_reinitialized_command('build')
+            build = self.reinitialize_command('build')
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -30,6 +30,7 @@
 from tokenize import detect_encoding
 from configparser import RawConfigParser
 
+from packaging import logger
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from packaging._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -124,7 +125,7 @@
         if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
-        print('\nERROR: You must select "Y" or "N".\n')
+        logger.error('You must select "Y" or "N".')
 
 
 # XXX use util.ask
@@ -147,10 +148,7 @@
     helptext = helptext.strip("\n")
 
     while True:
-        sys.stdout.write(prompt)
-        sys.stdout.flush()
-
-        line = sys.stdin.readline().strip()
+        line = input(prompt).strip()
         if line == '?':
             print('=' * 70)
             print(helptext)
@@ -271,9 +269,10 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print("ERROR: %(name)s.old backup exists, please check that "
-                      "current %(name)s is correct and remove %(name)s.old" %
-                      {'name': _FILENAME})
+                message = ("ERROR: %(name)s.old backup exists, please check "
+                           "that current %(name)s is correct and remove "
+                           "%(name)s.old" % {'name': _FILENAME})
+                logger.error(message)
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -320,7 +319,7 @@
             fp.write('\n')
 
         os.chmod(_FILENAME, 0o644)
-        print('Wrote "%s".' % _FILENAME)
+        logger.info('Wrote "%s".' % _FILENAME)
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -614,8 +613,8 @@
                         break
 
             if len(found_list) == 0:
-                print('ERROR: Could not find a matching license for "%s"' %
-                      license)
+                logger.error('Could not find a matching license for "%s"' %
+                             license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -636,8 +635,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print("ERROR: Invalid selection, type a number from the list "
-                      "above.")
+                logger.error(
+                    "Invalid selection, type a number from the list above.")
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -660,8 +659,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print("ERROR: Invalid selection, type a single digit "
-                          "number.")
+                    logger.error(
+                        "Invalid selection, type a single digit number.")
 
 
 def main():
@@ -675,7 +674,3 @@
     # program.write_setup_script()
     # packaging.util.cfg_to_args()
     program()
-
-
-if __name__ == '__main__':
-    main()
diff --git a/Lib/packaging/depgraph.py b/Lib/packaging/depgraph.py
--- a/Lib/packaging/depgraph.py
+++ b/Lib/packaging/depgraph.py
@@ -224,6 +224,7 @@
 
 
 def main():
+    # XXX move to run._graph
     from packaging.database import get_distributions
     tempout = StringIO()
     try:
@@ -267,7 +268,3 @@
     else:
         print('Supported option: -d [filename]')
         sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -636,9 +636,9 @@
             except ValueError as msg:
                 raise PackagingOptionError(msg)
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
+    def reinitialize_command(self, command, reinit_subcommands=False):
         """Reinitializes a command to the state it was in when first
-        returned by 'get_command_obj()': ie., initialized but not yet
+        returned by 'get_command_obj()': i.e., initialized but not yet
         finalized.  This provides the opportunity to sneak option
         values in programmatically, overriding or supplementing
         user-supplied values from the config files and command line.
@@ -650,10 +650,11 @@
         'reinit_subcommands' is true, also reinitializes the command's
         sub-commands, as declared by the 'sub_commands' class attribute (if
         it has one).  See the "install_dist" command for an example.  Only
-        reinitializes the sub-commands that actually matter, ie. those
-        whose test predicates return true.
+        reinitializes the sub-commands that actually matter, i.e. those
+        whose test predicate return true.
 
-        Returns the reinitialized command object.
+        Returns the reinitialized command object.  It will be the same
+        object as the one stored in the self.command_obj attribute.
         """
         if not isinstance(command, Command):
             command_name = command
@@ -671,7 +672,7 @@
 
         if reinit_subcommands:
             for sub in command.get_sub_commands():
-                self.get_reinitialized_command(sub, reinit_subcommands)
+                self.reinitialize_command(sub, reinit_subcommands)
 
         return command
 
diff --git a/Lib/packaging/install.py b/Lib/packaging/install.py
--- a/Lib/packaging/install.py
+++ b/Lib/packaging/install.py
@@ -527,12 +527,3 @@
             logger.info('%r conflicts with %s', project, ','.join(projects))
 
     return True
-
-
-def _main(**attrs):
-    if 'script_args' not in attrs:
-        attrs['requirements'] = sys.argv[1]
-    get_infos(**attrs)
-
-if __name__ == '__main__':
-    _main()
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -41,6 +41,9 @@
 import sysconfig
 
 from packaging.dist import Distribution
+from packaging.util import resolve_name
+from packaging.command import set_command, _COMMANDS
+
 from packaging.tests import unittest
 from test.support import requires_zlib, unlink
 
@@ -49,9 +52,10 @@
     # TestCase mixins
     'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
     # mocks
-    'DummyCommand', 'TestDistribution',
+    'DummyCommand', 'TestDistribution', 'Inputs',
     # misc. functions and decorators
-    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    'fake_dec', 'create_distribution', 'use_command',
+    'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
     'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
@@ -247,7 +251,7 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
-    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # XXX does not work with dist.reinitialize_command, which typechecks
     # and wants a finalized attribute
 
     def __init__(self, **kwargs):
@@ -270,6 +274,22 @@
         return self._config_files
 
 
+class Inputs:
+    """Fakes user inputs."""
+    # TODO document usage
+    # TODO use context manager or something for auto cleanup
+
+    def __init__(self, *answers):
+        self.answers = answers
+        self.index = 0
+
+    def __call__(self, prompt=''):
+        try:
+            return self.answers[self.index]
+        finally:
+            self.index += 1
+
+
 def create_distribution(configfiles=()):
     """Prepares a distribution with given config files parsed."""
     d = TestDistribution()
@@ -280,6 +300,15 @@
     return d
 
 
+def use_command(testcase, fullname):
+    """Register command at *fullname* for the duration of a test."""
+    set_command(fullname)
+    # XXX maybe set_command should return the class object
+    name = resolve_name(fullname).get_command_name()
+    # XXX maybe we need a public API to remove commands
+    testcase.addCleanup(_COMMANDS.__delitem__, name)
+
+
 def fake_dec(*args, **kw):
     """Fake decorator"""
     def _wrap(func):
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -3,7 +3,6 @@
 import site
 import sysconfig
 import textwrap
-from io import StringIO
 from packaging.dist import Distribution
 from packaging.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
@@ -11,7 +10,7 @@
 from packaging.compiler.extension import Extension
 
 from test.script_helper import assert_python_ok
-from packaging.tests import support, unittest, verbose
+from packaging.tests import support, unittest
 
 
 class BuildExtTestCase(support.TempdirManager,
@@ -37,18 +36,10 @@
         support.fixup_build_ext(cmd)
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
+        cmd.ensure_finalized()
+        cmd.run()
 
-        old_stdout = sys.stdout
-        if not verbose:
-            # silence compiler output
-            sys.stdout = StringIO()
-        try:
-            cmd.ensure_finalized()
-            cmd.run()
-        finally:
-            sys.stdout = old_stdout
-
-        code = """if 1:
+        code = textwrap.dedent("""\
             import sys
             sys.path.insert(0, %r)
 
@@ -63,7 +54,8 @@
             doc = 'This is a template module just for instruction.'
             assert xx.__doc__ == doc
             assert isinstance(xx.Null(), xx.Null)
-            assert isinstance(xx.Str(), xx.Str)"""
+            assert isinstance(xx.Str(), xx.Str)
+            """)
         code = code % self.tmp_dir
         assert_python_ok('-c', code)
 
@@ -388,16 +380,8 @@
         cmd.build_temp = self.tmp_dir
 
         try:
-            old_stdout = sys.stdout
-            if not verbose:
-                # silence compiler output
-                sys.stdout = StringIO()
-            try:
-                cmd.ensure_finalized()
-                cmd.run()
-            finally:
-                sys.stdout = old_stdout
-
+            cmd.ensure_finalized()
+            cmd.run()
         except CompileError:
             self.fail("Wrong deployment target during compilation")
 
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -67,8 +67,6 @@
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
         # create the distribution files.
         sources = self.mkdtemp()
         pkg = os.path.join(sources, 'pkg')
@@ -79,24 +77,16 @@
         open(os.path.join(testdir, "testfile"), "wb").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        #sys.stdout = StringIO.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": sources,
+                             "package_data": {"pkg": ["doc/*"]}})
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": sources,
-                                 "package_data": {"pkg": ["doc/*"]}})
-            dist.script_args = ["build"]
-            dist.parse_command_line()
-
-            try:
-                dist.run_commands()
-            except PackagingFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+            dist.run_commands()
+        except PackagingFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
     def test_byte_compile(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
diff --git a/Lib/packaging/tests/test_command_clean.py b/Lib/packaging/tests/test_command_clean.py
--- a/Lib/packaging/tests/test_command_clean.py
+++ b/Lib/packaging/tests/test_command_clean.py
@@ -36,8 +36,6 @@
                              '%r was not removed' % path)
 
         # let's run the command again (should spit warnings but succeed)
-        cmd.all = True
-        cmd.ensure_finalized()
         cmd.run()
 
 
diff --git a/Lib/packaging/tests/test_command_install_data.py b/Lib/packaging/tests/test_command_install_data.py
--- a/Lib/packaging/tests/test_command_install_data.py
+++ b/Lib/packaging/tests/test_command_install_data.py
@@ -62,6 +62,7 @@
 
         # let's try with warn_dir one
         cmd.warn_dir = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -80,6 +81,7 @@
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_install_distinfo.py b/Lib/packaging/tests/test_command_install_distinfo.py
--- a/Lib/packaging/tests/test_command_install_distinfo.py
+++ b/Lib/packaging/tests/test_command_install_distinfo.py
@@ -49,7 +49,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
@@ -76,7 +76,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.installer = 'bacon-python'
         cmd.ensure_finalized()
         cmd.run()
@@ -96,7 +96,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.requested = False
         cmd.ensure_finalized()
         cmd.run()
@@ -116,7 +116,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.no_record = True
         cmd.ensure_finalized()
         cmd.run()
@@ -214,7 +214,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_register.py b/Lib/packaging/tests/test_command_register.py
--- a/Lib/packaging/tests/test_command_register.py
+++ b/Lib/packaging/tests/test_command_register.py
@@ -12,6 +12,7 @@
     DOCUTILS_SUPPORT = False
 
 from packaging.tests import unittest, support
+from packaging.tests.support import Inputs
 from packaging.command import register as register_module
 from packaging.command.register import register
 from packaging.errors import PackagingSetupError
@@ -38,19 +39,6 @@
 """
 
 
-class Inputs:
-    """Fakes user inputs."""
-    def __init__(self, *answers):
-        self.answers = answers
-        self.index = 0
-
-    def __call__(self, prompt=''):
-        try:
-            return self.answers[self.index]
-        finally:
-            self.index += 1
-
-
 class FakeOpener:
     """Fakes a PyPI server"""
     def __init__(self):
@@ -143,6 +131,7 @@
 
         register_module.input = _no_way
         cmd.show_response = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -140,7 +140,7 @@
 
         # now trying a tar then a gztar
         cmd.formats = ['tar', 'gztar']
-
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -1,7 +1,6 @@
 """Tests for packaging.config."""
 import os
 import sys
-from io import StringIO
 
 from packaging import command
 from packaging.dist import Distribution
@@ -183,13 +182,14 @@
 
     def __init__(self, dist):
         self.distribution = dist
+        self._record = []
 
     @classmethod
     def get_command_name(cls):
         return 'foo'
 
     def run(self):
-        self.distribution.foo_was_here = True
+        self._record.append('foo has run')
 
     def nothing(self):
         pass
@@ -209,21 +209,11 @@
 
     def setUp(self):
         super(ConfigTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        sys.stdout = StringIO()
-        sys.stderr = StringIO()
-
-        self.addCleanup(os.chdir, os.getcwd())
         tempdir = self.mkdtemp()
         self.working_dir = os.getcwd()
         os.chdir(tempdir)
         self.tempdir = tempdir
 
-    def tearDown(self):
-        os.chdir(self.working_dir)
-        super(ConfigTestCase, self).tearDown()
-
     def write_setup(self, kwargs=None):
         opts = {'description-file': 'README', 'extra-files': '',
                 'setup-hooks': 'packaging.tests.test_config.version_hook'}
@@ -378,7 +368,7 @@
         self.assertIn('hooks', sys.modules)
 
     def test_missing_setup_hook_warns(self):
-        self.write_setup({'setup-hooks': 'this.does._not.exist'})
+        self.write_setup({'setup-hooks': 'does._not.exist'})
         self.write_file('README', 'yeah')
         self.get_dist()
         logs = self.get_logs()
@@ -491,10 +481,12 @@
             self.write_file((pkg, '__init__.py'), '#')
 
         # try to run the install command to see if foo is called
+        self.addCleanup(command._COMMANDS.__delitem__, 'foo')
         dist = self.get_dist()
-        self.assertIn('foo', command.get_command_names())
-        self.assertEqual('FooBarBazTest',
-                         dist.get_command_obj('foo').__class__.__name__)
+        dist.run_command('install_dist')
+        cmd = dist.get_command_obj('foo')
+        self.assertEqual(cmd.__class__.__name__, 'FooBarBazTest')
+        self.assertEqual(cmd._record, ['foo has run'])
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_create.py b/Lib/packaging/tests/test_create.py
--- a/Lib/packaging/tests/test_create.py
+++ b/Lib/packaging/tests/test_create.py
@@ -2,15 +2,17 @@
 import os
 import sys
 import sysconfig
-from io import StringIO
 from textwrap import dedent
+from packaging import create
 from packaging.create import MainProgram, ask_yn, ask, main
 
 from packaging.tests import support, unittest
+from packaging.tests.support import Inputs
 
 
 class CreateTestCase(support.TempdirManager,
                      support.EnvironRestorer,
+                     support.LoggingCatcher,
                      unittest.TestCase):
 
     maxDiff = None
@@ -18,11 +20,6 @@
 
     def setUp(self):
         super(CreateTestCase, self).setUp()
-        self._stdin = sys.stdin  # TODO use Inputs
-        self._stdout = sys.stdout
-        sys.stdin = StringIO()
-        sys.stdout = StringIO()
-        self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
         # patch sysconfig
@@ -32,29 +29,24 @@
             'doc': sys.prefix + '/share/doc/pyxfoil', }
 
     def tearDown(self):
-        sys.stdin = self._stdin
-        sys.stdout = self._stdout
-        os.chdir(self._cwd)
         sysconfig.get_paths = self._old_get_paths
+        if hasattr(create, 'input'):
+            del create.input
         super(CreateTestCase, self).tearDown()
 
     def test_ask_yn(self):
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         self.assertEqual('y', ask_yn('is this a test'))
 
     def test_ask(self):
-        sys.stdin.write('a\n')
-        sys.stdin.write('b\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('a', 'b')
         self.assertEqual('a', ask('is this a test'))
         self.assertEqual('b', ask(str(list(range(0, 70))), default='c',
                                   lengthy=True))
 
     def test_set_multi(self):
         mainprogram = MainProgram()
-        sys.stdin.write('aaaaa\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('aaaaa')
         mainprogram.data['author'] = []
         mainprogram._set_multi('_set_multi test', 'author')
         self.assertEqual(['aaaaa'], mainprogram.data['author'])
@@ -130,8 +122,7 @@
               scripts=['my_script', 'bin/run'],
               )
         """), encoding='utf-8')
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
@@ -206,9 +197,7 @@
 barbar is now in the public domain,
 ho, baby!
                         '''))
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
-        # FIXME Out of memory error.
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -6,30 +6,32 @@
 import packaging.dist
 
 from packaging.dist import Distribution
-from packaging.command import set_command, _COMMANDS
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
 from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
-from packaging.tests.support import create_distribution
+from packaging.tests.support import create_distribution, use_command
 from test.support import unload
 
 
 class test_dist(Command):
-    """Sample packaging extension command."""
+    """Custom command used for testing."""
 
     user_options = [
-        ("sample-option=", "S", "help text"),
+        ('sample-option=', 'S',
+         "help text"),
         ]
 
     def initialize_options(self):
         self.sample_option = None
+        self._record = []
 
     def finalize_options(self):
-        pass
+        if self.sample_option is None:
+            self.sample_option = 'default value'
 
     def run(self):
-        pass
+        self._record.append('test_dist has run')
 
 
 class DistributionTestCase(support.TempdirManager,
@@ -45,14 +47,10 @@
         # (defaulting to sys.argv)
         self.argv = sys.argv, sys.argv[:]
         del sys.argv[1:]
-        self._commands = _COMMANDS.copy()
 
     def tearDown(self):
         sys.argv = self.argv[0]
         sys.argv[:] = self.argv[1]
-        # XXX maybe we need a public API to remove commands
-        _COMMANDS.clear()
-        _COMMANDS.update(self._commands)
         super(DistributionTestCase, self).tearDown()
 
     @unittest.skip('needs to be updated')
@@ -181,7 +179,8 @@
         self.write_file((temp_home, "config2.cfg"),
                         '[test_dist]\npre-hook.b = type')
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
+
         dist = create_distribution(config_files)
         cmd = dist.get_command_obj("test_dist")
         self.assertEqual(cmd.pre_hook, {"a": 'type', "b": 'type'})
@@ -209,7 +208,7 @@
                 record.append('post-%s' % cmd.get_command_name())
             '''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
 
@@ -236,7 +235,7 @@
             [test_dist]
             pre-hook.test = nonexistent.dotted.name'''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
@@ -251,7 +250,7 @@
             [test_dist]
             pre-hook.test = packaging.tests.test_dist.__doc__'''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -33,11 +33,9 @@
 
     def setUp(self):
         super(RunTestCase, self).setUp()
-        self.old_stdout = sys.stdout
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
-        sys.stdout = self.old_stdout
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
diff --git a/Lib/packaging/tests/test_uninstall.py b/Lib/packaging/tests/test_uninstall.py
--- a/Lib/packaging/tests/test_uninstall.py
+++ b/Lib/packaging/tests/test_uninstall.py
@@ -1,6 +1,5 @@
 """Tests for the packaging.uninstall module."""
 import os
-import sys
 import logging
 import packaging.util
 
@@ -31,16 +30,12 @@
 
     def setUp(self):
         super(UninstallTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        self.addCleanup(os.chdir, os.getcwd())
         self.addCleanup(enable_cache)
         self.root_dir = self.mkdtemp()
         self.cwd = os.getcwd()
         disable_cache()
 
     def tearDown(self):
-        os.chdir(self.cwd)
         packaging.util._path_created.clear()
         super(UninstallTestCase, self).tearDown()
 
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -170,8 +170,8 @@
     def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
-        sys.old_stdout = self.old_stdout
-        sys.old_stderr = self.old_stderr
+        sys.stdout = self.old_stdout
+        sys.stderr = self.old_stderr
 
     def test_convert_path(self):
         # linux/mac
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -488,8 +488,7 @@
         # Setting 'userbase' is done below the call to the
         # init function to enable using 'get_config_var' in
         # the init-function.
-        if sys.version >= '2.6':
-            _CONFIG_VARS['userbase'] = _getuserbase()
+        _CONFIG_VARS['userbase'] = _getuserbase()
 
         if 'srcdir' not in _CONFIG_VARS:
             _CONFIG_VARS['srcdir'] = _PROJECT_BASE
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -172,6 +172,7 @@
 import json
 import logging
 import os
+import packaging.command
 import packaging.database
 import platform
 import random
@@ -967,7 +968,7 @@
                  'sys.warnoptions', 'threading._dangling',
                  'multiprocessing.process._dangling',
                  'sysconfig._CONFIG_VARS', 'sysconfig._SCHEMES',
-                 'packaging.database_caches',
+                 'packaging.command._COMMANDS', 'packaging.database_caches',
                 )
 
     def get_sys_argv(self):
@@ -1055,6 +1056,22 @@
         # Can't easily revert the logging state
         pass
 
+    def get_packaging_command__COMMANDS(self):
+        # registry mapping command names to full dotted path or to the actual
+        # class (resolved on demand); this check only looks at the names, not
+        # the types of the values (IOW, if a value changes from a string
+        # (dotted path) to a class it's okay but if a key (i.e. command class)
+        # is added we complain)
+        id_ = id(packaging.command._COMMANDS)
+        keys = set(packaging.command._COMMANDS)
+        return id_, keys
+    def restore_packaging_command__COMMANDS(self, saved):
+        # if command._COMMANDS was bound to another dict obhect, we can't
+        # restore the previous object and contents, because the get_ method
+        # above does not return the dict object (to ignore changes in values)
+        for key in packaging.command._COMMANDS.keys() - saved[1]:
+            del packaging.command._COMMANDS[key]
+
     def get_packaging_database_caches(self):
         # caching system used by the PEP 376 implementation
         # we have one boolean and four dictionaries, initially empty
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -294,8 +294,6 @@
 - Issue #11320: fix bogus memory management in Modules/getpath.c, leading to
   a possible crash when calling Py_SetPath().
 
-- _ast.__version__ is now a Mercurial hex revision.
-
 - Issue #11432: A bug was introduced in subprocess.Popen on posix systems with
   3.2.0 where the stdout or stderr file descriptor being the same as the stdin
   file descriptor would raise an exception. webbrowser.open would fail. fixed.
@@ -1555,8 +1553,6 @@
   signature.  Without this, architectures where sizeof void* != sizeof int are
   broken.  Patch given by Hallvard B Furuseth.
 
-- Issue #12221: Replace pyexpat.__version__ with the Python version.
-
 - Issue #12051: Fix segfault in json.dumps() while encoding highly-nested
   objects using the C accelerations.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 18:11:48 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 07 Nov 2011 18:11:48 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1RNSjc-0007PM-5H@dinsdale.python.org>

http://hg.python.org/cpython/rev/902ec9463312
changeset:   73432:902ec9463312
parent:      73431:78532ab28271
parent:      73430:4facbfdc7700
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 07 18:11:27 2011 +0100
summary:
  Merge 3.2

files:
  Doc/library/shutil.rst |  3 ++-
  Lib/http/cookiejar.py  |  2 +-
  2 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -295,7 +295,8 @@
    *owner* and *group* are used when creating a tar archive. By default,
    uses the current owner and group.
 
-   *logger* is an instance of :class:`logging.Logger`.
+   *logger* must be an object compatible with :pep:`282`, usually an instance of
+   :class:`logging.Logger`.
 
    .. versionadded:: 3.2
 
diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py
--- a/Lib/http/cookiejar.py
+++ b/Lib/http/cookiejar.py
@@ -1,4 +1,4 @@
-"""HTTP cookie handling for web clients.
+r"""HTTP cookie handling for web clients.
 
 This module has (now fairly distant) origins in Gisle Aas' Perl module
 HTTP::Cookies, from the libwww-perl library.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 19:43:34 2011
From: python-checkins at python.org (florent.xicluna)
Date: Mon, 07 Nov 2011 19:43:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_logging=3A_replace_codecs?=
 =?utf8?q?=2Eopen_with_builtins=2Eopen=2C_remove_=27=5Fencoded=27_sort=2C_?=
 =?utf8?q?add?=
Message-ID: <E1RNUAQ-0008HT-SN@dinsdale.python.org>

http://hg.python.org/cpython/rev/bbcba237bd51
changeset:   73433:bbcba237bd51
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Mon Nov 07 19:43:05 2011 +0100
summary:
  logging: replace codecs.open with builtins.open, remove '_encoded' sort, add some tests.

files:
  Lib/logging/__init__.py  |  13 +------------
  Lib/logging/config.py    |  12 ++++--------
  Lib/logging/handlers.py  |  12 ++----------
  Lib/test/test_logging.py |  23 ++++++++++++++++++-----
  4 files changed, 25 insertions(+), 35 deletions(-)


diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -36,11 +36,6 @@
            'getLogRecordFactory', 'setLogRecordFactory', 'lastResort']
 
 try:
-    import codecs
-except ImportError: #pragma: no cover
-    codecs = None
-
-try:
     import threading
 except ImportError: #pragma: no cover
     threading = None
@@ -954,8 +949,6 @@
         """
         #keep the absolute path, otherwise derived classes which use this
         #may come a cropper when the current directory changes
-        if codecs is None:  #pragma: no cover
-            encoding = None
         self.baseFilename = os.path.abspath(filename)
         self.mode = mode
         self.encoding = encoding
@@ -983,11 +976,7 @@
         Open the current base file with the (original) mode and encoding.
         Return the resulting stream.
         """
-        if self.encoding is None:
-            stream = open(self.baseFilename, self.mode)
-        else:
-            stream = codecs.open(self.baseFilename, self.mode, self.encoding)
-        return stream
+        return open(self.baseFilename, self.mode, encoding=self.encoding)
 
     def emit(self, record):
         """
diff --git a/Lib/logging/config.py b/Lib/logging/config.py
--- a/Lib/logging/config.py
+++ b/Lib/logging/config.py
@@ -24,8 +24,8 @@
 To use, simply 'import logging' and log away!
 """
 
-import sys, logging, logging.handlers, socket, struct, os, traceback, re
-import types, io
+import sys, logging, logging.handlers, socket, struct, traceback, re
+import io
 
 try:
     import _thread as thread
@@ -98,9 +98,6 @@
 def _strip_spaces(alist):
     return map(lambda x: x.strip(), alist)
 
-def _encoded(s):
-    return s if isinstance(s, str) else s.encode('utf-8')
-
 def _create_formatters(cp):
     """Create and return formatters"""
     flist = cp["formatters"]["keys"]
@@ -215,7 +212,7 @@
     #avoid disabling child loggers of explicitly
     #named loggers. With a sorted list it is easier
     #to find the child loggers.
-    existing.sort(key=_encoded)
+    existing.sort()
     #We'll keep the list of existing loggers
     #which are children of named loggers here...
     child_loggers = []
@@ -588,7 +585,7 @@
                 #avoid disabling child loggers of explicitly
                 #named loggers. With a sorted list it is easier
                 #to find the child loggers.
-                existing.sort(key=_encoded)
+                existing.sort()
                 #We'll keep the list of existing loggers
                 #which are children of named loggers here...
                 child_loggers = []
@@ -804,7 +801,6 @@
             struct.pack(">L", n), followed by the config file.
             Uses fileConfig() to do the grunt work.
             """
-            import tempfile
             try:
                 conn = self.connection
                 chunk = conn.recv(4)
diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py
--- a/Lib/logging/handlers.py
+++ b/Lib/logging/handlers.py
@@ -25,6 +25,7 @@
 """
 
 import logging, socket, os, pickle, struct, time, re
+from codecs import BOM_UTF8
 from stat import ST_DEV, ST_INO, ST_MTIME
 import queue
 try:
@@ -32,11 +33,6 @@
 except ImportError: #pragma: no cover
     threading = None
 
-try:
-    import codecs
-except ImportError: #pragma: no cover
-    codecs = None
-
 #
 # Some constants...
 #
@@ -60,8 +56,6 @@
         """
         Use the specified filename for streamed logging
         """
-        if codecs is None: #pragma: no cover
-            encoding = None
         logging.FileHandler.__init__(self, filename, mode, encoding, delay)
         self.mode = mode
         self.encoding = encoding
@@ -793,9 +787,7 @@
         prio = prio.encode('utf-8')
         # Message is a string. Convert to bytes as required by RFC 5424
         msg = msg.encode('utf-8')
-        if codecs:
-            msg = codecs.BOM_UTF8 + msg
-        msg = prio + msg
+        msg = prio + BOM_UTF8 + msg
         try:
             if self.unixsocket:
                 try:
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -49,6 +49,7 @@
 try:
     import threading
     # The following imports are needed only for tests which
+    # require threading
     import asynchat
     import asyncore
     import errno
@@ -95,9 +96,7 @@
         finally:
             logging._releaseLock()
 
-        # Set two unused loggers: one non-ASCII and one Unicode.
-        # This is to test correct operation when sorting existing
-        # loggers in the configuration code. See issue 8201.
+        # Set two unused loggers
         self.logger1 = logging.getLogger("\xab\xd7\xbb")
         self.logger2 = logging.getLogger("\u013f\u00d6\u0047")
 
@@ -310,8 +309,6 @@
             ('INF.BADPARENT', 'INFO', '4'),
         ])
 
-    def test_invalid_name(self):
-        self.assertRaises(TypeError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 
@@ -3514,6 +3511,22 @@
         self.addCleanup(setattr, self.logger.manager, 'disable', old_disable)
         self.assertFalse(self.logger.isEnabledFor(22))
 
+    def test_root_logger_aliases(self):
+        root = logging.getLogger()
+        self.assertIs(root, logging.root)
+        self.assertIs(root, logging.getLogger(None))
+        self.assertIs(root, logging.getLogger(''))
+        self.assertIs(root, logging.getLogger('foo').root)
+        self.assertIs(root, logging.getLogger('foo.bar').root)
+        self.assertIs(root, logging.getLogger('foo').parent)
+
+        self.assertIsNot(root, logging.getLogger('\0'))
+        self.assertIsNot(root, logging.getLogger('foo.bar').parent)
+
+    def test_invalid_names(self):
+        self.assertRaises(TypeError, logging.getLogger, any)
+        self.assertRaises(TypeError, logging.getLogger, b'foo')
+
 
 class BaseFileTest(BaseTest):
     "Base class for handler tests that write log files"

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 19:43:35 2011
From: python-checkins at python.org (florent.xicluna)
Date: Mon, 07 Nov 2011 19:43:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_fileinput=3A_replace_this_l?=
 =?utf8?q?ast_occurence_of_codecs=2Eopen_with_builtins=2Eopen=2E?=
Message-ID: <E1RNUAR-0008Hd-Om@dinsdale.python.org>

http://hg.python.org/cpython/rev/c2b3e8b437a4
changeset:   73434:c2b3e8b437a4
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Mon Nov 07 19:43:07 2011 +0100
summary:
  fileinput: replace this last occurence of codecs.open with builtins.open.

files:
  Lib/fileinput.py           |   3 +-
  Lib/test/test_fileinput.py |  28 ++++++++-----------------
  2 files changed, 10 insertions(+), 21 deletions(-)


diff --git a/Lib/fileinput.py b/Lib/fileinput.py
--- a/Lib/fileinput.py
+++ b/Lib/fileinput.py
@@ -398,9 +398,8 @@
 
 
 def hook_encoded(encoding):
-    import codecs
     def openhook(filename, mode):
-        return codecs.open(filename, mode, encoding)
+        return open(filename, mode, encoding=encoding)
     return openhook
 
 
diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py
--- a/Lib/test/test_fileinput.py
+++ b/Lib/test/test_fileinput.py
@@ -7,8 +7,7 @@
 import re
 import fileinput
 import collections
-import types
-import codecs
+import builtins
 import unittest
 
 try:
@@ -807,18 +806,8 @@
 
     @staticmethod
     def replace_builtin_open(new_open_func):
-        builtins_type = type(__builtins__)
-        if builtins_type is dict:
-            original_open = __builtins__["open"]
-            __builtins__["open"] = new_open_func
-        elif builtins_type is types.ModuleType:
-            original_open = __builtins__.open
-            __builtins__.open = new_open_func
-        else:
-            raise RuntimeError(
-                "unknown __builtins__ type: %r (unable to replace open)" %
-                builtins_type)
-
+        original_open = builtins.open
+        builtins.open = new_open_func
         return original_open
 
 class Test_hook_encoded(unittest.TestCase):
@@ -829,21 +818,22 @@
         result = fileinput.hook_encoded(encoding)
 
         fake_open = InvocationRecorder()
-        original_open = codecs.open
-        codecs.open = fake_open
+        original_open = builtins.open
+        builtins.open = fake_open
         try:
             filename = object()
             mode = object()
             open_result = result(filename, mode)
         finally:
-            codecs.open = original_open
+            builtins.open = original_open
 
         self.assertEqual(fake_open.invocation_count, 1)
 
-        args = fake_open.last_invocation[0]
+        args, kwargs = fake_open.last_invocation
         self.assertIs(args[0], filename)
         self.assertIs(args[1], mode)
-        self.assertIs(args[2], encoding)
+        self.assertIs(kwargs.pop('encoding'), encoding)
+        self.assertFalse(kwargs)
 
 def test_main():
     run_unittest(

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 19:49:21 2011
From: python-checkins at python.org (florent.xicluna)
Date: Mon, 07 Nov 2011 19:49:21 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_More_assertions_in_test=5Fu?=
 =?utf8?q?nicode=5Ffile=2C_to_chase_issue_=2313348=2E?=
Message-ID: <E1RNUG1-0000Ng-CV@dinsdale.python.org>

http://hg.python.org/cpython/rev/fcff91a7b397
changeset:   73435:fcff91a7b397
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Mon Nov 07 19:49:07 2011 +0100
summary:
  More assertions in test_unicode_file, to chase issue #13348.

files:
  Lib/test/test_unicode_file.py |  10 ++++++++--
  1 files changed, 8 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_unicode_file.py b/Lib/test/test_unicode_file.py
--- a/Lib/test/test_unicode_file.py
+++ b/Lib/test/test_unicode_file.py
@@ -56,16 +56,20 @@
         # Should be able to rename the file using either name.
         self.assertTrue(os.path.isfile(filename1)) # must exist.
         os.rename(filename1, filename2 + ".new")
-        self.assertTrue(os.path.isfile(filename1+".new"))
+        self.assertFalse(os.path.isfile(filename2))
+        self.assertTrue(os.path.isfile(filename1 + '.new'))
         os.rename(filename1 + ".new", filename2)
+        self.assertFalse(os.path.isfile(filename1 + '.new'))
         self.assertTrue(os.path.isfile(filename2))
 
         shutil.copy(filename1, filename2 + ".new")
         os.unlink(filename1 + ".new") # remove using equiv name.
         # And a couple of moves, one using each name.
         shutil.move(filename1, filename2 + ".new")
-        self.assertTrue(not os.path.exists(filename2))
+        self.assertFalse(os.path.exists(filename2))
+        self.assertTrue(os.path.exists(filename1 + '.new'))
         shutil.move(filename1 + ".new", filename2)
+        self.assertFalse(os.path.exists(filename2 + '.new'))
         self.assertTrue(os.path.exists(filename1))
         # Note - due to the implementation of shutil.move,
         # it tries a rename first.  This only fails on Windows when on
@@ -73,7 +77,9 @@
         # So we test the shutil.copy2 function, which is the thing most
         # likely to fail.
         shutil.copy2(filename1, filename2 + ".new")
+        self.assertTrue(os.path.isfile(filename1 + '.new'))
         os.unlink(filename1 + ".new")
+        self.assertFalse(os.path.exists(filename2 + '.new'))
 
     def _do_directory(self, make_name, chdir_name):
         cwd = os.getcwdb()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 21:20:09 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 21:20:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Adjust_None_handling_to_be_?=
 =?utf8?q?a_bit_more_clean=2E_Thanks_to_Benjamin?=
Message-ID: <E1RNVft-00017V-I8@dinsdale.python.org>

http://hg.python.org/cpython/rev/d0a73bd3162c
changeset:   73436:d0a73bd3162c
parent:      73420:fb73fe5d0ab1
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 10:51:18 2011 -0600
summary:
  Adjust None handling to be a bit more clean. Thanks to Benjamin
for pointing it out.

files:
  Modules/posixmodule.c |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -3543,7 +3543,7 @@
 posix_utime(PyObject *self, PyObject *args)
 {
 #ifdef MS_WINDOWS
-    PyObject *arg = NULL;
+    PyObject *arg = Py_None;
     PyObject *obwpath;
     wchar_t *wpath = NULL;
     PyObject *oapath;
@@ -3589,7 +3589,7 @@
         Py_DECREF(oapath);
     }
 
-    if (!arg || (arg == Py_None)) {
+    if (arg == Py_None) {
         SYSTEMTIME now;
         GetSystemTime(&now);
         if (!SystemTimeToFileTime(&now, &mtime) ||
@@ -3633,13 +3633,13 @@
     time_t atime, mtime;
     long ausec, musec;
     int res;
-    PyObject* arg = NULL;
+    PyObject* arg = Py_None;
 
     if (!PyArg_ParseTuple(args, "O&|O:utime",
                           PyUnicode_FSConverter, &opath, &arg))
         return NULL;
     path = PyBytes_AsString(opath);
-    if (!arg || (arg == Py_None)) {
+    if (arg == Py_None) {
         /* optional time values not given */
         Py_BEGIN_ALLOW_THREADS
         res = utime(path, NULL);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 21:20:10 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 21:20:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_13327=2E_Remove_explici?=
 =?utf8?q?t_None_arguments_from_futimes=2C_futimens=2C_futimesat=2C?=
Message-ID: <E1RNVfu-00017i-HA@dinsdale.python.org>

http://hg.python.org/cpython/rev/045e8757f10d
changeset:   73437:045e8757f10d
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 14:18:54 2011 -0600
summary:
  Fix 13327. Remove explicit None arguments from futimes, futimens, futimesat,
and lutimes.

files:
  Doc/library/os.rst     |  16 +++++--------
  Lib/test/test_posix.py |   4 +++
  Modules/posixmodule.c  |  35 ++++++++++++++---------------
  3 files changed, 27 insertions(+), 28 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -872,8 +872,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimesat(dirfd, path, (atime, mtime))
-              futimesat(dirfd, path, None)
+.. function:: futimesat(dirfd, path[, (atime, mtime)])
 
    Like :func:`utime` but if *path* is relative, it is taken as relative to *dirfd*.
    If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path*
@@ -884,12 +883,11 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimens(fd, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec))
-              futimens(fd, None, None)
+.. function:: futimens(fd[, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec)])
 
    Updates the timestamps of a file specified by the file descriptor *fd*, with
    nanosecond precision.
-   The second form sets *atime* and *mtime* to the current time.
+   If no second argument is given, set *atime* and *mtime* to the current time.
    If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_NOW`, the corresponding
    timestamp is updated to the current time.
    If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_OMIT`, the corresponding
@@ -911,11 +909,10 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimes(fd, (atime, mtime))
-              futimes(fd, None)
+.. function:: futimes(fd[, (atime, mtime)])
 
    Set the access and modified time of the file specified by the file
-   descriptor *fd* to the given values. If the second form is used, set the
+   descriptor *fd* to the given values. If no second argument is used, set the
    access and modified times to the current time.
 
    Availability: Unix.
@@ -1702,8 +1699,7 @@
       Added support for Windows 6.0 (Vista) symbolic links.
 
 
-.. function:: lutimes(path, (atime, mtime))
-              lutimes(path, None)
+.. function:: lutimes(path[, (atime, mtime)])
 
    Like :func:`utime`, but if *path* is a symbolic link, it is not
    dereferenced.
diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -235,6 +235,7 @@
         fd = os.open(support.TESTFN, os.O_RDONLY)
         try:
             posix.futimes(fd, None)
+            posix.futimes(fd)
             self.assertRaises(TypeError, posix.futimes, fd, (None, None))
             self.assertRaises(TypeError, posix.futimes, fd, (now, None))
             self.assertRaises(TypeError, posix.futimes, fd, (None, now))
@@ -252,6 +253,7 @@
         self.assertRaises(TypeError, posix.lutimes, support.TESTFN, (None, now))
         posix.lutimes(support.TESTFN, (int(now), int(now)))
         posix.lutimes(support.TESTFN, (now, now))
+        posix.lutimes(support.TESTFN)
 
     @unittest.skipUnless(hasattr(posix, 'futimens'), "test needs posix.futimens()")
     def test_futimens(self):
@@ -263,6 +265,7 @@
             self.assertRaises(TypeError, posix.futimens, fd, None, (now, 0))
             posix.futimens(fd, (int(now), int((now - int(now)) * 1e9)),
                     (int(now), int((now - int(now)) * 1e9)))
+            posix.futimens(fd)
         finally:
             os.close(fd)
 
@@ -691,6 +694,7 @@
         try:
             now = time.time()
             posix.futimesat(f, support.TESTFN, None)
+            posix.futimesat(f, support.TESTFN)
             self.assertRaises(TypeError, posix.futimesat, f, support.TESTFN, (None, None))
             self.assertRaises(TypeError, posix.futimesat, f, support.TESTFN, (now, None))
             self.assertRaises(TypeError, posix.futimesat, f, support.TESTFN, (None, now))
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -3534,10 +3534,10 @@
 }
 
 PyDoc_STRVAR(posix_utime__doc__,
-"utime(path, (atime, mtime))\n\
-utime(path, None)\n\n\
-Set the access and modified time of the file to the given values.  If the\n\
-second form is used, set the access and modified times to the current time.");
+"utime(path[, (atime, mtime)])\n\
+Set the access and modified time of the file to the given values.\n\
+If no second argument is used, set the access and modified times to\n\
+the current time.");
 
 static PyObject *
 posix_utime(PyObject *self, PyObject *args)
@@ -3706,21 +3706,20 @@
 
 #ifdef HAVE_FUTIMES
 PyDoc_STRVAR(posix_futimes__doc__,
-"futimes(fd, (atime, mtime))\n\
-futimes(fd, None)\n\n\
+"futimes(fd[, (atime, mtime)])\n\
 Set the access and modified time of the file specified by the file\n\
-descriptor fd to the given values. If the second form is used, set the\n\
+descriptor fd to the given values. If no second argument is used, set the\n\
 access and modified times to the current time.");
 
 static PyObject *
 posix_futimes(PyObject *self, PyObject *args)
 {
     int res, fd;
-    PyObject* arg;
+    PyObject* arg = Py_None;
     time_t atime, mtime;
     long ausec, musec;
 
-    if (!PyArg_ParseTuple(args, "iO:futimes", &fd, &arg))
+    if (!PyArg_ParseTuple(args, "i|O:futimes", &fd, &arg))
         return NULL;
 
     if (arg == Py_None) {
@@ -3771,20 +3770,20 @@
 
 #ifdef HAVE_LUTIMES
 PyDoc_STRVAR(posix_lutimes__doc__,
-"lutimes(path, (atime, mtime))\n\
-lutimes(path, None)\n\n\
+"lutimes(path[, (atime, mtime)])\n\
 Like utime(), but if path is a symbolic link, it is not dereferenced.");
 
 static PyObject *
 posix_lutimes(PyObject *self, PyObject *args)
 {
-    PyObject *opath, *arg;
+    PyObject *opath;
+    PyObject *arg = Py_None;
     const char *path;
     int res;
     time_t atime, mtime;
     long ausec, musec;
 
-    if (!PyArg_ParseTuple(args, "O&O:lutimes",
+    if (!PyArg_ParseTuple(args, "O&|O:lutimes",
             PyUnicode_FSConverter, &opath, &arg))
         return NULL;
     path = PyBytes_AsString(opath);
@@ -3840,11 +3839,10 @@
 
 #ifdef HAVE_FUTIMENS
 PyDoc_STRVAR(posix_futimens__doc__,
-"futimens(fd, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec))\n\
-futimens(fd, None, None)\n\n\
+"futimens(fd[, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec)])\n\
 Updates the timestamps of a file specified by the file descriptor fd, with\n\
 nanosecond precision.\n\
-The second form sets atime and mtime to the current time.\n\
+If no second argument is given, set atime and mtime to the current time.\n\
 If *_nsec is specified as UTIME_NOW, the timestamp is updated to the\n\
 current time.\n\
 If *_nsec is specified as UTIME_OMIT, the timestamp is not updated.");
@@ -3853,10 +3851,11 @@
 posix_futimens(PyObject *self, PyObject *args)
 {
     int res, fd;
-    PyObject *atime, *mtime;
+    PyObject *atime = Py_None;
+    PyObject *mtime = Py_None;
     struct timespec buf[2];
 
-    if (!PyArg_ParseTuple(args, "iOO:futimens",
+    if (!PyArg_ParseTuple(args, "i|OO:futimens",
             &fd, &atime, &mtime))
         return NULL;
     if (atime == Py_None && mtime == Py_None) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 21:20:11 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 21:20:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_branch_merge?=
Message-ID: <E1RNVfv-00017m-Fr@dinsdale.python.org>

http://hg.python.org/cpython/rev/d5187d904a08
changeset:   73438:d5187d904a08
parent:      73437:045e8757f10d
parent:      73435:fcff91a7b397
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 14:19:46 2011 -0600
summary:
  branch merge

files:
  Doc/library/shutil.rst                               |   3 +-
  Lib/fileinput.py                                     |   3 +-
  Lib/http/cookiejar.py                                |   2 +-
  Lib/logging/__init__.py                              |  13 +---
  Lib/logging/config.py                                |  12 +--
  Lib/logging/handlers.py                              |  12 +--
  Lib/packaging/command/bdist.py                       |   2 +-
  Lib/packaging/command/bdist_dumb.py                  |   4 +-
  Lib/packaging/command/bdist_msi.py                   |   6 +-
  Lib/packaging/command/bdist_wininst.py               |   5 +-
  Lib/packaging/command/cmd.py                         |   4 +-
  Lib/packaging/command/install_dist.py                |   4 +-
  Lib/packaging/command/install_distinfo.py            |  25 +++---
  Lib/packaging/command/test.py                        |   2 +-
  Lib/packaging/create.py                              |  33 ++++-----
  Lib/packaging/depgraph.py                            |   5 +-
  Lib/packaging/dist.py                                |  13 ++-
  Lib/packaging/install.py                             |   9 --
  Lib/packaging/tests/support.py                       |  35 +++++++++-
  Lib/packaging/tests/test_command_build_ext.py        |  32 ++------
  Lib/packaging/tests/test_command_build_py.py         |  26 ++-----
  Lib/packaging/tests/test_command_clean.py            |   2 -
  Lib/packaging/tests/test_command_install_data.py     |   2 +
  Lib/packaging/tests/test_command_install_distinfo.py |  10 +-
  Lib/packaging/tests/test_command_register.py         |  15 +---
  Lib/packaging/tests/test_command_sdist.py            |   2 +-
  Lib/packaging/tests/test_config.py                   |  24 ++----
  Lib/packaging/tests/test_create.py                   |  31 ++------
  Lib/packaging/tests/test_dist.py                     |  27 +++----
  Lib/packaging/tests/test_run.py                      |   2 -
  Lib/packaging/tests/test_uninstall.py                |   5 -
  Lib/packaging/tests/test_util.py                     |   4 +-
  Lib/sysconfig.py                                     |   3 +-
  Lib/test/regrtest.py                                 |  19 +++++-
  Lib/test/test_fileinput.py                           |  28 ++-----
  Lib/test/test_logging.py                             |  23 +++++-
  Lib/test/test_unicode_file.py                        |  10 ++-
  Misc/NEWS                                            |   4 -
  38 files changed, 202 insertions(+), 259 deletions(-)


diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst
--- a/Doc/library/shutil.rst
+++ b/Doc/library/shutil.rst
@@ -295,7 +295,8 @@
    *owner* and *group* are used when creating a tar archive. By default,
    uses the current owner and group.
 
-   *logger* is an instance of :class:`logging.Logger`.
+   *logger* must be an object compatible with :pep:`282`, usually an instance of
+   :class:`logging.Logger`.
 
    .. versionadded:: 3.2
 
diff --git a/Lib/fileinput.py b/Lib/fileinput.py
--- a/Lib/fileinput.py
+++ b/Lib/fileinput.py
@@ -398,9 +398,8 @@
 
 
 def hook_encoded(encoding):
-    import codecs
     def openhook(filename, mode):
-        return codecs.open(filename, mode, encoding)
+        return open(filename, mode, encoding=encoding)
     return openhook
 
 
diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py
--- a/Lib/http/cookiejar.py
+++ b/Lib/http/cookiejar.py
@@ -1,4 +1,4 @@
-"""HTTP cookie handling for web clients.
+r"""HTTP cookie handling for web clients.
 
 This module has (now fairly distant) origins in Gisle Aas' Perl module
 HTTP::Cookies, from the libwww-perl library.
diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py
--- a/Lib/logging/__init__.py
+++ b/Lib/logging/__init__.py
@@ -36,11 +36,6 @@
            'getLogRecordFactory', 'setLogRecordFactory', 'lastResort']
 
 try:
-    import codecs
-except ImportError: #pragma: no cover
-    codecs = None
-
-try:
     import threading
 except ImportError: #pragma: no cover
     threading = None
@@ -954,8 +949,6 @@
         """
         #keep the absolute path, otherwise derived classes which use this
         #may come a cropper when the current directory changes
-        if codecs is None:  #pragma: no cover
-            encoding = None
         self.baseFilename = os.path.abspath(filename)
         self.mode = mode
         self.encoding = encoding
@@ -983,11 +976,7 @@
         Open the current base file with the (original) mode and encoding.
         Return the resulting stream.
         """
-        if self.encoding is None:
-            stream = open(self.baseFilename, self.mode)
-        else:
-            stream = codecs.open(self.baseFilename, self.mode, self.encoding)
-        return stream
+        return open(self.baseFilename, self.mode, encoding=self.encoding)
 
     def emit(self, record):
         """
diff --git a/Lib/logging/config.py b/Lib/logging/config.py
--- a/Lib/logging/config.py
+++ b/Lib/logging/config.py
@@ -24,8 +24,8 @@
 To use, simply 'import logging' and log away!
 """
 
-import sys, logging, logging.handlers, socket, struct, os, traceback, re
-import types, io
+import sys, logging, logging.handlers, socket, struct, traceback, re
+import io
 
 try:
     import _thread as thread
@@ -98,9 +98,6 @@
 def _strip_spaces(alist):
     return map(lambda x: x.strip(), alist)
 
-def _encoded(s):
-    return s if isinstance(s, str) else s.encode('utf-8')
-
 def _create_formatters(cp):
     """Create and return formatters"""
     flist = cp["formatters"]["keys"]
@@ -215,7 +212,7 @@
     #avoid disabling child loggers of explicitly
     #named loggers. With a sorted list it is easier
     #to find the child loggers.
-    existing.sort(key=_encoded)
+    existing.sort()
     #We'll keep the list of existing loggers
     #which are children of named loggers here...
     child_loggers = []
@@ -588,7 +585,7 @@
                 #avoid disabling child loggers of explicitly
                 #named loggers. With a sorted list it is easier
                 #to find the child loggers.
-                existing.sort(key=_encoded)
+                existing.sort()
                 #We'll keep the list of existing loggers
                 #which are children of named loggers here...
                 child_loggers = []
@@ -804,7 +801,6 @@
             struct.pack(">L", n), followed by the config file.
             Uses fileConfig() to do the grunt work.
             """
-            import tempfile
             try:
                 conn = self.connection
                 chunk = conn.recv(4)
diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py
--- a/Lib/logging/handlers.py
+++ b/Lib/logging/handlers.py
@@ -25,6 +25,7 @@
 """
 
 import logging, socket, os, pickle, struct, time, re
+from codecs import BOM_UTF8
 from stat import ST_DEV, ST_INO, ST_MTIME
 import queue
 try:
@@ -32,11 +33,6 @@
 except ImportError: #pragma: no cover
     threading = None
 
-try:
-    import codecs
-except ImportError: #pragma: no cover
-    codecs = None
-
 #
 # Some constants...
 #
@@ -60,8 +56,6 @@
         """
         Use the specified filename for streamed logging
         """
-        if codecs is None: #pragma: no cover
-            encoding = None
         logging.FileHandler.__init__(self, filename, mode, encoding, delay)
         self.mode = mode
         self.encoding = encoding
@@ -793,9 +787,7 @@
         prio = prio.encode('utf-8')
         # Message is a string. Convert to bytes as required by RFC 5424
         msg = msg.encode('utf-8')
-        if codecs:
-            msg = codecs.BOM_UTF8 + msg
-        msg = prio + msg
+        msg = prio + BOM_UTF8 + msg
         try:
             if self.unixsocket:
                 try:
diff --git a/Lib/packaging/command/bdist.py b/Lib/packaging/command/bdist.py
--- a/Lib/packaging/command/bdist.py
+++ b/Lib/packaging/command/bdist.py
@@ -126,7 +126,7 @@
         # Reinitialize and run each command.
         for i in range(len(self.formats)):
             cmd_name = commands[i]
-            sub_cmd = self.get_reinitialized_command(cmd_name)
+            sub_cmd = self.reinitialize_command(cmd_name)
             sub_cmd.format = self.formats[i]
 
             # passing the owner and group names for tar archiving
diff --git a/Lib/packaging/command/bdist_dumb.py b/Lib/packaging/command/bdist_dumb.py
--- a/Lib/packaging/command/bdist_dumb.py
+++ b/Lib/packaging/command/bdist_dumb.py
@@ -80,8 +80,8 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
diff --git a/Lib/packaging/command/bdist_msi.py b/Lib/packaging/command/bdist_msi.py
--- a/Lib/packaging/command/bdist_msi.py
+++ b/Lib/packaging/command/bdist_msi.py
@@ -183,13 +183,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.prefix = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/Lib/packaging/command/bdist_wininst.py b/Lib/packaging/command/bdist_wininst.py
--- a/Lib/packaging/command/bdist_wininst.py
+++ b/Lib/packaging/command/bdist_wininst.py
@@ -115,14 +115,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install', reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
         install.plat_name = self.plat_name
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/Lib/packaging/command/cmd.py b/Lib/packaging/command/cmd.py
--- a/Lib/packaging/command/cmd.py
+++ b/Lib/packaging/command/cmd.py
@@ -318,8 +318,8 @@
         cmd_obj.ensure_finalized()
         return cmd_obj
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
-        return self.distribution.get_reinitialized_command(
+    def reinitialize_command(self, command, reinit_subcommands=False):
+        return self.distribution.reinitialize_command(
             command, reinit_subcommands)
 
     def run_command(self, command):
diff --git a/Lib/packaging/command/install_dist.py b/Lib/packaging/command/install_dist.py
--- a/Lib/packaging/command/install_dist.py
+++ b/Lib/packaging/command/install_dist.py
@@ -55,9 +55,7 @@
         ('install-data=', None,
          "installation directory for data files"),
 
-        # Byte-compilation options -- see install_lib.py for details, as
-        # these are duplicated from there (but only install_lib does
-        # anything with them).
+        # Byte-compilation options -- see install_lib for details
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
         ('optimize=', 'O',
diff --git a/Lib/packaging/command/install_distinfo.py b/Lib/packaging/command/install_distinfo.py
--- a/Lib/packaging/command/install_distinfo.py
+++ b/Lib/packaging/command/install_distinfo.py
@@ -16,8 +16,8 @@
     description = 'create a .dist-info directory for the distribution'
 
     user_options = [
-        ('distinfo-dir=', None,
-         "directory where the the .dist-info directory will be installed"),
+        ('install-dir=', None,
+         "directory where the the .dist-info directory will be created"),
         ('installer=', None,
          "the name of the installer"),
         ('requested', None,
@@ -35,7 +35,7 @@
     negative_opt = {'no-requested': 'requested'}
 
     def initialize_options(self):
-        self.distinfo_dir = None
+        self.install_dir = None
         self.installer = None
         self.requested = None
         self.no_record = None
@@ -46,8 +46,7 @@
         self.set_undefined_options('install_dist',
                                    'installer', 'requested', 'no_record')
 
-        self.set_undefined_options('install_lib',
-                                   ('install_dir', 'distinfo_dir'))
+        self.set_undefined_options('install_lib', 'install_dir')
 
         if self.installer is None:
             # FIXME distutils or packaging?
@@ -64,26 +63,26 @@
 
         basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
-        self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
+        self.install_dir = os.path.join(self.install_dir, basename)
 
     def run(self):
-        target = self.distinfo_dir
+        target = self.install_dir
 
         if os.path.isdir(target) and not os.path.islink(target):
             if not self.dry_run:
                 rmtree(target)
         elif os.path.exists(target):
-            self.execute(os.unlink, (self.distinfo_dir,),
+            self.execute(os.unlink, (self.install_dir,),
                          "removing " + target)
 
         self.execute(os.makedirs, (target,), "creating " + target)
 
-        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        metadata_path = os.path.join(self.install_dir, 'METADATA')
         self.execute(self.distribution.metadata.write, (metadata_path,),
                      "creating " + metadata_path)
         self.outfiles.append(metadata_path)
 
-        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        installer_path = os.path.join(self.install_dir, 'INSTALLER')
         logger.info('creating %s', installer_path)
         if not self.dry_run:
             with open(installer_path, 'w') as f:
@@ -91,7 +90,7 @@
         self.outfiles.append(installer_path)
 
         if self.requested:
-            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            requested_path = os.path.join(self.install_dir, 'REQUESTED')
             logger.info('creating %s', requested_path)
             if not self.dry_run:
                 open(requested_path, 'wb').close()
@@ -100,7 +99,7 @@
         if not self.no_resources:
             install_data = self.get_finalized_command('install_data')
             if install_data.get_resources_out() != []:
-                resources_path = os.path.join(self.distinfo_dir,
+                resources_path = os.path.join(self.install_dir,
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
@@ -114,7 +113,7 @@
                 self.outfiles.append(resources_path)
 
         if not self.no_record:
-            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            record_path = os.path.join(self.install_dir, 'RECORD')
             logger.info('creating %s', record_path)
             if not self.dry_run:
                 with open(record_path, 'w', encoding='utf-8') as f:
diff --git a/Lib/packaging/command/test.py b/Lib/packaging/command/test.py
--- a/Lib/packaging/command/test.py
+++ b/Lib/packaging/command/test.py
@@ -56,7 +56,7 @@
         prev_syspath = sys.path[:]
         try:
             # build release
-            build = self.get_reinitialized_command('build')
+            build = self.reinitialize_command('build')
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
diff --git a/Lib/packaging/create.py b/Lib/packaging/create.py
--- a/Lib/packaging/create.py
+++ b/Lib/packaging/create.py
@@ -30,6 +30,7 @@
 from tokenize import detect_encoding
 from configparser import RawConfigParser
 
+from packaging import logger
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from packaging._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -124,7 +125,7 @@
         if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
-        print('\nERROR: You must select "Y" or "N".\n')
+        logger.error('You must select "Y" or "N".')
 
 
 # XXX use util.ask
@@ -147,10 +148,7 @@
     helptext = helptext.strip("\n")
 
     while True:
-        sys.stdout.write(prompt)
-        sys.stdout.flush()
-
-        line = sys.stdin.readline().strip()
+        line = input(prompt).strip()
         if line == '?':
             print('=' * 70)
             print(helptext)
@@ -271,9 +269,10 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print("ERROR: %(name)s.old backup exists, please check that "
-                      "current %(name)s is correct and remove %(name)s.old" %
-                      {'name': _FILENAME})
+                message = ("ERROR: %(name)s.old backup exists, please check "
+                           "that current %(name)s is correct and remove "
+                           "%(name)s.old" % {'name': _FILENAME})
+                logger.error(message)
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -320,7 +319,7 @@
             fp.write('\n')
 
         os.chmod(_FILENAME, 0o644)
-        print('Wrote "%s".' % _FILENAME)
+        logger.info('Wrote "%s".' % _FILENAME)
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -614,8 +613,8 @@
                         break
 
             if len(found_list) == 0:
-                print('ERROR: Could not find a matching license for "%s"' %
-                      license)
+                logger.error('Could not find a matching license for "%s"' %
+                             license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -636,8 +635,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print("ERROR: Invalid selection, type a number from the list "
-                      "above.")
+                logger.error(
+                    "Invalid selection, type a number from the list above.")
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -660,8 +659,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print("ERROR: Invalid selection, type a single digit "
-                          "number.")
+                    logger.error(
+                        "Invalid selection, type a single digit number.")
 
 
 def main():
@@ -675,7 +674,3 @@
     # program.write_setup_script()
     # packaging.util.cfg_to_args()
     program()
-
-
-if __name__ == '__main__':
-    main()
diff --git a/Lib/packaging/depgraph.py b/Lib/packaging/depgraph.py
--- a/Lib/packaging/depgraph.py
+++ b/Lib/packaging/depgraph.py
@@ -224,6 +224,7 @@
 
 
 def main():
+    # XXX move to run._graph
     from packaging.database import get_distributions
     tempout = StringIO()
     try:
@@ -267,7 +268,3 @@
     else:
         print('Supported option: -d [filename]')
         sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()
diff --git a/Lib/packaging/dist.py b/Lib/packaging/dist.py
--- a/Lib/packaging/dist.py
+++ b/Lib/packaging/dist.py
@@ -636,9 +636,9 @@
             except ValueError as msg:
                 raise PackagingOptionError(msg)
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
+    def reinitialize_command(self, command, reinit_subcommands=False):
         """Reinitializes a command to the state it was in when first
-        returned by 'get_command_obj()': ie., initialized but not yet
+        returned by 'get_command_obj()': i.e., initialized but not yet
         finalized.  This provides the opportunity to sneak option
         values in programmatically, overriding or supplementing
         user-supplied values from the config files and command line.
@@ -650,10 +650,11 @@
         'reinit_subcommands' is true, also reinitializes the command's
         sub-commands, as declared by the 'sub_commands' class attribute (if
         it has one).  See the "install_dist" command for an example.  Only
-        reinitializes the sub-commands that actually matter, ie. those
-        whose test predicates return true.
+        reinitializes the sub-commands that actually matter, i.e. those
+        whose test predicate return true.
 
-        Returns the reinitialized command object.
+        Returns the reinitialized command object.  It will be the same
+        object as the one stored in the self.command_obj attribute.
         """
         if not isinstance(command, Command):
             command_name = command
@@ -671,7 +672,7 @@
 
         if reinit_subcommands:
             for sub in command.get_sub_commands():
-                self.get_reinitialized_command(sub, reinit_subcommands)
+                self.reinitialize_command(sub, reinit_subcommands)
 
         return command
 
diff --git a/Lib/packaging/install.py b/Lib/packaging/install.py
--- a/Lib/packaging/install.py
+++ b/Lib/packaging/install.py
@@ -527,12 +527,3 @@
             logger.info('%r conflicts with %s', project, ','.join(projects))
 
     return True
-
-
-def _main(**attrs):
-    if 'script_args' not in attrs:
-        attrs['requirements'] = sys.argv[1]
-    get_infos(**attrs)
-
-if __name__ == '__main__':
-    _main()
diff --git a/Lib/packaging/tests/support.py b/Lib/packaging/tests/support.py
--- a/Lib/packaging/tests/support.py
+++ b/Lib/packaging/tests/support.py
@@ -41,6 +41,9 @@
 import sysconfig
 
 from packaging.dist import Distribution
+from packaging.util import resolve_name
+from packaging.command import set_command, _COMMANDS
+
 from packaging.tests import unittest
 from test.support import requires_zlib, unlink
 
@@ -49,9 +52,10 @@
     # TestCase mixins
     'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
     # mocks
-    'DummyCommand', 'TestDistribution',
+    'DummyCommand', 'TestDistribution', 'Inputs',
     # misc. functions and decorators
-    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    'fake_dec', 'create_distribution', 'use_command',
+    'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
     'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
@@ -247,7 +251,7 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
-    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # XXX does not work with dist.reinitialize_command, which typechecks
     # and wants a finalized attribute
 
     def __init__(self, **kwargs):
@@ -270,6 +274,22 @@
         return self._config_files
 
 
+class Inputs:
+    """Fakes user inputs."""
+    # TODO document usage
+    # TODO use context manager or something for auto cleanup
+
+    def __init__(self, *answers):
+        self.answers = answers
+        self.index = 0
+
+    def __call__(self, prompt=''):
+        try:
+            return self.answers[self.index]
+        finally:
+            self.index += 1
+
+
 def create_distribution(configfiles=()):
     """Prepares a distribution with given config files parsed."""
     d = TestDistribution()
@@ -280,6 +300,15 @@
     return d
 
 
+def use_command(testcase, fullname):
+    """Register command at *fullname* for the duration of a test."""
+    set_command(fullname)
+    # XXX maybe set_command should return the class object
+    name = resolve_name(fullname).get_command_name()
+    # XXX maybe we need a public API to remove commands
+    testcase.addCleanup(_COMMANDS.__delitem__, name)
+
+
 def fake_dec(*args, **kw):
     """Fake decorator"""
     def _wrap(func):
diff --git a/Lib/packaging/tests/test_command_build_ext.py b/Lib/packaging/tests/test_command_build_ext.py
--- a/Lib/packaging/tests/test_command_build_ext.py
+++ b/Lib/packaging/tests/test_command_build_ext.py
@@ -3,7 +3,6 @@
 import site
 import sysconfig
 import textwrap
-from io import StringIO
 from packaging.dist import Distribution
 from packaging.errors import (UnknownFileError, CompileError,
                               PackagingPlatformError)
@@ -11,7 +10,7 @@
 from packaging.compiler.extension import Extension
 
 from test.script_helper import assert_python_ok
-from packaging.tests import support, unittest, verbose
+from packaging.tests import support, unittest
 
 
 class BuildExtTestCase(support.TempdirManager,
@@ -37,18 +36,10 @@
         support.fixup_build_ext(cmd)
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
+        cmd.ensure_finalized()
+        cmd.run()
 
-        old_stdout = sys.stdout
-        if not verbose:
-            # silence compiler output
-            sys.stdout = StringIO()
-        try:
-            cmd.ensure_finalized()
-            cmd.run()
-        finally:
-            sys.stdout = old_stdout
-
-        code = """if 1:
+        code = textwrap.dedent("""\
             import sys
             sys.path.insert(0, %r)
 
@@ -63,7 +54,8 @@
             doc = 'This is a template module just for instruction.'
             assert xx.__doc__ == doc
             assert isinstance(xx.Null(), xx.Null)
-            assert isinstance(xx.Str(), xx.Str)"""
+            assert isinstance(xx.Str(), xx.Str)
+            """)
         code = code % self.tmp_dir
         assert_python_ok('-c', code)
 
@@ -388,16 +380,8 @@
         cmd.build_temp = self.tmp_dir
 
         try:
-            old_stdout = sys.stdout
-            if not verbose:
-                # silence compiler output
-                sys.stdout = StringIO()
-            try:
-                cmd.ensure_finalized()
-                cmd.run()
-            finally:
-                sys.stdout = old_stdout
-
+            cmd.ensure_finalized()
+            cmd.run()
         except CompileError:
             self.fail("Wrong deployment target during compilation")
 
diff --git a/Lib/packaging/tests/test_command_build_py.py b/Lib/packaging/tests/test_command_build_py.py
--- a/Lib/packaging/tests/test_command_build_py.py
+++ b/Lib/packaging/tests/test_command_build_py.py
@@ -67,8 +67,6 @@
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
         # create the distribution files.
         sources = self.mkdtemp()
         pkg = os.path.join(sources, 'pkg')
@@ -79,24 +77,16 @@
         open(os.path.join(testdir, "testfile"), "wb").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        #sys.stdout = StringIO.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": sources,
+                             "package_data": {"pkg": ["doc/*"]}})
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": sources,
-                                 "package_data": {"pkg": ["doc/*"]}})
-            dist.script_args = ["build"]
-            dist.parse_command_line()
-
-            try:
-                dist.run_commands()
-            except PackagingFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+            dist.run_commands()
+        except PackagingFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
     def test_byte_compile(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
diff --git a/Lib/packaging/tests/test_command_clean.py b/Lib/packaging/tests/test_command_clean.py
--- a/Lib/packaging/tests/test_command_clean.py
+++ b/Lib/packaging/tests/test_command_clean.py
@@ -36,8 +36,6 @@
                              '%r was not removed' % path)
 
         # let's run the command again (should spit warnings but succeed)
-        cmd.all = True
-        cmd.ensure_finalized()
         cmd.run()
 
 
diff --git a/Lib/packaging/tests/test_command_install_data.py b/Lib/packaging/tests/test_command_install_data.py
--- a/Lib/packaging/tests/test_command_install_data.py
+++ b/Lib/packaging/tests/test_command_install_data.py
@@ -62,6 +62,7 @@
 
         # let's try with warn_dir one
         cmd.warn_dir = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -80,6 +81,7 @@
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_install_distinfo.py b/Lib/packaging/tests/test_command_install_distinfo.py
--- a/Lib/packaging/tests/test_command_install_distinfo.py
+++ b/Lib/packaging/tests/test_command_install_distinfo.py
@@ -49,7 +49,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
@@ -76,7 +76,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.installer = 'bacon-python'
         cmd.ensure_finalized()
         cmd.run()
@@ -96,7 +96,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.requested = False
         cmd.ensure_finalized()
         cmd.run()
@@ -116,7 +116,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.no_record = True
         cmd.ensure_finalized()
         cmd.run()
@@ -214,7 +214,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_register.py b/Lib/packaging/tests/test_command_register.py
--- a/Lib/packaging/tests/test_command_register.py
+++ b/Lib/packaging/tests/test_command_register.py
@@ -12,6 +12,7 @@
     DOCUTILS_SUPPORT = False
 
 from packaging.tests import unittest, support
+from packaging.tests.support import Inputs
 from packaging.command import register as register_module
 from packaging.command.register import register
 from packaging.errors import PackagingSetupError
@@ -38,19 +39,6 @@
 """
 
 
-class Inputs:
-    """Fakes user inputs."""
-    def __init__(self, *answers):
-        self.answers = answers
-        self.index = 0
-
-    def __call__(self, prompt=''):
-        try:
-            return self.answers[self.index]
-        finally:
-            self.index += 1
-
-
 class FakeOpener:
     """Fakes a PyPI server"""
     def __init__(self):
@@ -143,6 +131,7 @@
 
         register_module.input = _no_way
         cmd.show_response = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -140,7 +140,7 @@
 
         # now trying a tar then a gztar
         cmd.formats = ['tar', 'gztar']
-
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -1,7 +1,6 @@
 """Tests for packaging.config."""
 import os
 import sys
-from io import StringIO
 
 from packaging import command
 from packaging.dist import Distribution
@@ -183,13 +182,14 @@
 
     def __init__(self, dist):
         self.distribution = dist
+        self._record = []
 
     @classmethod
     def get_command_name(cls):
         return 'foo'
 
     def run(self):
-        self.distribution.foo_was_here = True
+        self._record.append('foo has run')
 
     def nothing(self):
         pass
@@ -209,21 +209,11 @@
 
     def setUp(self):
         super(ConfigTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        sys.stdout = StringIO()
-        sys.stderr = StringIO()
-
-        self.addCleanup(os.chdir, os.getcwd())
         tempdir = self.mkdtemp()
         self.working_dir = os.getcwd()
         os.chdir(tempdir)
         self.tempdir = tempdir
 
-    def tearDown(self):
-        os.chdir(self.working_dir)
-        super(ConfigTestCase, self).tearDown()
-
     def write_setup(self, kwargs=None):
         opts = {'description-file': 'README', 'extra-files': '',
                 'setup-hooks': 'packaging.tests.test_config.version_hook'}
@@ -378,7 +368,7 @@
         self.assertIn('hooks', sys.modules)
 
     def test_missing_setup_hook_warns(self):
-        self.write_setup({'setup-hooks': 'this.does._not.exist'})
+        self.write_setup({'setup-hooks': 'does._not.exist'})
         self.write_file('README', 'yeah')
         self.get_dist()
         logs = self.get_logs()
@@ -491,10 +481,12 @@
             self.write_file((pkg, '__init__.py'), '#')
 
         # try to run the install command to see if foo is called
+        self.addCleanup(command._COMMANDS.__delitem__, 'foo')
         dist = self.get_dist()
-        self.assertIn('foo', command.get_command_names())
-        self.assertEqual('FooBarBazTest',
-                         dist.get_command_obj('foo').__class__.__name__)
+        dist.run_command('install_dist')
+        cmd = dist.get_command_obj('foo')
+        self.assertEqual(cmd.__class__.__name__, 'FooBarBazTest')
+        self.assertEqual(cmd._record, ['foo has run'])
 
 
 def test_suite():
diff --git a/Lib/packaging/tests/test_create.py b/Lib/packaging/tests/test_create.py
--- a/Lib/packaging/tests/test_create.py
+++ b/Lib/packaging/tests/test_create.py
@@ -2,15 +2,17 @@
 import os
 import sys
 import sysconfig
-from io import StringIO
 from textwrap import dedent
+from packaging import create
 from packaging.create import MainProgram, ask_yn, ask, main
 
 from packaging.tests import support, unittest
+from packaging.tests.support import Inputs
 
 
 class CreateTestCase(support.TempdirManager,
                      support.EnvironRestorer,
+                     support.LoggingCatcher,
                      unittest.TestCase):
 
     maxDiff = None
@@ -18,11 +20,6 @@
 
     def setUp(self):
         super(CreateTestCase, self).setUp()
-        self._stdin = sys.stdin  # TODO use Inputs
-        self._stdout = sys.stdout
-        sys.stdin = StringIO()
-        sys.stdout = StringIO()
-        self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
         # patch sysconfig
@@ -32,29 +29,24 @@
             'doc': sys.prefix + '/share/doc/pyxfoil', }
 
     def tearDown(self):
-        sys.stdin = self._stdin
-        sys.stdout = self._stdout
-        os.chdir(self._cwd)
         sysconfig.get_paths = self._old_get_paths
+        if hasattr(create, 'input'):
+            del create.input
         super(CreateTestCase, self).tearDown()
 
     def test_ask_yn(self):
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         self.assertEqual('y', ask_yn('is this a test'))
 
     def test_ask(self):
-        sys.stdin.write('a\n')
-        sys.stdin.write('b\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('a', 'b')
         self.assertEqual('a', ask('is this a test'))
         self.assertEqual('b', ask(str(list(range(0, 70))), default='c',
                                   lengthy=True))
 
     def test_set_multi(self):
         mainprogram = MainProgram()
-        sys.stdin.write('aaaaa\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('aaaaa')
         mainprogram.data['author'] = []
         mainprogram._set_multi('_set_multi test', 'author')
         self.assertEqual(['aaaaa'], mainprogram.data['author'])
@@ -130,8 +122,7 @@
               scripts=['my_script', 'bin/run'],
               )
         """), encoding='utf-8')
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
@@ -206,9 +197,7 @@
 barbar is now in the public domain,
 ho, baby!
                         '''))
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
-        # FIXME Out of memory error.
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -6,30 +6,32 @@
 import packaging.dist
 
 from packaging.dist import Distribution
-from packaging.command import set_command, _COMMANDS
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
 from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
-from packaging.tests.support import create_distribution
+from packaging.tests.support import create_distribution, use_command
 from test.support import unload
 
 
 class test_dist(Command):
-    """Sample packaging extension command."""
+    """Custom command used for testing."""
 
     user_options = [
-        ("sample-option=", "S", "help text"),
+        ('sample-option=', 'S',
+         "help text"),
         ]
 
     def initialize_options(self):
         self.sample_option = None
+        self._record = []
 
     def finalize_options(self):
-        pass
+        if self.sample_option is None:
+            self.sample_option = 'default value'
 
     def run(self):
-        pass
+        self._record.append('test_dist has run')
 
 
 class DistributionTestCase(support.TempdirManager,
@@ -45,14 +47,10 @@
         # (defaulting to sys.argv)
         self.argv = sys.argv, sys.argv[:]
         del sys.argv[1:]
-        self._commands = _COMMANDS.copy()
 
     def tearDown(self):
         sys.argv = self.argv[0]
         sys.argv[:] = self.argv[1]
-        # XXX maybe we need a public API to remove commands
-        _COMMANDS.clear()
-        _COMMANDS.update(self._commands)
         super(DistributionTestCase, self).tearDown()
 
     @unittest.skip('needs to be updated')
@@ -181,7 +179,8 @@
         self.write_file((temp_home, "config2.cfg"),
                         '[test_dist]\npre-hook.b = type')
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
+
         dist = create_distribution(config_files)
         cmd = dist.get_command_obj("test_dist")
         self.assertEqual(cmd.pre_hook, {"a": 'type', "b": 'type'})
@@ -209,7 +208,7 @@
                 record.append('post-%s' % cmd.get_command_name())
             '''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
 
@@ -236,7 +235,7 @@
             [test_dist]
             pre-hook.test = nonexistent.dotted.name'''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
@@ -251,7 +250,7 @@
             [test_dist]
             pre-hook.test = packaging.tests.test_dist.__doc__'''))
 
-        set_command('packaging.tests.test_dist.test_dist')
+        use_command(self, 'packaging.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
diff --git a/Lib/packaging/tests/test_run.py b/Lib/packaging/tests/test_run.py
--- a/Lib/packaging/tests/test_run.py
+++ b/Lib/packaging/tests/test_run.py
@@ -33,11 +33,9 @@
 
     def setUp(self):
         super(RunTestCase, self).setUp()
-        self.old_stdout = sys.stdout
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
-        sys.stdout = self.old_stdout
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
diff --git a/Lib/packaging/tests/test_uninstall.py b/Lib/packaging/tests/test_uninstall.py
--- a/Lib/packaging/tests/test_uninstall.py
+++ b/Lib/packaging/tests/test_uninstall.py
@@ -1,6 +1,5 @@
 """Tests for the packaging.uninstall module."""
 import os
-import sys
 import logging
 import packaging.util
 
@@ -31,16 +30,12 @@
 
     def setUp(self):
         super(UninstallTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        self.addCleanup(os.chdir, os.getcwd())
         self.addCleanup(enable_cache)
         self.root_dir = self.mkdtemp()
         self.cwd = os.getcwd()
         disable_cache()
 
     def tearDown(self):
-        os.chdir(self.cwd)
         packaging.util._path_created.clear()
         super(UninstallTestCase, self).tearDown()
 
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -170,8 +170,8 @@
     def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
-        sys.old_stdout = self.old_stdout
-        sys.old_stderr = self.old_stderr
+        sys.stdout = self.old_stdout
+        sys.stderr = self.old_stderr
 
     def test_convert_path(self):
         # linux/mac
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -488,8 +488,7 @@
         # Setting 'userbase' is done below the call to the
         # init function to enable using 'get_config_var' in
         # the init-function.
-        if sys.version >= '2.6':
-            _CONFIG_VARS['userbase'] = _getuserbase()
+        _CONFIG_VARS['userbase'] = _getuserbase()
 
         if 'srcdir' not in _CONFIG_VARS:
             _CONFIG_VARS['srcdir'] = _PROJECT_BASE
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -172,6 +172,7 @@
 import json
 import logging
 import os
+import packaging.command
 import packaging.database
 import platform
 import random
@@ -967,7 +968,7 @@
                  'sys.warnoptions', 'threading._dangling',
                  'multiprocessing.process._dangling',
                  'sysconfig._CONFIG_VARS', 'sysconfig._SCHEMES',
-                 'packaging.database_caches',
+                 'packaging.command._COMMANDS', 'packaging.database_caches',
                 )
 
     def get_sys_argv(self):
@@ -1055,6 +1056,22 @@
         # Can't easily revert the logging state
         pass
 
+    def get_packaging_command__COMMANDS(self):
+        # registry mapping command names to full dotted path or to the actual
+        # class (resolved on demand); this check only looks at the names, not
+        # the types of the values (IOW, if a value changes from a string
+        # (dotted path) to a class it's okay but if a key (i.e. command class)
+        # is added we complain)
+        id_ = id(packaging.command._COMMANDS)
+        keys = set(packaging.command._COMMANDS)
+        return id_, keys
+    def restore_packaging_command__COMMANDS(self, saved):
+        # if command._COMMANDS was bound to another dict obhect, we can't
+        # restore the previous object and contents, because the get_ method
+        # above does not return the dict object (to ignore changes in values)
+        for key in packaging.command._COMMANDS.keys() - saved[1]:
+            del packaging.command._COMMANDS[key]
+
     def get_packaging_database_caches(self):
         # caching system used by the PEP 376 implementation
         # we have one boolean and four dictionaries, initially empty
diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py
--- a/Lib/test/test_fileinput.py
+++ b/Lib/test/test_fileinput.py
@@ -7,8 +7,7 @@
 import re
 import fileinput
 import collections
-import types
-import codecs
+import builtins
 import unittest
 
 try:
@@ -807,18 +806,8 @@
 
     @staticmethod
     def replace_builtin_open(new_open_func):
-        builtins_type = type(__builtins__)
-        if builtins_type is dict:
-            original_open = __builtins__["open"]
-            __builtins__["open"] = new_open_func
-        elif builtins_type is types.ModuleType:
-            original_open = __builtins__.open
-            __builtins__.open = new_open_func
-        else:
-            raise RuntimeError(
-                "unknown __builtins__ type: %r (unable to replace open)" %
-                builtins_type)
-
+        original_open = builtins.open
+        builtins.open = new_open_func
         return original_open
 
 class Test_hook_encoded(unittest.TestCase):
@@ -829,21 +818,22 @@
         result = fileinput.hook_encoded(encoding)
 
         fake_open = InvocationRecorder()
-        original_open = codecs.open
-        codecs.open = fake_open
+        original_open = builtins.open
+        builtins.open = fake_open
         try:
             filename = object()
             mode = object()
             open_result = result(filename, mode)
         finally:
-            codecs.open = original_open
+            builtins.open = original_open
 
         self.assertEqual(fake_open.invocation_count, 1)
 
-        args = fake_open.last_invocation[0]
+        args, kwargs = fake_open.last_invocation
         self.assertIs(args[0], filename)
         self.assertIs(args[1], mode)
-        self.assertIs(args[2], encoding)
+        self.assertIs(kwargs.pop('encoding'), encoding)
+        self.assertFalse(kwargs)
 
 def test_main():
     run_unittest(
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -49,6 +49,7 @@
 try:
     import threading
     # The following imports are needed only for tests which
+    # require threading
     import asynchat
     import asyncore
     import errno
@@ -95,9 +96,7 @@
         finally:
             logging._releaseLock()
 
-        # Set two unused loggers: one non-ASCII and one Unicode.
-        # This is to test correct operation when sorting existing
-        # loggers in the configuration code. See issue 8201.
+        # Set two unused loggers
         self.logger1 = logging.getLogger("\xab\xd7\xbb")
         self.logger2 = logging.getLogger("\u013f\u00d6\u0047")
 
@@ -310,8 +309,6 @@
             ('INF.BADPARENT', 'INFO', '4'),
         ])
 
-    def test_invalid_name(self):
-        self.assertRaises(TypeError, logging.getLogger, any)
 
 class BasicFilterTest(BaseTest):
 
@@ -3514,6 +3511,22 @@
         self.addCleanup(setattr, self.logger.manager, 'disable', old_disable)
         self.assertFalse(self.logger.isEnabledFor(22))
 
+    def test_root_logger_aliases(self):
+        root = logging.getLogger()
+        self.assertIs(root, logging.root)
+        self.assertIs(root, logging.getLogger(None))
+        self.assertIs(root, logging.getLogger(''))
+        self.assertIs(root, logging.getLogger('foo').root)
+        self.assertIs(root, logging.getLogger('foo.bar').root)
+        self.assertIs(root, logging.getLogger('foo').parent)
+
+        self.assertIsNot(root, logging.getLogger('\0'))
+        self.assertIsNot(root, logging.getLogger('foo.bar').parent)
+
+    def test_invalid_names(self):
+        self.assertRaises(TypeError, logging.getLogger, any)
+        self.assertRaises(TypeError, logging.getLogger, b'foo')
+
 
 class BaseFileTest(BaseTest):
     "Base class for handler tests that write log files"
diff --git a/Lib/test/test_unicode_file.py b/Lib/test/test_unicode_file.py
--- a/Lib/test/test_unicode_file.py
+++ b/Lib/test/test_unicode_file.py
@@ -56,16 +56,20 @@
         # Should be able to rename the file using either name.
         self.assertTrue(os.path.isfile(filename1)) # must exist.
         os.rename(filename1, filename2 + ".new")
-        self.assertTrue(os.path.isfile(filename1+".new"))
+        self.assertFalse(os.path.isfile(filename2))
+        self.assertTrue(os.path.isfile(filename1 + '.new'))
         os.rename(filename1 + ".new", filename2)
+        self.assertFalse(os.path.isfile(filename1 + '.new'))
         self.assertTrue(os.path.isfile(filename2))
 
         shutil.copy(filename1, filename2 + ".new")
         os.unlink(filename1 + ".new") # remove using equiv name.
         # And a couple of moves, one using each name.
         shutil.move(filename1, filename2 + ".new")
-        self.assertTrue(not os.path.exists(filename2))
+        self.assertFalse(os.path.exists(filename2))
+        self.assertTrue(os.path.exists(filename1 + '.new'))
         shutil.move(filename1 + ".new", filename2)
+        self.assertFalse(os.path.exists(filename2 + '.new'))
         self.assertTrue(os.path.exists(filename1))
         # Note - due to the implementation of shutil.move,
         # it tries a rename first.  This only fails on Windows when on
@@ -73,7 +77,9 @@
         # So we test the shutil.copy2 function, which is the thing most
         # likely to fail.
         shutil.copy2(filename1, filename2 + ".new")
+        self.assertTrue(os.path.isfile(filename1 + '.new'))
         os.unlink(filename1 + ".new")
+        self.assertFalse(os.path.exists(filename2 + '.new'))
 
     def _do_directory(self, make_name, chdir_name):
         cwd = os.getcwdb()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -294,8 +294,6 @@
 - Issue #11320: fix bogus memory management in Modules/getpath.c, leading to
   a possible crash when calling Py_SetPath().
 
-- _ast.__version__ is now a Mercurial hex revision.
-
 - Issue #11432: A bug was introduced in subprocess.Popen on posix systems with
   3.2.0 where the stdout or stderr file descriptor being the same as the stdin
   file descriptor would raise an exception. webbrowser.open would fail. fixed.
@@ -1555,8 +1553,6 @@
   signature.  Without this, architectures where sizeof void* != sizeof int are
   broken.  Patch given by Hallvard B Furuseth.
 
-- Issue #12221: Replace pyexpat.__version__ with the Python version.
-
 - Issue #12051: Fix segfault in json.dumps() while encoding highly-nested
   objects using the C accelerations.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 21:43:53 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 21:43:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Forgot_to_apply_the_futimes?=
	=?utf8?q?at_change=2E?=
Message-ID: <E1RNW2r-0003EK-8P@dinsdale.python.org>

http://hg.python.org/cpython/rev/d1375541d5d8
changeset:   73439:d1375541d5d8
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 14:38:24 2011 -0600
summary:
  Forgot to apply the futimesat change.

files:
  Modules/posixmodule.c |  7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -9686,8 +9686,7 @@
 
 #ifdef HAVE_FUTIMESAT
 PyDoc_STRVAR(posix_futimesat__doc__,
-"futimesat(dirfd, path, (atime, mtime))\n\
-futimesat(dirfd, path, None)\n\n\
+"futimesat(dirfd, path[, (atime, mtime)])\n\
 Like utime() but if path is relative, it is taken as relative to dirfd.\n\
 If path is relative and dirfd is the special value AT_FDCWD, then path\n\
 is interpreted relative to the current working directory.");
@@ -9698,11 +9697,11 @@
     PyObject *opath;
     char *path;
     int res, dirfd;
-    PyObject* arg;
+    PyObject* arg = Py_None;
     time_t atime, mtime;
     long ausec, musec;
 
-    if (!PyArg_ParseTuple(args, "iO&O:futimesat",
+    if (!PyArg_ParseTuple(args, "iO&|O:futimesat",
             &dirfd, PyUnicode_FSConverter, &opath, &arg))
         return NULL;
     path = PyBytes_AsString(opath);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 21:57:09 2011
From: python-checkins at python.org (florent.xicluna)
Date: Mon, 07 Nov 2011 21:57:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Closes_=2313366=3A_fix_test?=
 =?utf8?q?=5Fpep277_failure_on_Windows=2E?=
Message-ID: <E1RNWFh-0004je-OB@dinsdale.python.org>

http://hg.python.org/cpython/rev/655d65bcc939
changeset:   73440:655d65bcc939
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Mon Nov 07 21:56:17 2011 +0100
summary:
  Closes #13366: fix test_pep277 failure on Windows.

files:
  Lib/test/test_pep277.py |  19 ++++++++++++-------
  1 files changed, 12 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -96,9 +96,10 @@
         with self.assertRaises(expected_exception) as c:
             fn(filename)
         exc_filename = c.exception.filename
-        # the "filename" exception attribute may be encoded
-        if isinstance(exc_filename, bytes):
-            filename = filename.encode(sys.getfilesystemencoding())
+        # listdir may append a wildcard to the filename
+        if fn is os.listdir and sys.platform == 'win32':
+            exc_filename, _, wildcard = exc_filename.rpartition(os.sep)
+            self.assertEqual(wildcard, r'*.*')
         if check_filename:
             self.assertEqual(exc_filename, filename, "Function '%s(%a) failed "
                              "with bad filename in the exception: %a" %
@@ -113,8 +114,12 @@
             self._apply_failure(os.chdir, name)
             self._apply_failure(os.rmdir, name)
             self._apply_failure(os.remove, name)
-            # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, check_filename=False)
+            self._apply_failure(os.listdir, name)
+
+    if sys.platform == 'win32':
+        _listdir_failure = FileNotFoundError
+    else:
+        _listdir_failure = NotADirectoryError
 
     def test_open(self):
         for name in self.files:
@@ -122,7 +127,7 @@
             f.write((name+'\n').encode("utf-8"))
             f.close()
             os.stat(name)
-            self._apply_failure(os.listdir, name, NotADirectoryError)
+            self._apply_failure(os.listdir, name, self._listdir_failure)
 
     # Skip the test on darwin, because darwin does normalize the filename to
     # NFD (a variant of Unicode NFD form). Normalize the filename to NFC, NFKC,
@@ -142,7 +147,7 @@
             self._apply_failure(os.rmdir, name)
             self._apply_failure(os.remove, name)
             # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, check_filename=False)
+            self._apply_failure(os.listdir, name)
 
     # Skip the test on darwin, because darwin uses a normalization different
     # than Python NFD normalization: filenames are different even if we use

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 23:10:14 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 23:10:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_=2313327=2E_utimensat_n?=
 =?utf8?q?ow_has_the_atime_and_mtime_arguments_set_as_optional=2C?=
Message-ID: <E1RNXOQ-00032x-NA@dinsdale.python.org>

http://hg.python.org/cpython/rev/59dca1e2363d
changeset:   73441:59dca1e2363d
parent:      73439:d1375541d5d8
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 16:09:20 2011 -0600
summary:
  Fix #13327. utimensat now has the atime and mtime arguments set as optional,
defaulting to None like the other utimes family members. It now accepts
keyword arguments because, unlike other other functions in the family,
it has a `flags` value at the end of the argument list (which
retains its 0 default).

files:
  Doc/library/os.rst     |   8 ++++----
  Lib/test/test_posix.py |   5 +++++
  Modules/posixmodule.c  |  19 ++++++++++++-------
  3 files changed, 21 insertions(+), 11 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -1285,17 +1285,17 @@
    .. versionadded:: 3.3
 
 
-.. function:: utimensat(dirfd, path, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec), flags)
-              utimensat(dirfd, path, None, None, flags)
+.. function:: utimensat(dirfd, path[, atime=(atime_sec, atime_nsec), mtime=(mtime_sec, mtime_nsec), flags=0])
 
    Updates the timestamps of a file with nanosecond precision.
-   The second form sets *atime* and *mtime* to the current time.
+   The *atime* and *mtime* tuples default to ``None``, which sets those
+   values to the current time.
    If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_NOW`, the corresponding
    timestamp is updated to the current time.
    If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_OMIT`, the corresponding
    timestamp is not updated.
    If *path* is relative, it is taken as relative to *dirfd*.
-   *flags* is optional and may be 0 or :data:`AT_SYMLINK_NOFOLLOW`.
+   *flags* is optional and may be 0 (the default) or :data:`AT_SYMLINK_NOFOLLOW`.
    If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path*
    is interpreted relative to the current working directory.
 
diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -815,11 +815,16 @@
         try:
             now = time.time()
             posix.utimensat(f, support.TESTFN, None, None)
+            posix.utimensat(f, support.TESTFN)
+            posix.utimensat(f, support.TESTFN, flags=os.AT_SYMLINK_NOFOLLOW)
             self.assertRaises(TypeError, posix.utimensat, f, support.TESTFN, (None, None), (None, None))
             self.assertRaises(TypeError, posix.utimensat, f, support.TESTFN, (now, 0), None)
             self.assertRaises(TypeError, posix.utimensat, f, support.TESTFN, None, (now, 0))
             posix.utimensat(f, support.TESTFN, (int(now), int((now - int(now)) * 1e9)),
                     (int(now), int((now - int(now)) * 1e9)))
+            posix.utimensat(dirfd=f, path=support.TESTFN,
+                            atime=(int(now), int((now - int(now)) * 1e9)),
+                            mtime=(int(now), int((now - int(now)) * 1e9)))
         finally:
             posix.close(f)
 
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -10019,12 +10019,13 @@
 
 #ifdef HAVE_UTIMENSAT
 PyDoc_STRVAR(posix_utimensat__doc__,
-"utimensat(dirfd, path, (atime_sec, atime_nsec),\n\
-    (mtime_sec, mtime_nsec), flags)\n\
+"utimensat(dirfd, path[, atime=(atime_sec, atime_nsec),\n\
+    mtime=(mtime_sec, mtime_nsec), flags=0])\n\
 utimensat(dirfd, path, None, None, flags)\n\n\
 Updates the timestamps of a file with nanosecond precision. If path is\n\
 relative, it is taken as relative to dirfd.\n\
-The second form sets atime and mtime to the current time.\n\
+If atime and mtime are both None, which is the default, set atime and\n\
+mtime to the current time.\n\
 flags is optional and may be 0 or AT_SYMLINK_NOFOLLOW.\n\
 If path is relative and dirfd is the special value AT_FDCWD, then path\n\
 is interpreted relative to the current working directory.\n\
@@ -10033,16 +10034,19 @@
 If *_nsec is specified as UTIME_OMIT, the timestamp is not updated.");
 
 static PyObject *
-posix_utimensat(PyObject *self, PyObject *args)
+posix_utimensat(PyObject *self, PyObject *args, PyObject *kwargs)
 {
     PyObject *opath;
     char *path;
     int res, dirfd, flags = 0;
-    PyObject *atime, *mtime;
+    PyObject *atime = Py_None;
+    PyObject *mtime = Py_None;
+
+    static char *kwlist[] = {"dirfd", "path", "atime", "mtime", "flags", NULL};
 
     struct timespec buf[2];
 
-    if (!PyArg_ParseTuple(args, "iO&OO|i:utimensat",
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "iO&|OOi:utimensat", kwlist,
             &dirfd, PyUnicode_FSConverter, &opath, &atime, &mtime, &flags))
         return NULL;
     path = PyBytes_AsString(opath);
@@ -10939,7 +10943,8 @@
     {"unlinkat",        posix_unlinkat, METH_VARARGS, posix_unlinkat__doc__},
 #endif
 #ifdef HAVE_UTIMENSAT
-    {"utimensat",       posix_utimensat, METH_VARARGS, posix_utimensat__doc__},
+    {"utimensat",       posix_utimensat, METH_VARARGS | METH_KEYWORDS,
+                        posix_utimensat__doc__},
 #endif
 #ifdef HAVE_MKFIFOAT
     {"mkfifoat",        posix_mkfifoat, METH_VARARGS, posix_mkfifoat__doc__},

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 23:10:15 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 23:10:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_branch_merge=2E?=
Message-ID: <E1RNXOR-000331-GH@dinsdale.python.org>

http://hg.python.org/cpython/rev/c1e48f86b515
changeset:   73442:c1e48f86b515
parent:      73441:59dca1e2363d
parent:      73440:655d65bcc939
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 16:09:54 2011 -0600
summary:
  branch merge.

files:
  Lib/test/test_pep277.py |  19 ++++++++++++-------
  1 files changed, 12 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -96,9 +96,10 @@
         with self.assertRaises(expected_exception) as c:
             fn(filename)
         exc_filename = c.exception.filename
-        # the "filename" exception attribute may be encoded
-        if isinstance(exc_filename, bytes):
-            filename = filename.encode(sys.getfilesystemencoding())
+        # listdir may append a wildcard to the filename
+        if fn is os.listdir and sys.platform == 'win32':
+            exc_filename, _, wildcard = exc_filename.rpartition(os.sep)
+            self.assertEqual(wildcard, r'*.*')
         if check_filename:
             self.assertEqual(exc_filename, filename, "Function '%s(%a) failed "
                              "with bad filename in the exception: %a" %
@@ -113,8 +114,12 @@
             self._apply_failure(os.chdir, name)
             self._apply_failure(os.rmdir, name)
             self._apply_failure(os.remove, name)
-            # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, check_filename=False)
+            self._apply_failure(os.listdir, name)
+
+    if sys.platform == 'win32':
+        _listdir_failure = FileNotFoundError
+    else:
+        _listdir_failure = NotADirectoryError
 
     def test_open(self):
         for name in self.files:
@@ -122,7 +127,7 @@
             f.write((name+'\n').encode("utf-8"))
             f.close()
             os.stat(name)
-            self._apply_failure(os.listdir, name, NotADirectoryError)
+            self._apply_failure(os.listdir, name, self._listdir_failure)
 
     # Skip the test on darwin, because darwin does normalize the filename to
     # NFD (a variant of Unicode NFD form). Normalize the filename to NFC, NFKC,
@@ -142,7 +147,7 @@
             self._apply_failure(os.rmdir, name)
             self._apply_failure(os.remove, name)
             # listdir may append a wildcard to the filename, so dont check
-            self._apply_failure(os.listdir, name, check_filename=False)
+            self._apply_failure(os.listdir, name)
 
     # Skip the test on darwin, because darwin uses a normalization different
     # than Python NFD normalization: filenames are different even if we use

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 23:24:17 2011
From: python-checkins at python.org (florent.xicluna)
Date: Mon, 07 Nov 2011 23:24:17 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Some_win32_platforms_raise_?=
 =?utf8?q?NotADirectoryError=2C_others_FileNotFoundError=2E_Issue?=
Message-ID: <E1RNXc1-0004IH-1n@dinsdale.python.org>

http://hg.python.org/cpython/rev/2cd6b417e488
changeset:   73443:2cd6b417e488
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Mon Nov 07 23:24:08 2011 +0100
summary:
  Some win32 platforms raise NotADirectoryError, others FileNotFoundError. Issue #13366.

files:
  Lib/test/test_pep277.py |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -99,7 +99,7 @@
         # listdir may append a wildcard to the filename
         if fn is os.listdir and sys.platform == 'win32':
             exc_filename, _, wildcard = exc_filename.rpartition(os.sep)
-            self.assertEqual(wildcard, r'*.*')
+            self.assertEqual(wildcard, '*.*')
         if check_filename:
             self.assertEqual(exc_filename, filename, "Function '%s(%a) failed "
                              "with bad filename in the exception: %a" %
@@ -117,7 +117,8 @@
             self._apply_failure(os.listdir, name)
 
     if sys.platform == 'win32':
-        _listdir_failure = FileNotFoundError
+        # Windows is lunatic. Issue #13366.
+        _listdir_failure = NotADirectoryError, FileNotFoundError
     else:
         _listdir_failure = NotADirectoryError
 
@@ -146,7 +147,6 @@
             self._apply_failure(os.chdir, name)
             self._apply_failure(os.rmdir, name)
             self._apply_failure(os.remove, name)
-            # listdir may append a wildcard to the filename, so dont check
             self._apply_failure(os.listdir, name)
 
     # Skip the test on darwin, because darwin uses a normalization different

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov  7 23:30:08 2011
From: python-checkins at python.org (brian.curtin)
Date: Mon, 07 Nov 2011 23:30:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_News_updates_for_=2313327?=
	=?utf8?q?=2E?=
Message-ID: <E1RNXhg-0004rc-NW@dinsdale.python.org>

http://hg.python.org/cpython/rev/5e18ff5476e8
changeset:   73444:5e18ff5476e8
user:        Brian Curtin <brian at python.org>
date:        Mon Nov 07 16:30:01 2011 -0600
summary:
  News updates for #13327.

files:
  Misc/NEWS |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -11,7 +11,9 @@
 -----------------
 
 - Issue #13327: Remove the need for an explicit None as the second argument
-  to os.utime in order to update to the current time.
+  to os.utime, os.lutimes, os.futimes, os.futimens, os.futimesat, in
+  order to update to the current time. Also added keyword argument
+  handling to os.utimensat in order to remove the need for explicit None.
 
 - Issue #13350: Simplify some C code by replacing most usages of
   PyUnicode_Format by PyUnicode_FromFormat.

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue Nov  8 05:34:53 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 08 Nov 2011 05:34:53 +0100
Subject: [Python-checkins] Daily reference leaks (5e18ff5476e8): sum=0
Message-ID: <E1RNdOf-0003c2-Pb@ap.vmr.nerim.net>

results for 5e18ff5476e8 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogGN20bj', '-x']

From python-checkins at python.org  Tue Nov  8 13:12:01 2011
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 08 Nov 2011 13:12:01 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMjM3?=
 =?utf8?q?=3A_Forward_port_subprocess_module_updates_and_explicitly_docume?=
 =?utf8?q?nt?=
Message-ID: <E1RNkX3-0003zm-6z@dinsdale.python.org>

http://hg.python.org/cpython/rev/e929d2a96d9b
changeset:   73445:e929d2a96d9b
branch:      3.2
parent:      73430:4facbfdc7700
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue Nov 08 20:49:23 2011 +1000
summary:
  Issue #13237: Forward port subprocess module updates and explicitly document UTF-8 encoding assumption when universal_newlines=True

files:
  Doc/library/subprocess.rst |  440 +++++++++++++++++-------
  Misc/NEWS                  |    3 +
  2 files changed, 306 insertions(+), 137 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -25,7 +25,227 @@
 Using the subprocess Module
 ---------------------------
 
-This module defines one class called :class:`Popen`:
+The recommended approach to invoking subprocesses is to use the following
+convenience functions for all use cases they can handle. For more advanced
+use cases, the underlying :class:`Popen` interface can be used directly.
+
+
+.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
+
+   Run the command described by *args*.  Wait for command to complete, then
+   return the :attr:`returncode` attribute.
+
+   The arguments shown above are merely the most common ones, described below
+   in :ref:`frequently-used-arguments` (hence the slightly odd notation in
+   the abbreviated signature). The full function signature is the same as
+   that of the :class:`Popen` constructor - this functions passes all
+   supplied arguments directly through to that interface.
+
+   Examples::
+
+      >>> subprocess.call(["ls", "-l"])
+      0
+
+      >>> subprocess.call("exit 1", shell=True)
+      1
+
+   .. warning::
+
+      Invoking the system shell with ``shell=True`` can be a security hazard
+      if combined with untrusted input. See the warning under
+      :ref:`frequently-used-arguments` for details.
+
+   .. note::
+
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
+      the pipes are not being read in the current process, the child
+      process may block if it generates enough output to a pipe to fill up
+      the OS pipe buffer.
+
+
+.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
+
+   Run command with arguments.  Wait for command to complete. If the return
+   code was zero then return, otherwise raise :exc:`CalledProcessError`. The
+   :exc:`CalledProcessError` object will have the return code in the
+   :attr:`returncode` attribute.
+
+   The arguments shown above are merely the most common ones, described below
+   in :ref:`frequently-used-arguments` (hence the slightly odd notation in
+   the abbreviated signature). The full function signature is the same as
+   that of the :class:`Popen` constructor - this functions passes all
+   supplied arguments directly through to that interface.
+
+   Examples::
+
+      >>> subprocess.check_call(["ls", "-l"])
+      0
+
+      >>> subprocess.check_call("exit 1", shell=True)
+      Traceback (most recent call last):
+         ...
+      subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
+
+   .. versionadded:: 2.5
+
+   .. warning::
+
+      Invoking the system shell with ``shell=True`` can be a security hazard
+      if combined with untrusted input. See the warning under
+      :ref:`frequently-used-arguments` for details.
+
+   .. note::
+
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
+      the pipes are not being read in the current process, the child
+      process may block if it generates enough output to a pipe to fill up
+      the OS pipe buffer.
+
+
+.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False)
+
+   Run command with arguments and return its output as a byte string.
+
+   If the return code was non-zero it raises a :exc:`CalledProcessError`. The
+   :exc:`CalledProcessError` object will have the return code in the
+   :attr:`returncode` attribute and any output in the :attr:`output`
+   attribute.
+
+   The arguments shown above are merely the most common ones, described below
+   in :ref:`frequently-used-arguments` (hence the slightly odd notation in
+   the abbreviated signature). The full function signature is largely the
+   same as that of the :class:`Popen` constructor, except that *stdout* is
+   not permitted as it is used internally. All other supplied arguments are
+   passed directly through to the :class:`Popen` constructor.
+
+   Examples::
+
+      >>> subprocess.check_output(["echo", "Hello World!"])
+      b'Hello World!\n'
+
+      >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True)
+      'Hello World!\n'
+
+      >>> subprocess.check_output("exit 1", shell=True)
+      Traceback (most recent call last):
+         ...
+      subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
+
+   By default, this function will return the data as encoded bytes. The actual
+   encoding of the output data may depend on the command being invoked, so the
+   decoding to text will often need to be handled at the application level.
+
+   This behaviour may be overridden by setting *universal_newlines* to
+   :const:`True` as described below in :ref:`frequently-used-arguments`.
+
+   To also capture standard error in the result, use
+   ``stderr=subprocess.STDOUT``::
+
+      >>> subprocess.check_output(
+      ...     "ls non_existent_file; exit 0",
+      ...     stderr=subprocess.STDOUT,
+      ...     shell=True)
+      'ls: non_existent_file: No such file or directory\n'
+
+   .. versionadded:: 2.7
+
+   .. warning::
+
+      Invoking the system shell with ``shell=True`` can be a security hazard
+      if combined with untrusted input. See the warning under
+      :ref:`frequently-used-arguments` for details.
+
+   .. note::
+
+      Do not use ``stderr=PIPE`` with this function. As the pipe is not being
+      read in the current process, the child process may block if it
+      generates enough output to the pipe to fill up the OS pipe buffer.
+
+
+.. data:: PIPE
+
+   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
+   to :class:`Popen` and indicates that a pipe to the standard stream should be
+   opened.
+
+
+.. data:: STDOUT
+
+   Special value that can be used as the *stderr* argument to :class:`Popen` and
+   indicates that standard error should go into the same handle as standard
+   output.
+
+
+.. _frequently-used-arguments:
+
+Frequently Used Arguments
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To support a wide variety of use cases, the :class:`Popen` constructor (and
+the convenience functions) accept a large number of optional arguments. For
+most typical use cases, many of these arguments can be safely left at their
+default values. The arguments that are most commonly needed are:
+
+   *args* is required for all calls and should be a string, or a sequence of
+   program arguments. Providing a sequence of arguments is generally
+   preferred, as it allows the module to take care of any required escaping
+   and quoting of arguments (e.g. to permit spaces in file names). If passing
+   a single string, either *shell* must be :const:`True` (see below) or else
+   the string must simply name the program to be executed without specifying
+   any arguments.
+
+   *stdin*, *stdout* and *stderr* specify the executed program's standard input,
+   standard output and standard error file handles, respectively.  Valid values
+   are :data:`PIPE`, an existing file descriptor (a positive integer), an
+   existing file object, and ``None``.  :data:`PIPE` indicates that a new pipe
+   to the child should be created.  With the default settings of ``None``, no
+   redirection will occur; the child's file handles will be inherited from the
+   parent.  Additionally, *stderr* can be :data:`STDOUT`, which indicates that
+   the stderr data from the child process should be captured into the same file
+   handle as for stdout.
+
+   When *stdout* or *stderr* are pipes and *universal_newlines* is
+   :const:`True` then the output data is assumed to be encoded as UTF-8 and
+   will automatically be decoded to text. All line endings will be converted
+   to ``'\n'`` as described for the universal newlines `'U'`` mode argument
+   to :func:`open`.
+
+   If *shell* is :const:`True`, the specified command will be executed through
+   the shell. This can be useful if you are using Python primarily for the
+   enhanced control flow it offers over most system shells and still want
+   access to other shell features such as filename wildcards, shell pipes and
+   environment variable expansion.
+
+   .. warning::
+
+      Executing shell commands that incorporate unsanitized input from an
+      untrusted source makes a program vulnerable to `shell injection
+      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
+      a serious security flaw which can result in arbitrary command execution.
+      For this reason, the use of *shell=True* is **strongly discouraged** in cases
+      where the command string is constructed from external input::
+
+         >>> from subprocess import call
+         >>> filename = input("What file would you like to display?\n")
+         What file would you like to display?
+         non_existent; rm -rf / #
+         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
+
+      ``shell=False`` disables all shell based features, but does not suffer
+      from this vulnerability; see the Note in the :class:`Popen` constructor
+      documentation for helpful hints in getting ``shell=False`` to work.
+
+These options, along with all of the other options, are described in more
+detail in the :class:`Popen` constructor documentation.
+
+
+Popen Constuctor
+^^^^^^^^^^^^^^^^
+
+The underlying process creation and management in this module is handled by
+the :class:`Popen` class. It offers a lot of flexibility so that developers
+are able to handle the less common cases not covered by the convenience
+functions.
 
 
 .. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=())
@@ -78,21 +298,9 @@
 
    .. warning::
 
-      Executing shell commands that incorporate unsanitized input from an
-      untrusted source makes a program vulnerable to `shell injection
-      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
-      a serious security flaw which can result in arbitrary command execution.
-      For this reason, the use of *shell=True* is **strongly discouraged** in cases
-      where the command string is constructed from external input::
-
-         >>> from subprocess import call
-         >>> filename = input("What file would you like to display?\n")
-         What file would you like to display?
-         non_existent; rm -rf / #
-         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
-
-      *shell=False* does not suffer from this vulnerability; the above Note may be
-      helpful in getting code using *shell=False* to work.
+      Enabling this option can be a security hazard if combined with untrusted
+      input. See the warning under :ref:`frequently-used-arguments`
+      for details.
 
    On Windows: the :class:`Popen` class uses CreateProcess() to execute the
    child program, which operates on strings.  If *args* is a sequence, it will
@@ -121,14 +329,15 @@
    You don't need ``shell=True`` to run a batch file, nor to run a console-based
    executable.
 
-   *stdin*, *stdout* and *stderr* specify the executed programs' standard input,
+   *stdin*, *stdout* and *stderr* specify the executed program's standard input,
    standard output and standard error file handles, respectively.  Valid values
    are :data:`PIPE`, an existing file descriptor (a positive integer), an
    existing :term:`file object`, and ``None``.  :data:`PIPE` indicates that a
-   new pipe to the child should be created.  With ``None``, no redirection will
-   occur; the child's file handles will be inherited from the parent.  Additionally,
-   *stderr* can be :data:`STDOUT`, which indicates that the stderr data from the
-   applications should be captured into the same file handle as for stdout.
+   new pipe to the child should be created.  With the default settings of
+   ``None``, no redirection will occur; the child's file handles will be
+   inherited from the parent.  Additionally, *stderr* can be :data:`STDOUT`,
+   which indicates that the stderr data from the applications should be
+   captured into the same file handle as for stdout.
 
    If *preexec_fn* is set to a callable object, this object will be called in the
    child process just before the child is executed.
@@ -242,104 +451,6 @@
    output.
 
 
-Convenience Functions
-^^^^^^^^^^^^^^^^^^^^^
-
-This module also defines the following shortcut functions:
-
-
-.. function:: call(*popenargs, **kwargs)
-
-   Run command with arguments.  Wait for command to complete, then return the
-   :attr:`returncode` attribute.
-
-   The arguments are the same as for the :class:`Popen` constructor.  Example::
-
-      >>> retcode = subprocess.call(["ls", "-l"])
-
-   .. warning::
-
-      Like :meth:`Popen.wait`, this will deadlock when using
-      ``stdout=PIPE`` and/or ``stderr=PIPE`` and the child process
-      generates enough output to a pipe such that it blocks waiting
-      for the OS pipe buffer to accept more data.
-
-
-.. function:: check_call(*popenargs, **kwargs)
-
-   Run command with arguments.  Wait for command to complete. If the exit code was
-   zero then return, otherwise raise :exc:`CalledProcessError`. The
-   :exc:`CalledProcessError` object will have the return code in the
-   :attr:`returncode` attribute.
-
-   The arguments are the same as for the :class:`Popen` constructor.  Example::
-
-      >>> subprocess.check_call(["ls", "-l"])
-      0
-
-   .. warning::
-
-      See the warning for :func:`call`.
-
-
-.. function:: check_output(*popenargs, **kwargs)
-
-   Run command with arguments and return its output as a byte string.
-
-   If the exit code was non-zero it raises a :exc:`CalledProcessError`.  The
-   :exc:`CalledProcessError` object will have the return code in the
-   :attr:`returncode`
-   attribute and output in the :attr:`output` attribute.
-
-   The arguments are the same as for the :class:`Popen` constructor.  Example::
-
-      >>> subprocess.check_output(["ls", "-l", "/dev/null"])
-      b'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
-
-   The stdout argument is not allowed as it is used internally.
-   To capture standard error in the result, use ``stderr=subprocess.STDOUT``::
-
-      >>> subprocess.check_output(
-      ...     ["/bin/sh", "-c", "ls non_existent_file; exit 0"],
-      ...     stderr=subprocess.STDOUT)
-      b'ls: non_existent_file: No such file or directory\n'
-
-   .. versionadded:: 3.1
-
-
-.. function:: getstatusoutput(cmd)
-
-   Return ``(status, output)`` of executing *cmd* in a shell.
-
-   Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple
-   ``(status, output)``.  *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the
-   returned output will contain output or error messages.  A trailing newline is
-   stripped from the output.  The exit status for the command can be interpreted
-   according to the rules for the C function :c:func:`wait`.  Example::
-
-      >>> subprocess.getstatusoutput('ls /bin/ls')
-      (0, '/bin/ls')
-      >>> subprocess.getstatusoutput('cat /bin/junk')
-      (256, 'cat: /bin/junk: No such file or directory')
-      >>> subprocess.getstatusoutput('/bin/junk')
-      (256, 'sh: /bin/junk: not found')
-
-   Availability: UNIX.
-
-
-.. function:: getoutput(cmd)
-
-   Return output (stdout and stderr) of executing *cmd* in a shell.
-
-   Like :func:`getstatusoutput`, except the exit status is ignored and the return
-   value is a string containing the command's output.  Example::
-
-      >>> subprocess.getoutput('ls /bin/ls')
-      '/bin/ls'
-
-   Availability: UNIX.
-
-
 Exceptions
 ^^^^^^^^^^
 
@@ -355,16 +466,19 @@
 A :exc:`ValueError` will be raised if :class:`Popen` is called with invalid
 arguments.
 
-check_call() will raise :exc:`CalledProcessError`, if the called process returns
-a non-zero return code.
+:func:`check_call` and :func:`check_output` will raise
+:exc:`CalledProcessError` if the called process returns a non-zero return
+code.
 
 
 Security
 ^^^^^^^^
 
-Unlike some other popen functions, this implementation will never call /bin/sh
-implicitly.  This means that all characters, including shell metacharacters, can
-safely be passed to child processes.
+Unlike some other popen functions, this implementation will never call a
+system shell implicitly.  This means that all characters, including shell
+metacharacters, can safely be passed to child processes. Obviously, if the
+shell is invoked explicitly, then it is the application's responsibility to
+ensure that all whitespace and metacharacters are quoted appropriately.
 
 
 Popen Objects
@@ -592,15 +706,21 @@
 Replacing Older Functions with the subprocess Module
 ----------------------------------------------------
 
-In this section, "a ==> b" means that b can be used as a replacement for a.
+In this section, "a becomes b" means that b can be used as a replacement for a.
 
 .. note::
 
-   All functions in this section fail (more or less) silently if the executed
-   program cannot be found; this module raises an :exc:`OSError` exception.
+   All "a" functions in this section fail (more or less) silently if the
+   executed program cannot be found; the "b" replacements raise :exc:`OSError`
+   instead.
 
-In the following examples, we assume that the subprocess module is imported with
-"from subprocess import \*".
+   In addition, the replacements using :func:`check_output` will fail with a
+   :exc:`CalledProcessError` if the requested operation produces a non-zero
+   return code. The output is still available as the ``output`` attribute of
+   the raised exception.
+
+In the following examples, we assume that the relevant functions have already
+been imported from the subprocess module.
 
 
 Replacing /bin/sh shell backquote
@@ -609,8 +729,8 @@
 ::
 
    output=`mycmd myarg`
-   ==>
-   output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
+   # becomes
+   output = check_output(["mycmd", "myarg"])
 
 
 Replacing shell pipeline
@@ -619,7 +739,7 @@
 ::
 
    output=`dmesg | grep hda`
-   ==>
+   # becomes
    p1 = Popen(["dmesg"], stdout=PIPE)
    p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
    p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
@@ -628,22 +748,27 @@
 The p1.stdout.close() call after starting the p2 is important in order for p1
 to receive a SIGPIPE if p2 exits before p1.
 
+Alternatively, for trusted input, the shell's own pipeline support may still
+be used directly:
+
+   output=`dmesg | grep hda`
+   # becomes
+   output=check_output("dmesg | grep hda", shell=True)
+
+
 Replacing :func:`os.system`
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
    sts = os.system("mycmd" + " myarg")
-   ==>
-   p = Popen("mycmd" + " myarg", shell=True)
-   sts = os.waitpid(p.pid, 0)[1]
+   # becomes
+   sts = call("mycmd" + " myarg", shell=True)
 
 Notes:
 
 * Calling the program through the shell is usually not required.
 
-* It's easier to look at the :attr:`returncode` attribute than the exit status.
-
 A more realistic example would look like this::
 
    try:
@@ -768,6 +893,7 @@
   ``close_fds=True`` with :class:`Popen` to guarantee this behavior on
   all platforms or past Python versions.
 
+
 Notes
 -----
 
@@ -801,3 +927,43 @@
    described in rule 3.
 
 
+Legacy Shell Invocation Functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This module also provides the following legacy functions from the 2.x
+``commands`` module. These operations implicitly invoke the system shell and
+none of the guarantees described above regarding security and exception
+handling consistency are valid for these functions.
+
+.. function:: getstatusoutput(cmd)
+
+   Return ``(status, output)`` of executing *cmd* in a shell.
+
+   Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple
+   ``(status, output)``.  *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the
+   returned output will contain output or error messages.  A trailing newline is
+   stripped from the output.  The exit status for the command can be interpreted
+   according to the rules for the C function :c:func:`wait`.  Example::
+
+      >>> subprocess.getstatusoutput('ls /bin/ls')
+      (0, '/bin/ls')
+      >>> subprocess.getstatusoutput('cat /bin/junk')
+      (256, 'cat: /bin/junk: No such file or directory')
+      >>> subprocess.getstatusoutput('/bin/junk')
+      (256, 'sh: /bin/junk: not found')
+
+   Availability: UNIX.
+
+
+.. function:: getoutput(cmd)
+
+   Return output (stdout and stderr) of executing *cmd* in a shell.
+
+   Like :func:`getstatusoutput`, except the exit status is ignored and the return
+   value is a string containing the command's output.  Example::
+
+      >>> subprocess.getoutput('ls /bin/ls')
+      '/bin/ls'
+
+   Availability: UNIX.
+
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -259,6 +259,9 @@
 Documentation
 -------------
 
+- Issue #13237: Reorganise subprocess documentation to emphasise convenience
+  functions and the most commonly needed arguments to Popen.
+
 - Issue #13141: Demonstrate recommended style for socketserver examples.
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 13:12:01 2011
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 08 Nov 2011 13:12:01 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMjM3?=
 =?utf8?q?=3A_Remove_duplicate_data_value_descriptions_from_the_subprocess?=
 =?utf8?q?_docs?=
Message-ID: <E1RNkX3-0003zt-V3@dinsdale.python.org>

http://hg.python.org/cpython/rev/d3b159c43ee8
changeset:   73446:d3b159c43ee8
branch:      3.2
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue Nov 08 21:39:07 2011 +1000
summary:
  Issue #13237: Remove duplicate data value descriptions from the subprocess docs

files:
  Doc/library/subprocess.rst |  14 --------------
  1 files changed, 0 insertions(+), 14 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -437,20 +437,6 @@
       Added context manager support.
 
 
-.. data:: PIPE
-
-   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
-   to :class:`Popen` and indicates that a pipe to the standard stream should be
-   opened.
-
-
-.. data:: STDOUT
-
-   Special value that can be used as the *stderr* argument to :class:`Popen` and
-   indicates that standard error should go into the same handle as standard
-   output.
-
-
 Exceptions
 ^^^^^^^^^^
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 13:12:02 2011
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 08 Nov 2011 13:12:02 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMjM3?=
 =?utf8?q?=3A_Fix_formatting_error_-_the_legacy_shell_commands_weren=27t_m?=
 =?utf8?q?eant_to?=
Message-ID: <E1RNkX4-0003zx-Mm@dinsdale.python.org>

http://hg.python.org/cpython/rev/7545f4fb450c
changeset:   73447:7545f4fb450c
branch:      3.2
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue Nov 08 21:50:58 2011 +1000
summary:
  Issue #13237: Fix formatting error - the legacy shell commands weren't meant to be under the Notes heading

files:
  Doc/library/subprocess.rst |  83 ++++++++++++-------------
  1 files changed, 41 insertions(+), 42 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -880,6 +880,47 @@
   all platforms or past Python versions.
 
 
+Legacy Shell Invocation Functions
+---------------------------------
+
+This module also provides the following legacy functions from the 2.x
+``commands`` module. These operations implicitly invoke the system shell and
+none of the guarantees described above regarding security and exception
+handling consistency are valid for these functions.
+
+.. function:: getstatusoutput(cmd)
+
+   Return ``(status, output)`` of executing *cmd* in a shell.
+
+   Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple
+   ``(status, output)``.  *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the
+   returned output will contain output or error messages.  A trailing newline is
+   stripped from the output.  The exit status for the command can be interpreted
+   according to the rules for the C function :c:func:`wait`.  Example::
+
+      >>> subprocess.getstatusoutput('ls /bin/ls')
+      (0, '/bin/ls')
+      >>> subprocess.getstatusoutput('cat /bin/junk')
+      (256, 'cat: /bin/junk: No such file or directory')
+      >>> subprocess.getstatusoutput('/bin/junk')
+      (256, 'sh: /bin/junk: not found')
+
+   Availability: UNIX.
+
+
+.. function:: getoutput(cmd)
+
+   Return output (stdout and stderr) of executing *cmd* in a shell.
+
+   Like :func:`getstatusoutput`, except the exit status is ignored and the return
+   value is a string containing the command's output.  Example::
+
+      >>> subprocess.getoutput('ls /bin/ls')
+      '/bin/ls'
+
+   Availability: UNIX.
+
+
 Notes
 -----
 
@@ -911,45 +952,3 @@
    backslash.  If the number of backslashes is odd, the last
    backslash escapes the next double quotation mark as
    described in rule 3.
-
-
-Legacy Shell Invocation Functions
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-This module also provides the following legacy functions from the 2.x
-``commands`` module. These operations implicitly invoke the system shell and
-none of the guarantees described above regarding security and exception
-handling consistency are valid for these functions.
-
-.. function:: getstatusoutput(cmd)
-
-   Return ``(status, output)`` of executing *cmd* in a shell.
-
-   Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple
-   ``(status, output)``.  *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the
-   returned output will contain output or error messages.  A trailing newline is
-   stripped from the output.  The exit status for the command can be interpreted
-   according to the rules for the C function :c:func:`wait`.  Example::
-
-      >>> subprocess.getstatusoutput('ls /bin/ls')
-      (0, '/bin/ls')
-      >>> subprocess.getstatusoutput('cat /bin/junk')
-      (256, 'cat: /bin/junk: No such file or directory')
-      >>> subprocess.getstatusoutput('/bin/junk')
-      (256, 'sh: /bin/junk: not found')
-
-   Availability: UNIX.
-
-
-.. function:: getoutput(cmd)
-
-   Return output (stdout and stderr) of executing *cmd* in a shell.
-
-   Like :func:`getstatusoutput`, except the exit status is ignored and the return
-   value is a string containing the command's output.  Example::
-
-      >>> subprocess.getoutput('ls /bin/ls')
-      '/bin/ls'
-
-   Availability: UNIX.
-

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 13:12:03 2011
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 08 Nov 2011 13:12:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313237=3A_Forward_port_from_3=2E2_of_subprocess_docu?=
 =?utf8?q?mentation_updates=2E_Needed?=
Message-ID: <E1RNkX5-00040G-Ic@dinsdale.python.org>

http://hg.python.org/cpython/rev/0b50008bb953
changeset:   73448:0b50008bb953
parent:      73444:5e18ff5476e8
parent:      73447:7545f4fb450c
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue Nov 08 22:11:21 2011 +1000
summary:
  Issue #13237: Forward port from 3.2 of subprocess documentation updates. Needed quite a few adjustments to account for new features coming in 3.3

files:
  Doc/library/subprocess.rst |  523 ++++++++++++++++--------
  Misc/NEWS                  |    3 +
  2 files changed, 340 insertions(+), 186 deletions(-)


diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -25,7 +25,260 @@
 Using the subprocess Module
 ---------------------------
 
-This module defines one class called :class:`Popen`:
+The recommended approach to invoking subprocesses is to use the following
+convenience functions for all use cases they can handle. For more advanced
+use cases, the underlying :class:`Popen` interface can be used directly.
+
+
+.. function:: call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
+
+   Run the command described by *args*.  Wait for command to complete, then
+   return the :attr:`returncode` attribute.
+
+   The arguments shown above are merely the most common ones, described below
+   in :ref:`frequently-used-arguments` (hence the use of keyword-only notation
+   in the abbreviated signature). The full function signature is largely the
+   same as that of the :class:`Popen` constructor - this function passes all
+   supplied arguments other than *timeout* directly through to that interface.
+
+   The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout
+   expires, the child process will be killed and then waited for again.  The
+   :exc:`TimeoutExpired` exception will be re-raised after the child process
+   has terminated.
+
+   Examples::
+
+      >>> subprocess.call(["ls", "-l"])
+      0
+
+      >>> subprocess.call("exit 1", shell=True)
+      1
+
+   .. warning::
+
+      Invoking the system shell with ``shell=True`` can be a security hazard
+      if combined with untrusted input. See the warning under
+      :ref:`frequently-used-arguments` for details.
+
+   .. note::
+
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
+      the pipes are not being read in the current process, the child
+      process may block if it generates enough output to a pipe to fill up
+      the OS pipe buffer.
+
+   .. versionchanged:: 3.3
+      *timeout* was added.
+
+
+.. function:: check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
+
+   Run command with arguments.  Wait for command to complete. If the return
+   code was zero then return, otherwise raise :exc:`CalledProcessError`. The
+   :exc:`CalledProcessError` object will have the return code in the
+   :attr:`returncode` attribute.
+
+   The arguments shown above are merely the most common ones, described below
+   in :ref:`frequently-used-arguments` (hence the use of keyword-only notation
+   in the abbreviated signature). The full function signature is largely the
+   same as that of the :class:`Popen` constructor - this function passes all
+   supplied arguments other than *timeout* directly through to that interface.
+
+   The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout
+   expires, the child process will be killed and then waited for again.  The
+   :exc:`TimeoutExpired` exception will be re-raised after the child process
+   has terminated.
+
+   Examples::
+
+      >>> subprocess.check_call(["ls", "-l"])
+      0
+
+      >>> subprocess.check_call("exit 1", shell=True)
+      Traceback (most recent call last):
+         ...
+      subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
+
+   .. warning::
+
+      Invoking the system shell with ``shell=True`` can be a security hazard
+      if combined with untrusted input. See the warning under
+      :ref:`frequently-used-arguments` for details.
+
+   .. note::
+
+      Do not use ``stdout=PIPE`` or ``stderr=PIPE`` with this function. As
+      the pipes are not being read in the current process, the child
+      process may block if it generates enough output to a pipe to fill up
+      the OS pipe buffer.
+
+   .. versionchanged:: 3.3
+      *timeout* was added.
+
+
+.. function:: check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False, timeout=None)
+
+   Run command with arguments and return its output as a byte string.
+
+   If the return code was non-zero it raises a :exc:`CalledProcessError`. The
+   :exc:`CalledProcessError` object will have the return code in the
+   :attr:`returncode` attribute and any output in the :attr:`output`
+   attribute.
+
+   The arguments shown above are merely the most common ones, described below
+   in :ref:`frequently-used-arguments` (hence the use of keyword-only notation
+   in the abbreviated signature). The full function signature is largely the
+   same as that of the :class:`Popen` constructor - this functions passes all
+   supplied arguments other than *timeout* directly through to that interface.
+   In addition, *stdout* is not permitted as an argument, as it is used
+   internally to collect the output from the subprocess.
+
+   The *timeout* argument is passed to :meth:`Popen.wait`. If the timeout
+   expires, the child process will be killed and then waited for again.  The
+   :exc:`TimeoutExpired` exception will be re-raised after the child process
+   has terminated.
+
+   Examples::
+
+      >>> subprocess.check_output(["echo", "Hello World!"])
+      b'Hello World!\n'
+
+      >>> subprocess.check_output(["echo", "Hello World!"], universal_newlines=True)
+      'Hello World!\n'
+
+      >>> subprocess.check_output("exit 1", shell=True)
+      Traceback (most recent call last):
+         ...
+      subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
+
+   By default, this function will return the data as encoded bytes. The actual
+   encoding of the output data may depend on the command being invoked, so the
+   decoding to text will often need to be handled at the application level.
+
+   This behaviour may be overridden by setting *universal_newlines* to
+   :const:`True` as described below in :ref:`frequently-used-arguments`.
+
+   To also capture standard error in the result, use
+   ``stderr=subprocess.STDOUT``::
+
+      >>> subprocess.check_output(
+      ...     "ls non_existent_file; exit 0",
+      ...     stderr=subprocess.STDOUT,
+      ...     shell=True)
+      'ls: non_existent_file: No such file or directory\n'
+
+   .. versionadded:: 3.1
+
+   .. warning::
+
+      Invoking the system shell with ``shell=True`` can be a security hazard
+      if combined with untrusted input. See the warning under
+      :ref:`frequently-used-arguments` for details.
+
+   .. note::
+
+      Do not use ``stderr=PIPE`` with this function. As the pipe is not being
+      read in the current process, the child process may block if it
+      generates enough output to the pipe to fill up the OS pipe buffer.
+
+   .. versionchanged:: 3.3
+      *timeout* was added.
+
+
+.. data:: DEVNULL
+
+   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
+   to :class:`Popen` and indicates that the special file :data:`os.devnull`
+   will be used.
+
+   .. versionadded:: 3.3
+
+
+.. data:: PIPE
+
+   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
+   to :class:`Popen` and indicates that a pipe to the standard stream should be
+   opened.
+
+
+.. data:: STDOUT
+
+   Special value that can be used as the *stderr* argument to :class:`Popen` and
+   indicates that standard error should go into the same handle as standard
+   output.
+
+
+.. _frequently-used-arguments:
+
+Frequently Used Arguments
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To support a wide variety of use cases, the :class:`Popen` constructor (and
+the convenience functions) accept a large number of optional arguments. For
+most typical use cases, many of these arguments can be safely left at their
+default values. The arguments that are most commonly needed are:
+
+   *args* is required for all calls and should be a string, or a sequence of
+   program arguments. Providing a sequence of arguments is generally
+   preferred, as it allows the module to take care of any required escaping
+   and quoting of arguments (e.g. to permit spaces in file names). If passing
+   a single string, either *shell* must be :const:`True` (see below) or else
+   the string must simply name the program to be executed without specifying
+   any arguments.
+
+   *stdin*, *stdout* and *stderr* specify the executed program's standard input,
+   standard output and standard error file handles, respectively.  Valid values
+   are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive
+   integer), an existing file object, and ``None``.  :data:`PIPE` indicates
+   that a new pipe to the child should be created.  :data:`DEVNULL` indicates
+   that the special file :data:`os.devnull` will be used.  With the default
+   settings of ``None``, no redirection will occur; the child's file handles
+   will be inherited from the parent.  Additionally, *stderr* can be
+   :data:`STDOUT`, which indicates that the stderr data from the child
+   process should be captured into the same file handle as for *stdout*.
+
+   When *stdout* or *stderr* are pipes and *universal_newlines* is
+   :const:`True` then the output data is assumed to be encoded as UTF-8 and
+   will automatically be decoded to text. All line endings will be converted
+   to ``'\n'`` as described for the universal newlines `'U'`` mode argument
+   to :func:`open`.
+
+   If *shell* is :const:`True`, the specified command will be executed through
+   the shell. This can be useful if you are using Python primarily for the
+   enhanced control flow it offers over most system shells and still want
+   access to other shell features such as filename wildcards, shell pipes and
+   environment variable expansion.
+
+   .. warning::
+
+      Executing shell commands that incorporate unsanitized input from an
+      untrusted source makes a program vulnerable to `shell injection
+      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
+      a serious security flaw which can result in arbitrary command execution.
+      For this reason, the use of *shell=True* is **strongly discouraged** in cases
+      where the command string is constructed from external input::
+
+         >>> from subprocess import call
+         >>> filename = input("What file would you like to display?\n")
+         What file would you like to display?
+         non_existent; rm -rf / #
+         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
+
+      ``shell=False`` disables all shell based features, but does not suffer
+      from this vulnerability; see the Note in the :class:`Popen` constructor
+      documentation for helpful hints in getting ``shell=False`` to work.
+
+These options, along with all of the other options, are described in more
+detail in the :class:`Popen` constructor documentation.
+
+
+Popen Constuctor
+^^^^^^^^^^^^^^^^
+
+The underlying process creation and management in this module is handled by
+the :class:`Popen` class. It offers a lot of flexibility so that developers
+are able to handle the less common cases not covered by the convenience
+functions.
 
 
 .. class:: Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=())
@@ -78,22 +331,9 @@
 
    .. warning::
 
-      Executing shell commands that incorporate unsanitized input from an
-      untrusted source makes a program vulnerable to `shell injection
-      <http://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_,
-      a serious security flaw which can result in arbitrary command execution.
-      For this reason, the use of *shell=True* is **strongly discouraged** in cases
-      where the command string is constructed from external input::
-
-         >>> from subprocess import call
-         >>> filename = input("What file would you like to display?\n")
-         What file would you like to display?
-         non_existent; rm -rf / #
-         >>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...
-
-      *shell=False* does not suffer from this vulnerability; the above Note may be
-      helpful in getting code using *shell=False* to work.  See also
-      :func:`shlex.quote` for a function useful to quote filenames and commands.
+      Enabling this option can be a security hazard if combined with untrusted
+      input. See the warning under :ref:`frequently-used-arguments`
+      for details.
 
    On Windows: the :class:`Popen` class uses CreateProcess() to execute the
    child program, which operates on strings.  If *args* is a sequence, it will
@@ -122,16 +362,16 @@
    You don't need ``shell=True`` to run a batch file, nor to run a console-based
    executable.
 
-   *stdin*, *stdout* and *stderr* specify the executed programs' standard input,
+   *stdin*, *stdout* and *stderr* specify the executed program's standard input,
    standard output and standard error file handles, respectively.  Valid values
    are :data:`PIPE`, :data:`DEVNULL`, an existing file descriptor (a positive
    integer), an existing :term:`file object`, and ``None``.  :data:`PIPE`
    indicates that a new pipe to the child should be created.  :data:`DEVNULL`
-   indicates that the special file :data:`os.devnull` will be used. With ``None``,
-   no redirection will occur; the child's file handles will be inherited from
-   the parent.  Additionally, *stderr* can be :data:`STDOUT`, which indicates
-   that the stderr data from the applications should be captured into the same
-   file handle as for stdout.
+   indicates that the special file :data:`os.devnull` will be used. With the
+   default settings of ``None``, no redirection will occur; the child's file
+   handles will be inherited from the parent.  Additionally, *stderr* can be
+   :data:`STDOUT`, which indicates that the stderr data from the applications
+   should be captured into the same file handle as for stdout.
 
    If *preexec_fn* is set to a callable object, this object will be called in the
    child process just before the child is executed.
@@ -231,151 +471,6 @@
       Added context manager support.
 
 
-.. data:: DEVNULL
-
-   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
-   to :class:`Popen` and indicates that the special file :data:`os.devnull`
-   will be used.
-
-   .. versionadded:: 3.3
-
-
-.. data:: PIPE
-
-   Special value that can be used as the *stdin*, *stdout* or *stderr* argument
-   to :class:`Popen` and indicates that a pipe to the standard stream should be
-   opened.
-
-
-.. data:: STDOUT
-
-   Special value that can be used as the *stderr* argument to :class:`Popen` and
-   indicates that standard error should go into the same handle as standard
-   output.
-
-
-Convenience Functions
-^^^^^^^^^^^^^^^^^^^^^
-
-This module also defines the following shortcut functions:
-
-
-.. function:: call(*popenargs, timeout=None, **kwargs)
-
-   Run command with arguments.  Wait for command to complete, then return the
-   :attr:`returncode` attribute.
-
-   The arguments are the same as for the :class:`Popen` constructor, with the
-   exception of the *timeout* argument, which is given to :meth:`Popen.wait`.
-   Example::
-
-      >>> retcode = subprocess.call(["ls", "-l"])
-
-   If the timeout expires, the child process will be killed and then waited for
-   again.  The :exc:`TimeoutExpired` exception will be re-raised after the child
-   process has terminated.
-
-   .. warning::
-
-      Like :meth:`Popen.wait`, this will deadlock when using
-      ``stdout=PIPE`` and/or ``stderr=PIPE`` and the child process
-      generates enough output to a pipe such that it blocks waiting
-      for the OS pipe buffer to accept more data.
-
-   .. versionchanged:: 3.3
-      *timeout* was added.
-
-
-.. function:: check_call(*popenargs, timeout=None, **kwargs)
-
-   Run command with arguments.  Wait for command to complete. If the exit code was
-   zero then return, otherwise raise :exc:`CalledProcessError`. The
-   :exc:`CalledProcessError` object will have the return code in the
-   :attr:`returncode` attribute.
-
-   The arguments are the same as for the :func:`call` function.  Example::
-
-      >>> subprocess.check_call(["ls", "-l"])
-      0
-
-   As in the :func:`call` function, if the timeout expires, the child process
-   will be killed and the wait retried.  The :exc:`TimeoutExpired` exception
-   will be re-raised after the child process has terminated.
-
-   .. warning::
-
-      See the warning for :func:`call`.
-
-   .. versionchanged:: 3.3
-      *timeout* was added.
-
-
-.. function:: check_output(*popenargs, timeout=None, **kwargs)
-
-   Run command with arguments and return its output as a bytes object.
-
-   If the exit code was non-zero it raises a :exc:`CalledProcessError`.  The
-   :exc:`CalledProcessError` object will have the return code in the
-   :attr:`returncode` attribute and output in the :attr:`output` attribute.
-
-   The arguments are the same as for the :func:`call` function.  Example::
-
-      >>> subprocess.check_output(["ls", "-l", "/dev/null"])
-      b'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'
-
-   The stdout argument is not allowed as it is used internally.
-   To capture standard error in the result, use ``stderr=subprocess.STDOUT``::
-
-      >>> subprocess.check_output(
-      ...     ["/bin/sh", "-c", "ls non_existent_file; exit 0"],
-      ...     stderr=subprocess.STDOUT)
-      b'ls: non_existent_file: No such file or directory\n'
-
-   As in the :func:`call` function, if the timeout expires, the child process
-   will be killed and the wait retried.  The :exc:`TimeoutExpired` exception
-   will be re-raised after the child process has terminated.  The output from
-   the child process so far will be in the :attr:`output` attribute of the
-   exception.
-
-   .. versionadded:: 3.1
-
-   .. versionchanged:: 3.3
-      *timeout* was added.
-
-
-.. function:: getstatusoutput(cmd)
-
-   Return ``(status, output)`` of executing *cmd* in a shell.
-
-   Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple
-   ``(status, output)``.  *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the
-   returned output will contain output or error messages.  A trailing newline is
-   stripped from the output.  The exit status for the command can be interpreted
-   according to the rules for the C function :c:func:`wait`.  Example::
-
-      >>> subprocess.getstatusoutput('ls /bin/ls')
-      (0, '/bin/ls')
-      >>> subprocess.getstatusoutput('cat /bin/junk')
-      (256, 'cat: /bin/junk: No such file or directory')
-      >>> subprocess.getstatusoutput('/bin/junk')
-      (256, 'sh: /bin/junk: not found')
-
-   Availability: UNIX.
-
-
-.. function:: getoutput(cmd)
-
-   Return output (stdout and stderr) of executing *cmd* in a shell.
-
-   Like :func:`getstatusoutput`, except the exit status is ignored and the return
-   value is a string containing the command's output.  Example::
-
-      >>> subprocess.getoutput('ls /bin/ls')
-      '/bin/ls'
-
-   Availability: UNIX.
-
-
 Exceptions
 ^^^^^^^^^^
 
@@ -391,8 +486,9 @@
 A :exc:`ValueError` will be raised if :class:`Popen` is called with invalid
 arguments.
 
-check_call() will raise :exc:`CalledProcessError`, if the called process returns
-a non-zero return code.
+:func:`check_call` and :func:`check_output` will raise
+:exc:`CalledProcessError` if the called process returns a non-zero return
+code.
 
 All of the functions and methods that accept a *timeout* parameter, such as
 :func:`call` and :meth:`Popen.communicate` will raise :exc:`TimeoutExpired` if
@@ -407,9 +503,11 @@
 Security
 ^^^^^^^^
 
-Unlike some other popen functions, this implementation will never call /bin/sh
-implicitly.  This means that all characters, including shell metacharacters, can
-safely be passed to child processes.
+Unlike some other popen functions, this implementation will never call a
+system shell implicitly.  This means that all characters, including shell
+metacharacters, can safely be passed to child processes. Obviously, if the
+shell is invoked explicitly, then it is the application's responsibility to
+ensure that all whitespace and metacharacters are quoted appropriately.
 
 
 Popen Objects
@@ -663,15 +761,21 @@
 Replacing Older Functions with the subprocess Module
 ----------------------------------------------------
 
-In this section, "a ==> b" means that b can be used as a replacement for a.
+In this section, "a becomes b" means that b can be used as a replacement for a.
 
 .. note::
 
-   All functions in this section fail (more or less) silently if the executed
-   program cannot be found; this module raises an :exc:`OSError` exception.
+   All "a" functions in this section fail (more or less) silently if the
+   executed program cannot be found; the "b" replacements raise :exc:`OSError`
+   instead.
 
-In the following examples, we assume that the subprocess module is imported with
-"from subprocess import \*".
+   In addition, the replacements using :func:`check_output` will fail with a
+   :exc:`CalledProcessError` if the requested operation produces a non-zero
+   return code. The output is still available as the ``output`` attribute of
+   the raised exception.
+
+In the following examples, we assume that the relevant functions have already
+been imported from the subprocess module.
 
 
 Replacing /bin/sh shell backquote
@@ -680,8 +784,8 @@
 ::
 
    output=`mycmd myarg`
-   ==>
-   output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
+   # becomes
+   output = check_output(["mycmd", "myarg"])
 
 
 Replacing shell pipeline
@@ -690,7 +794,7 @@
 ::
 
    output=`dmesg | grep hda`
-   ==>
+   # becomes
    p1 = Popen(["dmesg"], stdout=PIPE)
    p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
    p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
@@ -699,22 +803,27 @@
 The p1.stdout.close() call after starting the p2 is important in order for p1
 to receive a SIGPIPE if p2 exits before p1.
 
+Alternatively, for trusted input, the shell's own pipeline support may still
+be used directly:
+
+   output=`dmesg | grep hda`
+   # becomes
+   output=check_output("dmesg | grep hda", shell=True)
+
+
 Replacing :func:`os.system`
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
    sts = os.system("mycmd" + " myarg")
-   ==>
-   p = Popen("mycmd" + " myarg", shell=True)
-   sts = os.waitpid(p.pid, 0)[1]
+   # becomes
+   sts = call("mycmd" + " myarg", shell=True)
 
 Notes:
 
 * Calling the program through the shell is usually not required.
 
-* It's easier to look at the :attr:`returncode` attribute than the exit status.
-
 A more realistic example would look like this::
 
    try:
@@ -839,6 +948,48 @@
   ``close_fds=True`` with :class:`Popen` to guarantee this behavior on
   all platforms or past Python versions.
 
+
+Legacy Shell Invocation Functions
+---------------------------------
+
+This module also provides the following legacy functions from the 2.x
+``commands`` module. These operations implicitly invoke the system shell and
+none of the guarantees described above regarding security and exception
+handling consistency are valid for these functions.
+
+.. function:: getstatusoutput(cmd)
+
+   Return ``(status, output)`` of executing *cmd* in a shell.
+
+   Execute the string *cmd* in a shell with :func:`os.popen` and return a 2-tuple
+   ``(status, output)``.  *cmd* is actually run as ``{ cmd ; } 2>&1``, so that the
+   returned output will contain output or error messages.  A trailing newline is
+   stripped from the output.  The exit status for the command can be interpreted
+   according to the rules for the C function :c:func:`wait`.  Example::
+
+      >>> subprocess.getstatusoutput('ls /bin/ls')
+      (0, '/bin/ls')
+      >>> subprocess.getstatusoutput('cat /bin/junk')
+      (256, 'cat: /bin/junk: No such file or directory')
+      >>> subprocess.getstatusoutput('/bin/junk')
+      (256, 'sh: /bin/junk: not found')
+
+   Availability: UNIX.
+
+
+.. function:: getoutput(cmd)
+
+   Return output (stdout and stderr) of executing *cmd* in a shell.
+
+   Like :func:`getstatusoutput`, except the exit status is ignored and the return
+   value is a string containing the command's output.  Example::
+
+      >>> subprocess.getoutput('ls /bin/ls')
+      '/bin/ls'
+
+   Availability: UNIX.
+
+
 Notes
 -----
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1780,6 +1780,9 @@
 Documentation
 -------------
 
+- Issue #13237: Reorganise subprocess documentation to emphasise convenience
+  functions and the most commonly needed arguments to Popen.
+
 - Issue #13141: Demonstrate recommended style for socketserver examples.
 
 - Issue #11818: Fix tempfile examples for Python 3.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 16:39:34 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 16:39:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Solved_a_potent?=
 =?utf8?q?ial_deadlock_in_test=5Ftelnetlib=2Epy=2E_Related_to_issue_=23118?=
 =?utf8?q?12?=
Message-ID: <E1RNnlu-0007we-3H@dinsdale.python.org>

http://hg.python.org/cpython/rev/76b6b85e4b78
changeset:   73449:76b6b85e4b78
branch:      2.7
parent:      73418:60dd1568bbd1
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 15:54:42 2011 +0100
summary:
  Solved a potential deadlock in test_telnetlib.py. Related to issue #11812

files:
  Lib/test/test_telnetlib.py |  8 --------
  Misc/NEWS                  |  2 ++
  2 files changed, 2 insertions(+), 8 deletions(-)


diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py
--- a/Lib/test/test_telnetlib.py
+++ b/Lib/test/test_telnetlib.py
@@ -15,7 +15,6 @@
         1) set evt to true to let the parent know we are ready
         2) [optional] if is not False, write the list of data from dataq.get()
            to the socket.
-        3) set evt to true to let the parent know we're done
     """
     serv.listen(5)
     evt.set()
@@ -40,7 +39,6 @@
         conn.close()
     finally:
         serv.close()
-        evt.set()
 
 class GeneralTests(TestCase):
 
@@ -52,11 +50,8 @@
         self.thread = threading.Thread(target=server, args=(self.evt,self.sock))
         self.thread.start()
         self.evt.wait()
-        self.evt.clear()
-        time.sleep(.1)
 
     def tearDown(self):
-        self.evt.wait()
         self.thread.join()
 
     def testBasic(self):
@@ -105,11 +100,8 @@
     self.thread = threading.Thread(target=server, args=(self.evt,self.sock, self.dataq))
     self.thread.start()
     self.evt.wait()
-    self.evt.clear()
-    time.sleep(.1)
 
 def _read_tearDown(self):
-    self.evt.wait()
     self.thread.join()
 
 class ReadTests(TestCase):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -359,6 +359,8 @@
 - Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary
   failure in name resolution.
 
+- Solved a potential deadlock in test_telnetlib.py. Related to issue #11812.
+
 - Avoid failing in test_robotparser when mueblesmoraleda.com is flaky and
   an overzealous DNS service (e.g. OpenDNS) redirects to a placeholder
   Web site.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 16:39:35 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 16:39:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Partial_patch_f?=
 =?utf8?q?or_issue_=2311812=3A_Take_care_of_test=5Ftelnetlib=2Epy?=
Message-ID: <E1RNnlv-0007xc-1b@dinsdale.python.org>

http://hg.python.org/cpython/rev/554802e562fa
changeset:   73450:554802e562fa
branch:      2.7
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 16:06:44 2011 +0100
summary:
  Partial patch for issue #11812: Take care of test_telnetlib.py

files:
  Lib/test/test_telnetlib.py |  12 ++++++------
  Misc/NEWS                  |   3 +++
  2 files changed, 9 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py
--- a/Lib/test/test_telnetlib.py
+++ b/Lib/test/test_telnetlib.py
@@ -33,10 +33,9 @@
                     data += item
                 written = conn.send(data)
                 data = data[written:]
+        conn.close()
     except socket.timeout:
         pass
-    else:
-        conn.close()
     finally:
         serv.close()
 
@@ -45,9 +44,10 @@
     def setUp(self):
         self.evt = threading.Event()
         self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        self.sock.settimeout(3)
+        self.sock.settimeout(60)  # Safety net. Look issue 11812
         self.port = test_support.bind_port(self.sock)
         self.thread = threading.Thread(target=server, args=(self.evt,self.sock))
+        self.thread.setDaemon(True)
         self.thread.start()
         self.evt.wait()
 
@@ -63,7 +63,7 @@
         self.assertTrue(socket.getdefaulttimeout() is None)
         socket.setdefaulttimeout(30)
         try:
-            telnet = telnetlib.Telnet("localhost", self.port)
+            telnet = telnetlib.Telnet(HOST, self.port)
         finally:
             socket.setdefaulttimeout(None)
         self.assertEqual(telnet.sock.gettimeout(), 30)
@@ -81,13 +81,13 @@
         telnet.sock.close()
 
     def testTimeoutValue(self):
-        telnet = telnetlib.Telnet("localhost", self.port, timeout=30)
+        telnet = telnetlib.Telnet(HOST, self.port, timeout=30)
         self.assertEqual(telnet.sock.gettimeout(), 30)
         telnet.sock.close()
 
     def testTimeoutOpen(self):
         telnet = telnetlib.Telnet()
-        telnet.open("localhost", self.port, timeout=30)
+        telnet.open(HOST, self.port, timeout=30)
         self.assertEqual(telnet.sock.gettimeout(), 30)
         telnet.sock.close()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -359,6 +359,9 @@
 - Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary
   failure in name resolution.
 
+- Issue #11812: Solve transient socket failure to connect to 'localhost'
+  in test_telnetlib.py.
+
 - Solved a potential deadlock in test_telnetlib.py. Related to issue #11812.
 
 - Avoid failing in test_robotparser when mueblesmoraleda.com is flaky and

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 16:39:35 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 16:39:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Solved_a_potent?=
 =?utf8?q?ial_deadlock_in_test=5Ftelnetlib=2Epy=2E_Related_to_issue_=23118?=
 =?utf8?q?12?=
Message-ID: <E1RNnlv-0007xg-UH@dinsdale.python.org>

http://hg.python.org/cpython/rev/f94533c9229d
changeset:   73451:f94533c9229d
branch:      3.2
parent:      73447:7545f4fb450c
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 16:20:46 2011 +0100
summary:
  Solved a potential deadlock in test_telnetlib.py. Related to issue #11812

files:
  Lib/test/test_telnetlib.py |  4 ----
  Misc/NEWS                  |  2 ++
  2 files changed, 2 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py
--- a/Lib/test/test_telnetlib.py
+++ b/Lib/test/test_telnetlib.py
@@ -21,7 +21,6 @@
         conn.close()
     finally:
         serv.close()
-        evt.set()
 
 class GeneralTests(TestCase):
 
@@ -33,11 +32,8 @@
         self.thread = threading.Thread(target=server, args=(self.evt,self.sock))
         self.thread.start()
         self.evt.wait()
-        self.evt.clear()
-        time.sleep(.1)
 
     def tearDown(self):
-        self.evt.wait()
         self.thread.join()
 
     def testBasic(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -527,6 +527,8 @@
 - Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary
   failure in name resolution.
 
+- Solved a potential deadlock in test_telnetlib.py. Related to issue #11812.
+
 - Avoid failing in test_urllibnet.test_bad_address when some overzealous
   DNS service (e.g. OpenDNS) resolves a non-existent domain name.  The test
   is now skipped instead.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 16:39:36 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 16:39:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Partial_patch_f?=
 =?utf8?q?or_issue_=2311812=3A_Take_care_of_test=5Ftelnetlib=2Epy?=
Message-ID: <E1RNnlw-0007xk-NE@dinsdale.python.org>

http://hg.python.org/cpython/rev/3b9f58f85d3e
changeset:   73452:3b9f58f85d3e
branch:      3.2
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 16:24:43 2011 +0100
summary:
  Partial patch for issue #11812: Take care of test_telnetlib.py

files:
  Lib/test/test_telnetlib.py |  12 ++++++------
  Misc/NEWS                  |   3 +++
  2 files changed, 9 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py
--- a/Lib/test/test_telnetlib.py
+++ b/Lib/test/test_telnetlib.py
@@ -15,10 +15,9 @@
     evt.set()
     try:
         conn, addr = serv.accept()
+        conn.close()
     except socket.timeout:
         pass
-    else:
-        conn.close()
     finally:
         serv.close()
 
@@ -27,9 +26,10 @@
     def setUp(self):
         self.evt = threading.Event()
         self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        self.sock.settimeout(3)
+        self.sock.settimeout(60)  # Safety net. Look issue 11812
         self.port = support.bind_port(self.sock)
         self.thread = threading.Thread(target=server, args=(self.evt,self.sock))
+        self.thread.setDaemon(True)
         self.thread.start()
         self.evt.wait()
 
@@ -45,7 +45,7 @@
         self.assertTrue(socket.getdefaulttimeout() is None)
         socket.setdefaulttimeout(30)
         try:
-            telnet = telnetlib.Telnet("localhost", self.port)
+            telnet = telnetlib.Telnet(HOST, self.port)
         finally:
             socket.setdefaulttimeout(None)
         self.assertEqual(telnet.sock.gettimeout(), 30)
@@ -63,13 +63,13 @@
         telnet.sock.close()
 
     def testTimeoutValue(self):
-        telnet = telnetlib.Telnet("localhost", self.port, timeout=30)
+        telnet = telnetlib.Telnet(HOST, self.port, timeout=30)
         self.assertEqual(telnet.sock.gettimeout(), 30)
         telnet.sock.close()
 
     def testTimeoutOpen(self):
         telnet = telnetlib.Telnet()
-        telnet.open("localhost", self.port, timeout=30)
+        telnet.open(HOST, self.port, timeout=30)
         self.assertEqual(telnet.sock.gettimeout(), 30)
         telnet.sock.close()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -527,6 +527,9 @@
 - Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary
   failure in name resolution.
 
+- Issue #11812: Solve transient socket failure to connect to 'localhost'
+  in test_telnetlib.py.
+
 - Solved a potential deadlock in test_telnetlib.py. Related to issue #11812.
 
 - Avoid failing in test_urllibnet.test_bad_address when some overzealous

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 16:39:37 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 16:39:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_MERGE=3A_Solved_a_potential_deadlock_in_test=5Ftelnetlib=2Ep?=
 =?utf8?q?y=2E_Related_to_issue_=2311812?=
Message-ID: <E1RNnlx-0007xo-GY@dinsdale.python.org>

http://hg.python.org/cpython/rev/85c10a905424
changeset:   73453:85c10a905424
parent:      73448:0b50008bb953
parent:      73451:f94533c9229d
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 16:34:22 2011 +0100
summary:
  MERGE: Solved a potential deadlock in test_telnetlib.py. Related to issue #11812

files:
  Lib/test/test_telnetlib.py |  4 ----
  Misc/NEWS                  |  2 ++
  2 files changed, 2 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py
--- a/Lib/test/test_telnetlib.py
+++ b/Lib/test/test_telnetlib.py
@@ -21,7 +21,6 @@
         conn.close()
     finally:
         serv.close()
-        evt.set()
 
 class GeneralTests(TestCase):
 
@@ -33,11 +32,8 @@
         self.thread = threading.Thread(target=server, args=(self.evt,self.sock))
         self.thread.start()
         self.evt.wait()
-        self.evt.clear()
-        time.sleep(.1)
 
     def tearDown(self):
-        self.evt.wait()
         self.thread.join()
         del self.thread  # Clear out any dangling Thread objects.
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1616,6 +1616,8 @@
 - Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary
   failure in name resolution.
 
+- Solved a potential deadlock in test_telnetlib.py. Related to issue #11812.
+
 - Avoid failing in test_robotparser when mueblesmoraleda.com is flaky and
   an overzealous DNS service (e.g. OpenDNS) redirects to a placeholder
   Web site.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 16:39:38 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 16:39:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_MERGE=3A_Partial_patch_for_issue_=2311812=3A_Take_care_of_te?=
 =?utf8?q?st=5Ftelnetlib=2Epy?=
Message-ID: <E1RNnly-0007xt-9h@dinsdale.python.org>

http://hg.python.org/cpython/rev/ca8a0dfb2176
changeset:   73454:ca8a0dfb2176
parent:      73453:85c10a905424
parent:      73452:3b9f58f85d3e
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 16:39:26 2011 +0100
summary:
  MERGE: Partial patch for issue #11812: Take care of test_telnetlib.py

files:
  Lib/test/test_telnetlib.py |  12 ++++++------
  Misc/NEWS                  |   3 +++
  2 files changed, 9 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py
--- a/Lib/test/test_telnetlib.py
+++ b/Lib/test/test_telnetlib.py
@@ -15,10 +15,9 @@
     evt.set()
     try:
         conn, addr = serv.accept()
+        conn.close()
     except socket.timeout:
         pass
-    else:
-        conn.close()
     finally:
         serv.close()
 
@@ -27,9 +26,10 @@
     def setUp(self):
         self.evt = threading.Event()
         self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        self.sock.settimeout(3)
+        self.sock.settimeout(60)  # Safety net. Look issue 11812
         self.port = support.bind_port(self.sock)
         self.thread = threading.Thread(target=server, args=(self.evt,self.sock))
+        self.thread.setDaemon(True)
         self.thread.start()
         self.evt.wait()
 
@@ -46,7 +46,7 @@
         self.assertTrue(socket.getdefaulttimeout() is None)
         socket.setdefaulttimeout(30)
         try:
-            telnet = telnetlib.Telnet("localhost", self.port)
+            telnet = telnetlib.Telnet(HOST, self.port)
         finally:
             socket.setdefaulttimeout(None)
         self.assertEqual(telnet.sock.gettimeout(), 30)
@@ -64,13 +64,13 @@
         telnet.sock.close()
 
     def testTimeoutValue(self):
-        telnet = telnetlib.Telnet("localhost", self.port, timeout=30)
+        telnet = telnetlib.Telnet(HOST, self.port, timeout=30)
         self.assertEqual(telnet.sock.gettimeout(), 30)
         telnet.sock.close()
 
     def testTimeoutOpen(self):
         telnet = telnetlib.Telnet()
-        telnet.open("localhost", self.port, timeout=30)
+        telnet.open(HOST, self.port, timeout=30)
         self.assertEqual(telnet.sock.gettimeout(), 30)
         telnet.sock.close()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1616,6 +1616,9 @@
 - Skip network tests when getaddrinfo() returns EAI_AGAIN, meaning a temporary
   failure in name resolution.
 
+- Issue #11812: Solve transient socket failure to connect to 'localhost'
+  in test_telnetlib.py.
+
 - Solved a potential deadlock in test_telnetlib.py. Related to issue #11812.
 
 - Avoid failing in test_robotparser when mueblesmoraleda.com is flaky and

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 17:28:11 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 08 Nov 2011 17:28:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Commit_59dca1e2363d_for_iss?=
 =?utf8?q?ue_=2313327_introduced_a_compilation_warning?=
Message-ID: <E1RNoWx-0005Sh-EL@dinsdale.python.org>

http://hg.python.org/cpython/rev/8907d646e0df
changeset:   73455:8907d646e0df
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 08 17:28:04 2011 +0100
summary:
  Commit 59dca1e2363d for issue #13327 introduced a compilation warning

files:
  Modules/posixmodule.c |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -10943,7 +10943,8 @@
     {"unlinkat",        posix_unlinkat, METH_VARARGS, posix_unlinkat__doc__},
 #endif
 #ifdef HAVE_UTIMENSAT
-    {"utimensat",       posix_utimensat, METH_VARARGS | METH_KEYWORDS,
+    {"utimensat",       (PyCFunction)posix_utimensat,
+                        METH_VARARGS | METH_KEYWORDS,
                         posix_utimensat__doc__},
 #endif
 #ifdef HAVE_MKFIFOAT

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 17:39:45 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Tue, 08 Nov 2011 17:39:45 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Change_decoders_to_use_Unic?=
 =?utf8?q?ode_API_instead_of_Py=5FUNICODE=2E?=
Message-ID: <E1RNoi9-0006fU-AN@dinsdale.python.org>

http://hg.python.org/cpython/rev/1a2a0821cf7b
changeset:   73456:1a2a0821cf7b
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Tue Nov 08 17:35:34 2011 +0100
summary:
  Change decoders to use Unicode API instead of Py_UNICODE.

files:
  Objects/unicodeobject.c |  742 +++++++++++----------------
  1 files changed, 307 insertions(+), 435 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1512,6 +1512,13 @@
     if (old_length == length)
         return 0;
 
+    if (length == 0) {
+        Py_DECREF(*p_unicode);
+        *p_unicode = unicode_empty;
+        Py_INCREF(*p_unicode);
+        return 0;
+    }
+
     if (!unicode_resizable(unicode)) {
         PyObject *copy = resize_copy(unicode, length);
         if (copy == NULL)
@@ -1540,8 +1547,7 @@
         return -1;
     }
     unicode = *p_unicode;
-    if (unicode == NULL || !PyUnicode_Check(unicode) || length < 0
-        || _PyUnicode_KIND(unicode) != PyUnicode_WCHAR_KIND)
+    if (unicode == NULL || !PyUnicode_Check(unicode) || length < 0)
     {
         PyErr_BadInternalCall();
         return -1;
@@ -1549,6 +1555,36 @@
     return unicode_resize(p_unicode, length);
 }
 
+static int
+unicode_widen(PyObject **p_unicode, int maxchar)
+{
+    PyObject *result;
+    assert(PyUnicode_IS_READY(*p_unicode));
+    if (maxchar <= PyUnicode_MAX_CHAR_VALUE(*p_unicode))
+        return 0;
+    result = PyUnicode_New(PyUnicode_GET_LENGTH(*p_unicode),
+                           maxchar);
+    if (result == NULL)
+        return -1;
+    PyUnicode_CopyCharacters(result, 0, *p_unicode, 0,
+                             PyUnicode_GET_LENGTH(*p_unicode));
+    Py_DECREF(*p_unicode);
+    *p_unicode = result;
+    return 0;
+}
+
+static int
+unicode_putchar(PyObject **p_unicode, Py_ssize_t *pos,
+                Py_UCS4 ch)
+{
+    if (unicode_widen(p_unicode, ch) < 0)
+        return -1;
+    PyUnicode_WRITE(PyUnicode_KIND(*p_unicode),
+                    PyUnicode_DATA(*p_unicode),
+                    (*pos)++, ch);
+    return 0;
+}
+
 static PyObject*
 get_latin1_char(unsigned char ch)
 {
@@ -3581,19 +3617,18 @@
                                  const char *encoding, const char *reason,
                                  const char **input, const char **inend, Py_ssize_t *startinpos,
                                  Py_ssize_t *endinpos, PyObject **exceptionObject, const char **inptr,
-                                 PyObject **output, Py_ssize_t *outpos, Py_UNICODE **outptr)
+                                 PyObject **output, Py_ssize_t *outpos)
 {
     static char *argparse = "O!n;decoding error handler must return (str, int) tuple";
 
     PyObject *restuple = NULL;
     PyObject *repunicode = NULL;
-    Py_ssize_t outsize = PyUnicode_GET_SIZE(*output);
+    Py_ssize_t outsize = PyUnicode_GET_LENGTH(*output);
     Py_ssize_t insize;
     Py_ssize_t requiredsize;
     Py_ssize_t newpos;
-    const Py_UNICODE *repptr;
     PyObject *inputobj = NULL;
-    Py_ssize_t repsize;
+    Py_ssize_t replen;
     int res = -1;
 
     if (*errorHandler == NULL) {
@@ -3619,6 +3654,8 @@
     }
     if (!PyArg_ParseTuple(restuple, argparse, &PyUnicode_Type, &repunicode, &newpos))
         goto onError;
+    if (PyUnicode_READY(repunicode) < 0)
+        goto onError;
 
     /* Copy back the bytes variables, which might have been modified by the
        callback */
@@ -3646,21 +3683,20 @@
        have+the replacement+the rest of the string (starting
        at the new input position), so we won't have to check space
        when there are no errors in the rest of the string) */
-    repptr = PyUnicode_AS_UNICODE(repunicode);
-    repsize = PyUnicode_GET_SIZE(repunicode);
-    requiredsize = *outpos + repsize + insize-newpos;
+    replen = PyUnicode_GET_LENGTH(repunicode);
+    requiredsize = *outpos + replen + insize-newpos;
     if (requiredsize > outsize) {
         if (requiredsize<2*outsize)
             requiredsize = 2*outsize;
-        if (PyUnicode_Resize(output, requiredsize) < 0)
+        if (unicode_resize(output, requiredsize) < 0)
             goto onError;
-        *outptr = PyUnicode_AS_UNICODE(*output) + *outpos;
-    }
+    }
+    if (unicode_widen(output, PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0)
+        goto onError;
     *endinpos = newpos;
     *inptr = *input + newpos;
-    Py_UNICODE_COPY(*outptr, repptr, repsize);
-    *outptr += repsize;
-    *outpos += repsize;
+    PyUnicode_CopyCharacters(*output, *outpos, repunicode, 0, replen);
+    *outpos += replen;
 
     /* we made it! */
     res = 0;
@@ -3778,17 +3814,17 @@
     Py_ssize_t outpos;
     const char *e;
     PyObject *unicode;
-    Py_UNICODE *p;
     const char *errmsg = "";
     int inShift = 0;
-    Py_UNICODE *shiftOutStart;
+    Py_ssize_t shiftOutStart;
     unsigned int base64bits = 0;
     unsigned long base64buffer = 0;
     Py_UNICODE surrogate = 0;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
 
-    unicode = (PyObject*)_PyUnicode_New(size);
+    /* Start off assuming it's all ASCII. Widen later as necessary. */
+    unicode = PyUnicode_New(size, 127);
     if (!unicode)
         return NULL;
     if (size == 0) {
@@ -3797,12 +3833,11 @@
         return unicode;
     }
 
-    p = PyUnicode_AS_UNICODE(unicode);
-    shiftOutStart = p;
+    shiftOutStart = outpos = 0;
     e = s + size;
 
     while (s < e) {
-        Py_UNICODE ch;
+        Py_UCS4 ch;
       restart:
         ch = (unsigned char) *s;
 
@@ -3820,13 +3855,10 @@
                     if (surrogate) {
                         /* expecting a second surrogate */
                         if (outCh >= 0xDC00 && outCh <= 0xDFFF) {
-#ifdef Py_UNICODE_WIDE
-                            *p++ = (((surrogate & 0x3FF)<<10)
-                                    | (outCh & 0x3FF)) + 0x10000;
-#else
-                            *p++ = surrogate;
-                            *p++ = outCh;
-#endif
+                            Py_UCS4 ch2 = (((surrogate & 0x3FF)<<10)
+                                           | (outCh & 0x3FF)) + 0x10000;
+                            if (unicode_putchar(&unicode, &outpos, ch2) < 0)
+                                goto onError;
                             surrogate = 0;
                         }
                         else {
@@ -3844,7 +3876,8 @@
                         goto utf7Error;
                     }
                     else {
-                        *p++ = outCh;
+                        if (unicode_putchar(&unicode, &outpos, outCh) < 0)
+                            goto onError;
                     }
                 }
             }
@@ -3872,7 +3905,8 @@
                 if (ch != '-') {
                     /* '-' is absorbed; other terminating
                        characters are preserved */
-                    *p++ = ch;
+                    if (unicode_putchar(&unicode, &outpos, ch) < 0)
+                        goto onError;
                 }
             }
         }
@@ -3881,16 +3915,18 @@
             s++; /* consume '+' */
             if (s < e && *s == '-') { /* '+-' encodes '+' */
                 s++;
-                *p++ = '+';
+                if (unicode_putchar(&unicode, &outpos, '+') < 0)
+                    goto onError;
             }
             else { /* begin base64-encoded section */
                 inShift = 1;
-                shiftOutStart = p;
+                shiftOutStart = outpos;
                 base64bits = 0;
             }
         }
         else if (DECODE_DIRECT(ch)) { /* character decodes as itself */
-            *p++ = ch;
+            if (unicode_putchar(&unicode, &outpos, ch) < 0)
+                goto onError;
             s++;
         }
         else {
@@ -3901,13 +3937,12 @@
         }
         continue;
 utf7Error:
-        outpos = p-PyUnicode_AS_UNICODE(unicode);
         endinpos = s-starts;
         if (unicode_decode_call_errorhandler(
                 errors, &errorHandler,
                 "utf7", errmsg,
                 &starts, &e, &startinpos, &endinpos, &exc, &s,
-                &unicode, &outpos, &p))
+                &unicode, &outpos))
             goto onError;
     }
 
@@ -3918,13 +3953,12 @@
         if (surrogate ||
                 (base64bits >= 6) ||
                 (base64bits > 0 && base64buffer != 0)) {
-            outpos = p-PyUnicode_AS_UNICODE(unicode);
             endinpos = size;
             if (unicode_decode_call_errorhandler(
                     errors, &errorHandler,
                     "utf7", "unterminated shift sequence",
                     &starts, &e, &startinpos, &endinpos, &exc, &s,
-                    &unicode, &outpos, &p))
+                    &unicode, &outpos))
                 goto onError;
             if (s < e)
                 goto restart;
@@ -3934,7 +3968,7 @@
     /* return state */
     if (consumed) {
         if (inShift) {
-            p = shiftOutStart; /* back off output */
+            outpos = shiftOutStart; /* back off output */
             *consumed = startinpos;
         }
         else {
@@ -3942,7 +3976,7 @@
         }
     }
 
-    if (PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
+    if (unicode_resize(&unicode, outpos) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
@@ -4208,7 +4242,7 @@
                         err = 1;
                     break;
                 }
-                for (cont = p + 1; cont < (p + n); ++cont) {
+                for (cont = p + 1; cont <= (p + n); ++cont) {
                     if ((*cont & 0xc0) != 0x80) {
                         err = 1;
                         break;
@@ -4229,19 +4263,23 @@
     return max_char;
 }
 
-/* Similar to PyUnicode_WRITE but can also write into wstr field
-   of the legacy unicode representation */
-#define WRITE_FLEXIBLE_OR_WSTR(kind, buf, index, value) \
-    do { \
-        const int k_ = (kind); \
-        if (k_ == PyUnicode_WCHAR_KIND) \
-            ((Py_UNICODE *)(buf))[(index)] = (Py_UNICODE)(value); \
-        else if (k_ == PyUnicode_1BYTE_KIND) \
-            ((unsigned char *)(buf))[(index)] = (unsigned char)(value); \
-        else if (k_ == PyUnicode_2BYTE_KIND) \
-            ((Py_UCS2 *)(buf))[(index)] = (Py_UCS2)(value); \
-        else \
-            ((Py_UCS4 *)(buf))[(index)] = (Py_UCS4)(value); \
+/* Similar to PyUnicode_WRITE but may attempt to widen and resize the string
+   in case of errors. Implicit parameters: unicode, kind, data, has_errors,
+   onError. Potential resizing overallocates, so the result needs to shrink
+   at the end.
+*/
+#define WRITE_MAYBE_FAIL(index, value)                                  \
+    do {                                                                \
+        if (has_errors) {                                               \
+            Py_ssize_t pos = index;                                     \
+            if (pos > PyUnicode_GET_LENGTH(unicode) &&                  \
+                unicode_resize(&unicode, pos + pos/8) < 0)              \
+                goto onError;                                           \
+            if (unicode_putchar(&unicode, &pos, value) < 0)             \
+                goto onError;                                           \
+        }                                                               \
+        else                                                            \
+            PyUnicode_WRITE(kind, data, index, value);                  \
     } while (0)
 
 PyObject *
@@ -4266,10 +4304,6 @@
     int kind;
     void *data;
     int has_errors;
-    Py_UNICODE *error_outptr;
-#if SIZEOF_WCHAR_T == 2
-    Py_ssize_t wchar_offset = 0;
-#endif
 
     if (size == 0) {
         if (consumed)
@@ -4278,28 +4312,23 @@
     }
     maxchar = utf8_max_char_size_and_has_errors(s, size, &unicode_size,
                                                 consumed, &has_errors);
-    if (has_errors) {
-        unicode = (PyObject*)_PyUnicode_New(size);
-        if (!unicode)
-            return NULL;
-        kind = PyUnicode_WCHAR_KIND;
-        data = PyUnicode_AS_UNICODE(unicode);
-        assert(data != NULL);
-    }
-    else {
+    if (has_errors)
+        /* maxchar and size computation might be incorrect; 
+           code below widens and resizes as necessary. */
+        unicode = PyUnicode_New(size, 127);
+    else
         unicode = PyUnicode_New(unicode_size, maxchar);
-        if (!unicode)
-            return NULL;
-        /* When the string is ASCII only, just use memcpy and return.
-           unicode_size may be != size if there is an incomplete UTF-8
-           sequence at the end of the ASCII block.  */
-        if (maxchar < 128 && size == unicode_size) {
-            Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
-            return unicode;
-        }
-        kind = PyUnicode_KIND(unicode);
-        data = PyUnicode_DATA(unicode);
-    }
+    if (!unicode)
+        return NULL;
+    /* When the string is ASCII only, just use memcpy and return.
+       unicode_size may be != size if there is an incomplete UTF-8
+       sequence at the end of the ASCII block.  */
+    if (!has_errors && maxchar < 128 && size == unicode_size) {
+        Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
+        return unicode;
+    }
+    kind = PyUnicode_KIND(unicode);
+    data = PyUnicode_DATA(unicode);
     /* Unpack UTF-8 encoded data */
     i = 0;
     e = s + size;
@@ -4327,15 +4356,15 @@
                     unsigned long value = *(unsigned long *) _s;
                     if (value & ASCII_CHAR_MASK)
                         break;
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+0, _s[0]);
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+1, _s[1]);
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+2, _s[2]);
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+3, _s[3]);
+                    WRITE_MAYBE_FAIL(_i+0, _s[0]);
+                    WRITE_MAYBE_FAIL(_i+1, _s[1]);
+                    WRITE_MAYBE_FAIL(_i+2, _s[2]);
+                    WRITE_MAYBE_FAIL(_i+3, _s[3]);
 #if (SIZEOF_LONG == 8)
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+4, _s[4]);
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+5, _s[5]);
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+6, _s[6]);
-                    WRITE_FLEXIBLE_OR_WSTR(kind, data, _i+7, _s[7]);
+                    WRITE_MAYBE_FAIL(_i+4, _s[4]);
+                    WRITE_MAYBE_FAIL(_i+5, _s[5]);
+                    WRITE_MAYBE_FAIL(_i+6, _s[6]);
+                    WRITE_MAYBE_FAIL(_i+7, _s[7]);
 #endif
                     _s += SIZEOF_LONG;
                     _i += SIZEOF_LONG;
@@ -4349,7 +4378,7 @@
         }
 
         if (ch < 0x80) {
-            WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
+            WRITE_MAYBE_FAIL(i++, ch);
             s++;
             continue;
         }
@@ -4392,7 +4421,7 @@
             }
             ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f);
             assert ((ch > 0x007F) && (ch <= 0x07FF));
-            WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
+            WRITE_MAYBE_FAIL(i++, ch);
             break;
 
         case 3:
@@ -4421,7 +4450,7 @@
             }
             ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f);
             assert ((ch > 0x07FF) && (ch <= 0xFFFF));
-            WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
+            WRITE_MAYBE_FAIL(i++, ch);
             break;
 
         case 4:
@@ -4446,86 +4475,56 @@
                  ((s[2] & 0x3f) << 6) + (s[3] & 0x3f);
             assert ((ch > 0xFFFF) && (ch <= 0x10ffff));
 
-            /* If the string is flexible or we have native UCS-4, write
-               directly.. */
-            if (sizeof(Py_UNICODE) > 2 || kind != PyUnicode_WCHAR_KIND)
-                WRITE_FLEXIBLE_OR_WSTR(kind, data, i++, ch);
-
-            else {
-                /* compute and append the two surrogates: */
-
-                /* translate from 10000..10FFFF to 0..FFFF */
-                ch -= 0x10000;
-
-                /* high surrogate = top 10 bits added to D800 */
-                WRITE_FLEXIBLE_OR_WSTR(kind, data, i++,
-                                       (Py_UNICODE)(0xD800 + (ch >> 10)));
-
-                /* low surrogate = bottom 10 bits added to DC00 */
-                WRITE_FLEXIBLE_OR_WSTR(kind, data, i++,
-                                       (Py_UNICODE)(0xDC00 + (ch & 0x03FF)));
-            }
-#if SIZEOF_WCHAR_T == 2
-            wchar_offset++;
-#endif
+            WRITE_MAYBE_FAIL(i++, ch);
             break;
         }
         s += n;
         continue;
 
       utf8Error:
-        /* If this is not yet a resizable string, make it one.. */
-        if (kind != PyUnicode_WCHAR_KIND) {
-            const Py_UNICODE *u;
-            PyObject *new_unicode = (PyObject*)_PyUnicode_New(size);
-            if (!new_unicode)
+        if (!has_errors) {
+            PyObject *tmp;
+            Py_ssize_t k;
+            /* We encountered some error that wasn't detected in the original scan,
+               e.g. an encoded surrogate character. The original maxchar computation may
+               have been incorrect, so redo it now. */
+            for (k = 0, maxchar = 0; k < i; k++)
+                maxchar = Py_MAX(maxchar, PyUnicode_READ(kind, data, k));
+            tmp = PyUnicode_New(PyUnicode_GET_LENGTH(unicode), maxchar);
+            if (tmp == NULL)
                 goto onError;
-            u = PyUnicode_AsUnicode(unicode);
-            if (!u)
-                goto onError;
-#if SIZEOF_WCHAR_T == 2
-            i += wchar_offset;
-#endif
-            Py_UNICODE_COPY(PyUnicode_AS_UNICODE(new_unicode), u, i);
+            PyUnicode_CopyCharacters(tmp, 0, unicode, 0, i);
             Py_DECREF(unicode);
-            unicode = new_unicode;
-            kind = 0;
-            data = PyUnicode_AS_UNICODE(new_unicode);
-            assert(data != NULL);
-        }
-        error_outptr = PyUnicode_AS_UNICODE(unicode) + i;
+            unicode = tmp;
+            has_errors = 1;
+        }
         if (unicode_decode_call_errorhandler(
                 errors, &errorHandler,
                 "utf8", errmsg,
                 &starts, &e, &startinpos, &endinpos, &exc, &s,
-                &unicode, &i, &error_outptr))
+                &unicode, &i))
             goto onError;
         /* Update data because unicode_decode_call_errorhandler might have
            re-created or resized the unicode object. */
-        data = PyUnicode_AS_UNICODE(unicode);
+        data = PyUnicode_DATA(unicode);
+        kind = PyUnicode_KIND(unicode);
         aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK);
     }
     /* Ensure the unicode_size calculation above was correct: */
-    assert(kind == PyUnicode_WCHAR_KIND || i == unicode_size);
+    assert(has_errors || i == unicode_size);
 
     if (consumed)
         *consumed = s-starts;
 
     /* Adjust length and ready string when it contained errors and
        is of the old resizable kind. */
-    if (kind == PyUnicode_WCHAR_KIND) {
+    if (has_errors) {
         if (PyUnicode_Resize(&unicode, i) < 0)
             goto onError;
     }
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&unicode)) {
-        Py_DECREF(unicode);
-        return NULL;
-    }
-#endif
     assert(_PyUnicode_CheckConsistency(unicode, 1));
     return unicode;
 
@@ -4536,7 +4535,7 @@
     return NULL;
 }
 
-#undef WRITE_FLEXIBLE_OR_WSTR
+#undef WRITE_MAYBE_FAIL
 
 #ifdef __APPLE__
 
@@ -4871,13 +4870,6 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     PyObject *unicode;
-    Py_UNICODE *p;
-#ifndef Py_UNICODE_WIDE
-    int pairs = 0;
-    const unsigned char *qq;
-#else
-    const int pairs = 0;
-#endif
     const unsigned char *q, *e;
     int bo = 0;       /* assume native ordering by default */
     const char *errmsg = "";
@@ -4941,23 +4933,13 @@
         iorder[3] = 0;
     }
 
-    /* On narrow builds we split characters outside the BMP into two
-       codepoints => count how much extra space we need. */
-#ifndef Py_UNICODE_WIDE
-    for (qq = q; qq < e; qq += 4)
-        if (qq[iorder[2]] != 0 || qq[iorder[3]] != 0)
-            pairs++;
-#endif
-
     /* This might be one to much, because of a BOM */
-    unicode = (PyObject*)_PyUnicode_New((size+3)/4+pairs);
+    unicode = PyUnicode_New((size+3)/4, 127);
     if (!unicode)
         return NULL;
     if (size == 0)
         return unicode;
-
-    /* Unpack UTF-32 encoded data */
-    p = PyUnicode_AS_UNICODE(unicode);
+    outpos = 0;
 
     while (q < e) {
         Py_UCS4 ch;
@@ -4982,24 +4964,16 @@
             endinpos = startinpos+4;
             goto utf32Error;
         }
-#ifndef Py_UNICODE_WIDE
-        if (ch >= 0x10000)
-        {
-            *p++ = 0xD800 | ((ch-0x10000) >> 10);
-            *p++ = 0xDC00 | ((ch-0x10000) & 0x3FF);
-        }
-        else
-#endif
-            *p++ = ch;
+        if (unicode_putchar(&unicode, &outpos, ch) < 0)
+            goto onError;
         q += 4;
         continue;
       utf32Error:
-        outpos = p-PyUnicode_AS_UNICODE(unicode);
         if (unicode_decode_call_errorhandler(
                 errors, &errorHandler,
                 "utf32", errmsg,
                 &starts, (const char **)&e, &startinpos, &endinpos, &exc, (const char **)&q,
-                &unicode, &outpos, &p))
+                &unicode, &outpos))
             goto onError;
     }
 
@@ -5010,7 +4984,7 @@
         *consumed = (const char *)q-starts;
 
     /* Adjust length */
-    if (PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
+    if (PyUnicode_Resize(&unicode, outpos) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
@@ -5171,7 +5145,6 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     PyObject *unicode;
-    Py_UNICODE *p;
     const unsigned char *q, *e, *aligned_end;
     int bo = 0;       /* assume native ordering by default */
     int native_ordering = 0;
@@ -5187,14 +5160,13 @@
 
     /* Note: size will always be longer than the resulting Unicode
        character count */
-    unicode = (PyObject*)_PyUnicode_New(size);
+    unicode = PyUnicode_New(size, 127);
     if (!unicode)
         return NULL;
     if (size == 0)
         return unicode;
-
-    /* Unpack UTF-16 encoded data */
-    p = PyUnicode_AS_UNICODE(unicode);
+    outpos = 0;
+
     q = (unsigned char *)s;
     e = q + size - 1;
 
@@ -5254,68 +5226,51 @@
         if (!((size_t) q & LONG_PTR_MASK)) {
             /* Fast path for runs of non-surrogate chars. */
             register const unsigned char *_q = q;
-            Py_UNICODE *_p = p;
-            if (native_ordering) {
-                /* Native ordering is simple: as long as the input cannot
-                   possibly contain a surrogate char, do an unrolled copy
-                   of several 16-bit code points to the target object.
-                   The non-surrogate check is done on several input bytes
-                   at a time (as many as a C 'long' can contain). */
-                while (_q < aligned_end) {
-                    unsigned long data = * (unsigned long *) _q;
-                    if (data & FAST_CHAR_MASK)
+            int kind = PyUnicode_KIND(unicode);
+            void *data = PyUnicode_DATA(unicode);
+            while (_q < aligned_end) {
+                unsigned long block = * (unsigned long *) _q;
+                unsigned short *pblock = (unsigned short*)&block;
+                Py_UCS4 maxch;
+                if (native_ordering) {
+                    /* Can use buffer directly */
+                    if (block & FAST_CHAR_MASK)
                         break;
-                    _p[0] = ((unsigned short *) _q)[0];
-                    _p[1] = ((unsigned short *) _q)[1];
+                }
+                else {
+                    /* Need to byte-swap */
+                    unsigned char *_p = (unsigned char*)pblock;
+                    if (block & SWAPPED_FAST_CHAR_MASK)
+                        break;
+                    _p[0] = _q[1];
+                    _p[1] = _q[0];
+                    _p[2] = _q[3];
+                    _p[3] = _q[2];
 #if (SIZEOF_LONG == 8)
-                    _p[2] = ((unsigned short *) _q)[2];
-                    _p[3] = ((unsigned short *) _q)[3];
-#endif
-                    _q += SIZEOF_LONG;
-                    _p += SIZEOF_LONG / 2;
-                }
-            }
-            else {
-                /* Byteswapped ordering is similar, but we must decompose
-                   the copy bytewise, and take care of zero'ing out the
-                   upper bytes if the target object is in 32-bit units
-                   (that is, in UCS-4 builds). */
-                while (_q < aligned_end) {
-                    unsigned long data = * (unsigned long *) _q;
-                    if (data & SWAPPED_FAST_CHAR_MASK)
-                        break;
-                    /* Zero upper bytes in UCS-4 builds */
-#if (Py_UNICODE_SIZE > 2)
-                    _p[0] = 0;
-                    _p[1] = 0;
-#if (SIZEOF_LONG == 8)
-                    _p[2] = 0;
-                    _p[3] = 0;
-#endif
-#endif
-                    /* Issue #4916; UCS-4 builds on big endian machines must
-                       fill the two last bytes of each 4-byte unit. */
-#if (!defined(BYTEORDER_IS_LITTLE_ENDIAN) && Py_UNICODE_SIZE > 2)
-# define OFF 2
-#else
-# define OFF 0
-#endif
-                    ((unsigned char *) _p)[OFF + 1] = _q[0];
-                    ((unsigned char *) _p)[OFF + 0] = _q[1];
-                    ((unsigned char *) _p)[OFF + 1 + Py_UNICODE_SIZE] = _q[2];
-                    ((unsigned char *) _p)[OFF + 0 + Py_UNICODE_SIZE] = _q[3];
-#if (SIZEOF_LONG == 8)
-                    ((unsigned char *) _p)[OFF + 1 + 2 * Py_UNICODE_SIZE] = _q[4];
-                    ((unsigned char *) _p)[OFF + 0 + 2 * Py_UNICODE_SIZE] = _q[5];
-                    ((unsigned char *) _p)[OFF + 1 + 3 * Py_UNICODE_SIZE] = _q[6];
-                    ((unsigned char *) _p)[OFF + 0 + 3 * Py_UNICODE_SIZE] = _q[7];
-#endif
-#undef OFF
-                    _q += SIZEOF_LONG;
-                    _p += SIZEOF_LONG / 2;
-                }
-            }
-            p = _p;
+                    _p[4] = _q[5];
+                    _p[5] = _q[4];
+                    _p[6] = _q[7];
+                    _p[7] = _q[6];
+#endif
+                }
+                maxch = Py_MAX(pblock[0], pblock[1]);
+#if SIZEOF_LONG == 8
+                maxch = Py_MAX(maxch, Py_MAX(pblock[2], pblock[3]));
+#endif
+                if (maxch > PyUnicode_MAX_CHAR_VALUE(unicode)) {
+                    if (unicode_widen(&unicode, maxch) < 0)
+                        goto onError;
+                    kind = PyUnicode_KIND(unicode);
+                    data = PyUnicode_DATA(unicode);
+                }
+                PyUnicode_WRITE(kind, data, outpos++, pblock[0]);
+                PyUnicode_WRITE(kind, data, outpos++, pblock[1]);
+#if SIZEOF_LONG == 8
+                PyUnicode_WRITE(kind, data, outpos++, pblock[2]);
+                PyUnicode_WRITE(kind, data, outpos++, pblock[3]);
+#endif
+                _q += SIZEOF_LONG;
+            }
             q = _q;
             if (q >= e)
                 break;
@@ -5325,7 +5280,8 @@
         q += 2;
 
         if (ch < 0xD800 || ch > 0xDFFF) {
-            *p++ = ch;
+            if (unicode_putchar(&unicode, &outpos, ch) < 0)
+                goto onError;
             continue;
         }
 
@@ -5340,12 +5296,10 @@
             Py_UNICODE ch2 = (q[ihi] << 8) | q[ilo];
             q += 2;
             if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
-#ifndef Py_UNICODE_WIDE
-                *p++ = ch;
-                *p++ = ch2;
-#else
-                *p++ = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000;
-#endif
+                if (unicode_putchar(&unicode, &outpos, 
+                                    (((ch & 0x3FF)<<10) | 
+                                     (ch2 & 0x3FF)) + 0x10000) < 0)
+                    goto onError;
                 continue;
             }
             else {
@@ -5362,7 +5316,6 @@
         /* Fall through to report the error */
 
       utf16Error:
-        outpos = p - PyUnicode_AS_UNICODE(unicode);
         if (unicode_decode_call_errorhandler(
                 errors,
                 &errorHandler,
@@ -5374,8 +5327,7 @@
                 &exc,
                 (const char **)&q,
                 &unicode,
-                &outpos,
-                &p))
+                &outpos))
             goto onError;
     }
     /* remaining byte at the end? (size should be even) */
@@ -5384,7 +5336,6 @@
             errmsg = "truncated data";
             startinpos = ((const char *)q) - starts;
             endinpos = ((const char *)e) + 1 - starts;
-            outpos = p - PyUnicode_AS_UNICODE(unicode);
             if (unicode_decode_call_errorhandler(
                     errors,
                     &errorHandler,
@@ -5396,8 +5347,7 @@
                     &exc,
                     (const char **)&q,
                     &unicode,
-                    &outpos,
-                    &p))
+                    &outpos))
                 goto onError;
             /* The remaining input chars are ignored if the callback
                chooses to skip the input */
@@ -5411,17 +5361,11 @@
         *consumed = (const char *)q-starts;
 
     /* Adjust length */
-    if (PyUnicode_Resize(&unicode, p - PyUnicode_AS_UNICODE(unicode)) < 0)
+    if (PyUnicode_Resize(&unicode, outpos) < 0)
         goto onError;
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&unicode)) {
-        Py_DECREF(unicode);
-        return NULL;
-    }
-#endif
     assert(_PyUnicode_CheckConsistency(unicode, 1));
     return unicode;
 
@@ -5613,31 +5557,26 @@
     Py_ssize_t endinpos;
     int j;
     PyObject *v;
-    Py_UNICODE *p;
     const char *end;
     char* message;
     Py_UCS4 chr = 0xffffffff; /* in case 'getcode' messes up */
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
-    Py_ssize_t ascii_length;
+    Py_ssize_t len;
     Py_ssize_t i;
-    int kind;
-    void *data;
-
-    ascii_length = length_of_escaped_ascii_string(s, size);
+
+    len = length_of_escaped_ascii_string(s, size);
 
     /* After length_of_escaped_ascii_string() there are two alternatives,
        either the string is pure ASCII with named escapes like \n, etc.
        and we determined it's exact size (common case)
        or it contains \x, \u, ... escape sequences.  then we create a
        legacy wchar string and resize it at the end of this function. */
-    if (ascii_length >= 0) {
-        v = PyUnicode_New(ascii_length, 127);
+    if (len >= 0) {
+        v = PyUnicode_New(len, 127);
         if (!v)
             goto onError;
         assert(PyUnicode_KIND(v) == PyUnicode_1BYTE_KIND);
-        kind = PyUnicode_1BYTE_KIND;
-        data = PyUnicode_DATA(v);
     }
     else {
         /* Escaped strings will always be longer than the resulting
@@ -5645,11 +5584,10 @@
            length after conversion to the true value.
            (but if the error callback returns a long replacement string
            we'll have to allocate more space) */
-        v = (PyObject*)_PyUnicode_New(size);
+        v = PyUnicode_New(size, 127);
         if (!v)
             goto onError;
-        kind = PyUnicode_WCHAR_KIND;
-        data = PyUnicode_AS_UNICODE(v);
+        len = size;
     }
 
     if (size == 0)
@@ -5662,18 +5600,14 @@
         Py_UNICODE x;
         int digits;
 
-        if (kind == PyUnicode_WCHAR_KIND) {
-            assert(i < _PyUnicode_WSTR_LENGTH(v));
-        }
-        else {
-            /* The only case in which i == ascii_length is a backslash
-               followed by a newline. */
-            assert(i <= ascii_length);
-        }
+        /* The only case in which i == ascii_length is a backslash
+           followed by a newline. */
+        assert(i <= len);
 
         /* Non-escape characters are interpreted as Unicode ordinals */
         if (*s != '\\') {
-            WRITE_ASCII_OR_WSTR(kind, data, i++, (unsigned char) *s++);
+            if (unicode_putchar(&v, &i, (unsigned char) *s++) < 0)
+                goto onError;
             continue;
         }
 
@@ -5684,32 +5618,33 @@
         if (s > end)
             c = '\0'; /* Invalid after \ */
 
-        if (kind == PyUnicode_WCHAR_KIND) {
-            assert(i < _PyUnicode_WSTR_LENGTH(v));
-        }
-        else {
-            /* The only case in which i == ascii_length is a backslash
-               followed by a newline. */
-            assert(i < ascii_length || (i == ascii_length && c == '\n'));
-        }
+        /* The only case in which i == ascii_length is a backslash
+           followed by a newline. */
+        assert(i < len || (i == len && c == '\n'));
 
         switch (c) {
 
             /* \x escapes */
+#define WRITECHAR(ch)                                   \
+            do {                                        \
+                if (unicode_putchar(&v, &i, ch) < 0)    \
+                    goto onError;                       \
+            }while(0)
+
         case '\n': break;
-        case '\\': WRITE_ASCII_OR_WSTR(kind, data, i++, '\\'); break;
-        case '\'': WRITE_ASCII_OR_WSTR(kind, data, i++, '\''); break;
-        case '\"': WRITE_ASCII_OR_WSTR(kind, data, i++, '\"'); break;
-        case 'b': WRITE_ASCII_OR_WSTR(kind, data, i++, '\b'); break;
+        case '\\': WRITECHAR('\\'); break;
+        case '\'': WRITECHAR('\''); break;
+        case '\"': WRITECHAR('\"'); break;
+        case 'b': WRITECHAR('\b'); break;
         /* FF */
-        case 'f': WRITE_ASCII_OR_WSTR(kind, data, i++, '\014'); break;
-        case 't': WRITE_ASCII_OR_WSTR(kind, data, i++, '\t'); break;
-        case 'n': WRITE_ASCII_OR_WSTR(kind, data, i++, '\n'); break;
-        case 'r': WRITE_ASCII_OR_WSTR(kind, data, i++, '\r'); break;
+        case 'f': WRITECHAR('\014'); break;
+        case 't': WRITECHAR('\t'); break;
+        case 'n': WRITECHAR('\n'); break;
+        case 'r': WRITECHAR('\r'); break;
         /* VT */
-        case 'v': WRITE_ASCII_OR_WSTR(kind, data, i++, '\013'); break;
+        case 'v': WRITECHAR('\013'); break;
         /* BEL, not classic C */
-        case 'a': WRITE_ASCII_OR_WSTR(kind, data, i++, '\007'); break;
+        case 'a': WRITECHAR('\007'); break;
 
             /* \OOO (octal) escapes */
         case '0': case '1': case '2': case '3':
@@ -5720,7 +5655,7 @@
                 if (s < end && '0' <= *s && *s <= '7')
                     x = (x<<3) + *s++ - '0';
             }
-            WRITE_WSTR(data, i++, x);
+            WRITECHAR(x);
             break;
 
             /* hex escapes */
@@ -5742,30 +5677,27 @@
             message = "truncated \\UXXXXXXXX escape";
         hexescape:
             chr = 0;
-            p = PyUnicode_AS_UNICODE(v) + i;
             if (s+digits>end) {
                 endinpos = size;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "unicodeescape", "end of string in escape sequence",
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &i, &p))
+                        &v, &i))
                     goto onError;
-                data = PyUnicode_AS_UNICODE(v);
                 goto nextByte;
             }
             for (j = 0; j < digits; ++j) {
                 c = (unsigned char) s[j];
                 if (!Py_ISXDIGIT(c)) {
                     endinpos = (s+j+1)-starts;
-                    p = PyUnicode_AS_UNICODE(v) + i;
                     if (unicode_decode_call_errorhandler(
                             errors, &errorHandler,
                             "unicodeescape", message,
                             &starts, &end, &startinpos, &endinpos, &exc, &s,
-                            &v, &i, &p))
+                            &v, &i))
                         goto onError;
-                    data = PyUnicode_AS_UNICODE(v);
+                    len = PyUnicode_GET_LENGTH(v);
                     goto nextByte;
                 }
                 chr = (chr<<4) & ~0xF;
@@ -5783,29 +5715,16 @@
                 break;
         store:
             /* when we get here, chr is a 32-bit unicode character */
-            if (chr <= 0xffff)
-                /* UCS-2 character */
-                WRITE_WSTR(data, i++, chr);
-            else if (chr <= 0x10ffff) {
-                /* UCS-4 character. Either store directly, or as
-                   surrogate pair. */
-#ifdef Py_UNICODE_WIDE
-                WRITE_WSTR(data, i++, chr);
-#else
-                chr -= 0x10000L;
-                WRITE_WSTR(data, i++, 0xD800 + (Py_UNICODE) (chr >> 10));
-                WRITE_WSTR(data, i++, 0xDC00 + (Py_UNICODE) (chr & 0x03FF));
-#endif
+            if (chr <= 0x10ffff) {
+                WRITECHAR(chr);
             } else {
                 endinpos = s-starts;
-                p = PyUnicode_AS_UNICODE(v) + i;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "unicodeescape", "illegal Unicode character",
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &i, &p))
+                        &v, &i))
                     goto onError;
-                data = PyUnicode_AS_UNICODE(v);
             }
             break;
 
@@ -5834,48 +5753,39 @@
                 }
             }
             endinpos = s-starts;
-            p = PyUnicode_AS_UNICODE(v) + i;
             if (unicode_decode_call_errorhandler(
                     errors, &errorHandler,
                     "unicodeescape", message,
                     &starts, &end, &startinpos, &endinpos, &exc, &s,
-                    &v, &i, &p))
+                    &v, &i))
                 goto onError;
-            data = PyUnicode_AS_UNICODE(v);
             break;
 
         default:
             if (s > end) {
-                assert(kind == PyUnicode_WCHAR_KIND);
                 message = "\\ at end of string";
                 s--;
                 endinpos = s-starts;
-                p = PyUnicode_AS_UNICODE(v) + i;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "unicodeescape", message,
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &i, &p))
+                        &v, &i))
                     goto onError;
-                data = PyUnicode_AS_UNICODE(v);
             }
             else {
-                WRITE_ASCII_OR_WSTR(kind, data, i++, '\\');
-                WRITE_ASCII_OR_WSTR(kind, data, i++, (unsigned char)s[-1]);
+                WRITECHAR('\\');
+                WRITECHAR(s[-1]);
             }
             break;
         }
       nextByte:
         ;
     }
-    /* Ensure the length prediction worked in case of ASCII strings */
-    assert(kind == PyUnicode_WCHAR_KIND || i == ascii_length);
-
-    if (kind == PyUnicode_WCHAR_KIND)
-    {
-        if (PyUnicode_Resize(&v, i) < 0)
-            goto onError;
-    }
+#undef WRITECHAR
+
+    if (PyUnicode_Resize(&v, i) < 0)
+        goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
 #ifndef DONT_MAKE_RESULT_READY
@@ -6081,7 +5991,6 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     PyObject *v;
-    Py_UNICODE *p;
     const char *end;
     const char *bs;
     PyObject *errorHandler = NULL;
@@ -6091,12 +6000,12 @@
        Unicode string, so we start with size here and then reduce the
        length after conversion to the true value. (But decoding error
        handler might have to resize the string) */
-    v = (PyObject*)_PyUnicode_New(size);
+    v = PyUnicode_New(size, 127);
     if (v == NULL)
         goto onError;
     if (size == 0)
         return v;
-    p = PyUnicode_AS_UNICODE(v);
+    outpos = 0;
     end = s + size;
     while (s < end) {
         unsigned char c;
@@ -6106,7 +6015,8 @@
 
         /* Non-escape characters are interpreted as Unicode ordinals */
         if (*s != '\\') {
-            *p++ = (unsigned char)*s++;
+            if (unicode_putchar(&v, &outpos, (unsigned char)*s++) < 0)
+                goto onError;
             continue;
         }
         startinpos = s-starts;
@@ -6117,19 +6027,19 @@
         for (;s < end;) {
             if (*s != '\\')
                 break;
-            *p++ = (unsigned char)*s++;
+            if (unicode_putchar(&v, &outpos, (unsigned char)*s++) < 0)
+                goto onError;
         }
         if (((s - bs) & 1) == 0 ||
             s >= end ||
             (*s != 'u' && *s != 'U')) {
             continue;
         }
-        p--;
+        outpos--;
         count = *s=='u' ? 4 : 8;
         s++;
 
         /* \uXXXX with 4 hex digits, \Uxxxxxxxx with 8 */
-        outpos = p-PyUnicode_AS_UNICODE(v);
         for (x = 0, i = 0; i < count; ++i, ++s) {
             c = (unsigned char)*s;
             if (!Py_ISXDIGIT(c)) {
@@ -6138,7 +6048,7 @@
                         errors, &errorHandler,
                         "rawunicodeescape", "truncated \\uXXXX",
                         &starts, &end, &startinpos, &endinpos, &exc, &s,
-                        &v, &outpos, &p))
+                        &v, &outpos))
                     goto onError;
                 goto nextByte;
             }
@@ -6150,42 +6060,25 @@
             else
                 x += 10 + c - 'A';
         }
-        if (x <= 0xffff)
-            /* UCS-2 character */
-            *p++ = (Py_UNICODE) x;
-        else if (x <= 0x10ffff) {
-            /* UCS-4 character. Either store directly, or as
-               surrogate pair. */
-#ifdef Py_UNICODE_WIDE
-            *p++ = (Py_UNICODE) x;
-#else
-            x -= 0x10000L;
-            *p++ = 0xD800 + (Py_UNICODE) (x >> 10);
-            *p++ = 0xDC00 + (Py_UNICODE) (x & 0x03FF);
-#endif
+        if (x <= 0x10ffff) {
+            if (unicode_putchar(&v, &outpos, x) < 0)
+                goto onError;
         } else {
             endinpos = s-starts;
-            outpos = p-PyUnicode_AS_UNICODE(v);
             if (unicode_decode_call_errorhandler(
                     errors, &errorHandler,
                     "rawunicodeescape", "\\Uxxxxxxxx out of range",
                     &starts, &end, &startinpos, &endinpos, &exc, &s,
-                    &v, &outpos, &p))
+                    &v, &outpos))
                 goto onError;
         }
       nextByte:
         ;
     }
-    if (PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
+    if (PyUnicode_Resize(&v, outpos) < 0)
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&v)) {
-        Py_DECREF(v);
-        return NULL;
-    }
-#endif
     assert(_PyUnicode_CheckConsistency(v, 1));
     return v;
 
@@ -6311,34 +6204,27 @@
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
     PyObject *v;
-    Py_UNICODE *p;
     const char *end;
     const char *reason;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
 
-#ifdef Py_UNICODE_WIDE
-    Py_UNICODE unimax = PyUnicode_GetMax();
-#endif
-
     /* XXX overflow detection missing */
-    v = (PyObject*)_PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE);
+    v = PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE, 127);
     if (v == NULL)
         goto onError;
-    /* Intentionally PyUnicode_GET_SIZE instead of PyUnicode_GET_LENGTH
-       as string was created with the old API. */
-    if (PyUnicode_GET_SIZE(v) == 0)
+    if (PyUnicode_GET_LENGTH(v) == 0)
         return v;
-    p = PyUnicode_AS_UNICODE(v);
+    outpos = 0;
     end = s + size;
 
     while (s < end) {
-        memcpy(p, s, sizeof(Py_UNICODE));
+        Py_UCS4 ch = *(Py_UNICODE*)s;
         /* We have to sanity check the raw data, otherwise doom looms for
            some malformed UCS-4 data. */
         if (
 #ifdef Py_UNICODE_WIDE
-            *p > unimax || *p < 0 ||
+            ch > 0x10ffff ||
 #endif
             end-s < Py_UNICODE_SIZE
             )
@@ -6352,31 +6238,25 @@
                 endinpos = s - starts + Py_UNICODE_SIZE;
                 reason = "illegal code point (> 0x10FFFF)";
             }
-            outpos = p - PyUnicode_AS_UNICODE(v);
             if (unicode_decode_call_errorhandler(
                     errors, &errorHandler,
                     "unicode_internal", reason,
                     &starts, &end, &startinpos, &endinpos, &exc, &s,
-                    &v, &outpos, &p)) {
+                    &v, &outpos)) {
                 goto onError;
             }
         }
         else {
-            p++;
+            if (unicode_putchar(&v, &outpos, ch) < 0)
+                goto onError;
             s += Py_UNICODE_SIZE;
         }
     }
 
-    if (PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
+    if (PyUnicode_Resize(&v, outpos) < 0)
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&v)) {
-        Py_DECREF(v);
-        return NULL;
-    }
-#endif
     assert(_PyUnicode_CheckConsistency(v, 1));
     return v;
 
@@ -6749,7 +6629,8 @@
 {
     const char *starts = s;
     PyObject *v;
-    Py_UNICODE *u;
+    int kind;
+    void *data;
     Py_ssize_t startinpos;
     Py_ssize_t endinpos;
     Py_ssize_t outpos;
@@ -6797,42 +6678,38 @@
     if (!has_error)
         return unicode_fromascii((const unsigned char *)s, size);
 
-    v = (PyObject*)_PyUnicode_New(size);
+    v = PyUnicode_New(size, 127);
     if (v == NULL)
         goto onError;
     if (size == 0)
         return v;
-    u = PyUnicode_AS_UNICODE(v);
+    kind = PyUnicode_KIND(v);
+    data = PyUnicode_DATA(v);
+    outpos = 0;
     e = s + size;
     while (s < e) {
         register unsigned char c = (unsigned char)*s;
         if (c < 128) {
-            *u++ = c;
+            PyUnicode_WRITE(kind, data, outpos++, c);
             ++s;
         }
         else {
             startinpos = s-starts;
             endinpos = startinpos + 1;
-            outpos = u - (Py_UNICODE *)PyUnicode_AS_UNICODE(v);
             if (unicode_decode_call_errorhandler(
                     errors, &errorHandler,
                     "ascii", "ordinal not in range(128)",
                     &starts, &e, &startinpos, &endinpos, &exc, &s,
-                    &v, &outpos, &u))
+                    &v, &outpos))
                 goto onError;
-        }
-    }
-    if (u - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v))
-        if (PyUnicode_Resize(&v, u - PyUnicode_AS_UNICODE(v)) < 0)
-            goto onError;
+            kind = PyUnicode_KIND(v);
+            data = PyUnicode_DATA(v);
+        }
+    }
+    if (PyUnicode_Resize(&v, outpos) < 0)
+        goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&v)) {
-        Py_DECREF(v);
-        return NULL;
-    }
-#endif
     assert(_PyUnicode_CheckConsistency(v, 1));
     return v;
 
@@ -7648,7 +7525,6 @@
     Py_ssize_t outpos;
     const char *e;
     PyObject *v;
-    Py_UNICODE *p;
     Py_ssize_t extrachars = 0;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
@@ -7659,12 +7535,12 @@
     if (mapping == NULL)
         return PyUnicode_DecodeLatin1(s, size, errors);
 
-    v = (PyObject*)_PyUnicode_New(size);
+    v = PyUnicode_New(size, 127);
     if (v == NULL)
         goto onError;
     if (size == 0)
         return v;
-    p = PyUnicode_AS_UNICODE(v);
+    outpos = 0;
     e = s + size;
     if (PyUnicode_CheckExact(mapping)) {
         mapstring = PyUnicode_AS_UNICODE(mapping);
@@ -7678,19 +7554,19 @@
 
             if (x == 0xfffe) {
                 /* undefined mapping */
-                outpos = p-PyUnicode_AS_UNICODE(v);
                 startinpos = s-starts;
                 endinpos = startinpos+1;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "charmap", "character maps to <undefined>",
                         &starts, &e, &startinpos, &endinpos, &exc, &s,
-                        &v, &outpos, &p)) {
+                        &v, &outpos)) {
                     goto onError;
                 }
                 continue;
             }
-            *p++ = x;
+            if (unicode_putchar(&v, &outpos, x) < 0)
+                goto onError;
             ++s;
         }
     }
@@ -7724,18 +7600,18 @@
                     Py_DECREF(x);
                     goto onError;
                 }
-                *p++ = (Py_UNICODE)value;
+                if (unicode_putchar(&v, &outpos, value) < 0)
+                    goto onError;
             }
             else if (x == Py_None) {
                 /* undefined mapping */
-                outpos = p-PyUnicode_AS_UNICODE(v);
                 startinpos = s-starts;
                 endinpos = startinpos+1;
                 if (unicode_decode_call_errorhandler(
                         errors, &errorHandler,
                         "charmap", "character maps to <undefined>",
                         &starts, &e, &startinpos, &endinpos, &exc, &s,
-                        &v, &outpos, &p)) {
+                        &v, &outpos)) {
                     Py_DECREF(x);
                     goto onError;
                 }
@@ -7743,32 +7619,36 @@
                 continue;
             }
             else if (PyUnicode_Check(x)) {
-                Py_ssize_t targetsize = PyUnicode_GET_SIZE(x);
-
-                if (targetsize == 1)
+                Py_ssize_t targetsize;
+
+                if (PyUnicode_READY(x) < 0)
+                    goto onError;
+                targetsize = PyUnicode_GET_LENGTH(x);
+
+                if (targetsize == 1) {
                     /* 1-1 mapping */
-                    *p++ = *PyUnicode_AS_UNICODE(x);
-
+                    if (unicode_putchar(&v, &outpos, 
+                                        PyUnicode_READ_CHAR(x, 0)) < 0)
+                        goto onError;
+                }
                 else if (targetsize > 1) {
                     /* 1-n mapping */
                     if (targetsize > extrachars) {
                         /* resize first */
-                        Py_ssize_t oldpos = p - PyUnicode_AS_UNICODE(v);
                         Py_ssize_t needed = (targetsize - extrachars) + \
                             (targetsize << 2);
                         extrachars += needed;
                         /* XXX overflow detection missing */
                         if (PyUnicode_Resize(&v,
-                                             PyUnicode_GET_SIZE(v) + needed) < 0) {
+                                             PyUnicode_GET_LENGTH(v) + needed) < 0) {
                             Py_DECREF(x);
                             goto onError;
                         }
-                        p = PyUnicode_AS_UNICODE(v) + oldpos;
                     }
-                    Py_UNICODE_COPY(p,
-                                    PyUnicode_AS_UNICODE(x),
-                                    targetsize);
-                    p += targetsize;
+                    if (unicode_widen(&v, PyUnicode_MAX_CHAR_VALUE(x)) < 0)
+                        goto onError;
+                    PyUnicode_CopyCharacters(v, outpos, x, 0, targetsize);
+                    outpos += targetsize;
                     extrachars -= targetsize;
                 }
                 /* 1-0 mapping: skip the character */
@@ -7784,17 +7664,9 @@
             ++s;
         }
     }
-    if (p - PyUnicode_AS_UNICODE(v) < PyUnicode_GET_SIZE(v))
-        if (PyUnicode_Resize(&v, p - PyUnicode_AS_UNICODE(v)) < 0)
-            goto onError;
+    if (PyUnicode_Resize(&v, outpos) < 0)
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&v)) {
-        Py_DECREF(v);
-        return NULL;
-    }
-#endif
     assert(_PyUnicode_CheckConsistency(v, 1));
     return v;
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 17:44:13 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Tue, 08 Nov 2011 17:44:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_=5FPyUnicode=5FFromId_?=
 =?utf8?q?return_borrowed_references=2E?=
Message-ID: <E1RNomT-0007F0-CN@dinsdale.python.org>

http://hg.python.org/cpython/rev/6ae8c68eb0c1
changeset:   73457:6ae8c68eb0c1
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Mon Nov 07 13:00:05 2011 +0100
summary:
  Make _PyUnicode_FromId return borrowed references.
http://mail.python.org/pipermail/python-dev/2011-November/114347.html

files:
  Include/unicodeobject.h |  2 +-
  Objects/object.c        |  9 +++------
  Objects/typeobject.c    |  3 +--
  Objects/unicodeobject.c |  1 -
  Python/_warnings.c      |  2 --
  5 files changed, 5 insertions(+), 12 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -2024,7 +2024,7 @@
    shutdown, all strings are released (through _PyUnicode_ClearStaticStrings).
 
    Alternatively, _Py_static_string allows to choose the variable name.
-   _PyUnicode_FromId returns a new reference to the interned string.
+   _PyUnicode_FromId returns a borrowed reference to the interned string.
    _PyObject_{Get,Set,Has}AttrId are __getattr__ versions using _Py_Identifier*.
 */
 typedef struct _Py_Identifier {
diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -814,11 +814,10 @@
 _PyObject_GetAttrId(PyObject *v, _Py_Identifier *name)
 {
     PyObject *result;
-    PyObject *oname = _PyUnicode_FromId(name);
+    PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
     if (!oname)
         return NULL;
     result = PyObject_GetAttr(v, oname);
-    Py_DECREF(oname);
     return result;
 }
 
@@ -826,11 +825,10 @@
 _PyObject_HasAttrId(PyObject *v, _Py_Identifier *name)
 {
     int result;
-    PyObject *oname = _PyUnicode_FromId(name);
+    PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
     if (!oname)
         return -1;
     result = PyObject_HasAttr(v, oname);
-    Py_DECREF(oname);
     return result;
 }
 
@@ -838,11 +836,10 @@
 _PyObject_SetAttrId(PyObject *v, _Py_Identifier *name, PyObject *w)
 {
     int result;
-    PyObject *oname = _PyUnicode_FromId(name);
+    PyObject *oname = _PyUnicode_FromId(name); /* borrowed */
     if (!oname)
         return -1;
     result = PyObject_SetAttr(v, oname, w);
-    Py_DECREF(oname);
     return result;
 }
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -1716,11 +1716,10 @@
     PyObject *dict_str;
     PyObject *descr;
 
-    dict_str = _PyUnicode_FromId(&PyId___dict__);
+    dict_str = _PyUnicode_FromId(&PyId___dict__); /* borrowed */
     if (dict_str == NULL)
         return NULL;
     descr = _PyType_Lookup(type, dict_str);
-    Py_DECREF(dict_str);
     if (descr == NULL || !PyDescr_IsData(descr))
         return NULL;
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1723,7 +1723,6 @@
         id->next = static_strings;
         static_strings = id;
     }
-    Py_INCREF(id->object);
     return id->object;
 }
 
diff --git a/Python/_warnings.c b/Python/_warnings.c
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -666,10 +666,8 @@
 
         if ((tmp = _PyUnicode_FromId(&PyId_get_source)) == NULL)
             return NULL;
-        Py_DECREF(tmp);
         if ((tmp = _PyUnicode_FromId(&PyId_splitlines)) == NULL)
             return NULL;
-        Py_DECREF(tmp);
 
         /* Check/get the requisite pieces needed for the loader. */
         loader = PyDict_GetItemString(module_globals, "__loader__");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 18:43:52 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 08 Nov 2011 18:43:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_missing_goto?=
Message-ID: <E1RNpiC-00057Q-8I@dinsdale.python.org>

http://hg.python.org/cpython/rev/8f1aabbd50d0
changeset:   73458:8f1aabbd50d0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 08 18:37:16 2011 +0100
summary:
  Fix missing goto

files:
  Objects/unicodeobject.c |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7664,6 +7664,7 @@
         }
     }
     if (PyUnicode_Resize(&v, outpos) < 0)
+        goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
     assert(_PyUnicode_CheckConsistency(v, 1));

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 18:43:53 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 08 Nov 2011 18:43:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_BytesWarnings_in_fnmatc?=
	=?utf8?q?h?=
Message-ID: <E1RNpiD-00057a-1t@dinsdale.python.org>

http://hg.python.org/cpython/rev/9cec4a4797b2
changeset:   73459:9cec4a4797b2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 08 18:39:15 2011 +0100
summary:
  Fix BytesWarnings in fnmatch

files:
  Lib/fnmatch.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/fnmatch.py b/Lib/fnmatch.py
--- a/Lib/fnmatch.py
+++ b/Lib/fnmatch.py
@@ -35,7 +35,7 @@
     pat = os.path.normcase(pat)
     return fnmatchcase(name, pat)
 
- at functools.lru_cache(maxsize=250)
+ at functools.lru_cache(maxsize=250, typed=True)
 def _compile_pattern(pat):
     if isinstance(pat, bytes):
         pat_str = str(pat, 'ISO-8859-1')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 19:44:06 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 08 Nov 2011 19:44:06 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogdGVzdF9pbXBvcnQ6?=
 =?utf8?q?_test=5Fexecute=5Fbit=5Fnot=5Fcopied=28=29_was_actually_a_no-op?=
 =?utf8?b?OiBmaXggaXQu?=
Message-ID: <E1RNqeU-0002fh-Vi@dinsdale.python.org>

http://hg.python.org/cpython/rev/f0b1d8e4de4f
changeset:   73460:f0b1d8e4de4f
branch:      3.2
parent:      73452:3b9f58f85d3e
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 08 19:42:02 2011 +0100
summary:
  test_import: test_execute_bit_not_copied() was actually a no-op: fix it.

files:
  Lib/test/test_import.py |  7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -112,10 +112,9 @@
                 if not os.path.exists(fn):
                     self.fail("__import__ did not result in creation of "
                               "either a .pyc or .pyo file")
-                    s = os.stat(fn)
-                    self.assertEqual(
-                        stat.S_IMODE(s.st_mode),
-                        stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
+                s = os.stat(fn)
+                self.assertEqual(stat.S_IMODE(s.st_mode),
+                                 stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
             finally:
                 del sys.path[0]
                 remove_files(TESTFN)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 19:44:07 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 08 Nov 2011 19:44:07 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiB0ZXN0X2ltcG9ydDogdGVzdF9leGVjdXRlX2JpdF9ub3RfY29waWVkKCkgd2Fz?=
 =?utf8?q?_actually_a_no-op=3A_enable_it=2E?=
Message-ID: <E1RNqeV-0002fr-OX@dinsdale.python.org>

http://hg.python.org/cpython/rev/b6336ba796d4
changeset:   73461:b6336ba796d4
parent:      73459:9cec4a4797b2
parent:      73460:f0b1d8e4de4f
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 08 19:43:09 2011 +0100
summary:
  test_import: test_execute_bit_not_copied() was actually a no-op: enable it.

files:
  Lib/test/test_import.py |  7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -112,10 +112,9 @@
                 if not os.path.exists(fn):
                     self.fail("__import__ did not result in creation of "
                               "either a .pyc or .pyo file")
-                    s = os.stat(fn)
-                    self.assertEqual(
-                        stat.S_IMODE(s.st_mode),
-                        stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
+                s = os.stat(fn)
+                self.assertEqual(stat.S_IMODE(s.st_mode),
+                                 stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
             finally:
                 del sys.path[0]
                 remove_files(TESTFN)

-- 
Repository URL: http://hg.python.org/cpython

From tjreedy at udel.edu  Tue Nov  8 19:20:18 2011
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 08 Nov 2011 13:20:18 -0500
Subject: [Python-checkins] cpython (3.2): Issue #13237: Forward port
 subprocess module updates and explicitly document
In-Reply-To: <E1RNkX3-0003zm-6z@dinsdale.python.org>
References: <E1RNkX3-0003zm-6z@dinsdale.python.org>
Message-ID: <4EB972E2.5090001@udel.edu>

On 11/8/2011 7:12 AM, nick.coghlan wrote:
> http://hg.python.org/cpython/rev/e929d2a96d9b
> changeset:   73445:e929d2a96d9b
> branch:      3.2
> parent:      73430:4facbfdc7700
> user:        Nick Coghlan<ncoghlan at gmail.com>
> date:        Tue Nov 08 20:49:23 2011 +1000
> summary:
>    Issue #13237: Forward port subprocess module updates and explicitly document UTF-8 encoding assumption when universal_newlines=True
> +
> +>>>  subprocess.check_call("exit 1", shell=True)
> +      Traceback (most recent call last):
> +         ...
> +      subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
> +
> +   .. versionadded:: 2.5

> +>>>  subprocess.check_output(
> +      ...     "ls non_existent_file; exit 0",
> +      ...     stderr=subprocess.STDOUT,
> +      ...     shell=True)
> +      'ls: non_existent_file: No such file or directory\n'
> +
> +   .. versionadded:: 2.7

The two 2.x added notes should be deleted.


From python-checkins at python.org  Tue Nov  8 20:39:38 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 08 Nov 2011 20:39:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Back_out_changeset_b6336ba7?=
 =?utf8?q?96d4_until_fix_for_=2313303=2E?=
Message-ID: <E1RNrWE-0008Ko-AF@dinsdale.python.org>

http://hg.python.org/cpython/rev/1238cdd288d2
changeset:   73462:1238cdd288d2
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 08 20:38:11 2011 +0100
summary:
  Back out changeset b6336ba796d4 until fix for #13303.

files:
  Lib/test/test_import.py |  7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -112,9 +112,10 @@
                 if not os.path.exists(fn):
                     self.fail("__import__ did not result in creation of "
                               "either a .pyc or .pyo file")
-                s = os.stat(fn)
-                self.assertEqual(stat.S_IMODE(s.st_mode),
-                                 stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
+                    s = os.stat(fn)
+                    self.assertEqual(
+                        stat.S_IMODE(s.st_mode),
+                        stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
             finally:
                 del sys.path[0]
                 remove_files(TESTFN)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 21:30:16 2011
From: python-checkins at python.org (brian.curtin)
Date: Tue, 08 Nov 2011 21:30:16 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_the_old_style_=5B=2E?=
 =?utf8?q?=2E=2E=5D_to_denote_optional_args_and_show_the_defaults=2E?=
Message-ID: <E1RNsJE-0004wZ-OL@dinsdale.python.org>

http://hg.python.org/cpython/rev/60ae7979fec8
changeset:   73463:60ae7979fec8
user:        Brian Curtin <brian at python.org>
date:        Tue Nov 08 14:30:02 2011 -0600
summary:
  Remove the old style [...] to denote optional args and show the defaults.

files:
  Doc/library/os.rst |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -872,7 +872,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimesat(dirfd, path[, (atime, mtime)])
+.. function:: futimesat(dirfd, path, (atime, mtime)=None)
 
    Like :func:`utime` but if *path* is relative, it is taken as relative to *dirfd*.
    If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path*
@@ -883,7 +883,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimens(fd[, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec)])
+.. function:: futimens(fd, (atime_sec, atime_nsec)=None, (mtime_sec, mtime_nsec)=None)
 
    Updates the timestamps of a file specified by the file descriptor *fd*, with
    nanosecond precision.
@@ -909,7 +909,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimes(fd[, (atime, mtime)])
+.. function:: futimes(fd, (atime, mtime)=None)
 
    Set the access and modified time of the file specified by the file
    descriptor *fd* to the given values. If no second argument is used, set the
@@ -1285,7 +1285,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: utimensat(dirfd, path[, atime=(atime_sec, atime_nsec), mtime=(mtime_sec, mtime_nsec), flags=0])
+.. function:: utimensat(dirfd, path, atime=(atime_sec, atime_nsec), mtime=(mtime_sec, mtime_nsec), flags=0)
 
    Updates the timestamps of a file with nanosecond precision.
    The *atime* and *mtime* tuples default to ``None``, which sets those
@@ -1699,7 +1699,7 @@
       Added support for Windows 6.0 (Vista) symbolic links.
 
 
-.. function:: lutimes(path[, (atime, mtime)])
+.. function:: lutimes(path, (atime, mtime)=None)
 
    Like :func:`utime`, but if *path* is a symbolic link, it is not
    dereferenced.
@@ -2130,7 +2130,7 @@
    Availability: Unix, Windows.
 
 
-.. function:: utime(path[, times])
+.. function:: utime(path, times=None)
 
    Set the access and modified times of the file specified by *path*. If *times*
    is ``None`` or not specified, then the file's access and modified times are

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 21:54:56 2011
From: python-checkins at python.org (brian.curtin)
Date: Tue, 08 Nov 2011 21:54:56 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Backed_out_changeset_60ae79?=
	=?utf8?q?79fec8?=
Message-ID: <E1RNsh6-0007Ll-8v@dinsdale.python.org>

http://hg.python.org/cpython/rev/2636df45b630
changeset:   73464:2636df45b630
user:        Brian Curtin <brian at python.org>
date:        Tue Nov 08 14:54:02 2011 -0600
summary:
  Backed out changeset 60ae7979fec8

files:
  Doc/library/os.rst |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -872,7 +872,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimesat(dirfd, path, (atime, mtime)=None)
+.. function:: futimesat(dirfd, path[, (atime, mtime)])
 
    Like :func:`utime` but if *path* is relative, it is taken as relative to *dirfd*.
    If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path*
@@ -883,7 +883,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimens(fd, (atime_sec, atime_nsec)=None, (mtime_sec, mtime_nsec)=None)
+.. function:: futimens(fd[, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec)])
 
    Updates the timestamps of a file specified by the file descriptor *fd*, with
    nanosecond precision.
@@ -909,7 +909,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimes(fd, (atime, mtime)=None)
+.. function:: futimes(fd[, (atime, mtime)])
 
    Set the access and modified time of the file specified by the file
    descriptor *fd* to the given values. If no second argument is used, set the
@@ -1285,7 +1285,7 @@
    .. versionadded:: 3.3
 
 
-.. function:: utimensat(dirfd, path, atime=(atime_sec, atime_nsec), mtime=(mtime_sec, mtime_nsec), flags=0)
+.. function:: utimensat(dirfd, path[, atime=(atime_sec, atime_nsec), mtime=(mtime_sec, mtime_nsec), flags=0])
 
    Updates the timestamps of a file with nanosecond precision.
    The *atime* and *mtime* tuples default to ``None``, which sets those
@@ -1699,7 +1699,7 @@
       Added support for Windows 6.0 (Vista) symbolic links.
 
 
-.. function:: lutimes(path, (atime, mtime)=None)
+.. function:: lutimes(path[, (atime, mtime)])
 
    Like :func:`utime`, but if *path* is a symbolic link, it is not
    dereferenced.
@@ -2130,7 +2130,7 @@
    Availability: Unix, Windows.
 
 
-.. function:: utime(path, times=None)
+.. function:: utime(path[, times])
 
    Set the access and modified times of the file specified by *path*. If *times*
    is ``None`` or not specified, then the file's access and modified times are

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov  8 23:26:11 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 08 Nov 2011 23:26:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Refactor_functions_signatur?=
	=?utf8?q?es_in_the_doc=2E?=
Message-ID: <E1RNu7P-00086A-T2@dinsdale.python.org>

http://hg.python.org/cpython/rev/22ff621e3805
changeset:   73465:22ff621e3805
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Wed Nov 09 00:25:47 2011 +0200
summary:
  Refactor functions signatures in the doc.

files:
  Doc/library/os.rst |  23 +++++++++++++++--------
  1 files changed, 15 insertions(+), 8 deletions(-)


diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -872,22 +872,26 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimesat(dirfd, path[, (atime, mtime)])
+.. function:: futimesat(dirfd, path[, times])
 
    Like :func:`utime` but if *path* is relative, it is taken as relative to *dirfd*.
    If *path* is relative and *dirfd* is the special value :data:`AT_FDCWD`, then *path*
-   is interpreted relative to the current working directory.
+   is interpreted relative to the current working directory.  *times* must be a
+   2-tuple of numbers, of the form ``(atime, mtime)``, or None.
 
    Availability: Unix.
 
    .. versionadded:: 3.3
 
 
-.. function:: futimens(fd[, (atime_sec, atime_nsec), (mtime_sec, mtime_nsec)])
+.. function:: futimens(fd[, atimes, mtimes])
 
    Updates the timestamps of a file specified by the file descriptor *fd*, with
    nanosecond precision.
    If no second argument is given, set *atime* and *mtime* to the current time.
+   *atimes* and *mtimes* must be 2-tuples of numbers, of the form
+   ``(atime_sec, atime_nsec)`` and ``(mtime_sec, mtime_nsec)`` respectively,
+   or ``None``.
    If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_NOW`, the corresponding
    timestamp is updated to the current time.
    If *atime_nsec* or *mtime_nsec* is specified as :data:`UTIME_OMIT`, the corresponding
@@ -909,11 +913,12 @@
    .. versionadded:: 3.3
 
 
-.. function:: futimes(fd[, (atime, mtime)])
+.. function:: futimes(fd[, times])
 
    Set the access and modified time of the file specified by the file
-   descriptor *fd* to the given values. If no second argument is used, set the
-   access and modified times to the current time.
+   descriptor *fd* to the given values.  *atimes* must be a 2-tuple of numbers,
+   of the form ``(atime, mtime)``, or None.  If no second argument is used,
+   set the access and modified times to the current time.
 
    Availability: Unix.
 
@@ -1699,10 +1704,12 @@
       Added support for Windows 6.0 (Vista) symbolic links.
 
 
-.. function:: lutimes(path[, (atime, mtime)])
+.. function:: lutimes(path[, times])
 
    Like :func:`utime`, but if *path* is a symbolic link, it is not
-   dereferenced.
+   dereferenced.  *times* must be a 2-tuple of numbers, of the form
+   ``(atime, mtime)``, or None.
+
 
    Availability: Unix.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  9 00:02:17 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 09 Nov 2011 00:02:17 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_the_code_page_decoder?=
Message-ID: <E1RNugL-0002pn-7E@dinsdale.python.org>

http://hg.python.org/cpython/rev/9042d71962d6
changeset:   73466:9042d71962d6
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 09 00:02:18 2011 +0100
summary:
  Fix the code page decoder

 * unicode_decode_call_errorhandler() now supports the PyUnicode_WCHAR_KIND
   kind
 * unicode_decode_call_errorhandler() calls copy_characters() instead of
   PyUnicode_CopyCharacters()

files:
  Objects/unicodeobject.c |  63 ++++++++++++++++++++--------
  1 files changed, 45 insertions(+), 18 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3622,14 +3622,18 @@
 
     PyObject *restuple = NULL;
     PyObject *repunicode = NULL;
-    Py_ssize_t outsize = PyUnicode_GET_LENGTH(*output);
+    Py_ssize_t outsize;
     Py_ssize_t insize;
     Py_ssize_t requiredsize;
     Py_ssize_t newpos;
     PyObject *inputobj = NULL;
-    Py_ssize_t replen;
     int res = -1;
 
+    if (_PyUnicode_KIND(*output) != PyUnicode_WCHAR_KIND)
+        outsize = PyUnicode_GET_LENGTH(*output);
+    else
+        outsize = _PyUnicode_WSTR_LENGTH(*output);
+
     if (*errorHandler == NULL) {
         *errorHandler = PyCodec_LookupError(errors);
         if (*errorHandler == NULL)
@@ -3678,24 +3682,46 @@
         goto onError;
     }
 
-    /* need more space? (at least enough for what we
-       have+the replacement+the rest of the string (starting
-       at the new input position), so we won't have to check space
-       when there are no errors in the rest of the string) */
-    replen = PyUnicode_GET_LENGTH(repunicode);
-    requiredsize = *outpos + replen + insize-newpos;
-    if (requiredsize > outsize) {
-        if (requiredsize<2*outsize)
-            requiredsize = 2*outsize;
-        if (unicode_resize(output, requiredsize) < 0)
+    if (_PyUnicode_KIND(*output) != PyUnicode_WCHAR_KIND) {
+        /* need more space? (at least enough for what we
+           have+the replacement+the rest of the string (starting
+           at the new input position), so we won't have to check space
+           when there are no errors in the rest of the string) */
+        Py_ssize_t replen = PyUnicode_GET_LENGTH(repunicode);
+        requiredsize = *outpos + replen + insize-newpos;
+        if (requiredsize > outsize) {
+            if (requiredsize<2*outsize)
+                requiredsize = 2*outsize;
+            if (unicode_resize(output, requiredsize) < 0)
+                goto onError;
+        }
+        if (unicode_widen(output, PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0)
             goto onError;
-    }
-    if (unicode_widen(output, PyUnicode_MAX_CHAR_VALUE(repunicode)) < 0)
-        goto onError;
+        copy_characters(*output, *outpos, repunicode, 0, replen);
+        *outpos += replen;
+    }
+    else {
+        wchar_t *repwstr;
+        Py_ssize_t repwlen;
+        repwstr = PyUnicode_AsUnicodeAndSize(repunicode, &repwlen);
+        if (repwstr == NULL)
+            goto onError;
+        /* need more space? (at least enough for what we
+           have+the replacement+the rest of the string (starting
+           at the new input position), so we won't have to check space
+           when there are no errors in the rest of the string) */
+        requiredsize = *outpos + repwlen + insize-newpos;
+        if (requiredsize > outsize) {
+            if (requiredsize < 2*outsize)
+                requiredsize = 2*outsize;
+            if (unicode_resize(output, requiredsize) < 0)
+                goto onError;
+        }
+        wcsncpy(_PyUnicode_WSTR(*output) + *outpos, repwstr, repwlen);
+        *outpos += repwlen;
+    }
     *endinpos = newpos;
     *inptr = *input + newpos;
-    PyUnicode_CopyCharacters(*output, *outpos, repunicode, 0, replen);
-    *outpos += replen;
 
     /* we made it! */
     res = 0;
@@ -6976,10 +7002,11 @@
                     errors, &errorHandler,
                     encoding, reason,
                     &startin, &endin, &startinpos, &endinpos, &exc, &in,
-                    v, &outpos, &out))
+                    v, &outpos))
             {
                 goto error;
             }
+            out = PyUnicode_AS_UNICODE(*v) + outpos;
         }
         else {
             in += insize;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  9 00:02:18 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 09 Nov 2011 00:02:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_compiler_warning=3A_u?=
 =?utf8?q?se_unsiged_for_maxchar_in_unicode=5Fwiden=28=29?=
Message-ID: <E1RNugM-0002q0-LL@dinsdale.python.org>

http://hg.python.org/cpython/rev/349c6576d5d4
changeset:   73467:349c6576d5d4
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 09 00:02:42 2011 +0100
summary:
  Fix a compiler warning: use unsiged for maxchar in unicode_widen()

files:
  Objects/unicodeobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1556,7 +1556,7 @@
 }
 
 static int
-unicode_widen(PyObject **p_unicode, int maxchar)
+unicode_widen(PyObject **p_unicode, unsigned int maxchar)
 {
     PyObject *result;
     assert(PyUnicode_IS_READY(*p_unicode));

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  9 00:02:19 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 09 Nov 2011 00:02:19 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Strip_trailing_spaces?=
Message-ID: <E1RNugN-0002q4-Hp@dinsdale.python.org>

http://hg.python.org/cpython/rev/f01123c71c57
changeset:   73468:f01123c71c57
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 09 00:03:45 2011 +0100
summary:
  Strip trailing spaces

files:
  Objects/unicodeobject.c |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4338,7 +4338,7 @@
     maxchar = utf8_max_char_size_and_has_errors(s, size, &unicode_size,
                                                 consumed, &has_errors);
     if (has_errors)
-        /* maxchar and size computation might be incorrect; 
+        /* maxchar and size computation might be incorrect;
            code below widens and resizes as necessary. */
         unicode = PyUnicode_New(size, 127);
     else
@@ -5321,8 +5321,8 @@
             Py_UNICODE ch2 = (q[ihi] << 8) | q[ilo];
             q += 2;
             if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
-                if (unicode_putchar(&unicode, &outpos, 
-                                    (((ch & 0x3FF)<<10) | 
+                if (unicode_putchar(&unicode, &outpos,
+                                    (((ch & 0x3FF)<<10) |
                                      (ch2 & 0x3FF)) + 0x10000) < 0)
                     goto onError;
                 continue;
@@ -7653,7 +7653,7 @@
 
                 if (targetsize == 1) {
                     /* 1-1 mapping */
-                    if (unicode_putchar(&v, &outpos, 
+                    if (unicode_putchar(&v, &outpos,
                                         PyUnicode_READ_CHAR(x, 0)) < 0)
                         goto onError;
                 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  9 01:12:03 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 09 Nov 2011 01:12:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313374=3A_Use_Unico?=
 =?utf8?q?de_filenames_instead_of_bytes_filenames?=
Message-ID: <E1RNvlr-0000bw-Az@dinsdale.python.org>

http://hg.python.org/cpython/rev/6bf07db23445
changeset:   73469:6bf07db23445
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 09 01:13:45 2011 +0100
summary:
  Issue #13374: Use Unicode filenames instead of bytes filenames

getcwdb() => getcwd()

files:
  Lib/test/test_unicode_file.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_unicode_file.py b/Lib/test/test_unicode_file.py
--- a/Lib/test/test_unicode_file.py
+++ b/Lib/test/test_unicode_file.py
@@ -82,7 +82,7 @@
         self.assertFalse(os.path.exists(filename2 + '.new'))
 
     def _do_directory(self, make_name, chdir_name):
-        cwd = os.getcwdb()
+        cwd = os.getcwd()
         if os.path.isdir(make_name):
             rmtree(make_name)
         os.mkdir(make_name)

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Nov  9 05:47:46 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 09 Nov 2011 05:47:46 +0100
Subject: [Python-checkins] Daily reference leaks (6bf07db23445): sum=0
Message-ID: <E1RO04g-00044O-Dp@ap.vmr.nerim.net>

results for 6bf07db23445 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog9vToyd', '-x']

From python-checkins at python.org  Wed Nov  9 19:45:56 2011
From: python-checkins at python.org (barry.warsaw)
Date: Wed, 09 Nov 2011 19:45:56 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Un-release_schedule_for_Python?=
	=?utf8?b?IDIuOC4=?=
Message-ID: <E1ROD9o-0004Ur-6Q@dinsdale.python.org>

http://hg.python.org/peps/rev/c832164e7a99
changeset:   3982:c832164e7a99
user:        Barry Warsaw <barry at python.org>
date:        Wed Nov 09 13:45:51 2011 -0500
summary:
  Un-release schedule for Python 2.8.

files:
  pep-0373.txt |   8 ++--
  pep-0405.txt |  64 ++++++++++++++++++++++++++++++++++++++++
  2 files changed, 68 insertions(+), 4 deletions(-)


diff --git a/pep-0373.txt b/pep-0373.txt
--- a/pep-0373.txt
+++ b/pep-0373.txt
@@ -13,10 +13,10 @@
 Abstract
 ========
 
-This document describes the development and release schedule for Python 2.7.
-The schedule primarily concerns itself with PEP-sized items.  Small features may
-be added up to and including the first beta release.  Bugs may be fixed until
-the final release.
+This document describes the development and release schedule for
+Python 2.7.  The schedule primarily concerns itself with PEP-sized
+items.  Small features may be added up to and including the first beta
+release.  Bugs may be fixed until the final release.
 
 
 Release Manager and Crew
diff --git a/pep-0405.txt b/pep-0405.txt
new file mode 100644
--- /dev/null
+++ b/pep-0405.txt
@@ -0,0 +1,64 @@
+PEP: 405
+Title: Python 2.8 Un-release Schedule
+Version: $Revision$
+Last-Modified: $Date$
+Author: Barry Warsaw <barry at python.org>
+Status: Final
+Type: Informational
+Content-Type: text/x-rst
+Created: 2011-11-09
+Python-Version: 2.8
+
+
+Abstract
+========
+
+This document describes the un-development and un-release schedule for Python
+2.8.
+
+
+Un-release Manager and Crew
+===========================
+
+============================ ==================
+Position                     Name
+============================ ==================
+2.8 Un-release Manager       Barry Warsaw
+============================ ==================
+
+
+Un-release Schedule
+===================
+
+The current un-schedule is:
+
+ - 2.8 final Never
+
+
+Official pronouncement
+======================
+
+There will never be an official Python 2.8 release.
+
+
+Upgrade path
+============
+
+The official upgrade path from Python 2.7 is to Python 3.
+
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+
+
+..
+  Local Variables:
+  mode: indented-text
+  indent-tabs-mode: nil
+  sentence-end-double-space: t
+  fill-column: 70
+  coding: utf-8
+  End:

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Wed Nov  9 22:12:30 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 09 Nov 2011 22:12:30 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312442=3A_nt=2E=5Fg?=
 =?utf8?q?etdiskusage=28=29_is_now_using_the_Windows_Unicode_API?=
Message-ID: <E1ROFRe-0002BW-Ri@dinsdale.python.org>

http://hg.python.org/cpython/rev/bdb3f0e7e268
changeset:   73470:bdb3f0e7e268
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 09 22:14:14 2011 +0100
summary:
  Issue #12442: nt._getdiskusage() is now using the Windows Unicode API

files:
  Modules/posixmodule.c |  14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -8163,13 +8163,13 @@
 {
     BOOL retval;
     ULARGE_INTEGER _, total, free;
-    LPCTSTR path;
-
-    if (! PyArg_ParseTuple(args, "s", &path))
-        return NULL;
-
-    Py_BEGIN_ALLOW_THREADS
-    retval = GetDiskFreeSpaceEx(path, &_, &total, &free);
+    const wchar_t *path;
+
+    if (! PyArg_ParseTuple(args, "u", &path))
+        return NULL;
+
+    Py_BEGIN_ALLOW_THREADS
+    retval = GetDiskFreeSpaceExW(path, &_, &total, &free);
     Py_END_ALLOW_THREADS
     if (retval == 0)
         return PyErr_SetFromWindowsErr(0);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov  9 22:53:16 2011
From: python-checkins at python.org (barry.warsaw)
Date: Wed, 09 Nov 2011 22:53:16 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Er=2C_refinement=2E?=
Message-ID: <E1ROG56-00062i-Ne@dinsdale.python.org>

http://hg.python.org/peps/rev/6e2586dd623b
changeset:   3983:6e2586dd623b
user:        Barry Warsaw <barry at python.org>
date:        Wed Nov 09 16:53:10 2011 -0500
summary:
  Er, refinement.

files:
  pep-0405.txt |  5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)


diff --git a/pep-0405.txt b/pep-0405.txt
--- a/pep-0405.txt
+++ b/pep-0405.txt
@@ -23,7 +23,7 @@
 ============================ ==================
 Position                     Name
 ============================ ==================
-2.8 Un-release Manager       Barry Warsaw
+2.8 Un-release Manager       Cardinal Biggles
 ============================ ==================
 
 
@@ -38,7 +38,8 @@
 Official pronouncement
 ======================
 
-There will never be an official Python 2.8 release.
+Rule number six: there is *no* official Python 2.8 release.  There never will
+be an official Python 2.8 release.  It is an ex-release.
 
 
 Upgrade path

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Nov 10 00:45:22 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 10 Nov 2011 00:45:22 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMzcz?=
 =?utf8?q?=3A_multiprocessing=2EQueue=2Eget=28=29_could_sometimes_block_in?=
 =?utf8?q?definitely?=
Message-ID: <E1ROHpa-0007Wa-9h@dinsdale.python.org>

http://hg.python.org/cpython/rev/e7b6dca28a2f
changeset:   73471:e7b6dca28a2f
branch:      2.7
parent:      73450:554802e562fa
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 10 00:33:50 2011 +0100
summary:
  Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
when called with a timeout.  Patch by Arnaud Ysmal.

files:
  Lib/multiprocessing/queues.py |  6 +++++-
  Misc/ACKS                     |  1 +
  Misc/NEWS                     |  3 +++
  3 files changed, 9 insertions(+), 1 deletions(-)


diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -126,7 +126,11 @@
             if not self._rlock.acquire(block, timeout):
                 raise Empty
             try:
-                if not self._poll(block and (deadline-time.time()) or 0.0):
+                if block:
+                    timeout = deadline - time.time()
+                    if timeout < 0 or not self._poll(timeout):
+                        raise Empty
+                elif not self._poll():
                     raise Empty
                 res = self._recv()
                 self._sem.release()
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -924,6 +924,7 @@
 Danny Yoo
 George Yoshida
 Masazumi Yoshikawa
+Arnaud Ysmal
 Bernard Yue
 Moshe Zadka
 Milan Zamazal
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -76,6 +76,9 @@
 Library
 -------
 
+- Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
+  when called with a timeout.  Patch by Arnaud Ysmal.
+
 - Issue #3067: Enhance the documentation and docstring of
   locale.setlocale().
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 00:45:24 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 10 Nov 2011 00:45:24 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzcz?=
 =?utf8?q?=3A_multiprocessing=2EQueue=2Eget=28=29_could_sometimes_block_in?=
 =?utf8?q?definitely?=
Message-ID: <E1ROHpc-0007We-Jh@dinsdale.python.org>

http://hg.python.org/cpython/rev/9328080a19c0
changeset:   73472:9328080a19c0
branch:      3.2
parent:      73460:f0b1d8e4de4f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 10 00:37:09 2011 +0100
summary:
  Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
when called with a timeout.  Patch by Arnaud Ysmal.

files:
  Lib/multiprocessing/queues.py |  6 +++++-
  Misc/ACKS                     |  1 +
  Misc/NEWS                     |  3 +++
  3 files changed, 9 insertions(+), 1 deletions(-)


diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -126,7 +126,11 @@
             if not self._rlock.acquire(block, timeout):
                 raise Empty
             try:
-                if not self._poll(block and (deadline-time.time()) or 0.0):
+                if block:
+                    timeout = deadline - time.time()
+                    if timeout < 0 or not self._poll(timeout):
+                        raise Empty
+                elif not self._poll():
                     raise Empty
                 res = self._recv()
                 self._sem.release()
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1001,6 +1001,7 @@
 Danny Yoo
 George Yoshida
 Masazumi Yoshikawa
+Arnaud Ysmal
 Bernard Yue
 Moshe Zadka
 Milan Zamazal
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
 Library
 -------
 
+- Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
+  when called with a timeout.  Patch by Arnaud Ysmal.
+
 - Issue #13254: Fix Maildir initialization so that maildir contents
   are read correctly.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 00:45:25 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 10 Nov 2011 00:45:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313373=3A_multiprocessing=2EQueue=2Eget=28=29_could_?=
 =?utf8?q?sometimes_block_indefinitely?=
Message-ID: <E1ROHpd-0007Wj-Ke@dinsdale.python.org>

http://hg.python.org/cpython/rev/f34b4e250b44
changeset:   73473:f34b4e250b44
parent:      73470:bdb3f0e7e268
parent:      73472:9328080a19c0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 10 00:38:25 2011 +0100
summary:
  Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
when called with a timeout.  Patch by Arnaud Ysmal.

files:
  Lib/multiprocessing/queues.py |  6 +++++-
  Misc/ACKS                     |  1 +
  Misc/NEWS                     |  3 +++
  3 files changed, 9 insertions(+), 1 deletions(-)


diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -129,7 +129,11 @@
             if not self._rlock.acquire(block, timeout):
                 raise Empty
             try:
-                if not self._poll(block and (deadline-time.time()) or 0.0):
+                if block:
+                    timeout = deadline - time.time()
+                    if timeout < 0 or not self._poll(timeout):
+                        raise Empty
+                elif not self._poll():
                     raise Empty
                 res = self._recv()
                 self._sem.release()
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1068,6 +1068,7 @@
 Danny Yoo
 George Yoshida
 Masazumi Yoshikawa
+Arnaud Ysmal
 Bernard Yue
 Moshe Zadka
 Milan Zamazal
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -365,6 +365,9 @@
 Library
 -------
 
+- Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
+  when called with a timeout.  Patch by Arnaud Ysmal.
+
 - Issue #13254: Fix Maildir initialization so that maildir contents
   are read correctly.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Nov 10 05:32:54 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 10 Nov 2011 05:32:54 +0100
Subject: [Python-checkins] Daily reference leaks (f34b4e250b44): sum=0
Message-ID: <E1ROMJq-0004Vq-S9@ap.vmr.nerim.net>

results for f34b4e250b44 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogKaup2R', '-x']

From python-checkins at python.org  Thu Nov 10 08:38:02 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 10 Nov 2011 08:38:02 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEzMzc5OiB1cGRh?=
 =?utf8?q?te_Unicode_version_in_unicodedata_docstrings_and_comments=2E?=
Message-ID: <E1ROPD0-0000iv-Rh@dinsdale.python.org>

http://hg.python.org/cpython/rev/6ddda809ea8c
changeset:   73474:6ddda809ea8c
branch:      3.2
parent:      73472:9328080a19c0
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 10 09:36:34 2011 +0200
summary:
  #13379: update Unicode version in unicodedata docstrings and comments.

files:
  Modules/unicodedata.c |  11 ++++++-----
  1 files changed, 6 insertions(+), 5 deletions(-)


diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -1,8 +1,9 @@
 /* ------------------------------------------------------------------------
 
-   unicodedata -- Provides access to the Unicode 5.2 data base.
+   unicodedata -- Provides access to the Unicode database.
 
-   Data was extracted from the Unicode 5.2 UnicodeData.txt file.
+   Data was extracted from the UnicodeData.txt file.
+   The current version number is reported in the unidata_version constant.
 
    Written by Marc-Andre Lemburg (mal at lemburg.com).
    Modified for Python 2.0 by Fredrik Lundh (fredrik at pythonware.com)
@@ -1241,11 +1242,11 @@
 "This module provides access to the Unicode Character Database which\n\
 defines character properties for all Unicode characters. The data in\n\
 this database is based on the UnicodeData.txt file version\n\
-5.2.0 which is publically available from ftp://ftp.unicode.org/.\n\
+6.0.0 which is publically available from ftp://ftp.unicode.org/.\n\
 \n\
 The module uses the same names and symbols as defined by the\n\
-UnicodeData File Format 5.2.0 (see\n\
-http://www.unicode.org/reports/tr44/tr44-4.html).");
+UnicodeData File Format 6.0.0 (see\n\
+http://www.unicode.org/reports/tr44/tr44-6.html).");
 
 
 static struct PyModuleDef unicodedatamodule = {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 08:38:03 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 10 Nov 2011 08:38:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2313379=3A_merge_with_3=2E2=2E?=
Message-ID: <E1ROPD1-0000jR-NA@dinsdale.python.org>

http://hg.python.org/cpython/rev/5fdd0c8032d0
changeset:   73475:5fdd0c8032d0
parent:      73473:f34b4e250b44
parent:      73474:6ddda809ea8c
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 10 09:37:43 2011 +0200
summary:
  #13379: merge with 3.2.

files:
  Modules/unicodedata.c |  11 ++++++-----
  1 files changed, 6 insertions(+), 5 deletions(-)


diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c
--- a/Modules/unicodedata.c
+++ b/Modules/unicodedata.c
@@ -1,8 +1,9 @@
 /* ------------------------------------------------------------------------
 
-   unicodedata -- Provides access to the Unicode 5.2 data base.
+   unicodedata -- Provides access to the Unicode database.
 
-   Data was extracted from the Unicode 5.2 UnicodeData.txt file.
+   Data was extracted from the UnicodeData.txt file.
+   The current version number is reported in the unidata_version constant.
 
    Written by Marc-Andre Lemburg (mal at lemburg.com).
    Modified for Python 2.0 by Fredrik Lundh (fredrik at pythonware.com)
@@ -1318,11 +1319,11 @@
 "This module provides access to the Unicode Character Database which\n\
 defines character properties for all Unicode characters. The data in\n\
 this database is based on the UnicodeData.txt file version\n\
-5.2.0 which is publically available from ftp://ftp.unicode.org/.\n\
+6.0.0 which is publically available from ftp://ftp.unicode.org/.\n\
 \n\
 The module uses the same names and symbols as defined by the\n\
-UnicodeData File Format 5.2.0 (see\n\
-http://www.unicode.org/reports/tr44/tr44-4.html).");
+UnicodeData File Format 6.0.0 (see\n\
+http://www.unicode.org/reports/tr44/tr44-6.html).");
 
 
 static struct PyModuleDef unicodedatamodule = {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 18:26:54 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 10 Nov 2011 18:26:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Port_encoders_from_Py=5FUNI?=
 =?utf8?q?CODE_API_to_unicode_object_API=2E?=
Message-ID: <E1ROYOs-0004wS-Ml@dinsdale.python.org>

http://hg.python.org/cpython/rev/b3e532c1485e
changeset:   73476:b3e532c1485e
user:        Martin v. L?wis <martin at v.loewis.de>
date:        Thu Nov 10 18:24:32 2011 +0100
summary:
  Port encoders from Py_UNICODE API to unicode object API.

files:
  Include/unicodeobject.h |   16 +
  Modules/_codecsmodule.c |  120 +++-----
  Objects/unicodeobject.c |  401 +++++++++++++--------------
  3 files changed, 261 insertions(+), 276 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1088,6 +1088,12 @@
     int base64WhiteSpace,       /* Encode whitespace (sp, ht, nl, cr) in base64 */
     const char *errors          /* error handling */
     );
+PyAPI_FUNC(PyObject*) _PyUnicode_EncodeUTF7(
+    PyObject *unicode,          /* Unicode object */
+    int base64SetO,             /* Encode RFC2152 Set O characters in base64 */
+    int base64WhiteSpace,       /* Encode whitespace (sp, ht, nl, cr) in base64 */
+    const char *errors          /* error handling */
+    );
 #endif
 
 /* --- UTF-8 Codecs ------------------------------------------------------- */
@@ -1195,6 +1201,11 @@
     const char *errors,         /* error handling */
     int byteorder               /* byteorder to use 0=BOM+native;-1=LE,1=BE */
     );
+PyAPI_FUNC(PyObject*) _PyUnicode_EncodeUTF32(
+    PyObject *object,           /* Unicode object */
+    const char *errors,         /* error handling */
+    int byteorder               /* byteorder to use 0=BOM+native;-1=LE,1=BE */
+    );
 #endif
 
 /* --- UTF-16 Codecs ------------------------------------------------------ */
@@ -1275,6 +1286,11 @@
     const char *errors,         /* error handling */
     int byteorder               /* byteorder to use 0=BOM+native;-1=LE,1=BE */
     );
+PyAPI_FUNC(PyObject*) _PyUnicode_EncodeUTF16(
+    PyObject* unicode,          /* Unicode object */
+    const char *errors,         /* error handling */
+    int byteorder               /* byteorder to use 0=BOM+native;-1=LE,1=BE */
+    );
 #endif
 
 /* --- Unicode-Escape Codecs ---------------------------------------------- */
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -235,8 +235,10 @@
         return NULL;
 
     if (PyUnicode_Check(obj)) {
+        if (PyUnicode_READY(obj) < 0)
+            return NULL;
         Py_INCREF(obj);
-        return codec_tuple(obj, PyUnicode_GET_SIZE(obj));
+        return codec_tuple(obj, PyUnicode_GET_LENGTH(obj));
     }
     else {
         if (PyObject_AsReadBuffer(obj, (const void **)&data, &size))
@@ -676,10 +678,12 @@
         return NULL;
 
     if (PyUnicode_Check(obj)) {
+        if (PyUnicode_READY(obj) < 0)
+            return NULL;
         data = PyUnicode_AS_DATA(obj);
         size = PyUnicode_GET_DATA_SIZE(obj);
         return codec_tuple(PyBytes_FromStringAndSize(data, size),
-                           PyUnicode_GET_SIZE(obj));
+                           PyUnicode_GET_LENGTH(obj));
     }
     else {
         if (PyObject_AsReadBuffer(obj, (const void **)&data, &size))
@@ -700,14 +704,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF7(PyUnicode_AS_UNICODE(str),
-                                         PyUnicode_GET_SIZE(str),
-                                         0,
-                                         0,
-                                         errors),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF7(str, 0, 0, errors),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -752,13 +752,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(str),
-                                          PyUnicode_GET_SIZE(str),
-                                          errors,
-                                          byteorder),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF16(str, errors, byteorder),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -775,13 +772,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(str),
-                                             PyUnicode_GET_SIZE(str),
-                                             errors,
-                                             -1),
-                       PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF16(str, errors, -1),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -798,13 +792,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(str),
-                                          PyUnicode_GET_SIZE(str),
-                                          errors,
-                                          +1),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF16(str, errors, +1),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -829,13 +820,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(str),
-                                          PyUnicode_GET_SIZE(str),
-                                          errors,
-                                          byteorder),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF32(str, errors, byteorder),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -852,13 +840,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(str),
-                                             PyUnicode_GET_SIZE(str),
-                                             errors,
-                                             -1),
-                       PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF32(str, errors, -1),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -875,13 +860,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(str),
-                                          PyUnicode_GET_SIZE(str),
-                                          errors,
-                                          +1),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_EncodeUTF32(str, errors, +1),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -898,11 +880,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeUnicodeEscape(PyUnicode_AS_UNICODE(str),
-                                                  PyUnicode_GET_SIZE(str)),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(PyUnicode_AsUnicodeEscapeString(str),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -919,12 +900,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeRawUnicodeEscape(
-                               PyUnicode_AS_UNICODE(str),
-                               PyUnicode_GET_SIZE(str)),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(PyUnicode_AsRawUnicodeEscapeString(str),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -941,13 +920,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeLatin1(
-                               PyUnicode_AS_UNICODE(str),
-                               PyUnicode_GET_SIZE(str),
-                               errors),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_AsLatin1String(str, errors),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -964,13 +940,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeASCII(
-                               PyUnicode_AS_UNICODE(str),
-                               PyUnicode_GET_SIZE(str),
-                               errors),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(_PyUnicode_AsASCIIString(str, errors),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -990,10 +963,10 @@
         mapping = NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
     v = codec_tuple(_PyUnicode_EncodeCharmap(str, mapping, errors),
-                    PyUnicode_GET_SIZE(str));
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -1021,13 +994,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
-    v = codec_tuple(PyUnicode_EncodeMBCS(
-                               PyUnicode_AS_UNICODE(str),
-                               PyUnicode_GET_SIZE(str),
-                               errors),
-                    PyUnicode_GET_SIZE(str));
+    v = codec_tuple(PyUnicode_EncodeCodePage(CP_ACP, str, errors),
+                    PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
     return v;
 }
@@ -1045,7 +1015,7 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL)
+    if (str == NULL || PyUnicode_READY(str) < 0)
         return NULL;
     v = codec_tuple(PyUnicode_EncodeCodePage(code_page,
                                              str,
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4024,26 +4024,35 @@
 
 
 PyObject *
-PyUnicode_EncodeUTF7(const Py_UNICODE *s,
-                     Py_ssize_t size,
-                     int base64SetO,
-                     int base64WhiteSpace,
-                     const char *errors)
-{
+_PyUnicode_EncodeUTF7(PyObject *str,
+                      int base64SetO,
+                      int base64WhiteSpace,
+                      const char *errors)
+{
+    int kind;
+    void *data;
+    Py_ssize_t len;
     PyObject *v;
-    /* It might be possible to tighten this worst case */
-    Py_ssize_t allocated = 8 * size;
+    Py_ssize_t allocated;
     int inShift = 0;
-    Py_ssize_t i = 0;
+    Py_ssize_t i;
     unsigned int base64bits = 0;
     unsigned long base64buffer = 0;
     char * out;
     char * start;
 
-    if (size == 0)
+    if (PyUnicode_READY(str) < 0)
+        return NULL;
+    kind = PyUnicode_KIND(str);
+    data = PyUnicode_DATA(str);
+    len = PyUnicode_GET_LENGTH(str);
+
+    if (len == 0)
         return PyBytes_FromStringAndSize(NULL, 0);
 
-    if (allocated / 8 != size)
+    /* It might be possible to tighten this worst case */
+    allocated = 8 * len;
+    if (allocated / 8 != len)
         return PyErr_NoMemory();
 
     v = PyBytes_FromStringAndSize(NULL, allocated);
@@ -4051,8 +4060,8 @@
         return NULL;
 
     start = out = PyBytes_AS_STRING(v);
-    for (;i < size; ++i) {
-        Py_UNICODE ch = s[i];
+    for (i = 0; i < len; ++i) {
+        Py_UNICODE ch = PyUnicode_READ(kind, data, i);
 
         if (inShift) {
             if (ENCODE_DIRECT(ch, !base64SetO, !base64WhiteSpace)) {
@@ -4118,6 +4127,22 @@
         return NULL;
     return v;
 }
+PyObject *
+PyUnicode_EncodeUTF7(const Py_UNICODE *s,
+                     Py_ssize_t size,
+                     int base64SetO,
+                     int base64WhiteSpace,
+                     const char *errors)
+{
+    PyObject *result;
+    PyObject *tmp = PyUnicode_FromUnicode(s, size);
+    if (tmp == NULL)
+        return NULL;
+    result = _PyUnicode_EncodeUTF7(tmp, base64SetO, 
+                                   base64WhiteSpace, errors);
+    Py_DECREF(tmp);
+    return result;
+}
 
 #undef IS_BASE64
 #undef FROM_BASE64
@@ -5031,19 +5056,16 @@
 }
 
 PyObject *
-PyUnicode_EncodeUTF32(const Py_UNICODE *s,
-                      Py_ssize_t size,
-                      const char *errors,
-                      int byteorder)
-{
+_PyUnicode_EncodeUTF32(PyObject *str,
+                       const char *errors,
+                       int byteorder)
+{
+    int kind;
+    void *data;
+    Py_ssize_t len;
     PyObject *v;
     unsigned char *p;
-    Py_ssize_t nsize, bytesize;
-#ifndef Py_UNICODE_WIDE
-    Py_ssize_t i, pairs;
-#else
-    const int pairs = 0;
-#endif
+    Py_ssize_t nsize, bytesize, i;
     /* Offsets from p for storing byte pairs in the right order. */
 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
     int iorder[] = {0, 1, 2, 3};
@@ -5060,15 +5082,17 @@
         p += 4;                                 \
     } while(0)
 
-    /* In narrow builds we can output surrogate pairs as one codepoint,
-       so we need less space. */
-#ifndef Py_UNICODE_WIDE
-    for (i = pairs = 0; i < size-1; i++)
-        if (0xD800 <= s[i] && s[i] <= 0xDBFF &&
-            0xDC00 <= s[i+1] && s[i+1] <= 0xDFFF)
-            pairs++;
-#endif
-    nsize = (size - pairs + (byteorder == 0));
+    if (!PyUnicode_Check(str)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
+    if (PyUnicode_READY(str) < 0)
+        return NULL;
+    kind = PyUnicode_KIND(str);
+    data = PyUnicode_DATA(str);
+    len = PyUnicode_GET_LENGTH(str);
+
+    nsize = len + (byteorder == 0);
     bytesize = nsize * 4;
     if (bytesize / 4 != nsize)
         return PyErr_NoMemory();
@@ -5079,7 +5103,7 @@
     p = (unsigned char *)PyBytes_AS_STRING(v);
     if (byteorder == 0)
         STORECHAR(0xFEFF);
-    if (size == 0)
+    if (len == 0)
         goto done;
 
     if (byteorder == -1) {
@@ -5097,20 +5121,8 @@
         iorder[3] = 0;
     }
 
-    while (size-- > 0) {
-        Py_UCS4 ch = *s++;
-#ifndef Py_UNICODE_WIDE
-        if (0xD800 <= ch && ch <= 0xDBFF && size > 0) {
-            Py_UCS4 ch2 = *s;
-            if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
-                ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000;
-                s++;
-                size--;
-            }
-        }
-#endif
-        STORECHAR(ch);
-    }
+    for (i = 0; i < len; i++)
+        STORECHAR(PyUnicode_READ(kind, data, i));
 
   done:
     return v;
@@ -5118,12 +5130,23 @@
 }
 
 PyObject *
+PyUnicode_EncodeUTF32(const Py_UNICODE *s,
+                      Py_ssize_t size,
+                      const char *errors,
+                      int byteorder)
+{
+    PyObject *result;
+    PyObject *tmp = PyUnicode_FromUnicode(s, size);
+    if (tmp == NULL)
+        return NULL;
+    result = _PyUnicode_EncodeUTF32(tmp, errors, byteorder);
+    Py_DECREF(tmp);
+    return result;
+}
+
+PyObject *
 PyUnicode_AsUTF32String(PyObject *unicode)
 {
-    if (!PyUnicode_Check(unicode)) {
-        PyErr_BadArgument();
-        return NULL;
-    }
     return PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(unicode),
                                  PyUnicode_GET_SIZE(unicode),
                                  NULL,
@@ -5405,19 +5428,17 @@
 #undef SWAPPED_FAST_CHAR_MASK
 
 PyObject *
-PyUnicode_EncodeUTF16(const Py_UNICODE *s,
-                      Py_ssize_t size,
-                      const char *errors,
-                      int byteorder)
-{
+_PyUnicode_EncodeUTF16(PyObject *str,
+                       const char *errors,
+                       int byteorder)
+{
+    int kind;
+    void *data;
+    Py_ssize_t len;
     PyObject *v;
     unsigned char *p;
     Py_ssize_t nsize, bytesize;
-#ifdef Py_UNICODE_WIDE
     Py_ssize_t i, pairs;
-#else
-    const int pairs = 0;
-#endif
     /* Offsets from p for storing byte pairs in the right order. */
 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
     int ihi = 1, ilo = 0;
@@ -5432,16 +5453,25 @@
         p += 2;                                 \
     } while(0)
 
-#ifdef Py_UNICODE_WIDE
-    for (i = pairs = 0; i < size; i++)
-        if (s[i] >= 0x10000)
-            pairs++;
-#endif
-    /* 2 * (size + pairs + (byteorder == 0)) */
-    if (size > PY_SSIZE_T_MAX ||
-        size > PY_SSIZE_T_MAX - pairs - (byteorder == 0))
+    if (!PyUnicode_Check(str)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
+    if (PyUnicode_READY(str) < 0)
+        return NULL;
+    kind = PyUnicode_KIND(str);
+    data = PyUnicode_DATA(str);
+    len = PyUnicode_GET_LENGTH(str);
+    
+    pairs = 0;
+    if (kind == PyUnicode_4BYTE_KIND)
+        for (i = 0; i < len; i++)
+            if (PyUnicode_READ(kind, data, i) >= 0x10000)
+                pairs++;
+    /* 2 * (len + pairs + (byteorder == 0)) */
+    if (len > PY_SSIZE_T_MAX - pairs - (byteorder == 0))
         return PyErr_NoMemory();
-    nsize = size + pairs + (byteorder == 0);
+    nsize = len + pairs + (byteorder == 0);
     bytesize = nsize * 2;
     if (bytesize / 2 != nsize)
         return PyErr_NoMemory();
@@ -5452,7 +5482,7 @@
     p = (unsigned char *)PyBytes_AS_STRING(v);
     if (byteorder == 0)
         STORECHAR(0xFEFF);
-    if (size == 0)
+    if (len == 0)
         goto done;
 
     if (byteorder == -1) {
@@ -5466,15 +5496,13 @@
         ilo = 1;
     }
 
-    while (size-- > 0) {
-        Py_UNICODE ch = *s++;
-        Py_UNICODE ch2 = 0;
-#ifdef Py_UNICODE_WIDE
+    for (i = 0; i < len; i++) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        Py_UCS4 ch2 = 0;
         if (ch >= 0x10000) {
             ch2 = 0xDC00 | ((ch-0x10000) & 0x3FF);
             ch  = 0xD800 | ((ch-0x10000) >> 10);
         }
-#endif
         STORECHAR(ch);
         if (ch2)
             STORECHAR(ch2);
@@ -5486,16 +5514,24 @@
 }
 
 PyObject *
+PyUnicode_EncodeUTF16(const Py_UNICODE *s,
+                      Py_ssize_t size,
+                      const char *errors,
+                      int byteorder)
+{
+    PyObject *result;
+    PyObject *tmp = PyUnicode_FromUnicode(s, size);
+    if (tmp == NULL)
+        return NULL;
+    result = _PyUnicode_EncodeUTF16(tmp, errors, byteorder);
+    Py_DECREF(tmp);
+    return result;
+}
+
+PyObject *
 PyUnicode_AsUTF16String(PyObject *unicode)
 {
-    if (!PyUnicode_Check(unicode)) {
-        PyErr_BadArgument();
-        return NULL;
-    }
-    return PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(unicode),
-                                 PyUnicode_GET_SIZE(unicode),
-                                 NULL,
-                                 0);
+    return _PyUnicode_EncodeUTF16(unicode, NULL, 0);
 }
 
 /* --- Unicode Escape Codec ----------------------------------------------- */
@@ -5850,22 +5886,15 @@
 */
 
 PyObject *
-PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s,
-                              Py_ssize_t size)
-{
+PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
+{
+    Py_ssize_t i, len;
     PyObject *repr;
     char *p;
-
-#ifdef Py_UNICODE_WIDE
-    const Py_ssize_t expandsize = 10;
-#else
-    const Py_ssize_t expandsize = 6;
-#endif
-
-    /* XXX(nnorwitz): rather than over-allocating, it would be
-       better to choose a different scheme.  Perhaps scan the
-       first N-chars of the string and allocate based on that size.
-    */
+    int kind;
+    void *data;
+    Py_ssize_t expandsize = 0;
+
     /* Initial allocation is based on the longest-possible unichr
        escape.
 
@@ -5880,23 +5909,38 @@
        escape.
     */
 
-    if (size == 0)
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
+    if (PyUnicode_READY(unicode) < 0)
+        return NULL;
+    len = PyUnicode_GET_LENGTH(unicode);
+    kind = PyUnicode_KIND(unicode);
+    data = PyUnicode_DATA(unicode);
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND: expandsize = 4; break;
+    case PyUnicode_2BYTE_KIND: expandsize = 6; break;
+    case PyUnicode_4BYTE_KIND: expandsize = 10; break;
+    }
+
+    if (len == 0)
         return PyBytes_FromStringAndSize(NULL, 0);
 
-    if (size > (PY_SSIZE_T_MAX - 2 - 1) / expandsize)
+    if (len > (PY_SSIZE_T_MAX - 2 - 1) / expandsize)
         return PyErr_NoMemory();
 
     repr = PyBytes_FromStringAndSize(NULL,
                                      2
-                                     + expandsize*size
+                                     + expandsize*len
                                      + 1);
     if (repr == NULL)
         return NULL;
 
     p = PyBytes_AS_STRING(repr);
 
-    while (size-- > 0) {
-        Py_UNICODE ch = *s++;
+    for (i = 0; i < len; i++) {
+        Py_UNICODE ch = PyUnicode_READ(kind, data, i);
 
         /* Escape backslashes */
         if (ch == '\\') {
@@ -5905,7 +5949,6 @@
             continue;
         }
 
-#ifdef Py_UNICODE_WIDE
         /* Map 21-bit characters to '\U00xxxxxx' */
         else if (ch >= 0x10000) {
             *p++ = '\\';
@@ -5920,33 +5963,6 @@
             *p++ = Py_hexdigits[ch & 0x0000000F];
             continue;
         }
-#else
-        /* Map UTF-16 surrogate pairs to '\U00xxxxxx' */
-        else if (ch >= 0xD800 && ch < 0xDC00) {
-            Py_UNICODE ch2;
-            Py_UCS4 ucs;
-
-            ch2 = *s++;
-            size--;
-            if (ch2 >= 0xDC00 && ch2 <= 0xDFFF) {
-                ucs = (((ch & 0x03FF) << 10) | (ch2 & 0x03FF)) + 0x00010000;
-                *p++ = '\\';
-                *p++ = 'U';
-                *p++ = Py_hexdigits[(ucs >> 28) & 0x0000000F];
-                *p++ = Py_hexdigits[(ucs >> 24) & 0x0000000F];
-                *p++ = Py_hexdigits[(ucs >> 20) & 0x0000000F];
-                *p++ = Py_hexdigits[(ucs >> 16) & 0x0000000F];
-                *p++ = Py_hexdigits[(ucs >> 12) & 0x0000000F];
-                *p++ = Py_hexdigits[(ucs >> 8) & 0x0000000F];
-                *p++ = Py_hexdigits[(ucs >> 4) & 0x0000000F];
-                *p++ = Py_hexdigits[ucs & 0x0000000F];
-                continue;
-            }
-            /* Fall through: isolated surrogates are copied as-is */
-            s--;
-            size++;
-        }
-#endif
 
         /* Map 16-bit characters to '\uxxxx' */
         if (ch >= 256) {
@@ -5992,16 +6008,16 @@
 }
 
 PyObject *
-PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
-{
-    PyObject *s;
-    if (!PyUnicode_Check(unicode)) {
-        PyErr_BadArgument();
-        return NULL;
-    }
-    s = PyUnicode_EncodeUnicodeEscape(PyUnicode_AS_UNICODE(unicode),
-                                      PyUnicode_GET_SIZE(unicode));
-    return s;
+PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s,
+                              Py_ssize_t size)
+{
+    PyObject *result;
+    PyObject *tmp = PyUnicode_FromUnicode(s, size);
+    if (tmp == NULL)
+        return NULL;
+    result = PyUnicode_AsUnicodeEscapeString(tmp);
+    Py_DECREF(tmp);
+    return result;
 }
 
 /* --- Raw Unicode Escape Codec ------------------------------------------- */
@@ -6114,33 +6130,46 @@
     return NULL;
 }
 
-PyObject *
-PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s,
-                                 Py_ssize_t size)
+
+PyObject *
+PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
 {
     PyObject *repr;
     char *p;
     char *q;
-
-#ifdef Py_UNICODE_WIDE
-    const Py_ssize_t expandsize = 10;
-#else
-    const Py_ssize_t expandsize = 6;
-#endif
-
-    if (size > PY_SSIZE_T_MAX / expandsize)
+    Py_ssize_t expandsize, pos;
+    int kind;
+    void *data;
+    Py_ssize_t len;
+
+    if (!PyUnicode_Check(unicode)) {
+        PyErr_BadArgument();
+        return NULL;
+    }
+    if (PyUnicode_READY(unicode) < 0)
+        return NULL;
+    kind = PyUnicode_KIND(unicode);
+    data = PyUnicode_DATA(unicode);
+    len = PyUnicode_GET_LENGTH(unicode);
+
+    switch(kind) {
+    case PyUnicode_1BYTE_KIND: expandsize = 4; break;
+    case PyUnicode_2BYTE_KIND: expandsize = 6; break;
+    case PyUnicode_4BYTE_KIND: expandsize = 10; break;
+    }
+        
+    if (len > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();
 
-    repr = PyBytes_FromStringAndSize(NULL, expandsize * size);
+    repr = PyBytes_FromStringAndSize(NULL, expandsize * len);
     if (repr == NULL)
         return NULL;
-    if (size == 0)
+    if (len == 0)
         return repr;
 
     p = q = PyBytes_AS_STRING(repr);
-    while (size-- > 0) {
-        Py_UNICODE ch = *s++;
-#ifdef Py_UNICODE_WIDE
+    for (pos = 0; pos < len; pos++) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, pos);
         /* Map 32-bit characters to '\Uxxxxxxxx' */
         if (ch >= 0x10000) {
             *p++ = '\\';
@@ -6154,36 +6183,8 @@
             *p++ = Py_hexdigits[(ch >> 4) & 0xf];
             *p++ = Py_hexdigits[ch & 15];
         }
-        else
-#else
-            /* Map UTF-16 surrogate pairs to '\U00xxxxxx' */
-            if (ch >= 0xD800 && ch < 0xDC00) {
-                Py_UNICODE ch2;
-                Py_UCS4 ucs;
-
-                ch2 = *s++;
-                size--;
-                if (ch2 >= 0xDC00 && ch2 <= 0xDFFF) {
-                    ucs = (((ch & 0x03FF) << 10) | (ch2 & 0x03FF)) + 0x00010000;
-                    *p++ = '\\';
-                    *p++ = 'U';
-                    *p++ = Py_hexdigits[(ucs >> 28) & 0xf];
-                    *p++ = Py_hexdigits[(ucs >> 24) & 0xf];
-                    *p++ = Py_hexdigits[(ucs >> 20) & 0xf];
-                    *p++ = Py_hexdigits[(ucs >> 16) & 0xf];
-                    *p++ = Py_hexdigits[(ucs >> 12) & 0xf];
-                    *p++ = Py_hexdigits[(ucs >> 8) & 0xf];
-                    *p++ = Py_hexdigits[(ucs >> 4) & 0xf];
-                    *p++ = Py_hexdigits[ucs & 0xf];
-                    continue;
-                }
-                /* Fall through: isolated surrogates are copied as-is */
-                s--;
-                size++;
-            }
-#endif
         /* Map 16-bit characters to '\uxxxx' */
-        if (ch >= 256) {
+        else if (ch >= 256) {
             *p++ = '\\';
             *p++ = 'u';
             *p++ = Py_hexdigits[(ch >> 12) & 0xf];
@@ -6195,26 +6196,24 @@
         else
             *p++ = (char) ch;
     }
-    size = p - q;
-
-    assert(size > 0);
-    if (_PyBytes_Resize(&repr, size) < 0)
+
+    assert(p > q);
+    if (_PyBytes_Resize(&repr, p - q) < 0)
         return NULL;
     return repr;
 }
 
 PyObject *
-PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
-{
-    PyObject *s;
-    if (!PyUnicode_Check(unicode)) {
-        PyErr_BadArgument();
-        return NULL;
-    }
-    s = PyUnicode_EncodeRawUnicodeEscape(PyUnicode_AS_UNICODE(unicode),
-                                         PyUnicode_GET_SIZE(unicode));
-
-    return s;
+PyUnicode_EncodeRawUnicodeEscape(const Py_UNICODE *s,
+                                 Py_ssize_t size)
+{
+    PyObject *result;
+    PyObject *tmp = PyUnicode_FromUnicode(s, size);
+    if (tmp == NULL)
+        return NULL;
+    result = PyUnicode_AsRawUnicodeEscapeString(tmp);
+    Py_DECREF(tmp);
+    return result;
 }
 
 /* --- Unicode Internal Codec ------------------------------------------- */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 19:23:25 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 10 Nov 2011 19:23:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313303=3A_Fix_bytec?=
 =?utf8?q?ode_file_default_permission=2E?=
Message-ID: <E1ROZHZ-00023I-Mm@dinsdale.python.org>

http://hg.python.org/cpython/rev/a9f10c3eff69
changeset:   73477:a9f10c3eff69
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Nov 10 19:12:29 2011 +0100
summary:
  Issue #13303: Fix bytecode file default permission.

files:
  Lib/importlib/_bootstrap.py |   2 +-
  Lib/test/test_import.py     |  17 +++++++----------
  Python/import.c             |  13 ++++++-------
  3 files changed, 14 insertions(+), 18 deletions(-)


diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py
+++ b/Lib/importlib/_bootstrap.py
@@ -88,7 +88,7 @@
         # On POSIX-like platforms, renaming is atomic. id() is used to generate
         # a pseudo-random filename.
         path_tmp = '{}.{}'.format(path, id(path))
-        fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY)
+        fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, 0o666)
         try:
             with _io.FileIO(fd, 'wb') as file:
                 file.write(data)
diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -97,25 +97,22 @@
 
     @unittest.skipUnless(os.name == 'posix',
                          "test meaningful only on posix systems")
-    def test_execute_bit_not_copied(self):
-        # Issue 6070: under posix .pyc files got their execute bit set if
-        # the .py file had the execute bit set, but they aren't executable.
-        with temp_umask(0o022):
+    def test_creation_mode(self):
+        mask = 0o022
+        with temp_umask(mask):
             sys.path.insert(0, os.curdir)
             try:
                 fname = TESTFN + os.extsep + "py"
                 create_empty_file(fname)
-                os.chmod(fname, (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH |
-                                 stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH))
                 __import__(TESTFN)
                 fn = imp.cache_from_source(fname)
                 if not os.path.exists(fn):
                     self.fail("__import__ did not result in creation of "
                               "either a .pyc or .pyo file")
-                    s = os.stat(fn)
-                    self.assertEqual(
-                        stat.S_IMODE(s.st_mode),
-                        stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
+                s = os.stat(fn)
+                # Check that the umask is respected, and the executable bits
+                # aren't set.
+                self.assertEqual(stat.S_IMODE(s.st_mode), 0o666 & ~mask)
             finally:
                 del sys.path[0]
                 remove_files(TESTFN)
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1202,12 +1202,10 @@
                       S_IXUSR | S_IXGRP | S_IXOTH |
                       S_IWUSR | S_IWGRP | S_IWOTH);
     PyObject *dirbytes;
-#endif
-    int fd;
-#ifndef MS_WINDOWS
     PyObject *cpathbytes, *cpathbytes_tmp;
     Py_ssize_t cpathbytes_len;
 #endif
+    int fd;
     PyObject *dirname;
     Py_UCS4 *dirsep;
     int res, ok;
@@ -1275,7 +1273,7 @@
         return;
     }
     cpathbytes_len = PyBytes_GET_SIZE(cpathbytes);
-    cpathbytes_tmp = PyBytes_FromStringAndSize(NULL, cpathbytes_len + 6);
+    cpathbytes_tmp = PyBytes_FromStringAndSize(NULL, cpathbytes_len + 4);
     if (cpathbytes_tmp == NULL) {
         Py_DECREF(cpathbytes);
         PyErr_Clear();
@@ -1283,9 +1281,10 @@
     }
     memcpy(PyBytes_AS_STRING(cpathbytes_tmp), PyBytes_AS_STRING(cpathbytes),
            cpathbytes_len);
-    memcpy(PyBytes_AS_STRING(cpathbytes_tmp) + cpathbytes_len, "XXXXXX", 6);
-
-    fd = mkstemp(PyBytes_AS_STRING(cpathbytes_tmp));
+    memcpy(PyBytes_AS_STRING(cpathbytes_tmp) + cpathbytes_len, ".tmp", 4);
+
+    fd = open(PyBytes_AS_STRING(cpathbytes_tmp),
+              O_CREAT | O_EXCL | O_WRONLY, 0666);
     if (0 <= fd)
         fp = fdopen(fd, "wb");
     else

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 19:23:27 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 10 Nov 2011 19:23:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=237777=3A_socket=3A_?=
 =?utf8?q?Add_Reliable_Datagram_Sockets_=28PF=5FRDS=29_support=2E?=
Message-ID: <E1ROZHb-00023N-OC@dinsdale.python.org>

http://hg.python.org/cpython/rev/2293ca739223
changeset:   73478:2293ca739223
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Nov 10 19:21:37 2011 +0100
summary:
  Issue #7777: socket: Add Reliable Datagram Sockets (PF_RDS) support.

files:
  Doc/library/socket.rst  |   24 +++-
  Doc/whatsnew/3.3.rst    |    3 +
  Lib/test/test_socket.py |  159 ++++++++++++++++++++++++++++
  Misc/NEWS               |    2 +
  Modules/socketmodule.c  |   77 +++++++++++++
  5 files changed, 260 insertions(+), 5 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -236,6 +236,19 @@
    .. versionadded:: 3.3
 
 
+.. data:: AF_RDS
+          PF_RDS
+          SOL_RDS
+          RDS_*
+
+   Many constants of these forms, documented in the Linux documentation, are
+   also defined in the socket module.
+
+   Availability: Linux >= 2.6.30.
+
+   .. versionadded:: 3.3
+
+
 .. data:: SIO_*
           RCVALL_*
 
@@ -407,14 +420,15 @@
 
    Create a new socket using the given address family, socket type and protocol
    number.  The address family should be :const:`AF_INET` (the default),
-   :const:`AF_INET6`, :const:`AF_UNIX` or :const:`AF_CAN`. The socket type
-   should be :const:`SOCK_STREAM` (the default), :const:`SOCK_DGRAM`,
-   :const:`SOCK_RAW` or perhaps one of the other ``SOCK_`` constants. The
-   protocol number is usually zero and may be omitted in that case or
-   :const:`CAN_RAW` in case the address family is :const:`AF_CAN`.
+   :const:`AF_INET6`, :const:`AF_UNIX`, :const:`AF_CAN` or :const:`AF_RDS`. The
+   socket type should be :const:`SOCK_STREAM` (the default),
+   :const:`SOCK_DGRAM`, :const:`SOCK_RAW` or perhaps one of the other ``SOCK_``
+   constants. The protocol number is usually zero and may be omitted in that
+   case or :const:`CAN_RAW` in case the address family is :const:`AF_CAN`.
 
    .. versionchanged:: 3.3
       The AF_CAN family was added.
+      The AF_RDS family was added.
 
 
 .. function:: socketpair([family[, type[, proto]]])
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -495,6 +495,9 @@
 
   (Contributed by Matthias Fuchs, updated by Tiago Gon?alves in :issue:`10141`)
 
+* The :class:`~socket.socket` class now supports the PF_RDS protocol family
+  (http://en.wikipedia.org/wiki/Reliable_Datagram_Sockets and
+  http://oss.oracle.com/projects/rds/).
 
 ssl
 ---
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -47,8 +47,20 @@
         s.close()
     return True
 
+def _have_socket_rds():
+    """Check whether RDS sockets are supported on this host."""
+    try:
+        s = socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0)
+    except (AttributeError, OSError):
+        return False
+    else:
+        s.close()
+    return True
+
 HAVE_SOCKET_CAN = _have_socket_can()
 
+HAVE_SOCKET_RDS = _have_socket_rds()
+
 # Size in bytes of the int type
 SIZEOF_INT = array.array("i").itemsize
 
@@ -113,6 +125,23 @@
             self.skipTest('network interface `%s` does not exist' %
                            self.interface)
 
+
+class SocketRDSTest(unittest.TestCase):
+
+    """To be able to run this test, the `rds` kernel module must be loaded:
+    # modprobe rds
+    """
+    bufsize = 8192
+
+    def setUp(self):
+        self.serv = socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0)
+        self.addCleanup(self.serv.close)
+        try:
+            self.port = support.bind_port(self.serv)
+        except OSError:
+            self.skipTest('unable to bind RDS socket')
+
+
 class ThreadableTest:
     """Threadable Test class
 
@@ -271,6 +300,29 @@
         self.cli = None
         ThreadableTest.clientTearDown(self)
 
+class ThreadedRDSSocketTest(SocketRDSTest, ThreadableTest):
+
+    def __init__(self, methodName='runTest'):
+        SocketRDSTest.__init__(self, methodName=methodName)
+        ThreadableTest.__init__(self)
+        self.evt = threading.Event()
+
+    def clientSetUp(self):
+        self.cli = socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0)
+        try:
+            # RDS sockets must be bound explicitly to send or receive data
+            self.cli.bind((HOST, 0))
+            self.cli_addr = self.cli.getsockname()
+        except OSError:
+            # skipTest should not be called here, and will be called in the
+            # server instead
+            pass
+
+    def clientTearDown(self):
+        self.cli.close()
+        self.cli = None
+        ThreadableTest.clientTearDown(self)
+
 class SocketConnectedTest(ThreadedTCPSocketTest):
     """Socket tests for client-server connection.
 
@@ -1239,6 +1291,112 @@
         self.cli.send(self.cf2)
 
 
+ at unittest.skipUnless(HAVE_SOCKET_RDS, 'RDS sockets required for this test.')
+class BasicRDSTest(unittest.TestCase):
+
+    def testCrucialConstants(self):
+        socket.AF_RDS
+        socket.PF_RDS
+
+    def testCreateSocket(self):
+        with socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0) as s:
+            pass
+
+    def testSocketBufferSize(self):
+        bufsize = 16384
+        with socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0) as s:
+            s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, bufsize)
+            s.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, bufsize)
+
+
+ at unittest.skipUnless(HAVE_SOCKET_RDS, 'RDS sockets required for this test.')
+ at unittest.skipUnless(thread, 'Threading required for this test.')
+class RDSTest(ThreadedRDSSocketTest):
+
+    def __init__(self, methodName='runTest'):
+        ThreadedRDSSocketTest.__init__(self, methodName=methodName)
+
+    def testSendAndRecv(self):
+        data, addr = self.serv.recvfrom(self.bufsize)
+        self.assertEqual(self.data, data)
+        self.assertEqual(self.cli_addr, addr)
+
+    def _testSendAndRecv(self):
+        self.data = b'spam'
+        self.cli.sendto(self.data, 0, (HOST, self.port))
+
+    def testPeek(self):
+        data, addr = self.serv.recvfrom(self.bufsize, socket.MSG_PEEK)
+        self.assertEqual(self.data, data)
+        data, addr = self.serv.recvfrom(self.bufsize)
+        self.assertEqual(self.data, data)
+
+    def _testPeek(self):
+        self.data = b'spam'
+        self.cli.sendto(self.data, 0, (HOST, self.port))
+
+    @requireAttrs(socket.socket, 'recvmsg')
+    def testSendAndRecvMsg(self):
+        data, ancdata, msg_flags, addr = self.serv.recvmsg(self.bufsize)
+        self.assertEqual(self.data, data)
+
+    @requireAttrs(socket.socket, 'sendmsg')
+    def _testSendAndRecvMsg(self):
+        self.data = b'hello ' * 10
+        self.cli.sendmsg([self.data], (), 0, (HOST, self.port))
+
+    def testSendAndRecvMulti(self):
+        data, addr = self.serv.recvfrom(self.bufsize)
+        self.assertEqual(self.data1, data)
+
+        data, addr = self.serv.recvfrom(self.bufsize)
+        self.assertEqual(self.data2, data)
+
+    def _testSendAndRecvMulti(self):
+        self.data1 = b'bacon'
+        self.cli.sendto(self.data1, 0, (HOST, self.port))
+
+        self.data2 = b'egg'
+        self.cli.sendto(self.data2, 0, (HOST, self.port))
+
+    def testSelect(self):
+        r, w, x = select.select([self.serv], [], [], 3.0)
+        self.assertIn(self.serv, r)
+        data, addr = self.serv.recvfrom(self.bufsize)
+        self.assertEqual(self.data, data)
+
+    def _testSelect(self):
+        self.data = b'select'
+        self.cli.sendto(self.data, 0, (HOST, self.port))
+
+    def testCongestion(self):
+        # wait until the sender is done
+        self.evt.wait()
+
+    def _testCongestion(self):
+        # test the behavior in case of congestion
+        self.data = b'fill'
+        self.cli.setblocking(False)
+        try:
+            # try to lower the receiver's socket buffer size
+            self.cli.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
+        except OSError:
+            pass
+        with self.assertRaises(OSError) as cm:
+            try:
+                # fill the receiver's socket buffer
+                while True:
+                    self.cli.sendto(self.data, 0, (HOST, self.port))
+            finally:
+                # signal the receiver we're done
+                self.evt.set()
+        # sendto() should have failed with ENOBUFS
+        self.assertEqual(cm.exception.errno, errno.ENOBUFS)
+        # and we should have received a congestion notification through poll
+        r, w, x = select.select([self.serv], [], [], 3.0)
+        self.assertIn(self.serv, r)
+
+
 @unittest.skipUnless(thread, 'Threading required for this test.')
 class BasicTCPTest(SocketConnectedTest):
 
@@ -4362,6 +4520,7 @@
         tests.append(TIPCTest)
         tests.append(TIPCThreadableTest)
     tests.extend([BasicCANTest, CANTest])
+    tests.extend([BasicRDSTest, RDSTest])
     tests.extend([
         CmsgMacroTests,
         SendmsgUDPTest,
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1516,6 +1516,8 @@
 Extension Modules
 -----------------
 
+- Issue #7777: socket: Add Reliable Datagram Sockets (PF_RDS) support.
+
 - Issue #13159: FileIO and BZ2Compressor/BZ2Decompressor now use a linear-time
   buffer growth strategy instead of a quadratic-time one.
 
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -1328,6 +1328,11 @@
     }
 #endif
 
+#ifdef AF_RDS
+    case AF_RDS:
+        /* RDS sockets use sockaddr_in: fall-through */
+#endif
+
     case AF_INET:
     {
         struct sockaddr_in* addr;
@@ -1686,6 +1691,11 @@
        }
 #endif
 
+#ifdef AF_RDS
+    case AF_RDS:
+        /* RDS sockets use sockaddr_in: fall-through */
+#endif
+
     case AF_INET:
     {
         *len_ret = sizeof (struct sockaddr_in);
@@ -5614,6 +5624,14 @@
     PyModule_AddIntConstant(m, "PF_CAN", PF_CAN);
 #endif
 
+/* Reliable Datagram Sockets */
+#ifdef AF_RDS
+    PyModule_AddIntConstant(m, "AF_RDS", AF_RDS);
+#endif
+#ifdef PF_RDS
+    PyModule_AddIntConstant(m, "PF_RDS", PF_RDS);
+#endif
+
 #ifdef AF_PACKET
     PyModule_AddIntMacro(m, AF_PACKET);
 #endif
@@ -5909,6 +5927,27 @@
     PyModule_AddIntConstant(m, "CAN_RAW_LOOPBACK", CAN_RAW_LOOPBACK);
     PyModule_AddIntConstant(m, "CAN_RAW_RECV_OWN_MSGS", CAN_RAW_RECV_OWN_MSGS);
 #endif
+#ifdef SOL_RDS
+    PyModule_AddIntConstant(m, "SOL_RDS", SOL_RDS);
+#endif
+#ifdef RDS_CANCEL_SENT_TO
+    PyModule_AddIntConstant(m, "RDS_CANCEL_SENT_TO", RDS_CANCEL_SENT_TO);
+#endif
+#ifdef RDS_GET_MR
+    PyModule_AddIntConstant(m, "RDS_GET_MR", RDS_GET_MR);
+#endif
+#ifdef RDS_FREE_MR
+    PyModule_AddIntConstant(m, "RDS_FREE_MR", RDS_FREE_MR);
+#endif
+#ifdef RDS_RECVERR
+    PyModule_AddIntConstant(m, "RDS_RECVERR", RDS_RECVERR);
+#endif
+#ifdef RDS_CONG_MONITOR
+    PyModule_AddIntConstant(m, "RDS_CONG_MONITOR", RDS_CONG_MONITOR);
+#endif
+#ifdef RDS_GET_MR_FOR_DEST
+    PyModule_AddIntConstant(m, "RDS_GET_MR_FOR_DEST", RDS_GET_MR_FOR_DEST);
+#endif
 #ifdef  IPPROTO_IP
     PyModule_AddIntConstant(m, "IPPROTO_IP", IPPROTO_IP);
 #else
@@ -6261,6 +6300,44 @@
     PyModule_AddIntConstant(m, "IPX_TYPE", IPX_TYPE);
 #endif
 
+/* Reliable Datagram Sockets */
+#ifdef RDS_CMSG_RDMA_ARGS
+    PyModule_AddIntConstant(m, "RDS_CMSG_RDMA_ARGS", RDS_CMSG_RDMA_ARGS);
+#endif
+#ifdef RDS_CMSG_RDMA_DEST
+    PyModule_AddIntConstant(m, "RDS_CMSG_RDMA_DEST", RDS_CMSG_RDMA_DEST);
+#endif
+#ifdef RDS_CMSG_RDMA_MAP
+    PyModule_AddIntConstant(m, "RDS_CMSG_RDMA_MAP", RDS_CMSG_RDMA_MAP);
+#endif
+#ifdef RDS_CMSG_RDMA_STATUS
+    PyModule_AddIntConstant(m, "RDS_CMSG_RDMA_STATUS", RDS_CMSG_RDMA_STATUS);
+#endif
+#ifdef RDS_CMSG_RDMA_UPDATE
+    PyModule_AddIntConstant(m, "RDS_CMSG_RDMA_UPDATE", RDS_CMSG_RDMA_UPDATE);
+#endif
+#ifdef RDS_RDMA_READWRITE
+    PyModule_AddIntConstant(m, "RDS_RDMA_READWRITE", RDS_RDMA_READWRITE);
+#endif
+#ifdef RDS_RDMA_FENCE
+    PyModule_AddIntConstant(m, "RDS_RDMA_FENCE", RDS_RDMA_FENCE);
+#endif
+#ifdef RDS_RDMA_INVALIDATE
+    PyModule_AddIntConstant(m, "RDS_RDMA_INVALIDATE", RDS_RDMA_INVALIDATE);
+#endif
+#ifdef RDS_RDMA_USE_ONCE
+    PyModule_AddIntConstant(m, "RDS_RDMA_USE_ONCE", RDS_RDMA_USE_ONCE);
+#endif
+#ifdef RDS_RDMA_DONTWAIT
+    PyModule_AddIntConstant(m, "RDS_RDMA_DONTWAIT", RDS_RDMA_DONTWAIT);
+#endif
+#ifdef RDS_RDMA_NOTIFY_ME
+    PyModule_AddIntConstant(m, "RDS_RDMA_NOTIFY_ME", RDS_RDMA_NOTIFY_ME);
+#endif
+#ifdef RDS_RDMA_SILENT
+    PyModule_AddIntConstant(m, "RDS_RDMA_SILENT", RDS_RDMA_SILENT);
+#endif
+
     /* get{addr,name}info parameters */
 #ifdef EAI_ADDRFAMILY
     PyModule_AddIntConstant(m, "EAI_ADDRFAMILY", EAI_ADDRFAMILY);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:04:08 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:04:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_build_on_Windows?=
Message-ID: <E1ROZuy-0005pV-6z@dinsdale.python.org>

http://hg.python.org/cpython/rev/27f5757cca0b
changeset:   73479:27f5757cca0b
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:05:55 2011 +0100
summary:
  Fix build on Windows

files:
  Modules/_codecsmodule.c |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -38,6 +38,10 @@
 #define PY_SSIZE_T_CLEAN
 #include "Python.h"
 
+#ifdef MS_WINDOWS
+#include <windows.h>
+#endif
+
 /* --- Registry ----------------------------------------------------------- */
 
 PyDoc_STRVAR(register__doc__,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:11:05 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:11:05 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_UTF-7_encoder_on_Window?=
	=?utf8?q?s?=
Message-ID: <E1ROa1h-0006bW-In@dinsdale.python.org>

http://hg.python.org/cpython/rev/24282b901ad6
changeset:   73480:24282b901ad6
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:12:49 2011 +0100
summary:
  Fix UTF-7 encoder on Windows

files:
  Objects/unicodeobject.c |  10 ++++------
  1 files changed, 4 insertions(+), 6 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4061,7 +4061,7 @@
 
     start = out = PyBytes_AS_STRING(v);
     for (i = 0; i < len; ++i) {
-        Py_UNICODE ch = PyUnicode_READ(kind, data, i);
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
 
         if (inShift) {
             if (ENCODE_DIRECT(ch, !base64SetO, !base64WhiteSpace)) {
@@ -4099,7 +4099,6 @@
         }
         continue;
 encode_char:
-#ifdef Py_UNICODE_WIDE
         if (ch >= 0x10000) {
             /* code first surrogate */
             base64bits += 16;
@@ -4111,7 +4110,6 @@
             /* prepare second surrogate */
             ch =  0xDC00 | ((ch-0x10000) & 0x3FF);
         }
-#endif
         base64bits += 16;
         base64buffer = (base64buffer << 16) | ch;
         while (base64bits >= 6) {
@@ -4138,7 +4136,7 @@
     PyObject *tmp = PyUnicode_FromUnicode(s, size);
     if (tmp == NULL)
         return NULL;
-    result = _PyUnicode_EncodeUTF7(tmp, base64SetO, 
+    result = _PyUnicode_EncodeUTF7(tmp, base64SetO,
                                    base64WhiteSpace, errors);
     Py_DECREF(tmp);
     return result;
@@ -5462,7 +5460,7 @@
     kind = PyUnicode_KIND(str);
     data = PyUnicode_DATA(str);
     len = PyUnicode_GET_LENGTH(str);
-    
+
     pairs = 0;
     if (kind == PyUnicode_4BYTE_KIND)
         for (i = 0; i < len; i++)
@@ -6157,7 +6155,7 @@
     case PyUnicode_2BYTE_KIND: expandsize = 6; break;
     case PyUnicode_4BYTE_KIND: expandsize = 10; break;
     }
-        
+
     if (len > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:13:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:13:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_=22unicode=5Fescape=22_?=
	=?utf8?q?encoder?=
Message-ID: <E1ROa4B-0006oa-6B@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a9248c32b87
changeset:   73481:5a9248c32b87
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:15:25 2011 +0100
summary:
  Fix "unicode_escape" encoder

files:
  Objects/unicodeobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -5938,7 +5938,7 @@
     p = PyBytes_AS_STRING(repr);
 
     for (i = 0; i < len; i++) {
-        Py_UNICODE ch = PyUnicode_READ(kind, data, i);
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
 
         /* Escape backslashes */
         if (ch == '\\') {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:20:02 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:20:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_PyUnicode=5FAS=5FUNIC?=
	=?utf8?q?ODE_in_the_UTF-8_encoder?=
Message-ID: <E1ROaAM-0007PG-AT@dinsdale.python.org>

http://hg.python.org/cpython/rev/7474bbee98cd
changeset:   73482:7474bbee98cd
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:21:49 2011 +0100
summary:
  Avoid PyUnicode_AS_UNICODE in the UTF-8 encoder

files:
  Objects/unicodeobject.c |  15 +++++++++++----
  1 files changed, 11 insertions(+), 4 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4820,11 +4820,18 @@
                 for(k = repsize; k > 0; k--)
                     *p++ = *prep++;
             } else /* rep is unicode */ {
-                const Py_UNICODE *prep = PyUnicode_AS_UNICODE(rep);
-                Py_UNICODE c;
+                enum PyUnicode_Kind repkind;
+                void *repdata;
+
+                if (PyUnicode_READY(rep) < 0) {
+                    Py_DECREF(rep);
+                    goto error;
+                }
+                repkind = PyUnicode_KIND(rep);
+                repdata = PyUnicode_DATA(rep);
 
                 for(k=0; k<repsize; k++) {
-                    c = prep[k];
+                    Py_UCS4 c = PyUnicode_READ(repkind, repdata, k);
                     if (0x80 <= c) {
                         raise_encode_exception(&exc, "utf-8",
                                                unicode,
@@ -4832,7 +4839,7 @@
                                                "surrogates not allowed");
                         goto error;
                     }
-                    *p++ = (char)prep[k];
+                    *p++ = (char)c;
                 }
             }
             Py_DECREF(rep);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:30:12 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:30:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FDecodeCharmap?=
 =?utf8?q?=28=29_uses_the_new_Unicode_API?=
Message-ID: <E1ROaKC-0000DK-Kr@dinsdale.python.org>

http://hg.python.org/cpython/rev/8797854b9c38
changeset:   73483:8797854b9c38
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:30:22 2011 +0100
summary:
  PyUnicode_DecodeCharmap() uses the new Unicode API

files:
  Objects/unicodeobject.c |  26 ++++++++++++++++++--------
  1 files changed, 18 insertions(+), 8 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7558,8 +7558,6 @@
     Py_ssize_t extrachars = 0;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
-    Py_UNICODE *mapstring = NULL;
-    Py_ssize_t maplen = 0;
 
     /* Default to Latin-1 */
     if (mapping == NULL)
@@ -7573,16 +7571,27 @@
     outpos = 0;
     e = s + size;
     if (PyUnicode_CheckExact(mapping)) {
-        mapstring = PyUnicode_AS_UNICODE(mapping);
-        maplen = PyUnicode_GET_SIZE(mapping);
+        Py_ssize_t maplen;
+        enum PyUnicode_Kind kind;
+        void *data;
+        Py_UCS4 x;
+
+        if (PyUnicode_READY(mapping) < 0)
+            return NULL;
+
+        maplen = PyUnicode_GET_LENGTH(mapping);
+        data = PyUnicode_DATA(mapping);
+        kind = PyUnicode_KIND(mapping);
         while (s < e) {
             unsigned char ch = *s;
-            Py_UNICODE x = 0xfffe; /* illegal value */
 
             if (ch < maplen)
-                x = mapstring[ch];
-
-            if (x == 0xfffe) {
+                x = PyUnicode_READ(kind, data, ch);
+            else
+                x = 0xfffe; /* invalid value */
+
+            if (x == 0xfffe)
+            {
                 /* undefined mapping */
                 startinpos = s-starts;
                 endinpos = startinpos+1;
@@ -7595,6 +7604,7 @@
                 }
                 continue;
             }
+
             if (unicode_putchar(&v, &outpos, x) < 0)
                 goto onError;
             ++s;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:30:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:30:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Prefer_Py=5FUCS4_or_wchar?=
	=?utf8?q?=5Ft_over_Py=5FUNICODE?=
Message-ID: <E1ROaKD-0000DU-EE@dinsdale.python.org>

http://hg.python.org/cpython/rev/2bd055bb8f95
changeset:   73484:2bd055bb8f95
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:31:37 2011 +0100
summary:
  Prefer Py_UCS4 or wchar_t over Py_UNICODE

files:
  Objects/unicodeobject.c |  15 +++++++--------
  1 files changed, 7 insertions(+), 8 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3844,7 +3844,7 @@
     Py_ssize_t shiftOutStart;
     unsigned int base64bits = 0;
     unsigned long base64buffer = 0;
-    Py_UNICODE surrogate = 0;
+    Py_UCS4 surrogate = 0;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
 
@@ -3873,8 +3873,7 @@
                 s++;
                 if (base64bits >= 16) {
                     /* we have enough bits for a UTF-16 value */
-                    Py_UNICODE outCh = (Py_UNICODE)
-                                       (base64buffer >> (base64bits-16));
+                    Py_UCS4 outCh = (Py_UCS4)(base64buffer >> (base64bits-16));
                     base64bits -= 16;
                     base64buffer &= (1 << base64bits) - 1; /* clear high bits */
                     if (surrogate) {
@@ -5232,7 +5231,7 @@
        stream as-is (giving a ZWNBSP character). */
     if (bo == 0) {
         if (size >= 2) {
-            const Py_UNICODE bom = (q[ihi] << 8) | q[ilo];
+            const Py_UCS4 bom = (q[ihi] << 8) | q[ilo];
 #ifdef BYTEORDER_IS_LITTLE_ENDIAN
             if (bom == 0xFEFF) {
                 q += 2;
@@ -5273,7 +5272,7 @@
 
     aligned_end = (const unsigned char *) ((size_t) e & ~LONG_PTR_MASK);
     while (q < e) {
-        Py_UNICODE ch;
+        Py_UCS4 ch;
         /* First check for possible aligned read of a C 'long'. Unaligned
            reads are more expensive, better to defer to another iteration. */
         if (!((size_t) q & LONG_PTR_MASK)) {
@@ -5663,7 +5662,7 @@
 
     while (s < end) {
         unsigned char c;
-        Py_UNICODE x;
+        Py_UCS4 x;
         int digits;
 
         /* The only case in which i == ascii_length is a backslash
@@ -6863,7 +6862,7 @@
                         int insize)
 {
     const DWORD flags = decode_code_page_flags(code_page);
-    Py_UNICODE *out;
+    wchar_t *out;
     DWORD outsize;
 
     /* First get the size of the result */
@@ -7177,7 +7176,7 @@
     BOOL *pusedDefaultChar = &usedDefaultChar;
     int outsize;
     PyObject *exc = NULL;
-    Py_UNICODE *p;
+    wchar_t *p;
     Py_ssize_t size;
     const DWORD flags = encode_code_page_flags(code_page, NULL);
     char *out;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:34:14 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Thu, 10 Nov 2011 20:34:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_test=5Fsocket_when_buil?=
	=?utf8?q?t_whithout_threads=2E?=
Message-ID: <E1ROaO6-0000US-U4@dinsdale.python.org>

http://hg.python.org/cpython/rev/fa9bbfc0bcec
changeset:   73485:fa9bbfc0bcec
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Thu Nov 10 20:33:36 2011 +0100
summary:
  Fix test_socket when built whithout threads.

files:
  Lib/test/test_socket.py |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -305,7 +305,6 @@
     def __init__(self, methodName='runTest'):
         SocketRDSTest.__init__(self, methodName=methodName)
         ThreadableTest.__init__(self)
-        self.evt = threading.Event()
 
     def clientSetUp(self):
         self.cli = socket.socket(socket.PF_RDS, socket.SOCK_SEQPACKET, 0)
@@ -1316,6 +1315,10 @@
     def __init__(self, methodName='runTest'):
         ThreadedRDSSocketTest.__init__(self, methodName=methodName)
 
+    def setUp(self):
+        super().setUp()
+        self.evt = threading.Event()
+
     def testSendAndRecv(self):
         data, addr = self.serv.recvfrom(self.bufsize)
         self.assertEqual(self.data, data)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 20:55:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 10 Nov 2011 20:55:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_and_deprecated_the_unic?=
	=?utf8?q?ode=5Finternal_codec?=
Message-ID: <E1ROaip-0002hK-OO@dinsdale.python.org>

http://hg.python.org/cpython/rev/6e660d9fd509
changeset:   73486:6e660d9fd509
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 10 20:56:30 2011 +0100
summary:
  Fix and deprecated the unicode_internal codec

unicode_internal codec uses Py_UNICODE instead of the real internal
representation (PEP 393: Py_UCS1, Py_UCS2 or Py_UCS4) for backward
compatibility.

files:
  Doc/library/codecs.rst  |   2 +
  Doc/whatsnew/3.3.rst    |   2 +
  Modules/_codecsmodule.c |  20 ++++++++++++++---
  Objects/unicodeobject.c |  32 +++++++++++++++++++++-------
  4 files changed, 44 insertions(+), 12 deletions(-)


diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -1173,6 +1173,8 @@
 | unicode_internal   |         | Return the internal       |
 |                    |         | representation of the     |
 |                    |         | operand                   |
+|                    |         |                           |
+|                    |         | .. deprecated:: 3.3       |
 +--------------------+---------+---------------------------+
 
 The following codecs provide bytes-to-bytes mappings.
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -250,6 +250,8 @@
 
 (:issue:`12100`)
 
+The ``unicode_internal`` codec has been deprecated.
+
 crypt
 -----
 
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -675,18 +675,30 @@
     PyObject *obj;
     const char *errors = NULL;
     const char *data;
-    Py_ssize_t size;
+    Py_ssize_t len, size;
 
     if (!PyArg_ParseTuple(args, "O|z:unicode_internal_encode",
                           &obj, &errors))
         return NULL;
 
     if (PyUnicode_Check(obj)) {
+        Py_UNICODE *u;
+
         if (PyUnicode_READY(obj) < 0)
             return NULL;
-        data = PyUnicode_AS_DATA(obj);
-        size = PyUnicode_GET_DATA_SIZE(obj);
-        return codec_tuple(PyBytes_FromStringAndSize(data, size),
+
+        if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                         "unicode_internal codecs has been deprecated",
+                         1))
+            return NULL;
+
+        u = PyUnicode_AsUnicodeAndSize(obj, &len);
+        if (u == NULL)
+            return NULL;
+        if (len > PY_SSIZE_T_MAX / sizeof(Py_UNICODE))
+            return PyErr_NoMemory();
+        size = len * sizeof(Py_UNICODE);
+        return codec_tuple(PyBytes_FromStringAndSize((const char*)u, size),
                            PyUnicode_GET_LENGTH(obj));
     }
     else {
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6237,6 +6237,11 @@
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
 
+    if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                     "unicode_internal codecs has been deprecated",
+                     1))
+        return NULL;
+
     /* XXX overflow detection missing */
     v = PyUnicode_New((size+Py_UNICODE_SIZE-1)/ Py_UNICODE_SIZE, 127);
     if (v == NULL)
@@ -6270,15 +6275,26 @@
                     errors, &errorHandler,
                     "unicode_internal", reason,
                     &starts, &end, &startinpos, &endinpos, &exc, &s,
-                    &v, &outpos)) {
+                    &v, &outpos))
                 goto onError;
-            }
-        }
-        else {
-            if (unicode_putchar(&v, &outpos, ch) < 0)
-                goto onError;
-            s += Py_UNICODE_SIZE;
-        }
+            continue;
+        }
+
+        s += Py_UNICODE_SIZE;
+#ifndef Py_UNICODE_WIDE
+        if (ch >= 0xD800 && ch <= 0xDBFF && s < end)
+        {
+            Py_UCS4 ch2 = *(Py_UNICODE*)s;
+            if (ch2 >= 0xDC00 && ch2 <= 0xDFFF)
+            {
+                ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000;
+                s += Py_UNICODE_SIZE;
+            }
+        }
+#endif
+
+        if (unicode_putchar(&v, &outpos, ch) < 0)
+            goto onError;
     }
 
     if (PyUnicode_Resize(&v, outpos) < 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 21:56:43 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 10 Nov 2011 21:56:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313149=3A_Speed_up_?=
 =?utf8?q?append-only_StringIO_objects=2E?=
Message-ID: <E1RObfv-0008WP-7r@dinsdale.python.org>

http://hg.python.org/cpython/rev/8d9a869db675
changeset:   73487:8d9a869db675
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 10 21:47:38 2011 +0100
summary:
  Issue #13149: Speed up append-only StringIO objects.
This is very similar to the "lazy strings" idea.

files:
  Misc/NEWS               |    2 +
  Modules/_io/stringio.c  |  109 ++++++++++++++++++++++++++-
  Objects/unicodeobject.c |    2 +-
  3 files changed, 108 insertions(+), 5 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -365,6 +365,8 @@
 Library
 -------
 
+- Issue #13149: Speed up append-only StringIO objects.
+
 - Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
   when called with a timeout.  Patch by Arnaud Ysmal.
 
diff --git a/Modules/_io/stringio.c b/Modules/_io/stringio.c
--- a/Modules/_io/stringio.c
+++ b/Modules/_io/stringio.c
@@ -7,6 +7,9 @@
    than the enclosed string, for proper functioning of _PyIO_find_line_ending.
 */
 
+#define STATE_REALIZED 1
+#define STATE_ACCUMULATING 2
+
 typedef struct {
     PyObject_HEAD
     Py_UCS4 *buf;
@@ -14,6 +17,15 @@
     Py_ssize_t string_size;
     size_t buf_size;
 
+    /* The stringio object can be in two states: accumulating or realized.
+       In accumulating state, the internal buffer contains nothing and
+       the contents are given by the embedded _PyAccu structure.
+       In realized state, the internal buffer is meaningful and the
+       _PyAccu is destroyed.
+    */
+    int state;
+    _PyAccu accu;
+
     char ok; /* initialized? */
     char closed;
     char readuniversal;
@@ -40,6 +52,11 @@
         return NULL; \
     }
 
+#define ENSURE_REALIZED(self) \
+    if (realize(self) < 0) { \
+        return NULL; \
+    }
+
 PyDoc_STRVAR(stringio_doc,
     "Text I/O implementation using an in-memory buffer.\n"
     "\n"
@@ -102,6 +119,54 @@
     return -1;
 }
 
+static PyObject *
+make_intermediate(stringio *self)
+{
+    PyObject *intermediate = _PyAccu_Finish(&self->accu);
+    self->state = STATE_REALIZED;
+    if (intermediate == NULL)
+        return NULL;
+    if (_PyAccu_Init(&self->accu) ||
+        _PyAccu_Accumulate(&self->accu, intermediate)) {
+        Py_DECREF(intermediate);
+        return NULL;
+    }
+    self->state = STATE_ACCUMULATING;
+    return intermediate;
+}
+
+static int
+realize(stringio *self)
+{
+    Py_ssize_t len;
+    PyObject *intermediate;
+
+    if (self->state == STATE_REALIZED)
+        return 0;
+    assert(self->state == STATE_ACCUMULATING);
+    self->state = STATE_REALIZED;
+
+    intermediate = _PyAccu_Finish(&self->accu);
+    if (intermediate == NULL)
+        return -1;
+
+    /* Append the intermediate string to the internal buffer.
+       The length should be equal to the current cursor position.
+     */
+    len = PyUnicode_GET_LENGTH(intermediate);
+    if (resize_buffer(self, len) < 0) {
+        Py_DECREF(intermediate);
+        return -1;
+    }
+    if (!PyUnicode_AsUCS4(intermediate, self->buf, len, 0)) {
+        Py_DECREF(intermediate);
+        return -1;
+    }
+
+    Py_DECREF(intermediate);
+    return 0;
+}
+
 /* Internal routine for writing a whole PyUnicode object to the buffer of a
    StringIO object. Returns 0 on success, or -1 on error. */
 static Py_ssize_t
@@ -136,7 +201,6 @@
         return -1;
     }
     len = PyUnicode_GET_LENGTH(decoded);
-
     assert(len >= 0);
 
     /* This overflow check is not strictly necessary. However, it avoids us to
@@ -147,6 +211,17 @@
                         "new position too large");
         goto fail;
     }
+
+    if (self->state == STATE_ACCUMULATING) {
+        if (self->string_size == self->pos) {
+            if (_PyAccu_Accumulate(&self->accu, decoded))
+                goto fail;
+            goto success;
+        }
+        if (realize(self))
+            goto fail;
+    }
+
     if (self->pos + len > self->string_size) {
         if (resize_buffer(self, self->pos + len) < 0)
             goto fail;
@@ -174,6 +249,7 @@
                           0))
         goto fail;
 
+success:
     /* Set the new length of the internal string if it has changed. */
     self->pos += len;
     if (self->string_size < self->pos)
@@ -195,6 +271,8 @@
 {
     CHECK_INITIALIZED(self);
     CHECK_CLOSED(self);
+    if (self->state == STATE_ACCUMULATING)
+        return make_intermediate(self);
     return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, self->buf,
                                      self->string_size);
 }
@@ -251,6 +329,14 @@
             size = 0;
     }
 
+    /* Optimization for seek(0); read() */
+    if (self->state == STATE_ACCUMULATING && self->pos == 0 && size == n) {
+        PyObject *result = make_intermediate(self);
+        self->pos = self->string_size;
+        return result;
+    }
+
+    ENSURE_REALIZED(self);
     output = self->buf + self->pos;
     self->pos += size;
     return PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, output, size);
@@ -301,6 +387,7 @@
     if (!PyArg_ParseTuple(args, "|O:readline", &arg))
         return NULL;
     CHECK_CLOSED(self);
+    ENSURE_REALIZED(self);
 
     if (PyNumber_Check(arg)) {
         limit = PyNumber_AsSsize_t(arg, PyExc_OverflowError);
@@ -322,6 +409,7 @@
 
     CHECK_INITIALIZED(self);
     CHECK_CLOSED(self);
+    ENSURE_REALIZED(self);
 
     if (Py_TYPE(self) == &PyStringIO_Type) {
         /* Skip method call overhead for speed */
@@ -392,6 +480,7 @@
     }
 
     if (size < self->string_size) {
+        ENSURE_REALIZED(self);
         if (resize_buffer(self, size) < 0)
             return NULL;
         self->string_size = size;
@@ -492,6 +581,7 @@
     /* Free up some memory */
     if (resize_buffer(self, 0) < 0)
         return NULL;
+    _PyAccu_Destroy(&self->accu);
     Py_CLEAR(self->readnl);
     Py_CLEAR(self->writenl);
     Py_CLEAR(self->decoder);
@@ -521,6 +611,7 @@
         PyMem_Free(self->buf);
         self->buf = NULL;
     }
+    _PyAccu_Destroy(&self->accu);
     Py_CLEAR(self->readnl);
     Py_CLEAR(self->writenl);
     Py_CLEAR(self->decoder);
@@ -559,6 +650,7 @@
     PyObject *value = NULL;
     PyObject *newline_obj = NULL;
     char *newline = "\n";
+    Py_ssize_t value_len;
 
     if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OO:__init__", kwlist,
                                      &value, &newline_obj))
@@ -600,6 +692,7 @@
 
     self->ok = 0;
 
+    _PyAccu_Destroy(&self->accu);
     Py_CLEAR(self->readnl);
     Py_CLEAR(self->writenl);
     Py_CLEAR(self->decoder);
@@ -636,19 +729,27 @@
     /* Now everything is set up, resize buffer to size of initial value,
        and copy it */
     self->string_size = 0;
-    if (value && value != Py_None) {
-        Py_ssize_t len = PyUnicode_GetSize(value);
+    if (value && value != Py_None)
+        value_len = PyUnicode_GetSize(value);
+    else
+        value_len = 0;
+    if (value_len > 0) {
         /* This is a heuristic, for newline translation might change
            the string length. */
-        if (resize_buffer(self, len) < 0)
+        if (resize_buffer(self, 0) < 0)
             return -1;
+        self->state = STATE_REALIZED;
         self->pos = 0;
         if (write_str(self, value) < 0)
             return -1;
     }
     else {
+        /* Empty stringio object, we can start by accumulating */
         if (resize_buffer(self, 0) < 0)
             return -1;
+        if (_PyAccu_Init(&self->accu))
+            return -1;
+        self->state = STATE_ACCUMULATING;
     }
     self->pos = 0;
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -2055,7 +2055,7 @@
 PyUnicode_AsUCS4(PyObject *string, Py_UCS4 *target, Py_ssize_t targetsize,
                  int copy_null)
 {
-    if (target == NULL || targetsize < 1) {
+    if (target == NULL || targetsize < 0) {
         PyErr_BadInternalCall();
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 10 22:53:52 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 10 Nov 2011 22:53:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Speed_up_IDNA_for_the_commo?=
	=?utf8?q?n_case?=
Message-ID: <E1ROcZE-0005JF-W5@dinsdale.python.org>

http://hg.python.org/cpython/rev/6e1071ed4c66
changeset:   73488:6e1071ed4c66
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 10 22:49:20 2011 +0100
summary:
  Speed up IDNA for the common case

files:
  Lib/encodings/idna.py |  22 ++++++++++++++++++++++
  1 files changed, 22 insertions(+), 0 deletions(-)


diff --git a/Lib/encodings/idna.py b/Lib/encodings/idna.py
--- a/Lib/encodings/idna.py
+++ b/Lib/encodings/idna.py
@@ -153,6 +153,20 @@
         if not input:
             return b'', 0
 
+        try:
+            result = input.encode('ascii')
+        except UnicodeEncodeError:
+            pass
+        else:
+            # ASCII name: fast path
+            labels = result.split(b'.')
+            for label in labels[:-1]:
+                if not (0 < len(label) < 64):
+                    raise UnicodeError("label empty or too long")
+            if len(labels[-1]) >= 64:
+                raise UnicodeError("label too long")
+            return result, len(input)
+
         result = bytearray()
         labels = dots.split(input)
         if labels and not labels[-1]:
@@ -179,6 +193,14 @@
         if not isinstance(input, bytes):
             # XXX obviously wrong, see #3232
             input = bytes(input)
+
+        if ace_prefix not in input:
+            # Fast path
+            try:
+                return input.decode('ascii'), len(input)
+            except UnicodeDecodeError:
+                pass
+
         labels = input.split(b".")
 
         if labels and len(labels[-1]) == 0:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 03:09:10 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 11 Nov 2011 03:09:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_crashing_because_of_a?=
	=?utf8?q?n_unaligned_word_access?=
Message-ID: <E1ROgYI-0007c8-3F@dinsdale.python.org>

http://hg.python.org/cpython/rev/82c29c65435e
changeset:   73489:82c29c65435e
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 11 02:59:42 2011 +0100
summary:
  Avoid crashing because of an unaligned word access

files:
  Objects/unicodeobject.c |  10 +++++++++-
  1 files changed, 9 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6252,7 +6252,15 @@
     end = s + size;
 
     while (s < end) {
-        Py_UCS4 ch = *(Py_UNICODE*)s;
+        Py_UCS4 ch;
+        /* We copy the raw representation one byte at a time because the
+           pointer may be unaligned (see test_codeccallbacks). */
+        ((char *) &ch)[0] = s[0];
+        ((char *) &ch)[1] = s[1];
+#ifdef Py_UNICODE_WIDE
+        ((char *) &ch)[2] = s[2];
+        ((char *) &ch)[3] = s[3];
+#endif
         /* We have to sanity check the raw data, otherwise doom looms for
            some malformed UCS-4 data. */
         if (

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 03:09:10 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 11 Nov 2011 03:09:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Avoid_a_glibc_bug_in_test?=
	=?utf8?q?=5Ftime_=28issue_=2313309=29?=
Message-ID: <E1ROgYI-0007cC-S4@dinsdale.python.org>

http://hg.python.org/cpython/rev/05164831011e
changeset:   73490:05164831011e
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 11 03:04:35 2011 +0100
summary:
  Avoid a glibc bug in test_time (issue #13309)

files:
  Lib/test/test_time.py |  8 +++++---
  1 files changed, 5 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -5,6 +5,7 @@
 import sysconfig
 import sys
 import warnings
+import platform
 
 # Max year is only limited by the size of C int.
 SIZEOF_INT = sysconfig.get_config_var('SIZEOF_INT') or 4
@@ -313,13 +314,14 @@
         # It may not be possible to reliably make mktime return error
         # on all platfom.  This will make sure that no other exception
         # than OverflowError is raised for an extreme value.
+        if platform.libc_ver()[0] == 'glibc':
+            # Issue #13309: passing extreme values to mktime() or localtime()
+            # borks the glibc's internal timezone data.
+            return
         try:
             time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))
         except OverflowError:
             pass
-        msg = "Issue #13309: the '%Z' specifier reports erroneous timezone"
-        msg += " after time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))."
-        self.assertEqual(time.strftime('%Z', tt), tzname, msg)
 
 
 class TestLocale(unittest.TestCase):

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Nov 11 05:33:05 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 11 Nov 2011 05:33:05 +0100
Subject: [Python-checkins] Daily reference leaks (05164831011e): sum=462
Message-ID: <E1ROinZ-0005BP-BR@ap.vmr.nerim.net>

results for 05164831011e on branch "default"
--------------------------------------------

test_smtplib leaked [154, 154, 154] references, sum=462


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloggOSL98', '-x']

From python-checkins at python.org  Fri Nov 11 09:34:42 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 09:34:42 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzY1?=
 =?utf8?q?=3A_correct_an_error_in_the_documentation_of_str=2Eexpandtabs?=
Message-ID: <E1ROmZO-0001Qf-Qm@dinsdale.python.org>

http://hg.python.org/cpython/rev/84f803fdc0d2
changeset:   73491:84f803fdc0d2
branch:      3.2
parent:      73474:6ddda809ea8c
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 10:40:14 2011 +0200
summary:
  Issue #13365: correct an error in the documentation of str.expandtabs

files:
  Doc/library/stdtypes.rst |  2 +-
  Misc/ACKS                |  1 +
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1033,7 +1033,7 @@
 
 .. method:: str.expandtabs([tabsize])
 
-   Return a copy of the string where all tab characters are replaced by one or
+   Return a copy of the string where all tab characters are replaced by zero or
    more spaces, depending on the current column and the given tab size.  The
    column number is reset to zero after each newline occurring in the string.
    If *tabsize* is not given, a tab size of ``8`` characters is assumed.  This
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -285,6 +285,7 @@
 Niels Ferguson
 Sebastian Fernandez
 Florian Festi
+John Feuerstein
 Vincent Fiack
 Tomer Filiba
 Jeffrey Finkelstein

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 09:34:43 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 09:34:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313365=3A_correct_an_error_in_the_documentation_of_s?=
 =?utf8?q?tr=2Eexpandtabs=2E_Patch_by?=
Message-ID: <E1ROmZP-0001Qj-Je@dinsdale.python.org>

http://hg.python.org/cpython/rev/25191fe10da9
changeset:   73492:25191fe10da9
parent:      73490:05164831011e
parent:      73491:84f803fdc0d2
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 10:44:22 2011 +0200
summary:
  Issue #13365: correct an error in the documentation of str.expandtabs. Patch by John Feuerstein

files:
  Doc/library/stdtypes.rst |  2 +-
  Misc/ACKS                |  1 +
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -1037,7 +1037,7 @@
 
 .. method:: str.expandtabs([tabsize])
 
-   Return a copy of the string where all tab characters are replaced by one or
+   Return a copy of the string where all tab characters are replaced by zero or
    more spaces, depending on the current column and the given tab size.  The
    column number is reset to zero after each newline occurring in the string.
    If *tabsize* is not given, a tab size of ``8`` characters is assumed.  This
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -306,6 +306,7 @@
 Niels Ferguson
 Sebastian Fernandez
 Florian Festi
+John Feuerstein
 Vincent Fiack
 Tomer Filiba
 Jeffrey Finkelstein

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 09:48:54 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 09:48:54 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMTkx?=
	=?utf8?q?=3A_typo_in_argparse_docs?=
Message-ID: <E1ROmn8-0002VH-Rb@dinsdale.python.org>

http://hg.python.org/cpython/rev/61976390763f
changeset:   73493:61976390763f
branch:      3.2
parent:      73491:84f803fdc0d2
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 10:57:01 2011 +0200
summary:
  Issue #13191: typo in argparse docs

files:
  Doc/library/argparse.rst |  2 +-
  Misc/ACKS                |  1 +
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -1060,7 +1060,7 @@
 value as the "name" of each object.  By default, for positional argument
 actions, the dest_ value is used directly, and for optional argument actions,
 the dest_ value is uppercased.  So, a single positional argument with
-``dest='bar'`` will that argument will be referred to as ``bar``. A single
+``dest='bar'`` will be referred to as ``bar``. A single
 optional argument ``--foo`` that should be followed by a single command-line argument
 will be referred to as ``FOO``.  An example::
 
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -411,6 +411,7 @@
 Jan Hosang
 Ken Howard
 Brad Howes
+Mike Hoy
 Chih-Hao Huang
 Lawrence Hudson
 Michael Hudson

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 09:48:55 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 09:48:55 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313191=3A_typo_in_argparse_docs?=
Message-ID: <E1ROmn9-0002VL-KX@dinsdale.python.org>

http://hg.python.org/cpython/rev/edf944ab87c5
changeset:   73494:edf944ab87c5
parent:      73492:25191fe10da9
parent:      73493:61976390763f
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 10:58:36 2011 +0200
summary:
  Issue #13191: typo in argparse docs

files:
  Doc/library/argparse.rst |  2 +-
  Misc/ACKS                |  1 +
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -1079,7 +1079,7 @@
 value as the "name" of each object.  By default, for positional argument
 actions, the dest_ value is used directly, and for optional argument actions,
 the dest_ value is uppercased.  So, a single positional argument with
-``dest='bar'`` will that argument will be referred to as ``bar``. A single
+``dest='bar'`` will be referred to as ``bar``. A single
 optional argument ``--foo`` that should be followed by a single command-line argument
 will be referred to as ``FOO``.  An example::
 
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -440,6 +440,7 @@
 Jan Hosang
 Ken Howard
 Brad Howes
+Mike Hoy
 Chih-Hao Huang
 Christian Hudon
 Lawrence Hudson

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 13:36:37 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 11 Nov 2011 13:36:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_regression_on_2-byte_wc?=
 =?utf8?q?har=5Ft_systems_=28Windows=29?=
Message-ID: <E1ROqLV-0004i3-3h@dinsdale.python.org>

http://hg.python.org/cpython/rev/6f3f306fb8f3
changeset:   73495:6f3f306fb8f3
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 11 13:29:12 2011 +0100
summary:
  Fix regression on 2-byte wchar_t systems (Windows)

files:
  Objects/unicodeobject.c |  21 +++++++++++++--------
  1 files changed, 13 insertions(+), 8 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6252,15 +6252,18 @@
     end = s + size;
 
     while (s < end) {
+        Py_UNICODE uch;
         Py_UCS4 ch;
         /* We copy the raw representation one byte at a time because the
            pointer may be unaligned (see test_codeccallbacks). */
-        ((char *) &ch)[0] = s[0];
-        ((char *) &ch)[1] = s[1];
+        ((char *) &uch)[0] = s[0];
+        ((char *) &uch)[1] = s[1];
 #ifdef Py_UNICODE_WIDE
-        ((char *) &ch)[2] = s[2];
-        ((char *) &ch)[3] = s[3];
-#endif
+        ((char *) &uch)[2] = s[2];
+        ((char *) &uch)[3] = s[3];
+#endif
+        ch = uch;
+
         /* We have to sanity check the raw data, otherwise doom looms for
            some malformed UCS-4 data. */
         if (
@@ -6292,10 +6295,12 @@
 #ifndef Py_UNICODE_WIDE
         if (ch >= 0xD800 && ch <= 0xDBFF && s < end)
         {
-            Py_UCS4 ch2 = *(Py_UNICODE*)s;
-            if (ch2 >= 0xDC00 && ch2 <= 0xDFFF)
+            Py_UNICODE uch2;
+            ((char *) &uch2)[0] = s[0];
+            ((char *) &uch2)[1] = s[1];
+            if (uch2 >= 0xDC00 && uch2 <= 0xDFFF)
             {
-                ch = (((ch & 0x3FF)<<10) | (ch2 & 0x3FF)) + 0x10000;
+                ch = (((uch & 0x3FF)<<10) | (uch2 & 0x3FF)) + 0x10000;
                 s += Py_UNICODE_SIZE;
             }
         }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 13:36:37 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 11 Nov 2011 13:36:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Enable_commented_out_test?=
Message-ID: <E1ROqLV-0004iG-Rz@dinsdale.python.org>

http://hg.python.org/cpython/rev/0d24b85a3c43
changeset:   73496:0d24b85a3c43
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 11 13:31:59 2011 +0100
summary:
  Enable commented out test

files:
  Lib/test/test_unicode.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1403,7 +1403,7 @@
         # Roundtrip safety for non-BMP (just a few chars)
         u = '\U00010001\U00020002\U00030003\U00040004\U00050005'
         for encoding in ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be',
-                         #'raw_unicode_escape',
+                         'raw_unicode_escape',
                          'unicode_escape', 'unicode_internal'):
             self.assertEqual(str(u.encode(encoding),encoding), u)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 13:40:17 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 11 Nov 2011 13:40:17 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_test_more_inclusive?=
Message-ID: <E1ROqP3-00054V-R5@dinsdale.python.org>

http://hg.python.org/cpython/rev/bd5c4dda6ad0
changeset:   73497:bd5c4dda6ad0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 11 13:35:44 2011 +0100
summary:
  Make test more inclusive

files:
  Lib/test/test_unicode.py |  10 ++++------
  1 files changed, 4 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1407,12 +1407,10 @@
                          'unicode_escape', 'unicode_internal'):
             self.assertEqual(str(u.encode(encoding),encoding), u)
 
-        # UTF-8 must be roundtrip safe for all UCS-2 code points
-        # This excludes surrogates: in the full range, there would be
-        # a surrogate pair (\udbff\udc00), which gets converted back
-        # to a non-BMP character (\U0010fc00)
-        u = ''.join(map(chr, list(range(0,0xd800)) +
-                             list(range(0xe000,0x10000))))
+        # UTF-8 must be roundtrip safe for all code points
+        # (except surrogates, which are forbidden).
+        u = ''.join(map(chr, list(range(0, 0xd800)) +
+                             list(range(0xe000, 0x10ffff))))
         for encoding in ('utf-8',):
             self.assertEqual(str(u.encode(encoding),encoding), u)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 15:32:25 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 15:32:25 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMTkx?=
	=?utf8?q?=3A_typo_in_argparse_docs?=
Message-ID: <E1ROs9Z-0005eQ-76@dinsdale.python.org>

http://hg.python.org/cpython/rev/477f633aa09d
changeset:   73498:477f633aa09d
branch:      2.7
parent:      73471:e7b6dca28a2f
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 16:42:11 2011 +0200
summary:
  Issue #13191: typo in argparse docs

files:
  Doc/library/argparse.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -1056,7 +1056,7 @@
 value as the "name" of each object.  By default, for positional argument
 actions, the dest_ value is used directly, and for optional argument actions,
 the dest_ value is uppercased.  So, a single positional argument with
-``dest='bar'`` will that argument will be referred to as ``bar``. A single
+``dest='bar'`` will be referred to as ``bar``. A single
 optional argument ``--foo`` that should be followed by a single command-line argument
 will be referred to as ``FOO``.  An example::
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 15:42:30 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 15:42:30 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMTYx?=
 =?utf8?q?=3A_fix_doc_strings_of_=5F=5Fi*=5F=5F_operators?=
Message-ID: <E1ROsJK-0006XU-TG@dinsdale.python.org>

http://hg.python.org/cpython/rev/369487785e9f
changeset:   73499:369487785e9f
branch:      2.7
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 16:52:16 2011 +0200
summary:
  Issue #13161: fix doc strings of __i*__ operators

files:
  Objects/typeobject.c |  22 +++++++++++-----------
  1 files changed, 11 insertions(+), 11 deletions(-)


diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -5937,27 +5937,27 @@
     NBSLOT("__index__", nb_index, slot_nb_index, wrap_unaryfunc,
            "x[y:z] <==> x[y.__index__():z.__index__()]"),
     IBSLOT("__iadd__", nb_inplace_add, slot_nb_inplace_add,
-           wrap_binaryfunc, "+"),
+           wrap_binaryfunc, "+="),
     IBSLOT("__isub__", nb_inplace_subtract, slot_nb_inplace_subtract,
-           wrap_binaryfunc, "-"),
+           wrap_binaryfunc, "-="),
     IBSLOT("__imul__", nb_inplace_multiply, slot_nb_inplace_multiply,
-           wrap_binaryfunc, "*"),
+           wrap_binaryfunc, "*="),
     IBSLOT("__idiv__", nb_inplace_divide, slot_nb_inplace_divide,
-           wrap_binaryfunc, "/"),
+           wrap_binaryfunc, "/="),
     IBSLOT("__imod__", nb_inplace_remainder, slot_nb_inplace_remainder,
-           wrap_binaryfunc, "%"),
+           wrap_binaryfunc, "%="),
     IBSLOT("__ipow__", nb_inplace_power, slot_nb_inplace_power,
-           wrap_binaryfunc, "**"),
+           wrap_binaryfunc, "**="),
     IBSLOT("__ilshift__", nb_inplace_lshift, slot_nb_inplace_lshift,
-           wrap_binaryfunc, "<<"),
+           wrap_binaryfunc, "<<="),
     IBSLOT("__irshift__", nb_inplace_rshift, slot_nb_inplace_rshift,
-           wrap_binaryfunc, ">>"),
+           wrap_binaryfunc, ">>="),
     IBSLOT("__iand__", nb_inplace_and, slot_nb_inplace_and,
-           wrap_binaryfunc, "&"),
+           wrap_binaryfunc, "&="),
     IBSLOT("__ixor__", nb_inplace_xor, slot_nb_inplace_xor,
-           wrap_binaryfunc, "^"),
+           wrap_binaryfunc, "^="),
     IBSLOT("__ior__", nb_inplace_or, slot_nb_inplace_or,
-           wrap_binaryfunc, "|"),
+           wrap_binaryfunc, "|="),
     BINSLOT("__floordiv__", nb_floor_divide, slot_nb_floor_divide, "//"),
     RBINSLOT("__rfloordiv__", nb_floor_divide, slot_nb_floor_divide, "//"),
     BINSLOT("__truediv__", nb_true_divide, slot_nb_true_divide, "/"),

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 15:52:31 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 15:52:31 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMTYx?=
 =?utf8?q?=3A_fix_doc_strings_of_=5F=5Fi*=5F=5F_operators?=
Message-ID: <E1ROsT1-0007LO-OM@dinsdale.python.org>

http://hg.python.org/cpython/rev/9fbaa190f011
changeset:   73500:9fbaa190f011
branch:      3.2
parent:      73493:61976390763f
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 16:57:05 2011 +0200
summary:
  Issue #13161: fix doc strings of __i*__ operators

files:
  Objects/typeobject.c |  20 ++++++++++----------
  1 files changed, 10 insertions(+), 10 deletions(-)


diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -5556,25 +5556,25 @@
     NBSLOT("__index__", nb_index, slot_nb_index, wrap_unaryfunc,
            "x[y:z] <==> x[y.__index__():z.__index__()]"),
     IBSLOT("__iadd__", nb_inplace_add, slot_nb_inplace_add,
-           wrap_binaryfunc, "+"),
+           wrap_binaryfunc, "+="),
     IBSLOT("__isub__", nb_inplace_subtract, slot_nb_inplace_subtract,
-           wrap_binaryfunc, "-"),
+           wrap_binaryfunc, "-="),
     IBSLOT("__imul__", nb_inplace_multiply, slot_nb_inplace_multiply,
-           wrap_binaryfunc, "*"),
+           wrap_binaryfunc, "*="),
     IBSLOT("__imod__", nb_inplace_remainder, slot_nb_inplace_remainder,
-           wrap_binaryfunc, "%"),
+           wrap_binaryfunc, "%="),
     IBSLOT("__ipow__", nb_inplace_power, slot_nb_inplace_power,
-           wrap_binaryfunc, "**"),
+           wrap_binaryfunc, "**="),
     IBSLOT("__ilshift__", nb_inplace_lshift, slot_nb_inplace_lshift,
-           wrap_binaryfunc, "<<"),
+           wrap_binaryfunc, "<<="),
     IBSLOT("__irshift__", nb_inplace_rshift, slot_nb_inplace_rshift,
-           wrap_binaryfunc, ">>"),
+           wrap_binaryfunc, ">>="),
     IBSLOT("__iand__", nb_inplace_and, slot_nb_inplace_and,
-           wrap_binaryfunc, "&"),
+           wrap_binaryfunc, "&="),
     IBSLOT("__ixor__", nb_inplace_xor, slot_nb_inplace_xor,
-           wrap_binaryfunc, "^"),
+           wrap_binaryfunc, "^="),
     IBSLOT("__ior__", nb_inplace_or, slot_nb_inplace_or,
-           wrap_binaryfunc, "|"),
+           wrap_binaryfunc, "|="),
     BINSLOT("__floordiv__", nb_floor_divide, slot_nb_floor_divide, "//"),
     RBINSLOT("__rfloordiv__", nb_floor_divide, slot_nb_floor_divide, "//"),
     BINSLOT("__truediv__", nb_true_divide, slot_nb_true_divide, "/"),

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 15:52:32 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 15:52:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313161=3A_fix_doc_strings_of_=5F=5Fi*=5F=5F_operator?=
 =?utf8?q?s=2E_Closes_=2313161?=
Message-ID: <E1ROsT2-0007LS-Hn@dinsdale.python.org>

http://hg.python.org/cpython/rev/d58de3e9870a
changeset:   73501:d58de3e9870a
parent:      73497:bd5c4dda6ad0
parent:      73500:9fbaa190f011
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 17:02:16 2011 +0200
summary:
  Issue #13161: fix doc strings of __i*__ operators. Closes #13161

files:
  Objects/typeobject.c |  20 ++++++++++----------
  1 files changed, 10 insertions(+), 10 deletions(-)


diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -5717,25 +5717,25 @@
     NBSLOT("__index__", nb_index, slot_nb_index, wrap_unaryfunc,
            "x[y:z] <==> x[y.__index__():z.__index__()]"),
     IBSLOT("__iadd__", nb_inplace_add, slot_nb_inplace_add,
-           wrap_binaryfunc, "+"),
+           wrap_binaryfunc, "+="),
     IBSLOT("__isub__", nb_inplace_subtract, slot_nb_inplace_subtract,
-           wrap_binaryfunc, "-"),
+           wrap_binaryfunc, "-="),
     IBSLOT("__imul__", nb_inplace_multiply, slot_nb_inplace_multiply,
-           wrap_binaryfunc, "*"),
+           wrap_binaryfunc, "*="),
     IBSLOT("__imod__", nb_inplace_remainder, slot_nb_inplace_remainder,
-           wrap_binaryfunc, "%"),
+           wrap_binaryfunc, "%="),
     IBSLOT("__ipow__", nb_inplace_power, slot_nb_inplace_power,
-           wrap_binaryfunc, "**"),
+           wrap_binaryfunc, "**="),
     IBSLOT("__ilshift__", nb_inplace_lshift, slot_nb_inplace_lshift,
-           wrap_binaryfunc, "<<"),
+           wrap_binaryfunc, "<<="),
     IBSLOT("__irshift__", nb_inplace_rshift, slot_nb_inplace_rshift,
-           wrap_binaryfunc, ">>"),
+           wrap_binaryfunc, ">>="),
     IBSLOT("__iand__", nb_inplace_and, slot_nb_inplace_and,
-           wrap_binaryfunc, "&"),
+           wrap_binaryfunc, "&="),
     IBSLOT("__ixor__", nb_inplace_xor, slot_nb_inplace_xor,
-           wrap_binaryfunc, "^"),
+           wrap_binaryfunc, "^="),
     IBSLOT("__ior__", nb_inplace_or, slot_nb_inplace_or,
-           wrap_binaryfunc, "|"),
+           wrap_binaryfunc, "|="),
     BINSLOT("__floordiv__", nb_floor_divide, slot_nb_floor_divide, "//"),
     RBINSLOT("__rfloordiv__", nb_floor_divide, slot_nb_floor_divide, "//"),
     BINSLOT("__truediv__", nb_true_divide, slot_nb_true_divide, "/"),

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 16:01:09 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 11 Nov 2011 16:01:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_range_in_test=2E?=
Message-ID: <E1ROsbN-00086b-2q@dinsdale.python.org>

http://hg.python.org/cpython/rev/ed0944c5e2d5
changeset:   73502:ed0944c5e2d5
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 11 17:00:46 2011 +0200
summary:
  Fix range in test.

files:
  Lib/test/test_unicode.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1410,7 +1410,7 @@
         # UTF-8 must be roundtrip safe for all code points
         # (except surrogates, which are forbidden).
         u = ''.join(map(chr, list(range(0, 0xd800)) +
-                             list(range(0xe000, 0x10ffff))))
+                             list(range(0xe000, 0x110000))))
         for encoding in ('utf-8',):
             self.assertEqual(str(u.encode(encoding),encoding), u)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 16:41:26 2011
From: python-checkins at python.org (brian.curtin)
Date: Fri, 11 Nov 2011 16:41:26 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogRml4ICMxMzM4NC4g?=
 =?utf8?q?Remove_=5F=5Ffuture=5F=5F_import_in_3=2Ex_code=2E?=
Message-ID: <E1ROtEM-0003Rj-ST@dinsdale.python.org>

http://hg.python.org/cpython/rev/3fdc5a75d6e1
changeset:   73503:3fdc5a75d6e1
branch:      3.2
parent:      73500:9fbaa190f011
user:        Brian Curtin <brian at python.org>
date:        Fri Nov 11 09:37:25 2011 -0600
summary:
  Fix #13384. Remove __future__ import in 3.x code.

files:
  Lib/random.py |  1 -
  Misc/NEWS     |  2 ++
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/random.py b/Lib/random.py
--- a/Lib/random.py
+++ b/Lib/random.py
@@ -36,7 +36,6 @@
 
 """
 
-from __future__ import division
 from warnings import warn as _warn
 from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType
 from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,8 @@
 Library
 -------
 
+- Issue #13384: Remove unnecessary __future__ import in Lib/random.py
+
 - Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
   when called with a timeout.  Patch by Arnaud Ysmal.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 16:41:27 2011
From: python-checkins at python.org (brian.curtin)
Date: Fri, 11 Nov 2011 16:41:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1ROtEN-0003Rn-SF@dinsdale.python.org>

http://hg.python.org/cpython/rev/4ef224fbf34e
changeset:   73504:4ef224fbf34e
parent:      73502:ed0944c5e2d5
parent:      73503:3fdc5a75d6e1
user:        Brian Curtin <brian at python.org>
date:        Fri Nov 11 09:41:17 2011 -0600
summary:
  Merge 3.2

files:
  Lib/random.py |  1 -
  Misc/NEWS     |  2 ++
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/random.py b/Lib/random.py
--- a/Lib/random.py
+++ b/Lib/random.py
@@ -36,7 +36,6 @@
 
 """
 
-from __future__ import division
 from warnings import warn as _warn
 from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType
 from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -365,6 +365,8 @@
 Library
 -------
 
+- Issue #13384: Remove unnecessary __future__ import in Lib/random.py
+
 - Issue #13149: Speed up append-only StringIO objects.
 
 - Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:00:00 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:00:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_unittest=2EskipUnless_t?=
 =?utf8?q?o_skip_the_test_related_to_the_glibc_bug=2C_issue_=2313309=2E?=
Message-ID: <E1ROvOS-0008JM-Up@dinsdale.python.org>

http://hg.python.org/cpython/rev/bcd347cd6bf2
changeset:   73505:bcd347cd6bf2
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 18:59:30 2011 +0100
summary:
  Use unittest.skipUnless to skip the test related to the glibc bug, issue #13309.

files:
  Lib/test/test_time.py |  25 +++++++++++++------------
  1 files changed, 13 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py
--- a/Lib/test/test_time.py
+++ b/Lib/test/test_time.py
@@ -4,7 +4,6 @@
 import locale
 import sysconfig
 import sys
-import warnings
 import platform
 
 # Max year is only limited by the size of C int.
@@ -200,8 +199,8 @@
             else:
                 self.assertEqual(time.ctime(testval)[20:], str(year))
 
-    @unittest.skipIf(not hasattr(time, "tzset"),
-        "time module has no attribute tzset")
+    @unittest.skipUnless(hasattr(time, "tzset"),
+                         "time module has no attribute tzset")
     def test_tzset(self):
 
         from os import environ
@@ -298,8 +297,7 @@
         t1 = time.mktime(lt1)
         self.assertAlmostEqual(t1, t0, delta=0.2)
 
-    # XXX run last to work around issue #13309 on Gentoo
-    def test_zzz_mktime(self):
+    def test_mktime(self):
         # Issue #1726687
         for t in (-2, -1, 0, 1):
             try:
@@ -308,20 +306,23 @@
                 pass
             else:
                 self.assertEqual(time.mktime(tt), t)
+
+    # Issue #13309: passing extreme values to mktime() or localtime()
+    # borks the glibc's internal timezone data.
+    @unittest.skipUnless(platform.libc_ver()[0] != 'glibc',
+                         "disabled because of a bug in glibc. Issue #13309")
+    def test_mktime_error(self):
+        # It may not be possible to reliably make mktime return error
+        # on all platfom.  This will make sure that no other exception
+        # than OverflowError is raised for an extreme value.
         tt = time.gmtime(self.t)
         tzname = time.strftime('%Z', tt)
         self.assertNotEqual(tzname, 'LMT')
-        # It may not be possible to reliably make mktime return error
-        # on all platfom.  This will make sure that no other exception
-        # than OverflowError is raised for an extreme value.
-        if platform.libc_ver()[0] == 'glibc':
-            # Issue #13309: passing extreme values to mktime() or localtime()
-            # borks the glibc's internal timezone data.
-            return
         try:
             time.mktime((-1, 1, 1, 0, 0, 0, -1, -1, -1))
         except OverflowError:
             pass
+        self.assertEqual(time.strftime('%Z', tt), tzname)
 
 
 class TestLocale(unittest.TestCase):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:31:31 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:31:31 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Remove_duplicat?=
	=?utf8?q?e_ACKS=2E?=
Message-ID: <E1ROvsx-00030c-P0@dinsdale.python.org>

http://hg.python.org/cpython/rev/8a9482e5f5cc
changeset:   73506:8a9482e5f5cc
branch:      3.2
parent:      73503:3fdc5a75d6e1
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:21:22 2011 +0100
summary:
  Remove duplicate ACKS.

files:
  Misc/ACKS |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -871,7 +871,6 @@
 Nathan Sullivan
 Mark Summerfield
 Hisao Suzuki
-Andrew Svetlov
 Kalle Svensson
 Andrew Svetlov
 Paul Swartz
@@ -980,7 +979,6 @@
 Dik Winter
 Blake Winton
 Jean-Claude Wippler
-Frank Wierzbicki
 Lars Wirzenius
 John Wiseman
 Chris Withers

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:31:32 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:31:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=2E_Remove_duplicate_ACKS_and_reorder_slightly_to?=
 =?utf8?q?_prevent_future_dups=2E?=
Message-ID: <E1ROvsy-00030m-HE@dinsdale.python.org>

http://hg.python.org/cpython/rev/da2042c89d8a
changeset:   73507:da2042c89d8a
parent:      73505:bcd347cd6bf2
parent:      73506:8a9482e5f5cc
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:31:02 2011 +0100
summary:
  Merge 3.2. Remove duplicate ACKS and reorder slightly to prevent future dups.

files:
  Misc/ACKS |  79 ++++++++++++++++++++----------------------
  1 files changed, 38 insertions(+), 41 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -11,8 +11,8 @@
 PS: In the standard Python distribution, this file is encoded in UTF-8
 and the list is in rough alphabetical order by last names.
 
+Rajiv Abraham
 David Abrahams
-Rajiv Abraham
 Ron Adam
 Ali Afshar
 Jim Ahlstrom
@@ -40,11 +40,10 @@
 Chris AtLee
 Aymeric Augustin
 John Aycock
-Jan-Hein B?hrman
 Donovan Baarda
 Attila Babo
+Marcin Bachry
 Alfonso Baciero
-Marcin Bachry
 Dwayne Bailey
 Stig Bakken
 Greg Ball
@@ -64,14 +63,14 @@
 Pior Bastida
 Nick Bastin
 Jeff Bauer
-Mike Bayer
 Michael R Bax
 Anthony Baxter
+Mike Bayer
 Samuel L. Bayer
 Donald Beaudry
 David Beazley
+Neal Becker
 Robin Becker
-Neal Becker
 Torsten Becker
 Bill Bedford
 Stefan Behnel
@@ -104,8 +103,8 @@
 Paul Boddie
 Matthew Boedicker
 David Bolen
+Gawain Bolton
 Forest Bond
-Gawain Bolton
 Gregory Bond
 Matias Bordese
 Jurjen Bos
@@ -135,6 +134,7 @@
 Francisco Mart?n Brugu?
 Stan Bubrouski
 Erik de Bueger
+Jan-Hein B?hrman
 Dick Bulterman
 Bill Bumgarner
 Jimmy Burgett
@@ -150,6 +150,7 @@
 Tony Campbell
 Brett Cannon
 Mike Carlton
+Pierre Carrier
 Terry Carroll
 Lorenzo M. Catucci
 Donn Cave
@@ -160,10 +161,10 @@
 John Chandler
 Hye-Shik Chang
 Jeffrey Chang
+Godefroid Chapelle
+Brad Chapman
+Greg Chapman
 Mitch Chapman
-Greg Chapman
-Brad Chapman
-Godefroid Chapelle
 David Chaum
 Nicolas Chauvat
 Jerry Chen
@@ -190,8 +191,8 @@
 Robert Collins
 Paul Colomiets
 Christophe Combelles
+Geremy Condra
 Denver Coneybeare
-Geremy Condra
 Juan Jos? Conti
 Matt Conway
 David M. Cooke
@@ -221,10 +222,10 @@
 Lars Damerow
 Evan Dandrea
 Eric Daniel
-Pierre-Yves David
 Scott David Daniels
 Ben Darnell
 Jonathan Dasteel
+Pierre-Yves David
 Xavier de Gaye
 John DeGood
 Ned Deily
@@ -287,8 +288,8 @@
 Andr? Espaze
 Stefan Esser
 Nicolas Estibals
+Carey Evans
 Stephen D Evans
-Carey Evans
 Tim Everett
 Paul Everitt
 David Everly
@@ -299,7 +300,6 @@
 Clovis Fabricio
 Andreas Faerber
 Bill Fancher
-Andrew Francis
 Troy J. Farrell
 Mark Favas
 Boris Feld
@@ -319,6 +319,7 @@
 Amaury Forgeot d'Arc
 Doug Fort
 John Fouhy
+Andrew Francis
 Martin Franklin
 Robin Friedrich
 Ivan Frohne
@@ -401,6 +402,7 @@
 Malte Helmert
 Lance Finn Helsten
 Jonathan Hendry
+Michael Henry
 James Henstridge
 Kasun Herath
 Chris Herborth
@@ -416,7 +418,6 @@
 Jason Hildebrand
 Richie Hindle
 Konrad Hinsen
-Michael Henry
 David Hobley
 Tim Hochberg
 Joerg-Cyril Hoehle
@@ -486,36 +487,35 @@
 Richard Jones
 Irmen de Jong
 Lucas de Jonge
+Jens B. Jorgensen
 John Jorgensen
-Jens B. Jorgensen
 Sijin Joseph
 Andreas Jung
 Tattoo Mabonzo K.
 Bob Kahn
 Kurt B. Kaiser
 Tamito Kajiyama
+Jan Kaliszewski
 Peter van Kampen
 Rafe Kaplan
 Jacob Kaplan-Moss
-Jan Kaliszewski
-Arkady Koplyarov
 Lou Kates
 Hiroaki Kawai
 Sebastien Keim
 Ryan Kelly
 Dan Kenigsberg
+Randall Kern
 Robert Kern
-Randall Kern
 Magnus Kessler
 Lawrence Kesteloot
 Vivek Khera
-Akira Kitada
 Mads Kiilerich
 Taek Joo Kim
 W. Trevor King
 Paul Kippes
 Steve Kirsch
 Sebastian Kirsche
+Akira Kitada
 Ron Klatchko
 Reid Kleckner
 Bastian Kleineidam
@@ -530,6 +530,7 @@
 Damon Kohler
 Jacek Konieczny
 ???? ?????????
+Arkady Koplyarov
 Vlad Korolev
 Joseph Koshy
 Maksim Kozyarchuk
@@ -559,11 +560,11 @@
 Chris Lawrence
 Brian Leair
 Mathieu Leduc-Hamel
+Christopher Lee
+Inyeol Lee
 James Lee
 John J. Lee
-Inyeol Lee
 Thomas Lee
-Christopher Lee
 Tennessee Leeuwenburg
 Luc Lefebvre
 Vincent Legoll
@@ -634,8 +635,8 @@
 Michael McLay
 Mark Mc Mahon
 Gordon McMillan
+Andrew McNamara
 Caolan McNamara
-Andrew McNamara
 Craig McPheeters
 Lambert Meertens
 Bill van Melle
@@ -645,20 +646,21 @@
 Luke Mewburn
 Carl Meyer
 Mike Meyer
+Piotr Meyer
 Alexis M?taireau
 Steven Miale
 Trent Mick
 Stan Mihai
 Aristotelis Mikropoulos
+Chad Miller
 Damien Miller
-Chad Miller
 Jason V. Miller
 Jay T. Miller
 Roman Milner
 Julien Miotte
 Andrii V. Mishkovskyi
+Dom Mitchell
 Dustin J. Mitchell
-Dom Mitchell
 Zubin Mithra
 Doug Moen
 The Dragon De Monsyne
@@ -670,21 +672,20 @@
 Pablo Mouzo
 Mher Movsisyan
 Ruslan Mstoi
+Michael Mulich
+Sape Mullender
 Sjoerd Mullender
-Sape Mullender
 Michael Muller
 Neil Muller
-Michael Mulich
 Louis Munro
 R. David Murray
-Piotr Meyer
 John Nagle
 Takahiro Nakayama
 Travers Naran
 Charles-Fran?ois Natali
 Fredrik Nehr
+Tony Nelson
 Trent Nelson
-Tony Nelson
 Chad Netzer
 Max Neunh?ffer
 George Neville-Neil
@@ -772,8 +773,8 @@
 Donovan Preston
 Jyrki Pulliainen
 Steve Purcell
+Eduardo P?rez
 Fernando P?rez
-Eduardo P?rez
 Pierre Quentel
 Brian Quinlan
 Anders Qvist
@@ -800,8 +801,8 @@
 Steven Reiz
 Roeland Rengelink
 Antoine Reversat
+Francesco Ricciardi
 Tim Rice
-Francesco Ricciardi
 Jan Pieter Riegel
 Armin Rigo
 Nicholas Riley
@@ -811,8 +812,8 @@
 Davide Rizzo
 Anthony Roach
 Mark Roberts
+Andy Robinson
 Jim Robinson
-Andy Robinson
 Mark Roddy
 Kevin Rodgers
 Giampaolo Rodola
@@ -869,8 +870,8 @@
 Stefan Schwarzer
 Dietmar Schwertberger
 Federico Schwindt
+Barry Scott
 Steven Scott
-Barry Scott
 Nick Seidenman
 ?iga Seilnach
 Yury Selivanov
@@ -899,8 +900,8 @@
 George Sipe
 J. Sipprell
 Kragen Sitaker
+Christopher Smith
 Eric V. Smith
-Christopher Smith
 Gregory P. Smith
 Mark Smith
 Rafal Smotrzyk
@@ -936,7 +937,6 @@
 Nathan Sullivan
 Mark Summerfield
 Hisao Suzuki
-Andrew Svetlov
 Kalle Svensson
 Andrew Svetlov
 Paul Swartz
@@ -949,8 +949,8 @@
 William Tanksley
 Christian Tanzer
 Steven Taschuk
+Amy Taylor
 Monty Taylor
-Amy Taylor
 Anatoly Techtonik
 Mikhail Terekhov
 Richard M. Tew
@@ -967,8 +967,8 @@
 Jason Tishler
 Christian Tismer
 Frank J. Tobin
+Bennett Todd
 R Lindsay Todd
-Bennett Todd
 Eugene Toder
 Matias Torchinsky
 Sandro Tosi
@@ -1035,10 +1035,10 @@
 Gerry Wiener
 Frank Wierzbicki
 Bryce "Zooko" Wilcox-O'Hearn
+Gerald S. Williams
 Jason Williams
 John Williams
 Sue Williams
-Gerald S. Williams
 Frank Willison
 Greg V. Wilson
 Jody Winston
@@ -1046,7 +1046,6 @@
 Dik Winter
 Blake Winton
 Jean-Claude Wippler
-Frank Wierzbicki
 Lars Wirzenius
 John Wiseman
 Chris Withers
@@ -1057,8 +1056,8 @@
 Dan Wolfe
 Richard Wolff
 Adam Woodbeck
+Gordon Worley
 Darren Worrall
-Gordon Worley
 Thomas Wouters
 Heiko Wundram
 Doug Wyatt
@@ -1082,5 +1081,3 @@
 Kai Zhu
 Tarek Ziad?
 Peter ?strand
-Torsten Becker
-Pierre Carrier

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:39:54 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:39:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_few_typos?=
	=?utf8?q?=2E?=
Message-ID: <E1ROw14-0003g4-CN@dinsdale.python.org>

http://hg.python.org/cpython/rev/27fe09ff14b7
changeset:   73508:27fe09ff14b7
branch:      3.2
parent:      73506:8a9482e5f5cc
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:35:42 2011 +0100
summary:
  Fix few typos.

files:
  Lib/email/_parseaddr.py |  2 +-
  Lib/imaplib.py          |  2 +-
  Lib/test/test_ast.py    |  2 +-
  Modules/Setup.dist      |  2 +-
  4 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Lib/email/_parseaddr.py b/Lib/email/_parseaddr.py
--- a/Lib/email/_parseaddr.py
+++ b/Lib/email/_parseaddr.py
@@ -178,7 +178,7 @@
     front of you.
 
     Note: this class interface is deprecated and may be removed in the future.
-    Use rfc822.AddressList instead.
+    Use email.utils.AddressList instead.
     """
 
     def __init__(self, field):
diff --git a/Lib/imaplib.py b/Lib/imaplib.py
--- a/Lib/imaplib.py
+++ b/Lib/imaplib.py
@@ -1385,7 +1385,7 @@
     """Convert date_time to IMAP4 INTERNALDATE representation.
 
     Return string in form: '"DD-Mmm-YYYY HH:MM:SS +HHMM"'.  The
-    date_time argument can be a number (int or float) represening
+    date_time argument can be a number (int or float) representing
     seconds since epoch (as returned by time.time()), a 9-tuple
     representing local time (as returned by time.localtime()), or a
     double-quoted string.  In the last case, it is assumed to already
diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -290,7 +290,7 @@
         self.assertEqual(x.body, body)
 
     def test_nodeclasses(self):
-        # Zero arguments constructor explicitely allowed
+        # Zero arguments constructor explicitly allowed
         x = ast.BinOp()
         self.assertEqual(x._fields, ('left', 'op', 'right'))
 
diff --git a/Modules/Setup.dist b/Modules/Setup.dist
--- a/Modules/Setup.dist
+++ b/Modules/Setup.dist
@@ -291,7 +291,7 @@
 #syslog syslogmodule.c		# syslog daemon interface
 
 
-# Curses support, requring the System V version of curses, often
+# Curses support, requiring the System V version of curses, often
 # provided by the ncurses library.  e.g. on Linux, link with -lncurses
 # instead of -lcurses).
 #

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:39:55 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:39:55 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=2C_fix_typos=2E?=
Message-ID: <E1ROw15-0003gE-5m@dinsdale.python.org>

http://hg.python.org/cpython/rev/74fa415dc715
changeset:   73509:74fa415dc715
parent:      73507:da2042c89d8a
parent:      73508:27fe09ff14b7
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:39:25 2011 +0100
summary:
  Merge 3.2, fix typos.

files:
  Lib/email/_parseaddr.py      |  2 +-
  Lib/imaplib.py               |  2 +-
  Lib/packaging/pypi/dist.py   |  2 +-
  Lib/packaging/pypi/simple.py |  2 +-
  Lib/test/test_ast.py         |  2 +-
  Lib/test/test_logging.py     |  2 +-
  Lib/test/test_posix.py       |  2 +-
  Modules/Setup.dist           |  2 +-
  8 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Lib/email/_parseaddr.py b/Lib/email/_parseaddr.py
--- a/Lib/email/_parseaddr.py
+++ b/Lib/email/_parseaddr.py
@@ -203,7 +203,7 @@
     front of you.
 
     Note: this class interface is deprecated and may be removed in the future.
-    Use rfc822.AddressList instead.
+    Use email.utils.AddressList instead.
     """
 
     def __init__(self, field):
diff --git a/Lib/imaplib.py b/Lib/imaplib.py
--- a/Lib/imaplib.py
+++ b/Lib/imaplib.py
@@ -1392,7 +1392,7 @@
     """Convert date_time to IMAP4 INTERNALDATE representation.
 
     Return string in form: '"DD-Mmm-YYYY HH:MM:SS +HHMM"'.  The
-    date_time argument can be a number (int or float) represening
+    date_time argument can be a number (int or float) representing
     seconds since epoch (as returned by time.time()), a 9-tuple
     representing local time (as returned by time.localtime()), or a
     double-quoted string.  In the last case, it is assumed to already
diff --git a/Lib/packaging/pypi/dist.py b/Lib/packaging/pypi/dist.py
--- a/Lib/packaging/pypi/dist.py
+++ b/Lib/packaging/pypi/dist.py
@@ -427,7 +427,7 @@
         """Sort the results with the given properties.
 
         The `prefer_final` argument can be used to specify if final
-        distributions (eg. not dev, bet or alpha) would be prefered or not.
+        distributions (eg. not dev, beta or alpha) would be preferred or not.
 
         Results can be inverted by using `reverse`.
 
diff --git a/Lib/packaging/pypi/simple.py b/Lib/packaging/pypi/simple.py
--- a/Lib/packaging/pypi/simple.py
+++ b/Lib/packaging/pypi/simple.py
@@ -269,7 +269,7 @@
     def _register_release(self, release=None, release_info={}):
         """Register a new release.
 
-        Both a release or a dict of release_info can be provided, the prefered
+        Both a release or a dict of release_info can be provided, the preferred
         way (eg. the quicker) is the dict one.
 
         Return the list of existing releases for the given project.
diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -295,7 +295,7 @@
         self.assertEqual(x.body, body)
 
     def test_nodeclasses(self):
-        # Zero arguments constructor explicitely allowed
+        # Zero arguments constructor explicitly allowed
         x = ast.BinOp()
         self.assertEqual(x._fields, ('left', 'op', 'right'))
 
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -3222,7 +3222,7 @@
         # style defaults to %
         self.assertIsInstance(formatter._style, logging.PercentStyle)
 
-        # level is not explicitely set
+        # level is not explicitly set
         self.assertEqual(logging.root.level, self.original_logging_level)
 
     def test_filename(self):
diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -190,7 +190,7 @@
             os.write(fd, b'test')
             os.lseek(fd, 0, os.SEEK_SET)
             self.assertEqual(b'es', posix.pread(fd, 2, 1))
-            # the first pread() shoudn't disturb the file offset
+            # the first pread() shouldn't disturb the file offset
             self.assertEqual(b'te', posix.read(fd, 2))
         finally:
             os.close(fd)
diff --git a/Modules/Setup.dist b/Modules/Setup.dist
--- a/Modules/Setup.dist
+++ b/Modules/Setup.dist
@@ -294,7 +294,7 @@
 #syslog syslogmodule.c		# syslog daemon interface
 
 
-# Curses support, requring the System V version of curses, often
+# Curses support, requiring the System V version of curses, often
 # provided by the ncurses library.  e.g. on Linux, link with -lncurses
 # instead of -lcurses).
 #

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:59:04 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:59:04 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Few_typos_in_th?=
	=?utf8?q?e_documentation=2E?=
Message-ID: <E1ROwJc-0005UE-Re@dinsdale.python.org>

http://hg.python.org/cpython/rev/fbb76e68467c
changeset:   73510:fbb76e68467c
branch:      3.2
parent:      73508:27fe09ff14b7
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:55:21 2011 +0100
summary:
  Few typos in the documentation.

files:
  Doc/library/builtins.rst |  2 +-
  Doc/library/imaplib.rst  |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/builtins.rst b/Doc/library/builtins.rst
--- a/Doc/library/builtins.rst
+++ b/Doc/library/builtins.rst
@@ -36,6 +36,6 @@
 
 As an implementation detail, most modules have the name ``__builtins__`` made
 available as part of their globals.  The value of ``__builtins__`` is normally
-either this module or the value of this modules's :attr:`__dict__` attribute.
+either this module or the value of this module's :attr:`__dict__` attribute.
 Since this is an implementation detail, it may not be used by alternate
 implementations of Python.
diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst
--- a/Doc/library/imaplib.rst
+++ b/Doc/library/imaplib.rst
@@ -109,7 +109,7 @@
    Convert *date_time* to an IMAP4 ``INTERNALDATE`` representation.  The
    return value is a string in the form: ``"DD-Mmm-YYYY HH:MM:SS
    +HHMM"`` (including double-quotes).  The *date_time* argument can be a
-   number (int or float) represening seconds since epoch (as returned
+   number (int or float) representing seconds since epoch (as returned
    by :func:`time.time`), a 9-tuple representing local time (as returned by
    :func:`time.localtime`), or a double-quoted string.  In the last case, it
    is assumed to already be in the correct format.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:59:05 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:59:05 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Reformat_paragr?=
	=?utf8?q?aphs=2E?=
Message-ID: <E1ROwJd-0005UO-Js@dinsdale.python.org>

http://hg.python.org/cpython/rev/f2b7db0c578b
changeset:   73511:f2b7db0c578b
branch:      3.2
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:56:26 2011 +0100
summary:
  Reformat paragraphs.

files:
  Doc/library/socketserver.rst |  25 ++++++++++++-----------
  1 files changed, 13 insertions(+), 12 deletions(-)


diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst
--- a/Doc/library/socketserver.rst
+++ b/Doc/library/socketserver.rst
@@ -39,11 +39,12 @@
 
 When inheriting from :class:`ThreadingMixIn` for threaded connection behavior,
 you should explicitly declare how you want your threads to behave on an abrupt
-shutdown. The :class:`ThreadingMixIn` class defines an attribute
+shutdown.  The :class:`ThreadingMixIn` class defines an attribute
 *daemon_threads*, which indicates whether or not the server should wait for
-thread termination. You should set the flag explicitly if you would like threads
-to behave autonomously; the default is :const:`False`, meaning that Python will
-not exit until all threads created by :class:`ThreadingMixIn` have exited.
+thread termination.  You should set the flag explicitly if you would like
+threads to behave autonomously; the default is :const:`False`, meaning that
+Python will not exit until all threads created by :class:`ThreadingMixIn` have
+exited.
 
 Server classes have the same external methods and attributes, no matter what
 network protocol they use.
@@ -115,8 +116,8 @@
 finished requests and to use :func:`select` to decide which request to work on
 next (or whether to handle a new incoming request).  This is particularly
 important for stream services where each client can potentially be connected for
-a long time (if threads or subprocesses cannot be used). See :mod:`asyncore` for
-another way to manage this.
+a long time (if threads or subprocesses cannot be used).  See :mod:`asyncore`
+for another way to manage this.
 
 .. XXX should data and methods be intermingled, or separate?
    how should the distinction between class and instance variables be drawn?
@@ -192,7 +193,7 @@
 
 .. attribute:: BaseServer.allow_reuse_address
 
-   Whether the server will allow the reuse of an address. This defaults to
+   Whether the server will allow the reuse of an address.  This defaults to
    :const:`False`, and can be set in subclasses to change the policy.
 
 
@@ -269,7 +270,7 @@
 .. method:: BaseServer.server_activate()
 
    Called by the server's constructor to activate the server.  The default behavior
-   just :meth:`listen`\ s to the server's socket. May be overridden.
+   just :meth:`listen`\ s to the server's socket.  May be overridden.
 
 
 .. method:: BaseServer.server_bind()
@@ -280,10 +281,10 @@
 
 .. method:: BaseServer.verify_request(request, client_address)
 
-   Must return a Boolean value; if the value is :const:`True`, the request will be
-   processed, and if it's :const:`False`, the request will be denied. This function
-   can be overridden to implement access controls for a server. The default
-   implementation always returns :const:`True`.
+   Must return a Boolean value; if the value is :const:`True`, the request will
+   be processed, and if it's :const:`False`, the request will be denied.  This
+   function can be overridden to implement access controls for a server. The
+   default implementation always returns :const:`True`.
 
 
 RequestHandler Objects

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 19:59:06 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 19:59:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_3=2E2=2C_doc_typos=2E?=
Message-ID: <E1ROwJe-0005US-D8@dinsdale.python.org>

http://hg.python.org/cpython/rev/3e1fe0477538
changeset:   73512:3e1fe0477538
parent:      73509:74fa415dc715
parent:      73511:f2b7db0c578b
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 19:58:53 2011 +0100
summary:
  Merge 3.2, doc typos.

files:
  Doc/glossary.rst                      |   2 +-
  Doc/library/builtins.rst              |   2 +-
  Doc/library/imaplib.rst               |   2 +-
  Doc/library/packaging.pypi.simple.rst |   2 +-
  Doc/library/packaging.pypi.xmlrpc.rst |   2 +-
  Doc/library/socketserver.rst          |  29 +++++++-------
  6 files changed, 20 insertions(+), 19 deletions(-)


diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -587,7 +587,7 @@
       as :keyword:`if`, :keyword:`while` or :keyword:`for`.
 
    struct sequence
-      A tuple with named elements. Struct sequences expose an interface similiar
+      A tuple with named elements. Struct sequences expose an interface similar
       to :term:`named tuple` in that elements can either be accessed either by
       index or as an attribute. However, they do not have any of the named tuple
       methods like :meth:`~collections.somenamedtuple._make` or
diff --git a/Doc/library/builtins.rst b/Doc/library/builtins.rst
--- a/Doc/library/builtins.rst
+++ b/Doc/library/builtins.rst
@@ -36,6 +36,6 @@
 
 As an implementation detail, most modules have the name ``__builtins__`` made
 available as part of their globals.  The value of ``__builtins__`` is normally
-either this module or the value of this modules's :attr:`__dict__` attribute.
+either this module or the value of this module's :attr:`__dict__` attribute.
 Since this is an implementation detail, it may not be used by alternate
 implementations of Python.
diff --git a/Doc/library/imaplib.rst b/Doc/library/imaplib.rst
--- a/Doc/library/imaplib.rst
+++ b/Doc/library/imaplib.rst
@@ -116,7 +116,7 @@
    Convert *date_time* to an IMAP4 ``INTERNALDATE`` representation.  The
    return value is a string in the form: ``"DD-Mmm-YYYY HH:MM:SS
    +HHMM"`` (including double-quotes).  The *date_time* argument can be a
-   number (int or float) represening seconds since epoch (as returned
+   number (int or float) representing seconds since epoch (as returned
    by :func:`time.time`), a 9-tuple representing local time (as returned by
    :func:`time.localtime`), or a double-quoted string.  In the last case, it
    is assumed to already be in the correct format.
diff --git a/Doc/library/packaging.pypi.simple.rst b/Doc/library/packaging.pypi.simple.rst
--- a/Doc/library/packaging.pypi.simple.rst
+++ b/Doc/library/packaging.pypi.simple.rst
@@ -30,7 +30,7 @@
 
       The first two parameters control the query results.  *prefer_final*
       indicates whether a final version (not alpha, beta or candidate) is to be
-      prefered over a newer but non-final version (for example, whether to pick
+      preferred over a newer but non-final version (for example, whether to pick
       up 1.0 over 2.0a3).  It is used only for queries that don't give a version
       argument.  Likewise, *prefer_source* tells whether to prefer a source
       distribution over a binary one, if no distribution argument was prodived.
diff --git a/Doc/library/packaging.pypi.xmlrpc.rst b/Doc/library/packaging.pypi.xmlrpc.rst
--- a/Doc/library/packaging.pypi.xmlrpc.rst
+++ b/Doc/library/packaging.pypi.xmlrpc.rst
@@ -94,7 +94,7 @@
 Getting metadata information
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-XML-RPC is a prefered way to retrieve metadata information from indexes.
+XML-RPC is a preferred way to retrieve metadata information from indexes.
 It's really simple to do so::
 
    >>> client = xmlrpc.Client()
diff --git a/Doc/library/socketserver.rst b/Doc/library/socketserver.rst
--- a/Doc/library/socketserver.rst
+++ b/Doc/library/socketserver.rst
@@ -39,11 +39,12 @@
 
 When inheriting from :class:`ThreadingMixIn` for threaded connection behavior,
 you should explicitly declare how you want your threads to behave on an abrupt
-shutdown. The :class:`ThreadingMixIn` class defines an attribute
+shutdown.  The :class:`ThreadingMixIn` class defines an attribute
 *daemon_threads*, which indicates whether or not the server should wait for
-thread termination. You should set the flag explicitly if you would like threads
-to behave autonomously; the default is :const:`False`, meaning that Python will
-not exit until all threads created by :class:`ThreadingMixIn` have exited.
+thread termination.  You should set the flag explicitly if you would like
+threads to behave autonomously; the default is :const:`False`, meaning that
+Python will not exit until all threads created by :class:`ThreadingMixIn` have
+exited.
 
 Server classes have the same external methods and attributes, no matter what
 network protocol they use.
@@ -115,8 +116,8 @@
 finished requests and to use :func:`select` to decide which request to work on
 next (or whether to handle a new incoming request).  This is particularly
 important for stream services where each client can potentially be connected for
-a long time (if threads or subprocesses cannot be used). See :mod:`asyncore` for
-another way to manage this.
+a long time (if threads or subprocesses cannot be used).  See :mod:`asyncore`
+for another way to manage this.
 
 .. XXX should data and methods be intermingled, or separate?
    how should the distinction between class and instance variables be drawn?
@@ -153,9 +154,9 @@
 .. method:: BaseServer.serve_forever(poll_interval=0.5)
 
    Handle requests until an explicit :meth:`shutdown` request.  Polls for
-   shutdown every *poll_interval* seconds. It also calls
+   shutdown every *poll_interval* seconds.  It also calls
    :meth:`service_actions` which may be used by a subclass or Mixin to provide
-   various cleanup actions. For e.g. ForkingMixin class uses
+   various cleanup actions.  For e.g. ForkingMixin class uses
    :meth:`service_actions` to cleanup the zombie child processes.
 
    .. versionchanged:: 3.3
@@ -205,7 +206,7 @@
 
 .. attribute:: BaseServer.allow_reuse_address
 
-   Whether the server will allow the reuse of an address. This defaults to
+   Whether the server will allow the reuse of an address.  This defaults to
    :const:`False`, and can be set in subclasses to change the policy.
 
 
@@ -282,7 +283,7 @@
 .. method:: BaseServer.server_activate()
 
    Called by the server's constructor to activate the server.  The default behavior
-   just :meth:`listen`\ s to the server's socket. May be overridden.
+   just :meth:`listen`\ s to the server's socket.  May be overridden.
 
 
 .. method:: BaseServer.server_bind()
@@ -293,10 +294,10 @@
 
 .. method:: BaseServer.verify_request(request, client_address)
 
-   Must return a Boolean value; if the value is :const:`True`, the request will be
-   processed, and if it's :const:`False`, the request will be denied. This function
-   can be overridden to implement access controls for a server. The default
-   implementation always returns :const:`True`.
+   Must return a Boolean value; if the value is :const:`True`, the request will
+   be processed, and if it's :const:`False`, the request will be denied.  This
+   function can be overridden to implement access controls for a server. The
+   default implementation always returns :const:`True`.
 
 
 RequestHandler Objects

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 20:07:01 2011
From: python-checkins at python.org (florent.xicluna)
Date: Fri, 11 Nov 2011 20:07:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_unused_or_redundant_?=
 =?utf8?q?imports_in_concurrent=2Efutures_and_multiprocessing=2E?=
Message-ID: <E1ROwRJ-0006Gf-7E@dinsdale.python.org>

http://hg.python.org/cpython/rev/60fd58a56f44
changeset:   73513:60fd58a56f44
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Fri Nov 11 20:05:50 2011 +0100
summary:
  Remove unused or redundant imports in concurrent.futures and multiprocessing.

files:
  Lib/concurrent/futures/_base.py       |  1 -
  Lib/concurrent/futures/process.py     |  3 +--
  Lib/concurrent/futures/thread.py      |  2 +-
  Lib/multiprocessing/dummy/__init__.py |  2 --
  Lib/multiprocessing/forking.py        |  8 ++------
  Lib/multiprocessing/heap.py           |  1 -
  Lib/multiprocessing/managers.py       |  6 +-----
  Lib/multiprocessing/queues.py         |  1 -
  Lib/multiprocessing/reduction.py      |  2 --
  Lib/multiprocessing/sharedctypes.py   |  1 -
  Lib/multiprocessing/synchronize.py    |  5 +----
  Lib/multiprocessing/util.py           |  3 +--
  12 files changed, 7 insertions(+), 28 deletions(-)


diff --git a/Lib/concurrent/futures/_base.py b/Lib/concurrent/futures/_base.py
--- a/Lib/concurrent/futures/_base.py
+++ b/Lib/concurrent/futures/_base.py
@@ -4,7 +4,6 @@
 __author__ = 'Brian Quinlan (brian at sweetapp.com)'
 
 import collections
-import functools
 import logging
 import threading
 import time
diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -221,7 +221,7 @@
         assert sentinels
         try:
             result_item = result_queue.get(sentinels=sentinels)
-        except SentinelReady as e:
+        except SentinelReady:
             # Mark the process pool broken so that submits fail right now.
             executor = executor_reference()
             if executor is not None:
@@ -291,7 +291,6 @@
             raise NotImplementedError(_system_limited)
     _system_limits_checked = True
     try:
-        import os
         nsems_max = os.sysconf("SC_SEM_NSEMS_MAX")
     except (AttributeError, ValueError):
         # sysconf not available or setting not available
diff --git a/Lib/concurrent/futures/thread.py b/Lib/concurrent/futures/thread.py
--- a/Lib/concurrent/futures/thread.py
+++ b/Lib/concurrent/futures/thread.py
@@ -74,7 +74,7 @@
                 work_queue.put(None)
                 return
             del executor
-    except BaseException as e:
+    except BaseException:
         _base.LOGGER.critical('Exception in worker', exc_info=True)
 
 class ThreadPoolExecutor(_base.Executor):
diff --git a/Lib/multiprocessing/dummy/__init__.py b/Lib/multiprocessing/dummy/__init__.py
--- a/Lib/multiprocessing/dummy/__init__.py
+++ b/Lib/multiprocessing/dummy/__init__.py
@@ -46,9 +46,7 @@
 import sys
 import weakref
 import array
-import itertools
 
-from multiprocessing import TimeoutError, cpu_count
 from multiprocessing.dummy.connection import Pipe
 from threading import Lock, RLock, Semaphore, BoundedSemaphore
 from threading import Event, Condition
diff --git a/Lib/multiprocessing/forking.py b/Lib/multiprocessing/forking.py
--- a/Lib/multiprocessing/forking.py
+++ b/Lib/multiprocessing/forking.py
@@ -35,7 +35,6 @@
 import os
 import sys
 import signal
-import select
 
 from multiprocessing import util, process
 
@@ -101,7 +100,6 @@
 #
 
 if sys.platform != 'win32':
-    import time
     import select
 
     exit = os._exit
@@ -170,7 +168,7 @@
             if self.returncode is None:
                 try:
                     os.kill(self.pid, signal.SIGTERM)
-                except OSError as e:
+                except OSError:
                     if self.wait(timeout=0.1) is None:
                         raise
 
@@ -186,11 +184,9 @@
     import _thread
     import msvcrt
     import _subprocess
-    import time
 
-    from pickle import dump, load, HIGHEST_PROTOCOL
+    from pickle import load, HIGHEST_PROTOCOL
     from _multiprocessing import win32
-    from .util import Finalize
 
     def dump(obj, file, protocol=None):
         ForkingPickler(file, protocol).dump(obj)
diff --git a/Lib/multiprocessing/heap.py b/Lib/multiprocessing/heap.py
--- a/Lib/multiprocessing/heap.py
+++ b/Lib/multiprocessing/heap.py
@@ -34,7 +34,6 @@
 
 import bisect
 import mmap
-import tempfile
 import os
 import sys
 import threading
diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py
--- a/Lib/multiprocessing/managers.py
+++ b/Lib/multiprocessing/managers.py
@@ -39,19 +39,15 @@
 # Imports
 #
 
-import os
 import sys
-import weakref
 import threading
 import array
 import queue
 
 from traceback import format_exc
-from pickle import PicklingError
 from multiprocessing import Process, current_process, active_children, Pool, util, connection
 from multiprocessing.process import AuthenticationString
-from multiprocessing.forking import exit, Popen, assert_spawning, ForkingPickler
-from multiprocessing.util import Finalize, info
+from multiprocessing.forking import exit, Popen, ForkingPickler
 
 #
 # Register some things for pickling
diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -39,7 +39,6 @@
 import threading
 import collections
 import time
-import atexit
 import weakref
 import errno
 
diff --git a/Lib/multiprocessing/reduction.py b/Lib/multiprocessing/reduction.py
--- a/Lib/multiprocessing/reduction.py
+++ b/Lib/multiprocessing/reduction.py
@@ -41,7 +41,6 @@
 import threading
 import struct
 
-import _multiprocessing
 from multiprocessing import current_process
 from multiprocessing.forking import Popen, duplicate, close, ForkingPickler
 from multiprocessing.util import register_after_fork, debug, sub_debug
@@ -61,7 +60,6 @@
 #
 
 if sys.platform == 'win32':
-    import _subprocess
     from _multiprocessing import win32
 
     def send_handle(conn, handle, destination_pid):
diff --git a/Lib/multiprocessing/sharedctypes.py b/Lib/multiprocessing/sharedctypes.py
--- a/Lib/multiprocessing/sharedctypes.py
+++ b/Lib/multiprocessing/sharedctypes.py
@@ -32,7 +32,6 @@
 # SUCH DAMAGE.
 #
 
-import sys
 import ctypes
 import weakref
 
diff --git a/Lib/multiprocessing/synchronize.py b/Lib/multiprocessing/synchronize.py
--- a/Lib/multiprocessing/synchronize.py
+++ b/Lib/multiprocessing/synchronize.py
@@ -37,14 +37,11 @@
     ]
 
 import threading
-import os
 import sys
 
-from time import time as _time, sleep as _sleep
-
 import _multiprocessing
 from multiprocessing.process import current_process
-from multiprocessing.util import Finalize, register_after_fork, debug
+from multiprocessing.util import register_after_fork, debug
 from multiprocessing.forking import assert_spawning, Popen
 
 # Try to import the mp.synchronize module cleanly, if it fails
diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py
--- a/Lib/multiprocessing/util.py
+++ b/Lib/multiprocessing/util.py
@@ -36,7 +36,6 @@
 import itertools
 import weakref
 import atexit
-import select
 import threading        # we want threading to install it's
                         # cleanup function before multiprocessing does
 
@@ -86,7 +85,7 @@
     Returns logger used by multiprocessing
     '''
     global _logger
-    import logging, atexit
+    import logging
 
     logging._acquireLock()
     try:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 11 20:16:06 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 11 Nov 2011 20:16:06 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyODc1?=
 =?utf8?q?=3A_explicitly_specify_default_value_of_the_optional_=27flags=27?=
 =?utf8?q?_argument?=
Message-ID: <E1ROwa6-00078k-Sl@dinsdale.python.org>

http://hg.python.org/cpython/rev/02e4d3ebbb02
changeset:   73514:02e4d3ebbb02
branch:      2.7
parent:      73499:369487785e9f
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 11 21:25:56 2011 +0200
summary:
  Issue #12875: explicitly specify default value of the optional 'flags' argument to re.* functions. Closes #12875

files:
  Doc/library/re.rst |  14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/Doc/library/re.rst b/Doc/library/re.rst
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -431,7 +431,7 @@
 form.
 
 
-.. function:: compile(pattern[, flags])
+.. function:: compile(pattern[, flags=0])
 
    Compile a regular expression pattern into a regular expression object, which
    can be used for matching using its :func:`match` and :func:`search` methods,
@@ -521,7 +521,7 @@
       b = re.compile(r"\d+\.\d*")
 
 
-.. function:: search(pattern, string[, flags])
+.. function:: search(pattern, string[, flags=0])
 
    Scan through *string* looking for a location where the regular expression
    *pattern* produces a match, and return a corresponding :class:`MatchObject`
@@ -530,7 +530,7 @@
    string.
 
 
-.. function:: match(pattern, string[, flags])
+.. function:: match(pattern, string[, flags=0])
 
    If zero or more characters at the beginning of *string* match the regular
    expression *pattern*, return a corresponding :class:`MatchObject` instance.
@@ -584,7 +584,7 @@
       Added the optional flags argument.
 
 
-.. function:: findall(pattern, string[, flags])
+.. function:: findall(pattern, string[, flags=0])
 
    Return all non-overlapping matches of *pattern* in *string*, as a list of
    strings.  The *string* is scanned left-to-right, and matches are returned in
@@ -599,7 +599,7 @@
       Added the optional flags argument.
 
 
-.. function:: finditer(pattern, string[, flags])
+.. function:: finditer(pattern, string[, flags=0])
 
    Return an :term:`iterator` yielding :class:`MatchObject` instances over all
    non-overlapping matches for the RE *pattern* in *string*.  The *string* is
@@ -613,7 +613,7 @@
       Added the optional flags argument.
 
 
-.. function:: sub(pattern, repl, string[, count, flags])
+.. function:: sub(pattern, repl, string[, count, flags=0])
 
    Return the string obtained by replacing the leftmost non-overlapping occurrences
    of *pattern* in *string* by the replacement *repl*.  If the pattern isn't found,
@@ -662,7 +662,7 @@
       Added the optional flags argument.
 
 
-.. function:: subn(pattern, repl, string[, count, flags])
+.. function:: subn(pattern, repl, string[, count, flags=0])
 
    Perform the same operation as :func:`sub`, but return a tuple ``(new_string,
    number_of_subs_made)``.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 01:38:39 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 01:38:39 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMTkz?=
 =?utf8?q?=3A_fix_distutils=2Efilelist=2EFileList_under_Windows?=
Message-ID: <E1RP1cF-0006Pt-7c@dinsdale.python.org>

http://hg.python.org/cpython/rev/0a94e2f807c7
changeset:   73515:0a94e2f807c7
branch:      3.2
parent:      73511:f2b7db0c578b
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 01:20:45 2011 +0100
summary:
  Issue #13193: fix distutils.filelist.FileList under Windows

files:
  Lib/distutils/filelist.py |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/filelist.py b/Lib/distutils/filelist.py
--- a/Lib/distutils/filelist.py
+++ b/Lib/distutils/filelist.py
@@ -313,7 +313,10 @@
         # ditch end of pattern character
         empty_pattern = glob_to_re('')
         prefix_re = (glob_to_re(prefix))[:-len(empty_pattern)]
-        pattern_re = "^" + os.path.join(prefix_re, ".*" + pattern_re)
+        # match both path separators, as in Postel's principle
+        sep_pat = "[" + re.escape(os.path.sep + os.path.altsep
+                                  if os.path.altsep else os.path.sep) + "]"
+        pattern_re = "^" + sep_pat.join([prefix_re, ".*" + pattern_re])
     else:                               # no prefix -- respect anchor flag
         if anchor:
             pattern_re = "^" + pattern_re

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 01:38:40 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 01:38:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_NEWS_entry_?=
	=?utf8?q?for_=2313193?=
Message-ID: <E1RP1cG-0006Px-3F@dinsdale.python.org>

http://hg.python.org/cpython/rev/80d5040f2a78
changeset:   73516:80d5040f2a78
branch:      3.2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 01:22:11 2011 +0100
summary:
  Add NEWS entry for #13193

files:
  Misc/NEWS |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
 Library
 -------
 
+- Issue #13193: Fix distutils.filelist.FileList under Windows.  The
+  "recursive-include" directive now recognizes both legal path separators.
+
 - Issue #13384: Remove unnecessary __future__ import in Lib/random.py
 
 - Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 01:38:40 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 01:38:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313193=3A_Fix_distutils=2Efilelist=2EFileList_and?=
Message-ID: <E1RP1cG-0006Q3-W5@dinsdale.python.org>

http://hg.python.org/cpython/rev/64485e0700ba
changeset:   73517:64485e0700ba
parent:      73513:60fd58a56f44
parent:      73516:80d5040f2a78
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 01:27:19 2011 +0100
summary:
  Issue #13193: Fix distutils.filelist.FileList and
packaging.manifest.Manifest under Windows.  The "recursive-include"
directive now recognizes both legal path separators.

files:
  Lib/distutils/filelist.py |  5 ++++-
  Lib/packaging/manifest.py |  5 ++++-
  Misc/NEWS                 |  4 ++++
  3 files changed, 12 insertions(+), 2 deletions(-)


diff --git a/Lib/distutils/filelist.py b/Lib/distutils/filelist.py
--- a/Lib/distutils/filelist.py
+++ b/Lib/distutils/filelist.py
@@ -313,7 +313,10 @@
         # ditch end of pattern character
         empty_pattern = glob_to_re('')
         prefix_re = (glob_to_re(prefix))[:-len(empty_pattern)]
-        pattern_re = "^" + os.path.join(prefix_re, ".*" + pattern_re)
+        # match both path separators, as in Postel's principle
+        sep_pat = "[" + re.escape(os.path.sep + os.path.altsep
+                                  if os.path.altsep else os.path.sep) + "]"
+        pattern_re = "^" + sep_pat.join([prefix_re, ".*" + pattern_re])
     else:                               # no prefix -- respect anchor flag
         if anchor:
             pattern_re = "^" + pattern_re
diff --git a/Lib/packaging/manifest.py b/Lib/packaging/manifest.py
--- a/Lib/packaging/manifest.py
+++ b/Lib/packaging/manifest.py
@@ -366,7 +366,10 @@
         # ditch end of pattern character
         empty_pattern = _glob_to_re('')
         prefix_re = _glob_to_re(prefix)[:-len(empty_pattern)]
-        pattern_re = "^" + os.path.join(prefix_re, ".*" + pattern_re)
+        # match both path separators, as in Postel's principle
+        sep_pat = "[" + re.escape(os.path.sep + os.path.altsep
+                                  if os.path.altsep else os.path.sep) + "]"
+        pattern_re = "^" + sep_pat.join([prefix_re, ".*" + pattern_re])
     else:                               # no prefix -- respect anchor flag
         if anchor:
             pattern_re = "^" + pattern_re
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -365,6 +365,10 @@
 Library
 -------
 
+- Issue #13193: Fix distutils.filelist.FileList and
+  packaging.manifest.Manifest under Windows.  The "recursive-include"
+  directive now recognizes both legal path separators.
+
 - Issue #13384: Remove unnecessary __future__ import in Lib/random.py
 
 - Issue #13149: Speed up append-only StringIO objects.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 01:38:41 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 01:38:41 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMTkz?=
 =?utf8?q?=3A_Fix_distutils=2Efilelist=2EFileList_under_Windows=2E__The?=
Message-ID: <E1RP1cH-0006Q8-P1@dinsdale.python.org>

http://hg.python.org/cpython/rev/557a973709de
changeset:   73518:557a973709de
branch:      2.7
parent:      73514:02e4d3ebbb02
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 01:33:59 2011 +0100
summary:
  Issue #13193: Fix distutils.filelist.FileList under Windows.  The
"recursive-include" directive now recognizes both legal path separators.

files:
  Lib/distutils/filelist.py |  5 ++++-
  Misc/NEWS                 |  3 +++
  2 files changed, 7 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/filelist.py b/Lib/distutils/filelist.py
--- a/Lib/distutils/filelist.py
+++ b/Lib/distutils/filelist.py
@@ -328,7 +328,10 @@
         # ditch end of pattern character
         empty_pattern = glob_to_re('')
         prefix_re = glob_to_re(prefix)[:-len(empty_pattern)]
-        pattern_re = "^" + os.path.join(prefix_re, ".*" + pattern_re)
+        # match both path separators, as in Postel's principle
+        sep_pat = "[" + re.escape(os.path.sep + os.path.altsep
+                                  if os.path.altsep else os.path.sep) + "]"
+        pattern_re = "^" + sep_pat.join([prefix_re, ".*" + pattern_re])
     else:                               # no prefix -- respect anchor flag
         if anchor:
             pattern_re = "^" + pattern_re
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -76,6 +76,9 @@
 Library
 -------
 
+- Issue #13193: Fix distutils.filelist.FileList under Windows.  The
+  "recursive-include" directive now recognizes both legal path separators.
+
 - Issue #13373: multiprocessing.Queue.get() could sometimes block indefinitely
   when called with a timeout.  Patch by Arnaud Ysmal.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat Nov 12 05:37:11 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 12 Nov 2011 05:37:11 +0100
Subject: [Python-checkins] Daily reference leaks (64485e0700ba): sum=462
Message-ID: <E1RP5L5-0005JX-8m@ap.vmr.nerim.net>

results for 64485e0700ba on branch "default"
--------------------------------------------

test_smtplib leaked [154, 154, 154] references, sum=462


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogxz0C_t', '-x']

From python-checkins at python.org  Sat Nov 12 18:14:28 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 18:14:28 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Plug_some_=28unlikely=29_re?=
	=?utf8?q?fleaks=2E?=
Message-ID: <E1RPH9w-0007jq-4t@dinsdale.python.org>

http://hg.python.org/cpython/rev/9688c634f0d9
changeset:   73519:9688c634f0d9
parent:      73517:64485e0700ba
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 18:05:15 2011 +0100
summary:
  Plug some (unlikely) refleaks.

files:
  Modules/_codecsmodule.c |  60 +++++++++++++++++++++-------
  1 files changed, 45 insertions(+), 15 deletions(-)


diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -720,8 +720,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF7(str, 0, 0, errors),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -740,8 +742,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) == -1)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(PyUnicode_AsEncodedString(str, "utf-8", errors),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -768,8 +772,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF16(str, errors, byteorder),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -788,8 +794,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF16(str, errors, -1),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -808,8 +816,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF16(str, errors, +1),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -836,8 +846,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF32(str, errors, byteorder),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -856,8 +868,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF32(str, errors, -1),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -876,8 +890,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeUTF32(str, errors, +1),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -896,8 +912,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(PyUnicode_AsUnicodeEscapeString(str),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -916,8 +934,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(PyUnicode_AsRawUnicodeEscapeString(str),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -936,8 +956,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_AsLatin1String(str, errors),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -956,8 +978,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_AsASCIIString(str, errors),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -979,8 +1003,10 @@
         mapping = NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(_PyUnicode_EncodeCharmap(str, mapping, errors),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -1010,8 +1036,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(PyUnicode_EncodeCodePage(CP_ACP, str, errors),
                     PyUnicode_GET_LENGTH(str));
     Py_DECREF(str);
@@ -1031,8 +1059,10 @@
         return NULL;
 
     str = PyUnicode_FromObject(str);
-    if (str == NULL || PyUnicode_READY(str) < 0)
+    if (str == NULL || PyUnicode_READY(str) < 0) {
+        Py_XDECREF(str);
         return NULL;
+    }
     v = codec_tuple(PyUnicode_EncodeCodePage(code_page,
                                              str,
                                              errors),

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 18:43:11 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 18:43:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Sanitize_reference_manageme?=
	=?utf8?q?nt_in_the_utf-8_encoder?=
Message-ID: <E1RPHbj-0001Uj-Ie@dinsdale.python.org>

http://hg.python.org/cpython/rev/e3a84fa5cf63
changeset:   73520:e3a84fa5cf63
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 18:35:19 2011 +0100
summary:
  Sanitize reference management in the utf-8 encoder

files:
  Objects/unicodeobject.c |  9 ++++-----
  1 files changed, 4 insertions(+), 5 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4722,6 +4722,7 @@
     int kind;
     void *data;
     Py_ssize_t size;
+    PyObject *rep = NULL;
 
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadArgument();
@@ -4774,7 +4775,6 @@
             *p++ = (char)(0x80 | (ch & 0x3f));
         } else if (0xD800 <= ch && ch <= 0xDFFF) {
             Py_ssize_t newpos;
-            PyObject *rep;
             Py_ssize_t repsize, k, startpos;
             startpos = i-1;
             rep = unicode_encode_call_errorhandler(
@@ -4822,10 +4822,8 @@
                 enum PyUnicode_Kind repkind;
                 void *repdata;
 
-                if (PyUnicode_READY(rep) < 0) {
-                    Py_DECREF(rep);
+                if (PyUnicode_READY(rep) < 0)
                     goto error;
-                }
                 repkind = PyUnicode_KIND(rep);
                 repdata = PyUnicode_DATA(rep);
 
@@ -4841,7 +4839,7 @@
                     *p++ = (char)c;
                 }
             }
-            Py_DECREF(rep);
+            Py_CLEAR(rep);
         } else if (ch < 0x10000) {
             *p++ = (char)(0xe0 | (ch >> 12));
             *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
@@ -4872,6 +4870,7 @@
     Py_XDECREF(exc);
     return result;
  error:
+    Py_XDECREF(rep);
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
     Py_XDECREF(result);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 19:36:52 2011
From: python-checkins at python.org (eli.bendersky)
Date: Sat, 12 Nov 2011 19:36:52 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNzY3?=
 =?utf8?q?=3A_documenting_threading=2ECondition=2Enotify?=
Message-ID: <E1RPIRg-0005Ke-Tu@dinsdale.python.org>

http://hg.python.org/cpython/rev/63a24bff6f36
changeset:   73521:63a24bff6f36
branch:      3.2
parent:      73516:80d5040f2a78
user:        Eli Bendersky <eliben at gmail.com>
date:        Sat Nov 12 20:44:25 2011 +0200
summary:
  Issue #12767: documenting threading.Condition.notify

files:
  Doc/library/threading.rst |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -634,20 +634,21 @@
 
       .. versionadded:: 3.2
 
-   .. method:: notify()
+   .. method:: notify(n=1)
 
-      Wake up a thread waiting on this condition, if any.  If the calling thread
-      has not acquired the lock when this method is called, a
+      By default, wake up one thread waiting on this condition, if any.  If the
+      calling thread has not acquired the lock when this method is called, a
       :exc:`RuntimeError` is raised.
 
-      This method wakes up one of the threads waiting for the condition
-      variable, if any are waiting; it is a no-op if no threads are waiting.
+      This method wakes up at most *n* of the threads waiting for the condition
+      variable; it is a no-op if no threads are waiting.
 
-      The current implementation wakes up exactly one thread, if any are
-      waiting.  However, it's not safe to rely on this behavior.  A future,
-      optimized implementation may occasionally wake up more than one thread.
+      The current implementation wakes up exactly *n* threads, if at least *n*
+      threads are waiting.  However, it's not safe to rely on this behavior.
+      A future, optimized implementation may occasionally wake up more than
+      *n* threads.
 
-      Note: the awakened thread does not actually return from its :meth:`wait`
+      Note: an awakened thread does not actually return from its :meth:`wait`
       call until it can reacquire the lock.  Since :meth:`notify` does not
       release the lock, its caller should.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 19:36:53 2011
From: python-checkins at python.org (eli.bendersky)
Date: Sat, 12 Nov 2011 19:36:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_12767=3A_document_the_argument_of_threading=2EConditio?=
 =?utf8?q?n=2Enotify?=
Message-ID: <E1RPIRh-0005Ki-MZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/ac12dcea69e1
changeset:   73522:ac12dcea69e1
parent:      73520:e3a84fa5cf63
parent:      73521:63a24bff6f36
user:        Eli Bendersky <eliben at gmail.com>
date:        Sat Nov 12 20:46:54 2011 +0200
summary:
  Issue 12767: document the argument of threading.Condition.notify

files:
  Doc/library/threading.rst |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -656,20 +656,21 @@
 
       .. versionadded:: 3.2
 
-   .. method:: notify()
+   .. method:: notify(n=1)
 
-      Wake up a thread waiting on this condition, if any.  If the calling thread
-      has not acquired the lock when this method is called, a
+      By default, wake up one thread waiting on this condition, if any.  If the
+      calling thread has not acquired the lock when this method is called, a
       :exc:`RuntimeError` is raised.
 
-      This method wakes up one of the threads waiting for the condition
-      variable, if any are waiting; it is a no-op if no threads are waiting.
+      This method wakes up at most *n* of the threads waiting for the condition
+      variable; it is a no-op if no threads are waiting.
 
-      The current implementation wakes up exactly one thread, if any are
-      waiting.  However, it's not safe to rely on this behavior.  A future,
-      optimized implementation may occasionally wake up more than one thread.
+      The current implementation wakes up exactly *n* threads, if at least *n*
+      threads are waiting.  However, it's not safe to rely on this behavior.
+      A future, optimized implementation may occasionally wake up more than
+      *n* threads.
 
-      Note: the awakened thread does not actually return from its :meth:`wait`
+      Note: an awakened thread does not actually return from its :meth:`wait`
       call until it can reacquire the lock.  Since :meth:`notify` does not
       release the lock, its caller should.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 19:41:50 2011
From: python-checkins at python.org (eli.bendersky)
Date: Sat, 12 Nov 2011 19:41:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_issue_12?=
 =?utf8?q?767=3A_document_the_argument_of_threading=2ECondition=2Enotify?=
Message-ID: <E1RPIWU-0005iY-LI@dinsdale.python.org>

http://hg.python.org/cpython/rev/63a00d019bb2
changeset:   73523:63a00d019bb2
branch:      2.7
parent:      73518:557a973709de
user:        Eli Bendersky <eliben at gmail.com>
date:        Sat Nov 12 20:51:54 2011 +0200
summary:
  Closes issue 12767: document the argument of threading.Condition.notify

files:
  Doc/library/threading.rst |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -573,20 +573,21 @@
       interface is then used to restore the recursion level when the lock is
       reacquired.
 
-   .. method:: notify()
+   .. method:: notify(n=1)
 
-      Wake up a thread waiting on this condition, if any.  If the calling thread
-      has not acquired the lock when this method is called, a
+      By default, wake up one thread waiting on this condition, if any.  If the
+      calling thread has not acquired the lock when this method is called, a
       :exc:`RuntimeError` is raised.
 
-      This method wakes up one of the threads waiting for the condition
-      variable, if any are waiting; it is a no-op if no threads are waiting.
+      This method wakes up at most *n* of the threads waiting for the condition
+      variable; it is a no-op if no threads are waiting.
 
-      The current implementation wakes up exactly one thread, if any are
-      waiting.  However, it's not safe to rely on this behavior.  A future,
-      optimized implementation may occasionally wake up more than one thread.
+      The current implementation wakes up exactly *n* threads, if at least *n*
+      threads are waiting.  However, it's not safe to rely on this behavior.
+      A future, optimized implementation may occasionally wake up more than
+      *n* threads.
 
-      Note: the awakened thread does not actually return from its :meth:`wait`
+      Note: an awakened thread does not actually return from its :meth:`wait`
       call until it can reacquire the lock.  Since :meth:`notify` does not
       release the lock, its caller should.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:28:32 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 12 Nov 2011 20:28:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Update_mailbox?=
	=?utf8?q?=2EMaildir_tests?=
Message-ID: <E1RPJFg-0000Z6-GG@dinsdale.python.org>

http://hg.python.org/cpython/rev/d2b0751174f6
changeset:   73524:d2b0751174f6
branch:      2.7
parent:      73518:557a973709de
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 12 21:02:42 2011 +0200
summary:
  Update mailbox.Maildir tests

Remove a sleep to fix transient test failures. Use skewfactor of -3 to
make it work on systems that have 1 second precision for time.time().

Closes #11999
Refs #13254

files:
  Lib/test/test_mailbox.py |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -697,7 +697,7 @@
         # skew factor to make _refresh think that the filesystem
         # safety period has passed and re-reading the _toc is only
         # required if mtimes differ.
-        self._box._skewfactor = -2
+        self._box._skewfactor = -3
 
         self._box._refresh()
         self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
@@ -790,7 +790,12 @@
         # refresh is done unconditionally if called for within
         # two-second-plus-a-bit of the last one, just in case the mbox has
         # changed; so now we have to wait for that interval to expire.
-        time.sleep(2.01 + self._box._skewfactor)
+        #
+        # Because this is a test, emulate sleeping. Instead of
+        # sleeping for 2 seconds, use the skew factor to make _refresh
+        # think that 2 seconds have passed and re-reading the _toc is
+        # only required if mtimes differ.
+        self._box._skewfactor = -3
 
         # Re-reading causes the ._toc attribute to be assigned a new dictionary
         # object, so we'll check that the ._toc attribute isn't a different
@@ -803,7 +808,8 @@
         self.assertFalse(refreshed())
 
         # Now, write something into cur and remove it.  This changes
-        # the mtime and should cause a re-read.
+        # the mtime and should cause a re-read. Note that "sleep
+        # emulation" is still in effect, as skewfactor is -3.
         filename = os.path.join(self._path, 'cur', 'stray-file')
         f = open(filename, 'w')
         f.close()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:28:33 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 12 Nov 2011 20:28:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Update_mailbox?=
	=?utf8?q?=2EMaildir_tests?=
Message-ID: <E1RPJFh-0000ZA-8m@dinsdale.python.org>

http://hg.python.org/cpython/rev/b3c5e1c62839
changeset:   73525:b3c5e1c62839
branch:      3.2
parent:      73516:80d5040f2a78
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 12 21:02:42 2011 +0200
summary:
  Update mailbox.Maildir tests

Remove a sleep to fix transient test failures. Use skewfactor of -3 to
make it work on systems that have 1 second precision for time.time().

Closes #11999
Refs #13254

files:
  Lib/test/test_mailbox.py |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -815,7 +815,7 @@
         # skew factor to make _refresh think that the filesystem
         # safety period has passed and re-reading the _toc is only
         # required if mtimes differ.
-        self._box._skewfactor = -2
+        self._box._skewfactor = -3
 
         self._box._refresh()
         self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
@@ -908,7 +908,12 @@
         # refresh is done unconditionally if called for within
         # two-second-plus-a-bit of the last one, just in case the mbox has
         # changed; so now we have to wait for that interval to expire.
-        time.sleep(2.01 + self._box._skewfactor)
+        #
+        # Because this is a test, emulate sleeping. Instead of
+        # sleeping for 2 seconds, use the skew factor to make _refresh
+        # think that 2 seconds have passed and re-reading the _toc is
+        # only required if mtimes differ.
+        self._box._skewfactor = -3
 
         # Re-reading causes the ._toc attribute to be assigned a new dictionary
         # object, so we'll check that the ._toc attribute isn't a different
@@ -921,7 +926,8 @@
         self.assertFalse(refreshed())
 
         # Now, write something into cur and remove it.  This changes
-        # the mtime and should cause a re-read.
+        # the mtime and should cause a re-read. Note that "sleep
+        # emulation" is still in effect, as skewfactor is -3.
         filename = os.path.join(self._path, 'cur', 'stray-file')
         f = open(filename, 'w')
         f.close()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:28:34 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 12 Nov 2011 20:28:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_branch_3=2E2?=
Message-ID: <E1RPJFi-0000ZE-0Z@dinsdale.python.org>

http://hg.python.org/cpython/rev/4c11dac0e9d5
changeset:   73526:4c11dac0e9d5
parent:      73520:e3a84fa5cf63
parent:      73525:b3c5e1c62839
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 12 21:15:12 2011 +0200
summary:
  Merge branch 3.2

files:
  Lib/test/test_mailbox.py |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -815,7 +815,7 @@
         # skew factor to make _refresh think that the filesystem
         # safety period has passed and re-reading the _toc is only
         # required if mtimes differ.
-        self._box._skewfactor = -2
+        self._box._skewfactor = -3
 
         self._box._refresh()
         self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
@@ -908,7 +908,12 @@
         # refresh is done unconditionally if called for within
         # two-second-plus-a-bit of the last one, just in case the mbox has
         # changed; so now we have to wait for that interval to expire.
-        time.sleep(2.01 + self._box._skewfactor)
+        #
+        # Because this is a test, emulate sleeping. Instead of
+        # sleeping for 2 seconds, use the skew factor to make _refresh
+        # think that 2 seconds have passed and re-reading the _toc is
+        # only required if mtimes differ.
+        self._box._skewfactor = -3
 
         # Re-reading causes the ._toc attribute to be assigned a new dictionary
         # object, so we'll check that the ._toc attribute isn't a different
@@ -921,7 +926,8 @@
         self.assertFalse(refreshed())
 
         # Now, write something into cur and remove it.  This changes
-        # the mtime and should cause a re-read.
+        # the mtime and should cause a re-read. Note that "sleep
+        # emulation" is still in effect, as skewfactor is -3.
         filename = os.path.join(self._path, 'cur', 'stray-file')
         support.create_empty_file(filename)
         os.unlink(filename)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:28:34 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 12 Nov 2011 20:28:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge_heads?=
Message-ID: <E1RPJFi-0000ZI-O2@dinsdale.python.org>

http://hg.python.org/cpython/rev/9fb636764fac
changeset:   73527:9fb636764fac
parent:      73526:4c11dac0e9d5
parent:      73522:ac12dcea69e1
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 12 21:23:28 2011 +0200
summary:
  Merge heads

files:
  Doc/library/threading.rst |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -656,20 +656,21 @@
 
       .. versionadded:: 3.2
 
-   .. method:: notify()
+   .. method:: notify(n=1)
 
-      Wake up a thread waiting on this condition, if any.  If the calling thread
-      has not acquired the lock when this method is called, a
+      By default, wake up one thread waiting on this condition, if any.  If the
+      calling thread has not acquired the lock when this method is called, a
       :exc:`RuntimeError` is raised.
 
-      This method wakes up one of the threads waiting for the condition
-      variable, if any are waiting; it is a no-op if no threads are waiting.
+      This method wakes up at most *n* of the threads waiting for the condition
+      variable; it is a no-op if no threads are waiting.
 
-      The current implementation wakes up exactly one thread, if any are
-      waiting.  However, it's not safe to rely on this behavior.  A future,
-      optimized implementation may occasionally wake up more than one thread.
+      The current implementation wakes up exactly *n* threads, if at least *n*
+      threads are waiting.  However, it's not safe to rely on this behavior.
+      A future, optimized implementation may occasionally wake up more than
+      *n* threads.
 
-      Note: the awakened thread does not actually return from its :meth:`wait`
+      Note: an awakened thread does not actually return from its :meth:`wait`
       call until it can reacquire the lock.  Since :meth:`notify` does not
       release the lock, its caller should.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:28:35 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 12 Nov 2011 20:28:35 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Merge_heads?=
Message-ID: <E1RPJFj-0000ZM-Fq@dinsdale.python.org>

http://hg.python.org/cpython/rev/bf21980c12f1
changeset:   73528:bf21980c12f1
branch:      3.2
parent:      73525:b3c5e1c62839
parent:      73521:63a24bff6f36
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 12 21:23:46 2011 +0200
summary:
  Merge heads

files:
  Doc/library/threading.rst |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -634,20 +634,21 @@
 
       .. versionadded:: 3.2
 
-   .. method:: notify()
+   .. method:: notify(n=1)
 
-      Wake up a thread waiting on this condition, if any.  If the calling thread
-      has not acquired the lock when this method is called, a
+      By default, wake up one thread waiting on this condition, if any.  If the
+      calling thread has not acquired the lock when this method is called, a
       :exc:`RuntimeError` is raised.
 
-      This method wakes up one of the threads waiting for the condition
-      variable, if any are waiting; it is a no-op if no threads are waiting.
+      This method wakes up at most *n* of the threads waiting for the condition
+      variable; it is a no-op if no threads are waiting.
 
-      The current implementation wakes up exactly one thread, if any are
-      waiting.  However, it's not safe to rely on this behavior.  A future,
-      optimized implementation may occasionally wake up more than one thread.
+      The current implementation wakes up exactly *n* threads, if at least *n*
+      threads are waiting.  However, it's not safe to rely on this behavior.
+      A future, optimized implementation may occasionally wake up more than
+      *n* threads.
 
-      Note: the awakened thread does not actually return from its :meth:`wait`
+      Note: an awakened thread does not actually return from its :meth:`wait`
       call until it can reacquire the lock.  Since :meth:`notify` does not
       release the lock, its caller should.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:28:36 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 12 Nov 2011 20:28:36 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMi43IC0+IDIuNyk6?=
	=?utf8?q?_Merge_heads?=
Message-ID: <E1RPJFk-0000ZQ-9d@dinsdale.python.org>

http://hg.python.org/cpython/rev/c3b063c82ae5
changeset:   73529:c3b063c82ae5
branch:      2.7
parent:      73524:d2b0751174f6
parent:      73523:63a00d019bb2
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 12 21:24:00 2011 +0200
summary:
  Merge heads

files:
  Doc/library/threading.rst |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -573,20 +573,21 @@
       interface is then used to restore the recursion level when the lock is
       reacquired.
 
-   .. method:: notify()
+   .. method:: notify(n=1)
 
-      Wake up a thread waiting on this condition, if any.  If the calling thread
-      has not acquired the lock when this method is called, a
+      By default, wake up one thread waiting on this condition, if any.  If the
+      calling thread has not acquired the lock when this method is called, a
       :exc:`RuntimeError` is raised.
 
-      This method wakes up one of the threads waiting for the condition
-      variable, if any are waiting; it is a no-op if no threads are waiting.
+      This method wakes up at most *n* of the threads waiting for the condition
+      variable; it is a no-op if no threads are waiting.
 
-      The current implementation wakes up exactly one thread, if any are
-      waiting.  However, it's not safe to rely on this behavior.  A future,
-      optimized implementation may occasionally wake up more than one thread.
+      The current implementation wakes up exactly *n* threads, if at least *n*
+      threads are waiting.  However, it's not safe to rely on this behavior.
+      A future, optimized implementation may occasionally wake up more than
+      *n* threads.
 
-      Note: the awakened thread does not actually return from its :meth:`wait`
+      Note: an awakened thread does not actually return from its :meth:`wait`
       call until it can reacquire the lock.  Since :meth:`notify` does not
       release the lock, its caller should.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:41:27 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 20:41:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Restore_smtpd=2EDEBUGSTREAM?=
 =?utf8?q?_at_the_end_of_test=5Fsmtpd=2E?=
Message-ID: <E1RPJSB-0001Nu-OA@dinsdale.python.org>

http://hg.python.org/cpython/rev/6b9f547e92d8
changeset:   73530:6b9f547e92d8
parent:      73522:ac12dcea69e1
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 20:36:29 2011 +0100
summary:
  Restore smtpd.DEBUGSTREAM at the end of test_smtpd.
This fixes a subsequent refleak in test_smtplib.

files:
  Lib/test/test_smtpd.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py
--- a/Lib/test/test_smtpd.py
+++ b/Lib/test/test_smtpd.py
@@ -52,6 +52,7 @@
 class SMTPDChannelTest(TestCase):
     def setUp(self):
         smtpd.socket = asyncore.socket = mock_socket
+        self.old_debugstream = smtpd.DEBUGSTREAM
         self.debug = smtpd.DEBUGSTREAM = io.StringIO()
         self.server = DummyServer('a', 'b')
         conn, addr = self.server.accept()
@@ -60,6 +61,7 @@
     def tearDown(self):
         asyncore.close_all()
         asyncore.socket = smtpd.socket = socket
+        smtpd.DEBUGSTREAM = self.old_debugstream
 
     def write_line(self, line):
         self.channel.socket.queue_recv(line)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:41:28 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 20:41:28 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge?=
Message-ID: <E1RPJSC-0001O4-Gp@dinsdale.python.org>

http://hg.python.org/cpython/rev/a116c700b0d1
changeset:   73531:a116c700b0d1
parent:      73530:6b9f547e92d8
parent:      73527:9fb636764fac
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 20:36:51 2011 +0100
summary:
  Merge

files:
  Lib/test/test_mailbox.py |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py
--- a/Lib/test/test_mailbox.py
+++ b/Lib/test/test_mailbox.py
@@ -815,7 +815,7 @@
         # skew factor to make _refresh think that the filesystem
         # safety period has passed and re-reading the _toc is only
         # required if mtimes differ.
-        self._box._skewfactor = -2
+        self._box._skewfactor = -3
 
         self._box._refresh()
         self.assertEqual(sorted(self._box._toc.keys()), sorted([key0, key1]))
@@ -908,7 +908,12 @@
         # refresh is done unconditionally if called for within
         # two-second-plus-a-bit of the last one, just in case the mbox has
         # changed; so now we have to wait for that interval to expire.
-        time.sleep(2.01 + self._box._skewfactor)
+        #
+        # Because this is a test, emulate sleeping. Instead of
+        # sleeping for 2 seconds, use the skew factor to make _refresh
+        # think that 2 seconds have passed and re-reading the _toc is
+        # only required if mtimes differ.
+        self._box._skewfactor = -3
 
         # Re-reading causes the ._toc attribute to be assigned a new dictionary
         # object, so we'll check that the ._toc attribute isn't a different
@@ -921,7 +926,8 @@
         self.assertFalse(refreshed())
 
         # Now, write something into cur and remove it.  This changes
-        # the mtime and should cause a re-read.
+        # the mtime and should cause a re-read. Note that "sleep
+        # emulation" is still in effect, as skewfactor is -3.
         filename = os.path.join(self._path, 'cur', 'stray-file')
         support.create_empty_file(filename)
         os.unlink(filename)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:43:49 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 20:43:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Restore_smtpd?=
 =?utf8?q?=2EDEBUGSTREAM_at_the_end_of_test=5Fsmtpd=2E?=
Message-ID: <E1RPJUT-0001Sz-SE@dinsdale.python.org>

http://hg.python.org/cpython/rev/71f0875300b0
changeset:   73532:71f0875300b0
branch:      3.2
parent:      73528:bf21980c12f1
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 20:36:29 2011 +0100
summary:
  Restore smtpd.DEBUGSTREAM at the end of test_smtpd.
(backport of 6b9f547e92d8)

files:
  Lib/test/test_smtpd.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_smtpd.py b/Lib/test/test_smtpd.py
--- a/Lib/test/test_smtpd.py
+++ b/Lib/test/test_smtpd.py
@@ -52,6 +52,7 @@
 class SMTPDChannelTest(TestCase):
     def setUp(self):
         smtpd.socket = asyncore.socket = mock_socket
+        self.old_debugstream = smtpd.DEBUGSTREAM
         self.debug = smtpd.DEBUGSTREAM = io.StringIO()
         self.server = DummyServer('a', 'b')
         conn, addr = self.server.accept()
@@ -60,6 +61,7 @@
     def tearDown(self):
         asyncore.close_all()
         asyncore.socket = smtpd.socket = socket
+        smtpd.DEBUGSTREAM = self.old_debugstream
 
     def write_line(self, line):
         self.channel.socket.queue_recv(line)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 20:43:50 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 20:43:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Null_merge?=
Message-ID: <E1RPJUU-0001T9-Ki@dinsdale.python.org>

http://hg.python.org/cpython/rev/b5c1e9c7f09c
changeset:   73533:b5c1e9c7f09c
parent:      73531:a116c700b0d1
parent:      73532:71f0875300b0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 20:39:02 2011 +0100
summary:
  Null merge

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 21:20:30 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 21:20:30 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_the_small_object_alloca?=
	=?utf8?q?tor_for_small_bytearrays?=
Message-ID: <E1RPK3y-0003uC-Bb@dinsdale.python.org>

http://hg.python.org/cpython/rev/9b26fa7f9adf
changeset:   73534:9b26fa7f9adf
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 21:15:28 2011 +0100
summary:
  Use the small object allocator for small bytearrays

files:
  Objects/bytearrayobject.c |  10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -139,7 +139,7 @@
     }
     else {
         alloc = size + 1;
-        new->ob_bytes = PyMem_Malloc(alloc);
+        new->ob_bytes = PyObject_Malloc(alloc);
         if (new->ob_bytes == NULL) {
             Py_DECREF(new);
             return PyErr_NoMemory();
@@ -209,7 +209,7 @@
         alloc = size + 1;
     }
 
-    sval = PyMem_Realloc(((PyByteArrayObject *)self)->ob_bytes, alloc);
+    sval = PyObject_Realloc(((PyByteArrayObject *)self)->ob_bytes, alloc);
     if (sval == NULL) {
         PyErr_NoMemory();
         return -1;
@@ -870,7 +870,7 @@
     }
 
     newsize = 15 + length * 4;
-    buffer = PyMem_Malloc(newsize);
+    buffer = PyObject_Malloc(newsize);
     if (buffer == NULL) {
         PyErr_NoMemory();
         return NULL;
@@ -924,7 +924,7 @@
     }
 
     v = PyUnicode_DecodeASCII(buffer, p - buffer, NULL);
-    PyMem_Free(buffer);
+    PyObject_Free(buffer);
     return v;
 }
 
@@ -1020,7 +1020,7 @@
         PyErr_Print();
     }
     if (self->ob_bytes != 0) {
-        PyMem_Free(self->ob_bytes);
+        PyObject_Free(self->ob_bytes);
     }
     Py_TYPE(self)->tp_free((PyObject *)self);
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 12 22:39:35 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 12 Nov 2011 22:39:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_memory_leak_in_io=2EStr?=
	=?utf8?q?ingIO?=
Message-ID: <E1RPLIV-0000mC-BK@dinsdale.python.org>

http://hg.python.org/cpython/rev/c6dafa2e2594
changeset:   73535:c6dafa2e2594
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 12 22:34:28 2011 +0100
summary:
  Fix memory leak in io.StringIO

files:
  Modules/_io/textio.c |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -455,6 +455,7 @@
             }
             Py_DECREF(output);
             output = PyUnicode_FromKindAndData(kind, translated, out);
+            PyMem_Free(translated);
             if (!output)
                 goto error;
         }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 13 01:08:16 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 13 Nov 2011 01:08:16 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_memory_leak?=
 =?utf8?q?_with_FLUFL-related_syntax_errors_=28!=29?=
Message-ID: <E1RPNcO-00040S-SF@dinsdale.python.org>

http://hg.python.org/cpython/rev/11f92e6d8871
changeset:   73536:11f92e6d8871
branch:      3.2
parent:      73532:71f0875300b0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 13 01:01:23 2011 +0100
summary:
  Fix memory leak with FLUFL-related syntax errors (!)

files:
  Parser/parsetok.c |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Parser/parsetok.c b/Parser/parsetok.c
--- a/Parser/parsetok.c
+++ b/Parser/parsetok.c
@@ -183,11 +183,13 @@
         if (type == NOTEQUAL) {
             if (!(ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                             strcmp(str, "!=")) {
+                PyObject_FREE(str);
                 err_ret->error = E_SYNTAX;
                 break;
             }
             else if ((ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                             strcmp(str, "<>")) {
+                PyObject_FREE(str);
                 err_ret->text = "with Barry as BDFL, use '<>' "
                                 "instead of '!='";
                 err_ret->error = E_SYNTAX;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 13 01:08:17 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 13 Nov 2011 01:08:17 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Fix_memory_leak_with_FLUFL-related_syntax_errors_=28!=29?=
Message-ID: <E1RPNcP-00040c-L9@dinsdale.python.org>

http://hg.python.org/cpython/rev/0feb5a5dbaeb
changeset:   73537:0feb5a5dbaeb
parent:      73535:c6dafa2e2594
parent:      73536:11f92e6d8871
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 13 01:02:02 2011 +0100
summary:
  Fix memory leak with FLUFL-related syntax errors (!)

files:
  Parser/parsetok.c |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Parser/parsetok.c b/Parser/parsetok.c
--- a/Parser/parsetok.c
+++ b/Parser/parsetok.c
@@ -190,11 +190,13 @@
         if (type == NOTEQUAL) {
             if (!(ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                             strcmp(str, "!=")) {
+                PyObject_FREE(str);
                 err_ret->error = E_SYNTAX;
                 break;
             }
             else if ((ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
                             strcmp(str, "<>")) {
+                PyObject_FREE(str);
                 err_ret->text = "with Barry as BDFL, use '<>' "
                                 "instead of '!='";
                 err_ret->error = E_SYNTAX;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 13 04:20:16 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 13 Nov 2011 04:20:16 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_In_text_I/O=2C_optimize_sca?=
 =?utf8?q?nning_for_new_lines_with_1-byte_unicode_chars?=
Message-ID: <E1RPQcC-0007Ta-R9@dinsdale.python.org>

http://hg.python.org/cpython/rev/6a357b949df1
changeset:   73538:6a357b949df1
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 13 03:53:42 2011 +0100
summary:
  In text I/O, optimize scanning for new lines with 1-byte unicode chars

files:
  Modules/_io/textio.c |  32 ++++++++++++++++++++------------
  1 files changed, 20 insertions(+), 12 deletions(-)


diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -365,19 +365,23 @@
             */
             if (seennl == 0 &&
                 memchr(in_str, '\n', kind * len) != NULL) {
-                Py_ssize_t i = 0;
-                for (;;) {
-                    Py_UCS4 c;
-                    /* Fast loop for non-control characters */
-                    while (PyUnicode_READ(kind, in_str, i) > '\n')
-                        i++;
-                    c = PyUnicode_READ(kind, in_str, i++);
-                    if (c == '\n') {
-                        seennl |= SEEN_LF;
-                        break;
+                if (kind == PyUnicode_1BYTE_KIND)
+                    seennl |= SEEN_LF;
+                else {
+                    Py_ssize_t i = 0;
+                    for (;;) {
+                        Py_UCS4 c;
+                        /* Fast loop for non-control characters */
+                        while (PyUnicode_READ(kind, in_str, i) > '\n')
+                            i++;
+                        c = PyUnicode_READ(kind, in_str, i++);
+                        if (c == '\n') {
+                            seennl |= SEEN_LF;
+                            break;
+                        }
+                        if (i >= len)
+                            break;
                     }
-                    if (i >= len)
-                        break;
                 }
             }
             /* Finished: we have scanned for newlines, and none of them
@@ -1597,6 +1601,10 @@
 static char *
 find_control_char(int kind, char *s, char *end, Py_UCS4 ch)
 {
+    if (kind == PyUnicode_1BYTE_KIND) {
+        assert(ch < 256);
+        return (char *) memchr((void *) s, (char) ch, end - s);
+    }
     for (;;) {
         while (PyUnicode_READ(kind, s, 0) > ch)
             s += kind;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 13 04:20:17 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 13 Nov 2011 04:20:17 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Restore_performance_of_spec?=
 =?utf8?q?ial_casings_for_utf-16_and_utf-32_in_TextIOWrapper?=
Message-ID: <E1RPQcD-0007Tk-JQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/27adb952813b
changeset:   73539:27adb952813b
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 13 04:11:37 2011 +0100
summary:
  Restore performance of special casings for utf-16 and utf-32 in TextIOWrapper

files:
  Modules/_io/textio.c |  30 ++++++++++++------------------
  1 files changed, 12 insertions(+), 18 deletions(-)


diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -725,17 +725,15 @@
 static PyObject *
 utf16be_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors), 1);
+    return _PyUnicode_EncodeUTF16(text,
+                                  PyBytes_AS_STRING(self->errors), 1);
 }
 
 static PyObject *
 utf16le_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors), -1);
+    return _PyUnicode_EncodeUTF16(text,
+                                  PyBytes_AS_STRING(self->errors), -1);
 }
 
 static PyObject *
@@ -749,25 +747,22 @@
         return utf16le_encode(self, text);
 #endif
     }
-    return PyUnicode_EncodeUTF16(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors), 0);
+    return _PyUnicode_EncodeUTF16(text,
+                                  PyBytes_AS_STRING(self->errors), 0);
 }
 
 static PyObject *
 utf32be_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors), 1);
+    return _PyUnicode_EncodeUTF32(text,
+                                  PyBytes_AS_STRING(self->errors), 1);
 }
 
 static PyObject *
 utf32le_encode(textio *self, PyObject *text)
 {
-    return PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors), -1);
+    return _PyUnicode_EncodeUTF32(text,
+                                  PyBytes_AS_STRING(self->errors), -1);
 }
 
 static PyObject *
@@ -781,9 +776,8 @@
         return utf32le_encode(self, text);
 #endif
     }
-    return PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(text),
-                                 PyUnicode_GET_SIZE(text),
-                                 PyBytes_AS_STRING(self->errors), 0);
+    return _PyUnicode_EncodeUTF32(text,
+                                  PyBytes_AS_STRING(self->errors), 0);
 }
 
 static PyObject *

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Nov 13 05:36:51 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 13 Nov 2011 05:36:51 +0100
Subject: [Python-checkins] Daily reference leaks (0feb5a5dbaeb): sum=0
Message-ID: <E1RPRoJ-0003CY-If@ap.vmr.nerim.net>

results for 0feb5a5dbaeb on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogAlQqYX', '-x']

From python-checkins at python.org  Sun Nov 13 06:04:20 2011
From: python-checkins at python.org (nick.coghlan)
Date: Sun, 13 Nov 2011 06:04:20 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Finally_publish_the_import_eng?=
	=?utf8?q?ine_PEP_on_python=2Eorg?=
Message-ID: <E1RPSEu-0004pD-38@dinsdale.python.org>

http://hg.python.org/peps/rev/51b56ff80426
changeset:   3984:51b56ff80426
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sun Nov 13 15:04:07 2011 +1000
summary:
  Finally publish the import engine PEP on python.org

files:
  pep-0406.txt |  247 +++++++++++++++++++++++++++++++++++++++
  1 files changed, 247 insertions(+), 0 deletions(-)


diff --git a/pep-0406.txt b/pep-0406.txt
new file mode 100644
--- /dev/null
+++ b/pep-0406.txt
@@ -0,0 +1,247 @@
+PEP: 406
+Title: Improved Encapsulation of Import State
+Version: $Revision$
+Last-Modified: $Date$
+Author: Nick Coghlan <ncoghlan at gmail.com>, Greg Slodkowicz <jergosh at gmail.com>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Created: 4-Jul-2011
+Post-History: 31-Jul-2011, 13-Nov-2011
+
+Abstract
+========
+
+This PEP proposes the introduction of a new 'ImportEngine' class as part of
+``importlib`` which would encapsulate all state related to importing modules
+into a single object. Creating new instances of this object would then provide
+an alternative to completely replacing the built-in implementation of the
+import statement, by overriding the ``__import__()`` function. To work with
+the builtin import functionality and importing via import engine objects,
+module importers and loaders must accept an optional ``engine`` parameter. In
+that sense, this PEP constitutes a revision of finder and loader interfaces
+described in PEP 302 [1]_. However, the standard import process will not
+supply the additional argument, so this proposal remains fully backwards
+compatible.
+
+The PEP also proposes inclusion of a ``GlobalImportEngine`` subclass and a
+globally accessible instance of that class, which "writes through" to the
+process global state and invokes importers and loaders without the additional
+``engine`` argument. This provides a backwards compatible bridge between the
+proposed encapsulated API and the legacy process global state.
+
+
+Rationale
+=========
+
+Currently, most state related to the import system is stored as module level
+attributes in the ``sys`` module. The one exception is the import lock, which
+is not accessible directly, but only via the related functions in the ``imp``
+module. The current process global import state comprises:
+
+* sys.modules
+* sys.path
+* sys.path_hooks
+* sys.meta_path
+* sys.path_importer_cache
+* the import lock (imp.lock_held()/acquire_lock()/release_lock())
+
+Isolating this state would allow multiple import states to be
+conveniently stored within a process. Placing the import functionality
+in a self-contained object would also allow subclassing to add additional
+features (e.g. module import notifications or fine-grained control
+over which modules can be imported). The engine would also be
+subclassed to make it possible to use the import engine API to
+interact with the existing process-global state.
+
+The namespace PEPs (especially PEP 402) raise a potential need for
+*additional* process global state, in order to correctly update package paths
+as ``sys.path`` is modified.
+
+
+Proposal
+========
+
+We propose introducing an ImportEngine class to encapsulate import
+functionality. This includes an ``__import__()`` method which can
+be used as an alternative to the built-in ``__import__()`` when
+desired and also an ``import_module()`` method, equivalent to
+``importlib.import_module()`` [3]_.
+
+Since the new style finders and loaders should also have the option to
+modify the global import state, we introduce a ``GlobalImportState``
+class with an interface identical to ``ImportEngine`` but taking
+advantage of the global state. This can be easily implemented using
+class properties.
+
+
+Specification
+=============
+
+ImportEngine API
+~~~~~~~~~~~~~~~~
+
+The proposed extension consists of the following objects:
+
+``importlib.engine.ImportEngine``
+
+    ``from_engine(self, other)``
+
+        Create a new import object from another ImportEngine instance. The
+        new object is initialised with a copy of the state in ``other``. When
+        called on ``importlib engine.sysengine``, ``from_engine()`` can be
+        used to create an ``ImportEngine`` object with a **copy** of the
+        global import state.
+
+    ``__import__(self, name, globals={}, locals={}, fromlist=[], level=0)``
+
+        Reimplementation of the builtin ``__import__()`` function. The
+        import of a module will proceed using the state stored in the
+        ImportEngine instance rather than the global import state. For full
+        documentation of ``__import__`` funtionality, see [2]_ .
+        ``__import__()`` from ``ImportEngine`` and its subclasses can be used
+        to customise the behaviour of the ``import`` statement by replacing
+        ``__builtin__.__import__`` with ``ImportEngine().__import__``.
+
+    ``import_module(name, package=None)``
+
+        A reimplementation of ``importlib.import_module()`` which uses the
+        import state stored in the ImportEngine instance. See [3]_ for a full
+        reference.
+
+    ``modules, path, path_hooks, meta_path, path_importer_cache``
+
+        Instance-specific versions of their process global ``sys`` equivalents
+
+
+``importlib.engine.GlobalImportEngine(ImportEngine)``
+
+    Convenience class to provide engine-like access to the global state.
+    Provides ``__import__()``, ``import_module()`` and ``from_engine()``
+    methods like ``ImportEngine`` but writes through to the global state
+    in ``sys``.
+
+
+Global variables
+~~~~~~~~~~~~~~~~
+
+``importlib.engine.sysengine``
+
+    A precreated instance of ``GlobalImportEngine``. Intended for use by
+    importers and loaders that have been updated to accept optional ``engine``
+    parameters and with ``ImportEngine.from_engine(sysengine)`` to start with
+    a copy of the process global import state.
+
+
+Necessary changes to finder/loader interfaces:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``find_module (cls, fullname, path=None, engine=None)``
+
+``load_module (cls, fullname, path=None, engine=None)``
+
+The only difference between engine compatible and PEP 302 compatible
+finders/loaders is the presence of an additional ``engine`` parameter.
+This is intended to specify an ImportEngine instance or subclass thereof. 
+This parameter is optional so that engine compatible finders and
+loaders can be made backwards compatible with PEP 302 calling conventions by
+falling back on ``engine.sysengine`` with the following simple pattern::
+
+    def find_module(cls, fullname, path=None, engine=None):
+        if not engine:
+            engine = importlib.engine.sysengine
+        ...
+
+
+Open Issues
+===========
+
+
+API design for falling back to global import state
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The current proposal relies on the ``from_engine()`` API to fall back to the
+global import state. It may be desirable to offer a variant that instead falls
+back to the global import state dynamically.
+
+However, one big advantage of starting with an "as isolated as possible"
+design is that it becomes possible to experiment with subclasses that blur
+the boundaries between the engine instance state and the process global state
+in various ways.
+
+
+Builtin and extension modules must be process global
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Due to platform limitations, only one copy of each builtin and extension
+module can readily exist in each process. Accordingly, it is impossible for
+each ``ImportEngine`` instance to load such modules independently.
+
+The simplest solution is for ``ImportEngine`` to refuse to load such modules,
+raising ``ImportError``. ``GlobalImportEngine`` would be able to load them
+normally.
+
+``ImportEngine`` will still return such modules from a prepopulated module
+cache - it's only loading them directly which causes problems.
+
+
+Nested imports
+~~~~~~~~~~~~~~
+
+The reference implementation currently applies only to the outermost import.
+Any imports by the module being imported will be handled using the standard
+import machinery.
+
+One way to handle this is to place the burden on the implementation of module
+loaders to set ``module.__dict__["__import__"] = engine.__import__`` before
+running the module's code. The ``importlib`` design facilities this by
+allowing the change to be made in one place (``_LoaderBasics._load_module``).
+
+
+Scope of API updates
+~~~~~~~~~~~~~~~~~~~~
+
+The reference implementation focuses on finding and loading modules. There
+may be other PEP 302 APIs that should also be updated to accept an optional
+``engine`` parameter.
+
+
+Reference Implementation
+========================
+
+A reference implementation [4]_ based on Brett Cannon's importlib has been
+developed by Greg Slodkowicz as part of the 2011 Google Summer of Code. Note
+that the current implementation avoids modifying existing code, and hence
+duplicates a lot of things unnecessarily. An actual implementation would just
+modify any such affected code in place.
+
+
+References
+==========
+
+.. [1] PEP 302, New Import Hooks, J van Rossum, Moore
+  (http://www.python.org/dev/peps/pep-0302)
+
+.. [2] __import__() builtin function, The Python Standard Library documentation
+  (http://docs.python.org/library/functions.html#__import__)
+
+.. [3] Importlib documentation, Cannon
+  (http://docs.python.org/dev/library/importlib)
+
+.. [4] Reference implentation
+  (https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py)
+
+
+Copyright
+=========
+
+This document has been placed in the public domain.
+
+..
+  Local Variables:
+  mode: indented-text
+  indent-tabs-mode: nil
+  sentence-end-double-space: t
+  fill-column: 70
+  coding: utf-8
+  End:

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sun Nov 13 17:36:28 2011
From: python-checkins at python.org (ezio.melotti)
Date: Sun, 13 Nov 2011 17:36:28 +0100
Subject: [Python-checkins] =?utf8?q?hooks=3A_Allow_more_than_one_issue_num?=
 =?utf8?q?ber_in_the_hgroundup_hook=2E?=
Message-ID: <E1RPd2i-0006e1-Hi@dinsdale.python.org>

http://hg.python.org/hooks/rev/ebdd1c1c365b
changeset:   78:ebdd1c1c365b
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Sun Nov 13 18:36:08 2011 +0200
summary:
  Allow more than one issue number in the hgroundup hook.

files:
  hgroundup.py |  34 +++++++++++++++++++---------------
  1 files changed, 19 insertions(+), 15 deletions(-)


diff --git a/hgroundup.py b/hgroundup.py
--- a/hgroundup.py
+++ b/hgroundup.py
@@ -1,7 +1,7 @@
 """Mercurial hook to update a Roundup issue.
 
-Update a Roundup issue via email for changesets where commit messages
-mention an issue anywhere in the commit message in the following way:
+Update Roundup issue(s) via email for changesets where commit messages
+mention one or more issues anywhere in the commit message in the following way:
 
    #12345
    issue12345
@@ -83,19 +83,23 @@
     for rev in xrange(start, len(repo)):
         ctx = repo[rev]
         description = fromlocal(ctx.description().strip())
-        match = ISSUE_PATTERN.search(description)
-        ui.debug('match in commit msg: %s\n' % (match and match.groupdict() or 'no'))
-        if not match:
-            continue
-        data = match.groupdict()
-        comment = Template(COMMENT_TEMPLATE).substitute({
-            'author': fromlocal(person(ctx.user())),
-            'branch': ctx.branch(),
-            'changeset_id': str(ctx),
-            'changeset_url': posixpath.join(repourl, str(ctx)),
-            'commit_msg': description.splitlines()[0],
-        })
-        add_comment(issues, data, comment)
+        matches = ISSUE_PATTERN.finditer(description)
+        ids = set()
+        for match in matches:
+            data = match.groupdict()
+            ui.debug('match in commit msg: %s\n' % data)
+            # check for duplicated issue numbers in the same commit msg
+            if data['issue_id'] in ids:
+                continue
+            ids.add(data['issue_id'])
+            comment = Template(COMMENT_TEMPLATE).substitute({
+                'author': fromlocal(person(ctx.user())),
+                'branch': ctx.branch(),
+                'changeset_id': str(ctx),
+                'changeset_url': posixpath.join(repourl, str(ctx)),
+                'commit_msg': description.splitlines()[0],
+            })
+            add_comment(issues, data, comment)
     if issues:
         try:
             send_comments(mailrelay, fromaddr, toaddr, issues)

-- 
Repository URL: http://hg.python.org/hooks

From python-checkins at python.org  Sun Nov 13 17:59:38 2011
From: python-checkins at python.org (barry.warsaw)
Date: Sun, 13 Nov 2011 17:59:38 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_With_Carl=27s_permission=2C_sw?=
	=?utf8?q?ap_PEPs_404_and_405=2E?=
Message-ID: <E1RPdP8-0008Oq-21@dinsdale.python.org>

http://hg.python.org/peps/rev/7e9647d691e6
changeset:   3985:7e9647d691e6
user:        Barry Warsaw <barry at python.org>
date:        Sun Nov 13 11:59:21 2011 -0500
summary:
  With Carl's permission, swap PEPs 404 and 405.

files:
  pep-0404.txt |  647 ++------------------------------------
  pep-0405.txt |  661 ++++++++++++++++++++++++++++++++++++--
  2 files changed, 661 insertions(+), 647 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -1,643 +1,51 @@
-PEP: 404
-Title: Python Virtual Environments
+PEP: 405
+Title: Python 2.8 Un-release Schedule
 Version: $Revision$
 Last-Modified: $Date$
-Author: Carl Meyer <carl at oddbird.net>
-Status: Draft
-Type: Standards Track
+Author: Barry Warsaw <barry at python.org>
+Status: Final
+Type: Informational
 Content-Type: text/x-rst
-Created: 13-Jun-2011
-Python-Version: 3.3
-Post-History: 24-Oct-2011, 28-Oct-2011
+Created: 2011-11-09
+Python-Version: 2.8
 
 
 Abstract
 ========
 
-This PEP proposes to add to Python a mechanism for lightweight
-"virtual environments" with their own site directories, optionally
-isolated from system site directories.  Each virtual environment has
-its own Python binary (allowing creation of environments with various
-Python versions) and can have its own independent set of installed
-Python packages in its site directories, but shares the standard
-library with the base installed Python.
+This document describes the un-development and un-release schedule for Python
+2.8.
 
 
-Motivation
-==========
+Un-release Manager and Crew
+===========================
 
-The utility of Python virtual environments has already been well
-established by the popularity of existing third-party
-virtual-environment tools, primarily Ian Bicking's `virtualenv`_.
-Virtual environments are already widely used for dependency management
-and isolation, ease of installing and using Python packages without
-system-administrator access, and automated testing of Python software
-across multiple Python versions, among other uses.
+============================ ==================
+Position                     Name
+============================ ==================
+2.8 Un-release Manager       Cardinal Biggles
+============================ ==================
 
-Existing virtual environment tools suffer from lack of support from
-the behavior of Python itself.  Tools such as `rvirtualenv`_, which do
-not copy the Python binary into the virtual environment, cannot
-provide reliable isolation from system site directories.  Virtualenv,
-which does copy the Python binary, is forced to duplicate much of
-Python's ``site`` module and manually symlink/copy an ever-changing
-set of standard-library modules into the virtual environment in order
-to perform a delicate boot-strapping dance at every startup.
-(Virtualenv copies the binary because symlinking it does not provide
-isolation, as Python dereferences a symlinked executable before
-searching for ``sys.prefix``.)
 
-The ``PYTHONHOME`` environment variable, Python's only existing
-built-in solution for virtual environments, requires
-copying/symlinking the entire standard library into every environment.
-Copying the whole standard library is not a lightweight solution, and
-cross-platform support for symlinks remains inconsistent (even on
-Windows platforms that do support them, creating them often requires
-administrator privileges).
+Un-release Schedule
+===================
 
-A virtual environment mechanism integrated with Python and drawing on
-years of experience with existing third-party tools can be lower
-maintenance, more reliable, and more easily available to all Python
-users.
+The current un-schedule is:
 
-.. _virtualenv: http://www.virtualenv.org
+ - 2.8 final Never
 
-.. _rvirtualenv: https://github.com/kvbik/rvirtualenv
 
+Official pronouncement
+======================
 
-Specification
-=============
+Rule number six: there is *no* official Python 2.8 release.  There never will
+be an official Python 2.8 release.  It is an ex-release.
 
-When the Python binary is executed, it attempts to determine its
-prefix (which it stores in ``sys.prefix``), which is then used to find
-the standard library and other key files, and by the ``site`` module
-to determine the location of the site-package directories.  Currently
-the prefix is found (assuming ``PYTHONHOME`` is not set) by first
-walking up the filesystem tree looking for a marker file (``os.py``)
-that signifies the presence of the standard library, and if none is
-found, falling back to the build-time prefix hardcoded in the binary.
 
-This PEP proposes to add a new first step to this search.  If a
-``pyvenv.cfg`` file is found either adjacent to the Python executable,
-or one directory above it, this file is scanned for lines of the form
-``key = value``.  If a ``home`` key is found, this signifies that the
-Python binary belongs to a virtual environment, and the value of the
-``home`` key is the directory containing the Python executable used to
-create this virtual environment.
+Upgrade path
+============
 
-In this case, prefix-finding continues as normal using the value of
-the ``home`` key as the effective Python binary location, which
-results in ``sys.prefix`` being set to the system installation prefix,
-while ``sys.site_prefix`` is set to the directory containing
-``pyvenv.cfg``.
-
-(If ``pyvenv.cfg`` is not found or does not contain the ``home`` key,
-prefix-finding continues normally, and ``sys.site_prefix`` will be
-equal to ``sys.prefix``.)
-
-The ``site`` and ``sysconfig`` standard-library modules are modified
-such that site-package directories ("purelib" and "platlib", in
-``sysconfig`` terms) are found relative to ``sys.site_prefix``, while
-other directories (the standard library, include files) are still
-found relative to ``sys.prefix``.
-
-(Also, ``sys.site_exec_prefix`` is added, and handled similarly with
-regard to ``sys.exec_prefix``.)
-
-Thus, a Python virtual environment in its simplest form would consist
-of nothing more than a copy or symlink of the Python binary
-accompanied by a ``pyvenv.cfg`` file and a site-packages directory.
-
-
-Isolation from system site-packages
------------------------------------
-
-By default, a virtual environment is entirely isolated from the
-system-level site-packages directories.
-
-If the ``pyvenv.cfg`` file also contains a key
-``include-system-site-packages`` with a value of ``true`` (not case
-sensitive), the ``site`` module will also add the system site
-directories to ``sys.path`` after the virtual environment site
-directories.  Thus system-installed packages will still be importable,
-but a package of the same name installed in the virtual environment
-will take precedence.
-
-:pep:`370` user-level site-packages are considered part of the system
-site-packages for venv purposes: they are not available from an
-isolated venv, but are available from an
-``include-system-site-packages = true`` venv.
-
-
-Creating virtual environments
------------------------------
-
-This PEP also proposes adding a new ``venv`` module to the standard
-library which implements the creation of virtual environments.  This
-module can be executed using the ``-m`` flag::
-
-    python3 -m venv /path/to/new/virtual/environment
-
-A ``pyvenv`` installed script is also provided to make this more
-convenient::
-
-    pyvenv /path/to/new/virtual/environment
-
-Running this command creates the target directory (creating any parent
-directories that don't exist already) and places a ``pyvenv.cfg`` file
-in it with a ``home`` key pointing to the Python installation the
-command was run from.  It also creates a ``bin/`` (or ``Scripts`` on
-Windows) subdirectory containing a copy (or symlink) of the
-``python3`` executable, and the ``pysetup3`` script from the
-``packaging`` standard library module (to facilitate easy installation
-of packages from PyPI into the new virtualenv).  And it creates an
-(initially empty) ``lib/pythonX.Y/site-packages`` (or
-``Lib\site-packages`` on Windows) subdirectory.
-
-If the target directory already exists an error will be raised, unless
-the ``--clear`` option was provided, in which case the target
-directory will be deleted and virtual environment creation will
-proceed as usual.
-
-The created ``pyvenv.cfg`` file also includes the
-``include-system-site-packages`` key, set to ``true`` if ``venv`` is
-run with the ``--system-site-packages`` option, ``false`` by default.
-
-Multiple paths can be given to ``pyvenv``, in which case an identical
-virtualenv will be created, according to the given options, at each
-provided path.
-
-The ``venv`` module also adds a ``pysetup3`` script into each venv.
-In order to allow ``pysetup`` and other Python package managers to
-install packages into the virtual environment the same way they would
-install into a normal Python installation, and avoid special-casing
-virtual environments in ``sysconfig`` beyond using ``sys.site_prefix``
-in place of ``sys.prefix``, the internal virtual environment layout
-mimics the layout of the Python installation itself on each platform.
-So a typical virtual environment layout on a POSIX system would be::
-
-    pyvenv.cfg
-    bin/python3
-    bin/python
-    bin/pysetup3
-    lib/python3.3/site-packages/
-
-While on a Windows system::
-
-    pyvenv.cfg
-    Scripts/python.exe
-    Scripts/python3.dll
-    Scripts/pysetup3.exe
-    Scripts/pysetup3-script.py
-    ... other DLLs and pyds...
-    Lib/site-packages/
-
-Third-party packages installed into the virtual environment will have
-their Python modules placed in the ``site-packages`` directory, and
-their executables placed in ``bin/`` or ``Scripts\``.
-
-.. note::
-
-    On a normal Windows system-level installation, the Python binary
-    itself wouldn't go inside the "Scripts/" subdirectory, as it does
-    in the default venv layout.  This is useful in a virtual
-    environment so that a user only has to add a single directory to
-    their shell PATH in order to effectively "activate" the virtual
-    environment.
-
-.. note::
-
-    On Windows, it is necessary to also copy or symlink DLLs and pyd
-    files from compiled stdlib modules into the env, because if the
-    venv is created from a non-system-wide Python installation,
-    Windows won't be able to find the Python installation's copies of
-    those files when Python is run from the venv.
-
-
-Copies versus symlinks
-----------------------
-
-The technique in this PEP works equally well in general with a copied
-or symlinked Python binary (and other needed DLLs on Windows).  Some
-users prefer a copied binary (for greater isolation from system
-changes) and some prefer a symlinked one (so that e.g. security
-updates automatically propagate to virtual environments).
-
-There are some cross-platform difficulties with symlinks:
-
-* Not all Windows versions support symlinks, and even on those that
-  do, creating them often requires administrator privileges.
-
-* On OSX framework builds of Python, sys.executable is just a stub
-  that executes the real Python binary.  Symlinking this stub does not
-  work with the implementation in this PEP; it must be copied.
-  (Fortunately the stub is also small, so copying it is not an issue).
-
-Because of these issues, this PEP proposes to copy the Python binary
-by default, to maintain cross-platform consistency in the default
-behavior.
-
-The ``pyvenv`` script accepts a ``--symlink`` option.  If this option
-is provided, the script will attempt to symlink instead of copy.  If a
-symlink fails (e.g. because they are not supported by the platform, or
-additional privileges are needed), the script will warn the user and
-fall back to a copy.
-
-On OSX framework builds, where a symlink of the executable would
-succeed but create a non-functional virtual environment, the script
-will fail with an error message that symlinking is not supported on
-OSX framework builds.
-
-
-API
----
-
-The high-level method described above makes use of a simple API which
-provides mechanisms for third-party virtual environment creators to
-customize environment creation according to their needs.
-
-The ``venv`` module contains an ``EnvBuilder`` class which accepts the
-following keyword arguments on instantiation:
-
-* ``system_site_packages`` - A Boolean value indicating that the
-  system Python site-packages should be available to the environment.
-  Defaults to ``False``.
-
-* ``clear`` - A Boolean value which, if true, will delete any existing
-  target directory instead of raising an exception.  Defaults to
-  ``False``.
-
-* ``use_symlinks`` - A Boolean value indicating whether to attempt to
-  symlink the Python binary (and any necessary DLLs or other binaries,
-  e.g. ``pythonw.exe``), rather than copying.  Defaults to ``False``.
-
-The returned env-builder is an object with a ``create`` method, which
-takes as required argument the path (absolute or relative to the
-current directory) of the target directory which is to contain the
-virtual environment.  The ``create`` method either creates the
-environment in the specified directory, or raises an appropriate
-exception.
-
-Creators of third-party virtual environment tools are free to use the
-provided ``EnvBuilder`` class as a base class.
-
-The ``venv`` module also provides a module-level function as a
-convenience::
-
-    def create(env_dir,
-               system_site_packages=False, clear=False, use_symlinks=False):
-        builder = EnvBuilder(
-            system_site_packages=system_site_packages,
-            clear=clear,
-            use_symlinks=use_symlinks)
-        builder.create(env_dir)
-
-The ``create`` method of the ``EnvBuilder`` class illustrates the
-hooks available for customization::
-
-    def create(self, env_dir):
-        """
-        Create a virtualized Python environment in a directory.
-
-        :param env_dir: The target directory to create an environment in.
-
-        """
-        env_dir = os.path.abspath(env_dir)
-        context = self.create_directories(env_dir)
-        self.create_configuration(context)
-        self.setup_python(context)
-        self.post_setup(context)
-
-Each of the methods ``create_directories``, ``create_configuration``,
-``setup_python``, and ``post_setup`` can be overridden.  The functions
-of these methods are:
-
-* ``create_directories`` - creates the environment directory and all
-  necessary directories, and returns a context object.  This is just a
-  holder for attributes (such as paths), for use by the other methods.
-
-* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration
-  file in the environment.
-
-* ``setup_python`` - creates a copy of the Python executable (and,
-  under Windows, DLLs) in the environment.
-
-* ``post_setup`` - A (no-op by default) hook method which can be
-  overridden in third party implementations to pre-install packages or
-  install scripts in the virtual environment.
-
-In addition, ``EnvBuilder`` provides a utility method that can be
-called from ``post_setup`` in subclasses to assist in installing
-scripts into the virtual environment.  The method ``install_scripts``
-accepts as arguments the ``context`` object (see above) and a
-bytestring.  The bytestring should be a base64-encoded zip file
-containing directories "common", "posix", "nt", each containing
-scripts destined for the bin directory in the environment.  The
-contents of "common" and the directory corresponding to ``os.name``
-are copied after doing some text replacement of placeholders:
-
-* ``__VENV_DIR__`` is replaced with absolute path of the environment
-  directory.
-
-* ``__VENV_NAME__`` is replaced with the environment name (final path
-  segment of environment directory).
-
-* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
-  (either ``bin`` or ``Scripts``).
-
-* ``__VENV_PYTHON__`` is replaced with the absolute path of the
-  environment's executable.
-
-The ``DistributeEnvBuilder`` subclass in the reference implementation
-illustrates how the customization hook can be used in practice to
-pre-install Distribute and shell activation scripts into the virtual
-environment.  It's not envisaged that ``DistributeEnvBuilder`` will be
-actually added to Python core, but it makes the reference
-implementation more immediately useful for testing and exploratory
-purposes.
-
-The "shell activation scripts" provided by ``DistributeEnvBuilder``
-simply add the virtual environment's ``bin/`` (or ``Scripts\``)
-directory to the front of the user's shell PATH.  This is not strictly
-necessary for use of a virtual environment (as an explicit path to the
-venv's python binary or scripts can just as well be used), but it is
-convenient.
-
-This PEP does not propose that the ``venv`` module in core Python will
-add such activation scripts by default, as they are shell-specific.
-Adding activation scripts for the wide variety of possible shells is
-an added maintenance burden, and is left to third-party extension
-tools.
-
-No doubt the process of PEP review will show up any customization
-requirements which have not yet been considered.
-
-
-Backwards Compatibility
-=======================
-
-Splitting the meanings of ``sys.prefix``
-----------------------------------------
-
-Any virtual environment tool along these lines (which attempts to
-isolate site-packages, while still making use of the base Python's
-standard library with no need for it to be symlinked into the virtual
-environment) is proposing a split between two different meanings
-(among others) that are currently both wrapped up in ``sys.prefix``:
-the answers to the questions "Where is the standard library?" and
-"Where is the site-packages location where third-party modules should
-be installed?"
-
-This split could be handled by introducing a new ``sys`` attribute for
-either the former prefix or the latter prefix.  Either option
-potentially introduces some backwards-incompatibility with software
-written to assume the other meaning for ``sys.prefix``.  (Such
-software should preferably be using the APIs in the ``site`` and
-``sysconfig`` modules to answer these questions rather than using
-``sys.prefix`` directly, in which case there is no
-backwards-compatibility issue, but in practice ``sys.prefix`` is
-sometimes used.)
-
-The `documentation`__ for ``sys.prefix`` describes it as "A string
-giving the site-specific directory prefix where the platform
-independent Python files are installed," and specifically mentions the
-standard library and header files as found under ``sys.prefix``.  It
-does not mention ``site-packages``.
-
-__ http://docs.python.org/dev/library/sys.html#sys.prefix
-
-This PEP currently proposes to leave ``sys.prefix`` pointing to the
-base system installation (which is where the standard library and
-header files are found), and introduce a new value in ``sys``
-(``sys.site_prefix``) to point to the prefix for ``site-packages``.
-This maintains the documented semantics of ``sys.prefix``, but risks
-breaking isolation if third-party code uses ``sys.prefix`` rather than
-``sys.site_prefix`` or the appropriate ``site`` API to find
-site-packages directories.
-
-The most notable case is probably `setuptools`_ and its fork
-`distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but
-do use ``sys.prefix`` directly to build up a list of site directories
-for pre-flight checking where ``pth`` files can usefully be placed.
-It would be trivial to modify these tools (currently only
-`distribute`_ is Python 3 compatible) to check ``sys.site_prefix`` and
-fall back to ``sys.prefix`` if it doesn't exist (for earlier versions
-of Python).  If Distribute is modified in this way and released before
-Python 3.3 is released with the ``venv`` module, there would be no
-likely reason for an older version of Distribute to ever be installed
-in a virtual environment.
-
-In terms of other third-party usage, a `Google Code Search`_ turns up
-what appears to be a roughly even mix of usage between packages using
-``sys.prefix`` to build up a site-packages path and packages using it
-to e.g. eliminate the standard-library from code-execution tracing.
-Either choice that's made here will require one or the other of these
-uses to be updated.
-
-.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
-.. _distribute: http://packages.python.org/distribute/
-.. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs
-
-
-Open Questions
-==============
-
-Naming of the new ``sys`` prefix attributes
--------------------------------------------
-
-The name ``sys.site_prefix`` was chosen with the following
-considerations in mind:
-
-* Inasmuch as "site" has a meaning in Python, it means a combination
-  of Python version, standard library, and specific set of
-  site-packages.  This is, fundamentally, what a venv is (although it
-  shares the standard library with its "base" site).
-
-* It is the Python ``site`` module which implements adding
-  site-packages directories to ``sys.path``, so ``sys.site_prefix`` is
-  a prefix used (and set) primarily by the ``site`` module.
-
-A concern has been raised that the term ``site`` in Python is already
-overloaded and of unclear meaning, and this usage will increase the
-overload.
-
-One proposed alternative is ``sys.venv_prefix``, which has the
-advantage of being clearly related to the venv implementation.  The
-downside of this proposal is that it implies the attribute is only
-useful/relevant when in a venv and should be absent or ``None`` when
-not in a venv.  This imposes an unnecessary extra burden on code using
-the attribute: ``sys.venv_prefix if sys.venv_prefix else sys.prefix``.
-The prefix attributes are more usable and general if they are always
-present and set, and split by meaning (stdlib vs site-packages,
-roughly), rather than specifically tied to venv.  Also, third-party
-code should be encouraged to not know or care whether it is running in
-a virtual environment or not; this option seems to work against that
-goal.
-
-Another option would be ``sys.local_prefix``, which has both the
-advantage and disadvantage, depending on perspective, that it
-introduces the new term "local" rather than drawing on existing
-associations with the term "site".
-
-
-Why not modify sys.prefix?
---------------------------
-
-As discussed above under `Backwards Compatibility`_, this PEP proposes
-to add ``sys.site_prefix`` as "the prefix relative to which
-site-package directories are found".  This maintains compatibility
-with the documented meaning of ``sys.prefix`` (as the location
-relative to which the standard library can be found), but means that
-code assuming that site-packages directories are found relative to
-``sys.prefix`` will not respect the virtual environment correctly.
-
-Since it is unable to modify ``distutils``/``sysconfig``,
-`virtualenv`_ is forced to instead re-point ``sys.prefix`` at the
-virtual environment.
-
-An argument could be made that this PEP should follow virtualenv's
-lead here (and introduce something like ``sys.base_prefix`` to point
-to the standard library and header files), since virtualenv already
-does this and it doesn't appear to have caused major problems with
-existing code.
-
-Another argument in favor of this is that it would be preferable to
-err on the side of greater, rather than lesser, isolation.  Changing
-``sys.prefix`` to point to the virtual environment and introducing a
-new ``sys.base_prefix`` attribute would err on the side of greater
-isolation in the face of existing code's use of ``sys.prefix``.
-
-
-What about include files?
--------------------------
-
-For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in
-``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default
-in ``$VE/include/pythonX.Y``.  With virtualenv, everything works
-because the PythonX.Y include is symlinked, so everything that's
-needed is in ``$VE/include``.  At the moment the reference
-implementation doesn't do anything with include files, besides
-creating the include directory; this might need to change, to
-copy/symlink ``$VE/include/pythonX.Y``.
-
-As in Python there's no abstraction for a site-specific include
-directory, other than for platform-specific stuff, then the user
-expectation would seem to be that all include files anyone could ever
-want should be found in one of just two locations, with sysconfig
-labels "include" & "platinclude".
-
-There's another issue: what if includes are Python-version-specific?
-For example, SIP installs by default into ``$VE/include/pythonX.Y``
-rather than ``$VE/include``, presumably because there's
-version-specific stuff in there - but even if that's not the case with
-SIP, it could be the case with some other package.  And the problem
-that gives is that you can't just symlink the ``include/pythonX.Y``
-directory, but actually have to provide a writable directory and
-symlink/copy the contents from the system ``include/pythonX.Y``.  Of
-course this is not hard to do, but it does seem inelegant.  OTOH it's
-really because there's no supporting concept in ``Python/sysconfig``.
-
-
-Interface with packaging tools
-------------------------------
-
-Some work will be needed in packaging tools (Python 3.3 packaging,
-Distribute) to support implementation of this PEP.  For example:
-
-* How Distribute and packaging use ``sys.prefix`` and/or
-  ``sys.site_prefix``.  Clearly, in practice we'll need to use
-  Distribute for a while, until packages have migrated over to usage
-  of setup.cfg.
-
-* How packaging and Distribute set up shebang lines in scripts which they
-  install in virtual environments.
-
-
-Testability and Source Build Issues
------------------------------------
-
-Currently in the reference implementation, virtual environments must
-be created with an installed Python, rather than a source build, as
-the base installation.  In order to be able to fully test the ``venv``
-module in the Python regression test suite, some anomalies in how
-sysconfig data is configured in source builds will need to be removed.
-For example, ``sysconfig.get_paths()`` in a source build gives
-(partial output)::
-
-    {
-     'include': '/home/vinay/tools/pythonv/Include',
-     'libdir': '/usr/lib  ; or /usr/lib64 on a multilib system',
-     'platinclude': '/home/vinay/tools/pythonv',
-     'platlib': '/usr/local/lib/python3.3/site-packages',
-     'platstdlib': '/usr/local/lib/python3.3',
-     'purelib': '/usr/local/lib/python3.3/site-packages',
-     'stdlib': '/usr/local/lib/python3.3'
-    }
-
-
-Need for ``install_name_tool`` on OSX?
---------------------------------------
-
-`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode
-developer tools, to modify the copied executable on OSX.  We need
-input from OSX developers on whether this is actually necessary in
-this PEP's implementation of virtual environments, and if so, if there
-is an alternative to ``install_name_tool`` that would allow ``venv``
-to not require that Xcode is installed.
-
-.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168
-
-
-Activation and Utility Scripts
-------------------------------
-
-Virtualenv provides shell "activation" scripts as a user convenience,
-to put the virtual environment's Python binary first on the shell
-PATH.  This is a maintenance burden, as separate activation scripts
-need to be provided and maintained for every supported shell.  For
-this reason, this PEP proposes to leave such scripts to be provided by
-third-party extensions; virtual environments created by the core
-functionality would be used by directly invoking the environment's
-Python binary or scripts.
-
-If we are going to rely on external code to provide these
-conveniences, we need to check with existing third-party projects in
-this space (virtualenv, zc.buildout) and ensure that the proposed API
-meets their needs.
-
-(Virtualenv would be fine with the proposed API; it would become a
-relatively thin wrapper with a subclass of the env builder that adds
-shell activation and automatic installation of ``pip`` inside the
-virtual environment).
-
-
-Provide a mode that is isolated only from user site packages?
--------------------------------------------------------------
-
-Is there sufficient rationale for providing a mode that isolates the
-venv from :pep:`370` user site packages, but not from the system-level
-site-packages?
-
-
-Other Python implementations?
------------------------------
-
-We should get feedback from Jython, IronPython, and PyPy about whether
-there's anything in this PEP that they foresee as a difficulty for
-their implementation.
-
-
-Reference Implementation
-========================
-
-The in-progress reference implementation is found in `a clone of the
-CPython Mercurial repository`_.  To test it, build and install it (the
-virtual environment tool currently does not run from a source tree).
-From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv``
-to create a virtual environment.
-
-The reference implementation (like this PEP!) is a work in progress.
-
-.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv
+The official upgrade path from Python 2.7 is to Python 3.
 
 
 Copyright
@@ -648,11 +56,10 @@
 
 
 ..
-   Local Variables:
-   mode: indented-text
-   indent-tabs-mode: nil
-   sentence-end-double-space: t
-   fill-column: 70
-   coding: utf-8
-   End:
-
+  Local Variables:
+  mode: indented-text
+  indent-tabs-mode: nil
+  sentence-end-double-space: t
+  fill-column: 70
+  coding: utf-8
+  End:
diff --git a/pep-0405.txt b/pep-0405.txt
--- a/pep-0405.txt
+++ b/pep-0405.txt
@@ -1,51 +1,643 @@
-PEP: 405
-Title: Python 2.8 Un-release Schedule
+PEP: 404
+Title: Python Virtual Environments
 Version: $Revision$
 Last-Modified: $Date$
-Author: Barry Warsaw <barry at python.org>
-Status: Final
-Type: Informational
+Author: Carl Meyer <carl at oddbird.net>
+Status: Draft
+Type: Standards Track
 Content-Type: text/x-rst
-Created: 2011-11-09
-Python-Version: 2.8
+Created: 13-Jun-2011
+Python-Version: 3.3
+Post-History: 24-Oct-2011, 28-Oct-2011
 
 
 Abstract
 ========
 
-This document describes the un-development and un-release schedule for Python
-2.8.
+This PEP proposes to add to Python a mechanism for lightweight
+"virtual environments" with their own site directories, optionally
+isolated from system site directories.  Each virtual environment has
+its own Python binary (allowing creation of environments with various
+Python versions) and can have its own independent set of installed
+Python packages in its site directories, but shares the standard
+library with the base installed Python.
 
 
-Un-release Manager and Crew
-===========================
+Motivation
+==========
 
-============================ ==================
-Position                     Name
-============================ ==================
-2.8 Un-release Manager       Cardinal Biggles
-============================ ==================
+The utility of Python virtual environments has already been well
+established by the popularity of existing third-party
+virtual-environment tools, primarily Ian Bicking's `virtualenv`_.
+Virtual environments are already widely used for dependency management
+and isolation, ease of installing and using Python packages without
+system-administrator access, and automated testing of Python software
+across multiple Python versions, among other uses.
 
+Existing virtual environment tools suffer from lack of support from
+the behavior of Python itself.  Tools such as `rvirtualenv`_, which do
+not copy the Python binary into the virtual environment, cannot
+provide reliable isolation from system site directories.  Virtualenv,
+which does copy the Python binary, is forced to duplicate much of
+Python's ``site`` module and manually symlink/copy an ever-changing
+set of standard-library modules into the virtual environment in order
+to perform a delicate boot-strapping dance at every startup.
+(Virtualenv copies the binary because symlinking it does not provide
+isolation, as Python dereferences a symlinked executable before
+searching for ``sys.prefix``.)
 
-Un-release Schedule
-===================
+The ``PYTHONHOME`` environment variable, Python's only existing
+built-in solution for virtual environments, requires
+copying/symlinking the entire standard library into every environment.
+Copying the whole standard library is not a lightweight solution, and
+cross-platform support for symlinks remains inconsistent (even on
+Windows platforms that do support them, creating them often requires
+administrator privileges).
 
-The current un-schedule is:
+A virtual environment mechanism integrated with Python and drawing on
+years of experience with existing third-party tools can be lower
+maintenance, more reliable, and more easily available to all Python
+users.
 
- - 2.8 final Never
+.. _virtualenv: http://www.virtualenv.org
 
+.. _rvirtualenv: https://github.com/kvbik/rvirtualenv
 
-Official pronouncement
-======================
 
-Rule number six: there is *no* official Python 2.8 release.  There never will
-be an official Python 2.8 release.  It is an ex-release.
+Specification
+=============
 
+When the Python binary is executed, it attempts to determine its
+prefix (which it stores in ``sys.prefix``), which is then used to find
+the standard library and other key files, and by the ``site`` module
+to determine the location of the site-package directories.  Currently
+the prefix is found (assuming ``PYTHONHOME`` is not set) by first
+walking up the filesystem tree looking for a marker file (``os.py``)
+that signifies the presence of the standard library, and if none is
+found, falling back to the build-time prefix hardcoded in the binary.
 
-Upgrade path
-============
+This PEP proposes to add a new first step to this search.  If a
+``pyvenv.cfg`` file is found either adjacent to the Python executable,
+or one directory above it, this file is scanned for lines of the form
+``key = value``.  If a ``home`` key is found, this signifies that the
+Python binary belongs to a virtual environment, and the value of the
+``home`` key is the directory containing the Python executable used to
+create this virtual environment.
 
-The official upgrade path from Python 2.7 is to Python 3.
+In this case, prefix-finding continues as normal using the value of
+the ``home`` key as the effective Python binary location, which
+results in ``sys.prefix`` being set to the system installation prefix,
+while ``sys.site_prefix`` is set to the directory containing
+``pyvenv.cfg``.
+
+(If ``pyvenv.cfg`` is not found or does not contain the ``home`` key,
+prefix-finding continues normally, and ``sys.site_prefix`` will be
+equal to ``sys.prefix``.)
+
+The ``site`` and ``sysconfig`` standard-library modules are modified
+such that site-package directories ("purelib" and "platlib", in
+``sysconfig`` terms) are found relative to ``sys.site_prefix``, while
+other directories (the standard library, include files) are still
+found relative to ``sys.prefix``.
+
+(Also, ``sys.site_exec_prefix`` is added, and handled similarly with
+regard to ``sys.exec_prefix``.)
+
+Thus, a Python virtual environment in its simplest form would consist
+of nothing more than a copy or symlink of the Python binary
+accompanied by a ``pyvenv.cfg`` file and a site-packages directory.
+
+
+Isolation from system site-packages
+-----------------------------------
+
+By default, a virtual environment is entirely isolated from the
+system-level site-packages directories.
+
+If the ``pyvenv.cfg`` file also contains a key
+``include-system-site-packages`` with a value of ``true`` (not case
+sensitive), the ``site`` module will also add the system site
+directories to ``sys.path`` after the virtual environment site
+directories.  Thus system-installed packages will still be importable,
+but a package of the same name installed in the virtual environment
+will take precedence.
+
+:pep:`370` user-level site-packages are considered part of the system
+site-packages for venv purposes: they are not available from an
+isolated venv, but are available from an
+``include-system-site-packages = true`` venv.
+
+
+Creating virtual environments
+-----------------------------
+
+This PEP also proposes adding a new ``venv`` module to the standard
+library which implements the creation of virtual environments.  This
+module can be executed using the ``-m`` flag::
+
+    python3 -m venv /path/to/new/virtual/environment
+
+A ``pyvenv`` installed script is also provided to make this more
+convenient::
+
+    pyvenv /path/to/new/virtual/environment
+
+Running this command creates the target directory (creating any parent
+directories that don't exist already) and places a ``pyvenv.cfg`` file
+in it with a ``home`` key pointing to the Python installation the
+command was run from.  It also creates a ``bin/`` (or ``Scripts`` on
+Windows) subdirectory containing a copy (or symlink) of the
+``python3`` executable, and the ``pysetup3`` script from the
+``packaging`` standard library module (to facilitate easy installation
+of packages from PyPI into the new virtualenv).  And it creates an
+(initially empty) ``lib/pythonX.Y/site-packages`` (or
+``Lib\site-packages`` on Windows) subdirectory.
+
+If the target directory already exists an error will be raised, unless
+the ``--clear`` option was provided, in which case the target
+directory will be deleted and virtual environment creation will
+proceed as usual.
+
+The created ``pyvenv.cfg`` file also includes the
+``include-system-site-packages`` key, set to ``true`` if ``venv`` is
+run with the ``--system-site-packages`` option, ``false`` by default.
+
+Multiple paths can be given to ``pyvenv``, in which case an identical
+virtualenv will be created, according to the given options, at each
+provided path.
+
+The ``venv`` module also adds a ``pysetup3`` script into each venv.
+In order to allow ``pysetup`` and other Python package managers to
+install packages into the virtual environment the same way they would
+install into a normal Python installation, and avoid special-casing
+virtual environments in ``sysconfig`` beyond using ``sys.site_prefix``
+in place of ``sys.prefix``, the internal virtual environment layout
+mimics the layout of the Python installation itself on each platform.
+So a typical virtual environment layout on a POSIX system would be::
+
+    pyvenv.cfg
+    bin/python3
+    bin/python
+    bin/pysetup3
+    lib/python3.3/site-packages/
+
+While on a Windows system::
+
+    pyvenv.cfg
+    Scripts/python.exe
+    Scripts/python3.dll
+    Scripts/pysetup3.exe
+    Scripts/pysetup3-script.py
+    ... other DLLs and pyds...
+    Lib/site-packages/
+
+Third-party packages installed into the virtual environment will have
+their Python modules placed in the ``site-packages`` directory, and
+their executables placed in ``bin/`` or ``Scripts\``.
+
+.. note::
+
+    On a normal Windows system-level installation, the Python binary
+    itself wouldn't go inside the "Scripts/" subdirectory, as it does
+    in the default venv layout.  This is useful in a virtual
+    environment so that a user only has to add a single directory to
+    their shell PATH in order to effectively "activate" the virtual
+    environment.
+
+.. note::
+
+    On Windows, it is necessary to also copy or symlink DLLs and pyd
+    files from compiled stdlib modules into the env, because if the
+    venv is created from a non-system-wide Python installation,
+    Windows won't be able to find the Python installation's copies of
+    those files when Python is run from the venv.
+
+
+Copies versus symlinks
+----------------------
+
+The technique in this PEP works equally well in general with a copied
+or symlinked Python binary (and other needed DLLs on Windows).  Some
+users prefer a copied binary (for greater isolation from system
+changes) and some prefer a symlinked one (so that e.g. security
+updates automatically propagate to virtual environments).
+
+There are some cross-platform difficulties with symlinks:
+
+* Not all Windows versions support symlinks, and even on those that
+  do, creating them often requires administrator privileges.
+
+* On OSX framework builds of Python, sys.executable is just a stub
+  that executes the real Python binary.  Symlinking this stub does not
+  work with the implementation in this PEP; it must be copied.
+  (Fortunately the stub is also small, so copying it is not an issue).
+
+Because of these issues, this PEP proposes to copy the Python binary
+by default, to maintain cross-platform consistency in the default
+behavior.
+
+The ``pyvenv`` script accepts a ``--symlink`` option.  If this option
+is provided, the script will attempt to symlink instead of copy.  If a
+symlink fails (e.g. because they are not supported by the platform, or
+additional privileges are needed), the script will warn the user and
+fall back to a copy.
+
+On OSX framework builds, where a symlink of the executable would
+succeed but create a non-functional virtual environment, the script
+will fail with an error message that symlinking is not supported on
+OSX framework builds.
+
+
+API
+---
+
+The high-level method described above makes use of a simple API which
+provides mechanisms for third-party virtual environment creators to
+customize environment creation according to their needs.
+
+The ``venv`` module contains an ``EnvBuilder`` class which accepts the
+following keyword arguments on instantiation:
+
+* ``system_site_packages`` - A Boolean value indicating that the
+  system Python site-packages should be available to the environment.
+  Defaults to ``False``.
+
+* ``clear`` - A Boolean value which, if true, will delete any existing
+  target directory instead of raising an exception.  Defaults to
+  ``False``.
+
+* ``use_symlinks`` - A Boolean value indicating whether to attempt to
+  symlink the Python binary (and any necessary DLLs or other binaries,
+  e.g. ``pythonw.exe``), rather than copying.  Defaults to ``False``.
+
+The returned env-builder is an object with a ``create`` method, which
+takes as required argument the path (absolute or relative to the
+current directory) of the target directory which is to contain the
+virtual environment.  The ``create`` method either creates the
+environment in the specified directory, or raises an appropriate
+exception.
+
+Creators of third-party virtual environment tools are free to use the
+provided ``EnvBuilder`` class as a base class.
+
+The ``venv`` module also provides a module-level function as a
+convenience::
+
+    def create(env_dir,
+               system_site_packages=False, clear=False, use_symlinks=False):
+        builder = EnvBuilder(
+            system_site_packages=system_site_packages,
+            clear=clear,
+            use_symlinks=use_symlinks)
+        builder.create(env_dir)
+
+The ``create`` method of the ``EnvBuilder`` class illustrates the
+hooks available for customization::
+
+    def create(self, env_dir):
+        """
+        Create a virtualized Python environment in a directory.
+
+        :param env_dir: The target directory to create an environment in.
+
+        """
+        env_dir = os.path.abspath(env_dir)
+        context = self.create_directories(env_dir)
+        self.create_configuration(context)
+        self.setup_python(context)
+        self.post_setup(context)
+
+Each of the methods ``create_directories``, ``create_configuration``,
+``setup_python``, and ``post_setup`` can be overridden.  The functions
+of these methods are:
+
+* ``create_directories`` - creates the environment directory and all
+  necessary directories, and returns a context object.  This is just a
+  holder for attributes (such as paths), for use by the other methods.
+
+* ``create_configuration`` - creates the ``pyvenv.cfg`` configuration
+  file in the environment.
+
+* ``setup_python`` - creates a copy of the Python executable (and,
+  under Windows, DLLs) in the environment.
+
+* ``post_setup`` - A (no-op by default) hook method which can be
+  overridden in third party implementations to pre-install packages or
+  install scripts in the virtual environment.
+
+In addition, ``EnvBuilder`` provides a utility method that can be
+called from ``post_setup`` in subclasses to assist in installing
+scripts into the virtual environment.  The method ``install_scripts``
+accepts as arguments the ``context`` object (see above) and a
+bytestring.  The bytestring should be a base64-encoded zip file
+containing directories "common", "posix", "nt", each containing
+scripts destined for the bin directory in the environment.  The
+contents of "common" and the directory corresponding to ``os.name``
+are copied after doing some text replacement of placeholders:
+
+* ``__VENV_DIR__`` is replaced with absolute path of the environment
+  directory.
+
+* ``__VENV_NAME__`` is replaced with the environment name (final path
+  segment of environment directory).
+
+* ``__VENV_BIN_NAME__`` is replaced with the name of the bin directory
+  (either ``bin`` or ``Scripts``).
+
+* ``__VENV_PYTHON__`` is replaced with the absolute path of the
+  environment's executable.
+
+The ``DistributeEnvBuilder`` subclass in the reference implementation
+illustrates how the customization hook can be used in practice to
+pre-install Distribute and shell activation scripts into the virtual
+environment.  It's not envisaged that ``DistributeEnvBuilder`` will be
+actually added to Python core, but it makes the reference
+implementation more immediately useful for testing and exploratory
+purposes.
+
+The "shell activation scripts" provided by ``DistributeEnvBuilder``
+simply add the virtual environment's ``bin/`` (or ``Scripts\``)
+directory to the front of the user's shell PATH.  This is not strictly
+necessary for use of a virtual environment (as an explicit path to the
+venv's python binary or scripts can just as well be used), but it is
+convenient.
+
+This PEP does not propose that the ``venv`` module in core Python will
+add such activation scripts by default, as they are shell-specific.
+Adding activation scripts for the wide variety of possible shells is
+an added maintenance burden, and is left to third-party extension
+tools.
+
+No doubt the process of PEP review will show up any customization
+requirements which have not yet been considered.
+
+
+Backwards Compatibility
+=======================
+
+Splitting the meanings of ``sys.prefix``
+----------------------------------------
+
+Any virtual environment tool along these lines (which attempts to
+isolate site-packages, while still making use of the base Python's
+standard library with no need for it to be symlinked into the virtual
+environment) is proposing a split between two different meanings
+(among others) that are currently both wrapped up in ``sys.prefix``:
+the answers to the questions "Where is the standard library?" and
+"Where is the site-packages location where third-party modules should
+be installed?"
+
+This split could be handled by introducing a new ``sys`` attribute for
+either the former prefix or the latter prefix.  Either option
+potentially introduces some backwards-incompatibility with software
+written to assume the other meaning for ``sys.prefix``.  (Such
+software should preferably be using the APIs in the ``site`` and
+``sysconfig`` modules to answer these questions rather than using
+``sys.prefix`` directly, in which case there is no
+backwards-compatibility issue, but in practice ``sys.prefix`` is
+sometimes used.)
+
+The `documentation`__ for ``sys.prefix`` describes it as "A string
+giving the site-specific directory prefix where the platform
+independent Python files are installed," and specifically mentions the
+standard library and header files as found under ``sys.prefix``.  It
+does not mention ``site-packages``.
+
+__ http://docs.python.org/dev/library/sys.html#sys.prefix
+
+This PEP currently proposes to leave ``sys.prefix`` pointing to the
+base system installation (which is where the standard library and
+header files are found), and introduce a new value in ``sys``
+(``sys.site_prefix``) to point to the prefix for ``site-packages``.
+This maintains the documented semantics of ``sys.prefix``, but risks
+breaking isolation if third-party code uses ``sys.prefix`` rather than
+``sys.site_prefix`` or the appropriate ``site`` API to find
+site-packages directories.
+
+The most notable case is probably `setuptools`_ and its fork
+`distribute`_, which mostly use ``distutils``/``sysconfig`` APIs, but
+do use ``sys.prefix`` directly to build up a list of site directories
+for pre-flight checking where ``pth`` files can usefully be placed.
+It would be trivial to modify these tools (currently only
+`distribute`_ is Python 3 compatible) to check ``sys.site_prefix`` and
+fall back to ``sys.prefix`` if it doesn't exist (for earlier versions
+of Python).  If Distribute is modified in this way and released before
+Python 3.3 is released with the ``venv`` module, there would be no
+likely reason for an older version of Distribute to ever be installed
+in a virtual environment.
+
+In terms of other third-party usage, a `Google Code Search`_ turns up
+what appears to be a roughly even mix of usage between packages using
+``sys.prefix`` to build up a site-packages path and packages using it
+to e.g. eliminate the standard-library from code-execution tracing.
+Either choice that's made here will require one or the other of these
+uses to be updated.
+
+.. _setuptools: http://peak.telecommunity.com/DevCenter/setuptools
+.. _distribute: http://packages.python.org/distribute/
+.. _Google Code Search: http://www.google.com/codesearch#search/&q=sys\.prefix&p=1&type=cs
+
+
+Open Questions
+==============
+
+Naming of the new ``sys`` prefix attributes
+-------------------------------------------
+
+The name ``sys.site_prefix`` was chosen with the following
+considerations in mind:
+
+* Inasmuch as "site" has a meaning in Python, it means a combination
+  of Python version, standard library, and specific set of
+  site-packages.  This is, fundamentally, what a venv is (although it
+  shares the standard library with its "base" site).
+
+* It is the Python ``site`` module which implements adding
+  site-packages directories to ``sys.path``, so ``sys.site_prefix`` is
+  a prefix used (and set) primarily by the ``site`` module.
+
+A concern has been raised that the term ``site`` in Python is already
+overloaded and of unclear meaning, and this usage will increase the
+overload.
+
+One proposed alternative is ``sys.venv_prefix``, which has the
+advantage of being clearly related to the venv implementation.  The
+downside of this proposal is that it implies the attribute is only
+useful/relevant when in a venv and should be absent or ``None`` when
+not in a venv.  This imposes an unnecessary extra burden on code using
+the attribute: ``sys.venv_prefix if sys.venv_prefix else sys.prefix``.
+The prefix attributes are more usable and general if they are always
+present and set, and split by meaning (stdlib vs site-packages,
+roughly), rather than specifically tied to venv.  Also, third-party
+code should be encouraged to not know or care whether it is running in
+a virtual environment or not; this option seems to work against that
+goal.
+
+Another option would be ``sys.local_prefix``, which has both the
+advantage and disadvantage, depending on perspective, that it
+introduces the new term "local" rather than drawing on existing
+associations with the term "site".
+
+
+Why not modify sys.prefix?
+--------------------------
+
+As discussed above under `Backwards Compatibility`_, this PEP proposes
+to add ``sys.site_prefix`` as "the prefix relative to which
+site-package directories are found".  This maintains compatibility
+with the documented meaning of ``sys.prefix`` (as the location
+relative to which the standard library can be found), but means that
+code assuming that site-packages directories are found relative to
+``sys.prefix`` will not respect the virtual environment correctly.
+
+Since it is unable to modify ``distutils``/``sysconfig``,
+`virtualenv`_ is forced to instead re-point ``sys.prefix`` at the
+virtual environment.
+
+An argument could be made that this PEP should follow virtualenv's
+lead here (and introduce something like ``sys.base_prefix`` to point
+to the standard library and header files), since virtualenv already
+does this and it doesn't appear to have caused major problems with
+existing code.
+
+Another argument in favor of this is that it would be preferable to
+err on the side of greater, rather than lesser, isolation.  Changing
+``sys.prefix`` to point to the virtual environment and introducing a
+new ``sys.base_prefix`` attribute would err on the side of greater
+isolation in the face of existing code's use of ``sys.prefix``.
+
+
+What about include files?
+-------------------------
+
+For example, ZeroMQ installs ``zmq.h`` and ``zmq_utils.h`` in
+``$VE/include``, whereas SIP (part of PyQt4) installs sip.h by default
+in ``$VE/include/pythonX.Y``.  With virtualenv, everything works
+because the PythonX.Y include is symlinked, so everything that's
+needed is in ``$VE/include``.  At the moment the reference
+implementation doesn't do anything with include files, besides
+creating the include directory; this might need to change, to
+copy/symlink ``$VE/include/pythonX.Y``.
+
+As in Python there's no abstraction for a site-specific include
+directory, other than for platform-specific stuff, then the user
+expectation would seem to be that all include files anyone could ever
+want should be found in one of just two locations, with sysconfig
+labels "include" & "platinclude".
+
+There's another issue: what if includes are Python-version-specific?
+For example, SIP installs by default into ``$VE/include/pythonX.Y``
+rather than ``$VE/include``, presumably because there's
+version-specific stuff in there - but even if that's not the case with
+SIP, it could be the case with some other package.  And the problem
+that gives is that you can't just symlink the ``include/pythonX.Y``
+directory, but actually have to provide a writable directory and
+symlink/copy the contents from the system ``include/pythonX.Y``.  Of
+course this is not hard to do, but it does seem inelegant.  OTOH it's
+really because there's no supporting concept in ``Python/sysconfig``.
+
+
+Interface with packaging tools
+------------------------------
+
+Some work will be needed in packaging tools (Python 3.3 packaging,
+Distribute) to support implementation of this PEP.  For example:
+
+* How Distribute and packaging use ``sys.prefix`` and/or
+  ``sys.site_prefix``.  Clearly, in practice we'll need to use
+  Distribute for a while, until packages have migrated over to usage
+  of setup.cfg.
+
+* How packaging and Distribute set up shebang lines in scripts which they
+  install in virtual environments.
+
+
+Testability and Source Build Issues
+-----------------------------------
+
+Currently in the reference implementation, virtual environments must
+be created with an installed Python, rather than a source build, as
+the base installation.  In order to be able to fully test the ``venv``
+module in the Python regression test suite, some anomalies in how
+sysconfig data is configured in source builds will need to be removed.
+For example, ``sysconfig.get_paths()`` in a source build gives
+(partial output)::
+
+    {
+     'include': '/home/vinay/tools/pythonv/Include',
+     'libdir': '/usr/lib  ; or /usr/lib64 on a multilib system',
+     'platinclude': '/home/vinay/tools/pythonv',
+     'platlib': '/usr/local/lib/python3.3/site-packages',
+     'platstdlib': '/usr/local/lib/python3.3',
+     'purelib': '/usr/local/lib/python3.3/site-packages',
+     'stdlib': '/usr/local/lib/python3.3'
+    }
+
+
+Need for ``install_name_tool`` on OSX?
+--------------------------------------
+
+`Virtualenv uses`_ ``install_name_tool``, a tool provided in the Xcode
+developer tools, to modify the copied executable on OSX.  We need
+input from OSX developers on whether this is actually necessary in
+this PEP's implementation of virtual environments, and if so, if there
+is an alternative to ``install_name_tool`` that would allow ``venv``
+to not require that Xcode is installed.
+
+.. _Virtualenv uses: https://github.com/pypa/virtualenv/issues/168
+
+
+Activation and Utility Scripts
+------------------------------
+
+Virtualenv provides shell "activation" scripts as a user convenience,
+to put the virtual environment's Python binary first on the shell
+PATH.  This is a maintenance burden, as separate activation scripts
+need to be provided and maintained for every supported shell.  For
+this reason, this PEP proposes to leave such scripts to be provided by
+third-party extensions; virtual environments created by the core
+functionality would be used by directly invoking the environment's
+Python binary or scripts.
+
+If we are going to rely on external code to provide these
+conveniences, we need to check with existing third-party projects in
+this space (virtualenv, zc.buildout) and ensure that the proposed API
+meets their needs.
+
+(Virtualenv would be fine with the proposed API; it would become a
+relatively thin wrapper with a subclass of the env builder that adds
+shell activation and automatic installation of ``pip`` inside the
+virtual environment).
+
+
+Provide a mode that is isolated only from user site packages?
+-------------------------------------------------------------
+
+Is there sufficient rationale for providing a mode that isolates the
+venv from :pep:`370` user site packages, but not from the system-level
+site-packages?
+
+
+Other Python implementations?
+-----------------------------
+
+We should get feedback from Jython, IronPython, and PyPy about whether
+there's anything in this PEP that they foresee as a difficulty for
+their implementation.
+
+
+Reference Implementation
+========================
+
+The in-progress reference implementation is found in `a clone of the
+CPython Mercurial repository`_.  To test it, build and install it (the
+virtual environment tool currently does not run from a source tree).
+From the installed Python, run ``bin/pyvenv /path/to/new/virtualenv``
+to create a virtual environment.
+
+The reference implementation (like this PEP!) is a work in progress.
+
+.. _a clone of the CPython Mercurial repository: https://bitbucket.org/vinay.sajip/pythonv
 
 
 Copyright
@@ -56,10 +648,11 @@
 
 
 ..
-  Local Variables:
-  mode: indented-text
-  indent-tabs-mode: nil
-  sentence-end-double-space: t
-  fill-column: 70
-  coding: utf-8
-  End:
+   Local Variables:
+   mode: indented-text
+   indent-tabs-mode: nil
+   sentence-end-double-space: t
+   fill-column: 70
+   coding: utf-8
+   End:
+

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sun Nov 13 18:00:03 2011
From: python-checkins at python.org (barry.warsaw)
Date: Sun, 13 Nov 2011 18:00:03 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_PEP_numbers=2E?=
Message-ID: <E1RPdPX-00004s-Cw@dinsdale.python.org>

http://hg.python.org/peps/rev/1346b214ab8a
changeset:   3986:1346b214ab8a
user:        Barry Warsaw <barry at python.org>
date:        Sun Nov 13 11:59:59 2011 -0500
summary:
  Fix PEP numbers.

files:
  pep-0404.txt |  2 +-
  pep-0405.txt |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -1,4 +1,4 @@
-PEP: 405
+PEP: 404
 Title: Python 2.8 Un-release Schedule
 Version: $Revision$
 Last-Modified: $Date$
diff --git a/pep-0405.txt b/pep-0405.txt
--- a/pep-0405.txt
+++ b/pep-0405.txt
@@ -1,4 +1,4 @@
-PEP: 404
+PEP: 405
 Title: Python Virtual Environments
 Version: $Revision$
 Last-Modified: $Date$

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sun Nov 13 19:42:37 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sun, 13 Nov 2011 19:42:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313217=3A_add_missi?=
 =?utf8?q?ng_header_dependencies_in_the_Makefile_for?=
Message-ID: <E1RPf0n-0007L7-Ut@dinsdale.python.org>

http://hg.python.org/cpython/rev/36375075d6aa
changeset:   73540:36375075d6aa
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sun Nov 13 19:37:58 2011 +0100
summary:
  Issue #13217: add missing header dependencies in the Makefile for unicodeobject.o.
Patch by John O'Connor.

files:
  Makefile.pre.in |  14 +++++++++++---
  1 files changed, 11 insertions(+), 3 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -640,14 +640,22 @@
 		$(srcdir)/Objects/stringlib/stringdefs.h \
 		$(srcdir)/Objects/stringlib/transmogrify.h \
 		$(srcdir)/Objects/stringlib/unicodedefs.h \
-		$(srcdir)/Objects/stringlib/localeutil.h
+		$(srcdir)/Objects/stringlib/localeutil.h \
+		$(srcdir)/Objects/stringlib/undef.h
+
+UNICODE_DEPS = $(BYTESTR_DEPS) \
+		$(srcdir)/Objects/stringlib/asciilib.h \
+		$(srcdir)/Objects/stringlib/ucs1lib.h \
+		$(srcdir)/Objects/stringlib/ucs2lib.h \
+		$(srcdir)/Objects/stringlib/ucs4lib.h \
+		$(srcdir)/Objects/stringlib/unicode_format.h \
+		$(srcdir)/Objects/stringlib/unicodedefs.h
 
 Objects/bytesobject.o: $(srcdir)/Objects/bytesobject.c $(BYTESTR_DEPS)
 
 Objects/bytearrayobject.o: $(srcdir)/Objects/bytearrayobject.c $(BYTESTR_DEPS)
 
-Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c \
-				$(BYTESTR_DEPS) $(srcdir)/Objects/stringlib/unicode_format.h
+Objects/unicodeobject.o: $(srcdir)/Objects/unicodeobject.c $(UNICODE_DEPS)
 
 Objects/dictobject.o: $(srcdir)/Objects/stringlib/eq.h
 Objects/setobject.o: $(srcdir)/Objects/stringlib/eq.h

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 13 22:33:28 2011
From: python-checkins at python.org (barry.warsaw)
Date: Sun, 13 Nov 2011 22:33:28 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_And_now_for_something_complete?=
	=?utf8?q?ly_different=2E?=
Message-ID: <E1RPhg8-0002fd-0T@dinsdale.python.org>

http://hg.python.org/peps/rev/804b6389c282
changeset:   3987:804b6389c282
user:        Barry Warsaw <barry at python.org>
date:        Sun Nov 13 16:33:25 2011 -0500
summary:
  And now for something completely different.

files:
  pep-0404.txt |  102 ++++++++++++++++++++++++++++++++++++++-
  1 files changed, 101 insertions(+), 1 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -39,7 +39,8 @@
 ======================
 
 Rule number six: there is *no* official Python 2.8 release.  There never will
-be an official Python 2.8 release.  It is an ex-release.
+be an official Python 2.8 release.  It is an ex-release.  Python 2.7
+is the end of the Python 2 line of development.
 
 
 Upgrade path
@@ -48,12 +49,111 @@
 The official upgrade path from Python 2.7 is to Python 3.
 
 
+And Now For Something Completely Different
+==========================================
+
+In all seriousness, there are important reasons why there won't be an
+official Python 2.8 release, and why you should plan to migrate
+instead to Python 3.
+
+Python is (as of this writing) nearly 20 years old, and Guido and the
+community has learned a lot in those intervening years.  Guido's
+original concept for Python 3 was to make changes to the language
+primarily to remove the warts that had grown in the preceding
+versions.  Python 3 was not to be a complete redesign, but instead an
+evolution of the language, and while maintaining backward
+compatibility with Python 2 was explicitly off-the-table, neither were
+gratuitous changes in syntax or semantics acceptable.  In most cases,
+Python 2 code can be translated fairly easily to Python 3, sometimes
+entirely mechanically by such tools as `2to3`_.
+
+Because maintaining multiple versions of Python is a significant drag
+on the resources of the Python developers, and because the
+improvements to the language and libraries embodied in Python 3 are so
+important, it was decided to end the Python 2 lineage with Python
+2.7.  Thus, all new development occurs in the Python 3 line of
+development, and there will never be an official Python 2.8 release.
+Python 2.7 will however be maintained for longer than the usual period
+of time.
+
+Here are some highlights of the significant improvements in Python 3.
+You can read in more detail on the differences_ between Python 2 and
+Python 3.  There are also many good guides on porting_ from Python 2
+to Python 3.
+
+
+Strings and bytes
+-----------------
+
+Python 2's basic original string type are called 8-bit strings, and
+they play a dual role in Python 2 as both ASCII text and as byte
+arrays.  While Python 2 also has a unicode string type, the
+fundamental ambiguity of the core string type, coupled with Python 2's
+default behavior of supporting automatic coercion from 8-bit strings
+to unicodes when the two are combined, often leads to `UnicodeError`s.
+Python 3's standard string type is a unicode, and Python 3 adds a
+bytes type, but critically, no automatic coercion between bytes and
+unicodes is provided.  Thus, the core interpreter, its I/O libraries,
+module names, etc. are clear in their distinction between unicode
+strings and bytes.  This clarity is often a source of difficulty in
+transitioning existing code to Python 3, because many third party
+libraries and applications are themselves ambiguous in this
+distinction.  Once migrated though, most `UnicodeError`s can be
+eliminated.
+
+
+Numbers
+-------
+
+Python 2 has two basic integer types, a native machine-sized `int`
+type, and an arbitrary length `long` type.  These have been merged in
+Python 3 into a single `int` type analogous to Python 2's `long`
+type.
+
+In addition, integer division now produces floating point numbers for
+non-integer results.
+
+
+Multiple spellings
+------------------
+
+There are many cases in Python 2 where multiple spellings of some
+constructs exist, such as `repr()` and *backticks*, or the two
+inequality operators `!=` and `<>`.  In all cases, Python 3 has chosen
+exactly one spelling and removed the other (e.g. `repr()` and `!=`
+were kept).
+
+
+Imports
+-------
+
+In Python 3, star imports (e.g. ``from x import *``) are only
+premitted in module level code.  Also, only absolute imports are
+supported.
+
+Also, some areas of the standard library have been reorganized to make
+the naming scheme more intuitive.  Some rarely used builtins have been
+relocated to standard library modules.
+
+
+Iterators and views
+-------------------
+
+Many APIs, which in Python 2 returned concrete lists, in Python 3 now
+return iterators or lightweight *views*.
+
+
 Copyright
 =========
 
 This document has been placed in the public domain.
 
 
+.. _`2to3`: http://docs.python.org/library/2to3.html
+.. _differences: http://docs.python.org/release/3.0.1/whatsnew/3.0.html
+.. _porting: http://python3porting.com/
+
+
 
 ..
   Local Variables:

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sun Nov 13 23:52:00 2011
From: python-checkins at python.org (eli.bendersky)
Date: Sun, 13 Nov 2011 23:52:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Normalize_the_k?=
 =?utf8?q?eyword_arguments_documentation_notation_in_re=2Erst=2E_Closes_is?=
 =?utf8?q?sue?=
Message-ID: <E1RPiu8-0008Ve-IQ@dinsdale.python.org>

http://hg.python.org/cpython/rev/87ecfd5cd5d1
changeset:   73541:87ecfd5cd5d1
branch:      2.7
parent:      73529:c3b063c82ae5
user:        Eli Bendersky <eliben at gmail.com>
date:        Mon Nov 14 01:02:20 2011 +0200
summary:
  Normalize the keyword arguments documentation notation in re.rst. Closes issue #12875

files:
  Doc/library/re.rst |  22 +++++++++++-----------
  1 files changed, 11 insertions(+), 11 deletions(-)


diff --git a/Doc/library/re.rst b/Doc/library/re.rst
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -431,7 +431,7 @@
 form.
 
 
-.. function:: compile(pattern[, flags=0])
+.. function:: compile(pattern, flags=0)
 
    Compile a regular expression pattern into a regular expression object, which
    can be used for matching using its :func:`match` and :func:`search` methods,
@@ -521,7 +521,7 @@
       b = re.compile(r"\d+\.\d*")
 
 
-.. function:: search(pattern, string[, flags=0])
+.. function:: search(pattern, string, flags=0)
 
    Scan through *string* looking for a location where the regular expression
    *pattern* produces a match, and return a corresponding :class:`MatchObject`
@@ -530,7 +530,7 @@
    string.
 
 
-.. function:: match(pattern, string[, flags=0])
+.. function:: match(pattern, string, flags=0)
 
    If zero or more characters at the beginning of *string* match the regular
    expression *pattern*, return a corresponding :class:`MatchObject` instance.
@@ -543,7 +543,7 @@
       instead.
 
 
-.. function:: split(pattern, string[, maxsplit=0, flags=0])
+.. function:: split(pattern, string, maxsplit=0, flags=0)
 
    Split *string* by the occurrences of *pattern*.  If capturing parentheses are
    used in *pattern*, then the text of all groups in the pattern are also returned
@@ -584,7 +584,7 @@
       Added the optional flags argument.
 
 
-.. function:: findall(pattern, string[, flags=0])
+.. function:: findall(pattern, string, flags=0)
 
    Return all non-overlapping matches of *pattern* in *string*, as a list of
    strings.  The *string* is scanned left-to-right, and matches are returned in
@@ -599,7 +599,7 @@
       Added the optional flags argument.
 
 
-.. function:: finditer(pattern, string[, flags=0])
+.. function:: finditer(pattern, string, flags=0)
 
    Return an :term:`iterator` yielding :class:`MatchObject` instances over all
    non-overlapping matches for the RE *pattern* in *string*.  The *string* is
@@ -613,7 +613,7 @@
       Added the optional flags argument.
 
 
-.. function:: sub(pattern, repl, string[, count, flags=0])
+.. function:: sub(pattern, repl, string, count=0, flags=0)
 
    Return the string obtained by replacing the leftmost non-overlapping occurrences
    of *pattern* in *string* by the replacement *repl*.  If the pattern isn't found,
@@ -662,7 +662,7 @@
       Added the optional flags argument.
 
 
-.. function:: subn(pattern, repl, string[, count, flags=0])
+.. function:: subn(pattern, repl, string, count=0, flags=0)
 
    Perform the same operation as :func:`sub`, but return a tuple ``(new_string,
    number_of_subs_made)``.
@@ -747,7 +747,7 @@
       <_sre.SRE_Match object at ...>
 
 
-   .. method:: RegexObject.split(string[, maxsplit=0])
+   .. method:: RegexObject.split(string, maxsplit=0)
 
       Identical to the :func:`split` function, using the compiled pattern.
 
@@ -766,12 +766,12 @@
       region like for :meth:`match`.
 
 
-   .. method:: RegexObject.sub(repl, string[, count=0])
+   .. method:: RegexObject.sub(repl, string, count=0)
 
       Identical to the :func:`sub` function, using the compiled pattern.
 
 
-   .. method:: RegexObject.subn(repl, string[, count=0])
+   .. method:: RegexObject.subn(repl, string, count=0)
 
       Identical to the :func:`subn` function, using the compiled pattern.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 00:08:12 2011
From: python-checkins at python.org (eli.bendersky)
Date: Mon, 14 Nov 2011 00:08:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Clarify_the_exi?=
 =?utf8?q?stence_of_the_=3C=3E_operator_in_Grammar/Grammar_with_a_comment?=
 =?utf8?q?=2C_for?=
Message-ID: <E1RPj9o-00018j-U2@dinsdale.python.org>

http://hg.python.org/cpython/rev/a259511351d9
changeset:   73542:a259511351d9
branch:      3.2
parent:      73536:11f92e6d8871
user:        Eli Bendersky <eliben at gmail.com>
date:        Mon Nov 14 01:16:31 2011 +0200
summary:
  Clarify the existence of the <> operator in Grammar/Grammar with a comment, for issue 13239

files:
  Grammar/Grammar |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Grammar/Grammar b/Grammar/Grammar
--- a/Grammar/Grammar
+++ b/Grammar/Grammar
@@ -88,6 +88,8 @@
 and_test: not_test ('and' not_test)*
 not_test: 'not' not_test | comparison
 comparison: expr (comp_op expr)*
+# <> isn't actually a valid comparison operator in Python. It's here for the
+# sake of a __future__ import described in PEP 401
 comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'
 star_expr: '*' expr
 expr: xor_expr ('|' xor_expr)*

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 00:08:13 2011
From: python-checkins at python.org (eli.bendersky)
Date: Mon, 14 Nov 2011 00:08:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Clarify_the_existence_of_the_=3C=3E_operator_in_Grammar/Gram?=
 =?utf8?q?mar_with_a_comment=2E?=
Message-ID: <E1RPj9p-00018o-Lh@dinsdale.python.org>

http://hg.python.org/cpython/rev/410115400838
changeset:   73543:410115400838
parent:      73540:36375075d6aa
parent:      73542:a259511351d9
user:        Eli Bendersky <eliben at gmail.com>
date:        Mon Nov 14 01:18:24 2011 +0200
summary:
  Clarify the existence of the <> operator in Grammar/Grammar with a comment. Closes issue 13239

files:
  Grammar/Grammar |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Grammar/Grammar b/Grammar/Grammar
--- a/Grammar/Grammar
+++ b/Grammar/Grammar
@@ -88,6 +88,8 @@
 and_test: not_test ('and' not_test)*
 not_test: 'not' not_test | comparison
 comparison: expr (comp_op expr)*
+# <> isn't actually a valid comparison operator in Python. It's here for the
+# sake of a __future__ import described in PEP 401
 comp_op: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in'|'is'|'is' 'not'
 star_expr: '*' expr
 expr: xor_expr ('|' xor_expr)*

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Mon Nov 14 05:38:40 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 14 Nov 2011 05:38:40 +0100
Subject: [Python-checkins] Daily reference leaks (410115400838): sum=0
Message-ID: <E1RPoJc-0003mF-Gv@ap.vmr.nerim.net>

results for 410115400838 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogz_LHXV', '-x']

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_developers_notes?=
Message-ID: <E1RPxSB-00071w-FQ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/88ab3211b002
changeset:   1221:88ab3211b002
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:38:07 2011 +0100
summary:
  Update developers notes

files:
  DEVNOTES.txt |  23 ++++++++++++++++++++---
  1 files changed, 20 insertions(+), 3 deletions(-)


diff --git a/DEVNOTES.txt b/DEVNOTES.txt
--- a/DEVNOTES.txt
+++ b/DEVNOTES.txt
@@ -1,9 +1,20 @@
-Notes for developers
+====================
+Notes for Developers
 ====================
 
-- Distutils2 runs on Python from 2.4 to 2.7 so make sure you don't use code
+- Distutils2 has been merged into the CPython repository under the module name
+  "packaging", for release with Python 3.3.  The Distutils2 repository is used
+  to maintain the backport released on PyPI for older Python versions.  It is
+  recommended to work in the CPython repository if possible, but you but you
+  can make patches for the Distutils2 repository if you prefer.
+
+  Repo: http://hg.python.org/cpython (default branch)
+  More info: http://wiki.python.org/moin/Distutils/Contributing
+
+- Distutils2 runs on Python from 2.4 to 2.7, so make sure you don't use code
   that doesn't work under one of these Python versions.  There is also a
-  version compatible with 3.1-3.3 in the "python3" branch.
+  version compatible with 3.1-3.3 in the "python3" branch.  When merging
+  default into python3, please use modern 3.x idioms.
 
 - For 2.4, you need to run "python2.4 setup.py build" before you can try out
   pysetup or run tests (unless you use the runtests.py script which will call
@@ -12,3 +23,9 @@
 - Always run tests.sh before you commit a change.  This implies that you have
   all Python versions installed from 2.4 to 2.7.  Be sure to also have docutils
   installed on all Python versions to avoid skipping tests.
+
+- To merge with the Python 3 version, update to the python3 branch, merge and
+  run tests.sh to test with Python 3.1, 3.2 and 3.3 (you'll need to build the
+  last one from source).  If you can't test, let someone else do the merge.
+
+- Don't forget to update CONTRIBUTORS.txt and CHANGES.txt.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fill_in_missing_entries_?=
	=?utf8?q?for_last_year?=
Message-ID: <E1RPxSB-00071t-EN@dinsdale.python.org>

http://hg.python.org/distutils2/rev/70c220cd2f59
changeset:   1220:70c220cd2f59
parent:      1218:5949563b9f1c
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:37:45 2011 +0100
summary:
  Fill in missing entries for last year

files:
  CHANGES.txt      |  165 ++++++++++++++++++++++++++++++++--
  CONTRIBUTORS.txt |   41 +++++++-
  2 files changed, 185 insertions(+), 21 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,16 +1,154 @@
-=======
-CHANGES
-=======
+====================
+Distutils2 Changelog
+====================
 
-1.0a4 - ?
----------
+This file should list all changes made to the user-visible behavior and the
+public API, all important internal changes to help developers merge changes with
+their clones, and all changes that have a bug report.  Contributors' first names
+(and last name initial when needed) are given for each item; see
+CONTRIBUTORS.txt for full names.  Bug numbers refer to http://bugs.python.org/.
 
-- The setup runner supports more options:
-- XXX fill changes done in commands + compilers [tarek]
-- Issue #10409: Fixed the Licence selector in mkcfg [tarek]
-- Issue #9558: Fix build_ext with VS 8.0 [?ric]
-- Issue #6007: Add disclaimer about MinGW compatibility in docs [?ric]
-- Renamed DistributionMetadata to Metadata [ccomb]
+
+1.0a4 - 2011-12-??
+------------------
+
+- Remove type check for commands in favor of minimal duck type check [tarek]
+- Add setup_hook, run between setup.cfg parsing and any other action [tarek]
+- Allow configuring command classes in setup.cfg [tarek]
+- Add description-file to setup.cfg [tarek]
+- Make Manifest.read_template accept file objects [tarek]
+- Replace MANIFEST.in with extra_files field in setup.cfg for sdist [tarek]
+- Abandon converter idea for d1-d2 migration [tarek]
+- Make 'include' default action for extra_files lines [tarek]
+- Rename py_modules to modules in setup.cfg [tarek]
+- Add detection of files in mkcfg [tarek]
+- Remove core module [tarek]
+- Remove log module, use standard logging with 'distutils2' logger name [tarek]
+- Allow configuring sub-commands in setup.cfg [tarek]
+- Add manifest_builders field [tarek]
+- Move useful functions from compiler.ccompiler to compiler [tarek]
+- Compiler classes should now define a description attribute [tarek]
+- Compiler classes are now loaded with their fully qualified names [tarek]
+- Allow registering extra compiler classes in setup.cfg [tarek]
+- The EMX compiler is gone [tarek]
+- Standard command classes are now registered with set_commands;
+  command_packages is gone [tarek]
+- Move extension module to compiler.extension [tarek]
+- The compiler_type attribute of compiler classes is now name [tarek]
+- Document the setup.cfg file format in a specification that can be used by
+  non-Python developers [tarek, ?ric, julien j]
+- #10409: Fixed the License selector in mkcfg [tarek]
+- #9558: Fix build_ext with VS 8.0 [?ric]
+- #6007: Add disclaimer about MinGW compatibility in docs [?ric]
+- #11038: Add strict parameter to metadata.check, use it in check command [ga?l]
+- Support multiple files in description-file [ga?l]
+- Simplify and rename package_dir to packages_root: only one root directory for
+  all packages and modules is allowed [tarek]
+- Add util.generate_setup_py to create a setup.py script compatible with
+  distutils that takes information from a setup.cfg [julien m]
+- README or README.txt and test/test*.py are no longer included in sdists
+  [godefroid]
+- Move PEP 345 markers interpreter from metadata to markers [tarek]
+- #11057: Fix two NameErrors [?ric]
+- Add install.remove, the uninstall feature [ga?l]
+- Reuse info from existing setup.py when creating a setup.cfg with mkcfg [alain]
+- Add support for extension modules in setup.cfg [andr?]
+- Arguments that specify the target directory for install-related functions in
+  install don't have default values anymore [yannick]
+- Add paths argument to install.install* functions to allow working outside of
+  sys.path [tarek]
+- Metadata.check and check command now want an author field, even if maintainer
+  is supplied [godefroid]
+- Renamed DistributionMetadata to Metadata [chistophe]
+- Make --help-commands work again [?ric]
+- Fix index.dist.DistInfo.unpack to really use the path given as argument
+  [kelsey]
+- Introduce the resources system, a major overhaul of data_files installation
+  that supports putting files in various directories compliant with OS policies
+  and pain-free use of these files by Python code [boris, pierre-yves]
+- New util.iglob function supporting more patterns than stdlib glob
+  [pierre-yves]
+- Add 'pysetup create' to prompt the user and create a setup.cfg file with the
+  answers [tarek]
+- Convert and import the code into the CPython 3.3 repository [tarek, arc,
+  vinay, elson, walker, michael, kelsey, jason, alexis, ?ric, victor]; in the
+  distutils2 backport, change some names to match the CPython repo: exception
+  names start with "Packaging", there is a util.is_packaging function, etc.
+- Add bdist_wininst helpers compiled with Visual Studio 10 [jason]
+- 'pysetup install' now accepts a patch to a directory or to an archive in
+  addition to a PyPI project name [kelsey]
+- Rename mkcfg to create [?ric]
+- Add function to convert an egg-info file or directory to dist-info [kelsey]
+- Add functions to detect if a project uses distutils, setuptools or packaging
+  [kelsey, hugo]
+- Config fields that support environment markers (PEP 345) can now check
+  'platform.python_implementation' [alexis]
+- Use True and False instead of 0 and 1 (compatible change that makes the intent
+  of the code clearer) [?ric]
+- Rename 'pysetup list' the action used to look for installed projects [alexis]
+- Add 'pysetup search' to look for projects on an index/catalog [alexis]
+- Rename packaging.index back to packaging.pypi [kelsey]
+- Clean up logging setup, improve tests.support.LoggingCatcher [?ric]
+- Replace warnings by logging, remove display_warning argument of
+  Metadata.__init__, kill warn/announce/debug_print methods [?ric]
+- Improve EnvironGuard and rename it EnvironRestorer [?ric]
+- Move PEP 376 implementation from pkgutil to packaging.database [?ric]
+- Add version attribute to database.*Distribution classes [?ric]
+- #10419, #6011: Make sure build_scripts can handle non-ASCII path for the
+  Python interpreter used in shebangs [victor]
+- #12112, #12320, #9561: Use UTF-8 to read or write setup.cfg, setup.py and
+  METADATA files instead of the default, locale-dependent encoding [victor]
+- #12114: Fix potential deadlock or zombification in util._find_exe_version
+  [victor]
+- Use / as path separator in setup.cfg created by 'pysetup create' even on
+  Windows [tarek]
+- Use / as path separator in RECORD file even on Windows [tarek]
+- #6459: Fix the import symbol in extension modules [tarek]
+- #10126: Fix for python built in shared mode on Unix [tarek]
+- #10359: ';' after function definition is invalid in ISO C [?ric]
+- Remove the resources submodule, move its functions into resources [tarek]
+- Update the docs and move them to the CPython repository [kelsey, elson,
+  guillermoo, ?ric]
+- Add 'pysetup generate-setup' to expose util.generate_setup_py [tarek]
+- #11092: Make sdist always include setup.cfg [?ric]
+- #12246: Don?t try to install something when running from an uninstalled Python
+  built in its checkout [tschepang, ?ric]
+- Add packaging.util.split_multiline [julien m, erik]
+- #11595: Fix assorted bugs in packaging.util.cfg_to_args [erik, ?ric]
+- #12240: Allow multiple setup hooks [erik, ?ric]
+- #11637: Fix support for importing setup hooks from the project directory
+  [vinay, ?ric]
+- #9516: Revise deployment target processing for OS X [ned]
+- #12169, #10510: Factor out code used by various commands to make HTTP POST
+  requests, and make sure it uses CRLF [john, ?ric]
+- #12504: Close file handles in a timely manner in database; this fixes a bug
+  with the remove (uninstall) feature on Windows [thomas]
+- #11409, #12222: Let all pysetup actions return a meaningful 0 or 1 exit code
+  [kelsey, ?ric]
+- Add filesafe argument to Metadata.get_fullname and Distribution.get_fullname
+  [jeremy]
+- Change distutils2's setup.py script to get info from the setup.cfg [jeremy]
+- Add support for building OpenSSL on Windows (for _backports.hashlib) [jeremy]
+- Print all fields when calling 'pysetup metadata' without options, remove --all
+  option for metadata and list actions [?ric]
+- Remove display options (--name, etc.) from the Distribution class; this has
+  the side effect that 'url' is no longer accepted as key in the attrs argument
+  of the class' constructor, it needs to be 'home-page' to be recognized as
+  valid metadata field [?ric]
+- #10946: Make bdist_dumb, bdist_wininst and bdist_msi respect a --skip-build
+  option given to bdist [?ric]
+- The right-hand part in [extension: foo] (in a setup.cfg) is now used as the
+  name of the extension module [?ric]
+- #8933: METADATA files will now correctly report Metadata-Version: 1.1 instead
+  of 1.0 if a Classifier or Download-URL field is present [filip, ?ric]
+- Create a branch for a Python 3 version of distutils2 [?ric]
+- #10359: Make C code in one test comply with ISO C [hallvard]
+- #11254: Fix byte-compilation to comply with PEP 3147 on Python 3.2+ [?ric]
+- #13114: Add tests for Unicode handling in check and register [?ric]
+- #13170: Revert one of Jeremy's changes to config to fix a bug [?ric]
+- #13170: Kludge around shlex not supporting unicode in 2.x [?ric]
+- #13205: Fix and improve generated setup scripts [david, ?ric]
+
 
 1.0a3 - 2010-10-08
 ------------------
@@ -19,8 +157,8 @@
 - Fixed the installation when using easy_install and Pip by switching
   setup.py to distutils1 [holger/tarek]
 - Added missing c/h files in the MANIFEST so they are always present
-  no matter which Python version was used to build it. [holger/tarek]
-- Added the new setup runner that uses only setup.cfg
+  no matter which Python version was used to build it [holger/tarek]
+- Added pysetup, the new setup runner that uses only setup.cfg
 - Renamed mkpkg to mkcfg [tarek]
 - Renamed install_tools to install [alexis]
 
@@ -44,7 +182,7 @@
 - Remove PyPIRCCommand, move its helper code into util [tarek]
 - Remove Mac OS 9 support [?ric]
 - Start adding docstrings to interface methods [jeremy]
-- Move documentation from the stdlib [ali, ?ric]
+- Copy documentation from the stdlib [ali, ?ric]
 - Lots of bug fixes, cleanups, tests [everyone]
 
 
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -1,44 +1,69 @@
-============
-Contributors
-============
+=======================
+Distutils2 Contributors
+=======================
 
-Distutils2 is a project that was started and that is maintained by
-Tarek Ziad?, and many people are contributing to the project.
+The Distutils2 project was started by Tarek Ziad? and is currently
+maintained by ?ric Araujo.  Many people have contributed to the project.
 
-If you did, please add your name below in alphabetical order!
+If you're making a patch, please add your name below in alphabetical order,
+and welcome into the Fellowship of the Packaging!
 
 Thanks to:
 
 - Rajiv Abraham
 - Ali Afshar
-- ?ric Araujo
+- David Barnett
 - Pior Bastida
 - Anthony Baxter
-- Titus Brown
+- Erik Bray
+- C. Titus Brown
 - Nicolas Cadou
+- Godefroid Chapelle
+- Christophe Combelles
+- Jason R. Coombs
+- Pierre-Yves David
+- Ned Deily
 - Konrad Delong
 - Josip Djolonga
+- John Edmonds
+- Andr? Espaze
+- Boris Feld
 - Andrew Francis
+- Hallvard B Furuseth
 - Yannick Gingras
+- Filip Gruszczy?ski
+- guillermoo
+- Walker Hale IV
 - Alexandre Hamelin
 - Kelsey Hightower
+- Thomas Holmes
 - Christian Hudon
+- Julien Jehannet
 - Jeremy Kloth
 - Amos Latteier
 - Mathieu Leduc-Hamel
+- Tshepang Lekhonkhobe
+- Alain Leufroy
 - Martin von L?wis
+- Hugo Lopes Tavares
 - Simon Mathieu
 - Carl Meyer
 - Alexis M?taireau
+- Julien Miotte
 - Zubin Mithra
 - Derek McTavish Mounce
 - Michael Mulich
 - Louis Munro
+- Ga?l Pasgrimaud
 - George Peristerakis
 - Mathieu Perreault
 - Sean Reifschneider
 - Antoine Reversat
+- Arc Riley
+- Elson Rodriguez
 - Luis Rojas
 - Erik Rose
 - Brian Rosner
+- Vinay Sajip
+- Victor Stinner
 - Alexandre Vassalotti

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_and_expand_README?=
Message-ID: <E1RPxSB-000720-Fr@dinsdale.python.org>

http://hg.python.org/distutils2/rev/9602fadd902a
changeset:   1222:9602fadd902a
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:38:59 2011 +0100
summary:
  Update and expand README

files:
  README.txt |  53 ++++++++++++++++++++++++++++-------------
  1 files changed, 36 insertions(+), 17 deletions(-)


diff --git a/README.txt b/README.txt
--- a/README.txt
+++ b/README.txt
@@ -1,24 +1,43 @@
-==========
-Distutils2
-==========
+=======================
+ Welcome to Distutils2
+=======================
 
-Welcome to Distutils2!
+Distutils2 is the packaging library that supersedes Distutils.  It has three
+main audiences:
 
-Distutils2 is the new version of Distutils. It's not backward compatible with
-Distutils but provides more features, and implement most new packaging
-standards.
+- Python authors who want to distribute their code
+- End users who want to install modules or applications
+- Developers of packaging-related tools who need a support library to
+  build on
 
-See the documentation at http://packages.python.org/Distutils2 for more info.
+Authors will have to write a :file:`setup.cfg` file and run a few
+commands to package and distribute their code.  End users will be able to
+search for, install and remove Python projects with the included
+``pysetup`` program.  Last, developers will be able to reuse classes and
+functions in their tools.
 
-If you want to contribute, please have a look to 
-http://distutils2.notmyidea.org/contributing.html
+The Distutils2 codebase is a fork of Distutils.  It is not backward compatible
+with Distutils and does not depend on it.  It provides more features and
+implements new packaging standards.  In Python 3.3, Distutils2 is included in
+the standard library under the module name "packaging".  Documentation is
+provided at http://docs.python.org/dev/packaging --for ease of maintenance, it
+is not duplicated in this repository.  You can use the Packaging documentation
+to use Distutils2; only the package name is different (packaging vs.
+distutils2), all modules, classes and functions have the same name.
 
-**Beware that Distutils2 is in its early stage and should not be used in
-production. Its API is subject to changes**
+If you want to contribute, please have a look at DEVNOTES.txt or
+http://wiki.python.org/Distutils2/Contributing .
 
-Useful further links:
+Beware that Distutils2 is still in alpha stage and its API is subject to
+change.  It should be not used for critical deployments.  That said, it
+is possible to start using it while keeping compatiblity with tools based
+on the old Distutils or Setuptools, and the developers are eager to get
+feedback from authors, end users and developers.
 
-Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/
-Documentation: http://packages.python.org/Distutils2
-Repository: http://hg.python.org/distutils2
-Bug tracker: http://bugs.python.org
+Useful links:
+
+- Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/
+- Mailing list with friendly tutors to guide new contributors:
+  http://mail.python.org/mailman/listinfo/core-mentorship
+- Repository: http://hg.python.org/distutils2
+- Bug tracker: http://bugs.python.org/ (component "Distutils2")

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_pyflakes_is_more_importa?=
 =?utf8?q?nt_than_pep8=2C_run_it_first?=
Message-ID: <E1RPxSB-000728-HS@dinsdale.python.org>

http://hg.python.org/distutils2/rev/13e8ceab09ca
changeset:   1224:13e8ceab09ca
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:41:36 2011 +0100
summary:
  pyflakes is more important than pep8, run it first

files:
  check.sh |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/check.sh b/check.sh
--- a/check.sh
+++ b/check.sh
@@ -1,2 +1,4 @@
-pep8 distutils2
+# TODO exclude backports
 pyflakes distutils2
+pep8 -r distutils2
+# maybe:  exit $?

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Propagate_error_code_fro?=
	=?utf8?q?m_Python?=
Message-ID: <E1RPxSB-000724-Gr@dinsdale.python.org>

http://hg.python.org/distutils2/rev/0aa491a3dca3
changeset:   1223:0aa491a3dca3
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:39:36 2011 +0100
summary:
  Propagate error code from Python

files:
  tests.sh |  10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/tests.sh b/tests.sh
--- a/tests.sh
+++ b/tests.sh
@@ -5,7 +5,7 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python2.4 -Wd runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
@@ -15,7 +15,7 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python2.5 -Wd runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
@@ -25,7 +25,7 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python2.6 -Wd runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
@@ -35,14 +35,14 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python2.7 -Wd -bb -3 runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
 
 if [ $? -ne 0 ];then
     echo Failed
-    exit 1
+    exit $?
 else
     echo "Good job, commit now! (or add tests)"
 fi

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Avoid_diverging_from_ups?=
	=?utf8?q?tream_in_backports?=
Message-ID: <E1RPxSB-00072C-II@dinsdale.python.org>

http://hg.python.org/distutils2/rev/2d9af3034632
changeset:   1225:2d9af3034632
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:44:26 2011 +0100
summary:
  Avoid diverging from upstream in backports

files:
  distutils2/_backport/shutil.py |  7 +------
  1 files changed, 1 insertions(+), 6 deletions(-)


diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -202,12 +202,7 @@
     else:
         ignored_names = set()
 
-    try:
-        os.makedirs(dst)
-    except OSError, e:
-        if e.errno != errno.EEXIST:
-            raise
-
+    os.makedirs(dst)
     errors = []
     for name in names:
         if name in ignored_names:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Credit_David_Barnett_for?=
	=?utf8?q?_his_help?=
Message-ID: <E1RPxSB-00072K-JZ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/ad87858b5589
changeset:   1227:ad87858b5589
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:49:08 2011 +0100
summary:
  Credit David Barnett for his help

files:
  CHANGES.txt |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -145,8 +145,8 @@
 - #10359: Make C code in one test comply with ISO C [hallvard]
 - #11254: Fix byte-compilation to comply with PEP 3147 on Python 3.2+ [?ric]
 - #13114: Add tests for Unicode handling in check and register [?ric]
-- #13170: Revert one of Jeremy's changes to config to fix a bug [?ric]
-- #13170: Kludge around shlex not supporting unicode in 2.x [?ric]
+- #13170: Revert one of Jeremy's changes to config to fix a bug, kludge around
+  shlex not supporting unicode in 2.x, fix wrong shutil import [david, ?ric]
 - #13205: Fix and improve generated setup scripts [david, ?ric]
 
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_import_in_install=5F?=
 =?utf8?q?data_=28=2313170=29=2E__Thanks_to_David_Barnett=2E?=
Message-ID: <E1RPxSB-00072G-J4@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5b096fc6e65d
changeset:   1226:5b096fc6e65d
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 10:46:17 2011 +0100
summary:
  Fix import in install_data (#13170).  Thanks to David Barnett.

files:
  distutils2/command/install_data.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -3,10 +3,10 @@
 # Contributed by Bastian Kleineidam
 
 import os
-from shutil import Error
 from distutils2 import logger
 from distutils2.util import convert_path
 from distutils2.command.cmd import Command
+from distutils2._backport.shutil import Error
 from distutils2._backport.sysconfig import get_paths, format_value
 
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_writing_of_the_RESOU?=
 =?utf8?q?RCES_file_=28=2312386=29=2E?=
Message-ID: <E1RPxSB-00072O-KF@dinsdale.python.org>

http://hg.python.org/distutils2/rev/cb49bc384957
changeset:   1228:cb49bc384957
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 11:41:17 2011 +0100
summary:
  Fix writing of the RESOURCES file (#12386).

This was not broken in d2 because codecs.open allows 'b' in mode.

files:
  distutils2/command/install_distinfo.py            |   2 +-
  distutils2/tests/test_command_install_data.py     |  70 +++++++++-
  distutils2/tests/test_command_install_distinfo.py |   5 +-
  3 files changed, 72 insertions(+), 5 deletions(-)


diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -111,7 +111,7 @@
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
-                    f = open(resources_path, 'wb')
+                    f = open(resources_path, 'w')
                     try:
                         writer = csv.writer(f, delimiter=',',
                                             lineterminator='\n',
diff --git a/distutils2/tests/test_command_install_data.py b/distutils2/tests/test_command_install_data.py
--- a/distutils2/tests/test_command_install_data.py
+++ b/distutils2/tests/test_command_install_data.py
@@ -1,28 +1,37 @@
 """Tests for distutils2.command.install_data."""
 import os
+import sys
+import distutils2.database
 from distutils2._backport import sysconfig
 from distutils2._backport.sysconfig import _get_default_scheme
 from distutils2.tests import unittest, support
 from distutils2.command.install_data import install_data
+from distutils2.command.install_dist import install_dist
+from distutils2.command.install_distinfo import install_distinfo
 
 
 class InstallDataTestCase(support.TempdirManager,
                           support.LoggingCatcher,
                           unittest.TestCase):
 
-    def test_simple_run(self):
+    def setUp(self):
+        super(InstallDataTestCase, self).setUp()
         scheme = _get_default_scheme()
         old_items = sysconfig._SCHEMES.items(scheme)
+
         def restore():
             sysconfig._SCHEMES.remove_section(scheme)
             sysconfig._SCHEMES.add_section(scheme)
             for option, value in old_items:
                 sysconfig._SCHEMES.set(scheme, option, value)
+
         self.addCleanup(restore)
 
+    def test_simple_run(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_data(dist)
         cmd.install_dir = inst = os.path.join(pkg_dir, 'inst')
+        scheme = _get_default_scheme()
 
         sysconfig._SCHEMES.set(scheme, 'inst',
                                os.path.join(pkg_dir, 'inst'))
@@ -67,8 +76,7 @@
         three = os.path.join(cmd.install_dir, 'three')
         self.write_file(three, 'xx')
 
-        sysconfig._SCHEMES.set(scheme, 'inst3',
-                               cmd.install_dir)
+        sysconfig._SCHEMES.set(scheme, 'inst3', cmd.install_dir)
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
@@ -80,6 +88,62 @@
         self.assertTrue(os.path.exists(os.path.join(inst2, rtwo)))
         self.assertTrue(os.path.exists(os.path.join(inst, rone)))
 
+    def test_resources(self):
+        install_dir = self.mkdtemp()
+        scripts_dir = self.mkdtemp()
+        project_dir, dist = self.create_dist(
+            name='Spamlib', version='0.1',
+            data_files={'spamd': '{scripts}/spamd'})
+
+        os.chdir(project_dir)
+        self.write_file('spamd', '# Python script')
+        sysconfig._SCHEMES.set(_get_default_scheme(), 'scripts', scripts_dir)
+        sys.path.insert(0, install_dir)
+        distutils2.database.disable_cache()
+        self.addCleanup(sys.path.remove, install_dir)
+        self.addCleanup(distutils2.database.enable_cache)
+
+        cmd = install_dist(dist)
+        cmd.outputs = ['spamd']
+        cmd.install_lib = install_dir
+        dist.command_obj['install_dist'] = cmd
+
+        cmd = install_data(dist)
+        cmd.install_dir = install_dir
+        cmd.ensure_finalized()
+        dist.command_obj['install_data'] = cmd
+        cmd.run()
+
+        cmd = install_distinfo(dist)
+        cmd.ensure_finalized()
+        dist.command_obj['install_distinfo'] = cmd
+        cmd.run()
+
+        # first a few sanity checks
+        self.assertEqual(os.listdir(scripts_dir), ['spamd'])
+        self.assertEqual(os.listdir(install_dir), ['Spamlib-0.1.dist-info'])
+
+        # now the real test
+        fn = os.path.join(install_dir, 'Spamlib-0.1.dist-info', 'RESOURCES')
+        fp = open(fn)
+        try:
+            content = fp.read().strip()
+        finally:
+            fp.close()
+
+        expected = 'spamd,%s' % os.path.join(scripts_dir, 'spamd')
+        self.assertEqual(content, expected)
+
+        # just to be sure, we also test that get_file works here, even though
+        # packaging.database has its own test file
+        fp = distutils2.database.get_file('Spamlib', 'spamd')
+        try:
+            content = fp.read()
+        finally:
+            fp.close()
+
+        self.assertEqual('# Python script', content)
+
 
 def test_suite():
     return unittest.makeSuite(InstallDataTestCase)
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -1,4 +1,7 @@
-"""Tests for ``distutils2.command.install_distinfo``. """
+"""Tests for ``distutils2.command.install_distinfo``.
+
+Writing of the RESOURCES file is tested in test_command_install_data.
+"""
 
 import os
 import csv

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_More_information_about_t?=
	=?utf8?q?he_Python_3_port=2E?=
Message-ID: <E1RPxSB-00072S-Km@dinsdale.python.org>

http://hg.python.org/distutils2/rev/c8720f11e768
changeset:   1229:c8720f11e768
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 12:01:41 2011 +0100
summary:
  More information about the Python 3 port.

It?s more useful if this file contains all information in both branches.

files:
  DEVNOTES.txt |  20 +++++++++++---------
  1 files changed, 11 insertions(+), 9 deletions(-)


diff --git a/DEVNOTES.txt b/DEVNOTES.txt
--- a/DEVNOTES.txt
+++ b/DEVNOTES.txt
@@ -12,20 +12,22 @@
   More info: http://wiki.python.org/moin/Distutils/Contributing
 
 - Distutils2 runs on Python from 2.4 to 2.7, so make sure you don't use code
-  that doesn't work under one of these Python versions.  There is also a
-  version compatible with 3.1-3.3 in the "python3" branch.  When merging
-  default into python3, please use modern 3.x idioms.
+  that doesn't work under one of these Python versions.  The version in the
+  "python3" branch is compatible with all version from 3.1 to 3.3.
+
+- When merging default into python3, don't keep maximum compatibility with
+  Python 2 but use idiomatic 3.x code, as long as it's compatible with all 3.x
+  versions.  For difficult conversions like encoding handling with I/O, you can
+  have a look or use a diff tool with the same file in distutils or packaging
+  from Python 3.3.  If you can't run tests, let someone else do the merge.
 
 - For 2.4, you need to run "python2.4 setup.py build" before you can try out
   pysetup or run tests (unless you use the runtests.py script which will call
   "setup.py build" automatically).
 
 - Always run tests.sh before you commit a change.  This implies that you have
-  all Python versions installed from 2.4 to 2.7.  Be sure to also have docutils
-  installed on all Python versions to avoid skipping tests.
-
-- To merge with the Python 3 version, update to the python3 branch, merge and
-  run tests.sh to test with Python 3.1, 3.2 and 3.3 (you'll need to build the
-  last one from source).  If you can't test, let someone else do the merge.
+  all Python versions installed from 2.4 to 2.7, as well as 3.1-.3.3 if you
+  merge into the python3 branch.  Be sure to also have docutils installed on all
+  Python versions to avoid skipping tests.
 
 - Don't forget to update CONTRIBUTORS.txt and CHANGES.txt.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Avoid_matching_=27=27_or?=
 =?utf8?q?_=27yn=27_when_asking_for_=27y=27_or_=27n=27_in_interactive_code?=
Message-ID: <E1RPxSB-00072a-MX@dinsdale.python.org>

http://hg.python.org/distutils2/rev/f40475ea7a1a
changeset:   1231:f40475ea7a1a
parent:      1229:c8720f11e768
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 23:15:09 2011 +0100
summary:
  Avoid matching '' or 'yn' when asking for 'y' or 'n' in interactive code

files:
  distutils2/command/register.py |  2 +-
  distutils2/create.py           |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -177,7 +177,7 @@
                         'will be faster.\n(the login will be stored in %s)',
                         get_pypirc_path())
                     choice = 'X'
-                    while choice.lower() not in 'yn':
+                    while choice.lower() not in ('y', 'n'):
                         choice = raw_input('Save your login (y/N)?')
                         if not choice:
                             choice = 'n'
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -136,7 +136,7 @@
     question += ' (y/n)'
     while True:
         answer = ask(question, default, helptext, required=True)
-        if answer and answer[0].lower() in 'yn':
+        if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
         print '\nERROR: You must select "Y" or "N".\n'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Expand_tests_and_fix_bug?=
 =?utf8?q?s_in_util=2Eresolve=5Fname=2E?=
Message-ID: <E1RPxSB-00072e-NK@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5df1065ddb8b
changeset:   1232:5df1065ddb8b
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 23:26:49 2011 +0100
summary:
  Expand tests and fix bugs in util.resolve_name.

The code is still ugly, but at least it works better now.  Patches to
make it easier to read are welcome, as well as support in #12915.

files:
  distutils2/tests/test_util.py |  65 +++++++++++++---------
  distutils2/util.py            |  24 ++++++--
  2 files changed, 57 insertions(+), 32 deletions(-)


diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -380,35 +380,48 @@
                                         'pkg1.pkg3.pkg6']))
 
     def test_resolve_name(self):
-        self.assertIs(str, resolve_name('__builtin__.str'))
-        self.assertEqual(
-            UtilTestCase.__name__,
-            resolve_name("distutils2.tests.test_util.UtilTestCase").__name__)
-        self.assertEqual(
-            UtilTestCase.test_resolve_name.__name__,
-            resolve_name("distutils2.tests.test_util.UtilTestCase."
-                         "test_resolve_name").__name__)
+        # test raw module name
+        tmpdir = self.mkdtemp()
+        sys.path.append(tmpdir)
+        self.addCleanup(sys.path.remove, tmpdir)
+        self.write_file((tmpdir, 'hello.py'), '')
 
-        self.assertRaises(ImportError, resolve_name,
-                          "distutils2.tests.test_util.UtilTestCaseNot")
-        self.assertRaises(ImportError, resolve_name,
-                          "distutils2.tests.test_util.UtilTestCase."
-                          "nonexistent_attribute")
+        os.makedirs(os.path.join(tmpdir, 'a', 'b'))
+        self.write_file((tmpdir, 'a', '__init__.py'), '')
+        self.write_file((tmpdir, 'a', 'b', '__init__.py'), '')
+        self.write_file((tmpdir, 'a', 'b', 'c.py'), 'class Foo: pass')
+        self.write_file((tmpdir, 'a', 'b', 'd.py'), textwrap.dedent("""\
+                         class FooBar:
+                             class Bar:
+                                 def baz(self):
+                                     pass
+                            """))
 
-    def test_import_nested_first_time(self):
-        tmp_dir = self.mkdtemp()
-        os.makedirs(os.path.join(tmp_dir, 'a', 'b'))
-        self.write_file(os.path.join(tmp_dir, 'a', '__init__.py'), '')
-        self.write_file(os.path.join(tmp_dir, 'a', 'b', '__init__.py'), '')
-        self.write_file(os.path.join(tmp_dir, 'a', 'b', 'c.py'),
-                                    'class Foo: pass')
+        # check Python, C and built-in module
+        self.assertEqual(resolve_name('hello').__name__, 'hello')
+        self.assertEqual(resolve_name('_csv').__name__, '_csv')
+        self.assertEqual(resolve_name('sys').__name__, 'sys')
 
-        try:
-            sys.path.append(tmp_dir)
-            resolve_name("a.b.c.Foo")
-            # assert nothing raised
-        finally:
-            sys.path.remove(tmp_dir)
+        # test module.attr
+        self.assertIs(resolve_name('__builtin__.str'), str)
+        self.assertIsNone(resolve_name('hello.__doc__'))
+        self.assertEqual(resolve_name('a.b.c.Foo').__name__, 'Foo')
+        self.assertEqual(resolve_name('a.b.d.FooBar.Bar.baz').__name__, 'baz')
+
+        # error if module not found
+        self.assertRaises(ImportError, resolve_name, 'nonexistent')
+        self.assertRaises(ImportError, resolve_name, 'non.existent')
+        self.assertRaises(ImportError, resolve_name, 'a.no')
+        self.assertRaises(ImportError, resolve_name, 'a.b.no')
+        self.assertRaises(ImportError, resolve_name, 'a.b.no.no')
+        self.assertRaises(ImportError, resolve_name, 'inva-lid')
+
+        # looking up built-in names is not supported
+        self.assertRaises(ImportError, resolve_name, 'str')
+
+        # error if module found but not attr
+        self.assertRaises(ImportError, resolve_name, 'a.b.Spam')
+        self.assertRaises(ImportError, resolve_name, 'a.b.c.Spam')
 
     @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
     def test_run_2to3_on_code(self):
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -638,22 +638,35 @@
 def resolve_name(name):
     """Resolve a name like ``module.object`` to an object and return it.
 
-    Raise ImportError if the module or name is not found.
+    This functions supports packages and attributes without depth limitation:
+    ``package.package.module.class.class.function.attr`` is valid input.
+    However, looking up builtins is not directly supported: use
+    ``builtins.name``.
+
+    Raises ImportError if importing the module fails or if one requested
+    attribute is not found.
     """
+    if '.' not in name:
+        # shortcut
+        __import__(name)
+        return sys.modules[name]
+
+    # FIXME clean up this code!
     parts = name.split('.')
     cursor = len(parts)
     module_name = parts[:cursor]
+    ret = ''
 
     while cursor > 0:
         try:
             ret = __import__('.'.join(module_name))
             break
         except ImportError:
-            if cursor == 0:
-                raise
             cursor -= 1
             module_name = parts[:cursor]
-            ret = ''
+
+    if ret == '':
+        raise ImportError(parts[0])
 
     for part in parts[1:]:
         try:
@@ -1469,8 +1482,7 @@
 
     Returns (content_type: bytes, body: bytes) ready for httplib.HTTP.
     """
-    # Taken from
-    # http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
+    # Taken from http://code.activestate.com/recipes/146306
 
     if boundary is None:
         boundary = '--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
 =?utf8?q?n3=29=3A_Merge_fixes_for_=2313170_and_=2312386_and_other_misc=2E?=
 =?utf8?q?_changes_from_default?=
Message-ID: <E1RPxSB-00072X-Lu@dinsdale.python.org>

http://hg.python.org/distutils2/rev/2d469ccfe30e
changeset:   1230:2d469ccfe30e
branch:      python3
parent:      1219:eb845a9a00b7
parent:      1229:c8720f11e768
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 12:16:16 2011 +0100
summary:
  Merge fixes for #13170 and #12386 and other misc. changes from default

files:
  CHANGES.txt                                       |  165 +++++++++-
  CONTRIBUTORS.txt                                  |   41 ++-
  DEVNOTES.txt                                      |   37 +-
  README.txt                                        |   53 ++-
  check.sh                                          |    4 +-
  distutils2/_backport/shutil.py                    |    7 +-
  distutils2/command/install_data.py                |    2 +-
  distutils2/command/install_distinfo.py            |    2 +-
  distutils2/tests/test_command_install_data.py     |   70 ++++-
  distutils2/tests/test_command_install_distinfo.py |    5 +-
  distutils2/tests/test_pypi_simple.py              |    6 +-
  tests.sh                                          |    8 +-
  12 files changed, 334 insertions(+), 66 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,16 +1,154 @@
-=======
-CHANGES
-=======
+====================
+Distutils2 Changelog
+====================
 
-1.0a4 - ?
----------
+This file should list all changes made to the user-visible behavior and the
+public API, all important internal changes to help developers merge changes with
+their clones, and all changes that have a bug report.  Contributors' first names
+(and last name initial when needed) are given for each item; see
+CONTRIBUTORS.txt for full names.  Bug numbers refer to http://bugs.python.org/.
 
-- The setup runner supports more options:
-- XXX fill changes done in commands + compilers [tarek]
-- Issue #10409: Fixed the Licence selector in mkcfg [tarek]
-- Issue #9558: Fix build_ext with VS 8.0 [?ric]
-- Issue #6007: Add disclaimer about MinGW compatibility in docs [?ric]
-- Renamed DistributionMetadata to Metadata [ccomb]
+
+1.0a4 - 2011-12-??
+------------------
+
+- Remove type check for commands in favor of minimal duck type check [tarek]
+- Add setup_hook, run between setup.cfg parsing and any other action [tarek]
+- Allow configuring command classes in setup.cfg [tarek]
+- Add description-file to setup.cfg [tarek]
+- Make Manifest.read_template accept file objects [tarek]
+- Replace MANIFEST.in with extra_files field in setup.cfg for sdist [tarek]
+- Abandon converter idea for d1-d2 migration [tarek]
+- Make 'include' default action for extra_files lines [tarek]
+- Rename py_modules to modules in setup.cfg [tarek]
+- Add detection of files in mkcfg [tarek]
+- Remove core module [tarek]
+- Remove log module, use standard logging with 'distutils2' logger name [tarek]
+- Allow configuring sub-commands in setup.cfg [tarek]
+- Add manifest_builders field [tarek]
+- Move useful functions from compiler.ccompiler to compiler [tarek]
+- Compiler classes should now define a description attribute [tarek]
+- Compiler classes are now loaded with their fully qualified names [tarek]
+- Allow registering extra compiler classes in setup.cfg [tarek]
+- The EMX compiler is gone [tarek]
+- Standard command classes are now registered with set_commands;
+  command_packages is gone [tarek]
+- Move extension module to compiler.extension [tarek]
+- The compiler_type attribute of compiler classes is now name [tarek]
+- Document the setup.cfg file format in a specification that can be used by
+  non-Python developers [tarek, ?ric, julien j]
+- #10409: Fixed the License selector in mkcfg [tarek]
+- #9558: Fix build_ext with VS 8.0 [?ric]
+- #6007: Add disclaimer about MinGW compatibility in docs [?ric]
+- #11038: Add strict parameter to metadata.check, use it in check command [ga?l]
+- Support multiple files in description-file [ga?l]
+- Simplify and rename package_dir to packages_root: only one root directory for
+  all packages and modules is allowed [tarek]
+- Add util.generate_setup_py to create a setup.py script compatible with
+  distutils that takes information from a setup.cfg [julien m]
+- README or README.txt and test/test*.py are no longer included in sdists
+  [godefroid]
+- Move PEP 345 markers interpreter from metadata to markers [tarek]
+- #11057: Fix two NameErrors [?ric]
+- Add install.remove, the uninstall feature [ga?l]
+- Reuse info from existing setup.py when creating a setup.cfg with mkcfg [alain]
+- Add support for extension modules in setup.cfg [andr?]
+- Arguments that specify the target directory for install-related functions in
+  install don't have default values anymore [yannick]
+- Add paths argument to install.install* functions to allow working outside of
+  sys.path [tarek]
+- Metadata.check and check command now want an author field, even if maintainer
+  is supplied [godefroid]
+- Renamed DistributionMetadata to Metadata [chistophe]
+- Make --help-commands work again [?ric]
+- Fix index.dist.DistInfo.unpack to really use the path given as argument
+  [kelsey]
+- Introduce the resources system, a major overhaul of data_files installation
+  that supports putting files in various directories compliant with OS policies
+  and pain-free use of these files by Python code [boris, pierre-yves]
+- New util.iglob function supporting more patterns than stdlib glob
+  [pierre-yves]
+- Add 'pysetup create' to prompt the user and create a setup.cfg file with the
+  answers [tarek]
+- Convert and import the code into the CPython 3.3 repository [tarek, arc,
+  vinay, elson, walker, michael, kelsey, jason, alexis, ?ric, victor]; in the
+  distutils2 backport, change some names to match the CPython repo: exception
+  names start with "Packaging", there is a util.is_packaging function, etc.
+- Add bdist_wininst helpers compiled with Visual Studio 10 [jason]
+- 'pysetup install' now accepts a patch to a directory or to an archive in
+  addition to a PyPI project name [kelsey]
+- Rename mkcfg to create [?ric]
+- Add function to convert an egg-info file or directory to dist-info [kelsey]
+- Add functions to detect if a project uses distutils, setuptools or packaging
+  [kelsey, hugo]
+- Config fields that support environment markers (PEP 345) can now check
+  'platform.python_implementation' [alexis]
+- Use True and False instead of 0 and 1 (compatible change that makes the intent
+  of the code clearer) [?ric]
+- Rename 'pysetup list' the action used to look for installed projects [alexis]
+- Add 'pysetup search' to look for projects on an index/catalog [alexis]
+- Rename packaging.index back to packaging.pypi [kelsey]
+- Clean up logging setup, improve tests.support.LoggingCatcher [?ric]
+- Replace warnings by logging, remove display_warning argument of
+  Metadata.__init__, kill warn/announce/debug_print methods [?ric]
+- Improve EnvironGuard and rename it EnvironRestorer [?ric]
+- Move PEP 376 implementation from pkgutil to packaging.database [?ric]
+- Add version attribute to database.*Distribution classes [?ric]
+- #10419, #6011: Make sure build_scripts can handle non-ASCII path for the
+  Python interpreter used in shebangs [victor]
+- #12112, #12320, #9561: Use UTF-8 to read or write setup.cfg, setup.py and
+  METADATA files instead of the default, locale-dependent encoding [victor]
+- #12114: Fix potential deadlock or zombification in util._find_exe_version
+  [victor]
+- Use / as path separator in setup.cfg created by 'pysetup create' even on
+  Windows [tarek]
+- Use / as path separator in RECORD file even on Windows [tarek]
+- #6459: Fix the import symbol in extension modules [tarek]
+- #10126: Fix for python built in shared mode on Unix [tarek]
+- #10359: ';' after function definition is invalid in ISO C [?ric]
+- Remove the resources submodule, move its functions into resources [tarek]
+- Update the docs and move them to the CPython repository [kelsey, elson,
+  guillermoo, ?ric]
+- Add 'pysetup generate-setup' to expose util.generate_setup_py [tarek]
+- #11092: Make sdist always include setup.cfg [?ric]
+- #12246: Don?t try to install something when running from an uninstalled Python
+  built in its checkout [tschepang, ?ric]
+- Add packaging.util.split_multiline [julien m, erik]
+- #11595: Fix assorted bugs in packaging.util.cfg_to_args [erik, ?ric]
+- #12240: Allow multiple setup hooks [erik, ?ric]
+- #11637: Fix support for importing setup hooks from the project directory
+  [vinay, ?ric]
+- #9516: Revise deployment target processing for OS X [ned]
+- #12169, #10510: Factor out code used by various commands to make HTTP POST
+  requests, and make sure it uses CRLF [john, ?ric]
+- #12504: Close file handles in a timely manner in database; this fixes a bug
+  with the remove (uninstall) feature on Windows [thomas]
+- #11409, #12222: Let all pysetup actions return a meaningful 0 or 1 exit code
+  [kelsey, ?ric]
+- Add filesafe argument to Metadata.get_fullname and Distribution.get_fullname
+  [jeremy]
+- Change distutils2's setup.py script to get info from the setup.cfg [jeremy]
+- Add support for building OpenSSL on Windows (for _backports.hashlib) [jeremy]
+- Print all fields when calling 'pysetup metadata' without options, remove --all
+  option for metadata and list actions [?ric]
+- Remove display options (--name, etc.) from the Distribution class; this has
+  the side effect that 'url' is no longer accepted as key in the attrs argument
+  of the class' constructor, it needs to be 'home-page' to be recognized as
+  valid metadata field [?ric]
+- #10946: Make bdist_dumb, bdist_wininst and bdist_msi respect a --skip-build
+  option given to bdist [?ric]
+- The right-hand part in [extension: foo] (in a setup.cfg) is now used as the
+  name of the extension module [?ric]
+- #8933: METADATA files will now correctly report Metadata-Version: 1.1 instead
+  of 1.0 if a Classifier or Download-URL field is present [filip, ?ric]
+- Create a branch for a Python 3 version of distutils2 [?ric]
+- #10359: Make C code in one test comply with ISO C [hallvard]
+- #11254: Fix byte-compilation to comply with PEP 3147 on Python 3.2+ [?ric]
+- #13114: Add tests for Unicode handling in check and register [?ric]
+- #13170: Revert one of Jeremy's changes to config to fix a bug, kludge around
+  shlex not supporting unicode in 2.x, fix wrong shutil import [david, ?ric]
+- #13205: Fix and improve generated setup scripts [david, ?ric]
+
 
 1.0a3 - 2010-10-08
 ------------------
@@ -19,8 +157,8 @@
 - Fixed the installation when using easy_install and Pip by switching
   setup.py to distutils1 [holger/tarek]
 - Added missing c/h files in the MANIFEST so they are always present
-  no matter which Python version was used to build it. [holger/tarek]
-- Added the new setup runner that uses only setup.cfg
+  no matter which Python version was used to build it [holger/tarek]
+- Added pysetup, the new setup runner that uses only setup.cfg
 - Renamed mkpkg to mkcfg [tarek]
 - Renamed install_tools to install [alexis]
 
@@ -44,7 +182,7 @@
 - Remove PyPIRCCommand, move its helper code into util [tarek]
 - Remove Mac OS 9 support [?ric]
 - Start adding docstrings to interface methods [jeremy]
-- Move documentation from the stdlib [ali, ?ric]
+- Copy documentation from the stdlib [ali, ?ric]
 - Lots of bug fixes, cleanups, tests [everyone]
 
 
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -1,44 +1,69 @@
-============
-Contributors
-============
+=======================
+Distutils2 Contributors
+=======================
 
-Distutils2 is a project that was started and that is maintained by
-Tarek Ziad?, and many people are contributing to the project.
+The Distutils2 project was started by Tarek Ziad? and is currently
+maintained by ?ric Araujo.  Many people have contributed to the project.
 
-If you did, please add your name below in alphabetical order!
+If you're making a patch, please add your name below in alphabetical order,
+and welcome into the Fellowship of the Packaging!
 
 Thanks to:
 
 - Rajiv Abraham
 - Ali Afshar
-- ?ric Araujo
+- David Barnett
 - Pior Bastida
 - Anthony Baxter
-- Titus Brown
+- Erik Bray
+- C. Titus Brown
 - Nicolas Cadou
+- Godefroid Chapelle
+- Christophe Combelles
+- Jason R. Coombs
+- Pierre-Yves David
+- Ned Deily
 - Konrad Delong
 - Josip Djolonga
+- John Edmonds
+- Andr? Espaze
+- Boris Feld
 - Andrew Francis
+- Hallvard B Furuseth
 - Yannick Gingras
+- Filip Gruszczy?ski
+- guillermoo
+- Walker Hale IV
 - Alexandre Hamelin
 - Kelsey Hightower
+- Thomas Holmes
 - Christian Hudon
+- Julien Jehannet
 - Jeremy Kloth
 - Amos Latteier
 - Mathieu Leduc-Hamel
+- Tshepang Lekhonkhobe
+- Alain Leufroy
 - Martin von L?wis
+- Hugo Lopes Tavares
 - Simon Mathieu
 - Carl Meyer
 - Alexis M?taireau
+- Julien Miotte
 - Zubin Mithra
 - Derek McTavish Mounce
 - Michael Mulich
 - Louis Munro
+- Ga?l Pasgrimaud
 - George Peristerakis
 - Mathieu Perreault
 - Sean Reifschneider
 - Antoine Reversat
+- Arc Riley
+- Elson Rodriguez
 - Luis Rojas
 - Erik Rose
 - Brian Rosner
+- Vinay Sajip
+- Victor Stinner
 - Alexandre Vassalotti
diff --git a/DEVNOTES.txt b/DEVNOTES.txt
--- a/DEVNOTES.txt
+++ b/DEVNOTES.txt
@@ -1,16 +1,33 @@
-Notes for developers
+====================
+Notes for Developers
 ====================
 
-- Distutils2 runs on Python from 3.1 to 3.2 so make sure you don't use code
-  that doesn't work under one of these Python versions.  The version compatible
-  with 2.4-2.7 in the "default" branch.
+- Distutils2 has been merged into the CPython repository under the module name
+  "packaging", for release with Python 3.3.  The Distutils2 repository is used
+  to maintain the backport released on PyPI for older Python versions.  It is
+  recommended to work in the CPython repository if possible, but you but you
+  can make patches for the Distutils2 repository if you prefer.
+
+  Repo: http://hg.python.org/cpython (default branch)
+  More info: http://wiki.python.org/moin/Distutils/Contributing
+
+- Distutils2 runs on Python from 2.4 to 2.7, so make sure you don't use code
+  that doesn't work under one of these Python versions.  The version in the
+  "python3" branch is compatible with all version from 3.1 to 3.3.
 
 - When merging default into python3, don't keep maximum compatibility with
-  Python 2 but use idiomatic 3.x code.  For difficult conversions like encoding
-  handling with I/O, you can have a look or use a diff tool with the same file
-  in distutils or packaging from Python 3.3.
+  Python 2 but use idiomatic 3.x code, as long as it's compatible with all 3.x
+  versions.  For difficult conversions like encoding handling with I/O, you can
+  have a look or use a diff tool with the same file in distutils or packaging
+  from Python 3.3.  If you can't run tests, let someone else do the merge.
+
+- For 2.4, you need to run "python2.4 setup.py build" before you can try out
+  pysetup or run tests (unless you use the runtests.py script which will call
+  "setup.py build" automatically).
 
 - Always run tests.sh before you commit a change.  This implies that you have
-  Python 3.1 and 3.2 installed as well as a checkout of Python 3.3 with its
-  executable available as "python3.3" in your $PATH.  Be sure to also have
-  docutils installed on all Python versions to avoid skipping tests.
+  all Python versions installed from 2.4 to 2.7, as well as 3.1-.3.3 if you
+  merge into the python3 branch.  Be sure to also have docutils installed on all
+  Python versions to avoid skipping tests.
+
+- Don't forget to update CONTRIBUTORS.txt and CHANGES.txt.
diff --git a/README.txt b/README.txt
--- a/README.txt
+++ b/README.txt
@@ -1,24 +1,43 @@
-==========
-Distutils2
-==========
+=======================
+ Welcome to Distutils2
+=======================
 
-Welcome to Distutils2!
+Distutils2 is the packaging library that supersedes Distutils.  It has three
+main audiences:
 
-Distutils2 is the new version of Distutils. It's not backward compatible with
-Distutils but provides more features, and implement most new packaging
-standards.
+- Python authors who want to distribute their code
+- End users who want to install modules or applications
+- Developers of packaging-related tools who need a support library to
+  build on
 
-See the documentation at http://packages.python.org/Distutils2 for more info.
+Authors will have to write a :file:`setup.cfg` file and run a few
+commands to package and distribute their code.  End users will be able to
+search for, install and remove Python projects with the included
+``pysetup`` program.  Last, developers will be able to reuse classes and
+functions in their tools.
 
-If you want to contribute, please have a look to 
-http://distutils2.notmyidea.org/contributing.html
+The Distutils2 codebase is a fork of Distutils.  It is not backward compatible
+with Distutils and does not depend on it.  It provides more features and
+implements new packaging standards.  In Python 3.3, Distutils2 is included in
+the standard library under the module name "packaging".  Documentation is
+provided at http://docs.python.org/dev/packaging --for ease of maintenance, it
+is not duplicated in this repository.  You can use the Packaging documentation
+to use Distutils2; only the package name is different (packaging vs.
+distutils2), all modules, classes and functions have the same name.
 
-**Beware that Distutils2 is in its early stage and should not be used in
-production. Its API is subject to changes**
+If you want to contribute, please have a look at DEVNOTES.txt or
+http://wiki.python.org/Distutils2/Contributing .
 
-Useful further links:
+Beware that Distutils2 is still in alpha stage and its API is subject to
+change.  It should be not used for critical deployments.  That said, it
+is possible to start using it while keeping compatiblity with tools based
+on the old Distutils or Setuptools, and the developers are eager to get
+feedback from authors, end users and developers.
 
-Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/
-Documentation: http://packages.python.org/Distutils2
-Repository: http://hg.python.org/distutils2
-Bug tracker: http://bugs.python.org
+Useful links:
+
+- Mailing list: http://mail.python.org/mailman/listinfo/distutils-sig/
+- Mailing list with friendly tutors to guide new contributors:
+  http://mail.python.org/mailman/listinfo/core-mentorship
+- Repository: http://hg.python.org/distutils2
+- Bug tracker: http://bugs.python.org/ (component "Distutils2")
diff --git a/check.sh b/check.sh
--- a/check.sh
+++ b/check.sh
@@ -1,2 +1,4 @@
-pep8 distutils2
+# TODO exclude backports
 pyflakes distutils2
+pep8 -r distutils2
+# maybe:  exit $?
diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -197,12 +197,7 @@
     else:
         ignored_names = set()
 
-    try:
-        os.makedirs(dst)
-    except OSError as e:
-        if e.errno != errno.EEXIST:
-            raise
-
+    os.makedirs(dst)
     errors = []
     for name in names:
         if name in ignored_names:
diff --git a/distutils2/command/install_data.py b/distutils2/command/install_data.py
--- a/distutils2/command/install_data.py
+++ b/distutils2/command/install_data.py
@@ -3,10 +3,10 @@
 # Contributed by Bastian Kleineidam
 
 import os
-from shutil import Error
 from distutils2 import logger
 from distutils2.util import convert_path
 from distutils2.command.cmd import Command
+from distutils2._backport.shutil import Error
 from distutils2._backport.sysconfig import get_paths, format_value
 
 
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -104,7 +104,7 @@
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
-                    with open(resources_path, 'wb') as f:
+                    with open(resources_path, 'w') as f:
                         writer = csv.writer(f, delimiter=',',
                                             lineterminator='\n',
                                             quotechar='"')
diff --git a/distutils2/tests/test_command_install_data.py b/distutils2/tests/test_command_install_data.py
--- a/distutils2/tests/test_command_install_data.py
+++ b/distutils2/tests/test_command_install_data.py
@@ -1,28 +1,37 @@
 """Tests for distutils2.command.install_data."""
 import os
+import sys
+import distutils2.database
 from distutils2._backport import sysconfig
 from distutils2._backport.sysconfig import _get_default_scheme
 from distutils2.tests import unittest, support
 from distutils2.command.install_data import install_data
+from distutils2.command.install_dist import install_dist
+from distutils2.command.install_distinfo import install_distinfo
 
 
 class InstallDataTestCase(support.TempdirManager,
                           support.LoggingCatcher,
                           unittest.TestCase):
 
-    def test_simple_run(self):
+    def setUp(self):
+        super(InstallDataTestCase, self).setUp()
         scheme = _get_default_scheme()
         old_items = sysconfig._SCHEMES.items(scheme)
+
         def restore():
             sysconfig._SCHEMES.remove_section(scheme)
             sysconfig._SCHEMES.add_section(scheme)
             for option, value in old_items:
                 sysconfig._SCHEMES.set(scheme, option, value)
+
         self.addCleanup(restore)
 
+    def test_simple_run(self):
         pkg_dir, dist = self.create_dist()
         cmd = install_data(dist)
         cmd.install_dir = inst = os.path.join(pkg_dir, 'inst')
+        scheme = _get_default_scheme()
 
         sysconfig._SCHEMES.set(scheme, 'inst',
                                os.path.join(pkg_dir, 'inst'))
@@ -67,8 +76,7 @@
         three = os.path.join(cmd.install_dir, 'three')
         self.write_file(three, 'xx')
 
-        sysconfig._SCHEMES.set(scheme, 'inst3',
-                               cmd.install_dir)
+        sysconfig._SCHEMES.set(scheme, 'inst3', cmd.install_dir)
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
@@ -80,6 +88,62 @@
         self.assertTrue(os.path.exists(os.path.join(inst2, rtwo)))
         self.assertTrue(os.path.exists(os.path.join(inst, rone)))
 
+    def test_resources(self):
+        install_dir = self.mkdtemp()
+        scripts_dir = self.mkdtemp()
+        project_dir, dist = self.create_dist(
+            name='Spamlib', version='0.1',
+            data_files={'spamd': '{scripts}/spamd'})
+
+        os.chdir(project_dir)
+        self.write_file('spamd', '# Python script')
+        sysconfig._SCHEMES.set(_get_default_scheme(), 'scripts', scripts_dir)
+        sys.path.insert(0, install_dir)
+        distutils2.database.disable_cache()
+        self.addCleanup(sys.path.remove, install_dir)
+        self.addCleanup(distutils2.database.enable_cache)
+
+        cmd = install_dist(dist)
+        cmd.outputs = ['spamd']
+        cmd.install_lib = install_dir
+        dist.command_obj['install_dist'] = cmd
+
+        cmd = install_data(dist)
+        cmd.install_dir = install_dir
+        cmd.ensure_finalized()
+        dist.command_obj['install_data'] = cmd
+        cmd.run()
+
+        cmd = install_distinfo(dist)
+        cmd.ensure_finalized()
+        dist.command_obj['install_distinfo'] = cmd
+        cmd.run()
+
+        # first a few sanity checks
+        self.assertEqual(os.listdir(scripts_dir), ['spamd'])
+        self.assertEqual(os.listdir(install_dir), ['Spamlib-0.1.dist-info'])
+
+        # now the real test
+        fn = os.path.join(install_dir, 'Spamlib-0.1.dist-info', 'RESOURCES')
+        fp = open(fn)
+        try:
+            content = fp.read().strip()
+        finally:
+            fp.close()
+
+        expected = 'spamd,%s' % os.path.join(scripts_dir, 'spamd')
+        self.assertEqual(content, expected)
+
+        # just to be sure, we also test that get_file works here, even though
+        # packaging.database has its own test file
+        fp = distutils2.database.get_file('Spamlib', 'spamd')
+        try:
+            content = fp.read()
+        finally:
+            fp.close()
+
+        self.assertEqual('# Python script', content)
+
 
 def test_suite():
     return unittest.makeSuite(InstallDataTestCase)
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -1,4 +1,7 @@
-"""Tests for ``distutils2.command.install_distinfo``. """
+"""Tests for ``distutils2.command.install_distinfo``.
+
+Writing of the RESOURCES file is tested in test_command_install_data.
+"""
 
 import os
 import csv
diff --git a/distutils2/tests/test_pypi_simple.py b/distutils2/tests/test_pypi_simple.py
--- a/distutils2/tests/test_pypi_simple.py
+++ b/distutils2/tests/test_pypi_simple.py
@@ -87,7 +87,11 @@
         try:
             crawler._open_url(url)
         except Exception as v:
-            self.assertIn('nonnumeric port', str(v))
+            if sys.version_info[:2] < (3, 3):
+                wanted = 'nonnumeric port'
+            else:
+                wanted = 'Download error'
+            self.assertIn(wanted, str(v))
 
         # issue #160
         url = server.full_address
diff --git a/tests.sh b/tests.sh
--- a/tests.sh
+++ b/tests.sh
@@ -4,7 +4,7 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python3.1 -Wd runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
@@ -14,7 +14,7 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python3.2 -Wd runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
@@ -24,14 +24,14 @@
 if [ $? -ne 0 ];then
     echo Failed, re-running
     python3.3 -Wd runtests.py
-    exit 1
+    exit $?
 else
     echo Success
 fi
 
 if [ $? -ne 0 ];then
     echo Failed
-    exit 1
+    exit $?
 else
     echo "Good job, commit now! (or add tests)"
 fi

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Increase_test_coverage_f?=
 =?utf8?b?b3IgbWFuaWZlc3QgKCMxMTc1MSku?=
Message-ID: <E1RPxSB-00072i-Nt@dinsdale.python.org>

http://hg.python.org/distutils2/rev/3dda26cfc1d7
changeset:   1233:3dda26cfc1d7
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 11 23:36:39 2011 +0100
summary:
  Increase test coverage for manifest (#11751).

One test fails on versions older than 2.6 because of a change in
fnmatch; we?re discussing how best to fix that on the bug tracker and
another commit will follow up.

files:
  CHANGES.txt                       |    1 +
  CONTRIBUTORS.txt                  |    1 +
  distutils2/manifest.py            |    4 +-
  distutils2/tests/test_manifest.py |  201 +++++++++++++++++-
  4 files changed, 199 insertions(+), 8 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -148,6 +148,7 @@
 - #13170: Revert one of Jeremy's changes to config to fix a bug, kludge around
   shlex not supporting unicode in 2.x, fix wrong shutil import [david, ?ric]
 - #13205: Fix and improve generated setup scripts [david, ?ric]
+- #11751: Improve test coverage for manifest [justin]
 
 
 1.0a3 - 2010-10-08
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -46,6 +46,7 @@
 - Alain Leufroy
 - Martin von L?wis
 - Hugo Lopes Tavares
+- Justin Love
 - Simon Mathieu
 - Carl Meyer
 - Alexis M?taireau
diff --git a/distutils2/manifest.py b/distutils2/manifest.py
--- a/distutils2/manifest.py
+++ b/distutils2/manifest.py
@@ -153,7 +153,9 @@
 
     def _parse_template_line(self, line):
         words = line.split()
-        if len(words) == 1:
+        if len(words) == 1 and words[0] not in (
+              'include', 'exclude', 'global-include', 'global-exclude',
+              'recursive-include', 'recursive-exclude', 'graft', 'prune'):
             # no action given, let's use the default 'include'
             words.insert(0, 'include')
 
diff --git a/distutils2/tests/test_manifest.py b/distutils2/tests/test_manifest.py
--- a/distutils2/tests/test_manifest.py
+++ b/distutils2/tests/test_manifest.py
@@ -1,8 +1,10 @@
 """Tests for distutils2.manifest."""
 import os
+import re
 import logging
 from StringIO import StringIO
-from distutils2.manifest import Manifest
+from distutils2.errors import PackagingTemplateError
+from distutils2.manifest import Manifest, _translate_pattern, _glob_to_re
 
 from distutils2.tests import unittest, support
 
@@ -26,13 +28,11 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
 
-    def setUp(self):
-        super(ManifestTestCase, self).setUp()
-        self.cwd = os.getcwd()
+    def assertNoWarnings(self):
+        self.assertEqual(self.get_logs(logging.WARNING), [])
 
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(ManifestTestCase, self).tearDown()
+    def assertWarnings(self):
+        self.assertGreater(len(self.get_logs(logging.WARNING)), 0)
 
     def test_manifest_reader(self):
         tmpdir = self.mkdtemp()
@@ -75,6 +75,193 @@
         manifest.read_template(content)
         self.assertEqual(['README', 'file1'], manifest.files)
 
+    def test_glob_to_re(self):
+        # simple cases
+        self.assertEqual(_glob_to_re('foo*'), 'foo[^/]*\\Z(?ms)')
+        self.assertEqual(_glob_to_re('foo?'), 'foo[^/]\\Z(?ms)')
+        self.assertEqual(_glob_to_re('foo??'), 'foo[^/][^/]\\Z(?ms)')
+
+        # special cases
+        self.assertEqual(_glob_to_re(r'foo\\*'), r'foo\\\\[^/]*\Z(?ms)')
+        self.assertEqual(_glob_to_re(r'foo\\\*'), r'foo\\\\\\[^/]*\Z(?ms)')
+        self.assertEqual(_glob_to_re('foo????'), r'foo[^/][^/][^/][^/]\Z(?ms)')
+        self.assertEqual(_glob_to_re(r'foo\\??'), r'foo\\\\[^/][^/]\Z(?ms)')
+
+    def test_remove_duplicates(self):
+        manifest = Manifest()
+        manifest.files = ['a', 'b', 'a', 'g', 'c', 'g']
+        # files must be sorted beforehand
+        manifest.sort()
+        manifest.remove_duplicates()
+        self.assertEqual(manifest.files, ['a', 'b', 'c', 'g'])
+
+    def test_translate_pattern(self):
+        # blackbox test of a private function
+
+        # not regex
+        pattern = _translate_pattern('a', anchor=True, is_regex=False)
+        self.assertTrue(hasattr(pattern, 'search'))
+
+        # is a regex
+        regex = re.compile('a')
+        pattern = _translate_pattern(regex, anchor=True, is_regex=True)
+        self.assertEqual(pattern, regex)
+
+        # plain string flagged as regex
+        pattern = _translate_pattern('a', anchor=True, is_regex=True)
+        self.assertTrue(hasattr(pattern, 'search'))
+
+        # glob support
+        pattern = _translate_pattern('*.py', anchor=True, is_regex=False)
+        self.assertTrue(pattern.search('filelist.py'))
+
+    def test_exclude_pattern(self):
+        # return False if no match
+        manifest = Manifest()
+        self.assertFalse(manifest.exclude_pattern('*.py'))
+
+        # return True if files match
+        manifest = Manifest()
+        manifest.files = ['a.py', 'b.py']
+        self.assertTrue(manifest.exclude_pattern('*.py'))
+
+        # test excludes
+        manifest = Manifest()
+        manifest.files = ['a.py', 'a.txt']
+        manifest.exclude_pattern('*.py')
+        self.assertEqual(manifest.files, ['a.txt'])
+
+    def test_include_pattern(self):
+        # return False if no match
+        manifest = Manifest()
+        manifest.allfiles = []
+        self.assertFalse(manifest._include_pattern('*.py'))
+
+        # return True if files match
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt']
+        self.assertTrue(manifest._include_pattern('*.py'))
+
+        # test * matches all files
+        manifest = Manifest()
+        self.assertIsNone(manifest.allfiles)
+        manifest.allfiles = ['a.py', 'b.txt']
+        manifest._include_pattern('*')
+        self.assertEqual(manifest.allfiles, ['a.py', 'b.txt'])
+
+    def test_process_template(self):
+        # invalid lines
+        manifest = Manifest()
+        for action in ('include', 'exclude', 'global-include',
+                       'global-exclude', 'recursive-include',
+                       'recursive-exclude', 'graft', 'prune'):
+            self.assertRaises(PackagingTemplateError,
+                              manifest._process_template_line, action)
+
+        # implicit include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('*.py')
+        self.assertEqual(manifest.files, ['a.py'])
+        self.assertNoWarnings()
+
+        # include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('include *.py')
+        self.assertEqual(manifest.files, ['a.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('include *.rb')
+        self.assertEqual(manifest.files, ['a.py'])
+        self.assertWarnings()
+
+        # exclude
+        manifest = Manifest()
+        manifest.files = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('exclude *.py')
+        self.assertEqual(manifest.files, ['b.txt', 'd/c.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('exclude *.rb')
+        self.assertEqual(manifest.files, ['b.txt', 'd/c.py'])
+        self.assertWarnings()
+
+        # global-include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('global-include *.py')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('global-include *.rb')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.py'])
+        self.assertWarnings()
+
+        # global-exclude
+        manifest = Manifest()
+        manifest.files = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('global-exclude *.py')
+        self.assertEqual(manifest.files, ['b.txt'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('global-exclude *.rb')
+        self.assertEqual(manifest.files, ['b.txt'])
+        self.assertWarnings()
+
+        # recursive-include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'd/b.py', 'd/c.txt', 'd/d/e.py']
+
+        manifest._process_template_line('recursive-include d *.py')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('recursive-include e *.py')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertWarnings()
+
+        # recursive-exclude
+        manifest = Manifest()
+        manifest.files = ['a.py', 'd/b.py', 'd/c.txt', 'd/d/e.py']
+
+        manifest._process_template_line('recursive-exclude d *.py')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.txt'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('recursive-exclude e *.py')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.txt'])
+        self.assertWarnings()
+
+        # graft
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'd/b.py', 'd/d/e.py', 'f/f.py']
+
+        manifest._process_template_line('graft d')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('graft e')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertWarnings()
+
+        # prune
+        manifest = Manifest()
+        manifest.files = ['a.py', 'd/b.py', 'd/d/e.py', 'f/f.py']
+
+        manifest._process_template_line('prune d')
+        self.assertEqual(manifest.files, ['a.py', 'f/f.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('prune e')
+        self.assertEqual(manifest.files, ['a.py', 'f/f.py'])
+        self.assertWarnings()
+
 
 def test_suite():
     return unittest.makeSuite(ManifestTestCase)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_test_to_account_f?=
	=?utf8?q?or_recent_Python_bugfix=2E?=
Message-ID: <E1RPxSB-00072q-PN@dinsdale.python.org>

http://hg.python.org/distutils2/rev/529dbfb375cd
changeset:   1235:529dbfb375cd
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 00:39:02 2011 +0100
summary:
  Update test to account for recent Python bugfix.

This does not mean that we can run tests successfully with a Python 2.7
built from its repository: The version that will become 2.7.3 reports
itself as 2.7.2 (except for a ?+? in sys.version, but that?s an
implementation detail).  Some tests need access to Python.h and that
isn?t found when running from an uninstalled Python anyway.

files:
  distutils2/tests/test_pypi_simple.py |  6 +++++-
  1 files changed, 5 insertions(+), 1 deletions(-)


diff --git a/distutils2/tests/test_pypi_simple.py b/distutils2/tests/test_pypi_simple.py
--- a/distutils2/tests/test_pypi_simple.py
+++ b/distutils2/tests/test_pypi_simple.py
@@ -86,7 +86,11 @@
         try:
             crawler._open_url(url)
         except Exception, v:
-            self.assertIn('nonnumeric port', str(v))
+            if sys.version_info[:3] < (2, 7, 3):
+                wanted = 'nonnumeric port'
+            else:
+                wanted = 'Download error'
+            self.assertIn(wanted, str(v))
 
         # issue #160
         url = server.full_address

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Clean_up_some_idioms_in_?=
	=?utf8?q?tests=2E?=
Message-ID: <E1RPxSB-00072n-Os@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5d860a0bddae
changeset:   1234:5d860a0bddae
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 00:33:44 2011 +0100
summary:
  Clean up some idioms in tests.

- Use os.makedirs (I had forgotten about it!)
- Let TempdirManager.write_file call os.path.join for us
- Let TempdirManager.tearDown go back to the previous working directory
  for us
- Use a skip instead of hiding a method with an underscore
- Reset the finalized attribute of command objects before calling
  ensure_finalized a second time, so that it?s not a no-op
- Address pyflakes warnings

files:
  distutils2/tests/__main__.py                  |   1 -
  distutils2/tests/test_command_clean.py        |   9 +-
  distutils2/tests/test_command_install_data.py |   2 +
  distutils2/tests/test_command_install_dist.py |   9 +-
  distutils2/tests/test_command_test.py         |   3 +-
  distutils2/tests/test_command_upload.py       |   3 +-
  distutils2/tests/test_command_upload_docs.py  |   6 +-
  distutils2/tests/test_create.py               |   3 +-
  distutils2/tests/test_uninstall.py            |   8 +--
  distutils2/tests/test_util.py                 |  28 ++++-----
  distutils2/tests/test_version.py              |   4 +-
  11 files changed, 32 insertions(+), 44 deletions(-)


diff --git a/distutils2/tests/__main__.py b/distutils2/tests/__main__.py
--- a/distutils2/tests/__main__.py
+++ b/distutils2/tests/__main__.py
@@ -3,7 +3,6 @@
 # Ripped from importlib tests, thanks Brett!
 
 import os
-import sys
 from test.test_support import reap_children, reap_threads, run_unittest
 
 from distutils2.tests import unittest
diff --git a/distutils2/tests/test_command_clean.py b/distutils2/tests/test_command_clean.py
--- a/distutils2/tests/test_command_clean.py
+++ b/distutils2/tests/test_command_clean.py
@@ -5,7 +5,8 @@
 from distutils2.tests import unittest, support
 
 
-class cleanTestCase(support.TempdirManager, support.LoggingCatcher,
+class CleanTestCase(support.TempdirManager,
+                    support.LoggingCatcher,
                     unittest.TestCase):
 
     def test_simple_run(self):
@@ -23,7 +24,7 @@
             if name == 'build_base':
                 continue
             for f in ('one', 'two', 'three'):
-                self.write_file(os.path.join(path, f))
+                self.write_file((path, f))
 
         # let's run the command
         cmd.all = True
@@ -36,13 +37,11 @@
                              '%r was not removed' % path)
 
         # let's run the command again (should spit warnings but succeed)
-        cmd.all = True
-        cmd.ensure_finalized()
         cmd.run()
 
 
 def test_suite():
-    return unittest.makeSuite(cleanTestCase)
+    return unittest.makeSuite(CleanTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/distutils2/tests/test_command_install_data.py b/distutils2/tests/test_command_install_data.py
--- a/distutils2/tests/test_command_install_data.py
+++ b/distutils2/tests/test_command_install_data.py
@@ -62,6 +62,7 @@
 
         # let's try with warn_dir one
         cmd.warn_dir = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -80,6 +81,7 @@
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -93,21 +93,20 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        try:
-            # this is the actual test
-            self._test_user_site()
-        finally:
+        def cleanup():
             _CONFIG_VARS['userbase'] = self.old_user_base
             _SCHEMES.set(scheme, 'purelib', self.old_user_site)
             os.path.expanduser = self.old_expand
 
-    def _test_user_site(self):
+        self.addCleanup(cleanup)
+
         schemes = get_scheme_names()
         for key in ('nt_user', 'posix_user', 'os2_home'):
             self.assertIn(key, schemes)
 
         dist = Distribution({'name': 'xx'})
         cmd = install_dist(dist)
+
         # making sure the user option is there
         options = [name for name, short, lable in
                    cmd.user_options]
diff --git a/distutils2/tests/test_command_test.py b/distutils2/tests/test_command_test.py
--- a/distutils2/tests/test_command_test.py
+++ b/distutils2/tests/test_command_test.py
@@ -140,7 +140,8 @@
         cmd.run()
         self.assertEqual(['build has run'], record)
 
-    def _test_works_with_2to3(self):
+    @unittest.skip('needs to be written')
+    def test_works_with_2to3(self):
         pass
 
     def test_checks_requires(self):
diff --git a/distutils2/tests/test_command_upload.py b/distutils2/tests/test_command_upload.py
--- a/distutils2/tests/test_command_upload.py
+++ b/distutils2/tests/test_command_upload.py
@@ -44,6 +44,7 @@
 """
 
 
+#@skip if no threading, see end of file
 class UploadTestCase(support.TempdirManager, support.EnvironRestorer,
                      support.LoggingCatcher, PyPIServerTestCase):
 
@@ -129,7 +130,7 @@
         dist_files = [(command, pyversion, filename)]
         docs_path = os.path.join(self.tmp_dir, "build", "docs")
         os.makedirs(docs_path)
-        self.write_file(os.path.join(docs_path, "index.html"), "yellow")
+        self.write_file((docs_path, "index.html"), "yellow")
         self.write_file(self.rc, PYPIRC)
 
         # let's run it
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -32,6 +32,7 @@
 """
 
 
+#@skip if no threading, see end of file
 class UploadDocsTestCase(support.TempdirManager,
                          support.EnvironRestorer,
                          support.LoggingCatcher,
@@ -69,9 +70,8 @@
         if sample_dir is None:
             sample_dir = self.mkdtemp()
         os.mkdir(os.path.join(sample_dir, "docs"))
-        self.write_file(os.path.join(sample_dir, "docs", "index.html"),
-                        "Ce mortel ennui")
-        self.write_file(os.path.join(sample_dir, "index.html"), "Oh la la")
+        self.write_file((sample_dir, "docs", "index.html"), "Ce mortel ennui")
+        self.write_file((sample_dir, "index.html"), "Oh la la")
         return sample_dir
 
     def test_zip_dir(self):
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -82,8 +82,7 @@
             os.mkdir(os.path.join(tempdir, dir_))
 
         for file_ in files:
-            path = os.path.join(tempdir, file_)
-            self.write_file(path, 'xxx')
+            self.write_file((tempdir, file_), 'xxx')
 
         mainprogram._find_files()
         mainprogram.data['packages'].sort()
diff --git a/distutils2/tests/test_uninstall.py b/distutils2/tests/test_uninstall.py
--- a/distutils2/tests/test_uninstall.py
+++ b/distutils2/tests/test_uninstall.py
@@ -1,6 +1,5 @@
 """Tests for the distutils2.uninstall module."""
 import os
-import sys
 import logging
 import distutils2.util
 
@@ -31,16 +30,12 @@
 
     def setUp(self):
         super(UninstallTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        self.addCleanup(os.chdir, os.getcwd())
         self.addCleanup(enable_cache)
         self.root_dir = self.mkdtemp()
         self.cwd = os.getcwd()
         disable_cache()
 
     def tearDown(self):
-        os.chdir(self.cwd)
         distutils2.util._path_created.clear()
         super(UninstallTestCase, self).tearDown()
 
@@ -61,8 +56,7 @@
         kw['pkg'] = pkg
 
         pkg_dir = os.path.join(project_dir, pkg)
-        os.mkdir(pkg_dir)
-        os.mkdir(os.path.join(pkg_dir, 'sub'))
+        os.makedirs(os.path.join(pkg_dir, 'sub'))
 
         self.write_file((project_dir, 'setup.cfg'), SETUP_CFG % kw)
         self.write_file((pkg_dir, '__init__.py'), '#')
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -360,24 +360,20 @@
         #
         root = self.mkdtemp()
         pkg1 = os.path.join(root, 'pkg1')
-        os.mkdir(pkg1)
-        self.write_file(os.path.join(pkg1, '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg2'))
-        self.write_file(os.path.join(pkg1, 'pkg2', '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg3'))
-        self.write_file(os.path.join(pkg1, 'pkg3', '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg3', 'pkg6'))
-        self.write_file(os.path.join(pkg1, 'pkg3', 'pkg6', '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg4'))
-        os.mkdir(os.path.join(pkg1, 'pkg4', 'pkg8'))
-        self.write_file(os.path.join(pkg1, 'pkg4', 'pkg8', '__init__.py'))
-        pkg5 = os.path.join(root, 'pkg5')
-        os.mkdir(pkg5)
-        self.write_file(os.path.join(pkg5, '__init__.py'))
+        os.makedirs(os.path.join(pkg1, 'pkg2'))
+        os.makedirs(os.path.join(pkg1, 'pkg3', 'pkg6'))
+        os.makedirs(os.path.join(pkg1, 'pkg4', 'pkg8'))
+        os.makedirs(os.path.join(root, 'pkg5'))
+        self.write_file((pkg1, '__init__.py'))
+        self.write_file((pkg1, 'pkg2', '__init__.py'))
+        self.write_file((pkg1, 'pkg3', '__init__.py'))
+        self.write_file((pkg1, 'pkg3', 'pkg6', '__init__.py'))
+        self.write_file((pkg1, 'pkg4', 'pkg8', '__init__.py'))
+        self.write_file((root, 'pkg5', '__init__.py'))
 
         res = find_packages([root], ['pkg1.pkg2'])
-        self.assertEqual(set(res), set(['pkg1', 'pkg5', 'pkg1.pkg3',
-                                        'pkg1.pkg3.pkg6']))
+        self.assertEqual(sorted(res),
+                         ['pkg1', 'pkg1.pkg3', 'pkg1.pkg3.pkg6', 'pkg5'])
 
     def test_resolve_name(self):
         # test raw module name
diff --git a/distutils2/tests/test_version.py b/distutils2/tests/test_version.py
--- a/distutils2/tests/test_version.py
+++ b/distutils2/tests/test_version.py
@@ -1,6 +1,5 @@
 """Tests for distutils2.version."""
 import doctest
-import os
 
 from distutils2.version import NormalizedVersion as V
 from distutils2.version import HugeMajorVersionNumError, IrrationalVersionError
@@ -46,7 +45,6 @@
     def test_from_parts(self):
 
         for v, s in self.versions:
-            parts = v.parts
             v2 = V.from_parts(*v.parts)
             self.assertEqual(v, v2)
             self.assertEqual(str(v), str(v2))
@@ -192,7 +190,7 @@
                       'Hey (>=2.5,<2.7)')
 
         for predicate in predicates:
-            v = VersionPredicate(predicate)
+            VersionPredicate(predicate)
 
         self.assertTrue(VersionPredicate('Hey (>=2.5,<2.7)').match('2.6'))
         self.assertTrue(VersionPredicate('Ho').match('2.6'))

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Bring_back_callable=2E?=
Message-ID: <E1RPxSB-00072u-Pv@dinsdale.python.org>

http://hg.python.org/distutils2/rev/18aa8ff947f4
changeset:   1236:18aa8ff947f4
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 00:42:41 2011 +0100
summary:
  Bring back callable.

hasattr is subtly wrong, as special methods are looked up on the class,
and given that the callable builtin is back in Python 3.2 we may as well
use it.  Regular users shouldn?t see the DeprecationWarnings, and us
developers can just ignore thme when running tests.

files:
  distutils2/dist.py |  7 +++----
  distutils2/run.py  |  4 ++--
  2 files changed, 5 insertions(+), 6 deletions(-)


diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -409,13 +409,12 @@
             for help_option, short, desc, func in cmd_class.help_options:
                 if hasattr(opts, help_option.replace('-', '_')):
                     help_option_found = True
-                    if hasattr(func, '__call__'):
-                        func()
-                    else:
+                    if not callable(func):
                         raise PackagingClassError(
                             "invalid help function %r for help option %r: "
                             "must be a callable object (function, etc.)"
                             % (func, help_option))
+                    func()
 
             if help_option_found:
                 return
@@ -733,7 +732,7 @@
             else:
                 hook_obj = hook
 
-            if not hasattr(hook_obj, '__call__'):
+            if not callable(hook_obj):
                 raise PackagingOptionError('hook %r is not callable' % hook)
 
             logger.info('running %s %s for command %s',
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -368,7 +368,7 @@
     ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
-    ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),
+    ('generate-setup', 'Generate a backward-compatible setup.py', _generate),
 ]
 
 
@@ -500,7 +500,7 @@
             for help_option, short, desc, func in cmd_class.help_options:
                 if hasattr(opts, help_option.replace('-', '_')):
                     help_option_found = True
-                    if hasattr(func, '__call__'):
+                    if callable(func):
                         func()
                     else:
                         raise PackagingClassError(

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Change_signature_of_test?=
 =?utf8?q?s=2Esupport=2ELoggingCatcher=2Eget=5Flogs=2E?=
Message-ID: <E1RPxSB-000732-Sp@dinsdale.python.org>

http://hg.python.org/distutils2/rev/be248fc83a60
changeset:   1238:be248fc83a60
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 03:18:15 2011 +0100
summary:
  Change signature of tests.support.LoggingCatcher.get_logs.

I need this for some tests, and it makes code clearer.  This commit also
changes some assertEqual calls to use (actual, expected) order and fix
some pyflakes warnings.

files:
  CHANGES.txt                                  |   2 +
  distutils2/tests/support.py                  |  39 ++++----
  distutils2/tests/test_command_check.py       |  45 ++++-----
  distutils2/tests/test_command_cmd.py         |   3 +-
  distutils2/tests/test_command_sdist.py       |  21 ++--
  distutils2/tests/test_command_test.py        |   3 +-
  distutils2/tests/test_command_upload_docs.py |  10 +-
  distutils2/tests/test_config.py              |   8 +-
  distutils2/tests/test_dist.py                |   6 +-
  distutils2/tests/test_manifest.py            |   7 +-
  distutils2/tests/test_metadata.py            |   5 +-
  11 files changed, 72 insertions(+), 77 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -151,6 +151,8 @@
 - #11751: Improve test coverage for manifest [justin]
 - Byte compilation is now isolated from the calling Python -B or -O options
   [?ric]
+- The signature of tests.support.LoggingCatcher.get_logs changed, see
+  docstring [?ric]
 
 
 1.0a3 - 2010-10-08
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -90,10 +90,13 @@
     configured to record all messages logged to the 'distutils2' logger.
 
     Use get_logs to retrieve messages and self.loghandler.flush to discard
-    them.  get_logs automatically flushes the logs; if you test code that
-    generates logging messages but don't use get_logs, you have to flush
-    manually before doing other checks on logging message, otherwise you
-    will get irrelevant results.  See example in test_command_check.
+    them.  get_logs automatically flushes the logs, unless you pass
+    *flush=False*, for example to make multiple calls to the method with
+    different level arguments.  If your test calls some code that generates
+    logging message and then you don't call get_logs, you will need to flush
+    manually before testing other code in the same test_* method, otherwise
+    get_logs in the next lines will see messages from the previous lines.
+    See example in test_command_check.
     """
 
     def setUp(self):
@@ -117,25 +120,23 @@
         logger2to3.setLevel(self._old_levels[1])
         super(LoggingCatcher, self).tearDown()
 
-    def get_logs(self, *levels):
-        """Return all log messages with level in *levels*.
+    def get_logs(self, level=logging.WARNING, flush=True):
+        """Return all log messages with given level.
 
-        Without explicit levels given, returns all messages.  *levels* defaults
-        to all levels.  For log calls with arguments (i.e.
-        logger.info('bla bla %r', arg)), the messages will be formatted before
-        being returned (e.g. "bla bla 'thing'").
+        *level* defaults to logging.WARNING.
+
+        For log calls with arguments (i.e.  logger.info('bla bla %r', arg)),
+        the messages will be formatted before being returned (e.g. "bla bla
+        'thing'").
 
         Returns a list.  Automatically flushes the loghandler after being
-        called.
-
-        Example: self.get_logs(logging.WARN, logging.DEBUG).
+        called, unless *flush* is False (this is useful to get e.g. all
+        warnings then all info messages).
         """
-        if not levels:
-            messages = [log.getMessage() for log in self.loghandler.buffer]
-        else:
-            messages = [log.getMessage() for log in self.loghandler.buffer
-                        if log.levelno in levels]
-        self.loghandler.flush()
+        messages = [log.getMessage() for log in self.loghandler.buffer
+                    if log.levelno == level]
+        if flush:
+            self.loghandler.flush()
         return messages
 
 
diff --git a/distutils2/tests/test_command_check.py b/distutils2/tests/test_command_check.py
--- a/distutils2/tests/test_command_check.py
+++ b/distutils2/tests/test_command_check.py
@@ -1,6 +1,5 @@
 """Tests for distutils.command.check."""
 
-import logging
 from distutils2.command.check import check
 from distutils2.metadata import _HAS_DOCUTILS
 from distutils2.errors import PackagingSetupError, MetadataMissingError
@@ -27,11 +26,11 @@
         # let's run the command with no metadata at all
         # by default, check is checking the metadata
         # should have some warnings
-        cmd = self._run()
+        self._run()
         # trick: using assertNotEqual with an empty list will give us a more
         # useful error message than assertGreater(.., 0) when the code change
         # and the test fails
-        self.assertNotEqual([], self.get_logs(logging.WARNING))
+        self.assertNotEqual(self.get_logs(), [])
 
         # now let's add the required fields
         # and run it again, to make sure we don't get
@@ -40,8 +39,8 @@
                     'author_email': 'xxx',
                     'name': 'xxx', 'version': '4.2',
                     }
-        cmd = self._run(metadata)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata)
+        self.assertEqual(self.get_logs(), [])
 
         # now with the strict mode, we should
         # get an error if there are missing metadata
@@ -53,8 +52,8 @@
         self.loghandler.flush()
 
         # and of course, no error when all metadata fields are present
-        cmd = self._run(metadata, strict=True)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata, strict=True)
+        self.assertEqual(self.get_logs(), [])
 
         # now a test with non-ASCII characters
         metadata = {'home_page': u'xxx', 'author': u'\u00c9ric',
@@ -62,15 +61,15 @@
                     'version': u'1.2',
                     'summary': u'Something about esszet \u00df',
                     'description': u'More things about esszet \u00df'}
-        cmd = self._run(metadata)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata)
+        self.assertEqual(self.get_logs(), [])
 
     def test_check_metadata_1_2(self):
         # let's run the command with no metadata at all
         # by default, check is checking the metadata
         # should have some warnings
-        cmd = self._run()
-        self.assertNotEqual([], self.get_logs(logging.WARNING))
+        self._run()
+        self.assertNotEqual(self.get_logs(), [])
 
         # now let's add the required fields and run it again, to make sure we
         # don't get any warning anymore let's use requires_python as a marker
@@ -80,8 +79,8 @@
                     'name': 'xxx', 'version': '4.2',
                     'requires_python': '2.4',
                     }
-        cmd = self._run(metadata)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata)
+        self.assertEqual(self.get_logs(), [])
 
         # now with the strict mode, we should
         # get an error if there are missing metadata
@@ -99,8 +98,8 @@
 
         # now with correct version format again
         metadata['version'] = '4.2'
-        cmd = self._run(metadata, strict=True)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata, strict=True)
+        self.assertEqual(self.get_logs(), [])
 
     @unittest.skipUnless(_HAS_DOCUTILS, "requires docutils")
     def test_check_restructuredtext(self):
@@ -109,9 +108,7 @@
         pkg_info, dist = self.create_dist(description=broken_rest)
         cmd = check(dist)
         cmd.check_restructuredtext()
-        self.assertEqual(len(self.get_logs(logging.WARNING)), 1)
-        # clear warnings from the previous call
-        self.loghandler.flush()
+        self.assertEqual(len(self.get_logs()), 1)
 
         # let's see if we have an error with strict=1
         metadata = {'home_page': 'xxx', 'author': 'xxx',
@@ -126,7 +123,7 @@
         dist = self.create_dist(description=u'title\n=====\n\ntest \u00df')[1]
         cmd = check(dist)
         cmd.check_restructuredtext()
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertEqual(self.get_logs(), [])
 
     def test_check_all(self):
         self.assertRaises(PackagingSetupError, self._run,
@@ -143,18 +140,18 @@
         }
         cmd = check(dist)
         cmd.check_hooks_resolvable()
-        self.assertEqual(len(self.get_logs(logging.WARNING)), 1)
+        self.assertEqual(len(self.get_logs()), 1)
 
     def test_warn(self):
         _, dist = self.create_dist()
         cmd = check(dist)
-        self.assertEqual([], self.get_logs())
+        self.assertEqual(self.get_logs(), [])
         cmd.warn('hello')
-        self.assertEqual(['check: hello'], self.get_logs())
+        self.assertEqual(self.get_logs(), ['check: hello'])
         cmd.warn('hello %s', 'world')
-        self.assertEqual(['check: hello world'], self.get_logs())
+        self.assertEqual(self.get_logs(), ['check: hello world'])
         cmd.warn('hello %s %s', 'beautiful', 'world')
-        self.assertEqual(['check: hello beautiful world'], self.get_logs())
+        self.assertEqual(self.get_logs(), ['check: hello beautiful world'])
 
 
 def test_suite():
diff --git a/distutils2/tests/test_command_cmd.py b/distutils2/tests/test_command_cmd.py
--- a/distutils2/tests/test_command_cmd.py
+++ b/distutils2/tests/test_command_cmd.py
@@ -1,5 +1,6 @@
 """Tests for distutils.cmd."""
 import os
+import logging
 
 from distutils2.command.cmd import Command
 from distutils2.dist import Distribution
@@ -43,7 +44,7 @@
 
         wanted = ["command options for 'MyCmd':", '  option1 = 1',
                   '  option2 = 1']
-        msgs = self.get_logs()
+        msgs = self.get_logs(logging.INFO)
         self.assertEqual(msgs, wanted)
 
     def test_ensure_string(self):
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -1,7 +1,6 @@
 """Tests for distutils2.command.sdist."""
 import os
 import zipfile
-import logging
 
 from distutils2.tests.support import requires_zlib
 
@@ -226,12 +225,14 @@
         # testing the `check-metadata` option
         dist, cmd = self.get_cmd(metadata={'name': 'xxx', 'version': 'xxx'})
 
-        # this should raise some warnings
-        # with the check subcommand
+        # this should cause the check subcommand to log two warnings:
+        # version is invalid, home-page and author are missing
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(logging.WARN)
-        self.assertEqual(len(warnings), 4)
+        warnings = self.get_logs()
+        check_warnings = [msg for msg in warnings if
+                          not msg.startswith('sdist:')]
+        self.assertEqual(len(check_warnings), 2, warnings)
 
         # trying with a complete set of metadata
         self.loghandler.flush()
@@ -239,13 +240,10 @@
         cmd.ensure_finalized()
         cmd.metadata_check = False
         cmd.run()
-        warnings = self.get_logs(logging.WARN)
-        # removing manifest generated warnings
-        warnings = [warn for warn in warnings if
-                    not warn.endswith('-- skipping')]
-        # the remaining warnings are about the use of the default file list and
-        # the absence of setup.cfg
+        warnings = self.get_logs()
         self.assertEqual(len(warnings), 2)
+        self.assertIn('using default file list', warnings[0])
+        self.assertIn("'setup.cfg' file not found", warnings[1])
 
     def test_show_formats(self):
         __, stdout = captured_stdout(show_formats)
@@ -257,7 +255,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
diff --git a/distutils2/tests/test_command_test.py b/distutils2/tests/test_command_test.py
--- a/distutils2/tests/test_command_test.py
+++ b/distutils2/tests/test_command_test.py
@@ -2,7 +2,6 @@
 import re
 import sys
 import shutil
-import logging
 import unittest as ut1
 import distutils2.database
 
@@ -150,7 +149,7 @@
         phony_project = 'ohno_ohno-impossible_1234-name_stop-that!'
         cmd.tests_require = [phony_project]
         cmd.ensure_finalized()
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertIn(phony_project, logs[-1])
 
     def prepare_a_module(self):
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -1,6 +1,7 @@
 """Tests for distutils2.command.upload_docs."""
 import os
 import shutil
+import logging
 import zipfile
 try:
     import _ssl
@@ -140,13 +141,16 @@
         self.pypi.default_response_status = '403 Forbidden'
         self.prepare_command()
         self.cmd.run()
-        self.assertIn('Upload failed (403): Forbidden', self.get_logs()[-1])
+        errors = self.get_logs(logging.ERROR)
+        self.assertEqual(len(errors), 1)
+        self.assertIn('Upload failed (403): Forbidden', errors[0])
 
         self.pypi.default_response_status = '301 Moved Permanently'
         self.pypi.default_response_headers.append(
             ("Location", "brand_new_location"))
         self.cmd.run()
-        self.assertIn('brand_new_location', self.get_logs()[-1])
+        lastlog = self.get_logs(logging.INFO)[-1]
+        self.assertIn('brand_new_location', lastlog)
 
     def test_reads_pypirc_data(self):
         self.write_file(self.rc, PYPIRC % self.pypi.full_address)
@@ -170,7 +174,7 @@
         self.prepare_command()
         self.cmd.show_response = True
         self.cmd.run()
-        record = self.get_logs()[-1]
+        record = self.get_logs(logging.INFO)[-1]
         self.assertTrue(record, "should report the response")
         self.assertIn(self.pypi.default_response_data, record)
 
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -2,7 +2,6 @@
 """Tests for distutils2.config."""
 import os
 import sys
-import logging
 from StringIO import StringIO
 
 from distutils2 import command
@@ -376,15 +375,14 @@
         self.write_file('README', 'yeah')
         self.write_file('hooks.py', HOOKS_MODULE)
         self.get_dist()
-        logs = self.get_logs(logging.WARNING)
-        self.assertEqual(['logging_hook called'], logs)
+        self.assertEqual(['logging_hook called'], self.get_logs())
         self.assertIn('hooks', sys.modules)
 
     def test_missing_setup_hook_warns(self):
         self.write_setup({'setup-hooks': 'this.does._not.exist'})
         self.write_file('README', 'yeah')
         self.get_dist()
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(1, len(logs))
         self.assertIn('cannot find setup hook', logs[0])
 
@@ -398,7 +396,7 @@
         dist = self.get_dist()
 
         self.assertEqual(['haven', 'first', 'third'], dist.py_modules)
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(1, len(logs))
         self.assertIn('cannot find setup hook', logs[0])
 
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -1,8 +1,6 @@
 """Tests for distutils2.dist."""
 import os
 import sys
-import codecs
-import logging
 import textwrap
 
 import distutils2.dist
@@ -78,7 +76,7 @@
                             'version': '1.2',
                             'home_page': 'xxxx',
                             'badoptname': 'xxx'})
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(len(logs), 1)
         self.assertIn('unknown argument', logs[0])
 
@@ -89,7 +87,7 @@
                                    'version': '1.2', 'home_page': 'xxxx',
                                    'options': {}})
 
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertEqual(self.get_logs(), [])
         self.assertNotIn('options', dir(dist))
 
     def test_non_empty_options(self):
diff --git a/distutils2/tests/test_manifest.py b/distutils2/tests/test_manifest.py
--- a/distutils2/tests/test_manifest.py
+++ b/distutils2/tests/test_manifest.py
@@ -1,7 +1,6 @@
 """Tests for distutils2.manifest."""
 import os
 import re
-import logging
 from StringIO import StringIO
 from distutils2.errors import PackagingTemplateError
 from distutils2.manifest import Manifest, _translate_pattern, _glob_to_re
@@ -29,10 +28,10 @@
                        unittest.TestCase):
 
     def assertNoWarnings(self):
-        self.assertEqual(self.get_logs(logging.WARNING), [])
+        self.assertEqual(self.get_logs(), [])
 
     def assertWarnings(self):
-        self.assertGreater(len(self.get_logs(logging.WARNING)), 0)
+        self.assertNotEqual(self.get_logs(), [])
 
     def test_manifest_reader(self):
         tmpdir = self.mkdtemp()
@@ -46,7 +45,7 @@
         manifest = Manifest()
         manifest.read_template(MANIFEST)
 
-        warnings = self.get_logs(logging.WARNING)
+        warnings = self.get_logs()
         # the manifest should have been read and 3 warnings issued
         # (we didn't provide the files)
         self.assertEqual(3, len(warnings))
diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py
--- a/distutils2/tests/test_metadata.py
+++ b/distutils2/tests/test_metadata.py
@@ -3,7 +3,6 @@
 import os
 import sys
 import codecs
-import logging
 from textwrap import dedent
 from StringIO import StringIO
 
@@ -318,7 +317,7 @@
                           'name': 'xxx',
                           'version': 'xxx',
                           'home_page': 'xxxx'})
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(1, len(logs))
         self.assertIn('not a valid version', logs[0])
 
@@ -440,7 +439,7 @@
         # XXX check PEP and see if 3 == 3.0
         metadata['Requires-Python'] = '>=2.6, <3.0'
         metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertEqual(self.get_logs(), [])
 
     @unittest.skip('needs to be implemented')
     def test_requires_illegal(self):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Minor_assorted_cleanups?=
	=?utf8?q?=2E?=
Message-ID: <E1RPxSB-00073C-V6@dinsdale.python.org>

http://hg.python.org/distutils2/rev/b65e794099e3
changeset:   1241:b65e794099e3
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 04:29:11 2011 +0100
summary:
  Minor assorted cleanups.

- Remove __main__ blocks obsoleted by pysetup
- Fix typo ?seperate?
- Add one test that was promised but not written
- Reorganize one file

files:
  distutils2/command/build.py                 |   2 +-
  distutils2/create.py                        |   4 -
  distutils2/depgraph.py                      |   5 +-
  distutils2/dist.py                          |   2 +-
  distutils2/install.py                       |   9 -
  distutils2/tests/test_command_bdist_dumb.py |  17 ++-
  distutils2/tests/test_command_register.py   |   1 +
  distutils2/tests/test_command_sdist.py      |  47 ++++-----
  8 files changed, 39 insertions(+), 48 deletions(-)


diff --git a/distutils2/command/build.py b/distutils2/command/build.py
--- a/distutils2/command/build.py
+++ b/distutils2/command/build.py
@@ -41,7 +41,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -698,7 +698,3 @@
     # program.write_setup_script()
     # distutils2.util.cfg_to_args()
     program()
-
-
-if __name__ == '__main__':
-    main()
diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py
--- a/distutils2/depgraph.py
+++ b/distutils2/depgraph.py
@@ -224,6 +224,7 @@
 
 
 def main():
+    # XXX move to run._graph
     from distutils2.database import get_distributions
     tempout = StringIO()
     try:
@@ -270,7 +271,3 @@
     else:
         print 'Supported option: -d [filename]'
         sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -69,7 +69,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ]
     display_option_names = [x[0].replace('-', '_') for x in display_options]
 
diff --git a/distutils2/install.py b/distutils2/install.py
--- a/distutils2/install.py
+++ b/distutils2/install.py
@@ -531,12 +531,3 @@
             logger.info('%r conflicts with %s', project, ','.join(projects))
 
     return True
-
-
-def _main(**attrs):
-    if 'script_args' not in attrs:
-        attrs['requirements'] = sys.argv[1]
-    get_infos(**attrs)
-
-if __name__ == '__main__':
-    _main()
diff --git a/distutils2/tests/test_command_bdist_dumb.py b/distutils2/tests/test_command_bdist_dumb.py
--- a/distutils2/tests/test_command_bdist_dumb.py
+++ b/distutils2/tests/test_command_bdist_dumb.py
@@ -1,6 +1,7 @@
 """Tests for distutils.command.bdist_dumb."""
 
 import os
+import zipfile
 import distutils2.util
 
 from distutils2.dist import Distribution
@@ -49,15 +50,23 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        fp = zipfile.ZipFile(os.path.join('dist', base))
+        try:
+            contents = fp.namelist()
+        finally:
+            fp.close()
+
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo.py', 'foo.pyc',
+                  'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(contents, sorted(wanted))
 
     def test_finalize_options(self):
         pkg_dir, dist = self.create_dist()
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -144,6 +144,7 @@
 
         register_module.input = _no_way
         cmd.show_response = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -2,8 +2,6 @@
 import os
 import zipfile
 
-from distutils2.tests.support import requires_zlib
-
 try:
     import grp
     import pwd
@@ -12,17 +10,17 @@
     UID_GID_SUPPORT = False
 
 from os.path import join
-from distutils2.tests import captured_stdout
-from distutils2.command.sdist import sdist
-from distutils2.command.sdist import show_formats
 from distutils2.dist import Distribution
-from distutils2.tests import unittest
+from distutils2.util import find_executable
 from distutils2.errors import PackagingOptionError
-from distutils2.util import find_executable
-from distutils2.tests import support
+from distutils2.command.sdist import sdist, show_formats
 from distutils2._backport import tarfile
 from distutils2._backport.shutil import get_archive_formats
 
+from distutils2.tests import support, unittest
+from distutils2.tests import captured_stdout
+from distutils2.tests.support import requires_zlib
+
 
 MANIFEST = """\
 # file GENERATED by distutils2, do NOT edit
@@ -88,7 +86,6 @@
 
         # creating VCS directories with some files in them
         os.mkdir(join(self.tmp_dir, 'somecode', '.svn'))
-
         self.write_file((self.tmp_dir, 'somecode', '.svn', 'ok.py'), 'xxx')
 
         os.mkdir(join(self.tmp_dir, 'somecode', '.hg'))
@@ -146,7 +143,7 @@
 
         # now trying a tar then a gztar
         cmd.formats = ['tar', 'gztar']
-
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -274,6 +271,21 @@
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     @requires_zlib
+    def test_template(self):
+        dist, cmd = self.get_cmd()
+        dist.extra_files = ['include yeah']
+        cmd.ensure_finalized()
+        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
+        cmd.run()
+        f = open(cmd.manifest)
+        try:
+            content = f.read()
+        finally:
+            f.close()
+
+        self.assertIn('yeah', content)
+
+    @requires_zlib
     @unittest.skipUnless(UID_GID_SUPPORT, "requires grp and pwd support")
     @unittest.skipIf(find_executable('tar') is None or
                      find_executable('gzip') is None,
@@ -395,21 +407,6 @@
         self.assertEqual(manifest, ['README.manual'])
 
     @requires_zlib
-    def test_template(self):
-        dist, cmd = self.get_cmd()
-        dist.extra_files = ['include yeah']
-        cmd.ensure_finalized()
-        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
-        cmd.run()
-        f = open(cmd.manifest)
-        try:
-            content = f.read()
-        finally:
-            f.close()
-
-        self.assertIn('yeah', content)
-
-    @requires_zlib
     def test_manifest_builder(self):
         dist, cmd = self.get_cmd()
         cmd.manifest_builders = 'distutils2.tests.test_command_sdist.builder'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Make_sure_tests_that_reg?=
 =?utf8?q?ister_custom_commands_also_clear_them?=
Message-ID: <E1RPxSB-00073G-WB@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7223eb4f1b5c
changeset:   1242:7223eb4f1b5c
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 04:59:36 2011 +0100
summary:
  Make sure tests that register custom commands also clear them

files:
  distutils2/tests/support.py     |  15 +++++++++++-
  distutils2/tests/test_config.py |  24 ++++++------------
  distutils2/tests/test_dist.py   |  26 +++++++++++++-------
  3 files changed, 39 insertions(+), 26 deletions(-)


diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -49,6 +49,9 @@
     zlib = None
 
 from distutils2.dist import Distribution
+from distutils2.util import resolve_name
+from distutils2.command import set_command, _COMMANDS
+
 from distutils2.tests import unittest
 from distutils2._backport import sysconfig
 
@@ -59,7 +62,8 @@
     # mocks
     'DummyCommand', 'TestDistribution',
     # misc. functions and decorators
-    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    'fake_dec', 'create_distribution', 'use_command',
+    'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
     'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
@@ -291,6 +295,15 @@
     return d
 
 
+def use_command(testcase, fullname):
+    """Register command at *fullname* for the duration of a test."""
+    set_command(fullname)
+    # XXX maybe set_command should return the class object
+    name = resolve_name(fullname).get_command_name()
+    # XXX maybe we need a public API to remove commands
+    testcase.addCleanup(_COMMANDS.__delitem__, name)
+
+
 def fake_dec(*args, **kw):
     """Fake decorator"""
     def _wrap(func):
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -2,7 +2,6 @@
 """Tests for distutils2.config."""
 import os
 import sys
-from StringIO import StringIO
 
 from distutils2 import command
 from distutils2.dist import Distribution
@@ -184,13 +183,14 @@
 
     def __init__(self, dist):
         self.distribution = dist
+        self._record = []
 
     @classmethod
     def get_command_name(cls):
         return 'foo'
 
     def run(self):
-        self.distribution.foo_was_here = True
+        self._record.append('foo has run')
 
     def nothing(self):
         pass
@@ -210,21 +210,11 @@
 
     def setUp(self):
         super(ConfigTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        sys.stdout = StringIO()
-        sys.stderr = StringIO()
-
-        self.addCleanup(os.chdir, os.getcwd())
         tempdir = self.mkdtemp()
         self.working_dir = os.getcwd()
         os.chdir(tempdir)
         self.tempdir = tempdir
 
-    def tearDown(self):
-        os.chdir(self.working_dir)
-        super(ConfigTestCase, self).tearDown()
-
     def write_setup(self, kwargs=None):
         opts = {'description-file': 'README', 'extra-files': '',
                 'setup-hooks': 'distutils2.tests.test_config.version_hook'}
@@ -379,7 +369,7 @@
         self.assertIn('hooks', sys.modules)
 
     def test_missing_setup_hook_warns(self):
-        self.write_setup({'setup-hooks': 'this.does._not.exist'})
+        self.write_setup({'setup-hooks': 'does._not.exist'})
         self.write_file('README', 'yeah')
         self.get_dist()
         logs = self.get_logs()
@@ -495,10 +485,12 @@
             self.write_file((pkg, '__init__.py'), '#')
 
         # try to run the install command to see if foo is called
+        self.addCleanup(command._COMMANDS.__delitem__, 'foo')
         dist = self.get_dist()
-        self.assertIn('foo', command.get_command_names())
-        self.assertEqual('FooBarBazTest',
-                         dist.get_command_obj('foo').__class__.__name__)
+        dist.run_command('install_dist')
+        cmd = dist.get_command_obj('foo')
+        self.assertEqual(cmd.__class__.__name__, 'FooBarBazTest')
+        self.assertEqual(cmd._record, ['foo has run'])
 
 
 def test_suite():
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -6,27 +6,32 @@
 import distutils2.dist
 
 from distutils2.dist import Distribution
-from distutils2.command import set_command
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingModuleError, PackagingOptionError
 from distutils2.tests import captured_stdout
 from distutils2.tests import support, unittest
-from distutils2.tests.support import create_distribution
+from distutils2.tests.support import create_distribution, use_command
 from distutils2.tests.support import unload
 
 
 class test_dist(Command):
-    """Sample distutils2 extension command."""
+    """Custom command used for testing."""
 
     user_options = [
-        ("sample-option=", "S", "help text"),
+        ('sample-option=', 'S',
+         "help text"),
         ]
 
     def initialize_options(self):
         self.sample_option = None
+        self._record = []
 
     def finalize_options(self):
-        pass
+        if self.sample_option is None:
+            self.sample_option = 'default value'
+
+    def run(self):
+        self._record.append('test_dist has run')
 
 
 class DistributionTestCase(support.TempdirManager,
@@ -38,6 +43,8 @@
 
     def setUp(self):
         super(DistributionTestCase, self).setUp()
+        # XXX this is ugly, we should fix the functions to accept args
+        # (defaulting to sys.argv)
         self.argv = sys.argv, sys.argv[:]
         del sys.argv[1:]
 
@@ -181,7 +188,8 @@
         self.write_file((temp_home, "config2.cfg"),
                         '[test_dist]\npre-hook.b = type')
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
+
         dist = create_distribution(config_files)
         cmd = dist.get_command_obj("test_dist")
         self.assertEqual(cmd.pre_hook, {"a": 'type', "b": 'type'})
@@ -209,7 +217,7 @@
                 record.append('post-%s' % cmd.get_command_name())
             '''))
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
 
@@ -236,7 +244,7 @@
             [test_dist]
             pre-hook.test = nonexistent.dotted.name'''))
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
@@ -251,7 +259,7 @@
             [test_dist]
             pre-hook.test = distutils2.tests.test_dist.__doc__'''))
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:08 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Undo_potentially_confusi?=
	=?utf8?q?ng_name_change=2E?=
Message-ID: <E1RPxSC-00073K-0m@dinsdale.python.org>

http://hg.python.org/distutils2/rev/b9c9600aaca2
changeset:   1243:b9c9600aaca2
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 05:08:01 2011 +0100
summary:
  Undo potentially confusing name change.

This method was named reinitialize_command in distutils and accompanied
by a comment suggesting to change it to get_reinitialized_command.
Following that, I did the change for distutils2, but it proved
confusing: The Distribution object has an internal cache of command
objects, to make sure only one instance is ever used, and the name
get_reinitialized_command could suggest that the object returned was
independent of that cache, which it was not.  I?m reverting the name
change to make code clearer.

files:
  CHANGES.txt                         |   1 +
  distutils2/command/bdist.py         |   2 +-
  distutils2/command/bdist_dumb.py    |   4 ++--
  distutils2/command/bdist_msi.py     |   6 +++---
  distutils2/command/bdist_wininst.py |   5 ++---
  distutils2/command/cmd.py           |   4 ++--
  distutils2/command/test.py          |   2 +-
  distutils2/dist.py                  |  15 ++++++++-------
  distutils2/tests/support.py         |   2 +-
  9 files changed, 21 insertions(+), 20 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -153,6 +153,7 @@
   [?ric]
 - The signature of tests.support.LoggingCatcher.get_logs changed, see
   docstring [?ric]
+- Rename get_reinitialized_command back to reinitialize_command [?ric]
 
 
 1.0a3 - 2010-10-08
diff --git a/distutils2/command/bdist.py b/distutils2/command/bdist.py
--- a/distutils2/command/bdist.py
+++ b/distutils2/command/bdist.py
@@ -126,7 +126,7 @@
         # Reinitialize and run each command.
         for i in range(len(self.formats)):
             cmd_name = commands[i]
-            sub_cmd = self.get_reinitialized_command(cmd_name)
+            sub_cmd = self.reinitialize_command(cmd_name)
             sub_cmd.format = self.formats[i]
 
             # passing the owner and group names for tar archiving
diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -80,8 +80,8 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -183,13 +183,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.prefix = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -117,14 +117,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install', reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
         install.plat_name = self.plat_name
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -318,8 +318,8 @@
         cmd_obj.ensure_finalized()
         return cmd_obj
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
-        return self.distribution.get_reinitialized_command(
+    def reinitialize_command(self, command, reinit_subcommands=False):
+        return self.distribution.reinitialize_command(
             command, reinit_subcommands)
 
     def run_command(self, command):
diff --git a/distutils2/command/test.py b/distutils2/command/test.py
--- a/distutils2/command/test.py
+++ b/distutils2/command/test.py
@@ -56,7 +56,7 @@
         prev_syspath = sys.path[:]
         try:
             # build release
-            build = self.get_reinitialized_command('build')
+            build = self.reinitialize_command('build')
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -5,9 +5,9 @@
 
 from distutils2 import logger
 from distutils2.util import strtobool, resolve_name
+from distutils2.config import Config
 from distutils2.errors import (PackagingOptionError, PackagingArgError,
                                PackagingModuleError, PackagingClassError)
-from distutils2.config import Config
 from distutils2.command import get_command_class, STANDARD_COMMANDS
 from distutils2.command.cmd import Command
 from distutils2.metadata import Metadata
@@ -635,9 +635,9 @@
             except ValueError, msg:
                 raise PackagingOptionError(msg)
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
+    def reinitialize_command(self, command, reinit_subcommands=False):
         """Reinitializes a command to the state it was in when first
-        returned by 'get_command_obj()': ie., initialized but not yet
+        returned by 'get_command_obj()': i.e., initialized but not yet
         finalized.  This provides the opportunity to sneak option
         values in programmatically, overriding or supplementing
         user-supplied values from the config files and command line.
@@ -649,10 +649,11 @@
         'reinit_subcommands' is true, also reinitializes the command's
         sub-commands, as declared by the 'sub_commands' class attribute (if
         it has one).  See the "install_dist" command for an example.  Only
-        reinitializes the sub-commands that actually matter, ie. those
-        whose test predicates return true.
+        reinitializes the sub-commands that actually matter, i.e. those
+        whose test predicate return true.
 
-        Returns the reinitialized command object.
+        Returns the reinitialized command object.  It will be the same
+        object as the one stored in the self.command_obj attribute.
         """
         if not isinstance(command, Command):
             command_name = command
@@ -670,7 +671,7 @@
 
         if reinit_subcommands:
             for sub in command.get_sub_commands():
-                self.get_reinitialized_command(sub, reinit_subcommands)
+                self.reinitialize_command(sub, reinit_subcommands)
 
         return command
 
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -262,7 +262,7 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
-    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # XXX does not work with dist.reinitialize_command, which typechecks
     # and wants a finalized attribute
 
     def __init__(self, **kwargs):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:08 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Clean_up_mocking_of_stdo?=
	=?utf8?q?ut_and_stdin_in_tests=2E?=
Message-ID: <E1RPxSC-00073O-1o@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5971f625f8a7
changeset:   1244:5971f625f8a7
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 05:30:34 2011 +0100
summary:
  Clean up mocking of stdout and stdin in tests.

In addition, update the create module to use logging.

files:
  distutils2/create.py                       |  29 ++++----
  distutils2/tests/support.py                |  18 +++++-
  distutils2/tests/test_command_build_ext.py |  33 ++-------
  distutils2/tests/test_command_build_py.py  |  26 ++-----
  distutils2/tests/test_command_register.py  |  25 +------
  distutils2/tests/test_create.py            |  30 +++------
  distutils2/tests/test_run.py               |   2 -
  distutils2/tests/test_uninstall.py         |   9 +--
  distutils2/tests/test_util.py              |   8 --
  9 files changed, 64 insertions(+), 116 deletions(-)


diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -28,6 +28,7 @@
 from textwrap import dedent
 from ConfigParser import RawConfigParser
 
+from distutils2 import logger
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -139,7 +140,7 @@
         if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
-        print '\nERROR: You must select "Y" or "N".\n'
+        logger.error('You must select "Y" or "N".')
 
 
 # XXX use util.ask
@@ -162,10 +163,7 @@
     helptext = helptext.strip("\n")
 
     while True:
-        sys.stdout.write(prompt)
-        sys.stdout.flush()
-
-        line = sys.stdin.readline().strip()
+        line = raw_input(prompt).strip()
         if line == '?':
             print '=' * 70
             print helptext
@@ -286,9 +284,10 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print ('ERROR: %(name)s.old backup exists, please check that '
-                       'current %(name)s is correct and remove %(name)s.old' %
-                       {'name': _FILENAME})
+                message = ("ERROR: %(name)s.old backup exists, please check "
+                           "that current %(name)s is correct and remove "
+                           "%(name)s.old" % {'name': _FILENAME})
+                logger.error(message)
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -339,7 +338,7 @@
             fp.close()
 
         os.chmod(_FILENAME, 0644)
-        print 'Wrote %r.' % _FILENAME
+        logger.info('Wrote "%s".' % _FILENAME)
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -637,8 +636,8 @@
                         break
 
             if len(found_list) == 0:
-                print ('ERROR: Could not find a matching license for "%s"' %
-                       license)
+                logger.error('Could not find a matching license for "%s"' %
+                             license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -659,8 +658,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print ('ERROR: Invalid selection, type a number from the list '
-                       'above.')
+                logger.error(
+                    "Invalid selection, type a number from the list above.")
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -683,8 +682,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print ('ERROR: Invalid selection, type a single digit '
-                           'number.')
+                    logger.error(
+                        "Invalid selection, type a single digit number.")
 
 
 def main():
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -60,7 +60,7 @@
     # TestCase mixins
     'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
     # mocks
-    'DummyCommand', 'TestDistribution',
+    'DummyCommand', 'TestDistribution', 'Inputs',
     # misc. functions and decorators
     'fake_dec', 'create_distribution', 'use_command',
     'copy_xxmodule_c', 'fixup_build_ext',
@@ -285,6 +285,22 @@
         return self._config_files
 
 
+class Inputs(object):
+    """Fakes user inputs."""
+    # TODO document usage
+    # TODO use context manager or something for auto cleanup
+
+    def __init__(self, *answers):
+        self.answers = answers
+        self.index = 0
+
+    def __call__(self, prompt=''):
+        try:
+            return self.answers[self.index]
+        finally:
+            self.index += 1
+
+
 def create_distribution(configfiles=()):
     """Prepares a distribution with given config files parsed."""
     d = TestDistribution()
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -1,9 +1,7 @@
 import os
 import sys
 import site
-import shutil
 import textwrap
-from StringIO import StringIO
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
                                PackagingPlatformError)
@@ -11,7 +9,7 @@
 from distutils2.compiler.extension import Extension
 from distutils2._backport import sysconfig
 
-from distutils2.tests import support, unittest, verbose
+from distutils2.tests import support, unittest
 from distutils2.tests.support import assert_python_ok
 
 
@@ -43,18 +41,10 @@
         support.fixup_build_ext(cmd)
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
+        cmd.ensure_finalized()
+        cmd.run()
 
-        old_stdout = sys.stdout
-        if not verbose:
-            # silence compiler output
-            sys.stdout = StringIO()
-        try:
-            cmd.ensure_finalized()
-            cmd.run()
-        finally:
-            sys.stdout = old_stdout
-
-        code = """if 1:
+        code = textwrap.dedent("""\
             import sys
             sys.path.insert(0, %r)
 
@@ -69,7 +59,8 @@
             doc = 'This is a template module just for instruction.'
             assert xx.__doc__ == doc
             assert isinstance(xx.Null(), xx.Null)
-            assert isinstance(xx.Str(), xx.Str)"""
+            assert isinstance(xx.Str(), xx.Str)
+            """)
         code = code % self.tmp_dir
         assert_python_ok('-c', code)
 
@@ -398,16 +389,8 @@
         cmd.build_temp = self.tmp_dir
 
         try:
-            old_stdout = sys.stdout
-            if not verbose:
-                # silence compiler output
-                sys.stdout = StringIO()
-            try:
-                cmd.ensure_finalized()
-                cmd.run()
-            finally:
-                sys.stdout = old_stdout
-
+            cmd.ensure_finalized()
+            cmd.run()
         except CompileError:
             self.fail("Wrong deployment target during compilation")
 
diff --git a/distutils2/tests/test_command_build_py.py b/distutils2/tests/test_command_build_py.py
--- a/distutils2/tests/test_command_build_py.py
+++ b/distutils2/tests/test_command_build_py.py
@@ -63,8 +63,6 @@
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
         # create the distribution files.
         sources = self.mkdtemp()
         pkg = os.path.join(sources, 'pkg')
@@ -75,24 +73,16 @@
         open(os.path.join(testdir, "testfile"), "wb").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        #sys.stdout = StringIO.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": sources,
+                             "package_data": {"pkg": ["doc/*"]}})
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": sources,
-                                 "package_data": {"pkg": ["doc/*"]}})
-            dist.script_args = ["build"]
-            dist.parse_command_line()
-
-            try:
-                dist.run_commands()
-            except PackagingFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+            dist.run_commands()
+        except PackagingFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
     def test_byte_compile(self):
         project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -10,6 +10,7 @@
     DOCUTILS_SUPPORT = False
 
 from distutils2.tests import unittest, support
+from distutils2.tests.support import Inputs
 from distutils2.command import register as register_module
 from distutils2.command.register import register
 from distutils2.errors import PackagingSetupError
@@ -36,19 +37,6 @@
 """
 
 
-class Inputs(object):
-    """Fakes user inputs."""
-    def __init__(self, *answers):
-        self.answers = answers
-        self.index = 0
-
-    def __call__(self, prompt=''):
-        try:
-            return self.answers[self.index]
-        finally:
-            self.index += 1
-
-
 class FakeOpener(object):
     """Fakes a PyPI server"""
     def __init__(self):
@@ -202,10 +190,8 @@
 
     @unittest.skipUnless(DOCUTILS_SUPPORT, 'needs docutils')
     def test_strict(self):
-        # testing the script option
-        # when on, the register command stops if
-        # the metadata is incomplete or if
-        # long_description is not reSt compliant
+        # testing the strict option: when on, the register command stops if the
+        # metadata is incomplete or if description contains bad reST
 
         # empty metadata
         cmd = self._get_cmd({'name': 'xxx', 'version': 'xxx'})
@@ -215,16 +201,15 @@
         register_module.raw_input = inputs
         self.assertRaises(PackagingSetupError, cmd.run)
 
-        # metadata is OK but long_description is broken
+        # metadata is OK but description is broken
         metadata = {'home_page': 'xxx', 'author': 'xxx',
                     'author_email': '?x?x?',
-                    'name': 'xxx', 'version': 'xxx',
+                    'name': 'xxx', 'version': '4.2',
                     'description': 'title\n==\n\ntext'}
 
         cmd = self._get_cmd(metadata)
         cmd.ensure_finalized()
         cmd.strict = True
-
         self.assertRaises(PackagingSetupError, cmd.run)
 
         # now something that works
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -5,14 +5,17 @@
 import codecs
 from StringIO import StringIO
 from textwrap import dedent
+from distutils2 import create
 from distutils2.create import MainProgram, ask_yn, ask, main
 from distutils2._backport import sysconfig
 
 from distutils2.tests import support, unittest
+from distutils2.tests.support import Inputs
 
 
 class CreateTestCase(support.TempdirManager,
                      support.EnvironRestorer,
+                     support.LoggingCatcher,
                      unittest.TestCase):
 
     maxDiff = None
@@ -20,11 +23,6 @@
 
     def setUp(self):
         super(CreateTestCase, self).setUp()
-        self._stdin = sys.stdin  # TODO use Inputs
-        self._stdout = sys.stdout
-        sys.stdin = StringIO()
-        sys.stdout = StringIO()
-        self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
         # patch sysconfig
@@ -34,29 +32,24 @@
             'doc': sys.prefix + '/share/doc/pyxfoil', }
 
     def tearDown(self):
-        sys.stdin = self._stdin
-        sys.stdout = self._stdout
-        os.chdir(self._cwd)
         sysconfig.get_paths = self._old_get_paths
+        if hasattr(create, 'raw_input'):
+            del create.raw_input
         super(CreateTestCase, self).tearDown()
 
     def test_ask_yn(self):
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.raw_input = Inputs('y')
         self.assertEqual('y', ask_yn('is this a test'))
 
     def test_ask(self):
-        sys.stdin.write('a\n')
-        sys.stdin.write('b\n')
-        sys.stdin.seek(0)
+        create.raw_input = Inputs('a', 'b')
         self.assertEqual('a', ask('is this a test'))
         self.assertEqual('b', ask(str(list(range(0, 70))), default='c',
                                   lengthy=True))
 
     def test_set_multi(self):
         mainprogram = MainProgram()
-        sys.stdin.write('aaaaa\n')
-        sys.stdin.seek(0)
+        create.raw_input = Inputs('aaaaa')
         mainprogram.data['author'] = []
         mainprogram._set_multi('_set_multi test', 'author')
         self.assertEqual(['aaaaa'], mainprogram.data['author'])
@@ -132,8 +125,7 @@
               scripts=['my_script', 'bin/run'],
               )
         """), encoding='utf-8')
-        sys.stdin.write(u'y\n')
-        sys.stdin.seek(0)
+        create.raw_input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
@@ -214,9 +206,7 @@
 barbar is now in the public domain,
 ho, baby!
                         '''))
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
-        # FIXME Out of memory error.
+        create.raw_input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py
--- a/distutils2/tests/test_run.py
+++ b/distutils2/tests/test_run.py
@@ -33,11 +33,9 @@
 
     def setUp(self):
         super(RunTestCase, self).setUp()
-        self.old_stdout = sys.stdout
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
-        sys.stdout = self.old_stdout
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
diff --git a/distutils2/tests/test_uninstall.py b/distutils2/tests/test_uninstall.py
--- a/distutils2/tests/test_uninstall.py
+++ b/distutils2/tests/test_uninstall.py
@@ -31,14 +31,9 @@
     def setUp(self):
         super(UninstallTestCase, self).setUp()
         self.addCleanup(enable_cache)
-        self.root_dir = self.mkdtemp()
-        self.cwd = os.getcwd()
+        self.addCleanup(distutils2.util._path_created.clear)
         disable_cache()
 
-    def tearDown(self):
-        distutils2.util._path_created.clear()
-        super(UninstallTestCase, self).tearDown()
-
     def get_path(self, dist, name):
         # the dist argument must contain an install_dist command correctly
         # initialized with a prefix option and finalized befored this method
@@ -79,7 +74,7 @@
         dist.parse_config_files()
         dist.finalize_options()
         dist.run_command('install_dist',
-                         {'prefix': ('command line', self.root_dir)})
+                         {'prefix': ('command line', self.mkdtemp())})
 
         site_packages = self.get_path(dist, 'purelib')
         return dist, site_packages
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -606,14 +606,6 @@
 
 class GlobTestCase(GlobTestCaseBase):
 
-    def setUp(self):
-        super(GlobTestCase, self).setUp()
-        self.cwd = os.getcwd()
-
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(GlobTestCase, self).tearDown()
-
     def assertGlobMatch(self, glob, spec):
         tempdir = self.build_files_tree(spec)
         expected = self.clean_tree(spec)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:08 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_missing_entry?=
Message-ID: <E1RPxSC-00073R-2Y@dinsdale.python.org>

http://hg.python.org/distutils2/rev/bfae20822a7a
changeset:   1245:bfae20822a7a
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 05:49:50 2011 +0100
summary:
  Add missing entry

files:
  CHANGES.txt |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -148,6 +148,7 @@
 - #13170: Revert one of Jeremy's changes to config to fix a bug, kludge around
   shlex not supporting unicode in 2.x, fix wrong shutil import [david, ?ric]
 - #13205: Fix and improve generated setup scripts [david, ?ric]
+- #12386: Fix writing of the RESOURCES file [?ric]
 - #11751: Improve test coverage for manifest [justin]
 - Byte compilation is now isolated from the calling Python -B or -O options
   [?ric]

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:08 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Use_more_usual_name_for_?=
 =?utf8?q?one_option_of_install=5Fdistinfo?=
Message-ID: <E1RPxSC-00073V-3r@dinsdale.python.org>

http://hg.python.org/distutils2/rev/2657c291ec81
changeset:   1246:2657c291ec81
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 05:51:01 2011 +0100
summary:
  Use more usual name for one option of install_distinfo

files:
  CHANGES.txt                                       |   2 +
  distutils2/command/install_dist.py                |   4 +-
  distutils2/command/install_distinfo.py            |  25 ++++-----
  distutils2/tests/test_command_install_distinfo.py |  10 ++--
  4 files changed, 20 insertions(+), 21 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -155,6 +155,8 @@
 - The signature of tests.support.LoggingCatcher.get_logs changed, see
   docstring [?ric]
 - Rename get_reinitialized_command back to reinitialize_command [?ric]
+- Rename install_distinfo's option from distinfo-dir to the more usual
+  install_dir [?ric]
 
 
 1.0a3 - 2010-10-08
diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -58,9 +58,7 @@
         ('install-data=', None,
          "installation directory for data files"),
 
-        # Byte-compilation options -- see install_lib.py for details, as
-        # these are duplicated from there (but only install_lib does
-        # anything with them).
+        # Byte-compilation options -- see install_lib for details
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
         ('optimize=', 'O',
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -20,8 +20,8 @@
     description = 'create a .dist-info directory for the distribution'
 
     user_options = [
-        ('distinfo-dir=', None,
-         "directory where the the .dist-info directory will be installed"),
+        ('install-dir=', None,
+         "directory where the the .dist-info directory will be created"),
         ('installer=', None,
          "the name of the installer"),
         ('requested', None,
@@ -39,7 +39,7 @@
     negative_opt = {'no-requested': 'requested'}
 
     def initialize_options(self):
-        self.distinfo_dir = None
+        self.install_dir = None
         self.installer = None
         self.requested = None
         self.no_record = None
@@ -50,8 +50,7 @@
         self.set_undefined_options('install_dist',
                                    'installer', 'requested', 'no_record')
 
-        self.set_undefined_options('install_lib',
-                                   ('install_dir', 'distinfo_dir'))
+        self.set_undefined_options('install_lib', 'install_dir')
 
         if self.installer is None:
             # FIXME distutils or packaging or distutils2?
@@ -68,26 +67,26 @@
 
         basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
-        self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
+        self.install_dir = os.path.join(self.install_dir, basename)
 
     def run(self):
-        target = self.distinfo_dir
+        target = self.install_dir
 
         if os.path.isdir(target) and not os.path.islink(target):
             if not self.dry_run:
                 rmtree(target)
         elif os.path.exists(target):
-            self.execute(os.unlink, (self.distinfo_dir,),
+            self.execute(os.unlink, (self.install_dir,),
                          "removing " + target)
 
         self.execute(os.makedirs, (target,), "creating " + target)
 
-        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        metadata_path = os.path.join(self.install_dir, 'METADATA')
         self.execute(self.distribution.metadata.write, (metadata_path,),
                      "creating " + metadata_path)
         self.outfiles.append(metadata_path)
 
-        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        installer_path = os.path.join(self.install_dir, 'INSTALLER')
         logger.info('creating %s', installer_path)
         if not self.dry_run:
             f = open(installer_path, 'w')
@@ -98,7 +97,7 @@
         self.outfiles.append(installer_path)
 
         if self.requested:
-            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            requested_path = os.path.join(self.install_dir, 'REQUESTED')
             logger.info('creating %s', requested_path)
             if not self.dry_run:
                 open(requested_path, 'wb').close()
@@ -107,7 +106,7 @@
         if not self.no_resources:
             install_data = self.get_finalized_command('install_data')
             if install_data.get_resources_out() != []:
-                resources_path = os.path.join(self.distinfo_dir,
+                resources_path = os.path.join(self.install_dir,
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
@@ -124,7 +123,7 @@
                 self.outfiles.append(resources_path)
 
         if not self.no_record:
-            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            record_path = os.path.join(self.install_dir, 'RECORD')
             logger.info('creating %s', record_path)
             if not self.dry_run:
                 f = codecs.open(record_path, 'w', encoding='utf-8')
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -53,7 +53,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
@@ -88,7 +88,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.installer = 'bacon-python'
         cmd.ensure_finalized()
         cmd.run()
@@ -111,7 +111,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.requested = False
         cmd.ensure_finalized()
         cmd.run()
@@ -131,7 +131,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.no_record = True
         cmd.ensure_finalized()
         cmd.run()
@@ -235,7 +235,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:08 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Fix_module_name?=
Message-ID: <E1RPxSC-00073a-4e@dinsdale.python.org>

http://hg.python.org/distutils2/rev/259745e16c33
changeset:   1247:259745e16c33
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 06:27:14 2011 +0100
summary:
  Fix module name

files:
  distutils2/util.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -638,7 +638,7 @@
     This functions supports packages and attributes without depth limitation:
     ``package.package.module.class.class.function.attr`` is valid input.
     However, looking up builtins is not directly supported: use
-    ``builtins.name``.
+    ``__builtin__.name``.
 
     Raises ImportError if importing the module fails or if one requested
     attribute is not found.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:08 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
	=?utf8?q?n3=29=3A_Ye_olde_merge=2E?=
Message-ID: <E1RPxSC-00073e-8S@dinsdale.python.org>

http://hg.python.org/distutils2/rev/1c4fe27018fd
changeset:   1248:1c4fe27018fd
branch:      python3
parent:      1230:2d469ccfe30e
parent:      1246:2657c291ec81
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 07:33:45 2011 +0100
summary:
  Ye olde merge.

I broke test_mixin2to3 somehow; distutils2-default is okay and packaging too,
so I don?t see an obvious reason right now, I?ll investigate later.

files:
  CHANGES.txt                                       |    9 +
  CONTRIBUTORS.txt                                  |    1 +
  distutils2/command/bdist.py                       |    2 +-
  distutils2/command/bdist_dumb.py                  |    4 +-
  distutils2/command/bdist_msi.py                   |    8 +-
  distutils2/command/bdist_wininst.py               |    6 +-
  distutils2/command/build.py                       |    2 +-
  distutils2/command/build_py.py                    |   77 +--
  distutils2/command/cmd.py                         |   24 +-
  distutils2/command/install_dist.py                |    4 +-
  distutils2/command/install_distinfo.py            |   25 +-
  distutils2/command/install_lib.py                 |   72 +--
  distutils2/command/register.py                    |    2 +-
  distutils2/command/test.py                        |    2 +-
  distutils2/compat.py                              |   17 +
  distutils2/compiler/bcppcompiler.py               |    2 +-
  distutils2/compiler/cygwinccompiler.py            |   25 +-
  distutils2/compiler/msvc9compiler.py              |    2 +-
  distutils2/compiler/msvccompiler.py               |    2 +-
  distutils2/create.py                              |   34 +-
  distutils2/depgraph.py                            |    5 +-
  distutils2/dist.py                                |   25 +-
  distutils2/errors.py                              |    4 -
  distutils2/install.py                             |    9 -
  distutils2/manifest.py                            |    4 +-
  distutils2/metadata.py                            |   12 +-
  distutils2/run.py                                 |    5 +-
  distutils2/tests/__main__.py                      |    1 -
  distutils2/tests/pypi_server.py                   |   12 +-
  distutils2/tests/support.py                       |   83 ++-
  distutils2/tests/test_command_bdist_dumb.py       |   23 +-
  distutils2/tests/test_command_build_ext.py        |   33 +-
  distutils2/tests/test_command_build_py.py         |   94 ++-
  distutils2/tests/test_command_check.py            |   45 +-
  distutils2/tests/test_command_clean.py            |    9 +-
  distutils2/tests/test_command_cmd.py              |    3 +-
  distutils2/tests/test_command_install_data.py     |   12 +-
  distutils2/tests/test_command_install_dist.py     |   25 +-
  distutils2/tests/test_command_install_distinfo.py |   11 +-
  distutils2/tests/test_command_install_lib.py      |   72 +-
  distutils2/tests/test_command_register.py         |   28 +-
  distutils2/tests/test_command_sdist.py            |   72 +--
  distutils2/tests/test_command_test.py             |    6 +-
  distutils2/tests/test_command_upload.py           |    2 +-
  distutils2/tests/test_command_upload_docs.py      |   15 +-
  distutils2/tests/test_config.py                   |   32 +-
  distutils2/tests/test_create.py                   |   34 +-
  distutils2/tests/test_dist.py                     |   31 +-
  distutils2/tests/test_manifest.py                 |  204 +++++++++-
  distutils2/tests/test_metadata.py                 |    5 +-
  distutils2/tests/test_mixin2to3.py                |    2 +-
  distutils2/tests/test_pypi_simple.py              |    2 +-
  distutils2/tests/test_run.py                      |    4 +-
  distutils2/tests/test_uninstall.py                |   17 +-
  distutils2/tests/test_util.py                     |  159 +++---
  distutils2/tests/test_version.py                  |    4 +-
  distutils2/util.py                                |   64 +-
  57 files changed, 818 insertions(+), 670 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -148,6 +148,15 @@
 - #13170: Revert one of Jeremy's changes to config to fix a bug, kludge around
   shlex not supporting unicode in 2.x, fix wrong shutil import [david, ?ric]
 - #13205: Fix and improve generated setup scripts [david, ?ric]
+- #12386: Fix writing of the RESOURCES file [?ric]
+- #11751: Improve test coverage for manifest [justin]
+- Byte compilation is now isolated from the calling Python -B or -O options
+  [?ric]
+- The signature of tests.support.LoggingCatcher.get_logs changed, see
+  docstring [?ric]
+- Rename get_reinitialized_command back to reinitialize_command [?ric]
+- Rename install_distinfo's option from distinfo-dir to the more usual
+  install_dir [?ric]
 
 
 1.0a3 - 2010-10-08
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -46,6 +46,7 @@
 - Alain Leufroy
 - Martin von L?wis
 - Hugo Lopes Tavares
+- Justin Love
 - Simon Mathieu
 - Carl Meyer
 - Alexis M?taireau
diff --git a/distutils2/command/bdist.py b/distutils2/command/bdist.py
--- a/distutils2/command/bdist.py
+++ b/distutils2/command/bdist.py
@@ -126,7 +126,7 @@
         # Reinitialize and run each command.
         for i in range(len(self.formats)):
             cmd_name = commands[i]
-            sub_cmd = self.get_reinitialized_command(cmd_name)
+            sub_cmd = self.reinitialize_command(cmd_name)
             sub_cmd.format = self.formats[i]
 
             # passing the owner and group names for tar archiving
diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -80,8 +80,8 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -35,7 +35,7 @@
     def __init__(self, *args, **kw):
         """Dialog(database, name, x, y, w, h, attributes, title, first,
         default, cancel, bitmap=true)"""
-        Dialog.__init__(self, *args)
+        super(PyDialog, self).__init__(*args)
         ruler = self.h - 36
         #if kw.get("bitmap", True):
         #    self.bitmap("Bitmap", 0, 0, bmwidth, ruler, "PythonWin")
@@ -183,13 +183,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install_dist',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install_dist',
+                                            reinit_subcommands=True)
         install.prefix = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -2,7 +2,6 @@
 
 import sys
 import os
-import codecs
 
 from shutil import rmtree
 
@@ -117,14 +116,13 @@
         if not self.skip_build:
             self.run_command('build')
 
-        install = self.get_reinitialized_command('install',
-                                                 reinit_subcommands=True)
+        install = self.reinitialize_command('install', reinit_subcommands=True)
         install.root = self.bdist_dir
         install.skip_build = self.skip_build
         install.warn_dir = False
         install.plat_name = self.plat_name
 
-        install_lib = self.get_reinitialized_command('install_lib')
+        install_lib = self.reinitialize_command('install_lib')
         # we do not want to include pyc or pyo files
         install_lib.compile = False
         install_lib.optimize = 0
diff --git a/distutils2/command/build.py b/distutils2/command/build.py
--- a/distutils2/command/build.py
+++ b/distutils2/command/build.py
@@ -41,7 +41,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
diff --git a/distutils2/command/build_py.py b/distutils2/command/build_py.py
--- a/distutils2/command/build_py.py
+++ b/distutils2/command/build_py.py
@@ -1,22 +1,25 @@
 """Build pure Python modules (just copy to build directory)."""
 
 import os
-import sys
 from glob import glob
 
 from distutils2 import logger
+from distutils2.util import convert_path
+from distutils2.compat import Mixin2to3, cache_from_source
+from distutils2.errors import PackagingOptionError, PackagingFileError
 from distutils2.command.cmd import Command
-from distutils2.errors import PackagingOptionError, PackagingFileError
-from distutils2.util import convert_path
-from distutils2.compat import Mixin2to3
 
 # marking public APIs
 __all__ = ['build_py']
 
+
 class build_py(Command, Mixin2to3):
 
     description = "build pure Python modules (copy to build directory)"
 
+    # The options for controlling byte compilation are two independent sets;
+    # more info in install_lib or the reST docs
+
     user_options = [
         ('build-lib=', 'd', "directory to build (copy) to"),
         ('compile', 'c', "compile .py to .pyc"),
@@ -28,13 +31,14 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
 
     boolean_options = ['compile', 'force']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         self.build_lib = None
@@ -108,14 +112,15 @@
             self.run_2to3(self._updated_files, self._doctests_2to3,
                                             self.use_2to3_fixers)
 
-        self.byte_compile(self.get_outputs(include_bytecode=False))
+        self.byte_compile(self.get_outputs(include_bytecode=False),
+                          prefix=self.build_lib)
 
     # -- Top-level worker functions ------------------------------------
 
     def get_data_files(self):
         """Generate list of '(package,src_dir,build_dir,filenames)' tuples.
 
-        Helper function for `finalize_options()`.
+        Helper function for finalize_options.
         """
         data = []
         if not self.packages:
@@ -130,7 +135,7 @@
             # Length of path to strip from found files
             plen = 0
             if src_dir:
-                plen = len(src_dir)+1
+                plen = len(src_dir) + 1
 
             # Strip directory from globbed filenames
             filenames = [
@@ -142,7 +147,7 @@
     def find_data_files(self, package, src_dir):
         """Return filenames for package's data files in 'src_dir'.
 
-        Helper function for `get_data_files()`.
+        Helper function for get_data_files.
         """
         globs = (self.package_data.get('', [])
                  + self.package_data.get(package, []))
@@ -157,7 +162,7 @@
     def build_package_data(self):
         """Copy data files into build directory.
 
-        Helper function for `run()`.
+        Helper function for run.
         """
         # FIXME add tests for this method
         for package, src_dir, build_dir, filenames in self.data_files:
@@ -167,16 +172,17 @@
                 self.mkpath(os.path.dirname(target))
                 outf, copied = self.copy_file(srcfile,
                                target, preserve_mode=False)
-                if copied and srcfile in self.distribution.convert_2to3.doctests:
+                doctests = self.distribution.convert_2to3_doctests
+                if copied and srcfile in doctests:
                     self._doctests_2to3.append(outf)
 
     # XXX - this should be moved to the Distribution class as it is not
     # only needed for build_py. It also has no dependencies on this class.
     def get_package_dir(self, package):
         """Return the directory, relative to the top of the source
-           distribution, where package 'package' should be found
-           (at least according to the 'package_dir' option, if any)."""
-
+        distribution, where package 'package' should be found
+        (at least according to the 'package_dir' option, if any).
+        """
         path = package.split('.')
         if self.package_dir is not None:
             path.insert(0, self.package_dir)
@@ -187,8 +193,7 @@
         return ''
 
     def check_package(self, package, package_dir):
-        """Helper function for `find_package_modules()` and `find_modules()'.
-        """
+        """Helper function for find_package_modules and find_modules."""
         # Empty dir name means current directory, which we can probably
         # assume exists.  Also, os.path.exists and isdir don't know about
         # my "empty string means current dir" convention, so we have to
@@ -208,8 +213,8 @@
             if os.path.isfile(init_py):
                 return init_py
             else:
-                logger.warning(("package init file '%s' not found " +
-                                "(or not a regular file)"), init_py)
+                logger.warning("package init file %r not found "
+                               "(or not a regular file)", init_py)
 
         # Either not in a package at all (__init__.py not expected), or
         # __init__.py doesn't exist -- so don't return the filename.
@@ -217,7 +222,7 @@
 
     def check_module(self, module, module_file):
         if not os.path.isfile(module_file):
-            logger.warning("file %s (for module %s) not found",
+            logger.warning("file %r (for module %r) not found",
                            module_file, module)
             return False
         else:
@@ -238,7 +243,7 @@
                 module = os.path.splitext(os.path.basename(f))[0]
                 modules.append((package, module, f))
             else:
-                logger.debug("excluding %s", setup_script)
+                logger.debug("excluding %r", setup_script)
         return modules
 
     def find_modules(self):
@@ -330,9 +335,9 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(filename + "c")
-                if self.optimize > 0:
-                    outputs.append(filename + "o")
+                    outputs.append(cache_from_source(filename, True))
+                if self.optimize:
+                    outputs.append(cache_from_source(filename, False))
 
         outputs += [
             os.path.join(build_dir, filename)
@@ -359,7 +364,6 @@
     def build_modules(self):
         modules = self.find_modules()
         for package, module, module_file in modules:
-
             # Now "build" the module -- ie. copy the source file to
             # self.build_lib (the build directory for Python source).
             # (Actually, it gets copied to the directory for this package
@@ -368,7 +372,6 @@
 
     def build_packages(self):
         for package in self.packages:
-
             # Get list of (package, module, module_file) tuples based on
             # scanning the package directory.  'package' is only included
             # in the tuple so that 'find_modules()' and
@@ -386,25 +389,3 @@
             for package_, module, module_file in modules:
                 assert package == package_
                 self.build_module(module, module_file, package)
-
-    def byte_compile(self, files):
-        if sys.dont_write_bytecode:
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
-        from distutils2.util import byte_compile  # FIXME use compileall
-        prefix = self.build_lib
-        if prefix[-1] != os.sep:
-            prefix = prefix + os.sep
-
-        # XXX this code is essentially the same as the 'byte_compile()
-        # method of the "install_lib" command, except for the determination
-        # of the 'prefix' string.  Hmmm.
-
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -10,7 +10,7 @@
 
 class Command:
     """Abstract base class for defining command classes, the "worker bees"
-    of the Packaging.  A useful analogy for command classes is to think of
+    of Packaging.  A useful analogy for command classes is to think of
     them as subroutines with local variables called "options".  The options
     are "declared" in 'initialize_options()' and "defined" (given their
     final values, aka "finalized") in 'finalize_options()', both of which
@@ -318,8 +318,8 @@
         cmd_obj.ensure_finalized()
         return cmd_obj
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
-        return self.distribution.get_reinitialized_command(
+    def reinitialize_command(self, command, reinit_subcommands=False):
+        return self.distribution.reinitialize_command(
             command, reinit_subcommands)
 
     def run_command(self, command):
@@ -386,7 +386,6 @@
         if self.dry_run:
             return  # see if we want to display something
 
-
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
 
@@ -439,3 +438,20 @@
         # Otherwise, print the "skip" message
         else:
             logger.debug(skip_msg)
+
+    def byte_compile(self, files, prefix=None):
+        """Byte-compile files to pyc and/or pyo files.
+
+        This method requires that the calling class define compile and
+        optimize options, like build_py and install_lib.  It also
+        automatically respects the force and dry-run options.
+
+        prefix, if given, is a string that will be stripped off the
+        filenames encoded in bytecode files.
+        """
+        if self.compile:
+            util.byte_compile(files, optimize=False, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
+        if self.optimize:
+            util.byte_compile(files, optimize=self.optimize, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -55,9 +55,7 @@
         ('install-data=', None,
          "installation directory for data files"),
 
-        # Byte-compilation options -- see install_lib.py for details, as
-        # these are duplicated from there (but only install_lib does
-        # anything with them).
+        # Byte-compilation options -- see install_lib for details
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
         ('optimize=', 'O',
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -16,8 +16,8 @@
     description = 'create a .dist-info directory for the distribution'
 
     user_options = [
-        ('distinfo-dir=', None,
-         "directory where the the .dist-info directory will be installed"),
+        ('install-dir=', None,
+         "directory where the the .dist-info directory will be created"),
         ('installer=', None,
          "the name of the installer"),
         ('requested', None,
@@ -35,7 +35,7 @@
     negative_opt = {'no-requested': 'requested'}
 
     def initialize_options(self):
-        self.distinfo_dir = None
+        self.install_dir = None
         self.installer = None
         self.requested = None
         self.no_record = None
@@ -46,8 +46,7 @@
         self.set_undefined_options('install_dist',
                                    'installer', 'requested', 'no_record')
 
-        self.set_undefined_options('install_lib',
-                                   ('install_dir', 'distinfo_dir'))
+        self.set_undefined_options('install_lib', 'install_dir')
 
         if self.installer is None:
             # FIXME distutils or packaging or distutils2?
@@ -64,26 +63,26 @@
 
         basename = metadata.get_fullname(filesafe=True) + ".dist-info"
 
-        self.distinfo_dir = os.path.join(self.distinfo_dir, basename)
+        self.install_dir = os.path.join(self.install_dir, basename)
 
     def run(self):
-        target = self.distinfo_dir
+        target = self.install_dir
 
         if os.path.isdir(target) and not os.path.islink(target):
             if not self.dry_run:
                 rmtree(target)
         elif os.path.exists(target):
-            self.execute(os.unlink, (self.distinfo_dir,),
+            self.execute(os.unlink, (self.install_dir,),
                          "removing " + target)
 
         self.execute(os.makedirs, (target,), "creating " + target)
 
-        metadata_path = os.path.join(self.distinfo_dir, 'METADATA')
+        metadata_path = os.path.join(self.install_dir, 'METADATA')
         self.execute(self.distribution.metadata.write, (metadata_path,),
                      "creating " + metadata_path)
         self.outfiles.append(metadata_path)
 
-        installer_path = os.path.join(self.distinfo_dir, 'INSTALLER')
+        installer_path = os.path.join(self.install_dir, 'INSTALLER')
         logger.info('creating %s', installer_path)
         if not self.dry_run:
             with open(installer_path, 'w') as f:
@@ -91,7 +90,7 @@
         self.outfiles.append(installer_path)
 
         if self.requested:
-            requested_path = os.path.join(self.distinfo_dir, 'REQUESTED')
+            requested_path = os.path.join(self.install_dir, 'REQUESTED')
             logger.info('creating %s', requested_path)
             if not self.dry_run:
                 open(requested_path, 'wb').close()
@@ -100,7 +99,7 @@
         if not self.no_resources:
             install_data = self.get_finalized_command('install_data')
             if install_data.get_resources_out() != []:
-                resources_path = os.path.join(self.distinfo_dir,
+                resources_path = os.path.join(self.install_dir,
                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
@@ -114,7 +113,7 @@
                 self.outfiles.append(resources_path)
 
         if not self.no_record:
-            record_path = os.path.join(self.distinfo_dir, 'RECORD')
+            record_path = os.path.join(self.install_dir, 'RECORD')
             logger.info('creating %s', record_path)
             if not self.dry_run:
                 with open(record_path, 'w', encoding='utf-8') as f:
diff --git a/distutils2/command/install_lib.py b/distutils2/command/install_lib.py
--- a/distutils2/command/install_lib.py
+++ b/distutils2/command/install_lib.py
@@ -1,34 +1,26 @@
 """Install all modules (extensions and pure Python)."""
 
 import os
-import sys
-import logging
 
 from distutils2 import logger
+from distutils2.compat import cache_from_source
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingOptionError
 
 
 # Extension for Python source files.
+# XXX dead code?  most of the codebase checks for literal '.py'
 if hasattr(os, 'extsep'):
     PYTHON_SOURCE_EXTENSION = os.extsep + "py"
 else:
     PYTHON_SOURCE_EXTENSION = ".py"
 
+
 class install_lib(Command):
 
     description = "install all modules (extensions and pure Python)"
 
-    # The byte-compilation options are a tad confusing.  Here are the
-    # possible scenarios:
-    #   1) no compilation at all (--no-compile --no-optimize)
-    #   2) compile .pyc only (--compile --no-optimize; default)
-    #   3) compile .pyc and "level 1" .pyo (--compile --optimize)
-    #   4) compile "level 1" .pyo only (--no-compile --optimize)
-    #   5) compile .pyc and "level 2" .pyo (--compile --optimize-more)
-    #   6) compile "level 2" .pyo only (--no-compile --optimize-more)
-    #
-    # The UI for this is two option, 'compile' and 'optimize'.
+    # The options for controlling byte compilation are two independent sets:
     # 'compile' is strictly boolean, and only decides whether to
     # generate .pyc files.  'optimize' is three-way (0, 1, or 2), and
     # decides both whether to generate .pyo files and what level of
@@ -36,7 +28,7 @@
 
     user_options = [
         ('install-dir=', 'd', "directory to install to"),
-        ('build-dir=','b', "build directory (where to install from)"),
+        ('build-dir=', 'b', "build directory (where to install from)"),
         ('force', 'f', "force installation (overwrite existing files)"),
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
@@ -47,7 +39,8 @@
         ]
 
     boolean_options = ['force', 'compile', 'skip-build']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         # let the 'install_dist' command dictate our installation directory
@@ -65,7 +58,8 @@
         self.set_undefined_options('install_dist',
                                    ('build_lib', 'build_dir'),
                                    ('install_lib', 'install_dir'),
-                                   'force', 'compile', 'optimize', 'skip_build')
+                                   'force', 'compile', 'optimize',
+                                   'skip_build')
 
         if self.compile is None:
             self.compile = True
@@ -89,9 +83,14 @@
         # having a build directory!)
         outfiles = self.install()
 
-        # (Optionally) compile .py to .pyc
+        # (Optionally) compile .py to .pyc and/or .pyo
         if outfiles is not None and self.distribution.has_pure_modules():
-            self.byte_compile(outfiles)
+            # XXX comment from distutils: "This [prefix stripping] is far from
+            # complete, but it should at least generate usable bytecode in RPM
+            # distributions." -> need to find exact requirements for
+            # byte-compiled files and fix it
+            install_root = self.get_finalized_command('install_dist').root
+            self.byte_compile(outfiles, prefix=install_root)
 
     # -- Top-level worker functions ------------------------------------
     # (called from 'run()')
@@ -113,38 +112,6 @@
             return
         return outfiles
 
-    def byte_compile(self, files):
-        if sys.dont_write_bytecode:
-            # XXX do we want this?  because a Python runs without bytecode
-            # doesn't mean that the *dists should not contain bytecode
-            #--or does it?
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
-        from distutils2.util import byte_compile  # FIXME use compileall
-
-        # Get the "--root" directory supplied to the "install_dist" command,
-        # and use it as a prefix to strip off the purported filename
-        # encoded in bytecode files.  This is far from complete, but it
-        # should at least generate usable bytecode in RPM distributions.
-        install_root = self.get_finalized_command('install_dist').root
-
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=install_root,
-                         dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=install_root,
-                         verbose=verbose,
-                         dry_run=self.dry_run)
-
-
     # -- Utility methods -----------------------------------------------
 
     def _mutate_outputs(self, has_any, build_cmd, cmd_option, output_dir):
@@ -172,13 +139,12 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(py_file + "c")
-            if self.optimize > 0:
-                bytecode_files.append(py_file + "o")
+                bytecode_files.append(cache_from_source(py_file, True))
+            if self.optimize:
+                bytecode_files.append(cache_from_source(py_file, False))
 
         return bytecode_files
 
-
     # -- External interface --------------------------------------------
     # (called by outsiders)
 
diff --git a/distutils2/command/register.py b/distutils2/command/register.py
--- a/distutils2/command/register.py
+++ b/distutils2/command/register.py
@@ -178,7 +178,7 @@
                         'will be faster.\n(the login will be stored in %s)',
                         get_pypirc_path())
                     choice = 'X'
-                    while choice.lower() not in 'yn':
+                    while choice.lower() not in ('y', 'n'):
                         choice = input('Save your login (y/N)?')
                         if not choice:
                             choice = 'n'
diff --git a/distutils2/command/test.py b/distutils2/command/test.py
--- a/distutils2/command/test.py
+++ b/distutils2/command/test.py
@@ -56,7 +56,7 @@
         prev_syspath = sys.path[:]
         try:
             # build release
-            build = self.get_reinitialized_command('build')
+            build = self.reinitialize_command('build')
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -68,3 +68,20 @@
     else:
         raise TypeError("expect bytes or str, not %s" %
                         type(filename).__name__)
+
+
+try:
+    callable = callable
+except NameError:
+    from collections import Callable
+
+    def callable(obj):
+        return isinstance(obj, Callable)
+
+
+try:
+    from imp import cache_from_source
+except ImportError:
+    def cache_from_source(py_file, debug=__debug__):
+        ext = debug and 'c' or 'o'
+        return py_file + ext
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -48,7 +48,7 @@
 
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-        CCompiler.__init__(self, verbose, dry_run, force)
+        super(BCPPCompiler, self).__init__(verbose, dry_run, force)
 
         # These executables are assumed to all be in the path.
         # Borland doesn't seem to use any special registry settings to
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -93,8 +93,7 @@
     exe_extension = ".exe"
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-
-        UnixCCompiler.__init__(self, verbose, dry_run, force)
+        super(CygwinCCompiler, self).__init__(verbose, dry_run, force)
 
         status, details = check_config_h()
         logger.debug("Python's GCC status: %s (details: %s)", status, details)
@@ -234,12 +233,11 @@
         if not debug:
             extra_preargs.append("-s")
 
-        UnixCCompiler.link(self, target_desc, objects, output_filename,
-                           output_dir, libraries, library_dirs,
-                           runtime_library_dirs,
-                           None, # export_symbols, we do this in our def-file
-                           debug, extra_preargs, extra_postargs, build_temp,
-                           target_lang)
+        super(CygwinCCompiler, self).link(
+            target_desc, objects, output_filename, output_dir, libraries,
+            library_dirs, runtime_library_dirs,
+            None, # export_symbols, we do this in our def-file
+            debug, extra_preargs, extra_postargs, build_temp, target_lang)
 
     # -- Miscellaneous methods -----------------------------------------
 
@@ -255,14 +253,14 @@
             if ext not in (self.src_extensions + ['.rc','.res']):
                 raise UnknownFileError("unknown file type '%s' (from '%s')" % (ext, src_name))
             if strip_dir:
-                base = os.path.basename (base)
+                base = os.path.basename(base)
             if ext in ('.res', '.rc'):
                 # these need to be compiled to object files
-                obj_names.append (os.path.join(output_dir,
+                obj_names.append(os.path.join(output_dir,
                                               base + ext + self.obj_extension))
             else:
-                obj_names.append (os.path.join(output_dir,
-                                               base + self.obj_extension))
+                obj_names.append(os.path.join(output_dir,
+                                              base + self.obj_extension))
         return obj_names
 
 # the same as cygwin plus some additional parameters
@@ -273,8 +271,7 @@
     description = 'MinGW32 compiler'
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-
-        CygwinCCompiler.__init__ (self, verbose, dry_run, force)
+        super(Mingw32CCompiler, self).__init__(verbose, dry_run, force)
 
         # ld_version >= "2.13" support -shared so use it instead of
         # -mdll -static
diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py
--- a/distutils2/compiler/msvc9compiler.py
+++ b/distutils2/compiler/msvc9compiler.py
@@ -310,7 +310,7 @@
     exe_extension = '.exe'
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-        CCompiler.__init__(self, verbose, dry_run, force)
+        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
         self.__version = VERSION
         self.__root = r"Software\Microsoft\VisualStudio"
         # self.__macros = MACROS
diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py
--- a/distutils2/compiler/msvccompiler.py
+++ b/distutils2/compiler/msvccompiler.py
@@ -237,7 +237,7 @@
     exe_extension = '.exe'
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-        CCompiler.__init__(self, verbose, dry_run, force)
+        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
         self.__version = get_build_version()
         self.__arch = get_build_architecture()
         if self.__arch == "Intel":
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -29,6 +29,7 @@
 from tokenize import detect_encoding
 from configparser import RawConfigParser
 
+from distutils2 import logger
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
@@ -125,7 +126,7 @@
         if answer and answer[0].lower() in ('y', 'n'):
             return answer[0].lower()
 
-        print('\nERROR: You must select "Y" or "N".\n')
+        logger.error('You must select "Y" or "N".')
 
 
 # XXX use util.ask
@@ -148,10 +149,7 @@
     helptext = helptext.strip("\n")
 
     while True:
-        sys.stdout.write(prompt)
-        sys.stdout.flush()
-
-        line = sys.stdin.readline().strip()
+        line = input(prompt).strip()
         if line == '?':
             print('=' * 70)
             print(helptext)
@@ -272,9 +270,10 @@
     def _write_cfg(self):
         if os.path.exists(_FILENAME):
             if os.path.exists('%s.old' % _FILENAME):
-                print("ERROR: %(name)s.old backup exists, please check that "
-                      "current %(name)s is correct and remove %(name)s.old" %
-                      {'name': _FILENAME})
+                message = ("ERROR: %(name)s.old backup exists, please check "
+                           "that current %(name)s is correct and remove "
+                           "%(name)s.old" % {'name': _FILENAME})
+                logger.error(message)
                 return
             shutil.move(_FILENAME, '%s.old' % _FILENAME)
 
@@ -321,7 +320,7 @@
             fp.write('\n')
 
         os.chmod(_FILENAME, 0o644)
-        print('Wrote "%s".' % _FILENAME)
+        logger.info('Wrote "%s".' % _FILENAME)
 
     def convert_py_to_cfg(self):
         """Generate a setup.cfg from an existing setup.py.
@@ -351,7 +350,6 @@
                       ('long_description', 'description'),
                       ('url', 'home_page'),
                       ('platforms', 'platform'),
-                      # backport only for 2.5+
                       ('provides', 'provides-dist'),
                       ('obsoletes', 'obsoletes-dist'),
                       ('requires', 'requires-dist'))
@@ -615,8 +613,8 @@
                         break
 
             if len(found_list) == 0:
-                print('ERROR: Could not find a matching license for "%s"' %
-                      license)
+                logger.error('Could not find a matching license for "%s"' %
+                             license)
                 continue
 
             question = 'Matching licenses:\n\n'
@@ -637,8 +635,8 @@
             try:
                 index = found_list[int(choice) - 1]
             except ValueError:
-                print("ERROR: Invalid selection, type a number from the list "
-                      "above.")
+                logger.error(
+                    "Invalid selection, type a number from the list above.")
 
             classifiers.add(_CLASSIFIERS_LIST[index])
 
@@ -661,8 +659,8 @@
                     classifiers.add(key)
                     return
                 except (IndexError, ValueError):
-                    print("ERROR: Invalid selection, type a single digit "
-                          "number.")
+                    logger.error(
+                        "Invalid selection, type a single digit number.")
 
 
 def main():
@@ -676,7 +674,3 @@
     # program.write_setup_script()
     # distutils2.util.cfg_to_args()
     program()
-
-
-if __name__ == '__main__':
-    main()
diff --git a/distutils2/depgraph.py b/distutils2/depgraph.py
--- a/distutils2/depgraph.py
+++ b/distutils2/depgraph.py
@@ -224,6 +224,7 @@
 
 
 def main():
+    # XXX move to run._graph
     from distutils2.database import get_distributions
     tempout = StringIO()
     try:
@@ -267,7 +268,3 @@
     else:
         print('Supported option: -d [filename]')
         sys.exit(1)
-
-
-if __name__ == '__main__':
-    main()
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -5,9 +5,10 @@
 
 from distutils2 import logger
 from distutils2.util import strtobool, resolve_name
+from distutils2.compat import callable
+from distutils2.config import Config
 from distutils2.errors import (PackagingOptionError, PackagingArgError,
                                PackagingModuleError, PackagingClassError)
-from distutils2.config import Config
 from distutils2.command import get_command_class, STANDARD_COMMANDS
 from distutils2.command.cmd import Command
 from distutils2.metadata import Metadata
@@ -69,7 +70,7 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ]
     display_option_names = [x[0].replace('-', '_') for x in display_options]
 
@@ -409,13 +410,12 @@
             for help_option, short, desc, func in cmd_class.help_options:
                 if hasattr(opts, help_option.replace('-', '_')):
                     help_option_found = True
-                    if hasattr(func, '__call__'):
-                        func()
-                    else:
+                    if not callable(func):
                         raise PackagingClassError(
                             "invalid help function %r for help option %r: "
                             "must be a callable object (function, etc.)"
                             % (func, help_option))
+                    func()
 
             if help_option_found:
                 return
@@ -636,9 +636,9 @@
             except ValueError as msg:
                 raise PackagingOptionError(msg)
 
-    def get_reinitialized_command(self, command, reinit_subcommands=False):
+    def reinitialize_command(self, command, reinit_subcommands=False):
         """Reinitializes a command to the state it was in when first
-        returned by 'get_command_obj()': ie., initialized but not yet
+        returned by 'get_command_obj()': i.e., initialized but not yet
         finalized.  This provides the opportunity to sneak option
         values in programmatically, overriding or supplementing
         user-supplied values from the config files and command line.
@@ -650,10 +650,11 @@
         'reinit_subcommands' is true, also reinitializes the command's
         sub-commands, as declared by the 'sub_commands' class attribute (if
         it has one).  See the "install_dist" command for an example.  Only
-        reinitializes the sub-commands that actually matter, ie. those
-        whose test predicates return true.
+        reinitializes the sub-commands that actually matter, i.e. those
+        whose test predicate return true.
 
-        Returns the reinitialized command object.
+        Returns the reinitialized command object.  It will be the same
+        object as the one stored in the self.command_obj attribute.
         """
         if not isinstance(command, Command):
             command_name = command
@@ -671,7 +672,7 @@
 
         if reinit_subcommands:
             for sub in command.get_sub_commands():
-                self.get_reinitialized_command(sub, reinit_subcommands)
+                self.reinitialize_command(sub, reinit_subcommands)
 
         return command
 
@@ -733,7 +734,7 @@
             else:
                 hook_obj = hook
 
-            if not hasattr(hook_obj, '__call__'):
+            if not callable(hook_obj):
                 raise PackagingOptionError('hook %r is not callable' % hook)
 
             logger.info('running %s %s for command %s',
diff --git a/distutils2/errors.py b/distutils2/errors.py
--- a/distutils2/errors.py
+++ b/distutils2/errors.py
@@ -72,10 +72,6 @@
     """Syntax error in a file list template."""
 
 
-class PackagingByteCompileError(PackagingError):
-    """Byte compile error."""
-
-
 class PackagingPyPIError(PackagingError):
     """Any problem occuring during using the indexes."""
 
diff --git a/distutils2/install.py b/distutils2/install.py
--- a/distutils2/install.py
+++ b/distutils2/install.py
@@ -528,12 +528,3 @@
             logger.info('%r conflicts with %s', project, ','.join(projects))
 
     return True
-
-
-def _main(**attrs):
-    if 'script_args' not in attrs:
-        attrs['requirements'] = sys.argv[1]
-    get_infos(**attrs)
-
-if __name__ == '__main__':
-    _main()
diff --git a/distutils2/manifest.py b/distutils2/manifest.py
--- a/distutils2/manifest.py
+++ b/distutils2/manifest.py
@@ -147,7 +147,9 @@
 
     def _parse_template_line(self, line):
         words = line.split()
-        if len(words) == 1:
+        if len(words) == 1 and words[0] not in (
+              'include', 'exclude', 'global-include', 'global-exclude',
+              'recursive-include', 'recursive-exclude', 'graft', 'prune'):
             # no action given, let's use the default 'include'
             words.insert(0, 'include')
 
diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -28,8 +28,9 @@
         def __init__(self, source, report_level, halt_level, stream=None,
                      debug=0, encoding='ascii', error_handler='replace'):
             self.messages = []
-            Reporter.__init__(self, source, report_level, halt_level, stream,
-                              debug, encoding, error_handler)
+            super(SilentReporter, self).__init__(
+                source, report_level, halt_level, stream,
+                debug, encoding, error_handler)
 
         def system_message(self, level, message, *children, **kwargs):
             self.messages.append((level, message, children, kwargs))
@@ -184,6 +185,7 @@
 
 _FILESAFE = re.compile('[^A-Za-z0-9.]+')
 
+
 class Metadata:
     """The metadata of a release.
 
@@ -227,10 +229,8 @@
 
     def __delitem__(self, name):
         field_name = self._convert_name(name)
-        try:
-            del self._fields[field_name]
-        except KeyError:
-            raise KeyError(name)
+        # we let a KeyError propagate
+        del self._fields[field_name]
         self._set_best_version()
 
     def __contains__(self, name):
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -9,6 +9,7 @@
 from distutils2 import logger
 from distutils2.dist import Distribution
 from distutils2.util import _is_archive_file, generate_setup_py
+from distutils2.compat import callable
 from distutils2.command import get_command_class, STANDARD_COMMANDS
 from distutils2.install import install, install_local_project, remove
 from distutils2.database import get_distribution, get_distributions
@@ -368,7 +369,7 @@
     ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
-    ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),
+    ('generate-setup', 'Generate a backward-compatible setup.py', _generate),
 ]
 
 
@@ -500,7 +501,7 @@
             for help_option, short, desc, func in cmd_class.help_options:
                 if hasattr(opts, help_option.replace('-', '_')):
                     help_option_found = True
-                    if hasattr(func, '__call__'):
+                    if callable(func):
                         func()
                     else:
                         raise PackagingClassError(
diff --git a/distutils2/tests/__main__.py b/distutils2/tests/__main__.py
--- a/distutils2/tests/__main__.py
+++ b/distutils2/tests/__main__.py
@@ -3,7 +3,6 @@
 # Ripped from importlib tests, thanks Brett!
 
 import os
-import sys
 from test.support import reap_children, reap_threads, run_unittest
 
 from distutils2.tests import unittest
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -33,7 +33,6 @@
 import queue
 import select
 import threading
-import socketserver
 from functools import wraps
 from http.server import HTTPServer, SimpleHTTPRequestHandler
 from xmlrpc.server import SimpleXMLRPCServer
@@ -103,7 +102,7 @@
         """
         # we want to launch the server in a new dedicated thread, to not freeze
         # tests.
-        threading.Thread.__init__(self)
+        super(PyPIServer, self).__init__()
         self._run = True
         self._serve_xmlrpc = serve_xmlrpc
         if static_filesystem_paths is None:
@@ -270,7 +269,7 @@
 class PyPIXMLRPCServer(SimpleXMLRPCServer):
     def server_bind(self):
         """Override server_bind to store the server name."""
-        socketserver.TCPServer.server_bind(self)
+        super(PyPIXMLRPCServer, self).server_bind()
         host, port = self.socket.getsockname()[:2]
         self.server_port = port
 
@@ -371,12 +370,13 @@
             'requires_python': self.requires_python,
             'classifiers': [],
             'name': self.name,
-            'licence': self.licence,
+            'licence': self.licence,  # XXX licence or license?
             'summary': self.summary,
             'home_page': self.homepage,
             'stable_version': self.stable_version,
-            'provides_dist': self.provides_dist or "%s (%s)" % (self.name,
-                                                              self.version),
+            # FIXME doesn't that reproduce the bug from 6527d3106e9f?
+            'provides_dist': (self.provides_dist or
+                             "%s (%s)" % (self.name, self.version)),
             'requires': self.requires,
             'cheesecake_installability_id': self.cheesecake_installability_id,
         }
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -36,7 +36,6 @@
 import re
 import sys
 import errno
-import codecs
 import shutil
 import logging
 import logging.handlers
@@ -49,6 +48,9 @@
     zlib = None
 
 from distutils2.dist import Distribution
+from distutils2.util import resolve_name
+from distutils2.command import set_command, _COMMANDS
+
 from distutils2.tests import unittest
 from distutils2._backport import sysconfig
 
@@ -57,11 +59,12 @@
     # TestCase mixins
     'LoggingCatcher', 'TempdirManager', 'EnvironRestorer',
     # mocks
-    'DummyCommand', 'TestDistribution',
+    'DummyCommand', 'TestDistribution', 'Inputs',
     # misc. functions and decorators
-    'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
+    'fake_dec', 'create_distribution', 'use_command',
+    'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
-    'unittest', 'requires_zlib', 'skip_unless_symlink',
+    'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
 
 
@@ -73,7 +76,7 @@
     # stolen and adapted from test.support
 
     def __init__(self):
-        logging.handlers.BufferingHandler.__init__(self, 0)
+        super(_TestHandler, self).__init__(0)
         self.setLevel(logging.DEBUG)
 
     def shouldFlush(self):
@@ -90,10 +93,13 @@
     configured to record all messages logged to the 'distutils2' logger.
 
     Use get_logs to retrieve messages and self.loghandler.flush to discard
-    them.  get_logs automatically flushes the logs; if you test code that
-    generates logging messages but don't use get_logs, you have to flush
-    manually before doing other checks on logging message, otherwise you
-    will get irrelevant results.  See example in test_command_check.
+    them.  get_logs automatically flushes the logs, unless you pass
+    *flush=False*, for example to make multiple calls to the method with
+    different level arguments.  If your test calls some code that generates
+    logging message and then you don't call get_logs, you will need to flush
+    manually before testing other code in the same test_* method, otherwise
+    get_logs in the next lines will see messages from the previous lines.
+    See example in test_command_check.
     """
 
     def setUp(self):
@@ -117,25 +123,23 @@
         logger2to3.setLevel(self._old_levels[1])
         super(LoggingCatcher, self).tearDown()
 
-    def get_logs(self, *levels):
-        """Return all log messages with level in *levels*.
+    def get_logs(self, level=logging.WARNING, flush=True):
+        """Return all log messages with given level.
 
-        Without explicit levels given, returns all messages.  *levels* defaults
-        to all levels.  For log calls with arguments (i.e.
-        logger.info('bla bla %r', arg)), the messages will be formatted before
-        being returned (e.g. "bla bla 'thing'").
+        *level* defaults to logging.WARNING.
+
+        For log calls with arguments (i.e.  logger.info('bla bla %r', arg)),
+        the messages will be formatted before being returned (e.g. "bla bla
+        'thing'").
 
         Returns a list.  Automatically flushes the loghandler after being
-        called.
-
-        Example: self.get_logs(logging.WARN, logging.DEBUG).
+        called, unless *flush* is False (this is useful to get e.g. all
+        warnings then all info messages).
         """
-        if not levels:
-            messages = [log.getMessage() for log in self.loghandler.buffer]
-        else:
-            messages = [log.getMessage() for log in self.loghandler.buffer
-                        if log.levelno in levels]
-        self.loghandler.flush()
+        messages = [log.getMessage() for log in self.loghandler.buffer
+                    if log.levelno == level]
+        if flush:
+            self.loghandler.flush()
         return messages
 
 
@@ -254,7 +258,7 @@
     Useful for mocking one dependency command in the tests for another
     command, see e.g. the dummy build command in test_build_scripts.
     """
-    # XXX does not work with dist.get_reinitialized_command, which typechecks
+    # XXX does not work with dist.reinitialize_command, which typechecks
     # and wants a finalized attribute
 
     def __init__(self, **kwargs):
@@ -277,6 +281,22 @@
         return self._config_files
 
 
+class Inputs:
+    """Fakes user inputs."""
+    # TODO document usage
+    # TODO use context manager or something for auto cleanup
+
+    def __init__(self, *answers):
+        self.answers = answers
+        self.index = 0
+
+    def __call__(self, prompt=''):
+        try:
+            return self.answers[self.index]
+        finally:
+            self.index += 1
+
+
 def create_distribution(configfiles=()):
     """Prepares a distribution with given config files parsed."""
     d = TestDistribution()
@@ -287,6 +307,15 @@
     return d
 
 
+def use_command(testcase, fullname):
+    """Register command at *fullname* for the duration of a test."""
+    set_command(fullname)
+    # XXX maybe set_command should return the class object
+    name = resolve_name(fullname).get_command_name()
+    # XXX maybe we need a public API to remove commands
+    testcase.addCleanup(_COMMANDS.__delitem__, name)
+
+
 def fake_dec(*args, **kw):
     """Fake decorator"""
     def _wrap(func):
@@ -369,6 +398,10 @@
         'requires test.support.skip_unless_symlink')
 
 
+skip_2to3_optimize = unittest.skipIf(sys.flags.optimize,
+                                     "2to3 doesn't work under -O")
+
+
 requires_zlib = unittest.skipUnless(zlib, 'requires zlib')
 
 
diff --git a/distutils2/tests/test_command_bdist_dumb.py b/distutils2/tests/test_command_bdist_dumb.py
--- a/distutils2/tests/test_command_bdist_dumb.py
+++ b/distutils2/tests/test_command_bdist_dumb.py
@@ -1,6 +1,9 @@
 """Tests for distutils.command.bdist_dumb."""
 
 import os
+import imp
+import sys
+import zipfile
 import distutils2.util
 
 from distutils2.dist import Distribution
@@ -49,15 +52,27 @@
 
         # see what we have
         dist_created = os.listdir(os.path.join(pkg_dir, 'dist'))
-        base = "%s.%s" % (dist.get_fullname(), cmd.plat_name)
+        base = "%s.%s.zip" % (dist.get_fullname(), cmd.plat_name)
         if os.name == 'os2':
             base = base.replace(':', '-')
 
-        wanted = ['%s.zip' % base]
-        self.assertEqual(dist_created, wanted)
+        self.assertEqual(dist_created, [base])
 
         # now let's check what we have in the zip file
-        # XXX to be done
+        fp = zipfile.ZipFile(os.path.join('dist', base))
+        try:
+            contents = fp.namelist()
+        finally:
+            fp.close
+
+        if sys.version_info[1] == 1:
+            pyc = 'foo.pyc'
+        else:
+            pyc = 'foo.%s.pyc' % imp.get_tag()
+        contents = sorted(os.path.basename(fn) for fn in contents)
+        wanted = ['foo.py', pyc,
+                  'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(contents, sorted(wanted))
 
     def test_finalize_options(self):
         pkg_dir, dist = self.create_dist()
diff --git a/distutils2/tests/test_command_build_ext.py b/distutils2/tests/test_command_build_ext.py
--- a/distutils2/tests/test_command_build_ext.py
+++ b/distutils2/tests/test_command_build_ext.py
@@ -1,9 +1,7 @@
 import os
 import sys
 import site
-import shutil
 import textwrap
-from io import StringIO
 from distutils2.dist import Distribution
 from distutils2.errors import (UnknownFileError, CompileError,
                                PackagingPlatformError)
@@ -11,7 +9,7 @@
 from distutils2.compiler.extension import Extension
 from distutils2._backport import sysconfig
 
-from distutils2.tests import support, unittest, verbose
+from distutils2.tests import support, unittest
 from distutils2.tests.support import assert_python_ok
 
 
@@ -38,18 +36,10 @@
         support.fixup_build_ext(cmd)
         cmd.build_lib = self.tmp_dir
         cmd.build_temp = self.tmp_dir
+        cmd.ensure_finalized()
+        cmd.run()
 
-        old_stdout = sys.stdout
-        if not verbose:
-            # silence compiler output
-            sys.stdout = StringIO()
-        try:
-            cmd.ensure_finalized()
-            cmd.run()
-        finally:
-            sys.stdout = old_stdout
-
-        code = """if 1:
+        code = textwrap.dedent("""\
             import sys
             sys.path.insert(0, %r)
 
@@ -64,7 +54,8 @@
             doc = 'This is a template module just for instruction.'
             assert xx.__doc__ == doc
             assert isinstance(xx.Null(), xx.Null)
-            assert isinstance(xx.Str(), xx.Str)"""
+            assert isinstance(xx.Str(), xx.Str)
+            """)
         code = code % self.tmp_dir
         assert_python_ok('-c', code)
 
@@ -389,16 +380,8 @@
         cmd.build_temp = self.tmp_dir
 
         try:
-            old_stdout = sys.stdout
-            if not verbose:
-                # silence compiler output
-                sys.stdout = StringIO()
-            try:
-                cmd.ensure_finalized()
-                cmd.run()
-            finally:
-                sys.stdout = old_stdout
-
+            cmd.ensure_finalized()
+            cmd.run()
         except CompileError:
             self.fail("Wrong deployment target during compilation")
 
diff --git a/distutils2/tests/test_command_build_py.py b/distutils2/tests/test_command_build_py.py
--- a/distutils2/tests/test_command_build_py.py
+++ b/distutils2/tests/test_command_build_py.py
@@ -55,30 +55,20 @@
         # This makes sure the list of outputs includes byte-compiled
         # files for Python modules but not for package data files
         # (there shouldn't *be* byte-code files for those!).
-        #
         self.assertEqual(len(cmd.get_outputs()), 3)
         pkgdest = os.path.join(destination, "pkg")
         files = os.listdir(pkgdest)
-        pycache_dir = os.path.join(pkgdest, "__pycache__")
         self.assertIn("__init__.py", files)
         self.assertIn("README.txt", files)
-        if sys.dont_write_bytecode:
-            if sys.version_info[1] == 1:
-                self.assertNotIn("__init__.pyc", files)
-            else:
-                self.assertFalse(os.path.exists(pycache_dir))
+        if sys.version_info[1] == 1:
+            self.assertIn("__init__.pyc", files)
         else:
-            # XXX even with -O, distutils2 writes pyc, not pyo; bug?
-            if sys.version_info[1] == 1:
-                self.assertIn("__init__.pyc", files)
-            else:
-                pyc_files = os.listdir(pycache_dir)
-                self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
+            pycache_dir = os.path.join(pkgdest, "__pycache__")
+            pyc_files = os.listdir(pycache_dir)
+            self.assertIn("__init__.%s.pyc" % imp.get_tag(), pyc_files)
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
-        cwd = os.getcwd()
-
         # create the distribution files.
         sources = self.mkdtemp()
         pkg = os.path.join(sources, 'pkg')
@@ -89,40 +79,66 @@
         open(os.path.join(testdir, "testfile"), "wb").close()
 
         os.chdir(sources)
-        old_stdout = sys.stdout
-        #sys.stdout = StringIO.StringIO()
+        dist = Distribution({"packages": ["pkg"],
+                             "package_dir": sources,
+                             "package_data": {"pkg": ["doc/*"]}})
+        dist.script_args = ["build"]
+        dist.parse_command_line()
 
         try:
-            dist = Distribution({"packages": ["pkg"],
-                                 "package_dir": sources,
-                                 "package_data": {"pkg": ["doc/*"]}})
-            dist.script_args = ["build"]
-            dist.parse_command_line()
+            dist.run_commands()
+        except PackagingFileError:
+            self.fail("failed package_data test when package_dir is ''")
 
-            try:
-                dist.run_commands()
-            except PackagingFileError:
-                self.fail("failed package_data test when package_dir is ''")
-        finally:
-            # Restore state.
-            os.chdir(cwd)
-            sys.stdout = old_stdout
+    def test_byte_compile(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = True
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
 
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+        found = os.listdir(cmd.build_lib)
+        if sys.version_info[1] == 1:
+            self.assertEqual(sorted(found),
+                             ['boiledeggs.py', 'boiledeggs.pyc'])
+        else:
+            self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+            found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+            self.assertEqual(found, ['boiledeggs.%s.pyc' % imp.get_tag()])
+
+    def test_byte_compile_optimized(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
         cmd = build_py(dist)
         cmd.compile = True
         cmd.optimize = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
 
-        old_dont_write_bytecode = sys.dont_write_bytecode
+        found = os.listdir(cmd.build_lib)
+        if sys.version_info[1] == 1:
+            self.assertEqual(sorted(found), ['boiledeggs.py', 'boiledeggs.pyc',
+                                             'boiledeggs.pyo'])
+        else:
+            self.assertEqual(sorted(found), ['__pycache__', 'boiledeggs.py'])
+            found = os.listdir(os.path.join(cmd.build_lib, '__pycache__'))
+            self.assertEqual(sorted(found),
+                             ['boiledeggs.%s.pyc' % imp.get_tag(),
+                              'boiledeggs.%s.pyo' % imp.get_tag()])
+
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            cmd.byte_compile([])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
+        self.test_byte_compile()
+        self.test_byte_compile_optimized()
 
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
 
 def test_suite():
     return unittest.makeSuite(BuildPyTestCase)
diff --git a/distutils2/tests/test_command_check.py b/distutils2/tests/test_command_check.py
--- a/distutils2/tests/test_command_check.py
+++ b/distutils2/tests/test_command_check.py
@@ -1,6 +1,5 @@
 """Tests for distutils.command.check."""
 
-import logging
 from distutils2.command.check import check
 from distutils2.metadata import _HAS_DOCUTILS
 from distutils2.errors import PackagingSetupError, MetadataMissingError
@@ -27,11 +26,11 @@
         # let's run the command with no metadata at all
         # by default, check is checking the metadata
         # should have some warnings
-        cmd = self._run()
+        self._run()
         # trick: using assertNotEqual with an empty list will give us a more
         # useful error message than assertGreater(.., 0) when the code change
         # and the test fails
-        self.assertNotEqual([], self.get_logs(logging.WARNING))
+        self.assertNotEqual(self.get_logs(), [])
 
         # now let's add the required fields
         # and run it again, to make sure we don't get
@@ -40,8 +39,8 @@
                     'author_email': 'xxx',
                     'name': 'xxx', 'version': '4.2',
                     }
-        cmd = self._run(metadata)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata)
+        self.assertEqual(self.get_logs(), [])
 
         # now with the strict mode, we should
         # get an error if there are missing metadata
@@ -53,8 +52,8 @@
         self.loghandler.flush()
 
         # and of course, no error when all metadata fields are present
-        cmd = self._run(metadata, strict=True)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata, strict=True)
+        self.assertEqual(self.get_logs(), [])
 
         # now a test with non-ASCII characters
         metadata = {'home_page': 'xxx', 'author': '\u00c9ric',
@@ -62,15 +61,15 @@
                     'version': '1.2',
                     'summary': 'Something about esszet \u00df',
                     'description': 'More things about esszet \u00df'}
-        cmd = self._run(metadata)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata)
+        self.assertEqual(self.get_logs(), [])
 
     def test_check_metadata_1_2(self):
         # let's run the command with no metadata at all
         # by default, check is checking the metadata
         # should have some warnings
-        cmd = self._run()
-        self.assertNotEqual([], self.get_logs(logging.WARNING))
+        self._run()
+        self.assertNotEqual(self.get_logs(), [])
 
         # now let's add the required fields and run it again, to make sure we
         # don't get any warning anymore let's use requires_python as a marker
@@ -80,8 +79,8 @@
                     'name': 'xxx', 'version': '4.2',
                     'requires_python': '2.4',
                     }
-        cmd = self._run(metadata)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata)
+        self.assertEqual(self.get_logs(), [])
 
         # now with the strict mode, we should
         # get an error if there are missing metadata
@@ -99,8 +98,8 @@
 
         # now with correct version format again
         metadata['version'] = '4.2'
-        cmd = self._run(metadata, strict=True)
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self._run(metadata, strict=True)
+        self.assertEqual(self.get_logs(), [])
 
     @unittest.skipUnless(_HAS_DOCUTILS, "requires docutils")
     def test_check_restructuredtext(self):
@@ -109,9 +108,7 @@
         pkg_info, dist = self.create_dist(description=broken_rest)
         cmd = check(dist)
         cmd.check_restructuredtext()
-        self.assertEqual(len(self.get_logs(logging.WARNING)), 1)
-        # clear warnings from the previous call
-        self.loghandler.flush()
+        self.assertEqual(len(self.get_logs()), 1)
 
         # let's see if we have an error with strict=1
         metadata = {'home_page': 'xxx', 'author': 'xxx',
@@ -126,7 +123,7 @@
         dist = self.create_dist(description='title\n=====\n\ntest \u00df')[1]
         cmd = check(dist)
         cmd.check_restructuredtext()
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertEqual(self.get_logs(), [])
 
     def test_check_all(self):
         self.assertRaises(PackagingSetupError, self._run,
@@ -143,18 +140,18 @@
         }
         cmd = check(dist)
         cmd.check_hooks_resolvable()
-        self.assertEqual(len(self.get_logs(logging.WARNING)), 1)
+        self.assertEqual(len(self.get_logs()), 1)
 
     def test_warn(self):
         _, dist = self.create_dist()
         cmd = check(dist)
-        self.assertEqual([], self.get_logs())
+        self.assertEqual(self.get_logs(), [])
         cmd.warn('hello')
-        self.assertEqual(['check: hello'], self.get_logs())
+        self.assertEqual(self.get_logs(), ['check: hello'])
         cmd.warn('hello %s', 'world')
-        self.assertEqual(['check: hello world'], self.get_logs())
+        self.assertEqual(self.get_logs(), ['check: hello world'])
         cmd.warn('hello %s %s', 'beautiful', 'world')
-        self.assertEqual(['check: hello beautiful world'], self.get_logs())
+        self.assertEqual(self.get_logs(), ['check: hello beautiful world'])
 
 
 def test_suite():
diff --git a/distutils2/tests/test_command_clean.py b/distutils2/tests/test_command_clean.py
--- a/distutils2/tests/test_command_clean.py
+++ b/distutils2/tests/test_command_clean.py
@@ -5,7 +5,8 @@
 from distutils2.tests import unittest, support
 
 
-class cleanTestCase(support.TempdirManager, support.LoggingCatcher,
+class CleanTestCase(support.TempdirManager,
+                    support.LoggingCatcher,
                     unittest.TestCase):
 
     def test_simple_run(self):
@@ -23,7 +24,7 @@
             if name == 'build_base':
                 continue
             for f in ('one', 'two', 'three'):
-                self.write_file(os.path.join(path, f))
+                self.write_file((path, f))
 
         # let's run the command
         cmd.all = True
@@ -36,13 +37,11 @@
                              '%r was not removed' % path)
 
         # let's run the command again (should spit warnings but succeed)
-        cmd.all = True
-        cmd.ensure_finalized()
         cmd.run()
 
 
 def test_suite():
-    return unittest.makeSuite(cleanTestCase)
+    return unittest.makeSuite(CleanTestCase)
 
 if __name__ == "__main__":
     unittest.main(defaultTest="test_suite")
diff --git a/distutils2/tests/test_command_cmd.py b/distutils2/tests/test_command_cmd.py
--- a/distutils2/tests/test_command_cmd.py
+++ b/distutils2/tests/test_command_cmd.py
@@ -1,5 +1,6 @@
 """Tests for distutils.cmd."""
 import os
+import logging
 
 from distutils2.command.cmd import Command
 from distutils2.dist import Distribution
@@ -43,7 +44,7 @@
 
         wanted = ["command options for 'MyCmd':", '  option1 = 1',
                   '  option2 = 1']
-        msgs = self.get_logs()
+        msgs = self.get_logs(logging.INFO)
         self.assertEqual(msgs, wanted)
 
     def test_ensure_string(self):
diff --git a/distutils2/tests/test_command_install_data.py b/distutils2/tests/test_command_install_data.py
--- a/distutils2/tests/test_command_install_data.py
+++ b/distutils2/tests/test_command_install_data.py
@@ -62,6 +62,7 @@
 
         # let's try with warn_dir one
         cmd.warn_dir = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -80,6 +81,7 @@
 
         cmd.data_files = {one: '{inst}/one', two: '{inst2}/two',
                           three: '{inst3}/three'}
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -125,22 +127,16 @@
 
         # now the real test
         fn = os.path.join(install_dir, 'Spamlib-0.1.dist-info', 'RESOURCES')
-        fp = open(fn)
-        try:
+        with open(fn, encoding='utf-8') as fp:
             content = fp.read().strip()
-        finally:
-            fp.close()
 
         expected = 'spamd,%s' % os.path.join(scripts_dir, 'spamd')
         self.assertEqual(content, expected)
 
         # just to be sure, we also test that get_file works here, even though
         # packaging.database has its own test file
-        fp = distutils2.database.get_file('Spamlib', 'spamd')
-        try:
+        with distutils2.database.get_file('Spamlib', 'spamd') as fp:
             content = fp.read()
-        finally:
-            fp.close()
 
         self.assertEqual('# Python script', content)
 
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -1,6 +1,7 @@
 """Tests for distutils2.command.install."""
 
 import os
+import imp
 import sys
 
 from distutils2.command.build_ext import build_ext
@@ -92,21 +93,20 @@
         self.old_expand = os.path.expanduser
         os.path.expanduser = _expanduser
 
-        try:
-            # this is the actual test
-            self._test_user_site()
-        finally:
+        def cleanup():
             _CONFIG_VARS['userbase'] = self.old_user_base
             _SCHEMES.set(scheme, 'purelib', self.old_user_site)
             os.path.expanduser = self.old_expand
 
-    def _test_user_site(self):
+        self.addCleanup(cleanup)
+
         schemes = get_scheme_names()
         for key in ('nt_user', 'posix_user', 'os2_home'):
             self.assertIn(key, schemes)
 
         dist = Distribution({'name': 'xx'})
         cmd = install_dist(dist)
+
         # making sure the user option is there
         options = [name for name, short, lable in
                    cmd.user_options]
@@ -181,9 +181,11 @@
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
         install_dir = self.mkdtemp()
-        project_dir, dist = self.create_dist(scripts=['hello'])
+        project_dir, dist = self.create_dist(py_modules=['hello'],
+                                             scripts=['sayhi'])
         os.chdir(project_dir)
-        self.write_file('hello', "print('o hai')")
+        self.write_file('hello.py', "def main(): print('o hai')")
+        self.write_file('sayhi', 'from hello import main; main()')
 
         cmd = install_dist(dist)
         dist.command_obj['install_dist'] = cmd
@@ -195,9 +197,14 @@
         with open(cmd.record) as f:
             content = f.read()
 
+        if sys.version_info[1] == 1:
+            pyc = 'hello.pyc'
+        else:
+            pyc = 'hello.%s.pyc' % imp.get_tag()
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['hello', 'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
-        self.assertEqual(found, expected)
+        expected = ['hello.py', pyc, 'sayhi',
+                    'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(sorted(found), sorted(expected))
 
         # XXX test that fancy_getopt is okay with options named
         # record and no-record but unrelated
diff --git a/distutils2/tests/test_command_install_distinfo.py b/distutils2/tests/test_command_install_distinfo.py
--- a/distutils2/tests/test_command_install_distinfo.py
+++ b/distutils2/tests/test_command_install_distinfo.py
@@ -49,7 +49,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
@@ -60,6 +60,7 @@
                         ['METADATA', 'RECORD', 'REQUESTED', 'INSTALLER'])
         with open(os.path.join(dist_info, 'INSTALLER')) as fp:
             self.assertEqual(fp.read(), 'distutils')
+
         with open(os.path.join(dist_info, 'REQUESTED')) as fp:
             self.assertEqual(fp.read(), '')
         meta_path = os.path.join(dist_info, 'METADATA')
@@ -76,7 +77,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.installer = 'bacon-python'
         cmd.ensure_finalized()
         cmd.run()
@@ -96,7 +97,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.requested = False
         cmd.ensure_finalized()
         cmd.run()
@@ -116,7 +117,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.no_record = True
         cmd.ensure_finalized()
         cmd.run()
@@ -214,7 +215,7 @@
         cmd = install_distinfo(dist)
         dist.command_obj['install_distinfo'] = cmd
 
-        cmd.distinfo_dir = install_dir
+        cmd.install_dir = install_dir
         cmd.ensure_finalized()
         cmd.run()
 
diff --git a/distutils2/tests/test_command_install_lib.py b/distutils2/tests/test_command_install_lib.py
--- a/distutils2/tests/test_command_install_lib.py
+++ b/distutils2/tests/test_command_install_lib.py
@@ -17,7 +17,7 @@
     restore_environ = ['PYTHONPATH']
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -34,75 +34,77 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(sys.dont_write_bytecode, 'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
-        os.chdir(pkg_dir)
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = True
         cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
         if sys.version_info[1] == 1:
             pyc_file = 'foo.pyc'
             pyo_file = 'foo.pyo'
         else:
-            pyc_file = imp.cache_from_source('foo.py')
-            pyo_file = imp.cache_from_source('foo.py', debug_override=False)
+            pyc_file = imp.cache_from_source('foo.py', True)
+            pyo_file = imp.cache_from_source('foo.py', False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
+        sys.dont_write_bytecode = True
+        self.test_byte_compile()
+
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # make sure the build_lib is set the temp dir
-        build_dir = os.path.split(pkg_dir)[0]
+        # make sure the build_lib is set the temp dir  # XXX what?  this is not
+        # needed in the same distutils test and should work without manual
+        # intervention
+        build_dir = os.path.split(project_dir)[0]
         cmd.get_finalized_command('build_py').build_lib = build_dir
 
-        # get_output should return 4 elements
-        self.assertEqual(len(cmd.get_outputs()), 4)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo*.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
-
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
-        cmd = install_lib(dist)
-        cmd.compile = True
-        cmd.optimize = 1
-
-        self.addCleanup(setattr, sys, 'dont_write_bytecode',
-                        sys.dont_write_bytecode)
-        sys.dont_write_bytecode = True
-        cmd.byte_compile([])
-
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo*.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
 
 def test_suite():
diff --git a/distutils2/tests/test_command_register.py b/distutils2/tests/test_command_register.py
--- a/distutils2/tests/test_command_register.py
+++ b/distutils2/tests/test_command_register.py
@@ -12,6 +12,7 @@
     DOCUTILS_SUPPORT = False
 
 from distutils2.tests import unittest, support
+from distutils2.tests.support import Inputs
 from distutils2.command import register as register_module
 from distutils2.command.register import register
 from distutils2.errors import PackagingSetupError
@@ -38,19 +39,6 @@
 """
 
 
-class Inputs:
-    """Fakes user inputs."""
-    def __init__(self, *answers):
-        self.answers = answers
-        self.index = 0
-
-    def __call__(self, prompt=''):
-        try:
-            return self.answers[self.index]
-        finally:
-            self.index += 1
-
-
 class FakeOpener:
     """Fakes a PyPI server"""
     def __init__(self):
@@ -143,6 +131,7 @@
 
         register_module.input = _no_way
         cmd.show_response = True
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -200,12 +189,10 @@
 
     @unittest.skipUnless(DOCUTILS_SUPPORT, 'needs docutils')
     def test_strict(self):
-        # testing the script option
-        # when on, the register command stops if
-        # the metadata is incomplete or if
-        # long_description is not reSt compliant
+        # testing the strict option: when on, the register command stops if the
+        # metadata is incomplete or if description contains bad reST
 
-        # empty metadata
+        # empty metadata  # XXX this is not really empty..
         cmd = self._get_cmd({'name': 'xxx', 'version': 'xxx'})
         cmd.ensure_finalized()
         cmd.strict = True
@@ -213,16 +200,15 @@
         register_module.input = inputs
         self.assertRaises(PackagingSetupError, cmd.run)
 
-        # metadata is OK but long_description is broken
+        # metadata is OK but description is broken
         metadata = {'home_page': 'xxx', 'author': 'xxx',
                     'author_email': '?x?x?',
-                    'name': 'xxx', 'version': 'xxx',
+                    'name': 'xxx', 'version': '4.2',
                     'description': 'title\n==\n\ntext'}
 
         cmd = self._get_cmd(metadata)
         cmd.ensure_finalized()
         cmd.strict = True
-
         self.assertRaises(PackagingSetupError, cmd.run)
 
         # now something that works
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -1,9 +1,6 @@
 """Tests for distutils2.command.sdist."""
 import os
 import zipfile
-import logging
-
-from distutils2.tests.support import requires_zlib
 
 try:
     import grp
@@ -13,17 +10,17 @@
     UID_GID_SUPPORT = False
 
 from os.path import join
-from distutils2.tests import captured_stdout
-from distutils2.command.sdist import sdist
-from distutils2.command.sdist import show_formats
 from distutils2.dist import Distribution
-from distutils2.tests import unittest
+from distutils2.util import find_executable
 from distutils2.errors import PackagingOptionError
-from distutils2.util import find_executable
-from distutils2.tests import support
+from distutils2.command.sdist import sdist, show_formats
 from distutils2._backport import tarfile
 from distutils2._backport.shutil import get_archive_formats
 
+from distutils2.tests import support, unittest
+from distutils2.tests import captured_stdout
+from distutils2.tests.support import requires_zlib
+
 
 MANIFEST = """\
 # file GENERATED by distutils2, do NOT edit
@@ -89,7 +86,6 @@
 
         # creating VCS directories with some files in them
         os.mkdir(join(self.tmp_dir, 'somecode', '.svn'))
-
         self.write_file((self.tmp_dir, 'somecode', '.svn', 'ok.py'), 'xxx')
 
         os.mkdir(join(self.tmp_dir, 'somecode', '.hg'))
@@ -147,7 +143,7 @@
 
         # now trying a tar then a gztar
         cmd.formats = ['tar', 'gztar']
-
+        cmd.finalized = False
         cmd.ensure_finalized()
         cmd.run()
 
@@ -223,12 +219,14 @@
         # testing the `check-metadata` option
         dist, cmd = self.get_cmd(metadata={'name': 'xxx', 'version': 'xxx'})
 
-        # this should raise some warnings
-        # with the check subcommand
+        # this should cause the check subcommand to log two warnings:
+        # version is invalid, home-page and author are missing
         cmd.ensure_finalized()
         cmd.run()
-        warnings = self.get_logs(logging.WARN)
-        self.assertEqual(len(warnings), 4)
+        warnings = self.get_logs()
+        check_warnings = [msg for msg in warnings if
+                          not msg.startswith('sdist:')]
+        self.assertEqual(len(check_warnings), 2, warnings)
 
         # trying with a complete set of metadata
         self.loghandler.flush()
@@ -236,13 +234,10 @@
         cmd.ensure_finalized()
         cmd.metadata_check = False
         cmd.run()
-        warnings = self.get_logs(logging.WARN)
-        # removing manifest generated warnings
-        warnings = [warn for warn in warnings if
-                    not warn.endswith('-- skipping')]
-        # the remaining warnings are about the use of the default file list and
-        # the absence of setup.cfg
+        warnings = self.get_logs()
         self.assertEqual(len(warnings), 2)
+        self.assertIn('using default file list', warnings[0])
+        self.assertIn("'setup.cfg' file not found", warnings[1])
 
     def test_show_formats(self):
         __, stdout = captured_stdout(show_formats)
@@ -254,7 +249,6 @@
         self.assertEqual(len(output), num_formats)
 
     def test_finalize_options(self):
-
         dist, cmd = self.get_cmd()
         cmd.finalize_options()
 
@@ -274,6 +268,18 @@
         self.assertRaises(PackagingOptionError, cmd.finalize_options)
 
     @requires_zlib
+    def test_template(self):
+        dist, cmd = self.get_cmd()
+        dist.extra_files = ['include yeah']
+        cmd.ensure_finalized()
+        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
+        cmd.run()
+        with open(cmd.manifest) as f:
+            content = f.read()
+
+        self.assertIn('yeah', content)
+
+    @requires_zlib
     @unittest.skipUnless(UID_GID_SUPPORT, "requires grp and pwd support")
     @unittest.skipIf(find_executable('tar') is None or
                      find_executable('gzip') is None,
@@ -291,13 +297,10 @@
 
         # making sure we have the good rights
         archive_name = join(self.tmp_dir, 'dist', 'fake-1.0.tar.gz')
-        archive = tarfile.open(archive_name)
-        try:
+        with tarfile.open(archive_name) as archive:
             for member in archive.getmembers():
                 self.assertEqual(member.uid, 0)
                 self.assertEqual(member.gid, 0)
-        finally:
-            archive.close()
 
         # building a sdist again
         dist, cmd = self.get_cmd()
@@ -309,15 +312,12 @@
 
         # making sure we have the good rights
         archive_name = join(self.tmp_dir, 'dist', 'fake-1.0.tar.gz')
-        archive = tarfile.open(archive_name)
-        try:
+        with tarfile.open(archive_name) as archive:
             # note that we are not testing the group ownership here
             # because, depending on the platforms and the container
             # rights (see #7408)
             for member in archive.getmembers():
                 self.assertEqual(member.uid, os.getuid())
-        finally:
-            archive.close()
 
     @requires_zlib
     def test_get_file_list(self):
@@ -383,18 +383,6 @@
         self.assertEqual(manifest, ['README.manual'])
 
     @requires_zlib
-    def test_template(self):
-        dist, cmd = self.get_cmd()
-        dist.extra_files = ['include yeah']
-        cmd.ensure_finalized()
-        self.write_file((self.tmp_dir, 'yeah'), 'xxx')
-        cmd.run()
-        with open(cmd.manifest) as f:
-            content = f.read()
-
-        self.assertIn('yeah', content)
-
-    @requires_zlib
     def test_manifest_builder(self):
         dist, cmd = self.get_cmd()
         cmd.manifest_builders = 'distutils2.tests.test_command_sdist.builder'
diff --git a/distutils2/tests/test_command_test.py b/distutils2/tests/test_command_test.py
--- a/distutils2/tests/test_command_test.py
+++ b/distutils2/tests/test_command_test.py
@@ -2,7 +2,6 @@
 import re
 import sys
 import shutil
-import logging
 import unittest as ut1
 import distutils2.database
 
@@ -140,7 +139,8 @@
         cmd.run()
         self.assertEqual(['build has run'], record)
 
-    def _test_works_with_2to3(self):
+    @unittest.skip('needs to be written')
+    def test_works_with_2to3(self):
         pass
 
     def test_checks_requires(self):
@@ -149,7 +149,7 @@
         phony_project = 'ohno_ohno-impossible_1234-name_stop-that!'
         cmd.tests_require = [phony_project]
         cmd.ensure_finalized()
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertIn(phony_project, logs[-1])
 
     def prepare_a_module(self):
diff --git a/distutils2/tests/test_command_upload.py b/distutils2/tests/test_command_upload.py
--- a/distutils2/tests/test_command_upload.py
+++ b/distutils2/tests/test_command_upload.py
@@ -129,7 +129,7 @@
         dist_files = [(command, pyversion, filename)]
         docs_path = os.path.join(self.tmp_dir, "build", "docs")
         os.makedirs(docs_path)
-        self.write_file(os.path.join(docs_path, "index.html"), "yellow")
+        self.write_file((docs_path, "index.html"), "yellow")
         self.write_file(self.rc, PYPIRC)
 
         # let's run it
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -1,6 +1,7 @@
 """Tests for distutils2.command.upload_docs."""
 import os
 import shutil
+import logging
 import zipfile
 try:
     import _ssl
@@ -70,9 +71,8 @@
         if sample_dir is None:
             sample_dir = self.mkdtemp()
         os.mkdir(os.path.join(sample_dir, "docs"))
-        self.write_file(os.path.join(sample_dir, "docs", "index.html"),
-                        "Ce mortel ennui")
-        self.write_file(os.path.join(sample_dir, "index.html"), "Oh la la")
+        self.write_file((sample_dir, "docs", "index.html"), "Ce mortel ennui")
+        self.write_file((sample_dir, "index.html"), "Oh la la")
         return sample_dir
 
     def test_zip_dir(self):
@@ -141,13 +141,16 @@
         self.pypi.default_response_status = '403 Forbidden'
         self.prepare_command()
         self.cmd.run()
-        self.assertIn('Upload failed (403): Forbidden', self.get_logs()[-1])
+        errors = self.get_logs(logging.ERROR)
+        self.assertEqual(len(errors), 1)
+        self.assertIn('Upload failed (403): Forbidden', errors[0])
 
         self.pypi.default_response_status = '301 Moved Permanently'
         self.pypi.default_response_headers.append(
             ("Location", "brand_new_location"))
         self.cmd.run()
-        self.assertIn('brand_new_location', self.get_logs()[-1])
+        lastlog = self.get_logs(logging.INFO)[-1]
+        self.assertIn('brand_new_location', lastlog)
 
     def test_reads_pypirc_data(self):
         self.write_file(self.rc, PYPIRC % self.pypi.full_address)
@@ -171,7 +174,7 @@
         self.prepare_command()
         self.cmd.show_response = True
         self.cmd.run()
-        record = self.get_logs()[-1]
+        record = self.get_logs(logging.INFO)[-1]
         self.assertTrue(record, "should report the response")
         self.assertIn(self.pypi.default_response_data, record)
 
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -1,8 +1,6 @@
 """Tests for distutils2.config."""
 import os
 import sys
-import logging
-from io import StringIO
 
 from distutils2 import command
 from distutils2.dist import Distribution
@@ -184,13 +182,14 @@
 
     def __init__(self, dist):
         self.distribution = dist
+        self._record = []
 
     @classmethod
     def get_command_name(cls):
         return 'foo'
 
     def run(self):
-        self.distribution.foo_was_here = True
+        self._record.append('foo has run')
 
     def nothing(self):
         pass
@@ -210,21 +209,11 @@
 
     def setUp(self):
         super(ConfigTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        sys.stdout = StringIO()
-        sys.stderr = StringIO()
-
-        self.addCleanup(os.chdir, os.getcwd())
         tempdir = self.mkdtemp()
         self.working_dir = os.getcwd()
         os.chdir(tempdir)
         self.tempdir = tempdir
 
-    def tearDown(self):
-        os.chdir(self.working_dir)
-        super(ConfigTestCase, self).tearDown()
-
     def write_setup(self, kwargs=None):
         opts = {'description-file': 'README', 'extra-files': '',
                 'setup-hooks': 'distutils2.tests.test_config.version_hook'}
@@ -375,15 +364,14 @@
         self.write_file('README', 'yeah')
         self.write_file('hooks.py', HOOKS_MODULE)
         self.get_dist()
-        logs = self.get_logs(logging.WARNING)
-        self.assertEqual(['logging_hook called'], logs)
+        self.assertEqual(['logging_hook called'], self.get_logs())
         self.assertIn('hooks', sys.modules)
 
     def test_missing_setup_hook_warns(self):
-        self.write_setup({'setup-hooks': 'this.does._not.exist'})
+        self.write_setup({'setup-hooks': 'does._not.exist'})
         self.write_file('README', 'yeah')
         self.get_dist()
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(1, len(logs))
         self.assertIn('cannot find setup hook', logs[0])
 
@@ -397,7 +385,7 @@
         dist = self.get_dist()
 
         self.assertEqual(['haven', 'first', 'third'], dist.py_modules)
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(1, len(logs))
         self.assertIn('cannot find setup hook', logs[0])
 
@@ -493,10 +481,12 @@
             self.write_file((pkg, '__init__.py'), '#')
 
         # try to run the install command to see if foo is called
+        self.addCleanup(command._COMMANDS.__delitem__, 'foo')
         dist = self.get_dist()
-        self.assertIn('foo', command.get_command_names())
-        self.assertEqual('FooBarBazTest',
-                         dist.get_command_obj('foo').__class__.__name__)
+        dist.run_command('install_dist')
+        cmd = dist.get_command_obj('foo')
+        self.assertEqual(cmd.__class__.__name__, 'FooBarBazTest')
+        self.assertEqual(cmd._record, ['foo has run'])
 
 
 def test_suite():
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -1,16 +1,18 @@
 """Tests for distutils2.create."""
 import os
 import sys
-from io import StringIO
 from textwrap import dedent
+from distutils2 import create
 from distutils2.create import MainProgram, ask_yn, ask, main
 from distutils2._backport import sysconfig
 
 from distutils2.tests import support, unittest
+from distutils2.tests.support import Inputs
 
 
 class CreateTestCase(support.TempdirManager,
                      support.EnvironRestorer,
+                     support.LoggingCatcher,
                      unittest.TestCase):
 
     maxDiff = None
@@ -18,11 +20,6 @@
 
     def setUp(self):
         super(CreateTestCase, self).setUp()
-        self._stdin = sys.stdin  # TODO use Inputs
-        self._stdout = sys.stdout
-        sys.stdin = StringIO()
-        sys.stdout = StringIO()
-        self._cwd = os.getcwd()
         self.wdir = self.mkdtemp()
         os.chdir(self.wdir)
         # patch sysconfig
@@ -32,29 +29,24 @@
             'doc': sys.prefix + '/share/doc/pyxfoil', }
 
     def tearDown(self):
-        sys.stdin = self._stdin
-        sys.stdout = self._stdout
-        os.chdir(self._cwd)
         sysconfig.get_paths = self._old_get_paths
+        if hasattr(create, 'input'):
+            del create.input
         super(CreateTestCase, self).tearDown()
 
     def test_ask_yn(self):
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         self.assertEqual('y', ask_yn('is this a test'))
 
     def test_ask(self):
-        sys.stdin.write('a\n')
-        sys.stdin.write('b\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('a', 'b')
         self.assertEqual('a', ask('is this a test'))
         self.assertEqual('b', ask(str(list(range(0, 70))), default='c',
                                   lengthy=True))
 
     def test_set_multi(self):
         mainprogram = MainProgram()
-        sys.stdin.write('aaaaa\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('aaaaa')
         mainprogram.data['author'] = []
         mainprogram._set_multi('_set_multi test', 'author')
         self.assertEqual(['aaaaa'], mainprogram.data['author'])
@@ -80,8 +72,7 @@
             os.mkdir(os.path.join(tempdir, dir_))
 
         for file_ in files:
-            path = os.path.join(tempdir, file_)
-            self.write_file(path, 'xxx')
+            self.write_file((tempdir, file_), 'xxx')
 
         mainprogram._find_files()
         mainprogram.data['packages'].sort()
@@ -131,8 +122,7 @@
               scripts=['my_script', 'bin/run'],
               )
         """), encoding='utf-8')
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
@@ -207,9 +197,7 @@
 barbar is now in the public domain,
 ho, baby!
                         '''))
-        sys.stdin.write('y\n')
-        sys.stdin.seek(0)
-        # FIXME Out of memory error.
+        create.input = Inputs('y')
         main()
 
         path = os.path.join(self.wdir, 'setup.cfg')
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -1,33 +1,37 @@
 """Tests for distutils2.dist."""
 import os
 import sys
-import logging
 import textwrap
 
 import distutils2.dist
 
 from distutils2.dist import Distribution
-from distutils2.command import set_command
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingModuleError, PackagingOptionError
 from distutils2.tests import captured_stdout
 from distutils2.tests import support, unittest
-from distutils2.tests.support import create_distribution
+from distutils2.tests.support import create_distribution, use_command
 from distutils2.tests.support import unload
 
 
 class test_dist(Command):
-    """Sample distutils2 extension command."""
+    """Custom command used for testing."""
 
     user_options = [
-        ("sample-option=", "S", "help text"),
+        ('sample-option=', 'S',
+         "help text"),
         ]
 
     def initialize_options(self):
         self.sample_option = None
+        self._record = []
 
     def finalize_options(self):
-        pass
+        if self.sample_option is None:
+            self.sample_option = 'default value'
+
+    def run(self):
+        self._record.append('test_dist has run')
 
 
 class DistributionTestCase(support.TempdirManager,
@@ -39,6 +43,8 @@
 
     def setUp(self):
         super(DistributionTestCase, self).setUp()
+        # XXX this is ugly, we should fix the functions to accept args
+        # (defaulting to sys.argv)
         self.argv = sys.argv, sys.argv[:]
         del sys.argv[1:]
 
@@ -74,7 +80,7 @@
                             'version': '1.2',
                             'home_page': 'xxxx',
                             'badoptname': 'xxx'})
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(len(logs), 1)
         self.assertIn('unknown argument', logs[0])
 
@@ -85,7 +91,7 @@
                                    'version': '1.2', 'home_page': 'xxxx',
                                    'options': {}})
 
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertEqual(self.get_logs(), [])
         self.assertNotIn('options', dir(dist))
 
     def test_non_empty_options(self):
@@ -173,7 +179,8 @@
         self.write_file((temp_home, "config2.cfg"),
                         '[test_dist]\npre-hook.b = type')
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
+
         dist = create_distribution(config_files)
         cmd = dist.get_command_obj("test_dist")
         self.assertEqual(cmd.pre_hook, {"a": 'type', "b": 'type'})
@@ -201,7 +208,7 @@
                 record.append('post-%s' % cmd.get_command_name())
             '''))
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
 
@@ -228,7 +235,7 @@
             [test_dist]
             pre-hook.test = nonexistent.dotted.name'''))
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
@@ -243,7 +250,7 @@
             [test_dist]
             pre-hook.test = distutils2.tests.test_dist.__doc__'''))
 
-        set_command('distutils2.tests.test_dist.test_dist')
+        use_command(self, 'distutils2.tests.test_dist.test_dist')
         d = create_distribution([config_file])
         cmd = d.get_command_obj("test_dist")
         cmd.ensure_finalized()
diff --git a/distutils2/tests/test_manifest.py b/distutils2/tests/test_manifest.py
--- a/distutils2/tests/test_manifest.py
+++ b/distutils2/tests/test_manifest.py
@@ -1,8 +1,9 @@
 """Tests for distutils2.manifest."""
 import os
-import logging
+import re
 from io import StringIO
-from distutils2.manifest import Manifest
+from distutils2.errors import PackagingTemplateError
+from distutils2.manifest import Manifest, _translate_pattern, _glob_to_re
 
 from distutils2.tests import unittest, support
 
@@ -26,13 +27,11 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
 
-    def setUp(self):
-        super(ManifestTestCase, self).setUp()
-        self.cwd = os.getcwd()
+    def assertNoWarnings(self):
+        self.assertEqual(self.get_logs(), [])
 
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(ManifestTestCase, self).tearDown()
+    def assertWarnings(self):
+        self.assertNotEqual(self.get_logs(), [])
 
     def test_manifest_reader(self):
         tmpdir = self.mkdtemp()
@@ -43,7 +42,7 @@
         manifest = Manifest()
         manifest.read_template(MANIFEST)
 
-        warnings = self.get_logs(logging.WARNING)
+        warnings = self.get_logs()
         # the manifest should have been read and 3 warnings issued
         # (we didn't provide the files)
         self.assertEqual(3, len(warnings))
@@ -69,6 +68,193 @@
         manifest.read_template(content)
         self.assertEqual(['README', 'file1'], manifest.files)
 
+    def test_glob_to_re(self):
+        # simple cases
+        self.assertEqual(_glob_to_re('foo*'), 'foo[^/]*\\Z(?ms)')
+        self.assertEqual(_glob_to_re('foo?'), 'foo[^/]\\Z(?ms)')
+        self.assertEqual(_glob_to_re('foo??'), 'foo[^/][^/]\\Z(?ms)')
+
+        # special cases
+        self.assertEqual(_glob_to_re(r'foo\\*'), r'foo\\\\[^/]*\Z(?ms)')
+        self.assertEqual(_glob_to_re(r'foo\\\*'), r'foo\\\\\\[^/]*\Z(?ms)')
+        self.assertEqual(_glob_to_re('foo????'), r'foo[^/][^/][^/][^/]\Z(?ms)')
+        self.assertEqual(_glob_to_re(r'foo\\??'), r'foo\\\\[^/][^/]\Z(?ms)')
+
+    def test_remove_duplicates(self):
+        manifest = Manifest()
+        manifest.files = ['a', 'b', 'a', 'g', 'c', 'g']
+        # files must be sorted beforehand
+        manifest.sort()
+        manifest.remove_duplicates()
+        self.assertEqual(manifest.files, ['a', 'b', 'c', 'g'])
+
+    def test_translate_pattern(self):
+        # blackbox test of a private function
+
+        # not regex
+        pattern = _translate_pattern('a', anchor=True, is_regex=False)
+        self.assertTrue(hasattr(pattern, 'search'))
+
+        # is a regex
+        regex = re.compile('a')
+        pattern = _translate_pattern(regex, anchor=True, is_regex=True)
+        self.assertEqual(pattern, regex)
+
+        # plain string flagged as regex
+        pattern = _translate_pattern('a', anchor=True, is_regex=True)
+        self.assertTrue(hasattr(pattern, 'search'))
+
+        # glob support
+        pattern = _translate_pattern('*.py', anchor=True, is_regex=False)
+        self.assertTrue(pattern.search('filelist.py'))
+
+    def test_exclude_pattern(self):
+        # return False if no match
+        manifest = Manifest()
+        self.assertFalse(manifest.exclude_pattern('*.py'))
+
+        # return True if files match
+        manifest = Manifest()
+        manifest.files = ['a.py', 'b.py']
+        self.assertTrue(manifest.exclude_pattern('*.py'))
+
+        # test excludes
+        manifest = Manifest()
+        manifest.files = ['a.py', 'a.txt']
+        manifest.exclude_pattern('*.py')
+        self.assertEqual(manifest.files, ['a.txt'])
+
+    def test_include_pattern(self):
+        # return False if no match
+        manifest = Manifest()
+        manifest.allfiles = []
+        self.assertFalse(manifest._include_pattern('*.py'))
+
+        # return True if files match
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt']
+        self.assertTrue(manifest._include_pattern('*.py'))
+
+        # test * matches all files
+        manifest = Manifest()
+        self.assertIsNone(manifest.allfiles)
+        manifest.allfiles = ['a.py', 'b.txt']
+        manifest._include_pattern('*')
+        self.assertEqual(manifest.allfiles, ['a.py', 'b.txt'])
+
+    def test_process_template(self):
+        # invalid lines
+        manifest = Manifest()
+        for action in ('include', 'exclude', 'global-include',
+                       'global-exclude', 'recursive-include',
+                       'recursive-exclude', 'graft', 'prune'):
+            self.assertRaises(PackagingTemplateError,
+                              manifest._process_template_line, action)
+
+        # implicit include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('*.py')
+        self.assertEqual(manifest.files, ['a.py'])
+        self.assertNoWarnings()
+
+        # include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('include *.py')
+        self.assertEqual(manifest.files, ['a.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('include *.rb')
+        self.assertEqual(manifest.files, ['a.py'])
+        self.assertWarnings()
+
+        # exclude
+        manifest = Manifest()
+        manifest.files = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('exclude *.py')
+        self.assertEqual(manifest.files, ['b.txt', 'd/c.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('exclude *.rb')
+        self.assertEqual(manifest.files, ['b.txt', 'd/c.py'])
+        self.assertWarnings()
+
+        # global-include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('global-include *.py')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('global-include *.rb')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.py'])
+        self.assertWarnings()
+
+        # global-exclude
+        manifest = Manifest()
+        manifest.files = ['a.py', 'b.txt', 'd/c.py']
+
+        manifest._process_template_line('global-exclude *.py')
+        self.assertEqual(manifest.files, ['b.txt'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('global-exclude *.rb')
+        self.assertEqual(manifest.files, ['b.txt'])
+        self.assertWarnings()
+
+        # recursive-include
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'd/b.py', 'd/c.txt', 'd/d/e.py']
+
+        manifest._process_template_line('recursive-include d *.py')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('recursive-include e *.py')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertWarnings()
+
+        # recursive-exclude
+        manifest = Manifest()
+        manifest.files = ['a.py', 'd/b.py', 'd/c.txt', 'd/d/e.py']
+
+        manifest._process_template_line('recursive-exclude d *.py')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.txt'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('recursive-exclude e *.py')
+        self.assertEqual(manifest.files, ['a.py', 'd/c.txt'])
+        self.assertWarnings()
+
+        # graft
+        manifest = Manifest()
+        manifest.allfiles = ['a.py', 'd/b.py', 'd/d/e.py', 'f/f.py']
+
+        manifest._process_template_line('graft d')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('graft e')
+        self.assertEqual(manifest.files, ['d/b.py', 'd/d/e.py'])
+        self.assertWarnings()
+
+        # prune
+        manifest = Manifest()
+        manifest.files = ['a.py', 'd/b.py', 'd/d/e.py', 'f/f.py']
+
+        manifest._process_template_line('prune d')
+        self.assertEqual(manifest.files, ['a.py', 'f/f.py'])
+        self.assertNoWarnings()
+
+        manifest._process_template_line('prune e')
+        self.assertEqual(manifest.files, ['a.py', 'f/f.py'])
+        self.assertWarnings()
+
 
 def test_suite():
     return unittest.makeSuite(ManifestTestCase)
diff --git a/distutils2/tests/test_metadata.py b/distutils2/tests/test_metadata.py
--- a/distutils2/tests/test_metadata.py
+++ b/distutils2/tests/test_metadata.py
@@ -1,7 +1,6 @@
 """Tests for distutils2.metadata."""
 import os
 import sys
-import logging
 from textwrap import dedent
 from io import StringIO
 
@@ -302,7 +301,7 @@
                           'name': 'xxx',
                           'version': 'xxx',
                           'home_page': 'xxxx'})
-        logs = self.get_logs(logging.WARNING)
+        logs = self.get_logs()
         self.assertEqual(1, len(logs))
         self.assertIn('not a valid version', logs[0])
 
@@ -418,7 +417,7 @@
         # XXX check PEP and see if 3 == 3.0
         metadata['Requires-Python'] = '>=2.6, <3.0'
         metadata['Requires-Dist'] = ['Foo (>=2.6, <3.0)']
-        self.assertEqual([], self.get_logs(logging.WARNING))
+        self.assertEqual(self.get_logs(), [])
 
     @unittest.skip('needs to be implemented')
     def test_requires_illegal(self):
diff --git a/distutils2/tests/test_mixin2to3.py b/distutils2/tests/test_mixin2to3.py
--- a/distutils2/tests/test_mixin2to3.py
+++ b/distutils2/tests/test_mixin2to3.py
@@ -1,4 +1,3 @@
-import sys
 import textwrap
 
 from distutils2.tests import unittest, support
@@ -9,6 +8,7 @@
                         support.LoggingCatcher,
                         unittest.TestCase):
 
+    @support.skip_2to3_optimize
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
diff --git a/distutils2/tests/test_pypi_simple.py b/distutils2/tests/test_pypi_simple.py
--- a/distutils2/tests/test_pypi_simple.py
+++ b/distutils2/tests/test_pypi_simple.py
@@ -87,7 +87,7 @@
         try:
             crawler._open_url(url)
         except Exception as v:
-            if sys.version_info[:2] < (3, 3):
+            if sys.version_info[:2] < (3, 2, 3):
                 wanted = 'nonnumeric port'
             else:
                 wanted = 'Download error'
diff --git a/distutils2/tests/test_run.py b/distutils2/tests/test_run.py
--- a/distutils2/tests/test_run.py
+++ b/distutils2/tests/test_run.py
@@ -33,11 +33,9 @@
 
     def setUp(self):
         super(RunTestCase, self).setUp()
-        self.old_stdout = sys.stdout
         self.old_argv = sys.argv, sys.argv[:]
 
     def tearDown(self):
-        sys.stdout = self.old_stdout
         sys.argv = self.old_argv[0]
         sys.argv[:] = self.old_argv[1]
         super(RunTestCase, self).tearDown()
@@ -67,7 +65,7 @@
         pythonpath = os.environ.get('PYTHONPATH')
         d2parent = os.path.dirname(os.path.dirname(__file__))
         if pythonpath is not None:
-            pythonpath =  os.pathsep.join((pythonpath, d2parent))
+            pythonpath = os.pathsep.join((pythonpath, d2parent))
         else:
             pythonpath = d2parent
 
diff --git a/distutils2/tests/test_uninstall.py b/distutils2/tests/test_uninstall.py
--- a/distutils2/tests/test_uninstall.py
+++ b/distutils2/tests/test_uninstall.py
@@ -1,6 +1,5 @@
 """Tests for the distutils2.uninstall module."""
 import os
-import sys
 import logging
 import distutils2.util
 
@@ -31,19 +30,10 @@
 
     def setUp(self):
         super(UninstallTestCase, self).setUp()
-        self.addCleanup(setattr, sys, 'stdout', sys.stdout)
-        self.addCleanup(setattr, sys, 'stderr', sys.stderr)
-        self.addCleanup(os.chdir, os.getcwd())
         self.addCleanup(enable_cache)
-        self.root_dir = self.mkdtemp()
-        self.cwd = os.getcwd()
+        self.addCleanup(distutils2.util._path_created.clear)
         disable_cache()
 
-    def tearDown(self):
-        os.chdir(self.cwd)
-        distutils2.util._path_created.clear()
-        super(UninstallTestCase, self).tearDown()
-
     def get_path(self, dist, name):
         # the dist argument must contain an install_dist command correctly
         # initialized with a prefix option and finalized befored this method
@@ -61,8 +51,7 @@
         kw['pkg'] = pkg
 
         pkg_dir = os.path.join(project_dir, pkg)
-        os.mkdir(pkg_dir)
-        os.mkdir(os.path.join(pkg_dir, 'sub'))
+        os.makedirs(os.path.join(pkg_dir, 'sub'))
 
         self.write_file((project_dir, 'setup.cfg'), SETUP_CFG % kw)
         self.write_file((pkg_dir, '__init__.py'), '#')
@@ -85,7 +74,7 @@
         dist.parse_config_files()
         dist.finalize_options()
         dist.run_command('install_dist',
-                         {'prefix': ('command line', self.root_dir)})
+                         {'prefix': ('command line', self.mkdtemp())})
 
         site_packages = self.get_path(dist, 'purelib')
         return dist, site_packages
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -10,7 +10,7 @@
 from io import StringIO
 
 from distutils2.errors import (
-    PackagingPlatformError, PackagingByteCompileError, PackagingFileError,
+    PackagingPlatformError, PackagingFileError,
     PackagingExecError, InstallationException)
 from distutils2 import util
 from distutils2.dist import Distribution
@@ -138,15 +138,8 @@
             self._uname = None
         os.uname = self._get_uname
 
-        # patching POpen
-        self.old_find_executable = util.find_executable
-        util.find_executable = self._find_executable
-        self._exes = {}
-        self.old_popen = subprocess.Popen
-        self.old_stdout = sys.stdout
-        self.old_stderr = sys.stderr
-        FakePopen.test_class = self
-        subprocess.Popen = FakePopen
+    def _get_uname(self):
+        return self._uname
 
     def tearDown(self):
         # getting back the environment
@@ -161,17 +154,24 @@
             os.uname = self.uname
         else:
             del os.uname
+        super(UtilTestCase, self).tearDown()
+
+    def mock_popen(self):
+        self.old_find_executable = util.find_executable
+        util.find_executable = self._find_executable
+        self._exes = {}
+        self.old_popen = subprocess.Popen
+        self.old_stdout = sys.stdout
+        self.old_stderr = sys.stderr
+        FakePopen.test_class = self
+        subprocess.Popen = FakePopen
+        self.addCleanup(self.unmock_popen)
+
+    def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
-        sys.old_stdout = self.old_stdout
-        sys.old_stderr = self.old_stderr
-        super(UtilTestCase, self).tearDown()
-
-    def _set_uname(self, uname):
-        self._uname = uname
-
-    def _get_uname(self):
-        return self._uname
+        sys.stdout = self.old_stdout
+        sys.stderr = self.old_stderr
 
     def test_convert_path(self):
         # linux/mac
@@ -283,6 +283,7 @@
         return None
 
     def test_get_compiler_versions(self):
+        self.mock_popen()
         # get_versions calls distutils.spawn.find_executable on
         # 'gcc', 'ld' and 'dllwrap'
         self.assertEqual(get_compiler_versions(), (None, None, None))
@@ -323,15 +324,12 @@
         res = get_compiler_versions()
         self.assertEqual(res[2], None)
 
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile raise a PackagingError
-        # if sys.dont_write_bytecode is True
-        old_dont_write_bytecode = sys.dont_write_bytecode
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            self.assertRaises(PackagingByteCompileError, byte_compile, [])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
+        byte_compile([])
 
     def test_newer(self):
         self.assertRaises(PackagingFileError, util.newer, 'xxx', 'xxx')
@@ -359,56 +357,66 @@
         #
         root = self.mkdtemp()
         pkg1 = os.path.join(root, 'pkg1')
-        os.mkdir(pkg1)
-        self.write_file(os.path.join(pkg1, '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg2'))
-        self.write_file(os.path.join(pkg1, 'pkg2', '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg3'))
-        self.write_file(os.path.join(pkg1, 'pkg3', '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg3', 'pkg6'))
-        self.write_file(os.path.join(pkg1, 'pkg3', 'pkg6', '__init__.py'))
-        os.mkdir(os.path.join(pkg1, 'pkg4'))
-        os.mkdir(os.path.join(pkg1, 'pkg4', 'pkg8'))
-        self.write_file(os.path.join(pkg1, 'pkg4', 'pkg8', '__init__.py'))
-        pkg5 = os.path.join(root, 'pkg5')
-        os.mkdir(pkg5)
-        self.write_file(os.path.join(pkg5, '__init__.py'))
+        os.makedirs(os.path.join(pkg1, 'pkg2'))
+        os.makedirs(os.path.join(pkg1, 'pkg3', 'pkg6'))
+        os.makedirs(os.path.join(pkg1, 'pkg4', 'pkg8'))
+        os.makedirs(os.path.join(root, 'pkg5'))
+        self.write_file((pkg1, '__init__.py'))
+        self.write_file((pkg1, 'pkg2', '__init__.py'))
+        self.write_file((pkg1, 'pkg3', '__init__.py'))
+        self.write_file((pkg1, 'pkg3', 'pkg6', '__init__.py'))
+        self.write_file((pkg1, 'pkg4', 'pkg8', '__init__.py'))
+        self.write_file((root, 'pkg5', '__init__.py'))
 
         res = find_packages([root], ['pkg1.pkg2'])
-        self.assertEqual(set(res), set(['pkg1', 'pkg5', 'pkg1.pkg3',
-                                        'pkg1.pkg3.pkg6']))
+        self.assertEqual(sorted(res),
+                         ['pkg1', 'pkg1.pkg3', 'pkg1.pkg3.pkg6', 'pkg5'])
 
     def test_resolve_name(self):
-        self.assertIs(str, resolve_name('builtins.str'))
-        self.assertEqual(
-            UtilTestCase.__name__,
-            resolve_name("distutils2.tests.test_util.UtilTestCase").__name__)
-        self.assertEqual(
-            UtilTestCase.test_resolve_name.__name__,
-            resolve_name("distutils2.tests.test_util.UtilTestCase."
-                         "test_resolve_name").__name__)
+        # test raw module name
+        tmpdir = self.mkdtemp()
+        sys.path.append(tmpdir)
+        self.addCleanup(sys.path.remove, tmpdir)
+        self.write_file((tmpdir, 'hello.py'), '')
 
-        self.assertRaises(ImportError, resolve_name,
-                          "distutils2.tests.test_util.UtilTestCaseNot")
-        self.assertRaises(ImportError, resolve_name,
-                          "distutils2.tests.test_util.UtilTestCase."
-                          "nonexistent_attribute")
+        os.makedirs(os.path.join(tmpdir, 'a', 'b'))
+        self.write_file((tmpdir, 'a', '__init__.py'), '')
+        self.write_file((tmpdir, 'a', 'b', '__init__.py'), '')
+        self.write_file((tmpdir, 'a', 'b', 'c.py'), 'class Foo: pass')
+        self.write_file((tmpdir, 'a', 'b', 'd.py'), textwrap.dedent("""\
+                         class FooBar:
+                             class Bar:
+                                 def baz(self):
+                                     pass
+                            """))
 
-    def test_import_nested_first_time(self):
-        tmp_dir = self.mkdtemp()
-        os.makedirs(os.path.join(tmp_dir, 'a', 'b'))
-        self.write_file(os.path.join(tmp_dir, 'a', '__init__.py'), '')
-        self.write_file(os.path.join(tmp_dir, 'a', 'b', '__init__.py'), '')
-        self.write_file(os.path.join(tmp_dir, 'a', 'b', 'c.py'),
-                                    'class Foo: pass')
+        # check Python, C and built-in module
+        self.assertEqual(resolve_name('hello').__name__, 'hello')
+        self.assertEqual(resolve_name('_csv').__name__, '_csv')
+        self.assertEqual(resolve_name('sys').__name__, 'sys')
 
-        try:
-            sys.path.append(tmp_dir)
-            resolve_name("a.b.c.Foo")
-            # assert nothing raised
-        finally:
-            sys.path.remove(tmp_dir)
+        # test module.attr
+        self.assertIs(resolve_name('builtins.str'), str)
+        self.assertIsNone(resolve_name('hello.__doc__'))
+        self.assertEqual(resolve_name('a.b.c.Foo').__name__, 'Foo')
+        self.assertEqual(resolve_name('a.b.d.FooBar.Bar.baz').__name__, 'baz')
 
+        # error if module not found
+        self.assertRaises(ImportError, resolve_name, 'nonexistent')
+        self.assertRaises(ImportError, resolve_name, 'non.existent')
+        self.assertRaises(ImportError, resolve_name, 'a.no')
+        self.assertRaises(ImportError, resolve_name, 'a.b.no')
+        self.assertRaises(ImportError, resolve_name, 'a.b.no.no')
+        self.assertRaises(ImportError, resolve_name, 'inva-lid')
+
+        # looking up built-in names is not supported
+        self.assertRaises(ImportError, resolve_name, 'str')
+
+        # error if module found but not attr
+        self.assertRaises(ImportError, resolve_name, 'a.b.Spam')
+        self.assertRaises(ImportError, resolve_name, 'a.b.c.Spam')
+
+    @support.skip_2to3_optimize
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -422,6 +430,7 @@
         file_handle.close()
         self.assertEqual(new_content, converted_content)
 
+    @support.skip_2to3_optimize
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
@@ -439,8 +448,6 @@
     @unittest.skipUnless(os.name in ('nt', 'posix'),
                          'runs only under posix or nt')
     def test_spawn(self):
-        # no patching of Popen here
-        subprocess.Popen = self.old_popen
         tmpdir = self.mkdtemp()
 
         # creating something executable
@@ -538,8 +545,6 @@
         self.assertEqual(args['py_modules'], dist.py_modules)
 
     def test_generate_setup_py(self):
-        # undo subprocess.Popen monkey-patching before using assert_python_*
-        subprocess.Popen = self.old_popen
         os.chdir(self.mkdtemp())
         self.write_file('setup.cfg', textwrap.dedent("""\
             [metadata]
@@ -599,14 +604,6 @@
 
 class GlobTestCase(GlobTestCaseBase):
 
-    def setUp(self):
-        super(GlobTestCase, self).setUp()
-        self.cwd = os.getcwd()
-
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(GlobTestCase, self).tearDown()
-
     def assertGlobMatch(self, glob, spec):
         tempdir = self.build_files_tree(spec)
         expected = self.clean_tree(spec)
diff --git a/distutils2/tests/test_version.py b/distutils2/tests/test_version.py
--- a/distutils2/tests/test_version.py
+++ b/distutils2/tests/test_version.py
@@ -1,6 +1,5 @@
 """Tests for distutils2.version."""
 import doctest
-import os
 
 from distutils2.version import NormalizedVersion as V
 from distutils2.version import HugeMajorVersionNumError, IrrationalVersionError
@@ -46,7 +45,6 @@
     def test_from_parts(self):
 
         for v, s in self.versions:
-            parts = v.parts
             v2 = V.from_parts(*v.parts)
             self.assertEqual(v, v2)
             self.assertEqual(str(v), str(v2))
@@ -192,7 +190,7 @@
                       'Hey (>=2.5,<2.7)')
 
         for predicate in predicates:
-            v = VersionPredicate(predicate)
+            VersionPredicate(predicate)
 
         self.assertTrue(VersionPredicate('Hey (>=2.5,<2.7)').match('2.6'))
         self.assertTrue(VersionPredicate('Ho').match('2.6'))
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -18,9 +18,10 @@
 from configparser import RawConfigParser
 
 from distutils2 import logger
+from distutils2.compat import cache_from_source
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
-                               PackagingByteCompileError, PackagingExecError,
-                               InstallationException, PackagingInternalError)
+                               PackagingExecError, InstallationException,
+                               PackagingInternalError)
 from distutils2._backport import sysconfig
 
 __all__ = [
@@ -296,7 +297,7 @@
 
 
 def byte_compile(py_files, optimize=0, force=False, prefix=None,
-                 base_dir=None, verbose=0, dry_run=False, direct=None):
+                 base_dir=None, dry_run=False, direct=None):
     """Byte-compile a collection of Python source files to either .pyc
     or .pyo files in the same directory (Python 3.1) or in a __pycache__
     subdirectory (3.2 and newer).
@@ -306,6 +307,9 @@
       0 - don't optimize (generate .pyc)
       1 - normal optimization (like "python -O")
       2 - extra optimization (like "python -OO")
+    This function is independent from the running Python's -O or -B options;
+    it is fully controlled by the parameters passed in.
+
     If 'force' is true, all files are recompiled regardless of
     timestamps.
 
@@ -327,10 +331,7 @@
     generated in indirect mode; unless you know what you're doing, leave
     it set to None.
     """
-    # nothing is done if sys.dont_write_bytecode is True
-    # FIXME this should not raise an error
-    if sys.dont_write_bytecode:
-        raise PackagingByteCompileError('byte-compiling is disabled.')
+    # FIXME use compileall + remove direct/indirect shenanigans
 
     # First, if the caller didn't force us into direct or indirect mode,
     # figure out which mode we should be in.  We take a conservative
@@ -384,15 +385,11 @@
                 script.write("""
 byte_compile(files, optimize=%r, force=%r,
              prefix=%r, base_dir=%r,
-             verbose=%r, dry_run=False,
+             dry_run=False,
              direct=True)
-""" % (optimize, force, prefix, base_dir, verbose))
+""" % (optimize, force, prefix, base_dir))
 
         cmd = [sys.executable, script_name]
-        if optimize == 1:
-            cmd.insert(1, "-O")
-        elif optimize == 2:
-            cmd.insert(1, "-OO")
 
         env = os.environ.copy()
         env['PYTHONPATH'] = os.path.pathsep.join(sys.path)
@@ -418,17 +415,12 @@
             # Terminology from the py_compile module:
             #   cfile - byte-compiled file
             #   dfile - purported source filename (same as 'file' by default)
-            if sys.version_info[1] == 1:
-                cfile = file + (__debug__ and "c" or "o")
-            else:
-                # comply with PEP 3147 in 3.2+
-                if optimize >= 0:
-                    cfile = imp.cache_from_source(file,
-                                                  debug_override=not optimize)
-                else:
-                    cfile = imp.cache_from_source(file)
+            # The second argument to cache_from_source forces the extension to
+            # be .pyc (if true) or .pyo (if false); without it, the extension
+            # would depend on the calling Python's -O option
+            cfile = cache_from_source(file, not optimize)
+            dfile = file
 
-            dfile = file
             if prefix:
                 if file[:len(prefix)] != prefix:
                     raise ValueError("invalid prefix: filename %r doesn't "
@@ -638,22 +630,35 @@
 def resolve_name(name):
     """Resolve a name like ``module.object`` to an object and return it.
 
-    Raise ImportError if the module or name is not found.
+    This functions supports packages and attributes without depth limitation:
+    ``package.package.module.class.class.function.attr`` is valid input.
+    However, looking up builtins is not directly supported: use
+    ``builtins.name``.
+
+    Raises ImportError if importing the module fails or if one requested
+    attribute is not found.
     """
+    if '.' not in name:
+        # shortcut
+        __import__(name)
+        return sys.modules[name]
+
+    # FIXME clean up this code!
     parts = name.split('.')
     cursor = len(parts)
     module_name = parts[:cursor]
+    ret = ''
 
     while cursor > 0:
         try:
             ret = __import__('.'.join(module_name))
             break
         except ImportError:
-            if cursor == 0:
-                raise
             cursor -= 1
             module_name = parts[:cursor]
-            ret = ''
+
+    if ret == '':
+        raise ImportError(parts[0])
 
     for part in parts[1:]:
         try:
@@ -1331,6 +1336,8 @@
 def copy_tree(src, dst, preserve_mode=True, preserve_times=True,
               preserve_symlinks=False, update=False, verbose=True,
               dry_run=False):
+    # FIXME use of this function is why we get spurious logging message on
+    # stdout when tests run; kill and replace by shutil!
     from distutils.file_util import copy_file
 
     if not dry_run and not os.path.isdir(src):
@@ -1444,8 +1451,7 @@
 
     Returns (content_type: bytes, body: bytes) ready for http.client.HTTP.
     """
-    # Taken from
-    # http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
+    # Taken from http://code.activestate.com/recipes/146306
 
     if boundary is None:
         boundary = b'--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Code_simplification?=
Message-ID: <E1RPxSB-000739-UG@dinsdale.python.org>

http://hg.python.org/distutils2/rev/8f69359881aa
changeset:   1240:8f69359881aa
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 03:47:29 2011 +0100
summary:
  Code simplification

files:
  distutils2/metadata.py |  6 ++----
  1 files changed, 2 insertions(+), 4 deletions(-)


diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -230,10 +230,8 @@
 
     def __delitem__(self, name):
         field_name = self._convert_name(name)
-        try:
-            del self._fields[field_name]
-        except KeyError:
-            raise KeyError(name)
+        # we let a KeyError propagate
+        del self._fields[field_name]
         self._set_best_version()
 
     def __contains__(self, name):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Super_considered_super?=
	=?utf8?q?=2E?=
Message-ID: <E1RPxSB-000736-Tn@dinsdale.python.org>

http://hg.python.org/distutils2/rev/6d968e324bc5
changeset:   1239:6d968e324bc5
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 03:46:43 2011 +0100
summary:
  Super considered super.

I think I?ve got all of them (I?ll clean up compat after); if someone
knows a lint tool that can detect missing super calls, please let me
know.

files:
  distutils2/command/bdist_msi.py        |   2 +-
  distutils2/compiler/bcppcompiler.py    |   2 +-
  distutils2/compiler/cygwinccompiler.py |  25 ++++++-------
  distutils2/compiler/msvc9compiler.py   |   2 +-
  distutils2/compiler/msvccompiler.py    |   2 +-
  distutils2/metadata.py                 |   8 ++-
  distutils2/tests/pypi_server.py        |  15 +++----
  distutils2/tests/support.py            |   4 +-
  8 files changed, 29 insertions(+), 31 deletions(-)


diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -35,7 +35,7 @@
     def __init__(self, *args, **kw):
         """Dialog(database, name, x, y, w, h, attributes, title, first,
         default, cancel, bitmap=true)"""
-        Dialog.__init__(self, *args)
+        super(PyDialog, self).__init__(*args)
         ruler = self.h - 36
         #if kw.get("bitmap", True):
         #    self.bitmap("Bitmap", 0, 0, bmwidth, ruler, "PythonWin")
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -48,7 +48,7 @@
 
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-        CCompiler.__init__(self, verbose, dry_run, force)
+        super(BCPPCompiler, self).__init__(verbose, dry_run, force)
 
         # These executables are assumed to all be in the path.
         # Borland doesn't seem to use any special registry settings to
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -93,8 +93,7 @@
     exe_extension = ".exe"
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-
-        UnixCCompiler.__init__(self, verbose, dry_run, force)
+        super(CygwinCCompiler, self).__init__(verbose, dry_run, force)
 
         status, details = check_config_h()
         logger.debug("Python's GCC status: %s (details: %s)", status, details)
@@ -234,12 +233,11 @@
         if not debug:
             extra_preargs.append("-s")
 
-        UnixCCompiler.link(self, target_desc, objects, output_filename,
-                           output_dir, libraries, library_dirs,
-                           runtime_library_dirs,
-                           None, # export_symbols, we do this in our def-file
-                           debug, extra_preargs, extra_postargs, build_temp,
-                           target_lang)
+        super(CygwinCCompiler, self).link(
+            target_desc, objects, output_filename, output_dir, libraries,
+            library_dirs, runtime_library_dirs,
+            None, # export_symbols, we do this in our def-file
+            debug, extra_preargs, extra_postargs, build_temp, target_lang)
 
     # -- Miscellaneous methods -----------------------------------------
 
@@ -255,14 +253,14 @@
             if ext not in (self.src_extensions + ['.rc','.res']):
                 raise UnknownFileError("unknown file type '%s' (from '%s')" % (ext, src_name))
             if strip_dir:
-                base = os.path.basename (base)
+                base = os.path.basename(base)
             if ext in ('.res', '.rc'):
                 # these need to be compiled to object files
-                obj_names.append (os.path.join(output_dir,
+                obj_names.append(os.path.join(output_dir,
                                               base + ext + self.obj_extension))
             else:
-                obj_names.append (os.path.join(output_dir,
-                                               base + self.obj_extension))
+                obj_names.append(os.path.join(output_dir,
+                                              base + self.obj_extension))
         return obj_names
 
 # the same as cygwin plus some additional parameters
@@ -273,8 +271,7 @@
     description = 'MinGW32 compiler'
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-
-        CygwinCCompiler.__init__ (self, verbose, dry_run, force)
+        super(Mingw32CCompiler, self).__init__(verbose, dry_run, force)
 
         # ld_version >= "2.13" support -shared so use it instead of
         # -mdll -static
diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py
--- a/distutils2/compiler/msvc9compiler.py
+++ b/distutils2/compiler/msvc9compiler.py
@@ -310,7 +310,7 @@
     exe_extension = '.exe'
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-        CCompiler.__init__(self, verbose, dry_run, force)
+        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
         self.__version = VERSION
         self.__root = r"Software\Microsoft\VisualStudio"
         # self.__macros = MACROS
diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py
--- a/distutils2/compiler/msvccompiler.py
+++ b/distutils2/compiler/msvccompiler.py
@@ -237,7 +237,7 @@
     exe_extension = '.exe'
 
     def __init__(self, verbose=0, dry_run=False, force=False):
-        CCompiler.__init__(self, verbose, dry_run, force)
+        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
         self.__version = get_build_version()
         self.__arch = get_build_architecture()
         if self.__arch == "Intel":
diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -24,13 +24,14 @@
     from docutils import frontend
     from docutils import nodes
 
-    class SilentReporter(Reporter):
+    class SilentReporter(Reporter, object):
 
         def __init__(self, source, report_level, halt_level, stream=None,
                      debug=0, encoding='ascii', error_handler='replace'):
             self.messages = []
-            Reporter.__init__(self, source, report_level, halt_level, stream,
-                              debug, encoding, error_handler)
+            super(SilentReporter, self).__init__(
+                source, report_level, halt_level, stream,
+                debug, encoding, error_handler)
 
         def system_message(self, level, message, *children, **kwargs):
             self.messages.append((level, message, children, kwargs))
@@ -185,6 +186,7 @@
 
 _FILESAFE = re.compile('[^A-Za-z0-9.]+')
 
+
 class Metadata(object):
     """The metadata of a release.
 
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -33,7 +33,6 @@
 import Queue
 import select
 import threading
-import SocketServer
 from BaseHTTPServer import HTTPServer
 from SimpleHTTPServer import SimpleHTTPRequestHandler
 from SimpleXMLRPCServer import SimpleXMLRPCServer
@@ -42,7 +41,6 @@
 from distutils2.compat import wraps
 
 
-
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
     os.path.dirname(os.path.abspath(__file__)), 'pypiserver')
 
@@ -105,7 +103,7 @@
         """
         # we want to launch the server in a new dedicated thread, to not freeze
         # tests.
-        threading.Thread.__init__(self)
+        super(PyPIServer, self).__init__()
         self._run = True
         self._serve_xmlrpc = serve_xmlrpc
         if static_filesystem_paths is None:
@@ -275,10 +273,10 @@
         self.wfile.write(data)
 
 
-class PyPIXMLRPCServer(SimpleXMLRPCServer):
+class PyPIXMLRPCServer(SimpleXMLRPCServer, object):
     def server_bind(self):
         """Override server_bind to store the server name."""
-        SocketServer.TCPServer.server_bind(self)
+        super(PyPIXMLRPCServer, self).server_bind()
         host, port = self.socket.getsockname()[:2]
         self.server_port = port
 
@@ -379,12 +377,13 @@
             'requires_python': self.requires_python,
             'classifiers': [],
             'name': self.name,
-            'licence': self.licence,
+            'licence': self.licence,  # XXX licence or license?
             'summary': self.summary,
             'home_page': self.homepage,
             'stable_version': self.stable_version,
-            'provides_dist': self.provides_dist or "%s (%s)" % (self.name,
-                                                              self.version),
+            # FIXME doesn't that reproduce the bug from 6527d3106e9f?
+            'provides_dist': (self.provides_dist or
+                             "%s (%s)" % (self.name, self.version)),
             'requires': self.requires,
             'cheesecake_installability_id': self.cheesecake_installability_id,
         }
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -69,11 +69,11 @@
 logger2to3 = logging.getLogger('RefactoringTool')
 
 
-class _TestHandler(logging.handlers.BufferingHandler):
+class _TestHandler(logging.handlers.BufferingHandler, object):
     # stolen and adapted from test.support
 
     def __init__(self):
-        logging.handlers.BufferingHandler.__init__(self, 0)
+        super(_TestHandler, self).__init__(0)
         self.setLevel(logging.DEBUG)
 
     def shouldFlush(self):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 15:24:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 15:24:07 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Improve_byte-compilation?=
 =?utf8?q?_to_be_independent_of_-O_or_-B=2E?=
Message-ID: <E1RPxSB-00072z-RL@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7c0a88497b5c
changeset:   1237:7c0a88497b5c
user:        ?ric Araujo <merwok at netwok.org>
date:        Sat Nov 12 02:58:14 2011 +0100
summary:
  Improve byte-compilation to be independent of -O or -B.

All code (util.byte_compile, build_py, install_lib) can now create .pyc
and/or.pyo files according to options given by users, without
interference from the calling Python?s own optimize mode or from the
sys.dont_write_bytecode switch.

The rationale is that packaging gives control over the creation of
.pyc/.pyo files to the user with its own explicit option, and the
behavior should not be changed if the calling Python happens to run with
-B or -O for whatever reason.

This is actually a bug fix, not an improvement: Digging into the early
history of distutils shows that the original author wanted this behavior
(see for example comments in build_py in r12940).

files:
  CHANGES.txt                                   |   2 +
  distutils2/command/build_py.py                |  67 +++-----
  distutils2/command/cmd.py                     |  20 ++-
  distutils2/command/install_lib.py             |  67 ++------
  distutils2/errors.py                          |   4 -
  distutils2/tests/support.py                   |   6 +-
  distutils2/tests/test_command_build_py.py     |  50 ++++--
  distutils2/tests/test_command_install_dist.py |  11 +-
  distutils2/tests/test_command_install_lib.py  |  78 +++++----
  distutils2/tests/test_mixin2to3.py            |   1 +
  distutils2/tests/test_util.py                 |  58 +++---
  distutils2/util.py                            |  25 +-
  12 files changed, 184 insertions(+), 205 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -149,6 +149,8 @@
   shlex not supporting unicode in 2.x, fix wrong shutil import [david, ?ric]
 - #13205: Fix and improve generated setup scripts [david, ?ric]
 - #11751: Improve test coverage for manifest [justin]
+- Byte compilation is now isolated from the calling Python -B or -O options
+  [?ric]
 
 
 1.0a3 - 2010-10-08
diff --git a/distutils2/command/build_py.py b/distutils2/command/build_py.py
--- a/distutils2/command/build_py.py
+++ b/distutils2/command/build_py.py
@@ -1,7 +1,6 @@
 """Build pure Python modules (just copy to build directory)."""
 
 import os
-import sys
 from glob import glob
 
 from distutils2 import logger
@@ -13,10 +12,14 @@
 # marking public APIs
 __all__ = ['build_py']
 
+
 class build_py(Command, Mixin2to3):
 
     description = "build pure Python modules (copy to build directory)"
 
+    # The options for controlling byte compilation are two independent sets;
+    # more info in install_lib or the reST docs
+
     user_options = [
         ('build-lib=', 'd', "directory to build (copy) to"),
         ('compile', 'c', "compile .py to .pyc"),
@@ -28,13 +31,14 @@
         ('use-2to3', None,
          "use 2to3 to make source python 3.x compatible"),
         ('convert-2to3-doctests', None,
-         "use 2to3 to convert doctests in seperate text files"),
+         "use 2to3 to convert doctests in separate text files"),
         ('use-2to3-fixers', None,
          "list additional fixers opted for during 2to3 conversion"),
         ]
 
     boolean_options = ['compile', 'force']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         self.build_lib = None
@@ -108,14 +112,15 @@
             self.run_2to3(self._updated_files, self._doctests_2to3,
                                             self.use_2to3_fixers)
 
-        self.byte_compile(self.get_outputs(include_bytecode=False))
+        self.byte_compile(self.get_outputs(include_bytecode=False),
+                          prefix=self.build_lib)
 
     # -- Top-level worker functions ------------------------------------
 
     def get_data_files(self):
         """Generate list of '(package,src_dir,build_dir,filenames)' tuples.
 
-        Helper function for `finalize_options()`.
+        Helper function for finalize_options.
         """
         data = []
         if not self.packages:
@@ -130,7 +135,7 @@
             # Length of path to strip from found files
             plen = 0
             if src_dir:
-                plen = len(src_dir)+1
+                plen = len(src_dir) + 1
 
             # Strip directory from globbed filenames
             filenames = [
@@ -142,7 +147,7 @@
     def find_data_files(self, package, src_dir):
         """Return filenames for package's data files in 'src_dir'.
 
-        Helper function for `get_data_files()`.
+        Helper function for get_data_files.
         """
         globs = (self.package_data.get('', [])
                  + self.package_data.get(package, []))
@@ -157,7 +162,7 @@
     def build_package_data(self):
         """Copy data files into build directory.
 
-        Helper function for `run()`.
+        Helper function for run.
         """
         # FIXME add tests for this method
         for package, src_dir, build_dir, filenames in self.data_files:
@@ -167,16 +172,17 @@
                 self.mkpath(os.path.dirname(target))
                 outf, copied = self.copy_file(srcfile,
                                target, preserve_mode=False)
-                if copied and srcfile in self.distribution.convert_2to3.doctests:
+                doctests = self.distribution.convert_2to3_doctests
+                if copied and srcfile in doctests:
                     self._doctests_2to3.append(outf)
 
     # XXX - this should be moved to the Distribution class as it is not
     # only needed for build_py. It also has no dependencies on this class.
     def get_package_dir(self, package):
         """Return the directory, relative to the top of the source
-           distribution, where package 'package' should be found
-           (at least according to the 'package_dir' option, if any)."""
-
+        distribution, where package 'package' should be found
+        (at least according to the 'package_dir' option, if any).
+        """
         path = package.split('.')
         if self.package_dir is not None:
             path.insert(0, self.package_dir)
@@ -187,8 +193,7 @@
         return ''
 
     def check_package(self, package, package_dir):
-        """Helper function for `find_package_modules()` and `find_modules()'.
-        """
+        """Helper function for find_package_modules and find_modules."""
         # Empty dir name means current directory, which we can probably
         # assume exists.  Also, os.path.exists and isdir don't know about
         # my "empty string means current dir" convention, so we have to
@@ -208,8 +213,8 @@
             if os.path.isfile(init_py):
                 return init_py
             else:
-                logger.warning(("package init file '%s' not found " +
-                                "(or not a regular file)"), init_py)
+                logger.warning("package init file %r not found "
+                               "(or not a regular file)", init_py)
 
         # Either not in a package at all (__init__.py not expected), or
         # __init__.py doesn't exist -- so don't return the filename.
@@ -217,7 +222,7 @@
 
     def check_module(self, module, module_file):
         if not os.path.isfile(module_file):
-            logger.warning("file %s (for module %s) not found",
+            logger.warning("file %r (for module %r) not found",
                            module_file, module)
             return False
         else:
@@ -238,7 +243,7 @@
                 module = os.path.splitext(os.path.basename(f))[0]
                 modules.append((package, module, f))
             else:
-                logger.debug("excluding %s", setup_script)
+                logger.debug("excluding %r", setup_script)
         return modules
 
     def find_modules(self):
@@ -331,7 +336,7 @@
             if include_bytecode:
                 if self.compile:
                     outputs.append(filename + "c")
-                if self.optimize > 0:
+                if self.optimize:
                     outputs.append(filename + "o")
 
         outputs += [
@@ -359,7 +364,6 @@
     def build_modules(self):
         modules = self.find_modules()
         for package, module, module_file in modules:
-
             # Now "build" the module -- ie. copy the source file to
             # self.build_lib (the build directory for Python source).
             # (Actually, it gets copied to the directory for this package
@@ -368,7 +372,6 @@
 
     def build_packages(self):
         for package in self.packages:
-
             # Get list of (package, module, module_file) tuples based on
             # scanning the package directory.  'package' is only included
             # in the tuple so that 'find_modules()' and
@@ -386,25 +389,3 @@
             for package_, module, module_file in modules:
                 assert package == package_
                 self.build_module(module, module_file, package)
-
-    def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode', False):
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
-        from distutils2.util import byte_compile  # FIXME use compileall
-        prefix = self.build_lib
-        if prefix[-1] != os.sep:
-            prefix = prefix + os.sep
-
-        # XXX this code is essentially the same as the 'byte_compile()
-        # method of the "install_lib" command, except for the determination
-        # of the 'prefix' string.  Hmmm.
-
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -10,7 +10,7 @@
 
 class Command(object):
     """Abstract base class for defining command classes, the "worker bees"
-    of the Packaging.  A useful analogy for command classes is to think of
+    of Packaging.  A useful analogy for command classes is to think of
     them as subroutines with local variables called "options".  The options
     are "declared" in 'initialize_options()' and "defined" (given their
     final values, aka "finalized") in 'finalize_options()', both of which
@@ -386,7 +386,6 @@
         if self.dry_run:
             return  # see if we want to display something
 
-
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
 
@@ -439,3 +438,20 @@
         # Otherwise, print the "skip" message
         else:
             logger.debug(skip_msg)
+
+    def byte_compile(self, files, prefix=None):
+        """Byte-compile files to pyc and/or pyo files.
+
+        This method requires that the calling class define compile and
+        optimize options, like build_py and install_lib.  It also
+        automatically respects the force and dry-run options.
+
+        prefix, if given, is a string that will be stripped off the
+        filenames encoded in bytecode files.
+        """
+        if self.compile:
+            util.byte_compile(files, optimize=False, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
+        if self.optimize:
+            util.byte_compile(files, optimize=self.optimize, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
diff --git a/distutils2/command/install_lib.py b/distutils2/command/install_lib.py
--- a/distutils2/command/install_lib.py
+++ b/distutils2/command/install_lib.py
@@ -1,8 +1,6 @@
 """Install all modules (extensions and pure Python)."""
 
 import os
-import sys
-import logging
 
 from distutils2 import logger
 from distutils2.command.cmd import Command
@@ -10,25 +8,18 @@
 
 
 # Extension for Python source files.
+# XXX dead code?  most of the codebase checks for literal '.py'
 if hasattr(os, 'extsep'):
     PYTHON_SOURCE_EXTENSION = os.extsep + "py"
 else:
     PYTHON_SOURCE_EXTENSION = ".py"
 
+
 class install_lib(Command):
 
     description = "install all modules (extensions and pure Python)"
 
-    # The byte-compilation options are a tad confusing.  Here are the
-    # possible scenarios:
-    #   1) no compilation at all (--no-compile --no-optimize)
-    #   2) compile .pyc only (--compile --no-optimize; default)
-    #   3) compile .pyc and "level 1" .pyo (--compile --optimize)
-    #   4) compile "level 1" .pyo only (--no-compile --optimize)
-    #   5) compile .pyc and "level 2" .pyo (--compile --optimize-more)
-    #   6) compile "level 2" .pyo only (--no-compile --optimize-more)
-    #
-    # The UI for this is two option, 'compile' and 'optimize'.
+    # The options for controlling byte compilation are two independent sets:
     # 'compile' is strictly boolean, and only decides whether to
     # generate .pyc files.  'optimize' is three-way (0, 1, or 2), and
     # decides both whether to generate .pyo files and what level of
@@ -36,7 +27,7 @@
 
     user_options = [
         ('install-dir=', 'd', "directory to install to"),
-        ('build-dir=','b', "build directory (where to install from)"),
+        ('build-dir=', 'b', "build directory (where to install from)"),
         ('force', 'f', "force installation (overwrite existing files)"),
         ('compile', 'c', "compile .py to .pyc [default]"),
         ('no-compile', None, "don't compile .py files"),
@@ -47,7 +38,8 @@
         ]
 
     boolean_options = ['force', 'compile', 'skip-build']
-    negative_opt = {'no-compile' : 'compile'}
+
+    negative_opt = {'no-compile': 'compile'}
 
     def initialize_options(self):
         # let the 'install_dist' command dictate our installation directory
@@ -65,7 +57,8 @@
         self.set_undefined_options('install_dist',
                                    ('build_lib', 'build_dir'),
                                    ('install_lib', 'install_dir'),
-                                   'force', 'compile', 'optimize', 'skip_build')
+                                   'force', 'compile', 'optimize',
+                                   'skip_build')
 
         if self.compile is None:
             self.compile = True
@@ -89,9 +82,14 @@
         # having a build directory!)
         outfiles = self.install()
 
-        # (Optionally) compile .py to .pyc
+        # (Optionally) compile .py to .pyc and/or .pyo
         if outfiles is not None and self.distribution.has_pure_modules():
-            self.byte_compile(outfiles)
+            # XXX comment from distutils: "This [prefix stripping] is far from
+            # complete, but it should at least generate usable bytecode in RPM
+            # distributions." -> need to find exact requirements for
+            # byte-compiled files and fix it
+            install_root = self.get_finalized_command('install_dist').root
+            self.byte_compile(outfiles, prefix=install_root)
 
     # -- Top-level worker functions ------------------------------------
     # (called from 'run()')
@@ -113,38 +111,6 @@
             return
         return outfiles
 
-    def byte_compile(self, files):
-        if getattr(sys, 'dont_write_bytecode', False):
-            # XXX do we want this?  because a Python runs without bytecode
-            # doesn't mean that the *dists should not contain bytecode
-            #--or does it?
-            logger.warning('%s: byte-compiling is disabled, skipping.',
-                           self.get_command_name())
-            return
-
-        from distutils2.util import byte_compile  # FIXME use compileall
-
-        # Get the "--root" directory supplied to the "install_dist" command,
-        # and use it as a prefix to strip off the purported filename
-        # encoded in bytecode files.  This is far from complete, but it
-        # should at least generate usable bytecode in RPM distributions.
-        install_root = self.get_finalized_command('install_dist').root
-
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=install_root,
-                         dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=install_root,
-                         verbose=verbose,
-                         dry_run=self.dry_run)
-
-
     # -- Utility methods -----------------------------------------------
 
     def _mutate_outputs(self, has_any, build_cmd, cmd_option, output_dir):
@@ -173,12 +139,11 @@
                 continue
             if self.compile:
                 bytecode_files.append(py_file + "c")
-            if self.optimize > 0:
+            if self.optimize:
                 bytecode_files.append(py_file + "o")
 
         return bytecode_files
 
-
     # -- External interface --------------------------------------------
     # (called by outsiders)
 
diff --git a/distutils2/errors.py b/distutils2/errors.py
--- a/distutils2/errors.py
+++ b/distutils2/errors.py
@@ -72,10 +72,6 @@
     """Syntax error in a file list template."""
 
 
-class PackagingByteCompileError(PackagingError):
-    """Byte compile error."""
-
-
 class PackagingPyPIError(PackagingError):
     """Any problem occuring during using the indexes."""
 
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -61,7 +61,7 @@
     # misc. functions and decorators
     'fake_dec', 'create_distribution', 'copy_xxmodule_c', 'fixup_build_ext',
     # imported from this module for backport purposes
-    'unittest', 'requires_zlib', 'skip_unless_symlink',
+    'unittest', 'requires_zlib', 'skip_2to3_optimize', 'skip_unless_symlink',
 ]
 
 
@@ -362,6 +362,10 @@
         'requires test.test_support.skip_unless_symlink')
 
 
+skip_2to3_optimize = unittest.skipUnless(__debug__,
+                                         "2to3 doesn't work under -O")
+
+
 requires_zlib = unittest.skipUnless(zlib, 'requires zlib')
 
 
diff --git a/distutils2/tests/test_command_build_py.py b/distutils2/tests/test_command_build_py.py
--- a/distutils2/tests/test_command_build_py.py
+++ b/distutils2/tests/test_command_build_py.py
@@ -54,17 +54,12 @@
         # This makes sure the list of outputs includes byte-compiled
         # files for Python modules but not for package data files
         # (there shouldn't *be* byte-code files for those!).
-        #
         self.assertEqual(len(cmd.get_outputs()), 3)
         pkgdest = os.path.join(destination, "pkg")
         files = os.listdir(pkgdest)
         self.assertIn("__init__.py", files)
         self.assertIn("README.txt", files)
-        # XXX even with -O, distutils writes pyc, not pyo; bug?
-        if getattr(sys , 'dont_write_bytecode', False):
-            self.assertNotIn("__init__.pyc", files)
-        else:
-            self.assertIn("__init__.pyc", files)
+        self.assertIn("__init__.pyc", files)
 
     def test_empty_package_dir(self):
         # See SF 1668596/1720897.
@@ -99,23 +94,44 @@
             os.chdir(cwd)
             sys.stdout = old_stdout
 
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
+    def test_byte_compile(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
+        cmd = build_py(dist)
+        cmd.compile = True
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
+
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found), ['boiledeggs.py', 'boiledeggs.pyc'])
+
+    def test_byte_compile_optimized(self):
+        project_dir, dist = self.create_dist(py_modules=['boiledeggs'])
+        os.chdir(project_dir)
+        self.write_file('boiledeggs.py', 'import antigravity')
         cmd = build_py(dist)
         cmd.compile = True
         cmd.optimize = 1
+        cmd.build_lib = 'here'
+        cmd.finalize_options()
+        cmd.run()
 
-        old_dont_write_bytecode = sys.dont_write_bytecode
+        found = os.listdir(cmd.build_lib)
+        self.assertEqual(sorted(found),
+                         ['boiledeggs.py', 'boiledeggs.pyc', 'boiledeggs.pyo'])
+
+    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
+                         'sys.dont_write_bytecode not supported')
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            cmd.byte_compile([])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
+        self.test_byte_compile()
+        self.test_byte_compile_optimized()
 
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
 
 def test_suite():
     return unittest.makeSuite(BuildPyTestCase)
diff --git a/distutils2/tests/test_command_install_dist.py b/distutils2/tests/test_command_install_dist.py
--- a/distutils2/tests/test_command_install_dist.py
+++ b/distutils2/tests/test_command_install_dist.py
@@ -182,9 +182,11 @@
     def test_old_record(self):
         # test pre-PEP 376 --record option (outside dist-info dir)
         install_dir = self.mkdtemp()
-        project_dir, dist = self.create_dist(scripts=['hello'])
+        project_dir, dist = self.create_dist(py_modules=['hello'],
+                                             scripts=['sayhi'])
         os.chdir(project_dir)
-        self.write_file('hello', "print 'o hai'")
+        self.write_file('hello.py', "def main(): print 'o hai'")
+        self.write_file('sayhi', 'from hello import main; main()')
 
         cmd = install_dist(dist)
         dist.command_obj['install_dist'] = cmd
@@ -200,8 +202,9 @@
             f.close()
 
         found = [os.path.basename(line) for line in content.splitlines()]
-        expected = ['hello', 'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
-        self.assertEqual(found, expected)
+        expected = ['hello.py', 'hello.pyc', 'sayhi',
+                    'METADATA', 'INSTALLER', 'REQUESTED', 'RECORD']
+        self.assertEqual(sorted(found), sorted(expected))
 
         # XXX test that fancy_getopt is okay with options named
         # record and no-record but unrelated
diff --git a/distutils2/tests/test_command_install_lib.py b/distutils2/tests/test_command_install_lib.py
--- a/distutils2/tests/test_command_install_lib.py
+++ b/distutils2/tests/test_command_install_lib.py
@@ -1,6 +1,6 @@
 """Tests for distutils2.command.install_data."""
+import os
 import sys
-import os
 
 from distutils2.tests import unittest, support
 from distutils2.command.install_lib import install_lib
@@ -16,7 +16,7 @@
     restore_environ = ['PYTHONPATH']
 
     def test_finalize_options(self):
-        pkg_dir, dist = self.create_dist()
+        dist = self.create_dist()[1]
         cmd = install_lib(dist)
 
         cmd.finalize_options()
@@ -33,71 +33,73 @@
         cmd.finalize_options()
         self.assertEqual(cmd.optimize, 2)
 
-    @unittest.skipIf(getattr(sys, 'dont_write_bytecode', False),
-                     'byte-compile disabled')
     def test_byte_compile(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
         cmd = install_lib(dist)
         cmd.compile = True
         cmd.optimize = 1
 
-        f = os.path.join(pkg_dir, 'foo.py')
+        f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        self.assertTrue(os.path.exists(os.path.join(pkg_dir, 'foo.pyc')))
-        self.assertTrue(os.path.exists(os.path.join(pkg_dir, 'foo.pyo')))
+        self.assertTrue(os.path.exists(f + 'c'))
+        self.assertTrue(os.path.exists(f + 'o'))
+
+    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
+                         'sys.dont_write_bytecode not supported')
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
+        sys.dont_write_bytecode = True
+        self.test_byte_compile()
 
     def test_get_outputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # make sure the build_lib is set the temp dir
-        build_dir = os.path.split(pkg_dir)[0]
+        # make sure the build_lib is set the temp dir  # XXX what?  this is not
+        # needed in the same distutils test and should work without manual
+        # intervention
+        build_dir = os.path.split(project_dir)[0]
         cmd.get_finalized_command('build_py').build_lib = build_dir
 
-        # get_output should return 4 elements
-        self.assertEqual(len(cmd.get_outputs()), 4)
+        # get_outputs should return 4 elements: spam/__init__.py, .pyc and
+        # .pyo, foo.so / foo.pyd
+        outputs = cmd.get_outputs()
+        self.assertEqual(len(outputs), 4, outputs)
 
     def test_get_inputs(self):
-        pkg_dir, dist = self.create_dist()
+        project_dir, dist = self.create_dist()
+        os.chdir(project_dir)
+        os.mkdir('spam')
         cmd = install_lib(dist)
 
         # setting up a dist environment
         cmd.compile = True
         cmd.optimize = 1
-        cmd.install_dir = pkg_dir
-        f = os.path.join(pkg_dir, '__init__.py')
+        cmd.install_dir = self.mkdtemp()
+        f = os.path.join(project_dir, 'spam', '__init__.py')
         self.write_file(f, '# python package')
         cmd.distribution.ext_modules = [Extension('foo', ['xxx'])]
-        cmd.distribution.packages = [pkg_dir]
+        cmd.distribution.packages = ['spam']
 
-        # get_input should return 2 elements
-        self.assertEqual(len(cmd.get_inputs()), 2)
-
-    @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
-                         'sys.dont_write_bytecode not supported')
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile is not used
-        pkg_dir, dist = self.create_dist()
-        cmd = install_lib(dist)
-        cmd.compile = True
-        cmd.optimize = 1
-
-        self.addCleanup(setattr, sys, 'dont_write_bytecode',
-                        sys.dont_write_bytecode)
-        sys.dont_write_bytecode = True
-        cmd.byte_compile([])
-
-        self.assertIn('byte-compiling is disabled', self.get_logs()[0])
+        # get_inputs should return 2 elements: spam/__init__.py and
+        # foo.so / foo.pyd
+        inputs = cmd.get_inputs()
+        self.assertEqual(len(inputs), 2, inputs)
 
 
 def test_suite():
diff --git a/distutils2/tests/test_mixin2to3.py b/distutils2/tests/test_mixin2to3.py
--- a/distutils2/tests/test_mixin2to3.py
+++ b/distutils2/tests/test_mixin2to3.py
@@ -10,6 +10,7 @@
                         unittest.TestCase):
 
     @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
+    @support.skip_2to3_optimize
     def test_convert_code_only(self):
         # used to check if code gets converted properly.
         code = "print 'test'"
diff --git a/distutils2/tests/test_util.py b/distutils2/tests/test_util.py
--- a/distutils2/tests/test_util.py
+++ b/distutils2/tests/test_util.py
@@ -9,7 +9,7 @@
 from StringIO import StringIO
 
 from distutils2.errors import (
-    PackagingPlatformError, PackagingByteCompileError, PackagingFileError,
+    PackagingPlatformError, PackagingFileError,
     PackagingExecError, InstallationException)
 from distutils2 import util
 from distutils2.dist import Distribution
@@ -137,15 +137,8 @@
             self._uname = None
         os.uname = self._get_uname
 
-        # patching POpen
-        self.old_find_executable = util.find_executable
-        util.find_executable = self._find_executable
-        self._exes = {}
-        self.old_popen = subprocess.Popen
-        self.old_stdout = sys.stdout
-        self.old_stderr = sys.stderr
-        FakePopen.test_class = self
-        subprocess.Popen = FakePopen
+    def _get_uname(self):
+        return self._uname
 
     def tearDown(self):
         # getting back the environment
@@ -160,17 +153,24 @@
             os.uname = self.uname
         else:
             del os.uname
+        super(UtilTestCase, self).tearDown()
+
+    def mock_popen(self):
+        self.old_find_executable = util.find_executable
+        util.find_executable = self._find_executable
+        self._exes = {}
+        self.old_popen = subprocess.Popen
+        self.old_stdout = sys.stdout
+        self.old_stderr = sys.stderr
+        FakePopen.test_class = self
+        subprocess.Popen = FakePopen
+        self.addCleanup(self.unmock_popen)
+
+    def unmock_popen(self):
         util.find_executable = self.old_find_executable
         subprocess.Popen = self.old_popen
-        sys.old_stdout = self.old_stdout
-        sys.old_stderr = self.old_stderr
-        super(UtilTestCase, self).tearDown()
-
-    def _set_uname(self, uname):
-        self._uname = uname
-
-    def _get_uname(self):
-        return self._uname
+        sys.stdout = self.old_stdout
+        sys.stderr = self.old_stderr
 
     def test_convert_path(self):
         # linux/mac
@@ -282,6 +282,7 @@
         return None
 
     def test_get_compiler_versions(self):
+        self.mock_popen()
         # get_versions calls distutils.spawn.find_executable on
         # 'gcc', 'ld' and 'dllwrap'
         self.assertEqual(get_compiler_versions(), (None, None, None))
@@ -324,15 +325,12 @@
 
     @unittest.skipUnless(hasattr(sys, 'dont_write_bytecode'),
                          'sys.dont_write_bytecode not supported')
-    def test_dont_write_bytecode(self):
-        # makes sure byte_compile raise a PackagingError
-        # if sys.dont_write_bytecode is True
-        old_dont_write_bytecode = sys.dont_write_bytecode
+    def test_byte_compile_under_B(self):
+        # make sure byte compilation works under -B (dont_write_bytecode)
+        self.addCleanup(setattr, sys, 'dont_write_bytecode',
+                        sys.dont_write_bytecode)
         sys.dont_write_bytecode = True
-        try:
-            self.assertRaises(PackagingByteCompileError, byte_compile, [])
-        finally:
-            sys.dont_write_bytecode = old_dont_write_bytecode
+        byte_compile([])
 
     def test_newer(self):
         self.assertRaises(PackagingFileError, util.newer, 'xxx', 'xxx')
@@ -420,6 +418,7 @@
         self.assertRaises(ImportError, resolve_name, 'a.b.c.Spam')
 
     @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
+    @support.skip_2to3_optimize
     def test_run_2to3_on_code(self):
         content = "print 'test'"
         converted_content = "print('test')"
@@ -434,6 +433,7 @@
         self.assertEqual(new_content, converted_content)
 
     @unittest.skipIf(sys.version < '2.6', 'requires Python 2.6 or higher')
+    @support.skip_2to3_optimize
     def test_run_2to3_on_doctests(self):
         # to check if text files containing doctests only get converted.
         content = ">>> print 'test'\ntest\n"
@@ -451,8 +451,6 @@
     @unittest.skipUnless(os.name in ('nt', 'posix'),
                          'runs only under posix or nt')
     def test_spawn(self):
-        # no patching of Popen here
-        subprocess.Popen = self.old_popen
         tmpdir = self.mkdtemp()
 
         # creating something executable
@@ -549,8 +547,6 @@
         self.assertEqual(args['py_modules'], dist.py_modules)
 
     def test_generate_setup_py(self):
-        # undo subprocess.Popen monkey-patching before using assert_python_*
-        subprocess.Popen = self.old_popen
         os.chdir(self.mkdtemp())
         self.write_file('setup.cfg', textwrap.dedent("""\
             [metadata]
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -24,8 +24,8 @@
 
 from distutils2 import logger
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
-                               PackagingByteCompileError, PackagingExecError,
-                               InstallationException, PackagingInternalError)
+                               PackagingExecError, InstallationException,
+                               PackagingInternalError)
 from distutils2._backport import sysconfig
 
 __all__ = [
@@ -301,7 +301,7 @@
 
 
 def byte_compile(py_files, optimize=0, force=False, prefix=None,
-                 base_dir=None, verbose=0, dry_run=False, direct=None):
+                 base_dir=None, dry_run=False, direct=None):
     """Byte-compile a collection of Python source files to either .pyc
     or .pyo files in the same directory.
 
@@ -310,6 +310,9 @@
       0 - don't optimize (generate .pyc)
       1 - normal optimization (like "python -O")
       2 - extra optimization (like "python -OO")
+    This function is independent from the running Python's -O or -B options;
+    it is fully controlled by the parameters passed in.
+
     If 'force' is true, all files are recompiled regardless of
     timestamps.
 
@@ -331,10 +334,7 @@
     generated in indirect mode; unless you know what you're doing, leave
     it set to None.
     """
-    # nothing is done if sys.dont_write_bytecode is True
-    # FIXME this should not raise an error
-    if getattr(sys, 'dont_write_bytecode', False):
-        raise PackagingByteCompileError('byte-compiling is disabled.')
+    # FIXME use compileall + remove direct/indirect shenanigans
 
     # First, if the caller didn't force us into direct or indirect mode,
     # figure out which mode we should be in.  We take a conservative
@@ -388,17 +388,13 @@
                 script.write("""
 byte_compile(files, optimize=%r, force=%r,
              prefix=%r, base_dir=%r,
-             verbose=%r, dry_run=False,
+             dry_run=False,
              direct=True)
-""" % (optimize, force, prefix, base_dir, verbose))
+""" % (optimize, force, prefix, base_dir))
             finally:
                 script.close()
 
         cmd = [sys.executable, script_name]
-        if optimize == 1:
-            cmd.insert(1, "-O")
-        elif optimize == 2:
-            cmd.insert(1, "-OO")
 
         env = os.environ.copy()
         env['PYTHONPATH'] = os.path.pathsep.join(sys.path)
@@ -424,8 +420,9 @@
             # Terminology from the py_compile module:
             #   cfile - byte-compiled file
             #   dfile - purported source filename (same as 'file' by default)
-            cfile = file + (__debug__ and "c" or "o")
+            cfile = file + (optimize and 'o' or 'c')
             dfile = file
+
             if prefix:
                 if file[:len(prefix)] != prefix:
                     raise ValueError("invalid prefix: filename %r doesn't "

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 16:08:02 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 16:08:02 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Replace_nick_with_full_n?=
	=?utf8?q?ame?=
Message-ID: <E1RPy8g-0002zK-S9@dinsdale.python.org>

http://hg.python.org/distutils2/rev/67311c507b93
changeset:   1249:67311c507b93
parent:      1247:259745e16c33
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 15:57:32 2011 +0100
summary:
  Replace nick with full name

files:
  CONTRIBUTORS.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -32,7 +32,6 @@
 - Hallvard B Furuseth
 - Yannick Gingras
 - Filip Gruszczy?ski
-- guillermoo
 - Walker Hale IV
 - Alexandre Hamelin
 - Kelsey Hightower
@@ -46,6 +45,7 @@
 - Alain Leufroy
 - Martin von L?wis
 - Hugo Lopes Tavares
+- Guillermo L?pez-Anglada
 - Justin Love
 - Simon Mathieu
 - Carl Meyer

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 16:58:02 2011
From: python-checkins at python.org (barry.warsaw)
Date: Mon, 14 Nov 2011 16:58:02 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Add_some_additional_comments_b?=
 =?utf8?q?ased_on_feedback_from_Antoine_and_Victor=2E?=
Message-ID: <E1RPyv4-0008TA-Pj@dinsdale.python.org>

http://hg.python.org/peps/rev/430feff13d2a
changeset:   3988:430feff13d2a
user:        Barry Warsaw <barry at python.org>
date:        Mon Nov 14 10:58:00 2011 -0500
summary:
  Add some additional comments based on feedback from Antoine and Victor.

files:
  pep-0404.txt |  19 +++++++++++++++++--
  1 files changed, 17 insertions(+), 2 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -95,7 +95,10 @@
 bytes type, but critically, no automatic coercion between bytes and
 unicodes is provided.  Thus, the core interpreter, its I/O libraries,
 module names, etc. are clear in their distinction between unicode
-strings and bytes.  This clarity is often a source of difficulty in
+strings and bytes.  Python 3's unicode support even extends to the
+filesystem, so that non-ASCII file names are natively supported.
+
+This string/bytes clarity is often a source of difficulty in
 transitioning existing code to Python 3, because many third party
 libraries and applications are themselves ambiguous in this
 distinction.  Once migrated though, most `UnicodeError`s can be
@@ -114,6 +117,18 @@
 non-integer results.
 
 
+Classes
+-------
+
+Python 2 has two core class hierarchies, often called *classic
+classes* and *new-style classes*.  The latter allow for such things as
+inheriting from the built-in basic types.  However, confusion and
+inconsistencies between the two class types has led Python 3 to drop
+classic classes.  Now all classes in Python 3 are *new-style*
+(although that's a misnomer now).  There is no need to inherit from
+``object`` or set the default metatype to enable them.
+
+
 Multiple spellings
 ------------------
 
@@ -128,7 +143,7 @@
 -------
 
 In Python 3, star imports (e.g. ``from x import *``) are only
-premitted in module level code.  Also, only absolute imports are
+permitted in module level code.  Also, only absolute imports are
 supported.
 
 Also, some areas of the standard library have been reorganized to make

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon Nov 14 17:29:01 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 17:29:01 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_The_error_message_should?=
 =?utf8?q?_contain_the_key_as_given=2C_not_normalized=2E?=
Message-ID: <E1RPzP3-0003Vp-GS@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7eb32e910b85
changeset:   1250:7eb32e910b85
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 17:06:10 2011 +0100
summary:
  The error message should contain the key as given, not normalized.

Backout of 8f69359881aa.  Reported by Jeremy Kloth.

files:
  distutils2/metadata.py |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -230,8 +230,10 @@
 
     def __delitem__(self, name):
         field_name = self._convert_name(name)
-        # we let a KeyError propagate
-        del self._fields[field_name]
+        try:
+            del self._fields[field_name]
+        except KeyError:
+            raise KeyError(name)
         self._set_best_version()
 
     def __contains__(self, name):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 17:29:01 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 14 Nov 2011 17:29:01 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
	=?utf8?q?n3=29=3A_Merge_default?=
Message-ID: <E1RPzP3-0003Vz-HY@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7fe2f04a24cc
changeset:   1251:7fe2f04a24cc
branch:      python3
parent:      1248:1c4fe27018fd
parent:      1250:7eb32e910b85
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 17:25:52 2011 +0100
summary:
  Merge default

files:
  CONTRIBUTORS.txt       |  2 +-
  distutils2/metadata.py |  6 ++++--
  2 files changed, 5 insertions(+), 3 deletions(-)


diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -32,7 +32,6 @@
 - Hallvard B Furuseth
 - Yannick Gingras
 - Filip Gruszczy?ski
-- guillermoo
 - Walker Hale IV
 - Alexandre Hamelin
 - Kelsey Hightower
@@ -46,6 +45,7 @@
 - Alain Leufroy
 - Martin von L?wis
 - Hugo Lopes Tavares
+- Guillermo L?pez-Anglada
 - Justin Love
 - Simon Mathieu
 - Carl Meyer
diff --git a/distutils2/metadata.py b/distutils2/metadata.py
--- a/distutils2/metadata.py
+++ b/distutils2/metadata.py
@@ -229,8 +229,10 @@
 
     def __delitem__(self, name):
         field_name = self._convert_name(name)
-        # we let a KeyError propagate
-        del self._fields[field_name]
+        try:
+            del self._fields[field_name]
+        except KeyError:
+            raise KeyError(name)
         self._set_best_version()
 
     def __contains__(self, name):

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 14 17:57:02 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 14 Nov 2011 17:57:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Group_tests_abo?=
 =?utf8?q?ut_attributes_in_a_separate_class=2E?=
Message-ID: <E1RPzqA-0006YS-Np@dinsdale.python.org>

http://hg.python.org/cpython/rev/da8212a27e3c
changeset:   73544:da8212a27e3c
branch:      2.7
parent:      73541:87ecfd5cd5d1
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Tue Nov 08 02:07:18 2011 +0200
summary:
  Group tests about attributes in a separate class.

files:
  Lib/test/test_htmlparser.py |  120 +++++++++++------------
  1 files changed, 59 insertions(+), 61 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -181,60 +181,6 @@
             ("data", "this < text > contains < bare>pointy< brackets"),
             ])
 
-    def test_attr_syntax(self):
-        output = [
-          ("starttag", "a", [("b", "v"), ("c", "v"), ("d", "v"), ("e", None)])
-          ]
-        self._run_check("""<a b='v' c="v" d=v e>""", output)
-        self._run_check("""<a  b = 'v' c = "v" d = v e>""", output)
-        self._run_check("""<a\nb\n=\n'v'\nc\n=\n"v"\nd\n=\nv\ne>""", output)
-        self._run_check("""<a\tb\t=\t'v'\tc\t=\t"v"\td\t=\tv\te>""", output)
-
-    def test_attr_values(self):
-        self._run_check("""<a b='xxx\n\txxx' c="yyy\t\nyyy" d='\txyz\n'>""",
-                        [("starttag", "a", [("b", "xxx\n\txxx"),
-                                            ("c", "yyy\t\nyyy"),
-                                            ("d", "\txyz\n")])
-                         ])
-        self._run_check("""<a b='' c="">""", [
-            ("starttag", "a", [("b", ""), ("c", "")]),
-            ])
-        # Regression test for SF patch #669683.
-        self._run_check("<e a=rgb(1,2,3)>", [
-            ("starttag", "e", [("a", "rgb(1,2,3)")]),
-            ])
-        # Regression test for SF bug #921657.
-        self._run_check("<a href=mailto:xyz at example.com>", [
-            ("starttag", "a", [("href", "mailto:xyz at example.com")]),
-            ])
-
-    def test_attr_nonascii(self):
-        # see issue 7311
-        self._run_check(u"<img src=/foo/bar.png alt=\u4e2d\u6587>", [
-            ("starttag", "img", [("src", "/foo/bar.png"),
-                                 ("alt", u"\u4e2d\u6587")]),
-            ])
-        self._run_check(u"<a title='\u30c6\u30b9\u30c8' "
-                        u"href='\u30c6\u30b9\u30c8.html'>", [
-            ("starttag", "a", [("title", u"\u30c6\u30b9\u30c8"),
-                               ("href", u"\u30c6\u30b9\u30c8.html")]),
-            ])
-        self._run_check(u'<a title="\u30c6\u30b9\u30c8" '
-                        u'href="\u30c6\u30b9\u30c8.html">', [
-            ("starttag", "a", [("title", u"\u30c6\u30b9\u30c8"),
-                               ("href", u"\u30c6\u30b9\u30c8.html")]),
-            ])
-
-    def test_attr_entity_replacement(self):
-        self._run_check("""<a b='&amp;&gt;&lt;&quot;&apos;'>""", [
-            ("starttag", "a", [("b", "&><\"'")]),
-            ])
-
-    def test_attr_funky_names(self):
-        self._run_check("""<a a.b='v' c:d=v e-f=v>""", [
-            ("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")]),
-            ])
-
     def test_illegal_declarations(self):
         self._parse_error('<!spacer type="block" height="25">')
 
@@ -342,12 +288,6 @@
                                     ("data", content),
                                     ("endtag", element_lower)])
 
-
-    def test_entityrefs_in_attributes(self):
-        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>", [
-                ("starttag", "html", [("foo", u"\u20AC&aa&unsupported;")])
-                ])
-
     def test_malformatted_charref(self):
         self._run_check("<p>&#bad;</p>", [
             ("starttag", "p", []),
@@ -361,8 +301,66 @@
         self.assertEqual(parser.unescape('&#0038;'),'&')
 
 
+
+class AttributesTestCase(TestCaseBase):
+
+    def test_attr_syntax(self):
+        output = [
+          ("starttag", "a", [("b", "v"), ("c", "v"), ("d", "v"), ("e", None)])
+        ]
+        self._run_check("""<a b='v' c="v" d=v e>""", output)
+        self._run_check("""<a  b = 'v' c = "v" d = v e>""", output)
+        self._run_check("""<a\nb\n=\n'v'\nc\n=\n"v"\nd\n=\nv\ne>""", output)
+        self._run_check("""<a\tb\t=\t'v'\tc\t=\t"v"\td\t=\tv\te>""", output)
+
+    def test_attr_values(self):
+        self._run_check("""<a b='xxx\n\txxx' c="yyy\t\nyyy" d='\txyz\n'>""",
+                        [("starttag", "a", [("b", "xxx\n\txxx"),
+                                            ("c", "yyy\t\nyyy"),
+                                            ("d", "\txyz\n")])])
+        self._run_check("""<a b='' c="">""",
+                        [("starttag", "a", [("b", ""), ("c", "")])])
+        # Regression test for SF patch #669683.
+        self._run_check("<e a=rgb(1,2,3)>",
+                        [("starttag", "e", [("a", "rgb(1,2,3)")])])
+        # Regression test for SF bug #921657.
+        self._run_check(
+            "<a href=mailto:xyz at example.com>",
+            [("starttag", "a", [("href", "mailto:xyz at example.com")])])
+
+    def test_attr_nonascii(self):
+        # see issue 7311
+        self._run_check(
+            u"<img src=/foo/bar.png alt=\u4e2d\u6587>",
+            [("starttag", "img", [("src", "/foo/bar.png"),
+                                  ("alt", u"\u4e2d\u6587")])])
+        self._run_check(
+            u"<a title='\u30c6\u30b9\u30c8' href='\u30c6\u30b9\u30c8.html'>",
+            [("starttag", "a", [("title", u"\u30c6\u30b9\u30c8"),
+                                ("href", u"\u30c6\u30b9\u30c8.html")])])
+        self._run_check(
+            u'<a title="\u30c6\u30b9\u30c8" href="\u30c6\u30b9\u30c8.html">',
+            [("starttag", "a", [("title", u"\u30c6\u30b9\u30c8"),
+                                ("href", u"\u30c6\u30b9\u30c8.html")])])
+
+    def test_attr_entity_replacement(self):
+        self._run_check(
+            "<a b='&amp;&gt;&lt;&quot;&apos;'>",
+            [("starttag", "a", [("b", "&><\"'")])])
+
+    def test_attr_funky_names(self):
+        self._run_check(
+            "<a a.b='v' c:d=v e-f=v>",
+            [("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")])])
+
+
+    def test_entityrefs_in_attributes(self):
+        self._run_check(
+            "<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
+            [("starttag", "html", [("foo", u"\u20AC&aa&unsupported;")])])
+
 def test_main():
-    test_support.run_unittest(HTMLParserTestCase)
+    test_support.run_unittest(HTMLParserTestCase, AttributesTestCase)
 
 
 if __name__ == "__main__":

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 17:57:03 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 14 Nov 2011 17:57:03 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzE3NDU3NjEsICM3?=
 =?utf8?q?55670=2C_=2313357=2C_=2312629=2C_=231200313=3A_improve_attribute?=
 =?utf8?q?_handling_in?=
Message-ID: <E1RPzqB-0006YX-Jm@dinsdale.python.org>

http://hg.python.org/cpython/rev/3c3009f63700
changeset:   73545:3c3009f63700
branch:      2.7
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Mon Nov 14 18:04:05 2011 +0200
summary:
  #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser.

files:
  Lib/HTMLParser.py           |  20 +++--
  Lib/test/test_htmlparser.py |  76 ++++++++++++++++++++++++-
  Misc/NEWS                   |   3 +
  3 files changed, 87 insertions(+), 12 deletions(-)


diff --git a/Lib/HTMLParser.py b/Lib/HTMLParser.py
--- a/Lib/HTMLParser.py
+++ b/Lib/HTMLParser.py
@@ -24,22 +24,23 @@
 piclose = re.compile('>')
 commentclose = re.compile(r'--\s*>')
 tagfind = re.compile('[a-zA-Z][-.a-zA-Z0-9:_]*')
+
 attrfind = re.compile(
-    r'\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
-    r'(\'[^\']*\'|"[^"]*"|[^\s"\'=<>`]*))?')
+    r'\s*((?<=[\'"\s])[^\s/>][^\s/=>]*)(\s*=+\s*'
+    r'(\'[^\']*\'|"[^"]*"|(?![\'"])[^>\s]*))?')
 
 locatestarttagend = re.compile(r"""
   <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
   (?:\s+                             # whitespace before attribute name
-    (?:[a-zA-Z_][-.:a-zA-Z0-9_]*     # attribute name
-      (?:\s*=\s*                     # value indicator
+    (?:(?<=['"\s])[^\s/>][^\s/=>]*   # attribute name
+      (?:\s*=+\s*                    # value indicator
         (?:'[^']*'                   # LITA-enclosed value
-          |\"[^\"]*\"                # LIT-enclosed value
-          |[^'\">\s]+                # bare value
+          |"[^"]*"                   # LIT-enclosed value
+          |(?!['"])[^>\s]*           # bare value
          )
-       )?
-     )
-   )*
+       )?\s*
+     )*
+   )?
   \s*                                # trailing whitespace
 """, re.VERBOSE)
 endendtag = re.compile('>')
@@ -254,6 +255,7 @@
             elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
                  attrvalue[:1] == '"' == attrvalue[-1:]:
                 attrvalue = attrvalue[1:-1]
+            if attrvalue:
                 attrvalue = self.unescape(attrvalue)
             attrs.append((attrname.lower(), attrvalue))
             k = m.end()
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -226,13 +226,11 @@
         self._parse_error("<a<a>")
         self._parse_error("</a<a>")
         self._parse_error("<!")
-        self._parse_error("<a $>")
         self._parse_error("<a")
         self._parse_error("<a foo='bar'")
         self._parse_error("<a foo='bar")
         self._parse_error("<a foo='>'")
         self._parse_error("<a foo='>")
-        self._parse_error("<a foo=>")
 
     def test_declaration_junk_chars(self):
         self._parse_error("<!DOCTYPE foo $ >")
@@ -352,13 +350,85 @@
         self._run_check(
             "<a a.b='v' c:d=v e-f=v>",
             [("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")])])
-
+        self._run_check(
+            "<a $><b $=%><c \=/>",
+            [("starttag", "a", [("$", None)]),
+             ("starttag", "b", [("$", "%")]),
+             ("starttag", "c", [("\\", "/")])])
 
     def test_entityrefs_in_attributes(self):
         self._run_check(
             "<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
             [("starttag", "html", [("foo", u"\u20AC&aa&unsupported;")])])
 
+    def test_entities_in_attribute_value(self):
+        # see #1200313
+        for entity in ['&', '&amp;', '&#38;', '&#x26;']:
+            self._run_check('<a href="%s">' % entity,
+                            [("starttag", "a", [("href", "&")])])
+            self._run_check("<a href='%s'>" % entity,
+                            [("starttag", "a", [("href", "&")])])
+            self._run_check("<a href=%s>" % entity,
+                            [("starttag", "a", [("href", "&")])])
+
+    def test_malformed_attributes(self):
+        # see #13357
+        html = (
+            "<a href=test'style='color:red;bad1'>test - bad1</a>"
+            "<a href=test'+style='color:red;ba2'>test - bad2</a>"
+            "<a href=test'&nbsp;style='color:red;bad3'>test - bad3</a>"
+            "<a href = test'&nbsp;style='color:red;bad4'  >test - bad4</a>"
+        )
+        expected = [
+            ('starttag', 'a', [('href', "test'style='color:red;bad1'")]),
+            ('data', 'test - bad1'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'+style='color:red;ba2'")]),
+            ('data', 'test - bad2'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', u"test'\xa0style='color:red;bad3'")]),
+            ('data', 'test - bad3'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', u"test'\xa0style='color:red;bad4'")]),
+            ('data', 'test - bad4'), ('endtag', 'a')
+        ]
+        self._run_check(html, expected)
+
+    def test_malformed_adjacent_attributes(self):
+        # see #12629
+        self._run_check('<x><y z=""o"" /></x>',
+                        [('starttag', 'x', []),
+                            ('startendtag', 'y', [('z', ''), ('o""', None)]),
+                            ('endtag', 'x')])
+        self._run_check('<x><y z="""" /></x>',
+                        [('starttag', 'x', []),
+                            ('startendtag', 'y', [('z', ''), ('""', None)]),
+                            ('endtag', 'x')])
+
+    # see #755670 for the following 3 tests
+    def test_adjacent_attributes(self):
+        self._run_check('<a width="100%"cellspacing=0>',
+                        [("starttag", "a",
+                          [("width", "100%"), ("cellspacing","0")])])
+
+        self._run_check('<a id="foo"class="bar">',
+                        [("starttag", "a",
+                          [("id", "foo"), ("class","bar")])])
+
+    def test_missing_attribute_value(self):
+        self._run_check('<a v=>',
+                        [("starttag", "a", [("v", "")])])
+
+    def test_javascript_attribute_value(self):
+        self._run_check("<a href=javascript:popup('/popup/help.html')>",
+                        [("starttag", "a",
+                          [("href", "javascript:popup('/popup/help.html')")])])
+
+    def test_end_tag_in_attribute_value(self):
+        # see #1745761
+        self._run_check("<a href='http://www.example.org/\">;'>spam</a>",
+                        [("starttag", "a",
+                          [("href", "http://www.example.org/\">;")]),
+                         ("data", "spam"), ("endtag", "a")])
+
+
 def test_main():
     test_support.run_unittest(HTMLParserTestCase, AttributesTestCase)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -76,6 +76,9 @@
 Library
 -------
 
+- Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly
+  handles non-valid attributes, including adjacent and unquoted attributes.
+
 - Issue #13193: Fix distutils.filelist.FileList under Windows.  The
   "recursive-include" directive now recognizes both legal path separators.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 17:57:04 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 14 Nov 2011 17:57:04 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Group_tests_abo?=
 =?utf8?q?ut_attributes_in_a_separate_class=2E?=
Message-ID: <E1RPzqC-0006Yb-B7@dinsdale.python.org>

http://hg.python.org/cpython/rev/62d4dc5bb3ec
changeset:   73546:62d4dc5bb3ec
branch:      3.2
parent:      73542:a259511351d9
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Mon Nov 14 18:13:22 2011 +0200
summary:
  Group tests about attributes in a separate class.

files:
  Lib/test/test_htmlparser.py |  121 ++++++++++++-----------
  1 files changed, 62 insertions(+), 59 deletions(-)


diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -196,60 +196,6 @@
             ("data", "this < text > contains < bare>pointy< brackets"),
             ])
 
-    def test_attr_syntax(self):
-        output = [
-          ("starttag", "a", [("b", "v"), ("c", "v"), ("d", "v"), ("e", None)])
-          ]
-        self._run_check("""<a b='v' c="v" d=v e>""", output)
-        self._run_check("""<a  b = 'v' c = "v" d = v e>""", output)
-        self._run_check("""<a\nb\n=\n'v'\nc\n=\n"v"\nd\n=\nv\ne>""", output)
-        self._run_check("""<a\tb\t=\t'v'\tc\t=\t"v"\td\t=\tv\te>""", output)
-
-    def test_attr_values(self):
-        self._run_check("""<a b='xxx\n\txxx' c="yyy\t\nyyy" d='\txyz\n'>""",
-                        [("starttag", "a", [("b", "xxx\n\txxx"),
-                                            ("c", "yyy\t\nyyy"),
-                                            ("d", "\txyz\n")])
-                         ])
-        self._run_check("""<a b='' c="">""", [
-            ("starttag", "a", [("b", ""), ("c", "")]),
-            ])
-        # Regression test for SF patch #669683.
-        self._run_check("<e a=rgb(1,2,3)>", [
-            ("starttag", "e", [("a", "rgb(1,2,3)")]),
-            ])
-        # Regression test for SF bug #921657.
-        self._run_check("<a href=mailto:xyz at example.com>", [
-            ("starttag", "a", [("href", "mailto:xyz at example.com")]),
-            ])
-
-    def test_attr_nonascii(self):
-        # see issue 7311
-        self._run_check("<img src=/foo/bar.png alt=\u4e2d\u6587>", [
-            ("starttag", "img", [("src", "/foo/bar.png"),
-                                 ("alt", "\u4e2d\u6587")]),
-            ])
-        self._run_check("<a title='\u30c6\u30b9\u30c8' "
-                        "href='\u30c6\u30b9\u30c8.html'>", [
-            ("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
-                               ("href", "\u30c6\u30b9\u30c8.html")]),
-            ])
-        self._run_check('<a title="\u30c6\u30b9\u30c8" '
-                        'href="\u30c6\u30b9\u30c8.html">', [
-            ("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
-                               ("href", "\u30c6\u30b9\u30c8.html")]),
-            ])
-
-    def test_attr_entity_replacement(self):
-        self._run_check("""<a b='&amp;&gt;&lt;&quot;&apos;'>""", [
-            ("starttag", "a", [("b", "&><\"'")]),
-            ])
-
-    def test_attr_funky_names(self):
-        self._run_check("""<a a.b='v' c:d=v e-f=v>""", [
-            ("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")]),
-            ])
-
     def test_illegal_declarations(self):
         self._parse_error('<!spacer type="block" height="25">')
 
@@ -358,10 +304,6 @@
                                     ("endtag", element_lower)])
 
 
-    def test_entityrefs_in_attributes(self):
-        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
-                        [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
-
 
 class HTMLParserTolerantTestCase(HTMLParserStrictTestCase):
 
@@ -458,8 +400,69 @@
         self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
 
+class AttributesStrictTestCase(TestCaseBase):
+
+    def get_collector(self):
+        return EventCollector(strict=True)
+
+    def test_attr_syntax(self):
+        output = [
+          ("starttag", "a", [("b", "v"), ("c", "v"), ("d", "v"), ("e", None)])
+        ]
+        self._run_check("""<a b='v' c="v" d=v e>""", output)
+        self._run_check("""<a  b = 'v' c = "v" d = v e>""", output)
+        self._run_check("""<a\nb\n=\n'v'\nc\n=\n"v"\nd\n=\nv\ne>""", output)
+        self._run_check("""<a\tb\t=\t'v'\tc\t=\t"v"\td\t=\tv\te>""", output)
+
+    def test_attr_values(self):
+        self._run_check("""<a b='xxx\n\txxx' c="yyy\t\nyyy" d='\txyz\n'>""",
+                        [("starttag", "a", [("b", "xxx\n\txxx"),
+                                            ("c", "yyy\t\nyyy"),
+                                            ("d", "\txyz\n")])])
+        self._run_check("""<a b='' c="">""",
+                        [("starttag", "a", [("b", ""), ("c", "")])])
+        # Regression test for SF patch #669683.
+        self._run_check("<e a=rgb(1,2,3)>",
+                        [("starttag", "e", [("a", "rgb(1,2,3)")])])
+        # Regression test for SF bug #921657.
+        self._run_check(
+            "<a href=mailto:xyz at example.com>",
+            [("starttag", "a", [("href", "mailto:xyz at example.com")])])
+
+    def test_attr_nonascii(self):
+        # see issue 7311
+        self._run_check(
+            "<img src=/foo/bar.png alt=\u4e2d\u6587>",
+            [("starttag", "img", [("src", "/foo/bar.png"),
+                                  ("alt", "\u4e2d\u6587")])])
+        self._run_check(
+            "<a title='\u30c6\u30b9\u30c8' href='\u30c6\u30b9\u30c8.html'>",
+            [("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
+                                ("href", "\u30c6\u30b9\u30c8.html")])])
+        self._run_check(
+            '<a title="\u30c6\u30b9\u30c8" href="\u30c6\u30b9\u30c8.html">',
+            [("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
+                                ("href", "\u30c6\u30b9\u30c8.html")])])
+
+    def test_attr_entity_replacement(self):
+        self._run_check(
+            "<a b='&amp;&gt;&lt;&quot;&apos;'>",
+            [("starttag", "a", [("b", "&><\"'")])])
+
+    def test_attr_funky_names(self):
+        self._run_check(
+            "<a a.b='v' c:d=v e-f=v>",
+            [("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")])])
+
+    def test_entityrefs_in_attributes(self):
+        self._run_check(
+            "<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
+            [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
+
+
 def test_main():
-    support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase)
+    support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase,
+                         AttributesStrictTestCase)
 
 
 if __name__ == "__main__":

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 17:57:05 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 14 Nov 2011 17:57:05 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzE3NDU3NjEsICM3?=
 =?utf8?q?55670=2C_=2313357=2C_=2312629=2C_=231200313=3A_improve_attribute?=
 =?utf8?q?_handling_in?=
Message-ID: <E1RPzqD-0006Yf-7j@dinsdale.python.org>

http://hg.python.org/cpython/rev/16ed15ff0d7c
changeset:   73547:16ed15ff0d7c
branch:      3.2
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Mon Nov 14 18:53:33 2011 +0200
summary:
  #1745761, #755670, #13357, #12629, #1200313: improve attribute handling in HTMLParser.

files:
  Lib/html/parser.py          |   19 ++--
  Lib/test/test_htmlparser.py |  107 ++++++++++++++++++++---
  Misc/NEWS                   |    3 +
  3 files changed, 106 insertions(+), 23 deletions(-)


diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -30,8 +30,8 @@
     r'\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
     r'(\'[^\']*\'|"[^"]*"|[^\s"\'=<>`]*))?')
 attrfind_tolerant = re.compile(
-    r',?\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
-    r'(\'[^\']*\'|"[^"]*"|[^>\s]*))?')
+    r'\s*((?<=[\'"\s])[^\s/>][^\s/=>]*)(\s*=+\s*'
+    r'(\'[^\']*\'|"[^"]*"|(?![\'"])[^>\s]*))?')
 locatestarttagend = re.compile(r"""
   <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
   (?:\s+                             # whitespace before attribute name
@@ -49,16 +49,16 @@
 locatestarttagend_tolerant = re.compile(r"""
   <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
   (?:\s*                             # optional whitespace before attribute name
-    (?:[a-zA-Z_][-.:a-zA-Z0-9_]*     # attribute name
-      (?:\s*=\s*                     # value indicator
+    (?:(?<=['"\s])[^\s/>][^\s/=>]*   # attribute name
+      (?:\s*=+\s*                    # value indicator
         (?:'[^']*'                   # LITA-enclosed value
-          |\"[^\"]*\"                # LIT-enclosed value
-          |[^'\">\s]+                # bare value
+          |"[^"]*"                   # LIT-enclosed value
+          |(?!['"])[^>\s]*           # bare value
          )
          (?:\s*,)*                   # possibly followed by a comma
-       )?
-     )
-   )*
+       )?\s*
+     )*
+   )?
   \s*                                # trailing whitespace
 """, re.VERBOSE)
 endendtag = re.compile('>')
@@ -295,6 +295,7 @@
             elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
                  attrvalue[:1] == '"' == attrvalue[-1:]:
                 attrvalue = attrvalue[1:-1]
+            if attrvalue:
                 attrvalue = self.unescape(attrvalue)
             attrs.append((attrname.lower(), attrvalue))
             k = m.end()
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -241,13 +241,11 @@
         self._parse_error("<a<a>")
         self._parse_error("</a<a>")
         self._parse_error("<!")
-        self._parse_error("<a $>")
         self._parse_error("<a")
         self._parse_error("<a foo='bar'")
         self._parse_error("<a foo='bar")
         self._parse_error("<a foo='>'")
         self._parse_error("<a foo='>")
-        self._parse_error("<a foo=>")
 
     def test_declaration_junk_chars(self):
         self._parse_error("<!DOCTYPE foo $ >")
@@ -313,15 +311,14 @@
     def test_tolerant_parsing(self):
         self._run_check('<html <html>te>>xt&a<<bc</a></html>\n'
                         '<img src="URL><//img></html</html>', [
-                             ('data', '<html '),
-                             ('starttag', 'html', []),
-                             ('data', 'te>>xt'),
-                             ('entityref', 'a'),
-                             ('data', '<<bc'),
-                             ('endtag', 'a'),
-                             ('endtag', 'html'),
-                             ('data', '\n<img src="URL><//img></html'),
-                             ('endtag', 'html')])
+                            ('starttag', 'html', [('<html', None)]),
+                            ('data', 'te>>xt'),
+                            ('entityref', 'a'),
+                            ('data', '<<bc'),
+                            ('endtag', 'a'),
+                            ('endtag', 'html'),
+                            ('data', '\n<img src="URL><//img></html'),
+                            ('endtag', 'html')])
 
     def test_with_unquoted_attributes(self):
         # see #12008
@@ -352,7 +349,7 @@
                         'method="post">', [
                             ('starttag', 'form',
                                 [('action', '/xxx.php?a=1&b=2&amp'),
-                                 ('method', 'post')])])
+                                 (',', None), ('method', 'post')])])
 
     def test_weird_chars_in_unquoted_attribute_values(self):
         self._run_check('<form action=bogus|&#()value>', [
@@ -383,7 +380,7 @@
 
         html = '<div style="", foo = "bar" ><b>The <a href="some_url">rain</a>'
         expected = [
-            ('starttag', 'div', [('style', ''), ('foo', 'bar')]),
+            ('starttag', 'div', [('style', ''), (',', None), ('foo', 'bar')]),
             ('starttag', 'b', []),
             ('data', 'The '),
             ('starttag', 'a', [('href', 'some_url')]),
@@ -460,9 +457,91 @@
             [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
 
 
+
+class AttributesTolerantTestCase(AttributesStrictTestCase):
+
+    def get_collector(self):
+        return EventCollector(strict=False)
+
+    def test_attr_funky_names2(self):
+        self._run_check(
+            "<a $><b $=%><c \=/>",
+            [("starttag", "a", [("$", None)]),
+             ("starttag", "b", [("$", "%")]),
+             ("starttag", "c", [("\\", "/")])])
+
+    def test_entities_in_attribute_value(self):
+        # see #1200313
+        for entity in ['&', '&amp;', '&#38;', '&#x26;']:
+            self._run_check('<a href="%s">' % entity,
+                            [("starttag", "a", [("href", "&")])])
+            self._run_check("<a href='%s'>" % entity,
+                            [("starttag", "a", [("href", "&")])])
+            self._run_check("<a href=%s>" % entity,
+                            [("starttag", "a", [("href", "&")])])
+
+    def test_malformed_attributes(self):
+        # see #13357
+        html = (
+            "<a href=test'style='color:red;bad1'>test - bad1</a>"
+            "<a href=test'+style='color:red;ba2'>test - bad2</a>"
+            "<a href=test'&nbsp;style='color:red;bad3'>test - bad3</a>"
+            "<a href = test'&nbsp;style='color:red;bad4'  >test - bad4</a>"
+        )
+        expected = [
+            ('starttag', 'a', [('href', "test'style='color:red;bad1'")]),
+            ('data', 'test - bad1'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'+style='color:red;ba2'")]),
+            ('data', 'test - bad2'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'\xa0style='color:red;bad3'")]),
+            ('data', 'test - bad3'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'\xa0style='color:red;bad4'")]),
+            ('data', 'test - bad4'), ('endtag', 'a')
+        ]
+        self._run_check(html, expected)
+
+    def test_malformed_adjacent_attributes(self):
+        # see #12629
+        self._run_check('<x><y z=""o"" /></x>',
+                        [('starttag', 'x', []),
+                            ('startendtag', 'y', [('z', ''), ('o""', None)]),
+                            ('endtag', 'x')])
+        self._run_check('<x><y z="""" /></x>',
+                        [('starttag', 'x', []),
+                            ('startendtag', 'y', [('z', ''), ('""', None)]),
+                            ('endtag', 'x')])
+
+    # see #755670 for the following 3 tests
+    def test_adjacent_attributes(self):
+        self._run_check('<a width="100%"cellspacing=0>',
+                        [("starttag", "a",
+                          [("width", "100%"), ("cellspacing","0")])])
+
+        self._run_check('<a id="foo"class="bar">',
+                        [("starttag", "a",
+                          [("id", "foo"), ("class","bar")])])
+
+    def test_missing_attribute_value(self):
+        self._run_check('<a v=>',
+                        [("starttag", "a", [("v", "")])])
+
+    def test_javascript_attribute_value(self):
+        self._run_check("<a href=javascript:popup('/popup/help.html')>",
+                        [("starttag", "a",
+                          [("href", "javascript:popup('/popup/help.html')")])])
+
+    def test_end_tag_in_attribute_value(self):
+        # see #1745761
+        self._run_check("<a href='http://www.example.org/\">;'>spam</a>",
+                        [("starttag", "a",
+                          [("href", "http://www.example.org/\">;")]),
+                         ("data", "spam"), ("endtag", "a")])
+
+
+
 def test_main():
     support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase,
-                         AttributesStrictTestCase)
+                         AttributesStrictTestCase, AttributesTolerantTestCase)
 
 
 if __name__ == "__main__":
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -73,6 +73,9 @@
 Library
 -------
 
+- Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly
+  handles non-valid attributes, including adjacent and unquoted attributes.
+
 - Issue #13193: Fix distutils.filelist.FileList under Windows.  The
   "recursive-include" directive now recognizes both legal path separators.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 17:57:06 2011
From: python-checkins at python.org (ezio.melotti)
Date: Mon, 14 Nov 2011 17:57:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiAjMTc0NTc2MSwgIzc1NTY3MCwgIzEzMzU3LCAjMTI2MjksICMxMjAwMzEzOiBt?=
 =?utf8?q?erge_with_3=2E2=2E?=
Message-ID: <E1RPzqE-0006Yj-59@dinsdale.python.org>

http://hg.python.org/cpython/rev/426f7a2b1826
changeset:   73548:426f7a2b1826
parent:      73543:410115400838
parent:      73547:16ed15ff0d7c
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Mon Nov 14 18:56:11 2011 +0200
summary:
  #1745761, #755670, #13357, #12629, #1200313: merge with 3.2.

files:
  Lib/html/parser.py          |   19 +-
  Lib/test/test_htmlparser.py |  226 ++++++++++++++++-------
  Misc/NEWS                   |    3 +
  3 files changed, 167 insertions(+), 81 deletions(-)


diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -30,8 +30,8 @@
     r'\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
     r'(\'[^\']*\'|"[^"]*"|[^\s"\'=<>`]*))?')
 attrfind_tolerant = re.compile(
-    r',?\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*'
-    r'(\'[^\']*\'|"[^"]*"|[^>\s]*))?')
+    r'\s*((?<=[\'"\s])[^\s/>][^\s/=>]*)(\s*=+\s*'
+    r'(\'[^\']*\'|"[^"]*"|(?![\'"])[^>\s]*))?')
 locatestarttagend = re.compile(r"""
   <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
   (?:\s+                             # whitespace before attribute name
@@ -49,16 +49,16 @@
 locatestarttagend_tolerant = re.compile(r"""
   <[a-zA-Z][-.a-zA-Z0-9:_]*          # tag name
   (?:\s*                             # optional whitespace before attribute name
-    (?:[a-zA-Z_][-.:a-zA-Z0-9_]*     # attribute name
-      (?:\s*=\s*                     # value indicator
+    (?:(?<=['"\s])[^\s/>][^\s/=>]*   # attribute name
+      (?:\s*=+\s*                    # value indicator
         (?:'[^']*'                   # LITA-enclosed value
-          |\"[^\"]*\"                # LIT-enclosed value
-          |[^'\">\s]+                # bare value
+          |"[^"]*"                   # LIT-enclosed value
+          |(?!['"])[^>\s]*           # bare value
          )
          (?:\s*,)*                   # possibly followed by a comma
-       )?
-     )
-   )*
+       )?\s*
+     )*
+   )?
   \s*                                # trailing whitespace
 """, re.VERBOSE)
 endendtag = re.compile('>')
@@ -295,6 +295,7 @@
             elif attrvalue[:1] == '\'' == attrvalue[-1:] or \
                  attrvalue[:1] == '"' == attrvalue[-1:]:
                 attrvalue = attrvalue[1:-1]
+            if attrvalue:
                 attrvalue = self.unescape(attrvalue)
             attrs.append((attrname.lower(), attrvalue))
             k = m.end()
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -196,60 +196,6 @@
             ("data", "this < text > contains < bare>pointy< brackets"),
             ])
 
-    def test_attr_syntax(self):
-        output = [
-          ("starttag", "a", [("b", "v"), ("c", "v"), ("d", "v"), ("e", None)])
-          ]
-        self._run_check("""<a b='v' c="v" d=v e>""", output)
-        self._run_check("""<a  b = 'v' c = "v" d = v e>""", output)
-        self._run_check("""<a\nb\n=\n'v'\nc\n=\n"v"\nd\n=\nv\ne>""", output)
-        self._run_check("""<a\tb\t=\t'v'\tc\t=\t"v"\td\t=\tv\te>""", output)
-
-    def test_attr_values(self):
-        self._run_check("""<a b='xxx\n\txxx' c="yyy\t\nyyy" d='\txyz\n'>""",
-                        [("starttag", "a", [("b", "xxx\n\txxx"),
-                                            ("c", "yyy\t\nyyy"),
-                                            ("d", "\txyz\n")])
-                         ])
-        self._run_check("""<a b='' c="">""", [
-            ("starttag", "a", [("b", ""), ("c", "")]),
-            ])
-        # Regression test for SF patch #669683.
-        self._run_check("<e a=rgb(1,2,3)>", [
-            ("starttag", "e", [("a", "rgb(1,2,3)")]),
-            ])
-        # Regression test for SF bug #921657.
-        self._run_check("<a href=mailto:xyz at example.com>", [
-            ("starttag", "a", [("href", "mailto:xyz at example.com")]),
-            ])
-
-    def test_attr_nonascii(self):
-        # see issue 7311
-        self._run_check("<img src=/foo/bar.png alt=\u4e2d\u6587>", [
-            ("starttag", "img", [("src", "/foo/bar.png"),
-                                 ("alt", "\u4e2d\u6587")]),
-            ])
-        self._run_check("<a title='\u30c6\u30b9\u30c8' "
-                        "href='\u30c6\u30b9\u30c8.html'>", [
-            ("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
-                               ("href", "\u30c6\u30b9\u30c8.html")]),
-            ])
-        self._run_check('<a title="\u30c6\u30b9\u30c8" '
-                        'href="\u30c6\u30b9\u30c8.html">', [
-            ("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
-                               ("href", "\u30c6\u30b9\u30c8.html")]),
-            ])
-
-    def test_attr_entity_replacement(self):
-        self._run_check("""<a b='&amp;&gt;&lt;&quot;&apos;'>""", [
-            ("starttag", "a", [("b", "&><\"'")]),
-            ])
-
-    def test_attr_funky_names(self):
-        self._run_check("""<a a.b='v' c:d=v e-f=v>""", [
-            ("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")]),
-            ])
-
     def test_illegal_declarations(self):
         self._parse_error('<!spacer type="block" height="25">')
 
@@ -295,13 +241,11 @@
         self._parse_error("<a<a>")
         self._parse_error("</a<a>")
         self._parse_error("<!")
-        self._parse_error("<a $>")
         self._parse_error("<a")
         self._parse_error("<a foo='bar'")
         self._parse_error("<a foo='bar")
         self._parse_error("<a foo='>'")
         self._parse_error("<a foo='>")
-        self._parse_error("<a foo=>")
 
     def test_declaration_junk_chars(self):
         self._parse_error("<!DOCTYPE foo $ >")
@@ -358,10 +302,6 @@
                                     ("endtag", element_lower)])
 
 
-    def test_entityrefs_in_attributes(self):
-        self._run_check("<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
-                        [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
-
 
 class HTMLParserTolerantTestCase(HTMLParserStrictTestCase):
 
@@ -371,15 +311,14 @@
     def test_tolerant_parsing(self):
         self._run_check('<html <html>te>>xt&a<<bc</a></html>\n'
                         '<img src="URL><//img></html</html>', [
-                             ('data', '<html '),
-                             ('starttag', 'html', []),
-                             ('data', 'te>>xt'),
-                             ('entityref', 'a'),
-                             ('data', '<<bc'),
-                             ('endtag', 'a'),
-                             ('endtag', 'html'),
-                             ('data', '\n<img src="URL><//img></html'),
-                             ('endtag', 'html')])
+                            ('starttag', 'html', [('<html', None)]),
+                            ('data', 'te>>xt'),
+                            ('entityref', 'a'),
+                            ('data', '<<bc'),
+                            ('endtag', 'a'),
+                            ('endtag', 'html'),
+                            ('data', '\n<img src="URL><//img></html'),
+                            ('endtag', 'html')])
 
     def test_with_unquoted_attributes(self):
         # see #12008
@@ -410,7 +349,7 @@
                         'method="post">', [
                             ('starttag', 'form',
                                 [('action', '/xxx.php?a=1&b=2&amp'),
-                                 ('method', 'post')])])
+                                 (',', None), ('method', 'post')])])
 
     def test_weird_chars_in_unquoted_attribute_values(self):
         self._run_check('<form action=bogus|&#()value>', [
@@ -441,7 +380,7 @@
 
         html = '<div style="", foo = "bar" ><b>The <a href="some_url">rain</a>'
         expected = [
-            ('starttag', 'div', [('style', ''), ('foo', 'bar')]),
+            ('starttag', 'div', [('style', ''), (',', None), ('foo', 'bar')]),
             ('starttag', 'b', []),
             ('data', 'The '),
             ('starttag', 'a', [('href', 'some_url')]),
@@ -458,8 +397,151 @@
         self.assertEqual(p.unescape('&#123; ' * 1050), '{ ' * 1050)
 
 
+class AttributesStrictTestCase(TestCaseBase):
+
+    def get_collector(self):
+        return EventCollector(strict=True)
+
+    def test_attr_syntax(self):
+        output = [
+          ("starttag", "a", [("b", "v"), ("c", "v"), ("d", "v"), ("e", None)])
+        ]
+        self._run_check("""<a b='v' c="v" d=v e>""", output)
+        self._run_check("""<a  b = 'v' c = "v" d = v e>""", output)
+        self._run_check("""<a\nb\n=\n'v'\nc\n=\n"v"\nd\n=\nv\ne>""", output)
+        self._run_check("""<a\tb\t=\t'v'\tc\t=\t"v"\td\t=\tv\te>""", output)
+
+    def test_attr_values(self):
+        self._run_check("""<a b='xxx\n\txxx' c="yyy\t\nyyy" d='\txyz\n'>""",
+                        [("starttag", "a", [("b", "xxx\n\txxx"),
+                                            ("c", "yyy\t\nyyy"),
+                                            ("d", "\txyz\n")])])
+        self._run_check("""<a b='' c="">""",
+                        [("starttag", "a", [("b", ""), ("c", "")])])
+        # Regression test for SF patch #669683.
+        self._run_check("<e a=rgb(1,2,3)>",
+                        [("starttag", "e", [("a", "rgb(1,2,3)")])])
+        # Regression test for SF bug #921657.
+        self._run_check(
+            "<a href=mailto:xyz at example.com>",
+            [("starttag", "a", [("href", "mailto:xyz at example.com")])])
+
+    def test_attr_nonascii(self):
+        # see issue 7311
+        self._run_check(
+            "<img src=/foo/bar.png alt=\u4e2d\u6587>",
+            [("starttag", "img", [("src", "/foo/bar.png"),
+                                  ("alt", "\u4e2d\u6587")])])
+        self._run_check(
+            "<a title='\u30c6\u30b9\u30c8' href='\u30c6\u30b9\u30c8.html'>",
+            [("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
+                                ("href", "\u30c6\u30b9\u30c8.html")])])
+        self._run_check(
+            '<a title="\u30c6\u30b9\u30c8" href="\u30c6\u30b9\u30c8.html">',
+            [("starttag", "a", [("title", "\u30c6\u30b9\u30c8"),
+                                ("href", "\u30c6\u30b9\u30c8.html")])])
+
+    def test_attr_entity_replacement(self):
+        self._run_check(
+            "<a b='&amp;&gt;&lt;&quot;&apos;'>",
+            [("starttag", "a", [("b", "&><\"'")])])
+
+    def test_attr_funky_names(self):
+        self._run_check(
+            "<a a.b='v' c:d=v e-f=v>",
+            [("starttag", "a", [("a.b", "v"), ("c:d", "v"), ("e-f", "v")])])
+
+    def test_entityrefs_in_attributes(self):
+        self._run_check(
+            "<html foo='&euro;&amp;&#97;&#x61;&unsupported;'>",
+            [("starttag", "html", [("foo", "\u20AC&aa&unsupported;")])])
+
+
+
+class AttributesTolerantTestCase(AttributesStrictTestCase):
+
+    def get_collector(self):
+        return EventCollector(strict=False)
+
+    def test_attr_funky_names2(self):
+        self._run_check(
+            "<a $><b $=%><c \=/>",
+            [("starttag", "a", [("$", None)]),
+             ("starttag", "b", [("$", "%")]),
+             ("starttag", "c", [("\\", "/")])])
+
+    def test_entities_in_attribute_value(self):
+        # see #1200313
+        for entity in ['&', '&amp;', '&#38;', '&#x26;']:
+            self._run_check('<a href="%s">' % entity,
+                            [("starttag", "a", [("href", "&")])])
+            self._run_check("<a href='%s'>" % entity,
+                            [("starttag", "a", [("href", "&")])])
+            self._run_check("<a href=%s>" % entity,
+                            [("starttag", "a", [("href", "&")])])
+
+    def test_malformed_attributes(self):
+        # see #13357
+        html = (
+            "<a href=test'style='color:red;bad1'>test - bad1</a>"
+            "<a href=test'+style='color:red;ba2'>test - bad2</a>"
+            "<a href=test'&nbsp;style='color:red;bad3'>test - bad3</a>"
+            "<a href = test'&nbsp;style='color:red;bad4'  >test - bad4</a>"
+        )
+        expected = [
+            ('starttag', 'a', [('href', "test'style='color:red;bad1'")]),
+            ('data', 'test - bad1'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'+style='color:red;ba2'")]),
+            ('data', 'test - bad2'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'\xa0style='color:red;bad3'")]),
+            ('data', 'test - bad3'), ('endtag', 'a'),
+            ('starttag', 'a', [('href', "test'\xa0style='color:red;bad4'")]),
+            ('data', 'test - bad4'), ('endtag', 'a')
+        ]
+        self._run_check(html, expected)
+
+    def test_malformed_adjacent_attributes(self):
+        # see #12629
+        self._run_check('<x><y z=""o"" /></x>',
+                        [('starttag', 'x', []),
+                            ('startendtag', 'y', [('z', ''), ('o""', None)]),
+                            ('endtag', 'x')])
+        self._run_check('<x><y z="""" /></x>',
+                        [('starttag', 'x', []),
+                            ('startendtag', 'y', [('z', ''), ('""', None)]),
+                            ('endtag', 'x')])
+
+    # see #755670 for the following 3 tests
+    def test_adjacent_attributes(self):
+        self._run_check('<a width="100%"cellspacing=0>',
+                        [("starttag", "a",
+                          [("width", "100%"), ("cellspacing","0")])])
+
+        self._run_check('<a id="foo"class="bar">',
+                        [("starttag", "a",
+                          [("id", "foo"), ("class","bar")])])
+
+    def test_missing_attribute_value(self):
+        self._run_check('<a v=>',
+                        [("starttag", "a", [("v", "")])])
+
+    def test_javascript_attribute_value(self):
+        self._run_check("<a href=javascript:popup('/popup/help.html')>",
+                        [("starttag", "a",
+                          [("href", "javascript:popup('/popup/help.html')")])])
+
+    def test_end_tag_in_attribute_value(self):
+        # see #1745761
+        self._run_check("<a href='http://www.example.org/\">;'>spam</a>",
+                        [("starttag", "a",
+                          [("href", "http://www.example.org/\">;")]),
+                         ("data", "spam"), ("endtag", "a")])
+
+
+
 def test_main():
-    support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase)
+    support.run_unittest(HTMLParserStrictTestCase, HTMLParserTolerantTestCase,
+                         AttributesStrictTestCase, AttributesTolerantTestCase)
 
 
 if __name__ == "__main__":
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -365,6 +365,9 @@
 Library
 -------
 
+- Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly
+  handles non-valid attributes, including adjacent and unquoted attributes.
+
 - Issue #13193: Fix distutils.filelist.FileList and
   packaging.manifest.Manifest under Windows.  The "recursive-include"
   directive now recognizes both legal path separators.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 19:17:53 2011
From: python-checkins at python.org (jesus.cea)
Date: Mon, 14 Nov 2011 19:17:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=236397=3A_Support_?=
 =?utf8?q?=27/dev/poll=27_polling_objects_in_select_module=2C_under?=
Message-ID: <E1RQ16P-0006Z9-IN@dinsdale.python.org>

http://hg.python.org/cpython/rev/8f7ab4bf7ad9
changeset:   73549:8f7ab4bf7ad9
user:        Jesus Cea <jcea at jcea.es>
date:        Mon Nov 14 19:07:41 2011 +0100
summary:
  Issue #6397: Support '/dev/poll' polling objects in select module, under Solaris & derivatives.

files:
  Doc/library/select.rst |   84 ++++++-
  Lib/test/test_poll.py  |  123 ++-------
  Misc/NEWS              |    3 +
  Modules/selectmodule.c |  370 +++++++++++++++++++++++++++++
  configure              |    7 +-
  configure.in           |    7 +-
  pyconfig.h.in          |    3 +
  7 files changed, 497 insertions(+), 100 deletions(-)


diff --git a/Doc/library/select.rst b/Doc/library/select.rst
--- a/Doc/library/select.rst
+++ b/Doc/library/select.rst
@@ -6,7 +6,8 @@
 
 
 This module provides access to the :c:func:`select` and :c:func:`poll` functions
-available in most operating systems, :c:func:`epoll` available on Linux 2.5+ and
+available in most operating systems, :c:func:`devpoll` available on
+Solaris and derivatives, :c:func:`epoll` available on Linux 2.5+ and
 :c:func:`kqueue` available on most BSD.
 Note that on Windows, it only works for sockets; on other operating systems,
 it also works for other file types (in particular, on Unix, it works on pipes).
@@ -24,6 +25,19 @@
       Following :pep:`3151`, this class was made an alias of :exc:`OSError`.
 
 
+.. function:: devpoll()
+   (Only supported on Solaris and derivatives.)  Returns a ``/dev/poll``
+   polling object; see section :ref:`devpoll-objects` below for the
+   methods supported by devpoll objects.
+
+   :c:func:`devpoll` objects are linked to the number of file
+   descriptors allowed at the time of instantiation. If your program
+   reduces this value, :c:func:`devpoll` will fail. If your program
+   increases this value, c:func:`devpoll` may return an
+   incomplete list of active file descriptors.
+
+   .. versionadded:: 3.3
+
 .. function:: epoll(sizehint=-1)
 
    (Only supported on Linux 2.5.44 and newer.)  Returns an edge polling object,
@@ -107,6 +121,74 @@
    .. versionadded:: 3.2
 
 
+.. _devpoll-objects:
+
+``/dev/poll`` Polling Objects
+----------------------------------------------
+
+   http://developers.sun.com/solaris/articles/using_devpoll.html
+   http://developers.sun.com/solaris/articles/polling_efficient.html
+
+Solaris and derivatives have ``/dev/poll``. While :c:func:`select` is
+O(highest file descriptor) and :c:func:`poll` is O(number of file
+descriptors), ``/dev/poll`` is O(active file descriptors).
+
+``/dev/poll`` behaviour is very close to the standard :c:func:`poll`
+object.
+
+
+.. method:: devpoll.register(fd[, eventmask])
+
+   Register a file descriptor with the polling object.  Future calls to the
+   :meth:`poll` method will then check whether the file descriptor has any pending
+   I/O events.  *fd* can be either an integer, or an object with a :meth:`fileno`
+   method that returns an integer.  File objects implement :meth:`fileno`, so they
+   can also be used as the argument.
+
+   *eventmask* is an optional bitmask describing the type of events you want to
+   check for. The constants are the same that with :c:func:`poll`
+   object. The default value is a combination of the constants :const:`POLLIN`,
+   :const:`POLLPRI`, and :const:`POLLOUT`.
+
+   .. warning::
+
+      Registering a file descriptor that's already registered is not an
+      error, but the result is undefined. The appropiate action is to
+      unregister or modify it first. This is an important difference
+      compared with :c:func:`poll`.
+
+
+.. method:: devpoll.modify(fd[, eventmask])
+
+   This method does an :meth:`unregister` followed by a
+   :meth:`register`. It is (a bit) more efficient that doing the same
+   explicitly.
+
+
+.. method:: devpoll.unregister(fd)
+
+   Remove a file descriptor being tracked by a polling object.  Just like the
+   :meth:`register` method, *fd* can be an integer or an object with a
+   :meth:`fileno` method that returns an integer.
+
+   Attempting to remove a file descriptor that was never registered is
+   safely ignored.
+
+
+.. method:: devpoll.poll([timeout])
+
+   Polls the set of registered file descriptors, and returns a possibly-empty list
+   containing ``(fd, event)`` 2-tuples for the descriptors that have events or
+   errors to report. *fd* is the file descriptor, and *event* is a bitmask with
+   bits set for the reported events for that descriptor --- :const:`POLLIN` for
+   waiting input, :const:`POLLOUT` to indicate that the descriptor can be written
+   to, and so forth. An empty list indicates that the call timed out and no file
+   descriptors had any events to report. If *timeout* is given, it specifies the
+   length of time in milliseconds which the system will wait for events before
+   returning. If *timeout* is omitted, -1, or :const:`None`, the call will
+   block until there is an event for this poll object.
+
+
 .. _epoll-objects:
 
 Edge and Level Trigger Polling (epoll) Objects
diff --git a/Lib/test/test_poll.py b/Lib/test/test_devpoll.py
copy from Lib/test/test_poll.py
copy to Lib/test/test_devpoll.py
--- a/Lib/test/test_poll.py
+++ b/Lib/test/test_devpoll.py
@@ -1,12 +1,14 @@
-# Test case for the os.poll() function
+# Test case for the select.devpoll() function
 
-import os, select, random, unittest
+# Initial tests are copied as is from "test_poll.py"
+
+import os, select, random, unittest, sys
 from test.support import TESTFN, run_unittest
 
 try:
-    select.poll
+    select.devpoll
 except AttributeError:
-    raise unittest.SkipTest("select.poll not defined -- skipping test_poll")
+    raise unittest.SkipTest("select.devpoll not defined -- skipping test_devpoll")
 
 
 def find_ready_matching(ready, flag):
@@ -16,13 +18,13 @@
             match.append(fd)
     return match
 
-class PollTests(unittest.TestCase):
+class DevPollTests(unittest.TestCase):
 
-    def test_poll1(self):
+    def test_devpoll1(self):
         # Basic functional test of poll object
         # Create a bunch of pipe and test that poll works with them.
 
-        p = select.poll()
+        p = select.devpoll()
 
         NUM_PIPES = 12
         MSG = b" This is a test."
@@ -48,110 +50,45 @@
             ready = p.poll()
             ready_writers = find_ready_matching(ready, select.POLLOUT)
             if not ready_writers:
-                raise RuntimeError("no pipes ready for writing")
+                self.fail("no pipes ready for writing")
             wr = random.choice(ready_writers)
             os.write(wr, MSG)
 
             ready = p.poll()
             ready_readers = find_ready_matching(ready, select.POLLIN)
             if not ready_readers:
-                raise RuntimeError("no pipes ready for reading")
-            rd = random.choice(ready_readers)
+                self.fail("no pipes ready for reading")
+            self.assertEqual([w2r[wr]], ready_readers)
+            rd = ready_readers[0]
             buf = os.read(rd, MSG_LEN)
             self.assertEqual(len(buf), MSG_LEN)
             bufs.append(buf)
-            os.close(r2w[rd]) ; os.close( rd )
-            p.unregister( r2w[rd] )
-            p.unregister( rd )
+            os.close(r2w[rd]) ; os.close(rd)
+            p.unregister(r2w[rd])
+            p.unregister(rd)
             writers.remove(r2w[rd])
 
         self.assertEqual(bufs, [MSG] * NUM_PIPES)
 
-    def poll_unit_tests(self):
-        # returns NVAL for invalid file descriptor
-        FD = 42
-        try:
-            os.close(FD)
-        except OSError:
-            pass
-        p = select.poll()
-        p.register(FD)
-        r = p.poll()
-        self.assertEqual(r[0], (FD, select.POLLNVAL))
+    def test_timeout_overflow(self):
+        pollster = select.devpoll()
+        w, r = os.pipe()
+        pollster.register(w)
 
-        f = open(TESTFN, 'w')
-        fd = f.fileno()
-        p = select.poll()
-        p.register(f)
-        r = p.poll()
-        self.assertEqual(r[0][0], fd)
-        f.close()
-        r = p.poll()
-        self.assertEqual(r[0], (fd, select.POLLNVAL))
-        os.unlink(TESTFN)
+        pollster.poll(-1)
+        self.assertRaises(OverflowError, pollster.poll, -2)
+        self.assertRaises(OverflowError, pollster.poll, -1 << 31)
+        self.assertRaises(OverflowError, pollster.poll, -1 << 64)
 
-        # type error for invalid arguments
-        p = select.poll()
-        self.assertRaises(TypeError, p.register, p)
-        self.assertRaises(TypeError, p.unregister, p)
-
-        # can't unregister non-existent object
-        p = select.poll()
-        self.assertRaises(KeyError, p.unregister, 3)
-
-        # Test error cases
-        pollster = select.poll()
-        class Nope:
-            pass
-
-        class Almost:
-            def fileno(self):
-                return 'fileno'
-
-        self.assertRaises(TypeError, pollster.register, Nope(), 0)
-        self.assertRaises(TypeError, pollster.register, Almost(), 0)
-
-    # Another test case for poll().  This is copied from the test case for
-    # select(), modified to use poll() instead.
-
-    def test_poll2(self):
-        cmd = 'for i in 0 1 2 3 4 5 6 7 8 9; do echo testing...; sleep 1; done'
-        p = os.popen(cmd, 'r')
-        pollster = select.poll()
-        pollster.register( p, select.POLLIN )
-        for tout in (0, 1000, 2000, 4000, 8000, 16000) + (-1,)*10:
-            fdlist = pollster.poll(tout)
-            if (fdlist == []):
-                continue
-            fd, flags = fdlist[0]
-            if flags & select.POLLHUP:
-                line = p.readline()
-                if line != "":
-                    self.fail('error: pipe seems to be closed, but still returns data')
-                continue
-
-            elif flags & select.POLLIN:
-                line = p.readline()
-                if not line:
-                    break
-                continue
-            else:
-                self.fail('Unexpected return value from select.poll: %s' % fdlist)
-        p.close()
-
-    def test_poll3(self):
-        # test int overflow
-        pollster = select.poll()
-        pollster.register(1)
-
+        pollster.poll(0)
+        pollster.poll(1)
+        pollster.poll(1 << 30)
+        self.assertRaises(OverflowError, pollster.poll, 1 << 31)
+        self.assertRaises(OverflowError, pollster.poll, 1 << 63)
         self.assertRaises(OverflowError, pollster.poll, 1 << 64)
 
-        x = 2 + 3
-        if x != 5:
-            self.fail('Overflow must have occurred')
-
 def test_main():
-    run_unittest(PollTests)
+    run_unittest(DevPollTests)
 
 if __name__ == '__main__':
     test_main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -365,6 +365,9 @@
 Library
 -------
 
+- Issue #6397: Support "/dev/poll" polling objects in select module,
+  under Solaris & derivatives.
+
 - Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly
   handles non-valid attributes, including adjacent and unquoted attributes.
 
diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c
--- a/Modules/selectmodule.c
+++ b/Modules/selectmodule.c
@@ -7,6 +7,14 @@
 #include "Python.h"
 #include <structmember.h>
 
+#ifdef HAVE_SYS_DEVPOLL_H
+#include <sys/resource.h>
+#include <sys/devpoll.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#endif
+
 #ifdef __APPLE__
     /* Perform runtime testing for a broken poll on OSX to make it easier
      * to use the same binary on multiple releases of the OS.
@@ -648,6 +656,339 @@
     poll_methods,               /*tp_methods*/
 };
 
+#ifdef HAVE_SYS_DEVPOLL_H
+typedef struct {
+    PyObject_HEAD
+    int fd_devpoll;
+    int max_n_fds;
+    int n_fds;
+    struct pollfd *fds;
+} devpollObject;
+
+static PyTypeObject devpoll_Type;
+
+static int devpoll_flush(devpollObject *self)
+{
+    int size, n;
+
+    if (!self->n_fds) return 0;
+
+    size = sizeof(struct pollfd)*self->n_fds;
+    self->n_fds = 0;
+
+    Py_BEGIN_ALLOW_THREADS
+    n = write(self->fd_devpoll, self->fds, size);
+    Py_END_ALLOW_THREADS
+
+    if (n == -1 ) {
+        PyErr_SetFromErrno(PyExc_IOError);
+        return -1;
+    }
+    if (n < size) {
+        /*
+        ** Data writed to /dev/poll is a binary data structure. It is not
+        ** clear what to do if a partial write occurred. For now, raise
+        ** an exception and see if we actually found this problem in
+        ** the wild.
+        ** See http://bugs.python.org/issue6397.
+        */
+        PyErr_Format(PyExc_IOError, "failed to write all pollfds. "
+                "Please, report at http://bugs.python.org/. "
+                "Data to report: Size tried: %d, actual size written: %d.",
+                size, n);
+        return -1;
+    }
+    return 0;
+}
+
+static PyObject *
+internal_devpoll_register(devpollObject *self, PyObject *args, int remove)
+{
+    PyObject *o;
+    int fd, events = POLLIN | POLLPRI | POLLOUT;
+
+    if (!PyArg_ParseTuple(args, "O|i:register", &o, &events)) {
+        return NULL;
+    }
+
+    fd = PyObject_AsFileDescriptor(o);
+    if (fd == -1) return NULL;
+
+    if (remove) {
+        self->fds[self->n_fds].fd = fd;
+        self->fds[self->n_fds].events = POLLREMOVE;
+
+        if (++self->n_fds == self->max_n_fds) {
+            if (devpoll_flush(self))
+                return NULL;
+        }
+    }
+
+    self->fds[self->n_fds].fd = fd;
+    self->fds[self->n_fds].events = events;
+
+    if (++self->n_fds == self->max_n_fds) {
+        if (devpoll_flush(self))
+            return NULL;
+    }
+
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(devpoll_register_doc,
+"register(fd [, eventmask] ) -> None\n\n\
+Register a file descriptor with the polling object.\n\
+fd -- either an integer, or an object with a fileno() method returning an\n\
+      int.\n\
+events -- an optional bitmask describing the type of events to check for");
+
+static PyObject *
+devpoll_register(devpollObject *self, PyObject *args)
+{
+    return internal_devpoll_register(self, args, 0);
+}
+
+PyDoc_STRVAR(devpoll_modify_doc,
+"modify(fd[, eventmask]) -> None\n\n\
+Modify a possible already registered file descriptor.\n\
+fd -- either an integer, or an object with a fileno() method returning an\n\
+      int.\n\
+events -- an optional bitmask describing the type of events to check for");
+
+static PyObject *
+devpoll_modify(devpollObject *self, PyObject *args)
+{
+    return internal_devpoll_register(self, args, 1);
+}
+
+
+PyDoc_STRVAR(devpoll_unregister_doc,
+"unregister(fd) -> None\n\n\
+Remove a file descriptor being tracked by the polling object.");
+
+static PyObject *
+devpoll_unregister(devpollObject *self, PyObject *o)
+{
+    int fd;
+
+    fd = PyObject_AsFileDescriptor( o );
+    if (fd == -1)
+        return NULL;
+
+    self->fds[self->n_fds].fd = fd;
+    self->fds[self->n_fds].events = POLLREMOVE;
+
+    if (++self->n_fds == self->max_n_fds) {
+        if (devpoll_flush(self))
+            return NULL;
+    }
+
+    Py_RETURN_NONE;
+}
+
+PyDoc_STRVAR(devpoll_poll_doc,
+"poll( [timeout] ) -> list of (fd, event) 2-tuples\n\n\
+Polls the set of registered file descriptors, returning a list containing \n\
+any descriptors that have events or errors to report.");
+
+static PyObject *
+devpoll_poll(devpollObject *self, PyObject *args)
+{
+    struct dvpoll dvp;
+    PyObject *result_list = NULL, *tout = NULL;
+    int poll_result, i;
+    long timeout;
+    PyObject *value, *num1, *num2;
+
+    if (!PyArg_UnpackTuple(args, "poll", 0, 1, &tout)) {
+        return NULL;
+    }
+
+    /* Check values for timeout */
+    if (tout == NULL || tout == Py_None)
+        timeout = -1;
+    else if (!PyNumber_Check(tout)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "timeout must be an integer or None");
+        return NULL;
+    }
+    else {
+        tout = PyNumber_Long(tout);
+        if (!tout)
+            return NULL;
+        timeout = PyLong_AsLong(tout);
+        Py_DECREF(tout);
+        if (timeout == -1 && PyErr_Occurred())
+            return NULL;
+    }
+
+    if ((timeout < -1) || (timeout > INT_MAX)) {
+        PyErr_SetString(PyExc_OverflowError,
+                        "timeout is out of range");
+        return NULL;
+    }
+
+    if (devpoll_flush(self))
+        return NULL;
+
+    dvp.dp_fds = self->fds;
+    dvp.dp_nfds = self->max_n_fds;
+    dvp.dp_timeout = timeout;
+
+    /* call devpoll() */
+    Py_BEGIN_ALLOW_THREADS
+    poll_result = ioctl(self->fd_devpoll, DP_POLL, &dvp);
+    Py_END_ALLOW_THREADS
+
+    if (poll_result < 0) {
+        PyErr_SetFromErrno(PyExc_IOError);
+        return NULL;
+    }
+
+    /* build the result list */
+
+    result_list = PyList_New(poll_result);
+    if (!result_list)
+        return NULL;
+    else {
+        for (i = 0; i < poll_result; i++) {
+            num1 = PyLong_FromLong(self->fds[i].fd);
+            num2 = PyLong_FromLong(self->fds[i].revents);
+            if ((num1 == NULL) || (num2 == NULL)) {
+                Py_XDECREF(num1);
+                Py_XDECREF(num2);
+                goto error;
+            }
+            value = PyTuple_Pack(2, num1, num2);
+            Py_DECREF(num1);
+            Py_DECREF(num2);
+            if (value == NULL)
+                goto error;
+            if ((PyList_SetItem(result_list, i, value)) == -1) {
+                Py_DECREF(value);
+                goto error;
+            }
+        }
+    }
+
+    return result_list;
+
+  error:
+    Py_DECREF(result_list);
+    return NULL;
+}
+
+static PyMethodDef devpoll_methods[] = {
+    {"register",        (PyCFunction)devpoll_register,
+     METH_VARARGS,  devpoll_register_doc},
+    {"modify",          (PyCFunction)devpoll_modify,
+     METH_VARARGS,  devpoll_modify_doc},
+    {"unregister",      (PyCFunction)devpoll_unregister,
+     METH_O,        devpoll_unregister_doc},
+    {"poll",            (PyCFunction)devpoll_poll,
+     METH_VARARGS,  devpoll_poll_doc},
+    {NULL,              NULL}           /* sentinel */
+};
+
+static devpollObject *
+newDevPollObject(void)
+{
+    devpollObject *self;
+    int fd_devpoll, limit_result;
+    struct pollfd *fds;
+    struct rlimit limit;
+
+    Py_BEGIN_ALLOW_THREADS
+    /*
+    ** If we try to process more that getrlimit()
+    ** fds, the kernel will give an error, so
+    ** we set the limit here. It is a dynamic
+    ** value, because we can change rlimit() anytime.
+    */
+    limit_result = getrlimit(RLIMIT_NOFILE, &limit);
+    if (limit_result != -1)
+        fd_devpoll = open("/dev/poll", O_RDWR);
+    Py_END_ALLOW_THREADS
+
+    if (limit_result == -1) {
+        PyErr_SetFromErrno(PyExc_OSError);
+        return NULL;
+    }
+    if (fd_devpoll == -1) {
+        PyErr_SetFromErrnoWithFilename(PyExc_IOError, "/dev/poll");
+        return NULL;
+    }
+
+    fds = PyMem_NEW(struct pollfd, limit.rlim_cur);
+    if (fds == NULL) {
+        close(fd_devpoll);
+        PyErr_NoMemory();
+        return NULL;
+    }
+
+    self = PyObject_New(devpollObject, &devpoll_Type);
+    if (self == NULL) {
+        close(fd_devpoll);
+        PyMem_DEL(fds);
+        return NULL;
+    }
+    self->fd_devpoll = fd_devpoll;
+    self->max_n_fds = limit.rlim_cur;
+    self->n_fds = 0;
+    self->fds = fds;
+
+    return self;
+}
+
+static void
+devpoll_dealloc(devpollObject *self)
+{
+    Py_BEGIN_ALLOW_THREADS
+    close(self->fd_devpoll);
+    Py_END_ALLOW_THREADS
+
+    PyMem_DEL(self->fds);
+
+    PyObject_Del(self);
+}
+
+static PyTypeObject devpoll_Type = {
+    /* The ob_type field must be initialized in the module init function
+     * to be portable to Windows without using C++. */
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "select.devpoll",           /*tp_name*/
+    sizeof(devpollObject),      /*tp_basicsize*/
+    0,                          /*tp_itemsize*/
+    /* methods */
+    (destructor)devpoll_dealloc, /*tp_dealloc*/
+    0,                          /*tp_print*/
+    0,                          /*tp_getattr*/
+    0,                          /*tp_setattr*/
+    0,                          /*tp_reserved*/
+    0,                          /*tp_repr*/
+    0,                          /*tp_as_number*/
+    0,                          /*tp_as_sequence*/
+    0,                          /*tp_as_mapping*/
+    0,                          /*tp_hash*/
+    0,                          /*tp_call*/
+    0,                          /*tp_str*/
+    0,                          /*tp_getattro*/
+    0,                          /*tp_setattro*/
+    0,                          /*tp_as_buffer*/
+    Py_TPFLAGS_DEFAULT,         /*tp_flags*/
+    0,                          /*tp_doc*/
+    0,                          /*tp_traverse*/
+    0,                          /*tp_clear*/
+    0,                          /*tp_richcompare*/
+    0,                          /*tp_weaklistoffset*/
+    0,                          /*tp_iter*/
+    0,                          /*tp_iternext*/
+    devpoll_methods,            /*tp_methods*/
+};
+#endif  /* HAVE_SYS_DEVPOLL_H */
+
+
+
 PyDoc_STRVAR(poll_doc,
 "Returns a polling object, which supports registering and\n\
 unregistering file descriptors, and then polling them for I/O events.");
@@ -658,6 +999,19 @@
     return (PyObject *)newPollObject();
 }
 
+#ifdef HAVE_SYS_DEVPOLL_H
+PyDoc_STRVAR(devpoll_doc,
+"Returns a polling object, which supports registering and\n\
+unregistering file descriptors, and then polling them for I/O events.");
+
+static PyObject *
+select_devpoll(PyObject *self, PyObject *unused)
+{
+    return (PyObject *)newDevPollObject();
+}
+#endif
+
+
 #ifdef __APPLE__
 /*
  * On some systems poll() sets errno on invalid file descriptors. We test
@@ -1715,6 +2069,11 @@
 };
 
 #endif /* HAVE_KQUEUE */
+
+
+
+
+
 /* ************************************************************************ */
 
 PyDoc_STRVAR(select_doc,
@@ -1746,6 +2105,9 @@
 #ifdef HAVE_POLL
     {"poll",            select_poll,    METH_NOARGS,    poll_doc},
 #endif /* HAVE_POLL */
+#ifdef HAVE_SYS_DEVPOLL_H
+    {"devpoll",         select_devpoll, METH_NOARGS,    devpoll_doc},
+#endif
     {0,         0},     /* sentinel */
 };
 
@@ -1768,6 +2130,9 @@
     NULL
 };
 
+
+
+
 PyMODINIT_FUNC
 PyInit_select(void)
 {
@@ -1824,6 +2189,11 @@
     }
 #endif /* HAVE_POLL */
 
+#ifdef HAVE_SYS_DEVPOLL_H
+    if (PyType_Ready(&devpoll_Type) < 0)
+        return NULL;
+#endif
+
 #ifdef HAVE_EPOLL
     Py_TYPE(&pyEpoll_Type) = &PyType_Type;
     if (PyType_Ready(&pyEpoll_Type) < 0)
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -6139,12 +6139,13 @@
 
 for ac_header in asm/types.h conio.h curses.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
-ieeefp.h io.h langinfo.h libintl.h ncurses.h poll.h process.h pthread.h \
+ieeefp.h io.h langinfo.h libintl.h ncurses.h process.h pthread.h \
 sched.h shadow.h signal.h stdint.h stropts.h termios.h \
 unistd.h utime.h \
-sys/audioio.h sys/xattr.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \
+poll.h sys/devpoll.h sys/epoll.h sys/poll.h \
+sys/audioio.h sys/xattr.h sys/bsdtty.h sys/event.h sys/file.h sys/loadavg.h \
 sys/lock.h sys/mkdev.h sys/modem.h \
-sys/param.h sys/poll.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
+sys/param.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
 sys/stat.h sys/termio.h sys/time.h \
 sys/times.h sys/types.h sys/uio.h sys/un.h sys/utsname.h sys/wait.h pty.h \
 libutil.h sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -1329,12 +1329,13 @@
 AC_HEADER_STDC
 AC_CHECK_HEADERS(asm/types.h conio.h curses.h direct.h dlfcn.h errno.h \
 fcntl.h grp.h \
-ieeefp.h io.h langinfo.h libintl.h ncurses.h poll.h process.h pthread.h \
+ieeefp.h io.h langinfo.h libintl.h ncurses.h process.h pthread.h \
 sched.h shadow.h signal.h stdint.h stropts.h termios.h \
 unistd.h utime.h \
-sys/audioio.h sys/xattr.h sys/bsdtty.h sys/epoll.h sys/event.h sys/file.h sys/loadavg.h \
+poll.h sys/devpoll.h sys/epoll.h sys/poll.h \
+sys/audioio.h sys/xattr.h sys/bsdtty.h sys/event.h sys/file.h sys/loadavg.h \
 sys/lock.h sys/mkdev.h sys/modem.h \
-sys/param.h sys/poll.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
+sys/param.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \
 sys/stat.h sys/termio.h sys/time.h \
 sys/times.h sys/types.h sys/uio.h sys/un.h sys/utsname.h sys/wait.h pty.h \
 libutil.h sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -883,6 +883,9 @@
 /* Define to 1 if you have the <sys/bsdtty.h> header file. */
 #undef HAVE_SYS_BSDTTY_H
 
+/* Define to 1 if you have the <sys/devpoll.h> header file. */
+#undef HAVE_SYS_DEVPOLL_H
+
 /* Define to 1 if you have the <sys/dir.h> header file, and it defines `DIR'.
    */
 #undef HAVE_SYS_DIR_H

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 14 20:49:10 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 14 Nov 2011 20:49:10 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzc3MzI6?=
 =?utf8?q?_Try_to_fix_the_a_failing_test_on_Windows?=
Message-ID: <E1RQ2Wk-00076N-F5@dinsdale.python.org>

http://hg.python.org/cpython/rev/555871844962
changeset:   73550:555871844962
branch:      2.7
parent:      73545:3c3009f63700
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 14 20:50:36 2011 +0100
summary:
  Issue #7732: Try to fix the a failing test on Windows

It doesn't matter if imp.find_module() fails with ImportError or IOError, but
it should not crash.

files:
  Lib/test/test_import.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_import.py b/Lib/test/test_import.py
--- a/Lib/test/test_import.py
+++ b/Lib/test/test_import.py
@@ -269,7 +269,8 @@
         source = TESTFN + '.py'
         os.mkdir(source)
         try:
-            self.assertRaises(IOError, imp.find_module, TESTFN, ["."])
+            self.assertRaises((ImportError, IOError),
+                              imp.find_module, TESTFN, ["."])
         finally:
             os.rmdir(source)
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 00:10:25 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 00:10:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313389=3A_Full_garb?=
 =?utf8?q?age_collection_passes_now_clear_the_freelists_for?=
Message-ID: <E1RQ5fV-0001X2-3u@dinsdale.python.org>

http://hg.python.org/cpython/rev/910986542a75
changeset:   73551:910986542a75
parent:      73549:8f7ab4bf7ad9
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 00:00:12 2011 +0100
summary:
  Issue #13389: Full garbage collection passes now clear the freelists for
list and dict objects.  They already cleared other freelists in the
interpreter.

files:
  Doc/c-api/dict.rst   |   7 +++++++
  Doc/c-api/list.rst   |   7 +++++++
  Include/dictobject.h |   2 ++
  Include/listobject.h |   2 ++
  Misc/NEWS            |   4 ++++
  Modules/gcmodule.c   |   2 ++
  Objects/dictobject.c |  13 ++++++++++---
  Objects/listobject.c |  13 ++++++++++---
  8 files changed, 44 insertions(+), 6 deletions(-)


diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst
--- a/Doc/c-api/dict.rst
+++ b/Doc/c-api/dict.rst
@@ -209,3 +209,10 @@
           for key, value in seq2:
               if override or key not in a:
                   a[key] = value
+
+
+.. c:function:: int PyDict_ClearFreeList()
+
+   Clear the free list. Return the total number of freed items.
+
+   .. versionadded:: 3.3
diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -142,3 +142,10 @@
 
    Return a new tuple object containing the contents of *list*; equivalent to
    ``tuple(list)``.
+
+
+.. c:function:: int PyList_ClearFreeList()
+
+   Clear the free list. Return the total number of freed items.
+
+   .. versionadded:: 3.3
diff --git a/Include/dictobject.h b/Include/dictobject.h
--- a/Include/dictobject.h
+++ b/Include/dictobject.h
@@ -129,6 +129,8 @@
 PyAPI_FUNC(PyObject *) _PyDict_NewPresized(Py_ssize_t minused);
 PyAPI_FUNC(void) _PyDict_MaybeUntrack(PyObject *mp);
 PyAPI_FUNC(int) _PyDict_HasOnlyStringKeys(PyObject *mp);
+
+PyAPI_FUNC(int) PyDict_ClearFreeList(void);
 #endif
 
 /* PyDict_Update(mp, other) is equivalent to PyDict_Merge(mp, other, 1). */
diff --git a/Include/listobject.h b/Include/listobject.h
--- a/Include/listobject.h
+++ b/Include/listobject.h
@@ -62,6 +62,8 @@
 PyAPI_FUNC(PyObject *) PyList_AsTuple(PyObject *);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) _PyList_Extend(PyListObject *, PyObject *);
+
+PyAPI_FUNC(int) PyList_ClearFreeList(void);
 #endif
 
 /* Macro, trading safety for speed */
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13389: Full garbage collection passes now clear the freelists for
+  list and dict objects.  They already cleared other freelists in the
+  interpreter.
+
 - Issue #13327: Remove the need for an explicit None as the second argument
   to os.utime, os.lutimes, os.futimes, os.futimens, os.futimesat, in
   order to update to the current time. Also added keyword argument
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -762,6 +762,8 @@
     (void)PyTuple_ClearFreeList();
     (void)PyUnicode_ClearFreeList();
     (void)PyFloat_ClearFreeList();
+    (void)PyList_ClearFreeList();
+    (void)PyDict_ClearFreeList();
 }
 
 static double
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -217,16 +217,23 @@
 static PyDictObject *free_list[PyDict_MAXFREELIST];
 static int numfree = 0;
 
-void
-PyDict_Fini(void)
+int
+PyDict_ClearFreeList(void)
 {
     PyDictObject *op;
-
+    int ret = numfree;
     while (numfree) {
         op = free_list[--numfree];
         assert(PyDict_CheckExact(op));
         PyObject_GC_Del(op);
     }
+    return ret;
+}
+
+void
+PyDict_Fini(void)
+{
+    PyDict_ClearFreeList();
 }
 
 PyObject *
diff --git a/Objects/listobject.c b/Objects/listobject.c
--- a/Objects/listobject.c
+++ b/Objects/listobject.c
@@ -97,16 +97,23 @@
 static PyListObject *free_list[PyList_MAXFREELIST];
 static int numfree = 0;
 
-void
-PyList_Fini(void)
+int
+PyList_ClearFreeList(void)
 {
     PyListObject *op;
-
+    int ret = numfree;
     while (numfree) {
         op = free_list[--numfree];
         assert(PyList_CheckExact(op));
         PyObject_GC_Del(op);
     }
+    return ret;
+}
+
+void
+PyList_Fini(void)
+{
+    PyList_ClearFreeList();
 }
 
 PyObject *

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 01:54:55 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 01:54:55 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzMz?=
 =?utf8?q?=3A_The_UTF-7_decoder_now_accepts_lone_surrogates?=
Message-ID: <E1RQ7Id-00017T-Kw@dinsdale.python.org>

http://hg.python.org/cpython/rev/ddfcb0de564f
changeset:   73552:ddfcb0de564f
branch:      3.2
parent:      73547:16ed15ff0d7c
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 01:42:21 2011 +0100
summary:
  Issue #13333: The UTF-7 decoder now accepts lone surrogates
(the encoder already accepts them).

files:
  Lib/test/test_unicode.py |  14 +++++++++++---
  Misc/NEWS                |   3 +++
  Objects/unicodeobject.c  |  14 +++++---------
  3 files changed, 19 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1091,10 +1091,18 @@
         for (x, y) in utfTests:
             self.assertEqual(x.encode('utf-7'), y)
 
-        # Unpaired surrogates not supported
-        self.assertRaises(UnicodeError, str, b'+3ADYAA-', 'utf-7')
+        # Unpaired surrogates are passed through
+        self.assertEqual('\uD801'.encode('utf-7'), b'+2AE-')
+        self.assertEqual('\uD801x'.encode('utf-7'), b'+2AE-x')
+        self.assertEqual('\uDC01'.encode('utf-7'), b'+3AE-')
+        self.assertEqual('\uDC01x'.encode('utf-7'), b'+3AE-x')
+        self.assertEqual(b'+2AE-'.decode('utf-7'), '\uD801')
+        self.assertEqual(b'+2AE-x'.decode('utf-7'), '\uD801x')
+        self.assertEqual(b'+3AE-'.decode('utf-7'), '\uDC01')
+        self.assertEqual(b'+3AE-x'.decode('utf-7'), '\uDC01x')
 
-        self.assertEqual(str(b'+3ADYAA-', 'utf-7', 'replace'), '\ufffd\ufffd')
+        self.assertEqual('\uD801\U000abcde'.encode('utf-7'), b'+2AHab9ze-')
+        self.assertEqual(b'+2AHab9ze-'.decode('utf-7'), '\uD801\U000abcde')
 
         # Issue #2242: crash on some Windows/MSVC versions
         self.assertEqual(b'+\xc1'.decode('utf-7'), '\xc1')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder
+  already accepts them).
+
 - Issue #13342: input() used to ignore sys.stdin's and sys.stdout's unicode
   error handler in interactive mode (when calling into PyOS_Readline()).
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -2282,21 +2282,17 @@
                             *p++ = outCh;
 #endif
                             surrogate = 0;
+                            continue;
                         }
                         else {
+                            *p++ = surrogate;
                             surrogate = 0;
-                            errmsg = "second surrogate missing";
-                            goto utf7Error;
                         }
                     }
-                    else if (outCh >= 0xD800 && outCh <= 0xDBFF) {
+                    if (outCh >= 0xD800 && outCh <= 0xDBFF) {
                         /* first surrogate */
                         surrogate = outCh;
                     }
-                    else if (outCh >= 0xDC00 && outCh <= 0xDFFF) {
-                        errmsg = "unexpected second surrogate";
-                        goto utf7Error;
-                    }
                     else {
                         *p++ = outCh;
                     }
@@ -2306,8 +2302,8 @@
                 inShift = 0;
                 s++;
                 if (surrogate) {
-                    errmsg = "second surrogate missing at end of shift sequence";
-                    goto utf7Error;
+                    *p++ = surrogate;
+                    surrogate = 0;
                 }
                 if (base64bits > 0) { /* left-over bits */
                     if (base64bits >= 6) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 01:54:56 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 01:54:56 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313333=3A_The_UTF-7_decoder_now_accepts_lone_surroga?=
 =?utf8?q?tes?=
Message-ID: <E1RQ7Ie-00017X-Fr@dinsdale.python.org>

http://hg.python.org/cpython/rev/250091e60f28
changeset:   73553:250091e60f28
parent:      73551:910986542a75
parent:      73552:ddfcb0de564f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 01:44:16 2011 +0100
summary:
  Issue #13333: The UTF-7 decoder now accepts lone surrogates
(the encoder already accepts them).

files:
  Lib/test/test_unicode.py |  14 +++++++++++---
  Misc/NEWS                |   3 +++
  Objects/unicodeobject.c  |  16 +++++++---------
  3 files changed, 21 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1108,10 +1108,18 @@
         for (x, y) in utfTests:
             self.assertEqual(x.encode('utf-7'), y)
 
-        # Unpaired surrogates not supported
-        self.assertRaises(UnicodeError, str, b'+3ADYAA-', 'utf-7')
+        # Unpaired surrogates are passed through
+        self.assertEqual('\uD801'.encode('utf-7'), b'+2AE-')
+        self.assertEqual('\uD801x'.encode('utf-7'), b'+2AE-x')
+        self.assertEqual('\uDC01'.encode('utf-7'), b'+3AE-')
+        self.assertEqual('\uDC01x'.encode('utf-7'), b'+3AE-x')
+        self.assertEqual(b'+2AE-'.decode('utf-7'), '\uD801')
+        self.assertEqual(b'+2AE-x'.decode('utf-7'), '\uD801x')
+        self.assertEqual(b'+3AE-'.decode('utf-7'), '\uDC01')
+        self.assertEqual(b'+3AE-x'.decode('utf-7'), '\uDC01x')
 
-        self.assertEqual(str(b'+3ADYAA-', 'utf-7', 'replace'), '\ufffd\ufffd')
+        self.assertEqual('\uD801\U000abcde'.encode('utf-7'), b'+2AHab9ze-')
+        self.assertEqual(b'+2AHab9ze-'.decode('utf-7'), '\uD801\U000abcde')
 
         # Issue #2242: crash on some Windows/MSVC versions
         self.assertEqual(b'+\xc1'.decode('utf-7'), '\xc1')
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder
+  already accepts them).
+
 - Issue #13389: Full garbage collection passes now clear the freelists for
   list and dict objects.  They already cleared other freelists in the
   interpreter.
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3884,21 +3884,18 @@
                             if (unicode_putchar(&unicode, &outpos, ch2) < 0)
                                 goto onError;
                             surrogate = 0;
+                            continue;
                         }
                         else {
+                            if (unicode_putchar(&unicode, &outpos, surrogate) < 0)
+                                goto onError;
                             surrogate = 0;
-                            errmsg = "second surrogate missing";
-                            goto utf7Error;
                         }
                     }
-                    else if (outCh >= 0xD800 && outCh <= 0xDBFF) {
+                    if (outCh >= 0xD800 && outCh <= 0xDBFF) {
                         /* first surrogate */
                         surrogate = outCh;
                     }
-                    else if (outCh >= 0xDC00 && outCh <= 0xDFFF) {
-                        errmsg = "unexpected second surrogate";
-                        goto utf7Error;
-                    }
                     else {
                         if (unicode_putchar(&unicode, &outpos, outCh) < 0)
                             goto onError;
@@ -3909,8 +3906,9 @@
                 inShift = 0;
                 s++;
                 if (surrogate) {
-                    errmsg = "second surrogate missing at end of shift sequence";
-                    goto utf7Error;
+                    if (unicode_putchar(&unicode, &outpos, surrogate) < 0)
+                        goto onError;
+                    surrogate = 0;
                 }
                 if (base64bits > 0) { /* left-over bits */
                     if (base64bits >= 6) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 01:54:57 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 01:54:57 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMzMz?=
 =?utf8?q?=3A_The_UTF-7_decoder_now_accepts_lone_surrogates?=
Message-ID: <E1RQ7If-00017b-9s@dinsdale.python.org>

http://hg.python.org/cpython/rev/050772822bde
changeset:   73554:050772822bde
branch:      2.7
parent:      73550:555871844962
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 01:49:40 2011 +0100
summary:
  Issue #13333: The UTF-7 decoder now accepts lone surrogates
(the encoder already accepts them).

files:
  Lib/test/test_unicode.py |  14 +++++++++++---
  Misc/NEWS                |   3 +++
  Objects/unicodeobject.c  |  14 +++++---------
  3 files changed, 19 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -771,10 +771,18 @@
         for (x, y) in utfTests:
             self.assertEqual(x.encode('utf-7'), y)
 
-        # Unpaired surrogates not supported
-        self.assertRaises(UnicodeError, unicode, '+3ADYAA-', 'utf-7')
+        # Unpaired surrogates are passed through
+        self.assertEqual(u'\uD801'.encode('utf-7'), '+2AE-')
+        self.assertEqual(u'\uD801x'.encode('utf-7'), '+2AE-x')
+        self.assertEqual(u'\uDC01'.encode('utf-7'), '+3AE-')
+        self.assertEqual(u'\uDC01x'.encode('utf-7'), '+3AE-x')
+        self.assertEqual('+2AE-'.decode('utf-7'), u'\uD801')
+        self.assertEqual('+2AE-x'.decode('utf-7'), u'\uD801x')
+        self.assertEqual('+3AE-'.decode('utf-7'), u'\uDC01')
+        self.assertEqual('+3AE-x'.decode('utf-7'), u'\uDC01x')
 
-        self.assertEqual(unicode('+3ADYAA-', 'utf-7', 'replace'), u'\ufffd\ufffd')
+        self.assertEqual(u'\uD801\U000abcde'.encode('utf-7'), '+2AHab9ze-')
+        self.assertEqual('+2AHab9ze-'.decode('utf-7'), u'\uD801\U000abcde')
 
         # Direct encoded characters
         set_d = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'(),-./:?"
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -9,6 +9,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder
+  already accepts them).
+
 - Remove Py3k warning for callable.
 
 - Issue #10519: Avoid unnecessary recursive function calls in
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1628,21 +1628,17 @@
                             *p++ = outCh;
 #endif
                             surrogate = 0;
+                            continue;
                         }
                         else {
+                            *p++ = surrogate;
                             surrogate = 0;
-                            errmsg = "second surrogate missing";
-                            goto utf7Error;
                         }
                     }
-                    else if (outCh >= 0xD800 && outCh <= 0xDBFF) {
+                    if (outCh >= 0xD800 && outCh <= 0xDBFF) {
                         /* first surrogate */
                         surrogate = outCh;
                     }
-                    else if (outCh >= 0xDC00 && outCh <= 0xDFFF) {
-                        errmsg = "unexpected second surrogate";
-                        goto utf7Error;
-                    }
                     else {
                         *p++ = outCh;
                     }
@@ -1652,8 +1648,8 @@
                 inShift = 0;
                 s++;
                 if (surrogate) {
-                    errmsg = "second surrogate missing at end of shift sequence";
-                    goto utf7Error;
+                    *p++ = surrogate;
+                    surrogate = 0;
                 }
                 if (base64bits > 0) { /* left-over bits */
                     if (base64bits >= 6) {

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue Nov 15 05:34:55 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 15 Nov 2011 05:34:55 +0100
Subject: [Python-checkins] Daily reference leaks (250091e60f28): sum=0
Message-ID: <E1RQAjX-0004pi-LK@ap.vmr.nerim.net>

results for 250091e60f28 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogJvatB_', '-x']

From python-checkins at python.org  Tue Nov 15 05:43:05 2011
From: python-checkins at python.org (jesus.cea)
Date: Tue, 15 Nov 2011 05:43:05 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_reST_fixes_for_/dev/poll?=
Message-ID: <E1RQArR-0002WP-19@dinsdale.python.org>

http://hg.python.org/cpython/rev/ff7acde0faad
changeset:   73555:ff7acde0faad
parent:      73553:250091e60f28
user:        Jesus Cea <jcea at jcea.es>
date:        Tue Nov 15 05:42:59 2011 +0100
summary:
  reST fixes for /dev/poll

files:
  Doc/library/select.rst |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/select.rst b/Doc/library/select.rst
--- a/Doc/library/select.rst
+++ b/Doc/library/select.rst
@@ -26,6 +26,7 @@
 
 
 .. function:: devpoll()
+
    (Only supported on Solaris and derivatives.)  Returns a ``/dev/poll``
    polling object; see section :ref:`devpoll-objects` below for the
    methods supported by devpoll objects.
@@ -33,7 +34,7 @@
    :c:func:`devpoll` objects are linked to the number of file
    descriptors allowed at the time of instantiation. If your program
    reduces this value, :c:func:`devpoll` will fail. If your program
-   increases this value, c:func:`devpoll` may return an
+   increases this value, :c:func:`devpoll` may return an
    incomplete list of active file descriptors.
 
    .. versionadded:: 3.3

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 11:06:50 2011
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 15 Nov 2011 11:06:50 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Some_corrections_and_clarifica?=
 =?utf8?q?tions_regarding_the_changes_between_2=2Ex_and_3=2Ex=2E?=
Message-ID: <E1RQFuk-0003Dd-73@dinsdale.python.org>

http://hg.python.org/peps/rev/58c40f3cbef1
changeset:   3989:58c40f3cbef1
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue Nov 15 20:06:39 2011 +1000
summary:
  Some corrections and clarifications regarding the changes between 2.x and 3.x. Also, we celebrated Python's 20th birthday last PyCon...

files:
  pep-0404.txt |  49 +++++++++++++++++++++++++--------------
  1 files changed, 31 insertions(+), 18 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -56,16 +56,18 @@
 official Python 2.8 release, and why you should plan to migrate
 instead to Python 3.
 
-Python is (as of this writing) nearly 20 years old, and Guido and the
-community has learned a lot in those intervening years.  Guido's
+Python is (as of this writing) more than 20 years old, and Guido and the
+community have learned a lot in those intervening years.  Guido's
 original concept for Python 3 was to make changes to the language
 primarily to remove the warts that had grown in the preceding
 versions.  Python 3 was not to be a complete redesign, but instead an
-evolution of the language, and while maintaining backward
+evolution of the language, and while maintaining full backward
 compatibility with Python 2 was explicitly off-the-table, neither were
 gratuitous changes in syntax or semantics acceptable.  In most cases,
 Python 2 code can be translated fairly easily to Python 3, sometimes
-entirely mechanically by such tools as `2to3`_.
+entirely mechanically by such tools as `2to3`_ (there's also a non-trivial
+subset of the language that will run without modification on both 2.7 and
+3.x).
 
 Because maintaining multiple versions of Python is a significant drag
 on the resources of the Python developers, and because the
@@ -85,18 +87,20 @@
 Strings and bytes
 -----------------
 
-Python 2's basic original string type are called 8-bit strings, and
+Python 2's basic original strings are called 8-bit strings, and
 they play a dual role in Python 2 as both ASCII text and as byte
-arrays.  While Python 2 also has a unicode string type, the
+sequences.  While Python 2 also has a unicode string type, the
 fundamental ambiguity of the core string type, coupled with Python 2's
 default behavior of supporting automatic coercion from 8-bit strings
 to unicodes when the two are combined, often leads to `UnicodeError`s.
 Python 3's standard string type is a unicode, and Python 3 adds a
 bytes type, but critically, no automatic coercion between bytes and
-unicodes is provided.  Thus, the core interpreter, its I/O libraries,
-module names, etc. are clear in their distinction between unicode
-strings and bytes.  Python 3's unicode support even extends to the
-filesystem, so that non-ASCII file names are natively supported.
+unicodes is provided (the closest we get are a few text-based APIs that
+assume UTF-8 as the default encoding if no encoding is explicitly stated).
+Thus, the core interpreter, its I/O libraries, module names, etc. are clear
+in their distinction between unicode strings and bytes.  Python 3's unicode
+support even extends to the filesystem, so that non-ASCII file names are
+natively supported.
 
 This string/bytes clarity is often a source of difficulty in
 transitioning existing code to Python 3, because many third party
@@ -122,11 +126,19 @@
 
 Python 2 has two core class hierarchies, often called *classic
 classes* and *new-style classes*.  The latter allow for such things as
-inheriting from the built-in basic types.  However, confusion and
-inconsistencies between the two class types has led Python 3 to drop
-classic classes.  Now all classes in Python 3 are *new-style*
-(although that's a misnomer now).  There is no need to inherit from
-``object`` or set the default metatype to enable them.
+inheriting from the builtin basic types, support descriptor based tools
+like the ``property`` builtin and provide a generally more sane and coherent
+system for dealing with multiple inheritance.  Python 3 provided the
+opportunity to completely drop support for classic classes, so all classes
+in Python 3 automatically use the new-style semantics (although that's a
+misnomer now). There is no need to explicitly inherit from ``object`` or set
+the default metatype to enable them (in fact, setting a default metatype at
+the module level is no longer supported - the default metatype is always
+``object``).
+
+The mechanism for explicitly specifying a metaclass has also changed to use
+a ``metaclass`` keyword argument in the class header line rather than a
+``__metaclass__`` magic attribute in the class body.
 
 
 Multiple spellings
@@ -142,9 +154,10 @@
 Imports
 -------
 
-In Python 3, star imports (e.g. ``from x import *``) are only
-permitted in module level code.  Also, only absolute imports are
-supported.
+In Python 3, implicit relative imports within packages are no longer
+available - only absolute imports and explicit relative imports are
+supported. In addition, star imports (e.g. ``from x import *``) are only
+permitted in module level code.
 
 Also, some areas of the standard library have been reorganized to make
 the naming scheme more intuitive.  Some rarely used builtins have been

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Tue Nov 15 11:32:21 2011
From: python-checkins at python.org (nick.coghlan)
Date: Tue, 15 Nov 2011 11:32:21 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Grammar_fixes_in_the_unicode_v?=
	=?utf8?q?s_bytes_section_of_PEP_404?=
Message-ID: <E1RQGJR-0005Ud-Um@dinsdale.python.org>

http://hg.python.org/peps/rev/5aaa1fa3fde1
changeset:   3990:5aaa1fa3fde1
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Tue Nov 15 20:32:12 2011 +1000
summary:
  Grammar fixes in the unicode vs bytes section of PEP 404

files:
  pep-0404.txt |  15 ++++++++-------
  1 files changed, 8 insertions(+), 7 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -92,13 +92,14 @@
 sequences.  While Python 2 also has a unicode string type, the
 fundamental ambiguity of the core string type, coupled with Python 2's
 default behavior of supporting automatic coercion from 8-bit strings
-to unicodes when the two are combined, often leads to `UnicodeError`s.
-Python 3's standard string type is a unicode, and Python 3 adds a
-bytes type, but critically, no automatic coercion between bytes and
-unicodes is provided (the closest we get are a few text-based APIs that
-assume UTF-8 as the default encoding if no encoding is explicitly stated).
-Thus, the core interpreter, its I/O libraries, module names, etc. are clear
-in their distinction between unicode strings and bytes.  Python 3's unicode
+to unicode objects when the two are combined, often leads to
+`UnicodeError`s. Python 3's standard string type is Unicode based, and
+Python 3 adds a dedicated bytes type, but critically, no automatic coercion
+between bytes and unicode strings is provided. The closest the language gets
+to implicit coercion are a few text-based APIs that assume a default
+encoding (usually UTF-8) if no encoding is explicitly stated. Thus, the core
+interpreter, its I/O libraries, module names, etc. are clear in their
+distinction between unicode strings and bytes.  Python 3's unicode
 support even extends to the filesystem, so that non-ASCII file names are
 natively supported.
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Tue Nov 15 15:06:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 15:06:00 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_test_setup=2Ecfg_?=
 =?utf8?q?=28followup_to_e39d1b6f0856=29?=
Message-ID: <E1RQJeC-0000fx-HJ@dinsdale.python.org>

http://hg.python.org/distutils2/rev/3e65e4c748cb
changeset:   1253:3e65e4c748cb
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 11:27:33 2011 +0100
summary:
  Update test setup.cfg (followup to e39d1b6f0856)

files:
  distutils2/tests/test_config.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -120,7 +120,7 @@
     /usr/include/blitz
 extra_compile_args = -fPIC -O2
     -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
-    /DGECODE_VERSION='win32' -- sys.platform == 'win32'
+    /DGECODE_VERSION=win32 -- sys.platform == 'win32'
 language = cxx
 
 # corner case: if the parent package of an extension is declared but

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Nov 15 15:06:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 15:06:00 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_tests_to_check_initi?=
 =?utf8?q?al_contents_of_d2=2Edatabase_caches?=
Message-ID: <E1RQJeC-0000g1-I2@dinsdale.python.org>

http://hg.python.org/distutils2/rev/a27a2a237231
changeset:   1254:a27a2a237231
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 11:30:16 2011 +0100
summary:
  Add tests to check initial contents of d2.database caches

files:
  distutils2/tests/test_database.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py
--- a/distutils2/tests/test_database.py
+++ b/distutils2/tests/test_database.py
@@ -12,6 +12,7 @@
 from distutils2.tests.test_util import GlobTestCaseBase
 from distutils2.tests.support import requires_zlib
 
+import distutils2.database
 from distutils2.config import get_resources_dests
 from distutils2.errors import PackagingError
 from distutils2.metadata import Metadata
@@ -293,6 +294,12 @@
         sys.path.insert(0, self.fake_dists_path)
         self.addCleanup(sys.path.remove, self.fake_dists_path)
 
+    def test_caches(self):
+        # sanity check for internal caches
+        for name in ('_cache_name', '_cache_name_egg',
+                     '_cache_path', '_cache_path_egg'):
+            self.assertEqual(getattr(distutils2.database, name), {})
+
     def test_distinfo_dirname(self):
         # Given a name and a version, we expect the distinfo_dirname function
         # to return a standard distribution information directory name.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Nov 15 15:06:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 15:06:00 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Move_copies_of_stdlib_te?=
 =?utf8?q?st_code_from_tests=2E=5F=5Finit=5F=5F_to_runtests=2E?=
Message-ID: <E1RQJeC-0000fu-Go@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7df0c83e795f
changeset:   1252:7df0c83e795f
parent:      1250:7eb32e910b85
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 10:50:16 2011 +0100
summary:
  Move copies of stdlib test code from tests.__init__ to runtests.

I deleted captured_stdout because its usage felt clumsy to me; better to
use stdlib?s test.support.captured_stdout in packaging and just write
out the code for d2.

I checked that adding the import of d2.tests.unittest at the top level
did not change the coverage accuracy (see comment in the source and the
history of the file for more info).

files:
  distutils2/tests/__init__.py           |  107 +--------
  distutils2/tests/test_command_bdist.py |   12 +-
  distutils2/tests/test_command_sdist.py |   11 +-
  distutils2/tests/test_dist.py          |    1 -
  runtests.py                            |  164 +++++++++---
  5 files changed, 145 insertions(+), 150 deletions(-)


diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -7,24 +7,18 @@
 
 Utility code is included in distutils2.tests.support.
 
-Always import unittest from this module, it will be the right version
+Always import unittest from this module: it will be unittest from the
 standard library for packaging tests and unittest2 for distutils2 tests.
 """
 
 import os
 import sys
 import unittest2 as unittest
-from StringIO import StringIO
-
-# XXX move helpers to support, add tests for them, remove things that
-# duplicate test.support (or keep them for the backport; needs thinking)
-
-here = os.path.dirname(__file__) or os.curdir
-verbose = 1
 
 
 def test_suite():
     suite = unittest.TestSuite()
+    here = os.path.dirname(__file__) or os.curdir
     for fn in os.listdir(here):
         if fn.startswith("test") and fn.endswith(".py"):
             modname = "distutils2.tests." + fn[:-3]
@@ -32,100 +26,3 @@
             module = sys.modules[modname]
             suite.addTest(module.test_suite())
     return suite
-
-
-class Error(Exception):
-    """Base class for regression test exceptions."""
-
-
-class TestFailed(Error):
-    """Test failed."""
-
-
-class BasicTestRunner(object):
-    def run(self, test):
-        result = unittest.TestResult()
-        test(result)
-        return result
-
-
-def _run_suite(suite, verbose_=1):
-    """Run tests from a unittest.TestSuite-derived class."""
-    global verbose
-    verbose = verbose_
-    if verbose_:
-        runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
-    else:
-        runner = BasicTestRunner()
-
-    result = runner.run(suite)
-    if not result.wasSuccessful():
-        if len(result.errors) == 1 and not result.failures:
-            err = result.errors[0][1]
-        elif len(result.failures) == 1 and not result.errors:
-            err = result.failures[0][1]
-        else:
-            err = "errors occurred; run in verbose mode for details"
-        raise TestFailed(err)
-
-
-def run_unittest(classes, verbose_=1):
-    """Run tests from unittest.TestCase-derived classes.
-
-    Originally extracted from stdlib test.test_support and modified to
-    support unittest2.
-    """
-    valid_types = (unittest.TestSuite, unittest.TestCase)
-    suite = unittest.TestSuite()
-    for cls in classes:
-        if isinstance(cls, basestring):
-            if cls in sys.modules:
-                suite.addTest(unittest.findTestCases(sys.modules[cls]))
-            else:
-                raise ValueError("str arguments must be keys in sys.modules")
-        elif isinstance(cls, valid_types):
-            suite.addTest(cls)
-        else:
-            suite.addTest(unittest.makeSuite(cls))
-    _run_suite(suite, verbose_)
-
-
-def reap_children():
-    """Use this function at the end of test_main() whenever sub-processes
-    are started.  This will help ensure that no extra children (zombies)
-    stick around to hog resources and create problems when looking
-    for refleaks.
-
-    Extracted from stdlib test.support.
-    """
-
-    # Reap all our dead child processes so we don't leave zombies around.
-    # These hog resources and might be causing some of the buildbots to die.
-    if hasattr(os, 'waitpid'):
-        any_process = -1
-        while True:
-            try:
-                # This will raise an exception on Windows.  That's ok.
-                pid, status = os.waitpid(any_process, os.WNOHANG)
-                if pid == 0:
-                    break
-            except:
-                break
-
-
-def captured_stdout(func, *args, **kw):
-    orig_stdout = getattr(sys, 'stdout')
-    setattr(sys, 'stdout', StringIO())
-    try:
-        res = func(*args, **kw)
-        sys.stdout.seek(0)
-        return res, sys.stdout.read()
-    finally:
-        setattr(sys, 'stdout', orig_stdout)
-
-
-def unload(name):
-    try:
-        del sys.modules[name]
-    except KeyError:
-        pass
diff --git a/distutils2/tests/test_command_bdist.py b/distutils2/tests/test_command_bdist.py
--- a/distutils2/tests/test_command_bdist.py
+++ b/distutils2/tests/test_command_bdist.py
@@ -1,7 +1,9 @@
 """Tests for distutils.command.bdist."""
 import os
+import sys
+from StringIO import StringIO
 from distutils2.command.bdist import bdist, show_formats
-from distutils2.tests import unittest, support, captured_stdout
+from distutils2.tests import unittest, support
 
 
 class BuildTestCase(support.TempdirManager,
@@ -42,7 +44,13 @@
                             '%s should take --skip-build from bdist' % name)
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        saved = sys.stdout
+        sys.stdout = StringIO()
+        try:
+            show_formats()
+            stdout = sys.stdout.getvalue()
+        finally:
+            sys.stdout = saved
 
         # the output should be a header line + one line per format
         num_formats = len(bdist.format_commands)
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -1,5 +1,6 @@
 """Tests for distutils2.command.sdist."""
 import os
+import sys
 import zipfile
 
 try:
@@ -10,6 +11,7 @@
     UID_GID_SUPPORT = False
 
 from os.path import join
+from StringIO import StringIO
 from distutils2.dist import Distribution
 from distutils2.util import find_executable
 from distutils2.errors import PackagingOptionError
@@ -18,7 +20,6 @@
 from distutils2._backport.shutil import get_archive_formats
 
 from distutils2.tests import support, unittest
-from distutils2.tests import captured_stdout
 from distutils2.tests.support import requires_zlib
 
 
@@ -243,7 +244,13 @@
         self.assertIn("'setup.cfg' file not found", warnings[1])
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        saved = sys.stdout
+        sys.stdout = StringIO()
+        try:
+            show_formats()
+            stdout = sys.stdout.getvalue()
+        finally:
+            sys.stdout = saved
 
         # the output should be a header line + one line per format
         num_formats = len(get_archive_formats())
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -8,7 +8,6 @@
 from distutils2.dist import Distribution
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingModuleError, PackagingOptionError
-from distutils2.tests import captured_stdout
 from distutils2.tests import support, unittest
 from distutils2.tests.support import create_distribution, use_command
 from distutils2.tests.support import unload
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -1,15 +1,110 @@
 #!/usr/bin/env python
-"""Tests for distutils2.
+"""Test runner for distutils2.
 
 The tests for distutils2 are defined in the distutils2.tests package.
+They can also be executed with the unittest2 runner or nose.
 """
 
+import os
 import sys
 from os.path import dirname, islink, realpath, join, abspath
 from optparse import OptionParser
+from distutils2.tests import unittest
+
+
+# unittest machinery copied from stdlib's test.regrtest and test.support
+
+class TestFailed(Exception):
+    """Test failed."""
+
+
+class BasicTestRunner(object):
+    def run(self, test):
+        result = unittest.TestResult()
+        test(result)
+        return result
+
+
+def reap_children():
+    """Use this function at the end of test_main() whenever sub-processes
+    are started.  This will help ensure that no extra children (zombies)
+    stick around to hog resources and create problems when looking
+    for refleaks.
+    """
+
+    # Reap all our dead child processes so we don't leave zombies around.
+    # These hog resources and might be causing some of the buildbots to die.
+    if hasattr(os, 'waitpid'):
+        any_process = -1
+        while True:
+            try:
+                # This will raise an exception on Windows.  That's ok.
+                pid, status = os.waitpid(any_process, os.WNOHANG)
+                if pid == 0:
+                    break
+            except:
+                break
+
+
+def _run_suite(suite, verbose=True):
+    """Run tests from a unittest.TestSuite-derived class."""
+    if verbose:
+        runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
+    else:
+        runner = BasicTestRunner()
+
+    result = runner.run(suite)
+    if not result.wasSuccessful():
+        if len(result.errors) == 1 and not result.failures:
+            err = result.errors[0][1]
+        elif len(result.failures) == 1 and not result.errors:
+            err = result.failures[0][1]
+        else:
+            err = "errors occurred; run in verbose mode for details"
+        raise TestFailed(err)
+
+
+def run_unittest(classes, verbose=True):
+    """Run tests from unittest.TestCase-derived classes.
+
+    Originally extracted from stdlib test.test_support and modified to
+    support unittest2.
+    """
+    valid_types = (unittest.TestSuite, unittest.TestCase)
+    suite = unittest.TestSuite()
+    for cls in classes:
+        if isinstance(cls, basestring):
+            if cls in sys.modules:
+                suite.addTest(unittest.findTestCases(sys.modules[cls]))
+            else:
+                raise ValueError("str arguments must be keys in sys.modules")
+        elif isinstance(cls, valid_types):
+            suite.addTest(cls)
+        else:
+            suite.addTest(unittest.makeSuite(cls))
+    _run_suite(suite, verbose)
+
+
+def run_tests(verbose):
+    # do NOT import those at the top level, coverage will be inaccurate if
+    # distutils2 modules are imported before coverage magic is started
+    from distutils2.tests import test_suite
+    from distutils2._backport.tests import test_suite as btest_suite
+    try:
+        try:
+            run_unittest([test_suite(), btest_suite()], verbose=verbose)
+            return 0
+        except TestFailed:
+            return 1
+    finally:
+        reap_children()
+
+
+# coverage-related code
 
 COVERAGE_FILE = join(dirname(abspath(__file__)), '.coverage')
 
+
 def get_coverage():
     """ Return a usable coverage object. """
     # deferred import because coverage is optional
@@ -19,52 +114,34 @@
         cov = coverage.coverage(COVERAGE_FILE)
     return cov
 
+
 def ignore_prefixes(module):
     """ Return a list of prefixes to ignore in the coverage report if
     we want to completely skip `module`.
     """
-    # A function like that is needed because some GNU/Linux
-    # distributions, such a Ubuntu, really like to build link farm in
-    # /usr/lib in order to save a few bytes on the disk.
+    # A function like that is needed because some operating systems like Debian
+    # and derivatives use symlinks directory in order to save disk space
     dirnames = [dirname(module.__file__)]
 
-    pymod = module.__file__.rstrip("c")
+    pymod = module.__file__.rstrip('co')
     if islink(pymod):
         dirnames.append(dirname(realpath(pymod)))
     return dirnames
 
 
-def parse_opts():
-    parser = OptionParser(usage="%prog [OPTIONS]",
-                          description="run the distutils2 unittests")
-
-    parser.add_option("-q", "--quiet", help="do not print verbose messages",
-                      action="store_true", default=False)
-    parser.add_option("-c", "--coverage", action="store_true", default=False,
-                      help="produce a coverage report at the end of the run")
-    parser.add_option("-r", "--report", action="store_true", default=False,
-                      help="produce a coverage report from the last test run")
-    parser.add_option("-m", "--show-missing", action="store_true",
-                      default=False,
-                      help=("Show line numbers of statements in each module "
-                            "that weren't executed."))
-
-    opts, args = parser.parse_args()
-    return opts, args
-
-
 def coverage_report(opts):
     from distutils2.tests.support import unittest
     cov = get_coverage()
     if hasattr(cov, "load"):
         # running coverage 3.x
         cov.load()
+        # morfs means modules or files
         morfs = None
     else:
         # running coverage 2.x
         cov.cache = COVERAGE_FILE
         cov.restore()
-        morfs = [m for m in list(cov.cexecuted.keys()) if "distutils2" in m]
+        morfs = [m for m in cov.cexecuted if "distutils2" in m]
 
     prefixes = ["runtests", "distutils2/tests", "distutils2/_backport"]
     prefixes += ignore_prefixes(unittest)
@@ -93,6 +170,28 @@
                    omit=[p + "*" for p in prefixes],
                    show_missing=opts.show_missing)
 
+
+# command-line parsing
+
+def parse_opts():
+    parser = OptionParser(usage="%prog [OPTIONS]",
+                          description="run the distutils2 unittests")
+
+    parser.add_option("-q", "--quiet", help="do not print verbose messages",
+                      action="store_true", default=False)
+    parser.add_option("-c", "--coverage", action="store_true", default=False,
+                      help="produce a coverage report at the end of the run")
+    parser.add_option("-r", "--report", action="store_true", default=False,
+                      help="produce a coverage report from the last test run")
+    parser.add_option("-m", "--show-missing", action="store_true",
+                      default=False,
+                      help=("Show line numbers of statements in each module "
+                            "that weren't executed."))
+
+    opts, args = parser.parse_args()
+    return opts, args
+
+
 def test_main():
     opts, args = parse_opts()
     # FIXME when we run with --quiet, we still want to see errors and failures
@@ -115,21 +214,6 @@
     return ret
 
 
-def run_tests(verbose):
-    # do NOT import those at the top level, coverage will be inaccurate if
-    # modules are imported before its magic is started
-    from distutils2.tests import run_unittest, test_suite, reap_children, TestFailed
-    from distutils2._backport.tests import test_suite as btest_suite
-    try:
-        try:
-            run_unittest([test_suite(), btest_suite()], verbose_=verbose)
-            return 0
-        except TestFailed:
-            return 1
-    finally:
-        reap_children()
-
-
 if __name__ == "__main__":
     if sys.version < '2.5':
         try:

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Nov 15 15:06:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 15:06:00 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_=27verbose=27_arg?=
 =?utf8?q?uments=2C_obsoleted_by_logging?=
Message-ID: <E1RQJeC-0000g6-KK@dinsdale.python.org>

http://hg.python.org/distutils2/rev/83a3cc413139
changeset:   1255:83a3cc413139
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 11:50:00 2011 +0100
summary:
  Remove 'verbose' arguments, obsoleted by logging

files:
  CHANGES.txt                            |   2 +
  distutils2/command/build_ext.py        |   5 ---
  distutils2/command/cmd.py              |  18 ++++++----
  distutils2/command/sdist.py            |   5 +-
  distutils2/command/test.py             |   3 +-
  distutils2/compiler/__init__.py        |   9 +----
  distutils2/compiler/bcppcompiler.py    |   4 +-
  distutils2/compiler/ccompiler.py       |   3 +-
  distutils2/compiler/cygwinccompiler.py |   8 ++--
  distutils2/compiler/msvc9compiler.py   |   4 +-
  distutils2/compiler/msvccompiler.py    |   4 +-
  distutils2/util.py                     |  23 +++++--------
  12 files changed, 38 insertions(+), 50 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -157,6 +157,8 @@
 - Rename get_reinitialized_command back to reinitialize_command [?ric]
 - Rename install_distinfo's option from distinfo-dir to the more usual
   install_dir [?ric]
+- Remove verbose arguments for Command and Compiler classes as well as util
+  functions, obsoleted by logging [?ric]
 
 
 1.0a3 - 2010-10-08
diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -297,14 +297,9 @@
             self.libraries.extend(build_clib.get_library_names() or [])
             self.library_dirs.append(build_clib.build_clib)
 
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
         # Setup the CCompiler object that we'll use to do all the
         # compiling and linking
         self.compiler_obj = new_compiler(compiler=self.compiler,
-                                         verbose=verbose,
                                          dry_run=self.dry_run,
                                          force=self.force)
 
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -351,7 +351,7 @@
     def execute(self, func, args, msg=None, level=1):
         util.execute(func, args, msg, dry_run=self.dry_run)
 
-    def mkpath(self, name, mode=00777, dry_run=None, verbose=0):
+    def mkpath(self, name, mode=00777, dry_run=None):
         if dry_run is None:
             dry_run = self.dry_run
         name = os.path.normpath(name)
@@ -367,9 +367,11 @@
 
     def copy_file(self, infile, outfile,
                   preserve_mode=True, preserve_times=True, link=None, level=1):
-        """Copy a file respecting verbose, dry-run and force flags.  (The
-        former two default to whatever is in the Distribution object, and
-        the latter defaults to false for commands that don't define it.)"""
+        """Copy a file respecting dry-run and force flags.
+
+        (dry-run defaults to whatever is in the Distribution object, and
+        force to false for commands that don't define it.)
+        """
         if self.dry_run:
             # XXX add a comment
             return
@@ -380,11 +382,13 @@
 
     def copy_tree(self, infile, outfile, preserve_mode=True,
                   preserve_times=True, preserve_symlinks=False, level=1):
-        """Copy an entire directory tree respecting verbose, dry-run,
+        """Copy an entire directory tree respecting dry-run
         and force flags.
         """
         if self.dry_run:
-            return  # see if we want to display something
+            # XXX should not return but let copy_tree log and decide to execute
+            # or not based on its dry_run argument
+            return
 
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
@@ -392,7 +396,7 @@
     def move_file(self, src, dst, level=1):
         """Move a file respecting the dry-run flag."""
         if self.dry_run:
-            return  # XXX log ?
+            return  # XXX same thing
         return move(src, dst)
 
     def spawn(self, cmd, search_path=True, level=1):
diff --git a/distutils2/command/sdist.py b/distutils2/command/sdist.py
--- a/distutils2/command/sdist.py
+++ b/distutils2/command/sdist.py
@@ -337,12 +337,11 @@
         """
         return self.archive_files
 
-    def create_tree(self, base_dir, files, mode=0777, verbose=1,
-                    dry_run=False):
+    def create_tree(self, base_dir, files, mode=0777, dry_run=False):
         need_dir = set()
         for file in files:
             need_dir.add(os.path.join(base_dir, os.path.dirname(file)))
 
         # Now create them
         for dir in sorted(need_dir):
-            self.mkpath(dir, mode, verbose=verbose, dry_run=dry_run)
+            self.mkpath(dir, mode, dry_run=dry_run)
diff --git a/distutils2/command/test.py b/distutils2/command/test.py
--- a/distutils2/command/test.py
+++ b/distutils2/command/test.py
@@ -60,8 +60,7 @@
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
-            # Temporary kludge until we remove the verbose arguments and use
-            # logging everywhere
+            # XXX maybe we could pass the verbose argument of pysetup here
             logger = logging.getLogger('distutils2')
             verbose = logger.getEffectiveLevel() >= logging.DEBUG
             verbosity = verbose + 1
diff --git a/distutils2/compiler/__init__.py b/distutils2/compiler/__init__.py
--- a/distutils2/compiler/__init__.py
+++ b/distutils2/compiler/__init__.py
@@ -153,8 +153,7 @@
     pretty_printer.print_help("List of available compilers:")
 
 
-def new_compiler(plat=None, compiler=None, verbose=0, dry_run=False,
-                 force=False):
+def new_compiler(plat=None, compiler=None, dry_run=False, force=False):
     """Generate an instance of some CCompiler subclass for the supplied
     platform/compiler combination.  'plat' defaults to 'os.name'
     (eg. 'posix', 'nt'), and 'compiler' defaults to the default compiler
@@ -183,11 +182,7 @@
         cls = resolve_name(cls)
         _COMPILERS[compiler] = cls
 
-
-    # XXX The None is necessary to preserve backwards compatibility
-    # with classes that expect verbose to be the first positional
-    # argument.
-    return cls(None, dry_run, force)
+    return cls(dry_run, force)
 
 
 def gen_preprocess_options(macros, include_dirs):
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -47,8 +47,8 @@
     exe_extension = '.exe'
 
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(BCPPCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(BCPPCompiler, self).__init__(dry_run, force)
 
         # These executables are assumed to all be in the path.
         # Borland doesn't seem to use any special registry settings to
diff --git a/distutils2/compiler/ccompiler.py b/distutils2/compiler/ccompiler.py
--- a/distutils2/compiler/ccompiler.py
+++ b/distutils2/compiler/ccompiler.py
@@ -79,10 +79,9 @@
                    }
     language_order = ["c++", "objc", "c"]
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
+    def __init__(self, dry_run=False, force=False):
         self.dry_run = dry_run
         self.force = force
-        self.verbose = verbose
 
         # 'output_dir': a common output directory for object, library,
         # shared object, and shared library files
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -92,8 +92,8 @@
     shared_lib_format = "%s%s"
     exe_extension = ".exe"
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(CygwinCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(CygwinCCompiler, self).__init__(dry_run, force)
 
         status, details = check_config_h()
         logger.debug("Python's GCC status: %s (details: %s)", status, details)
@@ -270,8 +270,8 @@
     name = 'mingw32'
     description = 'MinGW32 compiler'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(Mingw32CCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(Mingw32CCompiler, self).__init__(dry_run, force)
 
         # ld_version >= "2.13" support -shared so use it instead of
         # -mdll -static
diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py
--- a/distutils2/compiler/msvc9compiler.py
+++ b/distutils2/compiler/msvc9compiler.py
@@ -309,8 +309,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = VERSION
         self.__root = r"Software\Microsoft\VisualStudio"
         # self.__macros = MACROS
diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py
--- a/distutils2/compiler/msvccompiler.py
+++ b/distutils2/compiler/msvccompiler.py
@@ -236,8 +236,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = get_build_version()
         self.__arch = get_build_architecture()
         if self.__arch == "Intel":
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -264,7 +264,7 @@
             if element]
 
 
-def execute(func, args, msg=None, verbose=0, dry_run=False):
+def execute(func, args, msg=None, dry_run=False):
     """Perform some action that affects the outside world.
 
     Some actions (e.g. writing to the filesystem) are special because
@@ -688,7 +688,7 @@
     _cfg_target_split = None
 
 
-def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
+def spawn(cmd, search_path=True, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
     'cmd' is just the argument list for the new process, ie.
@@ -1364,8 +1364,7 @@
 
 # XXX to be replaced by shutil.copytree
 def copy_tree(src, dst, preserve_mode=True, preserve_times=True,
-              preserve_symlinks=False, update=False, verbose=True,
-              dry_run=False):
+              preserve_symlinks=False, update=False, dry_run=False):
     from distutils.file_util import copy_file
 
     if not dry_run and not os.path.isdir(src):
@@ -1382,7 +1381,7 @@
                   "error listing files in '%s': %s" % (src, errstr))
 
     if not dry_run:
-        _mkpath(dst, verbose=verbose)
+        _mkpath(dst)
 
     outputs = []
 
@@ -1392,8 +1391,7 @@
 
         if preserve_symlinks and os.path.islink(src_name):
             link_dest = os.readlink(src_name)
-            if verbose >= 1:
-                logger.info("linking %s -> %s", dst_name, link_dest)
+            logger.info("linking %s -> %s", dst_name, link_dest)
             if not dry_run:
                 os.symlink(link_dest, dst_name)
             outputs.append(dst_name)
@@ -1402,11 +1400,10 @@
             outputs.extend(
                 copy_tree(src_name, dst_name, preserve_mode,
                           preserve_times, preserve_symlinks, update,
-                          verbose=verbose, dry_run=dry_run))
+                          dry_run=dry_run))
         else:
             copy_file(src_name, dst_name, preserve_mode,
-                      preserve_times, update, verbose=verbose,
-                      dry_run=dry_run)
+                      preserve_times, update, dry_run=dry_run)
             outputs.append(dst_name)
 
     return outputs
@@ -1419,7 +1416,7 @@
 # I don't use os.makedirs because a) it's new to Python 1.5.2, and
 # b) it blows up if the directory already exists (I want to silently
 # succeed in that case).
-def _mkpath(name, mode=0777, verbose=True, dry_run=False):
+def _mkpath(name, mode=0777, dry_run=False):
     # Detect a common bug -- name is None
     if not isinstance(name, basestring):
         raise PackagingInternalError(
@@ -1454,9 +1451,7 @@
         if abs_head in _path_created:
             continue
 
-        if verbose >= 1:
-            logger.info("creating %s", head)
-
+        logger.info("creating %s", head)
         if not dry_run:
             try:
                 os.mkdir(head, mode)

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Nov 15 15:06:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 15:06:00 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Add_tests_for_tests=2Esu?=
 =?utf8?q?pport_=28=2312659=29=2C_thanks_to_Francisco_Mart=C3=ADn_Brugu?=
 =?utf8?b?w6k=?=
Message-ID: <E1RQJeC-0000g9-Mi@dinsdale.python.org>

http://hg.python.org/distutils2/rev/659bf2a679d2
changeset:   1256:659bf2a679d2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 15:03:54 2011 +0100
summary:
  Add tests for tests.support (#12659), thanks to Francisco Mart?n Brugu?

files:
  CHANGES.txt                      |   1 +
  CONTRIBUTORS.txt                 |   1 +
  distutils2/tests/test_support.py |  85 ++++++++++++++++++++
  3 files changed, 87 insertions(+), 0 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -159,6 +159,7 @@
   install_dir [?ric]
 - Remove verbose arguments for Command and Compiler classes as well as util
   functions, obsoleted by logging [?ric]
+- #12659: Add tests for tests.support [francisco]
 
 
 1.0a3 - 2010-10-08
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -17,6 +17,7 @@
 - Anthony Baxter
 - Erik Bray
 - C. Titus Brown
+- Francisco Mart?n Brugu?
 - Nicolas Cadou
 - Godefroid Chapelle
 - Christophe Combelles
diff --git a/distutils2/tests/test_support.py b/distutils2/tests/test_support.py
new file mode 100644
--- /dev/null
+++ b/distutils2/tests/test_support.py
@@ -0,0 +1,85 @@
+import os
+import tempfile
+
+from distutils2.dist import Distribution
+from distutils2.tests import support, unittest
+
+
+class TestingSupportTestCase(unittest.TestCase):
+
+    def test_fake_dec(self):
+        @support.fake_dec(1, 2, k=3)
+        def func(arg0, *args, **kargs):
+            return arg0, args, kargs
+        self.assertEqual(func(-1, -2, k=-3), (-1, (-2,), {'k': -3}))
+
+    def test_TempdirManager(self):
+        files = {}
+
+        class Tester(support.TempdirManager, unittest.TestCase):
+
+            def runTest(self):
+                # empty method required for the backport
+                pass
+		
+            def test_mktempfile(self2):
+                tmpfile = self2.mktempfile()
+                files['test_mktempfile'] = tmpfile.name
+                self.assertTrue(os.path.isfile(tmpfile.name))
+
+            def test_mkdtemp(self2):
+                tmpdir = self2.mkdtemp()
+                files['test_mkdtemp'] = tmpdir
+                self.assertTrue(os.path.isdir(tmpdir))
+
+            def test_write_file(self2):
+                tmpdir = self2.mkdtemp()
+                files['test_write_file'] = tmpdir
+                self2.write_file((tmpdir, 'file1'), 'me file 1')
+                file1 = os.path.join(tmpdir, 'file1')
+                self.assertTrue(os.path.isfile(file1))
+                text = ''
+                f = open(file1, 'r')
+                try:
+                    text = f.read()
+                finally:
+                    f.close()
+                self.assertEqual(text, 'me file 1')
+
+            def test_create_dist(self2):
+                project_dir, dist = self2.create_dist()
+                files['test_create_dist'] = project_dir
+                self.assertTrue(os.path.isdir(project_dir))
+                self.assertIsInstance(dist, Distribution)
+
+            def test_assertIsFile(self2):
+                fd, fn = tempfile.mkstemp()
+                os.close(fd)
+                self.addCleanup(support.unlink, fn)
+                self2.assertIsFile(fn)
+                self.assertRaises(AssertionError, self2.assertIsFile, 'foO')
+
+            def test_assertIsNotFile(self2):
+                tmpdir = self2.mkdtemp()
+                self2.assertIsNotFile(tmpdir)
+
+        tester = Tester()
+        for name in ('test_mktempfile', 'test_mkdtemp', 'test_write_file',
+                     'test_create_dist', 'test_assertIsFile',
+                     'test_assertIsNotFile'):
+            tester.setUp()
+            try:
+                getattr(tester, name)()
+            finally:
+                tester.tearDown()
+
+            # check clean-up
+            if name in files:
+                self.assertFalse(os.path.exists(files[name]))
+
+
+def test_suite():
+    return unittest.makeSuite(TestingSupportTestCase)
+
+if __name__ == "__main__":
+    unittest.main(defaultTest="test_suite")

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Tue Nov 15 15:31:18 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 15:31:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Mark_test=5Fdevpoll_as_unex?=
	=?utf8?q?pected_on_most_platforms?=
Message-ID: <E1RQK2g-0003CF-Ha@dinsdale.python.org>

http://hg.python.org/cpython/rev/7f8ed2d04ef2
changeset:   73556:7f8ed2d04ef2
parent:      73553:250091e60f28
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 15:25:59 2011 +0100
summary:
  Mark test_devpoll as unexpected on most platforms

files:
  Lib/test/regrtest.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1462,6 +1462,7 @@
         test_crypt
         test_curses
         test_dbm
+        test_devpoll
         test_fcntl
         test_fork1
         test_epoll
@@ -1488,6 +1489,7 @@
     ('linux',
         """
         test_curses
+        test_devpoll
         test_largefile
         test_kqueue
         test_ossaudiodev
@@ -1538,6 +1540,7 @@
         """
         test__locale
         test_curses
+        test_devpoll
         test_epoll
         test_dbm_gnu
         test_gdb
@@ -1579,6 +1582,7 @@
         """
         test_curses
         test_dbm
+        test_devpoll
         test_epoll
         test_ioctl
         test_kqueue
@@ -1603,6 +1607,7 @@
         """),
     ('freebsd',
         """
+        test_devpoll
         test_epoll
         test_dbm_gnu
         test_locale
@@ -1636,6 +1641,7 @@
     ('openbsd',
         """
         test_ctypes
+        test_devpoll
         test_epoll
         test_dbm_gnu
         test_locale
@@ -1652,6 +1658,7 @@
         """
         test_ctypes
         test_curses
+        test_devpoll
         test_epoll
         test_dbm_gnu
         test_locale

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 15:31:19 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 15:31:19 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge?=
Message-ID: <E1RQK2h-0003CP-A2@dinsdale.python.org>

http://hg.python.org/cpython/rev/c512a7f5e0b3
changeset:   73557:c512a7f5e0b3
parent:      73556:7f8ed2d04ef2
parent:      73555:ff7acde0faad
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 15:26:32 2011 +0100
summary:
  Merge

files:
  Doc/library/select.rst |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/library/select.rst b/Doc/library/select.rst
--- a/Doc/library/select.rst
+++ b/Doc/library/select.rst
@@ -26,6 +26,7 @@
 
 
 .. function:: devpoll()
+
    (Only supported on Solaris and derivatives.)  Returns a ``/dev/poll``
    polling object; see section :ref:`devpoll-objects` below for the
    methods supported by devpoll objects.
@@ -33,7 +34,7 @@
    :c:func:`devpoll` objects are linked to the number of file
    descriptors allowed at the time of instantiation. If your program
    reduces this value, :c:func:`devpoll` will fail. If your program
-   increases this value, c:func:`devpoll` may return an
+   increases this value, :c:func:`devpoll` may return an
    incomplete list of active file descriptors.
 
    .. versionadded:: 3.3

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:48 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:48 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_markup?=
Message-ID: <E1RQKho-0007ml-Hx@dinsdale.python.org>

http://hg.python.org/cpython/rev/974c5ba31ca6
changeset:   73558:974c5ba31ca6
branch:      3.2
parent:      73547:16ed15ff0d7c
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 18:00:48 2011 +0100
summary:
  Fix markup

files:
  Doc/whatsnew/3.2.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst
--- a/Doc/whatsnew/3.2.rst
+++ b/Doc/whatsnew/3.2.rst
@@ -234,8 +234,8 @@
 namespace, *concurrent*.  Its first member is a *futures* package which provides
 a uniform high-level interface for managing threads and processes.
 
-The design for :mod:`concurrent.futures` was inspired by
-*java.util.concurrent.package*.  In that model, a running call and its result
+The design for :mod:`concurrent.futures` was inspired by the
+*java.util.concurrent* package.  In that model, a running call and its result
 are represented by a :class:`~concurrent.futures.Future` object that abstracts
 features common to threads, processes, and remote procedure calls.  That object
 supports status checks (running or done), timeouts, cancellations, adding

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:49 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Clean_up_byte-compilation_c?=
 =?utf8?q?ode_in_packaging_=28=2311254_followup=29=2E?=
Message-ID: <E1RQKhp-0007mq-A0@dinsdale.python.org>

http://hg.python.org/cpython/rev/c10946a17420
changeset:   73559:c10946a17420
parent:      73548:426f7a2b1826
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 18:10:19 2011 +0100
summary:
  Clean up byte-compilation code in packaging (#11254 followup).

- Don't use keyword arguments for debug_override; I find it more
  readable to have a comment explaining that True makes pyc and False
  pyo than to write out the non-obvious (when you haven?t read the doc)
  argument name

- Move duplicate code from build_py and install_lib into cmd

- Remove obsolete verbose argument of util.byte_compile

- Remove obsolete passing of -O/-OO to the Python process spawned by
  util.byte_compile (I?ll remove the whole spawning later, after I write
  more tests to check the contents of pyc and pyo files; now that
  byte_compile does not depend on the value of __debug__ in the calling
  Python, we can call py_compile or compileall directly)

files:
  Doc/library/packaging.util.rst                  |  11 +-
  Lib/packaging/command/build_py.py               |  29 +-----
  Lib/packaging/command/cmd.py                    |  20 ++++-
  Lib/packaging/command/install_lib.py            |  42 ++-------
  Lib/packaging/tests/test_command_install_lib.py |   4 +-
  Lib/packaging/util.py                           |  29 +++---
  6 files changed, 57 insertions(+), 78 deletions(-)


diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -90,7 +90,7 @@
    Search the path for a given executable name.
 
 
-.. function:: execute(func, args[, msg=None, verbose=0, dry_run=0])
+.. function:: execute(func, args, msg=None, verbose=0, dry_run=0)
 
    Perform some action that affects the outside world (for instance, writing to
    the filesystem).  Such actions are special because they are disabled by the
@@ -117,7 +117,8 @@
    :exc:`ValueError` if *val* is anything else.
 
 
-.. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])
+.. function:: byte_compile(py_files, optimize=0, force=0, prefix=None, \
+                           base_dir=None, dry_run=0, direct=None)
 
    Byte-compile a collection of Python source files to either :file:`.pyc` or
    :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`),
@@ -131,6 +132,9 @@
    * ``1`` - normal optimization (like ``python -O``)
    * ``2`` - extra optimization (like ``python -OO``)
 
+   This function is independent from the running Python's :option:`-O` or
+   :option:`-B` options; it is fully controlled by the parameters passed in.
+
    If *force* is true, all files are recompiled regardless of timestamps.
 
    The source filename encoded in each :term:`bytecode` file defaults to the filenames
@@ -149,6 +153,3 @@
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
-
-   This function is independent from the running Python's :option:`-O` or
-   :option:`-B` options; it is fully controlled by the parameters passed in.
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -18,7 +18,7 @@
 
     description = "build pure Python modules (copy to build directory)"
 
-    # The options for controlling byte compilations are two independent sets;
+    # The options for controlling byte compilation are two independent sets;
     # more info in install_lib or the reST docs
 
     user_options = [
@@ -113,7 +113,8 @@
             self.run_2to3(self._updated_files, self._doctests_2to3,
                                             self.use_2to3_fixers)
 
-        self.byte_compile(self.get_outputs(include_bytecode=False))
+        self.byte_compile(self.get_outputs(include_bytecode=False),
+                          prefix=self.build_lib)
 
     # -- Top-level worker functions ------------------------------------
 
@@ -335,11 +336,9 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(imp.cache_from_source(filename,
-                                                         debug_override=True))
-                if self.optimize > 0:
-                    outputs.append(imp.cache_from_source(filename,
-                                                         debug_override=False))
+                    outputs.append(imp.cache_from_source(filename, True))
+                if self.optimize:
+                    outputs.append(imp.cache_from_source(filename, False))
 
         outputs += [
             os.path.join(build_dir, filename)
@@ -391,19 +390,3 @@
             for package_, module, module_file in modules:
                 assert package == package_
                 self.build_module(module, module_file, package)
-
-    def byte_compile(self, files):
-        from packaging.util import byte_compile  # FIXME use compileall
-        prefix = self.build_lib
-        if prefix[-1] != os.sep:
-            prefix = prefix + os.sep
-
-        # XXX this code is essentially the same as the 'byte_compile()
-        # method of the "install_lib" command, except for the determination
-        # of the 'prefix' string.  Hmmm.
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
diff --git a/Lib/packaging/command/cmd.py b/Lib/packaging/command/cmd.py
--- a/Lib/packaging/command/cmd.py
+++ b/Lib/packaging/command/cmd.py
@@ -10,7 +10,7 @@
 
 class Command:
     """Abstract base class for defining command classes, the "worker bees"
-    of the Packaging.  A useful analogy for command classes is to think of
+    of Packaging.  A useful analogy for command classes is to think of
     them as subroutines with local variables called "options".  The options
     are "declared" in 'initialize_options()' and "defined" (given their
     final values, aka "finalized") in 'finalize_options()', both of which
@@ -386,7 +386,6 @@
         if self.dry_run:
             return  # see if we want to display something
 
-
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
 
@@ -439,3 +438,20 @@
         # Otherwise, print the "skip" message
         else:
             logger.debug(skip_msg)
+
+    def byte_compile(self, files, prefix=None):
+        """Byte-compile files to pyc and/or pyo files.
+
+        This method requires that the calling class define compile and
+        optimize options, like build_py and install_lib.  It also
+        automatically respects the force and dry-run options.
+
+        prefix, if given, is a string that will be stripped off the
+        filenames encoded in bytecode files.
+        """
+        if self.compile:
+            util.byte_compile(files, optimize=False, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
+        if self.optimize:
+            util.byte_compile(files, optimize=self.optimize, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -2,7 +2,6 @@
 
 import os
 import imp
-import logging
 
 from packaging import logger
 from packaging.command.cmd import Command
@@ -21,7 +20,7 @@
 
     description = "install all modules (extensions and pure Python)"
 
-    # The options for controlling byte compilations are two independent sets:
+    # The options for controlling byte compilation are two independent sets:
     # 'compile' is strictly boolean, and only decides whether to
     # generate .pyc files.  'optimize' is three-way (0, 1, or 2), and
     # decides both whether to generate .pyo files and what level of
@@ -84,9 +83,14 @@
         # having a build directory!)
         outfiles = self.install()
 
-        # (Optionally) compile .py to .pyc
+        # (Optionally) compile .py to .pyc and/or .pyo
         if outfiles is not None and self.distribution.has_pure_modules():
-            self.byte_compile(outfiles)
+            # XXX comment from distutils: "This [prefix stripping] is far from
+            # complete, but it should at least generate usable bytecode in RPM
+            # distributions." -> need to find exact requirements for
+            # byte-compiled files and fix it
+            install_root = self.get_finalized_command('install_dist').root
+            self.byte_compile(outfiles, prefix=install_root)
 
     # -- Top-level worker functions ------------------------------------
     # (called from 'run()')
@@ -108,28 +112,6 @@
             return
         return outfiles
 
-    def byte_compile(self, files):
-        from packaging.util import byte_compile  # FIXME use compileall
-
-        # Get the "--root" directory supplied to the "install_dist" command,
-        # and use it as a prefix to strip off the purported filename
-        # encoded in bytecode files.  This is far from complete, but it
-        # should at least generate usable bytecode in RPM distributions.
-        install_root = self.get_finalized_command('install_dist').root
-
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=install_root,
-                         verbose=verbose, dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=install_root,
-                         verbose=verbose, dry_run=self.dry_run)
-
     # -- Utility methods -----------------------------------------------
 
     def _mutate_outputs(self, has_any, build_cmd, cmd_option, output_dir):
@@ -157,11 +139,9 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(imp.cache_from_source(
-                    py_file, debug_override=True))
-            if self.optimize > 0:
-                bytecode_files.append(imp.cache_from_source(
-                    py_file, debug_override=False))
+                bytecode_files.append(imp.cache_from_source(py_file, True))
+            if self.optimize:
+                bytecode_files.append(imp.cache_from_source(py_file, False))
 
         return bytecode_files
 
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -44,8 +44,8 @@
         f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
-        pyo_file = imp.cache_from_source('foo.py', debug_override=False)
+        pyc_file = imp.cache_from_source('foo.py', True)
+        pyo_file = imp.cache_from_source('foo.py', False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -296,7 +296,7 @@
 
 
 def byte_compile(py_files, optimize=0, force=False, prefix=None,
-                 base_dir=None, verbose=0, dry_run=False, direct=None):
+                 base_dir=None, dry_run=False, direct=None):
     """Byte-compile a collection of Python source files to either .pyc
     or .pyo files in a __pycache__ subdirectory.
 
@@ -305,6 +305,9 @@
       0 - don't optimize (generate .pyc)
       1 - normal optimization (like "python -O")
       2 - extra optimization (like "python -OO")
+    This function is independent from the running Python's -O or -B options;
+    it is fully controlled by the parameters passed in.
+
     If 'force' is true, all files are recompiled regardless of
     timestamps.
 
@@ -325,10 +328,9 @@
     the source for details).  The 'direct' flag is used by the script
     generated in indirect mode; unless you know what you're doing, leave
     it set to None.
+    """
+    # FIXME use compileall + remove direct/indirect shenanigans
 
-    This function is independent from the running Python's -O or -B options;
-    it is fully controlled by the parameters passed in.
-    """
     # First, if the caller didn't force us into direct or indirect mode,
     # figure out which mode we should be in.  We take a conservative
     # approach: choose direct mode *only* if the current interpreter is
@@ -381,15 +383,11 @@
                 script.write("""
 byte_compile(files, optimize=%r, force=%r,
              prefix=%r, base_dir=%r,
-             verbose=%r, dry_run=False,
+             dry_run=False,
              direct=True)
-""" % (optimize, force, prefix, base_dir, verbose))
+""" % (optimize, force, prefix, base_dir))
 
         cmd = [sys.executable, script_name]
-        if optimize == 1:
-            cmd.insert(1, "-O")
-        elif optimize == 2:
-            cmd.insert(1, "-OO")
 
         env = os.environ.copy()
         env['PYTHONPATH'] = os.path.pathsep.join(sys.path)
@@ -415,8 +413,10 @@
             # Terminology from the py_compile module:
             #   cfile - byte-compiled file
             #   dfile - purported source filename (same as 'file' by default)
-            debug_override = not optimize
-            cfile = imp.cache_from_source(file, debug_override)
+            # The second argument to cache_from_source forces the extension to
+            # be .pyc (if true) or .pyo (if false); without it, the extension
+            # would depend on the calling Python's -O option
+            cfile = imp.cache_from_source(file, not optimize)
             dfile = file
 
             if prefix:
@@ -1334,7 +1334,7 @@
               preserve_symlinks=False, update=False, verbose=True,
               dry_run=False):
     # FIXME use of this function is why we get spurious logging message on
-    # stdout when tests run; kill and replace by shuil!
+    # stdout when tests run; kill and replace by shutil!
     from distutils.file_util import copy_file
 
     if not dry_run and not os.path.isdir(src):
@@ -1448,8 +1448,7 @@
 
     Returns (content_type: bytes, body: bytes) ready for http.client.HTTP.
     """
-    # Taken from
-    # http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
+    # Taken from http://code.activestate.com/recipes/146306
 
     if boundary is None:
         boundary = b'--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:50 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_NEWS_entries_for_pac?=
	=?utf8?q?kaging_test_suite_fixes=2E?=
Message-ID: <E1RQKhq-0007mu-6G@dinsdale.python.org>

http://hg.python.org/cpython/rev/6e3444d2b905
changeset:   73560:6e3444d2b905
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 18:13:24 2011 +0100
summary:
  Remove NEWS entries for packaging test suite fixes.

I haven?t updated NEWS for every change or fix in packaging, thinking
that the only entry we need for 3.3a1 is ?Add packaging module to the
stdlib?.  Other developers have however added entries for these fixes,
but I don?t think they are interesting for users.

Note that the distutils2 backport has a full changelog.

files:
  Misc/NEWS |  10 ----------
  1 files changed, 0 insertions(+), 10 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -953,13 +953,9 @@
 - Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl
   module.
 
-- Issue #12125: fixed the failures under Solaris due to improper test cleanup.
-
 - Issue #6501: os.device_encoding() returns None on Windows if the application
   has no console.
 
-- Issue #12132: Skip test_build_ext in case the xxmodule is not found.
-
 - Issue #12105: Add O_CLOEXEC to the os module.
 
 - Issue #12079: Decimal('Infinity').fma(Decimal('0'), (3.91224318126786e+19+0j))
@@ -1685,12 +1681,6 @@
 - Issue #12057: Add tests for ISO 2022 codecs (iso2022_jp, iso2022_jp_2,
   iso2022_kr).
 
-- Issue #12180: Fixed a few remaining errors in test_packaging when no
-  threading.
-
-- Issue #12120, #12119: skip a test in packaging and distutils
-  if sys.dont_write_bytecode is set to True.
-
 - Issue #12096: Fix a race condition in test_threading.test_waitfor(). Patch
   written by Charles-Fran?ois Natali.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:51 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_entries_for_distutils2_?=
 =?utf8?q?contributors_=28their_patches_are_in_packaging=29?=
Message-ID: <E1RQKhr-0007my-05@dinsdale.python.org>

http://hg.python.org/cpython/rev/d51eb3a757fc
changeset:   73561:d51eb3a757fc
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 18:14:09 2011 +0100
summary:
  Add entries for distutils2 contributors (their patches are in packaging)

files:
  Misc/ACKS |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -378,6 +378,7 @@
 Rasmus Hahn
 Peter Haight
 V?clav Haisman
+Walker Hale IV
 Bob Halley
 Jesse Hallio
 Jun Hamano
@@ -593,6 +594,7 @@
 Gregor Lingl
 Nick Lockwood
 Stephanie Lockwood
+Hugo Lopes Tavares
 Anne Lord
 Tom Loredo
 Justin Love
@@ -606,6 +608,7 @@
 Jim Lynch
 Mikael Lyngvig
 Martin von L?wis
+Guillermo L?pez-Anglada
 Andrew I MacIntyre
 Tim MacKenzie
 Nick Maclaren
@@ -805,6 +808,7 @@
 Tim Rice
 Jan Pieter Riegel
 Armin Rigo
+Arc Riley
 Nicholas Riley
 Jean-Claude Rimbault
 Vlad Riscutia
@@ -817,6 +821,7 @@
 Mark Roddy
 Kevin Rodgers
 Giampaolo Rodola
+Elson Rodriguez
 Luis Rojas
 Mike Romberg
 Armin Ronacher

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:51 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_few_typos?=
Message-ID: <E1RQKhr-0007n2-Qp@dinsdale.python.org>

http://hg.python.org/cpython/rev/f6f7bc5deca7
changeset:   73562:f6f7bc5deca7
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 18:18:15 2011 +0100
summary:
  Fix a few typos

files:
  Doc/install/pysetup.rst |  2 +-
  Doc/whatsnew/3.3.rst    |  2 +-
  Lib/packaging/run.py    |  2 +-
  Lib/test/regrtest.py    |  2 +-
  4 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Doc/install/pysetup.rst b/Doc/install/pysetup.rst
--- a/Doc/install/pysetup.rst
+++ b/Doc/install/pysetup.rst
@@ -149,7 +149,7 @@
        list: List installed projects
        graph: Display a graph
        create: Create a project
-       generate-setup: Generate a backward-comptatible setup.py
+       generate-setup: Generate a backward-compatible setup.py
 
    To get more help on an action, use:
 
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -596,7 +596,7 @@
    paths.  In previous versions, it did.  See changeset for doc changes in
    various files.  Contributed by Carl Meyer with editions by ?ric Araujo.
 
-.. Issue #10998: -Q command-line flags are related artifacts have been
+.. Issue #10998: the -Q command-line flag and related artifacts have been
    removed.  Code checking sys.flags.division_warning will need updating.
    Contributed by ?ric Araujo.
 
diff --git a/Lib/packaging/run.py b/Lib/packaging/run.py
--- a/Lib/packaging/run.py
+++ b/Lib/packaging/run.py
@@ -368,7 +368,7 @@
     ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
-    ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),
+    ('generate-setup', 'Generate a backward-compatible setup.py', _generate),
 ]
 
 
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1066,7 +1066,7 @@
         keys = set(packaging.command._COMMANDS)
         return id_, keys
     def restore_packaging_command__COMMANDS(self, saved):
-        # if command._COMMANDS was bound to another dict obhect, we can't
+        # if command._COMMANDS was bound to another dict object, we can't
         # restore the previous object and contents, because the get_ method
         # above does not return the dict object (to ignore changes in values)
         for key in packaging.command._COMMANDS.keys() - saved[1]:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:52 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Adapt_test_and_example_afte?=
	=?utf8?q?r_e39d1b6f0856=2E?=
Message-ID: <E1RQKhs-0007n6-Kw@dinsdale.python.org>

http://hg.python.org/cpython/rev/af8d17170065
changeset:   73563:af8d17170065
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 18:21:38 2011 +0100
summary:
  Adapt test and example after e39d1b6f0856.

Tarek?s commit fixed the way packaging configuration file markers are
split under Windows, but these two files were not edited.

files:
  Doc/packaging/setupcfg.rst         |  2 +-
  Lib/packaging/tests/test_config.py |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -779,7 +779,7 @@
    extra_compile_args =
        -fPIC -O2
        -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
-       /DGECODE_VERSION='win32' -- sys.platform == 'win32'
+       /DGECODE_VERSION=win32 -- sys.platform == 'win32'
 
 The section name must start with ``extension:``; the right-hand part is used as
 the full name (including a parent package, if any) of the extension.  Whitespace
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -119,7 +119,7 @@
     /usr/include/blitz
 extra_compile_args = -fPIC -O2
     -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
-    /DGECODE_VERSION='win32' -- sys.platform == 'win32'
+    /DGECODE_VERSION=win32 -- sys.platform == 'win32'
 language = cxx
 
 # corner case: if the parent package of an extension is declared but

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:53 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_packaging_cleanup=3A_A_few_?=
 =?utf8?q?super_I_missed_in_5ae03b1e147a?=
Message-ID: <E1RQKht-0007nA-E1@dinsdale.python.org>

http://hg.python.org/cpython/rev/10c866c5de0e
changeset:   73564:10c866c5de0e
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 19:40:31 2011 +0100
summary:
  packaging cleanup: A few super I missed in 5ae03b1e147a

files:
  Lib/packaging/compiler/cygwinccompiler.py |  11 +++++------
  Lib/packaging/tests/pypi_server.py        |  10 +++++-----
  2 files changed, 10 insertions(+), 11 deletions(-)


diff --git a/Lib/packaging/compiler/cygwinccompiler.py b/Lib/packaging/compiler/cygwinccompiler.py
--- a/Lib/packaging/compiler/cygwinccompiler.py
+++ b/Lib/packaging/compiler/cygwinccompiler.py
@@ -233,12 +233,11 @@
         if not debug:
             extra_preargs.append("-s")
 
-        UnixCCompiler.link(self, target_desc, objects, output_filename,
-                           output_dir, libraries, library_dirs,
-                           runtime_library_dirs,
-                           None, # export_symbols, we do this in our def-file
-                           debug, extra_preargs, extra_postargs, build_temp,
-                           target_lang)
+        super(CygwinCCompiler, self).link(
+            target_desc, objects, output_filename, output_dir, libraries,
+            library_dirs, runtime_library_dirs,
+            None, # export_symbols, we do this in our def-file
+            debug, extra_preargs, extra_postargs, build_temp, target_lang)
 
     # -- Miscellaneous methods -----------------------------------------
 
diff --git a/Lib/packaging/tests/pypi_server.py b/Lib/packaging/tests/pypi_server.py
--- a/Lib/packaging/tests/pypi_server.py
+++ b/Lib/packaging/tests/pypi_server.py
@@ -33,7 +33,6 @@
 import queue
 import select
 import threading
-import socketserver
 from functools import wraps
 from http.server import HTTPServer, SimpleHTTPRequestHandler
 from xmlrpc.server import SimpleXMLRPCServer
@@ -270,7 +269,7 @@
 class PyPIXMLRPCServer(SimpleXMLRPCServer):
     def server_bind(self):
         """Override server_bind to store the server name."""
-        socketserver.TCPServer.server_bind(self)
+        super(PyPIXMLRPCServer, self).server_bind()
         host, port = self.socket.getsockname()[:2]
         self.server_port = port
 
@@ -371,12 +370,13 @@
             'requires_python': self.requires_python,
             'classifiers': [],
             'name': self.name,
-            'licence': self.licence,
+            'licence': self.licence,  # XXX licence or license?
             'summary': self.summary,
             'home_page': self.homepage,
             'stable_version': self.stable_version,
-            'provides_dist': self.provides_dist or "%s (%s)" % (self.name,
-                                                              self.version),
+            # FIXME doesn't that reproduce the bug from 6527d3106e9f?
+            'provides_dist': (self.provides_dist or
+                             "%s (%s)" % (self.name, self.version)),
             'requires': self.requires,
             'cheesecake_installability_id': self.cheesecake_installability_id,
         }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:54 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_tests_to_check_initial_?=
 =?utf8?q?content_of_packaging=2Edatabase_caches?=
Message-ID: <E1RQKhu-0007ng-1N@dinsdale.python.org>

http://hg.python.org/cpython/rev/c1e04cf7aac4
changeset:   73565:c1e04cf7aac4
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 19:43:37 2011 +0100
summary:
  Add tests to check initial content of packaging.database caches

files:
  Lib/packaging/tests/test_database.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Lib/packaging/tests/test_database.py b/Lib/packaging/tests/test_database.py
--- a/Lib/packaging/tests/test_database.py
+++ b/Lib/packaging/tests/test_database.py
@@ -10,6 +10,7 @@
 from packaging.tests.test_util import GlobTestCaseBase
 from packaging.tests.support import requires_zlib
 
+import packaging.database
 from packaging.config import get_resources_dests
 from packaging.errors import PackagingError
 from packaging.metadata import Metadata
@@ -279,6 +280,12 @@
         sys.path.insert(0, self.fake_dists_path)
         self.addCleanup(sys.path.remove, self.fake_dists_path)
 
+    def test_caches(self):
+        # sanity check for internal caches
+        for name in ('_cache_name', '_cache_name_egg',
+                     '_cache_path', '_cache_path_egg'):
+            self.assertEqual(getattr(packaging.database, name), {})
+
     def test_distinfo_dirname(self):
         # Given a name and a version, we expect the distinfo_dirname function
         # to return a standard distribution information directory name.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:54 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_The_error_message_should_co?=
 =?utf8?q?ntain_the_key_as_given=2C_not_normalized=2E?=
Message-ID: <E1RQKhu-0007nk-Po@dinsdale.python.org>

http://hg.python.org/cpython/rev/c3c151519124
changeset:   73566:c3c151519124
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 19:45:30 2011 +0100
summary:
  The error message should contain the key as given, not normalized.

Backout of 2e047702df7f.  Reported by Jeremy Kloth.

files:
  Lib/packaging/metadata.py |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -229,8 +229,10 @@
 
     def __delitem__(self, name):
         field_name = self._convert_name(name)
-        # we let a KeyError propagate
-        del self._fields[field_name]
+        try:
+            del self._fields[field_name]
+        except KeyError:
+            raise KeyError(name)
         self._set_best_version()
 
     def __contains__(self, name):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:55 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:55 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_redundant_setUp/tear?=
 =?utf8?q?Down_methods_in_packaging_tests?=
Message-ID: <E1RQKhv-0007no-J4@dinsdale.python.org>

http://hg.python.org/cpython/rev/f1724c576905
changeset:   73567:f1724c576905
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 14 19:46:31 2011 +0100
summary:
  Remove redundant setUp/tearDown methods in packaging tests

files:
  Lib/packaging/tests/test_manifest.py  |  8 --------
  Lib/packaging/tests/test_uninstall.py |  9 ++-------
  Lib/packaging/tests/test_util.py      |  8 --------
  3 files changed, 2 insertions(+), 23 deletions(-)


diff --git a/Lib/packaging/tests/test_manifest.py b/Lib/packaging/tests/test_manifest.py
--- a/Lib/packaging/tests/test_manifest.py
+++ b/Lib/packaging/tests/test_manifest.py
@@ -27,14 +27,6 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
 
-    def setUp(self):
-        super(ManifestTestCase, self).setUp()
-        self.cwd = os.getcwd()
-
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(ManifestTestCase, self).tearDown()
-
     def assertNoWarnings(self):
         self.assertEqual(self.get_logs(), [])
 
diff --git a/Lib/packaging/tests/test_uninstall.py b/Lib/packaging/tests/test_uninstall.py
--- a/Lib/packaging/tests/test_uninstall.py
+++ b/Lib/packaging/tests/test_uninstall.py
@@ -31,14 +31,9 @@
     def setUp(self):
         super(UninstallTestCase, self).setUp()
         self.addCleanup(enable_cache)
-        self.root_dir = self.mkdtemp()
-        self.cwd = os.getcwd()
+        self.addCleanup(packaging.util._path_created.clear)
         disable_cache()
 
-    def tearDown(self):
-        packaging.util._path_created.clear()
-        super(UninstallTestCase, self).tearDown()
-
     def get_path(self, dist, name):
         # the dist argument must contain an install_dist command correctly
         # initialized with a prefix option and finalized befored this method
@@ -79,7 +74,7 @@
         dist.parse_config_files()
         dist.finalize_options()
         dist.run_command('install_dist',
-                         {'prefix': ('command line', self.root_dir)})
+                         {'prefix': ('command line', self.mkdtemp())})
 
         site_packages = self.get_path(dist, 'purelib')
         return dist, site_packages
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -602,14 +602,6 @@
 
 class GlobTestCase(GlobTestCaseBase):
 
-    def setUp(self):
-        super(GlobTestCase, self).setUp()
-        self.cwd = os.getcwd()
-
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(GlobTestCase, self).tearDown()
-
     def assertGlobMatch(self, glob, spec):
         tempdir = self.build_files_tree(spec)
         expected = self.clean_tree(spec)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:58 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_unused_code_from_pac?=
 =?utf8?b?a2FnaW5nLnRlc3RzLl9faW5pdF9f?=
Message-ID: <E1RQKhy-0007ns-MX@dinsdale.python.org>

http://hg.python.org/cpython/rev/f377dd55c097
changeset:   73568:f377dd55c097
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 10:48:36 2011 +0100
summary:
  Remove unused code from packaging.tests.__init__

files:
  Lib/packaging/tests/__init__.py           |  104 +---------
  Lib/packaging/tests/test_command_bdist.py |    7 +-
  Lib/packaging/tests/test_command_sdist.py |    6 +-
  Lib/packaging/tests/test_dist.py          |    1 -
  4 files changed, 10 insertions(+), 108 deletions(-)


diff --git a/Lib/packaging/tests/__init__.py b/Lib/packaging/tests/__init__.py
--- a/Lib/packaging/tests/__init__.py
+++ b/Lib/packaging/tests/__init__.py
@@ -14,16 +14,11 @@
 import os
 import sys
 import unittest
-from io import StringIO
 
-# XXX move helpers to support, add tests for them, remove things that
-# duplicate test.support (or keep them for the backport; needs thinking)
-
-here = os.path.dirname(__file__) or os.curdir
-verbose = 1
 
 def test_suite():
     suite = unittest.TestSuite()
+    here = os.path.dirname(__file__) or os.curdir
     for fn in os.listdir(here):
         if fn.startswith("test") and fn.endswith(".py"):
             modname = "packaging.tests." + fn[:-3]
@@ -31,100 +26,3 @@
             module = sys.modules[modname]
             suite.addTest(module.test_suite())
     return suite
-
-
-class Error(Exception):
-    """Base class for regression test exceptions."""
-
-
-class TestFailed(Error):
-    """Test failed."""
-
-
-class BasicTestRunner:
-    def run(self, test):
-        result = unittest.TestResult()
-        test(result)
-        return result
-
-
-def _run_suite(suite, verbose_=1):
-    """Run tests from a unittest.TestSuite-derived class."""
-    global verbose
-    verbose = verbose_
-    if verbose_:
-        runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
-    else:
-        runner = BasicTestRunner()
-
-    result = runner.run(suite)
-    if not result.wasSuccessful():
-        if len(result.errors) == 1 and not result.failures:
-            err = result.errors[0][1]
-        elif len(result.failures) == 1 and not result.errors:
-            err = result.failures[0][1]
-        else:
-            err = "errors occurred; run in verbose mode for details"
-        raise TestFailed(err)
-
-
-def run_unittest(classes, verbose_=1):
-    """Run tests from unittest.TestCase-derived classes.
-
-    Originally extracted from stdlib test.test_support and modified to
-    support unittest2.
-    """
-    valid_types = (unittest.TestSuite, unittest.TestCase)
-    suite = unittest.TestSuite()
-    for cls in classes:
-        if isinstance(cls, str):
-            if cls in sys.modules:
-                suite.addTest(unittest.findTestCases(sys.modules[cls]))
-            else:
-                raise ValueError("str arguments must be keys in sys.modules")
-        elif isinstance(cls, valid_types):
-            suite.addTest(cls)
-        else:
-            suite.addTest(unittest.makeSuite(cls))
-    _run_suite(suite, verbose_)
-
-
-def reap_children():
-    """Use this function at the end of test_main() whenever sub-processes
-    are started.  This will help ensure that no extra children (zombies)
-    stick around to hog resources and create problems when looking
-    for refleaks.
-
-    Extracted from stdlib test.support.
-    """
-
-    # Reap all our dead child processes so we don't leave zombies around.
-    # These hog resources and might be causing some of the buildbots to die.
-    if hasattr(os, 'waitpid'):
-        any_process = -1
-        while True:
-            try:
-                # This will raise an exception on Windows.  That's ok.
-                pid, status = os.waitpid(any_process, os.WNOHANG)
-                if pid == 0:
-                    break
-            except:
-                break
-
-
-def captured_stdout(func, *args, **kw):
-    orig_stdout = getattr(sys, 'stdout')
-    setattr(sys, 'stdout', StringIO())
-    try:
-        res = func(*args, **kw)
-        sys.stdout.seek(0)
-        return res, sys.stdout.read()
-    finally:
-        setattr(sys, 'stdout', orig_stdout)
-
-
-def unload(name):
-    try:
-        del sys.modules[name]
-    except KeyError:
-        pass
diff --git a/Lib/packaging/tests/test_command_bdist.py b/Lib/packaging/tests/test_command_bdist.py
--- a/Lib/packaging/tests/test_command_bdist.py
+++ b/Lib/packaging/tests/test_command_bdist.py
@@ -1,7 +1,8 @@
 """Tests for distutils.command.bdist."""
 import os
+from test.support import captured_stdout
 from packaging.command.bdist import bdist, show_formats
-from packaging.tests import unittest, support, captured_stdout
+from packaging.tests import unittest, support
 
 
 class BuildTestCase(support.TempdirManager,
@@ -42,7 +43,9 @@
                             '%s should take --skip-build from bdist' % name)
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        with captured_stdout() as stdout:
+            show_formats()
+        stdout = stdout.getvalue()
 
         # the output should be a header line + one line per format
         num_formats = len(bdist.format_commands)
diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -17,8 +17,8 @@
 from packaging.errors import PackagingOptionError
 from packaging.command.sdist import sdist, show_formats
 
+from test.support import captured_stdout
 from packaging.tests import support, unittest
-from packaging.tests import captured_stdout
 from packaging.tests.support import requires_zlib
 
 
@@ -234,7 +234,9 @@
         self.assertIn("'setup.cfg' file not found", warnings[1])
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        with captured_stdout() as stdout:
+            show_formats()
+        stdout = stdout.getvalue()
 
         # the output should be a header line + one line per format
         num_formats = len(get_archive_formats())
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -8,7 +8,6 @@
 from packaging.dist import Distribution
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
-from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
 from packaging.tests.support import create_distribution, use_command
 from test.support import unload

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:13:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:13:59 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_obsolete_verbose_arg?=
	=?utf8?q?uments_from_packaging=2E?=
Message-ID: <E1RQKhz-0007nw-E3@dinsdale.python.org>

http://hg.python.org/cpython/rev/fff38970e4e3
changeset:   73569:fff38970e4e3
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 11:43:20 2011 +0100
summary:
  Remove obsolete verbose arguments from packaging.

Logging replaces verbose arguments.  I haven?t fixed the example in
Doc/install/install.rst because I have major fixes and changes to the
oc under way and will fix or remove that example as part of that task.

files:
  Doc/install/install.rst                   |   2 +
  Doc/library/packaging.compiler.rst        |  10 ++--
  Doc/library/packaging.util.rst            |   2 +-
  Lib/packaging/command/build_ext.py        |   6 --
  Lib/packaging/command/cmd.py              |  18 +++++---
  Lib/packaging/command/sdist.py            |   5 +-
  Lib/packaging/command/test.py             |   3 +-
  Lib/packaging/compiler/__init__.py        |   9 +---
  Lib/packaging/compiler/bcppcompiler.py    |   4 +-
  Lib/packaging/compiler/ccompiler.py       |   3 +-
  Lib/packaging/compiler/cygwinccompiler.py |   8 +-
  Lib/packaging/compiler/msvc9compiler.py   |   4 +-
  Lib/packaging/compiler/msvccompiler.py    |   4 +-
  Lib/packaging/util.py                     |  23 ++++------
  14 files changed, 44 insertions(+), 57 deletions(-)


diff --git a/Doc/install/install.rst b/Doc/install/install.rst
--- a/Doc/install/install.rst
+++ b/Doc/install/install.rst
@@ -842,6 +842,8 @@
 Sections consist of one or more lines containing a single option specified as
 ``option = value``.
 
+.. XXX use dry-run in the next example or use a pysetup option as example
+
 For example, here's a complete configuration file that forces all commands to
 run quietly by default::
 
diff --git a/Doc/library/packaging.compiler.rst b/Doc/library/packaging.compiler.rst
--- a/Doc/library/packaging.compiler.rst
+++ b/Doc/library/packaging.compiler.rst
@@ -15,7 +15,7 @@
 Public functions
 ----------------
 
-.. function:: new_compiler(plat=None, compiler=None, verbose=0, dry_run=0, force=0)
+.. function:: new_compiler(plat=None, compiler=None, dry_run=False, force=False)
 
    Factory function to generate an instance of some
    :class:`~.ccompiler.CCompiler` subclass for the requested platform or
@@ -165,7 +165,7 @@
 options for the compiler --- macro definitions, include directories, link path,
 libraries and the like.
 
-.. class:: CCompiler([verbose=0, dry_run=0, force=0])
+.. class:: CCompiler(dry_run=False, force=False)
 
    The abstract base class :class:`CCompiler` defines the interface that must be
    implemented by real compiler classes.  The class also has some utility
@@ -180,11 +180,11 @@
    per-compilation or per-link basis.
 
    The constructor for each subclass creates an instance of the Compiler object.
-   Flags are *verbose* (show verbose output), *dry_run* (don't actually execute
+   Flags are *dry_run* (don't actually execute
    the steps) and *force* (rebuild everything, regardless of dependencies).  All
-   of these flags default to ``0`` (off). Note that you probably don't want to
+   of these flags default to ``False`` (off). Note that you probably don't want to
    instantiate :class:`CCompiler` or one of its subclasses directly - use the
-   :func:`packaging.CCompiler.new_compiler` factory function instead.
+   :func:`new_compiler` factory function instead.
 
    The following methods allow you to manually alter compiler options for the
    instance of the Compiler class.
diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -90,7 +90,7 @@
    Search the path for a given executable name.
 
 
-.. function:: execute(func, args, msg=None, verbose=0, dry_run=0)
+.. function:: execute(func, args, msg=None, dry_run=False)
 
    Perform some action that affects the outside world (for instance, writing to
    the filesystem).  Such actions are special because they are disabled by the
diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -4,7 +4,6 @@
 import re
 import sys
 import site
-import logging
 import sysconfig
 
 from packaging.util import get_platform
@@ -288,14 +287,9 @@
             self.libraries.extend(build_clib.get_library_names() or [])
             self.library_dirs.append(build_clib.build_clib)
 
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
         # Setup the CCompiler object that we'll use to do all the
         # compiling and linking
         self.compiler_obj = new_compiler(compiler=self.compiler,
-                                         verbose=verbose,
                                          dry_run=self.dry_run,
                                          force=self.force)
 
diff --git a/Lib/packaging/command/cmd.py b/Lib/packaging/command/cmd.py
--- a/Lib/packaging/command/cmd.py
+++ b/Lib/packaging/command/cmd.py
@@ -351,7 +351,7 @@
     def execute(self, func, args, msg=None, level=1):
         util.execute(func, args, msg, dry_run=self.dry_run)
 
-    def mkpath(self, name, mode=0o777, dry_run=None, verbose=0):
+    def mkpath(self, name, mode=0o777, dry_run=None):
         if dry_run is None:
             dry_run = self.dry_run
         name = os.path.normpath(name)
@@ -367,9 +367,11 @@
 
     def copy_file(self, infile, outfile,
                   preserve_mode=True, preserve_times=True, link=None, level=1):
-        """Copy a file respecting verbose, dry-run and force flags.  (The
-        former two default to whatever is in the Distribution object, and
-        the latter defaults to false for commands that don't define it.)"""
+        """Copy a file respecting dry-run and force flags.
+
+        (dry-run defaults to whatever is in the Distribution object, and
+        force to false for commands that don't define it.)
+        """
         if self.dry_run:
             # XXX add a comment
             return
@@ -380,11 +382,13 @@
 
     def copy_tree(self, infile, outfile, preserve_mode=True,
                   preserve_times=True, preserve_symlinks=False, level=1):
-        """Copy an entire directory tree respecting verbose, dry-run,
+        """Copy an entire directory tree respecting dry-run
         and force flags.
         """
         if self.dry_run:
-            return  # see if we want to display something
+            # XXX should not return but let copy_tree log and decide to execute
+            # or not based on its dry_run argument
+            return
 
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
@@ -392,7 +396,7 @@
     def move_file(self, src, dst, level=1):
         """Move a file respecting the dry-run flag."""
         if self.dry_run:
-            return  # XXX log ?
+            return  # XXX same thing
         return move(src, dst)
 
     def spawn(self, cmd, search_path=True, level=1):
diff --git a/Lib/packaging/command/sdist.py b/Lib/packaging/command/sdist.py
--- a/Lib/packaging/command/sdist.py
+++ b/Lib/packaging/command/sdist.py
@@ -337,12 +337,11 @@
         """
         return self.archive_files
 
-    def create_tree(self, base_dir, files, mode=0o777, verbose=1,
-                    dry_run=False):
+    def create_tree(self, base_dir, files, mode=0o777, dry_run=False):
         need_dir = set()
         for file in files:
             need_dir.add(os.path.join(base_dir, os.path.dirname(file)))
 
         # Now create them
         for dir in sorted(need_dir):
-            self.mkpath(dir, mode, verbose=verbose, dry_run=dry_run)
+            self.mkpath(dir, mode, dry_run=dry_run)
diff --git a/Lib/packaging/command/test.py b/Lib/packaging/command/test.py
--- a/Lib/packaging/command/test.py
+++ b/Lib/packaging/command/test.py
@@ -60,8 +60,7 @@
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
-            # Temporary kludge until we remove the verbose arguments and use
-            # logging everywhere
+            # XXX maybe we could pass the verbose argument of pysetup here
             logger = logging.getLogger('packaging')
             verbose = logger.getEffectiveLevel() >= logging.DEBUG
             verbosity = verbose + 1
diff --git a/Lib/packaging/compiler/__init__.py b/Lib/packaging/compiler/__init__.py
--- a/Lib/packaging/compiler/__init__.py
+++ b/Lib/packaging/compiler/__init__.py
@@ -153,8 +153,7 @@
     pretty_printer.print_help("List of available compilers:")
 
 
-def new_compiler(plat=None, compiler=None, verbose=0, dry_run=False,
-                 force=False):
+def new_compiler(plat=None, compiler=None, dry_run=False, force=False):
     """Generate an instance of some CCompiler subclass for the supplied
     platform/compiler combination.  'plat' defaults to 'os.name'
     (eg. 'posix', 'nt'), and 'compiler' defaults to the default compiler
@@ -183,11 +182,7 @@
         cls = resolve_name(cls)
         _COMPILERS[compiler] = cls
 
-
-    # XXX The None is necessary to preserve backwards compatibility
-    # with classes that expect verbose to be the first positional
-    # argument.
-    return cls(None, dry_run, force)
+    return cls(dry_run, force)
 
 
 def gen_preprocess_options(macros, include_dirs):
diff --git a/Lib/packaging/compiler/bcppcompiler.py b/Lib/packaging/compiler/bcppcompiler.py
--- a/Lib/packaging/compiler/bcppcompiler.py
+++ b/Lib/packaging/compiler/bcppcompiler.py
@@ -47,8 +47,8 @@
     exe_extension = '.exe'
 
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(BCPPCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(BCPPCompiler, self).__init__(dry_run, force)
 
         # These executables are assumed to all be in the path.
         # Borland doesn't seem to use any special registry settings to
diff --git a/Lib/packaging/compiler/ccompiler.py b/Lib/packaging/compiler/ccompiler.py
--- a/Lib/packaging/compiler/ccompiler.py
+++ b/Lib/packaging/compiler/ccompiler.py
@@ -79,10 +79,9 @@
                    }
     language_order = ["c++", "objc", "c"]
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
+    def __init__(self, dry_run=False, force=False):
         self.dry_run = dry_run
         self.force = force
-        self.verbose = verbose
 
         # 'output_dir': a common output directory for object, library,
         # shared object, and shared library files
diff --git a/Lib/packaging/compiler/cygwinccompiler.py b/Lib/packaging/compiler/cygwinccompiler.py
--- a/Lib/packaging/compiler/cygwinccompiler.py
+++ b/Lib/packaging/compiler/cygwinccompiler.py
@@ -92,8 +92,8 @@
     shared_lib_format = "%s%s"
     exe_extension = ".exe"
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(CygwinCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(CygwinCCompiler, self).__init__(dry_run, force)
 
         status, details = check_config_h()
         logger.debug("Python's GCC status: %s (details: %s)", status, details)
@@ -270,8 +270,8 @@
     name = 'mingw32'
     description = 'MinGW32 compiler'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(Mingw32CCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(Mingw32CCompiler, self).__init__(dry_run, force)
 
         # ld_version >= "2.13" support -shared so use it instead of
         # -mdll -static
diff --git a/Lib/packaging/compiler/msvc9compiler.py b/Lib/packaging/compiler/msvc9compiler.py
--- a/Lib/packaging/compiler/msvc9compiler.py
+++ b/Lib/packaging/compiler/msvc9compiler.py
@@ -309,8 +309,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = VERSION
         self.__root = r"Software\Microsoft\VisualStudio"
         # self.__macros = MACROS
diff --git a/Lib/packaging/compiler/msvccompiler.py b/Lib/packaging/compiler/msvccompiler.py
--- a/Lib/packaging/compiler/msvccompiler.py
+++ b/Lib/packaging/compiler/msvccompiler.py
@@ -236,8 +236,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = get_build_version()
         self.__arch = get_build_architecture()
         if self.__arch == "Intel":
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -259,7 +259,7 @@
             if element]
 
 
-def execute(func, args, msg=None, verbose=0, dry_run=False):
+def execute(func, args, msg=None, dry_run=False):
     """Perform some action that affects the outside world.
 
     Some actions (e.g. writing to the filesystem) are special because
@@ -681,7 +681,7 @@
     _cfg_target_split = None
 
 
-def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
+def spawn(cmd, search_path=True, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
     'cmd' is just the argument list for the new process, ie.
@@ -1331,8 +1331,7 @@
 
 # XXX to be replaced by shutil.copytree
 def copy_tree(src, dst, preserve_mode=True, preserve_times=True,
-              preserve_symlinks=False, update=False, verbose=True,
-              dry_run=False):
+              preserve_symlinks=False, update=False, dry_run=False):
     # FIXME use of this function is why we get spurious logging message on
     # stdout when tests run; kill and replace by shutil!
     from distutils.file_util import copy_file
@@ -1351,7 +1350,7 @@
                   "error listing files in '%s': %s" % (src, errstr))
 
     if not dry_run:
-        _mkpath(dst, verbose=verbose)
+        _mkpath(dst)
 
     outputs = []
 
@@ -1361,8 +1360,7 @@
 
         if preserve_symlinks and os.path.islink(src_name):
             link_dest = os.readlink(src_name)
-            if verbose >= 1:
-                logger.info("linking %s -> %s", dst_name, link_dest)
+            logger.info("linking %s -> %s", dst_name, link_dest)
             if not dry_run:
                 os.symlink(link_dest, dst_name)
             outputs.append(dst_name)
@@ -1371,11 +1369,10 @@
             outputs.extend(
                 copy_tree(src_name, dst_name, preserve_mode,
                           preserve_times, preserve_symlinks, update,
-                          verbose=verbose, dry_run=dry_run))
+                          dry_run=dry_run))
         else:
             copy_file(src_name, dst_name, preserve_mode,
-                      preserve_times, update, verbose=verbose,
-                      dry_run=dry_run)
+                      preserve_times, update, dry_run=dry_run)
             outputs.append(dst_name)
 
     return outputs
@@ -1388,7 +1385,7 @@
 # I don't use os.makedirs because a) it's new to Python 1.5.2, and
 # b) it blows up if the directory already exists (I want to silently
 # succeed in that case).
-def _mkpath(name, mode=0o777, verbose=True, dry_run=False):
+def _mkpath(name, mode=0o777, dry_run=False):
     # Detect a common bug -- name is None
     if not isinstance(name, str):
         raise PackagingInternalError(
@@ -1423,9 +1420,7 @@
         if abs_head in _path_created:
             continue
 
-        if verbose >= 1:
-            logger.info("creating %s", head)
-
+        logger.info("creating %s", head)
         if not dry_run:
             try:
                 os.mkdir(head, mode)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:14:00 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:14:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Branch_merge?=
Message-ID: <E1RQKi0-0007o0-Co@dinsdale.python.org>

http://hg.python.org/cpython/rev/69fb7933434a
changeset:   73570:69fb7933434a
parent:      73557:c512a7f5e0b3
parent:      73569:fff38970e4e3
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 16:12:22 2011 +0100
summary:
  Branch merge

files:
  Doc/install/install.rst                         |    2 +
  Doc/install/pysetup.rst                         |    2 +-
  Doc/library/packaging.compiler.rst              |   10 +-
  Doc/library/packaging.util.rst                  |   11 +-
  Doc/packaging/setupcfg.rst                      |    2 +-
  Doc/whatsnew/3.3.rst                            |    2 +-
  Lib/packaging/command/build_ext.py              |    6 -
  Lib/packaging/command/build_py.py               |   29 +--
  Lib/packaging/command/cmd.py                    |   38 ++-
  Lib/packaging/command/install_lib.py            |   42 +--
  Lib/packaging/command/sdist.py                  |    5 +-
  Lib/packaging/command/test.py                   |    3 +-
  Lib/packaging/compiler/__init__.py              |    9 +-
  Lib/packaging/compiler/bcppcompiler.py          |    4 +-
  Lib/packaging/compiler/ccompiler.py             |    3 +-
  Lib/packaging/compiler/cygwinccompiler.py       |   19 +-
  Lib/packaging/compiler/msvc9compiler.py         |    4 +-
  Lib/packaging/compiler/msvccompiler.py          |    4 +-
  Lib/packaging/metadata.py                       |    6 +-
  Lib/packaging/run.py                            |    2 +-
  Lib/packaging/tests/__init__.py                 |  104 +---------
  Lib/packaging/tests/pypi_server.py              |   10 +-
  Lib/packaging/tests/test_command_bdist.py       |    7 +-
  Lib/packaging/tests/test_command_install_lib.py |    4 +-
  Lib/packaging/tests/test_command_sdist.py       |    6 +-
  Lib/packaging/tests/test_config.py              |    2 +-
  Lib/packaging/tests/test_database.py            |    7 +
  Lib/packaging/tests/test_dist.py                |    1 -
  Lib/packaging/tests/test_manifest.py            |    8 -
  Lib/packaging/tests/test_uninstall.py           |    9 +-
  Lib/packaging/tests/test_util.py                |    8 -
  Lib/packaging/util.py                           |   52 ++--
  Lib/test/regrtest.py                            |    2 +-
  Misc/ACKS                                       |    5 +
  Misc/NEWS                                       |   10 -
  35 files changed, 144 insertions(+), 294 deletions(-)


diff --git a/Doc/install/install.rst b/Doc/install/install.rst
--- a/Doc/install/install.rst
+++ b/Doc/install/install.rst
@@ -842,6 +842,8 @@
 Sections consist of one or more lines containing a single option specified as
 ``option = value``.
 
+.. XXX use dry-run in the next example or use a pysetup option as example
+
 For example, here's a complete configuration file that forces all commands to
 run quietly by default::
 
diff --git a/Doc/install/pysetup.rst b/Doc/install/pysetup.rst
--- a/Doc/install/pysetup.rst
+++ b/Doc/install/pysetup.rst
@@ -149,7 +149,7 @@
        list: List installed projects
        graph: Display a graph
        create: Create a project
-       generate-setup: Generate a backward-comptatible setup.py
+       generate-setup: Generate a backward-compatible setup.py
 
    To get more help on an action, use:
 
diff --git a/Doc/library/packaging.compiler.rst b/Doc/library/packaging.compiler.rst
--- a/Doc/library/packaging.compiler.rst
+++ b/Doc/library/packaging.compiler.rst
@@ -15,7 +15,7 @@
 Public functions
 ----------------
 
-.. function:: new_compiler(plat=None, compiler=None, verbose=0, dry_run=0, force=0)
+.. function:: new_compiler(plat=None, compiler=None, dry_run=False, force=False)
 
    Factory function to generate an instance of some
    :class:`~.ccompiler.CCompiler` subclass for the requested platform or
@@ -165,7 +165,7 @@
 options for the compiler --- macro definitions, include directories, link path,
 libraries and the like.
 
-.. class:: CCompiler([verbose=0, dry_run=0, force=0])
+.. class:: CCompiler(dry_run=False, force=False)
 
    The abstract base class :class:`CCompiler` defines the interface that must be
    implemented by real compiler classes.  The class also has some utility
@@ -180,11 +180,11 @@
    per-compilation or per-link basis.
 
    The constructor for each subclass creates an instance of the Compiler object.
-   Flags are *verbose* (show verbose output), *dry_run* (don't actually execute
+   Flags are *dry_run* (don't actually execute
    the steps) and *force* (rebuild everything, regardless of dependencies).  All
-   of these flags default to ``0`` (off). Note that you probably don't want to
+   of these flags default to ``False`` (off). Note that you probably don't want to
    instantiate :class:`CCompiler` or one of its subclasses directly - use the
-   :func:`packaging.CCompiler.new_compiler` factory function instead.
+   :func:`new_compiler` factory function instead.
 
    The following methods allow you to manually alter compiler options for the
    instance of the Compiler class.
diff --git a/Doc/library/packaging.util.rst b/Doc/library/packaging.util.rst
--- a/Doc/library/packaging.util.rst
+++ b/Doc/library/packaging.util.rst
@@ -90,7 +90,7 @@
    Search the path for a given executable name.
 
 
-.. function:: execute(func, args[, msg=None, verbose=0, dry_run=0])
+.. function:: execute(func, args, msg=None, dry_run=False)
 
    Perform some action that affects the outside world (for instance, writing to
    the filesystem).  Such actions are special because they are disabled by the
@@ -117,7 +117,8 @@
    :exc:`ValueError` if *val* is anything else.
 
 
-.. function:: byte_compile(py_files[, optimize=0, force=0, prefix=None, base_dir=None, verbose=1, dry_run=0, direct=None])
+.. function:: byte_compile(py_files, optimize=0, force=0, prefix=None, \
+                           base_dir=None, dry_run=0, direct=None)
 
    Byte-compile a collection of Python source files to either :file:`.pyc` or
    :file:`.pyo` files in a :file:`__pycache__` subdirectory (see :pep:`3147`),
@@ -131,6 +132,9 @@
    * ``1`` - normal optimization (like ``python -O``)
    * ``2`` - extra optimization (like ``python -OO``)
 
+   This function is independent from the running Python's :option:`-O` or
+   :option:`-B` options; it is fully controlled by the parameters passed in.
+
    If *force* is true, all files are recompiled regardless of timestamps.
 
    The source filename encoded in each :term:`bytecode` file defaults to the filenames
@@ -149,6 +153,3 @@
    figure out to use direct compilation or not (see the source for details).
    The *direct* flag is used by the script generated in indirect mode; unless
    you know what you're doing, leave it set to ``None``.
-
-   This function is independent from the running Python's :option:`-O` or
-   :option:`-B` options; it is fully controlled by the parameters passed in.
diff --git a/Doc/packaging/setupcfg.rst b/Doc/packaging/setupcfg.rst
--- a/Doc/packaging/setupcfg.rst
+++ b/Doc/packaging/setupcfg.rst
@@ -779,7 +779,7 @@
    extra_compile_args =
        -fPIC -O2
        -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
-       /DGECODE_VERSION='win32' -- sys.platform == 'win32'
+       /DGECODE_VERSION=win32 -- sys.platform == 'win32'
 
 The section name must start with ``extension:``; the right-hand part is used as
 the full name (including a parent package, if any) of the extension.  Whitespace
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -596,7 +596,7 @@
    paths.  In previous versions, it did.  See changeset for doc changes in
    various files.  Contributed by Carl Meyer with editions by ?ric Araujo.
 
-.. Issue #10998: -Q command-line flags are related artifacts have been
+.. Issue #10998: the -Q command-line flag and related artifacts have been
    removed.  Code checking sys.flags.division_warning will need updating.
    Contributed by ?ric Araujo.
 
diff --git a/Lib/packaging/command/build_ext.py b/Lib/packaging/command/build_ext.py
--- a/Lib/packaging/command/build_ext.py
+++ b/Lib/packaging/command/build_ext.py
@@ -4,7 +4,6 @@
 import re
 import sys
 import site
-import logging
 import sysconfig
 
 from packaging.util import get_platform
@@ -288,14 +287,9 @@
             self.libraries.extend(build_clib.get_library_names() or [])
             self.library_dirs.append(build_clib.build_clib)
 
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
         # Setup the CCompiler object that we'll use to do all the
         # compiling and linking
         self.compiler_obj = new_compiler(compiler=self.compiler,
-                                         verbose=verbose,
                                          dry_run=self.dry_run,
                                          force=self.force)
 
diff --git a/Lib/packaging/command/build_py.py b/Lib/packaging/command/build_py.py
--- a/Lib/packaging/command/build_py.py
+++ b/Lib/packaging/command/build_py.py
@@ -18,7 +18,7 @@
 
     description = "build pure Python modules (copy to build directory)"
 
-    # The options for controlling byte compilations are two independent sets;
+    # The options for controlling byte compilation are two independent sets;
     # more info in install_lib or the reST docs
 
     user_options = [
@@ -113,7 +113,8 @@
             self.run_2to3(self._updated_files, self._doctests_2to3,
                                             self.use_2to3_fixers)
 
-        self.byte_compile(self.get_outputs(include_bytecode=False))
+        self.byte_compile(self.get_outputs(include_bytecode=False),
+                          prefix=self.build_lib)
 
     # -- Top-level worker functions ------------------------------------
 
@@ -335,11 +336,9 @@
             outputs.append(filename)
             if include_bytecode:
                 if self.compile:
-                    outputs.append(imp.cache_from_source(filename,
-                                                         debug_override=True))
-                if self.optimize > 0:
-                    outputs.append(imp.cache_from_source(filename,
-                                                         debug_override=False))
+                    outputs.append(imp.cache_from_source(filename, True))
+                if self.optimize:
+                    outputs.append(imp.cache_from_source(filename, False))
 
         outputs += [
             os.path.join(build_dir, filename)
@@ -391,19 +390,3 @@
             for package_, module, module_file in modules:
                 assert package == package_
                 self.build_module(module, module_file, package)
-
-    def byte_compile(self, files):
-        from packaging.util import byte_compile  # FIXME use compileall
-        prefix = self.build_lib
-        if prefix[-1] != os.sep:
-            prefix = prefix + os.sep
-
-        # XXX this code is essentially the same as the 'byte_compile()
-        # method of the "install_lib" command, except for the determination
-        # of the 'prefix' string.  Hmmm.
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=prefix, dry_run=self.dry_run)
diff --git a/Lib/packaging/command/cmd.py b/Lib/packaging/command/cmd.py
--- a/Lib/packaging/command/cmd.py
+++ b/Lib/packaging/command/cmd.py
@@ -10,7 +10,7 @@
 
 class Command:
     """Abstract base class for defining command classes, the "worker bees"
-    of the Packaging.  A useful analogy for command classes is to think of
+    of Packaging.  A useful analogy for command classes is to think of
     them as subroutines with local variables called "options".  The options
     are "declared" in 'initialize_options()' and "defined" (given their
     final values, aka "finalized") in 'finalize_options()', both of which
@@ -351,7 +351,7 @@
     def execute(self, func, args, msg=None, level=1):
         util.execute(func, args, msg, dry_run=self.dry_run)
 
-    def mkpath(self, name, mode=0o777, dry_run=None, verbose=0):
+    def mkpath(self, name, mode=0o777, dry_run=None):
         if dry_run is None:
             dry_run = self.dry_run
         name = os.path.normpath(name)
@@ -367,9 +367,11 @@
 
     def copy_file(self, infile, outfile,
                   preserve_mode=True, preserve_times=True, link=None, level=1):
-        """Copy a file respecting verbose, dry-run and force flags.  (The
-        former two default to whatever is in the Distribution object, and
-        the latter defaults to false for commands that don't define it.)"""
+        """Copy a file respecting dry-run and force flags.
+
+        (dry-run defaults to whatever is in the Distribution object, and
+        force to false for commands that don't define it.)
+        """
         if self.dry_run:
             # XXX add a comment
             return
@@ -380,12 +382,13 @@
 
     def copy_tree(self, infile, outfile, preserve_mode=True,
                   preserve_times=True, preserve_symlinks=False, level=1):
-        """Copy an entire directory tree respecting verbose, dry-run,
+        """Copy an entire directory tree respecting dry-run
         and force flags.
         """
         if self.dry_run:
-            return  # see if we want to display something
-
+            # XXX should not return but let copy_tree log and decide to execute
+            # or not based on its dry_run argument
+            return
 
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
@@ -393,7 +396,7 @@
     def move_file(self, src, dst, level=1):
         """Move a file respecting the dry-run flag."""
         if self.dry_run:
-            return  # XXX log ?
+            return  # XXX same thing
         return move(src, dst)
 
     def spawn(self, cmd, search_path=True, level=1):
@@ -439,3 +442,20 @@
         # Otherwise, print the "skip" message
         else:
             logger.debug(skip_msg)
+
+    def byte_compile(self, files, prefix=None):
+        """Byte-compile files to pyc and/or pyo files.
+
+        This method requires that the calling class define compile and
+        optimize options, like build_py and install_lib.  It also
+        automatically respects the force and dry-run options.
+
+        prefix, if given, is a string that will be stripped off the
+        filenames encoded in bytecode files.
+        """
+        if self.compile:
+            util.byte_compile(files, optimize=False, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
+        if self.optimize:
+            util.byte_compile(files, optimize=self.optimize, prefix=prefix,
+                              force=self.force, dry_run=self.dry_run)
diff --git a/Lib/packaging/command/install_lib.py b/Lib/packaging/command/install_lib.py
--- a/Lib/packaging/command/install_lib.py
+++ b/Lib/packaging/command/install_lib.py
@@ -2,7 +2,6 @@
 
 import os
 import imp
-import logging
 
 from packaging import logger
 from packaging.command.cmd import Command
@@ -21,7 +20,7 @@
 
     description = "install all modules (extensions and pure Python)"
 
-    # The options for controlling byte compilations are two independent sets:
+    # The options for controlling byte compilation are two independent sets:
     # 'compile' is strictly boolean, and only decides whether to
     # generate .pyc files.  'optimize' is three-way (0, 1, or 2), and
     # decides both whether to generate .pyo files and what level of
@@ -84,9 +83,14 @@
         # having a build directory!)
         outfiles = self.install()
 
-        # (Optionally) compile .py to .pyc
+        # (Optionally) compile .py to .pyc and/or .pyo
         if outfiles is not None and self.distribution.has_pure_modules():
-            self.byte_compile(outfiles)
+            # XXX comment from distutils: "This [prefix stripping] is far from
+            # complete, but it should at least generate usable bytecode in RPM
+            # distributions." -> need to find exact requirements for
+            # byte-compiled files and fix it
+            install_root = self.get_finalized_command('install_dist').root
+            self.byte_compile(outfiles, prefix=install_root)
 
     # -- Top-level worker functions ------------------------------------
     # (called from 'run()')
@@ -108,28 +112,6 @@
             return
         return outfiles
 
-    def byte_compile(self, files):
-        from packaging.util import byte_compile  # FIXME use compileall
-
-        # Get the "--root" directory supplied to the "install_dist" command,
-        # and use it as a prefix to strip off the purported filename
-        # encoded in bytecode files.  This is far from complete, but it
-        # should at least generate usable bytecode in RPM distributions.
-        install_root = self.get_finalized_command('install_dist').root
-
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
-        if self.compile:
-            byte_compile(files, optimize=0,
-                         force=self.force, prefix=install_root,
-                         verbose=verbose, dry_run=self.dry_run)
-        if self.optimize > 0:
-            byte_compile(files, optimize=self.optimize,
-                         force=self.force, prefix=install_root,
-                         verbose=verbose, dry_run=self.dry_run)
-
     # -- Utility methods -----------------------------------------------
 
     def _mutate_outputs(self, has_any, build_cmd, cmd_option, output_dir):
@@ -157,11 +139,9 @@
             if ext != PYTHON_SOURCE_EXTENSION:
                 continue
             if self.compile:
-                bytecode_files.append(imp.cache_from_source(
-                    py_file, debug_override=True))
-            if self.optimize > 0:
-                bytecode_files.append(imp.cache_from_source(
-                    py_file, debug_override=False))
+                bytecode_files.append(imp.cache_from_source(py_file, True))
+            if self.optimize:
+                bytecode_files.append(imp.cache_from_source(py_file, False))
 
         return bytecode_files
 
diff --git a/Lib/packaging/command/sdist.py b/Lib/packaging/command/sdist.py
--- a/Lib/packaging/command/sdist.py
+++ b/Lib/packaging/command/sdist.py
@@ -337,12 +337,11 @@
         """
         return self.archive_files
 
-    def create_tree(self, base_dir, files, mode=0o777, verbose=1,
-                    dry_run=False):
+    def create_tree(self, base_dir, files, mode=0o777, dry_run=False):
         need_dir = set()
         for file in files:
             need_dir.add(os.path.join(base_dir, os.path.dirname(file)))
 
         # Now create them
         for dir in sorted(need_dir):
-            self.mkpath(dir, mode, verbose=verbose, dry_run=dry_run)
+            self.mkpath(dir, mode, dry_run=dry_run)
diff --git a/Lib/packaging/command/test.py b/Lib/packaging/command/test.py
--- a/Lib/packaging/command/test.py
+++ b/Lib/packaging/command/test.py
@@ -60,8 +60,7 @@
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
-            # Temporary kludge until we remove the verbose arguments and use
-            # logging everywhere
+            # XXX maybe we could pass the verbose argument of pysetup here
             logger = logging.getLogger('packaging')
             verbose = logger.getEffectiveLevel() >= logging.DEBUG
             verbosity = verbose + 1
diff --git a/Lib/packaging/compiler/__init__.py b/Lib/packaging/compiler/__init__.py
--- a/Lib/packaging/compiler/__init__.py
+++ b/Lib/packaging/compiler/__init__.py
@@ -153,8 +153,7 @@
     pretty_printer.print_help("List of available compilers:")
 
 
-def new_compiler(plat=None, compiler=None, verbose=0, dry_run=False,
-                 force=False):
+def new_compiler(plat=None, compiler=None, dry_run=False, force=False):
     """Generate an instance of some CCompiler subclass for the supplied
     platform/compiler combination.  'plat' defaults to 'os.name'
     (eg. 'posix', 'nt'), and 'compiler' defaults to the default compiler
@@ -183,11 +182,7 @@
         cls = resolve_name(cls)
         _COMPILERS[compiler] = cls
 
-
-    # XXX The None is necessary to preserve backwards compatibility
-    # with classes that expect verbose to be the first positional
-    # argument.
-    return cls(None, dry_run, force)
+    return cls(dry_run, force)
 
 
 def gen_preprocess_options(macros, include_dirs):
diff --git a/Lib/packaging/compiler/bcppcompiler.py b/Lib/packaging/compiler/bcppcompiler.py
--- a/Lib/packaging/compiler/bcppcompiler.py
+++ b/Lib/packaging/compiler/bcppcompiler.py
@@ -47,8 +47,8 @@
     exe_extension = '.exe'
 
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(BCPPCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(BCPPCompiler, self).__init__(dry_run, force)
 
         # These executables are assumed to all be in the path.
         # Borland doesn't seem to use any special registry settings to
diff --git a/Lib/packaging/compiler/ccompiler.py b/Lib/packaging/compiler/ccompiler.py
--- a/Lib/packaging/compiler/ccompiler.py
+++ b/Lib/packaging/compiler/ccompiler.py
@@ -79,10 +79,9 @@
                    }
     language_order = ["c++", "objc", "c"]
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
+    def __init__(self, dry_run=False, force=False):
         self.dry_run = dry_run
         self.force = force
-        self.verbose = verbose
 
         # 'output_dir': a common output directory for object, library,
         # shared object, and shared library files
diff --git a/Lib/packaging/compiler/cygwinccompiler.py b/Lib/packaging/compiler/cygwinccompiler.py
--- a/Lib/packaging/compiler/cygwinccompiler.py
+++ b/Lib/packaging/compiler/cygwinccompiler.py
@@ -92,8 +92,8 @@
     shared_lib_format = "%s%s"
     exe_extension = ".exe"
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(CygwinCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(CygwinCCompiler, self).__init__(dry_run, force)
 
         status, details = check_config_h()
         logger.debug("Python's GCC status: %s (details: %s)", status, details)
@@ -233,12 +233,11 @@
         if not debug:
             extra_preargs.append("-s")
 
-        UnixCCompiler.link(self, target_desc, objects, output_filename,
-                           output_dir, libraries, library_dirs,
-                           runtime_library_dirs,
-                           None, # export_symbols, we do this in our def-file
-                           debug, extra_preargs, extra_postargs, build_temp,
-                           target_lang)
+        super(CygwinCCompiler, self).link(
+            target_desc, objects, output_filename, output_dir, libraries,
+            library_dirs, runtime_library_dirs,
+            None, # export_symbols, we do this in our def-file
+            debug, extra_preargs, extra_postargs, build_temp, target_lang)
 
     # -- Miscellaneous methods -----------------------------------------
 
@@ -271,8 +270,8 @@
     name = 'mingw32'
     description = 'MinGW32 compiler'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(Mingw32CCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(Mingw32CCompiler, self).__init__(dry_run, force)
 
         # ld_version >= "2.13" support -shared so use it instead of
         # -mdll -static
diff --git a/Lib/packaging/compiler/msvc9compiler.py b/Lib/packaging/compiler/msvc9compiler.py
--- a/Lib/packaging/compiler/msvc9compiler.py
+++ b/Lib/packaging/compiler/msvc9compiler.py
@@ -309,8 +309,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = VERSION
         self.__root = r"Software\Microsoft\VisualStudio"
         # self.__macros = MACROS
diff --git a/Lib/packaging/compiler/msvccompiler.py b/Lib/packaging/compiler/msvccompiler.py
--- a/Lib/packaging/compiler/msvccompiler.py
+++ b/Lib/packaging/compiler/msvccompiler.py
@@ -236,8 +236,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = get_build_version()
         self.__arch = get_build_architecture()
         if self.__arch == "Intel":
diff --git a/Lib/packaging/metadata.py b/Lib/packaging/metadata.py
--- a/Lib/packaging/metadata.py
+++ b/Lib/packaging/metadata.py
@@ -229,8 +229,10 @@
 
     def __delitem__(self, name):
         field_name = self._convert_name(name)
-        # we let a KeyError propagate
-        del self._fields[field_name]
+        try:
+            del self._fields[field_name]
+        except KeyError:
+            raise KeyError(name)
         self._set_best_version()
 
     def __contains__(self, name):
diff --git a/Lib/packaging/run.py b/Lib/packaging/run.py
--- a/Lib/packaging/run.py
+++ b/Lib/packaging/run.py
@@ -368,7 +368,7 @@
     ('list', 'List installed projects', _list),
     ('graph', 'Display a graph', _graph),
     ('create', 'Create a project', _create),
-    ('generate-setup', 'Generate a backward-comptatible setup.py', _generate),
+    ('generate-setup', 'Generate a backward-compatible setup.py', _generate),
 ]
 
 
diff --git a/Lib/packaging/tests/__init__.py b/Lib/packaging/tests/__init__.py
--- a/Lib/packaging/tests/__init__.py
+++ b/Lib/packaging/tests/__init__.py
@@ -14,16 +14,11 @@
 import os
 import sys
 import unittest
-from io import StringIO
 
-# XXX move helpers to support, add tests for them, remove things that
-# duplicate test.support (or keep them for the backport; needs thinking)
-
-here = os.path.dirname(__file__) or os.curdir
-verbose = 1
 
 def test_suite():
     suite = unittest.TestSuite()
+    here = os.path.dirname(__file__) or os.curdir
     for fn in os.listdir(here):
         if fn.startswith("test") and fn.endswith(".py"):
             modname = "packaging.tests." + fn[:-3]
@@ -31,100 +26,3 @@
             module = sys.modules[modname]
             suite.addTest(module.test_suite())
     return suite
-
-
-class Error(Exception):
-    """Base class for regression test exceptions."""
-
-
-class TestFailed(Error):
-    """Test failed."""
-
-
-class BasicTestRunner:
-    def run(self, test):
-        result = unittest.TestResult()
-        test(result)
-        return result
-
-
-def _run_suite(suite, verbose_=1):
-    """Run tests from a unittest.TestSuite-derived class."""
-    global verbose
-    verbose = verbose_
-    if verbose_:
-        runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
-    else:
-        runner = BasicTestRunner()
-
-    result = runner.run(suite)
-    if not result.wasSuccessful():
-        if len(result.errors) == 1 and not result.failures:
-            err = result.errors[0][1]
-        elif len(result.failures) == 1 and not result.errors:
-            err = result.failures[0][1]
-        else:
-            err = "errors occurred; run in verbose mode for details"
-        raise TestFailed(err)
-
-
-def run_unittest(classes, verbose_=1):
-    """Run tests from unittest.TestCase-derived classes.
-
-    Originally extracted from stdlib test.test_support and modified to
-    support unittest2.
-    """
-    valid_types = (unittest.TestSuite, unittest.TestCase)
-    suite = unittest.TestSuite()
-    for cls in classes:
-        if isinstance(cls, str):
-            if cls in sys.modules:
-                suite.addTest(unittest.findTestCases(sys.modules[cls]))
-            else:
-                raise ValueError("str arguments must be keys in sys.modules")
-        elif isinstance(cls, valid_types):
-            suite.addTest(cls)
-        else:
-            suite.addTest(unittest.makeSuite(cls))
-    _run_suite(suite, verbose_)
-
-
-def reap_children():
-    """Use this function at the end of test_main() whenever sub-processes
-    are started.  This will help ensure that no extra children (zombies)
-    stick around to hog resources and create problems when looking
-    for refleaks.
-
-    Extracted from stdlib test.support.
-    """
-
-    # Reap all our dead child processes so we don't leave zombies around.
-    # These hog resources and might be causing some of the buildbots to die.
-    if hasattr(os, 'waitpid'):
-        any_process = -1
-        while True:
-            try:
-                # This will raise an exception on Windows.  That's ok.
-                pid, status = os.waitpid(any_process, os.WNOHANG)
-                if pid == 0:
-                    break
-            except:
-                break
-
-
-def captured_stdout(func, *args, **kw):
-    orig_stdout = getattr(sys, 'stdout')
-    setattr(sys, 'stdout', StringIO())
-    try:
-        res = func(*args, **kw)
-        sys.stdout.seek(0)
-        return res, sys.stdout.read()
-    finally:
-        setattr(sys, 'stdout', orig_stdout)
-
-
-def unload(name):
-    try:
-        del sys.modules[name]
-    except KeyError:
-        pass
diff --git a/Lib/packaging/tests/pypi_server.py b/Lib/packaging/tests/pypi_server.py
--- a/Lib/packaging/tests/pypi_server.py
+++ b/Lib/packaging/tests/pypi_server.py
@@ -33,7 +33,6 @@
 import queue
 import select
 import threading
-import socketserver
 from functools import wraps
 from http.server import HTTPServer, SimpleHTTPRequestHandler
 from xmlrpc.server import SimpleXMLRPCServer
@@ -270,7 +269,7 @@
 class PyPIXMLRPCServer(SimpleXMLRPCServer):
     def server_bind(self):
         """Override server_bind to store the server name."""
-        socketserver.TCPServer.server_bind(self)
+        super(PyPIXMLRPCServer, self).server_bind()
         host, port = self.socket.getsockname()[:2]
         self.server_port = port
 
@@ -371,12 +370,13 @@
             'requires_python': self.requires_python,
             'classifiers': [],
             'name': self.name,
-            'licence': self.licence,
+            'licence': self.licence,  # XXX licence or license?
             'summary': self.summary,
             'home_page': self.homepage,
             'stable_version': self.stable_version,
-            'provides_dist': self.provides_dist or "%s (%s)" % (self.name,
-                                                              self.version),
+            # FIXME doesn't that reproduce the bug from 6527d3106e9f?
+            'provides_dist': (self.provides_dist or
+                             "%s (%s)" % (self.name, self.version)),
             'requires': self.requires,
             'cheesecake_installability_id': self.cheesecake_installability_id,
         }
diff --git a/Lib/packaging/tests/test_command_bdist.py b/Lib/packaging/tests/test_command_bdist.py
--- a/Lib/packaging/tests/test_command_bdist.py
+++ b/Lib/packaging/tests/test_command_bdist.py
@@ -1,7 +1,8 @@
 """Tests for distutils.command.bdist."""
 import os
+from test.support import captured_stdout
 from packaging.command.bdist import bdist, show_formats
-from packaging.tests import unittest, support, captured_stdout
+from packaging.tests import unittest, support
 
 
 class BuildTestCase(support.TempdirManager,
@@ -42,7 +43,9 @@
                             '%s should take --skip-build from bdist' % name)
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        with captured_stdout() as stdout:
+            show_formats()
+        stdout = stdout.getvalue()
 
         # the output should be a header line + one line per format
         num_formats = len(bdist.format_commands)
diff --git a/Lib/packaging/tests/test_command_install_lib.py b/Lib/packaging/tests/test_command_install_lib.py
--- a/Lib/packaging/tests/test_command_install_lib.py
+++ b/Lib/packaging/tests/test_command_install_lib.py
@@ -44,8 +44,8 @@
         f = os.path.join(project_dir, 'foo.py')
         self.write_file(f, '# python file')
         cmd.byte_compile([f])
-        pyc_file = imp.cache_from_source('foo.py', debug_override=True)
-        pyo_file = imp.cache_from_source('foo.py', debug_override=False)
+        pyc_file = imp.cache_from_source('foo.py', True)
+        pyo_file = imp.cache_from_source('foo.py', False)
         self.assertTrue(os.path.exists(pyc_file))
         self.assertTrue(os.path.exists(pyo_file))
 
diff --git a/Lib/packaging/tests/test_command_sdist.py b/Lib/packaging/tests/test_command_sdist.py
--- a/Lib/packaging/tests/test_command_sdist.py
+++ b/Lib/packaging/tests/test_command_sdist.py
@@ -17,8 +17,8 @@
 from packaging.errors import PackagingOptionError
 from packaging.command.sdist import sdist, show_formats
 
+from test.support import captured_stdout
 from packaging.tests import support, unittest
-from packaging.tests import captured_stdout
 from packaging.tests.support import requires_zlib
 
 
@@ -234,7 +234,9 @@
         self.assertIn("'setup.cfg' file not found", warnings[1])
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        with captured_stdout() as stdout:
+            show_formats()
+        stdout = stdout.getvalue()
 
         # the output should be a header line + one line per format
         num_formats = len(get_archive_formats())
diff --git a/Lib/packaging/tests/test_config.py b/Lib/packaging/tests/test_config.py
--- a/Lib/packaging/tests/test_config.py
+++ b/Lib/packaging/tests/test_config.py
@@ -119,7 +119,7 @@
     /usr/include/blitz
 extra_compile_args = -fPIC -O2
     -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
-    /DGECODE_VERSION='win32' -- sys.platform == 'win32'
+    /DGECODE_VERSION=win32 -- sys.platform == 'win32'
 language = cxx
 
 # corner case: if the parent package of an extension is declared but
diff --git a/Lib/packaging/tests/test_database.py b/Lib/packaging/tests/test_database.py
--- a/Lib/packaging/tests/test_database.py
+++ b/Lib/packaging/tests/test_database.py
@@ -10,6 +10,7 @@
 from packaging.tests.test_util import GlobTestCaseBase
 from packaging.tests.support import requires_zlib
 
+import packaging.database
 from packaging.config import get_resources_dests
 from packaging.errors import PackagingError
 from packaging.metadata import Metadata
@@ -279,6 +280,12 @@
         sys.path.insert(0, self.fake_dists_path)
         self.addCleanup(sys.path.remove, self.fake_dists_path)
 
+    def test_caches(self):
+        # sanity check for internal caches
+        for name in ('_cache_name', '_cache_name_egg',
+                     '_cache_path', '_cache_path_egg'):
+            self.assertEqual(getattr(packaging.database, name), {})
+
     def test_distinfo_dirname(self):
         # Given a name and a version, we expect the distinfo_dirname function
         # to return a standard distribution information directory name.
diff --git a/Lib/packaging/tests/test_dist.py b/Lib/packaging/tests/test_dist.py
--- a/Lib/packaging/tests/test_dist.py
+++ b/Lib/packaging/tests/test_dist.py
@@ -8,7 +8,6 @@
 from packaging.dist import Distribution
 from packaging.command.cmd import Command
 from packaging.errors import PackagingModuleError, PackagingOptionError
-from packaging.tests import captured_stdout
 from packaging.tests import support, unittest
 from packaging.tests.support import create_distribution, use_command
 from test.support import unload
diff --git a/Lib/packaging/tests/test_manifest.py b/Lib/packaging/tests/test_manifest.py
--- a/Lib/packaging/tests/test_manifest.py
+++ b/Lib/packaging/tests/test_manifest.py
@@ -27,14 +27,6 @@
                        support.LoggingCatcher,
                        unittest.TestCase):
 
-    def setUp(self):
-        super(ManifestTestCase, self).setUp()
-        self.cwd = os.getcwd()
-
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(ManifestTestCase, self).tearDown()
-
     def assertNoWarnings(self):
         self.assertEqual(self.get_logs(), [])
 
diff --git a/Lib/packaging/tests/test_uninstall.py b/Lib/packaging/tests/test_uninstall.py
--- a/Lib/packaging/tests/test_uninstall.py
+++ b/Lib/packaging/tests/test_uninstall.py
@@ -31,14 +31,9 @@
     def setUp(self):
         super(UninstallTestCase, self).setUp()
         self.addCleanup(enable_cache)
-        self.root_dir = self.mkdtemp()
-        self.cwd = os.getcwd()
+        self.addCleanup(packaging.util._path_created.clear)
         disable_cache()
 
-    def tearDown(self):
-        packaging.util._path_created.clear()
-        super(UninstallTestCase, self).tearDown()
-
     def get_path(self, dist, name):
         # the dist argument must contain an install_dist command correctly
         # initialized with a prefix option and finalized befored this method
@@ -79,7 +74,7 @@
         dist.parse_config_files()
         dist.finalize_options()
         dist.run_command('install_dist',
-                         {'prefix': ('command line', self.root_dir)})
+                         {'prefix': ('command line', self.mkdtemp())})
 
         site_packages = self.get_path(dist, 'purelib')
         return dist, site_packages
diff --git a/Lib/packaging/tests/test_util.py b/Lib/packaging/tests/test_util.py
--- a/Lib/packaging/tests/test_util.py
+++ b/Lib/packaging/tests/test_util.py
@@ -602,14 +602,6 @@
 
 class GlobTestCase(GlobTestCaseBase):
 
-    def setUp(self):
-        super(GlobTestCase, self).setUp()
-        self.cwd = os.getcwd()
-
-    def tearDown(self):
-        os.chdir(self.cwd)
-        super(GlobTestCase, self).tearDown()
-
     def assertGlobMatch(self, glob, spec):
         tempdir = self.build_files_tree(spec)
         expected = self.clean_tree(spec)
diff --git a/Lib/packaging/util.py b/Lib/packaging/util.py
--- a/Lib/packaging/util.py
+++ b/Lib/packaging/util.py
@@ -259,7 +259,7 @@
             if element]
 
 
-def execute(func, args, msg=None, verbose=0, dry_run=False):
+def execute(func, args, msg=None, dry_run=False):
     """Perform some action that affects the outside world.
 
     Some actions (e.g. writing to the filesystem) are special because
@@ -296,7 +296,7 @@
 
 
 def byte_compile(py_files, optimize=0, force=False, prefix=None,
-                 base_dir=None, verbose=0, dry_run=False, direct=None):
+                 base_dir=None, dry_run=False, direct=None):
     """Byte-compile a collection of Python source files to either .pyc
     or .pyo files in a __pycache__ subdirectory.
 
@@ -305,6 +305,9 @@
       0 - don't optimize (generate .pyc)
       1 - normal optimization (like "python -O")
       2 - extra optimization (like "python -OO")
+    This function is independent from the running Python's -O or -B options;
+    it is fully controlled by the parameters passed in.
+
     If 'force' is true, all files are recompiled regardless of
     timestamps.
 
@@ -325,10 +328,9 @@
     the source for details).  The 'direct' flag is used by the script
     generated in indirect mode; unless you know what you're doing, leave
     it set to None.
+    """
+    # FIXME use compileall + remove direct/indirect shenanigans
 
-    This function is independent from the running Python's -O or -B options;
-    it is fully controlled by the parameters passed in.
-    """
     # First, if the caller didn't force us into direct or indirect mode,
     # figure out which mode we should be in.  We take a conservative
     # approach: choose direct mode *only* if the current interpreter is
@@ -381,15 +383,11 @@
                 script.write("""
 byte_compile(files, optimize=%r, force=%r,
              prefix=%r, base_dir=%r,
-             verbose=%r, dry_run=False,
+             dry_run=False,
              direct=True)
-""" % (optimize, force, prefix, base_dir, verbose))
+""" % (optimize, force, prefix, base_dir))
 
         cmd = [sys.executable, script_name]
-        if optimize == 1:
-            cmd.insert(1, "-O")
-        elif optimize == 2:
-            cmd.insert(1, "-OO")
 
         env = os.environ.copy()
         env['PYTHONPATH'] = os.path.pathsep.join(sys.path)
@@ -415,8 +413,10 @@
             # Terminology from the py_compile module:
             #   cfile - byte-compiled file
             #   dfile - purported source filename (same as 'file' by default)
-            debug_override = not optimize
-            cfile = imp.cache_from_source(file, debug_override)
+            # The second argument to cache_from_source forces the extension to
+            # be .pyc (if true) or .pyo (if false); without it, the extension
+            # would depend on the calling Python's -O option
+            cfile = imp.cache_from_source(file, not optimize)
             dfile = file
 
             if prefix:
@@ -681,7 +681,7 @@
     _cfg_target_split = None
 
 
-def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
+def spawn(cmd, search_path=True, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
     'cmd' is just the argument list for the new process, ie.
@@ -1331,10 +1331,9 @@
 
 # XXX to be replaced by shutil.copytree
 def copy_tree(src, dst, preserve_mode=True, preserve_times=True,
-              preserve_symlinks=False, update=False, verbose=True,
-              dry_run=False):
+              preserve_symlinks=False, update=False, dry_run=False):
     # FIXME use of this function is why we get spurious logging message on
-    # stdout when tests run; kill and replace by shuil!
+    # stdout when tests run; kill and replace by shutil!
     from distutils.file_util import copy_file
 
     if not dry_run and not os.path.isdir(src):
@@ -1351,7 +1350,7 @@
                   "error listing files in '%s': %s" % (src, errstr))
 
     if not dry_run:
-        _mkpath(dst, verbose=verbose)
+        _mkpath(dst)
 
     outputs = []
 
@@ -1361,8 +1360,7 @@
 
         if preserve_symlinks and os.path.islink(src_name):
             link_dest = os.readlink(src_name)
-            if verbose >= 1:
-                logger.info("linking %s -> %s", dst_name, link_dest)
+            logger.info("linking %s -> %s", dst_name, link_dest)
             if not dry_run:
                 os.symlink(link_dest, dst_name)
             outputs.append(dst_name)
@@ -1371,11 +1369,10 @@
             outputs.extend(
                 copy_tree(src_name, dst_name, preserve_mode,
                           preserve_times, preserve_symlinks, update,
-                          verbose=verbose, dry_run=dry_run))
+                          dry_run=dry_run))
         else:
             copy_file(src_name, dst_name, preserve_mode,
-                      preserve_times, update, verbose=verbose,
-                      dry_run=dry_run)
+                      preserve_times, update, dry_run=dry_run)
             outputs.append(dst_name)
 
     return outputs
@@ -1388,7 +1385,7 @@
 # I don't use os.makedirs because a) it's new to Python 1.5.2, and
 # b) it blows up if the directory already exists (I want to silently
 # succeed in that case).
-def _mkpath(name, mode=0o777, verbose=True, dry_run=False):
+def _mkpath(name, mode=0o777, dry_run=False):
     # Detect a common bug -- name is None
     if not isinstance(name, str):
         raise PackagingInternalError(
@@ -1423,9 +1420,7 @@
         if abs_head in _path_created:
             continue
 
-        if verbose >= 1:
-            logger.info("creating %s", head)
-
+        logger.info("creating %s", head)
         if not dry_run:
             try:
                 os.mkdir(head, mode)
@@ -1448,8 +1443,7 @@
 
     Returns (content_type: bytes, body: bytes) ready for http.client.HTTP.
     """
-    # Taken from
-    # http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
+    # Taken from http://code.activestate.com/recipes/146306
 
     if boundary is None:
         boundary = b'--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1066,7 +1066,7 @@
         keys = set(packaging.command._COMMANDS)
         return id_, keys
     def restore_packaging_command__COMMANDS(self, saved):
-        # if command._COMMANDS was bound to another dict obhect, we can't
+        # if command._COMMANDS was bound to another dict object, we can't
         # restore the previous object and contents, because the get_ method
         # above does not return the dict object (to ignore changes in values)
         for key in packaging.command._COMMANDS.keys() - saved[1]:
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -378,6 +378,7 @@
 Rasmus Hahn
 Peter Haight
 V?clav Haisman
+Walker Hale IV
 Bob Halley
 Jesse Hallio
 Jun Hamano
@@ -593,6 +594,7 @@
 Gregor Lingl
 Nick Lockwood
 Stephanie Lockwood
+Hugo Lopes Tavares
 Anne Lord
 Tom Loredo
 Justin Love
@@ -606,6 +608,7 @@
 Jim Lynch
 Mikael Lyngvig
 Martin von L?wis
+Guillermo L?pez-Anglada
 Andrew I MacIntyre
 Tim MacKenzie
 Nick Maclaren
@@ -805,6 +808,7 @@
 Tim Rice
 Jan Pieter Riegel
 Armin Rigo
+Arc Riley
 Nicholas Riley
 Jean-Claude Rimbault
 Vlad Riscutia
@@ -817,6 +821,7 @@
 Mark Roddy
 Kevin Rodgers
 Giampaolo Rodola
+Elson Rodriguez
 Luis Rojas
 Mike Romberg
 Armin Ronacher
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -963,13 +963,9 @@
 - Issue #12049: Add RAND_bytes() and RAND_pseudo_bytes() functions to the ssl
   module.
 
-- Issue #12125: fixed the failures under Solaris due to improper test cleanup.
-
 - Issue #6501: os.device_encoding() returns None on Windows if the application
   has no console.
 
-- Issue #12132: Skip test_build_ext in case the xxmodule is not found.
-
 - Issue #12105: Add O_CLOEXEC to the os module.
 
 - Issue #12079: Decimal('Infinity').fma(Decimal('0'), (3.91224318126786e+19+0j))
@@ -1695,12 +1691,6 @@
 - Issue #12057: Add tests for ISO 2022 codecs (iso2022_jp, iso2022_jp_2,
   iso2022_kr).
 
-- Issue #12180: Fixed a few remaining errors in test_packaging when no
-  threading.
-
-- Issue #12120, #12119: skip a test in packaging and distutils
-  if sys.dont_write_bytecode is set to True.
-
 - Issue #12096: Fix a race condition in test_threading.test_waitfor(). Patch
   written by Charles-Fran?ois Natali.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:14:01 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:14:01 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Branch_merge?=
Message-ID: <E1RQKi1-0007o7-80@dinsdale.python.org>

http://hg.python.org/cpython/rev/a00bb30cf775
changeset:   73571:a00bb30cf775
branch:      3.2
parent:      73552:ddfcb0de564f
parent:      73558:974c5ba31ca6
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 16:12:49 2011 +0100
summary:
  Branch merge

files:
  Doc/whatsnew/3.2.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst
--- a/Doc/whatsnew/3.2.rst
+++ b/Doc/whatsnew/3.2.rst
@@ -234,8 +234,8 @@
 namespace, *concurrent*.  Its first member is a *futures* package which provides
 a uniform high-level interface for managing threads and processes.
 
-The design for :mod:`concurrent.futures` was inspired by
-*java.util.concurrent.package*.  In that model, a running call and its result
+The design for :mod:`concurrent.futures` was inspired by the
+*java.util.concurrent* package.  In that model, a running call and its result
 are represented by a :class:`~concurrent.futures.Future` object that abstracts
 features common to threads, processes, and remote procedure calls.  That object
 supports status checks (running or done), timeouts, cancellations, adding

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 16:14:01 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 15 Nov 2011 16:14:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1RQKi1-0007oE-Vi@dinsdale.python.org>

http://hg.python.org/cpython/rev/c7d1768d4fac
changeset:   73572:c7d1768d4fac
parent:      73570:69fb7933434a
parent:      73571:a00bb30cf775
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 15 16:13:16 2011 +0100
summary:
  Merge 3.2

files:
  Doc/whatsnew/3.2.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst
--- a/Doc/whatsnew/3.2.rst
+++ b/Doc/whatsnew/3.2.rst
@@ -234,8 +234,8 @@
 namespace, *concurrent*.  Its first member is a *futures* package which provides
 a uniform high-level interface for managing threads and processes.
 
-The design for :mod:`concurrent.futures` was inspired by
-*java.util.concurrent.package*.  In that model, a running call and its result
+The design for :mod:`concurrent.futures` was inspired by the
+*java.util.concurrent* package.  In that model, a running call and its result
 are represented by a :class:`~concurrent.futures.Future` object that abstracts
 features common to threads, processes, and remote procedure calls.  That object
 supports status checks (running or done), timeouts, cancellations, adding

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 19:20:01 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 19:20:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313392=3A_Writing_a?=
 =?utf8?q?_pyc_file_should_now_be_atomic_under_Windows_as_well=2E?=
Message-ID: <E1RQNc1-0001cI-SB@dinsdale.python.org>

http://hg.python.org/cpython/rev/b75b41237380
changeset:   73573:b75b41237380
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 19:15:19 2011 +0100
summary:
  Issue #13392: Writing a pyc file should now be atomic under Windows as well.

files:
  Lib/importlib/_bootstrap.py |  35 ++++++++++-------
  Misc/NEWS                   |   2 +
  Python/import.c             |  48 +++++++++++++++++++-----
  3 files changed, 60 insertions(+), 25 deletions(-)


diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py
+++ b/Lib/importlib/_bootstrap.py
@@ -84,24 +84,29 @@
     """Best-effort function to write data to a path atomically.
     Be prepared to handle a FileExistsError if concurrent writing of the
     temporary file is attempted."""
-    if not sys.platform.startswith('win'):
-        # On POSIX-like platforms, renaming is atomic. id() is used to generate
-        # a pseudo-random filename.
-        path_tmp = '{}.{}'.format(path, id(path))
-        fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, 0o666)
+    # Renaming should be atomic on most platforms (including Windows).
+    # Under Windows, the limitation is that we can't rename() to an existing
+    # path, while POSIX will overwrite it. But here we don't really care
+    # if there is a glimpse of time during which the final pyc file doesn't
+    # exist.
+    # id() is used to generate a pseudo-random filename.
+    path_tmp = '{}.{}'.format(path, id(path))
+    fd = _os.open(path_tmp, _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, 0o666)
+    try:
+        with _io.FileIO(fd, 'wb') as file:
+            file.write(data)
         try:
-            with _io.FileIO(fd, 'wb') as file:
-                file.write(data)
             _os.rename(path_tmp, path)
+        except FileExistsError:
+            # Windows (if we had access to MoveFileEx, we could overwrite)
+            _os.unlink(path)
+            _os.rename(path_tmp, path)
+    except OSError:
+        try:
+            _os.unlink(path_tmp)
         except OSError:
-            try:
-                _os.unlink(path_tmp)
-            except OSError:
-                pass
-            raise
-    else:
-        with _io.FileIO(path, 'wb') as file:
-            file.write(data)
+            pass
+        raise
 
 
 def _wrap(new, old):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,8 @@
 Core and Builtins
 -----------------
 
+- Issue #13392: Writing a pyc file should now be atomic under Windows as well.
+
 - Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder
   already accepts them).
 
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1197,6 +1197,8 @@
     time_t mtime = srcstat->st_mtime;
 #ifdef MS_WINDOWS   /* since Windows uses different permissions  */
     mode_t mode = srcstat->st_mode & ~S_IEXEC;
+    PyObject *cpathname_tmp;
+    Py_ssize_t cpathname_len;
 #else
     mode_t dirmode = (srcstat->st_mode |
                       S_IXUSR | S_IXGRP | S_IXOTH |
@@ -1255,18 +1257,29 @@
     }
     Py_DECREF(dirname);
 
+    /* We first write to a tmp file and then take advantage
+       of atomic renaming (which *should* be true even under Windows). */
 #ifdef MS_WINDOWS
-    (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname));
-    fd = _wopen(PyUnicode_AS_UNICODE(cpathname),
-                 O_EXCL | O_CREAT | O_WRONLY | O_TRUNC | O_BINARY,
-                 mode);
+    cpathname_len = PyUnicode_GET_LENGTH(cpathname);
+    cpathname_tmp = PyUnicode_New(cpathname_len + 4,
+                                  PyUnicode_MAX_CHAR_VALUE(cpathname));
+    if (cpathname_tmp == NULL) {
+        PyErr_Clear();
+        return;
+    }
+    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 0, '.');
+    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 1, 't');
+    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 2, 'm');
+    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 3, 'p');
+    (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
+    fd = _wopen(PyUnicode_AS_UNICODE(cpathname_tmp),
+                O_EXCL | O_CREAT | O_WRONLY | O_BINARY,
+                mode);
     if (0 <= fd)
         fp = fdopen(fd, "wb");
     else
         fp = NULL;
 #else
-    /* Under POSIX, we first write to a tmp file and then take advantage
-       of atomic renaming. */
     cpathbytes = PyUnicode_EncodeFSDefault(cpathname);
     if (cpathbytes == NULL) {
         PyErr_Clear();
@@ -1294,7 +1307,9 @@
         if (Py_VerboseFlag)
             PySys_FormatStderr(
                 "# can't create %R\n", cpathname);
-#ifndef MS_WINDOWS
+#ifdef MS_WINDOWS
+        Py_DECREF(cpathname_tmp);
+#else
         Py_DECREF(cpathbytes);
         Py_DECREF(cpathbytes_tmp);
 #endif
@@ -1315,7 +1330,8 @@
         /* Don't keep partial file */
         fclose(fp);
 #ifdef MS_WINDOWS
-        (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname));
+        (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
+        Py_DECREF(cpathname_tmp);
 #else
         (void) unlink(PyBytes_AS_STRING(cpathbytes_tmp));
         Py_DECREF(cpathbytes);
@@ -1324,8 +1340,20 @@
         return;
     }
     fclose(fp);
-    /* Under POSIX, do an atomic rename */
-#ifndef MS_WINDOWS
+    /* Do a (hopefully) atomic rename */
+#ifdef MS_WINDOWS
+    if (!MoveFileExW(PyUnicode_AS_UNICODE(cpathname_tmp),
+                     PyUnicode_AS_UNICODE(cpathname),
+                     MOVEFILE_REPLACE_EXISTING)) {
+        if (Py_VerboseFlag)
+            PySys_FormatStderr("# can't write %R\n", cpathname);
+        /* Don't keep tmp file */
+        (void) DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
+        Py_DECREF(cpathname_tmp);
+        return;
+    }
+    Py_DECREF(cpathname_tmp);
+#else
     if (rename(PyBytes_AS_STRING(cpathbytes_tmp),
                PyBytes_AS_STRING(cpathbytes))) {
         if (Py_VerboseFlag)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 20:47:52 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 20:47:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_regression_under_Window?=
 =?utf8?q?s_following_b75b41237380_=28from_issue_=2313392=29?=
Message-ID: <E1RQOz2-00028s-1e@dinsdale.python.org>

http://hg.python.org/cpython/rev/12940d9f8031
changeset:   73574:12940d9f8031
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 20:40:55 2011 +0100
summary:
  Fix regression under Windows following b75b41237380 (from issue #13392)

files:
  Python/import.c |  5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1267,6 +1267,11 @@
         PyErr_Clear();
         return;
     }
+    if (PyUnicode_CopyCharacters(cpathname_tmp, 0,
+                                 cpathname, 0, cpathname_len) < 0) {
+        PyErr_Clear();
+        return;
+    }
     PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 0, '.');
     PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 1, 't');
     PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 2, 'm');

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 20:54:00 2011
From: python-checkins at python.org (florent.xicluna)
Date: Tue, 15 Nov 2011 20:54:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Closes_=2313297=3A_use_byte?=
 =?utf8?q?s_type_to_send_and_receive_binary_data_through_XMLRPC=2E?=
Message-ID: <E1RQP4y-0002mg-V1@dinsdale.python.org>

http://hg.python.org/cpython/rev/0175883d9513
changeset:   73575:0175883d9513
user:        Florent Xicluna <florent.xicluna at gmail.com>
date:        Tue Nov 15 20:53:25 2011 +0100
summary:
  Closes #13297: use bytes type to send and receive binary data through XMLRPC.

files:
  Doc/library/xmlrpc.client.rst |  53 +++++++++++------
  Lib/test/test_xmlrpc.py       |  69 +++++++++++++++++++----
  Lib/xmlrpc/client.py          |  52 ++++++++++++-----
  Misc/NEWS                     |   2 +
  4 files changed, 130 insertions(+), 46 deletions(-)


diff --git a/Doc/library/xmlrpc.client.rst b/Doc/library/xmlrpc.client.rst
--- a/Doc/library/xmlrpc.client.rst
+++ b/Doc/library/xmlrpc.client.rst
@@ -8,7 +8,7 @@
 
 
 .. XXX Not everything is documented yet.  It might be good to describe
-   Marshaller, Unmarshaller, getparser, dumps, loads, and Transport.
+   Marshaller, Unmarshaller, getparser and Transport.
 
 **Source code:** :source:`Lib/xmlrpc/client.py`
 
@@ -21,7 +21,12 @@
 between conformable Python objects and XML on the wire.
 
 
-.. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, allow_none=False, use_datetime=False)
+.. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, \
+                       allow_none=False, use_datetime=False, \
+                       use_builtin_types=False)
+
+   .. versionchanged:: 3.3
+      The *use_builtin_types* flag was added.
 
    A :class:`ServerProxy` instance is an object that manages communication with a
    remote XML-RPC server.  The required first argument is a URI (Uniform Resource
@@ -34,9 +39,13 @@
    XML; the default behaviour is for ``None`` to raise a :exc:`TypeError`. This is
    a commonly-used extension to the XML-RPC specification, but isn't supported by
    all clients and servers; see http://ontosys.com/xml-rpc/extensions.php for a
-   description.  The *use_datetime* flag can be used to cause date/time values to
-   be presented as :class:`datetime.datetime` objects; this is false by default.
-   :class:`datetime.datetime` objects may be passed to calls.
+   description.  The *use_builtin_types* flag can be used to cause date/time values
+   to be presented as :class:`datetime.datetime` objects and binary data to be
+   presented as :class:`bytes` objects; this flag is false by default.
+   :class:`datetime.datetime` and :class:`bytes` objects may be passed to calls.
+
+   The obsolete *use_datetime* flag is similar to *use_builtin_types* but it
+   applies only to date/time values.
 
    Both the HTTP and HTTPS transports support the URL syntax extension for HTTP
    Basic Authentication: ``http://user:pass at host:port/path``.  The  ``user:pass``
@@ -78,12 +87,12 @@
    |                                 | only their *__dict__* attribute is          |
    |                                 | transmitted.                                |
    +---------------------------------+---------------------------------------------+
-   | :const:`dates`                  | in seconds since the epoch (pass in an      |
-   |                                 | instance of the :class:`DateTime` class) or |
+   | :const:`dates`                  | In seconds since the epoch.  Pass in an     |
+   |                                 | instance of the :class:`DateTime` class or  |
    |                                 | a :class:`datetime.datetime` instance.      |
    +---------------------------------+---------------------------------------------+
-   | :const:`binary data`            | pass in an instance of the :class:`Binary`  |
-   |                                 | wrapper class                               |
+   | :const:`binary data`            | Pass in an instance of the :class:`Binary`  |
+   |                                 | wrapper class or a :class:`bytes` instance. |
    +---------------------------------+---------------------------------------------+
 
    This is the full set of data types supported by XML-RPC.  Method calls may also
@@ -98,8 +107,9 @@
    ensure that the string is free of characters that aren't allowed in XML, such as
    the control characters with ASCII values between 0 and 31 (except, of course,
    tab, newline and carriage return); failing to do this will result in an XML-RPC
-   request that isn't well-formed XML.  If you have to pass arbitrary strings via
-   XML-RPC, use the :class:`Binary` wrapper class described below.
+   request that isn't well-formed XML.  If you have to pass arbitrary bytes
+   via XML-RPC, use the :class:`bytes` class or the class:`Binary` wrapper class
+   described below.
 
    :class:`Server` is retained as an alias for :class:`ServerProxy` for backwards
    compatibility.  New code should use :class:`ServerProxy`.
@@ -249,7 +259,7 @@
 Binary Objects
 --------------
 
-This class may be initialized from string data (which may include NULs). The
+This class may be initialized from bytes data (which may include NULs). The
 primary access to the content of a :class:`Binary` object is provided by an
 attribute:
 
@@ -257,15 +267,15 @@
 .. attribute:: Binary.data
 
    The binary data encapsulated by the :class:`Binary` instance.  The data is
-   provided as an 8-bit string.
+   provided as a :class:`bytes` object.
 
 :class:`Binary` objects have the following methods, supported mainly for
 internal use by the marshalling/unmarshalling code:
 
 
-.. method:: Binary.decode(string)
+.. method:: Binary.decode(bytes)
 
-   Accept a base64 string and decode it as the instance's new data.
+   Accept a base64 :class:`bytes` object and decode it as the instance's new data.
 
 
 .. method:: Binary.encode(out)
@@ -471,14 +481,21 @@
    it via an extension,  provide a true value for *allow_none*.
 
 
-.. function:: loads(data, use_datetime=False)
+.. function:: loads(data, use_datetime=False, use_builtin_types=False)
 
    Convert an XML-RPC request or response into Python objects, a ``(params,
    methodname)``.  *params* is a tuple of argument; *methodname* is a string, or
    ``None`` if no method name is present in the packet. If the XML-RPC packet
    represents a fault condition, this function will raise a :exc:`Fault` exception.
-   The *use_datetime* flag can be used to cause date/time values to be presented as
-   :class:`datetime.datetime` objects; this is false by default.
+   The *use_builtin_types* flag can be used to cause date/time values to be
+   presented as :class:`datetime.datetime` objects and binary data to be
+   presented as :class:`bytes` objects; this flag is false by default.
+
+   The obsolete *use_datetime* flag is similar to *use_builtin_types* but it
+   applies only to date/time values.
+
+   .. versionchanged:: 3.3
+      The *use_builtin_types* flag was added.
 
 
 .. _xmlrpc-client-example:
diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py
--- a/Lib/test/test_xmlrpc.py
+++ b/Lib/test/test_xmlrpc.py
@@ -24,6 +24,8 @@
           'ashortlong': 2,
           'anotherlist': ['.zyx.41'],
           'abase64': xmlrpclib.Binary(b"my dog has fleas"),
+          'b64bytes': b"my dog has fleas",
+          'b64bytearray': bytearray(b"my dog has fleas"),
           'boolean': False,
           'unicode': '\u4000\u6000\u8000',
           'ukey\u4000': 'regular value',
@@ -44,27 +46,54 @@
     def test_dump_bare_datetime(self):
         # This checks that an unwrapped datetime.date object can be handled
         # by the marshalling code.  This can't be done via test_dump_load()
-        # since with use_datetime set to 1 the unmarshaller would create
+        # since with use_builtin_types set to 1 the unmarshaller would create
         # datetime objects for the 'datetime[123]' keys as well
         dt = datetime.datetime(2005, 2, 10, 11, 41, 23)
+        self.assertEqual(dt, xmlrpclib.DateTime('20050210T11:41:23'))
         s = xmlrpclib.dumps((dt,))
-        (newdt,), m = xmlrpclib.loads(s, use_datetime=1)
+
+        result, m = xmlrpclib.loads(s, use_builtin_types=True)
+        (newdt,) = result
         self.assertEqual(newdt, dt)
-        self.assertEqual(m, None)
+        self.assertIs(type(newdt), datetime.datetime)
+        self.assertIsNone(m)
 
-        (newdt,), m = xmlrpclib.loads(s, use_datetime=0)
-        self.assertEqual(newdt, xmlrpclib.DateTime('20050210T11:41:23'))
+        result, m = xmlrpclib.loads(s, use_builtin_types=False)
+        (newdt,) = result
+        self.assertEqual(newdt, dt)
+        self.assertIs(type(newdt), xmlrpclib.DateTime)
+        self.assertIsNone(m)
+
+        result, m = xmlrpclib.loads(s, use_datetime=True)
+        (newdt,) = result
+        self.assertEqual(newdt, dt)
+        self.assertIs(type(newdt), datetime.datetime)
+        self.assertIsNone(m)
+
+        result, m = xmlrpclib.loads(s, use_datetime=False)
+        (newdt,) = result
+        self.assertEqual(newdt, dt)
+        self.assertIs(type(newdt), xmlrpclib.DateTime)
+        self.assertIsNone(m)
+
 
     def test_datetime_before_1900(self):
         # same as before but with a date before 1900
         dt = datetime.datetime(1,  2, 10, 11, 41, 23)
+        self.assertEqual(dt, xmlrpclib.DateTime('00010210T11:41:23'))
         s = xmlrpclib.dumps((dt,))
-        (newdt,), m = xmlrpclib.loads(s, use_datetime=1)
+
+        result, m = xmlrpclib.loads(s, use_builtin_types=True)
+        (newdt,) = result
         self.assertEqual(newdt, dt)
-        self.assertEqual(m, None)
+        self.assertIs(type(newdt), datetime.datetime)
+        self.assertIsNone(m)
 
-        (newdt,), m = xmlrpclib.loads(s, use_datetime=0)
-        self.assertEqual(newdt, xmlrpclib.DateTime('00010210T11:41:23'))
+        result, m = xmlrpclib.loads(s, use_builtin_types=False)
+        (newdt,) = result
+        self.assertEqual(newdt, dt)
+        self.assertIs(type(newdt), xmlrpclib.DateTime)
+        self.assertIsNone(m)
 
     def test_bug_1164912 (self):
         d = xmlrpclib.DateTime()
@@ -133,6 +162,25 @@
                           xmlrpclib.loads(strg)[0][0])
         self.assertRaises(TypeError, xmlrpclib.dumps, (arg1,))
 
+    def test_dump_bytes(self):
+        sample = b"my dog has fleas"
+        self.assertEqual(sample, xmlrpclib.Binary(sample))
+        for type_ in bytes, bytearray, xmlrpclib.Binary:
+            value = type_(sample)
+            s = xmlrpclib.dumps((value,))
+
+            result, m = xmlrpclib.loads(s, use_builtin_types=True)
+            (newvalue,) = result
+            self.assertEqual(newvalue, sample)
+            self.assertIs(type(newvalue), bytes)
+            self.assertIsNone(m)
+
+            result, m = xmlrpclib.loads(s, use_builtin_types=False)
+            (newvalue,) = result
+            self.assertEqual(newvalue, sample)
+            self.assertIs(type(newvalue), xmlrpclib.Binary)
+            self.assertIsNone(m)
+
     def test_get_host_info(self):
         # see bug #3613, this raised a TypeError
         transp = xmlrpc.client.Transport()
@@ -140,9 +188,6 @@
                           ('host.tld',
                            [('Authorization', 'Basic dXNlcg==')], {}))
 
-    def test_dump_bytes(self):
-        self.assertRaises(TypeError, xmlrpclib.dumps, (b"my dog has fleas",))
-
     def test_ssl_presence(self):
         try:
             import ssl
diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py
--- a/Lib/xmlrpc/client.py
+++ b/Lib/xmlrpc/client.py
@@ -386,8 +386,8 @@
         if data is None:
             data = b""
         else:
-            if not isinstance(data, bytes):
-                raise TypeError("expected bytes, not %s" %
+            if not isinstance(data, (bytes, bytearray)):
+                raise TypeError("expected bytes or bytearray, not %s" %
                                 data.__class__.__name__)
             data = bytes(data)  # Make a copy of the bytes!
         self.data = data
@@ -559,6 +559,14 @@
         write("</string></value>\n")
     dispatch[str] = dump_unicode
 
+    def dump_bytes(self, value, write):
+        write("<value><base64>\n")
+        encoded = base64.encodebytes(value)
+        write(encoded.decode('ascii'))
+        write("</base64></value>\n")
+    dispatch[bytes] = dump_bytes
+    dispatch[bytearray] = dump_bytes
+
     def dump_array(self, value, write):
         i = id(value)
         if i in self.memo:
@@ -629,7 +637,7 @@
     # and again, if you don't understand what's going on in here,
     # that's perfectly ok.
 
-    def __init__(self, use_datetime=False):
+    def __init__(self, use_datetime=False, use_builtin_types=False):
         self._type = None
         self._stack = []
         self._marks = []
@@ -637,7 +645,8 @@
         self._methodname = None
         self._encoding = "utf-8"
         self.append = self._stack.append
-        self._use_datetime = use_datetime
+        self._use_datetime = use_builtin_types or use_datetime
+        self._use_bytes = use_builtin_types
 
     def close(self):
         # return response tuple and target method
@@ -749,6 +758,8 @@
     def end_base64(self, data):
         value = Binary()
         value.decode(data.encode("ascii"))
+        if self._use_bytes:
+            value = value.data
         self.append(value)
         self._value = 0
     dispatch["base64"] = end_base64
@@ -860,21 +871,26 @@
 #
 # return A (parser, unmarshaller) tuple.
 
-def getparser(use_datetime=False):
+def getparser(use_datetime=False, use_builtin_types=False):
     """getparser() -> parser, unmarshaller
 
     Create an instance of the fastest available parser, and attach it
     to an unmarshalling object.  Return both objects.
     """
     if FastParser and FastUnmarshaller:
-        if use_datetime:
+        if use_builtin_types:
             mkdatetime = _datetime_type
+            mkbytes = base64.decodebytes
+        elif use_datetime:
+            mkdatetime = _datetime_type
+            mkbytes = _binary
         else:
             mkdatetime = _datetime
-        target = FastUnmarshaller(True, False, _binary, mkdatetime, Fault)
+            mkbytes = _binary
+        target = FastUnmarshaller(True, False, mkbytes, mkdatetime, Fault)
         parser = FastParser(target)
     else:
-        target = Unmarshaller(use_datetime=use_datetime)
+        target = Unmarshaller(use_datetime=use_datetime, use_builtin_types=use_builtin_types)
         if FastParser:
             parser = FastParser(target)
         else:
@@ -912,7 +928,7 @@
 
         encoding: the packet encoding (default is UTF-8)
 
-    All 8-bit strings in the data structure are assumed to use the
+    All byte strings in the data structure are assumed to use the
     packet encoding.  Unicode strings are automatically converted,
     where necessary.
     """
@@ -971,7 +987,7 @@
 #     (None if not present).
 # @see Fault
 
-def loads(data, use_datetime=False):
+def loads(data, use_datetime=False, use_builtin_types=False):
     """data -> unmarshalled data, method name
 
     Convert an XML-RPC packet to unmarshalled data plus a method
@@ -980,7 +996,7 @@
     If the XML-RPC packet represents a fault condition, this function
     raises a Fault exception.
     """
-    p, u = getparser(use_datetime=use_datetime)
+    p, u = getparser(use_datetime=use_datetime, use_builtin_types=use_builtin_types)
     p.feed(data)
     p.close()
     return u.close(), u.getmethodname()
@@ -1092,8 +1108,9 @@
     # that they can decode such a request
     encode_threshold = None #None = don't encode
 
-    def __init__(self, use_datetime=False):
+    def __init__(self, use_datetime=False, use_builtin_types=False):
         self._use_datetime = use_datetime
+        self._use_builtin_types = use_builtin_types
         self._connection = (None, None)
         self._extra_headers = []
 
@@ -1154,7 +1171,8 @@
 
     def getparser(self):
         # get parser and unmarshaller
-        return getparser(use_datetime=self._use_datetime)
+        return getparser(use_datetime=self._use_datetime,
+                         use_builtin_types=self._use_builtin_types)
 
     ##
     # Get authorization info from host parameter
@@ -1361,7 +1379,7 @@
     """
 
     def __init__(self, uri, transport=None, encoding=None, verbose=False,
-                 allow_none=False, use_datetime=False):
+                 allow_none=False, use_datetime=False, use_builtin_types=False):
         # establish a "logical" server connection
 
         # get the url
@@ -1375,9 +1393,11 @@
 
         if transport is None:
             if type == "https":
-                transport = SafeTransport(use_datetime=use_datetime)
+                handler = SafeTransport
             else:
-                transport = Transport(use_datetime=use_datetime)
+                handler = Transport
+            transport = handler(use_datetime=use_datetime,
+                                use_builtin_types=use_builtin_types)
         self.__transport = transport
 
         self.__encoding = encoding or 'utf-8'
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -374,6 +374,8 @@
 Library
 -------
 
+- Issue #13297: Use bytes type to send and receive binary data through XMLRPC.
+
 - Issue #6397: Support "/dev/poll" polling objects in select module,
   under Solaris & derivatives.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 22:25:43 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 15 Nov 2011 22:25:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313374=3A_The_Windo?=
 =?utf8?q?ws_bytes_API_has_been_deprecated_in_the_os_module=2E_Use?=
Message-ID: <E1RQQVj-0003Zy-Cw@dinsdale.python.org>

http://hg.python.org/cpython/rev/d42811b93357
changeset:   73576:d42811b93357
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 15 22:27:41 2011 +0100
summary:
  Issue #13374: The Windows bytes API has been deprecated in the os module. Use
Unicode filenames instead of bytes filenames to not depend on the ANSI code
page anymore and to support any filename.

files:
  Doc/whatsnew/3.3.rst         |    4 +
  Lib/test/test_genericpath.py |   21 +-
  Lib/test/test_ntpath.py      |    7 +-
  Lib/test/test_os.py          |   44 ++-
  Lib/test/test_pep277.py      |    9 +-
  Lib/test/test_posixpath.py   |   14 +-
  Misc/NEWS                    |    4 +
  Modules/posixmodule.c        |  380 ++++++++++++----------
  8 files changed, 289 insertions(+), 194 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -573,6 +573,10 @@
   with sys.platform.startswith('linux'), or directly sys.platform == 'linux' if
   you don't need to support older Python versions.
 
+* Issue #13374: The Windows bytes API has been deprecated in the :mod:`os`
+  module. Use Unicode filenames instead of bytes filenames to not depend on the
+  ANSI code page anymore and to support any filename.
+
 Porting C code
 --------------
 
diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py
--- a/Lib/test/test_genericpath.py
+++ b/Lib/test/test_genericpath.py
@@ -2,11 +2,12 @@
 Tests common to genericpath, macpath, ntpath and posixpath
 """
 
+import genericpath
+import os
+import sys
 import unittest
+import warnings
 from test import support
-import os
-import genericpath
-import sys
 
 
 def safe_rmdir(dirname):
@@ -258,7 +259,9 @@
 
     def test_abspath(self):
         self.assertIn("foo", self.pathmodule.abspath("foo"))
-        self.assertIn(b"foo", self.pathmodule.abspath(b"foo"))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertIn(b"foo", self.pathmodule.abspath(b"foo"))
 
         # Abspath returns bytes when the arg is bytes
         for path in (b'', b'foo', b'f\xf2\xf2', b'/foo', b'C:\\'):
@@ -266,7 +269,9 @@
 
     def test_realpath(self):
         self.assertIn("foo", self.pathmodule.realpath("foo"))
-        self.assertIn(b"foo", self.pathmodule.realpath(b"foo"))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertIn(b"foo", self.pathmodule.realpath(b"foo"))
 
     def test_normpath_issue5827(self):
         # Make sure normpath preserves unicode
@@ -296,8 +301,10 @@
         "Mac OS X denies the creation of a directory with an invalid utf8 name")
     def test_nonascii_abspath(self):
         # Test non-ASCII, non-UTF8 bytes in the path.
-        with support.temp_cwd(b'\xe7w\xf0'):
-            self.test_abspath()
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            with support.temp_cwd(b'\xe7w\xf0'):
+                self.test_abspath()
 
 
 def test_main():
diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py
--- a/Lib/test/test_ntpath.py
+++ b/Lib/test/test_ntpath.py
@@ -1,10 +1,11 @@
 import ntpath
 import os
 import sys
+import unittest
+import warnings
 from test.support import TestFailed
 from test import support, test_genericpath
 from tempfile import TemporaryFile
-import unittest
 
 
 def tester(fn, wantResult):
@@ -21,7 +22,9 @@
     fn = fn.replace('["', '[b"')
     fn = fn.replace(", '", ", b'")
     fn = fn.replace(', "', ', b"')
-    gotResult = eval(fn)
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore", DeprecationWarning)
+        gotResult = eval(fn)
     if isinstance(wantResult, str):
         wantResult = wantResult.encode('ascii')
     elif isinstance(wantResult, tuple):
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -213,7 +213,9 @@
             fname = self.fname.encode(sys.getfilesystemencoding())
         except UnicodeEncodeError:
             self.skipTest("cannot encode %a for the filesystem" % self.fname)
-        self.check_stat_attributes(fname)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.check_stat_attributes(fname)
 
     def test_statvfs_attributes(self):
         if not hasattr(os, "statvfs"):
@@ -838,7 +840,9 @@
         with open(file1, "w") as f1:
             f1.write("test")
 
-        os.link(file1, file2)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            os.link(file1, file2)
         with open(file1, "r") as f1, open(file2, "r") as f2:
             self.assertTrue(os.path.sameopenfile(f1.fileno(), f2.fileno()))
 
@@ -1160,8 +1164,10 @@
         self.assertNotEqual(os.lstat(link), os.stat(link))
 
         bytes_link = os.fsencode(link)
-        self.assertEqual(os.stat(bytes_link), os.stat(target))
-        self.assertNotEqual(os.lstat(bytes_link), os.stat(bytes_link))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertEqual(os.stat(bytes_link), os.stat(target))
+            self.assertNotEqual(os.lstat(bytes_link), os.stat(bytes_link))
 
     def test_12084(self):
         level1 = os.path.abspath(support.TESTFN)
@@ -1619,6 +1625,35 @@
         self._check_xattrs(getxattr, setxattr, removexattr, listxattr)
 
 
+ at unittest.skipUnless(sys.platform == "win32", "Win32 specific tests")
+class Win32DeprecatedBytesAPI(unittest.TestCase):
+    def test_deprecated(self):
+        import nt
+        filename = os.fsencode(support.TESTFN)
+        with warnings.catch_warnings():
+            warnings.simplefilter("error", DeprecationWarning)
+            for func, *args in (
+                (nt._getfullpathname, filename),
+                (nt._isdir, filename),
+                (os.access, filename, os.R_OK),
+                (os.chdir, filename),
+                (os.chmod, filename, 0o777),
+                (os.link, filename, filename),
+                (os.listdir, filename),
+                (os.lstat, filename),
+                (os.mkdir, filename),
+                (os.open, filename, os.O_RDONLY),
+                (os.rename, filename, filename),
+                (os.rmdir, filename),
+                (os.startfile, filename),
+                (os.stat, filename),
+                (os.symlink, filename, filename),
+                (os.unlink, filename),
+                (os.utime, filename),
+            ):
+                self.assertRaises(DeprecationWarning, func, *args)
+
+
 @support.reap_threads
 def test_main():
     support.run_unittest(
@@ -1643,6 +1678,7 @@
         TestSendfile,
         ProgramPriorityTests,
         ExtendedAttributeTests,
+        Win32DeprecatedBytesAPI,
     )
 
 if __name__ == "__main__":
diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -1,6 +1,9 @@
 # Test the Unicode versions of normal file functions
 # open, os.open, os.stat. os.listdir, os.rename, os.remove, os.mkdir, os.chdir, os.rmdir
-import sys, os, unittest
+import os
+import sys
+import unittest
+import warnings
 from unicodedata import normalize
 from test import support
 
@@ -155,7 +158,9 @@
     @unittest.skipIf(sys.platform == 'darwin', 'irrelevant test on Mac OS X')
     def test_listdir(self):
         sf0 = set(self.files)
-        f1 = os.listdir(support.TESTFN.encode(sys.getfilesystemencoding()))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            f1 = os.listdir(support.TESTFN.encode(sys.getfilesystemencoding()))
         f2 = os.listdir(support.TESTFN)
         sf2 = set(os.path.join(support.TESTFN, f) for f in f2)
         self.assertEqual(sf0, sf2, "%a != %a" % (sf0, sf2))
diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py
--- a/Lib/test/test_posixpath.py
+++ b/Lib/test/test_posixpath.py
@@ -1,11 +1,11 @@
+import os
+import posixpath
+import sys
 import unittest
+import warnings
+from posixpath import realpath, abspath, dirname, basename
 from test import support, test_genericpath
 
-import posixpath
-import os
-import sys
-from posixpath import realpath, abspath, dirname, basename
-
 try:
     import posix
 except ImportError:
@@ -231,7 +231,9 @@
 
     def test_ismount(self):
         self.assertIs(posixpath.ismount("/"), True)
-        self.assertIs(posixpath.ismount(b"/"), True)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertIs(posixpath.ismount(b"/"), True)
 
     def test_ismount_non_existent(self):
         # Non-existent mountpoint.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -374,6 +374,10 @@
 Library
 -------
 
+- Issue #13374: The Windows bytes API has been deprecated in the os module. Use
+  Unicode filenames instead of bytes filenames to not depend on the ANSI code
+  page anymore and to support any filename.
+
 - Issue #13297: Use bytes type to send and receive binary data through XMLRPC.
 
 - Issue #6397: Support "/dev/poll" polling objects in select module,
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -546,6 +546,34 @@
 
     return TRUE;
 }
+
+static int
+win32_warn_bytes_api()
+{
+    return PyErr_WarnEx(PyExc_DeprecationWarning,
+        "The Windows bytes API has been deprecated, "
+        "use Unicode filenames instead",
+        1);
+}
+
+static PyObject*
+win32_decode_filename(PyObject *obj)
+{
+    PyObject *unicode;
+    if (PyUnicode_Check(obj)) {
+        if (PyUnicode_READY(obj))
+            return NULL;
+        Py_INCREF(obj);
+        return obj;
+    }
+    if (!PyUnicode_FSDecoder(obj, &unicode))
+        return NULL;
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
+    return unicode;
+}
 #endif /* MS_WINDOWS */
 
 /* Return a dictionary corresponding to the POSIX environment table */
@@ -725,22 +753,6 @@
         return PyErr_SetFromWindowsErr(errno);
 }
 
-static int
-convert_to_unicode(PyObject **param)
-{
-    if (PyUnicode_CheckExact(*param))
-        Py_INCREF(*param);
-    else if (PyUnicode_Check(*param))
-        /* For a Unicode subtype that's not a Unicode object,
-           return a true Unicode object with the same data. */
-        *param = PyUnicode_Copy(*param);
-    else
-        *param = PyUnicode_FromEncodedObject(*param,
-                                             Py_FileSystemDefaultEncoding,
-                                             "strict");
-    return (*param) != NULL;
-}
-
 #endif /* MS_WINDOWS */
 
 #if defined(PYOS_OS2)
@@ -895,7 +907,7 @@
            char* wformat, BOOL (__stdcall *funcW)(LPWSTR))
 {
     PyObject *uni;
-    char *ansi;
+    const char *ansi;
     BOOL result;
 
     if (PyArg_ParseTuple(args, wformat, &uni))
@@ -915,6 +927,8 @@
 
     if (!PyArg_ParseTuple(args, format, &ansi))
         return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     result = funcA(ansi);
     Py_END_ALLOW_THREADS
@@ -1129,14 +1143,15 @@
 
 /* Grab GetFinalPathNameByHandle dynamically from kernel32 */
 static int has_GetFinalPathNameByHandle = 0;
-static DWORD (CALLBACK *Py_GetFinalPathNameByHandleA)(HANDLE, LPSTR, DWORD,
-                                                      DWORD);
 static DWORD (CALLBACK *Py_GetFinalPathNameByHandleW)(HANDLE, LPWSTR, DWORD,
                                                       DWORD);
 static int
 check_GetFinalPathNameByHandle()
 {
     HINSTANCE hKernel32;
+    DWORD (CALLBACK *Py_GetFinalPathNameByHandleA)(HANDLE, LPSTR, DWORD,
+                                                   DWORD);
+
     /* only recheck */
     if (!has_GetFinalPathNameByHandle)
     {
@@ -1853,6 +1868,12 @@
     if (!PyArg_ParseTuple(args, format,
                           PyUnicode_FSConverter, &opath))
         return NULL;
+#ifdef MS_WINDOWS
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(opath);
+        return NULL;
+    }
+#endif
     path = PyBytes_AsString(opath);
     Py_BEGIN_ALLOW_THREADS
     res = (*statfunc)(path, &st);
@@ -1885,8 +1906,7 @@
 static PyObject *
 posix_access(PyObject *self, PyObject *args)
 {
-    PyObject *opath;
-    char *path;
+    const char *path;
     int mode;
 
 #ifdef MS_WINDOWS
@@ -1904,14 +1924,13 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-    if (!PyArg_ParseTuple(args, "O&i:access",
-                          PyUnicode_FSConverter, &opath, &mode))
-        return NULL;
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "yi:access", &path, &mode))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     attr = GetFileAttributesA(path);
     Py_END_ALLOW_THREADS
-    Py_DECREF(opath);
 finish:
     if (attr == 0xFFFFFFFF)
         /* File does not exist, or cannot read attributes */
@@ -1923,6 +1942,7 @@
                            || !(attr & FILE_ATTRIBUTE_READONLY)
                            || (attr & FILE_ATTRIBUTE_DIRECTORY));
 #else
+    PyObject *opath;
     int res;
     if (!PyArg_ParseTuple(args, "O&i:access",
                           PyUnicode_FSConverter, &opath, &mode))
@@ -2042,7 +2062,7 @@
 posix_chmod(PyObject *self, PyObject *args)
 {
     PyObject *opath = NULL;
-    char *path = NULL;
+    const char *path = NULL;
     int i;
     int res;
 #ifdef MS_WINDOWS
@@ -2073,10 +2093,10 @@
        are also valid. */
     PyErr_Clear();
 
-    if (!PyArg_ParseTuple(args, "O&i:chmod", PyUnicode_FSConverter,
-                          &opath, &i))
-        return NULL;
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "yi:chmod", &path, &i))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     attr = GetFileAttributesA(path);
     if (attr != 0xFFFFFFFF) {
@@ -2091,10 +2111,8 @@
     Py_END_ALLOW_THREADS
     if (!res) {
         win32_error("chmod", path);
-        Py_DECREF(opath);
-        return NULL;
-    }
-    Py_DECREF(opath);
+        return NULL;
+    }
     Py_INCREF(Py_None);
     return Py_None;
 #else /* MS_WINDOWS */
@@ -2450,51 +2468,56 @@
 static PyObject *
 win32_link(PyObject *self, PyObject *args)
 {
-    PyObject *osrc, *odst;
-    char *src, *dst;
-    BOOL rslt;
-    PyObject *usrc, *udst;
-
-    if (PyArg_ParseTuple(args, "UU:link", &usrc, &udst))
+    PyObject *src, *dst;
+    BOOL ok;
+
+    if (PyArg_ParseTuple(args, "UU:link", &src, &dst))
     {
         wchar_t *wsrc, *wdst;
-        wsrc = PyUnicode_AsUnicode(usrc);
+
+        wsrc = PyUnicode_AsUnicode(src);
         if (wsrc == NULL)
+            goto error;
+        wdst = PyUnicode_AsUnicode(dst);
+        if (wdst == NULL)
+            goto error;
+
+        Py_BEGIN_ALLOW_THREADS
+        ok = CreateHardLinkW(wdst, wsrc, NULL);
+        Py_END_ALLOW_THREADS
+
+        if (!ok)
+            return win32_error("link", NULL);
+        Py_RETURN_NONE;
+    }
+    else {
+        PyErr_Clear();
+        if (!PyArg_ParseTuple(args, "O&O&:link",
+                              PyUnicode_FSConverter, &src,
+                              PyUnicode_FSConverter, &dst))
             return NULL;
-        wdst = PyUnicode_AsUnicode(udst);
-        if (wdst == NULL)
-            return NULL;
+
+        if (win32_warn_bytes_api())
+            goto error;
 
         Py_BEGIN_ALLOW_THREADS
-        rslt = CreateHardLinkW(wdst, wsrc, NULL);
+        ok = CreateHardLinkA(PyBytes_AS_STRING(dst),
+                             PyBytes_AS_STRING(src),
+                             NULL);
         Py_END_ALLOW_THREADS
 
-        if (rslt == 0)
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+
+        if (!ok)
             return win32_error("link", NULL);
-
         Py_RETURN_NONE;
-    }
-
-    /* Narrow strings also valid. */
-    PyErr_Clear();
-
-    if (!PyArg_ParseTuple(args, "O&O&:link", PyUnicode_FSConverter, &osrc,
-                          PyUnicode_FSConverter, &odst))
-        return NULL;
-
-    src = PyBytes_AsString(osrc);
-    dst = PyBytes_AsString(odst);
-
-    Py_BEGIN_ALLOW_THREADS
-    rslt = CreateHardLinkA(dst, src, NULL);
-    Py_END_ALLOW_THREADS
-
-    Py_DECREF(osrc);
-    Py_DECREF(odst);
-    if (rslt == 0)
-        return win32_error("link", NULL);
-
-    Py_RETURN_NONE;
+
+    error:
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+        return NULL;
+    }
 }
 #endif /* MS_WINDOWS */
 
@@ -2519,7 +2542,8 @@
     HANDLE hFindFile;
     BOOL result;
     WIN32_FIND_DATA FileData;
-    PyObject *opath;
+    const char *path;
+    Py_ssize_t pathlen;
     char namebuf[MAX_PATH+5]; /* Overallocate for \\*.*\0 */
     char *bufptr = namebuf;
     Py_ssize_t len = sizeof(namebuf)-5; /* only claim to have space for MAX_PATH */
@@ -2613,17 +2637,16 @@
        are also valid. */
     PyErr_Clear();
 
-    if (!PyArg_ParseTuple(args, "O&:listdir",
-                          PyUnicode_FSConverter, &opath))
-        return NULL;
-    if (PyBytes_GET_SIZE(opath)+1 > MAX_PATH) {
+    if (!PyArg_ParseTuple(args, "y#:listdir", &path, &pathlen))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
+    if (pathlen+1 > MAX_PATH) {
         PyErr_SetString(PyExc_ValueError, "path too long");
-        Py_DECREF(opath);
-        return NULL;
-    }
-    strcpy(namebuf, PyBytes_AsString(opath));
-    len = PyObject_Size(opath);
-    Py_DECREF(opath);
+        return NULL;
+    }
+    strcpy(namebuf, path);
+    len = pathlen;
     if (len > 0) {
         char ch = namebuf[len-1];
         if (ch != SEP && ch != ALTSEP && ch != ':')
@@ -2915,11 +2938,9 @@
 static PyObject *
 posix__getfullpathname(PyObject *self, PyObject *args)
 {
-    PyObject *opath;
-    char *path;
+    const char *path;
     char outbuf[MAX_PATH*2];
     char *temp;
-#ifdef MS_WINDOWS
     PyObject *po;
 
     if (PyArg_ParseTuple(args, "U|:_getfullpathname", &po))
@@ -2953,19 +2974,17 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-#endif
-
-    if (!PyArg_ParseTuple (args, "O&:_getfullpathname",
-                           PyUnicode_FSConverter, &opath))
-        return NULL;
-    path = PyBytes_AsString(opath);
+
+    if (!PyArg_ParseTuple (args, "y:_getfullpathname",
+                           &path))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     if (!GetFullPathName(path, Py_ARRAY_LENGTH(outbuf),
                          outbuf, &temp)) {
         win32_error("GetFullPathName", path);
-        Py_DECREF(opath);
-        return NULL;
-    }
-    Py_DECREF(opath);
+        return NULL;
+    }
     if (PyUnicode_Check(PyTuple_GetItem(args, 0))) {
         return PyUnicode_Decode(outbuf, strlen(outbuf),
                                 Py_FileSystemDefaultEncoding, NULL);
@@ -3069,8 +3088,7 @@
 static PyObject *
 posix__isdir(PyObject *self, PyObject *args)
 {
-    PyObject *opath;
-    char *path;
+    const char *path;
     PyObject *po;
     DWORD attributes;
 
@@ -3088,11 +3106,10 @@
        are also valid. */
     PyErr_Clear();
 
-    if (!PyArg_ParseTuple(args, "O&:_isdir",
-                          PyUnicode_FSConverter, &opath))
-        return NULL;
-
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "y:_isdir", &path))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     attributes = GetFileAttributesA(path);
     if (attributes == INVALID_FILE_ATTRIBUTES)
         Py_RETURN_FALSE;
@@ -3113,8 +3130,7 @@
 posix_mkdir(PyObject *self, PyObject *args)
 {
     int res;
-    PyObject *opath;
-    char *path;
+    const char *path;
     int mode = 0777;
 
 #ifdef MS_WINDOWS
@@ -3136,22 +3152,21 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-    if (!PyArg_ParseTuple(args, "O&|i:mkdir",
-                          PyUnicode_FSConverter, &opath, &mode))
-        return NULL;
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "y|i:mkdir", &path, &mode))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     res = CreateDirectoryA(path, NULL);
     Py_END_ALLOW_THREADS
     if (!res) {
         win32_error("mkdir", path);
-        Py_DECREF(opath);
-        return NULL;
-    }
-    Py_DECREF(opath);
+        return NULL;
+    }
     Py_INCREF(Py_None);
     return Py_None;
 #else
+    PyObject *opath;
 
     if (!PyArg_ParseTuple(args, "O&|i:mkdir",
                           PyUnicode_FSConverter, &opath, &mode))
@@ -3265,44 +3280,54 @@
 posix_rename(PyObject *self, PyObject *args)
 {
 #ifdef MS_WINDOWS
-    PyObject *o1, *o2;
-    wchar_t *w1, *w2;
-    char *p1, *p2;
+    PyObject *src, *dst;
     BOOL result;
-    if (!PyArg_ParseTuple(args, "OO:rename", &o1, &o2))
-        goto error;
-    if (!convert_to_unicode(&o1))
-        goto error;
-    if (!convert_to_unicode(&o2)) {
-        Py_DECREF(o1);
-        goto error;
-    }
-    w1 = PyUnicode_AsUnicode(o1);
-    if (w1 == NULL)
-        goto error;
-    w2 = PyUnicode_AsUnicode(o2);
-    if (w2 == NULL)
-        goto error;
-    Py_BEGIN_ALLOW_THREADS
-    result = MoveFileW(w1, w2);
-    Py_END_ALLOW_THREADS
-    Py_DECREF(o1);
-    Py_DECREF(o2);
-    if (!result)
-        return win32_error("rename", NULL);
-    Py_INCREF(Py_None);
-    return Py_None;
+    if (PyArg_ParseTuple(args, "UU:rename", &src, &dst))
+    {
+        wchar_t *wsrc, *wdst;
+
+        wsrc = PyUnicode_AsUnicode(src);
+        if (wsrc == NULL)
+            return NULL;
+        wdst = PyUnicode_AsUnicode(dst);
+        if (wdst == NULL)
+            return NULL;
+        Py_BEGIN_ALLOW_THREADS
+        result = MoveFileW(wsrc, wdst);
+        Py_END_ALLOW_THREADS
+        if (!result)
+            return win32_error("rename", NULL);
+        Py_INCREF(Py_None);
+        return Py_None;
+    }
+    else {
+        PyErr_Clear();
+        if (!PyArg_ParseTuple(args, "O&O&:rename",
+                              PyUnicode_FSConverter, &src,
+                              PyUnicode_FSConverter, &dst))
+            return NULL;
+
+        if (win32_warn_bytes_api())
+            goto error;
+
+        Py_BEGIN_ALLOW_THREADS
+        result = MoveFileA(PyBytes_AS_STRING(src),
+                           PyBytes_AS_STRING(dst));
+        Py_END_ALLOW_THREADS
+
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+
+        if (!result)
+            return win32_error("rename", NULL);
+        Py_INCREF(Py_None);
+        return Py_None;
+
 error:
-    PyErr_Clear();
-    if (!PyArg_ParseTuple(args, "ss:rename", &p1, &p2))
-        return NULL;
-    Py_BEGIN_ALLOW_THREADS
-    result = MoveFileA(p1, p2);
-    Py_END_ALLOW_THREADS
-    if (!result)
-        return win32_error("rename", NULL);
-    Py_INCREF(Py_None);
-    return Py_None;
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+        return NULL;
+    }
 #else
     return posix_2str(args, "O&O&:rename", rename);
 #endif
@@ -3546,8 +3571,7 @@
     PyObject *arg = Py_None;
     PyObject *obwpath;
     wchar_t *wpath = NULL;
-    PyObject *oapath;
-    char *apath;
+    const char *apath;
     HANDLE hFile;
     time_t atimesec, mtimesec;
     long ausec, musec;
@@ -3571,11 +3595,11 @@
            are also valid. */
         PyErr_Clear();
 
-        if (!PyArg_ParseTuple(args, "O&|O:utime",
-                        PyUnicode_FSConverter, &oapath, &arg))
+        if (!PyArg_ParseTuple(args, "y|O:utime", &apath, &arg))
             return NULL;
-
-        apath = PyBytes_AsString(oapath);
+        if (win32_warn_bytes_api())
+            return NULL;
+
         Py_BEGIN_ALLOW_THREADS
         hFile = CreateFileA(apath, FILE_WRITE_ATTRIBUTES, 0,
                             NULL, OPEN_EXISTING,
@@ -3583,10 +3607,8 @@
         Py_END_ALLOW_THREADS
         if (hFile == INVALID_HANDLE_VALUE) {
             win32_error("utime", apath);
-            Py_DECREF(oapath);
             return NULL;
         }
-        Py_DECREF(oapath);
     }
 
     if (arg == Py_None) {
@@ -6516,7 +6538,8 @@
 win_symlink(PyObject *self, PyObject *args, PyObject *kwargs)
 {
     static char *kwlist[] = {"src", "dest", "target_is_directory", NULL};
-    PyObject *src, *dest;
+    PyObject *osrc, *odest;
+    PyObject *usrc = NULL, *udest = NULL;
     wchar_t *wsrc, *wdest;
     int target_is_directory = 0;
     DWORD res;
@@ -6528,24 +6551,25 @@
         return PyErr_Format(PyExc_NotImplementedError,
             "CreateSymbolicLinkW not found");
     }
-    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "OO|i:symlink",
-        kwlist, &src, &dest, &target_is_directory))
-        return NULL;
+    if (!PyArg_ParseTupleAndKeywords(
+            args, kwargs, "OO|i:symlink", kwlist,
+            &osrc, &odest, &target_is_directory))
+        return NULL;
+
+    usrc = win32_decode_filename(osrc);
+    if (!usrc)
+        return NULL;
+    udest = win32_decode_filename(odest);
+    if (!udest)
+        goto error;
 
     if (win32_can_symlink == 0)
         return PyErr_Format(PyExc_OSError, "symbolic link privilege not held");
 
-    if (!convert_to_unicode(&src))
-        return NULL;
-    if (!convert_to_unicode(&dest)) {
-        Py_DECREF(src);
-        return NULL;
-    }
-
-    wsrc = PyUnicode_AsUnicode(src);
+    wsrc = PyUnicode_AsUnicode(usrc);
     if (wsrc == NULL)
         goto error;
-    wdest = PyUnicode_AsUnicode(dest);
+    wdest = PyUnicode_AsUnicode(udest);
     if (wsrc == NULL)
         goto error;
 
@@ -6563,17 +6587,17 @@
     res = Py_CreateSymbolicLinkW(wdest, wsrc, target_is_directory);
     Py_END_ALLOW_THREADS
 
-    Py_DECREF(src);
-    Py_DECREF(dest);
+    Py_DECREF(usrc);
+    Py_DECREF(udest);
     if (!res)
-        return win32_error_object("symlink", src);
+        return win32_error_object("symlink", osrc);
 
     Py_INCREF(Py_None);
     return Py_None;
 
 error:
-    Py_DECREF(src);
-    Py_DECREF(dest);
+    Py_XDECREF(usrc);
+    Py_XDECREF(udest);
     return NULL;
 }
 #endif /* defined(HAVE_SYMLINK) && defined(MS_WINDOWS) */
@@ -6791,6 +6815,12 @@
                           PyUnicode_FSConverter, &ofile,
                           &flag, &mode))
         return NULL;
+#ifdef MS_WINDOWS
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(ofile);
+        return NULL;
+    }
+#endif
     file = PyBytes_AsString(ofile);
     Py_BEGIN_ALLOW_THREADS
     fd = open(file, flag, mode);
@@ -9290,6 +9320,10 @@
                           PyUnicode_FSConverter, &ofilepath,
                           &operation))
         return NULL;
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(ofilepath);
+        return NULL;
+    }
     filepath = PyBytes_AsString(ofilepath);
     Py_BEGIN_ALLOW_THREADS
     rc = ShellExecute((HWND)0, operation, filepath,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 22:32:40 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 22:32:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_PyUnicode=5FFromFormat?=
 =?utf8?q?=28=29_to_create_the_temporary_file_name=2E?=
Message-ID: <E1RQQcS-0004FF-Ov@dinsdale.python.org>

http://hg.python.org/cpython/rev/cea2d28f2855
changeset:   73577:cea2d28f2855
parent:      73575:0175883d9513
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 22:27:32 2011 +0100
summary:
  Use PyUnicode_FromFormat() to create the temporary file name.
Also, as in importlib, append the id of an object to make the file name
pseudo-random.

files:
  Python/import.c |  43 ++++++++++++------------------------
  1 files changed, 15 insertions(+), 28 deletions(-)


diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1195,17 +1195,15 @@
     Py_UCS4 *cpathname_ucs4;
     FILE *fp;
     time_t mtime = srcstat->st_mtime;
+    PyObject *cpathname_tmp;
 #ifdef MS_WINDOWS   /* since Windows uses different permissions  */
     mode_t mode = srcstat->st_mode & ~S_IEXEC;
-    PyObject *cpathname_tmp;
-    Py_ssize_t cpathname_len;
 #else
     mode_t dirmode = (srcstat->st_mode |
                       S_IXUSR | S_IXGRP | S_IXOTH |
                       S_IWUSR | S_IWGRP | S_IWOTH);
     PyObject *dirbytes;
     PyObject *cpathbytes, *cpathbytes_tmp;
-    Py_ssize_t cpathbytes_len;
 #endif
     int fd;
     PyObject *dirname;
@@ -1258,24 +1256,18 @@
     Py_DECREF(dirname);
 
     /* We first write to a tmp file and then take advantage
-       of atomic renaming (which *should* be true even under Windows). */
-#ifdef MS_WINDOWS
-    cpathname_len = PyUnicode_GET_LENGTH(cpathname);
-    cpathname_tmp = PyUnicode_New(cpathname_len + 4,
-                                  PyUnicode_MAX_CHAR_VALUE(cpathname));
+       of atomic renaming (which *should* be true even under Windows).
+       As in importlib, we use id(something) to generate a pseudo-random
+       filename.  mkstemp() can't be used since it doesn't allow specifying
+       the file access permissions.
+    */
+    cpathname_tmp = PyUnicode_FromFormat("%U.%zd",
+                                         cpathname, (Py_ssize_t) co);
     if (cpathname_tmp == NULL) {
         PyErr_Clear();
         return;
     }
-    if (PyUnicode_CopyCharacters(cpathname_tmp, 0,
-                                 cpathname, 0, cpathname_len) < 0) {
-        PyErr_Clear();
-        return;
-    }
-    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 0, '.');
-    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 1, 't');
-    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 2, 'm');
-    PyUnicode_WriteChar(cpathname_tmp, cpathname_len + 3, 'p');
+#ifdef MS_WINDOWS
     (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
     fd = _wopen(PyUnicode_AS_UNICODE(cpathname_tmp),
                 O_EXCL | O_CREAT | O_WRONLY | O_BINARY,
@@ -1285,22 +1277,17 @@
     else
         fp = NULL;
 #else
+    cpathbytes_tmp = PyUnicode_EncodeFSDefault(cpathname_tmp);
+    Py_DECREF(cpathname_tmp);
+    if (cpathbytes_tmp == NULL) {
+        PyErr_Clear();
+        return;
+    }
     cpathbytes = PyUnicode_EncodeFSDefault(cpathname);
     if (cpathbytes == NULL) {
         PyErr_Clear();
         return;
     }
-    cpathbytes_len = PyBytes_GET_SIZE(cpathbytes);
-    cpathbytes_tmp = PyBytes_FromStringAndSize(NULL, cpathbytes_len + 4);
-    if (cpathbytes_tmp == NULL) {
-        Py_DECREF(cpathbytes);
-        PyErr_Clear();
-        return;
-    }
-    memcpy(PyBytes_AS_STRING(cpathbytes_tmp), PyBytes_AS_STRING(cpathbytes),
-           cpathbytes_len);
-    memcpy(PyBytes_AS_STRING(cpathbytes_tmp) + cpathbytes_len, ".tmp", 4);
-
     fd = open(PyBytes_AS_STRING(cpathbytes_tmp),
               O_CREAT | O_EXCL | O_WRONLY, 0666);
     if (0 <= fd)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 22:32:41 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 15 Nov 2011 22:32:41 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge?=
Message-ID: <E1RQQcT-0004FP-Ja@dinsdale.python.org>

http://hg.python.org/cpython/rev/b24bcb1b1314
changeset:   73578:b24bcb1b1314
parent:      73577:cea2d28f2855
parent:      73576:d42811b93357
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 15 22:27:43 2011 +0100
summary:
  Merge

files:
  Doc/whatsnew/3.3.rst         |    4 +
  Lib/test/test_genericpath.py |   21 +-
  Lib/test/test_ntpath.py      |    7 +-
  Lib/test/test_os.py          |   44 ++-
  Lib/test/test_pep277.py      |    9 +-
  Lib/test/test_posixpath.py   |   14 +-
  Misc/NEWS                    |    4 +
  Modules/posixmodule.c        |  380 ++++++++++++----------
  8 files changed, 289 insertions(+), 194 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -573,6 +573,10 @@
   with sys.platform.startswith('linux'), or directly sys.platform == 'linux' if
   you don't need to support older Python versions.
 
+* Issue #13374: The Windows bytes API has been deprecated in the :mod:`os`
+  module. Use Unicode filenames instead of bytes filenames to not depend on the
+  ANSI code page anymore and to support any filename.
+
 Porting C code
 --------------
 
diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py
--- a/Lib/test/test_genericpath.py
+++ b/Lib/test/test_genericpath.py
@@ -2,11 +2,12 @@
 Tests common to genericpath, macpath, ntpath and posixpath
 """
 
+import genericpath
+import os
+import sys
 import unittest
+import warnings
 from test import support
-import os
-import genericpath
-import sys
 
 
 def safe_rmdir(dirname):
@@ -258,7 +259,9 @@
 
     def test_abspath(self):
         self.assertIn("foo", self.pathmodule.abspath("foo"))
-        self.assertIn(b"foo", self.pathmodule.abspath(b"foo"))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertIn(b"foo", self.pathmodule.abspath(b"foo"))
 
         # Abspath returns bytes when the arg is bytes
         for path in (b'', b'foo', b'f\xf2\xf2', b'/foo', b'C:\\'):
@@ -266,7 +269,9 @@
 
     def test_realpath(self):
         self.assertIn("foo", self.pathmodule.realpath("foo"))
-        self.assertIn(b"foo", self.pathmodule.realpath(b"foo"))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertIn(b"foo", self.pathmodule.realpath(b"foo"))
 
     def test_normpath_issue5827(self):
         # Make sure normpath preserves unicode
@@ -296,8 +301,10 @@
         "Mac OS X denies the creation of a directory with an invalid utf8 name")
     def test_nonascii_abspath(self):
         # Test non-ASCII, non-UTF8 bytes in the path.
-        with support.temp_cwd(b'\xe7w\xf0'):
-            self.test_abspath()
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            with support.temp_cwd(b'\xe7w\xf0'):
+                self.test_abspath()
 
 
 def test_main():
diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py
--- a/Lib/test/test_ntpath.py
+++ b/Lib/test/test_ntpath.py
@@ -1,10 +1,11 @@
 import ntpath
 import os
 import sys
+import unittest
+import warnings
 from test.support import TestFailed
 from test import support, test_genericpath
 from tempfile import TemporaryFile
-import unittest
 
 
 def tester(fn, wantResult):
@@ -21,7 +22,9 @@
     fn = fn.replace('["', '[b"')
     fn = fn.replace(", '", ", b'")
     fn = fn.replace(', "', ', b"')
-    gotResult = eval(fn)
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore", DeprecationWarning)
+        gotResult = eval(fn)
     if isinstance(wantResult, str):
         wantResult = wantResult.encode('ascii')
     elif isinstance(wantResult, tuple):
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -213,7 +213,9 @@
             fname = self.fname.encode(sys.getfilesystemencoding())
         except UnicodeEncodeError:
             self.skipTest("cannot encode %a for the filesystem" % self.fname)
-        self.check_stat_attributes(fname)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.check_stat_attributes(fname)
 
     def test_statvfs_attributes(self):
         if not hasattr(os, "statvfs"):
@@ -838,7 +840,9 @@
         with open(file1, "w") as f1:
             f1.write("test")
 
-        os.link(file1, file2)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            os.link(file1, file2)
         with open(file1, "r") as f1, open(file2, "r") as f2:
             self.assertTrue(os.path.sameopenfile(f1.fileno(), f2.fileno()))
 
@@ -1160,8 +1164,10 @@
         self.assertNotEqual(os.lstat(link), os.stat(link))
 
         bytes_link = os.fsencode(link)
-        self.assertEqual(os.stat(bytes_link), os.stat(target))
-        self.assertNotEqual(os.lstat(bytes_link), os.stat(bytes_link))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertEqual(os.stat(bytes_link), os.stat(target))
+            self.assertNotEqual(os.lstat(bytes_link), os.stat(bytes_link))
 
     def test_12084(self):
         level1 = os.path.abspath(support.TESTFN)
@@ -1619,6 +1625,35 @@
         self._check_xattrs(getxattr, setxattr, removexattr, listxattr)
 
 
+ at unittest.skipUnless(sys.platform == "win32", "Win32 specific tests")
+class Win32DeprecatedBytesAPI(unittest.TestCase):
+    def test_deprecated(self):
+        import nt
+        filename = os.fsencode(support.TESTFN)
+        with warnings.catch_warnings():
+            warnings.simplefilter("error", DeprecationWarning)
+            for func, *args in (
+                (nt._getfullpathname, filename),
+                (nt._isdir, filename),
+                (os.access, filename, os.R_OK),
+                (os.chdir, filename),
+                (os.chmod, filename, 0o777),
+                (os.link, filename, filename),
+                (os.listdir, filename),
+                (os.lstat, filename),
+                (os.mkdir, filename),
+                (os.open, filename, os.O_RDONLY),
+                (os.rename, filename, filename),
+                (os.rmdir, filename),
+                (os.startfile, filename),
+                (os.stat, filename),
+                (os.symlink, filename, filename),
+                (os.unlink, filename),
+                (os.utime, filename),
+            ):
+                self.assertRaises(DeprecationWarning, func, *args)
+
+
 @support.reap_threads
 def test_main():
     support.run_unittest(
@@ -1643,6 +1678,7 @@
         TestSendfile,
         ProgramPriorityTests,
         ExtendedAttributeTests,
+        Win32DeprecatedBytesAPI,
     )
 
 if __name__ == "__main__":
diff --git a/Lib/test/test_pep277.py b/Lib/test/test_pep277.py
--- a/Lib/test/test_pep277.py
+++ b/Lib/test/test_pep277.py
@@ -1,6 +1,9 @@
 # Test the Unicode versions of normal file functions
 # open, os.open, os.stat. os.listdir, os.rename, os.remove, os.mkdir, os.chdir, os.rmdir
-import sys, os, unittest
+import os
+import sys
+import unittest
+import warnings
 from unicodedata import normalize
 from test import support
 
@@ -155,7 +158,9 @@
     @unittest.skipIf(sys.platform == 'darwin', 'irrelevant test on Mac OS X')
     def test_listdir(self):
         sf0 = set(self.files)
-        f1 = os.listdir(support.TESTFN.encode(sys.getfilesystemencoding()))
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            f1 = os.listdir(support.TESTFN.encode(sys.getfilesystemencoding()))
         f2 = os.listdir(support.TESTFN)
         sf2 = set(os.path.join(support.TESTFN, f) for f in f2)
         self.assertEqual(sf0, sf2, "%a != %a" % (sf0, sf2))
diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py
--- a/Lib/test/test_posixpath.py
+++ b/Lib/test/test_posixpath.py
@@ -1,11 +1,11 @@
+import os
+import posixpath
+import sys
 import unittest
+import warnings
+from posixpath import realpath, abspath, dirname, basename
 from test import support, test_genericpath
 
-import posixpath
-import os
-import sys
-from posixpath import realpath, abspath, dirname, basename
-
 try:
     import posix
 except ImportError:
@@ -231,7 +231,9 @@
 
     def test_ismount(self):
         self.assertIs(posixpath.ismount("/"), True)
-        self.assertIs(posixpath.ismount(b"/"), True)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            self.assertIs(posixpath.ismount(b"/"), True)
 
     def test_ismount_non_existent(self):
         # Non-existent mountpoint.
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -374,6 +374,10 @@
 Library
 -------
 
+- Issue #13374: The Windows bytes API has been deprecated in the os module. Use
+  Unicode filenames instead of bytes filenames to not depend on the ANSI code
+  page anymore and to support any filename.
+
 - Issue #13297: Use bytes type to send and receive binary data through XMLRPC.
 
 - Issue #6397: Support "/dev/poll" polling objects in select module,
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -546,6 +546,34 @@
 
     return TRUE;
 }
+
+static int
+win32_warn_bytes_api()
+{
+    return PyErr_WarnEx(PyExc_DeprecationWarning,
+        "The Windows bytes API has been deprecated, "
+        "use Unicode filenames instead",
+        1);
+}
+
+static PyObject*
+win32_decode_filename(PyObject *obj)
+{
+    PyObject *unicode;
+    if (PyUnicode_Check(obj)) {
+        if (PyUnicode_READY(obj))
+            return NULL;
+        Py_INCREF(obj);
+        return obj;
+    }
+    if (!PyUnicode_FSDecoder(obj, &unicode))
+        return NULL;
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(unicode);
+        return NULL;
+    }
+    return unicode;
+}
 #endif /* MS_WINDOWS */
 
 /* Return a dictionary corresponding to the POSIX environment table */
@@ -725,22 +753,6 @@
         return PyErr_SetFromWindowsErr(errno);
 }
 
-static int
-convert_to_unicode(PyObject **param)
-{
-    if (PyUnicode_CheckExact(*param))
-        Py_INCREF(*param);
-    else if (PyUnicode_Check(*param))
-        /* For a Unicode subtype that's not a Unicode object,
-           return a true Unicode object with the same data. */
-        *param = PyUnicode_Copy(*param);
-    else
-        *param = PyUnicode_FromEncodedObject(*param,
-                                             Py_FileSystemDefaultEncoding,
-                                             "strict");
-    return (*param) != NULL;
-}
-
 #endif /* MS_WINDOWS */
 
 #if defined(PYOS_OS2)
@@ -895,7 +907,7 @@
            char* wformat, BOOL (__stdcall *funcW)(LPWSTR))
 {
     PyObject *uni;
-    char *ansi;
+    const char *ansi;
     BOOL result;
 
     if (PyArg_ParseTuple(args, wformat, &uni))
@@ -915,6 +927,8 @@
 
     if (!PyArg_ParseTuple(args, format, &ansi))
         return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     result = funcA(ansi);
     Py_END_ALLOW_THREADS
@@ -1129,14 +1143,15 @@
 
 /* Grab GetFinalPathNameByHandle dynamically from kernel32 */
 static int has_GetFinalPathNameByHandle = 0;
-static DWORD (CALLBACK *Py_GetFinalPathNameByHandleA)(HANDLE, LPSTR, DWORD,
-                                                      DWORD);
 static DWORD (CALLBACK *Py_GetFinalPathNameByHandleW)(HANDLE, LPWSTR, DWORD,
                                                       DWORD);
 static int
 check_GetFinalPathNameByHandle()
 {
     HINSTANCE hKernel32;
+    DWORD (CALLBACK *Py_GetFinalPathNameByHandleA)(HANDLE, LPSTR, DWORD,
+                                                   DWORD);
+
     /* only recheck */
     if (!has_GetFinalPathNameByHandle)
     {
@@ -1853,6 +1868,12 @@
     if (!PyArg_ParseTuple(args, format,
                           PyUnicode_FSConverter, &opath))
         return NULL;
+#ifdef MS_WINDOWS
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(opath);
+        return NULL;
+    }
+#endif
     path = PyBytes_AsString(opath);
     Py_BEGIN_ALLOW_THREADS
     res = (*statfunc)(path, &st);
@@ -1885,8 +1906,7 @@
 static PyObject *
 posix_access(PyObject *self, PyObject *args)
 {
-    PyObject *opath;
-    char *path;
+    const char *path;
     int mode;
 
 #ifdef MS_WINDOWS
@@ -1904,14 +1924,13 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-    if (!PyArg_ParseTuple(args, "O&i:access",
-                          PyUnicode_FSConverter, &opath, &mode))
-        return NULL;
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "yi:access", &path, &mode))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     attr = GetFileAttributesA(path);
     Py_END_ALLOW_THREADS
-    Py_DECREF(opath);
 finish:
     if (attr == 0xFFFFFFFF)
         /* File does not exist, or cannot read attributes */
@@ -1923,6 +1942,7 @@
                            || !(attr & FILE_ATTRIBUTE_READONLY)
                            || (attr & FILE_ATTRIBUTE_DIRECTORY));
 #else
+    PyObject *opath;
     int res;
     if (!PyArg_ParseTuple(args, "O&i:access",
                           PyUnicode_FSConverter, &opath, &mode))
@@ -2042,7 +2062,7 @@
 posix_chmod(PyObject *self, PyObject *args)
 {
     PyObject *opath = NULL;
-    char *path = NULL;
+    const char *path = NULL;
     int i;
     int res;
 #ifdef MS_WINDOWS
@@ -2073,10 +2093,10 @@
        are also valid. */
     PyErr_Clear();
 
-    if (!PyArg_ParseTuple(args, "O&i:chmod", PyUnicode_FSConverter,
-                          &opath, &i))
-        return NULL;
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "yi:chmod", &path, &i))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     attr = GetFileAttributesA(path);
     if (attr != 0xFFFFFFFF) {
@@ -2091,10 +2111,8 @@
     Py_END_ALLOW_THREADS
     if (!res) {
         win32_error("chmod", path);
-        Py_DECREF(opath);
-        return NULL;
-    }
-    Py_DECREF(opath);
+        return NULL;
+    }
     Py_INCREF(Py_None);
     return Py_None;
 #else /* MS_WINDOWS */
@@ -2450,51 +2468,56 @@
 static PyObject *
 win32_link(PyObject *self, PyObject *args)
 {
-    PyObject *osrc, *odst;
-    char *src, *dst;
-    BOOL rslt;
-    PyObject *usrc, *udst;
-
-    if (PyArg_ParseTuple(args, "UU:link", &usrc, &udst))
+    PyObject *src, *dst;
+    BOOL ok;
+
+    if (PyArg_ParseTuple(args, "UU:link", &src, &dst))
     {
         wchar_t *wsrc, *wdst;
-        wsrc = PyUnicode_AsUnicode(usrc);
+
+        wsrc = PyUnicode_AsUnicode(src);
         if (wsrc == NULL)
+            goto error;
+        wdst = PyUnicode_AsUnicode(dst);
+        if (wdst == NULL)
+            goto error;
+
+        Py_BEGIN_ALLOW_THREADS
+        ok = CreateHardLinkW(wdst, wsrc, NULL);
+        Py_END_ALLOW_THREADS
+
+        if (!ok)
+            return win32_error("link", NULL);
+        Py_RETURN_NONE;
+    }
+    else {
+        PyErr_Clear();
+        if (!PyArg_ParseTuple(args, "O&O&:link",
+                              PyUnicode_FSConverter, &src,
+                              PyUnicode_FSConverter, &dst))
             return NULL;
-        wdst = PyUnicode_AsUnicode(udst);
-        if (wdst == NULL)
-            return NULL;
+
+        if (win32_warn_bytes_api())
+            goto error;
 
         Py_BEGIN_ALLOW_THREADS
-        rslt = CreateHardLinkW(wdst, wsrc, NULL);
+        ok = CreateHardLinkA(PyBytes_AS_STRING(dst),
+                             PyBytes_AS_STRING(src),
+                             NULL);
         Py_END_ALLOW_THREADS
 
-        if (rslt == 0)
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+
+        if (!ok)
             return win32_error("link", NULL);
-
         Py_RETURN_NONE;
-    }
-
-    /* Narrow strings also valid. */
-    PyErr_Clear();
-
-    if (!PyArg_ParseTuple(args, "O&O&:link", PyUnicode_FSConverter, &osrc,
-                          PyUnicode_FSConverter, &odst))
-        return NULL;
-
-    src = PyBytes_AsString(osrc);
-    dst = PyBytes_AsString(odst);
-
-    Py_BEGIN_ALLOW_THREADS
-    rslt = CreateHardLinkA(dst, src, NULL);
-    Py_END_ALLOW_THREADS
-
-    Py_DECREF(osrc);
-    Py_DECREF(odst);
-    if (rslt == 0)
-        return win32_error("link", NULL);
-
-    Py_RETURN_NONE;
+
+    error:
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+        return NULL;
+    }
 }
 #endif /* MS_WINDOWS */
 
@@ -2519,7 +2542,8 @@
     HANDLE hFindFile;
     BOOL result;
     WIN32_FIND_DATA FileData;
-    PyObject *opath;
+    const char *path;
+    Py_ssize_t pathlen;
     char namebuf[MAX_PATH+5]; /* Overallocate for \\*.*\0 */
     char *bufptr = namebuf;
     Py_ssize_t len = sizeof(namebuf)-5; /* only claim to have space for MAX_PATH */
@@ -2613,17 +2637,16 @@
        are also valid. */
     PyErr_Clear();
 
-    if (!PyArg_ParseTuple(args, "O&:listdir",
-                          PyUnicode_FSConverter, &opath))
-        return NULL;
-    if (PyBytes_GET_SIZE(opath)+1 > MAX_PATH) {
+    if (!PyArg_ParseTuple(args, "y#:listdir", &path, &pathlen))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
+    if (pathlen+1 > MAX_PATH) {
         PyErr_SetString(PyExc_ValueError, "path too long");
-        Py_DECREF(opath);
-        return NULL;
-    }
-    strcpy(namebuf, PyBytes_AsString(opath));
-    len = PyObject_Size(opath);
-    Py_DECREF(opath);
+        return NULL;
+    }
+    strcpy(namebuf, path);
+    len = pathlen;
     if (len > 0) {
         char ch = namebuf[len-1];
         if (ch != SEP && ch != ALTSEP && ch != ':')
@@ -2915,11 +2938,9 @@
 static PyObject *
 posix__getfullpathname(PyObject *self, PyObject *args)
 {
-    PyObject *opath;
-    char *path;
+    const char *path;
     char outbuf[MAX_PATH*2];
     char *temp;
-#ifdef MS_WINDOWS
     PyObject *po;
 
     if (PyArg_ParseTuple(args, "U|:_getfullpathname", &po))
@@ -2953,19 +2974,17 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-#endif
-
-    if (!PyArg_ParseTuple (args, "O&:_getfullpathname",
-                           PyUnicode_FSConverter, &opath))
-        return NULL;
-    path = PyBytes_AsString(opath);
+
+    if (!PyArg_ParseTuple (args, "y:_getfullpathname",
+                           &path))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     if (!GetFullPathName(path, Py_ARRAY_LENGTH(outbuf),
                          outbuf, &temp)) {
         win32_error("GetFullPathName", path);
-        Py_DECREF(opath);
-        return NULL;
-    }
-    Py_DECREF(opath);
+        return NULL;
+    }
     if (PyUnicode_Check(PyTuple_GetItem(args, 0))) {
         return PyUnicode_Decode(outbuf, strlen(outbuf),
                                 Py_FileSystemDefaultEncoding, NULL);
@@ -3069,8 +3088,7 @@
 static PyObject *
 posix__isdir(PyObject *self, PyObject *args)
 {
-    PyObject *opath;
-    char *path;
+    const char *path;
     PyObject *po;
     DWORD attributes;
 
@@ -3088,11 +3106,10 @@
        are also valid. */
     PyErr_Clear();
 
-    if (!PyArg_ParseTuple(args, "O&:_isdir",
-                          PyUnicode_FSConverter, &opath))
-        return NULL;
-
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "y:_isdir", &path))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     attributes = GetFileAttributesA(path);
     if (attributes == INVALID_FILE_ATTRIBUTES)
         Py_RETURN_FALSE;
@@ -3113,8 +3130,7 @@
 posix_mkdir(PyObject *self, PyObject *args)
 {
     int res;
-    PyObject *opath;
-    char *path;
+    const char *path;
     int mode = 0777;
 
 #ifdef MS_WINDOWS
@@ -3136,22 +3152,21 @@
     /* Drop the argument parsing error as narrow strings
        are also valid. */
     PyErr_Clear();
-    if (!PyArg_ParseTuple(args, "O&|i:mkdir",
-                          PyUnicode_FSConverter, &opath, &mode))
-        return NULL;
-    path = PyBytes_AsString(opath);
+    if (!PyArg_ParseTuple(args, "y|i:mkdir", &path, &mode))
+        return NULL;
+    if (win32_warn_bytes_api())
+        return NULL;
     Py_BEGIN_ALLOW_THREADS
     res = CreateDirectoryA(path, NULL);
     Py_END_ALLOW_THREADS
     if (!res) {
         win32_error("mkdir", path);
-        Py_DECREF(opath);
-        return NULL;
-    }
-    Py_DECREF(opath);
+        return NULL;
+    }
     Py_INCREF(Py_None);
     return Py_None;
 #else
+    PyObject *opath;
 
     if (!PyArg_ParseTuple(args, "O&|i:mkdir",
                           PyUnicode_FSConverter, &opath, &mode))
@@ -3265,44 +3280,54 @@
 posix_rename(PyObject *self, PyObject *args)
 {
 #ifdef MS_WINDOWS
-    PyObject *o1, *o2;
-    wchar_t *w1, *w2;
-    char *p1, *p2;
+    PyObject *src, *dst;
     BOOL result;
-    if (!PyArg_ParseTuple(args, "OO:rename", &o1, &o2))
-        goto error;
-    if (!convert_to_unicode(&o1))
-        goto error;
-    if (!convert_to_unicode(&o2)) {
-        Py_DECREF(o1);
-        goto error;
-    }
-    w1 = PyUnicode_AsUnicode(o1);
-    if (w1 == NULL)
-        goto error;
-    w2 = PyUnicode_AsUnicode(o2);
-    if (w2 == NULL)
-        goto error;
-    Py_BEGIN_ALLOW_THREADS
-    result = MoveFileW(w1, w2);
-    Py_END_ALLOW_THREADS
-    Py_DECREF(o1);
-    Py_DECREF(o2);
-    if (!result)
-        return win32_error("rename", NULL);
-    Py_INCREF(Py_None);
-    return Py_None;
+    if (PyArg_ParseTuple(args, "UU:rename", &src, &dst))
+    {
+        wchar_t *wsrc, *wdst;
+
+        wsrc = PyUnicode_AsUnicode(src);
+        if (wsrc == NULL)
+            return NULL;
+        wdst = PyUnicode_AsUnicode(dst);
+        if (wdst == NULL)
+            return NULL;
+        Py_BEGIN_ALLOW_THREADS
+        result = MoveFileW(wsrc, wdst);
+        Py_END_ALLOW_THREADS
+        if (!result)
+            return win32_error("rename", NULL);
+        Py_INCREF(Py_None);
+        return Py_None;
+    }
+    else {
+        PyErr_Clear();
+        if (!PyArg_ParseTuple(args, "O&O&:rename",
+                              PyUnicode_FSConverter, &src,
+                              PyUnicode_FSConverter, &dst))
+            return NULL;
+
+        if (win32_warn_bytes_api())
+            goto error;
+
+        Py_BEGIN_ALLOW_THREADS
+        result = MoveFileA(PyBytes_AS_STRING(src),
+                           PyBytes_AS_STRING(dst));
+        Py_END_ALLOW_THREADS
+
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+
+        if (!result)
+            return win32_error("rename", NULL);
+        Py_INCREF(Py_None);
+        return Py_None;
+
 error:
-    PyErr_Clear();
-    if (!PyArg_ParseTuple(args, "ss:rename", &p1, &p2))
-        return NULL;
-    Py_BEGIN_ALLOW_THREADS
-    result = MoveFileA(p1, p2);
-    Py_END_ALLOW_THREADS
-    if (!result)
-        return win32_error("rename", NULL);
-    Py_INCREF(Py_None);
-    return Py_None;
+        Py_XDECREF(src);
+        Py_XDECREF(dst);
+        return NULL;
+    }
 #else
     return posix_2str(args, "O&O&:rename", rename);
 #endif
@@ -3546,8 +3571,7 @@
     PyObject *arg = Py_None;
     PyObject *obwpath;
     wchar_t *wpath = NULL;
-    PyObject *oapath;
-    char *apath;
+    const char *apath;
     HANDLE hFile;
     time_t atimesec, mtimesec;
     long ausec, musec;
@@ -3571,11 +3595,11 @@
            are also valid. */
         PyErr_Clear();
 
-        if (!PyArg_ParseTuple(args, "O&|O:utime",
-                        PyUnicode_FSConverter, &oapath, &arg))
+        if (!PyArg_ParseTuple(args, "y|O:utime", &apath, &arg))
             return NULL;
-
-        apath = PyBytes_AsString(oapath);
+        if (win32_warn_bytes_api())
+            return NULL;
+
         Py_BEGIN_ALLOW_THREADS
         hFile = CreateFileA(apath, FILE_WRITE_ATTRIBUTES, 0,
                             NULL, OPEN_EXISTING,
@@ -3583,10 +3607,8 @@
         Py_END_ALLOW_THREADS
         if (hFile == INVALID_HANDLE_VALUE) {
             win32_error("utime", apath);
-            Py_DECREF(oapath);
             return NULL;
         }
-        Py_DECREF(oapath);
     }
 
     if (arg == Py_None) {
@@ -6516,7 +6538,8 @@
 win_symlink(PyObject *self, PyObject *args, PyObject *kwargs)
 {
     static char *kwlist[] = {"src", "dest", "target_is_directory", NULL};
-    PyObject *src, *dest;
+    PyObject *osrc, *odest;
+    PyObject *usrc = NULL, *udest = NULL;
     wchar_t *wsrc, *wdest;
     int target_is_directory = 0;
     DWORD res;
@@ -6528,24 +6551,25 @@
         return PyErr_Format(PyExc_NotImplementedError,
             "CreateSymbolicLinkW not found");
     }
-    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "OO|i:symlink",
-        kwlist, &src, &dest, &target_is_directory))
-        return NULL;
+    if (!PyArg_ParseTupleAndKeywords(
+            args, kwargs, "OO|i:symlink", kwlist,
+            &osrc, &odest, &target_is_directory))
+        return NULL;
+
+    usrc = win32_decode_filename(osrc);
+    if (!usrc)
+        return NULL;
+    udest = win32_decode_filename(odest);
+    if (!udest)
+        goto error;
 
     if (win32_can_symlink == 0)
         return PyErr_Format(PyExc_OSError, "symbolic link privilege not held");
 
-    if (!convert_to_unicode(&src))
-        return NULL;
-    if (!convert_to_unicode(&dest)) {
-        Py_DECREF(src);
-        return NULL;
-    }
-
-    wsrc = PyUnicode_AsUnicode(src);
+    wsrc = PyUnicode_AsUnicode(usrc);
     if (wsrc == NULL)
         goto error;
-    wdest = PyUnicode_AsUnicode(dest);
+    wdest = PyUnicode_AsUnicode(udest);
     if (wsrc == NULL)
         goto error;
 
@@ -6563,17 +6587,17 @@
     res = Py_CreateSymbolicLinkW(wdest, wsrc, target_is_directory);
     Py_END_ALLOW_THREADS
 
-    Py_DECREF(src);
-    Py_DECREF(dest);
+    Py_DECREF(usrc);
+    Py_DECREF(udest);
     if (!res)
-        return win32_error_object("symlink", src);
+        return win32_error_object("symlink", osrc);
 
     Py_INCREF(Py_None);
     return Py_None;
 
 error:
-    Py_DECREF(src);
-    Py_DECREF(dest);
+    Py_XDECREF(usrc);
+    Py_XDECREF(udest);
     return NULL;
 }
 #endif /* defined(HAVE_SYMLINK) && defined(MS_WINDOWS) */
@@ -6791,6 +6815,12 @@
                           PyUnicode_FSConverter, &ofile,
                           &flag, &mode))
         return NULL;
+#ifdef MS_WINDOWS
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(ofile);
+        return NULL;
+    }
+#endif
     file = PyBytes_AsString(ofile);
     Py_BEGIN_ALLOW_THREADS
     fd = open(file, flag, mode);
@@ -9290,6 +9320,10 @@
                           PyUnicode_FSConverter, &ofilepath,
                           &operation))
         return NULL;
+    if (win32_warn_bytes_api()) {
+        Py_DECREF(ofilepath);
+        return NULL;
+    }
     filepath = PyBytes_AsString(ofilepath);
     Py_BEGIN_ALLOW_THREADS
     rc = ShellExecute((HWND)0, operation, filepath,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 22:42:06 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 15 Nov 2011 22:42:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=22unicode=5Finternal=22_co?=
 =?utf8?q?dec_has_been_deprecated=3A_fix_related_tests?=
Message-ID: <E1RQQla-0005Db-1O@dinsdale.python.org>

http://hg.python.org/cpython/rev/ec8c0ac81706
changeset:   73579:ec8c0ac81706
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 15 22:44:05 2011 +0100
summary:
  "unicode_internal" codec has been deprecated: fix related tests

files:
  Lib/test/test_codeccallbacks.py |  31 +++++++++++-----
  Lib/test/test_codecs.py         |  39 ++++++++++++++-------
  Lib/test/test_unicode.py        |  22 ++++++++---
  3 files changed, 62 insertions(+), 30 deletions(-)


diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -1,5 +1,10 @@
-import test.support, unittest
-import sys, codecs, html.entities, unicodedata
+import codecs
+import html.entities
+import sys
+import test.support
+import unicodedata
+import unittest
+import warnings
 
 try:
     import ctypes
@@ -621,12 +626,15 @@
                 ("utf-7", b"+x-"),
                 ("unicode-internal", b"\x00"),
             ):
-                self.assertRaises(
-                    TypeError,
-                    bytes.decode,
-                    enc,
-                    "test.badhandler"
-                )
+                with warnings.catch_warnings():
+                    # unicode-internal has been deprecated
+                    warnings.simplefilter("ignore", DeprecationWarning)
+                    self.assertRaises(
+                        TypeError,
+                        bytes.decode,
+                        enc,
+                        "test.badhandler"
+                    )
 
     def test_lookup(self):
         self.assertEqual(codecs.strict_errors, codecs.lookup_error("strict"))
@@ -842,8 +850,11 @@
             else:
                 raise TypeError("don't know how to handle %r" % exc)
         codecs.register_error("test.replacing", replacing)
-        for (encoding, data) in baddata:
-            self.assertRaises(TypeError, data.decode, encoding, "test.replacing")
+        with warnings.catch_warnings():
+            # unicode-internal has been deprecated
+            warnings.simplefilter("ignore", DeprecationWarning)
+            for (encoding, data) in baddata:
+                self.assertRaises(TypeError, data.decode, encoding, "test.replacing")
 
         def mutating(exc):
             if isinstance(exc, UnicodeDecodeError):
diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1,8 +1,12 @@
+import _testcapi
+import codecs
+import io
+import locale
+import sys
+import unittest
+import warnings
+
 from test import support
-import unittest
-import codecs
-import locale
-import sys, _testcapi, io
 
 if sys.platform == 'win32':
     VISTA_OR_LATER = (sys.getwindowsversion().major >= 6)
@@ -1051,12 +1055,16 @@
         self.assertEqual(("ab", 12), ignored)
 
     def test_encode_length(self):
-        # Issue 3739
-        encoder = codecs.getencoder("unicode_internal")
-        self.assertEqual(encoder("a")[1], 1)
-        self.assertEqual(encoder("\xe9\u0142")[1], 2)
+        with warnings.catch_warnings():
+            # unicode-internal has been deprecated
+            warnings.simplefilter("ignore", DeprecationWarning)
 
-        self.assertEqual(codecs.escape_encode(br'\x00')[1], 4)
+            # Issue 3739
+            encoder = codecs.getencoder("unicode_internal")
+            self.assertEqual(encoder("a")[1], 1)
+            self.assertEqual(encoder("\xe9\u0142")[1], 2)
+
+            self.assertEqual(codecs.escape_encode(br'\x00')[1], 4)
 
 # From http://www.gnu.org/software/libidn/draft-josefsson-idn-test-vectors.html
 nameprep_tests = [
@@ -1512,10 +1520,15 @@
             elif encoding == "latin_1":
                 name = "latin_1"
             self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
-            (b, size) = codecs.getencoder(encoding)(s)
-            self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
-            (chars, size) = codecs.getdecoder(encoding)(b)
-            self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
+
+            with warnings.catch_warnings():
+                # unicode-internal has been deprecated
+                warnings.simplefilter("ignore", DeprecationWarning)
+
+                (b, size) = codecs.getencoder(encoding)(s)
+                self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
+                (chars, size) = codecs.getdecoder(encoding)(b)
+                self.assertEqual(chars, s, "%r != %r (encoding=%r)" % (chars, s, encoding))
 
             if encoding not in broken_unicode_with_streams:
                 # check stream reader/writer
diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -5,13 +5,13 @@
 (c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
 
 """#"
+import _string
 import codecs
 import struct
 import sys
 import unittest
 import warnings
 from test import support, string_tests
-import _string
 
 # Error handling (bad decoder return)
 def search_function(encoding):
@@ -1394,7 +1394,11 @@
             for encoding in ('utf-7', 'utf-8', 'utf-16', 'utf-16-le',
                              'utf-16-be', 'raw_unicode_escape',
                              'unicode_escape', 'unicode_internal'):
-                self.assertEqual(str(u.encode(encoding),encoding), u)
+                with warnings.catch_warnings():
+                    # unicode-internal has been deprecated
+                    warnings.simplefilter("ignore", DeprecationWarning)
+
+                    self.assertEqual(str(u.encode(encoding),encoding), u)
 
         # Roundtrip safety for BMP (just the first 256 chars)
         for c in range(256):
@@ -1409,11 +1413,15 @@
                 self.assertEqual(str(u.encode(encoding),encoding), u)
 
         # Roundtrip safety for non-BMP (just a few chars)
-        u = '\U00010001\U00020002\U00030003\U00040004\U00050005'
-        for encoding in ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be',
-                         'raw_unicode_escape',
-                         'unicode_escape', 'unicode_internal'):
-            self.assertEqual(str(u.encode(encoding),encoding), u)
+        with warnings.catch_warnings():
+            # unicode-internal has been deprecated
+            warnings.simplefilter("ignore", DeprecationWarning)
+
+            u = '\U00010001\U00020002\U00030003\U00040004\U00050005'
+            for encoding in ('utf-8', 'utf-16', 'utf-16-le', 'utf-16-be',
+                             'raw_unicode_escape',
+                             'unicode_escape', 'unicode_internal'):
+                self.assertEqual(str(u.encode(encoding),encoding), u)
 
         # UTF-8 must be roundtrip safe for all code points
         # (except surrogates, which are forbidden).

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 15 23:48:30 2011
From: python-checkins at python.org (eli.bendersky)
Date: Tue, 15 Nov 2011 23:48:30 +0100
Subject: [Python-checkins] =?utf8?q?devguide=3A_document_existing_hooks_fo?=
 =?utf8?q?r_commit_messages_mentioning_issues=2E_Closes_issue?=
Message-ID: <E1RQRnq-0002mQ-4Q@dinsdale.python.org>

http://hg.python.org/devguide/rev/21758f27f98f
changeset:   462:21758f27f98f
user:        Eli Bendersky <eliben at gmail.com>
date:        Wed Nov 16 00:48:23 2011 +0200
summary:
  document existing hooks for commit messages mentioning issues. Closes issue 13388

files:
  committing.rst |  21 +++++++++++++++++++++
  1 files changed, 21 insertions(+), 0 deletions(-)


diff --git a/committing.rst b/committing.rst
--- a/committing.rst
+++ b/committing.rst
@@ -68,6 +68,27 @@
   and the original NEWS entry remains valid, then no additional entry is
   needed.
 
+Mercurial hooks
+'''''''''''''''
+
+Special hooks have been added to the Mercurial repository to enable notifying
+the issue tracker of a commit related to an issue.
+
+A commit message can mention one or several issues in one of the following
+ways::
+
+   #12345
+   issue12345
+   issue 12345
+   bug12345
+   bug 12345
+
+where 12345 is the number of the issue. The commit details (including its
+changeset, branch and commit message) will then be posted as a message to the
+issue's page in the tracker, for each mentioned issue.
+
+If "closes" (or "closed", or "closing") is prepended, the issue is
+automatically closed as "fixed".
 
 Working with Mercurial_
 -----------------------

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Wed Nov 16 00:16:58 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 16 Nov 2011 00:16:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_What=27s_New_in_3=2E3=3A_Ad?=
 =?utf8?q?d_a_=22Deprecated_=2E=2E=2E=22_section?=
Message-ID: <E1RQSFO-0005D8-61@dinsdale.python.org>

http://hg.python.org/cpython/rev/a411f285362f
changeset:   73580:a411f285362f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 16 00:18:57 2011 +0100
summary:
  What's New in 3.3: Add a "Deprecated ..." section

files:
  Doc/whatsnew/3.3.rst |  22 +++++++++++++++++-----
  1 files changed, 17 insertions(+), 5 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -558,6 +558,22 @@
 are no longer supported due to maintenance burden.
 
 
+Deprecated modules, functions and methods
+=========================================
+
+* The :mod:`packaging` module replaces the :mod:`distutils` module
+* The ``unicode_internal`` codec has been deprecated because of the
+  :pep:`393`, use UTF-8, UTF-16 (``utf-16-le`` or ``utf-16-le``), or UTF-32
+  (``utf-32-le`` or ``utf-32-le``) instead.
+* :meth:`ftplib.FTP.nlst` and :meth:`ftplib.FTP.dir`: use
+  :meth:`ftplib.FTP.mlsd` instead.
+* :func:`platform.popen`: use the :mod:`subprocess` module. Check especially
+  the :ref:`subprocess-replacements` section.
+* :issue:`13374`: The Windows bytes API has been deprecated in the :mod:`os`
+  module. Use Unicode filenames instead of bytes filenames to not depend on
+  the ANSI code page anymore and to support any filename.
+
+
 Porting to Python 3.3
 =====================
 
@@ -567,16 +583,12 @@
 Porting Python code
 -------------------
 
-* Issue #12326: On Linux, sys.platform doesn't contain the major version
+* :issue:`12326`: On Linux, sys.platform doesn't contain the major version
   anymore. It is now always 'linux', instead of 'linux2' or 'linux3' depending
   on the Linux version used to build Python. Replace sys.platform == 'linux2'
   with sys.platform.startswith('linux'), or directly sys.platform == 'linux' if
   you don't need to support older Python versions.
 
-* Issue #13374: The Windows bytes API has been deprecated in the :mod:`os`
-  module. Use Unicode filenames instead of bytes filenames to not depend on the
-  ANSI code page anymore and to support any filename.
-
 Porting C code
 --------------
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 00:32:50 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 16 Nov 2011 00:32:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313374=3A_Skip_depr?=
 =?utf8?q?ecation_tests_for_os=2Esymlink=28=29_on_Windows_XP?=
Message-ID: <E1RQSUk-0006US-Do@dinsdale.python.org>

http://hg.python.org/cpython/rev/afc716e463a1
changeset:   73581:afc716e463a1
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 16 00:34:44 2011 +0100
summary:
  Issue #13374: Skip deprecation tests for os.symlink() on Windows XP

To avoid a NotImplementedError('CreateSymbolicLinkW not found') error.

files:
  Lib/test/test_os.py |  9 ++++++++-
  1 files changed, 8 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -1647,12 +1647,19 @@
                 (os.rmdir, filename),
                 (os.startfile, filename),
                 (os.stat, filename),
-                (os.symlink, filename, filename),
                 (os.unlink, filename),
                 (os.utime, filename),
             ):
                 self.assertRaises(DeprecationWarning, func, *args)
 
+    @support.skip_unless_symlink
+    def test_symlink(self):
+        filename = os.fsencode(support.TESTFN)
+        with warnings.catch_warnings():
+            warnings.simplefilter("error", DeprecationWarning)
+            self.assertRaises(DeprecationWarning,
+                              os.symlink, filename, filename)
+
 
 @support.reap_threads
 def test_main():

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 01:02:26 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Wed, 16 Nov 2011 01:02:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313393=3A_BufferedR?=
 =?utf8?q?eader=2Eread1=28=29_now_asks_the_full_requested_size_to?=
Message-ID: <E1RQSxO-0000RU-QI@dinsdale.python.org>

http://hg.python.org/cpython/rev/27bf3d0b8e5f
changeset:   73582:27bf3d0b8e5f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Wed Nov 16 00:56:10 2011 +0100
summary:
  Issue #13393: BufferedReader.read1() now asks the full requested size to
the raw stream instead of limiting itself to the buffer size.

files:
  Misc/NEWS                |   3 +
  Modules/_io/bufferedio.c |  49 +++++++++------------------
  2 files changed, 19 insertions(+), 33 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13393: BufferedReader.read1() now asks the full requested size to
+  the raw stream instead of limiting itself to the buffer size.
+
 - Issue #13392: Writing a pyc file should now be atomic under Windows as well.
 
 - Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -889,51 +889,34 @@
     if (n == 0)
         return PyBytes_FromStringAndSize(NULL, 0);
 
-    if (!ENTER_BUFFERED(self))
-        return NULL;
-
     /* Return up to n bytes.  If at least one byte is buffered, we
        only return buffered bytes.  Otherwise, we do one raw read. */
 
-    /* XXX: this mimicks the io.py implementation but is probably wrong.
-       If we need to read from the raw stream, then we could actually read
-       all `n` bytes asked by the caller (and possibly more, so as to fill
-       our buffer for the next reads). */
-
     have = Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t);
     if (have > 0) {
-        if (n > have)
-            n = have;
-        res = PyBytes_FromStringAndSize(self->buffer + self->pos, n);
-        if (res == NULL)
-            goto end;
-        self->pos += n;
-        goto end;
+        n = Py_MIN(have, n);
+        res = _bufferedreader_read_fast(self, n);
+        assert(res != Py_None);
+        return res;
     }
-
-    if (self->writable) {
-        res = buffered_flush_and_rewind_unlocked(self);
-        if (res == NULL)
-            goto end;
+    res = PyBytes_FromStringAndSize(NULL, n);
+    if (res == NULL)
+        return NULL;
+    if (!ENTER_BUFFERED(self)) {
         Py_DECREF(res);
+        return NULL;
     }
-
-    /* Fill the buffer from the raw stream, and copy it to the result. */
     _bufferedreader_reset_buf(self);
-    r = _bufferedreader_fill_buffer(self);
-    if (r == -1)
-        goto end;
+    r = _bufferedreader_raw_read(self, PyBytes_AS_STRING(res), n);
+    LEAVE_BUFFERED(self)
+    if (r == -1) {
+        Py_DECREF(res);
+        return NULL;
+    }
     if (r == -2)
         r = 0;
     if (n > r)
-        n = r;
-    res = PyBytes_FromStringAndSize(self->buffer, n);
-    if (res == NULL)
-        goto end;
-    self->pos = n;
-
-end:
-    LEAVE_BUFFERED(self)
+        _PyBytes_Resize(&res, r);
     return res;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 03:30:16 2011
From: python-checkins at python.org (ned.deily)
Date: Wed, 16 Nov 2011 03:30:16 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzg3OTM6?=
 =?utf8?q?_Prevent_IDLE_crash_in_2=2E7_when_given_strings_with?=
Message-ID: <E1RQVGS-0003CV-RR@dinsdale.python.org>

http://hg.python.org/cpython/rev/e277fe8380e0
changeset:   73583:e277fe8380e0
branch:      2.7
parent:      73554:050772822bde
user:        Ned Deily <nad at acm.org>
date:        Tue Nov 15 18:29:02 2011 -0800
summary:
  Issue #8793: Prevent IDLE crash in 2.7 when given strings with
invalid hex escape sequences.

files:
  Lib/idlelib/ScriptBinding.py |  2 +-
  Misc/NEWS                    |  3 +++
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/idlelib/ScriptBinding.py b/Lib/idlelib/ScriptBinding.py
--- a/Lib/idlelib/ScriptBinding.py
+++ b/Lib/idlelib/ScriptBinding.py
@@ -101,7 +101,7 @@
             try:
                 # If successful, return the compiled code
                 return compile(source, filename, "exec")
-            except (SyntaxError, OverflowError), err:
+            except (SyntaxError, OverflowError, ValueError), err:
                 try:
                     msg, (errorfilename, lineno, offset, line) = err
                     if not errorfilename:
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,9 @@
 Library
 -------
 
+- Issue #8793: Prevent IDLE crash when given strings with invalid hex escape
+  sequences.
+
 - Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly
   handles non-valid attributes, including adjacent and unquoted attributes.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 04:54:09 2011
From: python-checkins at python.org (eli.bendersky)
Date: Wed, 16 Nov 2011 04:54:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_remove_duplicat?=
 =?utf8?q?ed_paragraph_in_the_tutorial?=
Message-ID: <E1RQWZd-0001KX-PX@dinsdale.python.org>

http://hg.python.org/cpython/rev/ed8aace4db0e
changeset:   73584:ed8aace4db0e
branch:      2.7
user:        Eli Bendersky <eliben at gmail.com>
date:        Wed Nov 16 05:54:07 2011 +0200
summary:
  remove duplicated paragraph in the tutorial

files:
  Doc/tutorial/introduction.rst |  7 -------
  1 files changed, 0 insertions(+), 7 deletions(-)


diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst
--- a/Doc/tutorial/introduction.rst
+++ b/Doc/tutorial/introduction.rst
@@ -240,13 +240,6 @@
    This is a rather long string containing\n\
    several lines of text much as you would do in C.
 
-The interpreter prints the result of string operations in the same way as they
-are typed for input: inside quotes, and with quotes and other funny characters
-escaped by backslashes, to show the precise value.  The string is enclosed in
-double quotes if the string contains a single quote and no double quotes, else
-it's enclosed in single quotes.  (The :keyword:`print` statement, described
-later, can be used to write strings without quotes or escapes.)
-
 Strings can be concatenated (glued together) with the ``+`` operator, and
 repeated with ``*``::
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 05:01:16 2011
From: python-checkins at python.org (eli.bendersky)
Date: Wed, 16 Nov 2011 05:01:16 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_fixing_typos_in?=
	=?utf8?q?_optparse_doc?=
Message-ID: <E1RQWgW-0001tm-FW@dinsdale.python.org>

http://hg.python.org/cpython/rev/2a3b60d28f6c
changeset:   73585:2a3b60d28f6c
branch:      2.7
user:        Eli Bendersky <eliben at gmail.com>
date:        Wed Nov 16 06:01:14 2011 +0200
summary:
  fixing typos in optparse doc

files:
  Doc/library/optparse.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst
--- a/Doc/library/optparse.rst
+++ b/Doc/library/optparse.rst
@@ -610,8 +610,8 @@
 
        -g                  Group option.
 
-A bit more complete example might invole using more than one group: still
-extendind the previous example::
+A bit more complete example might involve using more than one group: still
+extending the previous example::
 
     group = OptionGroup(parser, "Dangerous Options",
                         "Caution: use these options at your own risk.  "

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 05:03:12 2011
From: python-checkins at python.org (eli.bendersky)
Date: Wed, 16 Nov 2011 05:03:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fixing_typos_in?=
	=?utf8?q?_optparse_doc?=
Message-ID: <E1RQWiO-0001yc-CO@dinsdale.python.org>

http://hg.python.org/cpython/rev/738c43484b47
changeset:   73586:738c43484b47
branch:      3.2
parent:      73571:a00bb30cf775
user:        Eli Bendersky <eliben at gmail.com>
date:        Wed Nov 16 06:02:21 2011 +0200
summary:
  fixing typos in optparse doc

files:
  Doc/library/optparse.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst
--- a/Doc/library/optparse.rst
+++ b/Doc/library/optparse.rst
@@ -607,8 +607,8 @@
 
        -g                  Group option.
 
-A bit more complete example might invole using more than one group: still
-extendind the previous example::
+A bit more complete example might involve using more than one group: still
+extending the previous example::
 
     group = OptionGroup(parser, "Dangerous Options",
                         "Caution: use these options at your own risk.  "

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 05:03:13 2011
From: python-checkins at python.org (eli.bendersky)
Date: Wed, 16 Nov 2011 05:03:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_fixing_typos_in_optparse_doc?=
Message-ID: <E1RQWiP-0001yg-4K@dinsdale.python.org>

http://hg.python.org/cpython/rev/72bdfef4aea7
changeset:   73587:72bdfef4aea7
parent:      73582:27bf3d0b8e5f
parent:      73586:738c43484b47
user:        Eli Bendersky <eliben at gmail.com>
date:        Wed Nov 16 06:03:09 2011 +0200
summary:
  fixing typos in optparse doc

files:
  Doc/library/optparse.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/library/optparse.rst b/Doc/library/optparse.rst
--- a/Doc/library/optparse.rst
+++ b/Doc/library/optparse.rst
@@ -607,8 +607,8 @@
 
        -g                  Group option.
 
-A bit more complete example might invole using more than one group: still
-extendind the previous example::
+A bit more complete example might involve using more than one group: still
+extending the previous example::
 
     group = OptionGroup(parser, "Dangerous Options",
                         "Caution: use these options at your own risk.  "

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Nov 16 05:36:04 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 16 Nov 2011 05:36:04 +0100
Subject: [Python-checkins] Daily reference leaks (27bf3d0b8e5f): sum=0
Message-ID: <E1RQXEC-0005jI-Ph@ap.vmr.nerim.net>

results for 27bf3d0b8e5f on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog_Z_2bY', '-x']

From python-checkins at python.org  Wed Nov 16 08:39:33 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 16 Nov 2011 08:39:33 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313406=3A_silence_depreca?=
 =?utf8?q?tion_warnings_in_test=5Fcodecs=2E?=
Message-ID: <E1RQa5l-0002ug-V9@dinsdale.python.org>

http://hg.python.org/cpython/rev/4f534cd40f54
changeset:   73588:4f534cd40f54
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Wed Nov 16 09:39:10 2011 +0200
summary:
  #13406: silence deprecation warnings in test_codecs.

files:
  Lib/test/test_codecs.py |  23 +++++++++++++++--------
  Modules/_codecsmodule.c |   2 +-
  Objects/unicodeobject.c |   2 +-
  3 files changed, 17 insertions(+), 10 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1025,17 +1025,22 @@
         for internal, uni in ok:
             if sys.byteorder == "little":
                 internal = bytes(reversed(internal))
-            self.assertEqual(uni, internal.decode("unicode_internal"))
+            with support.check_warnings():
+                self.assertEqual(uni, internal.decode("unicode_internal"))
         for internal in not_ok:
             if sys.byteorder == "little":
                 internal = bytes(reversed(internal))
-            self.assertRaises(UnicodeDecodeError, internal.decode,
-                "unicode_internal")
+            with support.check_warnings(('unicode_internal codecs has been '
+                                         'deprecated', DeprecationWarning)):
+                self.assertRaises(UnicodeDecodeError, internal.decode,
+                                  "unicode_internal")
 
     @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
     def test_decode_error_attributes(self):
         try:
-            b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
+            with support.check_warnings(('unicode_internal codecs has been '
+                                         'deprecated', DeprecationWarning)):
+                b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
         except UnicodeDecodeError as ex:
             self.assertEqual("unicode_internal", ex.encoding)
             self.assertEqual(b"\x00\x00\x00\x00\x00\x11\x11\x00", ex.object)
@@ -1048,10 +1053,12 @@
     def test_decode_callback(self):
         codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
         decoder = codecs.getdecoder("unicode_internal")
-        ab = "ab".encode("unicode_internal").decode()
-        ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
-                                "ascii"),
-                          "UnicodeInternalTest")
+        with support.check_warnings(('unicode_internal codecs has been '
+                                     'deprecated', DeprecationWarning)):
+            ab = "ab".encode("unicode_internal").decode()
+            ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),
+                                    "ascii"),
+                              "UnicodeInternalTest")
         self.assertEqual(("ab", 12), ignored)
 
     def test_encode_length(self):
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -688,7 +688,7 @@
             return NULL;
 
         if (PyErr_WarnEx(PyExc_DeprecationWarning,
-                         "unicode_internal codecs has been deprecated",
+                         "unicode_internal codec has been deprecated",
                          1))
             return NULL;
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6235,7 +6235,7 @@
     PyObject *exc = NULL;
 
     if (PyErr_WarnEx(PyExc_DeprecationWarning,
-                     "unicode_internal codecs has been deprecated",
+                     "unicode_internal codec has been deprecated",
                      1))
         return NULL;
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 08:54:52 2011
From: python-checkins at python.org (ezio.melotti)
Date: Wed, 16 Nov 2011 08:54:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313406=3A_correct_the_err?=
 =?utf8?q?or_message_in_check=5Fwarnings_too=2E?=
Message-ID: <E1RQaKa-00042W-BG@dinsdale.python.org>

http://hg.python.org/cpython/rev/028b61c4f926
changeset:   73589:028b61c4f926
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Wed Nov 16 09:54:19 2011 +0200
summary:
  #13406: correct the error message in check_warnings too.

files:
  Lib/test/test_codecs.py |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1030,7 +1030,7 @@
         for internal in not_ok:
             if sys.byteorder == "little":
                 internal = bytes(reversed(internal))
-            with support.check_warnings(('unicode_internal codecs has been '
+            with support.check_warnings(('unicode_internal codec has been '
                                          'deprecated', DeprecationWarning)):
                 self.assertRaises(UnicodeDecodeError, internal.decode,
                                   "unicode_internal")
@@ -1038,7 +1038,7 @@
     @unittest.skipUnless(SIZEOF_WCHAR_T == 4, 'specific to 32-bit wchar_t')
     def test_decode_error_attributes(self):
         try:
-            with support.check_warnings(('unicode_internal codecs has been '
+            with support.check_warnings(('unicode_internal codec has been '
                                          'deprecated', DeprecationWarning)):
                 b"\x00\x00\x00\x00\x00\x11\x11\x00".decode("unicode_internal")
         except UnicodeDecodeError as ex:
@@ -1053,7 +1053,7 @@
     def test_decode_callback(self):
         codecs.register_error("UnicodeInternalTest", codecs.ignore_errors)
         decoder = codecs.getdecoder("unicode_internal")
-        with support.check_warnings(('unicode_internal codecs has been '
+        with support.check_warnings(('unicode_internal codec has been '
                                      'deprecated', DeprecationWarning)):
             ab = "ab".encode("unicode_internal").decode()
             ignored = decoder(bytes("%s\x22\x22\x22\x22%s" % (ab[:4], ab[4:]),

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 16 23:42:31 2011
From: python-checkins at python.org (victor.stinner)
Date: Wed, 16 Nov 2011 23:42:31 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313374=3A_Deprecate?=
 =?utf8?q?_os=2Egetcwdb=28=29_on_Windows?=
Message-ID: <E1RQoBb-0004Mv-45@dinsdale.python.org>

http://hg.python.org/cpython/rev/5f239b0ba819
changeset:   73590:5f239b0ba819
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Wed Nov 16 23:43:07 2011 +0100
summary:
  Issue #13374: Deprecate os.getcwdb() on Windows

files:
  Lib/test/test_genericpath.py |  6 ++++--
  Lib/test/test_os.py          |  1 +
  Modules/posixmodule.c        |  3 +++
  3 files changed, 8 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_genericpath.py b/Lib/test/test_genericpath.py
--- a/Lib/test/test_genericpath.py
+++ b/Lib/test/test_genericpath.py
@@ -264,8 +264,10 @@
             self.assertIn(b"foo", self.pathmodule.abspath(b"foo"))
 
         # Abspath returns bytes when the arg is bytes
-        for path in (b'', b'foo', b'f\xf2\xf2', b'/foo', b'C:\\'):
-            self.assertIsInstance(self.pathmodule.abspath(path), bytes)
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore", DeprecationWarning)
+            for path in (b'', b'foo', b'f\xf2\xf2', b'/foo', b'C:\\'):
+                self.assertIsInstance(self.pathmodule.abspath(path), bytes)
 
     def test_realpath(self):
         self.assertIn("foo", self.pathmodule.realpath("foo"))
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -1638,6 +1638,7 @@
                 (os.access, filename, os.R_OK),
                 (os.chdir, filename),
                 (os.chmod, filename, 0o777),
+                (os.getcwdb,),
                 (os.link, filename, filename),
                 (os.listdir, filename),
                 (os.lstat, filename),
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -2410,6 +2410,9 @@
         if (wbuf2 != wbuf) free(wbuf2);
         return resobj;
     }
+
+    if (win32_warn_bytes_api())
+        return NULL;
 #endif
 
     Py_BEGIN_ALLOW_THREADS

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 00:47:00 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 17 Nov 2011 00:47:00 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_misused_of_=22PyUnicode?=
 =?utf8?q?Object=22_structure_name_in_unicodeobject=2Eh?=
Message-ID: <E1RQpC0-0001A7-Ie@dinsdale.python.org>

http://hg.python.org/cpython/rev/abe62c9d659d
changeset:   73591:abe62c9d659d
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 17 00:12:44 2011 +0100
summary:
  Fix misused of "PyUnicodeObject" structure name in unicodeobject.h

files:
  Include/unicodeobject.h |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -403,7 +403,7 @@
 
 /* --- Flexible String Representation Helper Macros (PEP 393) -------------- */
 
-/* Values for PyUnicodeObject.state: */
+/* Values for PyASCIIObject.state: */
 
 /* Interning state. */
 #define SSTATE_NOT_INTERNED 0
@@ -1564,7 +1564,7 @@
     );
 #endif
 
-/* Similar to PyUnicode_TransformDecimalToASCII(), but takes a PyUnicodeObject
+/* Similar to PyUnicode_TransformDecimalToASCII(), but takes a PyObject
    as argument instead of a raw buffer and length.  This function additionally
    transforms spaces to ASCII because this is what the callers in longobject,
    floatobject, and complexobject did anyways. */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 00:47:01 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 17 Nov 2011 00:47:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Catch_PyUnicode=5FAS=5FUNIC?=
	=?utf8?q?ODE=28=29_errors?=
Message-ID: <E1RQpC1-0001AH-D9@dinsdale.python.org>

http://hg.python.org/cpython/rev/a16d2a8a07cc
changeset:   73592:a16d2a8a07cc
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 17 00:45:54 2011 +0100
summary:
  Catch PyUnicode_AS_UNICODE() errors

files:
  Objects/unicodeobject.c |  33 +++++++++++++------
  Python/import.c         |  47 ++++++++++++++++++++--------
  2 files changed, 55 insertions(+), 25 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3051,9 +3051,13 @@
 PyUnicode_EncodeFSDefault(PyObject *unicode)
 {
 #ifdef HAVE_MBCS
-    return PyUnicode_EncodeMBCS(PyUnicode_AS_UNICODE(unicode),
-                                PyUnicode_GET_SIZE(unicode),
-                                NULL);
+    const Py_UNICODE *wstr;
+    Py_ssize_t wlen;
+
+    wstr = PyUnicode_AsUnicodeAndSize(unicode, &wlen);
+    if (wstr == NULL)
+        return NULL;
+    return PyUnicode_EncodeMBCS(wstr, wlen, NULL);
 #elif defined(__APPLE__)
     return _PyUnicode_AsUTF8String(unicode, "surrogateescape");
 #else
@@ -3137,10 +3141,15 @@
                  (strcmp(lower, "iso-8859-1") == 0))
             return _PyUnicode_AsLatin1String(unicode, errors);
 #ifdef HAVE_MBCS
-        else if (strcmp(lower, "mbcs") == 0)
-            return PyUnicode_EncodeMBCS(PyUnicode_AS_UNICODE(unicode),
-                                        PyUnicode_GET_SIZE(unicode),
-                                        errors);
+        else if (strcmp(lower, "mbcs") == 0) {
+            const Py_UNICODE *wstr;
+            Py_ssize_t wlen;
+
+            wstr = PyUnicode_AsUnicodeAndSize(unicode, &wlen);
+            if (wstr == NULL)
+                return NULL;
+            return PyUnicode_EncodeMBCS(wstr, wlen, errors);
+        }
 #endif
         else if (strcmp(lower, "ascii") == 0)
             return _PyUnicode_AsASCIIString(unicode, errors);
@@ -5148,10 +5157,12 @@
 PyObject *
 PyUnicode_AsUTF32String(PyObject *unicode)
 {
-    return PyUnicode_EncodeUTF32(PyUnicode_AS_UNICODE(unicode),
-                                 PyUnicode_GET_SIZE(unicode),
-                                 NULL,
-                                 0);
+    const Py_UNICODE *wstr;
+    Py_ssize_t wlen;
+    wstr = PyUnicode_AsUnicodeAndSize(unicode, &wlen);
+    if (wstr == NULL)
+        return NULL;
+    return PyUnicode_EncodeUTF32(wstr, wlen, NULL, 0);
 }
 
 /* --- UTF-16 Codec ------------------------------------------------------- */
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -1198,6 +1198,7 @@
     PyObject *cpathname_tmp;
 #ifdef MS_WINDOWS   /* since Windows uses different permissions  */
     mode_t mode = srcstat->st_mode & ~S_IEXEC;
+    wchar_t *wdirname, *wpathname, *wpathname_tmp;
 #else
     mode_t dirmode = (srcstat->st_mode |
                       S_IXUSR | S_IXGRP | S_IXOTH |
@@ -1230,7 +1231,12 @@
     }
 
 #ifdef MS_WINDOWS
-    res = CreateDirectoryW(PyUnicode_AS_UNICODE(dirname), NULL);
+    wdirname = PyUnicode_AsUnicode(dirname);
+    if (wdirname == NULL) {
+        PyErr_Clear();
+        return;
+    }
+    res = CreateDirectoryW(wdirname, NULL);
     ok = (res != 0);
     if (!ok && GetLastError() == ERROR_ALREADY_EXISTS)
         ok = 1;
@@ -1268,8 +1274,19 @@
         return;
     }
 #ifdef MS_WINDOWS
-    (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
-    fd = _wopen(PyUnicode_AS_UNICODE(cpathname_tmp),
+    wpathname = PyUnicode_AsUnicode(cpathname);
+    if (wpathname == NULL) {
+        PyErr_Clear();
+        return;
+    }
+    wpathname_tmp = PyUnicode_AsUnicode(cpathname_tmp);
+    if (wpathname_tmp == NULL) {
+        PyErr_Clear();
+        return;
+    }
+
+    (void)DeleteFileW(wpathname_tmp);
+    fd = _wopen(wpathname_tmp,
                 O_EXCL | O_CREAT | O_WRONLY | O_BINARY,
                 mode);
     if (0 <= fd)
@@ -1322,7 +1339,7 @@
         /* Don't keep partial file */
         fclose(fp);
 #ifdef MS_WINDOWS
-        (void)DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
+        (void)DeleteFileW(wpathname_tmp);
         Py_DECREF(cpathname_tmp);
 #else
         (void) unlink(PyBytes_AS_STRING(cpathbytes_tmp));
@@ -1334,13 +1351,11 @@
     fclose(fp);
     /* Do a (hopefully) atomic rename */
 #ifdef MS_WINDOWS
-    if (!MoveFileExW(PyUnicode_AS_UNICODE(cpathname_tmp),
-                     PyUnicode_AS_UNICODE(cpathname),
-                     MOVEFILE_REPLACE_EXISTING)) {
+    if (!MoveFileExW(wpathname_tmp, wpathname, MOVEFILE_REPLACE_EXISTING)) {
         if (Py_VerboseFlag)
             PySys_FormatStderr("# can't write %R\n", cpathname);
         /* Don't keep tmp file */
-        (void) DeleteFileW(PyUnicode_AS_UNICODE(cpathname_tmp));
+        (void) DeleteFileW(wpathname_tmp);
         Py_DECREF(cpathname_tmp);
         return;
     }
@@ -1819,10 +1834,10 @@
     PyUnicode_CopyCharacters(filename, 0, path_unicode, 0, len);
     pos = len;
     if (addsep)
-        PyUnicode_WRITE(PyUnicode_KIND(filename), 
+        PyUnicode_WRITE(PyUnicode_KIND(filename),
                         PyUnicode_DATA(filename),
                         pos++, SEP);
-    PyUnicode_CopyCharacters(filename, pos, name, 0, 
+    PyUnicode_CopyCharacters(filename, pos, name, 0,
                              PyUnicode_GET_LENGTH(name));
 
     /* Check for package import (buf holds a directory name,
@@ -2268,18 +2283,22 @@
     WIN32_FIND_DATAW data;
     HANDLE h;
     int cmp;
-    wchar_t *wname;
+    wchar_t *wfilename, *wname;
     Py_ssize_t wname_len;
 
     if (Py_GETENV("PYTHONCASEOK") != NULL)
         return 1;
 
-    h = FindFirstFileW(PyUnicode_AS_UNICODE(filename), &data);
+    wfilename = PyUnicode_AsUnicode(filename);
+    if (wfilename == NULL)
+        return -1;
+
+    h = FindFirstFileW(wfilename, &data);
     if (h == INVALID_HANDLE_VALUE) {
         PyErr_Format(PyExc_NameError,
           "Can't find file for module %R\n(filename %R)",
           name, filename);
-        return 0;
+        return -1;
     }
     FindClose(h);
 
@@ -2831,7 +2850,7 @@
 
     if (!ensure_fromlist(tail, fromlist, prefix, 0))
         goto out;
-    
+
     result = tail;
     Py_INCREF(result);
   out:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 00:53:09 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 17 Nov 2011 00:53:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Catch_PyUnicode=5FAS=5FUNIC?=
 =?utf8?q?ODE=28=29_errors_in_fileutils=2Ec?=
Message-ID: <E1RQpHx-0001Xf-DS@dinsdale.python.org>

http://hg.python.org/cpython/rev/94a38ad4aebf
changeset:   73593:94a38ad4aebf
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 17 00:51:38 2011 +0100
summary:
  Catch PyUnicode_AS_UNICODE() errors in fileutils.c

files:
  Python/fileutils.c |  13 +++++++++++--
  1 files changed, 11 insertions(+), 2 deletions(-)


diff --git a/Python/fileutils.c b/Python/fileutils.c
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -244,8 +244,12 @@
 #ifdef MS_WINDOWS
     int err;
     struct _stat wstatbuf;
+    wchar_t *wpath;
 
-    err = _wstat(PyUnicode_AS_UNICODE(path), &wstatbuf);
+    wpath = PyUnicode_AsUnicode(path);
+    if (wpath == NULL)
+        return -1;
+    err = _wstat(wpath, &wstatbuf);
     if (!err)
         statbuf->st_mode = wstatbuf.st_mode;
     return err;
@@ -297,14 +301,19 @@
 _Py_fopen(PyObject *path, const char *mode)
 {
 #ifdef MS_WINDOWS
+    wchar_t *wpath;
     wchar_t wmode[10];
     int usize;
 
+    wpath = PyUnicode_AsUnicode(path);
+    if (wpath == NULL)
+        return NULL;
+
     usize = MultiByteToWideChar(CP_ACP, 0, mode, -1, wmode, sizeof(wmode));
     if (usize == 0)
         return NULL;
 
-    return _wfopen(PyUnicode_AS_UNICODE(path), wmode);
+    return _wfopen(wpath, wmode);
 #else
     FILE *f;
     PyObject *bytes = PyUnicode_EncodeFSDefault(path);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 01:09:38 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 17 Nov 2011 01:09:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_socket=5Fgethostname=28=29_?=
 =?utf8?q?uses_a_wchar=5Ft*_with_PyMem=5FMalloc=28=29_to_avoid_the?=
Message-ID: <E1RQpXu-0002jA-Du@dinsdale.python.org>

http://hg.python.org/cpython/rev/488a2e41e89d
changeset:   73594:488a2e41e89d
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 17 01:11:36 2011 +0100
summary:
  socket_gethostname() uses a wchar_t* with PyMem_Malloc() to avoid the
old Unicode API.

files:
  Modules/socketmodule.c |  44 ++++++++++++++++++-----------
  1 files changed, 27 insertions(+), 17 deletions(-)


diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -3896,24 +3896,34 @@
        Otherwise, gethostname apparently also returns the DNS name. */
     wchar_t buf[MAX_COMPUTERNAME_LENGTH + 1];
     DWORD size = Py_ARRAY_LENGTH(buf);
+    wchar_t *name;
     PyObject *result;
-    if (!GetComputerNameExW(ComputerNamePhysicalDnsHostname, buf, &size)) {
-        if (GetLastError() == ERROR_MORE_DATA) {
-            /* MSDN says this may occur "because DNS allows longer names */
-            if (size == 0) /* XXX: I'm not sure how to handle this */
-                return PyUnicode_FromUnicode(NULL, 0);
-            result = PyUnicode_FromUnicode(NULL, size - 1);
-            if (!result)
-                return NULL;
-            if (GetComputerNameExW(ComputerNamePhysicalDnsHostname,
-                                   PyUnicode_AS_UNICODE(result),
-                                   &size))
-                return result;
-            Py_DECREF(result);
-        }
-        return PyErr_SetExcFromWindowsErr(PyExc_WindowsError, GetLastError());
-    }
-    return PyUnicode_FromUnicode(buf, size);
+
+    if (GetComputerNameExW(ComputerNamePhysicalDnsHostname, buf, &size))
+        return PyUnicode_FromUnicode(buf, size);
+
+    if (GetLastError() != ERROR_MORE_DATA)
+        return PyErr_SetFromWindowsErr(0);
+
+    if (size == 0)
+        return PyUnicode_New(0, 0);
+
+    /* MSDN says ERROR_MORE_DATA may occur because DNS allows longer
+       names */
+    name = PyMem_Malloc(size * sizeof(wchar_t));
+    if (!name)
+        return NULL;
+    if (!GetComputerNameExW(ComputerNamePhysicalDnsHostname,
+                           name,
+                           &size))
+    {
+        PyMem_Free(name);
+        return PyErr_SetFromWindowsErr(0);
+    }
+
+    result = PyUnicode_FromWideChar(name, size);
+    PyMem_Free(name);
+    return result;
 #else
     char buf[1024];
     int res;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 01:52:50 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 17 Nov 2011 01:52:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Be_a_bit_less_shy?=
Message-ID: <E1RQqDi-0005ou-1Y@dinsdale.python.org>

http://hg.python.org/cpython/rev/0db65a4cb906
changeset:   73595:0db65a4cb906
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 17 01:48:06 2011 +0100
summary:
  Be a bit less shy

files:
  Doc/whatsnew/3.3.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -102,7 +102,7 @@
   * indexing or slicing non-BMP characters returns the expected value,
     so ``'\U0010FFFF'[0]`` now returns ``'\U0010FFFF'`` and not ``'\uDBFF'``;
 
-  * several other functions in the standard library now handle correctly
+  * all other functions in the standard library now correctly handle
     non-BMP codepoints.
 
 * The value of :data:`sys.maxunicode` is now always ``1114111`` (``0x10FFFF``

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 02:05:02 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 17 Nov 2011 02:05:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Explain_concrete_=28resourc?=
 =?utf8?q?e_consumption=29_effects_of_PEP_393_a_bit=2E?=
Message-ID: <E1RQqPW-0006bW-0U@dinsdale.python.org>

http://hg.python.org/cpython/rev/bb28027fdba7
changeset:   73596:bb28027fdba7
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 17 01:59:51 2011 +0100
summary:
  Explain concrete (resource consumption) effects of PEP 393 a bit.

files:
  Doc/whatsnew/3.3.rst |  18 +++++++++++++-----
  1 files changed, 13 insertions(+), 5 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -84,11 +84,19 @@
 
   * non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint.
 
-.. The memory usage of Python 3.3 is two to three times smaller than Python 3.2,
-   and a little bit better than Python 2.7, on a `Django benchmark
-   <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
-   XXX The result should be moved in the PEP and a small summary about
-   performances and a link to the PEP should be added here.
+  The net effect is that for most applications, memory usage of string storage
+  should decrease significantly - especially compared to former wide unicode
+  builds - as, in many cases, strings will be pure ASCII even in international
+  contexts (because many strings store non-human language data, such as XML
+  fragments, HTTP headers, JSON-encoded data, etc.).  We also hope that it
+  will, for the same reasons, increase CPU cache efficiency on non-trivial
+  applications.
+
+   .. The memory usage of Python 3.3 is two to three times smaller than Python 3.2,
+      and a little bit better than Python 2.7, on a `Django benchmark
+      <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
+      XXX The result should be moved in the PEP and a link to the PEP should
+      be added here.
 
 * With the death of narrow builds, the problems specific to narrow builds have
   also been fixed, for example:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 02:05:02 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 17 Nov 2011 02:05:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_XXX=2C_as_the_functi?=
 =?utf8?q?ons_and_macros_are_mentioned_in_the_porting_section?=
Message-ID: <E1RQqPW-0006cc-OR@dinsdale.python.org>

http://hg.python.org/cpython/rev/622b98d3ea50
changeset:   73597:622b98d3ea50
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 17 02:00:19 2011 +0100
summary:
  Remove XXX, as the functions and macros are mentioned in the porting section

files:
  Doc/whatsnew/3.3.rst |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -120,8 +120,6 @@
 
 * The :file:`./configure` flag ``--with-wide-unicode`` has been removed.
 
-XXX mention new and deprecated functions and macros
-
 
 PEP 3151: Reworking the OS and IO exception hierarchy
 =====================================================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 02:13:56 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 17 Nov 2011 02:13:56 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_various_items_in_whatsn?=
	=?utf8?q?ew?=
Message-ID: <E1RQqY8-0007E1-Cs@dinsdale.python.org>

http://hg.python.org/cpython/rev/885f6f5ea607
changeset:   73598:885f6f5ea607
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 17 02:09:13 2011 +0100
summary:
  Add various items in whatsnew

files:
  Doc/whatsnew/3.3.rst |  21 ++++++++++++++++++++-
  1 files changed, 20 insertions(+), 1 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -510,12 +510,31 @@
 ssl
 ---
 
-The :mod:`ssl` module has new functions:
+* The :mod:`ssl` module has two new random generation functions:
 
   * :func:`~ssl.RAND_bytes`: generate cryptographically strong
     pseudo-random bytes.
   * :func:`~ssl.RAND_pseudo_bytes`: generate pseudo-random bytes.
 
+  (Contributed by Victor Stinner in :issue:`12049`)
+
+* The :mod:`ssl` module now exposes a finer-grained exception hierarchy
+  in order to make it easier to inspect the various kinds of errors.
+
+  (Contributed by Antoine Pitrou in :issue:`11183`)
+
+* :meth:`~ssl.SSLContext.load_cert_chain` now accepts a *password* argument
+  to be used if the private key is encrypted.
+
+  (Contributed by Adam Simpkins in :issue:`12803`)
+
+* SSL sockets have a new :meth:`~ssl.SSLSocket.get_channel_binding` method
+  allowing the implementation of certain authentication mechanisms such as
+  SCRAM-SHA-1-PLUS.
+
+  (Contributed by Jacek Konieczny in :issue:`12551`)
+
+
 
 shutil
 ------

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 02:25:34 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Thu, 17 Nov 2011 02:25:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_a_couple_other_whatsnew?=
	=?utf8?q?_items?=
Message-ID: <E1RQqjO-0008AU-Fn@dinsdale.python.org>

http://hg.python.org/cpython/rev/f57eca95cc2b
changeset:   73599:f57eca95cc2b
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Thu Nov 17 02:20:48 2011 +0100
summary:
  Add a couple other whatsnew items

files:
  Doc/whatsnew/3.3.rst |  20 ++++++++++++++++++--
  1 files changed, 18 insertions(+), 2 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -307,6 +307,15 @@
 (Contributed by Giampaolo Rodol? in :issue:`12139`)
 
 
+imaplib
+-------
+
+The :class:`~imaplib.IMAP4_SSL` constructor now accepts an SSLContext
+parameter to control parameters of the secure channel.
+
+(Contributed by Sijin Joseph in :issue:`8808`)
+
+
 math
 ----
 
@@ -534,8 +543,6 @@
 
   (Contributed by Jacek Konieczny in :issue:`12551`)
 
-
-
 shutil
 ------
 
@@ -547,6 +554,15 @@
     path also specifying the user/group names and not only their numeric
     ids. (Contributed by Sandro Tosi in :issue:`12191`)
 
+smtplib
+-------
+
+The :class:`~smtplib.SMTP_SSL` constructor and the :meth:`~smtplib.SMTP.starttls`
+method now accept an SSLContext parameter to control parameters of the secure
+channel.
+
+(Contributed by Kasun Herath in :issue:`8809`)
+
 urllib
 ------
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Nov 17 05:36:52 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 17 Nov 2011 05:36:52 +0100
Subject: [Python-checkins] Daily reference leaks (f57eca95cc2b): sum=0
Message-ID: <E1RQtiW-00068g-21@ap.vmr.nerim.net>

results for f57eca95cc2b on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogNwVpDB', '-x']

From python-checkins at python.org  Thu Nov 17 11:23:42 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 17 Nov 2011 11:23:42 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=2313406=3A_fix_more_deprec?=
 =?utf8?q?ation_warnings_and_move_the_deprecation_of?=
Message-ID: <E1RQz8A-000896-2x@dinsdale.python.org>

http://hg.python.org/cpython/rev/fe2be7d35660
changeset:   73600:fe2be7d35660
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 17 12:23:34 2011 +0200
summary:
  #13406: fix more deprecation warnings and move the deprecation of unicode-internal earlier in the code.

files:
  Lib/test/test_codeccallbacks.py |  57 +++++++++++---------
  Lib/test/test_codecs.py         |  14 ++--
  Modules/_codecsmodule.c         |  10 +-
  3 files changed, 43 insertions(+), 38 deletions(-)


diff --git a/Lib/test/test_codeccallbacks.py b/Lib/test/test_codeccallbacks.py
--- a/Lib/test/test_codeccallbacks.py
+++ b/Lib/test/test_codeccallbacks.py
@@ -205,33 +205,37 @@
         self.assertRaises(TypeError, codecs.charmap_encode, sin, "replace", charmap)
 
     def test_decodeunicodeinternal(self):
-        self.assertRaises(
-            UnicodeDecodeError,
-            b"\x00\x00\x00\x00\x00".decode,
-            "unicode-internal",
-        )
+        with test.support.check_warnings(('unicode_internal codec has been '
+                                          'deprecated', DeprecationWarning)):
+            self.assertRaises(
+                UnicodeDecodeError,
+                b"\x00\x00\x00\x00\x00".decode,
+                "unicode-internal",
+            )
         if SIZEOF_WCHAR_T == 4:
             def handler_unicodeinternal(exc):
                 if not isinstance(exc, UnicodeDecodeError):
                     raise TypeError("don't know how to handle %r" % exc)
                 return ("\x01", 1)
 
-            self.assertEqual(
-                b"\x00\x00\x00\x00\x00".decode("unicode-internal", "ignore"),
-                "\u0000"
-            )
+            with test.support.check_warnings(('unicode_internal codec has been '
+                                              'deprecated', DeprecationWarning)):
+                self.assertEqual(
+                    b"\x00\x00\x00\x00\x00".decode("unicode-internal", "ignore"),
+                    "\u0000"
+                )
 
-            self.assertEqual(
-                b"\x00\x00\x00\x00\x00".decode("unicode-internal", "replace"),
-                "\u0000\ufffd"
-            )
+                self.assertEqual(
+                    b"\x00\x00\x00\x00\x00".decode("unicode-internal", "replace"),
+                    "\u0000\ufffd"
+                )
 
-            codecs.register_error("test.hui", handler_unicodeinternal)
+                codecs.register_error("test.hui", handler_unicodeinternal)
 
-            self.assertEqual(
-                b"\x00\x00\x00\x00\x00".decode("unicode-internal", "test.hui"),
-                "\u0000\u0001\u0000"
-            )
+                self.assertEqual(
+                    b"\x00\x00\x00\x00\x00".decode("unicode-internal", "test.hui"),
+                    "\u0000\u0001\u0000"
+                )
 
     def test_callbacks(self):
         def handler1(exc):
@@ -626,9 +630,8 @@
                 ("utf-7", b"+x-"),
                 ("unicode-internal", b"\x00"),
             ):
-                with warnings.catch_warnings():
+                with test.support.check_warnings():
                     # unicode-internal has been deprecated
-                    warnings.simplefilter("ignore", DeprecationWarning)
                     self.assertRaises(
                         TypeError,
                         bytes.decode,
@@ -850,11 +853,12 @@
             else:
                 raise TypeError("don't know how to handle %r" % exc)
         codecs.register_error("test.replacing", replacing)
-        with warnings.catch_warnings():
+
+        with test.support.check_warnings():
             # unicode-internal has been deprecated
-            warnings.simplefilter("ignore", DeprecationWarning)
             for (encoding, data) in baddata:
-                self.assertRaises(TypeError, data.decode, encoding, "test.replacing")
+                with self.assertRaises(TypeError):
+                    data.decode(encoding, "test.replacing")
 
         def mutating(exc):
             if isinstance(exc, UnicodeDecodeError):
@@ -865,8 +869,11 @@
         codecs.register_error("test.mutating", mutating)
         # If the decoder doesn't pick up the modified input the following
         # will lead to an endless loop
-        for (encoding, data) in baddata:
-            self.assertRaises(TypeError, data.decode, encoding, "test.replacing")
+        with test.support.check_warnings():
+            # unicode-internal has been deprecated
+            for (encoding, data) in baddata:
+                with self.assertRaises(TypeError):
+                    data.decode(encoding, "test.replacing")
 
 def test_main():
     test.support.run_unittest(CodecCallbackTest)
diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -1062,10 +1062,8 @@
         self.assertEqual(("ab", 12), ignored)
 
     def test_encode_length(self):
-        with warnings.catch_warnings():
-            # unicode-internal has been deprecated
-            warnings.simplefilter("ignore", DeprecationWarning)
-
+        with support.check_warnings(('unicode_internal codec has been '
+                                     'deprecated', DeprecationWarning)):
             # Issue 3739
             encoder = codecs.getencoder("unicode_internal")
             self.assertEqual(encoder("a")[1], 1)
@@ -1528,10 +1526,8 @@
                 name = "latin_1"
             self.assertEqual(encoding.replace("_", "-"), name.replace("_", "-"))
 
-            with warnings.catch_warnings():
+            with support.check_warnings():
                 # unicode-internal has been deprecated
-                warnings.simplefilter("ignore", DeprecationWarning)
-
                 (b, size) = codecs.getencoder(encoding)(s)
                 self.assertEqual(size, len(s), "%r != %r (encoding=%r)" % (size, len(s), encoding))
                 (chars, size) = codecs.getdecoder(encoding)(b)
@@ -1639,7 +1635,9 @@
     def test_bad_encode_args(self):
         for encoding in all_unicode_encodings:
             encoder = codecs.getencoder(encoding)
-            self.assertRaises(TypeError, encoder)
+            with support.check_warnings():
+                # unicode-internal has been deprecated
+                self.assertRaises(TypeError, encoder)
 
     def test_encoding_map_type_initialized(self):
         from encodings import cp1140
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -677,6 +677,11 @@
     const char *data;
     Py_ssize_t len, size;
 
+    if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                     "unicode_internal codec has been deprecated",
+                     1))
+        return NULL;
+
     if (!PyArg_ParseTuple(args, "O|z:unicode_internal_encode",
                           &obj, &errors))
         return NULL;
@@ -687,11 +692,6 @@
         if (PyUnicode_READY(obj) < 0)
             return NULL;
 
-        if (PyErr_WarnEx(PyExc_DeprecationWarning,
-                         "unicode_internal codec has been deprecated",
-                         1))
-            return NULL;
-
         u = PyUnicode_AsUnicodeAndSize(obj, &len);
         if (u == NULL)
             return NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 17 17:45:43 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 17 Nov 2011 17:45:43 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_two_=22space_not_found=22_?=
	=?utf8?q?warnings_in_PEP-404=2E?=
Message-ID: <E1RR55r-0003Pf-S7@dinsdale.python.org>

http://hg.python.org/peps/rev/d6d7a35ff9f4
changeset:   3991:d6d7a35ff9f4
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 17 18:45:29 2011 +0200
summary:
  Fix two "space not found" warnings in PEP-404.

files:
  pep-0404.txt |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -93,7 +93,7 @@
 fundamental ambiguity of the core string type, coupled with Python 2's
 default behavior of supporting automatic coercion from 8-bit strings
 to unicode objects when the two are combined, often leads to
-`UnicodeError`s. Python 3's standard string type is Unicode based, and
+`UnicodeError`\ s. Python 3's standard string type is Unicode based, and
 Python 3 adds a dedicated bytes type, but critically, no automatic coercion
 between bytes and unicode strings is provided. The closest the language gets
 to implicit coercion are a few text-based APIs that assume a default
@@ -106,7 +106,7 @@
 This string/bytes clarity is often a source of difficulty in
 transitioning existing code to Python 3, because many third party
 libraries and applications are themselves ambiguous in this
-distinction.  Once migrated though, most `UnicodeError`s can be
+distinction.  Once migrated though, most `UnicodeError`\ s can be
 eliminated.
 
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Nov 17 17:58:18 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 17 Nov 2011 17:58:18 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_warnings_in_pep-395=2E?=
Message-ID: <E1RR5I2-0004hJ-1H@dinsdale.python.org>

http://hg.python.org/peps/rev/3e6f838fc878
changeset:   3992:3e6f838fc878
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 17 18:54:52 2011 +0200
summary:
  Fix warnings in pep-395.

files:
  pep-0395.txt |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/pep-0395.txt b/pep-0395.txt
--- a/pep-0395.txt
+++ b/pep-0395.txt
@@ -196,7 +196,7 @@
 
 The two current PEPs for namespace packages (PEP 382 and PEP 402) would both
 affect this part of the proposal. For PEP 382 (with its current suggestion of
-"*.pyp" package directories, this check would instead just walk up the
+"\*.pyp" package directories, this check would instead just walk up the
 supplied path, looking for the first non-package directory (this would not
 require any filesystem stat calls). Since PEP 402 deliberately omits explicit
 directory markers, it would need an alternative approach, based on checking
@@ -236,6 +236,7 @@
 ``__qname__`` attribute will be used to set ``__module__``.
 
 ``pydoc`` and ``inspect`` would also be updated appropriately to:
+
 - use ``__qname__`` instead of ``__name__`` and ``__qmodule__`` instead of
   ``__module__``where appropriate (e.g. ``inspect.getsource()`` would prefer
   the qualified variants)

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Nov 17 17:58:18 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 17 Nov 2011 17:58:18 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_a_warning_in_pep-403=2E?=
Message-ID: <E1RR5I2-0004hU-O7@dinsdale.python.org>

http://hg.python.org/peps/rev/136bc122bd3d
changeset:   3993:136bc122bd3d
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 17 18:58:12 2011 +0200
summary:
  Fix a warning in pep-403.

files:
  pep-0403.txt |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pep-0403.txt b/pep-0403.txt
--- a/pep-0403.txt
+++ b/pep-0403.txt
@@ -295,8 +295,8 @@
 References
 ==========
 
-[1] Start of python-ideas thread:
-    http://mail.python.org/pipermail/python-ideas/2011-October/012276.html
+.. [1] Start of python-ideas thread:
+   http://mail.python.org/pipermail/python-ideas/2011-October/012276.html
 
 
 Copyright

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Nov 17 18:03:11 2011
From: python-checkins at python.org (ezio.melotti)
Date: Thu, 17 Nov 2011 18:03:11 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Avoid_using_the_default_reST_r?=
	=?utf8?q?ole_in_PEP-404=2E?=
Message-ID: <E1RR5Ml-0005Co-AD@dinsdale.python.org>

http://hg.python.org/peps/rev/e04ddba93092
changeset:   3994:e04ddba93092
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Thu Nov 17 19:03:06 2011 +0200
summary:
  Avoid using the default reST role in PEP-404.

files:
  pep-0404.txt |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/pep-0404.txt b/pep-0404.txt
--- a/pep-0404.txt
+++ b/pep-0404.txt
@@ -93,7 +93,7 @@
 fundamental ambiguity of the core string type, coupled with Python 2's
 default behavior of supporting automatic coercion from 8-bit strings
 to unicode objects when the two are combined, often leads to
-`UnicodeError`\ s. Python 3's standard string type is Unicode based, and
+``UnicodeError``\ s. Python 3's standard string type is Unicode based, and
 Python 3 adds a dedicated bytes type, but critically, no automatic coercion
 between bytes and unicode strings is provided. The closest the language gets
 to implicit coercion are a few text-based APIs that assume a default
@@ -106,16 +106,16 @@
 This string/bytes clarity is often a source of difficulty in
 transitioning existing code to Python 3, because many third party
 libraries and applications are themselves ambiguous in this
-distinction.  Once migrated though, most `UnicodeError`\ s can be
+distinction.  Once migrated though, most ``UnicodeError``\ s can be
 eliminated.
 
 
 Numbers
 -------
 
-Python 2 has two basic integer types, a native machine-sized `int`
-type, and an arbitrary length `long` type.  These have been merged in
-Python 3 into a single `int` type analogous to Python 2's `long`
+Python 2 has two basic integer types, a native machine-sized ``int``
+type, and an arbitrary length ``long`` type.  These have been merged in
+Python 3 into a single ``int`` type analogous to Python 2's ``long``
 type.
 
 In addition, integer division now produces floating point numbers for
@@ -146,9 +146,9 @@
 ------------------
 
 There are many cases in Python 2 where multiple spellings of some
-constructs exist, such as `repr()` and *backticks*, or the two
-inequality operators `!=` and `<>`.  In all cases, Python 3 has chosen
-exactly one spelling and removed the other (e.g. `repr()` and `!=`
+constructs exist, such as ``repr()`` and *backticks*, or the two
+inequality operators ``!=`` and ``<>``.  In all cases, Python 3 has chosen
+exactly one spelling and removed the other (e.g. ``repr()`` and ``!=``
 were kept).
 
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Fri Nov 18 00:53:08 2011
From: python-checkins at python.org (jason.coombs)
Date: Fri, 18 Nov 2011 00:53:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_PDB_now_will_pr?=
 =?utf8?q?operly_escape_backslashes_in_the_names_of_modules_it_executes=2E?=
Message-ID: <E1RRBlU-0003e2-FH@dinsdale.python.org>

http://hg.python.org/cpython/rev/f7dd5178f36a
changeset:   73601:f7dd5178f36a
branch:      2.7
parent:      73585:2a3b60d28f6c
user:        Jason R. Coombs <jaraco at jaraco.com>
date:        Thu Nov 17 18:03:24 2011 -0500
summary:
  PDB now will properly escape backslashes in the names of modules it executes. Fixes #7750

files:
  Lib/pdb.py           |   2 +-
  Lib/test/test_pdb.py |  27 +++++++++++++++++++++++++++
  2 files changed, 28 insertions(+), 1 deletions(-)


diff --git a/Lib/pdb.py b/Lib/pdb.py
--- a/Lib/pdb.py
+++ b/Lib/pdb.py
@@ -1229,7 +1229,7 @@
         self._wait_for_mainpyfile = 1
         self.mainpyfile = self.canonic(filename)
         self._user_requested_quit = 0
-        statement = 'execfile( "%s")' % filename
+        statement = 'execfile(%r)' % filename
         self.run(statement)
 
 # Simplified interface
diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -3,6 +3,9 @@
 
 import imp
 import sys
+import os
+import unittest
+import subprocess
 
 from test import test_support
 # This little helper class is essential for testing pdb under doctest.
@@ -277,6 +280,29 @@
     4
     """
 
+class Tester7750(unittest.TestCase):
+    # if the filename has something that resolves to a python
+    #  escape character (such as \t), it will fail
+    test_fn = '.\\test7750.py'
+
+    msg = "issue7750 only applies when os.sep is a backslash"
+    @unittest.skipUnless(os.path.sep == '\\', msg)
+    def test_issue7750(self):
+        with open(self.test_fn, 'w') as f:
+            f.write('print("hello world")')
+        cmd = [sys.executable, '-m', 'pdb', self.test_fn,]
+        proc = subprocess.Popen(cmd,
+            stdout=subprocess.PIPE,
+            stdin=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            )
+        stdout, stderr = proc.communicate('quit\n')
+        self.assertNotIn('IOError', stdout, "pdb munged the filename")
+
+    def tearDown(self):
+        if os.path.isfile(self.test_fn):
+            os.remove(self.test_fn)
+
 
 def test_main():
     from test import test_pdb
@@ -285,3 +311,4 @@
 
 if __name__ == '__main__':
     test_main()
+    unittest.main()

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Fri Nov 18 05:37:59 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 18 Nov 2011 05:37:59 +0100
Subject: [Python-checkins] Daily reference leaks (fe2be7d35660): sum=0
Message-ID: <E1RRGD9-0006hK-0H@ap.vmr.nerim.net>

results for fe2be7d35660 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogRt5kJT', '-x']

From python-checkins at python.org  Fri Nov 18 12:52:53 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 12:52:53 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzEzNDI2OiBmaXgg?=
	=?utf8?q?typo_in_pickle_doc=2E?=
Message-ID: <E1RRN01-00015d-LR@dinsdale.python.org>

http://hg.python.org/cpython/rev/ce34e9223450
changeset:   73602:ce34e9223450
branch:      2.7
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 13:41:58 2011 +0200
summary:
  #13426: fix typo in pickle doc.

files:
  Doc/library/pickle.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst
--- a/Doc/library/pickle.rst
+++ b/Doc/library/pickle.rst
@@ -364,7 +364,7 @@
 
 Note that functions (built-in and user-defined) are pickled by "fully qualified"
 name reference, not by value.  This means that only the function name is
-pickled, along with the name of module the function is defined in.  Neither the
+pickled, along with the name of the module the function is defined in.  Neither the
 function's code, nor any of its function attributes are pickled.  Thus the
 defining module must be importable in the unpickling environment, and the module
 must contain the named object, otherwise an exception will be raised. [#]_

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 12:52:54 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 12:52:54 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEzNDI2OiBmaXgg?=
	=?utf8?q?typos_in_pickle_doc=2E?=
Message-ID: <E1RRN02-00015h-Fi@dinsdale.python.org>

http://hg.python.org/cpython/rev/1f31061afdaf
changeset:   73603:1f31061afdaf
branch:      3.2
parent:      73586:738c43484b47
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 13:51:10 2011 +0200
summary:
  #13426: fix typos in pickle doc.

files:
  Doc/library/pickle.rst |  10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst
--- a/Doc/library/pickle.rst
+++ b/Doc/library/pickle.rst
@@ -237,7 +237,7 @@
 
 .. exception:: UnpicklingError
 
-   Error raised when there a problem unpickling an object, such as a data
+   Error raised when there is a problem unpickling an object, such as a data
    corruption or a security violation.  It inherits :exc:`PickleError`.
 
    Note that other exceptions may also be raised during unpickling, including
@@ -324,11 +324,11 @@
 
    .. method:: persistent_load(pid)
 
-      Raise an :exc:`UnpickingError` by default.
+      Raise an :exc:`UnpicklingError` by default.
 
       If defined, :meth:`persistent_load` should return the object specified by
       the persistent ID *pid*.  If an invalid persistent ID is encountered, an
-      :exc:`UnpickingError` should be raised.
+      :exc:`UnpicklingError` should be raised.
 
       See :ref:`pickle-persistent` for details and examples of uses.
 
@@ -377,7 +377,7 @@
 
 Note that functions (built-in and user-defined) are pickled by "fully qualified"
 name reference, not by value.  This means that only the function name is
-pickled, along with the name of module the function is defined in.  Neither the
+pickled, along with the name of the module the function is defined in.  Neither the
 function's code, nor any of its function attributes are pickled.  Thus the
 defining module must be importable in the unpickling environment, and the module
 must contain the named object, otherwise an exception will be raised. [#]_
@@ -668,7 +668,7 @@
 For this reason, you may want to control what gets unpickled by customizing
 :meth:`Unpickler.find_class`.  Unlike its name suggests, :meth:`find_class` is
 called whenever a global (i.e., a class or a function) is requested.  Thus it is
-possible to either forbid completely globals or restrict them to a safe subset.
+possible to either completely forbid globals or restrict them to a safe subset.
 
 Here is an example of an unpickler allowing only few safe classes from the
 :mod:`builtins` module to be loaded::

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 12:52:55 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 12:52:55 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2313426=3A_merge_with_3=2E2=2E?=
Message-ID: <E1RRN03-00015l-Az@dinsdale.python.org>

http://hg.python.org/cpython/rev/7992f3247447
changeset:   73604:7992f3247447
parent:      73600:fe2be7d35660
parent:      73603:1f31061afdaf
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 13:52:37 2011 +0200
summary:
  #13426: merge with 3.2.

files:
  Doc/library/pickle.rst |  10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst
--- a/Doc/library/pickle.rst
+++ b/Doc/library/pickle.rst
@@ -237,7 +237,7 @@
 
 .. exception:: UnpicklingError
 
-   Error raised when there a problem unpickling an object, such as a data
+   Error raised when there is a problem unpickling an object, such as a data
    corruption or a security violation.  It inherits :exc:`PickleError`.
 
    Note that other exceptions may also be raised during unpickling, including
@@ -324,11 +324,11 @@
 
    .. method:: persistent_load(pid)
 
-      Raise an :exc:`UnpickingError` by default.
+      Raise an :exc:`UnpicklingError` by default.
 
       If defined, :meth:`persistent_load` should return the object specified by
       the persistent ID *pid*.  If an invalid persistent ID is encountered, an
-      :exc:`UnpickingError` should be raised.
+      :exc:`UnpicklingError` should be raised.
 
       See :ref:`pickle-persistent` for details and examples of uses.
 
@@ -377,7 +377,7 @@
 
 Note that functions (built-in and user-defined) are pickled by "fully qualified"
 name reference, not by value.  This means that only the function name is
-pickled, along with the name of module the function is defined in.  Neither the
+pickled, along with the name of the module the function is defined in.  Neither the
 function's code, nor any of its function attributes are pickled.  Thus the
 defining module must be importable in the unpickling environment, and the module
 must contain the named object, otherwise an exception will be raised. [#]_
@@ -668,7 +668,7 @@
 For this reason, you may want to control what gets unpickled by customizing
 :meth:`Unpickler.find_class`.  Unlike its name suggests, :meth:`find_class` is
 called whenever a global (i.e., a class or a function) is requested.  Thus it is
-possible to either forbid completely globals or restrict them to a safe subset.
+possible to either completely forbid globals or restrict them to a safe subset.
 
 Here is an example of an unpickler allowing only few safe classes from the
 :mod:`builtins` module to be loaded::

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 16:36:19 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 16:36:19 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzQxNDc6IG1pbmlk?=
 =?utf8?q?om=27s_toprettyxml_no_longer_adds_whitespace_around_a_text_node_?=
 =?utf8?q?when?=
Message-ID: <E1RRQUF-0005bG-DL@dinsdale.python.org>

http://hg.python.org/cpython/rev/7262f8f276ff
changeset:   73605:7262f8f276ff
branch:      2.7
parent:      73602:ce34e9223450
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 17:30:28 2011 +0200
summary:
  #4147: minidom's toprettyxml no longer adds whitespace around a text node when it is the only child of an element.  Initial patch by Dan Kenigsberg.

files:
  Lib/test/test_minidom.py |  37 ++++++++++++++++++++++++---
  Lib/xml/dom/minidom.py   |  14 ++++++---
  Misc/NEWS                |   4 +++
  3 files changed, 45 insertions(+), 10 deletions(-)


diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
--- a/Lib/test/test_minidom.py
+++ b/Lib/test/test_minidom.py
@@ -439,12 +439,39 @@
         dom.unlink()
         self.confirm(domstr == str.replace("\n", "\r\n"))
 
+    def test_toprettyxml_with_text_nodes(self):
+        # see issue #4147, text nodes are not indented
+        decl = '<?xml version="1.0" ?>\n'
+        self.assertEqual(parseString('<B>A</B>').toprettyxml(),
+                         decl + '<B>A</B>\n')
+        self.assertEqual(parseString('<C>A<B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\tA\n\t<B>A</B>\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B>A</C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\tA\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B><B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\t<B>A</B>\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B>A<B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\tA\n\t<B>A</B>\n</C>\n')
+
+    def test_toprettyxml_with_adjacent_text_nodes(self):
+        # see issue #4147, adjacent text nodes are indented normally
+        dom = Document()
+        elem = dom.createElement(u'elem')
+        elem.appendChild(dom.createTextNode(u'TEXT'))
+        elem.appendChild(dom.createTextNode(u'TEXT'))
+        dom.appendChild(elem)
+        decl = '<?xml version="1.0" ?>\n'
+        self.assertEqual(dom.toprettyxml(),
+                         decl + '<elem>\n\tTEXT\n\tTEXT\n</elem>\n')
+
     def test_toprettyxml_preserves_content_of_text_node(self):
-        str = '<A>B</A>'
-        dom = parseString(str)
-        dom2 = parseString(dom.toprettyxml())
-        self.assertEqual(dom.childNodes[0].childNodes[0].toxml(),
-                         dom2.childNodes[0].childNodes[0].toxml())
+        # see issue #4147
+        for str in ('<B>A</B>', '<A><B>C</B></A>'):
+            dom = parseString(str)
+            dom2 = parseString(dom.toprettyxml())
+            self.assertEqual(
+                dom.getElementsByTagName('B')[0].childNodes[0].toxml(),
+                dom2.getElementsByTagName('B')[0].childNodes[0].toxml())
 
     def testProcessingInstruction(self):
         dom = parseString('<e><?mypi \t\n data \t\n ?></e>')
diff --git a/Lib/xml/dom/minidom.py b/Lib/xml/dom/minidom.py
--- a/Lib/xml/dom/minidom.py
+++ b/Lib/xml/dom/minidom.py
@@ -807,11 +807,15 @@
             writer.write("\"")
         if self.childNodes:
             writer.write(">")
-            if self.childNodes[0].nodeType != Node.TEXT_NODE:
+            if (len(self.childNodes) == 1 and
+                self.childNodes[0].nodeType == Node.TEXT_NODE):
+                self.childNodes[0].writexml(writer, '', '', '')
+            else:
                 writer.write(newl)
-            for node in self.childNodes:
-                node.writexml(writer,indent+addindent,addindent,newl)
-            writer.write("%s</%s>%s" % (indent,self.tagName,newl))
+                for node in self.childNodes:
+                    node.writexml(writer, indent+addindent, addindent, newl)
+                writer.write(indent)
+            writer.write("</%s>%s" % (self.tagName, newl))
         else:
             writer.write("/>%s"%(newl))
 
@@ -1033,7 +1037,7 @@
         return newText
 
     def writexml(self, writer, indent="", addindent="", newl=""):
-        _write_data(writer, self.data)
+        _write_data(writer, "%s%s%s" % (indent, self.data, newl))
 
     # DOM Level 3 (WD 9 April 2002)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,10 @@
 Library
 -------
 
+- Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
+  node when it is the only child of an element.  Initial patch by Dan
+  Kenigsberg.
+
 - Issue #8793: Prevent IDLE crash when given strings with invalid hex escape
   sequences.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 16:36:20 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 16:36:20 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzQxNDc6IG1pbmlk?=
 =?utf8?q?om=27s_toprettyxml_no_longer_adds_whitespace_around_a_text_node_?=
 =?utf8?q?when?=
Message-ID: <E1RRQUG-0005bK-BO@dinsdale.python.org>

http://hg.python.org/cpython/rev/5401daa96a21
changeset:   73606:5401daa96a21
branch:      3.2
parent:      73603:1f31061afdaf
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 17:34:26 2011 +0200
summary:
  #4147: minidom's toprettyxml no longer adds whitespace around a text node when it is the only child of an element.  Initial patch by Dan Kenigsberg.

files:
  Lib/test/test_minidom.py |  37 ++++++++++++++++++++++++---
  Lib/xml/dom/minidom.py   |  14 ++++++---
  Misc/NEWS                |   4 +++
  3 files changed, 45 insertions(+), 10 deletions(-)


diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
--- a/Lib/test/test_minidom.py
+++ b/Lib/test/test_minidom.py
@@ -446,12 +446,39 @@
         dom.unlink()
         self.confirm(domstr == str.replace("\n", "\r\n"))
 
+    def test_toprettyxml_with_text_nodes(self):
+        # see issue #4147, text nodes are not indented
+        decl = '<?xml version="1.0" ?>\n'
+        self.assertEqual(parseString('<B>A</B>').toprettyxml(),
+                         decl + '<B>A</B>\n')
+        self.assertEqual(parseString('<C>A<B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\tA\n\t<B>A</B>\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B>A</C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\tA\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B><B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\t<B>A</B>\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B>A<B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\tA\n\t<B>A</B>\n</C>\n')
+
+    def test_toprettyxml_with_adjacent_text_nodes(self):
+        # see issue #4147, adjacent text nodes are indented normally
+        dom = Document()
+        elem = dom.createElement('elem')
+        elem.appendChild(dom.createTextNode('TEXT'))
+        elem.appendChild(dom.createTextNode('TEXT'))
+        dom.appendChild(elem)
+        decl = '<?xml version="1.0" ?>\n'
+        self.assertEqual(dom.toprettyxml(),
+                         decl + '<elem>\n\tTEXT\n\tTEXT\n</elem>\n')
+
     def test_toprettyxml_preserves_content_of_text_node(self):
-        str = '<A>B</A>'
-        dom = parseString(str)
-        dom2 = parseString(dom.toprettyxml())
-        self.assertEqual(dom.childNodes[0].childNodes[0].toxml(),
-                         dom2.childNodes[0].childNodes[0].toxml())
+        # see issue #4147
+        for str in ('<B>A</B>', '<A><B>C</B></A>'):
+            dom = parseString(str)
+            dom2 = parseString(dom.toprettyxml())
+            self.assertEqual(
+                dom.getElementsByTagName('B')[0].childNodes[0].toxml(),
+                dom2.getElementsByTagName('B')[0].childNodes[0].toxml())
 
     def testProcessingInstruction(self):
         dom = parseString('<e><?mypi \t\n data \t\n ?></e>')
diff --git a/Lib/xml/dom/minidom.py b/Lib/xml/dom/minidom.py
--- a/Lib/xml/dom/minidom.py
+++ b/Lib/xml/dom/minidom.py
@@ -837,11 +837,15 @@
             writer.write("\"")
         if self.childNodes:
             writer.write(">")
-            if self.childNodes[0].nodeType != Node.TEXT_NODE:
+            if (len(self.childNodes) == 1 and
+                self.childNodes[0].nodeType == Node.TEXT_NODE):
+                self.childNodes[0].writexml(writer, '', '', '')
+            else:
                 writer.write(newl)
-            for node in self.childNodes:
-                node.writexml(writer,indent+addindent,addindent,newl)
-            writer.write("%s</%s>%s" % (indent,self.tagName,newl))
+                for node in self.childNodes:
+                    node.writexml(writer, indent+addindent, addindent, newl)
+                writer.write(indent)
+            writer.write("</%s>%s" % (self.tagName, newl))
         else:
             writer.write("/>%s"%(newl))
 
@@ -1063,7 +1067,7 @@
         return newText
 
     def writexml(self, writer, indent="", addindent="", newl=""):
-        _write_data(writer, self.data)
+        _write_data(writer, "%s%s%s" % (indent, self.data, newl))
 
     # DOM Level 3 (WD 9 April 2002)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -76,6 +76,10 @@
 Library
 -------
 
+- Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
+  node when it is the only child of an element.  Initial patch by Dan
+  Kenigsberg.
+
 - Issues #1745761, #755670, #13357, #12629, #1200313: HTMLParser now correctly
   handles non-valid attributes, including adjacent and unquoted attributes.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 16:36:22 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 16:36:22 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiAjNDE0NzogbWVyZ2Ugd2l0aCAzLjIu?=
Message-ID: <E1RRQUI-0005cf-6t@dinsdale.python.org>

http://hg.python.org/cpython/rev/cb6614e3438b
changeset:   73607:cb6614e3438b
parent:      73604:7992f3247447
parent:      73606:5401daa96a21
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 17:36:07 2011 +0200
summary:
  #4147: merge with 3.2.

files:
  Lib/test/test_minidom.py |  37 ++++++++++++++++++++++++---
  Lib/xml/dom/minidom.py   |  14 ++++++---
  Misc/NEWS                |   4 +++
  3 files changed, 45 insertions(+), 10 deletions(-)


diff --git a/Lib/test/test_minidom.py b/Lib/test/test_minidom.py
--- a/Lib/test/test_minidom.py
+++ b/Lib/test/test_minidom.py
@@ -467,12 +467,39 @@
         dom.unlink()
         self.confirm(domstr == str.replace("\n", "\r\n"))
 
+    def test_toprettyxml_with_text_nodes(self):
+        # see issue #4147, text nodes are not indented
+        decl = '<?xml version="1.0" ?>\n'
+        self.assertEqual(parseString('<B>A</B>').toprettyxml(),
+                         decl + '<B>A</B>\n')
+        self.assertEqual(parseString('<C>A<B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\tA\n\t<B>A</B>\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B>A</C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\tA\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B><B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\t<B>A</B>\n</C>\n')
+        self.assertEqual(parseString('<C><B>A</B>A<B>A</B></C>').toprettyxml(),
+                         decl + '<C>\n\t<B>A</B>\n\tA\n\t<B>A</B>\n</C>\n')
+
+    def test_toprettyxml_with_adjacent_text_nodes(self):
+        # see issue #4147, adjacent text nodes are indented normally
+        dom = Document()
+        elem = dom.createElement('elem')
+        elem.appendChild(dom.createTextNode('TEXT'))
+        elem.appendChild(dom.createTextNode('TEXT'))
+        dom.appendChild(elem)
+        decl = '<?xml version="1.0" ?>\n'
+        self.assertEqual(dom.toprettyxml(),
+                         decl + '<elem>\n\tTEXT\n\tTEXT\n</elem>\n')
+
     def test_toprettyxml_preserves_content_of_text_node(self):
-        str = '<A>B</A>'
-        dom = parseString(str)
-        dom2 = parseString(dom.toprettyxml())
-        self.assertEqual(dom.childNodes[0].childNodes[0].toxml(),
-                         dom2.childNodes[0].childNodes[0].toxml())
+        # see issue #4147
+        for str in ('<B>A</B>', '<A><B>C</B></A>'):
+            dom = parseString(str)
+            dom2 = parseString(dom.toprettyxml())
+            self.assertEqual(
+                dom.getElementsByTagName('B')[0].childNodes[0].toxml(),
+                dom2.getElementsByTagName('B')[0].childNodes[0].toxml())
 
     def testProcessingInstruction(self):
         dom = parseString('<e><?mypi \t\n data \t\n ?></e>')
diff --git a/Lib/xml/dom/minidom.py b/Lib/xml/dom/minidom.py
--- a/Lib/xml/dom/minidom.py
+++ b/Lib/xml/dom/minidom.py
@@ -837,11 +837,15 @@
             writer.write("\"")
         if self.childNodes:
             writer.write(">")
-            if self.childNodes[0].nodeType != Node.TEXT_NODE:
+            if (len(self.childNodes) == 1 and
+                self.childNodes[0].nodeType == Node.TEXT_NODE):
+                self.childNodes[0].writexml(writer, '', '', '')
+            else:
                 writer.write(newl)
-            for node in self.childNodes:
-                node.writexml(writer,indent+addindent,addindent,newl)
-            writer.write("%s</%s>%s" % (indent,self.tagName,newl))
+                for node in self.childNodes:
+                    node.writexml(writer, indent+addindent, addindent, newl)
+                writer.write(indent)
+            writer.write("</%s>%s" % (self.tagName, newl))
         else:
             writer.write("/>%s"%(newl))
 
@@ -1063,7 +1067,7 @@
         return newText
 
     def writexml(self, writer, indent="", addindent="", newl=""):
-        _write_data(writer, self.data)
+        _write_data(writer, "%s%s%s" % (indent, self.data, newl))
 
     # DOM Level 3 (WD 9 April 2002)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -377,6 +377,10 @@
 Library
 -------
 
+- Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
+  node when it is the only child of an element.  Initial patch by Dan
+  Kenigsberg.
+
 - Issue #13374: The Windows bytes API has been deprecated in the os module. Use
   Unicode filenames instead of bytes filenames to not depend on the ANSI code
   page anymore and to support any filename.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 17:03:10 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 17:03:10 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzEzMzU4OiBIVE1M?=
 =?utf8?q?Parser_now_calls_handle=5Fdata_only_once_for_each_CDATA=2E?=
Message-ID: <E1RRQuE-0008Vd-OW@dinsdale.python.org>

http://hg.python.org/cpython/rev/91163aa3d5b4
changeset:   73608:91163aa3d5b4
branch:      2.7
parent:      73605:7262f8f276ff
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:00:40 2011 +0200
summary:
  #13358: HTMLParser now calls handle_data only once for each CDATA.

files:
  Lib/HTMLParser.py           |   7 ++++---
  Lib/test/test_htmlparser.py |  21 +++++++++++++++++++++
  Misc/NEWS                   |   2 ++
  3 files changed, 27 insertions(+), 3 deletions(-)


diff --git a/Lib/HTMLParser.py b/Lib/HTMLParser.py
--- a/Lib/HTMLParser.py
+++ b/Lib/HTMLParser.py
@@ -14,7 +14,6 @@
 # Regular expressions used for parsing
 
 interesting_normal = re.compile('[&<]')
-interesting_cdata = re.compile(r'<(/|\Z)')
 incomplete = re.compile('&[a-zA-Z#]')
 
 entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]')
@@ -125,8 +124,8 @@
         return self.__starttag_text
 
     def set_cdata_mode(self, elem):
-        self.interesting = interesting_cdata
         self.cdata_elem = elem.lower()
+        self.interesting = re.compile(r'</\s*%s\s*>' % self.cdata_elem, re.I)
 
     def clear_cdata_mode(self):
         self.interesting = interesting_normal
@@ -144,6 +143,8 @@
             if match:
                 j = match.start()
             else:
+                if self.cdata_elem:
+                    break
                 j = n
             if i < j: self.handle_data(rawdata[i:j])
             i = self.updatepos(i, j)
@@ -212,7 +213,7 @@
             else:
                 assert 0, "interesting.search() lied"
         # end while
-        if end and i < n:
+        if end and i < n and not self.cdata_elem:
             self.handle_data(rawdata[i:n])
             i = self.updatepos(i, n)
         self.rawdata = rawdata[i:]
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -286,6 +286,27 @@
                                     ("data", content),
                                     ("endtag", element_lower)])
 
+    def test_cdata_with_closing_tags(self):
+        # see issue #13358
+        # make sure that HTMLParser calls handle_data only once for each CDATA.
+        # The normal event collector normalizes the events in get_events,
+        # so we override it to return the original list of events.
+        class Collector(EventCollector):
+            def get_events(self):
+                return self.events
+
+        content = """<!-- not a comment --> &not-an-entity-ref;
+                  <a href="" /> </p><p> &amp; <span></span></style>
+                  '</script' + '>' </html> </head> </scripter>!"""
+        for element in [' script', 'script ', ' script ',
+                        '\nscript', 'script\n', '\nscript\n']:
+            s = u'<script>{content}</{element}>'.format(element=element,
+                                                        content=content)
+            self._run_check(s, [("starttag", "script", []),
+                                ("data", content),
+                                ("endtag", "script")],
+                            collector=Collector)
+
     def test_malformatted_charref(self):
         self._run_check("<p>&#bad;</p>", [
             ("starttag", "p", []),
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,8 @@
 Library
 -------
 
+- Issue #13358: HTMLParser now calls handle_data only once for each CDATA.
+
 - Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
   node when it is the only child of an element.  Initial patch by Dan
   Kenigsberg.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 17:03:11 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 17:03:11 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEzMzU4OiBIVE1M?=
 =?utf8?q?Parser_now_calls_handle=5Fdata_only_once_for_each_CDATA=2E?=
Message-ID: <E1RRQuF-0008Vh-IZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/0a32e7e3aa1f
changeset:   73609:0a32e7e3aa1f
branch:      3.2
parent:      73606:5401daa96a21
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:01:49 2011 +0200
summary:
  #13358: HTMLParser now calls handle_data only once for each CDATA.

files:
  Lib/html/parser.py          |   7 ++++---
  Lib/test/test_htmlparser.py |  20 ++++++++++++++++++++
  Misc/NEWS                   |   2 ++
  3 files changed, 26 insertions(+), 3 deletions(-)


diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -14,7 +14,6 @@
 # Regular expressions used for parsing
 
 interesting_normal = re.compile('[&<]')
-interesting_cdata = re.compile(r'<(/|\Z)')
 incomplete = re.compile('&[a-zA-Z#]')
 
 entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]')
@@ -149,8 +148,8 @@
         return self.__starttag_text
 
     def set_cdata_mode(self, elem):
-        self.interesting = interesting_cdata
         self.cdata_elem = elem.lower()
+        self.interesting = re.compile(r'</\s*%s\s*>' % self.cdata_elem, re.I)
 
     def clear_cdata_mode(self):
         self.interesting = interesting_normal
@@ -168,6 +167,8 @@
             if match:
                 j = match.start()
             else:
+                if self.cdata_elem:
+                    break
                 j = n
             if i < j: self.handle_data(rawdata[i:j])
             i = self.updatepos(i, j)
@@ -250,7 +251,7 @@
             else:
                 assert 0, "interesting.search() lied"
         # end while
-        if end and i < n:
+        if end and i < n and not self.cdata_elem:
             self.handle_data(rawdata[i:n])
             i = self.updatepos(i, n)
         self.rawdata = rawdata[i:]
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -301,7 +301,27 @@
                                     ("data", content),
                                     ("endtag", element_lower)])
 
+    def test_cdata_with_closing_tags(self):
+        # see issue #13358
+        # make sure that HTMLParser calls handle_data only once for each CDATA.
+        # The normal event collector normalizes  the events in get_events,
+        # so we override it to return the original list of events.
+        class Collector(EventCollector):
+            def get_events(self):
+                return self.events
 
+        content = """<!-- not a comment --> &not-an-entity-ref;
+                  <a href="" /> </p><p> <span></span></style>
+                  '</script' + '>'"""
+        for element in [' script', 'script ', ' script ',
+                        '\nscript', 'script\n', '\nscript\n']:
+            element_lower = element.lower().strip()
+            s = '<script>{content}</{element}>'.format(element=element,
+                                                       content=content)
+            self._run_check(s, [("starttag", element_lower, []),
+                                ("data", content),
+                                ("endtag", element_lower)],
+                            collector=Collector())
 
 class HTMLParserTolerantTestCase(HTMLParserStrictTestCase):
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -76,6 +76,8 @@
 Library
 -------
 
+- Issue #13358: HTMLParser now calls handle_data only once for each CDATA.
+
 - Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
   node when it is the only child of an element.  Initial patch by Dan
   Kenigsberg.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 17:03:12 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 17:03:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2313358=3A_merge_with_3=2E2=2E?=
Message-ID: <E1RRQuG-0008Vl-CF@dinsdale.python.org>

http://hg.python.org/cpython/rev/e12d2b9c88ef
changeset:   73610:e12d2b9c88ef
parent:      73607:cb6614e3438b
parent:      73609:0a32e7e3aa1f
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:02:59 2011 +0200
summary:
  #13358: merge with 3.2.

files:
  Lib/html/parser.py          |   7 ++++---
  Lib/test/test_htmlparser.py |  20 ++++++++++++++++++++
  Misc/NEWS                   |   2 ++
  3 files changed, 26 insertions(+), 3 deletions(-)


diff --git a/Lib/html/parser.py b/Lib/html/parser.py
--- a/Lib/html/parser.py
+++ b/Lib/html/parser.py
@@ -14,7 +14,6 @@
 # Regular expressions used for parsing
 
 interesting_normal = re.compile('[&<]')
-interesting_cdata = re.compile(r'<(/|\Z)')
 incomplete = re.compile('&[a-zA-Z#]')
 
 entityref = re.compile('&([a-zA-Z][-.a-zA-Z0-9]*)[^a-zA-Z0-9]')
@@ -149,8 +148,8 @@
         return self.__starttag_text
 
     def set_cdata_mode(self, elem):
-        self.interesting = interesting_cdata
         self.cdata_elem = elem.lower()
+        self.interesting = re.compile(r'</\s*%s\s*>' % self.cdata_elem, re.I)
 
     def clear_cdata_mode(self):
         self.interesting = interesting_normal
@@ -168,6 +167,8 @@
             if match:
                 j = match.start()
             else:
+                if self.cdata_elem:
+                    break
                 j = n
             if i < j: self.handle_data(rawdata[i:j])
             i = self.updatepos(i, j)
@@ -250,7 +251,7 @@
             else:
                 assert 0, "interesting.search() lied"
         # end while
-        if end and i < n:
+        if end and i < n and not self.cdata_elem:
             self.handle_data(rawdata[i:n])
             i = self.updatepos(i, n)
         self.rawdata = rawdata[i:]
diff --git a/Lib/test/test_htmlparser.py b/Lib/test/test_htmlparser.py
--- a/Lib/test/test_htmlparser.py
+++ b/Lib/test/test_htmlparser.py
@@ -301,7 +301,27 @@
                                     ("data", content),
                                     ("endtag", element_lower)])
 
+    def test_cdata_with_closing_tags(self):
+        # see issue #13358
+        # make sure that HTMLParser calls handle_data only once for each CDATA.
+        # The normal event collector normalizes  the events in get_events,
+        # so we override it to return the original list of events.
+        class Collector(EventCollector):
+            def get_events(self):
+                return self.events
 
+        content = """<!-- not a comment --> &not-an-entity-ref;
+                  <a href="" /> </p><p> <span></span></style>
+                  '</script' + '>'"""
+        for element in [' script', 'script ', ' script ',
+                        '\nscript', 'script\n', '\nscript\n']:
+            element_lower = element.lower().strip()
+            s = '<script>{content}</{element}>'.format(element=element,
+                                                       content=content)
+            self._run_check(s, [("starttag", element_lower, []),
+                                ("data", content),
+                                ("endtag", element_lower)],
+                            collector=Collector())
 
 class HTMLParserTolerantTestCase(HTMLParserStrictTestCase):
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -377,6 +377,8 @@
 Library
 -------
 
+- Issue #13358: HTMLParser now calls handle_data only once for each CDATA.
+
 - Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
   node when it is the only child of an element.  Initial patch by Dan
   Kenigsberg.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 17:36:05 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 17:36:05 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzExMTEyOiBGaXgg?=
 =?utf8?q?typo_in_a_base_class_in_test=5Fsocket=2E?=
Message-ID: <E1RRRQ5-0003Z1-WE@dinsdale.python.org>

http://hg.python.org/cpython/rev/0fdf7f7c353d
changeset:   73611:0fdf7f7c353d
branch:      2.7
parent:      73608:91163aa3d5b4
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:33:14 2011 +0200
summary:
  #11112: Fix typo in a base class in test_socket.

files:
  Lib/test/test_socket.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1379,7 +1379,7 @@
             # no alarm can be pending.  Safe to restore old handler.
             signal.signal(signal.SIGALRM, old_alarm)
 
-class UDPTimeoutTest(SocketTCPTest):
+class UDPTimeoutTest(SocketUDPTest):
 
     def testUDPTimeout(self):
         def raise_timeout(*args, **kwargs):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 17:36:06 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 17:36:06 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzExMTEyOiBGaXgg?=
 =?utf8?q?typo_in_a_base_class_in_test=5Fsocket=2E?=
Message-ID: <E1RRRQ6-0003Z5-OP@dinsdale.python.org>

http://hg.python.org/cpython/rev/b410bcd300a1
changeset:   73612:b410bcd300a1
branch:      3.2
parent:      73609:0a32e7e3aa1f
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:33:14 2011 +0200
summary:
  #11112: Fix typo in a base class in test_socket.

files:
  Lib/test/test_socket.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -1722,7 +1722,7 @@
             # no alarm can be pending.  Safe to restore old handler.
             signal.signal(signal.SIGALRM, old_alarm)
 
-class UDPTimeoutTest(SocketTCPTest):
+class UDPTimeoutTest(SocketUDPTest):
 
     def testUDPTimeout(self):
         def raise_timeout(*args, **kwargs):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 17:36:07 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 17:36:07 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2311112=3A_merge_with_3=2E2=2E?=
Message-ID: <E1RRRQ7-0003Z9-IP@dinsdale.python.org>

http://hg.python.org/cpython/rev/116ed0ce5756
changeset:   73613:116ed0ce5756
parent:      73610:e12d2b9c88ef
parent:      73612:b410bcd300a1
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:35:53 2011 +0200
summary:
  #11112: merge with 3.2.

files:
  Lib/test/test_socket.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -4184,7 +4184,7 @@
             # no alarm can be pending.  Safe to restore old handler.
             signal.signal(signal.SIGALRM, old_alarm)
 
-class UDPTimeoutTest(SocketTCPTest):
+class UDPTimeoutTest(SocketUDPTest):
 
     def testUDPTimeout(self):
         def raise_timeout(*args, **kwargs):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 18:01:09 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 18:01:09 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogIzEzMzg3OiBhZGQg?=
 =?utf8?q?note_about_checking_the_exact_type_in_assertIsInstance_doc=2E?=
Message-ID: <E1RRRoL-0005vh-Rc@dinsdale.python.org>

http://hg.python.org/cpython/rev/fd9d7a8e45bc
changeset:   73614:fd9d7a8e45bc
branch:      2.7
parent:      73611:0fdf7f7c353d
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:59:36 2011 +0200
summary:
  #13387: add note about checking the exact type in assertIsInstance doc.

files:
  Doc/library/unittest.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -895,6 +895,8 @@
 
       Test that *obj* is (or is not) an instance of *cls* (which can be a
       class or a tuple of classes, as supported by :func:`isinstance`).
+      To check for a specific type (without including superclasses) use
+      :func:`assertIs(type(obj), cls) <assertIs>`.
 
       .. versionadded:: 2.7
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 18:01:10 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 18:01:10 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogIzEzMzg3OiBhZGQg?=
 =?utf8?q?note_about_checking_the_exact_type_in_assertIsInstance_doc=2E?=
Message-ID: <E1RRRoM-0005vm-Ms@dinsdale.python.org>

http://hg.python.org/cpython/rev/583aff635ce1
changeset:   73615:583aff635ce1
branch:      3.2
parent:      73612:b410bcd300a1
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 18:59:36 2011 +0200
summary:
  #13387: add note about checking the exact type in assertIsInstance doc.

files:
  Doc/library/unittest.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -929,6 +929,8 @@
 
       Test that *obj* is (or is not) an instance of *cls* (which can be a
       class or a tuple of classes, as supported by :func:`isinstance`).
+      To check for a specific type (without including superclasses) use
+      :func:`assertIs(type(obj), cls) <assertIs>`.
 
       .. versionadded:: 3.2
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 18:01:11 2011
From: python-checkins at python.org (ezio.melotti)
Date: Fri, 18 Nov 2011 18:01:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=2313387=3A_merge_with_3=2E2=2E?=
Message-ID: <E1RRRoN-0005vq-I4@dinsdale.python.org>

http://hg.python.org/cpython/rev/196d485ed26d
changeset:   73616:196d485ed26d
parent:      73613:116ed0ce5756
parent:      73615:583aff635ce1
user:        Ezio Melotti <ezio.melotti at gmail.com>
date:        Fri Nov 18 19:01:01 2011 +0200
summary:
  #13387: merge with 3.2.

files:
  Doc/library/unittest.rst |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -933,6 +933,8 @@
 
       Test that *obj* is (or is not) an instance of *cls* (which can be a
       class or a tuple of classes, as supported by :func:`isinstance`).
+      To check for a specific type (without including superclasses) use
+      :func:`assertIs(type(obj), cls) <assertIs>`.
 
       .. versionadded:: 3.2
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 20:22:49 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 18 Nov 2011 20:22:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2310227=3A_Add_an_al?=
 =?utf8?q?location_cache_for_a_single_slice_object=2E?=
Message-ID: <E1RRU1R-0002B1-As@dinsdale.python.org>

http://hg.python.org/cpython/rev/fa2f8dd077e0
changeset:   73617:fa2f8dd077e0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 18 20:14:34 2011 +0100
summary:
  Issue #10227: Add an allocation cache for a single slice object.
Patch by Stefan Behnel.

files:
  Include/pythonrun.h   |   1 +
  Misc/NEWS             |   3 ++
  Objects/sliceobject.c |  36 ++++++++++++++++++++++++------
  Python/pythonrun.c    |   1 +
  4 files changed, 34 insertions(+), 7 deletions(-)


diff --git a/Include/pythonrun.h b/Include/pythonrun.h
--- a/Include/pythonrun.h
+++ b/Include/pythonrun.h
@@ -211,6 +211,7 @@
 PyAPI_FUNC(void) PyFloat_Fini(void);
 PyAPI_FUNC(void) PyOS_FiniInterrupts(void);
 PyAPI_FUNC(void) _PyGC_Fini(void);
+PyAPI_FUNC(void) PySlice_Fini(void);
 
 PyAPI_DATA(PyThreadState *) _Py_Finalizing;
 #endif
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #10227: Add an allocation cache for a single slice object.  Patch by
+  Stefan Behnel.
+
 - Issue #13393: BufferedReader.read1() now asks the full requested size to
   the raw stream instead of limiting itself to the buffer size.
 
diff --git a/Objects/sliceobject.c b/Objects/sliceobject.c
--- a/Objects/sliceobject.c
+++ b/Objects/sliceobject.c
@@ -80,19 +80,38 @@
 };
 
 
-/* Slice object implementation
+/* Slice object implementation */
 
-   start, stop, and step are python objects with None indicating no
+/* Using a cache is very effective since typically only a single slice is
+ * created and then deleted again
+ */
+static PySliceObject *slice_cache = NULL;
+void PySlice_Fini(void)
+{
+    PySliceObject *obj = slice_cache;
+    if (obj != NULL) {
+        slice_cache = NULL;
+        PyObject_Del(obj);
+    }
+}
+
+/* start, stop, and step are python objects with None indicating no
    index is present.
 */
 
 PyObject *
 PySlice_New(PyObject *start, PyObject *stop, PyObject *step)
 {
-    PySliceObject *obj = PyObject_New(PySliceObject, &PySlice_Type);
-
-    if (obj == NULL)
-        return NULL;
+    PySliceObject *obj;
+    if (slice_cache != NULL) {
+        obj = slice_cache;
+        slice_cache = NULL;
+        _Py_NewReference((PyObject *)obj);
+    } else {
+        obj = PyObject_New(PySliceObject, &PySlice_Type);
+        if (obj == NULL)
+            return NULL;
+    }
 
     if (step == NULL) step = Py_None;
     Py_INCREF(step);
@@ -260,7 +279,10 @@
     Py_DECREF(r->step);
     Py_DECREF(r->start);
     Py_DECREF(r->stop);
-    PyObject_Del(r);
+    if (slice_cache == NULL)
+        slice_cache = r;
+    else
+        PyObject_Del(r);
 }
 
 static PyObject *
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -531,6 +531,7 @@
     PyLong_Fini();
     PyFloat_Fini();
     PyDict_Fini();
+    PySlice_Fini();
 
     /* Cleanup Unicode implementation */
     _PyUnicode_Fini();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 18 20:55:34 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 18 Nov 2011 20:55:34 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Settle_on_=5F=5Fqualname=5F=5F?=
 =?utf8?q?_since_=5F=5Fqname=5F=5F_is_non-obvious_for_some_people=2E?=
Message-ID: <E1RRUX8-0005CO-D9@dinsdale.python.org>

http://hg.python.org/peps/rev/ae222922450a
changeset:   3995:ae222922450a
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 18 20:50:49 2011 +0100
summary:
  Settle on __qualname__ since __qname__ is non-obvious for some people.

files:
  pep-3155.txt |  40 +++++++++++++++++++++++++++++-----------
  1 files changed, 29 insertions(+), 11 deletions(-)


diff --git a/pep-3155.txt b/pep-3155.txt
--- a/pep-3155.txt
+++ b/pep-3155.txt
@@ -59,16 +59,16 @@
 Proposal
 ========
 
-This PEP proposes the addition of a ``__qname__`` attribute to
+This PEP proposes the addition of a ``__qualname__`` attribute to
 functions and classes.  For top-level functions and classes, the
-``__qname__`` attribute is equal to the ``__name__`` attribute.  For
-nested classed, methods, and nested functions, the ``__qname__``
+``__qualname__`` attribute is equal to the ``__name__`` attribute.  For
+nested classed, methods, and nested functions, the ``__qualname__``
 attribute contains a dotted path leading to the object from the module
 top-level.  A function's local namespace is represented in that dotted
 path by a component named ``<locals>``.
 
 The repr() and str() of functions and classes is modified to use
-``__qname__`` rather than ``__name__``.
+``__qualname__`` rather than ``__name__``.
 
 Example with nested classes
 ---------------------------
@@ -78,13 +78,13 @@
 ...   class D:
 ...     def g(): pass
 ...
->>> C.__qname__
+>>> C.__qualname__
 'C'
->>> C.f.__qname__
+>>> C.f.__qualname__
 'C.f'
->>> C.D.__qname__
+>>> C.D.__qualname__
 'C.D'
->>> C.D.g.__qname__
+>>> C.D.g.__qualname__
 'C.D.g'
 
 Example with nested functions
@@ -94,9 +94,9 @@
 ...   def g(): pass
 ...   return g
 ...
->>> f.__qname__
+>>> f.__qualname__
 'f'
->>> f().__qname__
+>>> f().__qualname__
 'f.<locals>.g'
 
 
@@ -108,16 +108,34 @@
 namespace is not available from the outside.  It will still be more
 helpful to the human reader than the bare ``__name__``.
 
-As the ``__name__`` attribute, the ``__qname__`` attribute is computed
+As the ``__name__`` attribute, the ``__qualname__`` attribute is computed
 statically and it will not automatically follow rebinding.
 
 
+Naming choice
+=============
+
+"Qualified name" is the best approximation, as a short phrase, of what the
+additional attribute is about.  It is not a "full name" or "fully qualified
+name" since it (deliberately) does not include the module name.  Calling
+it a "path" would risk confusion with filesystem paths and the ``__file__``
+attribute.
+
+The first proposal for the attribute name was to call it ``__qname__`` but
+many people (who are not aware of previous use of such jargon in e.g. the
+XML specification [2]_) found it obscure and non-obvious, which is why the
+slightly less short and more explicit ``__qualname__`` was finally chosen.
+
+
 References
 ==========
 
 .. [1] "pickle should support methods":
    http://bugs.python.org/issue9276
 
+.. [2] "QName" entry in Wikipedia:
+   http://en.wikipedia.org/wiki/QName
+
 
 Copyright
 =========

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sat Nov 19 00:43:48 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 19 Nov 2011 00:43:48 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313393=3A_In_TextIO?=
 =?utf8?q?Wrapper=2Eread=28n=29=2C_try_to_read_=60n=60_characters_as?=
Message-ID: <E1RRY60-0008Ot-Ja@dinsdale.python.org>

http://hg.python.org/cpython/rev/76d414cc3e38
changeset:   73618:76d414cc3e38
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 19 00:39:01 2011 +0100
summary:
  Issue #13393: In TextIOWrapper.read(n), try to read `n` characters as
once rather than limit ourselves to the default chunk size.

files:
  Modules/_io/textio.c |  14 +++++++++-----
  1 files changed, 9 insertions(+), 5 deletions(-)


diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -1388,7 +1388,7 @@
 /* Read and decode the next chunk of data from the BufferedReader.
  */
 static int
-textiowrapper_read_chunk(textio *self)
+textiowrapper_read_chunk(textio *self, Py_ssize_t size_hint)
 {
     PyObject *dec_buffer = NULL;
     PyObject *dec_flags = NULL;
@@ -1430,7 +1430,10 @@
     }
 
     /* Read a chunk, decode it, and put the result in self._decoded_chars. */
-    chunk_size = PyLong_FromSsize_t(self->chunk_size);
+    if (size_hint > 0) {
+        size_hint = Py_MAX(self->b2cratio, 1.0) * size_hint;
+    }
+    chunk_size = PyLong_FromSsize_t(Py_MAX(self->chunk_size, size_hint));
     if (chunk_size == NULL)
         goto fail;
     input_chunk = PyObject_CallMethodObjArgs(self->buffer,
@@ -1553,7 +1556,7 @@
 
         /* Keep reading chunks until we have n characters to return */
         while (remaining > 0) {
-            res = textiowrapper_read_chunk(self);
+            res = textiowrapper_read_chunk(self, remaining);
             if (res < 0)
                 goto fail;
             if (res == 0)  /* EOF */
@@ -1563,7 +1566,8 @@
                 if (chunks == NULL)
                     goto fail;
             }
-            if (PyList_Append(chunks, result) < 0)
+            if (PyUnicode_GET_LENGTH(result) > 0 &&
+                PyList_Append(chunks, result) < 0)
                 goto fail;
             Py_DECREF(result);
             result = textiowrapper_get_decoded_chars(self, remaining);
@@ -1720,7 +1724,7 @@
         res = 1;
         while (!self->decoded_chars ||
                !PyUnicode_GET_LENGTH(self->decoded_chars)) {
-            res = textiowrapper_read_chunk(self);
+            res = textiowrapper_read_chunk(self, 0);
             if (res < 0)
                 goto error;
             if (res == 0)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 00:49:44 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 19 Nov 2011 00:49:44 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Address_Barry=27s_comments?=
Message-ID: <E1RRYBk-0000QC-AT@dinsdale.python.org>

http://hg.python.org/peps/rev/63e69dec9ef1
changeset:   3996:63e69dec9ef1
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 19 00:43:54 2011 +0100
summary:
  Address Barry's comments

files:
  pep-3155.txt |  18 ++++++++++++++++++
  1 files changed, 18 insertions(+), 0 deletions(-)


diff --git a/pep-3155.txt b/pep-3155.txt
--- a/pep-3155.txt
+++ b/pep-3155.txt
@@ -112,6 +112,24 @@
 statically and it will not automatically follow rebinding.
 
 
+Discussion
+==========
+
+Excluding the module name
+-------------------------
+
+As ``__name__``, ``__qualname__`` doesn't include the module name.  This
+makes it independent of module aliasing and rebinding, and also allows to
+compute it at compile time.
+
+Reviving unbound methods
+------------------------
+
+Reviving unbound methods would only solve a fraction of the problems this
+PEP solves, at a higher price (an additional object type and an additional
+indirection, rather than an additional attribute).
+
+
 Naming choice
 =============
 

-- 
Repository URL: http://hg.python.org/peps

From solipsis at pitrou.net  Sat Nov 19 05:29:28 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 19 Nov 2011 05:29:28 +0100
Subject: [Python-checkins] Daily reference leaks (76d414cc3e38): sum=0
Message-ID: <E1RRcYS-0007LQ-M2@ap.vmr.nerim.net>

results for 76d414cc3e38 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogSq_3ln', '-x']

From python-checkins at python.org  Sat Nov 19 10:00:14 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sat, 19 Nov 2011 10:00:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313215=3A_multiproc?=
 =?utf8?q?essing=2EConnection=3A_don=27t_hammer_the_remote_end_with?=
Message-ID: <E1RRgmU-0002IV-Ig@dinsdale.python.org>

http://hg.python.org/cpython/rev/34fcc0d5c3c5
changeset:   73619:34fcc0d5c3c5
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sat Nov 19 09:59:43 2011 +0100
summary:
  Issue #13215: multiprocessing.Connection: don't hammer the remote end with
retries in case of ECONNREFUSED.

files:
  Lib/multiprocessing/connection.py |  16 +---------------
  1 files changed, 1 insertions(+), 15 deletions(-)


diff --git a/Lib/multiprocessing/connection.py b/Lib/multiprocessing/connection.py
--- a/Lib/multiprocessing/connection.py
+++ b/Lib/multiprocessing/connection.py
@@ -608,21 +608,7 @@
     '''
     family = address_type(address)
     with socket.socket( getattr(socket, family) ) as s:
-        t = _init_timeout()
-
-        while 1:
-            try:
-                s.connect(address)
-            except socket.error as e:
-                if e.args[0] != errno.ECONNREFUSED or _check_timeout(t):
-                    debug('failed to connect to address %s', address)
-                    raise
-                time.sleep(0.01)
-            else:
-                break
-        else:
-            raise
-
+        s.connect(address)
         fd = duplicate(s.fileno())
     conn = Connection(fd)
     return conn

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 13:18:50 2011
From: python-checkins at python.org (nick.coghlan)
Date: Sat, 19 Nov 2011 13:18:50 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Update_PEP_395_in_light_of_imp?=
 =?utf8?q?ort-sig_discussion_=28also_changes_qname-=3Equalname=29?=
Message-ID: <E1RRjsg-0005oN-OY@dinsdale.python.org>

http://hg.python.org/peps/rev/24bfca7b9b49
changeset:   3997:24bfca7b9b49
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sat Nov 19 22:18:45 2011 +1000
summary:
  Update PEP 395 in light of import-sig discussion (also changes qname->qualname)

files:
  pep-0395.txt |  520 +++++++++++++++++++++++++++++++-------
  1 files changed, 415 insertions(+), 105 deletions(-)


diff --git a/pep-0395.txt b/pep-0395.txt
--- a/pep-0395.txt
+++ b/pep-0395.txt
@@ -1,5 +1,5 @@
 PEP: 395
-Title: Module Aliasing
+Title: Qualifed Names for Modules
 Version: $Revision$
 Last-Modified: $Date$
 Author: Nick Coghlan <ncoghlan at gmail.com>
@@ -8,19 +8,36 @@
 Content-Type: text/x-rst
 Created: 4-Mar-2011
 Python-Version: 3.3
-Post-History: 5-Mar-2011
+Post-History: 5-Mar-2011, 19-Nov-2011
 
 
 Abstract
 ========
 
 This PEP proposes new mechanisms that eliminate some longstanding traps for
-the unwary when dealing with Python's import system, the pickle module and
-introspection interfaces.
+the unwary when dealing with Python's import system, as well as serialisation
+and introspection of functions and classes.
 
 It builds on the "Qualified Name" concept defined in PEP 3155.
 
 
+Relationship with Other PEPs
+----------------------------
+
+This PEP builds on the "qualified name" concept introduced by PEP 3155, and
+also shares in that PEP's aim of fixing some ugly corner cases when dealing
+with serialisation of arbitrary functions and classes.
+
+It is also affected by the two competing "namespace package" PEPs (PEP 382
+and PEP 402). This PEP would require some minor adjustments to accommodate
+PEP 382, but has some critical incompatibilities with respect to the namespace
+package mechanism proposed in PEP 402.
+
+Finally, PEP 328 eliminated implicit relative imports from imported modules.
+This PEP proposes that implicit relative imports from main modules also be
+eliminated.
+
+
 What's in a ``__name__``?
 =========================
 
@@ -48,35 +65,122 @@
 surprising when they do come up.
 
 
+Why are my imports broken?
+--------------------------
+
+There's a general principle that applies when modifying ``sys.path``: *never*
+put a package directory directly on ``sys.path``. The reason this is
+problematic is that every module in that directory is now potentially
+accessible under two different names: as a top level module (since the
+package directory is on ``sys.path``) and as a submodule of the package (if
+the higher level directory containing the package itself is also on
+``sys.path``).
+
+As an example, Django (up to and including version 1.3) is guilty of setting
+up exactly this situation for site-specific applications - the application
+ends up being accessible as both ``app`` and ``site.app`` in the module
+namespace, and these are actually two *different* copies of the module. This
+is a recipe for confusion if there is any meaningful mutable module level
+state, so this behaviour is being eliminated from the default site set up in
+version 1.4 (site-specific apps will always be fully qualified with the site
+name).
+
+However, it's hard to blame Django for this, when the same part of Python
+responsible for setting ``__name__ = "__main__"`` in the main module commits
+the exact same error when determining the value for ``sys.path[0]``.
+
+The impact of this can be seen relatively frequently if you follow the 
+"python" and "import" tags on Stack Overflow. When I had the time to follow
+it myself, I regularly encountered people struggling to understand the
+behaviour of straightforward package layouts like the following::
+
+    project/
+        setup.py
+        package/
+            __init__.py
+            foo.py
+            tests/
+                __init__.py
+                test_foo.py
+
+I would actually often see it without the ``__init__.py`` files first, but
+that's a trivial fix to explain. What's hard to explain is that all of the
+following ways to invoke ``test_foo.py`` *probably won't work* due to broken
+imports (either failing to find ``package`` for absolute imports, complaining
+about relative imports in a non-package for explicit relative imports, or
+issuing even more obscure errors if some other submodule happens to shadow
+the name of a top-level module, such as a ``package.json`` module that
+handled serialisation or a ``package.tests.unittest`` test runner)::
+
+    # working directory: project/package/tests
+    ./test_foo.py
+    python test_foo.py
+    python -m test_foo
+    python -c "from test_foo import main; main()"
+
+    # working directory: project/package
+    tests/test_foo.py
+    python tests/test_foo.py
+    python -m tests.test_foo
+    python -c "from tests.test_foo import main; main()"
+
+    # working directory: project
+    package/tests/test_foo.py
+    python package/tests/test_foo.py
+
+    # working directory: project/..
+    project/package/tests/test_foo.py
+    python project/package/tests/test_foo.py
+    # The -m and -c approaches don't work from here either, but the failure
+    # to find 'package' correctly is pretty easy to explain in this case
+
+That's right, that long list is of all the methods of invocation that will
+almost certainly *break* if you try them, and the error messages won't make
+any sense if you're not already intimately not only with the way Python's
+import system works, but also with how it gets initialised.
+
+For a long time, the only way to get ``sys.path`` right with that kind of
+setup was to either set it manually in ``test_foo.py`` itself (hardly
+something a novice, or even many veteran, Python programmers are going to
+know how to do) or else to make sure to import the module instead of
+executing it directly::
+
+    # working directory: project
+    python -c "from package.tests.test_foo import main; main()"
+
+Since the implementation of PEP 366 (which defined a mechanism that allows
+relative imports to work correctly when a module inside a package is executed
+via the ``-m`` switch), the following also works properly::
+
+    # working directory: project
+    python -m package.tests.test_foo
+
+The fact that most methods of invoking Python code from the command line
+break when that code is inside a package, and the two that do work are highly
+sensitive to the current working directory is all thoroughly confusing for a
+beginner, and I personally believe it is one of the key factors leading
+to the perception that Python packages are complicated and hard to get right.
+
+This problem isn't even limited to the command line - if ``test_foo.py`` is
+open in Idle and you attempt to run it by pressing F5, then it will fail in
+just the same way it would if run directly from the command line.
+
+There's a reason the general ``sys.path`` guideline mentioned above exists,
+and the fact that the interpreter itself doesn't follow it when determining
+``sys.path[0]`` is the root cause of all sorts of grief.
+
+
 Importing the main module twice
 -------------------------------
 
-The most venerable of these traps is the issue of (effectively) importing
-``__main__`` twice. This occurs when the main module is also imported under
-its real name, effectively creating two instances of the same module under
+Another venerable trap is the issue of (effectively) importing ``__main__``
+twice. This occurs when the main module is also imported under its real
+name, effectively creating two instances of the same module under
 different names.
 
-This problem used to be significantly worse due to implicit relative imports
-from the main module, but the switch to allowing only absolute imports and
-explicit relative imports means this issue is now restricted to affecting the
-main module itself.
-
-
-Why are my relative imports broken?
------------------------------------
-
-PEP 366 defines a mechanism that allows relative imports to work correctly
-when a module inside a package is executed via the ``-m`` switch.
-
-Unfortunately, many users still attempt to directly execute scripts inside
-packages. While this no longer silently does the wrong thing by
-creating duplicate copies of peer modules due to implicit relative imports, it
-now fails noisily at the first explicit relative import, even though the
-interpreter actually has sufficient information available on the filesystem to
-make it work properly.
-
-<TODO: Anyone want to place bets on how many Stack Overflow links I could find
-to put here if I really went looking?>
+If the state stored in ``__main__`` is significant to the correct operation
+of the program, then this duplication can cause obscure and surprising
+errors.
 
 
 In a bit of a pickle
@@ -91,21 +195,23 @@
 ``__main__`` module in any application that involves any form of object
 serialisation and persistence.
 
-Similarly, when creating a pseudo-module\*, pickles rely on the name of the
+Similarly, when creating a pseudo-module, pickles rely on the name of the
 module where a class is actually defined, rather than the officially
 documented location for that class in the module hierarchy.
 
-While this PEP focuses specifically on ``pickle`` as the principal
-serialisation scheme in the standard library, this issue may also affect
-other mechanisms that support serialisation of arbitrary class instances.
-
-\*For the purposes of this PEP, a "pseudo-module" is a package designed like
+For the purposes of this PEP, a "pseudo-module" is a package designed like
 the Python 3.2 ``unittest`` and ``concurrent.futures`` packages. These
 packages are documented as if they were single modules, but are in fact
 internally implemented as a package. This is *supposed* to be an
-implementation detail that users and other implementations don't need to worry
-about, but, thanks to ``pickle`` (and serialisation in general), the details
-are exposed and effectively become part of the public API.
+implementation detail that users and other implementations don't need to
+worry about, but, thanks to ``pickle`` (and serialisation in general),
+the details are often exposed and can effectively become part of the public
+API.
+
+While this PEP focuses specifically on ``pickle`` as the principal
+serialisation scheme in the standard library, this issue may also affect
+other mechanisms that support serialisation of arbitrary class instances
+and rely on ``__name__`` to determine how to handle deserialisation.
 
 
 Where's the source?
@@ -141,8 +247,30 @@
 multiprocessing module on other platforms.
 
 
-Proposed Changes
-================
+Qualified Names for Modules
+===========================
+
+To make it feasible to fix these problems once and for all, it is proposed
+to add a new module level attribute: ``__qualname__``. This abbreviation of
+"qualified name" is taken from PEP 3155, where it is used to store the naming
+path to a nested class or function definition relative to the top level
+module.
+
+If a module loader does not initialise ``__qualname__`` itself, then the
+import system will add it automatically (setting it to the same value as
+``__name__``).
+
+For modules, ``__qualname__`` will normally be the same as ``__name__``, just
+as it is for top-level functions and classes in PEP 3155. However, it will
+differ in some situations so that the above problems can be addressed.
+
+Specifically, whenever ``__name__`` is modified for some other purpose (such
+as to denote the main module), then ``__qualname__`` will remain unchanged,
+allowing code that needs it to access the original unmodified value.
+
+
+Eliminating the Traps
+=====================
 
 The following changes are interrelated and make the most sense when
 considered together. They collectively either completely eliminate the traps
@@ -150,105 +278,281 @@
 dealing with them.
 
 A rough draft of some of the concepts presented here was first posted on the
-python-ideas list [1], but they have evolved considerably since first being
-discussed in that thread.
+python-ideas list [1]_, but they have evolved considerably since first being
+discussed in that thread. Further discussion has subsequently taken place on 
+import-sig [2]_.
+
+
+Fixing main module imports inside packages
+------------------------------------------
+
+To eliminate this trap, it is proposed that an additional filesystem check be
+performed when determining a suitable value for ``sys.path[0]``. This check
+will look for Python's explicit package directory markers and use them to find
+the appropriate directory to add to ``sys.path``.
+
+The current algorithm for setting ``sys.path[0]`` in relevant cases is roughly
+as follows:
+
+    # Interactive prompt, -m switch, -c switch
+    sys.path.insert(0, '')
+
+    # Valid sys.path entry execution (i.e. directory and zip execution)
+    sys.path.insert(0, sys.argv[0])
+
+    # Direct script execution
+    sys.path.insert(0, os.path.dirname(sys.argv[0]))
+
+It is proposed that this initialisation process be modified to take
+package details stored on the filesystem into account::
+
+    # Interactive prompt, -c switch
+    in_package, path_entry, modname = split_path_module(os.getcwd(), '')
+    if in_package:
+        sys.path.insert(0, path_entry)
+    else:
+        sys.path.insert(0, '')
+    # Start interactive prompt or run -c command as usual
+    # __main__.__qualname__ is set to "__main__"
+
+    # -m switch
+    modname = <<argument to -m switch>>
+    in_package, path_entry, modname = split_path_module(os.getcwd(), modname)
+    if in_package:
+        sys.path.insert(0, path_entry)
+    else:
+        sys.path.insert(0, '')
+    # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
+    # __main__.__qualname__ is set to modname
+
+    # Valid sys.path entry execution (i.e. directory and zip execution)
+    modname = "__main__"
+    path_entry, modname = split_path_module(sys.argv[0], modname)
+    sys.path.insert(0, path_entry)
+    # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
+    # __main__.__qualname__ is set to modname
+
+    # Direct script execution
+    in_package, path_entry, modname = split_path_module(sys.argv[0])
+    sys.path.insert(0, path_entry)
+    if in_package:
+        # Pass modname to ``runpy._run_module_as_main()``
+    else:
+        # Run script directly
+    # __main__.__qualname__ is set to modname
+
+The ``split_path_module()`` supporting function used in the above pseudo-code
+would have the following semantics::
+
+    def _splitmodname(fspath):
+        path_entry, fname = os.path.split(fspath)
+        modname = os.path.splitext(fname)[0]
+        return path_entry, modname
+
+    def _is_package_dir(fspath):
+        return any(os.exists("__init__" + info[0]) for info
+                       in imp.get_suffixes())
+
+    def split_path_module(fspath, modname=None):
+        """Given a filesystem path and a relative module name, determine an
+           appropriate sys.path entry and a fully qualified module name.
+
+           Returns a 3-tuple of (package_depth, fspath, modname). A reported
+           package depth of 0 indicates that this would be a top level import.
+
+           If no relative module name is given, it is derived from the final
+           component in the supplied path with the extension stripped.
+        """
+        if modname is None:
+            fspath, modname = _splitmodname(fspath)
+        package_depth = 0
+        while _is_package_dir(fspath):
+            fspath, pkg = _splitmodname(fspath)
+            modname = pkg + '.' + modname
+        return package_depth, fspath, modname
+
+This PEP also proposes that the ``split_path_module()`` functionality be
+exposed directly to Python users via the ``runpy`` module.
+
+
+Compatibility with PEP 382
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Making this proposal compatible with the PEP 382 namespace packaging PEP is
+trivial. The semantics of ``_is_package_dir()`` are merely changed to be::
+
+    def _is_package_dir(fspath):
+        return (fspath.endswith(".pyp") or
+                any(os.exists("__init__" + info[0]) for info
+                        in imp.get_suffixes()))
+
+
+Incompatibility with PEP 402
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+PEP 402 proposes the elimination of explicit markers in the file system for
+Python packages. This fundamentally breaks the proposed concept of being able
+to take a filesystem path and a Python module name and work out an unambiguous
+mapping to the Python module namespace. Instead, the appropriate mapping
+would depend on the current values in ``sys.path``, rendering it impossible
+to ever fix the problems described above with the calculation of
+``sys.path[0]`` when the interpreter is initialised.
+
+While some aspects of this PEP could probably be salvaged if PEP 402 were
+adopted, the core concept of making import semantics from main and other
+modules more consistent would no longer be feasible.
+
+This incompatibility is discussed in more detail in the relevant import-sig
+thread [2]_.
+
+
+Potential incompatibilities with scripts stored in packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The proposed change to ``sys.path[0]`` initialisation *may* break some
+existing code. Specifically, it will break scripts stored in package
+directories that rely on the implicit relative imports from ``__main__`` in
+order to run correctly under Python 3.
+
+While such scripts could be imported in Python 2 (due to implicit relative
+imports) it is already the case that they cannot be imported in Python 3,
+as implicit relative imports are no longer permitted when a module is
+imported.
+
+By disallowing implicit relatives imports from the main module as well,
+such modules won't even work as scripts with this PEP. Switching them
+over to explicit relative imports will then get them working again as
+both executable scripts *and* as importable modules.
+
+To support earlier versions of Python, a script could be written to use
+different forms of import based on the Python version::
+
+    if __name__ == "__main__" and sys.version_info < (3, 3):
+        import peer # Implicit relative import
+    else:
+        from . import peer # explicit relative import
 
 
 Fixing dual imports of the main module
 --------------------------------------
 
-Two simple changes are proposed to fix this problem:
+Given the above proposal to get ``__qualname__`` consistently set correctly
+in the main module, one simple change is proposed to eliminate the problem
+of dual imports of the main module: the addition of a ``sys.metapath`` hook
+that detects attempts to import ``__main__`` under its real name and returns
+the original main module instead::
 
-1. In ``runpy``, modify the implementation of the ``-m`` switch handling to
-   install the specified module in ``sys.modules`` under both its real name
-   and the name ``__main__``. (Currently it is only installed as the latter)
-2. When directly executing a module, install it in ``sys.modules`` under
-   ``os.path.splitext(os.path.basename(__file__))[0]`` as well as under
-   ``__main__``.
+  class AliasImporter:
+    def __init__(self, module, alias):
+        self.module = module
+        self.alias = alias
 
-With the main module also stored under its "real" name, attempts to import it
-will pick it up from the ``sys.modules`` cache rather than reimporting it
-under the new name.
+    def __repr__(self):
+        fmt = "{0.__class__.__name__}({0.module.__name__}, {0.alias})"
+        return fmt.format(self)
 
+    def find_module(self, fullname, path=None):
+        if path is None and fullname == self.alias:
+            return self
+        return None
 
-Fixing direct execution inside packages
----------------------------------------
+    def load_module(self, fullname):
+        if fullname != self.alias:
+            raise ImportError("{!r} cannot load {!r}".format(self, fullname))
+        return self.main_module
 
-To fix this problem, it is proposed that an additional filesystem check be
-performed before proceeding with direct execution of a ``PY_SOURCE`` or
-``PY_COMPILED`` file that has been named on the command line.
+This metapath hook would be added automatically during import system
+initialisation based on the following logic::
 
-This additional check would look for an ``__init__`` file that is a peer to
-the specified file with a matching extension (either ``.py``, ``.pyc`` or
-``.pyo``, depending what was passed on the command line).
+    main = sys.modules["__main__"]
+    if main.__name__ != main.__qualname__:
+        sys.metapath.append(AliasImporter(main, main.__qualname__))
 
-If this check fails to find anything, direct execution proceeds as usual.
-
-If, however, it finds something, execution is handed over to a
-helper function in the ``runpy`` module that ``runpy.run_path`` also invokes
-in the same circumstances. That function will walk back up the
-directory hierarchy from the supplied path, looking for the first directory
-that doesn't contain an ``__init__`` file. Once that directory is found, it
-will be set to ``sys.path[0]``, ``sys.argv[0]`` will be set to ``-m`` and
-``runpy._run_module_as_main`` will be invoked with the appropriate module
-name (as calculated based on the original filename and the directories
-traversed while looking for a directory without an ``__init__`` file).
-
-The two current PEPs for namespace packages (PEP 382 and PEP 402) would both
-affect this part of the proposal. For PEP 382 (with its current suggestion of
-"\*.pyp" package directories, this check would instead just walk up the
-supplied path, looking for the first non-package directory (this would not
-require any filesystem stat calls). Since PEP 402 deliberately omits explicit
-directory markers, it would need an alternative approach, based on checking
-the supplied path against the contents of ``sys.path``. In both cases, the
-direct execution behaviour can still be corrected.
+This is probably the least important proposal in the PEP - it just
+closes off the last mechanism that is likely to lead to module duplication
+after the configuration of ``sys.path[0]`` at interpreter startup is
+addressed.
 
 
 Fixing pickling without breaking introspection
 ----------------------------------------------
 
-To fix this problem, it is proposed to add a new optional module level
-attribute: ``__qname__``. This abbreviation of "qualified name" is taken
-from PEP 3155, where it is used to store the naming path to a nested class
-or function definition relative to the top level module. By default,
-``__qname__`` will be the same as ``__name__``, which covers the typical
-case where there is a one-to-one correspondence between the documented API
-and the actual module implementation.
+To fix this problem, it is proposed to make use of the new module level
+``__qualname__`` attributes to determine the real module location when
+``__name__`` has been modified for any reason.
 
-Functions and classes will gain a corresponding ``__qmodule__`` attribute
-that refers to their module's ``__qname__``.
+In the main module, ``__qualname__`` will automatically be set to the main
+module's "real" name (as described above) by the interpreter.
 
 Pseudo-modules that adjust ``__name__`` to point to the public namespace will
-leave ``__qname__`` untouched, so the implementation location remains readily
+leave ``__qualname__`` untouched, so the implementation location remains readily
 accessible for introspection.
 
-In the main module, ``__qname__`` will automatically be set to the main
-module's "real" name (as described above under the fix to prevent duplicate
-imports of the main module) by the interpreter.
+If ``__name__`` is adjusted at the top of a module, then this will
+automatically adjust the ``__module__`` attribute for all functions and
+classes subsequently defined in that module.
 
-At the interactive prompt, both ``__name__`` and ``__qname__`` will be set
-to ``"__main__"``.
+Since multiple submodules may be set to use the same "public" namespace,
+functions and classes will be given a new ``__qualmodule__`` attribute
+that refers to the ``__qualname__`` of their module.
 
-These changes on their own will fix most pickling and serialisation problems,
-but one additional change is needed to fix the problem with serialisation of
-items in ``__main__``: as a slight adjustment to the definition process for
-functions and classes, in the ``__name__ == "__main__"`` case, the module
-``__qname__`` attribute will be used to set ``__module__``.
+This isn't strictly necessary for functions (you could find out their
+module's qualified name by looking in their globals dictionary), it is
+needed for classes, since they don't hold a reference to the globals of
+their defining module. Once a new attribute is added to classes, it is
+more convenient to keep the API consistent and add a new attribute to
+functions as well.
 
-``pydoc`` and ``inspect`` would also be updated appropriately to:
+These changes mean that adjusting ``__name__`` (and, either directly or
+indirectly, the corresponding function and class ``__module__`` attributes)
+becomes the officially sanctioned way to implement a namespace as a package,
+while exposing the API as if it were still a single module.
 
-- use ``__qname__`` instead of ``__name__`` and ``__qmodule__`` instead of
-  ``__module__``where appropriate (e.g. ``inspect.getsource()`` would prefer
-  the qualified variants)
-- report both the public names and the qualified names for affected objects
+All serialisation code that currently uses ``__name__`` and ``__module__``
+attributes will then avoid exposing implementation details by default.
+
+To correctly handle serialisation of items from the main module, the class
+and function definition logic will be updated to also use ``__qualname__``
+for the ``__module__`` attribute in the case where ``__name__ == "__main__"``.
+
+With ``__name__`` and ``__module__`` being officially blessed as being used
+for the *public* names of things, the introspection tools in the standard
+library will be updated to use ``__qualname__`` and ``__qualmodule__``
+where appropriate. For example:
+
+- ``pydoc`` will report both public and qualified names for modules
+- ``inspect.getsource()`` (and similar tools) will use the qualified names
+  that point to the implementation of the code
+- additional ``pydoc`` and/or ``inspect`` APIs may be provided that report
+  all modules with a given public ``__name__``.
+
 
 Fixing multiprocessing on Windows
 ---------------------------------
 
-With ``__qname__`` now available to tell ``multiprocessing`` the real
-name of the main module, it should be able to simply include it in the
+With ``__qualname__`` now available to tell ``multiprocessing`` the real
+name of the main module, it will be able to simply include it in the
 serialised information passed to the child process, eliminating the
-need for dubious reverse engineering of the ``__file__`` attribute.
+need for the current dubious introspection of the ``__file__`` attribute.
+
+For older Python versions, ``multiprocessing`` could be improved by applying
+the ``split_path_module()`` algorithm described above when attempting to
+work out how to execute the main module based on its ``__file__`` attribute.
+
+
+Explicit relative imports
+=========================
+
+This PEP proposes that ``__package__`` be unconditionally defined in the
+main module as ``__qualname__.rpartition('.')[0]``. Aside from that, it
+proposes that the behaviour of explicit relative imports be left alone.
+
+In particular, if ``__package__`` is not set in a module when an explicit
+relative import occurs, the automatically cached value  will continue to be
+derived from ``__name__`` rather than ``__qualname__``. This minimises any
+backwards incompatibilities with code that deliberately manipulates
+relative imports by adjusting ``__name__`` rather than setting ``__package__``
+directly.
 
 
 Reference Implementation
@@ -263,6 +567,10 @@
 .. [1] Module aliases and/or "real names"
    (http://mail.python.org/pipermail/python-ideas/2011-January/008983.html)
 
+.. [2] PEP 395 (Module aliasing) and the namespace PEPs
+   (http://mail.python.org/pipermail/import-sig/2011-November/000382.html)
+
+
 
 Copyright
 =========

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sat Nov 19 13:49:55 2011
From: python-checkins at python.org (nick.coghlan)
Date: Sat, 19 Nov 2011 13:49:55 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Minor_updates_after_additional?=
 =?utf8?q?_review_of_the_published_version_of_PEP_395?=
Message-ID: <E1RRkMl-0007nI-V8@dinsdale.python.org>

http://hg.python.org/peps/rev/adfe569e8379
changeset:   3998:adfe569e8379
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sat Nov 19 22:49:17 2011 +1000
summary:
  Minor updates after additional review of the published version of PEP 395

files:
  pep-0395.txt |  82 +++++++++++++++++++++++----------------
  1 files changed, 49 insertions(+), 33 deletions(-)


diff --git a/pep-0395.txt b/pep-0395.txt
--- a/pep-0395.txt
+++ b/pep-0395.txt
@@ -28,10 +28,14 @@
 also shares in that PEP's aim of fixing some ugly corner cases when dealing
 with serialisation of arbitrary functions and classes.
 
-It is also affected by the two competing "namespace package" PEPs (PEP 382
-and PEP 402). This PEP would require some minor adjustments to accommodate
-PEP 382, but has some critical incompatibilities with respect to the namespace
-package mechanism proposed in PEP 402.
+It also builds on PEP 366, which took initial tentative steps towards making
+explicit relative imports from the main module work correctly in at least
+*some* circumstances.
+
+This PEP is also affected by the two competing "namespace package" PEPs
+(PEP 382 and PEP 402). This PEP would require some minor adjustments to
+accommodate PEP 382, but has some critical incompatibilities with respect to
+the implicit namespace package mechanism proposed in PEP 402.
 
 Finally, PEP 328 eliminated implicit relative imports from imported modules.
 This PEP proposes that implicit relative imports from main modules also be
@@ -136,8 +140,8 @@
 
 That's right, that long list is of all the methods of invocation that will
 almost certainly *break* if you try them, and the error messages won't make
-any sense if you're not already intimately not only with the way Python's
-import system works, but also with how it gets initialised.
+any sense if you're not already intimately familiar not only with the way
+Python's import system works, but also with how it gets initialised.
 
 For a long time, the only way to get ``sys.path`` right with that kind of
 setup was to either set it manually in ``test_foo.py`` itself (hardly
@@ -173,31 +177,31 @@
 Importing the main module twice
 -------------------------------
 
-Another venerable trap is the issue of (effectively) importing ``__main__``
-twice. This occurs when the main module is also imported under its real
-name, effectively creating two instances of the same module under
-different names.
+Another venerable trap is the issue of importing ``__main__`` twice. This
+occurs when the main module is also imported under its real name, effectively
+creating two instances of the same module under different names.
 
 If the state stored in ``__main__`` is significant to the correct operation
-of the program, then this duplication can cause obscure and surprising
-errors.
+of the program, or if there is top-level code in the main module that has
+non-idempotent side effects, then this duplication can cause obscure and
+surprising errors.
 
 
 In a bit of a pickle
 --------------------
 
-Something many users may not realise is that the ``pickle`` module serialises
-objects based on the ``__name__`` of the containing module. So objects
-defined in ``__main__`` are pickled that way, and won't be unpickled
-correctly by another python instance that only imported that module instead
-of running it directly. This behaviour is the underlying reason for the
-advice from many Python veterans to do as little as possible in the 
-``__main__`` module in any application that involves any form of object
-serialisation and persistence.
+Something many users may not realise is that the ``pickle`` module sometimes
+relies on the ``__module__`` attribute when serialising instances of arbitrary
+classes. So instances of classes defined in ``__main__`` are pickled that way,
+and won't be unpickled correctly by another python instance that only imported
+that module instead of running it directly. This behaviour is the underlying
+reason for the advice from many Python veterans to do as little as possible
+in the  ``__main__`` module in any application that involves any form of
+object serialisation and persistence.
 
-Similarly, when creating a pseudo-module, pickles rely on the name of the
-module where a class is actually defined, rather than the officially
-documented location for that class in the module hierarchy.
+Similarly, when creating a pseudo-module (see next paragraph), pickles rely
+on the name of the module where a class is actually defined, rather than the
+officially documented location for that class in the module hierarchy.
 
 For the purposes of this PEP, a "pseudo-module" is a package designed like
 the Python 3.2 ``unittest`` and ``concurrent.futures`` packages. These
@@ -211,7 +215,8 @@
 While this PEP focuses specifically on ``pickle`` as the principal
 serialisation scheme in the standard library, this issue may also affect
 other mechanisms that support serialisation of arbitrary class instances
-and rely on ``__name__`` to determine how to handle deserialisation.
+and rely on ``__module__`` attributes to determine how to handle
+deserialisation.
 
 
 Where's the source?
@@ -240,7 +245,8 @@
 executed script or top-level module. Packages and non-top-level modules
 executed via the ``-m`` switch, as well as directly executed zipfiles or
 directories, are likely to make multiprocessing on Windows do the wrong thing
-(either quietly or noisily) when spawning a new process.
+(either quietly or noisily, depending on application details) when spawning a
+new process.
 
 While this issue currently only affects Windows directly, it also impacts
 any proposals to provide Windows-style "clean process" invocation via the
@@ -256,10 +262,6 @@
 path to a nested class or function definition relative to the top level
 module.
 
-If a module loader does not initialise ``__qualname__`` itself, then the
-import system will add it automatically (setting it to the same value as
-``__name__``).
-
 For modules, ``__qualname__`` will normally be the same as ``__name__``, just
 as it is for top-level functions and classes in PEP 3155. However, it will
 differ in some situations so that the above problems can be addressed.
@@ -268,6 +270,10 @@
 as to denote the main module), then ``__qualname__`` will remain unchanged,
 allowing code that needs it to access the original unmodified value.
 
+If a module loader does not initialise ``__qualname__`` itself, then the
+import system will add it automatically (setting it to the same value as
+``__name__``).
+
 
 Eliminating the Traps
 =====================
@@ -280,7 +286,7 @@
 A rough draft of some of the concepts presented here was first posted on the
 python-ideas list [1]_, but they have evolved considerably since first being
 discussed in that thread. Further discussion has subsequently taken place on 
-import-sig [2]_.
+the import-sig mailing list [2]_.
 
 
 Fixing main module imports inside packages
@@ -292,14 +298,18 @@
 the appropriate directory to add to ``sys.path``.
 
 The current algorithm for setting ``sys.path[0]`` in relevant cases is roughly
-as follows:
+as follows::
 
     # Interactive prompt, -m switch, -c switch
     sys.path.insert(0, '')
 
+::
+
     # Valid sys.path entry execution (i.e. directory and zip execution)
     sys.path.insert(0, sys.argv[0])
 
+::
+
     # Direct script execution
     sys.path.insert(0, os.path.dirname(sys.argv[0]))
 
@@ -315,6 +325,8 @@
     # Start interactive prompt or run -c command as usual
     # __main__.__qualname__ is set to "__main__"
 
+::
+
     # -m switch
     modname = <<argument to -m switch>>
     in_package, path_entry, modname = split_path_module(os.getcwd(), modname)
@@ -325,6 +337,8 @@
     # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
     # __main__.__qualname__ is set to modname
 
+::
+
     # Valid sys.path entry execution (i.e. directory and zip execution)
     modname = "__main__"
     path_entry, modname = split_path_module(sys.argv[0], modname)
@@ -332,6 +346,8 @@
     # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
     # __main__.__qualname__ is set to modname
 
+::
+
     # Direct script execution
     in_package, path_entry, modname = split_path_module(sys.argv[0])
     sys.path.insert(0, path_entry)
@@ -497,7 +513,7 @@
 that refers to the ``__qualname__`` of their module.
 
 This isn't strictly necessary for functions (you could find out their
-module's qualified name by looking in their globals dictionary), it is
+module's qualified name by looking in their globals dictionary), but it is
 needed for classes, since they don't hold a reference to the globals of
 their defining module. Once a new attribute is added to classes, it is
 more convenient to keep the API consistent and add a new attribute to
@@ -550,7 +566,7 @@
 In particular, if ``__package__`` is not set in a module when an explicit
 relative import occurs, the automatically cached value  will continue to be
 derived from ``__name__`` rather than ``__qualname__``. This minimises any
-backwards incompatibilities with code that deliberately manipulates
+backwards incompatibilities with existing code that deliberately manipulates
 relative imports by adjusting ``__name__`` rather than setting ``__package__``
 directly.
 

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sat Nov 19 17:27:37 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 19 Nov 2011 17:27:37 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyMjQ1?=
 =?utf8?q?=3A_Document_sys=2Efloat=5Finfo=2Erounds_better=2E?=
Message-ID: <E1RRnlR-0005o5-Nb@dinsdale.python.org>

http://hg.python.org/cpython/rev/de1ecda2afa2
changeset:   73620:de1ecda2afa2
branch:      2.7
parent:      73614:fd9d7a8e45bc
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Nov 19 16:26:08 2011 +0000
summary:
  Issue #12245: Document sys.float_info.rounds better.

files:
  Doc/library/sys.rst |  8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -328,8 +328,12 @@
    +---------------------+----------------+--------------------------------------------------+
    | :const:`radix`      | FLT_RADIX      | radix of exponent representation                 |
    +---------------------+----------------+--------------------------------------------------+
-   | :const:`rounds`     | FLT_ROUNDS     | constant representing rounding mode              |
-   |                     |                | used for arithmetic operations                   |
+   | :const:`rounds`     | FLT_ROUNDS     | integer constant representing the rounding mode  |
+   |                     |                | used for arithmetic operations.  This reflects   |
+   |                     |                | the value of the system FLT_ROUNDS macro at      |
+   |                     |                | interpreter startup time.  See section 5.2.4.2.2 |
+   |                     |                | of the C99 standard for an explanation of the    |
+   |                     |                | possible values and their meanings.              |
    +---------------------+----------------+--------------------------------------------------+
 
    The attribute :attr:`sys.float_info.dig` needs further explanation.  If

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 17:27:38 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 19 Nov 2011 17:27:38 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMjQ1?=
 =?utf8?q?=3A_Document_sys=2Efloat=5Finfo=2Erounds_better=2E?=
Message-ID: <E1RRnlS-0005oA-GC@dinsdale.python.org>

http://hg.python.org/cpython/rev/795c184b0282
changeset:   73621:795c184b0282
branch:      3.2
parent:      73615:583aff635ce1
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Nov 19 16:26:45 2011 +0000
summary:
  Issue #12245: Document sys.float_info.rounds better.

files:
  Doc/library/sys.rst |  8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -296,8 +296,12 @@
    +---------------------+----------------+--------------------------------------------------+
    | :const:`radix`      | FLT_RADIX      | radix of exponent representation                 |
    +---------------------+----------------+--------------------------------------------------+
-   | :const:`rounds`     | FLT_ROUNDS     | constant representing rounding mode              |
-   |                     |                | used for arithmetic operations                   |
+   | :const:`rounds`     | FLT_ROUNDS     | integer constant representing the rounding mode  |
+   |                     |                | used for arithmetic operations.  This reflects   |
+   |                     |                | the value of the system FLT_ROUNDS macro at      |
+   |                     |                | interpreter startup time.  See section 5.2.4.2.2 |
+   |                     |                | of the C99 standard for an explanation of the    |
+   |                     |                | possible values and their meanings.              |
    +---------------------+----------------+--------------------------------------------------+
 
    The attribute :attr:`sys.float_info.dig` needs further explanation.  If

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 17:27:39 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 19 Nov 2011 17:27:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312245_merge=2E?=
Message-ID: <E1RRnlT-0005oH-80@dinsdale.python.org>

http://hg.python.org/cpython/rev/5e45dfc421e4
changeset:   73622:5e45dfc421e4
parent:      73619:34fcc0d5c3c5
parent:      73621:795c184b0282
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Nov 19 16:27:22 2011 +0000
summary:
  Issue #12245 merge.

files:
  Doc/library/sys.rst |  8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)


diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -298,8 +298,12 @@
    +---------------------+----------------+--------------------------------------------------+
    | :const:`radix`      | FLT_RADIX      | radix of exponent representation                 |
    +---------------------+----------------+--------------------------------------------------+
-   | :const:`rounds`     | FLT_ROUNDS     | constant representing rounding mode              |
-   |                     |                | used for arithmetic operations                   |
+   | :const:`rounds`     | FLT_ROUNDS     | integer constant representing the rounding mode  |
+   |                     |                | used for arithmetic operations.  This reflects   |
+   |                     |                | the value of the system FLT_ROUNDS macro at      |
+   |                     |                | interpreter startup time.  See section 5.2.4.2.2 |
+   |                     |                | of the C99 standard for an explanation of the    |
+   |                     |                | possible values and their meanings.              |
    +---------------------+----------------+--------------------------------------------------+
 
    The attribute :attr:`sys.float_info.dig` needs further explanation.  If

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 18:58:46 2011
From: python-checkins at python.org (mark.dickinson)
Date: Sat, 19 Nov 2011 18:58:46 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=239530=3A_Fix_undefi?=
 =?utf8?q?ned_behaviour_due_to_signed_overflow_in_testcapi=5Flong=2Eh=2E?=
Message-ID: <E1RRpBe-0003nt-Iz@dinsdale.python.org>

http://hg.python.org/cpython/rev/71100ef4f7a2
changeset:   73623:71100ef4f7a2
user:        Mark Dickinson <mdickinson at enthought.com>
date:        Sat Nov 19 17:58:15 2011 +0000
summary:
  Issue #9530: Fix undefined behaviour due to signed overflow in testcapi_long.h.

files:
  Modules/testcapi_long.h |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Modules/testcapi_long.h b/Modules/testcapi_long.h
--- a/Modules/testcapi_long.h
+++ b/Modules/testcapi_long.h
@@ -33,8 +33,7 @@
             unsigned TYPENAME uin, uout;
 
             /* For 0, 1, 2 use base; for 3, 4, 5 use -base */
-            uin = j < 3 ? base
-                        : (unsigned TYPENAME)(-(TYPENAME)base);
+            uin = j < 3 ? base : 0U - base;
 
             /* For 0 & 3, subtract 1.
              * For 1 & 4, leave alone.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 21:10:57 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 19 Nov 2011 21:10:57 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzM4?=
 =?utf8?q?=3A_Handle_all_enumerations_in_=5FPy=5FANNOTATE=5FMEMORY=5FORDER?=
Message-ID: <E1RRrFZ-0004R3-1A@dinsdale.python.org>

http://hg.python.org/cpython/rev/f855f929bc48
changeset:   73624:f855f929bc48
branch:      3.2
parent:      73621:795c184b0282
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 19 22:03:10 2011 +0200
summary:
  Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER

This allows compiling extension modules with -Wswitch-enum on gcc.
Initial patch by Floris Bruynooghe.

files:
  Include/pyatomic.h |  6 ++++--
  Misc/ACKS          |  1 +
  Misc/NEWS          |  4 ++++
  3 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/Include/pyatomic.h b/Include/pyatomic.h
--- a/Include/pyatomic.h
+++ b/Include/pyatomic.h
@@ -64,7 +64,8 @@
     case _Py_memory_order_seq_cst:
         _Py_ANNOTATE_HAPPENS_BEFORE(address);
         break;
-    default:
+    case _Py_memory_order_relaxed:
+    case _Py_memory_order_acquire:
         break;
     }
     switch(order) {
@@ -73,7 +74,8 @@
     case _Py_memory_order_seq_cst:
         _Py_ANNOTATE_HAPPENS_AFTER(address);
         break;
-    default:
+    case _Py_memory_order_relaxed:
+    case _Py_memory_order_release:
         break;
     }
 }
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -124,6 +124,7 @@
 Oleg Broytmann
 Dave Brueck
 Francisco Mart?n Brugu?
+Floris Bruynooghe
 Stan Bubrouski
 Erik de Bueger
 Dick Bulterman
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
+  to allow compiling extension modules with -Wswitch-enum on gcc.
+  Initial patch by Floris Bruynooghe.
+
 - Issue #13333: The UTF-7 decoder now accepts lone surrogates (the encoder
   already accepts them).
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 19 21:10:57 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Sat, 19 Nov 2011 21:10:57 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_branch_3=2E2_=28closes_=2313338=29?=
Message-ID: <E1RRrFZ-0004R7-Qa@dinsdale.python.org>

http://hg.python.org/cpython/rev/c2e588a5237a
changeset:   73625:c2e588a5237a
parent:      73623:71100ef4f7a2
parent:      73624:f855f929bc48
user:        Petri Lehtinen <petri at digip.org>
date:        Sat Nov 19 22:04:33 2011 +0200
summary:
  Merge branch 3.2 (closes #13338)

files:
  Include/pyatomic.h |  6 ++++--
  Misc/ACKS          |  1 +
  Misc/NEWS          |  4 ++++
  3 files changed, 9 insertions(+), 2 deletions(-)


diff --git a/Include/pyatomic.h b/Include/pyatomic.h
--- a/Include/pyatomic.h
+++ b/Include/pyatomic.h
@@ -64,7 +64,8 @@
     case _Py_memory_order_seq_cst:
         _Py_ANNOTATE_HAPPENS_BEFORE(address);
         break;
-    default:
+    case _Py_memory_order_relaxed:
+    case _Py_memory_order_acquire:
         break;
     }
     switch(order) {
@@ -73,7 +74,8 @@
     case _Py_memory_order_seq_cst:
         _Py_ANNOTATE_HAPPENS_AFTER(address);
         break;
-    default:
+    case _Py_memory_order_relaxed:
+    case _Py_memory_order_release:
         break;
     }
 }
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -132,6 +132,7 @@
 Oleg Broytmann
 Dave Brueck
 Francisco Mart?n Brugu?
+Floris Bruynooghe
 Stan Bubrouski
 Erik de Bueger
 Jan-Hein B?hrman
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
+  to allow compiling extension modules with -Wswitch-enum on gcc.
+  Initial patch by Floris Bruynooghe.
+
 - Issue #10227: Add an allocation cache for a single slice object.  Patch by
   Stefan Behnel.
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Nov 20 05:30:06 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 20 Nov 2011 05:30:06 +0100
Subject: [Python-checkins] Daily reference leaks (c2e588a5237a): sum=0
Message-ID: <E1RRz2c-0007yz-Vy@ap.vmr.nerim.net>

results for c2e588a5237a on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogtgKJHN', '-x']

From python-checkins at python.org  Sun Nov 20 12:46:23 2011
From: python-checkins at python.org (nick.coghlan)
Date: Sun, 20 Nov 2011 12:46:23 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Discuss_alternative_terminolog?=
 =?utf8?q?y_in_PEP_395_and_show_an_=27after=27_example_of_what?=
Message-ID: <E1RS5qp-0004vM-8r@dinsdale.python.org>

http://hg.python.org/peps/rev/ab37bff186aa
changeset:   3999:ab37bff186aa
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sun Nov 20 21:46:12 2011 +1000
summary:
  Discuss alternative terminology in PEP 395 and show an 'after' example of what the PEP allows when it comes to command line invocation inside packages

files:
  pep-0395.txt |  67 ++++++++++++++++++++++++++++++++++++++++
  1 files changed, 67 insertions(+), 0 deletions(-)


diff --git a/pep-0395.txt b/pep-0395.txt
--- a/pep-0395.txt
+++ b/pep-0395.txt
@@ -275,6 +275,36 @@
 ``__name__``).
 
 
+Alternative Names
+-----------------
+
+Two alternative names were also considered for the new attribute: "full name"
+(``__fullname__``) and "implementation name" (``__implname__``).
+
+Either of those would actually be valid for the use case in this PEP.
+However, as a meta-issue, it seemed needlessly inconsistent to add *two*
+terms that were essentially "like ``__name__``, but different in some cases
+where ``__name__`` is missing necessary information" and those terms aren't
+accurate for the PEP 3155 function and class use case.
+
+PEP 3155 deliberately omits the module information, so the term "full name"
+is simply untrue, and "implementation name" implies that it may specify an
+object other than that specified by ``__name__``, and that is never the
+case for PEP 3155 (in that PEP, ``__name__`` and ``__qualname__`` always
+refer to the same function or class, it's just that ``__name__`` is
+insufficient to accurately identify nested functions and classes).
+
+If the relative inscrutability of "qualified name" and ``__qualname__``
+encourages interested developers to look them up at least once rather than
+assuming they know what they mean just from the name and guessing wrong,
+that's not necessarily a bad outcome.
+
+Besides, 99% of Python developers should never need to even care these extra
+attributes exist - they're really an implementation detail to let us fix a
+few problematic behaviours exhibited by imports, pickling and introspection,
+not something people are going to be dealing with on a regular basis.
+
+
 Eliminating the Traps
 =====================
 
@@ -390,6 +420,43 @@
 This PEP also proposes that the ``split_path_module()`` functionality be
 exposed directly to Python users via the ``runpy`` module.
 
+With this fix in place, and the same simple package layout described earlier,
+*all* of the following commands would invoke the test suite correctly::
+
+    # working directory: project/package/tests
+    ./test_foo.py
+    python test_foo.py
+    python -m test_foo
+    python -m tests.test_foo
+    python -m package.tests.test_foo
+    python -c "from .test_foo import main; main()"
+    python -c "from ..tests.test_foo import main; main()"
+    python -c "from package.tests.test_foo import main; main()"
+
+    # working directory: project/package
+    tests/test_foo.py
+    python tests/test_foo.py
+    python -m tests.test_foo
+    python -m package.tests.test_foo
+    python -c "from .tests.test_foo import main; main()"
+    python -c "from package.tests.test_foo import main; main()"
+
+    # working directory: project
+    package/tests/test_foo.py
+    python package/tests/test_foo.py
+    python -m package.tests.test_foo
+    python -c "from package.tests.test_foo import main; main()"
+
+    # working directory: project/..
+    project/package/tests/test_foo.py
+    python project/package/tests/test_foo.py
+    # The -m and -c approaches still don't work from here, but the failure
+    # to find 'package' correctly is pretty easy to explain in this case
+
+Depending on the details of how it invokes the script, Idle would likely also
+be able to run ``test_foo.py`` correctly with F5, without needing any Idle
+specific fixes.
+
 
 Compatibility with PEP 382
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Sun Nov 20 16:01:43 2011
From: python-checkins at python.org (eric.araujo)
Date: Sun, 20 Nov 2011 16:01:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_list_of_trove_classi?=
	=?utf8?q?fiers?=
Message-ID: <E1RS8tr-00018y-Vj@dinsdale.python.org>

http://hg.python.org/cpython/rev/f1cafe6f7834
changeset:   73626:f1cafe6f7834
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 20 16:01:35 2011 +0100
summary:
  Update list of trove classifiers

files:
  Lib/packaging/_trove.py |  12 ++++++++++++
  1 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/Lib/packaging/_trove.py b/Lib/packaging/_trove.py
--- a/Lib/packaging/_trove.py
+++ b/Lib/packaging/_trove.py
@@ -47,6 +47,12 @@
 'Framework :: IDLE',
 'Framework :: Paste',
 'Framework :: Plone',
+'Framework :: Plone :: 3.2',
+'Framework :: Plone :: 3.3',
+'Framework :: Plone :: 4.0',
+'Framework :: Plone :: 4.1',
+'Framework :: Plone :: 4.2',
+'Framework :: Plone :: 4.3',
 'Framework :: Pylons',
 'Framework :: Setuptools Plugin',
 'Framework :: Trac',
@@ -261,6 +267,12 @@
 'Programming Language :: Python :: 3.0',
 'Programming Language :: Python :: 3.1',
 'Programming Language :: Python :: 3.2',
+'Programming Language :: Python :: Implementation',
+'Programming Language :: Python :: Implementation :: CPython',
+'Programming Language :: Python :: Implementation :: IronPython',
+'Programming Language :: Python :: Implementation :: Jython',
+'Programming Language :: Python :: Implementation :: PyPy',
+'Programming Language :: Python :: Implementation :: Stackless',
 'Programming Language :: REBOL',
 'Programming Language :: Rexx',
 'Programming Language :: Ruby',

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 19:39:01 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Sun, 20 Nov 2011 19:39:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Make_an_error_m?=
 =?utf8?q?essage_more_understandable_and_consistent_with_other_error?=
Message-ID: <E1RSCI9-0007w3-NW@dinsdale.python.org>

http://hg.python.org/cpython/rev/70a949f2a1cd
changeset:   73627:70a949f2a1cd
branch:      2.7
parent:      73620:de1ecda2afa2
user:        Raymond Hettinger <python at rcn.com>
date:        Sun Nov 20 10:38:53 2011 -0800
summary:
  Make an error message more understandable and consistent with other error messages.

files:
  Objects/abstract.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -156,7 +156,7 @@
                               "be integer, not '%.200s'", key);
     }
 
-    return type_error("'%.200s' object is not subscriptable", o);
+    return type_error("'%.200s' object has no attribute '__getitem__'", o);
 }
 
 int

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 20:05:34 2011
From: python-checkins at python.org (raymond.hettinger)
Date: Sun, 20 Nov 2011 20:05:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Note_the_store?=
 =?utf8?q?=5Ftrue_and_store=5Ffalse_also_create_the_appropriate_defaults?=
 =?utf8?q?=2E?=
Message-ID: <E1RSChq-0001ah-J0@dinsdale.python.org>

http://hg.python.org/cpython/rev/49677cc6d83a
changeset:   73628:49677cc6d83a
branch:      2.7
user:        Raymond Hettinger <python at rcn.com>
date:        Sun Nov 20 11:05:23 2011 -0800
summary:
  Note the store_true and store_false also create the appropriate defaults.

files:
  Doc/library/argparse.rst |  10 ++++++----
  1 files changed, 6 insertions(+), 4 deletions(-)


diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -670,15 +670,17 @@
     >>> parser.parse_args('--foo'.split())
     Namespace(foo=42)
 
-* ``'store_true'`` and ``'store_false'`` - These store the values ``True`` and
-  ``False`` respectively.  These are special cases of ``'store_const'``.  For
-  example::
+* ``'store_true'`` and ``'store_false'`` - These are special cases of
+  ``'store_const'`` using for storing the values ``True`` and ``False``
+  respectively.  In addition, they create default values of *False* and *True*
+  respectively.  For example::
 
     >>> parser = argparse.ArgumentParser()
     >>> parser.add_argument('--foo', action='store_true')
     >>> parser.add_argument('--bar', action='store_false')
+    >>> parser.add_argument('--baz', action='store_false')
     >>> parser.parse_args('--foo --bar'.split())
-    Namespace(bar=False, foo=True)
+    Namespace(bar=False, baz=True, foo=True)
 
 * ``'append'`` - This stores a list, and appends each argument value to the
   list.  This is useful to allow an option to be specified multiple times.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:18 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_charmap_encoders_uses_Py=5F?=
	=?utf8?q?UCS4=2C_not_Py=5FUNICODE?=
Message-ID: <E1RSFKA-0006FE-1F@dinsdale.python.org>

http://hg.python.org/cpython/rev/0045a2d1506a
changeset:   73629:0045a2d1506a
parent:      73626:f1cafe6f7834
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 17:09:18 2011 +0100
summary:
  charmap encoders uses Py_UCS4, not Py_UNICODE

files:
  Objects/unicodeobject.c |  11 ++++-------
  1 files changed, 4 insertions(+), 7 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -7934,7 +7934,7 @@
 }
 
 static int
-encoding_map_lookup(Py_UNICODE c, PyObject *mapping)
+encoding_map_lookup(Py_UCS4 c, PyObject *mapping)
 {
     struct encoding_map *map = (struct encoding_map*)mapping;
     int l1 = c>>11;
@@ -7942,11 +7942,8 @@
     int l3 = c & 0x7F;
     int i;
 
-#ifdef Py_UNICODE_WIDE
-    if (c > 0xFFFF) {
+    if (c > 0xFFFF)
         return -1;
-    }
-#endif
     if (c == 0)
         return 0;
     /* level 1*/
@@ -7971,7 +7968,7 @@
    can't be found, Py_None is returned (or NULL, if another
    error occurred). */
 static PyObject *
-charmapencode_lookup(Py_UNICODE c, PyObject *mapping)
+charmapencode_lookup(Py_UCS4 c, PyObject *mapping)
 {
     PyObject *w = PyLong_FromLong((long)c);
     PyObject *x;
@@ -8036,7 +8033,7 @@
    (in which case no character was written) or NULL, if a
    reallocation error occurred. The caller must decref the result */
 static charmapencode_result
-charmapencode_output(Py_UNICODE c, PyObject *mapping,
+charmapencode_output(Py_UCS4 c, PyObject *mapping,
                      PyObject **outobj, Py_ssize_t *outpos)
 {
     PyObject *rep;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:18 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_PyUnicode=5FEncodeCodeP?=
 =?utf8?q?age=28=29_instead_of_PyUnicode=5FEncodeMBCS=28=29_with?=
Message-ID: <E1RSFKA-0006FO-R7@dinsdale.python.org>

http://hg.python.org/cpython/rev/8e0f91448d36
changeset:   73630:8e0f91448d36
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:27:03 2011 +0100
summary:
  Use PyUnicode_EncodeCodePage() instead of PyUnicode_EncodeMBCS() with
PyUnicode_AsUnicodeAndSize()

files:
  Objects/unicodeobject.c |  19 +++----------------
  1 files changed, 3 insertions(+), 16 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -3051,13 +3051,7 @@
 PyUnicode_EncodeFSDefault(PyObject *unicode)
 {
 #ifdef HAVE_MBCS
-    const Py_UNICODE *wstr;
-    Py_ssize_t wlen;
-
-    wstr = PyUnicode_AsUnicodeAndSize(unicode, &wlen);
-    if (wstr == NULL)
-        return NULL;
-    return PyUnicode_EncodeMBCS(wstr, wlen, NULL);
+    return PyUnicode_EncodeCodePage(CP_ACP, unicode, NULL);
 #elif defined(__APPLE__)
     return _PyUnicode_AsUTF8String(unicode, "surrogateescape");
 #else
@@ -3141,15 +3135,8 @@
                  (strcmp(lower, "iso-8859-1") == 0))
             return _PyUnicode_AsLatin1String(unicode, errors);
 #ifdef HAVE_MBCS
-        else if (strcmp(lower, "mbcs") == 0) {
-            const Py_UNICODE *wstr;
-            Py_ssize_t wlen;
-
-            wstr = PyUnicode_AsUnicodeAndSize(unicode, &wlen);
-            if (wstr == NULL)
-                return NULL;
-            return PyUnicode_EncodeMBCS(wstr, wlen, errors);
-        }
+        else if (strcmp(lower, "mbcs") == 0)
+            return PyUnicode_EncodeCodePage(CP_ACP, unicode, errors);
 #endif
         else if (strcmp(lower, "ascii") == 0)
             return _PyUnicode_AsASCIIString(unicode, errors);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:19 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:19 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Document_PyUnicode=5FWCHAR?=
	=?utf8?q?=5FKIND?=
Message-ID: <E1RSFKB-0006FS-Dy@dinsdale.python.org>

http://hg.python.org/cpython/rev/991456c523eb
changeset:   73631:991456c523eb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:27:28 2011 +0100
summary:
  Document PyUnicode_WCHAR_KIND

files:
  Doc/c-api/unicode.rst |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -129,7 +129,8 @@
    .. versionadded:: 3.3
 
 
-.. c:macro:: PyUnicode_1BYTE_KIND
+.. c:macro:: PyUnicode_WCHAR_KIND
+             PyUnicode_1BYTE_KIND
              PyUnicode_2BYTE_KIND
              PyUnicode_4BYTE_KIND
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:20 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Document_new_and_deprecated?=
	=?utf8?q?_Unicode_functions?=
Message-ID: <E1RSFKC-0006FX-5g@dinsdale.python.org>

http://hg.python.org/cpython/rev/a371e65feefb
changeset:   73632:a371e65feefb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:27:55 2011 +0100
summary:
  Document new and deprecated Unicode functions

files:
  Doc/whatsnew/3.3.rst |  103 +++++++++++++++++++++++++++---
  1 files changed, 92 insertions(+), 11 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -209,10 +209,8 @@
 (:issue:`13021`)
 
 
-New, Improved, and Deprecated Modules
-=====================================
-
-* Stub
+New and Improved Modules
+========================
 
 array
 -----
@@ -579,7 +577,11 @@
 
 Major performance enhancements have been added:
 
-* Stub
+* Thanks to the :pep:`393`, some operations on Unicode strings has been optimized:
+
+  * the memory footprint is divided by 2 to 4 depending on the text
+  * getting a substring of a latin1 strings is 4 times faster
+  * TODO
 
 
 Build and C API Changes
@@ -587,7 +589,27 @@
 
 Changes to Python's build process and to the C API include:
 
-* Stub
+* The :pep:`393` added new Unicode types, macros and functions:
+
+  * Py_UCS1, Py_UCS2, Py_UCS4 types
+  * PyASCIIObject and PyCompactUnicodeObject structures
+  * :c:func:`PyUnicode_New`
+  * :c:macro:`PyUnicode_READY`
+  * :c:func:`PyUnicode_FromKindAndData`
+  * :c:func:`PyUnicode_GetLength`, :c:macro:`PyUnicode_GET_LENGTH`
+  * :c:func:`PyUnicode_CopyCharacters`
+  * :c:func:`PyUnicode_ReadChar`, :c:func:`PyUnicode_WriteChar`
+  * :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsUCS4Copy`
+  * :c:func:`PyUnicode_FindChar`
+  * :c:func:`PyUnicode_Substring`
+  * :c:macro:`PyUnicode_1BYTE_DATA`, :c:macro:`PyUnicode_2BYTE_DATA`,
+    :c:macro:`PyUnicode_4BYTE_DATA`
+  * :c:macro:`PyUnicode_KIND` with :c:type:`PyUnicode_Kind` enum:
+    :c:data:`PyUnicode_WCHAR_KIND`, :c:data:`PyUnicode_1BYTE_KIND`,
+    :c:data:`PyUnicode_2BYTE_KIND`, :c:data:`PyUnicode_4BYTE_KIND`
+  * :c:macro:`PyUnicode_DATA`
+  * :c:macro:`PyUnicode_READ`, :c:macro:`PyUnicode_READ_CHAR`, :c:macro:`PyUnicode_WRITE`
+  * :c:macro:`PyUnicode_MAX_CHAR_VALUE`
 
 
 Unsupported Operating Systems
@@ -599,22 +621,81 @@
 are no longer supported due to maintenance burden.
 
 
-Deprecated modules, functions and methods
-=========================================
+Deprecated Python modules, functions and methods
+================================================
 
 * The :mod:`packaging` module replaces the :mod:`distutils` module
 * The ``unicode_internal`` codec has been deprecated because of the
   :pep:`393`, use UTF-8, UTF-16 (``utf-16-le`` or ``utf-16-le``), or UTF-32
-  (``utf-32-le`` or ``utf-32-le``) instead.
+  (``utf-32-le`` or ``utf-32-le``)
 * :meth:`ftplib.FTP.nlst` and :meth:`ftplib.FTP.dir`: use
-  :meth:`ftplib.FTP.mlsd` instead.
+  :meth:`ftplib.FTP.mlsd`
 * :func:`platform.popen`: use the :mod:`subprocess` module. Check especially
   the :ref:`subprocess-replacements` section.
 * :issue:`13374`: The Windows bytes API has been deprecated in the :mod:`os`
-  module. Use Unicode filenames instead of bytes filenames to not depend on
+  module. Use Unicode filenames, instead of bytes filenames, to not depend on
   the ANSI code page anymore and to support any filename.
 
 
+Deprecated functions and types of the C API
+===========================================
+
+The :c:type:`Py_UNICODE` has been deprecated by the :pep:`393` and will be
+removed in Python 4. All functions using this type are deprecated:
+
+Functions and macros manipulating Py_UNICODE* strings:
+
+ * :c:macro:`Py_UNICODE_strlen`: use :c:func:`PyUnicode_GetLength` or
+   :c:macro:`PyUnicode_GET_LENGTH`
+ * :c:macro:`Py_UNICODE_strcat`: use :c:func:`PyUnicode_CopyCharacters` or
+   :c:func:`PyUnicode_FromFormat`
+ * :c:macro:`Py_UNICODE_strcpy`, :c:macro:`Py_UNICODE_strncpy`,
+   :c:macro:`Py_UNICODE_COPY`: use :c:func:`PyUnicode_CopyCharacters` or
+   :c:func:`PyUnicode_Substring`
+ * :c:macro:`Py_UNICODE_strcmp`: use :c:func:`PyUnicode_Compare`
+ * :c:macro:`Py_UNICODE_strncmp`: use :c:func:`PyUnicode_Tailmatch`
+ * :c:macro:`Py_UNICODE_strchr`, :c:macro:`Py_UNICODE_strrchr`: use
+   :c:func:`PyUnicode_FindChar`
+ * :c:macro:`Py_UNICODE_FILL`
+
+Unicode functions and methods using :c:type:`Py_UNICODE` and
+:c:type:`Py_UNICODE*` types:
+
+ * :c:macro:`PyUnicode_FromUnicode`: use :c:func:`PyUnicode_FromWideChar` or
+   :c:func:`PyUnicode_FromKindAndData`
+ * :c:macro:`PyUnicode_AS_UNICODE`, :c:func:`PyUnicode_AsUnicode`,
+   :c:func:`PyUnicode_AsUnicodeAndSize`: use :c:func:`PyUnicode_AsWideCharString`
+ * :c:macro:`PyUnicode_AS_DATA`: use :c:macro:`PyUnicode_DATA` with
+   :c:macro:`PyUnicode_READ` and :c:macro:`PyUnicode_WRITE`
+ * :c:macro:`PyUnicode_GET_SIZE`, :c:func:`PyUnicode_GetSize`: use
+   :c:macro:`PyUnicode_GET_LENGTH` or :c:func:`PyUnicode_GetLength`
+ * :c:macro:`PyUnicode_GET_DATA_SIZE`: use
+   ``PyUnicode_GET_LENGTH(str) * PyUnicode_KIND(str)`` (only work on ready
+   strings)
+ * :c:func:`PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy`,
+   :c:func:`PyUnicode_AsWideCharString` or :c:func:`PyUnicode_Copy`
+
+Encoders:
+
+ * :c:func:`PyUnicode_Encode`: use :c:func:`PyUnicode_AsEncodedObject`
+ * :c:func:`PyUnicode_EncodeUTF7`
+ * :c:func:`PyUnicode_EncodeUTF8`: use :c:func:`PyUnicode_AsUTF8String`
+ * :c:func:`PyUnicode_EncodeUTF32`
+ * :c:func:`PyUnicode_EncodeUTF16`
+ * :c:func:`PyUnicode_EncodeUnicodeEscape:` use
+   :c:func:`PyUnicode_AsUnicodeEscapeString`
+ * :c:func:`PyUnicode_EncodeRawUnicodeEscape:` use
+   :c:func:`PyUnicode_AsRawUnicodeEscapeString`
+ * :c:func:`PyUnicode_EncodeLatin1`: use :c:func:`PyUnicode_AsLatin1String`
+ * :c:func:`PyUnicode_EncodeASCII`: use :c:func:`PyUnicode_AsASCIIString`
+ * :c:func:`PyUnicode_EncodeCharmap`
+ * :c:func:`PyUnicode_TranslateCharmap`
+ * :c:func:`PyUnicode_EncodeMBCS`: use :c:func:`PyUnicode_AsMBCSString` or
+   :c:func:`PyUnicode_EncodeCodePage` (with ``CP_ACP`` code_page)
+ * :c:func:`PyUnicode_EncodeDecimal`,
+   :c:func:`PyUnicode_TransformDecimalToASCII`
+
+
 Porting to Python 3.3
 =====================
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:20 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Ensure_that_Py=5FUCS4_is_32?=
 =?utf8?q?_bits_and_Py=5FUCS2_is_16_bits?=
Message-ID: <E1RSFKC-0006Fb-Tv@dinsdale.python.org>

http://hg.python.org/cpython/rev/1dd575504607
changeset:   73633:1dd575504607
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:28:29 2011 +0100
summary:
  Ensure that Py_UCS4 is 32 bits and Py_UCS2 is 16 bits

files:
  Include/unicodeobject.h |  9 +++++++--
  1 files changed, 7 insertions(+), 2 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -117,15 +117,20 @@
 
 /* Py_UCS4 and Py_UCS2 are typedefs for the respective
    unicode representations. */
-#if SIZEOF_INT >= 4
+#if SIZEOF_INT == 4
 typedef unsigned int Py_UCS4;
-#elif SIZEOF_LONG >= 4
+#elif SIZEOF_LONG == 4
 typedef unsigned long Py_UCS4;
 #else
 #error "Could not find a proper typedef for Py_UCS4"
 #endif
 
+#if SIZEOF_SHORT == 2
 typedef unsigned short Py_UCS2;
+#else
+#error "Could not find a proper typedef for Py_UCS2"
+#endif
+
 typedef unsigned char Py_UCS1;
 
 /* --- Internal Unicode Operations ---------------------------------------- */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:21 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:21 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_charmap=5Fencoding=5Ferror?=
 =?utf8?q?=28=29_uses_the_new_Unicode_API?=
Message-ID: <E1RSFKD-0006Ff-Me@dinsdale.python.org>

http://hg.python.org/cpython/rev/9599b0346505
changeset:   73634:9599b0346505
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:28:55 2011 +0100
summary:
  charmap_encoding_error() uses the new Unicode API

files:
  Objects/unicodeobject.c |  16 +++++++++++++---
  1 files changed, 13 insertions(+), 3 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8087,7 +8087,9 @@
     PyObject *repunicode = NULL; /* initialize to prevent gcc warning */
     Py_ssize_t size, repsize;
     Py_ssize_t newpos;
-    Py_UNICODE *uni2;
+    enum PyUnicode_Kind kind;
+    void *data;
+    Py_ssize_t index;
     /* startpos for collecting unencodable chars */
     Py_ssize_t collstartpos = *inpos;
     Py_ssize_t collendpos = *inpos+1;
@@ -8201,10 +8203,18 @@
             break;
         }
         /* generate replacement  */
+        if (PyUnicode_READY(repunicode) < 0) {
+            Py_DECREF(repunicode);
+            return -1;
+        }
         repsize = PyUnicode_GET_SIZE(repunicode);
-        for (uni2 = PyUnicode_AS_UNICODE(repunicode); repsize-->0; ++uni2) {
-            x = charmapencode_output(*uni2, mapping, res, respos);
+        data = PyUnicode_DATA(repunicode);
+        kind = PyUnicode_KIND(repunicode);
+        for (index = 0; index < repsize; index++) {
+            Py_UCS4 repch = PyUnicode_READ(kind, data, index);
+            x = charmapencode_output(repch, mapping, res, respos);
             if (x==enc_EXCEPTION) {
+                Py_DECREF(repunicode);
                 return -1;
             }
             else if (x==enc_FAILED) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:22 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:22 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_PyUnicode=5FCopyCharact?=
	=?utf8?b?ZXJzKCkgZG9j?=
Message-ID: <E1RSFKE-0006Fk-Im@dinsdale.python.org>

http://hg.python.org/cpython/rev/28f2c1c1b7c1
changeset:   73635:28f2c1c1b7c1
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:29:14 2011 +0100
summary:
  Fix PyUnicode_CopyCharacters() doc

files:
  Include/unicodeobject.h |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -756,8 +756,7 @@
    PyUnicode_FromUnicode). Unicode objects allocated by the new API (e.g.
    PyUnicode_New) cannot be resized by this function.
 
-   The length is a number of Py_UNICODE characters (and not the number of code
-   points).
+   The length is a number of characters (and not the number of Py_UNICODE characters).
 
    *unicode is modified to point to the new (resized) object and 0
    returned on success.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:23 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:23 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Reuse_surrogate_macros_in_U?=
	=?utf8?q?TF-16_decoder?=
Message-ID: <E1RSFKF-0006Fo-Df@dinsdale.python.org>

http://hg.python.org/cpython/rev/99e2ccd351c0
changeset:   73636:99e2ccd351c0
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:40:27 2011 +0100
summary:
  Reuse surrogate macros in UTF-16 decoder

files:
  Objects/unicodeobject.c |  9 ++++-----
  1 files changed, 4 insertions(+), 5 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -5339,13 +5339,12 @@
             endinpos = ((const char *)e) + 1 - starts;
             goto utf16Error;
         }
-        if (0xD800 <= ch && ch <= 0xDBFF) {
-            Py_UNICODE ch2 = (q[ihi] << 8) | q[ilo];
+        if (Py_UNICODE_IS_HIGH_SURROGATE(ch)) {
+            Py_UCS4 ch2 = (q[ihi] << 8) | q[ilo];
             q += 2;
-            if (0xDC00 <= ch2 && ch2 <= 0xDFFF) {
+            if (Py_UNICODE_IS_LOW_SURROGATE(ch2)) {
                 if (unicode_putchar(&unicode, &outpos,
-                                    (((ch & 0x3FF)<<10) |
-                                     (ch2 & 0x3FF)) + 0x10000) < 0)
+                                    Py_UNICODE_JOIN_SURROGATES(ch, ch2)) < 0)
                     goto onError;
                 continue;
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:24 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:24 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_the_two_ugly_and_unu?=
 =?utf8?q?sed_WRITE=5FASCII=5FOR=5FWSTR_and_WRITE=5FWSTR_macros?=
Message-ID: <E1RSFKG-0006Fs-6a@dinsdale.python.org>

http://hg.python.org/cpython/rev/71cf0ea0b341
changeset:   73637:71cf0ea0b341
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:41:31 2011 +0100
summary:
  Remove the two ugly and unused WRITE_ASCII_OR_WSTR and WRITE_WSTR macros

files:
  Objects/unicodeobject.c |  18 ------------------
  1 files changed, 0 insertions(+), 18 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -5589,21 +5589,6 @@
     return length;
 }
 
-/* Similar to PyUnicode_WRITE but either write into wstr field
-   or treat string as ASCII. */
-#define WRITE_ASCII_OR_WSTR(kind, buf, index, value) \
-    do { \
-        if ((kind) != PyUnicode_WCHAR_KIND) \
-            ((unsigned char *)(buf))[(index)] = (unsigned char)(value); \
-        else \
-            ((Py_UNICODE *)(buf))[(index)] = (Py_UNICODE)(value); \
-    } while (0)
-
-#define WRITE_WSTR(buf, index, value) \
-    assert(kind == PyUnicode_WCHAR_KIND), \
-    ((Py_UNICODE *)(buf))[(index)] = (Py_UNICODE)(value)
-
-
 static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL;
 
 PyObject *
@@ -5873,9 +5858,6 @@
     return NULL;
 }
 
-#undef WRITE_ASCII_OR_WSTR
-#undef WRITE_WSTR
-
 /* Return a Unicode-Escape string version of the Unicode object.
 
    If quotes is true, the string is enclosed in u"" or u'' quotes as

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:24 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:24 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=5FPyUnicode=5FCheckConsist?=
 =?utf8?q?ency=28=29_also_checks_maxchar_maximum_value=2C?=
Message-ID: <E1RSFKG-0006Fw-R8@dinsdale.python.org>

http://hg.python.org/cpython/rev/26af48f65ef3
changeset:   73638:26af48f65ef3
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 18:56:05 2011 +0100
summary:
  _PyUnicode_CheckConsistency() also checks maxchar maximum value,
not only its minimum value

files:
  Include/unicodeobject.h |  13 ++++++++-----
  Objects/unicodeobject.c |  12 +++++++++---
  2 files changed, 17 insertions(+), 8 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -303,19 +303,22 @@
            - PyUnicode_1BYTE_KIND (1):
 
              * character type = Py_UCS1 (8 bits, unsigned)
-             * if ascii is set, all characters must be in range
-               U+0000-U+007F, otherwise at least one character must be in range
+             * all characters are in the range U+0000-U+00FF (latin1)
+             * if ascii is set, all characters are in the range U+0000-U+007F
+               (ASCII), otherwise at least one character is in the range
                U+0080-U+00FF
 
            - PyUnicode_2BYTE_KIND (2):
 
              * character type = Py_UCS2 (16 bits, unsigned)
-             * at least one character must be in range U+0100-U+FFFF
+             * all characters are in the range U+0000-U+FFFF (BMP)
+             * at least one character is in the range U+0100-U+FFFF
 
            - PyUnicode_4BYTE_KIND (4):
 
              * character type = Py_UCS4 (32 bits, unsigned)
-             * at least one character must be in range U+10000-U+10FFFF
+             * all characters are in the range U+0000-U+10FFFF
+             * at least one character is in the range U+10000-U+10FFFF
          */
         unsigned int kind:3;
         /* Compact is with respect to the allocation scheme. Compact unicode
@@ -323,7 +326,7 @@
            one block for the PyUnicodeObject struct and another for its data
            buffer. */
         unsigned int compact:1;
-        /* The string only contains characters in range U+0000-U+007F (ASCII)
+        /* The string only contains characters in the range U+0000-U+007F (ASCII)
            and the kind is PyUnicode_1BYTE_KIND. If ascii is set and compact is
            set, use the PyASCIIObject structure. */
         unsigned int ascii:1;
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -392,15 +392,21 @@
                 maxchar = ch;
         }
         if (kind == PyUnicode_1BYTE_KIND) {
-            if (ascii->state.ascii == 0)
+            if (ascii->state.ascii == 0) {
                 assert(maxchar >= 128);
+                assert(maxchar <= 255);
+            }
             else
                 assert(maxchar < 128);
         }
-        else if (kind == PyUnicode_2BYTE_KIND)
+        else if (kind == PyUnicode_2BYTE_KIND) {
             assert(maxchar >= 0x100);
-        else
+            assert(maxchar <= 0xFFFF);
+        }
+        else {
             assert(maxchar >= 0x10000);
+            assert(maxchar <= 0x10FFFF);
+        }
     }
     if (check_content && !unicode_is_singleton(op))
         assert(ascii->hash == -1);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:25 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FAsUTF32String?=
 =?utf8?q?=28=29_calls_directly_=5FPyUnicode=5FEncodeUTF32=28=29=2C?=
Message-ID: <E1RSFKH-0006G1-Kg@dinsdale.python.org>

http://hg.python.org/cpython/rev/459336931f19
changeset:   73639:459336931f19
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 19:12:52 2011 +0100
summary:
  PyUnicode_AsUTF32String() calls directly _PyUnicode_EncodeUTF32(),
instead of calling the deprecated PyUnicode_EncodeUTF32() function

files:
  Objects/unicodeobject.c |  7 +------
  1 files changed, 1 insertions(+), 6 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -5150,12 +5150,7 @@
 PyObject *
 PyUnicode_AsUTF32String(PyObject *unicode)
 {
-    const Py_UNICODE *wstr;
-    Py_ssize_t wlen;
-    wstr = PyUnicode_AsUnicodeAndSize(unicode, &wlen);
-    if (wstr == NULL)
-        return NULL;
-    return PyUnicode_EncodeUTF32(wstr, wlen, NULL, 0);
+    return _PyUnicode_EncodeUTF32(unicode, NULL, 0);
 }
 
 /* --- UTF-16 Codec ------------------------------------------------------- */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:26 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_print=5Fexception=28=29_use?=
 =?utf8?q?s_PyUnicode=5FGetLength=28=29_instead_of_PyUnicode=5FGetSize=28?=
 =?utf8?q?=29?=
Message-ID: <E1RSFKI-0006G5-7M@dinsdale.python.org>

http://hg.python.org/cpython/rev/c4c1c8af0115
changeset:   73640:c4c1c8af0115
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 19:20:00 2011 +0100
summary:
  print_exception() uses PyUnicode_GetLength() instead of PyUnicode_GetSize()

files:
  Python/pythonrun.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -1641,7 +1641,7 @@
         if (s == NULL)
             err = -1;
         else if (!PyUnicode_Check(s) ||
-            PyUnicode_GetSize(s) != 0)
+            PyUnicode_GetLength(s) != 0)
             err = PyFile_WriteString(": ", f);
         if (err == 0)
           err = PyFile_WriteObject(s, f, Py_PRINT_RAW);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:26 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_stringlib=3A_remove_unused_?=
	=?utf8?q?STRINGLIB=5FFILL?=
Message-ID: <E1RSFKI-0006G9-QD@dinsdale.python.org>

http://hg.python.org/cpython/rev/6b3e871c003c
changeset:   73641:6b3e871c003c
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 19:30:15 2011 +0100
summary:
  stringlib: remove unused STRINGLIB_FILL

files:
  Objects/stringlib/asciilib.h    |  1 -
  Objects/stringlib/stringdefs.h  |  1 -
  Objects/stringlib/ucs1lib.h     |  1 -
  Objects/stringlib/ucs2lib.h     |  1 -
  Objects/stringlib/ucs4lib.h     |  1 -
  Objects/stringlib/unicodedefs.h |  1 -
  6 files changed, 0 insertions(+), 6 deletions(-)


diff --git a/Objects/stringlib/asciilib.h b/Objects/stringlib/asciilib.h
--- a/Objects/stringlib/asciilib.h
+++ b/Objects/stringlib/asciilib.h
@@ -17,7 +17,6 @@
 #define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
 #define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
 #define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
-#define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_1BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
 #define STRINGLIB_NEW            unicode_fromascii
diff --git a/Objects/stringlib/stringdefs.h b/Objects/stringlib/stringdefs.h
--- a/Objects/stringlib/stringdefs.h
+++ b/Objects/stringlib/stringdefs.h
@@ -20,7 +20,6 @@
 #define STRINGLIB_TODECIMAL(x)   (STRINGLIB_ISDECIMAL(x) ? (x - '0') : -1)
 #define STRINGLIB_TOUPPER        Py_TOUPPER
 #define STRINGLIB_TOLOWER        Py_TOLOWER
-#define STRINGLIB_FILL           memset
 #define STRINGLIB_STR            PyBytes_AS_STRING
 #define STRINGLIB_LEN            PyBytes_GET_SIZE
 #define STRINGLIB_NEW            PyBytes_FromStringAndSize
diff --git a/Objects/stringlib/ucs1lib.h b/Objects/stringlib/ucs1lib.h
--- a/Objects/stringlib/ucs1lib.h
+++ b/Objects/stringlib/ucs1lib.h
@@ -17,7 +17,6 @@
 #define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
 #define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
 #define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
-#define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_1BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
 #define STRINGLIB_NEW            _PyUnicode_FromUCS1
diff --git a/Objects/stringlib/ucs2lib.h b/Objects/stringlib/ucs2lib.h
--- a/Objects/stringlib/ucs2lib.h
+++ b/Objects/stringlib/ucs2lib.h
@@ -17,7 +17,6 @@
 #define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
 #define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
 #define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
-#define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_2BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
 #define STRINGLIB_NEW            _PyUnicode_FromUCS2
diff --git a/Objects/stringlib/ucs4lib.h b/Objects/stringlib/ucs4lib.h
--- a/Objects/stringlib/ucs4lib.h
+++ b/Objects/stringlib/ucs4lib.h
@@ -17,7 +17,6 @@
 #define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
 #define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
 #define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
-#define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_4BYTE_DATA
 #define STRINGLIB_LEN            PyUnicode_GET_LENGTH
 #define STRINGLIB_NEW            _PyUnicode_FromUCS4
diff --git a/Objects/stringlib/unicodedefs.h b/Objects/stringlib/unicodedefs.h
--- a/Objects/stringlib/unicodedefs.h
+++ b/Objects/stringlib/unicodedefs.h
@@ -20,7 +20,6 @@
 #define STRINGLIB_TODECIMAL      Py_UNICODE_TODECIMAL
 #define STRINGLIB_TOUPPER        Py_UNICODE_TOUPPER
 #define STRINGLIB_TOLOWER        Py_UNICODE_TOLOWER
-#define STRINGLIB_FILL           Py_UNICODE_FILL
 #define STRINGLIB_STR            PyUnicode_AS_UNICODE
 #define STRINGLIB_LEN            PyUnicode_GET_SIZE
 #define STRINGLIB_NEW            PyUnicode_FromUnicode

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:27 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:27 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_PyUnicode=5FCompareWith?=
 =?utf8?q?ASCIIString=28=29_instead_of_Py=5FUNICODE=5Fstrcmp=28=29_to_avoi?=
 =?utf8?q?d?=
Message-ID: <E1RSFKJ-0006GW-D6@dinsdale.python.org>

http://hg.python.org/cpython/rev/4700e768c7b5
changeset:   73642:4700e768c7b5
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 19:32:09 2011 +0100
summary:
  Use PyUnicode_CompareWithASCIIString() instead of Py_UNICODE_strcmp() to avoid
the deprecate Py_UNICODE type

files:
  Modules/_testcapimodule.c |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1803,13 +1803,12 @@
 {
     PyObject *result;
     char *msg;
-    static const Py_UNICODE one[] = {'1', 0};
 
 #define CHECK_1_FORMAT(FORMAT, TYPE)                                \
     result = PyUnicode_FromFormat(FORMAT, (TYPE)1);                 \
     if (result == NULL)                                             \
         return NULL;                                                \
-    if (Py_UNICODE_strcmp(PyUnicode_AS_UNICODE(result), one)) {     \
+    if (PyUnicode_CompareWithASCIIString(result, "1")) {     \
         msg = FORMAT " failed at 1";                                \
         goto Fail;                                                  \
     }                                                               \

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:28 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:28 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyObject=5FStr=28=29_ensure?=
 =?utf8?q?s_that_the_result_string_is_ready?=
Message-ID: <E1RSFKK-0006Gb-1i@dinsdale.python.org>

http://hg.python.org/cpython/rev/701165dfaae5
changeset:   73643:701165dfaae5
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 19:48:36 2011 +0100
summary:
  PyObject_Str() ensures that the result string is ready
and check the string consistency.

_PyUnicode_CheckConsistency() doesn't check the hash anymore. It should be
possible to call this function even if hash(str) was already called.

files:
  Objects/object.c        |  5 +++++
  Objects/unicodeobject.c |  2 --
  2 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -403,6 +403,8 @@
     if (v == NULL)
         return PyUnicode_FromString("<NULL>");
     if (PyUnicode_CheckExact(v)) {
+        if (PyUnicode_READY(v) < 0)
+            return NULL;
         Py_INCREF(v);
         return v;
     }
@@ -424,6 +426,9 @@
         Py_DECREF(res);
         return NULL;
     }
+    if (PyUnicode_READY(res) < 0)
+        return NULL;
+    assert(_PyUnicode_CheckConsistency(res, 1));
     return res;
 }
 
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -408,8 +408,6 @@
             assert(maxchar <= 0x10FFFF);
         }
     }
-    if (check_content && !unicode_is_singleton(op))
-        assert(ascii->hash == -1);
     return 1;
 }
 #endif

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 22:53:28 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 22:53:28 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_UnicodeEncodeError_uses_the?=
	=?utf8?q?_new_Unicode_API?=
Message-ID: <E1RSFKK-0006Gf-Ko@dinsdale.python.org>

http://hg.python.org/cpython/rev/5b426bf9260c
changeset:   73644:5b426bf9260c
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 22:50:23 2011 +0100
summary:
  UnicodeEncodeError uses the new Unicode API

The index is a character index, not a index in a Py_UNICODE* string.

files:
  Objects/exceptions.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -1542,8 +1542,8 @@
     if (encoding_str == NULL)
         goto done;
 
-    if (uself->start < PyUnicode_GET_SIZE(uself->object) && uself->end == uself->start+1) {
-        int badchar = (int)PyUnicode_AS_UNICODE(uself->object)[uself->start];
+    if (uself->start < PyUnicode_GET_LENGTH(uself->object) && uself->end == uself->start+1) {
+        Py_UCS4 badchar = PyUnicode_ReadChar(uself->object, uself->start);
         const char *fmt;
         if (badchar <= 0xff)
             fmt = "'%U' codec can't encode character '\\x%02x' in position %zd: %U";
@@ -1554,7 +1554,7 @@
         result = PyUnicode_FromFormat(
             fmt,
             encoding_str,
-            badchar,
+            (int)badchar,
             uself->start,
             reason_str);
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 23:09:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 23:09:39 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogQ2xvc2UgIzEzNDAx?=
 =?utf8?q?=3A_Skip_TestFileTypeW_of_test=5Fargparse_if_the_current_user_is?=
 =?utf8?q?_root?=
Message-ID: <E1RSFZz-0007IH-IB@dinsdale.python.org>

http://hg.python.org/cpython/rev/ba3da86d1c5d
changeset:   73645:ba3da86d1c5d
branch:      3.2
parent:      73624:f855f929bc48
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 23:09:09 2011 +0100
summary:
  Close #13401: Skip TestFileTypeW of test_argparse if the current user is root

Patch written by Arfrever Frehtes Taifersar Arahesis.

files:
  Lib/test/test_argparse.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py
--- a/Lib/test/test_argparse.py
+++ b/Lib/test/test_argparse.py
@@ -1502,6 +1502,8 @@
         return self.name == other.name
 
 
+ at unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+                 "non-root user required")
 class TestFileTypeW(TempDirMixin, ParserTestCase):
     """Test the FileType option/argument type for writing files"""
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 23:09:40 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 23:09:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=28Merge_3=2E2=29_Close_=2313401=3A_Skip_TestFileTypeW_of_te?=
 =?utf8?q?st=5Fargparse_if_the_current?=
Message-ID: <E1RSFa0-0007IL-BK@dinsdale.python.org>

http://hg.python.org/cpython/rev/50e788691eda
changeset:   73646:50e788691eda
parent:      73644:5b426bf9260c
parent:      73645:ba3da86d1c5d
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 23:10:12 2011 +0100
summary:
  (Merge 3.2) Close #13401: Skip TestFileTypeW of test_argparse if the current user is root

Patch written by Arfrever Frehtes Taifersar Arahesis.

files:
  Lib/test/test_argparse.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py
--- a/Lib/test/test_argparse.py
+++ b/Lib/test/test_argparse.py
@@ -1502,6 +1502,8 @@
         return self.name == other.name
 
 
+ at unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+                 "non-root user required")
 class TestFileTypeW(TempDirMixin, ParserTestCase):
     """Test the FileType option/argument type for writing files"""
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 20 23:09:41 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 20 Nov 2011 23:09:41 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogQ2xvc2UgIzEzNDAx?=
 =?utf8?q?=3A_Skip_TestFileTypeW_of_test=5Fargparse_if_the_current_user_is?=
 =?utf8?q?_root?=
Message-ID: <E1RSFa1-0007IP-CX@dinsdale.python.org>

http://hg.python.org/cpython/rev/6e1e1118adca
changeset:   73647:6e1e1118adca
branch:      2.7
parent:      73628:49677cc6d83a
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 20 23:09:09 2011 +0100
summary:
  Close #13401: Skip TestFileTypeW of test_argparse if the current user is root

Patch written by Arfrever Frehtes Taifersar Arahesis.

files:
  Lib/test/test_argparse.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py
--- a/Lib/test/test_argparse.py
+++ b/Lib/test/test_argparse.py
@@ -1509,6 +1509,8 @@
         return self.name == other.name
 
 
+ at unittest.skipIf(hasattr(os, 'geteuid') and os.geteuid() == 0,
+                 "non-root user required")
 class TestFileTypeW(TempDirMixin, ParserTestCase):
     """Test the FileType option/argument type for writing files"""
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 01:11:40 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 01:11:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313441=3A_Disable_t?=
 =?utf8?q?emporary_strxfrm=28=29_tests_on_Solaris?=
Message-ID: <E1RSHU4-0007Sx-HK@dinsdale.python.org>

http://hg.python.org/cpython/rev/31baf1363ba1
changeset:   73648:31baf1363ba1
parent:      73646:50e788691eda
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 01:02:23 2011 +0100
summary:
  Issue #13441: Disable temporary strxfrm() tests on Solaris

files:
  Lib/test/test_locale.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py
--- a/Lib/test/test_locale.py
+++ b/Lib/test/test_locale.py
@@ -343,6 +343,8 @@
         self.assertEqual(locale.strcoll('a', 'a'), 0)
         self.assertGreater(locale.strcoll('b', 'a'), 0)
 
+    @unittest.skipIf(sys.platform == 'sunos5',
+                     "http://bugs.python.org/issue13441")
     def test_strxfrm(self):
         self.assertLess(locale.strxfrm('a'), locale.strxfrm('b'))
 
@@ -364,6 +366,8 @@
     def test_strcoll_with_diacritic(self):
         self.assertLess(locale.strcoll('?', 'b'), 0)
 
+    @unittest.skipIf(sys.platform == 'sunos5',
+                     "http://bugs.python.org/issue13441")
     def test_strxfrm_with_diacritic(self):
         self.assertLess(locale.strxfrm('?'), locale.strxfrm('b'))
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 01:15:15 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 01:15:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_UnicodeTranslateError_uses_?=
	=?utf8?q?the_new_Unicode_API?=
Message-ID: <E1RSHXX-0007if-1F@dinsdale.python.org>

http://hg.python.org/cpython/rev/244d9159aa38
changeset:   73649:244d9159aa38
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 01:17:27 2011 +0100
summary:
  UnicodeTranslateError uses the new Unicode API

The index is a character index, not a index in a Py_UNICODE* string.

files:
  Objects/exceptions.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -1751,8 +1751,8 @@
     if (reason_str == NULL)
         goto done;
 
-    if (uself->start < PyUnicode_GET_SIZE(uself->object) && uself->end == uself->start+1) {
-        int badchar = (int)PyUnicode_AS_UNICODE(uself->object)[uself->start];
+    if (uself->start < PyUnicode_GET_LENGTH(uself->object) && uself->end == uself->start+1) {
+        Py_UCS4 badchar = PyUnicode_ReadChar(uself->object, uself->start);
         const char *fmt;
         if (badchar <= 0xff)
             fmt = "can't translate character '\\x%02x' in position %zd: %U";
@@ -1762,7 +1762,7 @@
             fmt = "can't translate character '\\U%08x' in position %zd: %U";
         result = PyUnicode_FromFormat(
             fmt,
-            badchar,
+            (int)badchar,
             uself->start,
             reason_str
         );

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:01:46 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:01:46 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Check_for_PyUnicode=5FAS=5F?=
	=?utf8?q?UNICODE=28=29_failure?=
Message-ID: <E1RSIGY-0002NA-Ji@dinsdale.python.org>

http://hg.python.org/cpython/rev/071cc767a28e
changeset:   73650:071cc767a28e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:01:41 2011 +0100
summary:
  Check for PyUnicode_AS_UNICODE() failure

files:
  Modules/_ctypes/cfield.c |   7 +++++--
  PC/_subprocess.c         |  14 +++++++++++++-
  PC/import_nt.c           |  17 ++++++++++++++---
  Python/dynload_win.c     |   7 ++++++-
  4 files changed, 38 insertions(+), 7 deletions(-)


diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c
--- a/Modules/_ctypes/cfield.c
+++ b/Modules/_ctypes/cfield.c
@@ -1472,12 +1472,15 @@
     /* create a BSTR from value */
     if (value) {
         Py_ssize_t size = PyUnicode_GET_SIZE(value);
+        wchar_t* wvalue;
         if ((unsigned) size != size) {
             PyErr_SetString(PyExc_ValueError, "String too long for BSTR");
             return NULL;
         }
-        bstr = SysAllocStringLen(PyUnicode_AS_UNICODE(value),
-                                 (unsigned)size);
+        wvalue = PyUnicode_AsUnicode(value);
+        if (wvalue == NULL)
+            return NULL;
+        bstr = SysAllocStringLen(wvalue, (unsigned)size);
         Py_DECREF(value);
     } else
         bstr = NULL;
diff --git a/PC/_subprocess.c b/PC/_subprocess.c
--- a/PC/_subprocess.c
+++ b/PC/_subprocess.c
@@ -417,6 +417,7 @@
     PROCESS_INFORMATION pi;
     STARTUPINFOW si;
     PyObject* environment;
+    wchar_t *wenvironment;
 
     Py_UNICODE* application_name;
     Py_UNICODE* command_line;
@@ -461,6 +462,17 @@
             return NULL;
     }
 
+    if (environment) {
+        wenvironment = PyUnicode_AsUnicode(environment)
+        if (wenvironment == NULL)
+        {
+            Py_XDECREF(environment);
+            return NULL;
+        }
+    }
+    else
+        wenvironment = NULL;
+
     Py_BEGIN_ALLOW_THREADS
     result = CreateProcessW(application_name,
                            command_line,
@@ -468,7 +480,7 @@
                            NULL,
                            inherit_handles,
                            creation_flags | CREATE_UNICODE_ENVIRONMENT,
-                           environment ? PyUnicode_AS_UNICODE(environment) : NULL,
+                           wenvironment,
                            current_directory,
                            &si,
                            &pi);
diff --git a/PC/import_nt.c b/PC/import_nt.c
--- a/PC/import_nt.c
+++ b/PC/import_nt.c
@@ -35,6 +35,7 @@
     wchar_t pathBuf[MAXPATHLEN+1];
     int pathLen = MAXPATHLEN+1;
     PyObject *path, *moduleKey, *suffix;
+    wchar_t *wmoduleKey, *wsuffix;
     struct filedescr *fdp;
     HKEY keyBase;
     int modNameSize;
@@ -52,17 +53,22 @@
         PyWin_DLLVersionString, moduleName);
     if (moduleKey == NULL)
         return NULL;
+    wmoduleKey = PyUnicode_AsUnicode(moduleKey);
+    if (wmoduleKey == NULL) {
+        Py_DECREF(moduleKey);
+        return NULL;
+    }
 
     keyBase = HKEY_CURRENT_USER;
     modNameSize = pathLen;
-    regStat = RegQueryValueW(keyBase, PyUnicode_AS_UNICODE(moduleKey),
+    regStat = RegQueryValueW(keyBase, wmoduleKey,
                              pathBuf, &modNameSize);
     if (regStat != ERROR_SUCCESS) {
         /* No user setting - lookup in machine settings */
         keyBase = HKEY_LOCAL_MACHINE;
         /* be anal - failure may have reset size param */
         modNameSize = pathLen;
-        regStat = RegQueryValueW(keyBase, PyUnicode_AS_UNICODE(moduleKey),
+        regStat = RegQueryValueW(keyBase, wmoduleKey,
                                  pathBuf, &modNameSize);
         if (regStat != ERROR_SUCCESS) {
             Py_DECREF(moduleKey);
@@ -80,10 +86,15 @@
         suffix = PyUnicode_FromString(fdp->suffix);
         if (suffix == NULL)
             return NULL;
+        wsuffix = PyUnicode_AsUnicode(suffix);
+        if (wsuffix == NULL) {
+            Py_DECREF(suffix);
+            return NULL;
+        }
         extLen = PyUnicode_GET_SIZE(suffix);
         if ((Py_ssize_t)modNameSize > extLen &&
             _wcsnicmp(pathBuf + ((Py_ssize_t)modNameSize-extLen-1),
-                      PyUnicode_AS_UNICODE(suffix),
+                      wsuffix,
                       extLen) == 0)
         {
             Py_DECREF(suffix);
diff --git a/Python/dynload_win.c b/Python/dynload_win.c
--- a/Python/dynload_win.c
+++ b/Python/dynload_win.c
@@ -176,11 +176,16 @@
 {
     dl_funcptr p;
     char funcname[258], *import_python;
+    wchar_t *wpathname;
 
 #ifndef _DEBUG
     _Py_CheckPython3();
 #endif
 
+    wpathname = PyUnicode_AsUnicode(pathname);
+    if (wpathname == NULL)
+        return NULL;
+
     PyOS_snprintf(funcname, sizeof(funcname), "PyInit_%.200s", shortname);
 
     {
@@ -195,7 +200,7 @@
         /* We use LoadLibraryEx so Windows looks for dependent DLLs
             in directory of pathname first. */
         /* XXX This call doesn't exist in Windows CE */
-        hDLL = LoadLibraryExW(PyUnicode_AS_UNICODE(pathname), NULL,
+        hDLL = LoadLibraryExW(wpathname, NULL,
                               LOAD_WITH_ALTERED_SEARCH_PATH);
         _Py_DeactivateActCtx(cookie);
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:14:56 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:14:56 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_test=5Fwidechar=28=29_uses_?=
	=?utf8?q?the_new_Unicode_API?=
Message-ID: <E1RSITI-0003FC-Dj@dinsdale.python.org>

http://hg.python.org/cpython/rev/893b16723c9f
changeset:   73651:893b16723c9f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:06:57 2011 +0100
summary:
  test_widechar() uses the new Unicode API

PyUnicode_GET_SIZE() => PyUnicode_GET_LENGTH()

files:
  Modules/_testcapimodule.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1425,7 +1425,7 @@
         return NULL;
     }
 
-    if (PyUnicode_GET_SIZE(wide) != PyUnicode_GET_SIZE(utf8)) {
+    if (PyUnicode_GET_LENGTH(wide) != PyUnicode_GET_LENGTH(utf8)) {
         Py_DECREF(wide);
         Py_DECREF(utf8);
         return raiseTestError("test_widechar",

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:14:57 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:14:57 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_ctypes_check_for_PyUnicode?=
	=?utf8?q?=5FGET=5FSIZE=28=29_failure?=
Message-ID: <E1RSITJ-0003FO-5q@dinsdale.python.org>

http://hg.python.org/cpython/rev/fbf76c5c5c66
changeset:   73652:fbf76c5c5c66
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:11:26 2011 +0100
summary:
  ctypes check for PyUnicode_GET_SIZE() failure

files:
  Modules/_ctypes/_ctypes.c |   8 +++++++-
  Modules/_ctypes/cfield.c  |  14 +++++++++-----
  2 files changed, 16 insertions(+), 6 deletions(-)


diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -1142,6 +1142,8 @@
 WCharArray_set_value(CDataObject *self, PyObject *value)
 {
     Py_ssize_t result = 0;
+    Py_UNICODE *wstr;
+    Py_ssize_t len;
 
     if (value == NULL) {
         PyErr_SetString(PyExc_TypeError,
@@ -1155,7 +1157,11 @@
         return -1;
     } else
         Py_INCREF(value);
-    if ((unsigned)PyUnicode_GET_SIZE(value) > self->b_size/sizeof(wchar_t)) {
+
+    wstr = PyUnicode_AsUnicodeAndSize(value, &len);
+    if (wstr == NULL)
+        return -1;
+    if ((unsigned)len > self->b_size/sizeof(wchar_t)) {
         PyErr_SetString(PyExc_ValueError,
                         "string too long");
         result = -1;
diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c
--- a/Modules/_ctypes/cfield.c
+++ b/Modules/_ctypes/cfield.c
@@ -1259,6 +1259,7 @@
 static PyObject *
 U_set(void *ptr, PyObject *value, Py_ssize_t length)
 {
+    Py_UNICODE *wstr;
     Py_ssize_t size;
 
     /* It's easier to calculate in characters than in bytes */
@@ -1271,7 +1272,10 @@
         return NULL;
     } else
         Py_INCREF(value);
-    size = PyUnicode_GET_SIZE(value);
+
+    wstr = PyUnicode_AsUnicodeAndSize(value, &size);
+    if (wstr == NULL)
+        return NULL;
     if (size > length) {
         PyErr_Format(PyExc_ValueError,
                      "string too long (%zd, maximum length %zd)",
@@ -1471,15 +1475,15 @@
 
     /* create a BSTR from value */
     if (value) {
-        Py_ssize_t size = PyUnicode_GET_SIZE(value);
         wchar_t* wvalue;
+        Py_ssize_t size;
+        wvalue = PyUnicode_AsUnicodeAndSize(value, &size);
+        if (wvalue == NULL)
+            return NULL;
         if ((unsigned) size != size) {
             PyErr_SetString(PyExc_ValueError, "String too long for BSTR");
             return NULL;
         }
-        wvalue = PyUnicode_AsUnicode(value);
-        if (wvalue == NULL)
-            return NULL;
         bstr = SysAllocStringLen(wvalue, (unsigned)size);
         Py_DECREF(value);
     } else

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:14:57 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:14:57 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_typo_in_PC/=5Fsubproc?=
	=?utf8?q?ess=2Ec?=
Message-ID: <E1RSITJ-0003FS-TL@dinsdale.python.org>

http://hg.python.org/cpython/rev/da608aa6f59e
changeset:   73653:da608aa6f59e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:17:08 2011 +0100
summary:
  Fix a typo in PC/_subprocess.c

files:
  PC/_subprocess.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/PC/_subprocess.c b/PC/_subprocess.c
--- a/PC/_subprocess.c
+++ b/PC/_subprocess.c
@@ -463,7 +463,7 @@
     }
 
     if (environment) {
-        wenvironment = PyUnicode_AsUnicode(environment)
+        wenvironment = PyUnicode_AsUnicode(environment);
         if (wenvironment == NULL)
         {
             Py_XDECREF(environment);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:50:22 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:50:22 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FGET=5FSIZE=28?=
 =?utf8?q?=29_checks_that_PyUnicode=5FAsUnicode=28=29_succeed?=
Message-ID: <E1RSJ1a-0005sc-Cc@dinsdale.python.org>

http://hg.python.org/cpython/rev/98b8849d1a30
changeset:   73654:98b8849d1a30
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:24:49 2011 +0100
summary:
  PyUnicode_GET_SIZE() checks that PyUnicode_AsUnicode() succeed
using an assertion

files:
  Include/unicodeobject.h |  13 +++++++------
  1 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -385,12 +385,13 @@
    If the Py_UNICODE representation is not available, it will be computed
    on request.  Use PyUnicode_GET_LENGTH() for the length in code points. */
 
-#define PyUnicode_GET_SIZE(op) \
-    (assert(PyUnicode_Check(op)), \
-     (((PyASCIIObject *)(op))->wstr) ? \
-        PyUnicode_WSTR_LENGTH(op) :                   \
-        ((void)PyUnicode_AsUnicode((PyObject *)(op)), \
-         PyUnicode_WSTR_LENGTH(op)))
+#define PyUnicode_GET_SIZE(op)                       \
+    (assert(PyUnicode_Check(op)),                    \
+     (((PyASCIIObject *)(op))->wstr) ?               \
+      PyUnicode_WSTR_LENGTH(op) :                    \
+      ((void)PyUnicode_AsUnicode((PyObject *)(op)),  \
+       assert(((PyASCIIObject *)(op))->wstr),        \
+       PyUnicode_WSTR_LENGTH(op)))
 
 #define PyUnicode_GET_DATA_SIZE(op) \
     (PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:50:23 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:50:23 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_misuse_of_PyUnicode=5FG?=
 =?utf8?b?RVRfU0laRSgpID0+IFB5VW5pY29kZV9HRVRfTEVOR1RIKCk=?=
Message-ID: <E1RSJ1b-0005sn-8D@dinsdale.python.org>

http://hg.python.org/cpython/rev/2e37fc70e2dd
changeset:   73655:2e37fc70e2dd
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:49:52 2011 +0100
summary:
  Fix misuse of PyUnicode_GET_SIZE() => PyUnicode_GET_LENGTH()

And PyUnicode_GetSize() => PyUnicode_GetLength()

files:
  Modules/_csv.c            |   2 +-
  Modules/_datetimemodule.c |   4 ++--
  Modules/_gestalt.c        |   2 +-
  Modules/_io/stringio.c    |   2 +-
  Modules/_io/textio.c      |   4 ++--
  Objects/bytesobject.c     |   2 +-
  Objects/exceptions.c      |   4 ++--
  Objects/unicodeobject.c   |   4 ++--
  PC/import_nt.c            |   3 +--
  Python/_warnings.c        |  10 +++++-----
  10 files changed, 18 insertions(+), 19 deletions(-)


diff --git a/Modules/_csv.c b/Modules/_csv.c
--- a/Modules/_csv.c
+++ b/Modules/_csv.c
@@ -207,7 +207,7 @@
         *target = '\0';
         if (src != Py_None) {
             Py_ssize_t len;
-            len = PyUnicode_GetSize(src);
+            len = PyUnicode_GetLength(src);
             if (len > 1) {
                 PyErr_Format(PyExc_TypeError,
                     "\"%s\" must be an 1-character string",
diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -1080,7 +1080,7 @@
     PyObject *tzinfo = get_tzinfo_member(object);
     PyObject *Zreplacement = PyUnicode_FromStringAndSize(NULL, 0);
     _Py_IDENTIFIER(replace);
-	
+
     if (Zreplacement == NULL)
         return NULL;
     if (tzinfo == Py_None || tzinfo == NULL)
@@ -2673,7 +2673,7 @@
         return NULL;
 
     /* if the format is zero length, return str(self) */
-    if (PyUnicode_GetSize(format) == 0)
+    if (PyUnicode_GetLength(format) == 0)
         return PyObject_Str((PyObject *)self);
 
     return _PyObject_CallMethodId((PyObject *)self, &PyId_strftime, "O", format);
diff --git a/Modules/_gestalt.c b/Modules/_gestalt.c
--- a/Modules/_gestalt.c
+++ b/Modules/_gestalt.c
@@ -33,7 +33,7 @@
 convert_to_OSType(PyObject *v, OSType *pr)
 {
     uint32_t tmp;
-    if (!PyUnicode_Check(v) || PyUnicode_GetSize(v) != 4) {
+    if (!PyUnicode_Check(v) || PyUnicode_GetLength(v) != 4) {
     PyErr_SetString(PyExc_TypeError,
                     "OSType arg must be string of 4 chars");
     return 0;
diff --git a/Modules/_io/stringio.c b/Modules/_io/stringio.c
--- a/Modules/_io/stringio.c
+++ b/Modules/_io/stringio.c
@@ -730,7 +730,7 @@
        and copy it */
     self->string_size = 0;
     if (value && value != Py_None)
-        value_len = PyUnicode_GetSize(value);
+        value_len = PyUnicode_GetLength(value);
     else
         value_len = 0;
     if (value_len > 0) {
diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -2144,7 +2144,7 @@
         textiowrapper_set_decoded_chars(self, decoded);
 
         /* Skip chars_to_skip of the decoded characters. */
-        if (PyUnicode_GetSize(self->decoded_chars) < cookie.chars_to_skip) {
+        if (PyUnicode_GetLength(self->decoded_chars) < cookie.chars_to_skip) {
             PyErr_SetString(PyExc_IOError, "can't restore logical file position");
             goto fail;
         }
@@ -2208,7 +2208,7 @@
         goto fail;
 
     if (self->decoder == NULL || self->snapshot == NULL) {
-        assert (self->decoded_chars == NULL || PyUnicode_GetSize(self->decoded_chars) == 0);
+        assert (self->decoded_chars == NULL || PyUnicode_GetLength(self->decoded_chars) == 0);
         return posobj;
     }
 
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -2941,7 +2941,7 @@
         PyErr_BadInternalCall();
         return NULL;
     }
-    llen = PyUnicode_GetSize(result);
+    llen = PyUnicode_GetLength(result);
     if (llen > INT_MAX) {
         PyErr_SetString(PyExc_ValueError,
                         "string too large in _PyBytes_FormatLong");
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -1273,7 +1273,7 @@
     if (!obj)
         return -1;
     *start = ((PyUnicodeErrorObject *)exc)->start;
-    size = PyUnicode_GET_SIZE(obj);
+    size = PyUnicode_GET_LENGTH(obj);
     if (*start<0)
         *start = 0; /*XXX check for values <0*/
     if (*start>=size)
@@ -1341,7 +1341,7 @@
     if (!obj)
         return -1;
     *end = ((PyUnicodeErrorObject *)exc)->end;
-    size = PyUnicode_GET_SIZE(obj);
+    size = PyUnicode_GET_LENGTH(obj);
     if (*end<1)
         *end = 1;
     if (*end>size)
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4784,7 +4784,7 @@
             if (PyBytes_Check(rep))
                 repsize = PyBytes_GET_SIZE(rep);
             else
-                repsize = PyUnicode_GET_SIZE(rep);
+                repsize = PyUnicode_GET_LENGTH(rep);
 
             if (repsize > 4) {
                 Py_ssize_t offset;
@@ -8187,7 +8187,7 @@
             Py_DECREF(repunicode);
             return -1;
         }
-        repsize = PyUnicode_GET_SIZE(repunicode);
+        repsize = PyUnicode_GET_LENGTH(repunicode);
         data = PyUnicode_DATA(repunicode);
         kind = PyUnicode_KIND(repunicode);
         for (index = 0; index < repsize; index++) {
diff --git a/PC/import_nt.c b/PC/import_nt.c
--- a/PC/import_nt.c
+++ b/PC/import_nt.c
@@ -86,12 +86,11 @@
         suffix = PyUnicode_FromString(fdp->suffix);
         if (suffix == NULL)
             return NULL;
-        wsuffix = PyUnicode_AsUnicode(suffix);
+        wsuffix = PyUnicode_AsUnicodeAndSize(suffix, &extLen);
         if (wsuffix == NULL) {
             Py_DECREF(suffix);
             return NULL;
         }
-        extLen = PyUnicode_GET_SIZE(suffix);
         if ((Py_ssize_t)modNameSize > extLen &&
             _wcsnicmp(pathBuf + ((Py_ssize_t)modNameSize-extLen-1),
                       wsuffix,
diff --git a/Python/_warnings.c b/Python/_warnings.c
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -203,13 +203,13 @@
 
     mod_str = _PyUnicode_AsString(filename);
     if (mod_str == NULL)
-            return NULL;
-    len = PyUnicode_GetSize(filename);
+        return NULL;
+    len = PyUnicode_GetLength(filename);
     if (len < 0)
         return NULL;
     if (len >= 3 &&
         strncmp(mod_str + (len - 3), ".py", 3) == 0) {
-        module = PyUnicode_FromStringAndSize(mod_str, len-3);
+        module = PyUnicode_Substring(filename, 0, len-3);
     }
     else {
         module = filename;
@@ -506,7 +506,7 @@
         if (PyUnicode_READY(*filename))
             goto handle_error;
 
-        len = PyUnicode_GetSize(*filename);
+        len = PyUnicode_GetLength(*filename);
         kind = PyUnicode_KIND(*filename);
         data = PyUnicode_DATA(*filename);
 
@@ -690,7 +690,7 @@
         }
 
         /* Split the source into lines. */
-        source_list = PyObject_CallMethodObjArgs(source, 
+        source_list = PyObject_CallMethodObjArgs(source,
                                                  PyId_splitlines.object,
                                                  NULL);
         Py_DECREF(source);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:50:23 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:50:23 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_MultibyteCodec=5FEncode=28?=
 =?utf8?q?=29_checks_if_PyUnicode=5FAS=5FUNICODE=28=29_failed?=
Message-ID: <E1RSJ1b-0005sr-Vv@dinsdale.python.org>

http://hg.python.org/cpython/rev/2d106508ff37
changeset:   73656:2d106508ff37
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 02:50:14 2011 +0100
summary:
  MultibyteCodec_Encode() checks if PyUnicode_AS_UNICODE() failed

files:
  Modules/cjkcodecs/multibytecodec.c |  7 +++++--
  1 files changed, 5 insertions(+), 2 deletions(-)


diff --git a/Modules/cjkcodecs/multibytecodec.c b/Modules/cjkcodecs/multibytecodec.c
--- a/Modules/cjkcodecs/multibytecodec.c
+++ b/Modules/cjkcodecs/multibytecodec.c
@@ -574,8 +574,11 @@
         }
     }
 
-    data = PyUnicode_AS_UNICODE(arg);
-    datalen = PyUnicode_GET_SIZE(arg);
+    data = PyUnicode_AsUnicodeAndSize(arg, &datalen);
+    if (data == NULL) {
+        Py_XDECREF(ucvt);
+        return NULL;
+    }
 
     errorcb = internal_error_callback(errors);
     if (errorcb == NULL) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 02:59:17 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 02:59:17 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_CJK_codecs_checks_for_conve?=
 =?utf8?q?rsion_to_Py=5FUNICODE*_failures?=
Message-ID: <E1RSJAD-0006RH-Cz@dinsdale.python.org>

http://hg.python.org/cpython/rev/abd505205a85
changeset:   73657:abd505205a85
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 03:01:27 2011 +0100
summary:
  CJK codecs checks for conversion to Py_UNICODE* failures

files:
  Modules/cjkcodecs/multibytecodec.c |  13 ++++++++++---
  1 files changed, 10 insertions(+), 3 deletions(-)


diff --git a/Modules/cjkcodecs/multibytecodec.c b/Modules/cjkcodecs/multibytecodec.c
--- a/Modules/cjkcodecs/multibytecodec.c
+++ b/Modules/cjkcodecs/multibytecodec.c
@@ -443,10 +443,12 @@
         goto errorexit;
     }
 
+    if (PyUnicode_AsUnicode(retuni) == NULL)
+        goto errorexit;
     retunisize = PyUnicode_GET_SIZE(retuni);
     if (retunisize > 0) {
         REQUIRE_DECODEBUFFER(buf, retunisize);
-        memcpy((char *)buf->outbuf, PyUnicode_AS_DATA(retuni),
+        memcpy((char *)buf->outbuf, PyUnicode_AS_UNICODE(retuni),
                         retunisize * Py_UNICODE_SIZE);
         buf->outbuf += retunisize;
     }
@@ -746,6 +748,7 @@
     PyObject *ucvt, *r = NULL;
     Py_UNICODE *inbuf, *inbuf_end, *inbuf_tmp = NULL;
     Py_ssize_t datalen, origpending;
+    wchar_t *data;
 
     if (PyUnicode_Check(unistr))
         ucvt = NULL;
@@ -761,7 +764,9 @@
         }
     }
 
-    datalen = PyUnicode_GET_SIZE(unistr);
+    data = PyUnicode_AsUnicodeAndSize(unistr, &datalen);
+    if (data == NULL)
+        goto errorexit;
     origpending = ctx->pendingsize;
 
     if (origpending > 0) {
@@ -852,7 +857,9 @@
         buf->outobj = PyUnicode_FromUnicode(NULL, size);
         if (buf->outobj == NULL)
             return -1;
-        buf->outbuf = PyUnicode_AS_UNICODE(buf->outobj);
+        buf->outbuf = PyUnicode_AsUnicode(buf->outobj);
+        if (buf->outbuf == NULL)
+            return -1;
         buf->outbuf_end = buf->outbuf +
                           PyUnicode_GET_SIZE(buf->outobj);
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 03:16:52 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 03:16:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313441=3A_Disable_t?=
 =?utf8?q?emporary_localeconv=28=29_tests_on_Solaris?=
Message-ID: <E1RSJRE-0007ky-Rd@dinsdale.python.org>

http://hg.python.org/cpython/rev/78123afb3ea4
changeset:   73658:78123afb3ea4
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 03:19:05 2011 +0100
summary:
  Issue #13441: Disable temporary localeconv() tests on Solaris

files:
  Lib/test/test__locale.py |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py
--- a/Lib/test/test__locale.py
+++ b/Lib/test/test__locale.py
@@ -113,6 +113,8 @@
                                                 nl_radixchar, li_radixchar,
                                                 loc, set_locale))
 
+    @unittest.skipIf(sys.platform == 'sunos5',
+                     "http://bugs.python.org/issue13441")
     def test_float_parsing(self):
         # Bug #1391872: Test whether float parsing is okay on European
         # locales.

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Mon Nov 21 05:25:20 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 21 Nov 2011 05:25:20 +0100
Subject: [Python-checkins] Daily reference leaks (78123afb3ea4): sum=0
Message-ID: <E1RSLRY-0000OP-NG@ap.vmr.nerim.net>

results for 78123afb3ea4 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogyKuUmE', '-x']

From python-checkins at python.org  Mon Nov 21 12:41:38 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 12:41:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_winreg_module_avoids_the_de?=
	=?utf8?q?precated_Unicode_API?=
Message-ID: <E1RSSFm-0008D9-75@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a66478eba50
changeset:   73659:5a66478eba50
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 12:43:50 2011 +0100
summary:
  winreg module avoids the deprecated Unicode API

files:
  PC/winreg.c |  60 ++++++++++++++++++++++++----------------
  1 files changed, 36 insertions(+), 24 deletions(-)


diff --git a/PC/winreg.c b/PC/winreg.c
--- a/PC/winreg.c
+++ b/PC/winreg.c
@@ -760,26 +760,27 @@
         case REG_SZ:
         case REG_EXPAND_SZ:
             {
-            if (value == Py_None)
-                *retDataSize = 1;
-            else {
-                if (!PyUnicode_Check(value))
-                    return FALSE;
-                *retDataSize = Py_SAFE_DOWNCAST(
-                                   2 + PyUnicode_GET_DATA_SIZE(value),
-                                   size_t, DWORD);
-            }
-            *retDataBuf = (BYTE *)PyMem_NEW(DWORD, *retDataSize);
-            if (*retDataBuf==NULL){
-                PyErr_NoMemory();
-                return FALSE;
-            }
-            if (value == Py_None)
-                wcscpy((wchar_t *)*retDataBuf, L"");
-            else
-                wcscpy((wchar_t *)*retDataBuf,
-                       PyUnicode_AS_UNICODE(value));
-            break;
+                if (value != Py_None) {
+                    Py_ssize_t len;
+                    if (!PyUnicode_Check(value))
+                        return FALSE;
+                    *retDataBuf = (BYTE*)PyUnicode_AsWideCharString(value, &len);
+                    if (*retDataBuf == NULL)
+                        return FALSE;
+                    *retDataSize = Py_SAFE_DOWNCAST(
+                        (len + 1) * sizeof(wchar_t),
+                        Py_ssize_t, DWORD);
+                }
+                else {
+                    *retDataBuf = (BYTE *)PyMem_NEW(wchar_t, 1);
+                    if (*retDataBuf == NULL) {
+                        PyErr_NoMemory();
+                        return FALSE;
+                    }
+                    ((wchar_t *)*retDataBuf)[0] = L'\0';
+                    *retDataSize = 1 * sizeof(wchar_t);
+                }
+                break;
             }
         case REG_MULTI_SZ:
             {
@@ -796,10 +797,16 @@
                 for (j = 0; j < i; j++)
                 {
                     PyObject *t;
+                    wchar_t *wstr;
+                    Py_ssize_t len;
+
                     t = PyList_GET_ITEM(value, j);
                     if (!PyUnicode_Check(t))
                         return FALSE;
-                    size += Py_SAFE_DOWNCAST(2 + PyUnicode_GET_DATA_SIZE(t),
+                    wstr = PyUnicode_AsUnicodeAndSize(t, &len);
+                    if (wstr == NULL)
+                        return FALSE;
+                    size += Py_SAFE_DOWNCAST((len + 1) * sizeof(wchar_t),
                                              size_t, DWORD);
                 }
 
@@ -815,10 +822,15 @@
                 for (j = 0; j < i; j++)
                 {
                     PyObject *t;
+                    wchar_t *wstr;
+                    Py_ssize_t len;
+
                     t = PyList_GET_ITEM(value, j);
-                    wcscpy(P, PyUnicode_AS_UNICODE(t));
-                    P += 1 + wcslen(
-                        PyUnicode_AS_UNICODE(t));
+                    wstr = PyUnicode_AsUnicodeAndSize(t, &len);
+                    if (wstr == NULL)
+                        return FALSE;
+                    wcscpy(P, wstr);
+                    P += (len + 1);
                 }
                 /* And doubly-terminate the list... */
                 *P = '\0';

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 13:12:30 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 13:12:30 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_What=27s_new_in_Python_3=2E?=
	=?utf8?q?3=3A_Rephrase_PEP_393_doc?=
Message-ID: <E1RSSje-0003E2-QT@dinsdale.python.org>

http://hg.python.org/cpython/rev/b195bfb681e0
changeset:   73660:b195bfb681e0
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 13:14:43 2011 +0100
summary:
  What's new in Python 3.3: Rephrase PEP 393 doc

files:
  Doc/whatsnew/3.3.rst |  79 +++++++++++++++++--------------
  1 files changed, 43 insertions(+), 36 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -580,8 +580,9 @@
 * Thanks to the :pep:`393`, some operations on Unicode strings has been optimized:
 
   * the memory footprint is divided by 2 to 4 depending on the text
+  * encode an ASCII string to UTF-8 doesn't need to encode characters anymore,
+    the UTF-8 representation is shared with the ASCII representation
   * getting a substring of a latin1 strings is 4 times faster
-  * TODO
 
 
 Build and C API Changes
@@ -591,25 +592,30 @@
 
 * The :pep:`393` added new Unicode types, macros and functions:
 
-  * Py_UCS1, Py_UCS2, Py_UCS4 types
-  * PyASCIIObject and PyCompactUnicodeObject structures
-  * :c:func:`PyUnicode_New`
-  * :c:macro:`PyUnicode_READY`
-  * :c:func:`PyUnicode_FromKindAndData`
-  * :c:func:`PyUnicode_GetLength`, :c:macro:`PyUnicode_GET_LENGTH`
-  * :c:func:`PyUnicode_CopyCharacters`
-  * :c:func:`PyUnicode_ReadChar`, :c:func:`PyUnicode_WriteChar`
-  * :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsUCS4Copy`
-  * :c:func:`PyUnicode_FindChar`
-  * :c:func:`PyUnicode_Substring`
-  * :c:macro:`PyUnicode_1BYTE_DATA`, :c:macro:`PyUnicode_2BYTE_DATA`,
-    :c:macro:`PyUnicode_4BYTE_DATA`
-  * :c:macro:`PyUnicode_KIND` with :c:type:`PyUnicode_Kind` enum:
-    :c:data:`PyUnicode_WCHAR_KIND`, :c:data:`PyUnicode_1BYTE_KIND`,
-    :c:data:`PyUnicode_2BYTE_KIND`, :c:data:`PyUnicode_4BYTE_KIND`
-  * :c:macro:`PyUnicode_DATA`
-  * :c:macro:`PyUnicode_READ`, :c:macro:`PyUnicode_READ_CHAR`, :c:macro:`PyUnicode_WRITE`
-  * :c:macro:`PyUnicode_MAX_CHAR_VALUE`
+  * High-level API:
+
+    * :c:func:`PyUnicode_CopyCharacters`
+    * :c:func:`PyUnicode_FindChar`
+    * :c:func:`PyUnicode_GetLength`, :c:macro:`PyUnicode_GET_LENGTH`
+    * :c:func:`PyUnicode_New`
+    * :c:func:`PyUnicode_Substring`
+    * :c:func:`PyUnicode_ReadChar`, :c:func:`PyUnicode_WriteChar`
+
+  * Low-level API:
+
+    * :c:type:`Py_UCS1`, :c:type:`Py_UCS2`, :c:type:`Py_UCS4` types
+    * :c:type:`PyASCIIObject` and :c:type:`PyCompactUnicodeObject` structures
+    * :c:macro:`PyUnicode_READY`
+    * :c:func:`PyUnicode_FromKindAndData`
+    * :c:func:`PyUnicode_AsUCS4`, :c:func:`PyUnicode_AsUCS4Copy`
+    * :c:macro:`PyUnicode_DATA`, :c:macro:`PyUnicode_1BYTE_DATA`,
+      :c:macro:`PyUnicode_2BYTE_DATA`, :c:macro:`PyUnicode_4BYTE_DATA`
+    * :c:macro:`PyUnicode_KIND` with :c:type:`PyUnicode_Kind` enum:
+      :c:data:`PyUnicode_WCHAR_KIND`, :c:data:`PyUnicode_1BYTE_KIND`,
+      :c:data:`PyUnicode_2BYTE_KIND`, :c:data:`PyUnicode_4BYTE_KIND`
+    * :c:macro:`PyUnicode_READ`, :c:macro:`PyUnicode_READ_CHAR`, :c:macro:`PyUnicode_WRITE`
+    * :c:macro:`PyUnicode_MAX_CHAR_VALUE`
+
 
 
 Unsupported Operating Systems
@@ -643,21 +649,6 @@
 The :c:type:`Py_UNICODE` has been deprecated by the :pep:`393` and will be
 removed in Python 4. All functions using this type are deprecated:
 
-Functions and macros manipulating Py_UNICODE* strings:
-
- * :c:macro:`Py_UNICODE_strlen`: use :c:func:`PyUnicode_GetLength` or
-   :c:macro:`PyUnicode_GET_LENGTH`
- * :c:macro:`Py_UNICODE_strcat`: use :c:func:`PyUnicode_CopyCharacters` or
-   :c:func:`PyUnicode_FromFormat`
- * :c:macro:`Py_UNICODE_strcpy`, :c:macro:`Py_UNICODE_strncpy`,
-   :c:macro:`Py_UNICODE_COPY`: use :c:func:`PyUnicode_CopyCharacters` or
-   :c:func:`PyUnicode_Substring`
- * :c:macro:`Py_UNICODE_strcmp`: use :c:func:`PyUnicode_Compare`
- * :c:macro:`Py_UNICODE_strncmp`: use :c:func:`PyUnicode_Tailmatch`
- * :c:macro:`Py_UNICODE_strchr`, :c:macro:`Py_UNICODE_strrchr`: use
-   :c:func:`PyUnicode_FindChar`
- * :c:macro:`Py_UNICODE_FILL`
-
 Unicode functions and methods using :c:type:`Py_UNICODE` and
 :c:type:`Py_UNICODE*` types:
 
@@ -675,11 +666,27 @@
  * :c:func:`PyUnicode_AsUnicodeCopy`: use :c:func:`PyUnicode_AsUCS4Copy`,
    :c:func:`PyUnicode_AsWideCharString` or :c:func:`PyUnicode_Copy`
 
+Functions and macros manipulating Py_UNICODE* strings:
+
+ * :c:macro:`Py_UNICODE_strlen`: use :c:func:`PyUnicode_GetLength` or
+   :c:macro:`PyUnicode_GET_LENGTH`
+ * :c:macro:`Py_UNICODE_strcat`: use :c:func:`PyUnicode_CopyCharacters` or
+   :c:func:`PyUnicode_FromFormat`
+ * :c:macro:`Py_UNICODE_strcpy`, :c:macro:`Py_UNICODE_strncpy`,
+   :c:macro:`Py_UNICODE_COPY`: use :c:func:`PyUnicode_CopyCharacters` or
+   :c:func:`PyUnicode_Substring`
+ * :c:macro:`Py_UNICODE_strcmp`: use :c:func:`PyUnicode_Compare`
+ * :c:macro:`Py_UNICODE_strncmp`: use :c:func:`PyUnicode_Tailmatch`
+ * :c:macro:`Py_UNICODE_strchr`, :c:macro:`Py_UNICODE_strrchr`: use
+   :c:func:`PyUnicode_FindChar`
+ * :c:macro:`Py_UNICODE_FILL`
+
 Encoders:
 
  * :c:func:`PyUnicode_Encode`: use :c:func:`PyUnicode_AsEncodedObject`
  * :c:func:`PyUnicode_EncodeUTF7`
- * :c:func:`PyUnicode_EncodeUTF8`: use :c:func:`PyUnicode_AsUTF8String`
+ * :c:func:`PyUnicode_EncodeUTF8`: use :c:func:`PyUnicode_AsUTF8` or
+   :c:func:`PyUnicode_AsUTF8String`
  * :c:func:`PyUnicode_EncodeUTF32`
  * :c:func:`PyUnicode_EncodeUTF16`
  * :c:func:`PyUnicode_EncodeUnicodeEscape:` use

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 14:21:37 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 21 Nov 2011 14:21:37 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_usage_of_stdlib_s?=
	=?utf8?q?hutil=2E?=
Message-ID: <E1RSToX-0001CH-Ki@dinsdale.python.org>

http://hg.python.org/distutils2/rev/c1b1b537196d
changeset:   1257:c1b1b537196d
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 20 18:51:32 2011 +0100
summary:
  Remove usage of stdlib shutil.

This should help avoiding issues like #13170.

Because of implicit relative imports, _backport.tarfile already imported
shutil from the backports instead of the stdlib; I have inlined the only
function it needed to remove the circular dependency between these
modules.

files:
  distutils2/_backport/shutil.py               |  7 +------
  distutils2/_backport/tarfile.py              |  7 +++++--
  distutils2/_backport/tests/test_sysconfig.py |  2 +-
  distutils2/command/bdist_dumb.py             |  2 +-
  distutils2/command/bdist_msi.py              |  3 +--
  distutils2/command/bdist_wininst.py          |  3 +--
  distutils2/command/clean.py                  |  5 +++--
  distutils2/command/install_distinfo.py       |  2 +-
  distutils2/compiler/ccompiler.py             |  2 +-
  distutils2/create.py                         |  3 +--
  distutils2/tests/support.py                  |  3 +--
  distutils2/tests/test_command_test.py        |  2 +-
  distutils2/tests/test_command_upload_docs.py |  2 +-
  distutils2/tests/test_database.py            |  2 +-
  distutils2/tests/test_pypi_dist.py           |  5 +++--
  distutils2/util.py                           |  3 +--
  16 files changed, 24 insertions(+), 29 deletions(-)


diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -10,6 +10,7 @@
 from os.path import abspath
 import fnmatch
 import errno
+from distutils2._backport import tarfile
 
 try:
     import bz2
@@ -407,10 +408,6 @@
             os.makedirs(archive_dir)
 
     # creating the tarball
-    # XXX late import because of circular dependency between shutil and
-    # tarfile :(
-    from distutils2._backport import tarfile
-
     if logger is not None:
         logger.info('Creating tar archive')
 
@@ -700,8 +697,6 @@
 def _unpack_tarfile(filename, extract_dir):
     """Unpack tar/tar.gz/tar.bz2 `filename` to `extract_dir`
     """
-    # late import because of circular dependency
-    from distutils2._backport import tarfile
     try:
         tarobj = tarfile.open(filename)
     except tarfile.TarError:
diff --git a/distutils2/_backport/tarfile.py b/distutils2/_backport/tarfile.py
--- a/distutils2/_backport/tarfile.py
+++ b/distutils2/_backport/tarfile.py
@@ -43,7 +43,6 @@
 #---------
 import sys
 import os
-import shutil
 import stat
 import errno
 import time
@@ -256,7 +255,11 @@
     if length == 0:
         return
     if length is None:
-        shutil.copyfileobj(src, dst)
+        while 1:
+            buf = src.read(16*1024)
+            if not buf:
+                break
+            dst.write(buf)
         return
 
     BUFSIZE = 16 * 1024
diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -1,10 +1,10 @@
 import os
 import sys
 import subprocess
-import shutil
 from copy import copy
 from StringIO import StringIO
 
+from distutils2._backport import shutil
 from distutils2._backport import sysconfig
 from distutils2._backport.sysconfig import (
     get_paths, get_platform, get_config_vars, get_path, get_path_names,
diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -5,12 +5,12 @@
 """
 
 import os
-from shutil import rmtree
 
 from distutils2.util import get_platform
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingPlatformError
 from distutils2 import logger
+from distutils2._backport.shutil import rmtree
 from distutils2._backport.sysconfig import get_python_version
 
 
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -7,9 +7,8 @@
 import os
 import msilib
 
-
+from distutils2._backport.shutil import rmtree
 from distutils2._backport.sysconfig import get_python_version
-from shutil import rmtree
 from distutils2.command.cmd import Command
 from distutils2.version import NormalizedVersion
 from distutils2.errors import PackagingOptionError
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -4,12 +4,11 @@
 import os
 import codecs
 
-from shutil import rmtree
-
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingOptionError, PackagingPlatformError
 from distutils2 import logger
 from distutils2.util import get_platform
+from distutils2._backport.shutil import rmtree
 from distutils2._backport.sysconfig import get_python_version
 
 
diff --git a/distutils2/command/clean.py b/distutils2/command/clean.py
--- a/distutils2/command/clean.py
+++ b/distutils2/command/clean.py
@@ -3,9 +3,10 @@
 # Contributed by Bastian Kleineidam <calvin at cs.uni-sb.de>
 
 import os
-from shutil import rmtree
+from distutils2 import logger
 from distutils2.command.cmd import Command
-from distutils2 import logger
+from distutils2._backport.shutil import rmtree
+
 
 class clean(Command):
 
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -5,7 +5,6 @@
 import os
 import csv
 import codecs
-from shutil import rmtree
 try:
     import hashlib
 except ImportError:
@@ -13,6 +12,7 @@
 
 from distutils2 import logger
 from distutils2.command.cmd import Command
+from distutils2._backport.shutil import rmtree
 
 
 class install_distinfo(Command):
diff --git a/distutils2/compiler/ccompiler.py b/distutils2/compiler/ccompiler.py
--- a/distutils2/compiler/ccompiler.py
+++ b/distutils2/compiler/ccompiler.py
@@ -5,11 +5,11 @@
 """
 
 import os
-from shutil import move
 from distutils2 import logger
 from distutils2.util import split_quoted, execute, newer_group, spawn
 from distutils2.errors import CompileError, LinkError, UnknownFileError
 from distutils2.compiler import gen_preprocess_options
+from distutils2._backport.shutil import move
 
 
 class CCompiler(object):
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -24,7 +24,6 @@
 import sys
 import glob
 import codecs
-import shutil
 from textwrap import dedent
 from ConfigParser import RawConfigParser
 
@@ -34,7 +33,7 @@
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
 from distutils2.compat import detect_encoding
 from distutils2.version import is_valid_version
-from distutils2._backport import sysconfig
+from distutils2._backport import shutil, sysconfig
 try:
     any
 except NameError:
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -37,7 +37,6 @@
 import sys
 import errno
 import codecs
-import shutil
 import logging
 import logging.handlers
 import subprocess
@@ -53,7 +52,7 @@
 from distutils2.command import set_command, _COMMANDS
 
 from distutils2.tests import unittest
-from distutils2._backport import sysconfig
+from distutils2._backport import shutil, sysconfig
 
 # define __all__ to make pydoc more useful
 __all__ = [
diff --git a/distutils2/tests/test_command_test.py b/distutils2/tests/test_command_test.py
--- a/distutils2/tests/test_command_test.py
+++ b/distutils2/tests/test_command_test.py
@@ -1,7 +1,6 @@
 import os
 import re
 import sys
-import shutil
 import unittest as ut1
 import distutils2.database
 
@@ -14,6 +13,7 @@
 from distutils2.command.test import test
 from distutils2.command import set_command
 from distutils2.dist import Distribution
+from distutils2._backport import shutil
 
 
 EXPECTED_OUTPUT_RE = r'''FAIL: test_blah \(myowntestmodule.SomeTest\)
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -1,6 +1,5 @@
 """Tests for distutils2.command.upload_docs."""
 import os
-import shutil
 import logging
 import zipfile
 try:
@@ -12,6 +11,7 @@
 from distutils2.command.upload_docs import upload_docs, zip_dir
 from distutils2.dist import Distribution
 from distutils2.errors import PackagingFileError, PackagingOptionError
+from distutils2._backport import shutil
 
 from distutils2.tests import unittest, support
 try:
diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py
--- a/distutils2/tests/test_database.py
+++ b/distutils2/tests/test_database.py
@@ -1,7 +1,6 @@
 import os
 import csv
 import sys
-import shutil
 import tempfile
 try:
     from hashlib import md5
@@ -22,6 +21,7 @@
     provides_distribution, obsoletes_distribution, get_file_users,
     enable_cache, disable_cache, distinfo_dirname, _yield_distributions,
     get_file, get_file_path)
+from distutils2._backport import shutil
 
 # TODO Add a test for getting a distribution provided by another distribution
 # TODO Add a test for absolute pathed RECORD items (e.g. /etc/myapp/config.ini)
diff --git a/distutils2/tests/test_pypi_dist.py b/distutils2/tests/test_pypi_dist.py
--- a/distutils2/tests/test_pypi_dist.py
+++ b/distutils2/tests/test_pypi_dist.py
@@ -1,11 +1,11 @@
 """Tests for the distutils2.pypi.dist module."""
 
 import os
-import shutil
 from distutils2.version import VersionPredicate
 from distutils2.pypi.dist import (ReleaseInfo, ReleasesList, DistInfo,
-                                 split_archive_name, get_infos_from_url)
+                                  split_archive_name, get_infos_from_url)
 from distutils2.pypi.errors import HashDoesNotMatch, UnsupportedHashName
+from distutils2._backport import shutil
 
 from distutils2.tests import unittest
 from distutils2.tests.support import TempdirManager, requires_zlib, fake_dec
@@ -38,6 +38,7 @@
         # should not fail
         release['sdist']
 
+    @unittest.skip('needs to be written')
     def test_get_unknown_distribution(self):
         # should raise a KeyError
         pass
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -6,7 +6,6 @@
 import sys
 import errno
 import codecs
-import shutil
 import string
 import posixpath
 import subprocess
@@ -26,7 +25,7 @@
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
                                PackagingExecError, InstallationException,
                                PackagingInternalError)
-from distutils2._backport import sysconfig
+from distutils2._backport import shutil, sysconfig
 
 __all__ = [
     # file dependencies

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 21 14:21:37 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 21 Nov 2011 14:21:37 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Make_test=5Fmanifest_pas?=
 =?utf8?q?s_on_2=2E4_and_2=2E5_=28fixes_=2311751=29=2E?=
Message-ID: <E1RSToX-0001CK-LC@dinsdale.python.org>

http://hg.python.org/distutils2/rev/723517bf6708
changeset:   1258:723517bf6708
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 20 19:02:08 2011 +0100
summary:
  Make test_manifest pass on 2.4 and 2.5 (fixes #11751).

The code added massages the output of fnmatch in old Pythons to make it
match more recent behavior (it was changed in 2.6 to append '\Z(?ms)'
instead of '$', to support newline characters in filenames).  I could
have changed the tests, but I prefer having the code do the same thing
for all versions as far as possible.

files:
  distutils2/manifest.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/distutils2/manifest.py b/distutils2/manifest.py
--- a/distutils2/manifest.py
+++ b/distutils2/manifest.py
@@ -346,6 +346,10 @@
     # Unix-only.
     pattern_re = re.sub(r'((?<!\\)(\\\\)*)\.', r'\1[^/]', pattern_re)
 
+    if pattern_re.endswith('$'):
+        # make it look like fnmatch in Python 2.6 and newer
+        pattern_re = pattern_re[:-1] + r'\Z(?ms)'
+
     return pattern_re
 
 

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 21 14:21:37 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 21 Nov 2011 14:21:37 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_unused_names?=
Message-ID: <E1RSToX-0001CP-Lk@dinsdale.python.org>

http://hg.python.org/distutils2/rev/c898ebb74f28
changeset:   1259:c898ebb74f28
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 20 19:04:39 2011 +0100
summary:
  Remove unused names

files:
  distutils2/command/build_ext.py    |  1 -
  distutils2/command/install_dist.py |  1 -
  distutils2/config.py               |  1 -
  distutils2/tests/test_create.py    |  1 -
  4 files changed, 0 insertions(+), 4 deletions(-)


diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -3,7 +3,6 @@
 import os
 import re
 import sys
-import logging
 
 from distutils2._backport import sysconfig
 from distutils2.util import get_platform
diff --git a/distutils2/command/install_dist.py b/distutils2/command/install_dist.py
--- a/distutils2/command/install_dist.py
+++ b/distutils2/command/install_dist.py
@@ -13,7 +13,6 @@
 from distutils2.util import convert_path, change_root, get_platform
 from distutils2.errors import PackagingOptionError
 
-import site
 if sys.version_info[:2] >= (2, 6):
     HAS_USER_SITE = True
 else:
diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -21,7 +21,6 @@
     if '.' not in name:
         return
     parts = name.split('.')
-    modname = parts[-1]
     parent = '.'.join(parts[:-1])
     if parent not in packages:
         # we could log a warning instead of raising, but what's the use
diff --git a/distutils2/tests/test_create.py b/distutils2/tests/test_create.py
--- a/distutils2/tests/test_create.py
+++ b/distutils2/tests/test_create.py
@@ -3,7 +3,6 @@
 import os
 import sys
 import codecs
-from StringIO import StringIO
 from textwrap import dedent
 from distutils2 import create
 from distutils2.create import MainProgram, ask_yn, ask, main

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 21 14:21:37 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 21 Nov 2011 14:21:37 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28python3=29=3A_Fix_sheban?=
	=?utf8?q?gs?=
Message-ID: <E1RSToX-0001CW-Ov@dinsdale.python.org>

http://hg.python.org/distutils2/rev/bf1b6478e134
changeset:   1261:bf1b6478e134
branch:      python3
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 20 19:46:13 2011 +0100
summary:
  Fix shebangs

files:
  pysetup     |  2 +-
  runtests.py |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pysetup b/pysetup
--- a/pysetup
+++ b/pysetup
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from distutils2.run import main
 
 if __name__ == "__main__":
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 """Test runner for distutils2.
 
 The tests for distutils2 are defined in the distutils2.tests package.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 21 14:21:37 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 21 Nov 2011 14:21:37 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
	=?utf8?q?n3=29=3A_Branch_merge?=
Message-ID: <E1RSToX-0001CT-OP@dinsdale.python.org>

http://hg.python.org/distutils2/rev/49e4d8a859c9
changeset:   1260:49e4d8a859c9
branch:      python3
parent:      1251:7fe2f04a24cc
parent:      1259:c898ebb74f28
user:        ?ric Araujo <merwok at netwok.org>
date:        Sun Nov 20 19:45:31 2011 +0100
summary:
  Branch merge

files:
  CHANGES.txt                                  |    3 +
  CONTRIBUTORS.txt                             |    1 +
  distutils2/_backport/shutil.py               |    7 +-
  distutils2/_backport/tarfile.py              |    7 +-
  distutils2/_backport/tests/test_sysconfig.py |    2 +-
  distutils2/command/bdist_dumb.py             |    2 +-
  distutils2/command/bdist_msi.py              |    3 +-
  distutils2/command/bdist_wininst.py          |    3 +-
  distutils2/command/build_ext.py              |    6 -
  distutils2/command/clean.py                  |    5 +-
  distutils2/command/cmd.py                    |   18 +-
  distutils2/command/install_distinfo.py       |    2 +-
  distutils2/command/sdist.py                  |    5 +-
  distutils2/command/test.py                   |    3 +-
  distutils2/compiler/__init__.py              |    9 +-
  distutils2/compiler/bcppcompiler.py          |    4 +-
  distutils2/compiler/ccompiler.py             |    5 +-
  distutils2/compiler/cygwinccompiler.py       |    8 +-
  distutils2/compiler/msvc9compiler.py         |    4 +-
  distutils2/compiler/msvccompiler.py          |    4 +-
  distutils2/config.py                         |    1 -
  distutils2/create.py                         |    3 +-
  distutils2/tests/__init__.py                 |  105 +------
  distutils2/tests/support.py                  |    3 +-
  distutils2/tests/test_command_bdist.py       |    7 +-
  distutils2/tests/test_command_sdist.py       |    7 +-
  distutils2/tests/test_command_test.py        |    2 +-
  distutils2/tests/test_command_upload_docs.py |    2 +-
  distutils2/tests/test_config.py              |    2 +-
  distutils2/tests/test_database.py            |    9 +-
  distutils2/tests/test_dist.py                |    1 -
  distutils2/tests/test_pypi_dist.py           |    5 +-
  distutils2/tests/test_support.py             |   85 +++++
  distutils2/util.py                           |   27 +-
  runtests.py                                  |  164 +++++++--
  35 files changed, 292 insertions(+), 232 deletions(-)


diff --git a/CHANGES.txt b/CHANGES.txt
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -157,6 +157,9 @@
 - Rename get_reinitialized_command back to reinitialize_command [?ric]
 - Rename install_distinfo's option from distinfo-dir to the more usual
   install_dir [?ric]
+- Remove verbose arguments for Command and Compiler classes as well as util
+  functions, obsoleted by logging [?ric]
+- #12659: Add tests for tests.support [francisco]
 
 
 1.0a3 - 2010-10-08
diff --git a/CONTRIBUTORS.txt b/CONTRIBUTORS.txt
--- a/CONTRIBUTORS.txt
+++ b/CONTRIBUTORS.txt
@@ -17,6 +17,7 @@
 - Anthony Baxter
 - Erik Bray
 - C. Titus Brown
+- Francisco Mart?n Brugu?
 - Nicolas Cadou
 - Godefroid Chapelle
 - Christophe Combelles
diff --git a/distutils2/_backport/shutil.py b/distutils2/_backport/shutil.py
--- a/distutils2/_backport/shutil.py
+++ b/distutils2/_backport/shutil.py
@@ -11,6 +11,7 @@
 import fnmatch
 import collections
 import errno
+from distutils2._backport import tarfile
 
 try:
     import bz2
@@ -402,10 +403,6 @@
             os.makedirs(archive_dir)
 
     # creating the tarball
-    # XXX late import because of circular dependency between shutil and
-    # tarfile :(
-    from distutils2._backport import tarfile
-
     if logger is not None:
         logger.info('Creating tar archive')
 
@@ -695,8 +692,6 @@
 def _unpack_tarfile(filename, extract_dir):
     """Unpack tar/tar.gz/tar.bz2 `filename` to `extract_dir`
     """
-    # late import because of circular dependency
-    from distutils2._backport import tarfile
     try:
         tarobj = tarfile.open(filename)
     except tarfile.TarError:
diff --git a/distutils2/_backport/tarfile.py b/distutils2/_backport/tarfile.py
--- a/distutils2/_backport/tarfile.py
+++ b/distutils2/_backport/tarfile.py
@@ -42,7 +42,6 @@
 #---------
 import sys
 import os
-import shutil
 import stat
 import errno
 import time
@@ -255,7 +254,11 @@
     if length == 0:
         return
     if length is None:
-        shutil.copyfileobj(src, dst)
+        while True:
+            buf = src.read(16*1024)
+            if not buf:
+                break
+            dst.write(buf)
         return
 
     BUFSIZE = 16 * 1024
diff --git a/distutils2/_backport/tests/test_sysconfig.py b/distutils2/_backport/tests/test_sysconfig.py
--- a/distutils2/_backport/tests/test_sysconfig.py
+++ b/distutils2/_backport/tests/test_sysconfig.py
@@ -1,10 +1,10 @@
 import os
 import sys
 import subprocess
-import shutil
 from copy import copy
 from io import StringIO
 
+from distutils2._backport import shutil
 from distutils2._backport import sysconfig
 from distutils2._backport.sysconfig import (
     get_paths, get_platform, get_config_vars, get_path, get_path_names,
diff --git a/distutils2/command/bdist_dumb.py b/distutils2/command/bdist_dumb.py
--- a/distutils2/command/bdist_dumb.py
+++ b/distutils2/command/bdist_dumb.py
@@ -5,12 +5,12 @@
 """
 
 import os
-from shutil import rmtree
 
 from distutils2.util import get_platform
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingPlatformError
 from distutils2 import logger
+from distutils2._backport.shutil import rmtree
 from distutils2._backport.sysconfig import get_python_version
 
 
diff --git a/distutils2/command/bdist_msi.py b/distutils2/command/bdist_msi.py
--- a/distutils2/command/bdist_msi.py
+++ b/distutils2/command/bdist_msi.py
@@ -7,9 +7,8 @@
 import os
 import msilib
 
-
+from distutils2._backport.shutil import rmtree
 from distutils2._backport.sysconfig import get_python_version
-from shutil import rmtree
 from distutils2.command.cmd import Command
 from distutils2.version import NormalizedVersion
 from distutils2.errors import PackagingOptionError
diff --git a/distutils2/command/bdist_wininst.py b/distutils2/command/bdist_wininst.py
--- a/distutils2/command/bdist_wininst.py
+++ b/distutils2/command/bdist_wininst.py
@@ -3,12 +3,11 @@
 import sys
 import os
 
-from shutil import rmtree
-
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingOptionError, PackagingPlatformError
 from distutils2 import logger
 from distutils2.util import get_platform
+from distutils2._backport.shutil import rmtree
 from distutils2._backport.sysconfig import get_python_version
 
 
diff --git a/distutils2/command/build_ext.py b/distutils2/command/build_ext.py
--- a/distutils2/command/build_ext.py
+++ b/distutils2/command/build_ext.py
@@ -4,7 +4,6 @@
 import re
 import sys
 import site
-import logging
 
 from distutils2._backport import sysconfig
 from distutils2.util import get_platform
@@ -288,14 +287,9 @@
             self.libraries.extend(build_clib.get_library_names() or [])
             self.library_dirs.append(build_clib.build_clib)
 
-        # Temporary kludge until we remove the verbose arguments and use
-        # logging everywhere
-        verbose = logger.getEffectiveLevel() >= logging.DEBUG
-
         # Setup the CCompiler object that we'll use to do all the
         # compiling and linking
         self.compiler_obj = new_compiler(compiler=self.compiler,
-                                         verbose=verbose,
                                          dry_run=self.dry_run,
                                          force=self.force)
 
diff --git a/distutils2/command/clean.py b/distutils2/command/clean.py
--- a/distutils2/command/clean.py
+++ b/distutils2/command/clean.py
@@ -3,9 +3,10 @@
 # Contributed by Bastian Kleineidam <calvin at cs.uni-sb.de>
 
 import os
-from shutil import rmtree
+from distutils2 import logger
 from distutils2.command.cmd import Command
-from distutils2 import logger
+from distutils2._backport.shutil import rmtree
+
 
 class clean(Command):
 
diff --git a/distutils2/command/cmd.py b/distutils2/command/cmd.py
--- a/distutils2/command/cmd.py
+++ b/distutils2/command/cmd.py
@@ -351,7 +351,7 @@
     def execute(self, func, args, msg=None, level=1):
         util.execute(func, args, msg, dry_run=self.dry_run)
 
-    def mkpath(self, name, mode=0o777, dry_run=None, verbose=0):
+    def mkpath(self, name, mode=0o777, dry_run=None):
         if dry_run is None:
             dry_run = self.dry_run
         name = os.path.normpath(name)
@@ -367,9 +367,11 @@
 
     def copy_file(self, infile, outfile,
                   preserve_mode=True, preserve_times=True, link=None, level=1):
-        """Copy a file respecting verbose, dry-run and force flags.  (The
-        former two default to whatever is in the Distribution object, and
-        the latter defaults to false for commands that don't define it.)"""
+        """Copy a file respecting dry-run and force flags.
+
+        (dry-run defaults to whatever is in the Distribution object, and
+        force to false for commands that don't define it.)
+        """
         if self.dry_run:
             # XXX add a comment
             return
@@ -380,11 +382,13 @@
 
     def copy_tree(self, infile, outfile, preserve_mode=True,
                   preserve_times=True, preserve_symlinks=False, level=1):
-        """Copy an entire directory tree respecting verbose, dry-run,
+        """Copy an entire directory tree respecting dry-run
         and force flags.
         """
         if self.dry_run:
-            return  # see if we want to display something
+            # XXX should not return but let copy_tree log and decide to execute
+            # or not based on its dry_run argument
+            return
 
         return util.copy_tree(infile, outfile, preserve_mode, preserve_times,
             preserve_symlinks, not self.force, dry_run=self.dry_run)
@@ -392,7 +396,7 @@
     def move_file(self, src, dst, level=1):
         """Move a file respecting the dry-run flag."""
         if self.dry_run:
-            return  # XXX log ?
+            return  # XXX same thing
         return move(src, dst)
 
     def spawn(self, cmd, search_path=True, level=1):
diff --git a/distutils2/command/install_distinfo.py b/distutils2/command/install_distinfo.py
--- a/distutils2/command/install_distinfo.py
+++ b/distutils2/command/install_distinfo.py
@@ -5,10 +5,10 @@
 import os
 import csv
 import hashlib
-from shutil import rmtree
 
 from distutils2 import logger
 from distutils2.command.cmd import Command
+from distutils2._backport.shutil import rmtree
 
 
 class install_distinfo(Command):
diff --git a/distutils2/command/sdist.py b/distutils2/command/sdist.py
--- a/distutils2/command/sdist.py
+++ b/distutils2/command/sdist.py
@@ -337,12 +337,11 @@
         """
         return self.archive_files
 
-    def create_tree(self, base_dir, files, mode=0o777, verbose=1,
-                    dry_run=False):
+    def create_tree(self, base_dir, files, mode=0o777, dry_run=False):
         need_dir = set()
         for file in files:
             need_dir.add(os.path.join(base_dir, os.path.dirname(file)))
 
         # Now create them
         for dir in sorted(need_dir):
-            self.mkpath(dir, mode, verbose=verbose, dry_run=dry_run)
+            self.mkpath(dir, mode, dry_run=dry_run)
diff --git a/distutils2/command/test.py b/distutils2/command/test.py
--- a/distutils2/command/test.py
+++ b/distutils2/command/test.py
@@ -60,8 +60,7 @@
             self.run_command('build')
             sys.path.insert(0, build.build_lib)
 
-            # Temporary kludge until we remove the verbose arguments and use
-            # logging everywhere
+            # XXX maybe we could pass the verbose argument of pysetup here
             logger = logging.getLogger('distutils2')
             verbose = logger.getEffectiveLevel() >= logging.DEBUG
             verbosity = verbose + 1
diff --git a/distutils2/compiler/__init__.py b/distutils2/compiler/__init__.py
--- a/distutils2/compiler/__init__.py
+++ b/distutils2/compiler/__init__.py
@@ -153,8 +153,7 @@
     pretty_printer.print_help("List of available compilers:")
 
 
-def new_compiler(plat=None, compiler=None, verbose=0, dry_run=False,
-                 force=False):
+def new_compiler(plat=None, compiler=None, dry_run=False, force=False):
     """Generate an instance of some CCompiler subclass for the supplied
     platform/compiler combination.  'plat' defaults to 'os.name'
     (eg. 'posix', 'nt'), and 'compiler' defaults to the default compiler
@@ -183,11 +182,7 @@
         cls = resolve_name(cls)
         _COMPILERS[compiler] = cls
 
-
-    # XXX The None is necessary to preserve backwards compatibility
-    # with classes that expect verbose to be the first positional
-    # argument.
-    return cls(None, dry_run, force)
+    return cls(dry_run, force)
 
 
 def gen_preprocess_options(macros, include_dirs):
diff --git a/distutils2/compiler/bcppcompiler.py b/distutils2/compiler/bcppcompiler.py
--- a/distutils2/compiler/bcppcompiler.py
+++ b/distutils2/compiler/bcppcompiler.py
@@ -47,8 +47,8 @@
     exe_extension = '.exe'
 
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(BCPPCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(BCPPCompiler, self).__init__(dry_run, force)
 
         # These executables are assumed to all be in the path.
         # Borland doesn't seem to use any special registry settings to
diff --git a/distutils2/compiler/ccompiler.py b/distutils2/compiler/ccompiler.py
--- a/distutils2/compiler/ccompiler.py
+++ b/distutils2/compiler/ccompiler.py
@@ -5,11 +5,11 @@
 """
 
 import os
-from shutil import move
 from distutils2 import logger
 from distutils2.util import split_quoted, execute, newer_group, spawn
 from distutils2.errors import CompileError, LinkError, UnknownFileError
 from distutils2.compiler import gen_preprocess_options
+from distutils2._backport.shutil import move
 
 
 class CCompiler:
@@ -79,10 +79,9 @@
                    }
     language_order = ["c++", "objc", "c"]
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
+    def __init__(self, dry_run=False, force=False):
         self.dry_run = dry_run
         self.force = force
-        self.verbose = verbose
 
         # 'output_dir': a common output directory for object, library,
         # shared object, and shared library files
diff --git a/distutils2/compiler/cygwinccompiler.py b/distutils2/compiler/cygwinccompiler.py
--- a/distutils2/compiler/cygwinccompiler.py
+++ b/distutils2/compiler/cygwinccompiler.py
@@ -92,8 +92,8 @@
     shared_lib_format = "%s%s"
     exe_extension = ".exe"
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(CygwinCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(CygwinCCompiler, self).__init__(dry_run, force)
 
         status, details = check_config_h()
         logger.debug("Python's GCC status: %s (details: %s)", status, details)
@@ -270,8 +270,8 @@
     name = 'mingw32'
     description = 'MinGW32 compiler'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(Mingw32CCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(Mingw32CCompiler, self).__init__(dry_run, force)
 
         # ld_version >= "2.13" support -shared so use it instead of
         # -mdll -static
diff --git a/distutils2/compiler/msvc9compiler.py b/distutils2/compiler/msvc9compiler.py
--- a/distutils2/compiler/msvc9compiler.py
+++ b/distutils2/compiler/msvc9compiler.py
@@ -309,8 +309,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = VERSION
         self.__root = r"Software\Microsoft\VisualStudio"
         # self.__macros = MACROS
diff --git a/distutils2/compiler/msvccompiler.py b/distutils2/compiler/msvccompiler.py
--- a/distutils2/compiler/msvccompiler.py
+++ b/distutils2/compiler/msvccompiler.py
@@ -236,8 +236,8 @@
     static_lib_format = shared_lib_format = '%s%s'
     exe_extension = '.exe'
 
-    def __init__(self, verbose=0, dry_run=False, force=False):
-        super(MSVCCompiler, self).__init__(verbose, dry_run, force)
+    def __init__(self, dry_run=False, force=False):
+        super(MSVCCompiler, self).__init__(dry_run, force)
         self.__version = get_build_version()
         self.__arch = get_build_architecture()
         if self.__arch == "Intel":
diff --git a/distutils2/config.py b/distutils2/config.py
--- a/distutils2/config.py
+++ b/distutils2/config.py
@@ -20,7 +20,6 @@
     if '.' not in name:
         return
     parts = name.split('.')
-    modname = parts[-1]
     parent = '.'.join(parts[:-1])
     if parent not in packages:
         # we could log a warning instead of raising, but what's the use
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -23,7 +23,6 @@
 import imp
 import sys
 import glob
-import shutil
 from hashlib import md5
 from textwrap import dedent
 from tokenize import detect_encoding
@@ -34,7 +33,7 @@
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
 from distutils2.version import is_valid_version
-from distutils2._backport import sysconfig
+from distutils2._backport import shutil, sysconfig
 
 
 _FILENAME = 'setup.cfg'
diff --git a/distutils2/tests/__init__.py b/distutils2/tests/__init__.py
--- a/distutils2/tests/__init__.py
+++ b/distutils2/tests/__init__.py
@@ -14,17 +14,11 @@
 import os
 import sys
 import unittest2 as unittest
-from io import StringIO
-
-# XXX move helpers to support, add tests for them, remove things that
-# duplicate test.support (or keep them for the backport; needs thinking)
-
-here = os.path.dirname(__file__) or os.curdir
-verbose = 1
 
 
 def test_suite():
     suite = unittest.TestSuite()
+    here = os.path.dirname(__file__) or os.curdir
     for fn in os.listdir(here):
         if fn.startswith("test") and fn.endswith(".py"):
             modname = "distutils2.tests." + fn[:-3]
@@ -32,100 +26,3 @@
             module = sys.modules[modname]
             suite.addTest(module.test_suite())
     return suite
-
-
-class Error(Exception):
-    """Base class for regression test exceptions."""
-
-
-class TestFailed(Error):
-    """Test failed."""
-
-
-class BasicTestRunner:
-    def run(self, test):
-        result = unittest.TestResult()
-        test(result)
-        return result
-
-
-def _run_suite(suite, verbose_=1):
-    """Run tests from a unittest.TestSuite-derived class."""
-    global verbose
-    verbose = verbose_
-    if verbose_:
-        runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
-    else:
-        runner = BasicTestRunner()
-
-    result = runner.run(suite)
-    if not result.wasSuccessful():
-        if len(result.errors) == 1 and not result.failures:
-            err = result.errors[0][1]
-        elif len(result.failures) == 1 and not result.errors:
-            err = result.failures[0][1]
-        else:
-            err = "errors occurred; run in verbose mode for details"
-        raise TestFailed(err)
-
-
-def run_unittest(classes, verbose_=1):
-    """Run tests from unittest.TestCase-derived classes.
-
-    Originally extracted from stdlib test.support and modified to
-    support unittest2.
-    """
-    valid_types = (unittest.TestSuite, unittest.TestCase)
-    suite = unittest.TestSuite()
-    for cls in classes:
-        if isinstance(cls, str):
-            if cls in sys.modules:
-                suite.addTest(unittest.findTestCases(sys.modules[cls]))
-            else:
-                raise ValueError("str arguments must be keys in sys.modules")
-        elif isinstance(cls, valid_types):
-            suite.addTest(cls)
-        else:
-            suite.addTest(unittest.makeSuite(cls))
-    _run_suite(suite, verbose_)
-
-
-def reap_children():
-    """Use this function at the end of test_main() whenever sub-processes
-    are started.  This will help ensure that no extra children (zombies)
-    stick around to hog resources and create problems when looking
-    for refleaks.
-
-    Extracted from stdlib test.support.
-    """
-
-    # Reap all our dead child processes so we don't leave zombies around.
-    # These hog resources and might be causing some of the buildbots to die.
-    if hasattr(os, 'waitpid'):
-        any_process = -1
-        while True:
-            try:
-                # This will raise an exception on Windows.  That's ok.
-                pid, status = os.waitpid(any_process, os.WNOHANG)
-                if pid == 0:
-                    break
-            except:
-                break
-
-
-def captured_stdout(func, *args, **kw):
-    orig_stdout = getattr(sys, 'stdout')
-    setattr(sys, 'stdout', StringIO())
-    try:
-        res = func(*args, **kw)
-        sys.stdout.seek(0)
-        return res, sys.stdout.read()
-    finally:
-        setattr(sys, 'stdout', orig_stdout)
-
-
-def unload(name):
-    try:
-        del sys.modules[name]
-    except KeyError:
-        pass
diff --git a/distutils2/tests/support.py b/distutils2/tests/support.py
--- a/distutils2/tests/support.py
+++ b/distutils2/tests/support.py
@@ -36,7 +36,6 @@
 import re
 import sys
 import errno
-import shutil
 import logging
 import logging.handlers
 import subprocess
@@ -52,7 +51,7 @@
 from distutils2.command import set_command, _COMMANDS
 
 from distutils2.tests import unittest
-from distutils2._backport import sysconfig
+from distutils2._backport import shutil, sysconfig
 
 # define __all__ to make pydoc more useful
 __all__ = [
diff --git a/distutils2/tests/test_command_bdist.py b/distutils2/tests/test_command_bdist.py
--- a/distutils2/tests/test_command_bdist.py
+++ b/distutils2/tests/test_command_bdist.py
@@ -1,7 +1,8 @@
 """Tests for distutils.command.bdist."""
 import os
+from test.support import captured_stdout
 from distutils2.command.bdist import bdist, show_formats
-from distutils2.tests import unittest, support, captured_stdout
+from distutils2.tests import unittest, support
 
 
 class BuildTestCase(support.TempdirManager,
@@ -42,7 +43,9 @@
                             '%s should take --skip-build from bdist' % name)
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        with captured_stdout() as stdout:
+            show_formats()
+        stdout = stdout.getvalue()
 
         # the output should be a header line + one line per format
         num_formats = len(bdist.format_commands)
diff --git a/distutils2/tests/test_command_sdist.py b/distutils2/tests/test_command_sdist.py
--- a/distutils2/tests/test_command_sdist.py
+++ b/distutils2/tests/test_command_sdist.py
@@ -1,5 +1,6 @@
 """Tests for distutils2.command.sdist."""
 import os
+import sys
 import zipfile
 
 try:
@@ -17,8 +18,8 @@
 from distutils2._backport import tarfile
 from distutils2._backport.shutil import get_archive_formats
 
+from test.support import captured_stdout
 from distutils2.tests import support, unittest
-from distutils2.tests import captured_stdout
 from distutils2.tests.support import requires_zlib
 
 
@@ -240,7 +241,9 @@
         self.assertIn("'setup.cfg' file not found", warnings[1])
 
     def test_show_formats(self):
-        __, stdout = captured_stdout(show_formats)
+        with captured_stdout() as stdout:
+            show_formats()
+        stdout = stdout.getvalue()
 
         # the output should be a header line + one line per format
         num_formats = len(get_archive_formats())
diff --git a/distutils2/tests/test_command_test.py b/distutils2/tests/test_command_test.py
--- a/distutils2/tests/test_command_test.py
+++ b/distutils2/tests/test_command_test.py
@@ -1,7 +1,6 @@
 import os
 import re
 import sys
-import shutil
 import unittest as ut1
 import distutils2.database
 
@@ -14,6 +13,7 @@
 from distutils2.command.test import test
 from distutils2.command import set_command
 from distutils2.dist import Distribution
+from distutils2._backport import shutil
 
 
 EXPECTED_OUTPUT_RE = r'''FAIL: test_blah \(myowntestmodule.SomeTest\)
diff --git a/distutils2/tests/test_command_upload_docs.py b/distutils2/tests/test_command_upload_docs.py
--- a/distutils2/tests/test_command_upload_docs.py
+++ b/distutils2/tests/test_command_upload_docs.py
@@ -1,6 +1,5 @@
 """Tests for distutils2.command.upload_docs."""
 import os
-import shutil
 import logging
 import zipfile
 try:
@@ -12,6 +11,7 @@
 from distutils2.command.upload_docs import upload_docs, zip_dir
 from distutils2.dist import Distribution
 from distutils2.errors import PackagingFileError, PackagingOptionError
+from distutils2._backport import shutil
 
 from distutils2.tests import unittest, support
 try:
diff --git a/distutils2/tests/test_config.py b/distutils2/tests/test_config.py
--- a/distutils2/tests/test_config.py
+++ b/distutils2/tests/test_config.py
@@ -119,7 +119,7 @@
     /usr/include/blitz
 extra_compile_args = -fPIC -O2
     -DGECODE_VERSION=$(./gecode_version) -- sys.platform != 'win32'
-    /DGECODE_VERSION='win32' -- sys.platform == 'win32'
+    /DGECODE_VERSION=win32 -- sys.platform == 'win32'
 language = cxx
 
 # corner case: if the parent package of an extension is declared but
diff --git a/distutils2/tests/test_database.py b/distutils2/tests/test_database.py
--- a/distutils2/tests/test_database.py
+++ b/distutils2/tests/test_database.py
@@ -2,7 +2,6 @@
 import io
 import csv
 import sys
-import shutil
 import tempfile
 from hashlib import md5
 from textwrap import dedent
@@ -10,6 +9,7 @@
 from distutils2.tests.test_util import GlobTestCaseBase
 from distutils2.tests.support import requires_zlib
 
+import distutils2.database
 from distutils2.config import get_resources_dests
 from distutils2.errors import PackagingError
 from distutils2.metadata import Metadata
@@ -19,6 +19,7 @@
     provides_distribution, obsoletes_distribution, get_file_users,
     enable_cache, disable_cache, distinfo_dirname, _yield_distributions,
     get_file, get_file_path)
+from distutils2._backport import shutil
 
 # TODO Add a test for getting a distribution provided by another distribution
 # TODO Add a test for absolute pathed RECORD items (e.g. /etc/myapp/config.ini)
@@ -279,6 +280,12 @@
         sys.path.insert(0, self.fake_dists_path)
         self.addCleanup(sys.path.remove, self.fake_dists_path)
 
+    def test_caches(self):
+        # sanity check for internal caches
+        for name in ('_cache_name', '_cache_name_egg',
+                     '_cache_path', '_cache_path_egg'):
+            self.assertEqual(getattr(distutils2.database, name), {})
+
     def test_distinfo_dirname(self):
         # Given a name and a version, we expect the distinfo_dirname function
         # to return a standard distribution information directory name.
diff --git a/distutils2/tests/test_dist.py b/distutils2/tests/test_dist.py
--- a/distutils2/tests/test_dist.py
+++ b/distutils2/tests/test_dist.py
@@ -8,7 +8,6 @@
 from distutils2.dist import Distribution
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingModuleError, PackagingOptionError
-from distutils2.tests import captured_stdout
 from distutils2.tests import support, unittest
 from distutils2.tests.support import create_distribution, use_command
 from distutils2.tests.support import unload
diff --git a/distutils2/tests/test_pypi_dist.py b/distutils2/tests/test_pypi_dist.py
--- a/distutils2/tests/test_pypi_dist.py
+++ b/distutils2/tests/test_pypi_dist.py
@@ -1,11 +1,11 @@
 """Tests for the distutils2.pypi.dist module."""
 
 import os
-import shutil
 from distutils2.version import VersionPredicate
 from distutils2.pypi.dist import (ReleaseInfo, ReleasesList, DistInfo,
-                                 split_archive_name, get_infos_from_url)
+                                  split_archive_name, get_infos_from_url)
 from distutils2.pypi.errors import HashDoesNotMatch, UnsupportedHashName
+from distutils2._backport import shutil
 
 from distutils2.tests import unittest
 from distutils2.tests.support import TempdirManager, requires_zlib, fake_dec
@@ -38,6 +38,7 @@
         # should not fail
         release['sdist']
 
+    @unittest.skip('needs to be written')
     def test_get_unknown_distribution(self):
         # should raise a KeyError
         pass
diff --git a/distutils2/tests/test_support.py b/distutils2/tests/test_support.py
new file mode 100644
--- /dev/null
+++ b/distutils2/tests/test_support.py
@@ -0,0 +1,85 @@
+import os
+import tempfile
+
+from distutils2.dist import Distribution
+from distutils2.tests import support, unittest
+
+
+class TestingSupportTestCase(unittest.TestCase):
+
+    def test_fake_dec(self):
+        @support.fake_dec(1, 2, k=3)
+        def func(arg0, *args, **kargs):
+            return arg0, args, kargs
+        self.assertEqual(func(-1, -2, k=-3), (-1, (-2,), {'k': -3}))
+
+    def test_TempdirManager(self):
+        files = {}
+
+        class Tester(support.TempdirManager, unittest.TestCase):
+
+            def runTest(self):
+                # empty method required for the backport
+                pass
+
+            def test_mktempfile(self2):
+                tmpfile = self2.mktempfile()
+                files['test_mktempfile'] = tmpfile.name
+                self.assertTrue(os.path.isfile(tmpfile.name))
+
+            def test_mkdtemp(self2):
+                tmpdir = self2.mkdtemp()
+                files['test_mkdtemp'] = tmpdir
+                self.assertTrue(os.path.isdir(tmpdir))
+
+            def test_write_file(self2):
+                tmpdir = self2.mkdtemp()
+                files['test_write_file'] = tmpdir
+                self2.write_file((tmpdir, 'file1'), 'me file 1')
+                file1 = os.path.join(tmpdir, 'file1')
+                self.assertTrue(os.path.isfile(file1))
+                text = ''
+                f = open(file1, 'r')
+                try:
+                    text = f.read()
+                finally:
+                    f.close()
+                self.assertEqual(text, 'me file 1')
+
+            def test_create_dist(self2):
+                project_dir, dist = self2.create_dist()
+                files['test_create_dist'] = project_dir
+                self.assertTrue(os.path.isdir(project_dir))
+                self.assertIsInstance(dist, Distribution)
+
+            def test_assertIsFile(self2):
+                fd, fn = tempfile.mkstemp()
+                os.close(fd)
+                self.addCleanup(support.unlink, fn)
+                self2.assertIsFile(fn)
+                self.assertRaises(AssertionError, self2.assertIsFile, 'foO')
+
+            def test_assertIsNotFile(self2):
+                tmpdir = self2.mkdtemp()
+                self2.assertIsNotFile(tmpdir)
+
+        tester = Tester()
+        for name in ('test_mktempfile', 'test_mkdtemp', 'test_write_file',
+                     'test_create_dist', 'test_assertIsFile',
+                     'test_assertIsNotFile'):
+            tester.setUp()
+            try:
+                getattr(tester, name)()
+            finally:
+                tester.tearDown()
+
+            # check clean-up
+            if name in files:
+                self.assertFalse(os.path.exists(files[name]))
+
+
+def test_suite():
+    return unittest.makeSuite(TestingSupportTestCase)
+
+if __name__ == "__main__":
+    unittest.main(defaultTest="test_suite")
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -3,11 +3,9 @@
 import os
 import re
 import csv
-import imp
 import sys
 import errno
 import codecs
-import shutil
 import string
 import hashlib
 import posixpath
@@ -22,7 +20,7 @@
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
                                PackagingExecError, InstallationException,
                                PackagingInternalError)
-from distutils2._backport import sysconfig
+from distutils2._backport import shutil, sysconfig
 
 __all__ = [
     # file dependencies
@@ -260,7 +258,7 @@
             if element]
 
 
-def execute(func, args, msg=None, verbose=0, dry_run=False):
+def execute(func, args, msg=None, dry_run=False):
     """Perform some action that affects the outside world.
 
     Some actions (e.g. writing to the filesystem) are special because
@@ -683,7 +681,7 @@
     _cfg_target_split = None
 
 
-def spawn(cmd, search_path=True, verbose=0, dry_run=False, env=None):
+def spawn(cmd, search_path=True, dry_run=False, env=None):
     """Run another program specified as a command list 'cmd' in a new process.
 
     'cmd' is just the argument list for the new process, ie.
@@ -1334,8 +1332,7 @@
 
 # XXX to be replaced by shutil.copytree
 def copy_tree(src, dst, preserve_mode=True, preserve_times=True,
-              preserve_symlinks=False, update=False, verbose=True,
-              dry_run=False):
+              preserve_symlinks=False, update=False, dry_run=False):
     # FIXME use of this function is why we get spurious logging message on
     # stdout when tests run; kill and replace by shutil!
     from distutils.file_util import copy_file
@@ -1354,7 +1351,7 @@
                   "error listing files in '%s': %s" % (src, errstr))
 
     if not dry_run:
-        _mkpath(dst, verbose=verbose)
+        _mkpath(dst)
 
     outputs = []
 
@@ -1364,8 +1361,7 @@
 
         if preserve_symlinks and os.path.islink(src_name):
             link_dest = os.readlink(src_name)
-            if verbose >= 1:
-                logger.info("linking %s -> %s", dst_name, link_dest)
+            logger.info("linking %s -> %s", dst_name, link_dest)
             if not dry_run:
                 os.symlink(link_dest, dst_name)
             outputs.append(dst_name)
@@ -1374,11 +1370,10 @@
             outputs.extend(
                 copy_tree(src_name, dst_name, preserve_mode,
                           preserve_times, preserve_symlinks, update,
-                          verbose=verbose, dry_run=dry_run))
+                          dry_run=dry_run))
         else:
             copy_file(src_name, dst_name, preserve_mode,
-                      preserve_times, update, verbose=verbose,
-                      dry_run=dry_run)
+                      preserve_times, update, dry_run=dry_run)
             outputs.append(dst_name)
 
     return outputs
@@ -1391,7 +1386,7 @@
 # I don't use os.makedirs because a) it's new to Python 1.5.2, and
 # b) it blows up if the directory already exists (I want to silently
 # succeed in that case).
-def _mkpath(name, mode=0o777, verbose=True, dry_run=False):
+def _mkpath(name, mode=0o777, dry_run=False):
     # Detect a common bug -- name is None
     if not isinstance(name, str):
         raise PackagingInternalError(
@@ -1426,9 +1421,7 @@
         if abs_head in _path_created:
             continue
 
-        if verbose >= 1:
-            logger.info("creating %s", head)
-
+        logger.info("creating %s", head)
         if not dry_run:
             try:
                 os.mkdir(head, mode)
diff --git a/runtests.py b/runtests.py
--- a/runtests.py
+++ b/runtests.py
@@ -1,15 +1,110 @@
 #!/usr/bin/env python
-"""Tests for distutils2.
+"""Test runner for distutils2.
 
 The tests for distutils2 are defined in the distutils2.tests package.
+They can also be executed with the unittest2 runner or nose.
 """
 
+import os
 import sys
 from os.path import dirname, islink, realpath, join, abspath
 from optparse import OptionParser
+from distutils2.tests import unittest
+
+
+# unittest machinery copied from stdlib's test.regrtest and test.support
+
+class TestFailed(Exception):
+    """Test failed."""
+
+
+class BasicTestRunner:
+    def run(self, test):
+        result = unittest.TestResult()
+        test(result)
+        return result
+
+
+def reap_children():
+    """Use this function at the end of test_main() whenever sub-processes
+    are started.  This will help ensure that no extra children (zombies)
+    stick around to hog resources and create problems when looking
+    for refleaks.
+    """
+
+    # Reap all our dead child processes so we don't leave zombies around.
+    # These hog resources and might be causing some of the buildbots to die.
+    if hasattr(os, 'waitpid'):
+        any_process = -1
+        while True:
+            try:
+                # This will raise an exception on Windows.  That's ok.
+                pid, status = os.waitpid(any_process, os.WNOHANG)
+                if pid == 0:
+                    break
+            except:
+                break
+
+
+def _run_suite(suite, verbose=True):
+    """Run tests from a unittest.TestSuite-derived class."""
+    if verbose:
+        runner = unittest.TextTestRunner(sys.stdout, verbosity=2)
+    else:
+        runner = BasicTestRunner()
+
+    result = runner.run(suite)
+    if not result.wasSuccessful():
+        if len(result.errors) == 1 and not result.failures:
+            err = result.errors[0][1]
+        elif len(result.failures) == 1 and not result.errors:
+            err = result.failures[0][1]
+        else:
+            err = "errors occurred; run in verbose mode for details"
+        raise TestFailed(err)
+
+
+def run_unittest(classes, verbose=True):
+    """Run tests from unittest.TestCase-derived classes.
+
+    Originally extracted from stdlib test.test_support and modified to
+    support unittest2.
+    """
+    valid_types = (unittest.TestSuite, unittest.TestCase)
+    suite = unittest.TestSuite()
+    for cls in classes:
+        if isinstance(cls, str):
+            if cls in sys.modules:
+                suite.addTest(unittest.findTestCases(sys.modules[cls]))
+            else:
+                raise ValueError("str arguments must be keys in sys.modules")
+        elif isinstance(cls, valid_types):
+            suite.addTest(cls)
+        else:
+            suite.addTest(unittest.makeSuite(cls))
+    _run_suite(suite, verbose)
+
+
+def run_tests(verbose):
+    # do NOT import those at the top level, coverage will be inaccurate if
+    # distutils2 modules are imported before coverage magic is started
+    from distutils2.tests import test_suite
+    from distutils2._backport.tests import test_suite as btest_suite
+    try:
+        try:
+            run_unittest([test_suite(), btest_suite()], verbose=verbose)
+            return 0
+        except TestFailed:
+            return 1
+    finally:
+        reap_children()
+
+
+# coverage-related code
 
 COVERAGE_FILE = join(dirname(abspath(__file__)), '.coverage')
 
+
 def get_coverage():
     """ Return a usable coverage object. """
     # deferred import because coverage is optional
@@ -19,52 +114,34 @@
         cov = coverage.coverage(COVERAGE_FILE)
     return cov
 
+
 def ignore_prefixes(module):
     """ Return a list of prefixes to ignore in the coverage report if
     we want to completely skip `module`.
     """
-    # A function like that is needed because some GNU/Linux
-    # distributions, such a Ubuntu, really like to build link farm in
-    # /usr/lib in order to save a few bytes on the disk.
+    # A function like that is needed because some operating systems like Debian
+    # and derivatives use symlinks directory in order to save disk space
     dirnames = [dirname(module.__file__)]
 
-    pymod = module.__file__.rstrip("c")
+    pymod = module.__file__.rstrip('co')
     if islink(pymod):
         dirnames.append(dirname(realpath(pymod)))
     return dirnames
 
 
-def parse_opts():
-    parser = OptionParser(usage="%prog [OPTIONS]",
-                          description="run the distutils2 unittests")
-
-    parser.add_option("-q", "--quiet", help="do not print verbose messages",
-                      action="store_true", default=False)
-    parser.add_option("-c", "--coverage", action="store_true", default=False,
-                      help="produce a coverage report at the end of the run")
-    parser.add_option("-r", "--report", action="store_true", default=False,
-                      help="produce a coverage report from the last test run")
-    parser.add_option("-m", "--show-missing", action="store_true",
-                      default=False,
-                      help=("Show line numbers of statements in each module "
-                            "that weren't executed."))
-
-    opts, args = parser.parse_args()
-    return opts, args
-
-
 def coverage_report(opts):
     from distutils2.tests.support import unittest
     cov = get_coverage()
     if hasattr(cov, "load"):
         # running coverage 3.x
         cov.load()
+        # morfs means modules or files
         morfs = None
     else:
         # running coverage 2.x
         cov.cache = COVERAGE_FILE
         cov.restore()
-        morfs = [m for m in list(cov.cexecuted.keys()) if "distutils2" in m]
+        morfs = [m for m in cov.cexecuted if "distutils2" in m]
 
     prefixes = ["runtests", "distutils2/tests", "distutils2/_backport"]
     prefixes += ignore_prefixes(unittest)
@@ -93,6 +170,28 @@
                    omit=[p + "*" for p in prefixes],
                    show_missing=opts.show_missing)
 
+
+# command-line parsing
+
+def parse_opts():
+    parser = OptionParser(usage="%prog [OPTIONS]",
+                          description="run the distutils2 unittests")
+
+    parser.add_option("-q", "--quiet", help="do not print verbose messages",
+                      action="store_true", default=False)
+    parser.add_option("-c", "--coverage", action="store_true", default=False,
+                      help="produce a coverage report at the end of the run")
+    parser.add_option("-r", "--report", action="store_true", default=False,
+                      help="produce a coverage report from the last test run")
+    parser.add_option("-m", "--show-missing", action="store_true",
+                      default=False,
+                      help=("Show line numbers of statements in each module "
+                            "that weren't executed."))
+
+    opts, args = parser.parse_args()
+    return opts, args
+
+
 def test_main():
     opts, args = parse_opts()
     # FIXME when we run with --quiet, we still want to see errors and failures
@@ -115,20 +214,5 @@
     return ret
 
 
-def run_tests(verbose):
-    # do NOT import those at the top level, coverage will be inaccurate if
-    # modules are imported before its magic is started
-    from distutils2.tests import run_unittest, test_suite, reap_children, TestFailed
-    from distutils2._backport.tests import test_suite as btest_suite
-    try:
-        try:
-            run_unittest([test_suite(), btest_suite()], verbose_=verbose)
-            return 0
-        except TestFailed:
-            return 1
-    finally:
-        reap_children()
-
-
 if __name__ == "__main__":
     sys.exit(test_main())

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Mon Nov 21 14:31:50 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 14:31:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313441=3A_=5FPyUnic?=
 =?utf8?q?ode=5FCheckConsistency=28=29_dumps_the_string_if_the_maximum?=
Message-ID: <E1RSTyQ-0002AN-Q1@dinsdale.python.org>

http://hg.python.org/cpython/rev/a19dad38d4e8
changeset:   73661:a19dad38d4e8
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 14:31:41 2011 +0100
summary:
  Issue #13441: _PyUnicode_CheckConsistency() dumps the string if the maximum
character is bigger than U+10FFFF and locale.localeconv() dumps the string
before decoding it.

Temporary hack to debug the issue #13441.

files:
  Modules/_localemodule.c |  26 ++++++++++++++++++++++++++
  Objects/unicodeobject.c |  13 +++++++++++++
  2 files changed, 39 insertions(+), 0 deletions(-)


diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -79,6 +79,23 @@
     return res2;
 }
 
+#ifdef Py_DEBUG
+void
+dump_str(const char *name, const char *value)
+{
+    size_t i, len = strlen(value);
+    printf("Decode localeconv() %s: {", name);
+    for (i=0; i<len; i++) {
+        unsigned char ch = value[i];
+        if (i)
+            printf(" 0x%02x", ch);
+        else
+            printf("0x%02x", ch);
+    }
+    printf("} (len=%u)\n", len);
+}
+#endif
+
 /* support functions for formatting floating point numbers */
 
 PyDoc_STRVAR(setlocale__doc__,
@@ -184,11 +201,20 @@
     /* hopefully, the localeconv result survives the C library calls
        involved herein */
 
+#ifdef Py_DEBUG
+#define RESULT_STRING(s)\
+    dump_str(#s, l->s); \
+    x = str2uni(l->s);   \
+    if (!x) goto failed;\
+    PyDict_SetItemString(result, #s, x);\
+    Py_XDECREF(x)
+#else
 #define RESULT_STRING(s)\
     x = str2uni(l->s);   \
     if (!x) goto failed;\
     PyDict_SetItemString(result, #s, x);\
     Py_XDECREF(x)
+#endif
 
 #define RESULT_INT(i)\
     x = PyLong_FromLong(l->i);\
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -391,6 +391,19 @@
             if (ch > maxchar)
                 maxchar = ch;
         }
+        if (maxchar > 0x10FFFF) {
+            printf("Invalid Unicode string! {");
+            for (i=0; i < ascii->length; i++)
+            {
+                Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+                if (i)
+                    printf(", U+%04x", ch);
+                else
+                    printf("U+%04x", ch);
+            }
+            printf("} (len=%u)\n", ascii->length);
+            abort();
+        }
         if (kind == PyUnicode_1BYTE_KIND) {
             if (ascii->state.ascii == 0) {
                 assert(maxchar >= 128);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 15:40:51 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 15:40:51 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Another_temporary_hack_to_d?=
	=?utf8?q?ebug_the_issue_=2313441?=
Message-ID: <E1RSV3D-0000uY-C5@dinsdale.python.org>

http://hg.python.org/cpython/rev/d1b3b1d00811
changeset:   73662:d1b3b1d00811
user:        Victor Stinner <vstinner at wyplay.com>
date:        Mon Nov 21 15:41:17 2011 +0100
summary:
  Another temporary hack to debug the issue #13441

Dump the wchar_t that we are going to decode and dump the locale

files:
  Modules/_localemodule.c |  20 ++++++++++++++++++++
  1 files changed, 20 insertions(+), 0 deletions(-)


diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -73,6 +73,20 @@
 #else
     assert(res1 == needed);
 #endif
+#ifdef Py_DEBUG
+    {
+        size_t i;
+        printf("Decode wchar_t {");
+        for (i=0; i<res1; i++) {
+            wchar_t ch = dest[i];
+            if (i)
+                printf(" U+%04x", ch);
+            else
+                printf("U+%04x", ch);
+        }
+        printf("} (len=%u)\n", res1);
+    }
+#endif
     res2 = PyUnicode_FromWideChar(dest, res1);
     if (dest != smallbuf)
         PyMem_Free(dest);
@@ -160,12 +174,18 @@
 
     if (locale) {
         /* set locale */
+#ifdef Py_DEBUG
+        printf("SET LOCALE \"%s\"\n", locale);
+#endif
         result = setlocale(category, locale);
         if (!result) {
             /* operation failed, no setting was changed */
             PyErr_SetString(Error, "unsupported locale setting");
             return NULL;
         }
+#ifdef Py_DEBUG
+        printf("SET LOCALE -> %s\n", result);
+#endif
         result_object = str2uni(result);
         if (!result_object)
             return NULL;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 16:01:02 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 16:01:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_temporary_hacks_for_?=
	=?utf8?q?the_issue_=2313441?=
Message-ID: <E1RSVMk-0002wV-EI@dinsdale.python.org>

http://hg.python.org/cpython/rev/acda16de630c
changeset:   73663:acda16de630c
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 16:01:27 2011 +0100
summary:
  Remove temporary hacks for the issue #13441

files:
  Modules/_localemodule.c |  46 -----------------------------
  1 files changed, 0 insertions(+), 46 deletions(-)


diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c
--- a/Modules/_localemodule.c
+++ b/Modules/_localemodule.c
@@ -73,43 +73,12 @@
 #else
     assert(res1 == needed);
 #endif
-#ifdef Py_DEBUG
-    {
-        size_t i;
-        printf("Decode wchar_t {");
-        for (i=0; i<res1; i++) {
-            wchar_t ch = dest[i];
-            if (i)
-                printf(" U+%04x", ch);
-            else
-                printf("U+%04x", ch);
-        }
-        printf("} (len=%u)\n", res1);
-    }
-#endif
     res2 = PyUnicode_FromWideChar(dest, res1);
     if (dest != smallbuf)
         PyMem_Free(dest);
     return res2;
 }
 
-#ifdef Py_DEBUG
-void
-dump_str(const char *name, const char *value)
-{
-    size_t i, len = strlen(value);
-    printf("Decode localeconv() %s: {", name);
-    for (i=0; i<len; i++) {
-        unsigned char ch = value[i];
-        if (i)
-            printf(" 0x%02x", ch);
-        else
-            printf("0x%02x", ch);
-    }
-    printf("} (len=%u)\n", len);
-}
-#endif
-
 /* support functions for formatting floating point numbers */
 
 PyDoc_STRVAR(setlocale__doc__,
@@ -174,18 +143,12 @@
 
     if (locale) {
         /* set locale */
-#ifdef Py_DEBUG
-        printf("SET LOCALE \"%s\"\n", locale);
-#endif
         result = setlocale(category, locale);
         if (!result) {
             /* operation failed, no setting was changed */
             PyErr_SetString(Error, "unsupported locale setting");
             return NULL;
         }
-#ifdef Py_DEBUG
-        printf("SET LOCALE -> %s\n", result);
-#endif
         result_object = str2uni(result);
         if (!result_object)
             return NULL;
@@ -221,20 +184,11 @@
     /* hopefully, the localeconv result survives the C library calls
        involved herein */
 
-#ifdef Py_DEBUG
-#define RESULT_STRING(s)\
-    dump_str(#s, l->s); \
-    x = str2uni(l->s);   \
-    if (!x) goto failed;\
-    PyDict_SetItemString(result, #s, x);\
-    Py_XDECREF(x)
-#else
 #define RESULT_STRING(s)\
     x = str2uni(l->s);   \
     if (!x) goto failed;\
     PyDict_SetItemString(result, #s, x);\
     Py_XDECREF(x)
-#endif
 
 #define RESULT_INT(i)\
     x = PyLong_FromLong(l->i);\

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 18:04:05 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 18:04:05 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313441=3A_Reenable_?=
 =?utf8?q?strxfrm=28=29_tests_on_Solaris?=
Message-ID: <E1RSXHp-0008LP-Ny@dinsdale.python.org>

http://hg.python.org/cpython/rev/d6d15fcf5eb6
changeset:   73664:d6d15fcf5eb6
user:        Victor Stinner <vstinner at wyplay.com>
date:        Mon Nov 21 18:04:30 2011 +0100
summary:
  Issue #13441: Reenable strxfrm() tests on Solaris

files:
  Lib/test/test__locale.py |  2 --
  Lib/test/test_locale.py  |  4 ----
  2 files changed, 0 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py
--- a/Lib/test/test__locale.py
+++ b/Lib/test/test__locale.py
@@ -113,8 +113,6 @@
                                                 nl_radixchar, li_radixchar,
                                                 loc, set_locale))
 
-    @unittest.skipIf(sys.platform == 'sunos5',
-                     "http://bugs.python.org/issue13441")
     def test_float_parsing(self):
         # Bug #1391872: Test whether float parsing is okay on European
         # locales.
diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py
--- a/Lib/test/test_locale.py
+++ b/Lib/test/test_locale.py
@@ -343,8 +343,6 @@
         self.assertEqual(locale.strcoll('a', 'a'), 0)
         self.assertGreater(locale.strcoll('b', 'a'), 0)
 
-    @unittest.skipIf(sys.platform == 'sunos5',
-                     "http://bugs.python.org/issue13441")
     def test_strxfrm(self):
         self.assertLess(locale.strxfrm('a'), locale.strxfrm('b'))
 
@@ -366,8 +364,6 @@
     def test_strcoll_with_diacritic(self):
         self.assertLess(locale.strcoll('?', 'b'), 0)
 
-    @unittest.skipIf(sys.platform == 'sunos5',
-                     "http://bugs.python.org/issue13441")
     def test_strxfrm_with_diacritic(self):
         self.assertLess(locale.strxfrm('?'), locale.strxfrm('b'))
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 20:31:11 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 20:31:11 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzIy?=
 =?utf8?q?=3A_Fix_BufferedWriter=2Ewrite=28=29_to_ensure_that_BlockingIOEr?=
 =?utf8?q?ror_is?=
Message-ID: <E1RSZaB-0005gA-U8@dinsdale.python.org>

http://hg.python.org/cpython/rev/ac2c4c62b486
changeset:   73665:ac2c4c62b486
branch:      3.2
parent:      73645:ba3da86d1c5d
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 20:16:44 2011 +0100
summary:
  Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
raised when the wrapped raw file is non-blocking and the write would block.
Previous code assumed that the raw write() would raise BlockingIOError, but
RawIOBase.write() is defined to returned None when the call would block.
Patch by sbt.

files:
  Lib/_pyio.py             |  51 ++++++++---------
  Lib/test/test_io.py      |  78 ++++++++++++++++++++++++++-
  Misc/ACKS                |   1 +
  Misc/NEWS                |   6 ++
  Modules/_io/bufferedio.c |  78 ++++++++++++++++-----------
  5 files changed, 152 insertions(+), 62 deletions(-)


diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -6,6 +6,7 @@
 import abc
 import codecs
 import warnings
+import errno
 # Import _thread instead of threading to reduce startup cost
 try:
     from _thread import allocate_lock as Lock
@@ -720,8 +721,11 @@
 
     def close(self):
         if self.raw is not None and not self.closed:
-            self.flush()
-            self.raw.close()
+            try:
+                # may raise BlockingIOError or BrokenPipeError etc
+                self.flush()
+            finally:
+                self.raw.close()
 
     def detach(self):
         if self.raw is None:
@@ -1080,13 +1084,9 @@
             # XXX we can implement some more tricks to try and avoid
             # partial writes
             if len(self._write_buf) > self.buffer_size:
-                # We're full, so let's pre-flush the buffer
-                try:
-                    self._flush_unlocked()
-                except BlockingIOError as e:
-                    # We can't accept anything else.
-                    # XXX Why not just let the exception pass through?
-                    raise BlockingIOError(e.errno, e.strerror, 0)
+                # We're full, so let's pre-flush the buffer.  (This may
+                # raise BlockingIOError with characters_written == 0.)
+                self._flush_unlocked()
             before = len(self._write_buf)
             self._write_buf.extend(b)
             written = len(self._write_buf) - before
@@ -1117,24 +1117,23 @@
     def _flush_unlocked(self):
         if self.closed:
             raise ValueError("flush of closed file")
-        written = 0
-        try:
-            while self._write_buf:
-                try:
-                    n = self.raw.write(self._write_buf)
-                except IOError as e:
-                    if e.errno != EINTR:
-                        raise
-                    continue
-                if n > len(self._write_buf) or n < 0:
-                    raise IOError("write() returned incorrect number of bytes")
-                del self._write_buf[:n]
-                written += n
-        except BlockingIOError as e:
-            n = e.characters_written
+        while self._write_buf:
+            try:
+                n = self.raw.write(self._write_buf)
+            except BlockingIOError:
+                raise RuntimeError("self.raw should implement RawIOBase: it "
+                                   "should not raise BlockingIOError")
+            except IOError as e:
+                if e.errno != EINTR:
+                    raise
+                continue
+            if n is None:
+                raise BlockingIOError(
+                    errno.EAGAIN,
+                    "write could not complete without blocking", 0)
+            if n > len(self._write_buf) or n < 0:
+                raise IOError("write() returned incorrect number of bytes")
             del self._write_buf[:n]
-            written += n
-            raise BlockingIOError(e.errno, e.strerror, written)
 
     def tell(self):
         return _BufferedIOMixin.tell(self) + len(self._write_buf)
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -42,7 +42,10 @@
     import threading
 except ImportError:
     threading = None
-
+try:
+    import fcntl
+except ImportError:
+    fcntl = None
 
 def _default_chunk_size():
     """Get the default TextIOWrapper chunk size"""
@@ -242,9 +245,14 @@
             except ValueError:
                 pass
             else:
-                self._blocker_char = None
-                self._write_stack.append(b[:n])
-                raise self.BlockingIOError(0, "test blocking", n)
+                if n > 0:
+                    # write data up to the first blocker
+                    self._write_stack.append(b[:n])
+                    return n
+                else:
+                    # cancel blocker and indicate would block
+                    self._blocker_char = None
+                    return None
         self._write_stack.append(b)
         return len(b)
 
@@ -2743,6 +2751,68 @@
                 with self.open(support.TESTFN, **kwargs) as f:
                     self.assertRaises(TypeError, pickle.dumps, f, protocol)
 
+    @unittest.skipUnless(fcntl, 'fcntl required for this test')
+    def test_nonblock_pipe_write_bigbuf(self):
+        self._test_nonblock_pipe_write(16*1024)
+
+    @unittest.skipUnless(fcntl, 'fcntl required for this test')
+    def test_nonblock_pipe_write_smallbuf(self):
+        self._test_nonblock_pipe_write(1024)
+
+    def _set_non_blocking(self, fd):
+        flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+        self.assertNotEqual(flags, -1)
+        res = fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+        self.assertEqual(res, 0)
+
+    def _test_nonblock_pipe_write(self, bufsize):
+        sent = []
+        received = []
+        r, w = os.pipe()
+        self._set_non_blocking(r)
+        self._set_non_blocking(w)
+
+        # To exercise all code paths in the C implementation we need
+        # to play with buffer sizes.  For instance, if we choose a
+        # buffer size less than or equal to _PIPE_BUF (4096 on Linux)
+        # then we will never get a partial write of the buffer.
+        rf = self.open(r, mode='rb', closefd=True, buffering=bufsize)
+        wf = self.open(w, mode='wb', closefd=True, buffering=bufsize)
+
+        with rf, wf:
+            for N in 9999, 73, 7574:
+                try:
+                    i = 0
+                    while True:
+                        msg = bytes([i % 26 + 97]) * N
+                        sent.append(msg)
+                        wf.write(msg)
+                        i += 1
+
+                except self.BlockingIOError as e:
+                    self.assertEqual(e.args[0], errno.EAGAIN)
+                    sent[-1] = sent[-1][:e.characters_written]
+                    received.append(rf.read())
+                    msg = b'BLOCKED'
+                    wf.write(msg)
+                    sent.append(msg)
+
+            while True:
+                try:
+                    wf.flush()
+                    break
+                except self.BlockingIOError as e:
+                    self.assertEqual(e.args[0], errno.EAGAIN)
+                    self.assertEqual(e.characters_written, 0)
+                    received.append(rf.read())
+
+            received += iter(rf.read, None)
+
+        sent, received = b''.join(sent), b''.join(received)
+        self.assertTrue(sent == received)
+        self.assertTrue(wf.closed)
+        self.assertTrue(rf.closed)
+
 class CMiscIOTest(MiscIOTest):
     io = io
 
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -796,6 +796,7 @@
 Mark Sapiro
 Ty Sarna
 Ben Sayer
+sbt
 Andrew Schaaf
 Michael Scharf
 Andreas Schawo
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -80,6 +80,12 @@
 Library
 -------
 
+- Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
+  raised when the wrapped raw file is non-blocking and the write would block.
+  Previous code assumed that the raw write() would raise BlockingIOError, but
+  RawIOBase.write() is defined to returned None when the call would block.
+  Patch by sbt.
+
 - Issue #13358: HTMLParser now calls handle_data only once for each CDATA.
 
 - Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -581,7 +581,7 @@
 
 /* Forward decls */
 static PyObject *
-_bufferedwriter_flush_unlocked(buffered *, int);
+_bufferedwriter_flush_unlocked(buffered *);
 static Py_ssize_t
 _bufferedreader_fill_buffer(buffered *self);
 static void
@@ -602,6 +602,18 @@
  * Helpers
  */
 
+/* Sets the current error to BlockingIOError */
+static void
+_set_BlockingIOError(char *msg, Py_ssize_t written)
+{
+    PyObject *err;
+    err = PyObject_CallFunction(PyExc_BlockingIOError, "isn",
+                                errno, msg, written);
+    if (err)
+        PyErr_SetObject(PyExc_BlockingIOError, err);
+    Py_XDECREF(err);
+}
+
 /* Returns the address of the `written` member if a BlockingIOError was
    raised, NULL otherwise. The error is always re-raised. */
 static Py_ssize_t *
@@ -756,7 +768,7 @@
 {
     PyObject *res;
 
-    res = _bufferedwriter_flush_unlocked(self, 0);
+    res = _bufferedwriter_flush_unlocked(self);
     if (res == NULL)
         return NULL;
     Py_DECREF(res);
@@ -1121,7 +1133,7 @@
 
     /* Fallback: invoke raw seek() method and clear buffer */
     if (self->writable) {
-        res = _bufferedwriter_flush_unlocked(self, 0);
+        res = _bufferedwriter_flush_unlocked(self);
         if (res == NULL)
             goto end;
         Py_CLEAR(res);
@@ -1714,6 +1726,7 @@
     Py_buffer buf;
     PyObject *memobj, *res;
     Py_ssize_t n;
+    int errnum;
     /* NOTE: the buffer needn't be released as its object is NULL. */
     if (PyBuffer_FillInfo(&buf, NULL, start, len, 1, PyBUF_CONTIG_RO) == -1)
         return -1;
@@ -1726,11 +1739,21 @@
        raised (see issue #10956).
     */
     do {
+        errno = 0;
         res = PyObject_CallMethodObjArgs(self->raw, _PyIO_str_write, memobj, NULL);
+        errnum = errno;
     } while (res == NULL && _trap_eintr());
     Py_DECREF(memobj);
     if (res == NULL)
         return -1;
+    if (res == Py_None) {
+        /* Non-blocking stream would have blocked. Special return code!
+           Being paranoid we reset errno in case it is changed by code
+           triggered by a decref.  errno is used by _set_BlockingIOError(). */
+        Py_DECREF(res);
+        errno = errnum;
+        return -2;
+    }
     n = PyNumber_AsSsize_t(res, PyExc_ValueError);
     Py_DECREF(res);
     if (n < 0 || n > len) {
@@ -1747,7 +1770,7 @@
 /* `restore_pos` is 1 if we need to restore the raw stream position at
    the end, 0 otherwise. */
 static PyObject *
-_bufferedwriter_flush_unlocked(buffered *self, int restore_pos)
+_bufferedwriter_flush_unlocked(buffered *self)
 {
     Py_ssize_t written = 0;
     Py_off_t n, rewind;
@@ -1769,14 +1792,11 @@
             Py_SAFE_DOWNCAST(self->write_end - self->write_pos,
                              Py_off_t, Py_ssize_t));
         if (n == -1) {
-            Py_ssize_t *w = _buffered_check_blocking_error();
-            if (w == NULL)
-                goto error;
-            self->write_pos += *w;
-            self->raw_pos = self->write_pos;
-            written += *w;
-            *w = written;
-            /* Already re-raised */
+            goto error;
+        }
+        else if (n == -2) {
+            _set_BlockingIOError("write could not complete without blocking",
+                                 0);
             goto error;
         }
         self->write_pos += n;
@@ -1789,16 +1809,6 @@
             goto error;
     }
 
-    if (restore_pos) {
-        Py_off_t forward = rewind - written;
-        if (forward != 0) {
-            n = _buffered_raw_seek(self, forward, 1);
-            if (n < 0) {
-                goto error;
-            }
-            self->raw_pos += forward;
-        }
-    }
     _bufferedwriter_reset_buf(self);
 
 end:
@@ -1851,7 +1861,7 @@
     }
 
     /* First write the current buffer */
-    res = _bufferedwriter_flush_unlocked(self, 0);
+    res = _bufferedwriter_flush_unlocked(self);
     if (res == NULL) {
         Py_ssize_t *w = _buffered_check_blocking_error();
         if (w == NULL)
@@ -1874,14 +1884,19 @@
             PyErr_Clear();
             memcpy(self->buffer + self->write_end, buf.buf, buf.len);
             self->write_end += buf.len;
+            self->pos += buf.len;
             written = buf.len;
             goto end;
         }
         /* Buffer as much as possible. */
         memcpy(self->buffer + self->write_end, buf.buf, avail);
         self->write_end += avail;
-        /* Already re-raised */
-        *w = avail;
+        self->pos += avail;
+        /* XXX Modifying the existing exception e using the pointer w
+           will change e.characters_written but not e.args[2].
+           Therefore we just replace with a new error. */
+        _set_BlockingIOError("write could not complete without blocking",
+                             avail);
         goto error;
     }
     Py_CLEAR(res);
@@ -1906,11 +1921,9 @@
         Py_ssize_t n = _bufferedwriter_raw_write(
             self, (char *) buf.buf + written, buf.len - written);
         if (n == -1) {
-            Py_ssize_t *w = _buffered_check_blocking_error();
-            if (w == NULL)
-                goto error;
-            written += *w;
-            remaining -= *w;
+            goto error;
+        } else if (n == -2) {
+            /* Write failed because raw file is non-blocking */
             if (remaining > self->buffer_size) {
                 /* Can't buffer everything, still buffer as much as possible */
                 memcpy(self->buffer,
@@ -1918,8 +1931,9 @@
                 self->raw_pos = 0;
                 ADJUST_POSITION(self, self->buffer_size);
                 self->write_end = self->buffer_size;
-                *w = written + self->buffer_size;
-                /* Already re-raised */
+                written += self->buffer_size;
+                _set_BlockingIOError("write could not complete without "
+                                     "blocking", written);
                 goto error;
             }
             PyErr_Clear();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 20:31:12 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 20:31:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313322=3A_Fix_BufferedWriter=2Ewrite=28=29_to_ensure?=
 =?utf8?q?_that_BlockingIOError_is?=
Message-ID: <E1RSZaC-0005gE-Ql@dinsdale.python.org>

http://hg.python.org/cpython/rev/3cd1985ed04f
changeset:   73666:3cd1985ed04f
parent:      73664:d6d15fcf5eb6
parent:      73665:ac2c4c62b486
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 20:22:01 2011 +0100
summary:
  Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
raised when the wrapped raw file is non-blocking and the write would block.
Previous code assumed that the raw write() would raise BlockingIOError, but
RawIOBase.write() is defined to returned None when the call would block.
Patch by sbt.

files:
  Lib/_pyio.py             |  47 ++++++++--------
  Lib/test/test_io.py      |  80 ++++++++++++++++++++++++++-
  Misc/ACKS                |   1 +
  Misc/NEWS                |   6 ++
  Modules/_io/bufferedio.c |  78 ++++++++++++++++-----------
  5 files changed, 152 insertions(+), 60 deletions(-)


diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -6,6 +6,7 @@
 import abc
 import codecs
 import warnings
+import errno
 # Import _thread instead of threading to reduce startup cost
 try:
     from _thread import allocate_lock as Lock
@@ -717,8 +718,11 @@
 
     def close(self):
         if self.raw is not None and not self.closed:
-            self.flush()
-            self.raw.close()
+            try:
+                # may raise BlockingIOError or BrokenPipeError etc
+                self.flush()
+            finally:
+                self.raw.close()
 
     def detach(self):
         if self.raw is None:
@@ -1077,13 +1081,9 @@
             # XXX we can implement some more tricks to try and avoid
             # partial writes
             if len(self._write_buf) > self.buffer_size:
-                # We're full, so let's pre-flush the buffer
-                try:
-                    self._flush_unlocked()
-                except BlockingIOError as e:
-                    # We can't accept anything else.
-                    # XXX Why not just let the exception pass through?
-                    raise BlockingIOError(e.errno, e.strerror, 0)
+                # We're full, so let's pre-flush the buffer.  (This may
+                # raise BlockingIOError with characters_written == 0.)
+                self._flush_unlocked()
             before = len(self._write_buf)
             self._write_buf.extend(b)
             written = len(self._write_buf) - before
@@ -1114,22 +1114,21 @@
     def _flush_unlocked(self):
         if self.closed:
             raise ValueError("flush of closed file")
-        written = 0
-        try:
-            while self._write_buf:
-                try:
-                    n = self.raw.write(self._write_buf)
-                except InterruptedError:
-                    continue
-                if n > len(self._write_buf) or n < 0:
-                    raise IOError("write() returned incorrect number of bytes")
-                del self._write_buf[:n]
-                written += n
-        except BlockingIOError as e:
-            n = e.characters_written
+        while self._write_buf:
+            try:
+                n = self.raw.write(self._write_buf)
+            except InterruptedError:
+                continue
+            except BlockingIOError:
+                raise RuntimeError("self.raw should implement RawIOBase: it "
+                                   "should not raise BlockingIOError")
+            if n is None:
+                raise BlockingIOError(
+                    errno.EAGAIN,
+                    "write could not complete without blocking", 0)
+            if n > len(self._write_buf) or n < 0:
+                raise IOError("write() returned incorrect number of bytes")
             del self._write_buf[:n]
-            written += n
-            raise BlockingIOError(e.errno, e.strerror, written)
 
     def tell(self):
         return _BufferedIOMixin.tell(self) + len(self._write_buf)
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -42,7 +42,10 @@
     import threading
 except ImportError:
     threading = None
-
+try:
+    import fcntl
+except ImportError:
+    fcntl = None
 
 def _default_chunk_size():
     """Get the default TextIOWrapper chunk size"""
@@ -242,9 +245,14 @@
             except ValueError:
                 pass
             else:
-                self._blocker_char = None
-                self._write_stack.append(b[:n])
-                raise self.BlockingIOError(0, "test blocking", n)
+                if n > 0:
+                    # write data up to the first blocker
+                    self._write_stack.append(b[:n])
+                    return n
+                else:
+                    # cancel blocker and indicate would block
+                    self._blocker_char = None
+                    return None
         self._write_stack.append(b)
         return len(b)
 
@@ -2753,6 +2761,70 @@
                 with self.open(support.TESTFN, **kwargs) as f:
                     self.assertRaises(TypeError, pickle.dumps, f, protocol)
 
+    @unittest.skipUnless(fcntl, 'fcntl required for this test')
+    def test_nonblock_pipe_write_bigbuf(self):
+        self._test_nonblock_pipe_write(16*1024)
+
+    @unittest.skipUnless(fcntl, 'fcntl required for this test')
+    def test_nonblock_pipe_write_smallbuf(self):
+        self._test_nonblock_pipe_write(1024)
+
+    def _set_non_blocking(self, fd):
+        flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+        self.assertNotEqual(flags, -1)
+        res = fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+        self.assertEqual(res, 0)
+
+    def _test_nonblock_pipe_write(self, bufsize):
+        sent = []
+        received = []
+        r, w = os.pipe()
+        self._set_non_blocking(r)
+        self._set_non_blocking(w)
+
+        # To exercise all code paths in the C implementation we need
+        # to play with buffer sizes.  For instance, if we choose a
+        # buffer size less than or equal to _PIPE_BUF (4096 on Linux)
+        # then we will never get a partial write of the buffer.
+        rf = self.open(r, mode='rb', closefd=True, buffering=bufsize)
+        wf = self.open(w, mode='wb', closefd=True, buffering=bufsize)
+
+        with rf, wf:
+            for N in 9999, 73, 7574:
+                try:
+                    i = 0
+                    while True:
+                        msg = bytes([i % 26 + 97]) * N
+                        sent.append(msg)
+                        wf.write(msg)
+                        i += 1
+
+                except self.BlockingIOError as e:
+                    self.assertEqual(e.args[0], errno.EAGAIN)
+                    self.assertEqual(e.args[2], e.characters_written)
+                    sent[-1] = sent[-1][:e.characters_written]
+                    received.append(rf.read())
+                    msg = b'BLOCKED'
+                    wf.write(msg)
+                    sent.append(msg)
+
+            while True:
+                try:
+                    wf.flush()
+                    break
+                except self.BlockingIOError as e:
+                    self.assertEqual(e.args[0], errno.EAGAIN)
+                    self.assertEqual(e.args[2], e.characters_written)
+                    self.assertEqual(e.characters_written, 0)
+                    received.append(rf.read())
+
+            received += iter(rf.read, None)
+
+        sent, received = b''.join(sent), b''.join(received)
+        self.assertTrue(sent == received)
+        self.assertTrue(wf.closed)
+        self.assertTrue(rf.closed)
+
 class CMiscIOTest(MiscIOTest):
     io = io
 
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -860,6 +860,7 @@
 Mark Sapiro
 Ty Sarna
 Ben Sayer
+sbt
 Andrew Schaaf
 Michael Scharf
 Andreas Schawo
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -384,6 +384,12 @@
 Library
 -------
 
+- Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
+  raised when the wrapped raw file is non-blocking and the write would block.
+  Previous code assumed that the raw write() would raise BlockingIOError, but
+  RawIOBase.write() is defined to returned None when the call would block.
+  Patch by sbt.
+
 - Issue #13358: HTMLParser now calls handle_data only once for each CDATA.
 
 - Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -596,7 +596,7 @@
 
 /* Forward decls */
 static PyObject *
-_bufferedwriter_flush_unlocked(buffered *, int);
+_bufferedwriter_flush_unlocked(buffered *);
 static Py_ssize_t
 _bufferedreader_fill_buffer(buffered *self);
 static void
@@ -618,6 +618,18 @@
  * Helpers
  */
 
+/* Sets the current error to BlockingIOError */
+static void
+_set_BlockingIOError(char *msg, Py_ssize_t written)
+{
+    PyObject *err;
+    err = PyObject_CallFunction(PyExc_BlockingIOError, "isn",
+                                errno, msg, written);
+    if (err)
+        PyErr_SetObject(PyExc_BlockingIOError, err);
+    Py_XDECREF(err);
+}
+
 /* Returns the address of the `written` member if a BlockingIOError was
    raised, NULL otherwise. The error is always re-raised. */
 static Py_ssize_t *
@@ -772,7 +784,7 @@
 {
     PyObject *res;
 
-    res = _bufferedwriter_flush_unlocked(self, 0);
+    res = _bufferedwriter_flush_unlocked(self);
     if (res == NULL)
         return NULL;
     Py_DECREF(res);
@@ -1188,7 +1200,7 @@
 
     /* Fallback: invoke raw seek() method and clear buffer */
     if (self->writable) {
-        res = _bufferedwriter_flush_unlocked(self, 0);
+        res = _bufferedwriter_flush_unlocked(self);
         if (res == NULL)
             goto end;
         Py_CLEAR(res);
@@ -1807,6 +1819,7 @@
     Py_buffer buf;
     PyObject *memobj, *res;
     Py_ssize_t n;
+    int errnum;
     /* NOTE: the buffer needn't be released as its object is NULL. */
     if (PyBuffer_FillInfo(&buf, NULL, start, len, 1, PyBUF_CONTIG_RO) == -1)
         return -1;
@@ -1819,11 +1832,21 @@
        raised (see issue #10956).
     */
     do {
+        errno = 0;
         res = PyObject_CallMethodObjArgs(self->raw, _PyIO_str_write, memobj, NULL);
+        errnum = errno;
     } while (res == NULL && _trap_eintr());
     Py_DECREF(memobj);
     if (res == NULL)
         return -1;
+    if (res == Py_None) {
+        /* Non-blocking stream would have blocked. Special return code!
+           Being paranoid we reset errno in case it is changed by code
+           triggered by a decref.  errno is used by _set_BlockingIOError(). */
+        Py_DECREF(res);
+        errno = errnum;
+        return -2;
+    }
     n = PyNumber_AsSsize_t(res, PyExc_ValueError);
     Py_DECREF(res);
     if (n < 0 || n > len) {
@@ -1840,7 +1863,7 @@
 /* `restore_pos` is 1 if we need to restore the raw stream position at
    the end, 0 otherwise. */
 static PyObject *
-_bufferedwriter_flush_unlocked(buffered *self, int restore_pos)
+_bufferedwriter_flush_unlocked(buffered *self)
 {
     Py_ssize_t written = 0;
     Py_off_t n, rewind;
@@ -1862,14 +1885,11 @@
             Py_SAFE_DOWNCAST(self->write_end - self->write_pos,
                              Py_off_t, Py_ssize_t));
         if (n == -1) {
-            Py_ssize_t *w = _buffered_check_blocking_error();
-            if (w == NULL)
-                goto error;
-            self->write_pos += *w;
-            self->raw_pos = self->write_pos;
-            written += *w;
-            *w = written;
-            /* Already re-raised */
+            goto error;
+        }
+        else if (n == -2) {
+            _set_BlockingIOError("write could not complete without blocking",
+                                 0);
             goto error;
         }
         self->write_pos += n;
@@ -1882,16 +1902,6 @@
             goto error;
     }
 
-    if (restore_pos) {
-        Py_off_t forward = rewind - written;
-        if (forward != 0) {
-            n = _buffered_raw_seek(self, forward, 1);
-            if (n < 0) {
-                goto error;
-            }
-            self->raw_pos += forward;
-        }
-    }
     _bufferedwriter_reset_buf(self);
 
 end:
@@ -1944,7 +1954,7 @@
     }
 
     /* First write the current buffer */
-    res = _bufferedwriter_flush_unlocked(self, 0);
+    res = _bufferedwriter_flush_unlocked(self);
     if (res == NULL) {
         Py_ssize_t *w = _buffered_check_blocking_error();
         if (w == NULL)
@@ -1967,14 +1977,19 @@
             PyErr_Clear();
             memcpy(self->buffer + self->write_end, buf.buf, buf.len);
             self->write_end += buf.len;
+            self->pos += buf.len;
             written = buf.len;
             goto end;
         }
         /* Buffer as much as possible. */
         memcpy(self->buffer + self->write_end, buf.buf, avail);
         self->write_end += avail;
-        /* Already re-raised */
-        *w = avail;
+        self->pos += avail;
+        /* XXX Modifying the existing exception e using the pointer w
+           will change e.characters_written but not e.args[2].
+           Therefore we just replace with a new error. */
+        _set_BlockingIOError("write could not complete without blocking",
+                             avail);
         goto error;
     }
     Py_CLEAR(res);
@@ -1999,11 +2014,9 @@
         Py_ssize_t n = _bufferedwriter_raw_write(
             self, (char *) buf.buf + written, buf.len - written);
         if (n == -1) {
-            Py_ssize_t *w = _buffered_check_blocking_error();
-            if (w == NULL)
-                goto error;
-            written += *w;
-            remaining -= *w;
+            goto error;
+        } else if (n == -2) {
+            /* Write failed because raw file is non-blocking */
             if (remaining > self->buffer_size) {
                 /* Can't buffer everything, still buffer as much as possible */
                 memcpy(self->buffer,
@@ -2011,8 +2024,9 @@
                 self->raw_pos = 0;
                 ADJUST_POSITION(self, self->buffer_size);
                 self->write_end = self->buffer_size;
-                *w = written + self->buffer_size;
-                /* Already re-raised */
+                written += self->buffer_size;
+                _set_BlockingIOError("write could not complete without "
+                                     "blocking", written);
                 goto error;
             }
             PyErr_Clear();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 20:31:13 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 20:31:13 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMzIy?=
 =?utf8?q?=3A_Fix_BufferedWriter=2Ewrite=28=29_to_ensure_that_BlockingIOEr?=
 =?utf8?q?ror_is?=
Message-ID: <E1RSZaD-0005gI-OE@dinsdale.python.org>

http://hg.python.org/cpython/rev/e84e17643eeb
changeset:   73667:e84e17643eeb
branch:      2.7
parent:      73647:6e1e1118adca
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 20:16:44 2011 +0100
summary:
  Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
raised when the wrapped raw file is non-blocking and the write would block.
Previous code assumed that the raw write() would raise BlockingIOError, but
RawIOBase.write() is defined to returned None when the call would block.
Patch by sbt.

files:
  Lib/_pyio.py             |  51 ++++++++---------
  Lib/test/test_io.py      |  77 ++++++++++++++++++++++++++-
  Misc/ACKS                |   1 +
  Misc/NEWS                |   6 ++
  Modules/_io/bufferedio.c |  78 ++++++++++++++++-----------
  5 files changed, 152 insertions(+), 61 deletions(-)


diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -8,6 +8,7 @@
 import abc
 import codecs
 import warnings
+import errno
 # Import thread instead of threading to reduce startup cost
 try:
     from thread import allocate_lock as Lock
@@ -720,8 +721,11 @@
 
     def close(self):
         if self.raw is not None and not self.closed:
-            self.flush()
-            self.raw.close()
+            try:
+                # may raise BlockingIOError or BrokenPipeError etc
+                self.flush()
+            finally:
+                self.raw.close()
 
     def detach(self):
         if self.raw is None:
@@ -1074,13 +1078,9 @@
             # XXX we can implement some more tricks to try and avoid
             # partial writes
             if len(self._write_buf) > self.buffer_size:
-                # We're full, so let's pre-flush the buffer
-                try:
-                    self._flush_unlocked()
-                except BlockingIOError as e:
-                    # We can't accept anything else.
-                    # XXX Why not just let the exception pass through?
-                    raise BlockingIOError(e.errno, e.strerror, 0)
+                # We're full, so let's pre-flush the buffer.  (This may
+                # raise BlockingIOError with characters_written == 0.)
+                self._flush_unlocked()
             before = len(self._write_buf)
             self._write_buf.extend(b)
             written = len(self._write_buf) - before
@@ -1111,24 +1111,23 @@
     def _flush_unlocked(self):
         if self.closed:
             raise ValueError("flush of closed file")
-        written = 0
-        try:
-            while self._write_buf:
-                try:
-                    n = self.raw.write(self._write_buf)
-                except IOError as e:
-                    if e.errno != EINTR:
-                        raise
-                    continue
-                if n > len(self._write_buf) or n < 0:
-                    raise IOError("write() returned incorrect number of bytes")
-                del self._write_buf[:n]
-                written += n
-        except BlockingIOError as e:
-            n = e.characters_written
+        while self._write_buf:
+            try:
+                n = self.raw.write(self._write_buf)
+            except BlockingIOError:
+                raise RuntimeError("self.raw should implement RawIOBase: it "
+                                   "should not raise BlockingIOError")
+            except IOError as e:
+                if e.errno != EINTR:
+                    raise
+                continue
+            if n is None:
+                raise BlockingIOError(
+                    errno.EAGAIN,
+                    "write could not complete without blocking", 0)
+            if n > len(self._write_buf) or n < 0:
+                raise IOError("write() returned incorrect number of bytes")
             del self._write_buf[:n]
-            written += n
-            raise BlockingIOError(e.errno, e.strerror, written)
 
     def tell(self):
         return _BufferedIOMixin.tell(self) + len(self._write_buf)
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -43,6 +43,10 @@
     import threading
 except ImportError:
     threading = None
+try:
+    import fcntl
+except ImportError:
+    fcntl = None
 
 __metaclass__ = type
 bytes = support.py3k_bytes
@@ -228,9 +232,14 @@
             except ValueError:
                 pass
             else:
-                self._blocker_char = None
-                self._write_stack.append(b[:n])
-                raise self.BlockingIOError(0, "test blocking", n)
+                if n > 0:
+                    # write data up to the first blocker
+                    self._write_stack.append(b[:n])
+                    return n
+                else:
+                    # cancel blocker and indicate would block
+                    self._blocker_char = None
+                    return None
         self._write_stack.append(b)
         return len(b)
 
@@ -2610,6 +2619,68 @@
         # baseline "io" module.
         self._check_abc_inheritance(io)
 
+    @unittest.skipUnless(fcntl, 'fcntl required for this test')
+    def test_nonblock_pipe_write_bigbuf(self):
+        self._test_nonblock_pipe_write(16*1024)
+
+    @unittest.skipUnless(fcntl, 'fcntl required for this test')
+    def test_nonblock_pipe_write_smallbuf(self):
+        self._test_nonblock_pipe_write(1024)
+
+    def _set_non_blocking(self, fd):
+        flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+        self.assertNotEqual(flags, -1)
+        res = fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+        self.assertEqual(res, 0)
+
+    def _test_nonblock_pipe_write(self, bufsize):
+        sent = []
+        received = []
+        r, w = os.pipe()
+        self._set_non_blocking(r)
+        self._set_non_blocking(w)
+
+        # To exercise all code paths in the C implementation we need
+        # to play with buffer sizes.  For instance, if we choose a
+        # buffer size less than or equal to _PIPE_BUF (4096 on Linux)
+        # then we will never get a partial write of the buffer.
+        rf = self.open(r, mode='rb', closefd=True, buffering=bufsize)
+        wf = self.open(w, mode='wb', closefd=True, buffering=bufsize)
+
+        with rf, wf:
+            for N in 9999, 73, 7574:
+                try:
+                    i = 0
+                    while True:
+                        msg = bytes([i % 26 + 97]) * N
+                        sent.append(msg)
+                        wf.write(msg)
+                        i += 1
+
+                except self.BlockingIOError as e:
+                    self.assertEqual(e.args[0], errno.EAGAIN)
+                    sent[-1] = sent[-1][:e.characters_written]
+                    received.append(rf.read())
+                    msg = b'BLOCKED'
+                    wf.write(msg)
+                    sent.append(msg)
+
+            while True:
+                try:
+                    wf.flush()
+                    break
+                except self.BlockingIOError as e:
+                    self.assertEqual(e.args[0], errno.EAGAIN)
+                    self.assertEqual(e.characters_written, 0)
+                    received.append(rf.read())
+
+            received += iter(rf.read, None)
+
+        sent, received = b''.join(sent), b''.join(received)
+        self.assertTrue(sent == received)
+        self.assertTrue(wf.closed)
+        self.assertTrue(rf.closed)
+
 class CMiscIOTest(MiscIOTest):
     io = io
 
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -735,6 +735,7 @@
 Mark Sapiro
 Ty Sarna
 Ben Sayer
+sbt
 Michael Scharf
 Neil Schemenauer
 David Scherer
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,12 @@
 Library
 -------
 
+- Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
+  raised when the wrapped raw file is non-blocking and the write would block.
+  Previous code assumed that the raw write() would raise BlockingIOError, but
+  RawIOBase.write() is defined to returned None when the call would block.
+  Patch by sbt.
+
 - Issue #13358: HTMLParser now calls handle_data only once for each CDATA.
 
 - Issue #4147: minidom's toprettyxml no longer adds whitespace around a text
diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c
+++ b/Modules/_io/bufferedio.c
@@ -550,7 +550,7 @@
 
 /* Forward decls */
 static PyObject *
-_bufferedwriter_flush_unlocked(buffered *, int);
+_bufferedwriter_flush_unlocked(buffered *);
 static Py_ssize_t
 _bufferedreader_fill_buffer(buffered *self);
 static void
@@ -571,6 +571,18 @@
  * Helpers
  */
 
+/* Sets the current error to BlockingIOError */
+static void
+_set_BlockingIOError(char *msg, Py_ssize_t written)
+{
+    PyObject *err;
+    err = PyObject_CallFunction(PyExc_BlockingIOError, "isn",
+                                errno, msg, written);
+    if (err)
+        PyErr_SetObject(PyExc_BlockingIOError, err);
+    Py_XDECREF(err);
+}
+
 /* Returns the address of the `written` member if a BlockingIOError was
    raised, NULL otherwise. The error is always re-raised. */
 static Py_ssize_t *
@@ -725,7 +737,7 @@
 {
     PyObject *res;
 
-    res = _bufferedwriter_flush_unlocked(self, 0);
+    res = _bufferedwriter_flush_unlocked(self);
     if (res == NULL)
         return NULL;
     Py_DECREF(res);
@@ -1087,7 +1099,7 @@
 
     /* Fallback: invoke raw seek() method and clear buffer */
     if (self->writable) {
-        res = _bufferedwriter_flush_unlocked(self, 0);
+        res = _bufferedwriter_flush_unlocked(self);
         if (res == NULL)
             goto end;
         Py_CLEAR(res);
@@ -1683,6 +1695,7 @@
     Py_buffer buf;
     PyObject *memobj, *res;
     Py_ssize_t n;
+    int errnum;
     /* NOTE: the buffer needn't be released as its object is NULL. */
     if (PyBuffer_FillInfo(&buf, NULL, start, len, 1, PyBUF_CONTIG_RO) == -1)
         return -1;
@@ -1695,11 +1708,21 @@
        raised (see issue #10956).
     */
     do {
+        errno = 0;
         res = PyObject_CallMethodObjArgs(self->raw, _PyIO_str_write, memobj, NULL);
+        errnum = errno;
     } while (res == NULL && _trap_eintr());
     Py_DECREF(memobj);
     if (res == NULL)
         return -1;
+    if (res == Py_None) {
+        /* Non-blocking stream would have blocked. Special return code!
+           Being paranoid we reset errno in case it is changed by code
+           triggered by a decref.  errno is used by _set_BlockingIOError(). */
+        Py_DECREF(res);
+        errno = errnum;
+        return -2;
+    }
     n = PyNumber_AsSsize_t(res, PyExc_ValueError);
     Py_DECREF(res);
     if (n < 0 || n > len) {
@@ -1716,7 +1739,7 @@
 /* `restore_pos` is 1 if we need to restore the raw stream position at
    the end, 0 otherwise. */
 static PyObject *
-_bufferedwriter_flush_unlocked(buffered *self, int restore_pos)
+_bufferedwriter_flush_unlocked(buffered *self)
 {
     Py_ssize_t written = 0;
     Py_off_t n, rewind;
@@ -1738,14 +1761,11 @@
             Py_SAFE_DOWNCAST(self->write_end - self->write_pos,
                              Py_off_t, Py_ssize_t));
         if (n == -1) {
-            Py_ssize_t *w = _buffered_check_blocking_error();
-            if (w == NULL)
-                goto error;
-            self->write_pos += *w;
-            self->raw_pos = self->write_pos;
-            written += *w;
-            *w = written;
-            /* Already re-raised */
+            goto error;
+        }
+        else if (n == -2) {
+            _set_BlockingIOError("write could not complete without blocking",
+                                 0);
             goto error;
         }
         self->write_pos += n;
@@ -1758,16 +1778,6 @@
             goto error;
     }
 
-    if (restore_pos) {
-        Py_off_t forward = rewind - written;
-        if (forward != 0) {
-            n = _buffered_raw_seek(self, forward, 1);
-            if (n < 0) {
-                goto error;
-            }
-            self->raw_pos += forward;
-        }
-    }
     _bufferedwriter_reset_buf(self);
 
 end:
@@ -1820,7 +1830,7 @@
     }
 
     /* First write the current buffer */
-    res = _bufferedwriter_flush_unlocked(self, 0);
+    res = _bufferedwriter_flush_unlocked(self);
     if (res == NULL) {
         Py_ssize_t *w = _buffered_check_blocking_error();
         if (w == NULL)
@@ -1843,14 +1853,19 @@
             PyErr_Clear();
             memcpy(self->buffer + self->write_end, buf.buf, buf.len);
             self->write_end += buf.len;
+            self->pos += buf.len;
             written = buf.len;
             goto end;
         }
         /* Buffer as much as possible. */
         memcpy(self->buffer + self->write_end, buf.buf, avail);
         self->write_end += avail;
-        /* Already re-raised */
-        *w = avail;
+        self->pos += avail;
+        /* XXX Modifying the existing exception e using the pointer w
+           will change e.characters_written but not e.args[2].
+           Therefore we just replace with a new error. */
+        _set_BlockingIOError("write could not complete without blocking",
+                             avail);
         goto error;
     }
     Py_CLEAR(res);
@@ -1875,11 +1890,9 @@
         Py_ssize_t n = _bufferedwriter_raw_write(
             self, (char *) buf.buf + written, buf.len - written);
         if (n == -1) {
-            Py_ssize_t *w = _buffered_check_blocking_error();
-            if (w == NULL)
-                goto error;
-            written += *w;
-            remaining -= *w;
+            goto error;
+        } else if (n == -2) {
+            /* Write failed because raw file is non-blocking */
             if (remaining > self->buffer_size) {
                 /* Can't buffer everything, still buffer as much as possible */
                 memcpy(self->buffer,
@@ -1887,8 +1900,9 @@
                 self->raw_pos = 0;
                 ADJUST_POSITION(self, self->buffer_size);
                 self->write_end = self->buffer_size;
-                *w = written + self->buffer_size;
-                /* Already re-raised */
+                written += self->buffer_size;
+                _set_BlockingIOError("write could not complete without "
+                                     "blocking", written);
                 goto error;
             }
             PyErr_Clear();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 20:44:06 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 20:44:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313417=3A_speed_up_?=
 =?utf8?q?utf-8_decoding_by_around_2x_for_the_non-fully-ASCII_case=2E?=
Message-ID: <E1RSZmg-0006iu-Du@dinsdale.python.org>

http://hg.python.org/cpython/rev/8e6c4acaf530
changeset:   73668:8e6c4acaf530
parent:      73666:3cd1985ed04f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 20:39:13 2011 +0100
summary:
  Issue #13417: speed up utf-8 decoding by around 2x for the non-fully-ASCII case.
This almost catches up with pre-PEP 393 performance, when decoding needed
only one pass.

files:
  Makefile.pre.in            |    1 +
  Objects/stringlib/codecs.h |  156 ++++++++++++++++
  Objects/unicodeobject.c    |  238 +++++++++++++-----------
  3 files changed, 283 insertions(+), 112 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -645,6 +645,7 @@
 
 UNICODE_DEPS = $(BYTESTR_DEPS) \
 		$(srcdir)/Objects/stringlib/asciilib.h \
+		$(srcdir)/Objects/stringlib/codecs.h \
 		$(srcdir)/Objects/stringlib/ucs1lib.h \
 		$(srcdir)/Objects/stringlib/ucs2lib.h \
 		$(srcdir)/Objects/stringlib/ucs4lib.h \
diff --git a/Objects/stringlib/codecs.h b/Objects/stringlib/codecs.h
new file mode 100644
--- /dev/null
+++ b/Objects/stringlib/codecs.h
@@ -0,0 +1,156 @@
+/* stringlib: codec implementations */
+
+#if STRINGLIB_IS_UNICODE
+
+/* Mask to check or force alignment of a pointer to C 'long' boundaries */
+#define LONG_PTR_MASK (size_t) (SIZEOF_LONG - 1)
+
+/* Mask to quickly check whether a C 'long' contains a
+   non-ASCII, UTF8-encoded char. */
+#if (SIZEOF_LONG == 8)
+# define ASCII_CHAR_MASK 0x8080808080808080L
+#elif (SIZEOF_LONG == 4)
+# define ASCII_CHAR_MASK 0x80808080L
+#else
+# error C 'long' size should be either 4 or 8!
+#endif
+
+Py_LOCAL_INLINE(int)
+STRINGLIB(utf8_try_decode)(const char *start, const char *end,
+                           STRINGLIB_CHAR *dest,
+                           const char **src_pos, Py_ssize_t *dest_index)
+{
+    int ret;
+    Py_ssize_t n;
+    const char *s = start;
+    const char *aligned_end = (const char *) ((size_t) end & ~LONG_PTR_MASK);
+    STRINGLIB_CHAR *p = dest;
+
+    while (s < end) {
+        Py_UCS4 ch = (unsigned char)*s;
+
+        if (ch < 0x80) {
+            /* Fast path for runs of ASCII characters. Given that common UTF-8
+               input will consist of an overwhelming majority of ASCII
+               characters, we try to optimize for this case by checking
+               as many characters as a C 'long' can contain.
+               First, check if we can do an aligned read, as most CPUs have
+               a penalty for unaligned reads.
+            */
+            if (!((size_t) s & LONG_PTR_MASK)) {
+                /* Help register allocation */
+                register const char *_s = s;
+                register STRINGLIB_CHAR *_p = p;
+                while (_s < aligned_end) {
+                    /* Read a whole long at a time (either 4 or 8 bytes),
+                       and do a fast unrolled copy if it only contains ASCII
+                       characters. */
+                    unsigned long value = *(unsigned long *) _s;
+                    if (value & ASCII_CHAR_MASK)
+                        break;
+                    _p[0] = _s[0];
+                    _p[1] = _s[1];
+                    _p[2] = _s[2];
+                    _p[3] = _s[3];
+#if (SIZEOF_LONG == 8)
+                    _p[4] = _s[4];
+                    _p[5] = _s[5];
+                    _p[6] = _s[6];
+                    _p[7] = _s[7];
+#endif
+                    _s += SIZEOF_LONG;
+                    _p += SIZEOF_LONG;
+                }
+                s = _s;
+                p = _p;
+                if (s == end)
+                    break;
+                ch = (unsigned char)*s;
+            }
+        }
+
+        if (ch < 0x80) {
+            s++;
+            *p++ = ch;
+            continue;
+        }
+
+        n = utf8_code_length[ch];
+
+        if (s + n > end) {
+            /* unexpected end of data: the caller will decide whether
+               it's an error or not */
+            goto _error;
+        }
+
+        switch (n) {
+        case 0:
+            /* invalid start byte */
+            goto _error;
+        case 1:
+            /* internal error */
+            goto _error;
+        case 2:
+            if ((s[1] & 0xc0) != 0x80)
+                /* invalid continuation byte */
+                goto _error;
+            ch = ((s[0] & 0x1f) << 6) + (s[1] & 0x3f);
+            assert ((ch > 0x007F) && (ch <= 0x07FF));
+            s += 2;
+            *p++ = ch;
+            break;
+
+        case 3:
+            /* Decoding UTF-8 sequences in range \xed\xa0\x80-\xed\xbf\xbf
+               will result in surrogates in range d800-dfff. Surrogates are
+               not valid UTF-8 so they are rejected.
+               See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf
+               (table 3-7) and http://www.rfc-editor.org/rfc/rfc3629.txt */
+            if ((s[1] & 0xc0) != 0x80 ||
+                (s[2] & 0xc0) != 0x80 ||
+                ((unsigned char)s[0] == 0xE0 &&
+                 (unsigned char)s[1] < 0xA0) ||
+                ((unsigned char)s[0] == 0xED &&
+                 (unsigned char)s[1] > 0x9F)) {
+                /* invalid continuation byte */
+                goto _error;
+            }
+            ch = ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + (s[2] & 0x3f);
+            assert ((ch > 0x07FF) && (ch <= 0xFFFF));
+            s += 3;
+            *p++ = ch;
+            break;
+
+        case 4:
+            if ((s[1] & 0xc0) != 0x80 ||
+                (s[2] & 0xc0) != 0x80 ||
+                (s[3] & 0xc0) != 0x80 ||
+                ((unsigned char)s[0] == 0xF0 &&
+                 (unsigned char)s[1] < 0x90) ||
+                ((unsigned char)s[0] == 0xF4 &&
+                 (unsigned char)s[1] > 0x8F)) {
+                /* invalid continuation byte */
+                goto _error;
+            }
+            ch = ((s[0] & 0x7) << 18) + ((s[1] & 0x3f) << 12) +
+                 ((s[2] & 0x3f) << 6) + (s[3] & 0x3f);
+            assert ((ch > 0xFFFF) && (ch <= 0x10ffff));
+            s += 4;
+            *p++ = ch;
+            break;
+        }
+    }
+    ret = 0;
+    goto _ok;
+_error:
+    ret = -1;
+_ok:
+    *src_pos = s;
+    *dest_index = p - dest;
+    return ret;
+}
+
+#undef LONG_PTR_MASK
+#undef ASCII_CHAR_MASK
+
+#endif /* STRINGLIB_IS_UNICODE */
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -523,6 +523,7 @@
 #include "stringlib/fastsearch.h"
 #include "stringlib/count.h"
 #include "stringlib/find.h"
+#include "stringlib/undef.h"
 
 /* --- Unicode Object ----------------------------------------------------- */
 
@@ -4190,6 +4191,18 @@
     return PyUnicode_DecodeUTF8Stateful(s, size, errors, NULL);
 }
 
+#include "stringlib/ucs1lib.h"
+#include "stringlib/codecs.h"
+#include "stringlib/undef.h"
+
+#include "stringlib/ucs2lib.h"
+#include "stringlib/codecs.h"
+#include "stringlib/undef.h"
+
+#include "stringlib/ucs4lib.h"
+#include "stringlib/codecs.h"
+#include "stringlib/undef.h"
+
 /* Mask to check or force alignment of a pointer to C 'long' boundaries */
 #define LONG_PTR_MASK (size_t) (SIZEOF_LONG - 1)
 
@@ -4203,33 +4216,41 @@
 # error C 'long' size should be either 4 or 8!
 #endif
 
-/* Scans a UTF-8 string and returns the maximum character to be expected,
-   the size of the decoded unicode string and if any major errors were
-   encountered.
-
-   This function does check basic UTF-8 sanity, it does however NOT CHECK
-   if the string contains surrogates, and if all continuation bytes are
-   within the correct ranges, these checks are performed in
+/* Scans a UTF-8 string and returns the maximum character to be expected
+   and the size of the decoded unicode string.
+
+   This function doesn't check for errors, these checks are performed in
    PyUnicode_DecodeUTF8Stateful.
-
-   If it sets has_errors to 1, it means the value of unicode_size and max_char
-   will be bogus and you should not rely on useful information in them.
    */
 static Py_UCS4
-utf8_max_char_size_and_has_errors(const char *s, Py_ssize_t string_size,
-                                  Py_ssize_t *unicode_size, Py_ssize_t* consumed,
-                                  int *has_errors)
-{
-    Py_ssize_t n;
+utf8_max_char_size_and_char_count(const char *s, Py_ssize_t string_size,
+                                  Py_ssize_t *unicode_size)
+{
     Py_ssize_t char_count = 0;
-    Py_UCS4 max_char = 127, new_max;
-    Py_UCS4 upper_bound;
     const unsigned char *p = (const unsigned char *)s;
     const unsigned char *end = p + string_size;
     const unsigned char *aligned_end = (const unsigned char *) ((size_t) end & ~LONG_PTR_MASK);
-    int err = 0;
-
-    for (; p < end && !err; ++p, ++char_count) {
+
+    assert(unicode_size != NULL);
+
+    /* By having a cascade of independent loops which fallback onto each
+       other, we minimize the amount of work done in the average loop
+       iteration, and we also maximize the CPU's ability to predict
+       branches correctly (because a given condition will have always the
+       same boolean outcome except perhaps in the last iteration of the
+       corresponding loop).
+       In the general case this brings us rather close to decoding
+       performance pre-PEP 393, despite the two-pass decoding.
+
+       Note that the pure ASCII loop is not duplicated once a non-ASCII
+       character has been encountered. It is actually a pessimization (by
+       a significant factor) to use this loop on text with many non-ASCII
+       characters, and it is important to avoid bad performance on valid
+       utf-8 data (invalid utf-8 being a different can of worms).
+    */
+
+    /* ASCII */
+    for (; p < end; ++p) {
         /* Only check value if it's not a ASCII char... */
         if (*p < 0x80) {
             /* Fast path, see below in PyUnicode_DecodeUTF8Stateful for
@@ -4249,76 +4270,59 @@
                     break;
             }
         }
-        if (*p >= 0x80) {
-            n = utf8_code_length[*p];
-            new_max = max_char;
-            switch (n) {
-            /* invalid start byte */
-            case 0:
-                err = 1;
-                break;
-            case 2:
-                /* Code points between 0x00FF and 0x07FF inclusive.
-                   Approximate the upper bound of the code point,
-                   if this flips over 255 we can be sure it will be more
-                   than 255 and the string will need 2 bytes per code coint,
-                   if it stays under or equal to 255, we can be sure 1 byte
-                   is enough.
-                   ((*p & 0b00011111) << 6) | 0b00111111 */
-                upper_bound = ((*p & 0x1F) << 6) | 0x3F;
-                if (max_char < upper_bound)
-                    new_max = upper_bound;
-                /* Ensure we track at least that we left ASCII space. */
-                if (new_max < 128)
-                    new_max = 128;
-                break;
-            case 3:
-                /* Between 0x0FFF and 0xFFFF inclusive, so values are
-                   always > 255 and <= 65535 and will always need 2 bytes. */
-                if (max_char < 65535)
-                    new_max = 65535;
-                break;
-            case 4:
-                /* Code point will be above 0xFFFF for sure in this case. */
-                new_max = 65537;
-                break;
-            /* Internal error, this should be caught by the first if */
-            case 1:
-            default:
-                assert(0 && "Impossible case in utf8_max_char_and_size");
-                err = 1;
-            }
-            /* Instead of number of overall bytes for this code point,
-               n contains the number of following bytes: */
-            --n;
-            /* Check if the follow up chars are all valid continuation bytes */
-            if (n >= 1) {
-                const unsigned char *cont;
-                if ((p + n) >= end) {
-                    if (consumed == 0)
-                        /* incomplete data, non-incremental decoding */
-                        err = 1;
-                    break;
-                }
-                for (cont = p + 1; cont <= (p + n); ++cont) {
-                    if ((*cont & 0xc0) != 0x80) {
-                        err = 1;
-                        break;
-                    }
-                }
-                p += n;
-            }
-            else
-                err = 1;
-            max_char = new_max;
-        }
-    }
-
-    if (unicode_size)
-        *unicode_size = char_count;
-    if (has_errors)
-        *has_errors = err;
-    return max_char;
+        if (*p < 0x80)
+            ++char_count;
+        else
+            goto _ucs1loop;
+    }
+    *unicode_size = char_count;
+    return 127;
+
+_ucs1loop:
+    for (; p < end; ++p) {
+        if (*p < 0xc4)
+            char_count += ((*p & 0xc0) != 0x80);
+        else
+            goto _ucs2loop;
+    }
+    *unicode_size = char_count;
+    return 255;
+
+_ucs2loop:
+    for (; p < end; ++p) {
+        if (*p < 0xf0)
+            char_count += ((*p & 0xc0) != 0x80);
+        else
+            goto _ucs4loop;
+    }
+    *unicode_size = char_count;
+    return 65535;
+
+_ucs4loop:
+    for (; p < end; ++p) {
+        char_count += ((*p & 0xc0) != 0x80);
+    }
+    *unicode_size = char_count;
+    return 65537;
+}
+
+/* Called when we encountered some error that wasn't detected in the original
+   scan, e.g. an encoded surrogate character. The original maxchar computation
+   may have been incorrect, so redo it. */
+static int
+refit_partial_string(PyObject **unicode, int kind, void *data, Py_ssize_t n)
+{
+    PyObject *tmp;
+    Py_ssize_t k, maxchar;
+    for (k = 0, maxchar = 0; k < n; k++)
+        maxchar = Py_MAX(maxchar, PyUnicode_READ(kind, data, k));
+    tmp = PyUnicode_New(PyUnicode_GET_LENGTH(*unicode), maxchar);
+    if (tmp == NULL)
+        return -1;
+    PyUnicode_CopyCharacters(tmp, 0, *unicode, 0, n);
+    Py_DECREF(*unicode);
+    *unicode = tmp;
+    return 0;
 }
 
 /* Similar to PyUnicode_WRITE but may attempt to widen and resize the string
@@ -4361,35 +4365,56 @@
     Py_ssize_t i;
     int kind;
     void *data;
-    int has_errors;
+    int has_errors = 0;
 
     if (size == 0) {
         if (consumed)
             *consumed = 0;
         return (PyObject *)PyUnicode_New(0, 0);
     }
-    maxchar = utf8_max_char_size_and_has_errors(s, size, &unicode_size,
-                                                consumed, &has_errors);
-    if (has_errors)
-        /* maxchar and size computation might be incorrect;
-           code below widens and resizes as necessary. */
-        unicode = PyUnicode_New(size, 127);
-    else
-        unicode = PyUnicode_New(unicode_size, maxchar);
+    maxchar = utf8_max_char_size_and_char_count(s, size, &unicode_size);
+    /* In case of errors, maxchar and size computation might be incorrect;
+       code below refits and resizes as necessary. */
+    unicode = PyUnicode_New(unicode_size, maxchar);
     if (!unicode)
         return NULL;
     /* When the string is ASCII only, just use memcpy and return.
        unicode_size may be != size if there is an incomplete UTF-8
        sequence at the end of the ASCII block.  */
-    if (!has_errors && maxchar < 128 && size == unicode_size) {
+    if (maxchar < 128 && size == unicode_size) {
         Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
         return unicode;
     }
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
+
     /* Unpack UTF-8 encoded data */
     i = 0;
     e = s + size;
+    switch (kind) {
+    case PyUnicode_1BYTE_KIND:
+        has_errors = ucs1lib_utf8_try_decode(s, e, (Py_UCS1 *) data, &s, &i);
+        break;
+    case PyUnicode_2BYTE_KIND:
+        has_errors = ucs2lib_utf8_try_decode(s, e, (Py_UCS2 *) data, &s, &i);
+        break;
+    case PyUnicode_4BYTE_KIND:
+        has_errors = ucs4lib_utf8_try_decode(s, e, (Py_UCS4 *) data, &s, &i);
+        break;
+    }
+    if (!has_errors) {
+        /* Ensure the unicode size calculation was correct */
+        assert(i == unicode_size);
+        assert(s == e);
+        if (consumed)
+            *consumed = s-starts;
+        return unicode;
+    }
+    /* Fall through to the generic decoding loop for the rest of
+       the string */
+    if (refit_partial_string(&unicode, kind, data, i) < 0)
+        goto onError;
+
     aligned_end = (const char *) ((size_t) e & ~LONG_PTR_MASK);
 
     while (s < e) {
@@ -4541,19 +4566,8 @@
 
       utf8Error:
         if (!has_errors) {
-            PyObject *tmp;
-            Py_ssize_t k;
-            /* We encountered some error that wasn't detected in the original scan,
-               e.g. an encoded surrogate character. The original maxchar computation may
-               have been incorrect, so redo it now. */
-            for (k = 0, maxchar = 0; k < i; k++)
-                maxchar = Py_MAX(maxchar, PyUnicode_READ(kind, data, k));
-            tmp = PyUnicode_New(PyUnicode_GET_LENGTH(unicode), maxchar);
-            if (tmp == NULL)
+            if (refit_partial_string(&unicode, kind, data, i) < 0)
                 goto onError;
-            PyUnicode_CopyCharacters(tmp, 0, unicode, 0, i);
-            Py_DECREF(unicode);
-            unicode = tmp;
             has_errors = 1;
         }
         if (unicode_decode_call_errorhandler(

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 20:53:55 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 20:53:55 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313411=3A_memoryvie?=
 =?utf8?q?w_objects_are_now_hashable_when_the_underlying_object_is?=
Message-ID: <E1RSZwB-0007ft-Gv@dinsdale.python.org>

http://hg.python.org/cpython/rev/215690b782f8
changeset:   73669:215690b782f8
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 20:46:33 2011 +0100
summary:
  Issue #13411: memoryview objects are now hashable when the underlying object is hashable.

files:
  Doc/library/stdtypes.rst    |  13 +++++++++
  Include/memoryobject.h      |   1 +
  Include/object.h            |   1 +
  Lib/test/test_memoryview.py |  27 +++++++++++++++++++
  Lib/test/test_sys.py        |   4 +-
  Misc/NEWS                   |   3 ++
  Objects/bytesobject.c       |  21 +++-----------
  Objects/memoryobject.c      |  34 ++++++++++++++++++++++++-
  Objects/object.c            |  15 +++++++++++
  9 files changed, 100 insertions(+), 19 deletions(-)


diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -2401,6 +2401,19 @@
 
    Notice how the size of the memoryview object cannot be changed.
 
+   Memoryviews of hashable (read-only) types are also hashable and their
+   hash value matches the corresponding bytes object::
+
+      >>> v = memoryview(b'abcefg')
+      >>> hash(v) == hash(b'abcefg')
+      True
+      >>> hash(v[2:4]) == hash(b'ce')
+      True
+
+   .. versionchanged:: 3.3
+      Memoryview objects are now hashable.
+
+
    :class:`memoryview` has several methods:
 
    .. method:: tobytes()
diff --git a/Include/memoryobject.h b/Include/memoryobject.h
--- a/Include/memoryobject.h
+++ b/Include/memoryobject.h
@@ -69,6 +69,7 @@
 typedef struct {
     PyObject_HEAD
     Py_buffer view;
+    Py_hash_t hash;
 } PyMemoryViewObject;
 #endif
 
diff --git a/Include/object.h b/Include/object.h
--- a/Include/object.h
+++ b/Include/object.h
@@ -519,6 +519,7 @@
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(Py_hash_t) _Py_HashDouble(double);
 PyAPI_FUNC(Py_hash_t) _Py_HashPointer(void*);
+PyAPI_FUNC(Py_hash_t) _Py_HashBytes(unsigned char*, Py_ssize_t);
 #endif
 
 /* Helper for passing objects to printf and the like */
diff --git a/Lib/test/test_memoryview.py b/Lib/test/test_memoryview.py
--- a/Lib/test/test_memoryview.py
+++ b/Lib/test/test_memoryview.py
@@ -283,6 +283,33 @@
         i = io.BytesIO(b'ZZZZ')
         self.assertRaises(TypeError, i.readinto, m)
 
+    def test_hash(self):
+        # Memoryviews of readonly (hashable) types are hashable, and they
+        # hash as the corresponding object.
+        tp = self.ro_type
+        if tp is None:
+            self.skipTest("no read-only type to test")
+        b = tp(self._source)
+        m = self._view(b)
+        self.assertEqual(hash(m), hash(b"abcdef"))
+        # Releasing the memoryview keeps the stored hash value (as with weakrefs)
+        m.release()
+        self.assertEqual(hash(m), hash(b"abcdef"))
+        # Hashing a memoryview for the first time after it is released
+        # results in an error (as with weakrefs).
+        m = self._view(b)
+        m.release()
+        self.assertRaises(ValueError, hash, m)
+
+    def test_hash_writable(self):
+        # Memoryviews of writable types are unhashable
+        tp = self.rw_type
+        if tp is None:
+            self.skipTest("no writable type to test")
+        b = tp(self._source)
+        m = self._view(b)
+        self.assertRaises(ValueError, hash, m)
+
 # Variations on source objects for the buffer: bytes-like objects, then arrays
 # with itemsize > 1.
 # NOTE: support for multi-dimensional objects is unimplemented.
diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -770,8 +770,8 @@
         check(int(PyLong_BASE), size(vh) + 2*self.longdigit)
         check(int(PyLong_BASE**2-1), size(vh) + 2*self.longdigit)
         check(int(PyLong_BASE**2), size(vh) + 3*self.longdigit)
-        # memory
-        check(memoryview(b''), size(h + 'PP2P2i7P'))
+        # memory (Py_buffer + hash value)
+        check(memoryview(b''), size(h + 'PP2P2i7P' + 'P'))
         # module
         check(unittest, size(h + '3P'))
         # None
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13411: memoryview objects are now hashable when the underlying
+  object is hashable.
+
 - Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
   to allow compiling extension modules with -Wswitch-enum on gcc.
   Initial patch by Floris Bruynooghe.
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -860,22 +860,11 @@
 static Py_hash_t
 bytes_hash(PyBytesObject *a)
 {
-    register Py_ssize_t len;
-    register unsigned char *p;
-    register Py_uhash_t x;
-
-    if (a->ob_shash != -1)
-        return a->ob_shash;
-    len = Py_SIZE(a);
-    p = (unsigned char *) a->ob_sval;
-    x = (Py_uhash_t)*p << 7;
-    while (--len >= 0)
-        x = (1000003U*x) ^ (Py_uhash_t)*p++;
-    x ^= (Py_uhash_t)Py_SIZE(a);
-    if (x == -1)
-        x = -2;
-    a->ob_shash = x;
-    return x;
+    if (a->ob_shash == -1) {
+        /* Can't fail */
+        a->ob_shash = _Py_HashBytes((unsigned char *) a->ob_sval, Py_SIZE(a));
+    }
+    return a->ob_shash;
 }
 
 static PyObject*
diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c
--- a/Objects/memoryobject.c
+++ b/Objects/memoryobject.c
@@ -84,6 +84,7 @@
         PyObject_GC_New(PyMemoryViewObject, &PyMemoryView_Type);
     if (mview == NULL)
         return NULL;
+    mview->hash = -1;
     dup_buffer(&mview->view, info);
     /* NOTE: mview->view.obj should already have been incref'ed as
        part of PyBuffer_FillInfo(). */
@@ -512,6 +513,37 @@
         return PyUnicode_FromFormat("<memory at %p>", self);
 }
 
+static Py_hash_t
+memory_hash(PyMemoryViewObject *self)
+{
+    if (self->hash == -1) {
+        Py_buffer *view = &self->view;
+        CHECK_RELEASED_INT(self);
+        if (view->ndim > 1) {
+            PyErr_SetString(PyExc_NotImplementedError,
+                            "can't hash multi-dimensional memoryview object");
+            return -1;
+        }
+        if (view->strides && view->strides[0] != view->itemsize) {
+            PyErr_SetString(PyExc_NotImplementedError,
+                            "can't hash strided memoryview object");
+            return -1;
+        }
+        if (!view->readonly) {
+            PyErr_SetString(PyExc_ValueError,
+                            "can't hash writable memoryview object");
+            return -1;
+        }
+        if (view->obj != NULL && PyObject_Hash(view->obj) == -1) {
+            /* Keep the original error message */
+            return -1;
+        }
+        /* Can't fail */
+        self->hash = _Py_HashBytes((unsigned char *) view->buf, view->len);
+    }
+    return self->hash;
+}
+
 /* Sequence methods */
 static Py_ssize_t
 memory_length(PyMemoryViewObject *self)
@@ -829,7 +861,7 @@
     0,                                        /* tp_as_number */
     &memory_as_sequence,                      /* tp_as_sequence */
     &memory_as_mapping,                       /* tp_as_mapping */
-    0,                                        /* tp_hash */
+    (hashfunc)memory_hash,                    /* tp_hash */
     0,                                        /* tp_call */
     0,                                        /* tp_str */
     PyObject_GenericGetAttr,                  /* tp_getattro */
diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -744,6 +744,21 @@
 }
 
 Py_hash_t
+_Py_HashBytes(unsigned char *p, Py_ssize_t len)
+{
+    Py_uhash_t x;
+    Py_ssize_t i;
+
+    x = (Py_uhash_t) *p << 7;
+    for (i = 0; i < len; i++)
+        x = (1000003U * x) ^ (Py_uhash_t) *p++;
+    x ^= (Py_uhash_t) len;
+    if (x == -1)
+        x = -2;
+    return x;
+}
+
+Py_hash_t
 PyObject_HashNotImplemented(PyObject *v)
 {
     PyErr_Format(PyExc_TypeError, "unhashable type: '%.200s'",

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 21:36:18 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 21:36:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312328=3A_Under_Win?=
 =?utf8?q?dows=2C_refactor_handling_of_Ctrl-C_events_and?=
Message-ID: <E1RSabC-0003Kv-MM@dinsdale.python.org>

http://hg.python.org/cpython/rev/08953a04b2e6
changeset:   73670:08953a04b2e6
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 21:26:56 2011 +0100
summary:
  Issue #12328: Under Windows, refactor handling of Ctrl-C events and
make _multiprocessing.win32.WaitForMultipleObjects interruptible when
the wait_flag parameter is false.  Patch by sbt.

files:
  Include/intrcheck.h                        |   6 +
  Misc/NEWS                                  |   4 +
  Modules/_multiprocessing/multiprocessing.c |  35 ------
  Modules/_multiprocessing/multiprocessing.h |   1 -
  Modules/_multiprocessing/semaphore.c       |  52 +++------
  Modules/_multiprocessing/win32_functions.c |  18 +++-
  Modules/signalmodule.c                     |  36 ++++++
  Modules/timemodule.c                       |  30 +-----
  8 files changed, 84 insertions(+), 98 deletions(-)


diff --git a/Include/intrcheck.h b/Include/intrcheck.h
--- a/Include/intrcheck.h
+++ b/Include/intrcheck.h
@@ -8,6 +8,12 @@
 PyAPI_FUNC(int) PyOS_InterruptOccurred(void);
 PyAPI_FUNC(void) PyOS_InitInterrupts(void);
 PyAPI_FUNC(void) PyOS_AfterFork(void);
+PyAPI_FUNC(int) _PyOS_IsMainThread(void);
+
+#ifdef MS_WINDOWS
+/* windows.h is not included by Python.h so use void* instead of HANDLE */
+PyAPI_FUNC(void*) _PyOS_SigintEvent(void);
+#endif
 
 #ifdef __cplusplus
 }
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -387,6 +387,10 @@
 Library
 -------
 
+- Issue #12328: Under Windows, refactor handling of Ctrl-C events and
+  make _multiprocessing.win32.WaitForMultipleObjects interruptible when
+  the wait_flag parameter is false.  Patch by sbt.
+
 - Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
   raised when the wrapped raw file is non-blocking and the write would block.
   Previous code assumed that the raw write() would raise BlockingIOError, but
diff --git a/Modules/_multiprocessing/multiprocessing.c b/Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c
+++ b/Modules/_multiprocessing/multiprocessing.c
@@ -53,30 +53,6 @@
 }
 
 
-/*
- * Windows only
- */
-
-#ifdef MS_WINDOWS
-
-/* On Windows we set an event to signal Ctrl-C; compare with timemodule.c */
-
-HANDLE sigint_event = NULL;
-
-static BOOL WINAPI
-ProcessingCtrlHandler(DWORD dwCtrlType)
-{
-    SetEvent(sigint_event);
-    return FALSE;
-}
-
-#endif /* MS_WINDOWS */
-
-
-/*
- * All platforms
- */
-
 static PyObject*
 multiprocessing_address_of_buffer(PyObject *self, PyObject *obj)
 {
@@ -165,17 +141,6 @@
     if (!temp)
         return NULL;
     PyModule_AddObject(module, "win32", temp);
-
-    /* Initialize the event handle used to signal Ctrl-C */
-    sigint_event = CreateEvent(NULL, TRUE, FALSE, NULL);
-    if (!sigint_event) {
-        PyErr_SetFromWindowsErr(0);
-        return NULL;
-    }
-    if (!SetConsoleCtrlHandler(ProcessingCtrlHandler, TRUE)) {
-        PyErr_SetFromWindowsErr(0);
-        return NULL;
-    }
 #endif
 
     /* Add configuration macros */
diff --git a/Modules/_multiprocessing/multiprocessing.h b/Modules/_multiprocessing/multiprocessing.h
--- a/Modules/_multiprocessing/multiprocessing.h
+++ b/Modules/_multiprocessing/multiprocessing.h
@@ -100,7 +100,6 @@
 extern PyObject *BufferTooShort;
 extern PyTypeObject SemLockType;
 extern PyTypeObject PipeConnectionType;
-extern HANDLE sigint_event;
 
 /*
  * Miscellaneous
diff --git a/Modules/_multiprocessing/semaphore.c b/Modules/_multiprocessing/semaphore.c
--- a/Modules/_multiprocessing/semaphore.c
+++ b/Modules/_multiprocessing/semaphore.c
@@ -62,7 +62,8 @@
     int blocking = 1;
     double timeout;
     PyObject *timeout_obj = Py_None;
-    DWORD res, full_msecs, msecs, start, ticks;
+    DWORD res, full_msecs, nhandles;
+    HANDLE handles[2], sigint_event;
 
     static char *kwlist[] = {"block", "timeout", NULL};
 
@@ -96,53 +97,40 @@
         Py_RETURN_TRUE;
     }
 
-    /* check whether we can acquire without blocking */
+    /* check whether we can acquire without releasing the GIL and blocking */
     if (WaitForSingleObject(self->handle, 0) == WAIT_OBJECT_0) {
         self->last_tid = GetCurrentThreadId();
         ++self->count;
         Py_RETURN_TRUE;
     }
 
-    msecs = full_msecs;
-    start = GetTickCount();
+    /* prepare list of handles */
+    nhandles = 0;
+    handles[nhandles++] = self->handle;
+    if (_PyOS_IsMainThread()) {
+        sigint_event = _PyOS_SigintEvent();
+        assert(sigint_event != NULL);
+        handles[nhandles++] = sigint_event;
+    }
 
-    for ( ; ; ) {
-        HANDLE handles[2] = {self->handle, sigint_event};
-
-        /* do the wait */
-        Py_BEGIN_ALLOW_THREADS
+    /* do the wait */
+    Py_BEGIN_ALLOW_THREADS
+    if (sigint_event != NULL)
         ResetEvent(sigint_event);
-        res = WaitForMultipleObjects(2, handles, FALSE, msecs);
-        Py_END_ALLOW_THREADS
-
-        /* handle result */
-        if (res != WAIT_OBJECT_0 + 1)
-            break;
-
-        /* got SIGINT so give signal handler a chance to run */
-        Sleep(1);
-
-        /* if this is main thread let KeyboardInterrupt be raised */
-        if (PyErr_CheckSignals())
-            return NULL;
-
-        /* recalculate timeout */
-        if (msecs != INFINITE) {
-            ticks = GetTickCount();
-            if ((DWORD)(ticks - start) >= full_msecs)
-                Py_RETURN_FALSE;
-            msecs = full_msecs - (ticks - start);
-        }
-    }
+    res = WaitForMultipleObjects(nhandles, handles, FALSE, full_msecs);
+    Py_END_ALLOW_THREADS
 
     /* handle result */
     switch (res) {
     case WAIT_TIMEOUT:
         Py_RETURN_FALSE;
-    case WAIT_OBJECT_0:
+    case WAIT_OBJECT_0 + 0:
         self->last_tid = GetCurrentThreadId();
         ++self->count;
         Py_RETURN_TRUE;
+    case WAIT_OBJECT_0 + 1:
+        errno = EINTR;
+        return PyErr_SetFromErrno(PyExc_IOError);
     case WAIT_FAILED:
         return PyErr_SetFromWindowsErr(0);
     default:
diff --git a/Modules/_multiprocessing/win32_functions.c b/Modules/_multiprocessing/win32_functions.c
--- a/Modules/_multiprocessing/win32_functions.c
+++ b/Modules/_multiprocessing/win32_functions.c
@@ -679,6 +679,7 @@
     DWORD result;
     PyObject *handle_seq;
     HANDLE handles[MAXIMUM_WAIT_OBJECTS];
+    HANDLE sigint_event = NULL;
     Py_ssize_t nhandles, i;
     int wait_flag;
     int milliseconds = INFINITE;
@@ -696,10 +697,10 @@
     nhandles = PySequence_Length(handle_seq);
     if (nhandles == -1)
         return NULL;
-    if (nhandles < 0 || nhandles >= MAXIMUM_WAIT_OBJECTS) {
+    if (nhandles < 0 || nhandles >= MAXIMUM_WAIT_OBJECTS - 1) {
         PyErr_Format(PyExc_ValueError,
                      "need at most %zd handles, got a sequence of length %zd",
-                     MAXIMUM_WAIT_OBJECTS, nhandles);
+                     MAXIMUM_WAIT_OBJECTS - 1, nhandles);
         return NULL;
     }
     for (i = 0; i < nhandles; i++) {
@@ -711,14 +712,27 @@
             return NULL;
         handles[i] = h;
     }
+    /* If this is the main thread then make the wait interruptible
+       by Ctrl-C unless we are waiting for *all* handles */
+    if (!wait_flag && _PyOS_IsMainThread()) {
+        sigint_event = _PyOS_SigintEvent();
+        assert(sigint_event != NULL);
+        handles[nhandles++] = sigint_event;
+    }
 
     Py_BEGIN_ALLOW_THREADS
+    if (sigint_event != NULL)
+        ResetEvent(sigint_event);
     result = WaitForMultipleObjects((DWORD) nhandles, handles,
                                     (BOOL) wait_flag, (DWORD) milliseconds);
     Py_END_ALLOW_THREADS
 
     if (result == WAIT_FAILED)
         return PyErr_SetExcFromWindowsErr(PyExc_IOError, 0);
+    else if (sigint_event != NULL && result == WAIT_OBJECT_0 + nhandles - 1) {
+        errno = EINTR;
+        return PyErr_SetFromErrno(PyExc_IOError);
+    }
 
     return PyLong_FromLong((int) result);
 }
diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c
--- a/Modules/signalmodule.c
+++ b/Modules/signalmodule.c
@@ -109,6 +109,10 @@
 
 static PyOS_sighandler_t old_siginthandler = SIG_DFL;
 
+#ifdef MS_WINDOWS
+static HANDLE sigint_event = NULL;
+#endif
+
 #ifdef HAVE_GETITIMER
 static PyObject *ItimerError;
 
@@ -229,6 +233,11 @@
     /* Issue #10311: asynchronously executing signal handlers should not
        mutate errno under the feet of unsuspecting C code. */
     errno = save_errno;
+
+#ifdef MS_WINDOWS
+    if (sig_num == SIGINT)
+        SetEvent(sigint_event);
+#endif
 }
 
 
@@ -1253,6 +1262,11 @@
     Py_DECREF(x);
 #endif
 
+#ifdef MS_WINDOWS
+    /* Create manual-reset event, initially unset */
+    sigint_event = CreateEvent(NULL, TRUE, FALSE, FALSE);
+#endif
+
     if (PyErr_Occurred()) {
         Py_DECREF(m);
         m = NULL;
@@ -1397,3 +1411,25 @@
     PyThread_ReInitTLS();
 #endif
 }
+
+int
+_PyOS_IsMainThread(void)
+{
+#ifdef WITH_THREAD
+    return PyThread_get_thread_ident() == main_thread;
+#else
+    return 1;
+#endif
+}
+
+#ifdef MS_WINDOWS
+void *_PyOS_SigintEvent(void)
+{
+    /* Returns a manual-reset event which gets tripped whenever
+       SIGINT is received.
+
+       Python.h does not include windows.h so we do cannot use HANDLE
+       as the return type of this function.  We use void* instead. */
+    return sigint_event;
+}
+#endif
diff --git a/Modules/timemodule.c b/Modules/timemodule.c
--- a/Modules/timemodule.c
+++ b/Modules/timemodule.c
@@ -21,19 +21,6 @@
 #include <windows.h>
 #include "pythread.h"
 
-/* helper to allow us to interrupt sleep() on Windows*/
-static HANDLE hInterruptEvent = NULL;
-static BOOL WINAPI PyCtrlHandler(DWORD dwCtrlType)
-{
-    SetEvent(hInterruptEvent);
-    /* allow other default handlers to be called.
-       Default Python handler will setup the
-       KeyboardInterrupt exception.
-    */
-    return FALSE;
-}
-static long main_thread;
-
 #if defined(__BORLANDC__)
 /* These overrides not needed for Win32 */
 #define timezone _timezone
@@ -955,15 +942,6 @@
     /* Set, or reset, module variables like time.timezone */
     PyInit_timezone(m);
 
-#ifdef MS_WINDOWS
-    /* Helper to allow interrupts for Windows.
-       If Ctrl+C event delivered while not sleeping
-       it will be ignored.
-    */
-    main_thread = PyThread_get_thread_ident();
-    hInterruptEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
-    SetConsoleCtrlHandler( PyCtrlHandler, TRUE);
-#endif /* MS_WINDOWS */
     if (!initialized) {
         PyStructSequence_InitType(&StructTimeType,
                                   &struct_time_type_desc);
@@ -1036,18 +1014,14 @@
          * by Guido, only the main thread can be interrupted.
          */
         ul_millis = (unsigned long)millisecs;
-        if (ul_millis == 0 ||
-            main_thread != PyThread_get_thread_ident())
+        if (ul_millis == 0 || !_PyOS_IsMainThread())
             Sleep(ul_millis);
         else {
             DWORD rc;
+            HANDLE hInterruptEvent = _PyOS_SigintEvent();
             ResetEvent(hInterruptEvent);
             rc = WaitForSingleObject(hInterruptEvent, ul_millis);
             if (rc == WAIT_OBJECT_0) {
-                /* Yield to make sure real Python signal
-                 * handler called.
-                 */
-                Sleep(1);
                 Py_BLOCK_THREADS
                 errno = EINTR;
                 PyErr_SetFromErrno(PyExc_IOError);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 22:11:12 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 22:11:12 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Mark_PEP_3155_accepted_-_thank?=
	=?utf8?q?_you_Guido!?=
Message-ID: <E1RSb8y-0006YX-5U@dinsdale.python.org>

http://hg.python.org/peps/rev/8a12e39f6c98
changeset:   4000:8a12e39f6c98
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 22:06:21 2011 +0100
summary:
  Mark PEP 3155 accepted - thank you Guido!

files:
  pep-3155.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-3155.txt b/pep-3155.txt
--- a/pep-3155.txt
+++ b/pep-3155.txt
@@ -3,7 +3,7 @@
 Version: $Revision$
 Last-Modified: $Date$
 Author: Antoine Pitrou <solipsis at pitrou.net>
-Status: Draft
+Status: Accepted
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 2011-10-29

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon Nov 21 22:19:38 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 21 Nov 2011 22:19:38 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Fill_resolution_field?=
Message-ID: <E1RSbH8-0007OZ-N0@dinsdale.python.org>

http://hg.python.org/peps/rev/d28bcd221582
changeset:   4001:d28bcd221582
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 21 22:14:48 2011 +0100
summary:
  Fill resolution field

files:
  pep-3155.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-3155.txt b/pep-3155.txt
--- a/pep-3155.txt
+++ b/pep-3155.txt
@@ -9,7 +9,7 @@
 Created: 2011-10-29
 Python-Version: 3.3
 Post-History:
-Resolution: TBD
+Resolution: http://mail.python.org/pipermail/python-dev/2011-November/114545.html
 
 
 Rationale

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon Nov 21 22:51:52 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 22:51:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Rewrite_PyUnicode=5FEncodeD?=
 =?utf8?q?ecimal=28=29_to_use_the_new_Unicode_API?=
Message-ID: <E1RSbmK-000268-Nm@dinsdale.python.org>

http://hg.python.org/cpython/rev/849e9277906a
changeset:   73671:849e9277906a
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 22:52:58 2011 +0100
summary:
  Rewrite PyUnicode_EncodeDecimal() to use the new Unicode API

Add tests for PyUnicode_EncodeDecimal() and
PyUnicode_TransformDecimalToASCII().

files:
  Lib/test/test_unicode.py  |  30 ++++++++
  Modules/_testcapimodule.c |  51 +++++++++++++-
  Objects/unicodeobject.c   |  97 ++++++++++++++------------
  3 files changed, 132 insertions(+), 46 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1806,6 +1806,36 @@
         s += "4"
         self.assertEqual(s, "3")
 
+    def test_encode_decimal(self):
+        from _testcapi import unicode_encodedecimal
+        self.assertEqual(unicode_encodedecimal('123'),
+                         b'123')
+        self.assertEqual(unicode_encodedecimal('\u0663.\u0661\u0664'),
+                         b'3.14')
+        self.assertEqual(unicode_encodedecimal("\N{EM SPACE}3.14\N{EN SPACE}"),
+                         b' 3.14 ')
+        self.assertRaises(UnicodeEncodeError,
+                          unicode_encodedecimal, "123\u20ac", "strict")
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "replace"),
+                         b'123?')
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "ignore"),
+                         b'123')
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "xmlcharrefreplace"),
+                         b'123&#8364;')
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "backslashreplace"),
+                         b'123\\u20ac')
+
+    def test_transform_decimal(self):
+        from _testcapi import unicode_transformdecimaltoascii as transform_decimal
+        self.assertEqual(transform_decimal('123'),
+                         '123')
+        self.assertEqual(transform_decimal('\u0663.\u0661\u0664'),
+                         '3.14')
+        self.assertEqual(transform_decimal("\N{EM SPACE}3.14\N{EN SPACE}"),
+                         "\N{EM SPACE}3.14\N{EN SPACE}")
+        self.assertEqual(transform_decimal('123\u20ac'),
+                         '123\u20ac')
+
 
 class StringModuleTest(unittest.TestCase):
     def test_formatter_parser(self):
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1500,6 +1500,51 @@
 }
 
 static PyObject *
+unicode_encodedecimal(PyObject *self, PyObject *args)
+{
+    Py_UNICODE *unicode;
+    Py_ssize_t length;
+    char *errors = NULL;
+    PyObject *decimal;
+    Py_ssize_t decimal_length, new_length;
+    int res;
+
+    if (!PyArg_ParseTuple(args, "u#|s", &unicode, &length, &errors))
+        return NULL;
+
+    decimal_length = length * 7; /* len('&#8364;') */
+    decimal = PyBytes_FromStringAndSize(NULL, decimal_length);
+    if (decimal == NULL)
+        return NULL;
+
+    res = PyUnicode_EncodeDecimal(unicode, length,
+                                  PyBytes_AS_STRING(decimal),
+                                  errors);
+    if (res < 0) {
+        Py_DECREF(decimal);
+        return NULL;
+    }
+
+    new_length = strlen(PyBytes_AS_STRING(decimal));
+    assert(new_length <= decimal_length);
+    res = _PyBytes_Resize(&decimal, new_length);
+    if (res < 0)
+        return NULL;
+
+    return decimal;
+}
+
+static PyObject *
+unicode_transformdecimaltoascii(PyObject *self, PyObject *args)
+{
+    Py_UNICODE *unicode;
+    Py_ssize_t length;
+    if (!PyArg_ParseTuple(args, "u#|s", &unicode, &length))
+        return NULL;
+    return PyUnicode_TransformDecimalToASCII(unicode, length);
+}
+
+static PyObject *
 getargs_w_star(PyObject *self, PyObject *args)
 {
     Py_buffer buffer;
@@ -2384,8 +2429,10 @@
     {"test_u_code",             (PyCFunction)test_u_code,        METH_NOARGS},
     {"test_Z_code",             (PyCFunction)test_Z_code,        METH_NOARGS},
     {"test_widechar",           (PyCFunction)test_widechar,      METH_NOARGS},
-    {"unicode_aswidechar",      unicode_aswidechar,                 METH_VARARGS},
-    {"unicode_aswidecharstring",unicode_aswidecharstring,           METH_VARARGS},
+    {"unicode_aswidechar",      unicode_aswidechar,              METH_VARARGS},
+    {"unicode_aswidecharstring",unicode_aswidecharstring,        METH_VARARGS},
+    {"unicode_encodedecimal",   unicode_encodedecimal,           METH_VARARGS},
+    {"unicode_transformdecimaltoascii", unicode_transformdecimaltoascii, METH_VARARGS},
 #ifdef WITH_THREAD
     {"_test_thread_state",      test_thread_state,               METH_VARARGS},
     {"_pending_threadfunc",     pending_threadfunc,              METH_VARARGS},
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8829,7 +8829,6 @@
                         char *output,
                         const char *errors)
 {
-    Py_UNICODE *p, *end;
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
     PyObject *unicode;
@@ -8838,47 +8837,50 @@
     /* the following variable is used for caching string comparisons
      * -1=not initialized, 0=unknown, 1=strict, 2=replace, 3=ignore, 4=xmlcharrefreplace */
     int known_errorHandler = -1;
+    Py_ssize_t i, j;
+    enum PyUnicode_Kind kind;
+    void *data;
 
     if (output == NULL) {
         PyErr_BadArgument();
         return -1;
     }
 
-    p = s;
-    end = s + length;
-    while (p < end) {
-        register Py_UNICODE ch = *p;
+    unicode = PyUnicode_FromUnicode(s, length);
+    if (unicode == NULL)
+        return -1;
+
+    if (PyUnicode_READY(unicode) < 0)
+        goto onError;
+    kind = PyUnicode_KIND(unicode);
+    data = PyUnicode_DATA(unicode);
+
+    for (i=0; i < length; i++) {
+        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
         int decimal;
-        PyObject *repunicode;
-        Py_ssize_t repsize;
-        Py_ssize_t newpos;
-        Py_UNICODE *uni2;
-        Py_UNICODE *collstart;
-        Py_UNICODE *collend;
+        Py_ssize_t startpos, endpos;
 
         if (Py_UNICODE_ISSPACE(ch)) {
             *output++ = ' ';
-            ++p;
             continue;
         }
         decimal = Py_UNICODE_TODECIMAL(ch);
         if (decimal >= 0) {
             *output++ = '0' + decimal;
-            ++p;
             continue;
         }
         if (0 < ch && ch < 256) {
             *output++ = (char)ch;
-            ++p;
             continue;
         }
         /* All other characters are considered unencodable */
-        collstart = p;
-        collend = p+1;
-        while (collend < end) {
-            if ((0 < *collend && *collend < 256) ||
-                !Py_UNICODE_ISSPACE(*collend) ||
-                Py_UNICODE_TODECIMAL(*collend))
+        startpos = i;
+        endpos = i+1;
+        for (; endpos < length; endpos++) {
+            ch = PyUnicode_READ(kind, data, endpos);
+            if ((0 < ch && ch < 256) ||
+                !Py_UNICODE_ISSPACE(ch) ||
+                Py_UNICODE_TODECIMAL(ch))
                 break;
         }
         /* cache callback name lookup
@@ -8897,33 +8899,33 @@
         }
         switch (known_errorHandler) {
         case 1: /* strict */
-            unicode = PyUnicode_FromUnicode(s, length);
-            if (unicode == NULL)
-                goto onError;
-            raise_encode_exception(&exc, encoding, unicode, collstart-s, collend-s, reason);
-            Py_DECREF(unicode);
+            raise_encode_exception(&exc, encoding, unicode, startpos, endpos, reason);
             goto onError;
         case 2: /* replace */
-            for (p = collstart; p < collend; ++p)
+            for (j=startpos; j < endpos; j++)
                 *output++ = '?';
             /* fall through */
         case 3: /* ignore */
-            p = collend;
+            i = endpos;
             break;
         case 4: /* xmlcharrefreplace */
-            /* generate replacement (temporarily (mis)uses p) */
-            for (p = collstart; p < collend; ++p)
-                output += sprintf(output, "&#%d;", (int)*p);
-            p = collend;
+            /* generate replacement */
+            for (j=startpos; j < endpos; j++) {
+                ch = PyUnicode_READ(kind, data, i);
+                output += sprintf(output, "&#%d;", (int)ch);
+                i++;
+            }
             break;
         default:
-            unicode = PyUnicode_FromUnicode(s, length);
-            if (unicode == NULL)
-                goto onError;
+        {
+            PyObject *repunicode;
+            Py_ssize_t repsize, newpos, k;
+            enum PyUnicode_Kind repkind;
+            void *repdata;
+
             repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
                                                           encoding, reason, unicode, &exc,
-                                                          collstart-s, collend-s, &newpos);
-            Py_DECREF(unicode);
+                                                          startpos, endpos, &newpos);
             if (repunicode == NULL)
                 goto onError;
             if (!PyUnicode_Check(repunicode)) {
@@ -8932,10 +8934,17 @@
                 Py_DECREF(repunicode);
                 goto onError;
             }
+            if (PyUnicode_READY(repunicode) < 0) {
+                Py_DECREF(repunicode);
+                goto onError;
+            }
+            repkind = PyUnicode_KIND(repunicode);
+            repdata = PyUnicode_DATA(repunicode);
+
             /* generate replacement  */
             repsize = PyUnicode_GET_SIZE(repunicode);
-            for (uni2 = PyUnicode_AS_UNICODE(repunicode); repsize-->0; ++uni2) {
-                Py_UNICODE ch = *uni2;
+            for (k=0; k<repsize; k++) {
+                ch = PyUnicode_READ(repkind, repdata, k);
                 if (Py_UNICODE_ISSPACE(ch))
                     *output++ = ' ';
                 else {
@@ -8946,29 +8955,29 @@
                         *output++ = (char)ch;
                     else {
                         Py_DECREF(repunicode);
-                        unicode = PyUnicode_FromUnicode(s, length);
-                        if (unicode == NULL)
-                            goto onError;
                         raise_encode_exception(&exc, encoding,
-                                               unicode, collstart-s, collend-s, reason);
-                        Py_DECREF(unicode);
+                                               unicode, startpos, endpos,
+                                               reason);
                         goto onError;
                     }
                 }
             }
-            p = s + newpos;
+            i = newpos;
             Py_DECREF(repunicode);
         }
+        }
     }
     /* 0-terminate the output string */
     *output++ = '\0';
     Py_XDECREF(exc);
     Py_XDECREF(errorHandler);
+    Py_DECREF(unicode);
     return 0;
 
   onError:
     Py_XDECREF(exc);
     Py_XDECREF(errorHandler);
+    Py_DECREF(unicode);
     return -1;
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 22:51:53 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 22:51:53 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_a_compiler_warning_in_?=
 =?utf8?q?=5FPyUnicode=5FCheckConsistency=28=29?=
Message-ID: <E1RSbmL-00026I-Hm@dinsdale.python.org>

http://hg.python.org/cpython/rev/e0df57330b83
changeset:   73672:e0df57330b83
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 22:54:05 2011 +0100
summary:
  Fix a compiler warning in _PyUnicode_CheckConsistency()

files:
  Objects/unicodeobject.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -401,7 +401,7 @@
                 else
                     printf("U+%04x", ch);
             }
-            printf("} (len=%u)\n", ascii->length);
+            printf("} (len=%lu)\n", ascii->length);
             abort();
         }
         if (kind == PyUnicode_1BYTE_KIND) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 23:02:34 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 23:02:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Simplify_PyUnicode=5FCopy?=
	=?utf8?b?KCk=?=
Message-ID: <E1RSbwg-00033N-W4@dinsdale.python.org>

http://hg.python.org/cpython/rev/3bb0dd83ff9f
changeset:   73673:3bb0dd83ff9f
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 23:03:47 2011 +0100
summary:
  Simplify PyUnicode_Copy()

USe PyUnicode_Copy() in fixup()

files:
  Objects/unicodeobject.c |  33 ++++++----------------------
  1 files changed, 7 insertions(+), 26 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1921,7 +1921,7 @@
 PyObject*
 PyUnicode_Copy(PyObject *unicode)
 {
-    Py_ssize_t size;
+    Py_ssize_t length;
     PyObject *copy;
     void *data;
 
@@ -1932,28 +1932,15 @@
     if (PyUnicode_READY(unicode))
         return NULL;
 
-    size = PyUnicode_GET_LENGTH(unicode);
-    copy = PyUnicode_New(size, PyUnicode_MAX_CHAR_VALUE(unicode));
+    length = PyUnicode_GET_LENGTH(unicode);
+    copy = PyUnicode_New(length, PyUnicode_MAX_CHAR_VALUE(unicode));
     if (!copy)
         return NULL;
     assert(PyUnicode_KIND(copy) == PyUnicode_KIND(unicode));
 
     data = PyUnicode_DATA(unicode);
-    switch (PyUnicode_KIND(unicode))
-    {
-    case PyUnicode_1BYTE_KIND:
-        memcpy(PyUnicode_1BYTE_DATA(copy), data, size);
-        break;
-    case PyUnicode_2BYTE_KIND:
-        memcpy(PyUnicode_2BYTE_DATA(copy), data, sizeof(Py_UCS2) * size);
-        break;
-    case PyUnicode_4BYTE_KIND:
-        memcpy(PyUnicode_4BYTE_DATA(copy), data, sizeof(Py_UCS4) * size);
-        break;
-    default:
-        assert(0);
-        break;
-    }
+    Py_MEMCPY(PyUnicode_DATA(copy), PyUnicode_DATA(unicode),
+              length * PyUnicode_KIND(unicode));
     assert(_PyUnicode_CheckConsistency(copy, 1));
     return copy;
 }
@@ -9344,16 +9331,10 @@
     PyObject *u;
     Py_UCS4 maxchar_old, maxchar_new = 0;
 
-    if (PyUnicode_READY(self) == -1)
-        return NULL;
-    maxchar_old = PyUnicode_MAX_CHAR_VALUE(self);
-    u = PyUnicode_New(PyUnicode_GET_LENGTH(self),
-                      maxchar_old);
+    u = PyUnicode_Copy(self);
     if (u == NULL)
         return NULL;
-
-    Py_MEMCPY(PyUnicode_1BYTE_DATA(u), PyUnicode_1BYTE_DATA(self),
-              PyUnicode_GET_LENGTH(u) * PyUnicode_KIND(u));
+    maxchar_old = PyUnicode_MAX_CHAR_VALUE(u);
 
     /* fix functions return the new maximum character in a string,
        if the kind of the resulting unicode object does not change,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 23:11:13 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 23:11:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_an_unused_variable_f?=
	=?utf8?q?rom_PyUnicode=5FCopy=28=29?=
Message-ID: <E1RSc53-0003xD-KL@dinsdale.python.org>

http://hg.python.org/cpython/rev/fb4756fac758
changeset:   73674:fb4756fac758
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 23:11:52 2011 +0100
summary:
  Remove an unused variable from PyUnicode_Copy()

files:
  Objects/unicodeobject.c |  2 --
  1 files changed, 0 insertions(+), 2 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1923,7 +1923,6 @@
 {
     Py_ssize_t length;
     PyObject *copy;
-    void *data;
 
     if (!PyUnicode_Check(unicode)) {
         PyErr_BadInternalCall();
@@ -1938,7 +1937,6 @@
         return NULL;
     assert(PyUnicode_KIND(copy) == PyUnicode_KIND(unicode));
 
-    data = PyUnicode_DATA(unicode);
     Py_MEMCPY(PyUnicode_DATA(copy), PyUnicode_DATA(unicode),
               length * PyUnicode_KIND(unicode));
     assert(_PyUnicode_CheckConsistency(copy, 1));

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 21 23:11:14 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 21 Nov 2011 23:11:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Rewrite_PyUnicode=5FTransfo?=
 =?utf8?q?rmDecimalToASCII=28=29_to_use_the_new_Unicode_API?=
Message-ID: <E1RSc54-0003xN-KJ@dinsdale.python.org>

http://hg.python.org/cpython/rev/5e916decdc74
changeset:   73675:5e916decdc74
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 21 23:12:56 2011 +0100
summary:
  Rewrite PyUnicode_TransformDecimalToASCII() to use the new Unicode API

files:
  Objects/unicodeobject.c |  51 +++++++++++++++++-----------
  1 files changed, 30 insertions(+), 21 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8779,32 +8779,41 @@
 PyUnicode_TransformDecimalToASCII(Py_UNICODE *s,
                                   Py_ssize_t length)
 {
-    PyObject *result;
-    Py_UNICODE *p; /* write pointer into result */
+    PyObject *decimal;
     Py_ssize_t i;
-    /* Copy to a new string */
-    result = (PyObject *)_PyUnicode_New(length);
-    Py_UNICODE_COPY(PyUnicode_AS_UNICODE(result), s, length);
-    if (result == NULL)
-        return result;
-    p = PyUnicode_AS_UNICODE(result);
-    /* Iterate over code points */
+    Py_UCS4 maxchar;
+    enum PyUnicode_Kind kind;
+    void *data;
+
+    maxchar = 0;
     for (i = 0; i < length; i++) {
-        Py_UNICODE ch =s[i];
+        Py_UNICODE ch = s[i];
         if (ch > 127) {
             int decimal = Py_UNICODE_TODECIMAL(ch);
             if (decimal >= 0)
-                p[i] = '0' + decimal;
-        }
-    }
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&result)) {
-        Py_DECREF(result);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(result, 1));
-    return result;
+                ch = '0' + decimal;
+        }
+        maxchar = Py_MAX(maxchar, ch);
+    }
+
+    /* Copy to a new string */
+    decimal = PyUnicode_New(length, maxchar);
+    if (decimal == NULL)
+        return decimal;
+    kind = PyUnicode_KIND(decimal);
+    data = PyUnicode_DATA(decimal);
+    /* Iterate over code points */
+    for (i = 0; i < length; i++) {
+        Py_UNICODE ch = s[i];
+        if (ch > 127) {
+            int decimal = Py_UNICODE_TODECIMAL(ch);
+            if (decimal >= 0)
+                ch = '0' + decimal;
+        }
+        PyUnicode_WRITE(kind, data, i, ch);
+    }
+    assert(_PyUnicode_CheckConsistency(decimal, 1));
+    return decimal;
 }
 /* --- Decimal Encoder ---------------------------------------------------- */
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:20:48 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:20:48 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_a_test_on_str=2E=5F=5Fg?=
	=?utf8?b?ZXRuZXdhcmdzX18oKQ==?=
Message-ID: <E1RSe6S-0005yM-Vp@dinsdale.python.org>

http://hg.python.org/cpython/rev/5d5133f518cb
changeset:   73676:5d5133f518cb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:06:15 2011 +0100
summary:
  Add a test on str.__getnewargs__()

It tests indirectly PyUnicode_Copy(): ensure that the string is a copy.

files:
  Lib/test/test_unicode.py |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1836,6 +1836,13 @@
         self.assertEqual(transform_decimal('123\u20ac'),
                          '123\u20ac')
 
+    def test_getnewargs(self):
+        text = 'abc'
+        args = text.__getnewargs__()
+        self.assertIsNot(args[0], text)
+        self.assertEqual(args[0], text)
+        self.assertEqual(len(args), 1)
+
 
 class StringModuleTest(unittest.TestCase):
     def test_formatter_parser(self):

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:20:49 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:20:49 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Replace_=5FPyUnicode=5FREAD?=
 =?utf8?q?Y=5FREPLACE=28=29_and_=5FPyUnicode=5FReadyReplace=28=29_with?=
Message-ID: <E1RSe6T-0005yW-Oz@dinsdale.python.org>

http://hg.python.org/cpython/rev/123f2dc08b3e
changeset:   73677:123f2dc08b3e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:22:34 2011 +0100
summary:
  Replace _PyUnicode_READY_REPLACE() and _PyUnicode_ReadyReplace() with unicode_ready()

 * unicode_ready() has a simpler API
 * try to reuse unicode_empty and latin1_char singleton everywhere
 * Fix a reference leak in _PyUnicode_TranslateCharmap()
 * PyUnicode_InternInPlace() doesn't try to get a singleton anymore, to avoid
   having to handle a failure

files:
  Objects/unicodeobject.c |  305 ++++++++++++++-------------
  1 files changed, 156 insertions(+), 149 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -46,10 +46,6 @@
 #include <windows.h>
 #endif
 
-#ifdef Py_DEBUG
-#  define DONT_MAKE_RESULT_READY
-#endif
-
 /* Endianness switches; defaults to little endian */
 
 #ifdef WORDS_BIGENDIAN
@@ -118,11 +114,6 @@
       0 :                                               \
       _PyUnicode_Ready(op)))
 
-#define _PyUnicode_READY_REPLACE(p_obj)                 \
-    (assert(_PyUnicode_CHECK(*p_obj)),                  \
-     (PyUnicode_IS_READY(*p_obj) ?                      \
-      0 : _PyUnicode_ReadyReplace((PyObject **)(p_obj))))
-
 #define _PyUnicode_SHARE_UTF8(op)                       \
     (assert(_PyUnicode_CHECK(op)),                      \
      assert(!PyUnicode_IS_COMPACT_ASCII(op)),           \
@@ -232,9 +223,6 @@
     PyObject *to, Py_ssize_t to_start,
     PyObject *from, Py_ssize_t from_start,
     Py_ssize_t how_many);
-#ifdef Py_DEBUG
-static int unicode_is_singleton(PyObject *unicode);
-#endif
 
 static PyObject *
 unicode_fromascii(const unsigned char *s, Py_ssize_t size);
@@ -425,6 +413,90 @@
 }
 #endif
 
+static PyObject*
+unicode_result_wchar(PyObject *unicode)
+{
+#ifndef Py_DEBUG
+    Py_ssize_t len;
+
+    assert(Py_REFCNT(unicode) == 1);
+
+    len = _PyUnicode_WSTR_LENGTH(unicode);
+    if (len == 0) {
+        Py_INCREF(unicode_empty);
+        Py_DECREF(unicode);
+        return unicode_empty;
+    }
+
+    if (len == 1) {
+        wchar_t ch = _PyUnicode_WSTR(unicode)[0];
+        if (ch < 256) {
+            PyObject *latin1_char = get_latin1_char((unsigned char)ch);
+            Py_DECREF(unicode);
+            return latin1_char;
+        }
+    }
+
+    if (_PyUnicode_Ready(unicode) < 0) {
+        Py_XDECREF(unicode);
+        return NULL;
+    }
+#else
+    /* don't make the result ready in debug mode to ensure that the caller
+       makes the string ready before using it */
+    assert(_PyUnicode_CheckConsistency(unicode, 1));
+#endif
+    return unicode;
+}
+
+static PyObject*
+unicode_result_ready(PyObject *unicode)
+{
+    Py_ssize_t length;
+
+    length = PyUnicode_GET_LENGTH(unicode);
+    if (length == 0) {
+        if (unicode != unicode_empty) {
+            Py_INCREF(unicode_empty);
+            Py_DECREF(unicode);
+        }
+        return unicode_empty;
+    }
+
+    if (length == 1) {
+        Py_UCS4 ch = PyUnicode_READ_CHAR(unicode, 0);
+        if (ch < 256) {
+            PyObject *latin1_char = unicode_latin1[ch];
+            if (latin1_char != NULL) {
+                if (unicode != latin1_char) {
+                    Py_INCREF(latin1_char);
+                    Py_DECREF(unicode);
+                }
+                return latin1_char;
+            }
+            else {
+                assert(_PyUnicode_CheckConsistency(unicode, 1));
+                Py_INCREF(unicode);
+                unicode_latin1[ch] = unicode;
+                return unicode;
+            }
+        }
+    }
+
+    assert(_PyUnicode_CheckConsistency(unicode, 1));
+    return unicode;
+}
+
+static PyObject*
+unicode_result(PyObject *unicode)
+{
+    assert(_PyUnicode_CHECK(unicode));
+    if (PyUnicode_IS_READY(unicode))
+        return unicode_result_ready(unicode);
+    else
+        return unicode_result_wchar(unicode);
+}
+
 #ifdef HAVE_MBCS
 static OSVERSIONINFOEX winver;
 #endif
@@ -1271,10 +1343,9 @@
 static int unicode_ready_calls = 0;
 #endif
 
-static int
-unicode_ready(PyObject **p_obj, int replace)
-{
-    PyObject *unicode;
+int
+_PyUnicode_Ready(PyObject *unicode)
+{
     wchar_t *end;
     Py_UCS4 maxchar = 0;
     Py_ssize_t num_surrogates;
@@ -1282,9 +1353,6 @@
     Py_ssize_t length_wo_surrogates;
 #endif
 
-    assert(p_obj != NULL);
-    unicode = *p_obj;
-
     /* _PyUnicode_Ready() is only intended for old-style API usage where
        strings were created using _PyObject_New() and where no canonical
        representation (the str field) has been set yet aka strings
@@ -1301,32 +1369,6 @@
     ++unicode_ready_calls;
 #endif
 
-#ifdef Py_DEBUG
-    assert(!replace || Py_REFCNT(unicode) == 1);
-#else
-    if (replace && Py_REFCNT(unicode) != 1)
-        replace = 0;
-#endif
-    if (replace) {
-        Py_ssize_t len = _PyUnicode_WSTR_LENGTH(unicode);
-        wchar_t *wstr = _PyUnicode_WSTR(unicode);
-        /* Optimization for empty strings */
-        if (len == 0) {
-            Py_INCREF(unicode_empty);
-            Py_DECREF(*p_obj);
-            *p_obj = unicode_empty;
-            return 0;
-        }
-        if (len == 1 && wstr[0] < 256) {
-            PyObject *latin1_char = get_latin1_char((unsigned char)wstr[0]);
-            if (latin1_char == NULL)
-                return -1;
-            Py_DECREF(*p_obj);
-            *p_obj = latin1_char;
-            return 0;
-        }
-    }
-
     end = _PyUnicode_WSTR(unicode) + _PyUnicode_WSTR_LENGTH(unicode);
     if (find_maxchar_surrogates(_PyUnicode_WSTR(unicode), end,
                                 &maxchar, &num_surrogates) == -1)
@@ -1430,18 +1472,6 @@
     return 0;
 }
 
-int
-_PyUnicode_ReadyReplace(PyObject **op)
-{
-    return unicode_ready(op, 1);
-}
-
-int
-_PyUnicode_Ready(PyObject *op)
-{
-    return unicode_ready(&op, 0);
-}
-
 static void
 unicode_dealloc(register PyObject *unicode)
 {
@@ -1681,8 +1711,7 @@
         assert(0 && "Impossible state");
     }
 
-    assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return unicode;
+    return unicode_result(unicode);
 }
 
 PyObject *
@@ -1755,6 +1784,8 @@
     }
 }
 
+/* Internal function, don't check maximum character */
+
 static PyObject*
 unicode_fromascii(const unsigned char* s, Py_ssize_t size)
 {
@@ -1795,11 +1826,16 @@
 _PyUnicode_FromUCS1(const unsigned char* u, Py_ssize_t size)
 {
     PyObject *res;
-    unsigned char max_char = 127;
-
-    assert(size >= 0);
+    unsigned char max_char;
+
+    if (size == 0) {
+        Py_INCREF(unicode_empty);
+        return unicode_empty;
+    }
+    assert(size > 0);
     if (size == 1)
         return get_latin1_char(u[0]);
+
     max_char = ucs1lib_find_max_char(u, u + size);
     res = PyUnicode_New(size, max_char);
     if (!res)
@@ -1813,11 +1849,16 @@
 _PyUnicode_FromUCS2(const Py_UCS2 *u, Py_ssize_t size)
 {
     PyObject *res;
-    Py_UCS2 max_char = 0;
-
-    assert(size >= 0);
+    Py_UCS2 max_char;
+
+    if (size == 0) {
+        Py_INCREF(unicode_empty);
+        return unicode_empty;
+    }
+    assert(size > 0);
     if (size == 1 && u[0] < 256)
         return get_latin1_char((unsigned char)u[0]);
+
     max_char = ucs2lib_find_max_char(u, u + size);
     res = PyUnicode_New(size, max_char);
     if (!res)
@@ -1836,11 +1877,16 @@
 _PyUnicode_FromUCS4(const Py_UCS4 *u, Py_ssize_t size)
 {
     PyObject *res;
-    Py_UCS4 max_char = 0;
-
-    assert(size >= 0);
+    Py_UCS4 max_char;
+
+    if (size == 0) {
+        Py_INCREF(unicode_empty);
+        return unicode_empty;
+    }
+    assert(size > 0);
     if (size == 1 && u[0] < 256)
-        return get_latin1_char(u[0]);
+        return get_latin1_char((unsigned char)u[0]);
+
     max_char = ucs4lib_find_max_char(u, u + size);
     res = PyUnicode_New(size, max_char);
     if (!res)
@@ -2640,8 +2686,7 @@
         PyObject_Free(callresults);
     if (numberresults)
         PyObject_Free(numberresults);
-    assert(_PyUnicode_CheckConsistency(string, 1));
-    return string;
+    return unicode_result(string);
   fail:
     if (callresults) {
         PyObject **callresult2 = callresults;
@@ -2936,14 +2981,7 @@
         goto onError;
     }
     Py_DECREF(buffer);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&unicode)) {
-        Py_DECREF(unicode);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return unicode;
+    return unicode_result(unicode);
 
   onError:
     Py_XDECREF(buffer);
@@ -2969,8 +3007,7 @@
     v = PyCodec_Decode(unicode, encoding, errors);
     if (v == NULL)
         goto onError;
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 
   onError:
     return NULL;
@@ -3002,8 +3039,7 @@
         Py_DECREF(v);
         goto onError;
     }
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 
   onError:
     return NULL;
@@ -4002,14 +4038,7 @@
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&unicode)) {
-        Py_DECREF(unicode);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return unicode;
+    return unicode_result(unicode);
 
   onError:
     Py_XDECREF(errorHandler);
@@ -4358,18 +4387,26 @@
         return (PyObject *)PyUnicode_New(0, 0);
     }
     maxchar = utf8_max_char_size_and_char_count(s, size, &unicode_size);
+    /* When the string is ASCII only, just use memcpy and return.
+       unicode_size may be != size if there is an incomplete UTF-8
+       sequence at the end of the ASCII block.  */
+    if (maxchar < 128 && size == unicode_size) {
+        if (size == 1)
+            return get_latin1_char((unsigned char)s[0]);
+
+        unicode = PyUnicode_New(unicode_size, maxchar);
+        if (!unicode)
+            return NULL;
+        Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
+        assert(_PyUnicode_CheckConsistency(unicode, 1));
+        return unicode;
+    }
+
     /* In case of errors, maxchar and size computation might be incorrect;
        code below refits and resizes as necessary. */
     unicode = PyUnicode_New(unicode_size, maxchar);
     if (!unicode)
         return NULL;
-    /* When the string is ASCII only, just use memcpy and return.
-       unicode_size may be != size if there is an incomplete UTF-8
-       sequence at the end of the ASCII block.  */
-    if (maxchar < 128 && size == unicode_size) {
-        Py_MEMCPY(PyUnicode_1BYTE_DATA(unicode), s, unicode_size);
-        return unicode;
-    }
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
 
@@ -5052,14 +5089,7 @@
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&unicode)) {
-        Py_DECREF(unicode);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return unicode;
+    return unicode_result(unicode);
 
   onError:
     Py_DECREF(unicode);
@@ -5423,8 +5453,7 @@
 
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-    assert(_PyUnicode_CheckConsistency(unicode, 1));
-    return unicode;
+    return unicode_result(unicode);
 
   onError:
     Py_DECREF(unicode);
@@ -5843,14 +5872,7 @@
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&v)) {
-        Py_DECREF(v);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 
   ucnhashError:
     PyErr_SetString(
@@ -6111,8 +6133,7 @@
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 
   onError:
     Py_XDECREF(v);
@@ -6301,8 +6322,7 @@
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 
   onError:
     Py_XDECREF(v);
@@ -6686,6 +6706,11 @@
     PyObject *errorHandler = NULL;
     PyObject *exc = NULL;
 
+    if (size == 0) {
+        Py_INCREF(unicode_empty);
+        return unicode_empty;
+    }
+
     /* ASCII is equivalent to the first 128 ordinals in Unicode. */
     if (size == 1 && (unsigned char)s[0] < 128)
         return get_latin1_char((unsigned char)s[0]);
@@ -7115,14 +7140,7 @@
         size -= converted;
     } while (!done);
 
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&v)) {
-        Py_DECREF(v);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 }
 
 PyObject *
@@ -7723,8 +7741,7 @@
         goto onError;
     Py_XDECREF(errorHandler);
     Py_XDECREF(exc);
-    assert(_PyUnicode_CheckConsistency(v, 1));
-    return v;
+    return unicode_result(v);
 
   onError:
     Py_XDECREF(errorHandler);
@@ -8658,8 +8675,12 @@
                 repunicode = unicode_translate_call_errorhandler(errors, &errorHandler,
                                                                  reason, input, &exc,
                                                                  collstart, collend, &newpos);
-                if (repunicode == NULL || _PyUnicode_READY_REPLACE(&repunicode))
+                if (repunicode == NULL)
                     goto onError;
+                if (PyUnicode_READY(repunicode) < 0) {
+                    Py_DECREF(repunicode);
+                    goto onError;
+                }
                 /* generate replacement  */
                 repsize = PyUnicode_GET_LENGTH(repunicode);
                 if (charmaptranslate_makespace(&output, &osize,
@@ -8812,8 +8833,7 @@
         }
         PyUnicode_WRITE(kind, data, i, ch);
     }
-    assert(_PyUnicode_CheckConsistency(decimal, 1));
-    return decimal;
+    return unicode_result(decimal);
 }
 /* --- Decimal Encoder ---------------------------------------------------- */
 
@@ -10801,8 +10821,7 @@
         if (!(PyUnicode_IS_ASCII(left) && !PyUnicode_IS_ASCII(right)))
         {
             unicode_append_inplace(p_left, right);
-            if (p_left != NULL)
-                assert(_PyUnicode_CheckConsistency(*p_left, 1));
+            assert(p_left == NULL || _PyUnicode_CheckConsistency(*p_left, 1));
             return;
         }
     }
@@ -11012,14 +11031,7 @@
         }
     }
     assert (j == PyUnicode_GET_LENGTH(u));
-#ifndef DONT_MAKE_RESULT_READY
-    if (_PyUnicode_READY_REPLACE(&u)) {
-        Py_DECREF(u);
-        return NULL;
-    }
-#endif
-    assert(_PyUnicode_CheckConsistency(u, 1));
-    return u;
+    return unicode_result(u);
 
   overflow:
     PyErr_SetString(PyExc_OverflowError, "new string is too long");
@@ -13876,9 +13888,9 @@
 
     /* Init the implementation */
     unicode_empty = PyUnicode_New(0, 0);
-    assert(_PyUnicode_CheckConsistency(unicode_empty, 1));
     if (!unicode_empty)
         Py_FatalError("Can't create empty string");
+    assert(_PyUnicode_CheckConsistency(unicode_empty, 1));
 
     for (i = 0; i < 256; i++)
         unicode_latin1[i] = NULL;
@@ -13946,11 +13958,6 @@
         return;
     if (PyUnicode_CHECK_INTERNED(s))
         return;
-    if (_PyUnicode_READY_REPLACE(p)) {
-        assert(0 && "_PyUnicode_READY_REPLACE fail in PyUnicode_InternInPlace");
-        return;
-    }
-    s = *p;
     if (interned == NULL) {
         interned = PyDict_New();
         if (interned == NULL) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:20:50 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:20:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_UTF-8_decoder=3A_set_consum?=
 =?utf8?q?ed_value_in_the_latin1_fast-path?=
Message-ID: <E1RSe6U-0005ya-Hz@dinsdale.python.org>

http://hg.python.org/cpython/rev/754d79aee7ab
changeset:   73678:754d79aee7ab
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:23:02 2011 +0100
summary:
  UTF-8 decoder: set consumed value in the latin1 fast-path

files:
  Objects/unicodeobject.c |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4391,6 +4391,9 @@
        unicode_size may be != size if there is an incomplete UTF-8
        sequence at the end of the ASCII block.  */
     if (maxchar < 128 && size == unicode_size) {
+        if (consumed)
+            *consumed = size;
+
         if (size == 1)
             return get_latin1_char((unsigned char)s[0]);
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:48:01 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:48:01 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PyUnicode=5FFromKindAndData?=
 =?utf8?q?=28=29_fails_with_a_ValueError_if_size_=3C_0?=
Message-ID: <E1RSeWn-0008DH-Ic@dinsdale.python.org>

http://hg.python.org/cpython/rev/0679344ffff2
changeset:   73679:0679344ffff2
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:29:14 2011 +0100
summary:
  PyUnicode_FromKindAndData() fails with a ValueError if size < 0

files:
  Objects/unicodeobject.c |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1906,6 +1906,10 @@
 PyObject*
 PyUnicode_FromKindAndData(int kind, const void *buffer, Py_ssize_t size)
 {
+    if (size < 0) {
+        PyErr_SetString(PyExc_ValueError, "size must be positive");
+        return NULL;
+    }
     switch(kind) {
     case PyUnicode_1BYTE_KIND:
         return _PyUnicode_FromUCS1(buffer, size);
@@ -1914,7 +1918,6 @@
     case PyUnicode_4BYTE_KIND:
         return _PyUnicode_FromUCS4(buffer, size);
     default:
-        assert(0 && "invalid kind");
         PyErr_SetString(PyExc_SystemError, "invalid kind");
         return NULL;
     }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:48:02 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:48:02 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMDkz?=
 =?utf8?q?=3A_Fix_error_handling_on_PyUnicode=5FEncodeDecimal=28=29?=
Message-ID: <E1RSeWo-0008DL-D0@dinsdale.python.org>

http://hg.python.org/cpython/rev/f29d7d597fae
changeset:   73680:f29d7d597fae
branch:      3.2
parent:      73665:ac2c4c62b486
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:45:37 2011 +0100
summary:
  Issue #13093: Fix error handling on PyUnicode_EncodeDecimal()

 * Add tests for PyUnicode_EncodeDecimal() and PyUnicode_TransformDecimalToASCII()
 * Remove the unused "e" variable in replace()

files:
  Lib/test/test_unicode.py  |  36 +++++++++++++++++++
  Modules/_testcapimodule.c |  51 +++++++++++++++++++++++++-
  Objects/unicodeobject.c   |  10 ++---
  3 files changed, 89 insertions(+), 8 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1767,6 +1767,42 @@
             ]])
         self.assertRaises(TypeError, _string.formatter_field_name_split, 1)
 
+    def test_encode_decimal(self):
+        from _testcapi import unicode_encodedecimal
+        self.assertEqual(unicode_encodedecimal('123'),
+                         b'123')
+        self.assertEqual(unicode_encodedecimal('\u0663.\u0661\u0664'),
+                         b'3.14')
+        self.assertEqual(unicode_encodedecimal("\N{EM SPACE}3.14\N{EN SPACE}"),
+                         b' 3.14 ')
+        self.assertRaises(UnicodeEncodeError,
+                          unicode_encodedecimal, "123\u20ac", "strict")
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "replace"),
+                         b'123?')
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "ignore"),
+                         b'123')
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "xmlcharrefreplace"),
+                         b'123&#8364;')
+        self.assertEqual(unicode_encodedecimal("123\u20ac", "backslashreplace"),
+                         b'123\\u20ac')
+        self.assertEqual(unicode_encodedecimal("123\u20ac\N{EM SPACE}", "replace"),
+                         b'123? ')
+        self.assertEqual(unicode_encodedecimal("123\u20ac\u20ac", "replace"),
+                         b'123??')
+        self.assertEqual(unicode_encodedecimal("123\u20ac\u0660", "replace"),
+                         b'123?0')
+
+    def test_transform_decimal(self):
+        from _testcapi import unicode_transformdecimaltoascii as transform_decimal
+        self.assertEqual(transform_decimal('123'),
+                         '123')
+        self.assertEqual(transform_decimal('\u0663.\u0661\u0664'),
+                         '3.14')
+        self.assertEqual(transform_decimal("\N{EM SPACE}3.14\N{EN SPACE}"),
+                         "\N{EM SPACE}3.14\N{EN SPACE}")
+        self.assertEqual(transform_decimal('123\u20ac'),
+                         '123\u20ac')
+
 
 def test_main():
     support.run_unittest(__name__)
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1438,6 +1438,51 @@
 }
 
 static PyObject *
+unicode_encodedecimal(PyObject *self, PyObject *args)
+{
+    Py_UNICODE *unicode;
+    Py_ssize_t length;
+    char *errors = NULL;
+    PyObject *decimal;
+    Py_ssize_t decimal_length, new_length;
+    int res;
+
+    if (!PyArg_ParseTuple(args, "u#|s", &unicode, &length, &errors))
+        return NULL;
+
+    decimal_length = length * 7; /* len('&#8364;') */
+    decimal = PyBytes_FromStringAndSize(NULL, decimal_length);
+    if (decimal == NULL)
+        return NULL;
+
+    res = PyUnicode_EncodeDecimal(unicode, length,
+                                  PyBytes_AS_STRING(decimal),
+                                  errors);
+    if (res < 0) {
+        Py_DECREF(decimal);
+        return NULL;
+    }
+
+    new_length = strlen(PyBytes_AS_STRING(decimal));
+    assert(new_length <= decimal_length);
+    res = _PyBytes_Resize(&decimal, new_length);
+    if (res < 0)
+        return NULL;
+
+    return decimal;
+}
+
+static PyObject *
+unicode_transformdecimaltoascii(PyObject *self, PyObject *args)
+{
+    Py_UNICODE *unicode;
+    Py_ssize_t length;
+    if (!PyArg_ParseTuple(args, "u#|s", &unicode, &length))
+        return NULL;
+    return PyUnicode_TransformDecimalToASCII(unicode, length);
+}
+
+static PyObject *
 getargs_w_star(PyObject *self, PyObject *args)
 {
     Py_buffer buffer;
@@ -2320,8 +2365,10 @@
     {"test_u_code",             (PyCFunction)test_u_code,        METH_NOARGS},
     {"test_Z_code",             (PyCFunction)test_Z_code,        METH_NOARGS},
     {"test_widechar",           (PyCFunction)test_widechar,      METH_NOARGS},
-    {"unicode_aswidechar",      unicode_aswidechar,                 METH_VARARGS},
-    {"unicode_aswidecharstring",unicode_aswidecharstring,           METH_VARARGS},
+    {"unicode_aswidechar",      unicode_aswidechar,              METH_VARARGS},
+    {"unicode_aswidecharstring",unicode_aswidecharstring,        METH_VARARGS},
+    {"unicode_encodedecimal",   unicode_encodedecimal,           METH_VARARGS},
+    {"unicode_transformdecimaltoascii", unicode_transformdecimaltoascii, METH_VARARGS},
 #ifdef WITH_THREAD
     {"_test_thread_state",      test_thread_state,               METH_VARARGS},
     {"_pending_threadfunc",     pending_threadfunc,              METH_VARARGS},
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6323,11 +6323,10 @@
         }
         /* All other characters are considered unencodable */
         collstart = p;
-        collend = p+1;
-        while (collend < end) {
+        for (collend = p+1; collend < end; collend++) {
             if ((0 < *collend && *collend < 256) ||
-                !Py_UNICODE_ISSPACE(*collend) ||
-                Py_UNICODE_TODECIMAL(*collend))
+                Py_UNICODE_ISSPACE(*collend) ||
+                0 <= Py_UNICODE_TODECIMAL(*collend))
                 break;
         }
         /* cache callback name lookup
@@ -7004,7 +7003,7 @@
         }
     } else {
 
-        Py_ssize_t n, i, j, e;
+        Py_ssize_t n, i, j;
         Py_ssize_t product, new_size, delta;
         Py_UNICODE *p;
 
@@ -7036,7 +7035,6 @@
             return NULL;
         i = 0;
         p = u->str;
-        e = self->length - str1->length;
         if (str1->length > 0) {
             while (n-- > 0) {
                 /* look for next match */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:48:03 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:48:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_=28Merge_3=2E2=29_Issue_=23?=
 =?utf8?q?13093=3A_Fix_error_handling_on_PyUnicode=5FEncodeDecimal=28=29?=
Message-ID: <E1RSeWp-0008DP-9j@dinsdale.python.org>

http://hg.python.org/cpython/rev/bc53c11804ab
changeset:   73681:bc53c11804ab
parent:      73679:0679344ffff2
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:50:07 2011 +0100
summary:
  (Merge 3.2) Issue #13093: Fix error handling on PyUnicode_EncodeDecimal()

files:
  Lib/test/test_unicode.py |   6 ++++++
  Objects/unicodeobject.c  |  12 ++++++++----
  2 files changed, 14 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1824,6 +1824,12 @@
                          b'123&#8364;')
         self.assertEqual(unicode_encodedecimal("123\u20ac", "backslashreplace"),
                          b'123\\u20ac')
+        self.assertEqual(unicode_encodedecimal("123\u20ac\N{EM SPACE}", "replace"),
+                         b'123? ')
+        self.assertEqual(unicode_encodedecimal("123\u20ac\u20ac", "replace"),
+                         b'123??')
+        self.assertEqual(unicode_encodedecimal("123\u20ac\u0660", "replace"),
+                         b'123?0')
 
     def test_transform_decimal(self):
         from _testcapi import unicode_transformdecimaltoascii as transform_decimal
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8875,22 +8875,25 @@
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
 
-    for (i=0; i < length; i++) {
+    for (i=0; i < length; ) {
         Py_UCS4 ch = PyUnicode_READ(kind, data, i);
         int decimal;
         Py_ssize_t startpos, endpos;
 
         if (Py_UNICODE_ISSPACE(ch)) {
             *output++ = ' ';
+            i++;
             continue;
         }
         decimal = Py_UNICODE_TODECIMAL(ch);
         if (decimal >= 0) {
             *output++ = '0' + decimal;
+            i++;
             continue;
         }
         if (0 < ch && ch < 256) {
             *output++ = (char)ch;
+            i++;
             continue;
         }
         /* All other characters are considered unencodable */
@@ -8899,8 +8902,8 @@
         for (; endpos < length; endpos++) {
             ch = PyUnicode_READ(kind, data, endpos);
             if ((0 < ch && ch < 256) ||
-                !Py_UNICODE_ISSPACE(ch) ||
-                Py_UNICODE_TODECIMAL(ch))
+                Py_UNICODE_ISSPACE(ch) ||
+                0 <= Py_UNICODE_TODECIMAL(ch))
                 break;
         }
         /* cache callback name lookup
@@ -8924,7 +8927,8 @@
         case 2: /* replace */
             for (j=startpos; j < endpos; j++)
                 *output++ = '?';
-            /* fall through */
+            i = endpos;
+            break;
         case 3: /* ignore */
             i = endpos;
             break;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 01:52:10 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 01:52:10 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMDkz?=
 =?utf8?q?=3A_Fix_error_handling_on_PyUnicode=5FEncodeDecimal=28=29?=
Message-ID: <E1RSeao-00008B-VV@dinsdale.python.org>

http://hg.python.org/cpython/rev/0cd197f13400
changeset:   73682:0cd197f13400
branch:      2.7
parent:      73667:e84e17643eeb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 01:54:19 2011 +0100
summary:
  Issue #13093: Fix error handling on PyUnicode_EncodeDecimal()

Add tests for PyUnicode_EncodeDecimal()

files:
  Lib/test/test_unicode.py  |  25 ++++++++++++++++++
  Modules/_testcapimodule.c |  36 +++++++++++++++++++++++++++
  Objects/unicodeobject.c   |   7 ++---
  3 files changed, 64 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1606,6 +1606,31 @@
         self.assertEqual("%s" % u, u'__unicode__ overridden')
         self.assertEqual("{}".format(u), '__unicode__ overridden')
 
+    def test_encode_decimal(self):
+        from _testcapi import unicode_encodedecimal
+        self.assertEqual(unicode_encodedecimal(u'123'),
+                         b'123')
+        self.assertEqual(unicode_encodedecimal(u'\u0663.\u0661\u0664'),
+                         b'3.14')
+        self.assertEqual(unicode_encodedecimal(u"\N{EM SPACE}3.14\N{EN SPACE}"),
+                         b' 3.14 ')
+        self.assertRaises(UnicodeEncodeError,
+                          unicode_encodedecimal, u"123\u20ac", "strict")
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac", "replace"),
+                         b'123?')
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac", "ignore"),
+                         b'123')
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac", "xmlcharrefreplace"),
+                         b'123&#8364;')
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac", "backslashreplace"),
+                         b'123\\u20ac')
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac\N{EM SPACE}", "replace"),
+                         b'123? ')
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac\u20ac", "replace"),
+                         b'123??')
+        self.assertEqual(unicode_encodedecimal(u"123\u20ac\u0660", "replace"),
+                         b'123?0')
+
 
 def test_main():
     test_support.run_unittest(__name__)
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1106,6 +1106,41 @@
 }
 
 static PyObject *
+unicode_encodedecimal(PyObject *self, PyObject *args)
+{
+    Py_UNICODE *unicode;
+    Py_ssize_t length;
+    char *errors = NULL;
+    PyObject *decimal;
+    Py_ssize_t decimal_length, new_length;
+    int res;
+
+    if (!PyArg_ParseTuple(args, "u#|s", &unicode, &length, &errors))
+        return NULL;
+
+    decimal_length = length * 7; /* len('&#8364;') */
+    decimal = PyBytes_FromStringAndSize(NULL, decimal_length);
+    if (decimal == NULL)
+        return NULL;
+
+    res = PyUnicode_EncodeDecimal(unicode, length,
+                                  PyBytes_AS_STRING(decimal),
+                                  errors);
+    if (res < 0) {
+        Py_DECREF(decimal);
+        return NULL;
+    }
+
+    new_length = strlen(PyBytes_AS_STRING(decimal));
+    assert(new_length <= decimal_length);
+    res = _PyBytes_Resize(&decimal, new_length);
+    if (res < 0)
+        return NULL;
+
+    return decimal;
+}
+
+static PyObject *
 test_empty_argparse(PyObject *self)
 {
     /* Test that formats can begin with '|'. See issue #4720. */
@@ -1698,6 +1733,7 @@
 #ifdef Py_USING_UNICODE
     {"test_u_code",             (PyCFunction)test_u_code,        METH_NOARGS},
     {"test_widechar",           (PyCFunction)test_widechar,      METH_NOARGS},
+    {"unicode_encodedecimal",   unicode_encodedecimal,           METH_VARARGS},
 #endif
 #ifdef WITH_THREAD
     {"_test_thread_state",  test_thread_state,                   METH_VARARGS},
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -5160,11 +5160,10 @@
         }
         /* All other characters are considered unencodable */
         collstart = p;
-        collend = p+1;
-        while (collend < end) {
+        for (collend = p+1; collend < end; collend++) {
             if ((0 < *collend && *collend < 256) ||
-                !Py_UNICODE_ISSPACE(*collend) ||
-                Py_UNICODE_TODECIMAL(*collend))
+                Py_UNICODE_ISSPACE(*collend) ||
+                0 <= Py_UNICODE_TODECIMAL(*collend))
                 break;
         }
         /* cache callback name lookup

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 02:12:19 2011
From: python-checkins at python.org (giampaolo.rodola)
Date: Tue, 22 Nov 2011 02:12:19 +0100
Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_myself_to_sched_and_sm?=
	=?utf8?q?tpd_modules?=
Message-ID: <E1RSeuJ-0001UY-77@dinsdale.python.org>

http://hg.python.org/devguide/rev/6996e096ec09
changeset:   463:6996e096ec09
user:        Giampaolo Rodola' <g.rodola at gmail.com>
date:        Tue Nov 22 02:12:13 2011 +0100
summary:
  Add myself to sched and smtpd modules

files:
  experts.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -185,14 +185,14 @@
 resource
 rlcompleter
 runpy                 ncoghlan
-sched
+sched                 giampaolo.rodola
 select
 shelve
 shlex
 shutil                tarek
 signal
 site
-smtpd
+smtpd                 giampaolo.rodola
 smtplib
 sndhdr
 socket

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Tue Nov 22 02:25:16 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 02:25:16 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Use_the_new_Unicode_API?=
Message-ID: <E1RSf6q-0002YT-2K@dinsdale.python.org>

http://hg.python.org/cpython/rev/d48b4a1c511a
changeset:   73683:d48b4a1c511a
parent:      73681:bc53c11804ab
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 02:27:30 2011 +0100
summary:
  Use the new Unicode API

 * Replace PyUnicode_FromUnicode(NULL, 0) by PyUnicode_New(0, 0)
 * Replce PyUnicode_FromUnicode(str, len) by PyUnicode_FromWideChar(str, len)
 * Replace Py_UNICODE by wchar_t
 * posix_putenv() uses PyUnicode_FromFormat() to create the string, instead
   of PyUnicode_FromUnicode() + _snwprintf()

files:
  Modules/_ctypes/_ctypes.c |   4 ++--
  Modules/posixmodule.c     |  26 +++++++++++++-------------
  Modules/socketmodule.c    |   2 +-
  Objects/abstract.c        |   2 +-
  PC/_msi.c                 |  10 +++++-----
  PC/_subprocess.c          |  17 +++++++----------
  PC/msvcrtmodule.c         |   4 ++--
  PC/winreg.c               |  10 +++++-----
  PC/winsound.c             |   2 +-
  Python/dynload_win.c      |   2 +-
  10 files changed, 38 insertions(+), 41 deletions(-)


diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -4255,7 +4255,7 @@
             wchar_t *dest;
 
             if (slicelen <= 0)
-                return PyUnicode_FromUnicode(NULL, 0);
+                return PyUnicode_New(0, 0);
             if (step == 1) {
                 return PyUnicode_FromWideChar(ptr + start,
                                               slicelen);
@@ -4930,7 +4930,7 @@
             wchar_t *dest;
 
             if (len <= 0)
-                return PyUnicode_FromUnicode(NULL, 0);
+                return PyUnicode_New(0, 0);
             if (step == 1) {
                 return PyUnicode_FromWideChar(ptr + start,
                                               len);
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -2599,7 +2599,7 @@
             /* Skip over . and .. */
             if (wcscmp(wFileData.cFileName, L".") != 0 &&
                 wcscmp(wFileData.cFileName, L"..") != 0) {
-                v = PyUnicode_FromUnicode(wFileData.cFileName, wcslen(wFileData.cFileName));
+                v = PyUnicode_FromWideChar(wFileData.cFileName, wcslen(wFileData.cFileName));
                 if (v == NULL) {
                     Py_DECREF(d);
                     d = NULL;
@@ -2967,7 +2967,7 @@
             result = GetFullPathNameW(wpath, result, woutbufp, &wtemp);
         }
         if (result)
-            v = PyUnicode_FromUnicode(woutbufp, wcslen(woutbufp));
+            v = PyUnicode_FromWideChar(woutbufp, wcslen(woutbufp));
         else
             v = win32_error_object("GetFullPathNameW", po);
         if (woutbufp != woutbuf)
@@ -3054,7 +3054,7 @@
         return win32_error_object("CloseHandle", po);
 
     target_path[result_length] = 0;
-    result = PyUnicode_FromUnicode(target_path, result_length);
+    result = PyUnicode_FromWideChar(target_path, result_length);
     free(target_path);
     return result;
 
@@ -7750,7 +7750,7 @@
 posix_putenv(PyObject *self, PyObject *args)
 {
 #ifdef MS_WINDOWS
-    wchar_t *s1, *s2;
+    PyObject *s1, *s2;
     wchar_t *newenv;
 #else
     PyObject *os1, *os2;
@@ -7762,7 +7762,7 @@
 
 #ifdef MS_WINDOWS
     if (!PyArg_ParseTuple(args,
-                          "uu:putenv",
+                          "UU:putenv",
                           &s1, &s2))
         return NULL;
 #else
@@ -7799,26 +7799,26 @@
     /* len includes space for a trailing \0; the size arg to
        PyBytes_FromStringAndSize does not count that */
 #ifdef MS_WINDOWS
-    len = wcslen(s1) + wcslen(s2) + 2;
-    newstr = PyUnicode_FromUnicode(NULL, (int)len - 1);
-#else
-    len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
-    newstr = PyBytes_FromStringAndSize(NULL, (int)len - 1);
-#endif
+    newstr = PyUnicode_FromFormat("%U=%U", s1, s2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
-#ifdef MS_WINDOWS
     newenv = PyUnicode_AsUnicode(newstr);
     if (newenv == NULL)
         goto error;
-    _snwprintf(newenv, len, L"%s=%s", s1, s2);
     if (_wputenv(newenv)) {
         posix_error();
         goto error;
     }
 #else
+    len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
+    newstr = PyBytes_FromStringAndSize(NULL, (int)len - 1);
+    if (newstr == NULL) {
+        PyErr_NoMemory();
+        goto error;
+    }
+
     newenv = PyBytes_AS_STRING(newstr);
     PyOS_snprintf(newenv, len, "%s=%s", s1, s2);
     if (putenv(newenv)) {
diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -3900,7 +3900,7 @@
     PyObject *result;
 
     if (GetComputerNameExW(ComputerNamePhysicalDnsHostname, buf, &size))
-        return PyUnicode_FromUnicode(buf, size);
+        return PyUnicode_FromWideChar(buf, size);
 
     if (GetLastError() != ERROR_MORE_DATA)
         return PyErr_SetFromWindowsErr(0);
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -701,7 +701,7 @@
 
     /* If no format_spec is provided, use an empty string */
     if (format_spec == NULL) {
-        empty = PyUnicode_FromUnicode(NULL, 0);
+        empty = PyUnicode_New(0, 0);
         format_spec = empty;
     }
 
diff --git a/PC/_msi.c b/PC/_msi.c
--- a/PC/_msi.c
+++ b/PC/_msi.c
@@ -18,7 +18,7 @@
 uuidcreate(PyObject* obj, PyObject*args)
 {
     UUID result;
-    unsigned short *cresult;
+    wchar_t *cresult;
     PyObject *oresult;
 
     /* May return ok, local only, and no address.
@@ -35,7 +35,7 @@
         return NULL;
     }
 
-    oresult = PyUnicode_FromUnicode(cresult, wcslen(cresult));
+    oresult = PyUnicode_FromWideChar(cresult, wcslen(cresult));
     RpcStringFreeW(&cresult);
     return oresult;
 
@@ -379,7 +379,7 @@
     }
     if (status != ERROR_SUCCESS)
         return msierror((int) status);
-    string = PyUnicode_FromUnicode(res, size);
+    string = PyUnicode_FromWideChar(res, size);
     if (buf != res)
         free(res);
     return string;
@@ -401,7 +401,7 @@
 {
     int status;
     int field;
-    Py_UNICODE *data;
+    wchar_t *data;
 
     if (!PyArg_ParseTuple(args, "iu:SetString", &field, &data))
         return NULL;
@@ -418,7 +418,7 @@
 {
     int status;
     int field;
-    Py_UNICODE *data;
+    wchar_t *data;
 
     if (!PyArg_ParseTuple(args, "iu:SetStream", &field, &data))
         return NULL;
diff --git a/PC/_subprocess.c b/PC/_subprocess.c
--- a/PC/_subprocess.c
+++ b/PC/_subprocess.c
@@ -419,14 +419,14 @@
     PyObject* environment;
     wchar_t *wenvironment;
 
-    Py_UNICODE* application_name;
-    Py_UNICODE* command_line;
+    wchar_t* application_name;
+    wchar_t* command_line;
     PyObject* process_attributes; /* ignored */
     PyObject* thread_attributes; /* ignored */
     int inherit_handles;
     int creation_flags;
     PyObject* env_mapping;
-    Py_UNICODE* current_directory;
+    wchar_t* current_directory;
     PyObject* startup_info;
 
     if (! PyArg_ParseTuple(args, "ZZOOiiOZO:CreateProcess",
@@ -454,15 +454,10 @@
     if (PyErr_Occurred())
         return NULL;
 
-    if (env_mapping == Py_None)
-        environment = NULL;
-    else {
+    if (env_mapping != Py_None) {
         environment = getenvironment(env_mapping);
         if (! environment)
             return NULL;
-    }
-
-    if (environment) {
         wenvironment = PyUnicode_AsUnicode(environment);
         if (wenvironment == NULL)
         {
@@ -470,8 +465,10 @@
             return NULL;
         }
     }
-    else
+    else {
+        environment = NULL;
         wenvironment = NULL;
+    }
 
     Py_BEGIN_ALLOW_THREADS
     result = CreateProcessW(application_name,
diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c
--- a/PC/msvcrtmodule.c
+++ b/PC/msvcrtmodule.c
@@ -211,7 +211,7 @@
 static PyObject *
 msvcrt_getwch(PyObject *self, PyObject *args)
 {
-    Py_UNICODE ch;
+    wchar_t ch;
 
     if (!PyArg_ParseTuple(args, ":getwch"))
         return NULL;
@@ -254,7 +254,7 @@
 static PyObject *
 msvcrt_getwche(PyObject *self, PyObject *args)
 {
-    Py_UNICODE ch;
+    wchar_t ch;
 
     if (!PyArg_ParseTuple(args, ":getwche"))
         return NULL;
diff --git a/PC/winreg.c b/PC/winreg.c
--- a/PC/winreg.c
+++ b/PC/winreg.c
@@ -1227,8 +1227,8 @@
 static PyObject *
 PyExpandEnvironmentStrings(PyObject *self, PyObject *args)
 {
-    Py_UNICODE *retValue = NULL;
-    Py_UNICODE *src;
+    wchar_t *retValue = NULL;
+    wchar_t *src;
     DWORD retValueSize;
     DWORD rc;
     PyObject *o;
@@ -1241,7 +1241,7 @@
         return PyErr_SetFromWindowsErrWithFunction(retValueSize,
                                         "ExpandEnvironmentStrings");
     }
-    retValue = (Py_UNICODE *)PyMem_Malloc(retValueSize * sizeof(Py_UNICODE));
+    retValue = (wchar_t *)PyMem_Malloc(retValueSize * sizeof(wchar_t));
     if (retValue == NULL) {
         return PyErr_NoMemory();
     }
@@ -1252,7 +1252,7 @@
         return PyErr_SetFromWindowsErrWithFunction(retValueSize,
                                         "ExpandEnvironmentStrings");
     }
-    o = PyUnicode_FromUnicode(retValue, wcslen(retValue));
+    o = PyUnicode_FromWideChar(retValue, wcslen(retValue));
     PyMem_Free(retValue);
     return o;
 }
@@ -1537,7 +1537,7 @@
 {
     HKEY hKey;
     PyObject *obKey;
-    Py_UNICODE *valueName;
+    wchar_t *valueName;
     PyObject *obRes;
     PyObject *value;
     BYTE *data;
diff --git a/PC/winsound.c b/PC/winsound.c
--- a/PC/winsound.c
+++ b/PC/winsound.c
@@ -72,7 +72,7 @@
 static PyObject *
 sound_playsound(PyObject *s, PyObject *args)
 {
-    Py_UNICODE *wsound;
+    wchar_t *wsound;
     int flags;
     int ok;
 
diff --git a/Python/dynload_win.c b/Python/dynload_win.c
--- a/Python/dynload_win.c
+++ b/Python/dynload_win.c
@@ -249,7 +249,7 @@
                     "DLL load failed: ");
 
                 PyUnicode_AppendAndDel(&message,
-                    PyUnicode_FromUnicode(
+                    PyUnicode_FromWideChar(
                         theInfo,
                         theLength));
             }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 03:25:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 03:25:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_compiler_warnings?=
Message-ID: <E1RSg3H-0007WQ-1O@dinsdale.python.org>

http://hg.python.org/cpython/rev/0212e83e7c41
changeset:   73684:0212e83e7c41
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 02:30:47 2011 +0100
summary:
  Fix compiler warnings

files:
  Modules/_io/textio.c    |  2 +-
  Modules/posixmodule.c   |  2 +-
  Objects/unicodeobject.c |  3 ++-
  3 files changed, 4 insertions(+), 3 deletions(-)


diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c
--- a/Modules/_io/textio.c
+++ b/Modules/_io/textio.c
@@ -1431,7 +1431,7 @@
 
     /* Read a chunk, decode it, and put the result in self._decoded_chars. */
     if (size_hint > 0) {
-        size_hint = Py_MAX(self->b2cratio, 1.0) * size_hint;
+        size_hint = (Py_ssize_t)(Py_MAX(self->b2cratio, 1.0) * size_hint);
     }
     chunk_size = PyLong_FromSsize_t(Py_MAX(self->chunk_size, size_hint));
     if (chunk_size == NULL)
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7756,9 +7756,9 @@
     PyObject *os1, *os2;
     char *s1, *s2;
     char *newenv;
+    size_t len;
 #endif
     PyObject *newstr = NULL;
-    size_t len;
 
 #ifdef MS_WINDOWS
     if (!PyArg_ParseTuple(args,
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -4330,7 +4330,8 @@
 refit_partial_string(PyObject **unicode, int kind, void *data, Py_ssize_t n)
 {
     PyObject *tmp;
-    Py_ssize_t k, maxchar;
+    Py_ssize_t k;
+    Py_UCS4 maxchar;
     for (k = 0, maxchar = 0; k < n; k++)
         maxchar = Py_MAX(maxchar, PyUnicode_READ(kind, data, k));
     tmp = PyUnicode_New(PyUnicode_GET_LENGTH(*unicode), maxchar);

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 03:25:39 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 03:25:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313441=3A_Disable_t?=
 =?utf8?q?emporary_the_check_on_the_maximum_character_until?=
Message-ID: <E1RSg3H-0007WW-Qx@dinsdale.python.org>

http://hg.python.org/cpython/rev/6f9af4e3c1db
changeset:   73685:6f9af4e3c1db
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 03:27:53 2011 +0100
summary:
  Issue #13441: Disable temporary the check on the maximum character until
the Solaris issue is solved.

But add assertion on the maximum character in various encoders: UTF-7, UTF-8,
wide character (wchar_t*, Py_UNICODE*), unicode-escape, raw-unicode-escape.

Fix also unicode_encode_ucs1() for backslashreplace error handler: Python is
now always "wide".

files:
  Objects/unicodeobject.c |  32 ++++++++++------------------
  1 files changed, 12 insertions(+), 20 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -379,19 +379,6 @@
             if (ch > maxchar)
                 maxchar = ch;
         }
-        if (maxchar > 0x10FFFF) {
-            printf("Invalid Unicode string! {");
-            for (i=0; i < ascii->length; i++)
-            {
-                Py_UCS4 ch = PyUnicode_READ(kind, data, i);
-                if (i)
-                    printf(", U+%04x", ch);
-                else
-                    printf("U+%04x", ch);
-            }
-            printf("} (len=%lu)\n", ascii->length);
-            abort();
-        }
         if (kind == PyUnicode_1BYTE_KIND) {
             if (ascii->state.ascii == 0) {
                 assert(maxchar >= 128);
@@ -406,7 +393,9 @@
         }
         else {
             assert(maxchar >= 0x10000);
-            assert(maxchar <= 0x10FFFF);
+            /* FIXME: Issue #13441: on Solaris, localeconv() and strxfrm()
+               return characters outside the range U+0000-U+10FFFF. */
+            /* assert(maxchar <= 0x10FFFF); */
         }
     }
     return 1;
@@ -3482,6 +3471,7 @@
             four_bytes = PyUnicode_4BYTE_DATA(unicode);
             for (; four_bytes < ucs4_end; ++four_bytes, ++w) {
                 if (*four_bytes > 0xFFFF) {
+                    assert(*four_bytes <= 0x10FFFF);
                     /* encode surrogate pair in this case */
                     *w++ = 0xD800 | ((*four_bytes - 0x10000) >> 10);
                     *w   = 0xDC00 | ((*four_bytes - 0x10000) & 0x3FF);
@@ -4128,6 +4118,8 @@
         continue;
 encode_char:
         if (ch >= 0x10000) {
+            assert(ch <= 0x10FFFF);
+
             /* code first surrogate */
             base64bits += 16;
             base64buffer = (base64buffer << 16) | 0xd800 | ((ch-0x10000) >> 10);
@@ -4899,6 +4891,7 @@
             *p++ = (char)(0x80 | ((ch >> 6) & 0x3f));
             *p++ = (char)(0x80 | (ch & 0x3f));
         } else /* ch >= 0x10000 */ {
+            assert(ch <= 0x10FFFF);
             /* Encode UCS4 Unicode ordinals */
             *p++ = (char)(0xf0 | (ch >> 18));
             *p++ = (char)(0x80 | ((ch >> 12) & 0x3f));
@@ -5971,6 +5964,7 @@
 
         /* Map 21-bit characters to '\U00xxxxxx' */
         else if (ch >= 0x10000) {
+            assert(ch <= 0x10FFFF);
             *p++ = '\\';
             *p++ = 'U';
             *p++ = Py_hexdigits[(ch >> 28) & 0x0000000F];
@@ -6191,6 +6185,7 @@
         Py_UCS4 ch = PyUnicode_READ(kind, data, pos);
         /* Map 32-bit characters to '\Uxxxxxxxx' */
         if (ch >= 0x10000) {
+            assert(ch <= 0x10FFFF);
             *p++ = '\\';
             *p++ = 'U';
             *p++ = Py_hexdigits[(ch >> 28) & 0xf];
@@ -6546,17 +6541,14 @@
                         repsize += 2+3+1;
                     else if (ch < 10000)
                         repsize += 2+4+1;
-#ifndef Py_UNICODE_WIDE
-                    else
-                        repsize += 2+5+1;
-#else
                     else if (ch < 100000)
                         repsize += 2+5+1;
                     else if (ch < 1000000)
                         repsize += 2+6+1;
-                    else
+                    else {
+                        assert(ch <= 0x10FFFF);
                         repsize += 2+7+1;
-#endif
+                    }
                 }
                 requiredsize = respos+repsize+(size-collend);
                 if (requiredsize > ressize) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 03:29:06 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 03:29:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Remove_=22=23ifdef_Py=5FUNI?=
 =?utf8?q?CODE=5FWIDE=22=3A_Python_is_now_always_wide?=
Message-ID: <E1RSg6c-0007i7-AJ@dinsdale.python.org>

http://hg.python.org/cpython/rev/59dcb6249034
changeset:   73686:59dcb6249034
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 03:31:20 2011 +0100
summary:
  Remove "#ifdef Py_UNICODE_WIDE": Python is now always wide

files:
  Modules/_sre.c       |  12 ++++--------
  Python/bltinmodule.c |  11 ++---------
  Python/traceback.c   |  10 ++--------
  3 files changed, 8 insertions(+), 25 deletions(-)


diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -500,7 +500,7 @@
     case SRE_OP_IN:
         /* repeated set */
         TRACE(("|%p|%p|COUNT IN\n", pattern, ptr));
-        while (ptr < end && 
+        while (ptr < end &&
                SRE_CHARSET(pattern + 2, SRE_CHARGET(state, ptr, 0)))
             ptr += state->charsize;
         break;
@@ -1030,7 +1030,7 @@
                 ctx->u.chr = ctx->pattern[ctx->pattern[0]+1];
                 for (;;) {
                     while (ctx->count >= (Py_ssize_t) ctx->pattern[1] &&
-                           (ctx->ptr >= end || 
+                           (ctx->ptr >= end ||
                             SRE_CHARGET(state, ctx->ptr, 0) != ctx->u.chr)) {
                         ctx->ptr -= state->charsize;
                         ctx->count--;
@@ -1302,7 +1302,7 @@
                     if (!p || !e || e < p)
                         RETURN_FAILURE;
                     while (p < e) {
-                        if (ctx->ptr >= end || 
+                        if (ctx->ptr >= end ||
                             SRE_CHARGET(state, ctx->ptr, 0) != SRE_CHARGET(state, p, 0))
                             RETURN_FAILURE;
                         p += state->charsize;
@@ -2664,7 +2664,7 @@
     if (pattern == Py_None) {
         self->logical_charsize = -1;
         self->charsize = -1;
-    } 
+    }
     else {
         Py_ssize_t p_length;
         if (!getstring(pattern, &p_length, &self->logical_charsize,
@@ -3021,10 +3021,8 @@
                 GET_ARG; max = arg;
                 if (min > max)
                     FAIL;
-#ifdef Py_UNICODE_WIDE
                 if (max > 65535)
                     FAIL;
-#endif
                 if (!_validate_inner(code, code+skip-4, groups))
                     FAIL;
                 code += skip-4;
@@ -3042,10 +3040,8 @@
                 GET_ARG; max = arg;
                 if (min > max)
                     FAIL;
-#ifdef Py_UNICODE_WIDE
                 if (max > 65535)
                     FAIL;
-#endif
                 if (!_validate_inner(code, code+skip-3, groups))
                     FAIL;
                 code += skip-3;
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -518,17 +518,10 @@
     return PyUnicode_FromOrdinal(x);
 }
 
-PyDoc_VAR(chr_doc) = PyDoc_STR(
+PyDoc_STRVAR(chr_doc,
 "chr(i) -> Unicode character\n\
 \n\
-Return a Unicode string of one character with ordinal i; 0 <= i <= 0x10ffff."
-)
-#ifndef Py_UNICODE_WIDE
-PyDoc_STR(
-"\nIf 0x10000 <= i, a surrogate pair is returned."
-)
-#endif
-;
+Return a Unicode string of one character with ordinal i; 0 <= i <= 0x10ffff.");
 
 
 static char *
diff --git a/Python/traceback.c b/Python/traceback.c
--- a/Python/traceback.c
+++ b/Python/traceback.c
@@ -526,23 +526,17 @@
             char c = (char)ch;
             write(fd, &c, 1);
         }
-        else if (ch < 256) {
+        else if (ch < 0xff) {
             PUTS(fd, "\\x");
             dump_hexadecimal(2, ch, fd);
         }
-        else
-#ifdef Py_UNICODE_WIDE
-        if (ch < 65536)
-#endif
-        {
+        else if (ch < 0xffff) {
             PUTS(fd, "\\u");
             dump_hexadecimal(4, ch, fd);
-#ifdef Py_UNICODE_WIDE
         }
         else {
             PUTS(fd, "\\U");
             dump_hexadecimal(8, ch, fd);
-#endif
         }
     }
     if (truncated)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 03:36:25 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 03:36:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_find=5Fmaxchar=5Fsurrogates?=
 =?utf8?q?=28=29_reuses_surrogate_macros?=
Message-ID: <E1RSgDh-0008Ia-LN@dinsdale.python.org>

http://hg.python.org/cpython/rev/8b25f0ff8e95
changeset:   73687:8b25f0ff8e95
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 03:38:40 2011 +0100
summary:
  find_maxchar_surrogates() reuses surrogate macros

files:
  Objects/unicodeobject.c |  8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1308,12 +1308,12 @@
 #endif
         }
 #if SIZEOF_WCHAR_T == 2
-        if (*iter >= 0xD800 && *iter <= 0xDBFF
-            && (iter+1) < end && iter[1] >= 0xDC00 && iter[1] <= 0xDFFF)
+        if (Py_UNICODE_IS_HIGH_SURROGATE(iter[0])
+            && (iter+1) < end
+            && Py_UNICODE_IS_LOW_SURROGATE(iter[1]))
         {
             Py_UCS4 surrogate_val;
-            surrogate_val = (((iter[0] & 0x3FF)<<10)
-                             | (iter[1] & 0x3FF)) + 0x10000;
+            surrogate_val = Py_UNICODE_JOIN_SURROGATES(iter[0], iter[1]);
             ++(*num_surrogates);
             if (surrogate_val > *maxchar)
                 *maxchar = surrogate_val;

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Tue Nov 22 05:30:21 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 22 Nov 2011 05:30:21 +0100
Subject: [Python-checkins] Daily reference leaks (59dcb6249034): sum=0
Message-ID: <E1RShzx-0001hi-Vo@ap.vmr.nerim.net>

results for 59dcb6249034 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflognsnDy1', '-x']

From python-checkins at python.org  Tue Nov 22 13:33:43 2011
From: python-checkins at python.org (giampaolo.rodola)
Date: Tue, 22 Nov 2011 13:33:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Fix_13245=3A?=
Message-ID: <E1RSpXj-0007uj-4g@dinsdale.python.org>

http://hg.python.org/cpython/rev/f1a21f2e3bec
changeset:   73688:f1a21f2e3bec
user:        Giampaolo Rodola' <g.rodola at gmail.com>
date:        Tue Nov 22 13:33:34 2011 +0100
summary:
  Fix 13245:
sched.scheduler class constructor's timefunc and delayfunct parameters are now optional.
scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter.

Patch contributed by Matt Mulsow.

files:
  Doc/library/sched.rst  |  25 ++++++++++++++++++++-----
  Doc/whatsnew/3.3.rst   |  17 ++++++++++++++++-
  Lib/sched.py           |  18 ++++++++++--------
  Lib/test/test_sched.py |  12 ++++++++++++
  Misc/ACKS              |   1 +
  Misc/NEWS              |   5 +++++
  6 files changed, 64 insertions(+), 14 deletions(-)


diff --git a/Doc/library/sched.rst b/Doc/library/sched.rst
--- a/Doc/library/sched.rst
+++ b/Doc/library/sched.rst
@@ -14,7 +14,7 @@
 The :mod:`sched` module defines a class which implements a general purpose event
 scheduler:
 
-.. class:: scheduler(timefunc, delayfunc)
+.. class:: scheduler(timefunc=time.time, delayfunc=time.sleep)
 
    The :class:`scheduler` class defines a generic interface to scheduling events.
    It needs two functions to actually deal with the "outside world" --- *timefunc*
@@ -25,6 +25,9 @@
    event is run to allow other threads an opportunity to run in multi-threaded
    applications.
 
+   .. versionchanged:: 3.3
+      *timefunc* and *delayfunc* parameters are optional.
+
 Example::
 
    >>> import sched, time
@@ -79,26 +82,38 @@
 :class:`scheduler` instances have the following methods and attributes:
 
 
-.. method:: scheduler.enterabs(time, priority, action, argument)
+.. method:: scheduler.enterabs(time, priority, action, argument=[], kwargs={})
 
    Schedule a new event. The *time* argument should be a numeric type compatible
    with the return value of the *timefunc* function passed  to the constructor.
    Events scheduled for the same *time* will be executed in the order of their
    *priority*.
 
-   Executing the event means executing ``action(*argument)``.  *argument* must be a
-   sequence holding the parameters for *action*.
+   Executing the event means executing ``action(*argument, **kwargs)``.
+   *argument* must be a sequence holding the parameters for *action*.
+   *kwargs* must be a dictionary holding the keyword parameters for *action*.
 
    Return value is an event which may be used for later cancellation of the event
    (see :meth:`cancel`).
 
+   .. versionchanged:: 3.3
+      *argument* parameter is optional.
 
-.. method:: scheduler.enter(delay, priority, action, argument)
+   .. versionadded:: 3.3
+      *kwargs* parameter was added.
+
+
+.. method:: scheduler.enter(delay, priority, action, argument=[], kwargs={})
 
    Schedule an event for *delay* more time units. Other than the relative time, the
    other arguments, the effect and the return value are the same as those for
    :meth:`enterabs`.
 
+   .. versionchanged:: 3.3
+      *argument* parameter is optional.
+
+   .. versionadded:: 3.3
+      *kwargs* parameter was added.
 
 .. method:: scheduler.cancel(event)
 
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -572,6 +572,22 @@
 
 (:issue:`1673007`)
 
+sched
+-----
+
+* *timefunc* and *delayfunct* parameters of :class:`~sched.scheduler` class
+  constructor are now optional and defaults to :func:`time.time` and
+  :func:`time.sleep` respectively.  (Contributed by Matt Mulsow in
+  :issue:`8809`)
+
+* :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs`
+  *argument* parameter is now optional.  (Contributed by Matt Mulsow in
+  :issue:`8809`)
+
+* :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs`
+  now accept a *kwargs* parameter.  (Contributed by Matt Mulsow in
+  :issue:`8809`)
+
 Optimizations
 =============
 
@@ -744,4 +760,3 @@
 .. Issue #10998: the -Q command-line flag and related artifacts have been
    removed.  Code checking sys.flags.division_warning will need updating.
    Contributed by ?ric Araujo.
-
diff --git a/Lib/sched.py b/Lib/sched.py
--- a/Lib/sched.py
+++ b/Lib/sched.py
@@ -28,12 +28,13 @@
 # XXX instead of having to define a module or class just to hold
 # XXX the global state of your particular time and delay functions.
 
+import time
 import heapq
 from collections import namedtuple
 
 __all__ = ["scheduler"]
 
-class Event(namedtuple('Event', 'time, priority, action, argument')):
+class Event(namedtuple('Event', 'time, priority, action, argument, kwargs')):
     def __eq__(s, o): return (s.time, s.priority) == (o.time, o.priority)
     def __ne__(s, o): return (s.time, s.priority) != (o.time, o.priority)
     def __lt__(s, o): return (s.time, s.priority) <  (o.time, o.priority)
@@ -42,32 +43,33 @@
     def __ge__(s, o): return (s.time, s.priority) >= (o.time, o.priority)
 
 class scheduler:
-    def __init__(self, timefunc, delayfunc):
+
+    def __init__(self, timefunc=time.time, delayfunc=time.sleep):
         """Initialize a new instance, passing the time and delay
         functions"""
         self._queue = []
         self.timefunc = timefunc
         self.delayfunc = delayfunc
 
-    def enterabs(self, time, priority, action, argument):
+    def enterabs(self, time, priority, action, argument=[], kwargs={}):
         """Enter a new event in the queue at an absolute time.
 
         Returns an ID for the event which can be used to remove it,
         if necessary.
 
         """
-        event = Event(time, priority, action, argument)
+        event = Event(time, priority, action, argument, kwargs)
         heapq.heappush(self._queue, event)
         return event # The ID
 
-    def enter(self, delay, priority, action, argument):
+    def enter(self, delay, priority, action, argument=[], kwargs={}):
         """A variant that specifies the time as a relative time.
 
         This is actually the more commonly used interface.
 
         """
         time = self.timefunc() + delay
-        return self.enterabs(time, priority, action, argument)
+        return self.enterabs(time, priority, action, argument, kwargs)
 
     def cancel(self, event):
         """Remove an event from the queue.
@@ -111,7 +113,7 @@
         timefunc = self.timefunc
         pop = heapq.heappop
         while q:
-            time, priority, action, argument = checked_event = q[0]
+            time, priority, action, argument, kwargs = checked_event = q[0]
             now = timefunc()
             if now < time:
                 delayfunc(time - now)
@@ -120,7 +122,7 @@
                 # Verify that the event was not removed or altered
                 # by another thread after we last looked at q[0].
                 if event is checked_event:
-                    action(*argument)
+                    action(*argument, **kwargs)
                     delayfunc(0)   # Let other threads run
                 else:
                     heapq.heappush(q, event)
diff --git a/Lib/test/test_sched.py b/Lib/test/test_sched.py
--- a/Lib/test/test_sched.py
+++ b/Lib/test/test_sched.py
@@ -72,6 +72,18 @@
         scheduler.run()
         self.assertEqual(scheduler._queue, [])
 
+    def test_args_kwargs(self):
+        flag = []
+
+        def fun(*a, **b):
+            flag.append(None)
+            self.assertEqual(a, (1,2,3))
+            self.assertEqual(b, {"foo":1})
+
+        scheduler = sched.scheduler(time.time, time.sleep)
+        z = scheduler.enterabs(0.01, 1, fun, argument=(1,2,3), kwargs={"foo":1})
+        scheduler.run()
+        self.assertEqual(flag, [None])
 
 def test_main():
     support.run_unittest(TestCase)
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -11,6 +11,7 @@
 PS: In the standard Python distribution, this file is encoded in UTF-8
 and the list is in rough alphabetical order by last names.
 
+Matt Mulsow
 Rajiv Abraham
 David Abrahams
 Ron Adam
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -387,6 +387,11 @@
 Library
 -------
 
+- Issue #13245: sched.scheduler class constructor's timefunc and
+  delayfunct parameters are now optional.
+  scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter.
+  Patch contributed by Matt Mulsow.
+
 - Issue #12328: Under Windows, refactor handling of Ctrl-C events and
   make _multiprocessing.win32.WaitForMultipleObjects interruptible when
   the wait_flag parameter is false.  Patch by sbt.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 13:38:03 2011
From: python-checkins at python.org (giampaolo.rodola)
Date: Tue, 22 Nov 2011 13:38:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_fix_wrong_credit_and_issue_?=
 =?utf8?q?id_given_in_previous_commit?=
Message-ID: <E1RSpbv-0008JW-2m@dinsdale.python.org>

http://hg.python.org/cpython/rev/dc52db0fa2e5
changeset:   73689:dc52db0fa2e5
user:        Giampaolo Rodola' <g.rodola at gmail.com>
date:        Tue Nov 22 13:37:58 2011 +0100
summary:
  fix wrong credit and issue id given in previous commit

files:
  Doc/whatsnew/3.3.rst |  12 ++++++------
  Misc/ACKS            |   2 +-
  Misc/NEWS            |   2 +-
  3 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -577,16 +577,16 @@
 
 * *timefunc* and *delayfunct* parameters of :class:`~sched.scheduler` class
   constructor are now optional and defaults to :func:`time.time` and
-  :func:`time.sleep` respectively.  (Contributed by Matt Mulsow in
-  :issue:`8809`)
+  :func:`time.sleep` respectively.  (Contributed by Chris Clark in
+  :issue:`13245`)
 
 * :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs`
-  *argument* parameter is now optional.  (Contributed by Matt Mulsow in
-  :issue:`8809`)
+  *argument* parameter is now optional.  (Contributed by Chris Clark in
+  :issue:`13245`)
 
 * :meth:`~sched.scheduler.enter` and :meth:`~sched.scheduler.enterabs`
-  now accept a *kwargs* parameter.  (Contributed by Matt Mulsow in
-  :issue:`8809`)
+  now accept a *kwargs* parameter.  (Contributed by Chris Clark in
+  :issue:`13245`)
 
 Optimizations
 =============
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -11,7 +11,7 @@
 PS: In the standard Python distribution, this file is encoded in UTF-8
 and the list is in rough alphabetical order by last names.
 
-Matt Mulsow
+Chris Clark
 Rajiv Abraham
 David Abrahams
 Ron Adam
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -390,7 +390,7 @@
 - Issue #13245: sched.scheduler class constructor's timefunc and
   delayfunct parameters are now optional.
   scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter.
-  Patch contributed by Matt Mulsow.
+  Patch contributed by Chris Clark.
 
 - Issue #12328: Under Windows, refactor handling of Ctrl-C events and
   make _multiprocessing.win32.WaitForMultipleObjects interruptible when

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 18:56:41 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 22 Nov 2011 18:56:41 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyMTU2?=
 =?utf8?q?=3A_Skip_test=5Fmultiprocessing_on_systems_which_don=27t_support?=
 =?utf8?q?_enough?=
Message-ID: <E1RSuaH-0006h1-H6@dinsdale.python.org>

http://hg.python.org/cpython/rev/ed2d3680f489
changeset:   73690:ed2d3680f489
branch:      2.7
parent:      73682:0cd197f13400
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 22 18:35:18 2011 +0100
summary:
  Issue #12156: Skip test_multiprocessing on systems which don't support enough
POSIX semaphores (among which FreeBSD < 8).

files:
  Lib/test/test_multiprocessing.py |  18 ++++++++++++++++++
  1 files changed, 18 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -95,6 +95,22 @@
     Structure = object
     c_int = c_double = None
 
+
+def check_enough_semaphores():
+    """Check that the system supports enough semaphores to run the test."""
+    # minimum number of semaphores available according to POSIX
+    nsems_min = 256
+    try:
+        nsems = os.sysconf("SC_SEM_NSEMS_MAX")
+    except (AttributeError, ValueError):
+        # sysconf not available or setting not available
+        return
+    if nsems == -1 or nsems >= nsems_min:
+        return
+    raise unittest.SkipTest("The OS doesn't support enough semaphores "
+                            "to run the test (required: %d)." % nsems_min)
+
+
 #
 # Creates a wrapper for a function which records the time it takes to finish
 #
@@ -2236,6 +2252,8 @@
         except OSError:
             raise unittest.SkipTest("OSError raises on RLock creation, see issue 3111!")
 
+    check_enough_semaphores()
+
     if run is None:
         from test.test_support import run_unittest as run
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 18:56:42 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 22 Nov 2011 18:56:42 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyMTU2?=
 =?utf8?q?=3A_Skip_test=5Fmultiprocessing_on_systems_which_don=27t_support?=
 =?utf8?q?_enough?=
Message-ID: <E1RSuaI-0006h5-BU@dinsdale.python.org>

http://hg.python.org/cpython/rev/311bb5e65b02
changeset:   73691:311bb5e65b02
branch:      3.2
parent:      73680:f29d7d597fae
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 22 18:36:39 2011 +0100
summary:
  Issue #12156: Skip test_multiprocessing on systems which don't support enough
POSIX semaphores (among which FreeBSD < 8).

files:
  Lib/test/test_multiprocessing.py |  18 ++++++++++++++++++
  1 files changed, 18 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -98,6 +98,22 @@
     Structure = object
     c_int = c_double = None
 
+
+def check_enough_semaphores():
+    """Check that the system supports enough semaphores to run the test."""
+    # minimum number of semaphores available according to POSIX
+    nsems_min = 256
+    try:
+        nsems = os.sysconf("SC_SEM_NSEMS_MAX")
+    except (AttributeError, ValueError):
+        # sysconf not available or setting not available
+        return
+    if nsems == -1 or nsems >= nsems_min:
+        return
+    raise unittest.SkipTest("The OS doesn't support enough semaphores "
+                            "to run the test (required: %d)." % nsems_min)
+
+
 #
 # Creates a wrapper for a function which records the time it takes to finish
 #
@@ -2294,6 +2310,8 @@
         except OSError:
             raise unittest.SkipTest("OSError raises on RLock creation, see issue 3111!")
 
+    check_enough_semaphores()
+
     if run is None:
         from test.support import run_unittest as run
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 18:56:43 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 22 Nov 2011 18:56:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313093=3A_Perform_a_real_merge=2E?=
Message-ID: <E1RSuaJ-0006hb-DG@dinsdale.python.org>

http://hg.python.org/cpython/rev/52fecdc1c5d8
changeset:   73692:52fecdc1c5d8
parent:      73689:dc52db0fa2e5
parent:      73691:311bb5e65b02
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 22 18:53:49 2011 +0100
summary:
  Issue #13093: Perform a real merge.

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 18:56:44 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 22 Nov 2011 18:56:44 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312156=3A_Skip_test?=
 =?utf8?q?=5Fmultiprocessing_on_systems_which_don=27t_support_enough?=
Message-ID: <E1RSuaK-0006hf-5p@dinsdale.python.org>

http://hg.python.org/cpython/rev/439ac346b20b
changeset:   73693:439ac346b20b
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 22 18:55:22 2011 +0100
summary:
  Issue #12156: Skip test_multiprocessing on systems which don't support enough
POSIX semaphores (among which FreeBSD < 8).

files:
  Lib/test/test_multiprocessing.py |  18 ++++++++++++++++++
  1 files changed, 18 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_multiprocessing.py b/Lib/test/test_multiprocessing.py
--- a/Lib/test/test_multiprocessing.py
+++ b/Lib/test/test_multiprocessing.py
@@ -115,6 +115,22 @@
     Structure = object
     c_int = c_double = None
 
+
+def check_enough_semaphores():
+    """Check that the system supports enough semaphores to run the test."""
+    # minimum number of semaphores available according to POSIX
+    nsems_min = 256
+    try:
+        nsems = os.sysconf("SC_SEM_NSEMS_MAX")
+    except (AttributeError, ValueError):
+        # sysconf not available or setting not available
+        return
+    if nsems == -1 or nsems >= nsems_min:
+        return
+    raise unittest.SkipTest("The OS doesn't support enough semaphores "
+                            "to run the test (required: %d)." % nsems_min)
+
+
 #
 # Creates a wrapper for a function which records the time it takes to finish
 #
@@ -2349,6 +2365,8 @@
         except OSError:
             raise unittest.SkipTest("OSError raises on RLock creation, see issue 3111!")
 
+    check_enough_semaphores()
+
     if run is None:
         from test.support import run_unittest as run
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 19:52:01 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 22 Nov 2011 19:52:01 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMTU2?=
 =?utf8?q?=3A_=5FPyGILState=5FReinit=28=29=3A_Re-associate_the_auto_thread?=
 =?utf8?q?_state_with_the?=
Message-ID: <E1RSvRp-00047K-I4@dinsdale.python.org>

http://hg.python.org/cpython/rev/ba90839c4993
changeset:   73694:ba90839c4993
branch:      3.2
parent:      73691:311bb5e65b02
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 22 19:49:51 2011 +0100
summary:
  Issue #13156: _PyGILState_Reinit(): Re-associate the auto thread state with the
TLS key only if the thread that called fork() had an associated auto thread
state (this might not be the case for example for a thread created outside of
Python calling into a subinterpreter).

files:
  Python/pystate.c |  9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)


diff --git a/Python/pystate.c b/Python/pystate.c
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -592,9 +592,9 @@
     autoInterpreterState = NULL;
 }
 
-/* Reset the TLS key - called by PyOS_AfterFork.
+/* Reset the TLS key - called by PyOS_AfterFork().
  * This should not be necessary, but some - buggy - pthread implementations
- * don't flush TLS on fork, see issue #10517.
+ * don't reset TLS upon fork(), see issue #10517.
  */
 void
 _PyGILState_Reinit(void)
@@ -604,8 +604,9 @@
     if ((autoTLSkey = PyThread_create_key()) == -1)
         Py_FatalError("Could not allocate TLS entry");
 
-    /* re-associate the current thread state with the new key */
-    if (PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0)
+    /* If the thread had an associated auto thread state, reassociate it with
+     * the new key. */
+    if (tstate && PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0)
         Py_FatalError("Couldn't create autoTLSkey mapping");
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 19:52:02 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Tue, 22 Nov 2011 19:52:02 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313156=3A_=5FPyGILState=5FReinit=28=29=3A_Re-associa?=
 =?utf8?q?te_the_auto_thread_state_with_the?=
Message-ID: <E1RSvRq-00047O-A8@dinsdale.python.org>

http://hg.python.org/cpython/rev/aa6ce09d2350
changeset:   73695:aa6ce09d2350
parent:      73693:439ac346b20b
parent:      73694:ba90839c4993
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Tue Nov 22 19:51:14 2011 +0100
summary:
  Issue #13156: _PyGILState_Reinit(): Re-associate the auto thread state with the
TLS key only if the thread that called fork() had an associated auto thread
state (this might not be the case for example for a thread created outside of
Python calling into a subinterpreter).

files:
  Python/pystate.c |  9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)


diff --git a/Python/pystate.c b/Python/pystate.c
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -592,9 +592,9 @@
     autoInterpreterState = NULL;
 }
 
-/* Reset the TLS key - called by PyOS_AfterFork.
+/* Reset the TLS key - called by PyOS_AfterFork().
  * This should not be necessary, but some - buggy - pthread implementations
- * don't flush TLS on fork, see issue #10517.
+ * don't reset TLS upon fork(), see issue #10517.
  */
 void
 _PyGILState_Reinit(void)
@@ -604,8 +604,9 @@
     if ((autoTLSkey = PyThread_create_key()) == -1)
         Py_FatalError("Could not allocate TLS entry");
 
-    /* re-associate the current thread state with the new key */
-    if (PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0)
+    /* If the thread had an associated auto thread state, reassociate it with
+     * the new key. */
+    if (tstate && PyThread_set_key_value(autoTLSkey, (void *)tstate) < 0)
         Py_FatalError("Couldn't create autoTLSkey mapping");
 }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 21:19:44 2011
From: python-checkins at python.org (giampaolo.rodola)
Date: Tue, 22 Nov 2011 21:19:44 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_sort_last_committed_name_in?=
	=?utf8?q?_alphabetical_order?=
Message-ID: <E1RSwoi-0003kT-Ad@dinsdale.python.org>

http://hg.python.org/cpython/rev/6a0da9b65e54
changeset:   73696:6a0da9b65e54
user:        Giampaolo Rodola' <g.rodola at gmail.com>
date:        Tue Nov 22 21:19:37 2011 +0100
summary:
  sort last committed name in alphabetical order

files:
  Lib/sched.py |  8 ++++++++
  Misc/ACKS    |  2 +-
  2 files changed, 9 insertions(+), 1 deletions(-)


diff --git a/Lib/sched.py b/Lib/sched.py
--- a/Lib/sched.py
+++ b/Lib/sched.py
@@ -35,6 +35,9 @@
 __all__ = ["scheduler"]
 
 class Event(namedtuple('Event', 'time, priority, action, argument, kwargs')):
+    def __init__(self, *args, **kwargs):
+        super(Event, self).__init__(*args, **kwargs)
+        self._scheduled = False
     def __eq__(s, o): return (s.time, s.priority) == (o.time, o.priority)
     def __ne__(s, o): return (s.time, s.priority) != (o.time, o.priority)
     def __lt__(s, o): return (s.time, s.priority) <  (o.time, o.priority)
@@ -59,6 +62,7 @@
 
         """
         event = Event(time, priority, action, argument, kwargs)
+        event._scheduled = True
         heapq.heappush(self._queue, event)
         return event # The ID
 
@@ -81,6 +85,9 @@
         self._queue.remove(event)
         heapq.heapify(self._queue)
 
+    def is_scheduled(self, event):
+        return event._scheduled
+
     def empty(self):
         """Check whether the queue is empty."""
         return not self._queue
@@ -122,6 +129,7 @@
                 # Verify that the event was not removed or altered
                 # by another thread after we last looked at q[0].
                 if event is checked_event:
+                    event._scheduled = False
                     action(*argument, **kwargs)
                     delayfunc(0)   # Let other threads run
                 else:
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -11,7 +11,6 @@
 PS: In the standard Python distribution, this file is encoded in UTF-8
 and the list is in rough alphabetical order by last names.
 
-Chris Clark
 Rajiv Abraham
 David Abrahams
 Ron Adam
@@ -179,6 +178,7 @@
 Vadim Chugunov
 David Cinege
 Craig Citro
+Chris Clark
 Mike Clarkson
 Andrew Clegg
 Brad Clements

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 21:29:43 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Tue, 22 Nov 2011 21:29:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_fix_compiler_warning_by_imp?=
 =?utf8?q?lementing_this_more_cleverly?=
Message-ID: <E1RSwyN-0004eI-I0@dinsdale.python.org>

http://hg.python.org/cpython/rev/77ab830930ae
changeset:   73697:77ab830930ae
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 15:29:32 2011 -0500
summary:
  fix compiler warning by implementing this more cleverly

files:
  Objects/unicodeobject.c |  7 +------
  1 files changed, 1 insertions(+), 6 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6164,12 +6164,7 @@
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
     len = PyUnicode_GET_LENGTH(unicode);
-
-    switch(kind) {
-    case PyUnicode_1BYTE_KIND: expandsize = 4; break;
-    case PyUnicode_2BYTE_KIND: expandsize = 6; break;
-    case PyUnicode_4BYTE_KIND: expandsize = 10; break;
-    }
+    expandsize = kind * 2 + 2;
 
     if (len > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:02:15 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Tue, 22 Nov 2011 22:02:15 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogYnl0ZXMoKSBjYW4n?=
 =?utf8?q?t_be_used_to_get_a_representation_of_an_object=2E?=
Message-ID: <E1RSxTr-0007vz-OD@dinsdale.python.org>

http://hg.python.org/cpython/rev/88047eaa428a
changeset:   73698:88047eaa428a
branch:      3.2
parent:      73694:ba90839c4993
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Tue Nov 22 19:34:08 2011 +0100
summary:
  bytes() can't be used to get a representation of an object.

files:
  Doc/library/exceptions.rst |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -40,9 +40,9 @@
 
    The base class for all built-in exceptions.  It is not meant to be directly
    inherited by user-defined classes (for that, use :exc:`Exception`).  If
-   :func:`bytes` or :func:`str` is called on an instance of this class, the
-   representation of the argument(s) to the instance are returned, or the empty
-   string when there were no arguments.
+   :func:`str` is called on an instance of this class, the representation of
+   the argument(s) to the instance are returned, or the empty string when
+   there were no arguments.
 
    .. attribute:: args
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:02:16 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Tue, 22 Nov 2011 22:02:16 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDM2?=
 =?utf8?q?=3A_Fix_a_bogus_error_message_when_an_AST_object_was_passed?=
Message-ID: <E1RSxTs-0007w3-Jo@dinsdale.python.org>

http://hg.python.org/cpython/rev/ffcdfc534942
changeset:   73699:ffcdfc534942
branch:      3.2
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Tue Nov 22 21:51:55 2011 +0100
summary:
  Issue #13436: Fix a bogus error message when an AST object was passed
an invalid integer value.

files:
  Lib/test/test_ast.py |  11 +++++++++++
  Misc/NEWS            |   3 +++
  Parser/asdl_c.py     |   6 +-----
  3 files changed, 15 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -486,6 +486,17 @@
         self.assertEqual(ast.literal_eval('10 + 2j'), 10 + 2j)
         self.assertEqual(ast.literal_eval('1.5 - 2j'), 1.5 - 2j)
 
+    def test_bad_integer(self):
+        # issue13436: Bad error message with invalid numeric values
+        body = [ast.ImportFrom(module='time',
+                               names=[ast.alias(name='sleep')],
+                               level=None,
+                               lineno=None, col_offset=None)]
+        mod = ast.Module(body)
+        with self.assertRaises(ValueError) as cm:
+            compile(mod, 'test', 'exec')
+        self.assertIn("invalid integer value: None", str(cm.exception))
+
 
 def test_main():
     support.run_unittest(AST_Tests, ASTHelpers_Test)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13436: Fix a bogus error message when an AST object was passed
+  an invalid integer value.
+
 - Issue #13338: Handle all enumerations in _Py_ANNOTATE_MEMORY_ORDER
   to allow compiling extension modules with -Wswitch-enum on gcc.
   Initial patch by Floris Bruynooghe.
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -816,11 +816,7 @@
 {
     int i;
     if (!PyLong_Check(obj)) {
-        PyObject *s = PyObject_Repr(obj);
-        if (s == NULL) return 1;
-        PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s",
-                     PyBytes_AS_STRING(s));
-        Py_DECREF(s);
+        PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj);
         return 1;
     }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:02:17 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Tue, 22 Nov 2011 22:02:17 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDM2?=
 =?utf8?q?=3A_commit_regenerated_Python-ast=2Ec?=
Message-ID: <E1RSxTt-0007w8-FU@dinsdale.python.org>

http://hg.python.org/cpython/rev/470f7d7c57ce
changeset:   73700:470f7d7c57ce
branch:      3.2
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Tue Nov 22 21:52:30 2011 +0100
summary:
  Issue #13436: commit regenerated Python-ast.c

files:
  Python/Python-ast.c |  6 +-----
  1 files changed, 1 insertions(+), 5 deletions(-)


diff --git a/Python/Python-ast.c b/Python/Python-ast.c
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -622,11 +622,7 @@
 {
     int i;
     if (!PyLong_Check(obj)) {
-        PyObject *s = PyObject_Repr(obj);
-        if (s == NULL) return 1;
-        PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s",
-                     PyBytes_AS_STRING(s));
-        Py_DECREF(s);
+        PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj);
         return 1;
     }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:02:18 2011
From: python-checkins at python.org (amaury.forgeotdarc)
Date: Tue, 22 Nov 2011 22:02:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_branch_3=2E2?=
Message-ID: <E1RSxTu-0007wD-CZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/53cc2668d54f
changeset:   73701:53cc2668d54f
parent:      73697:77ab830930ae
parent:      73700:470f7d7c57ce
user:        Amaury Forgeot d'Arc <amauryfa at gmail.com>
date:        Tue Nov 22 22:02:01 2011 +0100
summary:
  Merge branch 3.2

files:
  Doc/library/exceptions.rst |   6 +++---
  Lib/test/test_ast.py       |  11 +++++++++++
  Misc/NEWS                  |   3 +++
  Parser/asdl_c.py           |   6 +-----
  Python/Python-ast.c        |   6 +-----
  5 files changed, 19 insertions(+), 13 deletions(-)


diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -44,9 +44,9 @@
 
    The base class for all built-in exceptions.  It is not meant to be directly
    inherited by user-defined classes (for that, use :exc:`Exception`).  If
-   :func:`bytes` or :func:`str` is called on an instance of this class, the
-   representation of the argument(s) to the instance are returned, or the empty
-   string when there were no arguments.
+   :func:`str` is called on an instance of this class, the representation of
+   the argument(s) to the instance are returned, or the empty string when
+   there were no arguments.
 
    .. attribute:: args
 
diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -491,6 +491,17 @@
         self.assertEqual(ast.literal_eval('10 + 2j'), 10 + 2j)
         self.assertEqual(ast.literal_eval('1.5 - 2j'), 1.5 - 2j)
 
+    def test_bad_integer(self):
+        # issue13436: Bad error message with invalid numeric values
+        body = [ast.ImportFrom(module='time',
+                               names=[ast.alias(name='sleep')],
+                               level=None,
+                               lineno=None, col_offset=None)]
+        mod = ast.Module(body)
+        with self.assertRaises(ValueError) as cm:
+            compile(mod, 'test', 'exec')
+        self.assertIn("invalid integer value: None", str(cm.exception))
+
 
 class ASTValidatorTests(unittest.TestCase):
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13436: Fix a bogus error message when an AST object was passed
+  an invalid integer value.
+
 - Issue #13411: memoryview objects are now hashable when the underlying
   object is hashable.
 
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -844,11 +844,7 @@
 {
     int i;
     if (!PyLong_Check(obj)) {
-        PyObject *s = PyObject_Repr(obj);
-        if (s == NULL) return 1;
-        PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s",
-                     PyBytes_AS_STRING(s));
-        Py_DECREF(s);
+        PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj);
         return 1;
     }
 
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -690,11 +690,7 @@
 {
     int i;
     if (!PyLong_Check(obj)) {
-        PyObject *s = PyObject_Repr(obj);
-        if (s == NULL) return 1;
-        PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s",
-                     PyBytes_AS_STRING(s));
-        Py_DECREF(s);
+        PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj);
         return 1;
     }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:21:53 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:21:53 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDE1?=
 =?utf8?q?=3A_os=2Eunsetenv=28=29_doesn=27t_ignore_errors_anymore=2E?=
Message-ID: <E1RSxmr-0001HK-Cf@dinsdale.python.org>

http://hg.python.org/cpython/rev/3e892f428278
changeset:   73702:3e892f428278
branch:      3.2
parent:      73694:ba90839c4993
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:01:28 2011 +0100
summary:
  Issue #13415: os.unsetenv() doesn't ignore errors anymore.

files:
  Lib/test/test_os.py   |   9 +++++++++
  Misc/NEWS             |   2 ++
  Modules/posixmodule.c |  30 ++++++++++++------------------
  3 files changed, 23 insertions(+), 18 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -423,6 +423,15 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    def test_unset_error(self):
+        if sys.platform == "win32":
+            # an environment variable is limited to 32,767 characters
+            key = 'x' * 50000
+        else:
+            # "=" is not allowed in a variable name
+            key = 'key='
+        self.assertRaises(OSError, os.environ.__delitem__, key)
+
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -80,6 +80,8 @@
 Library
 -------
 
+- Issue #13415: os.unsetenv() doesn't ignore errors anymore.
+
 - Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
   raised when the wrapped raw file is non-blocking and the write would block.
   Previous code assumed that the raw write() would raise BlockingIOError, but
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -6106,6 +6106,12 @@
        PyBytes_FromStringAndSize does not count that */
 #ifdef MS_WINDOWS
     len = wcslen(s1) + wcslen(s2) + 2;
+    if (_MAX_ENV < (len - 1)) {
+        PyErr_Format(PyExc_ValueError,
+                     "the environment variable is longer than %u characters",
+                     _MAX_ENV);
+        goto error;
+    }
     newstr = PyUnicode_FromUnicode(NULL, (int)len - 1);
 #else
     len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
@@ -6177,42 +6183,30 @@
 static PyObject *
 posix_unsetenv(PyObject *self, PyObject *args)
 {
-#ifdef MS_WINDOWS
-    char *s1;
-
-    if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
-        return NULL;
-#else
     PyObject *os1;
     char *s1;
+    int err;
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
                           PyUnicode_FSConverter, &os1))
         return NULL;
     s1 = PyBytes_AsString(os1);
-#endif
-
-    unsetenv(s1);
+
+    err = unsetenv(s1);
+    if (err)
+        return posix_error();
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
      * happen after the real unsetenv() call because the
      * old value was still accessible until then.
      */
-    if (PyDict_DelItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0)
-#else
-                       os1
-#endif
-                       )) {
+    if (PyDict_DelItem(posix_putenv_garbage, os1)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
 
-#ifndef MS_WINDOWS
     Py_DECREF(os1);
-#endif
     Py_RETURN_NONE;
 }
 #endif /* unsetenv */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:21:54 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:21:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?b?OiAoTWVyZ2UgMy4yKSBJc3N1ZSAjMTM0MTU6IG9zLnVuc2V0ZW52KCkgZG9lc24n?=
 =?utf8?q?t_ignore_errors_anymore=2E?=
Message-ID: <E1RSxms-0001HO-Jp@dinsdale.python.org>

http://hg.python.org/cpython/rev/aa55b7dc43f7
changeset:   73703:aa55b7dc43f7
parent:      73697:77ab830930ae
parent:      73702:3e892f428278
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:16:17 2011 +0100
summary:
  (Merge 3.2) Issue #13415: os.unsetenv() doesn't ignore errors anymore.

files:
  Lib/test/test_os.py   |    9 ++
  Misc/NEWS             |    2 +
  Modules/posixmodule.c |  124 +++++++++--------------------
  3 files changed, 49 insertions(+), 86 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -447,6 +447,15 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    def test_unset_error(self):
+        if sys.platform == "win32":
+            # an environment variable is limited to 32,767 characters
+            key = 'x' * 50000
+        else:
+            # "=" is not allowed in a variable name
+            key = 'key='
+        self.assertRaises(OSError, os.environ.__delitem__, key)
+
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -387,6 +387,8 @@
 Library
 -------
 
+- Issue #13415: os.unsetenv() doesn't ignore errors anymore.
+
 - Issue #13245: sched.scheduler class constructor's timefunc and
   delayfunct parameters are now optional.
   scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7749,61 +7749,28 @@
 static PyObject *
 posix_putenv(PyObject *self, PyObject *args)
 {
+    PyObject *newstr = NULL;
 #ifdef MS_WINDOWS
-    PyObject *s1, *s2;
+    PyObject *os1, *os2;
     wchar_t *newenv;
-#else
-    PyObject *os1, *os2;
-    char *s1, *s2;
-    char *newenv;
-    size_t len;
-#endif
-    PyObject *newstr = NULL;
-
-#ifdef MS_WINDOWS
+
     if (!PyArg_ParseTuple(args,
                           "UU:putenv",
-                          &s1, &s2))
-        return NULL;
-#else
-    if (!PyArg_ParseTuple(args,
-                          "O&O&:putenv",
-                          PyUnicode_FSConverter, &os1,
-                          PyUnicode_FSConverter, &os2))
-        return NULL;
-    s1 = PyBytes_AsString(os1);
-    s2 = PyBytes_AsString(os2);
-#endif
-
-#if defined(PYOS_OS2)
-    if (stricmp(s1, "BEGINLIBPATH") == 0) {
-        APIRET rc;
-
-        rc = DosSetExtLIBPATH(s2, BEGIN_LIBPATH);
-        if (rc != NO_ERROR) {
-            os2_error(rc);
-            goto error;
-        }
-
-    } else if (stricmp(s1, "ENDLIBPATH") == 0) {
-        APIRET rc;
-
-        rc = DosSetExtLIBPATH(s2, END_LIBPATH);
-        if (rc != NO_ERROR) {
-            os2_error(rc);
-            goto error;
-        }
-    } else {
-#endif
-    /* XXX This can leak memory -- not easy to fix :-( */
-    /* len includes space for a trailing \0; the size arg to
-       PyBytes_FromStringAndSize does not count that */
-#ifdef MS_WINDOWS
-    newstr = PyUnicode_FromFormat("%U=%U", s1, s2);
+                          &os1, &os2))
+        return NULL;
+
+    newstr = PyUnicode_FromFormat("%U=%U", os1, os2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
+    if (_MAX_ENV < PyUnicode_GET_LENGTH(newstr)) {
+        PyErr_Format(PyExc_ValueError,
+                     "the environment variable is longer than %u characters",
+                     _MAX_ENV);
+        goto error;
+    }
+
     newenv = PyUnicode_AsUnicode(newstr);
     if (newenv == NULL)
         goto error;
@@ -7812,15 +7779,25 @@
         goto error;
     }
 #else
-    len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
-    newstr = PyBytes_FromStringAndSize(NULL, (int)len - 1);
+    PyObject *os1, *os2;
+    char *s1, *s2;
+    char *newenv;
+
+    if (!PyArg_ParseTuple(args,
+                          "O&O&:putenv",
+                          PyUnicode_FSConverter, &os1,
+                          PyUnicode_FSConverter, &os2))
+        return NULL;
+    s1 = PyBytes_AsString(os1);
+    s2 = PyBytes_AsString(os2);
+
+    newstr = PyBytes_FromFormat("%s=%s", s1, s2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
 
     newenv = PyBytes_AS_STRING(newstr);
-    PyOS_snprintf(newenv, len, "%s=%s", s1, s2);
     if (putenv(newenv)) {
         posix_error();
         goto error;
@@ -7831,13 +7808,7 @@
      * this will cause previous value to be collected.  This has to
      * happen after the real putenv() call because the old value
      * was still accessible until then. */
-    if (PyDict_SetItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0),
-#else
-                       os1,
-#endif
-                       newstr)) {
+    if (PyDict_SetItem(posix_putenv_garbage, os1, newstr)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
@@ -7845,10 +7816,6 @@
         Py_DECREF(newstr);
     }
 
-#if defined(PYOS_OS2)
-    }
-#endif
-
 #ifndef MS_WINDOWS
     Py_DECREF(os1);
     Py_DECREF(os2);
@@ -7873,42 +7840,27 @@
 static PyObject *
 posix_unsetenv(PyObject *self, PyObject *args)
 {
-#ifdef MS_WINDOWS
-    char *s1;
-
-    if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
-        return NULL;
-#else
-    PyObject *os1;
-    char *s1;
+    PyObject *name;
+    int err;
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
-                          PyUnicode_FSConverter, &os1))
-        return NULL;
-    s1 = PyBytes_AsString(os1);
-#endif
-
-    unsetenv(s1);
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+
+    err = unsetenv(PyBytes_AS_STRING(name));
+    if (err)
+        return posix_error();
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
      * happen after the real unsetenv() call because the
      * old value was still accessible until then.
      */
-    if (PyDict_DelItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0)
-#else
-                       os1
-#endif
-                       )) {
+    if (PyDict_DelItem(posix_putenv_garbage, name)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
-
-#ifndef MS_WINDOWS
-    Py_DECREF(os1);
-#endif
+    Py_DECREF(name);
     Py_RETURN_NONE;
 }
 #endif /* unsetenv */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:21:55 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:21:55 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzNDE1?=
 =?utf8?q?=3A_os=2Eunsetenv=28=29_doesn=27t_ignore_errors_anymore=2E?=
Message-ID: <E1RSxmt-0001HS-EW@dinsdale.python.org>

http://hg.python.org/cpython/rev/53cf6f9f374e
changeset:   73704:53cf6f9f374e
branch:      2.7
parent:      73690:ed2d3680f489
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:20:13 2011 +0100
summary:
  Issue #13415: os.unsetenv() doesn't ignore errors anymore.

files:
  Lib/test/test_os.py   |   9 +++++++++
  Misc/NEWS             |   2 ++
  Modules/posixmodule.c |  13 ++++++++++++-
  3 files changed, 23 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -361,6 +361,15 @@
                 value = popen.read().strip()
                 self.assertEqual(value, "World")
 
+    def test_unset_error(self):
+        if sys.platform == "win32":
+            # an environment variable is limited to 32,767 characters
+            key = 'x' * 50000
+        else:
+            # "=" is not allowed in a variable name
+            key = 'key='
+        self.assertRaises(OSError, os.environ.__delitem__, key)
+
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,8 @@
 Library
 -------
 
+- Issue #13415: os.unsetenv() doesn't ignore errors anymore.
+
 - Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
   raised when the wrapped raw file is non-blocking and the write would block.
   Previous code assumed that the raw write() would raise BlockingIOError, but
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -6994,6 +6994,14 @@
 
     /* XXX This can leak memory -- not easy to fix :-( */
     len = strlen(s1) + strlen(s2) + 2;
+#ifdef MS_WINDOWS
+    if (_MAX_ENV < (len - 1)) {
+        PyErr_Format(PyExc_ValueError,
+                     "the environment variable is longer than %u bytes",
+                     _MAX_ENV);
+        return NULL;
+    }
+#endif
     /* len includes space for a trailing \0; the size arg to
        PyString_FromStringAndSize does not count that */
     newstr = PyString_FromStringAndSize(NULL, (int)len - 1);
@@ -7036,11 +7044,14 @@
 posix_unsetenv(PyObject *self, PyObject *args)
 {
     char *s1;
+    int err;
 
     if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
         return NULL;
 
-    unsetenv(s1);
+    err = unsetenv(s1);
+    if (err)
+        return posix_error();
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:21:56 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:21:56 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAobWVyZ2UgMy4yIC0+IDMuMik6?=
	=?utf8?q?_Merge_branch_3=2E2?=
Message-ID: <E1RSxmu-0001HW-D7@dinsdale.python.org>

http://hg.python.org/cpython/rev/19999f594066
changeset:   73705:19999f594066
branch:      3.2
parent:      73700:470f7d7c57ce
parent:      73702:3e892f428278
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:22:26 2011 +0100
summary:
  Merge branch 3.2

files:
  Lib/test/test_os.py   |   9 +++++++++
  Misc/NEWS             |   2 ++
  Modules/posixmodule.c |  30 ++++++++++++------------------
  3 files changed, 23 insertions(+), 18 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -423,6 +423,15 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    def test_unset_error(self):
+        if sys.platform == "win32":
+            # an environment variable is limited to 32,767 characters
+            key = 'x' * 50000
+        else:
+            # "=" is not allowed in a variable name
+            key = 'key='
+        self.assertRaises(OSError, os.environ.__delitem__, key)
+
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -83,6 +83,8 @@
 Library
 -------
 
+- Issue #13415: os.unsetenv() doesn't ignore errors anymore.
+
 - Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
   raised when the wrapped raw file is non-blocking and the write would block.
   Previous code assumed that the raw write() would raise BlockingIOError, but
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -6106,6 +6106,12 @@
        PyBytes_FromStringAndSize does not count that */
 #ifdef MS_WINDOWS
     len = wcslen(s1) + wcslen(s2) + 2;
+    if (_MAX_ENV < (len - 1)) {
+        PyErr_Format(PyExc_ValueError,
+                     "the environment variable is longer than %u characters",
+                     _MAX_ENV);
+        goto error;
+    }
     newstr = PyUnicode_FromUnicode(NULL, (int)len - 1);
 #else
     len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
@@ -6177,42 +6183,30 @@
 static PyObject *
 posix_unsetenv(PyObject *self, PyObject *args)
 {
-#ifdef MS_WINDOWS
-    char *s1;
-
-    if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
-        return NULL;
-#else
     PyObject *os1;
     char *s1;
+    int err;
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
                           PyUnicode_FSConverter, &os1))
         return NULL;
     s1 = PyBytes_AsString(os1);
-#endif
-
-    unsetenv(s1);
+
+    err = unsetenv(s1);
+    if (err)
+        return posix_error();
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
      * happen after the real unsetenv() call because the
      * old value was still accessible until then.
      */
-    if (PyDict_DelItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0)
-#else
-                       os1
-#endif
-                       )) {
+    if (PyDict_DelItem(posix_putenv_garbage, os1)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
 
-#ifndef MS_WINDOWS
     Py_DECREF(os1);
-#endif
     Py_RETURN_NONE;
 }
 #endif /* unsetenv */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:21:57 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:21:57 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge_branch_default?=
Message-ID: <E1RSxmv-0001Ha-Dv@dinsdale.python.org>

http://hg.python.org/cpython/rev/b6268c4a0c99
changeset:   73706:b6268c4a0c99
parent:      73701:53cc2668d54f
parent:      73703:aa55b7dc43f7
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:23:23 2011 +0100
summary:
  Merge branch default

files:
  Lib/test/test_os.py   |    9 ++
  Misc/NEWS             |    2 +
  Modules/posixmodule.c |  124 +++++++++--------------------
  3 files changed, 49 insertions(+), 86 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -447,6 +447,15 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    def test_unset_error(self):
+        if sys.platform == "win32":
+            # an environment variable is limited to 32,767 characters
+            key = 'x' * 50000
+        else:
+            # "=" is not allowed in a variable name
+            key = 'key='
+        self.assertRaises(OSError, os.environ.__delitem__, key)
+
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -390,6 +390,8 @@
 Library
 -------
 
+- Issue #13415: os.unsetenv() doesn't ignore errors anymore.
+
 - Issue #13245: sched.scheduler class constructor's timefunc and
   delayfunct parameters are now optional.
   scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7749,61 +7749,28 @@
 static PyObject *
 posix_putenv(PyObject *self, PyObject *args)
 {
+    PyObject *newstr = NULL;
 #ifdef MS_WINDOWS
-    PyObject *s1, *s2;
+    PyObject *os1, *os2;
     wchar_t *newenv;
-#else
-    PyObject *os1, *os2;
-    char *s1, *s2;
-    char *newenv;
-    size_t len;
-#endif
-    PyObject *newstr = NULL;
-
-#ifdef MS_WINDOWS
+
     if (!PyArg_ParseTuple(args,
                           "UU:putenv",
-                          &s1, &s2))
-        return NULL;
-#else
-    if (!PyArg_ParseTuple(args,
-                          "O&O&:putenv",
-                          PyUnicode_FSConverter, &os1,
-                          PyUnicode_FSConverter, &os2))
-        return NULL;
-    s1 = PyBytes_AsString(os1);
-    s2 = PyBytes_AsString(os2);
-#endif
-
-#if defined(PYOS_OS2)
-    if (stricmp(s1, "BEGINLIBPATH") == 0) {
-        APIRET rc;
-
-        rc = DosSetExtLIBPATH(s2, BEGIN_LIBPATH);
-        if (rc != NO_ERROR) {
-            os2_error(rc);
-            goto error;
-        }
-
-    } else if (stricmp(s1, "ENDLIBPATH") == 0) {
-        APIRET rc;
-
-        rc = DosSetExtLIBPATH(s2, END_LIBPATH);
-        if (rc != NO_ERROR) {
-            os2_error(rc);
-            goto error;
-        }
-    } else {
-#endif
-    /* XXX This can leak memory -- not easy to fix :-( */
-    /* len includes space for a trailing \0; the size arg to
-       PyBytes_FromStringAndSize does not count that */
-#ifdef MS_WINDOWS
-    newstr = PyUnicode_FromFormat("%U=%U", s1, s2);
+                          &os1, &os2))
+        return NULL;
+
+    newstr = PyUnicode_FromFormat("%U=%U", os1, os2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
+    if (_MAX_ENV < PyUnicode_GET_LENGTH(newstr)) {
+        PyErr_Format(PyExc_ValueError,
+                     "the environment variable is longer than %u characters",
+                     _MAX_ENV);
+        goto error;
+    }
+
     newenv = PyUnicode_AsUnicode(newstr);
     if (newenv == NULL)
         goto error;
@@ -7812,15 +7779,25 @@
         goto error;
     }
 #else
-    len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
-    newstr = PyBytes_FromStringAndSize(NULL, (int)len - 1);
+    PyObject *os1, *os2;
+    char *s1, *s2;
+    char *newenv;
+
+    if (!PyArg_ParseTuple(args,
+                          "O&O&:putenv",
+                          PyUnicode_FSConverter, &os1,
+                          PyUnicode_FSConverter, &os2))
+        return NULL;
+    s1 = PyBytes_AsString(os1);
+    s2 = PyBytes_AsString(os2);
+
+    newstr = PyBytes_FromFormat("%s=%s", s1, s2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
 
     newenv = PyBytes_AS_STRING(newstr);
-    PyOS_snprintf(newenv, len, "%s=%s", s1, s2);
     if (putenv(newenv)) {
         posix_error();
         goto error;
@@ -7831,13 +7808,7 @@
      * this will cause previous value to be collected.  This has to
      * happen after the real putenv() call because the old value
      * was still accessible until then. */
-    if (PyDict_SetItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0),
-#else
-                       os1,
-#endif
-                       newstr)) {
+    if (PyDict_SetItem(posix_putenv_garbage, os1, newstr)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
@@ -7845,10 +7816,6 @@
         Py_DECREF(newstr);
     }
 
-#if defined(PYOS_OS2)
-    }
-#endif
-
 #ifndef MS_WINDOWS
     Py_DECREF(os1);
     Py_DECREF(os2);
@@ -7873,42 +7840,27 @@
 static PyObject *
 posix_unsetenv(PyObject *self, PyObject *args)
 {
-#ifdef MS_WINDOWS
-    char *s1;
-
-    if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
-        return NULL;
-#else
-    PyObject *os1;
-    char *s1;
+    PyObject *name;
+    int err;
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
-                          PyUnicode_FSConverter, &os1))
-        return NULL;
-    s1 = PyBytes_AsString(os1);
-#endif
-
-    unsetenv(s1);
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+
+    err = unsetenv(PyBytes_AS_STRING(name));
+    if (err)
+        return posix_error();
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
      * happen after the real unsetenv() call because the
      * old value was still accessible until then.
      */
-    if (PyDict_DelItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0)
-#else
-                       os1
-#endif
-                       )) {
+    if (PyDict_DelItem(posix_putenv_garbage, name)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
-
-#ifndef MS_WINDOWS
-    Py_DECREF(os1);
-#endif
+    Py_DECREF(name);
     Py_RETURN_NONE;
 }
 #endif /* unsetenv */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:30:44 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:30:44 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDM2?=
 =?utf8?q?=3A_Fix_unsetenv=28=29_test_on_Windows?=
Message-ID: <E1RSxvQ-0001ze-Vu@dinsdale.python.org>

http://hg.python.org/cpython/rev/2e5506d9a079
changeset:   73707:2e5506d9a079
branch:      3.2
parent:      73705:19999f594066
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:30:19 2011 +0100
summary:
  Issue #13436: Fix unsetenv() test on Windows

files:
  Lib/test/test_os.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -427,10 +427,11 @@
         if sys.platform == "win32":
             # an environment variable is limited to 32,767 characters
             key = 'x' * 50000
+            self.assertRaises(ValueError, os.environ.__delitem__, key)
         else:
             # "=" is not allowed in a variable name
             key = 'key='
-        self.assertRaises(OSError, os.environ.__delitem__, key)
+            self.assertRaises(OSError, os.environ.__delitem__, key)
 
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:30:45 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:30:45 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=28Merge_3=2E2=29_Issue_=2313436=3A_Fix_unsetenv=28=29_test_?=
 =?utf8?q?on_Windows?=
Message-ID: <E1RSxvR-0001zi-P2@dinsdale.python.org>

http://hg.python.org/cpython/rev/029ad97883ef
changeset:   73708:029ad97883ef
parent:      73706:b6268c4a0c99
parent:      73707:2e5506d9a079
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:32:28 2011 +0100
summary:
  (Merge 3.2) Issue #13436: Fix unsetenv() test on Windows

files:
  Lib/test/test_os.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -451,10 +451,11 @@
         if sys.platform == "win32":
             # an environment variable is limited to 32,767 characters
             key = 'x' * 50000
+            self.assertRaises(ValueError, os.environ.__delitem__, key)
         else:
             # "=" is not allowed in a variable name
             key = 'key='
-        self.assertRaises(OSError, os.environ.__delitem__, key)
+            self.assertRaises(OSError, os.environ.__delitem__, key)
 
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 22 22:30:46 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 22 Nov 2011 22:30:46 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzNDM2?=
 =?utf8?q?=3A_Fix_unsetenv=28=29_test_on_Windows?=
Message-ID: <E1RSxvS-0001zm-Ek@dinsdale.python.org>

http://hg.python.org/cpython/rev/e66ef9fa55de
changeset:   73709:e66ef9fa55de
branch:      2.7
parent:      73704:53cf6f9f374e
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 22 22:30:19 2011 +0100
summary:
  Issue #13436: Fix unsetenv() test on Windows

files:
  Lib/test/test_os.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -365,10 +365,11 @@
         if sys.platform == "win32":
             # an environment variable is limited to 32,767 characters
             key = 'x' * 50000
+            self.assertRaises(ValueError, os.environ.__delitem__, key)
         else:
             # "=" is not allowed in a variable name
             key = 'key='
-        self.assertRaises(OSError, os.environ.__delitem__, key)
+            self.assertRaises(OSError, os.environ.__delitem__, key)
 
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 00:03:21 2011
From: python-checkins at python.org (giampaolo.rodola)
Date: Wed, 23 Nov 2011 00:03:21 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_revert_cset_6a0da9b65e54_ag?=
 =?utf8?q?ainst_sched=2Epy_committed_by_accident?=
Message-ID: <E1RSzN3-0001po-Hp@dinsdale.python.org>

http://hg.python.org/cpython/rev/6132f46c2ab0
changeset:   73710:6132f46c2ab0
parent:      73708:029ad97883ef
user:        Giampaolo Rodola' <g.rodola at gmail.com>
date:        Wed Nov 23 00:03:15 2011 +0100
summary:
  revert cset 6a0da9b65e54 against sched.py committed by accident

files:
  Lib/sched.py |  8 --------
  1 files changed, 0 insertions(+), 8 deletions(-)


diff --git a/Lib/sched.py b/Lib/sched.py
--- a/Lib/sched.py
+++ b/Lib/sched.py
@@ -35,9 +35,6 @@
 __all__ = ["scheduler"]
 
 class Event(namedtuple('Event', 'time, priority, action, argument, kwargs')):
-    def __init__(self, *args, **kwargs):
-        super(Event, self).__init__(*args, **kwargs)
-        self._scheduled = False
     def __eq__(s, o): return (s.time, s.priority) == (o.time, o.priority)
     def __ne__(s, o): return (s.time, s.priority) != (o.time, o.priority)
     def __lt__(s, o): return (s.time, s.priority) <  (o.time, o.priority)
@@ -62,7 +59,6 @@
 
         """
         event = Event(time, priority, action, argument, kwargs)
-        event._scheduled = True
         heapq.heappush(self._queue, event)
         return event # The ID
 
@@ -85,9 +81,6 @@
         self._queue.remove(event)
         heapq.heapify(self._queue)
 
-    def is_scheduled(self, event):
-        return event._scheduled
-
     def empty(self):
         """Check whether the queue is empty."""
         return not self._queue
@@ -129,7 +122,6 @@
                 # Verify that the event was not removed or altered
                 # by another thread after we last looked at q[0].
                 if event is checked_event:
-                    event._scheduled = False
                     action(*argument, **kwargs)
                     delayfunc(0)   # Let other threads run
                 else:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 01:49:51 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Wed, 23 Nov 2011 01:49:51 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDU4?=
 =?utf8?q?=3A_Fix_a_memory_leak_in_the_ssl_module_when_decoding_a_certific?=
 =?utf8?q?ate?=
Message-ID: <E1RT127-00019w-Tv@dinsdale.python.org>

http://hg.python.org/cpython/rev/80d491aaeed2
changeset:   73711:80d491aaeed2
branch:      3.2
parent:      73707:2e5506d9a079
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Wed Nov 23 01:39:19 2011 +0100
summary:
  Issue #13458: Fix a memory leak in the ssl module when decoding a certificate with a subjectAltName.
Patch by Robert Xiao.

files:
  Misc/NEWS      |  3 +++
  Modules/_ssl.c |  1 +
  2 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -83,6 +83,9 @@
 Library
 -------
 
+- Issue #13458: Fix a memory leak in the ssl module when decoding a
+  certificate with a subjectAltName.  Patch by Robert Xiao.
+
 - Issue #13415: os.unsetenv() doesn't ignore errors anymore.
 
 - Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -679,6 +679,7 @@
             }
             Py_DECREF(t);
         }
+        sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free);
     }
     BIO_free(biobuf);
     if (peer_alt_names != Py_None) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 01:49:52 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Wed, 23 Nov 2011 01:49:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313458=3A_Fix_a_memory_leak_in_the_ssl_module_when_d?=
 =?utf8?q?ecoding_a_certificate?=
Message-ID: <E1RT128-0001A0-MG@dinsdale.python.org>

http://hg.python.org/cpython/rev/3b5fef34c8c7
changeset:   73712:3b5fef34c8c7
parent:      73710:6132f46c2ab0
parent:      73711:80d491aaeed2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Wed Nov 23 01:42:52 2011 +0100
summary:
  Issue #13458: Fix a memory leak in the ssl module when decoding a certificate with a subjectAltName.
Patch by Robert Xiao.

files:
  Misc/NEWS      |  3 +++
  Modules/_ssl.c |  1 +
  2 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -390,6 +390,9 @@
 Library
 -------
 
+- Issue #13458: Fix a memory leak in the ssl module when decoding a
+  certificate with a subjectAltName.  Patch by Robert Xiao.
+
 - Issue #13415: os.unsetenv() doesn't ignore errors anymore.
 
 - Issue #13245: sched.scheduler class constructor's timefunc and
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -707,6 +707,7 @@
             }
             Py_DECREF(t);
         }
+        sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free);
     }
     BIO_free(biobuf);
     if (peer_alt_names != Py_None) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 01:49:53 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Wed, 23 Nov 2011 01:49:53 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzNDU4?=
 =?utf8?q?=3A_Fix_a_memory_leak_in_the_ssl_module_when_decoding_a_certific?=
 =?utf8?q?ate?=
Message-ID: <E1RT129-0001A4-I3@dinsdale.python.org>

http://hg.python.org/cpython/rev/61a5d44020cd
changeset:   73713:61a5d44020cd
branch:      2.7
parent:      73709:e66ef9fa55de
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Wed Nov 23 01:39:19 2011 +0100
summary:
  Issue #13458: Fix a memory leak in the ssl module when decoding a certificate with a subjectAltName.
Patch by Robert Xiao.

files:
  Misc/NEWS      |  3 +++
  Modules/_ssl.c |  1 +
  2 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,9 @@
 Library
 -------
 
+- Issue #13458: Fix a memory leak in the ssl module when decoding a
+  certificate with a subjectAltName.  Patch by Robert Xiao.
+
 - Issue #13415: os.unsetenv() doesn't ignore errors anymore.
 
 - Issue #13322: Fix BufferedWriter.write() to ensure that BlockingIOError is
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -798,6 +798,7 @@
             }
             Py_DECREF(t);
         }
+        sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free);
     }
     BIO_free(biobuf);
     if (peer_alt_names != Py_None) {

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 02:06:34 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 02:06:34 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_cave_to_those_who_like_read?=
	=?utf8?q?able_code?=
Message-ID: <E1RT1II-0002HZ-47@dinsdale.python.org>

http://hg.python.org/cpython/rev/5c70f5215c24
changeset:   73714:5c70f5215c24
parent:      73697:77ab830930ae
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 19:05:49 2011 -0600
summary:
  cave to those who like readable code

files:
  Objects/unicodeobject.c |  16 +++++++++++++++-
  1 files changed, 15 insertions(+), 1 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6164,7 +6164,21 @@
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
     len = PyUnicode_GET_LENGTH(unicode);
-    expandsize = kind * 2 + 2;
+    expandsize = 0;
+    switch (kind) {
+    case PyUnicode_1BYTE_KIND:
+        expandsize = 4;
+        break;
+    case PyUnicode_2BYTE_KIND:
+        expandsize = 6;
+        break;
+    case PyUnicode_4BYTE_KIND:
+        expandsize = 10;
+        break;
+    default:
+        assert(0);
+        break;
+    }
 
     if (len > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 02:06:35 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 02:06:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_merge_heads?=
Message-ID: <E1RT1IJ-0002Hj-3n@dinsdale.python.org>

http://hg.python.org/cpython/rev/1b8db8aa5589
changeset:   73715:1b8db8aa5589
parent:      73714:5c70f5215c24
parent:      73712:3b5fef34c8c7
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 19:06:23 2011 -0600
summary:
  merge heads

files:
  Doc/library/exceptions.rst |    6 +-
  Lib/sched.py               |    8 -
  Lib/test/test_ast.py       |   11 ++
  Lib/test/test_os.py        |   10 ++
  Misc/NEWS                  |    8 +
  Modules/_ssl.c             |    1 +
  Modules/posixmodule.c      |  124 +++++++-----------------
  Parser/asdl_c.py           |    6 +-
  Python/Python-ast.c        |    6 +-
  9 files changed, 73 insertions(+), 107 deletions(-)


diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst
--- a/Doc/library/exceptions.rst
+++ b/Doc/library/exceptions.rst
@@ -44,9 +44,9 @@
 
    The base class for all built-in exceptions.  It is not meant to be directly
    inherited by user-defined classes (for that, use :exc:`Exception`).  If
-   :func:`bytes` or :func:`str` is called on an instance of this class, the
-   representation of the argument(s) to the instance are returned, or the empty
-   string when there were no arguments.
+   :func:`str` is called on an instance of this class, the representation of
+   the argument(s) to the instance are returned, or the empty string when
+   there were no arguments.
 
    .. attribute:: args
 
diff --git a/Lib/sched.py b/Lib/sched.py
--- a/Lib/sched.py
+++ b/Lib/sched.py
@@ -35,9 +35,6 @@
 __all__ = ["scheduler"]
 
 class Event(namedtuple('Event', 'time, priority, action, argument, kwargs')):
-    def __init__(self, *args, **kwargs):
-        super(Event, self).__init__(*args, **kwargs)
-        self._scheduled = False
     def __eq__(s, o): return (s.time, s.priority) == (o.time, o.priority)
     def __ne__(s, o): return (s.time, s.priority) != (o.time, o.priority)
     def __lt__(s, o): return (s.time, s.priority) <  (o.time, o.priority)
@@ -62,7 +59,6 @@
 
         """
         event = Event(time, priority, action, argument, kwargs)
-        event._scheduled = True
         heapq.heappush(self._queue, event)
         return event # The ID
 
@@ -85,9 +81,6 @@
         self._queue.remove(event)
         heapq.heapify(self._queue)
 
-    def is_scheduled(self, event):
-        return event._scheduled
-
     def empty(self):
         """Check whether the queue is empty."""
         return not self._queue
@@ -129,7 +122,6 @@
                 # Verify that the event was not removed or altered
                 # by another thread after we last looked at q[0].
                 if event is checked_event:
-                    event._scheduled = False
                     action(*argument, **kwargs)
                     delayfunc(0)   # Let other threads run
                 else:
diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -491,6 +491,17 @@
         self.assertEqual(ast.literal_eval('10 + 2j'), 10 + 2j)
         self.assertEqual(ast.literal_eval('1.5 - 2j'), 1.5 - 2j)
 
+    def test_bad_integer(self):
+        # issue13436: Bad error message with invalid numeric values
+        body = [ast.ImportFrom(module='time',
+                               names=[ast.alias(name='sleep')],
+                               level=None,
+                               lineno=None, col_offset=None)]
+        mod = ast.Module(body)
+        with self.assertRaises(ValueError) as cm:
+            compile(mod, 'test', 'exec')
+        self.assertIn("invalid integer value: None", str(cm.exception))
+
 
 class ASTValidatorTests(unittest.TestCase):
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -447,6 +447,16 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    def test_unset_error(self):
+        if sys.platform == "win32":
+            # an environment variable is limited to 32,767 characters
+            key = 'x' * 50000
+            self.assertRaises(ValueError, os.environ.__delitem__, key)
+        else:
+            # "=" is not allowed in a variable name
+            key = 'key='
+            self.assertRaises(OSError, os.environ.__delitem__, key)
+
 class WalkTests(unittest.TestCase):
     """Tests for os.walk()."""
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #13436: Fix a bogus error message when an AST object was passed
+  an invalid integer value.
+
 - Issue #13411: memoryview objects are now hashable when the underlying
   object is hashable.
 
@@ -387,6 +390,11 @@
 Library
 -------
 
+- Issue #13458: Fix a memory leak in the ssl module when decoding a
+  certificate with a subjectAltName.  Patch by Robert Xiao.
+
+- Issue #13415: os.unsetenv() doesn't ignore errors anymore.
+
 - Issue #13245: sched.scheduler class constructor's timefunc and
   delayfunct parameters are now optional.
   scheduler.enter and scheduler.enterabs methods gained a new kwargs parameter.
diff --git a/Modules/_ssl.c b/Modules/_ssl.c
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -707,6 +707,7 @@
             }
             Py_DECREF(t);
         }
+        sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free);
     }
     BIO_free(biobuf);
     if (peer_alt_names != Py_None) {
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7749,61 +7749,28 @@
 static PyObject *
 posix_putenv(PyObject *self, PyObject *args)
 {
+    PyObject *newstr = NULL;
 #ifdef MS_WINDOWS
-    PyObject *s1, *s2;
+    PyObject *os1, *os2;
     wchar_t *newenv;
-#else
-    PyObject *os1, *os2;
-    char *s1, *s2;
-    char *newenv;
-    size_t len;
-#endif
-    PyObject *newstr = NULL;
-
-#ifdef MS_WINDOWS
+
     if (!PyArg_ParseTuple(args,
                           "UU:putenv",
-                          &s1, &s2))
-        return NULL;
-#else
-    if (!PyArg_ParseTuple(args,
-                          "O&O&:putenv",
-                          PyUnicode_FSConverter, &os1,
-                          PyUnicode_FSConverter, &os2))
-        return NULL;
-    s1 = PyBytes_AsString(os1);
-    s2 = PyBytes_AsString(os2);
-#endif
-
-#if defined(PYOS_OS2)
-    if (stricmp(s1, "BEGINLIBPATH") == 0) {
-        APIRET rc;
-
-        rc = DosSetExtLIBPATH(s2, BEGIN_LIBPATH);
-        if (rc != NO_ERROR) {
-            os2_error(rc);
-            goto error;
-        }
-
-    } else if (stricmp(s1, "ENDLIBPATH") == 0) {
-        APIRET rc;
-
-        rc = DosSetExtLIBPATH(s2, END_LIBPATH);
-        if (rc != NO_ERROR) {
-            os2_error(rc);
-            goto error;
-        }
-    } else {
-#endif
-    /* XXX This can leak memory -- not easy to fix :-( */
-    /* len includes space for a trailing \0; the size arg to
-       PyBytes_FromStringAndSize does not count that */
-#ifdef MS_WINDOWS
-    newstr = PyUnicode_FromFormat("%U=%U", s1, s2);
+                          &os1, &os2))
+        return NULL;
+
+    newstr = PyUnicode_FromFormat("%U=%U", os1, os2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
+    if (_MAX_ENV < PyUnicode_GET_LENGTH(newstr)) {
+        PyErr_Format(PyExc_ValueError,
+                     "the environment variable is longer than %u characters",
+                     _MAX_ENV);
+        goto error;
+    }
+
     newenv = PyUnicode_AsUnicode(newstr);
     if (newenv == NULL)
         goto error;
@@ -7812,15 +7779,25 @@
         goto error;
     }
 #else
-    len = PyBytes_GET_SIZE(os1) + PyBytes_GET_SIZE(os2) + 2;
-    newstr = PyBytes_FromStringAndSize(NULL, (int)len - 1);
+    PyObject *os1, *os2;
+    char *s1, *s2;
+    char *newenv;
+
+    if (!PyArg_ParseTuple(args,
+                          "O&O&:putenv",
+                          PyUnicode_FSConverter, &os1,
+                          PyUnicode_FSConverter, &os2))
+        return NULL;
+    s1 = PyBytes_AsString(os1);
+    s2 = PyBytes_AsString(os2);
+
+    newstr = PyBytes_FromFormat("%s=%s", s1, s2);
     if (newstr == NULL) {
         PyErr_NoMemory();
         goto error;
     }
 
     newenv = PyBytes_AS_STRING(newstr);
-    PyOS_snprintf(newenv, len, "%s=%s", s1, s2);
     if (putenv(newenv)) {
         posix_error();
         goto error;
@@ -7831,13 +7808,7 @@
      * this will cause previous value to be collected.  This has to
      * happen after the real putenv() call because the old value
      * was still accessible until then. */
-    if (PyDict_SetItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0),
-#else
-                       os1,
-#endif
-                       newstr)) {
+    if (PyDict_SetItem(posix_putenv_garbage, os1, newstr)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
@@ -7845,10 +7816,6 @@
         Py_DECREF(newstr);
     }
 
-#if defined(PYOS_OS2)
-    }
-#endif
-
 #ifndef MS_WINDOWS
     Py_DECREF(os1);
     Py_DECREF(os2);
@@ -7873,42 +7840,27 @@
 static PyObject *
 posix_unsetenv(PyObject *self, PyObject *args)
 {
-#ifdef MS_WINDOWS
-    char *s1;
-
-    if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
-        return NULL;
-#else
-    PyObject *os1;
-    char *s1;
+    PyObject *name;
+    int err;
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
-                          PyUnicode_FSConverter, &os1))
-        return NULL;
-    s1 = PyBytes_AsString(os1);
-#endif
-
-    unsetenv(s1);
+                          PyUnicode_FSConverter, &name))
+        return NULL;
+
+    err = unsetenv(PyBytes_AS_STRING(name));
+    if (err)
+        return posix_error();
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
      * happen after the real unsetenv() call because the
      * old value was still accessible until then.
      */
-    if (PyDict_DelItem(posix_putenv_garbage,
-#ifdef MS_WINDOWS
-                       PyTuple_GET_ITEM(args, 0)
-#else
-                       os1
-#endif
-                       )) {
+    if (PyDict_DelItem(posix_putenv_garbage, name)) {
         /* really not much we can do; just leak */
         PyErr_Clear();
     }
-
-#ifndef MS_WINDOWS
-    Py_DECREF(os1);
-#endif
+    Py_DECREF(name);
     Py_RETURN_NONE;
 }
 #endif /* unsetenv */
diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -844,11 +844,7 @@
 {
     int i;
     if (!PyLong_Check(obj)) {
-        PyObject *s = PyObject_Repr(obj);
-        if (s == NULL) return 1;
-        PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s",
-                     PyBytes_AS_STRING(s));
-        Py_DECREF(s);
+        PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj);
         return 1;
     }
 
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -690,11 +690,7 @@
 {
     int i;
     if (!PyLong_Check(obj)) {
-        PyObject *s = PyObject_Repr(obj);
-        if (s == NULL) return 1;
-        PyErr_Format(PyExc_ValueError, "invalid integer value: %.400s",
-                     PyBytes_AS_STRING(s));
-        Py_DECREF(s);
+        PyErr_Format(PyExc_ValueError, "invalid integer value: %R", obj);
         return 1;
     }
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Wed Nov 23 05:30:25 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 23 Nov 2011 05:30:25 +0100
Subject: [Python-checkins] Daily reference leaks (1b8db8aa5589): sum=3
Message-ID: <E1RT4TZ-00063I-A6@ap.vmr.nerim.net>

results for 1b8db8aa5589 on branch "default"
--------------------------------------------

test_os leaked [1, 1, 1] references, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogPUf6xK', '-x']

From python-checkins at python.org  Wed Nov 23 06:17:12 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 06:17:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_plug_refleak?=
Message-ID: <E1RT5Cq-0004Hs-Em@dinsdale.python.org>

http://hg.python.org/cpython/rev/714b76882eb8
changeset:   73716:714b76882eb8
branch:      3.2
parent:      73711:80d491aaeed2
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 23:12:49 2011 -0600
summary:
  plug refleak

files:
  Modules/posixmodule.c |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -6193,8 +6193,10 @@
     s1 = PyBytes_AsString(os1);
 
     err = unsetenv(s1);
-    if (err)
+    if (err) {
+        Py_DECREF(s1);
         return posix_error();
+    }
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 06:17:13 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 06:17:13 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_merge_3=2E2?=
Message-ID: <E1RT5Cr-0004I2-7O@dinsdale.python.org>

http://hg.python.org/cpython/rev/f775e6d08410
changeset:   73717:f775e6d08410
parent:      73715:1b8db8aa5589
parent:      73716:714b76882eb8
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 23:14:47 2011 -0600
summary:
  merge 3.2

files:
  Modules/posixmodule.c |  8 ++++++--
  1 files changed, 6 insertions(+), 2 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7844,12 +7844,16 @@
     int err;
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
+
                           PyUnicode_FSConverter, &name))
         return NULL;
 
+
     err = unsetenv(PyBytes_AS_STRING(name));
-    if (err)
-        return posix_error();
+    if (err) {
+        Py_DECREF(name);
+        return posix_error();
+    }
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 06:17:14 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 06:17:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_plug_refleak?=
Message-ID: <E1RT5Cs-0004I6-0D@dinsdale.python.org>

http://hg.python.org/cpython/rev/2060f5e03683
changeset:   73718:2060f5e03683
branch:      2.7
parent:      73713:61a5d44020cd
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 23:12:49 2011 -0600
summary:
  plug refleak

files:
  Modules/posixmodule.c |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7050,8 +7050,10 @@
         return NULL;
 
     err = unsetenv(s1);
-    if (err)
+    if (err) {
+        Py_DECREF(s1);
         return posix_error();
+    }
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 06:56:36 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 06:56:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_no_python_objec?=
	=?utf8?q?ts_to_manage_here?=
Message-ID: <E1RT5oy-0008Fg-Oa@dinsdale.python.org>

http://hg.python.org/cpython/rev/123c959203ae
changeset:   73719:123c959203ae
branch:      2.7
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 23:56:06 2011 -0600
summary:
  no python objects to manage here

files:
  Modules/posixmodule.c |  4 +---
  1 files changed, 1 insertions(+), 3 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7050,10 +7050,8 @@
         return NULL;
 
     err = unsetenv(s1);
-    if (err) {
-        Py_DECREF(s1);
+    if (err)
         return posix_error();
-    }
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 06:58:21 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 06:58:21 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_decref_correct_?=
	=?utf8?q?object?=
Message-ID: <E1RT5qf-0008N2-5u@dinsdale.python.org>

http://hg.python.org/cpython/rev/78f7c232f88b
changeset:   73720:78f7c232f88b
branch:      3.2
parent:      73716:714b76882eb8
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 23:57:23 2011 -0600
summary:
  decref correct object

files:
  Modules/posixmodule.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -6194,7 +6194,7 @@
 
     err = unsetenv(s1);
     if (err) {
-        Py_DECREF(s1);
+        Py_DECREF(os1);
         return posix_error();
     }
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 06:58:21 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 06:58:21 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?b?OiBtZXJnZSAzLjIgKG51bGwp?=
Message-ID: <E1RT5qf-0008N6-U7@dinsdale.python.org>

http://hg.python.org/cpython/rev/958a4678377a
changeset:   73721:958a4678377a
parent:      73717:f775e6d08410
parent:      73720:78f7c232f88b
user:        Benjamin Peterson <benjamin at python.org>
date:        Tue Nov 22 23:57:50 2011 -0600
summary:
  merge 3.2 (null)

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 09:56:14 2011
From: python-checkins at python.org (vinay.sajip)
Date: Wed, 23 Nov 2011 09:56:14 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Closes_=2313459?=
 =?utf8?q?=3A_Clarified_documentation_on_Logger=2Epropagate=2E_Thanks_to_M?=
 =?utf8?q?ike?=
Message-ID: <E1RT8co-00055o-BF@dinsdale.python.org>

http://hg.python.org/cpython/rev/2c173769a957
changeset:   73722:2c173769a957
branch:      2.7
parent:      73719:123c959203ae
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Wed Nov 23 08:51:35 2011 +0000
summary:
  Closes #13459: Clarified documentation on Logger.propagate. Thanks to Mike Fogel for the patch.

files:
  Doc/library/logging.rst |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -59,9 +59,15 @@
 
 .. attribute:: Logger.propagate
 
-   If this evaluates to false, logging messages are not passed by this logger or by
-   its child loggers to the handlers of higher level (ancestor) loggers. The
-   constructor sets this attribute to 1.
+   If this evaluates to true, logging messages are passed by this logger and by
+   its child loggers to the handlers of higher level (ancestor) loggers.
+   Messages are passed directly to the ancestor loggers' handlers - neither the
+   level nor filters of the ancestor loggers in question are considered.
+
+   If this evaluates to false, logging messages are not passed to the handlers
+   of ancestor loggers.
+
+   The constructor sets this attribute to 1.
 
 
 .. method:: Logger.setLevel(lvl)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 09:56:15 2011
From: python-checkins at python.org (vinay.sajip)
Date: Wed, 23 Nov 2011 09:56:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_=2313459?=
 =?utf8?q?=3A_Clarified_documentation_on_Logger=2Epropagate=2E_Thanks_to_M?=
 =?utf8?q?ike?=
Message-ID: <E1RT8cp-00056K-2f@dinsdale.python.org>

http://hg.python.org/cpython/rev/a9f5639e18a1
changeset:   73723:a9f5639e18a1
branch:      3.2
parent:      73720:78f7c232f88b
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Wed Nov 23 08:54:22 2011 +0000
summary:
  Closes #13459: Clarified documentation on Logger.propagate. Thanks to Mike Fogel for the patch.

files:
  Doc/library/logging.rst |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -57,9 +57,15 @@
 
 .. attribute:: Logger.propagate
 
-   If this evaluates to false, logging messages are not passed by this logger or by
-   its child loggers to the handlers of higher level (ancestor) loggers. The
-   constructor sets this attribute to 1.
+   If this evaluates to true, logging messages are passed by this logger and by
+   its child loggers to the handlers of higher level (ancestor) loggers.
+   Messages are passed directly to the ancestor loggers' handlers - neither the
+   level nor filters of the ancestor loggers in question are considered.
+
+   If this evaluates to false, logging messages are not passed to the handlers
+   of ancestor loggers.
+
+   The constructor sets this attribute to 1.
 
 
 .. method:: Logger.setLevel(lvl)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 09:56:15 2011
From: python-checkins at python.org (vinay.sajip)
Date: Wed, 23 Nov 2011 09:56:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Closes_=2313459=3A_Merged_fix_from_3=2E2=2E?=
Message-ID: <E1RT8cp-00056O-UN@dinsdale.python.org>

http://hg.python.org/cpython/rev/cc6693fdf6d5
changeset:   73724:cc6693fdf6d5
parent:      73721:958a4678377a
parent:      73723:a9f5639e18a1
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Wed Nov 23 08:55:59 2011 +0000
summary:
  Closes #13459: Merged fix from 3.2.

files:
  Doc/library/logging.rst |  12 +++++++++---
  1 files changed, 9 insertions(+), 3 deletions(-)


diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -57,9 +57,15 @@
 
 .. attribute:: Logger.propagate
 
-   If this evaluates to false, logging messages are not passed by this logger or by
-   its child loggers to the handlers of higher level (ancestor) loggers. The
-   constructor sets this attribute to 1.
+   If this evaluates to true, logging messages are passed by this logger and by
+   its child loggers to the handlers of higher level (ancestor) loggers.
+   Messages are passed directly to the ancestor loggers' handlers - neither the
+   level nor filters of the ancestor loggers in question are considered.
+
+   If this evaluates to false, logging messages are not passed to the handlers
+   of ancestor loggers.
+
+   The constructor sets this attribute to 1.
 
 
 .. method:: Logger.setLevel(lvl)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 15:29:38 2011
From: python-checkins at python.org (vinay.sajip)
Date: Wed, 23 Nov 2011 15:29:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=282=2E7=29=3A_Added_a_configu?=
 =?utf8?q?ration_dictionary_example_to_the_logging_cookbook=2E?=
Message-ID: <E1RTDpS-0000pO-H5@dinsdale.python.org>

http://hg.python.org/cpython/rev/d1bdafa161e7
changeset:   73725:d1bdafa161e7
branch:      2.7
parent:      73722:2c173769a957
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Wed Nov 23 14:27:11 2011 +0000
summary:
  Added a configuration dictionary example to the logging cookbook.

files:
  Doc/howto/logging-cookbook.rst |  62 ++++++++++++++++++++++
  1 files changed, 62 insertions(+), 0 deletions(-)


diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
--- a/Doc/howto/logging-cookbook.rst
+++ b/Doc/howto/logging-cookbook.rst
@@ -683,3 +683,65 @@
 Obviously this example sets the log length much too small as an extreme
 example.  You would want to set *maxBytes* to an appropriate value.
 
+An example dictionary-based configuration
+-----------------------------------------
+
+Below is an example of a logging configuration dictionary - it's taken from
+the `documentation on the Django project <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_.
+This dictionary is passed to :func:`~logging.config.dictConfig` to put the configuration into effect::
+
+    LOGGING = {
+        'version': 1,
+        'disable_existing_loggers': True,
+        'formatters': {
+            'verbose': {
+                'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
+            },
+            'simple': {
+                'format': '%(levelname)s %(message)s'
+            },
+        },
+        'filters': {
+            'special': {
+                '()': 'project.logging.SpecialFilter',
+                'foo': 'bar',
+            }
+        },
+        'handlers': {
+            'null': {
+                'level':'DEBUG',
+                'class':'django.utils.log.NullHandler',
+            },
+            'console':{
+                'level':'DEBUG',
+                'class':'logging.StreamHandler',
+                'formatter': 'simple'
+            },
+            'mail_admins': {
+                'level': 'ERROR',
+                'class': 'django.utils.log.AdminEmailHandler',
+                'filters': ['special']
+            }
+        },
+        'loggers': {
+            'django': {
+                'handlers':['null'],
+                'propagate': True,
+                'level':'INFO',
+            },
+            'django.request': {
+                'handlers': ['mail_admins'],
+                'level': 'ERROR',
+                'propagate': False,
+            },
+            'myproject.custom': {
+                'handlers': ['console', 'mail_admins'],
+                'level': 'INFO',
+                'filters': ['special']
+            }
+        }
+    }
+
+For more information about this configuration, you can see the `relevant
+section <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_
+of the Django documentation.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 15:29:39 2011
From: python-checkins at python.org (vinay.sajip)
Date: Wed, 23 Nov 2011 15:29:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Added_a_configu?=
 =?utf8?q?ration_dictionary_example_to_the_logging_cookbook=2E?=
Message-ID: <E1RTDpT-0000pY-9B@dinsdale.python.org>

http://hg.python.org/cpython/rev/71cb7ca03bc5
changeset:   73726:71cb7ca03bc5
branch:      3.2
parent:      73723:a9f5639e18a1
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Wed Nov 23 14:27:54 2011 +0000
summary:
  Added a configuration dictionary example to the logging cookbook.

files:
  Doc/howto/logging-cookbook.rst |  64 ++++++++++++++++++++++
  1 files changed, 64 insertions(+), 0 deletions(-)


diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
--- a/Doc/howto/logging-cookbook.rst
+++ b/Doc/howto/logging-cookbook.rst
@@ -1038,3 +1038,67 @@
    :ref:`A basic logging tutorial <logging-basic-tutorial>`
 
    :ref:`A more advanced logging tutorial <logging-advanced-tutorial>`
+
+
+An example dictionary-based configuration
+-----------------------------------------
+
+Below is an example of a logging configuration dictionary - it's taken from
+the `documentation on the Django project <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_.
+This dictionary is passed to :func:`~logging.config.dictConfig` to put the configuration into effect::
+
+    LOGGING = {
+        'version': 1,
+        'disable_existing_loggers': True,
+        'formatters': {
+            'verbose': {
+                'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
+            },
+            'simple': {
+                'format': '%(levelname)s %(message)s'
+            },
+        },
+        'filters': {
+            'special': {
+                '()': 'project.logging.SpecialFilter',
+                'foo': 'bar',
+            }
+        },
+        'handlers': {
+            'null': {
+                'level':'DEBUG',
+                'class':'django.utils.log.NullHandler',
+            },
+            'console':{
+                'level':'DEBUG',
+                'class':'logging.StreamHandler',
+                'formatter': 'simple'
+            },
+            'mail_admins': {
+                'level': 'ERROR',
+                'class': 'django.utils.log.AdminEmailHandler',
+                'filters': ['special']
+            }
+        },
+        'loggers': {
+            'django': {
+                'handlers':['null'],
+                'propagate': True,
+                'level':'INFO',
+            },
+            'django.request': {
+                'handlers': ['mail_admins'],
+                'level': 'ERROR',
+                'propagate': False,
+            },
+            'myproject.custom': {
+                'handlers': ['console', 'mail_admins'],
+                'level': 'INFO',
+                'filters': ['special']
+            }
+        }
+    }
+
+For more information about this configuration, you can see the `relevant
+section <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_
+of the Django documentation.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 15:29:40 2011
From: python-checkins at python.org (vinay.sajip)
Date: Wed, 23 Nov 2011 15:29:40 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merged_documentation_addition_from_3=2E2=2E?=
Message-ID: <E1RTDpU-0000pc-0u@dinsdale.python.org>

http://hg.python.org/cpython/rev/6456f8d2b2e9
changeset:   73727:6456f8d2b2e9
parent:      73724:cc6693fdf6d5
parent:      73726:71cb7ca03bc5
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Wed Nov 23 14:29:01 2011 +0000
summary:
  Merged documentation addition from 3.2.

files:
  Doc/howto/logging-cookbook.rst |  64 ++++++++++++++++++++++
  1 files changed, 64 insertions(+), 0 deletions(-)


diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
--- a/Doc/howto/logging-cookbook.rst
+++ b/Doc/howto/logging-cookbook.rst
@@ -1038,3 +1038,67 @@
    :ref:`A basic logging tutorial <logging-basic-tutorial>`
 
    :ref:`A more advanced logging tutorial <logging-advanced-tutorial>`
+
+
+An example dictionary-based configuration
+-----------------------------------------
+
+Below is an example of a logging configuration dictionary - it's taken from
+the `documentation on the Django project <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_.
+This dictionary is passed to :func:`~logging.config.dictConfig` to put the configuration into effect::
+
+    LOGGING = {
+        'version': 1,
+        'disable_existing_loggers': True,
+        'formatters': {
+            'verbose': {
+                'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
+            },
+            'simple': {
+                'format': '%(levelname)s %(message)s'
+            },
+        },
+        'filters': {
+            'special': {
+                '()': 'project.logging.SpecialFilter',
+                'foo': 'bar',
+            }
+        },
+        'handlers': {
+            'null': {
+                'level':'DEBUG',
+                'class':'django.utils.log.NullHandler',
+            },
+            'console':{
+                'level':'DEBUG',
+                'class':'logging.StreamHandler',
+                'formatter': 'simple'
+            },
+            'mail_admins': {
+                'level': 'ERROR',
+                'class': 'django.utils.log.AdminEmailHandler',
+                'filters': ['special']
+            }
+        },
+        'loggers': {
+            'django': {
+                'handlers':['null'],
+                'propagate': True,
+                'level':'INFO',
+            },
+            'django.request': {
+                'handlers': ['mail_admins'],
+                'level': 'ERROR',
+                'propagate': False,
+            },
+            'myproject.custom': {
+                'handlers': ['console', 'mail_admins'],
+                'level': 'INFO',
+                'filters': ['special']
+            }
+        }
+    }
+
+For more information about this configuration, you can see the `relevant
+section <https://docs.djangoproject.com/en/1.3/topics/logging/#configuring-logging>`_
+of the Django documentation.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 23 16:45:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 23 Nov 2011 16:45:59 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
 =?utf8?q?n3=29=3A_Merge_compat/=5Fbackport_cleanup_from_default=2E?=
Message-ID: <E1RTF1L-0000aS-Rc@dinsdale.python.org>

http://hg.python.org/distutils2/rev/7e7f23d10484
changeset:   1263:7e7f23d10484
branch:      python3
parent:      1261:bf1b6478e134
parent:      1262:a382d149a0f0
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Nov 23 13:58:51 2011 +0100
summary:
  Merge compat/_backport cleanup from default.

For some reason, the diff for distutils/_backport/misc.py makes it seem
like it?s a new file instead of showing the changes I did during the
merge.

files:
  distutils2/_backport/__init__.py    |   6 +-
  distutils2/compat.py                |  87 +++-------------
  distutils2/command/build_py.py      |   3 +-
  distutils2/command/build_scripts.py |   2 +-
  distutils2/command/install_lib.py   |   2 +-
  distutils2/compat.py                |  38 +-------
  distutils2/dist.py                  |   2 +-
  distutils2/markers.py               |   6 +-
  distutils2/pypi/simple.py           |   4 +-
  distutils2/run.py                   |   2 +-
  distutils2/util.py                  |   2 +-
  11 files changed, 35 insertions(+), 119 deletions(-)


diff --git a/distutils2/_backport/__init__.py b/distutils2/_backport/__init__.py
--- a/distutils2/_backport/__init__.py
+++ b/distutils2/_backport/__init__.py
@@ -1,4 +1,6 @@
-"""Modules copied from Python 3 standard libraries.
+"""Modules copied from Python 3 standard libraries, for internal use only.
 
-Individual classes and objects like the any function are in compat.
+Individual classes and functions are found in d2._backport.misc.  Intended
+usage is to always import things missing from 3.1 from that module: the
+built-in/stdlib objects will be used if found.
 """
diff --git a/distutils2/compat.py b/distutils2/_backport/misc.py
copy from distutils2/compat.py
copy to distutils2/_backport/misc.py
--- a/distutils2/compat.py
+++ b/distutils2/_backport/misc.py
@@ -1,73 +1,17 @@
-"""Compatibility helpers.
+"""Backports for individual classes and functions."""
 
-This module provides individual classes or objects backported from
-Python 3.2, for internal use only.  Whole modules are in _backport.
-"""
+import os
+import sys
 
-import sys
-from distutils2 import logger
+__all__ = ['cache_from_source', 'callable', 'fsencode']
 
 
-# XXX Having two classes with the same name is not a good thing.
-# XXX 2to3-related code should move from util to this module
-
 try:
-    from distutils2.util import Mixin2to3 as _Mixin2to3
-    _CONVERT = True
-    _KLASS = _Mixin2to3
+    from imp import cache_from_source
 except ImportError:
-    _CONVERT = False
-    _KLASS = object
-
-__all__ = ['Mixin2to3']
-
-
-class Mixin2to3(_KLASS):
-    """ The base class which can be used for refactoring. When run under
-    Python 3.0, the run_2to3 method provided by Mixin2to3 is overridden.
-    When run on Python 2.x, it merely creates a class which overrides run_2to3,
-    yet does nothing in particular with it.
-    """
-    if _CONVERT:
-
-        def _run_2to3(self, files, doctests=[], fixers=[]):
-            """ Takes a list of files and doctests, and performs conversion
-            on those.
-              - First, the files which contain the code(`files`) are converted.
-              - Second, the doctests in `files` are converted.
-              - Thirdly, the doctests in `doctests` are converted.
-            """
-            if fixers:
-                self.fixer_names = fixers
-
-            logger.info('converting Python code')
-            _KLASS.run_2to3(self, files)
-
-            logger.info('converting doctests in Python files')
-            _KLASS.run_2to3(self, files, doctests_only=True)
-
-            if doctests != []:
-                logger.info('converting doctest in text files')
-                _KLASS.run_2to3(self, doctests, doctests_only=True)
-    else:
-        # If run on Python 2.x, there is nothing to do.
-
-        def _run_2to3(self, files, doctests=[], fixers=[]):
-            pass
-
-
-# The rest of this file does not exist in packaging
-# functions are sorted alphabetically and are not included in __all__
-
-
-def fsencode(filename):
-    if isinstance(filename, bytes):
-        return filename
-    elif isinstance(filename, str):
-        return filename.encode(sys.getfilesystemencoding())
-    else:
-        raise TypeError("expect bytes or str, not %s" %
-                        type(filename).__name__)
+    def cache_from_source(py_file, debug=__debug__):
+        ext = debug and 'c' or 'o'
+        return py_file + ext
 
 
 try:
@@ -80,8 +24,13 @@
 
 
 try:
-    from imp import cache_from_source
-except ImportError:
-    def cache_from_source(py_file, debug=__debug__):
-        ext = debug and 'c' or 'o'
-        return py_file + ext
+    fsencode = os.fsencode
+except AttributeError:
+    def fsencode(filename):
+        if isinstance(filename, bytes):
+            return filename
+        elif isinstance(filename, str):
+            return filename.encode(sys.getfilesystemencoding())
+        else:
+            raise TypeError("expect bytes or str, not %s" %
+                            type(filename).__name__)
diff --git a/distutils2/command/build_py.py b/distutils2/command/build_py.py
--- a/distutils2/command/build_py.py
+++ b/distutils2/command/build_py.py
@@ -5,9 +5,10 @@
 
 from distutils2 import logger
 from distutils2.util import convert_path
-from distutils2.compat import Mixin2to3, cache_from_source
+from distutils2.compat import Mixin2to3
 from distutils2.errors import PackagingOptionError, PackagingFileError
 from distutils2.command.cmd import Command
+from distutils2._backport.misc import cache_from_source
 
 # marking public APIs
 __all__ = ['build_py']
diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py
--- a/distutils2/command/build_scripts.py
+++ b/distutils2/command/build_scripts.py
@@ -8,8 +8,8 @@
 from distutils2.util import convert_path, newer
 from distutils2 import logger
 from distutils2.compat import Mixin2to3
-from distutils2.compat import fsencode
 from distutils2._backport import sysconfig
+from distutils2._backport.misc import fsencode
 
 
 # check if Python is called on the first line with this expression
diff --git a/distutils2/command/install_lib.py b/distutils2/command/install_lib.py
--- a/distutils2/command/install_lib.py
+++ b/distutils2/command/install_lib.py
@@ -3,9 +3,9 @@
 import os
 
 from distutils2 import logger
-from distutils2.compat import cache_from_source
 from distutils2.command.cmd import Command
 from distutils2.errors import PackagingOptionError
+from distutils2._backport.misc import cache_from_source
 
 
 # Extension for Python source files.
diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -1,10 +1,5 @@
-"""Compatibility helpers.
+"""Support for build-time 2to3 conversion."""
 
-This module provides individual classes or objects backported from
-Python 3.2, for internal use only.  Whole modules are in _backport.
-"""
-
-import sys
 from distutils2 import logger
 
 
@@ -54,34 +49,3 @@
 
         def _run_2to3(self, files, doctests=[], fixers=[]):
             pass
-
-
-# The rest of this file does not exist in packaging
-# functions are sorted alphabetically and are not included in __all__
-
-
-def fsencode(filename):
-    if isinstance(filename, bytes):
-        return filename
-    elif isinstance(filename, str):
-        return filename.encode(sys.getfilesystemencoding())
-    else:
-        raise TypeError("expect bytes or str, not %s" %
-                        type(filename).__name__)
-
-
-try:
-    callable = callable
-except NameError:
-    from collections import Callable
-
-    def callable(obj):
-        return isinstance(obj, Callable)
-
-
-try:
-    from imp import cache_from_source
-except ImportError:
-    def cache_from_source(py_file, debug=__debug__):
-        ext = debug and 'c' or 'o'
-        return py_file + ext
diff --git a/distutils2/dist.py b/distutils2/dist.py
--- a/distutils2/dist.py
+++ b/distutils2/dist.py
@@ -5,7 +5,6 @@
 
 from distutils2 import logger
 from distutils2.util import strtobool, resolve_name
-from distutils2.compat import callable
 from distutils2.config import Config
 from distutils2.errors import (PackagingOptionError, PackagingArgError,
                                PackagingModuleError, PackagingClassError)
@@ -13,6 +12,7 @@
 from distutils2.command.cmd import Command
 from distutils2.metadata import Metadata
 from distutils2.fancy_getopt import FancyGetopt
+from distutils2._backport.misc import callable
 
 # Regex to define acceptable Packaging command names.  This is not *quite*
 # the same as a Python name -- leading underscores are not allowed.  The fact
diff --git a/distutils2/markers.py b/distutils2/markers.py
--- a/distutils2/markers.py
+++ b/distutils2/markers.py
@@ -3,9 +3,8 @@
 import os
 import sys
 import platform
-
+from io import BytesIO
 from tokenize import tokenize, NAME, OP, STRING, ENDMARKER, ENCODING
-from io import BytesIO
 
 __all__ = ['interpret']
 
@@ -32,7 +31,8 @@
          'os.name': os.name,
          'platform.version': platform.version(),
          'platform.machine': platform.machine(),
-         'platform.python_implementation': platform.python_implementation()}
+         'platform.python_implementation': platform.python_implementation(),
+        }
 
 
 class _Operation:
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -18,9 +18,9 @@
 from fnmatch import translate
 from functools import wraps
 from distutils2 import logger
+from distutils2 import __version__ as distutils2_version
+from distutils2.version import get_version_predicate
 from distutils2.metadata import Metadata
-from distutils2.version import get_version_predicate
-from distutils2 import __version__ as distutils2_version
 from distutils2.pypi.base import BaseClient
 from distutils2.pypi.dist import (ReleasesList, EXTENSIONS,
                                   get_infos_from_url, MD5_HASH)
diff --git a/distutils2/run.py b/distutils2/run.py
--- a/distutils2/run.py
+++ b/distutils2/run.py
@@ -9,7 +9,6 @@
 from distutils2 import logger
 from distutils2.dist import Distribution
 from distutils2.util import _is_archive_file, generate_setup_py
-from distutils2.compat import callable
 from distutils2.command import get_command_class, STANDARD_COMMANDS
 from distutils2.install import install, install_local_project, remove
 from distutils2.database import get_distribution, get_distributions
@@ -18,6 +17,7 @@
 from distutils2.errors import (PackagingArgError, PackagingError,
                                PackagingModuleError, PackagingClassError,
                                CCompilerError)
+from distutils2._backport.misc import callable
 
 
 command_re = re.compile(r'^[a-zA-Z]([a-zA-Z0-9_]*)$')
diff --git a/distutils2/util.py b/distutils2/util.py
--- a/distutils2/util.py
+++ b/distutils2/util.py
@@ -16,11 +16,11 @@
 from configparser import RawConfigParser
 
 from distutils2 import logger
-from distutils2.compat import cache_from_source
 from distutils2.errors import (PackagingPlatformError, PackagingFileError,
                                PackagingExecError, InstallationException,
                                PackagingInternalError)
 from distutils2._backport import shutil, sysconfig
+from distutils2._backport.misc import cache_from_source
 
 __all__ = [
     # file dependencies

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Nov 23 16:45:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 23 Nov 2011 16:45:59 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Better_separate_compat_a?=
	=?utf8?q?nd_=5Fbackport=2E?=
Message-ID: <E1RTF1L-0000aI-QE@dinsdale.python.org>

http://hg.python.org/distutils2/rev/a382d149a0f0
changeset:   1262:a382d149a0f0
parent:      1259:c898ebb74f28
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Nov 23 13:31:27 2011 +0100
summary:
  Better separate compat and _backport.

This moves individual backported functions from compat into a new
_backport.misc module.  compat regains its original purpose of
containing 2to3 support code (i.e. compat for projects using distutils2,
not internal compat for d2 code) and all backports now live under the
private _backport subnamespace.

Also fix the definition of ?any? to bind it to the builtin in 2.5+
(fsencode and detect_encoding not changed, they never exist in
2.x).

files:
  distutils2/_backport/__init__.py          |    6 +-
  distutils2/compat.py                      |   63 +---
  distutils2/_backport/tests/test_shutil.py |    2 +-
  distutils2/command/build_scripts.py       |    2 +-
  distutils2/compat.py                      |  143 +---------
  distutils2/create.py                      |    6 +-
  distutils2/markers.py                     |    2 +-
  distutils2/pypi/simple.py                 |    6 +-
  distutils2/tests/pypi_server.py           |    2 +-
  distutils2/tests/test_markers.py          |    2 +-
  10 files changed, 20 insertions(+), 214 deletions(-)


diff --git a/distutils2/_backport/__init__.py b/distutils2/_backport/__init__.py
--- a/distutils2/_backport/__init__.py
+++ b/distutils2/_backport/__init__.py
@@ -1,4 +1,6 @@
-"""Modules copied from Python 3 standard libraries.
+"""Modules copied from Python 3 standard libraries, for internal use only.
 
-Individual classes and objects like the any function are in compat.
+Individual classes and functions are found in d2._backport.misc.  Intended
+usage is to always import things missing from 2.4 from that module: the
+built-in/stdlib objects will be used if found.
 """
diff --git a/distutils2/compat.py b/distutils2/_backport/misc.py
copy from distutils2/compat.py
copy to distutils2/_backport/misc.py
--- a/distutils2/compat.py
+++ b/distutils2/_backport/misc.py
@@ -1,69 +1,16 @@
-"""Compatibility helpers.
-
-This module provides individual classes or objects backported from
-Python 3.2, for internal use only.  Whole modules are in _backport.
-"""
+"""Backports for individual classes and functions."""
 
 import os
 import re
 import sys
 import codecs
-from distutils2 import logger
 
+__all__ = ['any', 'detect_encoding', 'fsencode', 'python_implementation',
+           'wraps']
 
-# XXX Having two classes with the same name is not a good thing.
-# XXX 2to3-related code should move from util to this module
 
 try:
-    from distutils2.util import Mixin2to3 as _Mixin2to3
-    _CONVERT = True
-    _KLASS = _Mixin2to3
-except ImportError:
-    _CONVERT = False
-    _KLASS = object
-
-__all__ = ['Mixin2to3']
-
-
-class Mixin2to3(_KLASS):
-    """ The base class which can be used for refactoring. When run under
-    Python 3.0, the run_2to3 method provided by Mixin2to3 is overridden.
-    When run on Python 2.x, it merely creates a class which overrides run_2to3,
-    yet does nothing in particular with it.
-    """
-    if _CONVERT:
-
-        def _run_2to3(self, files, doctests=[], fixers=[]):
-            """ Takes a list of files and doctests, and performs conversion
-            on those.
-              - First, the files which contain the code(`files`) are converted.
-              - Second, the doctests in `files` are converted.
-              - Thirdly, the doctests in `doctests` are converted.
-            """
-            if fixers:
-                self.fixer_names = fixers
-
-            logger.info('converting Python code')
-            _KLASS.run_2to3(self, files)
-
-            logger.info('converting doctests in Python files')
-            _KLASS.run_2to3(self, files, doctests_only=True)
-
-            if doctests != []:
-                logger.info('converting doctest in text files')
-                _KLASS.run_2to3(self, doctests, doctests_only=True)
-    else:
-        # If run on Python 2.x, there is nothing to do.
-
-        def _run_2to3(self, files, doctests=[], fixers=[]):
-            pass
-
-
-# The rest of this file does not exist in packaging
-# functions are sorted alphabetically and are not included in __all__
-
-try:
-    any
+    any = any
 except NameError:
     def any(seq):
         for elem in seq:
@@ -179,10 +126,12 @@
             return func
         return wrapped
 
+
 try:
     from platform import python_implementation
 except ImportError:
     def python_implementation():
+        """Return a string identifying the Python implementation."""
         if 'PyPy' in sys.version:
             return 'PyPy'
         if os.name == 'java':
diff --git a/distutils2/_backport/tests/test_shutil.py b/distutils2/_backport/tests/test_shutil.py
--- a/distutils2/_backport/tests/test_shutil.py
+++ b/distutils2/_backport/tests/test_shutil.py
@@ -6,13 +6,13 @@
 from StringIO import StringIO
 
 from distutils.spawn import find_executable, spawn
-from distutils2.compat import wraps
 from distutils2._backport import shutil, tarfile
 from distutils2._backport.shutil import (
     _make_tarball, _make_zipfile, make_archive, unpack_archive,
     register_archive_format, unregister_archive_format, get_archive_formats,
     register_unpack_format, unregister_unpack_format, get_unpack_formats,
     Error, RegistryError)
+from distutils2._backport.misc import wraps
 
 from distutils2.tests import unittest, support
 from test.test_support import TESTFN
diff --git a/distutils2/command/build_scripts.py b/distutils2/command/build_scripts.py
--- a/distutils2/command/build_scripts.py
+++ b/distutils2/command/build_scripts.py
@@ -7,8 +7,8 @@
 from distutils2.util import convert_path, newer
 from distutils2 import logger
 from distutils2.compat import Mixin2to3
-from distutils2.compat import detect_encoding, fsencode
 from distutils2._backport import sysconfig
+from distutils2._backport.misc import detect_encoding, fsencode
 
 
 # check if Python is called on the first line with this expression
diff --git a/distutils2/compat.py b/distutils2/compat.py
--- a/distutils2/compat.py
+++ b/distutils2/compat.py
@@ -1,13 +1,5 @@
-"""Compatibility helpers.
+"""Support for build-time 2to3 conversion."""
 
-This module provides individual classes or objects backported from
-Python 3.2, for internal use only.  Whole modules are in _backport.
-"""
-
-import os
-import re
-import sys
-import codecs
 from distutils2 import logger
 
 
@@ -57,136 +49,3 @@
 
         def _run_2to3(self, files, doctests=[], fixers=[]):
             pass
-
-
-# The rest of this file does not exist in packaging
-# functions are sorted alphabetically and are not included in __all__
-
-try:
-    any
-except NameError:
-    def any(seq):
-        for elem in seq:
-            if elem:
-                return True
-        return False
-
-
-_cookie_re = re.compile("coding[:=]\s*([-\w.]+)")
-
-
-def _get_normal_name(orig_enc):
-    """Imitates get_normal_name in tokenizer.c."""
-    # Only care about the first 12 characters.
-    enc = orig_enc[:12].lower().replace("_", "-")
-    if enc == "utf-8" or enc.startswith("utf-8-"):
-        return "utf-8"
-    if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or \
-       enc.startswith(("latin-1-", "iso-8859-1-", "iso-latin-1-")):
-        return "iso-8859-1"
-    return orig_enc
-
-
-def detect_encoding(readline):
-    """
-    The detect_encoding() function is used to detect the encoding that should
-    be used to decode a Python source file.  It requires one argment, readline,
-    in the same way as the tokenize() generator.
-
-    It will call readline a maximum of twice, and return the encoding used
-    (as a string) and a list of any lines (left as bytes) it has read in.
-
-    It detects the encoding from the presence of a utf-8 bom or an encoding
-    cookie as specified in pep-0263.  If both a bom and a cookie are present,
-    but disagree, a SyntaxError will be raised.  If the encoding cookie is an
-    invalid charset, raise a SyntaxError.  Note that if a utf-8 bom is found,
-    'utf-8-sig' is returned.
-
-    If no encoding is specified, then the default of 'utf-8' will be returned.
-    """
-    bom_found = False
-    encoding = None
-    default = 'utf-8'
-
-    def read_or_stop():
-        try:
-            return readline()
-        except StopIteration:
-            return ''
-
-    def find_cookie(line):
-        try:
-            line_string = line.decode('ascii')
-        except UnicodeDecodeError:
-            return None
-
-        matches = _cookie_re.findall(line_string)
-        if not matches:
-            return None
-        encoding = _get_normal_name(matches[0])
-        try:
-            codec = codecs.lookup(encoding)
-        except LookupError:
-            # This behaviour mimics the Python interpreter
-            raise SyntaxError("unknown encoding: " + encoding)
-
-        if bom_found:
-            if codec.name != 'utf-8':
-                # This behaviour mimics the Python interpreter
-                raise SyntaxError('encoding problem: utf-8')
-            encoding += '-sig'
-        return encoding
-
-    first = read_or_stop()
-    if first.startswith(codecs.BOM_UTF8):
-        bom_found = True
-        first = first[3:]
-        default = 'utf-8-sig'
-    if not first:
-        return default, []
-
-    encoding = find_cookie(first)
-    if encoding:
-        return encoding, [first]
-
-    second = read_or_stop()
-    if not second:
-        return default, [first]
-
-    encoding = find_cookie(second)
-    if encoding:
-        return encoding, [first, second]
-
-    return default, [first, second]
-
-
-def fsencode(filename):
-    if isinstance(filename, str):
-        return filename
-    elif isinstance(filename, unicode):
-        return filename.encode(sys.getfilesystemencoding())
-    else:
-        raise TypeError("expect bytes or str, not %s" %
-                        type(filename).__name__)
-
-
-try:
-    from functools import wraps
-except ImportError:
-    def wraps(func=None):
-        """No-op replacement for functools.wraps"""
-        def wrapped(func):
-            return func
-        return wrapped
-
-try:
-    from platform import python_implementation
-except ImportError:
-    def python_implementation():
-        if 'PyPy' in sys.version:
-            return 'PyPy'
-        if os.name == 'java':
-            return 'Jython'
-        if sys.version.startswith('IronPython'):
-            return 'IronPython'
-        return 'CPython'
diff --git a/distutils2/create.py b/distutils2/create.py
--- a/distutils2/create.py
+++ b/distutils2/create.py
@@ -31,13 +31,9 @@
 # importing this with an underscore as it should be replaced by the
 # dict form or another structures for all purposes
 from distutils2._trove import all_classifiers as _CLASSIFIERS_LIST
-from distutils2.compat import detect_encoding
 from distutils2.version import is_valid_version
 from distutils2._backport import shutil, sysconfig
-try:
-    any
-except NameError:
-    from distutils2.compat import any
+from distutils2._backport.misc import any, detect_encoding
 try:
     from hashlib import md5
 except ImportError:
diff --git a/distutils2/markers.py b/distutils2/markers.py
--- a/distutils2/markers.py
+++ b/distutils2/markers.py
@@ -6,7 +6,7 @@
 from tokenize import generate_tokens, NAME, OP, STRING, ENDMARKER
 from StringIO import StringIO as BytesIO
 
-from distutils2.compat import python_implementation
+from distutils2._backport.misc import python_implementation
 
 __all__ = ['interpret']
 
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -16,10 +16,9 @@
 
 from fnmatch import translate
 from distutils2 import logger
-from distutils2.compat import wraps
+from distutils2 import __version__ as distutils2_version
+from distutils2.version import get_version_predicate
 from distutils2.metadata import Metadata
-from distutils2.version import get_version_predicate
-from distutils2 import __version__ as distutils2_version
 from distutils2.pypi.base import BaseClient
 from distutils2.pypi.dist import (ReleasesList, EXTENSIONS,
                                   get_infos_from_url, MD5_HASH)
@@ -27,6 +26,7 @@
                                     UnableToDownload, CantParseArchiveName,
                                     ReleaseNotFound, ProjectNotFound)
 from distutils2.pypi.mirrors import get_mirrors
+from distutils2._backport.misc import wraps
 
 __all__ = ['Crawler', 'DEFAULT_SIMPLE_INDEX_URL']
 
diff --git a/distutils2/tests/pypi_server.py b/distutils2/tests/pypi_server.py
--- a/distutils2/tests/pypi_server.py
+++ b/distutils2/tests/pypi_server.py
@@ -38,7 +38,7 @@
 from SimpleXMLRPCServer import SimpleXMLRPCServer
 
 from distutils2.tests import unittest
-from distutils2.compat import wraps
+from distutils2._backport.misc import wraps
 
 
 PYPI_DEFAULT_STATIC_PATH = os.path.join(
diff --git a/distutils2/tests/test_markers.py b/distutils2/tests/test_markers.py
--- a/distutils2/tests/test_markers.py
+++ b/distutils2/tests/test_markers.py
@@ -2,8 +2,8 @@
 import os
 import sys
 import platform
-from distutils2.compat import python_implementation
 from distutils2.markers import interpret
+from distutils2._backport.misc import python_implementation
 
 from distutils2.tests import unittest
 from distutils2.tests.support import LoggingCatcher

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Nov 23 17:45:04 2011
From: python-checkins at python.org (benjamin.peterson)
Date: Wed, 23 Nov 2011 17:45:04 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_and_back_to_the_=22magic=22?=
 =?utf8?q?_formula_=28with_a_comment=29_it_is?=
Message-ID: <E1RTFwW-0006ye-HC@dinsdale.python.org>

http://hg.python.org/cpython/rev/14c979556d43
changeset:   73728:14c979556d43
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Nov 23 10:44:52 2011 -0600
summary:
  and back to the "magic" formula (with a comment) it is

files:
  Objects/unicodeobject.c |  18 +++---------------
  1 files changed, 3 insertions(+), 15 deletions(-)


diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -6164,21 +6164,9 @@
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
     len = PyUnicode_GET_LENGTH(unicode);
-    expandsize = 0;
-    switch (kind) {
-    case PyUnicode_1BYTE_KIND:
-        expandsize = 4;
-        break;
-    case PyUnicode_2BYTE_KIND:
-        expandsize = 6;
-        break;
-    case PyUnicode_4BYTE_KIND:
-        expandsize = 10;
-        break;
-    default:
-        assert(0);
-        break;
-    }
+    /* 4 byte characters can take up 10 bytes, 2 byte characters can take up 6
+       bytes, and 1 byte characters 4. */
+    expandsize = kind * 2 + 2;
 
     if (len > PY_SSIZE_T_MAX / expandsize)
         return PyErr_NoMemory();

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Thu Nov 24 05:32:11 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 24 Nov 2011 05:32:11 +0100
Subject: [Python-checkins] Daily reference leaks (14c979556d43): sum=0
Message-ID: <E1RTQyp-00079E-Mo@ap.vmr.nerim.net>

results for 14c979556d43 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloggelagq', '-x']

From python-checkins at python.org  Thu Nov 24 13:37:33 2011
From: python-checkins at python.org (nick.coghlan)
Date: Thu, 24 Nov 2011 13:37:33 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Update_for_latest_import-sig_d?=
 =?utf8?q?iscussion=3A_fix_proposed_-m_semantics=2C_better?=
Message-ID: <E1RTYYX-00039i-PU@dinsdale.python.org>

http://hg.python.org/peps/rev/d0eea091def7
changeset:   4002:d0eea091def7
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Thu Nov 24 22:37:18 2011 +1000
summary:
  Update for latest import-sig discussion: fix proposed -m semantics, better explanation of why we can change this now when we couldn't in 2.x

files:
  pep-0395.txt |  176 +++++++++++++++++++++++++-------------
  1 files changed, 116 insertions(+), 60 deletions(-)


diff --git a/pep-0395.txt b/pep-0395.txt
--- a/pep-0395.txt
+++ b/pep-0395.txt
@@ -38,8 +38,9 @@
 the implicit namespace package mechanism proposed in PEP 402.
 
 Finally, PEP 328 eliminated implicit relative imports from imported modules.
-This PEP proposes that implicit relative imports from main modules also be
-eliminated.
+This PEP proposes that the de facto implicit relative imports from main
+modules that are provided by the current initialisation behaviour for
+``sys.path[0]`` also be eliminated.
 
 
 What's in a ``__name__``?
@@ -62,11 +63,11 @@
 Traps for the Unwary
 ====================
 
-The overloading of the semantics of ``__name__`` have resulted in several
-traps for the unwary. These traps can be quite annoying in practice, as
-they are highly unobvious and can cause quite confusing behaviour. A lot of
-the time, you won't even notice them, which just makes them all the more
-surprising when they do come up.
+The overloading of the semantics of ``__name__``, along with some historically
+associated behaviour in the initialisation of ``sys.path[0]``, has resulted in
+several traps for the unwary. These traps can be quite annoying in practice,
+as they are highly unobvious (especially to beginners) and can cause quite
+confusing behaviour.
 
 
 Why are my imports broken?
@@ -96,47 +97,51 @@
 The impact of this can be seen relatively frequently if you follow the 
 "python" and "import" tags on Stack Overflow. When I had the time to follow
 it myself, I regularly encountered people struggling to understand the
-behaviour of straightforward package layouts like the following::
+behaviour of straightforward package layouts like the following (I actually
+use package layouts along these lines in my own projects)::
 
     project/
         setup.py
-        package/
+        example/
             __init__.py
             foo.py
             tests/
                 __init__.py
                 test_foo.py
 
-I would actually often see it without the ``__init__.py`` files first, but
-that's a trivial fix to explain. What's hard to explain is that all of the
-following ways to invoke ``test_foo.py`` *probably won't work* due to broken
-imports (either failing to find ``package`` for absolute imports, complaining
-about relative imports in a non-package for explicit relative imports, or
-issuing even more obscure errors if some other submodule happens to shadow
-the name of a top-level module, such as a ``package.json`` module that
-handled serialisation or a ``package.tests.unittest`` test runner)::
+While I would often see it without the ``__init__.py`` files first, that's a
+trivial fix to explain. What's hard to explain is that all of the following
+ways to invoke ``test_foo.py`` *probably won't work* due to broken imports
+(either failing to find ``example`` for absolute imports, complaining
+about relative imports in a non-package or beyond the toplevel package for
+explicit relative imports, or issuing even more obscure errors if some other
+submodule happens to shadow the name of a top-level module, such as an
+``example.json`` module that handled serialisation or an
+``example.tests.unittest`` test runner)::
 
-    # working directory: project/package/tests
+    # These commands will most likely *FAIL*, even if the code is correct
+
+    # working directory: project/example/tests
     ./test_foo.py
     python test_foo.py
-    python -m test_foo
-    python -c "from test_foo import main; main()"
+    python -m package.tests.test_foo
+    python -c "from package.tests.test_foo import main; main()"
 
     # working directory: project/package
     tests/test_foo.py
     python tests/test_foo.py
-    python -m tests.test_foo
-    python -c "from tests.test_foo import main; main()"
+    python -m package.tests.test_foo
+    python -c "from package.tests.test_foo import main; main()"
 
     # working directory: project
-    package/tests/test_foo.py
-    python package/tests/test_foo.py
+    example/tests/test_foo.py
+    python example/tests/test_foo.py
 
     # working directory: project/..
-    project/package/tests/test_foo.py
-    python project/package/tests/test_foo.py
+    project/example/tests/test_foo.py
+    python project/example/tests/test_foo.py
     # The -m and -c approaches don't work from here either, but the failure
-    # to find 'package' correctly is pretty easy to explain in this case
+    # to find 'package' correctly is easier to explain in this case
 
 That's right, that long list is of all the methods of invocation that will
 almost certainly *break* if you try them, and the error messages won't make
@@ -162,16 +167,24 @@
 The fact that most methods of invoking Python code from the command line
 break when that code is inside a package, and the two that do work are highly
 sensitive to the current working directory is all thoroughly confusing for a
-beginner, and I personally believe it is one of the key factors leading
+beginner. I personally believe it is one of the key factors leading
 to the perception that Python packages are complicated and hard to get right.
 
 This problem isn't even limited to the command line - if ``test_foo.py`` is
-open in Idle and you attempt to run it by pressing F5, then it will fail in
-just the same way it would if run directly from the command line.
+open in Idle and you attempt to run it by pressing F5, or if you try to run
+it by clicking on it in a graphical filebrowser, then it will fail in just
+the same way it would if run directly from the command line.
 
-There's a reason the general ``sys.path`` guideline mentioned above exists,
-and the fact that the interpreter itself doesn't follow it when determining
-``sys.path[0]`` is the root cause of all sorts of grief.
+There's a reason the general "no package directories on ``sys.path``"
+guideline exists, and the fact that the interpreter itself doesn't follow
+it when determining ``sys.path[0]`` is the root cause of all sorts of grief.
+
+In the past, this couldn't be fixed due to backwards compatibility concerns.
+However, scripts potentially affected by this problem will *already* require
+fixes when porting to the Python 3.x (due to the elimination of implicit
+relative imports when importing modules normally). This provides a convenient
+opportunity to implement a corresponding change in the initialisation
+semantics for ``sys.path[0]``.
 
 
 Importing the main module twice
@@ -282,8 +295,8 @@
 (``__fullname__``) and "implementation name" (``__implname__``).
 
 Either of those would actually be valid for the use case in this PEP.
-However, as a meta-issue, it seemed needlessly inconsistent to add *two*
-terms that were essentially "like ``__name__``, but different in some cases
+However, as a meta-issue, PEP 3155 is *also* adding a new attribute (for
+functions and classes) that is "like ``__name__``, but different in some cases
 where ``__name__`` is missing necessary information" and those terms aren't
 accurate for the PEP 3155 function and class use case.
 
@@ -294,6 +307,11 @@
 refer to the same function or class, it's just that ``__name__`` is
 insufficient to accurately identify nested functions and classes).
 
+Since it seems needlessly inconsistent to add *two* new terms for attributes
+that only exist because backwards compatibility concerns keep us from
+changing the behaviour of ``__name__`` itself, this PEP instead chose to
+adopt the PEP 3155 terminology.
+
 If the relative inscrutability of "qualified name" and ``__qualname__``
 encourages interested developers to look them up at least once rather than
 assuming they know what they mean just from the name and guessing wrong,
@@ -314,9 +332,9 @@
 dealing with them.
 
 A rough draft of some of the concepts presented here was first posted on the
-python-ideas list [1]_, but they have evolved considerably since first being
+python-ideas list ([1]_), but they have evolved considerably since first being
 discussed in that thread. Further discussion has subsequently taken place on 
-the import-sig mailing list [2]_.
+the import-sig mailing list ([2]_. [3]_).
 
 
 Fixing main module imports inside packages
@@ -346,26 +364,20 @@
 It is proposed that this initialisation process be modified to take
 package details stored on the filesystem into account::
 
-    # Interactive prompt, -c switch
-    in_package, path_entry, modname = split_path_module(os.getcwd(), '')
+    # Interactive prompt, -m switch, -c switch
+    in_package, path_entry, _ignored = split_path_module(os.getcwd(), '')
     if in_package:
         sys.path.insert(0, path_entry)
     else:
         sys.path.insert(0, '')
+
     # Start interactive prompt or run -c command as usual
-    # __main__.__qualname__ is set to "__main__"
+    #   __main__.__qualname__ is set to "__main__"
 
-::
-
-    # -m switch
-    modname = <<argument to -m switch>>
-    in_package, path_entry, modname = split_path_module(os.getcwd(), modname)
-    if in_package:
-        sys.path.insert(0, path_entry)
-    else:
-        sys.path.insert(0, '')
-    # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
-    # __main__.__qualname__ is set to modname
+    # The -m switches uses the same sys.path[0] calculation, but:
+    #   modname is the argument to the -m switch
+    #   modname is passed to ``runpy._run_module_as_main()`` as usual
+    #   __main__.__qualname__ is set to modname
 
 ::
 
@@ -373,6 +385,7 @@
     modname = "__main__"
     path_entry, modname = split_path_module(sys.argv[0], modname)
     sys.path.insert(0, path_entry)
+
     # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
     # __main__.__qualname__ is set to modname
 
@@ -423,11 +436,9 @@
 With this fix in place, and the same simple package layout described earlier,
 *all* of the following commands would invoke the test suite correctly::
 
-    # working directory: project/package/tests
+    # working directory: project/example/tests
     ./test_foo.py
     python test_foo.py
-    python -m test_foo
-    python -m tests.test_foo
     python -m package.tests.test_foo
     python -c "from .test_foo import main; main()"
     python -c "from ..tests.test_foo import main; main()"
@@ -436,27 +447,64 @@
     # working directory: project/package
     tests/test_foo.py
     python tests/test_foo.py
-    python -m tests.test_foo
     python -m package.tests.test_foo
     python -c "from .tests.test_foo import main; main()"
     python -c "from package.tests.test_foo import main; main()"
 
     # working directory: project
-    package/tests/test_foo.py
-    python package/tests/test_foo.py
+    example/tests/test_foo.py
+    python example/tests/test_foo.py
     python -m package.tests.test_foo
     python -c "from package.tests.test_foo import main; main()"
 
     # working directory: project/..
-    project/package/tests/test_foo.py
-    python project/package/tests/test_foo.py
+    project/example/tests/test_foo.py
+    python project/example/tests/test_foo.py
     # The -m and -c approaches still don't work from here, but the failure
     # to find 'package' correctly is pretty easy to explain in this case
 
+With these changes, clicking Python modules in a graphical file browser
+should always execute them correctly, even if they live inside a package.
 Depending on the details of how it invokes the script, Idle would likely also
 be able to run ``test_foo.py`` correctly with F5, without needing any Idle
 specific fixes.
 
+Optional addition: command line relative imports
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+With the above changes in place, it would be a fairly minor addition to allow
+explicit relative imports as arguments to the ``-m`` switch::
+
+    # working directory: project/example/tests
+    python -m .test_foo
+    python -m ..tests.test_foo
+
+    # working directory: project/example/
+    python -m .tests.test_foo
+
+With this addition, system initialisation for the ``-m`` switch would change
+as follows::
+
+    # -m switch (permitting explicit relative imports)
+    in_package, path_entry, pkg_name = split_path_module(os.getcwd(), '')
+    qualname= <<arguments to -m switch>>
+    if qualname.startswith('.'):
+        modname = qualname
+        while modname.startswith('.'):
+            modname = modname[1:]
+            pkg_name, sep, _ignored = pkg_name.rpartition('.')
+            if not sep:
+                raise ImportError("Attempted relative import beyond top level package")
+        qualname = pkg_name + '.' modname
+    if in_package:
+        sys.path.insert(0, path_entry)
+    else:
+        sys.path.insert(0, '')
+
+    # qualname is passed to ``runpy._run_module_as_main()``
+    # _main__.__qualname__ is set to qualname
+
+
 
 Compatibility with PEP 382
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -486,7 +534,7 @@
 modules more consistent would no longer be feasible.
 
 This incompatibility is discussed in more detail in the relevant import-sig
-thread [2]_.
+threads ([2]_, [3]_).
 
 
 Potential incompatibilities with scripts stored in packages
@@ -637,6 +685,10 @@
 relative imports by adjusting ``__name__`` rather than setting ``__package__``
 directly.
 
+This PEP does *not* propose that ``__package__`` be deprecated. While it is
+technically redundant following the introduction of ``__qualname__``, it just
+isn't worth the hassle of deprecating it within the lifetime of Python 3.x.
+
 
 Reference Implementation
 ========================
@@ -653,7 +705,11 @@
 .. [2] PEP 395 (Module aliasing) and the namespace PEPs
    (http://mail.python.org/pipermail/import-sig/2011-November/000382.html)
 
+.. [3] Updated PEP 395 (aka "Implicit Relative Imports Must Die!")
+   (http://mail.python.org/pipermail/import-sig/2011-November/000397.html)
 
+.. [4] Elaboration of compatibility problems between this PEP and PEP 402
+   (http://mail.python.org/pipermail/import-sig/2011-November/000403.html)
 
 Copyright
 =========

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Thu Nov 24 13:52:36 2011
From: python-checkins at python.org (victor.stinner)
Date: Thu, 24 Nov 2011 13:52:36 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2313415=3A_Test_in_c?=
 =?utf8?q?onfigure_if_unsetenv=28=29_has_a_return_value_or_not=2E?=
Message-ID: <E1RTYn6-0004LB-DS@dinsdale.python.org>

http://hg.python.org/cpython/rev/1b83fd683e28
changeset:   73729:1b83fd683e28
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Thu Nov 24 13:53:38 2011 +0100
summary:
  Close #13415: Test in configure if unsetenv() has a return value or not.

Patch written by Charles-Fran?ois Natali.

files:
  Modules/posixmodule.c |   7 ++++++-
  configure             |  28 ++++++++++++++++++++++++++++
  configure.in          |   9 +++++++++
  pyconfig.h.in         |   3 +++
  4 files changed, 46 insertions(+), 1 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7841,19 +7841,24 @@
 posix_unsetenv(PyObject *self, PyObject *args)
 {
     PyObject *name;
+#ifndef HAVE_BROKEN_UNSETENV
     int err;
+#endif
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
 
                           PyUnicode_FSConverter, &name))
         return NULL;
 
-
+#ifdef HAVE_BROKEN_UNSETENV
+    unsetenv(PyBytes_AS_STRING(name));
+#else
     err = unsetenv(PyBytes_AS_STRING(name));
     if (err) {
         Py_DECREF(name);
         return posix_error();
     }
+#endif
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -9812,6 +9812,34 @@
 fi
 rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken unsetenv" >&5
+$as_echo_n "checking for broken unsetenv... " >&6; }
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+#include <stdlib.h>
+
+int
+main ()
+{
+int res = unsetenv("DUMMY")
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_compile "$LINENO"; then :
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+else
+
+$as_echo "#define HAVE_BROKEN_UNSETENV 1" >>confdefs.h
+
+   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+
 for ac_prog in true
 do
   # Extract the first word of "$ac_prog", so it can be a program name with args.
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -2688,6 +2688,15 @@
   [AC_MSG_RESULT(no)
 ])
 
+AC_MSG_CHECKING(for broken unsetenv)
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+#include <stdlib.h>
+]], [[int res = unsetenv("DUMMY")]])],
+  [AC_MSG_RESULT(no)],
+  [AC_DEFINE(HAVE_BROKEN_UNSETENV, 1, Define if `unsetenv` does not return an int.)
+   AC_MSG_RESULT(yes)
+])
+
 dnl check for true
 AC_CHECK_PROGS(TRUE, true, /bin/true)
 
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -95,6 +95,9 @@
 /* define to 1 if your sem_getvalue is broken. */
 #undef HAVE_BROKEN_SEM_GETVALUE
 
+/* Define if `unsetenv` does not return an int. */
+#undef HAVE_BROKEN_UNSETENV
+
 /* Define this if you have the type _Bool. */
 #undef HAVE_C99_BOOL
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 24 19:09:38 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Thu, 24 Nov 2011 19:09:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_testFnNames_is_?=
	=?utf8?q?assigned_twice=2E?=
Message-ID: <E1RTdju-00085T-CD@dinsdale.python.org>

http://hg.python.org/cpython/rev/f29d256be73d
changeset:   73730:f29d256be73d
branch:      3.2
parent:      73726:71cb7ca03bc5
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Fri Nov 25 02:08:39 2011 +0800
summary:
  testFnNames is assigned twice.

files:
  Lib/unittest/loader.py |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Lib/unittest/loader.py b/Lib/unittest/loader.py
--- a/Lib/unittest/loader.py
+++ b/Lib/unittest/loader.py
@@ -139,8 +139,7 @@
                          prefix=self.testMethodPrefix):
             return attrname.startswith(prefix) and \
                 callable(getattr(testCaseClass, attrname))
-        testFnNames = testFnNames = list(filter(isTestMethod,
-                                                dir(testCaseClass)))
+        testFnNames = list(filter(isTestMethod, dir(testCaseClass)))
         if self.sortTestMethodsUsing:
             testFnNames.sort(key=functools.cmp_to_key(self.sortTestMethodsUsing))
         return testFnNames

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 24 19:09:39 2011
From: python-checkins at python.org (senthil.kumaran)
Date: Thu, 24 Nov 2011 19:09:39 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_merge_from_3=2E2_testFnNames_is_assigned_twice=2E?=
Message-ID: <E1RTdjv-00085d-5a@dinsdale.python.org>

http://hg.python.org/cpython/rev/89819a6393a5
changeset:   73731:89819a6393a5
parent:      73729:1b83fd683e28
parent:      73730:f29d256be73d
user:        Senthil Kumaran <senthil at uthcode.com>
date:        Fri Nov 25 02:09:24 2011 +0800
summary:
  merge from 3.2 testFnNames is assigned twice.

files:
  Lib/unittest/loader.py |  3 +--
  1 files changed, 1 insertions(+), 2 deletions(-)


diff --git a/Lib/unittest/loader.py b/Lib/unittest/loader.py
--- a/Lib/unittest/loader.py
+++ b/Lib/unittest/loader.py
@@ -139,8 +139,7 @@
                          prefix=self.testMethodPrefix):
             return attrname.startswith(prefix) and \
                 callable(getattr(testCaseClass, attrname))
-        testFnNames = testFnNames = list(filter(isTestMethod,
-                                                dir(testCaseClass)))
+        testFnNames = list(filter(isTestMethod, dir(testCaseClass)))
         if self.sortTestMethodsUsing:
             testFnNames.sort(key=functools.cmp_to_key(self.sortTestMethodsUsing))
         return testFnNames

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 24 21:04:15 2011
From: python-checkins at python.org (petri.lehtinen)
Date: Thu, 24 Nov 2011 21:04:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_a_=22What=27s_New=22_en?=
	=?utf8?q?try_for_=2312170?=
Message-ID: <E1RTfWp-0008HV-FX@dinsdale.python.org>

http://hg.python.org/cpython/rev/736b0aec412b
changeset:   73732:736b0aec412b
user:        Petri Lehtinen <petri at digip.org>
date:        Thu Nov 24 22:00:46 2011 +0200
summary:
  Add a "What's New" entry for #12170

files:
  Doc/whatsnew/3.3.rst |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -209,6 +209,13 @@
 (:issue:`13021`)
 
 
+The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()``
+methods of :class:`bytes` and :class:`bytearray` objects now accept an
+integer between 0 and 255 as their first argument.
+
+(:issue:`12170`)
+
+
 New and Improved Modules
 ========================
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Thu Nov 24 21:07:05 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 24 Nov 2011 21:07:05 +0100 (CET)
Subject: [Python-checkins] r88917 - in tracker/instances/python-dev:
	extensions/openid_login.py lib/openid2rp.py
Message-ID: <3SqBQK3YyyzNYy@mail.python.org>

Author: martin.v.loewis
Date: Thu Nov 24 21:07:05 2011
New Revision: 88917

Log:
Update openid2rp to support direct verification.


Modified:
   tracker/instances/python-dev/extensions/openid_login.py
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/extensions/openid_login.py
==============================================================================
--- tracker/instances/python-dev/extensions/openid_login.py	(original)
+++ tracker/instances/python-dev/extensions/openid_login.py	Thu Nov 24 21:07:05 2011
@@ -16,24 +16,16 @@
     'Helper class for OpenID'
 
     # Session management: Recycle expired session objects
-    def get_session(self, provider, discovered=None):
-        sessions = self.db.openid_session.filter(None, {'provider_id':provider})
+    def get_session(self, url, stypes):
+        sessions = self.db.openid_session.filter(None, {'url':url})
         for session_id in sessions:
             # Match may not have been exact
-            if self.db.openid_session.get(session_id, 'provider_id') != provider:
-                continue
-            if discovered and discovered[1] != self.db.openid_session.get(session_id, 'url'):
-                # User has changed provider; don't reuse session
+            if self.db.openid_session.get(session_id, 'url') != url:
                 continue
             expires = self.db.openid_session.get(session_id, 'expires')
             if  expires > date.Date('.')+date.Interval("1:00"):
                 # valid for another hour
                 return self.db.openid_session.getnode(session_id)
-        # need to create new session, or recycle an expired one
-        if discovered:
-            stypes, url, op_local = discovered
-        else:
-            stypes, url, op_local = openid2rp.discover(provider)
         now = date.Date('.')
         session_data = openid2rp.associate(stypes, url)
         # check whether a session has expired a day ago
@@ -44,40 +36,77 @@
         else:
             session_id = self.db.openid_session.create(assoc_handle=session_data['assoc_handle'])
             session = self.db.openid_session.getnode(session_id)
-        session.provider_id = provider
         session.url = url
-        session.stypes = " ".join(stypes)
         session.mac_key = session_data['mac_key']
         session.expires = now + date.Interval(int(session_data['expires_in']))
         self.db.commit()
         return session
 
-    def authenticate(self, session, query):
+    def discover(self, url):
+        '''Return cached discovery results or None.'''
+        try:
+            discovered = self.db.openid_discovery.lookup(url)
+        except KeyError:
+            return None
+        discovered = self.db.openid_discovery.getnode(discovered)
+        op_local = discovered.op_local
+        if op_local == '':
+            op_local = None
+        return discovered.services.split(), discovered.op_endpoint, op_local
+
+    def store_discovered(self, url, stypes, op_endpoint, op_local):
+        if op_local is None:
+            op_local = ''
+        try:
+            discovered = self.db.openid_discovery.lookup(url)
+        except KeyError:
+            self.db.openid_discovery.create(url=url, services=" ".join(stypes),
+                                            op_endpoint=op_endpoint, op_local=op_local)
+        else:
+            discovered = self.db.openid_discovery.getnode(discovered)
+            discovered.services = " ".join(stypes)
+            discovered.op_endpoint = op_endpoint
+            discovered.op_local = op_local
+            self.db.commit()
+
+    def find_association(self, handle):
+        try:
+            session = self.db.openid_session.lookup(handle)
+            session = self.db.openid_session.getnode(session)
+            return session
+        except KeyError:
+            return None
+
+    def nonce_seen(self, nonce):
+        try:
+            self.db.openid_nonce.lookup(nonce)
+            return True
+        except KeyError:
+            return False
+
+    def authenticate(self, query):
         '''Authenticate an OpenID indirect response, and return the claimed ID'''
         try:
-            signed = openid2rp.authenticate(session, query)
+            signed, claimed = openid2rp.verify(query, self.discover,
+                                               self.find_association,
+                                               self.nonce_seen)
         except Exception, e:
             raise ValueError, "Authentication failed: "+str(e)
-        if openid2rp.is_op_endpoint(session.stypes):
-            # Provider-guided login: provider ought to report claimed ID
-            if 'openid.claimed_id' in query:
-                claimed = query['openid.claimed_id'][0]
-            else:
-                raise ValueError, 'incomplete response'
-            # OpenID 11.2: verify that provider is authorized to assert ID
-            discovered = openid2rp.discover(claimed)
-            if not discovered or discovered[1] != session.url:
-                raise ValueError, "Provider %s is not authorized to make assertions about %s" % (session.url, claimed)
-        else:
-            # User entered claimed ID, stored in session object
-            claimed = session.provider_id
-            if not openid2rp.is_compat_1x(session.stypes):
-                # can only check correct claimed ID for OpenID 2.0
-                if 'openid.claimed_id' not in query or claimed != query['openid.claimed_id'][0]:
-                    # assertion is not about an ID, or about a different ID; refuse to accept
-                    raise ValueError, "Provider did not assert your ID"
         return claimed
 
+    def store_nonce(self, query):
+        '''Store a nonce in the database.'''
+        if 'openid.response_nonce' in query:
+            nonce = query['openid.response_nonce'][0]
+            stamp = openid2rp.parse_nonce(nonce)
+            # Consume nonce; reuse expired nonces
+            old = self.db.openid_nonce.filter(None, {'created':';.-1d'})
+            stamp = date.Date(stamp)
+            if old:
+                self.db.openid_nonce.set(old[0], created=stamp, nonce=nonce)
+            else:
+                self.db.openid_nonce.create(created=stamp, nonce=nonce)
+            self.db.commit()
         
 class OpenidLogin(LoginAction, Openid):
     'Extended versoin of LoginAction, supporting OpenID identifiers in username field.'
@@ -106,11 +135,12 @@
         discovered = openid2rp.discover(claimed)
         if not discovered:
             raise ValueError, "OpenID provider discovery failed"
+        self.store_discovered(claimed, *discovered)
         stypes, url, op_local = discovered
-        session = self.get_session(claimed, discovered) # one session per claimed id
+        session = self.get_session(url, stypes)
         realm = self.base+"?@action=openid_return"
         return_to = realm + "&__came_from=%s" % urllib.quote(self.client.path)
-        url = openid2rp.request_authentication(session.stypes, session.url,
+        url = openid2rp.request_authentication(stypes, url,
                                             session.assoc_handle, return_to, realm=realm,
                                             claimed=claimed, op_local=op_local)
         raise Redirect, url
@@ -127,14 +157,20 @@
             self.client.error_message.append(self._('Unsupported provider'))
             return
         provider_id = providers[provider][2]
+        # For most providers, it would be reasonable to cache the discovery
+        # results. However, the risk of login breaking if a provider does change
+        # its service URL outweighs the cost of another HTTP request to perform
+        # the discovery during login.
+        services, op_endpoint, op_local = openid2rp.discover(provider_id)
+        session = self.get_session(op_endpoint, services)
         try:
-            session = self.get_session(provider_id)
+            session = self.get_session(op_endpoint, services)
         except NoCertificate:
             self.client.error_message.append(self._('Peer did not return certificate'))
             return
         realm = self.base+"?@action=openid_return"
         return_to = realm + "&__came_from=%s" % urllib.quote(self.client.path)
-        url = openid2rp.request_authentication(session.stypes, session.url,
+        url = openid2rp.request_authentication(services, op_endpoint,
                                             session.assoc_handle, return_to, realm=realm)
         raise Redirect, url
 
@@ -156,40 +192,21 @@
             ''' % self.base
             self.client.additional_headers['Content-Type'] = 'application/xrds+xml'
             return payload
-        if 'openid.response_nonce' in query:
-            nonce = query['openid.response_nonce'][0]
-            stamp = openid2rp.parse_nonce(nonce)
-            utc = calendar.timegm(stamp.utctimetuple())
-            if utc < time.time()-3600:
-                # Old nonce
-                raise ValueError, "Replay detected"
-            try:
-                self.db.openid_nonce.lookup(nonce)
-            except KeyError:
-                pass
-            else:
-                raise ValueError, "Replay detected"
-            # Consume nonce; reuse expired nonces
-            old = self.db.openid_nonce.filter(None, {'created':';.-1d'})
-            stamp = date.Date(stamp)
-            if old:
-                self.db.openid_nonce.set(old[0], created=stamp, nonce=nonce)
-            else:
-                self.db.openid_nonce.create(created=stamp, nonce=nonce)
-            self.db.commit()
         handle = query['openid.assoc_handle'][0]
         try:
             session = self.db.openid_session.lookup(handle)
         except KeyError:
             raise ValueError, 'Not authenticated (no session)'
         session = self.db.openid_session.getnode(session)
-        claimed = self.authenticate(session, query)
+        claimed = self.authenticate(query)
         if self.user != 'anonymous':
             # Existing user claims OpenID
 
             # ID must be currently unassigned
             if self.db.user.filter(None, {'openids':claimed}):
                 raise ValueError, 'OpenID already claimed'
+            # Consume nonce
+            self.store_nonce(query)
             openids = self.db.user.get(self.userid, 'openids')
             if openids:
                 openids += ' '
@@ -203,6 +220,8 @@
         # Check whether this is a successful login
         user = self.db.user.filter(None, {'openids':claimed})
         if user:
+            # Consume nonce
+            self.store_nonce(query)
             # there should be only one user with that ID
             assert len(user)==1
             self.client.userid = user[0]
@@ -267,12 +286,6 @@
         query = {}
         if 'openid.identity' not in self.form:
             raise ValueError, "OpenID fields missing"
-        try:
-            handle = self.form['openid.assoc_handle'].value
-            session = self.db.openid_session.lookup(handle)
-            session = self.db.openid_session.getnode(session)
-        except Exception, e:
-            raise ValueError, "Not authenticated (no session): "+str(e)
         # re-authenticate fields
         for key in self.form:
             if key.startswith("openid"):
@@ -281,9 +294,13 @@
                     query[key].append(value)
                 except KeyError:
                     query[key] = [value]
-        claimed = self.authenticate(session, query)
+        claimed = self.authenticate(query)
         # OpenID signature is still authentic, now pass it on to the base
         # register method; also fake password
+
+        # Consume nonce first
+        self.store_nonce(query)
+        
         self.form.value.append(cgi.MiniFieldStorage('openids', claimed))
         pwd = password.generatePassword()
         self.form.value.append(cgi.MiniFieldStorage('password', pwd))

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Thu Nov 24 21:07:05 2011
@@ -6,7 +6,7 @@
 # This library implements OpenID Authentication 2.0,
 # in the role of a relying party
 
-import urlparse, urllib, httplib, time, cgi, htmllib, formatter
+import urlparse, urllib, httplib, time, cgi, HTMLParser
 import cStringIO, base64, hmac, hashlib, datetime, re, random
 import itertools, cPickle, sys
 
@@ -26,9 +26,51 @@
 if sys.version_info < (3,):
     def b(s):
         return s
+    # Convert byte to integer
+    b2i = ord
+    def bytes_from_ints(L):
+        return ''.join([chr(i) for i in L])
 else:
     def b(s):
-        return s.encode('ascii')
+        return s.encode('latin-1')
+    def b2i(char):
+        # 3.x: bytes are already sequences of integers
+        return char
+    bytes_from_ints = bytes
+
+class NotAuthenticated(Exception):
+    CONNECTION_REFUSED = 1
+    DIRECT_VERIFICATION_FAILED = 2
+    CANCELLED = 3
+    UNSUPPORTED_VERSION = 4
+    UNEXPECTED_MODE = 5
+    CLAIMED_ID_MISSING = 6
+    DISCOVERY_FAILED = 7
+    INCONSISTENT_IDS = 8
+    REPLAY_ATTACK = 9
+    MISSING_NONCE = 10
+    msgs = {
+        CONNECTION_REFUSED : 'OP refuses connection with status %d',
+        DIRECT_VERIFICATION_FAILED : 'OP doesn\'t assert that the signature of the verification request is valid',
+        CANCELLED : 'OP did not authenticate user (cancelled)',
+        UNSUPPORTED_VERSION : 'Unsupported OpenID version',
+        UNEXPECTED_MODE : 'Unexpected mode %s',
+        CLAIMED_ID_MISSING : 'Cannot determine claimed ID',
+        DISCOVERY_FAILED : 'Claimed ID %s cannot be rediscovered',
+        INCONSISTENT_IDS : 'Discovered and asserted endpoints differ',
+        REPLAY_ATTACK : 'Replay attack detected',
+        MISSING_NONCE : 'Nonce missing in OpenID 2 response',
+        }
+
+    def __init__(self, errno, *args):
+        msg = self.msgs[errno]
+        if args:
+            msg %= args
+        self.errno = errno
+        Exception.__init__(self, msg, errno, *args)
+
+    def __str__(self):
+        return self.args[0]
 
 def normalize_uri(uri):
     """Normalize an uri according to OpenID section 7.2. Return a pair
@@ -138,24 +180,60 @@
         res[k] = v
     return res
 
-class OpenIDParser(htmllib.HTMLParser):
+class OpenIDParser(HTMLParser.HTMLParser):
     def __init__(self):
-        htmllib.HTMLParser.__init__(self, formatter.NullFormatter())
+        HTMLParser.HTMLParser.__init__(self)
         self.links = {}
         self.xrds_location=None
 
-    def do_link(self, attrs):
-        attrs = dict(attrs)
-        try:
-            self.links[attrs['rel']] = attrs['href']
-        except KeyError:
-            pass
-
-    def do_meta(self, attrs):
-        attrs = dict(attrs)
-        # Yadis 6.2.5 option 1: meta tag
-        if attrs.get('http-equiv','').lower() == 'x-xrds-location':
-            self.xrds_location = attrs['content']
+    def handle_starttag(self, tag, attrs):
+        if tag == 'link':
+            attrs = dict(attrs)
+            try:
+                self.links[attrs['rel']] = attrs['href']
+            except KeyError:
+                pass
+        elif tag == 'meta':
+            attrs = dict(attrs)
+            # Yadis 6.2.5 option 1: meta tag
+            if attrs.get('http-equiv','').lower() == 'x-xrds-location':
+                self.xrds_location = attrs['content']
+
+def _extract_services(doc):
+    for svc in doc.findall(".//{xri://$xrd*($v*2.0)}Service"):
+        services = [x.text for x in svc.findall("{xri://$xrd*($v*2.0)}Type")]
+        if 'http://specs.openid.net/auth/2.0/server' in services:
+            # 7.3.2.1.1 OP Identifier Element
+            uri = svc.find("{xri://$xrd*($v*2.0)}URI")
+            if uri is not None:
+                op_local = None
+                op_endpoint = uri.text
+                break
+        elif 'http://specs.openid.net/auth/2.0/signon' in services:
+            # 7.3.2.1.2.  Claimed Identifier Element
+            op_local = svc.find("{xri://$xrd*($v*2.0)}LocalID")
+            if op_local is not None:
+                op_local = op_local.text
+            uri = svc.find("{xri://$xrd*($v*2.0)}URI")
+            if uri is not None:
+                op_endpoint = uri.text
+                break
+        elif 'http://openid.net/server/1.0' in services or \
+                'http://openid.net/server/1.1' in services or \
+                'http://openid.net/signon/1.0' in services or \
+                'http://openid.net/signon/1.1' in services:
+            # 14.2.1 says we also need to check for the 1.x types;
+            # XXX should check 1.x only if no 2.0 service is found
+            op_local = svc.find("{http://openid.net/xmlns/1.0}Delegate")
+            if op_local is not None:
+                op_local = op_local.text
+            uri = svc.find("{xri://$xrd*($v*2.0)}URI")
+            if uri is not None:
+                op_endpoint = uri.text
+                break
+    else:
+        return None # No OpenID 2.0 service found
+    return services, op_endpoint, op_local
 
 def discover(url):
     '''Perform service discovery on the OP URL.
@@ -205,7 +283,7 @@
 
     if content_type in ('text/html', 'application/xhtml+xml'):
         parser = OpenIDParser()
-        parser.feed(data)
+        parser.feed(data.decode('latin-1'))
         parser.close()
         # Yadis 6.2.5 option 1: meta tag
         if parser.xrds_location:
@@ -226,44 +304,48 @@
     elif content_type == 'application/xrds+xml':
         # Yadis 6.2.5 option 4
         doc = ElementTree.fromstring(data)
-        for svc in doc.findall(".//{xri://$xrd*($v*2.0)}Service"):
-            services = [x.text for x in svc.findall("{xri://$xrd*($v*2.0)}Type")]
-            if 'http://specs.openid.net/auth/2.0/server' in services:
-                # 7.3.2.1.1 OP Identifier Element
-                uri = svc.find("{xri://$xrd*($v*2.0)}URI")
-                if uri is not None:
-                    op_local = None
-                    op_endpoint = uri.text
-                    break
-            elif 'http://specs.openid.net/auth/2.0/signon' in services:
-                # 7.3.2.1.2.  Claimed Identifier Element
-                op_local = svc.find("{xri://$xrd*($v*2.0)}LocalID")
-                if op_local is not None:
-                    op_local = op_local.text
-                uri = svc.find("{xri://$xrd*($v*2.0)}URI")
-                if uri is not None:
-                    op_endpoint = uri.text
-                    break
-            elif 'http://openid.net/server/1.0' in services or \
-                 'http://openid.net/server/1.1' in services or \
-                 'http://openid.net/signon/1.0' in services or \
-                 'http://openid.net/signon/1.1' in services:
-                # 14.2.1 says we also need to check for the 1.x types;
-                # XXX should check 1.x only if no 2.0 service is found
-                op_local = svc.find("{http://openid.net/xmlns/1.0}Delegate")
-                if op_local is not None:
-                    op_local = op_local.text
-                uri = svc.find("{xri://$xrd*($v*2.0)}URI")
-                if uri is not None:
-                    op_endpoint = uri.text
-                    break
-        else:
-            return None # No OpenID 2.0 service found
+        return _extract_services(doc)
     else:
         # unknown content type
         return None
     return services, op_endpoint, op_local
 
+def resolve_xri(xri, proxy='xri.net'):
+    '''Perform XRI resolution of xri using a proxy resolver.
+    Return canonical ID, services, op endpoint, op local;
+    return None if an error occurred'''
+    xri = urllib.quote(xri, safe='=@*!+()')
+    # Ask explicitly for the specific service types, to
+    # avoid having to identify the correct XRD element in
+    # case the identifier is hierarchical
+    for stype in ('http://specs.openid.net/auth/2.0/signon',
+                  'http://openid.net/signon/1.0'):
+        conn = httplib.HTTPConnection(proxy)
+        try:
+            conn.connect()
+        except:
+            # DNS or TCP error
+            return None
+        conn.putrequest("GET", '/'+xri+'?_xrd_r=application/xrd+xml'
+                        +'&_xrd_t='+stype)
+        conn.putheader('Connection', 'Keep-Alive')
+        conn.endheaders()
+        res = conn.getresponse()
+        data = res.read()
+        doc = ElementTree.fromstring(data)
+        res = _extract_services(doc)
+        conn.close()
+        if res is not None:
+            break
+    else:
+        # No OpenID service found
+        return None
+    services, op_endpoint, op_local = res
+    canonical_id = doc.find(".//{xri://$xrd*($v*2.0)}CanonicalID")
+    if canonical_id is None:
+        return None
+    return canonical_id.text, services, op_endpoint, op_local
+
 def is_compat_1x(services):
     for uri in ('http://specs.openid.net/auth/2.0/signon',
                 'http://specs.openid.net/auth/2.0/server'):
@@ -291,13 +373,13 @@
     res = cPickle.dumps(l, 2)
     # Pickle result: proto 2, long1 (integer < 256 bytes)
     # number of bytes, little-endian integer, stop
-    assert res[:3] == '\x80\x02\x8a' 
+    assert res[:3] == b('\x80\x02\x8a')
     # btwoc ought to produce the shortest representation in two's
     # complement. Fortunately, pickle already does that.
-    return res[3+ord(res[3]):3:-1]
+    return res[3+b2i(res[3]):3:-1]
 
-def unbtwoc(b):
-    return cPickle.loads('\x80\x02\x8a'+chr(len(b))+b[::-1]+'.')
+def unbtwoc(B):
+    return cPickle.loads(b('\x80\x02\x8a')+bytes_from_ints([len(B)])+B[::-1]+(b'.'))
 
 # Appendix B; DH default prime
 dh_prime = """
@@ -311,8 +393,8 @@
 def string_xor(s1, s2):
     res = []
     for c1, c2 in itertools.izip(s1, s2):
-        res.append(chr(ord(c1) ^ ord(c2)))
-    return ''.join(res)
+        res.append(b2i(c1) ^ b2i(c2))
+    return bytes_from_ints(res)
 
 def associate(services, url):
     '''Create an association (OpenID section 8) between RP and OP.
@@ -338,23 +420,25 @@
         if data['openid.session_type'] == "no-encryption":
             data['openid.session_type'] = ''
         del data['openid.ns']
-    res = urllib.urlopen(url, urllib.urlencode(data))
+    res = urllib.urlopen(url, b(urllib.urlencode(data)))
+    if res.getcode() != 200:
+        raise ValueError, "OpenID provider refuses connection with status %d" % res.getcode()
     data = parse_response(res.read())
     if 'error' in data:
         raise ValueError, "associate failed: "+data['error']
     if url.startswith('http:'):
-        enc_mac_key = data.get('enc_mac_key')
+        enc_mac_key = b(data.get('enc_mac_key'))
         if not enc_mac_key:
             raise ValueError, "Provider protocol error: not using DH-SHA1"
-        enc_mac_key = base64.b64decode(data['enc_mac_key'])
-        dh_server_public = unbtwoc(base64.b64decode(data['dh_server_public']))
+        enc_mac_key = base64.b64decode(enc_mac_key)
+        dh_server_public = unbtwoc(base64.b64decode(b(data['dh_server_public'])))
         # shared secret: sha1(2^(server_priv*priv) mod prime) xor enc_mac_key
         shared_secret = btwoc(pow(dh_server_public, priv, dh_prime))
         shared_secret = hashlib.sha1(shared_secret).digest()
         if len(shared_secret) != len(enc_mac_key):
             raise ValueError, "incorrect DH key size"
         # Fake mac_key result
-        data['mac_key'] = base64.b64encode(string_xor(enc_mac_key, shared_secret))
+        data['mac_key'] = b(base64.b64encode(string_xor(enc_mac_key, shared_secret)))
     return data
 
 class _AX:
@@ -407,7 +491,7 @@
     if claimed is None:
         claimed = "http://specs.openid.net/auth/2.0/identifier_select"
     if op_local is None:
-        op_local = "http://specs.openid.net/auth/2.0/identifier_select"
+        op_local = claimed
     if realm is None:
         realm = return_to
     data = {
@@ -429,10 +513,15 @@
         if sreg_opt:
             data['openid.sreg.optional'] =  sreg11['openid.sreg11.optional'] =','.join(sreg_opt)
     if is_compat_1x(services):
+        # OpenID 1.1 does not communicate claimed_ids. Put them into the return URL
+        return_to += '&' if '?' in return_to else '?'
+        return_to += '&openid1=' + urllib.quote(claimed)
+        data['openid.return_to'] = return_to
         del data['openid.ns']
         del data['openid.claimed_id']
         del data['openid.realm']
-        data['openid.trust_root'] = return_to
+        trust_root = urlparse.urlparse(return_to)[:3] + (None,None,None)
+        data['openid.trust_root'] = urlparse.urlunparse(trust_root)
     ax_req, ax_opt = ax
     if "http://openid.net/srv/ax/1.0" in services and (ax_req or ax_opt):
         data.update({
@@ -455,8 +544,20 @@
     else:
         return url+"?"+urllib.urlencode(data)
 
-class NotAuthenticated(Exception):
-    pass
+# 11.4.2 Verifying Directly with the OpenID Provider
+def verify_signature_directly(op_endpoint, response):
+    '''Request that the OP verify the signature via Direct Verification'''
+
+    request = [('openid.mode', 'check_authentication')]
+    # Exact copies of all fields from the authentication response, except for
+    # "openid.mode"
+    request.extend((k, v) for k, (v,) in response.items() if 'openid.mode' != k)
+    res = urllib.urlopen(op_endpoint, urllib.urlencode(request))
+    if 200 != res.getcode():
+        raise NotAuthenticated(NotAuthenticated.CONNECTION_REFUSED, res.getcode())
+    response = parse_response(res.read())
+    if 'true' != response['is_valid']:
+        raise NotAuthenticated(NotAuthenticated.DIRECT_VERIFICATION_FAILED)
 
 def _prepare_response(response):
     if isinstance(response, str):
@@ -487,7 +588,7 @@
     if session['assoc_handle'] != response['openid.assoc_handle'][0]:
         raise ValueError('incorrect session')
     if response['openid.mode'][0] == 'cancel':
-        raise NotAuthenticated('provider did not authenticate user (cancelled)')
+        raise NotAuthenticated(NotAuthenticated.CANCELLED)
     if response['openid.mode'][0] != 'id_res':
         raise ValueError('invalid openid.mode')
     if  'openid.identity' not in response:
@@ -505,7 +606,7 @@
         query.append(value)
     query = b('').join(query)
 
-    mac_key = base64.decodestring(b(session['mac_key']))
+    mac_key = base64.decodestring(session['mac_key'])
     transmitted_sig = base64.decodestring(b(response['openid.sig'][0]))
     computed_sig = hmac.new(mac_key, query, hashlib.sha1).digest()
 
@@ -527,8 +628,75 @@
 
     return signed
 
+# td.total_seconds only works in 2.7
+def _total_seconds(td):
+    return td.days*24*3600 + td.seconds
+
+def verify(response, discovery_cache, find_association, nonce_seen):
+    response = _prepare_response(response)
+    if 'openid.ns' in response:
+        ns = response['openid.ns'][0]
+        if ns != 'http://specs.openid.net/auth/2.0':
+            raise NotAuthenticated(NotAuthenticate.UNSUPPORTED_VERSION)
+    else:
+        ns = None
+    mode = response['openid.mode'][0]
+    if mode == 'cancel':
+        raise NotAuthenticated(NotAuthenticated.CANCELLED)
+    if mode != 'id_res':
+        raise NotAuthenticated(NotAuthenticated.UNEXPECTED_MODE, mode)
+    # Establish claimed ID
+    if 'openid.claimed_id' in response:
+        claimed_id = response['openid.claimed_id'][0]
+        # 11.2. Drop Fragment from claimed_id
+        fragment = claimed_id.find('#')
+        if fragment != -1:
+            claimed_id = claimed_id[:fragment]
+    elif 'openid1' in response:
+        claimed_id = response['openid1'][0]
+    else:
+        raise NotAuthenticated(NotAuthenticated.CLAIMED_ID_MISSING)
+    discovered = discovery_cache(claimed_id)
+    if not discovered:
+        discovered = discover(claimed_id)
+        if not discovered:
+            raise NotAuthenticated(NotAuthenticated.DISCOVERY_FAILED, claimed_id)
+    services, op_endpoint, op_local = discovered
+    # For a provider-allocated claimed_id, there will be no op_local ID,
+    # and there is no point checking it.
+    if op_local and op_local != response['openid.identity'][0]:
+        raise NotAuthenticated('Discovered and asserted local identifiers differ')
+    # For OpenID 1.1, op_endpoint may not be included in the response
+    if ('openid.op_endpoint' in response and
+        op_endpoint != response['openid.op_endpoint'][0]):
+        raise NotAuthenticated(NotAuthenticated.INCONSISTENT_IDS)
+    # XXX verify protocol version, verify claimed_id wrt. original request,
+    # verify return_to URL
+    
+    # verify the signature
+    assoc_handle = response['openid.assoc_handle'][0]
+    session = find_association(assoc_handle)
+    if session:
+        signed = authenticate(session, response)
+    else:
+        verify_signature_directly(op_endpoint, response)
+        signed = response['openid.signed'][0].split(',')
+
+    # Check the nonce. OpenID 1.1 doesn't have them
+    if 'openid.response_nonce' in response:
+        nonce = response['openid.response_nonce'][0]
+        timestamp = parse_nonce(nonce)
+        if _total_seconds(datetime.datetime.utcnow() - timestamp) > 10:
+            # allow for at most 10s transmission time and time shift
+            raise NotAuthenticated(NotAuthenticated.REPLAY_ATTACK)
+        if nonce_seen(nonce):
+            raise NotAuthenticated(NotAuthenticated.REPLAY_ATTACK)
+    elif ns:
+        raise NotAuthenticated(NotAuthenticated.MISSING_NONCE)
+    return signed, claimed_id
+
 def parse_nonce(nonce):
-    '''Split a nonce into a (timestamp, ID) pair'''
+    '''Extract a datetime.datetime stamp from the nonce'''
     stamp = nonce.split('Z', 1)[0]
     stamp = time.strptime(stamp, "%Y-%m-%dT%H:%M:%S")[:6]
     stamp = datetime.datetime(*stamp)

From python-checkins at python.org  Thu Nov 24 21:13:12 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 24 Nov 2011 21:13:12 +0100 (CET)
Subject: [Python-checkins] r88918 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqBYN39CdzNZD@mail.python.org>

Author: martin.v.loewis
Date: Thu Nov 24 21:13:12 2011
New Revision: 88918

Log:
Fix syntax error.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Thu Nov 24 21:13:12 2011
@@ -379,7 +379,7 @@
     return res[3+b2i(res[3]):3:-1]
 
 def unbtwoc(B):
-    return cPickle.loads(b('\x80\x02\x8a')+bytes_from_ints([len(B)])+B[::-1]+(b'.'))
+    return cPickle.loads(b('\x80\x02\x8a')+bytes_from_ints([len(B)])+B[::-1]+b('.'))
 
 # Appendix B; DH default prime
 dh_prime = """

From python-checkins at python.org  Thu Nov 24 21:14:44 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Thu, 24 Nov 2011 21:14:44 +0100 (CET)
Subject: [Python-checkins] r88919 - tracker/instances/python-dev/schema.py
Message-ID: <3SqBb86QDTzNZT@mail.python.org>

Author: martin.v.loewis
Date: Thu Nov 24 21:14:44 2011
New Revision: 88919

Log:
Add new openid schema.


Modified:
   tracker/instances/python-dev/schema.py

Modified: tracker/instances/python-dev/schema.py
==============================================================================
--- tracker/instances/python-dev/schema.py	(original)
+++ tracker/instances/python-dev/schema.py	Thu Nov 24 21:14:44 2011
@@ -101,10 +101,16 @@
 db.security.addPermission(name='Register', klass='user',
                           description='User is allowed to register new user')
 
+openid_discovery = Class(db, 'openid_discovery',
+                         url=String(), # key
+                         services=String(), # space-separated list
+                         op_endpoint=String(),
+                         op_local=String())
+openid_discovery.setkey('url')
+openid_discovery.disableJournalling()
+
 openid_session = Class(db, 'openid_session',
-                       provider_id=String(), # or user id
                        url=String(),
-                       stypes=String(), # space-separated list of session types
                        assoc_handle=String(),
                        expires=Date(),
                        mac_key=String())

From python-checkins at python.org  Fri Nov 25 01:19:25 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 25 Nov 2011 01:19:25 +0100 (CET)
Subject: [Python-checkins] r88920 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqJ1T6z00zNVZ@mail.python.org>

Author: martin.v.loewis
Date: Fri Nov 25 01:19:25 2011
New Revision: 88920

Log:
Backport getcode to 2.5.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Fri Nov 25 01:19:25 2011
@@ -38,6 +38,21 @@
         return char
     bytes_from_ints = bytes
 
+if sys.version_info < (2,6):
+    # Add getcode into 2.5
+    class _addinfourl(urllib.addinfourl):
+        def __init__(self, fp, headers, url, code=None):
+            urllib.addinfourl(self, fp, headers, url)
+            self.code = code
+        def getcode(self):
+            return self.code
+
+    class FancyURLOpener(urllib.FancyURLopener):
+        def http_error_default(self, url, fp, errcode, errmsg, headers):
+            return _addinfourl(fp, headers, "http:" + url, errcode)
+else:
+    FancyURLopener = urllib.FancyURLopener
+
 class NotAuthenticated(Exception):
     CONNECTION_REFUSED = 1
     DIRECT_VERIFICATION_FAILED = 2
@@ -420,7 +435,7 @@
         if data['openid.session_type'] == "no-encryption":
             data['openid.session_type'] = ''
         del data['openid.ns']
-    res = urllib.urlopen(url, b(urllib.urlencode(data)))
+    res = FancyURLopener(url, b(urllib.urlencode(data)))
     if res.getcode() != 200:
         raise ValueError, "OpenID provider refuses connection with status %d" % res.getcode()
     data = parse_response(res.read())
@@ -552,7 +567,7 @@
     # Exact copies of all fields from the authentication response, except for
     # "openid.mode"
     request.extend((k, v) for k, (v,) in response.items() if 'openid.mode' != k)
-    res = urllib.urlopen(op_endpoint, urllib.urlencode(request))
+    res = FancyURLopener(op_endpoint, urllib.urlencode(request))
     if 200 != res.getcode():
         raise NotAuthenticated(NotAuthenticated.CONNECTION_REFUSED, res.getcode())
     response = parse_response(res.read())

From python-checkins at python.org  Fri Nov 25 01:21:51 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 25 Nov 2011 01:21:51 +0100 (CET)
Subject: [Python-checkins] r88921 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqJ4H3fLbzNrt@mail.python.org>

Author: martin.v.loewis
Date: Fri Nov 25 01:21:51 2011
New Revision: 88921

Log:
Fix typo.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Fri Nov 25 01:21:51 2011
@@ -47,7 +47,7 @@
         def getcode(self):
             return self.code
 
-    class FancyURLOpener(urllib.FancyURLopener):
+    class FancyURLopener(urllib.FancyURLopener):
         def http_error_default(self, url, fp, errcode, errmsg, headers):
             return _addinfourl(fp, headers, "http:" + url, errcode)
 else:

From python-checkins at python.org  Fri Nov 25 01:27:46 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 25 Nov 2011 01:27:46 +0100 (CET)
Subject: [Python-checkins] r88922 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqJC640MwzNZT@mail.python.org>

Author: martin.v.loewis
Date: Fri Nov 25 01:27:46 2011
New Revision: 88922

Log:
More typos.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Fri Nov 25 01:27:46 2011
@@ -42,7 +42,7 @@
     # Add getcode into 2.5
     class _addinfourl(urllib.addinfourl):
         def __init__(self, fp, headers, url, code=None):
-            urllib.addinfourl(self, fp, headers, url)
+            urllib.addinfourl.__init__(self, fp, headers, url)
             self.code = code
         def getcode(self):
             return self.code

From python-checkins at python.org  Fri Nov 25 01:36:04 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 25 Nov 2011 01:36:04 +0100 (CET)
Subject: [Python-checkins] r88923 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqJNh4bHZzPDv@mail.python.org>

Author: martin.v.loewis
Date: Fri Nov 25 01:36:04 2011
New Revision: 88923

Log:
Use FancyURLopener correctly.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Fri Nov 25 01:36:04 2011
@@ -435,7 +435,7 @@
         if data['openid.session_type'] == "no-encryption":
             data['openid.session_type'] = ''
         del data['openid.ns']
-    res = FancyURLopener(url, b(urllib.urlencode(data)))
+    res = FancyURLopener().open(url, b(urllib.urlencode(data)))
     if res.getcode() != 200:
         raise ValueError, "OpenID provider refuses connection with status %d" % res.getcode()
     data = parse_response(res.read())
@@ -567,7 +567,7 @@
     # Exact copies of all fields from the authentication response, except for
     # "openid.mode"
     request.extend((k, v) for k, (v,) in response.items() if 'openid.mode' != k)
-    res = FancyURLopener(op_endpoint, urllib.urlencode(request))
+    res = FancyURLopener().open(op_endpoint, urllib.urlencode(request))
     if 200 != res.getcode():
         raise NotAuthenticated(NotAuthenticated.CONNECTION_REFUSED, res.getcode())
     response = parse_response(res.read())

From python-checkins at python.org  Fri Nov 25 01:38:58 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 25 Nov 2011 01:38:58 +0100 (CET)
Subject: [Python-checkins] r88924 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqJS26jtZzPFs@mail.python.org>

Author: martin.v.loewis
Date: Fri Nov 25 01:38:58 2011
New Revision: 88924

Log:
Try to cope with base addinfourl objects.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Fri Nov 25 01:38:58 2011
@@ -436,7 +436,7 @@
             data['openid.session_type'] = ''
         del data['openid.ns']
     res = FancyURLopener().open(url, b(urllib.urlencode(data)))
-    if res.getcode() != 200:
+    if hasttr(res, 'getcode') and res.getcode() != 200:
         raise ValueError, "OpenID provider refuses connection with status %d" % res.getcode()
     data = parse_response(res.read())
     if 'error' in data:
@@ -568,7 +568,7 @@
     # "openid.mode"
     request.extend((k, v) for k, (v,) in response.items() if 'openid.mode' != k)
     res = FancyURLopener().open(op_endpoint, urllib.urlencode(request))
-    if 200 != res.getcode():
+    if hasttr(res, 'getcode') and 200 != res.getcode():
         raise NotAuthenticated(NotAuthenticated.CONNECTION_REFUSED, res.getcode())
     response = parse_response(res.read())
     if 'true' != response['is_valid']:

From python-checkins at python.org  Fri Nov 25 01:40:17 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Fri, 25 Nov 2011 01:40:17 +0100 (CET)
Subject: [Python-checkins] r88925 -
	tracker/instances/python-dev/lib/openid2rp.py
Message-ID: <3SqJTY0kXPzP36@mail.python.org>

Author: martin.v.loewis
Date: Fri Nov 25 01:40:16 2011
New Revision: 88925

Log:
Fix typo.


Modified:
   tracker/instances/python-dev/lib/openid2rp.py

Modified: tracker/instances/python-dev/lib/openid2rp.py
==============================================================================
--- tracker/instances/python-dev/lib/openid2rp.py	(original)
+++ tracker/instances/python-dev/lib/openid2rp.py	Fri Nov 25 01:40:16 2011
@@ -436,7 +436,7 @@
             data['openid.session_type'] = ''
         del data['openid.ns']
     res = FancyURLopener().open(url, b(urllib.urlencode(data)))
-    if hasttr(res, 'getcode') and res.getcode() != 200:
+    if hasattr(res, 'getcode') and res.getcode() != 200:
         raise ValueError, "OpenID provider refuses connection with status %d" % res.getcode()
     data = parse_response(res.read())
     if 'error' in data:
@@ -568,7 +568,7 @@
     # "openid.mode"
     request.extend((k, v) for k, (v,) in response.items() if 'openid.mode' != k)
     res = FancyURLopener().open(op_endpoint, urllib.urlencode(request))
-    if hasttr(res, 'getcode') and 200 != res.getcode():
+    if hasattr(res, 'getcode') and 200 != res.getcode():
         raise NotAuthenticated(NotAuthenticated.CONNECTION_REFUSED, res.getcode())
     response = parse_response(res.read())
     if 'true' != response['is_valid']:

From solipsis at pitrou.net  Fri Nov 25 05:33:10 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Fri, 25 Nov 2011 05:33:10 +0100
Subject: [Python-checkins] Daily reference leaks (736b0aec412b): sum=0
Message-ID: <E1RTnTK-0008Fi-E3@ap.vmr.nerim.net>

results for 736b0aec412b on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogvF3__4', '-x']

From python-checkins at python.org  Fri Nov 25 14:09:58 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 25 Nov 2011 14:09:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_fix_some_typos_?=
 =?utf8?q?in_Doc/c-api/memoryview=2Erst?=
Message-ID: <E1RTvXS-0002sk-7N@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f2e10596332
changeset:   73733:0f2e10596332
branch:      3.2
parent:      73730:f29d256be73d
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 25 15:07:50 2011 +0200
summary:
  fix some typos in Doc/c-api/memoryview.rst

files:
  Doc/c-api/memoryview.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst
--- a/Doc/c-api/memoryview.rst
+++ b/Doc/c-api/memoryview.rst
@@ -17,7 +17,7 @@
 
    Create a memoryview object from an object that provides the buffer interface.
    If *obj* supports writable buffer exports, the memoryview object will be
-   readable and writable, other it will be read-only.
+   readable and writable, otherwise it will be read-only.
 
 
 .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view)
@@ -33,7 +33,7 @@
    Create a memoryview object to a contiguous chunk of memory (in either
    'C' or 'F'ortran *order*) from an object that defines the buffer
    interface. If memory is contiguous, the memoryview object points to the
-   original memory. Otherwise copy is made and the memoryview points to a
+   original memory. Otherwise, a copy is made and the memoryview points to a
    new bytes object.
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 14:09:58 2011
From: python-checkins at python.org (eli.bendersky)
Date: Fri, 25 Nov 2011 14:09:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_fix_some_typos_in_Doc/c-api/memoryview=2Erst?=
Message-ID: <E1RTvXS-0002su-UZ@dinsdale.python.org>

http://hg.python.org/cpython/rev/db8a14e02342
changeset:   73734:db8a14e02342
parent:      73732:736b0aec412b
parent:      73733:0f2e10596332
user:        Eli Bendersky <eliben at gmail.com>
date:        Fri Nov 25 15:08:45 2011 +0200
summary:
  fix some typos in Doc/c-api/memoryview.rst

files:
  Doc/c-api/memoryview.rst |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst
--- a/Doc/c-api/memoryview.rst
+++ b/Doc/c-api/memoryview.rst
@@ -17,7 +17,7 @@
 
    Create a memoryview object from an object that provides the buffer interface.
    If *obj* supports writable buffer exports, the memoryview object will be
-   readable and writable, other it will be read-only.
+   readable and writable, otherwise it will be read-only.
 
 
 .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view)
@@ -33,7 +33,7 @@
    Create a memoryview object to a contiguous chunk of memory (in either
    'C' or 'F'ortran *order*) from an object that defines the buffer
    interface. If memory is contiguous, the memoryview object points to the
-   original memory. Otherwise copy is made and the memoryview points to a
+   original memory. Otherwise, a copy is made and the memoryview points to a
    new bytes object.
 
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 16:23:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 25 Nov 2011 16:23:59 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Don=E2=80=99t_display_sk?=
 =?utf8?q?ip_messages_for_tests_that_we=E2=80=99ll_never_fix?=
Message-ID: <E1RTxd9-0006Dh-Ib@dinsdale.python.org>

http://hg.python.org/distutils2/rev/1b8f84f165c3
changeset:   1264:1b8f84f165c3
parent:      1262:a382d149a0f0
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Nov 23 13:50:55 2011 +0100
summary:
  Don?t display skip messages for tests that we?ll never fix

files:
  distutils2/_backport/tests/test_shutil.py |  13 +++++-----
  1 files changed, 7 insertions(+), 6 deletions(-)


diff --git a/distutils2/_backport/tests/test_shutil.py b/distutils2/_backport/tests/test_shutil.py
--- a/distutils2/_backport/tests/test_shutil.py
+++ b/distutils2/_backport/tests/test_shutil.py
@@ -917,8 +917,9 @@
 
         self.assertRaises(IOError, shutil.copyfile, 'srcfile', 'destfile')
 
-    @unittest.skip("can't use the with statement and support 2.4")
-    def test_w_dest_open_fails(self):
+    # can't test usage of the with statement and support 2.4, so this
+    # test is renamed to disable it
+    def _test_w_dest_open_fails(self):
 
         srcfile = self.Faux()
 
@@ -937,8 +938,8 @@
         self.assertEqual(srcfile._exited_with[1].args,
                          ('Cannot open "destfile"',))
 
-    @unittest.skip("can't use the with statement and support 2.4")
-    def test_w_dest_close_fails(self):
+    # see above
+    def _test_w_dest_close_fails(self):
 
         srcfile = self.Faux()
         destfile = self.Faux(True)
@@ -960,8 +961,8 @@
         self.assertEqual(srcfile._exited_with[1].args,
                          ('Cannot close',))
 
-    @unittest.skip("can't use the with statement and support 2.4")
-    def test_w_source_close_fails(self):
+    # see above
+    def _test_w_source_close_fails(self):
 
         srcfile = self.Faux(True)
         destfile = self.Faux()

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Fri Nov 25 16:23:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 25 Nov 2011 16:23:59 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Kludge_for_hashlib?=
Message-ID: <E1RTxd9-0006Dq-J2@dinsdale.python.org>

http://hg.python.org/distutils2/rev/4bc6cdecea33
changeset:   1265:4bc6cdecea33
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 25 12:42:35 2011 +0100
summary:
  Kludge for hashlib

files:
  setup.py |  9 +++++++--
  1 files changed, 7 insertions(+), 2 deletions(-)


diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -194,8 +194,13 @@
                               sources=['distutils2/_backport/md5module.c',
                                        'distutils2/_backport/md5.c'],
                               depends=['distutils2/_backport/md5.h']) )
-    if openssl_ver < 0x908000:
-        # OpenSSL doesn't do these until 0.9.8 so we'll bring our own
+    # XXX always compile sha256 and sha512 to have a working hashlib (maybe
+    # I (merwok) can't compile _ssl and thus _hashlib for 2.4, maybe because of
+    # Debian multiarch, even though I set all needed build vars (and can
+    # compile _ssl for 2.6 for example)
+    if True:
+    #if openssl_ver < 0x908000:
+    #    # OpenSSL doesn't do these until 0.9.8 so we'll bring our own
         exts.append(Extension('distutils2._backport._sha256',
                               ['distutils2/_backport/sha256module.c']))
         exts.append(Extension('distutils2._backport._sha512',

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Fri Nov 25 16:23:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 25 Nov 2011 16:23:59 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Remove_duplicate_script?=
Message-ID: <E1RTxd9-0006Dv-Kc@dinsdale.python.org>

http://hg.python.org/distutils2/rev/3f8b40403811
changeset:   1266:3f8b40403811
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 25 12:54:13 2011 +0100
summary:
  Remove duplicate script

files:
  pysetup         |  5 ++---
  scripts/pysetup |  4 ----
  2 files changed, 2 insertions(+), 7 deletions(-)


diff --git a/pysetup b/pysetup
--- a/pysetup
+++ b/pysetup
@@ -1,5 +1,4 @@
 #!/usr/bin/env python
+import sys
 from distutils2.run import main
-
-if __name__ == "__main__":
-    main()
+sys.exit(main())
diff --git a/scripts/pysetup b/scripts/pysetup
deleted file mode 100644
--- a/scripts/pysetup
+++ /dev/null
@@ -1,4 +0,0 @@
-#! /usr/bin/env python
-import sys
-from distutils2.run import main
-sys.exit(main())

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Fri Nov 25 16:23:59 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 25 Nov 2011 16:23:59 +0100
Subject: [Python-checkins] =?utf8?q?distutils2_=28merge_default_-=3E_pytho?=
	=?utf8?q?n3=29=3A_Merge_cleanups?=
Message-ID: <E1RTxd9-0006Dy-LA@dinsdale.python.org>

http://hg.python.org/distutils2/rev/bcea7cbddd47
changeset:   1267:bcea7cbddd47
branch:      python3
parent:      1263:7e7f23d10484
parent:      1266:3f8b40403811
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 25 12:57:30 2011 +0100
summary:
  Merge cleanups

files:
  pysetup         |  5 ++---
  scripts/pysetup |  4 ----
  2 files changed, 2 insertions(+), 7 deletions(-)


diff --git a/pysetup b/pysetup
--- a/pysetup
+++ b/pysetup
@@ -1,5 +1,4 @@
 #!/usr/bin/env python3
+import sys
 from distutils2.run import main
-
-if __name__ == "__main__":
-    main()
+sys.exit(main())
diff --git a/scripts/pysetup b/scripts/pysetup
deleted file mode 100644
--- a/scripts/pysetup
+++ /dev/null
@@ -1,4 +0,0 @@
-#! /usr/bin/env python
-import sys
-from distutils2.run import main
-sys.exit(main())

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Fri Nov 25 16:39:25 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 16:39:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Clarify_concate?=
 =?utf8?q?nation_behaviour_of_immutable_strings=2C_and_remove_explicit?=
Message-ID: <E1RTxs5-0007Zz-Bb@dinsdale.python.org>

http://hg.python.org/cpython/rev/2d6f0e2fe034
changeset:   73735:2d6f0e2fe034
branch:      3.2
parent:      73733:0f2e10596332
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 16:33:53 2011 +0100
summary:
  Clarify concatenation behaviour of immutable strings, and remove explicit
mention of the CPython optimization hack.

files:
  Doc/faq/programming.rst  |  26 ++++++++++++++++++++++++++
  Doc/library/stdtypes.rst |  21 ++++++++++++---------
  2 files changed, 38 insertions(+), 9 deletions(-)


diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -989,6 +989,32 @@
 See the :ref:`unicode-howto`.
 
 
+What is the most efficient way to concatenate many strings together?
+--------------------------------------------------------------------
+
+:class:`str` and :class:`bytes` objects are immutable, therefore concatenating
+many strings together is inefficient as each concatenation creates a new
+object.  In the general case, the total runtime cost is quadratic in the
+total string length.
+
+To accumulate many :class:`str` objects, the recommended idiom is to place
+them into a list and call :meth:`str.join` at the end::
+
+   chunks = []
+   for s in my_strings:
+       chunks.append(s)
+   result = ''.join(chunks)
+
+(another reasonably efficient idiom is to use :class:`io.StringIO`)
+
+To accumulate many :class:`bytes` objects, the recommended idiom is to extend
+a :class:`bytearray` object using in-place concatenation (the ``+=`` operator)::
+
+   result = bytearray()
+   for b in my_bytes_objects:
+       result += b
+
+
 Sequences (Tuples/Lists)
 ========================
 
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -964,15 +964,18 @@
    If *k* is ``None``, it is treated like ``1``.
 
 (6)
-   .. impl-detail::
-
-      If *s* and *t* are both strings, some Python implementations such as
-      CPython can usually perform an in-place optimization for assignments of
-      the form ``s = s + t`` or ``s += t``.  When applicable, this optimization
-      makes quadratic run-time much less likely.  This optimization is both
-      version and implementation dependent.  For performance sensitive code, it
-      is preferable to use the :meth:`str.join` method which assures consistent
-      linear concatenation performance across versions and implementations.
+   Concatenating immutable strings always results in a new object.  This means
+   that building up a string by repeated concatenation will have a quadratic
+   runtime cost in the total string length.  To get a linear runtime cost,
+   you must switch to one of the alternatives below:
+
+   * if concatenating :class:`str` objects, you can build a list and use
+     :meth:`str.join` at the end;
+
+   * if concatenating :class:`bytes` objects, you can similarly use
+     :meth:`bytes.join`, or you can do in-place concatenation with a
+     :class:`bytearray` object.  :class:`bytearray` objects are mutable and
+     have an efficient overallocation mechanism.
 
 
 .. _string-methods:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 16:39:26 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 16:39:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Clarify_concatenation_behaviour_of_immutable_strings=2C_and_?=
 =?utf8?q?remove_explicit?=
Message-ID: <E1RTxs6-0007a9-8A@dinsdale.python.org>

http://hg.python.org/cpython/rev/905b6f1eca74
changeset:   73736:905b6f1eca74
parent:      73734:db8a14e02342
parent:      73735:2d6f0e2fe034
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 16:34:23 2011 +0100
summary:
  Clarify concatenation behaviour of immutable strings, and remove explicit
mention of the CPython optimization hack.

files:
  Doc/faq/programming.rst  |  26 ++++++++++++++++++++++++++
  Doc/library/stdtypes.rst |  21 ++++++++++++---------
  2 files changed, 38 insertions(+), 9 deletions(-)


diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -989,6 +989,32 @@
 See the :ref:`unicode-howto`.
 
 
+What is the most efficient way to concatenate many strings together?
+--------------------------------------------------------------------
+
+:class:`str` and :class:`bytes` objects are immutable, therefore concatenating
+many strings together is inefficient as each concatenation creates a new
+object.  In the general case, the total runtime cost is quadratic in the
+total string length.
+
+To accumulate many :class:`str` objects, the recommended idiom is to place
+them into a list and call :meth:`str.join` at the end::
+
+   chunks = []
+   for s in my_strings:
+       chunks.append(s)
+   result = ''.join(chunks)
+
+(another reasonably efficient idiom is to use :class:`io.StringIO`)
+
+To accumulate many :class:`bytes` objects, the recommended idiom is to extend
+a :class:`bytearray` object using in-place concatenation (the ``+=`` operator)::
+
+   result = bytearray()
+   for b in my_bytes_objects:
+       result += b
+
+
 Sequences (Tuples/Lists)
 ========================
 
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -968,15 +968,18 @@
    If *k* is ``None``, it is treated like ``1``.
 
 (6)
-   .. impl-detail::
-
-      If *s* and *t* are both strings, some Python implementations such as
-      CPython can usually perform an in-place optimization for assignments of
-      the form ``s = s + t`` or ``s += t``.  When applicable, this optimization
-      makes quadratic run-time much less likely.  This optimization is both
-      version and implementation dependent.  For performance sensitive code, it
-      is preferable to use the :meth:`str.join` method which assures consistent
-      linear concatenation performance across versions and implementations.
+   Concatenating immutable strings always results in a new object.  This means
+   that building up a string by repeated concatenation will have a quadratic
+   runtime cost in the total string length.  To get a linear runtime cost,
+   you must switch to one of the alternatives below:
+
+   * if concatenating :class:`str` objects, you can build a list and use
+     :meth:`str.join` at the end;
+
+   * if concatenating :class:`bytes` objects, you can similarly use
+     :meth:`bytes.join`, or you can do in-place concatenation with a
+     :class:`bytearray` object.  :class:`bytearray` objects are mutable and
+     have an efficient overallocation mechanism.
 
 
 .. _string-methods:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 17:44:10 2011
From: python-checkins at python.org (eric.araujo)
Date: Fri, 25 Nov 2011 17:44:10 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Fix_nesting_of_the_numerical_i?=
	=?utf8?b?bmRleCAoIzEyMzA3KS4=?=
Message-ID: <E1RTysk-0004lt-1J@dinsdale.python.org>

http://hg.python.org/peps/rev/6c7415a4f0f3
changeset:   4003:6c7415a4f0f3
user:        ?ric Araujo <merwok at netwok.org>
date:        Fri Nov 25 17:43:36 2011 +0100
summary:
  Fix nesting of the numerical index (#12307).

This extra space caused the numerical index to be parsed and displayed as a
subsection of the index by category instead of being an element at the same
level.  Patch by Eric Snow.

files:
  pep0/output.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep0/output.py b/pep0/output.py
--- a/pep0/output.py
+++ b/pep0/output.py
@@ -169,7 +169,7 @@
         print>>output, unicode(pep)
     print>>output
     print>>output
-    print>>output, u" Numerical Index"
+    print>>output, u"Numerical Index"
     print>>output
     write_column_headers(output)
     prev_pep = 0

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Fri Nov 25 18:08:10 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 18:08:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=239957=3A_SpooledTem?=
 =?utf8?q?poraryFile=2Etruncate=28=29_now_accepts_an_optional_size?=
Message-ID: <E1RTzFy-0006pA-61@dinsdale.python.org>

http://hg.python.org/cpython/rev/5a6911930bad
changeset:   73737:5a6911930bad
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 18:03:09 2011 +0100
summary:
  Issue #9957: SpooledTemporaryFile.truncate() now accepts an optional size parameter, as other file-like objects.
Patch by Ryan Kelly.

files:
  Lib/tempfile.py           |   9 +++++++--
  Lib/test/test_tempfile.py |  21 +++++++++++++++++++++
  Misc/NEWS                 |   4 +++-
  3 files changed, 31 insertions(+), 3 deletions(-)


diff --git a/Lib/tempfile.py b/Lib/tempfile.py
--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -578,8 +578,13 @@
     def tell(self):
         return self._file.tell()
 
-    def truncate(self):
-        self._file.truncate()
+    def truncate(self, size=None):
+        if size is None:
+            self._file.truncate()
+        else:
+            if size > self._max_size:
+                self.rollover()
+            self._file.truncate(size)
 
     def write(self, s):
         file = self._file
diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py
--- a/Lib/test/test_tempfile.py
+++ b/Lib/test/test_tempfile.py
@@ -846,6 +846,27 @@
                 pass
         self.assertRaises(ValueError, use_closed)
 
+    def test_truncate_with_size_parameter(self):
+        # A SpooledTemporaryFile can be truncated to zero size
+        f = tempfile.SpooledTemporaryFile(max_size=10)
+        f.write(b'abcdefg\n')
+        f.seek(0)
+        f.truncate()
+        self.assertFalse(f._rolled)
+        self.assertEqual(f._file.getvalue(), b'')
+        # A SpooledTemporaryFile can be truncated to a specific size
+        f = tempfile.SpooledTemporaryFile(max_size=10)
+        f.write(b'abcdefg\n')
+        f.truncate(4)
+        self.assertFalse(f._rolled)
+        self.assertEqual(f._file.getvalue(), b'abcd')
+        # A SpooledTemporaryFile rolls over if truncated to large size
+        f = tempfile.SpooledTemporaryFile(max_size=10)
+        f.write(b'abcdefg\n')
+        f.truncate(20)
+        self.assertTrue(f._rolled)
+        if has_stat:
+            self.assertEqual(os.fstat(f.fileno()).st_size, 20)
 
 test_classes.append(test_SpooledTemporaryFile)
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -386,10 +386,12 @@
 - Issue #12380: The rjust, ljust and center methods of bytes and bytearray
   now accept a bytearray argument.
 
-
 Library
 -------
 
+- Issue #9957: SpooledTemporaryFile.truncate() now accepts an optional size
+  parameter, as other file-like objects.  Patch by Ryan Kelly.
+
 - Issue #13458: Fix a memory leak in the ssl module when decoding a
   certificate with a subjectAltName.  Patch by Robert Xiao.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 19:03:54 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 19:03:54 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_PEP_3155_/_issue_=2313448?=
 =?utf8?q?=3A_Qualified_name_for_classes_and_functions=2E?=
Message-ID: <E1RU07u-0002xL-91@dinsdale.python.org>

http://hg.python.org/cpython/rev/e1dbc72bd97f
changeset:   73738:e1dbc72bd97f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 18:56:07 2011 +0100
summary:
  PEP 3155 / issue #13448: Qualified name for classes and functions.

files:
  Doc/c-api/function.rst      |   10 +
  Doc/data/refcounts.dat      |    5 +
  Doc/glossary.rst            |   18 +++
  Doc/library/stdtypes.rst    |    7 +
  Doc/reference/datamodel.rst |    5 +
  Include/funcobject.h        |    2 +
  Include/object.h            |    2 +-
  Lib/pydoc.py                |    2 +-
  Lib/test/test_code.py       |    2 +-
  Lib/test/test_descr.py      |   11 +-
  Lib/test/test_dis.py        |    1 +
  Lib/test/test_funcattrs.py  |   27 ++++
  Lib/test/test_metaclass.py  |    8 +-
  Lib/test/test_reprlib.py    |   13 +-
  Lib/test/test_sys.py        |    4 +-
  Misc/NEWS                   |    2 +
  Objects/funcobject.c        |   44 +++++++-
  Objects/typeobject.c        |   56 +++++++++-
  Python/ceval.c              |    4 +-
  Python/compile.c            |  139 ++++++++++++++++++++---
  Python/import.c             |    3 +-
  21 files changed, 322 insertions(+), 43 deletions(-)


diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -38,6 +38,16 @@
    object, the argument defaults and closure are set to *NULL*.
 
 
+.. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
+
+   As :c:func:`PyFunction_New`, but also allows to set the function object's
+   ``__qualname__`` attribute.  *qualname* should be a unicode object or NULL;
+   if NULL, the ``__qualname__`` attribute is set to the same value as its
+   ``__name__`` attribute.
+
+   .. versionadded:: 3.3
+
+
 .. c:function:: PyObject* PyFunction_GetCode(PyObject *op)
 
    Return the code object associated with the function object *op*.
diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat
--- a/Doc/data/refcounts.dat
+++ b/Doc/data/refcounts.dat
@@ -465,6 +465,11 @@
 PyFunction_New:PyObject*:code:+1:
 PyFunction_New:PyObject*:globals:+1:
 
+PyFunction_NewWithQualName:PyObject*::+1:
+PyFunction_NewWithQualName:PyObject*:code:+1:
+PyFunction_NewWithQualName:PyObject*:globals:+1:
+PyFunction_NewWithQualName:PyObject*:qualname:+1:
+
 PyFunction_SetClosure:int:::
 PyFunction_SetClosure:PyObject*:op:0:
 PyFunction_SetClosure:PyObject*:closure:+1:
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -544,6 +544,24 @@
          for piece in food:
              print(piece)
 
+   qualified name
+      A dotted name showing the "path" from a module's global scope to a
+      class, function or method defined in that module, as defined in
+      :pep:`3155`.  For top-level functions and classes, the qualified name
+      is the same as the object's name::
+
+         >>> class C:
+         ...     class D:
+         ...         def meth(self):
+         ...             pass
+         ...
+         >>> C.__qualname__
+         'C'
+         >>> C.D.__qualname__
+         'C.D'
+         >>> C.D.meth.__qualname__
+         'C.D.meth'
+
    reference count
       The number of references to an object.  When the reference count of an
       object drops to zero, it is deallocated.  Reference counting is
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -2824,6 +2824,13 @@
    The name of the class or type.
 
 
+.. attribute:: class.__qualname__
+
+   The :term:`qualified name` of the class or type.
+
+   .. versionadded:: 3.3
+
+
 .. attribute:: class.__mro__
 
    This attribute is a tuple of classes that are considered when looking for
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -448,6 +448,11 @@
       +-------------------------+-------------------------------+-----------+
       | :attr:`__name__`        | The function's name           | Writable  |
       +-------------------------+-------------------------------+-----------+
+      | :attr:`__qualname__`    | The function's                | Writable  |
+      |                         | :term:`qualified name`        |           |
+      |                         |                               |           |
+      |                         | .. versionadded:: 3.3         |           |
+      +-------------------------+-------------------------------+-----------+
       | :attr:`__module__`      | The name of the module the    | Writable  |
       |                         | function was defined in, or   |           |
       |                         | ``None`` if unavailable.      |           |
diff --git a/Include/funcobject.h b/Include/funcobject.h
--- a/Include/funcobject.h
+++ b/Include/funcobject.h
@@ -31,6 +31,7 @@
     PyObject *func_weakreflist;	/* List of weak references */
     PyObject *func_module;	/* The __module__ attribute, can be anything */
     PyObject *func_annotations;	/* Annotations, a dict or NULL */
+    PyObject *func_qualname;    /* The qualified name */
 
     /* Invariant:
      *     func_closure contains the bindings for func_code->co_freevars, so
@@ -44,6 +45,7 @@
 #define PyFunction_Check(op) (Py_TYPE(op) == &PyFunction_Type)
 
 PyAPI_FUNC(PyObject *) PyFunction_New(PyObject *, PyObject *);
+PyAPI_FUNC(PyObject *) PyFunction_NewWithQualName(PyObject *, PyObject *, PyObject *);
 PyAPI_FUNC(PyObject *) PyFunction_GetCode(PyObject *);
 PyAPI_FUNC(PyObject *) PyFunction_GetGlobals(PyObject *);
 PyAPI_FUNC(PyObject *) PyFunction_GetModule(PyObject *);
diff --git a/Include/object.h b/Include/object.h
--- a/Include/object.h
+++ b/Include/object.h
@@ -418,7 +418,7 @@
                                       a given operator (e.g. __getitem__).
                                       see add_operators() in typeobject.c . */
     PyBufferProcs as_buffer;
-    PyObject *ht_name, *ht_slots;
+    PyObject *ht_name, *ht_slots, *ht_qualname;
     /* here are optional user slots, followed by the members. */
 } PyHeapTypeObject;
 
diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -166,7 +166,7 @@
     if name in {'__builtins__', '__doc__', '__file__', '__path__',
                      '__module__', '__name__', '__slots__', '__package__',
                      '__cached__', '__author__', '__credits__', '__date__',
-                     '__version__'}:
+                     '__version__', '__qualname__'}:
         return 0
     # Private names are hidden, but special names are displayed.
     if name.startswith('__') and name.endswith('__'): return 1
diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py
--- a/Lib/test/test_code.py
+++ b/Lib/test/test_code.py
@@ -16,7 +16,7 @@
 freevars: ()
 nlocals: 2
 flags: 3
-consts: ('None', '<code object g>')
+consts: ('None', '<code object g>', "'f.<locals>.g'")
 
 >>> dump(f(4).__code__)
 name: g
diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py
--- a/Lib/test/test_descr.py
+++ b/Lib/test/test_descr.py
@@ -4492,9 +4492,14 @@
         self.assertEqual(type(C.__dict__), type(B.__dict__))
 
     def test_repr(self):
-        # Testing dict_proxy.__repr__
-        dict_ = {k: v for k, v in self.C.__dict__.items()}
-        self.assertEqual(repr(self.C.__dict__), 'dict_proxy({!r})'.format(dict_))
+        # Testing dict_proxy.__repr__.
+        # We can't blindly compare with the repr of another dict as ordering
+        # of keys and values is arbitrary and may differ.
+        r = repr(self.C.__dict__)
+        self.assertTrue(r.startswith('dict_proxy('), r)
+        self.assertTrue(r.endswith(')'), r)
+        for k, v in self.C.__dict__.items():
+            self.assertIn('{!r}: {!r}'.format(k, v), r)
 
 
 class PTypesLongInitTest(unittest.TestCase):
diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
--- a/Lib/test/test_dis.py
+++ b/Lib/test/test_dis.py
@@ -339,6 +339,7 @@
 Constants:
    0: None
    1: <code object f at (.*), file "(.*)", line (.*)>
+   2: 'tricky.<locals>.f'
 Variable names:
    0: x
    1: y
diff --git a/Lib/test/test_funcattrs.py b/Lib/test/test_funcattrs.py
--- a/Lib/test/test_funcattrs.py
+++ b/Lib/test/test_funcattrs.py
@@ -2,6 +2,15 @@
 import types
 import unittest
 
+
+def global_function():
+    def inner_function():
+        class LocalClass:
+            pass
+        return LocalClass
+    return lambda: inner_function
+
+
 class FuncAttrsTest(unittest.TestCase):
     def setUp(self):
         class F:
@@ -96,6 +105,24 @@
         self.assertEqual(self.fi.a.__name__, 'a')
         self.cannot_set_attr(self.fi.a, "__name__", 'a', AttributeError)
 
+    def test___qualname__(self):
+        # PEP 3155
+        self.assertEqual(self.b.__qualname__, 'FuncAttrsTest.setUp.<locals>.b')
+        self.assertEqual(FuncAttrsTest.setUp.__qualname__, 'FuncAttrsTest.setUp')
+        self.assertEqual(global_function.__qualname__, 'global_function')
+        self.assertEqual(global_function().__qualname__,
+                         'global_function.<locals>.<lambda>')
+        self.assertEqual(global_function()().__qualname__,
+                         'global_function.<locals>.inner_function')
+        self.assertEqual(global_function()()().__qualname__,
+                         'global_function.<locals>.inner_function.<locals>.LocalClass')
+        self.b.__qualname__ = 'c'
+        self.assertEqual(self.b.__qualname__, 'c')
+        self.b.__qualname__ = 'd'
+        self.assertEqual(self.b.__qualname__, 'd')
+        # __qualname__ must be a string
+        self.cannot_set_attr(self.b, '__qualname__', 7, TypeError)
+
     def test___code__(self):
         num_one, num_two = 7, 8
         def a(): pass
diff --git a/Lib/test/test_metaclass.py b/Lib/test/test_metaclass.py
--- a/Lib/test/test_metaclass.py
+++ b/Lib/test/test_metaclass.py
@@ -159,6 +159,7 @@
     ...     bar = 123
     ...
     d['__module__'] = 'test.test_metaclass'
+    d['__qualname__'] = 'C'
     d['foo'] = 4
     d['foo'] = 42
     d['bar'] = 123
@@ -177,12 +178,12 @@
     ...     b = 24
     ...
     meta: C ()
-    ns: [('__module__', 'test.test_metaclass'), ('a', 42), ('b', 24)]
+    ns: [('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('a', 42), ('b', 24)]
     kw: []
     >>> type(C) is dict
     True
     >>> print(sorted(C.items()))
-    [('__module__', 'test.test_metaclass'), ('a', 42), ('b', 24)]
+    [('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('a', 42), ('b', 24)]
     >>>
 
 And again, with a __prepare__ attribute.
@@ -199,11 +200,12 @@
     ...
     prepare: C () [('other', 'booh')]
     d['__module__'] = 'test.test_metaclass'
+    d['__qualname__'] = 'C'
     d['a'] = 1
     d['a'] = 2
     d['b'] = 3
     meta: C ()
-    ns: [('__module__', 'test.test_metaclass'), ('a', 2), ('b', 3)]
+    ns: [('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('a', 2), ('b', 3)]
     kw: [('other', 'booh')]
     >>>
 
diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py
--- a/Lib/test/test_reprlib.py
+++ b/Lib/test/test_reprlib.py
@@ -129,8 +129,8 @@
         self.assertIn(s.find("..."), [12, 13])
 
     def test_lambda(self):
-        self.assertTrue(repr(lambda x: x).startswith(
-            "<function <lambda"))
+        r = repr(lambda x: x)
+        self.assertTrue(r.startswith("<function ReprTests.test_lambda.<locals>.<lambda"), r)
         # XXX anonymous functions?  see func_repr
 
     def test_builtin_function(self):
@@ -278,13 +278,14 @@
 ''')
         from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import qux
         # Unbound methods first
-        self.assertTrue(repr(qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod).startswith(
-        '<function amethod'))
+        r = repr(qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod)
+        self.assertTrue(r.startswith('<function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod'), r)
         # Bound method next
         iqux = qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
-        self.assertTrue(repr(iqux.amethod).startswith(
+        r = repr(iqux.amethod)
+        self.assertTrue(r.startswith(
             '<bound method aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod of <%s.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa object at 0x' \
-            % (qux.__name__,) ))
+            % (qux.__name__,) ), r)
 
     def test_builtin_function(self):
         # XXX test built-in functions and methods with really long names
diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -730,7 +730,7 @@
         check(x, size(vh + '12P3i' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P'))
         # function
         def func(): pass
-        check(func, size(h + '11P'))
+        check(func, size(h + '12P'))
         class c():
             @staticmethod
             def foo():
@@ -828,7 +828,7 @@
         # type
         # (PyTypeObject + PyNumberMethods + PyMappingMethods +
         #  PySequenceMethods + PyBufferProcs)
-        s = size(vh + 'P2P15Pl4PP9PP11PI') + size('16Pi17P 3P 10P 2P 2P')
+        s = size(vh + 'P2P15Pl4PP9PP11PI') + size('16Pi17P 3P 10P 2P 3P')
         check(int, s)
         # class
         class newstyleclass(object): pass
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,8 @@
 Core and Builtins
 -----------------
 
+- PEP 3155 / issue #13448: Qualified name for classes and functions.
+
 - Issue #13436: Fix a bogus error message when an AST object was passed
   an invalid integer value.
 
diff --git a/Objects/funcobject.c b/Objects/funcobject.c
--- a/Objects/funcobject.c
+++ b/Objects/funcobject.c
@@ -6,7 +6,7 @@
 #include "structmember.h"
 
 PyObject *
-PyFunction_New(PyObject *code, PyObject *globals)
+PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
 {
     PyFunctionObject *op = PyObject_GC_New(PyFunctionObject,
                                         &PyFunction_Type);
@@ -54,6 +54,11 @@
             Py_INCREF(module);
             op->func_module = module;
         }
+        if (qualname)
+            op->func_qualname = qualname;
+        else
+            op->func_qualname = op->func_name;
+        Py_INCREF(op->func_qualname);
     }
     else
         return NULL;
@@ -62,6 +67,12 @@
 }
 
 PyObject *
+PyFunction_New(PyObject *code, PyObject *globals)
+{
+    return PyFunction_NewWithQualName(code, globals, NULL);
+}
+
+PyObject *
 PyFunction_GetCode(PyObject *op)
 {
     if (!PyFunction_Check(op)) {
@@ -334,6 +345,32 @@
 }
 
 static PyObject *
+func_get_qualname(PyFunctionObject *op)
+{
+    Py_INCREF(op->func_qualname);
+    return op->func_qualname;
+}
+
+static int
+func_set_qualname(PyFunctionObject *op, PyObject *value)
+{
+    PyObject *tmp;
+
+    /* Not legal to del f.__qualname__ or to set it to anything
+     * other than a string object. */
+    if (value == NULL || !PyUnicode_Check(value)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "__qualname__ must be set to a string object");
+        return -1;
+    }
+    tmp = op->func_qualname;
+    Py_INCREF(value);
+    op->func_qualname = value;
+    Py_DECREF(tmp);
+    return 0;
+}
+
+static PyObject *
 func_get_defaults(PyFunctionObject *op)
 {
     if (op->func_defaults == NULL) {
@@ -441,6 +478,7 @@
      (setter)func_set_annotations},
     {"__dict__", (getter)func_get_dict, (setter)func_set_dict},
     {"__name__", (getter)func_get_name, (setter)func_set_name},
+    {"__qualname__", (getter)func_get_qualname, (setter)func_set_qualname},
     {NULL} /* Sentinel */
 };
 
@@ -561,6 +599,7 @@
     Py_XDECREF(op->func_dict);
     Py_XDECREF(op->func_closure);
     Py_XDECREF(op->func_annotations);
+    Py_XDECREF(op->func_qualname);
     PyObject_GC_Del(op);
 }
 
@@ -568,7 +607,7 @@
 func_repr(PyFunctionObject *op)
 {
     return PyUnicode_FromFormat("<function %U at %p>",
-                               op->func_name, op);
+                               op->func_qualname, op);
 }
 
 static int
@@ -584,6 +623,7 @@
     Py_VISIT(f->func_dict);
     Py_VISIT(f->func_closure);
     Py_VISIT(f->func_annotations);
+    Py_VISIT(f->func_qualname);
     return 0;
 }
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -242,6 +242,19 @@
     }
 }
 
+static PyObject *
+type_qualname(PyTypeObject *type, void *context)
+{
+    if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) {
+        PyHeapTypeObject* et = (PyHeapTypeObject*)type;
+        Py_INCREF(et->ht_qualname);
+        return et->ht_qualname;
+    }
+    else {
+        return type_name(type, context);
+    }
+}
+
 static int
 type_set_name(PyTypeObject *type, PyObject *value, void *context)
 {
@@ -286,6 +299,25 @@
     return 0;
 }
 
+static int
+type_set_qualname(PyTypeObject *type, PyObject *value, void *context)
+{
+    PyHeapTypeObject* et;
+
+    if (!PyUnicode_Check(value)) {
+        PyErr_Format(PyExc_TypeError,
+                     "can only assign string to %s.__qualname__, not '%s'",
+                     type->tp_name, Py_TYPE(value)->tp_name);
+        return -1;
+    }
+
+    et = (PyHeapTypeObject*)type;
+    Py_INCREF(value);
+    Py_DECREF(et->ht_qualname);
+    et->ht_qualname = value;
+    return 0;
+}
+
 static PyObject *
 type_module(PyTypeObject *type, void *context)
 {
@@ -631,6 +663,7 @@
 
 static PyGetSetDef type_getsets[] = {
     {"__name__", (getter)type_name, (setter)type_set_name, NULL},
+    {"__qualname__", (getter)type_qualname, (setter)type_set_qualname, NULL},
     {"__bases__", (getter)type_get_bases, (setter)type_set_bases, NULL},
     {"__module__", (getter)type_module, (setter)type_set_module, NULL},
     {"__abstractmethods__", (getter)type_abstractmethods,
@@ -652,7 +685,7 @@
         Py_DECREF(mod);
         mod = NULL;
     }
-    name = type_name(type, NULL);
+    name = type_qualname(type, NULL);
     if (name == NULL)
         return NULL;
 
@@ -1955,7 +1988,7 @@
 {
     PyObject *name, *bases, *dict;
     static char *kwlist[] = {"name", "bases", "dict", 0};
-    PyObject *slots, *tmp, *newslots;
+    PyObject *qualname, *slots, *tmp, *newslots;
     PyTypeObject *type, *base, *tmptype, *winner;
     PyHeapTypeObject *et;
     PyMemberDef *mp;
@@ -2032,6 +2065,18 @@
         return NULL;
     }
 
+    /* Check for a __qualname__ variable in dict */
+    qualname = PyDict_GetItemString(dict, "__qualname__");
+    if (qualname == NULL) {
+        qualname = name;
+    }
+    else {
+        if (PyDict_DelItemString(dict, "__qualname__") < 0) {
+            Py_DECREF(bases);
+            return NULL;
+        }
+    }
+
     /* Check for a __slots__ sequence variable in dict, and count it */
     slots = PyDict_GetItemString(dict, "__slots__");
     nslots = 0;
@@ -2185,7 +2230,9 @@
     /* Keep name and slots alive in the extended type object */
     et = (PyHeapTypeObject *)type;
     Py_INCREF(name);
+    Py_INCREF(qualname);
     et->ht_name = name;
+    et->ht_qualname = qualname;
     et->ht_slots = slots;
 
     /* Initialize tp_flags */
@@ -2369,6 +2416,8 @@
     res->ht_name = PyUnicode_FromString(spec->name);
     if (!res->ht_name)
         goto fail;
+    res->ht_qualname = res->ht_name;
+    Py_INCREF(res->ht_qualname);
     res->ht_type.tp_name = _PyUnicode_AsString(res->ht_name);
     if (!res->ht_type.tp_name)
         goto fail;
@@ -2568,6 +2617,7 @@
      */
     PyObject_Free((char *)type->tp_doc);
     Py_XDECREF(et->ht_name);
+    Py_XDECREF(et->ht_qualname);
     Py_XDECREF(et->ht_slots);
     Py_TYPE(type)->tp_free((PyObject *)type);
 }
@@ -2983,7 +3033,7 @@
         Py_DECREF(mod);
         mod = NULL;
     }
-    name = type_name(type, NULL);
+    name = type_qualname(type, NULL);
     if (name == NULL)
         return NULL;
     if (mod != NULL && PyUnicode_CompareWithASCIIString(mod, "builtins"))
diff --git a/Python/ceval.c b/Python/ceval.c
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -2687,9 +2687,11 @@
             int kwdefaults = (oparg>>8) & 0xff;
             int num_annotations = (oparg >> 16) & 0x7fff;
 
+            w = POP(); /* qualname */
             v = POP(); /* code object */
-            x = PyFunction_New(v, f->f_globals);
+            x = PyFunction_NewWithQualName(v, f->f_globals, w);
             Py_DECREF(v);
+            Py_DECREF(w);
 
             if (x != NULL && opcode == MAKE_CLOSURE) {
                 v = POP();
diff --git a/Python/compile.c b/Python/compile.c
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -90,6 +90,13 @@
     basicblock *fb_block;
 };
 
+enum {
+    COMPILER_SCOPE_MODULE,
+    COMPILER_SCOPE_CLASS,
+    COMPILER_SCOPE_FUNCTION,
+    COMPILER_SCOPE_COMPREHENSION,
+};
+
 /* The following items change on entry and exit of code blocks.
    They must be saved and restored when returning to a block.
 */
@@ -97,6 +104,9 @@
     PySTEntryObject *u_ste;
 
     PyObject *u_name;
+    PyObject *u_qualname;  /* dot-separated qualified name (lazy) */
+    int u_scope_type;
+
     /* The following fields are dicts that map objects to
        the index of them in co_XXX.      The index is used as
        the argument for opcodes that refer to those collections.
@@ -149,7 +159,7 @@
     PyArena *c_arena;            /* pointer to memory allocation arena */
 };
 
-static int compiler_enter_scope(struct compiler *, identifier, void *, int);
+static int compiler_enter_scope(struct compiler *, identifier, int, void *, int);
 static void compiler_free(struct compiler *);
 static basicblock *compiler_new_block(struct compiler *);
 static int compiler_next_instr(struct compiler *, basicblock *);
@@ -457,6 +467,7 @@
     }
     Py_CLEAR(u->u_ste);
     Py_CLEAR(u->u_name);
+    Py_CLEAR(u->u_qualname);
     Py_CLEAR(u->u_consts);
     Py_CLEAR(u->u_names);
     Py_CLEAR(u->u_varnames);
@@ -467,8 +478,8 @@
 }
 
 static int
-compiler_enter_scope(struct compiler *c, identifier name, void *key,
-                     int lineno)
+compiler_enter_scope(struct compiler *c, identifier name,
+                     int scope_type, void *key, int lineno)
 {
     struct compiler_unit *u;
 
@@ -479,6 +490,7 @@
         return 0;
     }
     memset(u, 0, sizeof(struct compiler_unit));
+    u->u_scope_type = scope_type;
     u->u_argcount = 0;
     u->u_kwonlyargcount = 0;
     u->u_ste = PySymtable_Lookup(c->c_st, key);
@@ -566,6 +578,59 @@
 
 }
 
+static PyObject *
+compiler_scope_qualname(struct compiler *c)
+{
+    Py_ssize_t stack_size, i;
+    _Py_static_string(dot, ".");
+    _Py_static_string(locals, "<locals>");
+    struct compiler_unit *u;
+    PyObject *capsule, *name, *seq, *dot_str, *locals_str;
+
+    u = c->u;
+    if (u->u_qualname != NULL) {
+        Py_INCREF(u->u_qualname);
+        return u->u_qualname;
+    }
+
+    seq = PyList_New(0);
+    if (seq == NULL)
+        return NULL;
+
+    stack_size = PyList_GET_SIZE(c->c_stack);
+    for (i = 0; i < stack_size; i++) {
+        capsule = PyList_GET_ITEM(c->c_stack, i);
+        u = (struct compiler_unit *)PyCapsule_GetPointer(capsule, COMPILER_CAPSULE_NAME_COMPILER_UNIT);
+        assert(u);
+        if (u->u_scope_type == COMPILER_SCOPE_MODULE)
+            continue;
+        if (PyList_Append(seq, u->u_name))
+            goto _error;
+        if (u->u_scope_type == COMPILER_SCOPE_FUNCTION) {
+            locals_str = _PyUnicode_FromId(&locals);
+            if (locals_str == NULL)
+                goto _error;
+            if (PyList_Append(seq, locals_str))
+                goto _error;
+        }
+    }
+    u = c->u;
+    if (PyList_Append(seq, u->u_name))
+        goto _error;
+    dot_str = _PyUnicode_FromId(&dot);
+    if (dot_str == NULL)
+        goto _error;
+    name = PyUnicode_Join(dot_str, seq);
+    Py_DECREF(seq);
+    u->u_qualname = name;
+    Py_XINCREF(name);
+    return name;
+
+_error:
+    Py_XDECREF(seq);
+    return NULL;
+}
+
 /* Allocate a new block and return a pointer to it.
    Returns NULL on error.
 */
@@ -862,9 +927,9 @@
         case CALL_FUNCTION_VAR_KW:
             return -NARGS(oparg)-2;
         case MAKE_FUNCTION:
-            return -NARGS(oparg) - ((oparg >> 16) & 0xffff);
+            return -1 -NARGS(oparg) - ((oparg >> 16) & 0xffff);
         case MAKE_CLOSURE:
-            return -1 - NARGS(oparg) - ((oparg >> 16) & 0xffff);
+            return -2 - NARGS(oparg) - ((oparg >> 16) & 0xffff);
 #undef NARGS
         case BUILD_SLICE:
             if (oparg == 3)
@@ -1194,7 +1259,7 @@
             return NULL;
     }
     /* Use 0 for firstlineno initially, will fixup in assemble(). */
-    if (!compiler_enter_scope(c, module, mod, 0))
+    if (!compiler_enter_scope(c, module, COMPILER_SCOPE_MODULE, mod, 0))
         return NULL;
     switch (mod->kind) {
     case Module_kind:
@@ -1270,11 +1335,15 @@
 }
 
 static int
-compiler_make_closure(struct compiler *c, PyCodeObject *co, int args)
+compiler_make_closure(struct compiler *c, PyCodeObject *co, int args, PyObject *qualname)
 {
     int i, free = PyCode_GetNumFree(co);
+    if (qualname == NULL)
+        qualname = co->co_name;
+
     if (free == 0) {
         ADDOP_O(c, LOAD_CONST, (PyObject*)co, consts);
+        ADDOP_O(c, LOAD_CONST, qualname, consts);
         ADDOP_I(c, MAKE_FUNCTION, args);
         return 1;
     }
@@ -1311,6 +1380,7 @@
     }
     ADDOP_I(c, BUILD_TUPLE, free);
     ADDOP_O(c, LOAD_CONST, (PyObject*)co, consts);
+    ADDOP_O(c, LOAD_CONST, qualname, consts);
     ADDOP_I(c, MAKE_CLOSURE, args);
     return 1;
 }
@@ -1452,7 +1522,7 @@
 compiler_function(struct compiler *c, stmt_ty s)
 {
     PyCodeObject *co;
-    PyObject *first_const = Py_None;
+    PyObject *qualname, *first_const = Py_None;
     arguments_ty args = s->v.FunctionDef.args;
     expr_ty returns = s->v.FunctionDef.returns;
     asdl_seq* decos = s->v.FunctionDef.decorator_list;
@@ -1478,7 +1548,8 @@
         return 0;
     assert((num_annotations & 0xFFFF) == num_annotations);
 
-    if (!compiler_enter_scope(c, s->v.FunctionDef.name, (void *)s,
+    if (!compiler_enter_scope(c, s->v.FunctionDef.name,
+                              COMPILER_SCOPE_FUNCTION, (void *)s,
                               s->lineno))
         return 0;
 
@@ -1500,14 +1571,19 @@
         VISIT_IN_SCOPE(c, stmt, st);
     }
     co = assemble(c, 1);
+    qualname = compiler_scope_qualname(c);
     compiler_exit_scope(c);
-    if (co == NULL)
+    if (qualname == NULL || co == NULL) {
+        Py_XDECREF(qualname);
+        Py_XDECREF(co);
         return 0;
+    }
 
     arglength = asdl_seq_LEN(args->defaults);
     arglength |= kw_default_count << 8;
     arglength |= num_annotations << 16;
-    compiler_make_closure(c, co, arglength);
+    compiler_make_closure(c, co, arglength, qualname);
+    Py_DECREF(qualname);
     Py_DECREF(co);
 
     /* decorators */
@@ -1542,7 +1618,8 @@
     */
 
     /* 1. compile the class body into a code object */
-    if (!compiler_enter_scope(c, s->v.ClassDef.name, (void *)s, s->lineno))
+    if (!compiler_enter_scope(c, s->v.ClassDef.name,
+                              COMPILER_SCOPE_CLASS, (void *)s, s->lineno))
         return 0;
     /* this block represents what we do in the new scope */
     {
@@ -1572,6 +1649,21 @@
             return 0;
         }
         Py_DECREF(str);
+        /* store the __qualname__ */
+        str = compiler_scope_qualname(c);
+        if (!str) {
+            compiler_exit_scope(c);
+            return 0;
+        }
+        ADDOP_O(c, LOAD_CONST, str, consts);
+        Py_DECREF(str);
+        str = PyUnicode_InternFromString("__qualname__");
+        if (!str || !compiler_nameop(c, str, Store)) {
+            Py_XDECREF(str);
+            compiler_exit_scope(c);
+            return 0;
+        }
+        Py_DECREF(str);
         /* compile the body proper */
         if (!compiler_body(c, s->v.ClassDef.body)) {
             compiler_exit_scope(c);
@@ -1608,7 +1700,7 @@
     ADDOP(c, LOAD_BUILD_CLASS);
 
     /* 3. load a function (or closure) made from the code object */
-    compiler_make_closure(c, co, 0);
+    compiler_make_closure(c, co, 0, NULL);
     Py_DECREF(co);
 
     /* 4. load class name */
@@ -1659,6 +1751,7 @@
 compiler_lambda(struct compiler *c, expr_ty e)
 {
     PyCodeObject *co;
+    PyObject *qualname;
     static identifier name;
     int kw_default_count = 0, arglength;
     arguments_ty args = e->v.Lambda.args;
@@ -1678,7 +1771,8 @@
     }
     if (args->defaults)
         VISIT_SEQ(c, expr, args->defaults);
-    if (!compiler_enter_scope(c, name, (void *)e, e->lineno))
+    if (!compiler_enter_scope(c, name, COMPILER_SCOPE_FUNCTION,
+                              (void *)e, e->lineno))
         return 0;
 
     /* Make None the first constant, so the lambda can't have a
@@ -1696,13 +1790,15 @@
         ADDOP_IN_SCOPE(c, RETURN_VALUE);
     }
     co = assemble(c, 1);
+    qualname = compiler_scope_qualname(c);
     compiler_exit_scope(c);
-    if (co == NULL)
+    if (qualname == NULL || co == NULL)
         return 0;
 
     arglength = asdl_seq_LEN(args->defaults);
     arglength |= kw_default_count << 8;
-    compiler_make_closure(c, co, arglength);
+    compiler_make_closure(c, co, arglength, qualname);
+    Py_DECREF(qualname);
     Py_DECREF(co);
 
     return 1;
@@ -2916,11 +3012,13 @@
 {
     PyCodeObject *co = NULL;
     expr_ty outermost_iter;
+    PyObject *qualname = NULL;
 
     outermost_iter = ((comprehension_ty)
                       asdl_seq_GET(generators, 0))->iter;
 
-    if (!compiler_enter_scope(c, name, (void *)e, e->lineno))
+    if (!compiler_enter_scope(c, name, COMPILER_SCOPE_COMPREHENSION,
+                              (void *)e, e->lineno))
         goto error;
 
     if (type != COMP_GENEXP) {
@@ -2953,12 +3051,14 @@
     }
 
     co = assemble(c, 1);
+    qualname = compiler_scope_qualname(c);
     compiler_exit_scope(c);
-    if (co == NULL)
+    if (qualname == NULL || co == NULL)
         goto error;
 
-    if (!compiler_make_closure(c, co, 0))
+    if (!compiler_make_closure(c, co, 0, qualname))
         goto error;
+    Py_DECREF(qualname);
     Py_DECREF(co);
 
     VISIT(c, expr, outermost_iter);
@@ -2968,6 +3068,7 @@
 error_in_scope:
     compiler_exit_scope(c);
 error:
+    Py_XDECREF(qualname);
     Py_XDECREF(co);
     return 0;
 }
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -103,6 +103,7 @@
                      tag: cpython-32
        Python 3.2a2  3180 (add DELETE_DEREF)
        Python 3.3a0  3190 __class__ super closure changed
+       Python 3.3a0  3200 (__qualname__ added)
 */
 
 /* MAGIC must change whenever the bytecode emitted by the compiler may no
@@ -115,7 +116,7 @@
 #define STRIFY(name) QUOTE(name)
 #define MAJOR STRIFY(PY_MAJOR_VERSION)
 #define MINOR STRIFY(PY_MINOR_VERSION)
-#define MAGIC (3190 | ((long)'\r'<<16) | ((long)'\n'<<24))
+#define MAGIC (3200 | ((long)'\r'<<16) | ((long)'\n'<<24))
 #define TAG "cpython-" MAJOR MINOR;
 #define CACHEDIR "__pycache__"
 /* Current magic word and string tag as globals. */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 19:15:05 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 19:15:05 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Update_What=27s_new_for_PEP?=
	=?utf8?q?_3155?=
Message-ID: <E1RU0Ij-0003yR-LL@dinsdale.python.org>

http://hg.python.org/cpython/rev/064f8881eec7
changeset:   73739:064f8881eec7
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 19:10:05 2011 +0100
summary:
  Update What's new for PEP 3155

files:
  Doc/whatsnew/3.3.rst |  59 ++++++++++++++++++++++++++++++++
  1 files changed, 59 insertions(+), 0 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -189,6 +189,65 @@
         print("You are not allowed to read document.txt")
 
 
+PEP 3155: Qualified name for classes and functions
+==================================================
+
+:pep:`3155` - Qualified name for classes and functions
+ PEP written and implemented by Antoine Pitrou.
+
+Functions and class objects have a new ``__qualname__`` attribute representing
+the "path" from the module top-level to their definition.  For global functions
+and classes, this is the same as ``__name__``.  For other functions and classes,
+it provides better information about where they were actually defined, and
+how they might be accessible from the global scope.
+
+Example with (non-bound) methods::
+
+   >>> class C:
+   ...     def meth(self):
+   ...         pass
+   >>> C.meth.__name__
+   'meth'
+   >>> C.meth.__qualname__
+   'C.meth'
+
+Example with nested classes::
+
+   >>> class C:
+   ...     class D:
+   ...         def meth(self):
+   ...             pass
+   ...
+   >>> C.D.__name__
+   'D'
+   >>> C.D.__qualname__
+   'C.D'
+   >>> C.D.meth.__name__
+   'meth'
+   >>> C.D.meth.__qualname__
+   'C.D.meth'
+
+Example with nested functions::
+
+   >>> def outer():
+   ...     def inner():
+   ...         pass
+   ...     return inner
+   ...
+   >>> outer().__name__
+   'inner'
+   >>> outer().__qualname__
+   'outer.<locals>.inner'
+
+The string representation of those objects is also changed to included the
+new, more precise information::
+
+   >>> str(C.D)
+   "<class '__main__.C.D'>"
+   >>> str(C.D.meth)
+   '<function C.D.meth at 0x7f46b9fe31e0>'
+
+
 Other Language Changes
 ======================
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 19:16:25 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 19:16:25 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Typo?=
Message-ID: <E1RU0K1-000425-Uy@dinsdale.python.org>

http://hg.python.org/cpython/rev/ad6e995f598c
changeset:   73740:ad6e995f598c
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 19:11:26 2011 +0100
summary:
  Typo

files:
  Doc/whatsnew/3.3.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -239,7 +239,7 @@
    >>> outer().__qualname__
    'outer.<locals>.inner'
 
-The string representation of those objects is also changed to included the
+The string representation of those objects is also changed to include the
 new, more precise information::
 
    >>> str(C.D)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 20:06:50 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 25 Nov 2011 20:06:50 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Close_=2313093=3A_PyUnicode?=
 =?utf8?q?=5FEncodeDecimal=28=29_doesn=27t_support_error_handlers?=
Message-ID: <E1RU16o-0007yV-RT@dinsdale.python.org>

http://hg.python.org/cpython/rev/a20fae95618c
changeset:   73741:a20fae95618c
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 25 20:09:01 2011 +0100
summary:
  Close #13093: PyUnicode_EncodeDecimal() doesn't support error handlers
different than "strict" anymore. The caller was unable to compute the
size of the output buffer: it depends on the error handler.

files:
  Lib/test/test_unicode.py |   18 +--
  Misc/NEWS                |    4 +
  Objects/unicodeobject.c  |  131 +++-----------------------
  3 files changed, 26 insertions(+), 127 deletions(-)


diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1816,20 +1816,10 @@
                          b' 3.14 ')
         self.assertRaises(UnicodeEncodeError,
                           unicode_encodedecimal, "123\u20ac", "strict")
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "replace"),
-                         b'123?')
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "ignore"),
-                         b'123')
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "xmlcharrefreplace"),
-                         b'123&#8364;')
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "backslashreplace"),
-                         b'123\\u20ac')
-        self.assertEqual(unicode_encodedecimal("123\u20ac\N{EM SPACE}", "replace"),
-                         b'123? ')
-        self.assertEqual(unicode_encodedecimal("123\u20ac\u20ac", "replace"),
-                         b'123??')
-        self.assertEqual(unicode_encodedecimal("123\u20ac\u0660", "replace"),
-                         b'123?0')
+        self.assertRaisesRegex(
+            ValueError,
+            "^'decimal' codec can't encode character",
+            unicode_encodedecimal, "123\u20ac", "replace")
 
     def test_transform_decimal(self):
         from _testcapi import unicode_transformdecimaltoascii as transform_decimal
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #13093: PyUnicode_EncodeDecimal() doesn't support error handlers
+  different than "strict" anymore. The caller was unable to compute the
+  size of the output buffer: it depends on the error handler.
+
 - PEP 3155 / issue #13448: Qualified name for classes and functions.
 
 - Issue #13436: Fix a bogus error message when an AST object was passed
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8839,15 +8839,8 @@
                         char *output,
                         const char *errors)
 {
-    PyObject *errorHandler = NULL;
-    PyObject *exc = NULL;
     PyObject *unicode;
-    const char *encoding = "decimal";
-    const char *reason = "invalid decimal Unicode string";
-    /* the following variable is used for caching string comparisons
-     * -1=not initialized, 0=unknown, 1=strict, 2=replace, 3=ignore, 4=xmlcharrefreplace */
-    int known_errorHandler = -1;
-    Py_ssize_t i, j;
+    Py_ssize_t i;
     enum PyUnicode_Kind kind;
     void *data;
 
@@ -8860,15 +8853,20 @@
     if (unicode == NULL)
         return -1;
 
-    if (PyUnicode_READY(unicode) < 0)
-        goto onError;
+    if (PyUnicode_READY(unicode) < 0) {
+        Py_DECREF(unicode);
+        return -1;
+    }
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
 
     for (i=0; i < length; ) {
-        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        PyObject *exc;
+        Py_UCS4 ch;
         int decimal;
-        Py_ssize_t startpos, endpos;
+        Py_ssize_t startpos;
+
+        ch = PyUnicode_READ(kind, data, i);
 
         if (Py_UNICODE_ISSPACE(ch)) {
             *output++ = ' ';
@@ -8886,113 +8884,20 @@
             i++;
             continue;
         }
-        /* All other characters are considered unencodable */
+
         startpos = i;
-        endpos = i+1;
-        for (; endpos < length; endpos++) {
-            ch = PyUnicode_READ(kind, data, endpos);
-            if ((0 < ch && ch < 256) ||
-                Py_UNICODE_ISSPACE(ch) ||
-                0 <= Py_UNICODE_TODECIMAL(ch))
-                break;
-        }
-        /* cache callback name lookup
-         * (if not done yet, i.e. it's the first error) */
-        if (known_errorHandler==-1) {
-            if ((errors==NULL) || (!strcmp(errors, "strict")))
-                known_errorHandler = 1;
-            else if (!strcmp(errors, "replace"))
-                known_errorHandler = 2;
-            else if (!strcmp(errors, "ignore"))
-                known_errorHandler = 3;
-            else if (!strcmp(errors, "xmlcharrefreplace"))
-                known_errorHandler = 4;
-            else
-                known_errorHandler = 0;
-        }
-        switch (known_errorHandler) {
-        case 1: /* strict */
-            raise_encode_exception(&exc, encoding, unicode, startpos, endpos, reason);
-            goto onError;
-        case 2: /* replace */
-            for (j=startpos; j < endpos; j++)
-                *output++ = '?';
-            i = endpos;
-            break;
-        case 3: /* ignore */
-            i = endpos;
-            break;
-        case 4: /* xmlcharrefreplace */
-            /* generate replacement */
-            for (j=startpos; j < endpos; j++) {
-                ch = PyUnicode_READ(kind, data, i);
-                output += sprintf(output, "&#%d;", (int)ch);
-                i++;
-            }
-            break;
-        default:
-        {
-            PyObject *repunicode;
-            Py_ssize_t repsize, newpos, k;
-            enum PyUnicode_Kind repkind;
-            void *repdata;
-
-            repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
-                                                          encoding, reason, unicode, &exc,
-                                                          startpos, endpos, &newpos);
-            if (repunicode == NULL)
-                goto onError;
-            if (!PyUnicode_Check(repunicode)) {
-                /* Byte results not supported, since they have no decimal property. */
-                PyErr_SetString(PyExc_TypeError, "error handler should return unicode");
-                Py_DECREF(repunicode);
-                goto onError;
-            }
-            if (PyUnicode_READY(repunicode) < 0) {
-                Py_DECREF(repunicode);
-                goto onError;
-            }
-            repkind = PyUnicode_KIND(repunicode);
-            repdata = PyUnicode_DATA(repunicode);
-
-            /* generate replacement  */
-            repsize = PyUnicode_GET_SIZE(repunicode);
-            for (k=0; k<repsize; k++) {
-                ch = PyUnicode_READ(repkind, repdata, k);
-                if (Py_UNICODE_ISSPACE(ch))
-                    *output++ = ' ';
-                else {
-                    decimal = Py_UNICODE_TODECIMAL(ch);
-                    if (decimal >= 0)
-                        *output++ = '0' + decimal;
-                    else if (0 < ch && ch < 256)
-                        *output++ = (char)ch;
-                    else {
-                        Py_DECREF(repunicode);
-                        raise_encode_exception(&exc, encoding,
-                                               unicode, startpos, endpos,
-                                               reason);
-                        goto onError;
-                    }
-                }
-            }
-            i = newpos;
-            Py_DECREF(repunicode);
-        }
-        }
+        exc = NULL;
+        raise_encode_exception(&exc, "decimal", unicode,
+                               startpos, startpos+1,
+                               "invalid decimal Unicode string");
+        Py_XDECREF(exc);
+        Py_DECREF(unicode);
+        return -1;
     }
     /* 0-terminate the output string */
     *output++ = '\0';
-    Py_XDECREF(exc);
-    Py_XDECREF(errorHandler);
     Py_DECREF(unicode);
     return 0;
-
-  onError:
-    Py_XDECREF(exc);
-    Py_XDECREF(errorHandler);
-    Py_DECREF(unicode);
-    return -1;
 }
 
 /* --- Helpers ------------------------------------------------------------ */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 20:09:32 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 25 Nov 2011 20:09:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_NEWS=3A_fix_the_issue_numbe?=
 =?utf8?q?r_for_PyUnicode=5FEncodeDecimal_change?=
Message-ID: <E1RU19Q-00084X-J3@dinsdale.python.org>

http://hg.python.org/cpython/rev/9a712ad593bb
changeset:   73742:9a712ad593bb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 25 20:11:54 2011 +0100
summary:
  NEWS: fix the issue number for PyUnicode_EncodeDecimal change

files:
  Misc/NEWS |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,7 +10,7 @@
 Core and Builtins
 -----------------
 
-- Issue #13093: PyUnicode_EncodeDecimal() doesn't support error handlers
+- Issue #13452: PyUnicode_EncodeDecimal() doesn't support error handlers
   different than "strict" anymore. The caller was unable to compute the
   size of the output buffer: it depends on the error handler.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 21:38:19 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 21:38:19 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyODU2?=
 =?utf8?q?=3A_Ensure_child_processes_do_not_inherit_the_parent=27s_random_?=
 =?utf8?q?seed?=
Message-ID: <E1RU2XL-0006J0-Dh@dinsdale.python.org>

http://hg.python.org/cpython/rev/3c9ddd93c983
changeset:   73743:3c9ddd93c983
branch:      3.2
parent:      73735:2d6f0e2fe034
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 21:28:15 2011 +0100
summary:
  Issue #12856: Ensure child processes do not inherit the parent's random seed for filename generation in the tempfile module.
Patch by Brian Harring.

files:
  Lib/tempfile.py           |   9 +++++-
  Lib/test/test_tempfile.py |  32 +++++++++++++++++++++++++++
  Misc/NEWS                 |   4 +++
  3 files changed, 43 insertions(+), 2 deletions(-)


diff --git a/Lib/tempfile.py b/Lib/tempfile.py
--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -112,8 +112,13 @@
 
     characters = "abcdefghijklmnopqrstuvwxyz0123456789_"
 
-    def __init__(self):
-        self.rng = _Random()
+    @property
+    def rng(self):
+        cur_pid = _os.getpid()
+        if cur_pid != getattr(self, '_rng_pid', None):
+            self._rng = _Random()
+            self._rng_pid = cur_pid
+        return self._rng
 
     def __iter__(self):
         return self
diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py
--- a/Lib/test/test_tempfile.py
+++ b/Lib/test/test_tempfile.py
@@ -1,6 +1,7 @@
 # tempfile.py unit tests.
 import tempfile
 import os
+import signal
 import sys
 import re
 import warnings
@@ -135,6 +136,37 @@
         except:
             self.failOnException("iteration")
 
+    @unittest.skipUnless(hasattr(os, 'fork'),
+        "os.fork is required for this test")
+    def test_process_awareness(self):
+        # ensure that the random source differs between
+        # child and parent.
+        read_fd, write_fd = os.pipe()
+        pid = None
+        try:
+            pid = os.fork()
+            if not pid:
+                os.close(read_fd)
+                os.write(write_fd, next(self.r).encode("ascii"))
+                os.close(write_fd)
+                # bypass the normal exit handlers- leave those to
+                # the parent.
+                os._exit(0)
+            parent_value = next(self.r)
+            child_value = os.read(read_fd, len(parent_value)).decode("ascii")
+        finally:
+            if pid:
+                # best effort to ensure the process can't bleed out
+                # via any bugs above
+                try:
+                    os.kill(pid, signal.SIGKILL)
+                except EnvironmentError:
+                    pass
+            os.close(read_fd)
+            os.close(write_fd)
+        self.assertNotEqual(child_value, parent_value)
+
+
 test_classes.append(test__RandomNameSequence)
 
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -83,6 +83,10 @@
 Library
 -------
 
+- Issue #12856: Ensure child processes do not inherit the parent's random
+  seed for filename generation in the tempfile module.  Patch by Brian
+  Harring.
+
 - Issue #13458: Fix a memory leak in the ssl module when decoding a
   certificate with a subjectAltName.  Patch by Robert Xiao.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 21:38:20 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 21:38:20 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312856=3A_Ensure_child_processes_do_not_inherit_the_?=
 =?utf8?q?parent=27s_random_seed?=
Message-ID: <E1RU2XM-0006JW-5T@dinsdale.python.org>

http://hg.python.org/cpython/rev/588087429809
changeset:   73744:588087429809
parent:      73742:9a712ad593bb
parent:      73743:3c9ddd93c983
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 21:29:27 2011 +0100
summary:
  Issue #12856: Ensure child processes do not inherit the parent's random seed for filename generation in the tempfile module.
Patch by Brian Harring.

files:
  Lib/tempfile.py           |   9 +++++-
  Lib/test/test_tempfile.py |  32 +++++++++++++++++++++++++++
  Misc/NEWS                 |   4 +++
  3 files changed, 43 insertions(+), 2 deletions(-)


diff --git a/Lib/tempfile.py b/Lib/tempfile.py
--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -108,8 +108,13 @@
 
     characters = "abcdefghijklmnopqrstuvwxyz0123456789_"
 
-    def __init__(self):
-        self.rng = _Random()
+    @property
+    def rng(self):
+        cur_pid = _os.getpid()
+        if cur_pid != getattr(self, '_rng_pid', None):
+            self._rng = _Random()
+            self._rng_pid = cur_pid
+        return self._rng
 
     def __iter__(self):
         return self
diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py
--- a/Lib/test/test_tempfile.py
+++ b/Lib/test/test_tempfile.py
@@ -1,6 +1,7 @@
 # tempfile.py unit tests.
 import tempfile
 import os
+import signal
 import sys
 import re
 import warnings
@@ -135,6 +136,37 @@
         except:
             self.failOnException("iteration")
 
+    @unittest.skipUnless(hasattr(os, 'fork'),
+        "os.fork is required for this test")
+    def test_process_awareness(self):
+        # ensure that the random source differs between
+        # child and parent.
+        read_fd, write_fd = os.pipe()
+        pid = None
+        try:
+            pid = os.fork()
+            if not pid:
+                os.close(read_fd)
+                os.write(write_fd, next(self.r).encode("ascii"))
+                os.close(write_fd)
+                # bypass the normal exit handlers- leave those to
+                # the parent.
+                os._exit(0)
+            parent_value = next(self.r)
+            child_value = os.read(read_fd, len(parent_value)).decode("ascii")
+        finally:
+            if pid:
+                # best effort to ensure the process can't bleed out
+                # via any bugs above
+                try:
+                    os.kill(pid, signal.SIGKILL)
+                except EnvironmentError:
+                    pass
+            os.close(read_fd)
+            os.close(write_fd)
+        self.assertNotEqual(child_value, parent_value)
+
+
 test_classes.append(test__RandomNameSequence)
 
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -395,6 +395,10 @@
 Library
 -------
 
+- Issue #12856: Ensure child processes do not inherit the parent's random
+  seed for filename generation in the tempfile module.  Patch by Brian
+  Harring.
+
 - Issue #9957: SpooledTemporaryFile.truncate() now accepts an optional size
   parameter, as other file-like objects.  Patch by Ryan Kelly.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 21:38:20 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Fri, 25 Nov 2011 21:38:20 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyODU2?=
 =?utf8?q?=3A_Ensure_child_processes_do_not_inherit_the_parent=27s_random_?=
 =?utf8?q?seed?=
Message-ID: <E1RU2XM-0006Ja-US@dinsdale.python.org>

http://hg.python.org/cpython/rev/e42be90eb9c5
changeset:   73745:e42be90eb9c5
branch:      2.7
parent:      73725:d1bdafa161e7
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Fri Nov 25 21:28:15 2011 +0100
summary:
  Issue #12856: Ensure child processes do not inherit the parent's random seed for filename generation in the tempfile module.
Patch by Brian Harring.

files:
  Lib/tempfile.py           |   9 ++++++-
  Lib/test/test_tempfile.py |  32 +++++++++++++++++++++++++++
  Misc/NEWS                 |   4 +++
  3 files changed, 44 insertions(+), 1 deletions(-)


diff --git a/Lib/tempfile.py b/Lib/tempfile.py
--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -118,9 +118,16 @@
 
     def __init__(self):
         self.mutex = _allocate_lock()
-        self.rng = _Random()
         self.normcase = _os.path.normcase
 
+    @property
+    def rng(self):
+        cur_pid = _os.getpid()
+        if cur_pid != getattr(self, '_rng_pid', None):
+            self._rng = _Random()
+            self._rng_pid = cur_pid
+        return self._rng
+
     def __iter__(self):
         return self
 
diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py
--- a/Lib/test/test_tempfile.py
+++ b/Lib/test/test_tempfile.py
@@ -1,6 +1,7 @@
 # tempfile.py unit tests.
 import tempfile
 import os
+import signal
 import sys
 import re
 import warnings
@@ -126,6 +127,37 @@
         except:
             self.failOnException("iteration")
 
+    @unittest.skipUnless(hasattr(os, 'fork'),
+        "os.fork is required for this test")
+    def test_process_awareness(self):
+        # ensure that the random source differs between
+        # child and parent.
+        read_fd, write_fd = os.pipe()
+        pid = None
+        try:
+            pid = os.fork()
+            if not pid:
+                os.close(read_fd)
+                os.write(write_fd, next(self.r).encode("ascii"))
+                os.close(write_fd)
+                # bypass the normal exit handlers- leave those to
+                # the parent.
+                os._exit(0)
+            parent_value = next(self.r)
+            child_value = os.read(read_fd, len(parent_value)).decode("ascii")
+        finally:
+            if pid:
+                # best effort to ensure the process can't bleed out
+                # via any bugs above
+                try:
+                    os.kill(pid, signal.SIGKILL)
+                except EnvironmentError:
+                    pass
+            os.close(read_fd)
+            os.close(write_fd)
+        self.assertNotEqual(child_value, parent_value)
+
+
 test_classes.append(test__RandomNameSequence)
 
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -79,6 +79,10 @@
 Library
 -------
 
+- Issue #12856: Ensure child processes do not inherit the parent's random
+  seed for filename generation in the tempfile module.  Patch by Brian
+  Harring.
+
 - Issue #13458: Fix a memory leak in the ssl module when decoding a
   certificate with a subjectAltName.  Patch by Robert Xiao.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Fri Nov 25 22:08:32 2011
From: python-checkins at python.org (victor.stinner)
Date: Fri, 25 Nov 2011 22:08:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2312567=3A_The_curse?=
 =?utf8?q?s_module_uses_Unicode_functions_for_Unicode_arguments?=
Message-ID: <E1RU30a-0008Jb-Ui@dinsdale.python.org>

http://hg.python.org/cpython/rev/c3581ca21a57
changeset:   73746:c3581ca21a57
parent:      73744:588087429809
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Fri Nov 25 22:10:02 2011 +0100
summary:
  Issue #12567: The curses module uses Unicode functions for Unicode arguments
when it is linked to the ncurses library. It encodes also Unicode strings to
the locale encoding instead of UTF-8.

files:
  Doc/library/curses.rst  |   12 +-
  Doc/whatsnew/3.3.rst    |    5 +
  Include/py_curses.h     |    1 +
  Lib/test/test_curses.py |   41 +-
  Misc/NEWS               |    4 +
  Modules/_cursesmodule.c |  580 ++++++++++++++++++++++-----
  6 files changed, 521 insertions(+), 122 deletions(-)


diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -653,7 +653,7 @@
 --------------
 
 Window objects, as returned by :func:`initscr` and :func:`newwin` above, have
-the following methods:
+the following methods and attributes:
 
 
 .. method:: window.addch([y, x,] ch[, attr])
@@ -834,6 +834,16 @@
    event.
 
 
+.. attribute:: window.encoding
+
+   Encoding used to encode method arguments (Unicode strings and characters).
+   The encoding attribute is inherited from by parent window when a subwindow
+   is created, for example with :meth:`window.subwin`. By default, the locale
+   encoding is used (see :func:`locale.getpreferredencoding`).
+
+   .. versionadded:: 3.3
+
+
 .. method:: window.erase()
 
    Clear the window.
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -333,6 +333,11 @@
 curses
 ------
 
+ * If the :mod:`curses` module is linked to the ncursesw library, use Unicode
+   functions when Unicode strings or characters are passed (e.g.
+   :c:func:`waddwstr`), and bytes functions otherwise (e.g. :c:func:`waddstr`).
+ * Use the locale encoding instead of ``utf-8`` to encode Unicode strings.
+ * :class:`curses.window` has a new :attr:`curses.window.encoding` attribute.
  * The :class:`curses.window` class has a new :meth:`~curses.window.get_wch`
    method to get a wide character
  * The :mod:`curses` module has a new :meth:`~curses.unget_wch` function to
diff --git a/Include/py_curses.h b/Include/py_curses.h
--- a/Include/py_curses.h
+++ b/Include/py_curses.h
@@ -76,6 +76,7 @@
 typedef struct {
 	PyObject_HEAD
 	WINDOW *win;
+	char *encoding;
 } PyCursesWindowObject;
 
 #define PyCursesWindow_Check(v)	 (Py_TYPE(v) == &PyCursesWindow_Type)
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -267,24 +267,42 @@
 def test_unget_wch(stdscr):
     if not hasattr(curses, 'unget_wch'):
         return
-    ch = 'a'
-    curses.unget_wch(ch)
-    read = stdscr.get_wch()
-    read = chr(read)
-    if read != ch:
-        raise AssertionError("%r != %r" % (read, ch))
+    for ch in ('a', '\xe9', '\u20ac', '\U0010FFFF'):
+        curses.unget_wch(ch)
+        read = stdscr.get_wch()
+        read = chr(read)
+        if read != ch:
+            raise AssertionError("%r != %r" % (read, ch))
 
-    ch = ord('a')
-    curses.unget_wch(ch)
-    read = stdscr.get_wch()
-    if read != ch:
-        raise AssertionError("%r != %r" % (read, ch))
+        code = ord(ch)
+        curses.unget_wch(code)
+        read = stdscr.get_wch()
+        if read != code:
+            raise AssertionError("%r != %r" % (read, code))
 
 def test_issue10570():
     b = curses.tparm(curses.tigetstr("cup"), 5, 3)
     assert type(b) is bytes
     curses.putp(b)
 
+def test_encoding(stdscr):
+    import codecs
+    encoding = stdscr.encoding
+    codecs.lookup(encoding)
+    try:
+        stdscr.encoding = 10
+    except TypeError:
+        pass
+    else:
+        raise AssertionError("TypeError not raised")
+    stdscr.encoding = encoding
+    try:
+        del stdscr.encoding
+    except TypeError:
+        pass
+    else:
+        raise AssertionError("TypeError not raised")
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -295,6 +313,7 @@
         test_issue6243(stdscr)
         test_unget_wch(stdscr)
         test_issue10570()
+        test_encoding(stdscr)
     finally:
         curses.resetty()
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -395,6 +395,10 @@
 Library
 -------
 
+- Issue #12567: The curses module uses Unicode functions for Unicode arguments
+  when it is linked to the ncurses library. It encodes also Unicode strings to
+  the locale encoding instead of UTF-8.
+
 - Issue #12856: Ensure child processes do not inherit the parent's random
   seed for filename generation in the tempfile module.  Patch by Brian
   Harring.
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -121,6 +121,10 @@
 #include <term.h>
 #endif
 
+#ifdef HAVE_LANGINFO_H
+#include <langinfo.h>
+#endif
+
 #if !defined(HAVE_NCURSES_H) && (defined(sgi) || defined(__sun) || defined(SCO5))
 #define STRICT_SYSV_CURSES       /* Don't use ncurses extensions */
 typedef chtype attr_t;           /* No attr_t type is available */
@@ -143,6 +147,8 @@
 /* Tells whether start_color() has been called to initialise color usage. */
 static int initialisedcolors = FALSE;
 
+static char *screen_encoding = NULL;
+
 /* Utility Macros */
 #define PyCursesSetupTermCalled                                         \
     if (initialised_setupterm != TRUE) {                                \
@@ -175,7 +181,7 @@
  */
 
 static PyObject *
-PyCursesCheckERR(int code, char *fname)
+PyCursesCheckERR(int code, const char *fname)
 {
     if (code != ERR) {
         Py_INCREF(Py_None);
@@ -190,28 +196,193 @@
     }
 }
 
+/* Convert an object to a byte (an integer of type chtype):
+
+   - int
+   - bytes of length 1
+   - str of length 1
+
+   Return 1 on success, 0 on error (invalid type or integer overflow). */
 static int
-PyCurses_ConvertToChtype(PyObject *obj, chtype *ch)
+PyCurses_ConvertToChtype(PyCursesWindowObject *win, PyObject *obj, chtype *ch)
 {
-    if (PyLong_CheckExact(obj)) {
-        int overflow;
-        /* XXX should the truncation by the cast also be reported
-           as an error? */
-        *ch = (chtype) PyLong_AsLongAndOverflow(obj, &overflow);
-        if (overflow)
+    long value;
+    if(PyBytes_Check(obj) && PyBytes_Size(obj) == 1) {
+        value = (unsigned char)PyBytes_AsString(obj)[0];
+    }
+    else if (PyUnicode_Check(obj)) {
+        if (PyUnicode_GetLength(obj) != 1) {
+            PyErr_Format(PyExc_TypeError,
+                         "expect bytes or str of length 1, or int, "
+                         "got a str of length %zi",
+                         PyUnicode_GET_LENGTH(obj));
             return 0;
-    } else if(PyBytes_Check(obj)
-              && (PyBytes_Size(obj) == 1)) {
-        *ch = (chtype) *PyBytes_AsString(obj);
-    } else if (PyUnicode_Check(obj) && PyUnicode_GET_LENGTH(obj) == 1) {
-        Py_UCS4 ucs = PyUnicode_READ(PyUnicode_KIND(obj),
-                                     PyUnicode_DATA(obj),
-                                     0);
-        *ch = (chtype)ucs;
-    } else {
+        }
+        value = PyUnicode_READ_CHAR(obj, 0);
+        if (128 < value) {
+            PyObject *bytes;
+            const char *encoding;
+            if (win)
+                encoding = win->encoding;
+            else
+                encoding = screen_encoding;
+            bytes = PyUnicode_AsEncodedObject(obj, encoding, NULL);
+            if (bytes == NULL)
+                return 0;
+            if (PyBytes_GET_SIZE(bytes) == 1)
+                value = (unsigned char)PyBytes_AS_STRING(bytes)[0];
+            else
+                value = -1;
+            Py_DECREF(bytes);
+            if (value < 0)
+                goto overflow;
+        }
+    }
+    else if (PyLong_CheckExact(obj)) {
+        int long_overflow;
+        value = PyLong_AsLongAndOverflow(obj, &long_overflow);
+        if (long_overflow)
+            goto overflow;
+    }
+    else {
+        PyErr_Format(PyExc_TypeError,
+                     "expect bytes or str of length 1, or int, got %s",
+                     Py_TYPE(obj)->tp_name);
         return 0;
     }
+    *ch = (chtype)value;
+    if ((long)*ch != value)
+        goto overflow;
     return 1;
+
+overflow:
+    PyErr_SetString(PyExc_OverflowError,
+                    "byte doesn't fit in chtype");
+    return 0;
+}
+
+/* Convert an object to a byte (chtype) or a character (cchar_t):
+
+    - int
+    - bytes of length 1
+    - str of length 1
+
+   Return:
+
+    - 2 if obj is a character (written into *wch)
+    - 1 if obj is a byte (written into *ch)
+    - 0 on error: raise an exception */
+static int
+PyCurses_ConvertToCchar_t(PyCursesWindowObject *win, PyObject *obj,
+                          chtype *ch
+#ifdef HAVE_NCURSESW
+                          , cchar_t *wch
+#endif
+                          )
+{
+    int ret = 0;
+    long value;
+#ifdef HAVE_NCURSESW
+    wchar_t buffer[2];
+#endif
+
+    if (PyUnicode_Check(obj)) {
+#ifdef HAVE_NCURSESW
+        if (PyUnicode_AsWideChar(obj, buffer, 2) != 1) {
+            PyErr_Format(PyExc_TypeError,
+                         "expect bytes or str of length 1, or int, "
+                         "got a str of length %zi",
+                         PyUnicode_GET_LENGTH(obj));
+            return 0;
+        }
+        memset(wch->chars, 0, sizeof(wch->chars));
+        wch->chars[0] = buffer[0];
+        return 2;
+#else
+        return PyCurses_ConvertToChtype(win, obj, ch);
+#endif
+    }
+    else if(PyBytes_Check(obj) && PyBytes_Size(obj) == 1) {
+        value = (unsigned char)PyBytes_AsString(obj)[0];
+        ret = 1;
+    }
+    else if (PyLong_CheckExact(obj)) {
+        int overflow;
+        value = PyLong_AsLongAndOverflow(obj, &overflow);
+        if (overflow) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "int doesn't fit in long");
+            return 0;
+        }
+#ifdef HAVE_NCURSESW
+        ret = 2;
+#else
+        ret = 1;
+#endif
+    }
+    else {
+        PyErr_Format(PyExc_TypeError,
+                     "expect bytes or str of length 1, or int, got %s",
+                     Py_TYPE(obj)->tp_name);
+        return 0;
+    }
+#ifdef HAVE_NCURSESW
+    if (ret == 2) {
+        memset(wch->chars, 0, sizeof(wch->chars));
+        wch->chars[0] = (wchar_t)value;
+        if ((long)wch->chars[0] != value) {
+            PyErr_Format(PyExc_OverflowError,
+                         "character doesn't fit in wchar_t");
+            return 0;
+        }
+    }
+    else
+#endif
+    {
+        *ch = (chtype)value;
+        if ((long)*ch != value || value < 0 || value > 255) {
+            PyErr_Format(PyExc_OverflowError,
+                         "byte doesn't fit in chtype");
+            return 0;
+        }
+    }
+    return ret;
+}
+
+/* Convert an object to a byte string (char*) or a wide character string
+   (wchar_t*). Return:
+
+    - 2 if obj is a character string (written into *wch)
+    - 1 if obj is a byte string (written into *bytes)
+    - 0 on error: raise an exception */
+static int
+PyCurses_ConvertToString(PyCursesWindowObject *win, PyObject *obj,
+                         PyObject **bytes, wchar_t **wstr)
+{
+    if (PyUnicode_Check(obj)) {
+#ifdef HAVE_NCURSESW
+        assert (wstr != NULL);
+        *wstr = PyUnicode_AsWideCharString(obj, NULL);
+        if (*wstr == NULL)
+            return 0;
+        return 2;
+#else
+        assert (wstr == NULL);
+        *bytes = PyUnicode_AsEncodedObject(obj, win->encoding, NULL);
+        if (*bytes == NULL)
+            return 0;
+        return 1;
+#endif
+    }
+    else if (PyBytes_Check(obj)) {
+        Py_INCREF(obj);
+        *bytes = obj;
+        return 1;
+    }
+
+    PyErr_Format(PyExc_TypeError, "expect bytes or str, got %s",
+                 Py_TYPE(obj)->tp_name);
+    return 0;
 }
 
 /* Function versions of the 3 functions for testing whether curses has been
@@ -357,13 +528,37 @@
 /* Allocation and deallocation of Window Objects */
 
 static PyObject *
-PyCursesWindow_New(WINDOW *win)
+PyCursesWindow_New(WINDOW *win, const char *encoding)
 {
     PyCursesWindowObject *wo;
 
+    if (encoding == NULL) {
+#if defined(MS_WINDOWS)
+        char *buffer[100];
+        UINT cp;
+        cp = GetConsoleOutputCP();
+        if (cp != 0) {
+            PyOS_snprintf(buffer, sizeof(buffer), "cp%u", cp);
+            encoding = buffer;
+        }
+#elif defined(CODESET)
+        const char *codeset = nl_langinfo(CODESET);
+        if (codeset != NULL && codeset[0] != 0)
+            encoding = codeset;
+#endif
+        if (encoding == NULL)
+            encoding = "utf-8";
+    }
+
     wo = PyObject_NEW(PyCursesWindowObject, &PyCursesWindow_Type);
     if (wo == NULL) return NULL;
     wo->win = win;
+    wo->encoding = strdup(encoding);
+    if (wo->encoding == NULL) {
+        Py_DECREF(wo);
+        PyErr_NoMemory();
+        return NULL;
+    }
     return (PyObject *)wo;
 }
 
@@ -371,6 +566,8 @@
 PyCursesWindow_Dealloc(PyCursesWindowObject *wo)
 {
     if (wo->win != stdscr) delwin(wo->win);
+    if (wo->encoding != NULL)
+        free(wo->encoding);
     PyObject_DEL(wo);
 }
 
@@ -380,29 +577,34 @@
 PyCursesWindow_AddCh(PyCursesWindowObject *self, PyObject *args)
 {
     int rtn, x, y, use_xy = FALSE;
-    PyObject *temp;
-    chtype ch = 0;
+    PyObject *chobj;
+    int type;
+    chtype ch;
+#ifdef HAVE_NCURSESW
+    cchar_t wch;
+#endif
     attr_t attr = A_NORMAL;
     long lattr;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 1:
-        if (!PyArg_ParseTuple(args, "O;ch or int", &temp))
+        if (!PyArg_ParseTuple(args, "O;ch or int", &chobj))
             return NULL;
         break;
     case 2:
-        if (!PyArg_ParseTuple(args, "Ol;ch or int,attr", &temp, &lattr))
+        if (!PyArg_ParseTuple(args, "Ol;ch or int,attr", &chobj, &lattr))
             return NULL;
         attr = lattr;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"iiO;y,x,ch or int", &y, &x, &temp))
+        if (!PyArg_ParseTuple(args,"iiO;y,x,ch or int", &y, &x, &chobj))
             return NULL;
         use_xy = TRUE;
         break;
     case 4:
         if (!PyArg_ParseTuple(args,"iiOl;y,x,ch or int, attr",
-                              &y, &x, &temp, &lattr))
+                              &y, &x, &chobj, &lattr))
             return NULL;
         attr = lattr;
         use_xy = TRUE;
@@ -412,17 +614,33 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 or 3 must be a ch or an int");
+#ifdef HAVE_NCURSESW
+    type = PyCurses_ConvertToCchar_t(self, chobj, &ch, &wch);
+    if (type == 2) {
+        funcname = "add_wch";
+        wch.attr = attr;
+        if (use_xy == TRUE)
+            rtn = mvwadd_wch(self->win,y,x, &wch);
+        else {
+            rtn = wadd_wch(self->win, &wch);
+        }
+    }
+    else
+#else
+    type = PyCurses_ConvertToCchar_t(self, chobj, &ch);
+#endif
+    if (type == 1) {
+        funcname = "addch";
+        if (use_xy == TRUE)
+            rtn = mvwaddch(self->win,y,x, ch | attr);
+        else {
+            rtn = waddch(self->win, ch | attr);
+        }
+    }
+    else {
         return NULL;
     }
-
-    if (use_xy == TRUE)
-        rtn = mvwaddch(self->win,y,x, ch | attr);
-    else {
-        rtn = waddch(self->win, ch | attr);
-    }
-    return PyCursesCheckERR(rtn, "addch");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
@@ -430,29 +648,34 @@
 {
     int rtn;
     int x, y;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 1:
-        if (!PyArg_ParseTuple(args,"s;str", &str))
+        if (!PyArg_ParseTuple(args,"O;str", &strobj))
             return NULL;
         break;
     case 2:
-        if (!PyArg_ParseTuple(args,"sl;str,attr", &str, &lattr))
+        if (!PyArg_ParseTuple(args,"Ol;str,attr", &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"iis;int,int,str", &y, &x, &str))
+        if (!PyArg_ParseTuple(args,"iiO;int,int,str", &y, &x, &strobj))
             return NULL;
         use_xy = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisl;int,int,str,attr", &y, &x, &str, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOl;int,int,str,attr", &y, &x, &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -461,47 +684,74 @@
         PyErr_SetString(PyExc_TypeError, "addstr requires 1 to 4 arguments");
         return NULL;
     }
-
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwaddstr(self->win,y,x,str);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "addwstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddwstr(self->win,y,x,wstr);
+        else
+            rtn = waddwstr(self->win,wstr);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = waddstr(self->win,str);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "addstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddstr(self->win,y,x,str);
+        else
+            rtn = waddstr(self->win,str);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "addstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
 PyCursesWindow_AddNStr(PyCursesWindowObject *self, PyObject *args)
 {
     int rtn, x, y, n;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 2:
-        if (!PyArg_ParseTuple(args,"si;str,n", &str, &n))
+        if (!PyArg_ParseTuple(args,"Oi;str,n", &strobj, &n))
             return NULL;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"sil;str,n,attr", &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"Oil;str,n,attr", &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisi;y,x,str,n", &y, &x, &str, &n))
+        if (!PyArg_ParseTuple(args,"iiOi;y,x,str,n", &y, &x, &strobj, &n))
             return NULL;
         use_xy = TRUE;
         break;
     case 5:
-        if (!PyArg_ParseTuple(args,"iisil;y,x,str,n,attr", &y, &x, &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOil;y,x,str,n,attr", &y, &x, &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -510,18 +760,41 @@
         PyErr_SetString(PyExc_TypeError, "addnstr requires 2 to 5 arguments");
         return NULL;
     }
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
 
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwaddnstr(self->win,y,x,str,n);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "addnwstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddnwstr(self->win,y,x,wstr,n);
+        else
+            rtn = waddnwstr(self->win,wstr,n);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = waddnstr(self->win,str,n);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "addnstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddnstr(self->win,y,x,str,n);
+        else
+            rtn = waddnstr(self->win,str,n);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "addnstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
@@ -547,10 +820,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &bkgd)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 or 3 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &bkgd))
         return NULL;
-    }
 
     return PyCursesCheckERR(wbkgd(self->win, bkgd | attr), "bkgd");
 }
@@ -605,10 +876,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &bkgd)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &bkgd))
         return NULL;
-    }
 
     wbkgdset(self->win, bkgd | attr);
     return PyCursesCheckERR(0, "bkgdset");
@@ -633,11 +902,8 @@
         return NULL;
 
     for(i=0; i<8; i++) {
-        if (temp[i] != NULL && !PyCurses_ConvertToChtype(temp[i], &ch[i])) {
-            PyErr_Format(PyExc_TypeError,
-                         "argument %i must be a ch or an int", i+1);
+        if (temp[i] != NULL && !PyCurses_ConvertToChtype(self, temp[i], &ch[i]))
             return NULL;
-        }
     }
 
     wborder(self->win,
@@ -782,7 +1048,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -810,10 +1076,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &ch))
         return NULL;
-    }
 
 #ifdef WINDOW_HAS_FLAGS
     if (self->win->_flags & _ISPAD)
@@ -1034,11 +1298,8 @@
     }
 
     if (code != ERR) {
-        if (!PyCurses_ConvertToChtype(temp, &ch)) {
-            PyErr_SetString(PyExc_TypeError,
-                            "argument 1 or 3 must be a ch or an int");
+        if (!PyCurses_ConvertToChtype(self, temp, &ch))
             return NULL;
-        }
         return PyCursesCheckERR(whline(self->win, ch | attr, n), "hline");
     } else
         return PyCursesCheckERR(code, "wmove");
@@ -1079,11 +1340,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError,
-                        "argument 1 or 3 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &ch))
         return NULL;
-    }
 
     if (use_xy == TRUE)
         rtn = mvwinsch(self->win,y,x, ch | attr);
@@ -1154,29 +1412,34 @@
 {
     int rtn;
     int x, y;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 1:
-        if (!PyArg_ParseTuple(args,"s;str", &str))
+        if (!PyArg_ParseTuple(args,"O;str", &strobj))
             return NULL;
         break;
     case 2:
-        if (!PyArg_ParseTuple(args,"sl;str,attr", &str, &lattr))
+        if (!PyArg_ParseTuple(args,"Ol;str,attr", &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"iis;y,x,str", &y, &x, &str))
+        if (!PyArg_ParseTuple(args,"iiO;y,x,str", &y, &x, &strobj))
             return NULL;
         use_xy = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisl;y,x,str,attr", &y, &x, &str, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOl;y,x,str,attr", &y, &x, &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -1186,46 +1449,75 @@
         return NULL;
     }
 
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
+
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwinsstr(self->win,y,x,str);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "inswstr";
+        if (use_xy == TRUE)
+            rtn = mvwins_wstr(self->win,y,x,wstr);
+        else
+            rtn = wins_wstr(self->win,wstr);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = winsstr(self->win,str);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "insstr";
+        if (use_xy == TRUE)
+            rtn = mvwinsstr(self->win,y,x,str);
+        else
+            rtn = winsstr(self->win,str);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "insstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
 PyCursesWindow_InsNStr(PyCursesWindowObject *self, PyObject *args)
 {
     int rtn, x, y, n;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 2:
-        if (!PyArg_ParseTuple(args,"si;str,n", &str, &n))
+        if (!PyArg_ParseTuple(args,"Oi;str,n", &strobj, &n))
             return NULL;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"sil;str,n,attr", &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"Oil;str,n,attr", &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisi;y,x,str,n", &y, &x, &str, &n))
+        if (!PyArg_ParseTuple(args,"iiOi;y,x,str,n", &y, &x, &strobj, &n))
             return NULL;
         use_xy = TRUE;
         break;
     case 5:
-        if (!PyArg_ParseTuple(args,"iisil;y,x,str,n,attr", &y, &x, &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOil;y,x,str,n,attr", &y, &x, &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -1235,17 +1527,41 @@
         return NULL;
     }
 
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
+
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwinsnstr(self->win,y,x,str,n);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "insn_wstr";
+        if (use_xy == TRUE)
+            rtn = mvwins_nwstr(self->win,y,x,wstr,n);
+        else
+            rtn = wins_nwstr(self->win,wstr,n);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = winsnstr(self->win,str,n);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "insnstr";
+        if (use_xy == TRUE)
+            rtn = mvwinsnstr(self->win,y,x,str,n);
+        else
+            rtn = winsnstr(self->win,str,n);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "insnstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
@@ -1528,7 +1844,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, self->encoding);
 }
 
 static PyObject *
@@ -1604,16 +1920,51 @@
     }
 
     if (code != ERR) {
-        if (!PyCurses_ConvertToChtype(temp, &ch)) {
-            PyErr_SetString(PyExc_TypeError,
-                            "argument 1 or 3 must be a ch or an int");
+        if (!PyCurses_ConvertToChtype(self, temp, &ch))
             return NULL;
-        }
         return PyCursesCheckERR(wvline(self->win, ch | attr, n), "vline");
     } else
         return PyCursesCheckERR(code, "wmove");
 }
 
+static PyObject *
+PyCursesWindow_get_encoding(PyCursesWindowObject *self, void *closure)
+{
+    return PyUnicode_FromString(self->encoding);
+}
+
+static int
+PyCursesWindow_set_encoding(PyCursesWindowObject *self, PyObject *value)
+{
+    PyObject *ascii;
+    char *encoding;
+
+    /* It is illegal to del win.encoding */
+    if (value == NULL) {
+        PyErr_SetString(PyExc_TypeError,
+                        "encoding may not be deleted");
+        return -1;
+    }
+
+    if (!PyUnicode_Check(value)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "setting encoding to a non-string");
+        return -1;
+    }
+    ascii = PyUnicode_AsASCIIString(value);
+    if (ascii == NULL)
+        return -1;
+    encoding = strdup(PyBytes_AS_STRING(ascii));
+    if (encoding == NULL) {
+        PyErr_NoMemory();
+        return -1;
+    }
+    free(self->encoding);
+    self->encoding = encoding;
+    return 0;
+}
+
+
 static PyMethodDef PyCursesWindow_Methods[] = {
     {"addch",           (PyCFunction)PyCursesWindow_AddCh, METH_VARARGS},
     {"addnstr",         (PyCFunction)PyCursesWindow_AddNStr, METH_VARARGS},
@@ -1701,6 +2052,13 @@
     {NULL,                  NULL}   /* sentinel */
 };
 
+static PyGetSetDef PyCursesWindow_getsets[] = {
+    {"encoding",
+     (getter)PyCursesWindow_get_encoding,
+     (setter)PyCursesWindow_set_encoding,
+     "the typecode character used to create the array"}
+};
+
 /* -------------------------------------------------------*/
 
 PyTypeObject PyCursesWindow_Type = {
@@ -1733,6 +2091,8 @@
     0,                          /*tp_iter*/
     0,                          /*tp_iternext*/
     PyCursesWindow_Methods,     /*tp_methods*/
+    0,                          /* tp_members */
+    PyCursesWindow_getsets,     /* tp_getset */
 };
 
 /*********************************************************************
@@ -1956,7 +2316,7 @@
         PyErr_SetString(PyCursesError, catchall_NULL);
         return NULL;
     }
-    return PyCursesWindow_New(win);
+    return PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -2034,10 +2394,11 @@
 PyCurses_InitScr(PyObject *self)
 {
     WINDOW *win;
+    PyCursesWindowObject *winobj;
 
     if (initialised == TRUE) {
         wrefresh(stdscr);
-        return (PyObject *)PyCursesWindow_New(stdscr);
+        return (PyObject *)PyCursesWindow_New(stdscr, NULL);
     }
 
     win = initscr();
@@ -2129,7 +2490,9 @@
     SetDictInt("LINES", LINES);
     SetDictInt("COLS", COLS);
 
-    return (PyObject *)PyCursesWindow_New(win);
+    winobj = (PyCursesWindowObject *)PyCursesWindow_New(win, NULL);
+    screen_encoding = winobj->encoding;
+    return (PyObject *)winobj;
 }
 
 static PyObject *
@@ -2331,7 +2694,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -2363,7 +2726,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -2680,10 +3043,8 @@
 
     if (!PyArg_ParseTuple(args,"O;ch or int",&temp)) return NULL;
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(NULL, temp, &ch))
         return NULL;
-    }
 
     return PyBytes_FromString(unctrl(ch));
 }
@@ -2696,12 +3057,11 @@
 
     PyCursesInitialised;
 
-    if (!PyArg_ParseTuple(args,"O;ch or int",&temp)) return NULL;
-
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument must be a ch or an int");
+    if (!PyArg_ParseTuple(args,"O;ch or int",&temp))
         return NULL;
-    }
+
+    if (!PyCurses_ConvertToChtype(NULL, temp, &ch))
+        return NULL;
 
     return PyCursesCheckERR(ungetch(ch), "ungetch");
 }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 01:23:41 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 26 Nov 2011 01:23:41 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Better_resolution_for_issue?=
 =?utf8?q?_=2311849=3A_Ensure_that_free=28=29d_memory_arenas_are?=
Message-ID: <E1RU63R-0003SB-OA@dinsdale.python.org>

http://hg.python.org/cpython/rev/e7aa72e6aad4
changeset:   73747:e7aa72e6aad4
parent:      73732:736b0aec412b
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 26 01:11:02 2011 +0100
summary:
  Better resolution for issue #11849: Ensure that free()d memory arenas are really released
on POSIX systems supporting anonymous memory mappings.  Patch by Charles-Fran?ois Natali.

files:
  Misc/NEWS          |   8 +++---
  Objects/obmalloc.c |  37 ++++++++++++++++++++-------------
  configure          |  36 +++-----------------------------
  configure.in       |  13 +----------
  pyconfig.h.in      |   6 ++--
  5 files changed, 35 insertions(+), 65 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #11849: Ensure that free()d memory arenas are really released
+  on POSIX systems supporting anonymous memory mappings.  Patch by
+  Charles-Fran?ois Natali.
+
 - Issue #13436: Fix a bogus error message when an AST object was passed
   an invalid integer value.
 
@@ -260,10 +264,6 @@
   interpreter is shutting down; instead, these threads are now killed when
   they try to take the GIL.
 
-- Issue #11849: Make it more likely for the system allocator to release
-  free()d memory arenas on glibc-based systems.  Patch by Charles-Fran?ois
-  Natali.
-
 - Issue #9756: When calling a method descriptor or a slot wrapper descriptor,
   the check of the object type doesn't read the __class__ attribute anymore.
   Fix a crash if a class override its __class__ attribute (e.g. a proxy of the
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -2,8 +2,11 @@
 
 #ifdef WITH_PYMALLOC
 
-#ifdef HAVE_MALLOPT_MMAP_THRESHOLD
-  #include <malloc.h>
+#ifdef HAVE_MMAP
+ #include <sys/mman.h>
+ #ifdef MAP_ANONYMOUS
+  #define ARENAS_USE_MMAP
+ #endif
 #endif
 
 #ifdef WITH_VALGRIND
@@ -183,15 +186,15 @@
 /*
  * The allocator sub-allocates <Big> blocks of memory (called arenas) aligned
  * on a page boundary. This is a reserved virtual address space for the
- * current process (obtained through a malloc call). In no way this means
- * that the memory arenas will be used entirely. A malloc(<Big>) is usually
- * an address range reservation for <Big> bytes, unless all pages within this
- * space are referenced subsequently. So malloc'ing big blocks and not using
- * them does not mean "wasting memory". It's an addressable range wastage...
+ * current process (obtained through a malloc()/mmap() call). In no way this
+ * means that the memory arenas will be used entirely. A malloc(<Big>) is
+ * usually an address range reservation for <Big> bytes, unless all pages within
+ * this space are referenced subsequently. So malloc'ing big blocks and not
+ * using them does not mean "wasting memory". It's an addressable range
+ * wastage...
  *
- * Therefore, allocating arenas with malloc is not optimal, because there is
- * some address space wastage, but this is the most portable way to request
- * memory from the system across various platforms.
+ * Arenas are allocated with mmap() on systems supporting anonymous memory
+ * mappings to reduce heap fragmentation.
  */
 #define ARENA_SIZE              (256 << 10)     /* 256KB */
 
@@ -557,11 +560,6 @@
         if (numarenas > PY_SIZE_MAX / sizeof(*arenas))
             return NULL;                /* overflow */
 #endif
-#ifdef HAVE_MALLOPT_MMAP_THRESHOLD
-        /* Ensure arenas are allocated by mmap to avoid heap fragmentation. */
-        if (numarenas == INITIAL_ARENA_OBJECTS)
-            mallopt(M_MMAP_THRESHOLD, ARENA_SIZE);
-#endif
         nbytes = numarenas * sizeof(*arenas);
         arenaobj = (struct arena_object *)realloc(arenas, nbytes);
         if (arenaobj == NULL)
@@ -594,7 +592,12 @@
     arenaobj = unused_arena_objects;
     unused_arena_objects = arenaobj->nextarena;
     assert(arenaobj->address == 0);
+#ifdef ARENAS_USE_MMAP
+    arenaobj->address = (uptr)mmap(NULL, ARENA_SIZE, PROT_READ|PROT_WRITE,
+                                   MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+#else
     arenaobj->address = (uptr)malloc(ARENA_SIZE);
+#endif
     if (arenaobj->address == 0) {
         /* The allocation failed: return NULL after putting the
          * arenaobj back.
@@ -1071,7 +1074,11 @@
                 unused_arena_objects = ao;
 
                 /* Free the entire arena. */
+#ifdef ARENAS_USE_MMAP
+                munmap((void *)ao->address, ARENA_SIZE);
+#else
                 free((void *)ao->address);
+#endif
                 ao->address = 0;                        /* mark unassociated */
                 --narenas_currently_allocated;
 
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -9401,8 +9401,8 @@
  getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \
  getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \
  if_nameindex \
- initgroups kill killpg lchmod lchown lockf linkat lstat lutimes memrchr \
- mbrtowc mkdirat mkfifo \
+ initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
+ memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
  posix_fallocate posix_fadvise pread \
  pthread_init pthread_kill putenv pwrite readlink readlinkat readv realpath renameat \
@@ -9784,34 +9784,6 @@
 fi
 rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
 
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether mallopt can set malloc mmap threshold" >&5
-$as_echo_n "checking whether mallopt can set malloc mmap threshold... " >&6; }
-cat confdefs.h - <<_ACEOF >conftest.$ac_ext
-/* end confdefs.h.  */
-
-#include <malloc.h>
-
-int
-main ()
-{
-mallopt(M_MMAP_THRESHOLD, 256 * 1024)
-  ;
-  return 0;
-}
-_ACEOF
-if ac_fn_c_try_compile "$LINENO"; then :
-
-$as_echo "#define HAVE_MALLOPT_MMAP_THRESHOLD 1" >>confdefs.h
-
-   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
-$as_echo "yes" >&6; }
-else
-  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
-$as_echo "no" >&6; }
-
-fi
-rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
-
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken unsetenv" >&5
 $as_echo_n "checking for broken unsetenv... " >&6; }
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -14591,8 +14563,8 @@
 
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
 # Files that config.status was made for.
-config_files="$ac_config_files"
-config_headers="$ac_config_headers"
+config_files="`echo $ac_config_files`"
+config_headers="`echo $ac_config_headers`"
 
 _ACEOF
 
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -2567,8 +2567,8 @@
  getgrouplist getgroups getlogin getloadavg getpeername getpgid getpid \
  getpriority getresuid getresgid getpwent getspnam getspent getsid getwd \
  if_nameindex \
- initgroups kill killpg lchmod lchown lockf linkat lstat lutimes memrchr \
- mbrtowc mkdirat mkfifo \
+ initgroups kill killpg lchmod lchown lockf linkat lstat lutimes mmap \
+ memrchr mbrtowc mkdirat mkfifo \
  mkfifoat mknod mknodat mktime mremap nice openat pathconf pause pipe2 plock poll \
  posix_fallocate posix_fadvise pread \
  pthread_init pthread_kill putenv pwrite readlink readlinkat readv realpath renameat \
@@ -2679,15 +2679,6 @@
   [AC_MSG_RESULT(no)
 ])
 
-AC_MSG_CHECKING(whether mallopt can set malloc mmap threshold)
-AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
-#include <malloc.h>
-]], [[mallopt(M_MMAP_THRESHOLD, 256 * 1024)]])],
-  [AC_DEFINE(HAVE_MALLOPT_MMAP_THRESHOLD, 1, Define if mallopt can set malloc mmap threshold.)
-   AC_MSG_RESULT(yes)],
-  [AC_MSG_RESULT(no)
-])
-
 AC_MSG_CHECKING(for broken unsetenv)
 AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
 #include <stdlib.h>
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -512,9 +512,6 @@
 /* Define this if you have the makedev macro. */
 #undef HAVE_MAKEDEV
 
-/* Define if mallopt can set malloc mmap threshold. */
-#undef HAVE_MALLOPT_MMAP_THRESHOLD
-
 /* Define to 1 if you have the `mbrtowc' function. */
 #undef HAVE_MBRTOWC
 
@@ -545,6 +542,9 @@
 /* Define to 1 if you have the `mktime' function. */
 #undef HAVE_MKTIME
 
+/* Define to 1 if you have the `mmap' function. */
+#undef HAVE_MMAP
+
 /* Define to 1 if you have the `mremap' function. */
 #undef HAVE_MREMAP
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 01:23:42 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 26 Nov 2011 01:23:42 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_default_-=3E_default?=
	=?utf8?q?=29=3A_Merge_heads?=
Message-ID: <E1RU63S-0003SG-Q6@dinsdale.python.org>

http://hg.python.org/cpython/rev/7c2a22b94e6b
changeset:   73748:7c2a22b94e6b
parent:      73747:e7aa72e6aad4
parent:      73746:c3581ca21a57
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 26 01:13:12 2011 +0100
summary:
  Merge heads

files:
  Doc/c-api/function.rst      |   10 +
  Doc/c-api/memoryview.rst    |    4 +-
  Doc/data/refcounts.dat      |    5 +
  Doc/faq/programming.rst     |   26 +
  Doc/glossary.rst            |   18 +
  Doc/library/curses.rst      |   12 +-
  Doc/library/stdtypes.rst    |   28 +-
  Doc/reference/datamodel.rst |    5 +
  Doc/whatsnew/3.3.rst        |   64 ++
  Include/funcobject.h        |    2 +
  Include/object.h            |    2 +-
  Include/py_curses.h         |    1 +
  Lib/pydoc.py                |    2 +-
  Lib/tempfile.py             |   18 +-
  Lib/test/test_code.py       |    2 +-
  Lib/test/test_curses.py     |   41 +-
  Lib/test/test_descr.py      |   11 +-
  Lib/test/test_dis.py        |    1 +
  Lib/test/test_funcattrs.py  |   27 +
  Lib/test/test_metaclass.py  |    8 +-
  Lib/test/test_reprlib.py    |   13 +-
  Lib/test/test_sys.py        |    4 +-
  Lib/test/test_tempfile.py   |   53 ++
  Lib/test/test_unicode.py    |   18 +-
  Misc/NEWS                   |   18 +-
  Modules/_cursesmodule.c     |  580 +++++++++++++++++++----
  Objects/funcobject.c        |   44 +-
  Objects/typeobject.c        |   56 ++-
  Objects/unicodeobject.c     |  131 +----
  Python/ceval.c              |    4 +-
  Python/compile.c            |  139 ++++-
  Python/import.c             |    3 +-
  32 files changed, 1042 insertions(+), 308 deletions(-)


diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -38,6 +38,16 @@
    object, the argument defaults and closure are set to *NULL*.
 
 
+.. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
+
+   As :c:func:`PyFunction_New`, but also allows to set the function object's
+   ``__qualname__`` attribute.  *qualname* should be a unicode object or NULL;
+   if NULL, the ``__qualname__`` attribute is set to the same value as its
+   ``__name__`` attribute.
+
+   .. versionadded:: 3.3
+
+
 .. c:function:: PyObject* PyFunction_GetCode(PyObject *op)
 
    Return the code object associated with the function object *op*.
diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst
--- a/Doc/c-api/memoryview.rst
+++ b/Doc/c-api/memoryview.rst
@@ -17,7 +17,7 @@
 
    Create a memoryview object from an object that provides the buffer interface.
    If *obj* supports writable buffer exports, the memoryview object will be
-   readable and writable, other it will be read-only.
+   readable and writable, otherwise it will be read-only.
 
 
 .. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view)
@@ -33,7 +33,7 @@
    Create a memoryview object to a contiguous chunk of memory (in either
    'C' or 'F'ortran *order*) from an object that defines the buffer
    interface. If memory is contiguous, the memoryview object points to the
-   original memory. Otherwise copy is made and the memoryview points to a
+   original memory. Otherwise, a copy is made and the memoryview points to a
    new bytes object.
 
 
diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat
--- a/Doc/data/refcounts.dat
+++ b/Doc/data/refcounts.dat
@@ -465,6 +465,11 @@
 PyFunction_New:PyObject*:code:+1:
 PyFunction_New:PyObject*:globals:+1:
 
+PyFunction_NewWithQualName:PyObject*::+1:
+PyFunction_NewWithQualName:PyObject*:code:+1:
+PyFunction_NewWithQualName:PyObject*:globals:+1:
+PyFunction_NewWithQualName:PyObject*:qualname:+1:
+
 PyFunction_SetClosure:int:::
 PyFunction_SetClosure:PyObject*:op:0:
 PyFunction_SetClosure:PyObject*:closure:+1:
diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst
--- a/Doc/faq/programming.rst
+++ b/Doc/faq/programming.rst
@@ -989,6 +989,32 @@
 See the :ref:`unicode-howto`.
 
 
+What is the most efficient way to concatenate many strings together?
+--------------------------------------------------------------------
+
+:class:`str` and :class:`bytes` objects are immutable, therefore concatenating
+many strings together is inefficient as each concatenation creates a new
+object.  In the general case, the total runtime cost is quadratic in the
+total string length.
+
+To accumulate many :class:`str` objects, the recommended idiom is to place
+them into a list and call :meth:`str.join` at the end::
+
+   chunks = []
+   for s in my_strings:
+       chunks.append(s)
+   result = ''.join(chunks)
+
+(another reasonably efficient idiom is to use :class:`io.StringIO`)
+
+To accumulate many :class:`bytes` objects, the recommended idiom is to extend
+a :class:`bytearray` object using in-place concatenation (the ``+=`` operator)::
+
+   result = bytearray()
+   for b in my_bytes_objects:
+       result += b
+
+
 Sequences (Tuples/Lists)
 ========================
 
diff --git a/Doc/glossary.rst b/Doc/glossary.rst
--- a/Doc/glossary.rst
+++ b/Doc/glossary.rst
@@ -544,6 +544,24 @@
          for piece in food:
              print(piece)
 
+   qualified name
+      A dotted name showing the "path" from a module's global scope to a
+      class, function or method defined in that module, as defined in
+      :pep:`3155`.  For top-level functions and classes, the qualified name
+      is the same as the object's name::
+
+         >>> class C:
+         ...     class D:
+         ...         def meth(self):
+         ...             pass
+         ...
+         >>> C.__qualname__
+         'C'
+         >>> C.D.__qualname__
+         'C.D'
+         >>> C.D.meth.__qualname__
+         'C.D.meth'
+
    reference count
       The number of references to an object.  When the reference count of an
       object drops to zero, it is deallocated.  Reference counting is
diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -653,7 +653,7 @@
 --------------
 
 Window objects, as returned by :func:`initscr` and :func:`newwin` above, have
-the following methods:
+the following methods and attributes:
 
 
 .. method:: window.addch([y, x,] ch[, attr])
@@ -834,6 +834,16 @@
    event.
 
 
+.. attribute:: window.encoding
+
+   Encoding used to encode method arguments (Unicode strings and characters).
+   The encoding attribute is inherited from by parent window when a subwindow
+   is created, for example with :meth:`window.subwin`. By default, the locale
+   encoding is used (see :func:`locale.getpreferredencoding`).
+
+   .. versionadded:: 3.3
+
+
 .. method:: window.erase()
 
    Clear the window.
diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst
--- a/Doc/library/stdtypes.rst
+++ b/Doc/library/stdtypes.rst
@@ -968,15 +968,18 @@
    If *k* is ``None``, it is treated like ``1``.
 
 (6)
-   .. impl-detail::
-
-      If *s* and *t* are both strings, some Python implementations such as
-      CPython can usually perform an in-place optimization for assignments of
-      the form ``s = s + t`` or ``s += t``.  When applicable, this optimization
-      makes quadratic run-time much less likely.  This optimization is both
-      version and implementation dependent.  For performance sensitive code, it
-      is preferable to use the :meth:`str.join` method which assures consistent
-      linear concatenation performance across versions and implementations.
+   Concatenating immutable strings always results in a new object.  This means
+   that building up a string by repeated concatenation will have a quadratic
+   runtime cost in the total string length.  To get a linear runtime cost,
+   you must switch to one of the alternatives below:
+
+   * if concatenating :class:`str` objects, you can build a list and use
+     :meth:`str.join` at the end;
+
+   * if concatenating :class:`bytes` objects, you can similarly use
+     :meth:`bytes.join`, or you can do in-place concatenation with a
+     :class:`bytearray` object.  :class:`bytearray` objects are mutable and
+     have an efficient overallocation mechanism.
 
 
 .. _string-methods:
@@ -2821,6 +2824,13 @@
    The name of the class or type.
 
 
+.. attribute:: class.__qualname__
+
+   The :term:`qualified name` of the class or type.
+
+   .. versionadded:: 3.3
+
+
 .. attribute:: class.__mro__
 
    This attribute is a tuple of classes that are considered when looking for
diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -448,6 +448,11 @@
       +-------------------------+-------------------------------+-----------+
       | :attr:`__name__`        | The function's name           | Writable  |
       +-------------------------+-------------------------------+-----------+
+      | :attr:`__qualname__`    | The function's                | Writable  |
+      |                         | :term:`qualified name`        |           |
+      |                         |                               |           |
+      |                         | .. versionadded:: 3.3         |           |
+      +-------------------------+-------------------------------+-----------+
       | :attr:`__module__`      | The name of the module the    | Writable  |
       |                         | function was defined in, or   |           |
       |                         | ``None`` if unavailable.      |           |
diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -189,6 +189,65 @@
         print("You are not allowed to read document.txt")
 
 
+PEP 3155: Qualified name for classes and functions
+==================================================
+
+:pep:`3155` - Qualified name for classes and functions
+ PEP written and implemented by Antoine Pitrou.
+
+Functions and class objects have a new ``__qualname__`` attribute representing
+the "path" from the module top-level to their definition.  For global functions
+and classes, this is the same as ``__name__``.  For other functions and classes,
+it provides better information about where they were actually defined, and
+how they might be accessible from the global scope.
+
+Example with (non-bound) methods::
+
+   >>> class C:
+   ...     def meth(self):
+   ...         pass
+   >>> C.meth.__name__
+   'meth'
+   >>> C.meth.__qualname__
+   'C.meth'
+
+Example with nested classes::
+
+   >>> class C:
+   ...     class D:
+   ...         def meth(self):
+   ...             pass
+   ...
+   >>> C.D.__name__
+   'D'
+   >>> C.D.__qualname__
+   'C.D'
+   >>> C.D.meth.__name__
+   'meth'
+   >>> C.D.meth.__qualname__
+   'C.D.meth'
+
+Example with nested functions::
+
+   >>> def outer():
+   ...     def inner():
+   ...         pass
+   ...     return inner
+   ...
+   >>> outer().__name__
+   'inner'
+   >>> outer().__qualname__
+   'outer.<locals>.inner'
+
+The string representation of those objects is also changed to include the
+new, more precise information::
+
+   >>> str(C.D)
+   "<class '__main__.C.D'>"
+   >>> str(C.D.meth)
+   '<function C.D.meth at 0x7f46b9fe31e0>'
+
+
 Other Language Changes
 ======================
 
@@ -274,6 +333,11 @@
 curses
 ------
 
+ * If the :mod:`curses` module is linked to the ncursesw library, use Unicode
+   functions when Unicode strings or characters are passed (e.g.
+   :c:func:`waddwstr`), and bytes functions otherwise (e.g. :c:func:`waddstr`).
+ * Use the locale encoding instead of ``utf-8`` to encode Unicode strings.
+ * :class:`curses.window` has a new :attr:`curses.window.encoding` attribute.
  * The :class:`curses.window` class has a new :meth:`~curses.window.get_wch`
    method to get a wide character
  * The :mod:`curses` module has a new :meth:`~curses.unget_wch` function to
diff --git a/Include/funcobject.h b/Include/funcobject.h
--- a/Include/funcobject.h
+++ b/Include/funcobject.h
@@ -31,6 +31,7 @@
     PyObject *func_weakreflist;	/* List of weak references */
     PyObject *func_module;	/* The __module__ attribute, can be anything */
     PyObject *func_annotations;	/* Annotations, a dict or NULL */
+    PyObject *func_qualname;    /* The qualified name */
 
     /* Invariant:
      *     func_closure contains the bindings for func_code->co_freevars, so
@@ -44,6 +45,7 @@
 #define PyFunction_Check(op) (Py_TYPE(op) == &PyFunction_Type)
 
 PyAPI_FUNC(PyObject *) PyFunction_New(PyObject *, PyObject *);
+PyAPI_FUNC(PyObject *) PyFunction_NewWithQualName(PyObject *, PyObject *, PyObject *);
 PyAPI_FUNC(PyObject *) PyFunction_GetCode(PyObject *);
 PyAPI_FUNC(PyObject *) PyFunction_GetGlobals(PyObject *);
 PyAPI_FUNC(PyObject *) PyFunction_GetModule(PyObject *);
diff --git a/Include/object.h b/Include/object.h
--- a/Include/object.h
+++ b/Include/object.h
@@ -418,7 +418,7 @@
                                       a given operator (e.g. __getitem__).
                                       see add_operators() in typeobject.c . */
     PyBufferProcs as_buffer;
-    PyObject *ht_name, *ht_slots;
+    PyObject *ht_name, *ht_slots, *ht_qualname;
     /* here are optional user slots, followed by the members. */
 } PyHeapTypeObject;
 
diff --git a/Include/py_curses.h b/Include/py_curses.h
--- a/Include/py_curses.h
+++ b/Include/py_curses.h
@@ -76,6 +76,7 @@
 typedef struct {
 	PyObject_HEAD
 	WINDOW *win;
+	char *encoding;
 } PyCursesWindowObject;
 
 #define PyCursesWindow_Check(v)	 (Py_TYPE(v) == &PyCursesWindow_Type)
diff --git a/Lib/pydoc.py b/Lib/pydoc.py
--- a/Lib/pydoc.py
+++ b/Lib/pydoc.py
@@ -166,7 +166,7 @@
     if name in {'__builtins__', '__doc__', '__file__', '__path__',
                      '__module__', '__name__', '__slots__', '__package__',
                      '__cached__', '__author__', '__credits__', '__date__',
-                     '__version__'}:
+                     '__version__', '__qualname__'}:
         return 0
     # Private names are hidden, but special names are displayed.
     if name.startswith('__') and name.endswith('__'): return 1
diff --git a/Lib/tempfile.py b/Lib/tempfile.py
--- a/Lib/tempfile.py
+++ b/Lib/tempfile.py
@@ -108,8 +108,13 @@
 
     characters = "abcdefghijklmnopqrstuvwxyz0123456789_"
 
-    def __init__(self):
-        self.rng = _Random()
+    @property
+    def rng(self):
+        cur_pid = _os.getpid()
+        if cur_pid != getattr(self, '_rng_pid', None):
+            self._rng = _Random()
+            self._rng_pid = cur_pid
+        return self._rng
 
     def __iter__(self):
         return self
@@ -578,8 +583,13 @@
     def tell(self):
         return self._file.tell()
 
-    def truncate(self):
-        self._file.truncate()
+    def truncate(self, size=None):
+        if size is None:
+            self._file.truncate()
+        else:
+            if size > self._max_size:
+                self.rollover()
+            self._file.truncate(size)
 
     def write(self, s):
         file = self._file
diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py
--- a/Lib/test/test_code.py
+++ b/Lib/test/test_code.py
@@ -16,7 +16,7 @@
 freevars: ()
 nlocals: 2
 flags: 3
-consts: ('None', '<code object g>')
+consts: ('None', '<code object g>', "'f.<locals>.g'")
 
 >>> dump(f(4).__code__)
 name: g
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -267,24 +267,42 @@
 def test_unget_wch(stdscr):
     if not hasattr(curses, 'unget_wch'):
         return
-    ch = 'a'
-    curses.unget_wch(ch)
-    read = stdscr.get_wch()
-    read = chr(read)
-    if read != ch:
-        raise AssertionError("%r != %r" % (read, ch))
+    for ch in ('a', '\xe9', '\u20ac', '\U0010FFFF'):
+        curses.unget_wch(ch)
+        read = stdscr.get_wch()
+        read = chr(read)
+        if read != ch:
+            raise AssertionError("%r != %r" % (read, ch))
 
-    ch = ord('a')
-    curses.unget_wch(ch)
-    read = stdscr.get_wch()
-    if read != ch:
-        raise AssertionError("%r != %r" % (read, ch))
+        code = ord(ch)
+        curses.unget_wch(code)
+        read = stdscr.get_wch()
+        if read != code:
+            raise AssertionError("%r != %r" % (read, code))
 
 def test_issue10570():
     b = curses.tparm(curses.tigetstr("cup"), 5, 3)
     assert type(b) is bytes
     curses.putp(b)
 
+def test_encoding(stdscr):
+    import codecs
+    encoding = stdscr.encoding
+    codecs.lookup(encoding)
+    try:
+        stdscr.encoding = 10
+    except TypeError:
+        pass
+    else:
+        raise AssertionError("TypeError not raised")
+    stdscr.encoding = encoding
+    try:
+        del stdscr.encoding
+    except TypeError:
+        pass
+    else:
+        raise AssertionError("TypeError not raised")
+
 def main(stdscr):
     curses.savetty()
     try:
@@ -295,6 +313,7 @@
         test_issue6243(stdscr)
         test_unget_wch(stdscr)
         test_issue10570()
+        test_encoding(stdscr)
     finally:
         curses.resetty()
 
diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py
--- a/Lib/test/test_descr.py
+++ b/Lib/test/test_descr.py
@@ -4492,9 +4492,14 @@
         self.assertEqual(type(C.__dict__), type(B.__dict__))
 
     def test_repr(self):
-        # Testing dict_proxy.__repr__
-        dict_ = {k: v for k, v in self.C.__dict__.items()}
-        self.assertEqual(repr(self.C.__dict__), 'dict_proxy({!r})'.format(dict_))
+        # Testing dict_proxy.__repr__.
+        # We can't blindly compare with the repr of another dict as ordering
+        # of keys and values is arbitrary and may differ.
+        r = repr(self.C.__dict__)
+        self.assertTrue(r.startswith('dict_proxy('), r)
+        self.assertTrue(r.endswith(')'), r)
+        for k, v in self.C.__dict__.items():
+            self.assertIn('{!r}: {!r}'.format(k, v), r)
 
 
 class PTypesLongInitTest(unittest.TestCase):
diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py
--- a/Lib/test/test_dis.py
+++ b/Lib/test/test_dis.py
@@ -339,6 +339,7 @@
 Constants:
    0: None
    1: <code object f at (.*), file "(.*)", line (.*)>
+   2: 'tricky.<locals>.f'
 Variable names:
    0: x
    1: y
diff --git a/Lib/test/test_funcattrs.py b/Lib/test/test_funcattrs.py
--- a/Lib/test/test_funcattrs.py
+++ b/Lib/test/test_funcattrs.py
@@ -2,6 +2,15 @@
 import types
 import unittest
 
+
+def global_function():
+    def inner_function():
+        class LocalClass:
+            pass
+        return LocalClass
+    return lambda: inner_function
+
+
 class FuncAttrsTest(unittest.TestCase):
     def setUp(self):
         class F:
@@ -96,6 +105,24 @@
         self.assertEqual(self.fi.a.__name__, 'a')
         self.cannot_set_attr(self.fi.a, "__name__", 'a', AttributeError)
 
+    def test___qualname__(self):
+        # PEP 3155
+        self.assertEqual(self.b.__qualname__, 'FuncAttrsTest.setUp.<locals>.b')
+        self.assertEqual(FuncAttrsTest.setUp.__qualname__, 'FuncAttrsTest.setUp')
+        self.assertEqual(global_function.__qualname__, 'global_function')
+        self.assertEqual(global_function().__qualname__,
+                         'global_function.<locals>.<lambda>')
+        self.assertEqual(global_function()().__qualname__,
+                         'global_function.<locals>.inner_function')
+        self.assertEqual(global_function()()().__qualname__,
+                         'global_function.<locals>.inner_function.<locals>.LocalClass')
+        self.b.__qualname__ = 'c'
+        self.assertEqual(self.b.__qualname__, 'c')
+        self.b.__qualname__ = 'd'
+        self.assertEqual(self.b.__qualname__, 'd')
+        # __qualname__ must be a string
+        self.cannot_set_attr(self.b, '__qualname__', 7, TypeError)
+
     def test___code__(self):
         num_one, num_two = 7, 8
         def a(): pass
diff --git a/Lib/test/test_metaclass.py b/Lib/test/test_metaclass.py
--- a/Lib/test/test_metaclass.py
+++ b/Lib/test/test_metaclass.py
@@ -159,6 +159,7 @@
     ...     bar = 123
     ...
     d['__module__'] = 'test.test_metaclass'
+    d['__qualname__'] = 'C'
     d['foo'] = 4
     d['foo'] = 42
     d['bar'] = 123
@@ -177,12 +178,12 @@
     ...     b = 24
     ...
     meta: C ()
-    ns: [('__module__', 'test.test_metaclass'), ('a', 42), ('b', 24)]
+    ns: [('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('a', 42), ('b', 24)]
     kw: []
     >>> type(C) is dict
     True
     >>> print(sorted(C.items()))
-    [('__module__', 'test.test_metaclass'), ('a', 42), ('b', 24)]
+    [('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('a', 42), ('b', 24)]
     >>>
 
 And again, with a __prepare__ attribute.
@@ -199,11 +200,12 @@
     ...
     prepare: C () [('other', 'booh')]
     d['__module__'] = 'test.test_metaclass'
+    d['__qualname__'] = 'C'
     d['a'] = 1
     d['a'] = 2
     d['b'] = 3
     meta: C ()
-    ns: [('__module__', 'test.test_metaclass'), ('a', 2), ('b', 3)]
+    ns: [('__module__', 'test.test_metaclass'), ('__qualname__', 'C'), ('a', 2), ('b', 3)]
     kw: [('other', 'booh')]
     >>>
 
diff --git a/Lib/test/test_reprlib.py b/Lib/test/test_reprlib.py
--- a/Lib/test/test_reprlib.py
+++ b/Lib/test/test_reprlib.py
@@ -129,8 +129,8 @@
         self.assertIn(s.find("..."), [12, 13])
 
     def test_lambda(self):
-        self.assertTrue(repr(lambda x: x).startswith(
-            "<function <lambda"))
+        r = repr(lambda x: x)
+        self.assertTrue(r.startswith("<function ReprTests.test_lambda.<locals>.<lambda"), r)
         # XXX anonymous functions?  see func_repr
 
     def test_builtin_function(self):
@@ -278,13 +278,14 @@
 ''')
         from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import qux
         # Unbound methods first
-        self.assertTrue(repr(qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod).startswith(
-        '<function amethod'))
+        r = repr(qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod)
+        self.assertTrue(r.startswith('<function aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod'), r)
         # Bound method next
         iqux = qux.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
-        self.assertTrue(repr(iqux.amethod).startswith(
+        r = repr(iqux.amethod)
+        self.assertTrue(r.startswith(
             '<bound method aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.amethod of <%s.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa object at 0x' \
-            % (qux.__name__,) ))
+            % (qux.__name__,) ), r)
 
     def test_builtin_function(self):
         # XXX test built-in functions and methods with really long names
diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -730,7 +730,7 @@
         check(x, size(vh + '12P3i' + CO_MAXBLOCKS*'3i' + 'P' + extras*'P'))
         # function
         def func(): pass
-        check(func, size(h + '11P'))
+        check(func, size(h + '12P'))
         class c():
             @staticmethod
             def foo():
@@ -828,7 +828,7 @@
         # type
         # (PyTypeObject + PyNumberMethods + PyMappingMethods +
         #  PySequenceMethods + PyBufferProcs)
-        s = size(vh + 'P2P15Pl4PP9PP11PI') + size('16Pi17P 3P 10P 2P 2P')
+        s = size(vh + 'P2P15Pl4PP9PP11PI') + size('16Pi17P 3P 10P 2P 3P')
         check(int, s)
         # class
         class newstyleclass(object): pass
diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py
--- a/Lib/test/test_tempfile.py
+++ b/Lib/test/test_tempfile.py
@@ -1,6 +1,7 @@
 # tempfile.py unit tests.
 import tempfile
 import os
+import signal
 import sys
 import re
 import warnings
@@ -135,6 +136,37 @@
         except:
             self.failOnException("iteration")
 
+    @unittest.skipUnless(hasattr(os, 'fork'),
+        "os.fork is required for this test")
+    def test_process_awareness(self):
+        # ensure that the random source differs between
+        # child and parent.
+        read_fd, write_fd = os.pipe()
+        pid = None
+        try:
+            pid = os.fork()
+            if not pid:
+                os.close(read_fd)
+                os.write(write_fd, next(self.r).encode("ascii"))
+                os.close(write_fd)
+                # bypass the normal exit handlers- leave those to
+                # the parent.
+                os._exit(0)
+            parent_value = next(self.r)
+            child_value = os.read(read_fd, len(parent_value)).decode("ascii")
+        finally:
+            if pid:
+                # best effort to ensure the process can't bleed out
+                # via any bugs above
+                try:
+                    os.kill(pid, signal.SIGKILL)
+                except EnvironmentError:
+                    pass
+            os.close(read_fd)
+            os.close(write_fd)
+        self.assertNotEqual(child_value, parent_value)
+
+
 test_classes.append(test__RandomNameSequence)
 
 
@@ -846,6 +878,27 @@
                 pass
         self.assertRaises(ValueError, use_closed)
 
+    def test_truncate_with_size_parameter(self):
+        # A SpooledTemporaryFile can be truncated to zero size
+        f = tempfile.SpooledTemporaryFile(max_size=10)
+        f.write(b'abcdefg\n')
+        f.seek(0)
+        f.truncate()
+        self.assertFalse(f._rolled)
+        self.assertEqual(f._file.getvalue(), b'')
+        # A SpooledTemporaryFile can be truncated to a specific size
+        f = tempfile.SpooledTemporaryFile(max_size=10)
+        f.write(b'abcdefg\n')
+        f.truncate(4)
+        self.assertFalse(f._rolled)
+        self.assertEqual(f._file.getvalue(), b'abcd')
+        # A SpooledTemporaryFile rolls over if truncated to large size
+        f = tempfile.SpooledTemporaryFile(max_size=10)
+        f.write(b'abcdefg\n')
+        f.truncate(20)
+        self.assertTrue(f._rolled)
+        if has_stat:
+            self.assertEqual(os.fstat(f.fileno()).st_size, 20)
 
 test_classes.append(test_SpooledTemporaryFile)
 
diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py
--- a/Lib/test/test_unicode.py
+++ b/Lib/test/test_unicode.py
@@ -1816,20 +1816,10 @@
                          b' 3.14 ')
         self.assertRaises(UnicodeEncodeError,
                           unicode_encodedecimal, "123\u20ac", "strict")
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "replace"),
-                         b'123?')
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "ignore"),
-                         b'123')
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "xmlcharrefreplace"),
-                         b'123&#8364;')
-        self.assertEqual(unicode_encodedecimal("123\u20ac", "backslashreplace"),
-                         b'123\\u20ac')
-        self.assertEqual(unicode_encodedecimal("123\u20ac\N{EM SPACE}", "replace"),
-                         b'123? ')
-        self.assertEqual(unicode_encodedecimal("123\u20ac\u20ac", "replace"),
-                         b'123??')
-        self.assertEqual(unicode_encodedecimal("123\u20ac\u0660", "replace"),
-                         b'123?0')
+        self.assertRaisesRegex(
+            ValueError,
+            "^'decimal' codec can't encode character",
+            unicode_encodedecimal, "123\u20ac", "replace")
 
     def test_transform_decimal(self):
         from _testcapi import unicode_transformdecimaltoascii as transform_decimal
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -14,6 +14,12 @@
   on POSIX systems supporting anonymous memory mappings.  Patch by
   Charles-Fran?ois Natali.
 
+- Issue #13452: PyUnicode_EncodeDecimal() doesn't support error handlers
+  different than "strict" anymore. The caller was unable to compute the
+  size of the output buffer: it depends on the error handler.
+
+- PEP 3155 / issue #13448: Qualified name for classes and functions.
+
 - Issue #13436: Fix a bogus error message when an AST object was passed
   an invalid integer value.
 
@@ -386,10 +392,20 @@
 - Issue #12380: The rjust, ljust and center methods of bytes and bytearray
   now accept a bytearray argument.
 
-
 Library
 -------
 
+- Issue #12567: The curses module uses Unicode functions for Unicode arguments
+  when it is linked to the ncurses library. It encodes also Unicode strings to
+  the locale encoding instead of UTF-8.
+
+- Issue #12856: Ensure child processes do not inherit the parent's random
+  seed for filename generation in the tempfile module.  Patch by Brian
+  Harring.
+
+- Issue #9957: SpooledTemporaryFile.truncate() now accepts an optional size
+  parameter, as other file-like objects.  Patch by Ryan Kelly.
+
 - Issue #13458: Fix a memory leak in the ssl module when decoding a
   certificate with a subjectAltName.  Patch by Robert Xiao.
 
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -121,6 +121,10 @@
 #include <term.h>
 #endif
 
+#ifdef HAVE_LANGINFO_H
+#include <langinfo.h>
+#endif
+
 #if !defined(HAVE_NCURSES_H) && (defined(sgi) || defined(__sun) || defined(SCO5))
 #define STRICT_SYSV_CURSES       /* Don't use ncurses extensions */
 typedef chtype attr_t;           /* No attr_t type is available */
@@ -143,6 +147,8 @@
 /* Tells whether start_color() has been called to initialise color usage. */
 static int initialisedcolors = FALSE;
 
+static char *screen_encoding = NULL;
+
 /* Utility Macros */
 #define PyCursesSetupTermCalled                                         \
     if (initialised_setupterm != TRUE) {                                \
@@ -175,7 +181,7 @@
  */
 
 static PyObject *
-PyCursesCheckERR(int code, char *fname)
+PyCursesCheckERR(int code, const char *fname)
 {
     if (code != ERR) {
         Py_INCREF(Py_None);
@@ -190,28 +196,193 @@
     }
 }
 
+/* Convert an object to a byte (an integer of type chtype):
+
+   - int
+   - bytes of length 1
+   - str of length 1
+
+   Return 1 on success, 0 on error (invalid type or integer overflow). */
 static int
-PyCurses_ConvertToChtype(PyObject *obj, chtype *ch)
+PyCurses_ConvertToChtype(PyCursesWindowObject *win, PyObject *obj, chtype *ch)
 {
-    if (PyLong_CheckExact(obj)) {
-        int overflow;
-        /* XXX should the truncation by the cast also be reported
-           as an error? */
-        *ch = (chtype) PyLong_AsLongAndOverflow(obj, &overflow);
-        if (overflow)
+    long value;
+    if(PyBytes_Check(obj) && PyBytes_Size(obj) == 1) {
+        value = (unsigned char)PyBytes_AsString(obj)[0];
+    }
+    else if (PyUnicode_Check(obj)) {
+        if (PyUnicode_GetLength(obj) != 1) {
+            PyErr_Format(PyExc_TypeError,
+                         "expect bytes or str of length 1, or int, "
+                         "got a str of length %zi",
+                         PyUnicode_GET_LENGTH(obj));
             return 0;
-    } else if(PyBytes_Check(obj)
-              && (PyBytes_Size(obj) == 1)) {
-        *ch = (chtype) *PyBytes_AsString(obj);
-    } else if (PyUnicode_Check(obj) && PyUnicode_GET_LENGTH(obj) == 1) {
-        Py_UCS4 ucs = PyUnicode_READ(PyUnicode_KIND(obj),
-                                     PyUnicode_DATA(obj),
-                                     0);
-        *ch = (chtype)ucs;
-    } else {
+        }
+        value = PyUnicode_READ_CHAR(obj, 0);
+        if (128 < value) {
+            PyObject *bytes;
+            const char *encoding;
+            if (win)
+                encoding = win->encoding;
+            else
+                encoding = screen_encoding;
+            bytes = PyUnicode_AsEncodedObject(obj, encoding, NULL);
+            if (bytes == NULL)
+                return 0;
+            if (PyBytes_GET_SIZE(bytes) == 1)
+                value = (unsigned char)PyBytes_AS_STRING(bytes)[0];
+            else
+                value = -1;
+            Py_DECREF(bytes);
+            if (value < 0)
+                goto overflow;
+        }
+    }
+    else if (PyLong_CheckExact(obj)) {
+        int long_overflow;
+        value = PyLong_AsLongAndOverflow(obj, &long_overflow);
+        if (long_overflow)
+            goto overflow;
+    }
+    else {
+        PyErr_Format(PyExc_TypeError,
+                     "expect bytes or str of length 1, or int, got %s",
+                     Py_TYPE(obj)->tp_name);
         return 0;
     }
+    *ch = (chtype)value;
+    if ((long)*ch != value)
+        goto overflow;
     return 1;
+
+overflow:
+    PyErr_SetString(PyExc_OverflowError,
+                    "byte doesn't fit in chtype");
+    return 0;
+}
+
+/* Convert an object to a byte (chtype) or a character (cchar_t):
+
+    - int
+    - bytes of length 1
+    - str of length 1
+
+   Return:
+
+    - 2 if obj is a character (written into *wch)
+    - 1 if obj is a byte (written into *ch)
+    - 0 on error: raise an exception */
+static int
+PyCurses_ConvertToCchar_t(PyCursesWindowObject *win, PyObject *obj,
+                          chtype *ch
+#ifdef HAVE_NCURSESW
+                          , cchar_t *wch
+#endif
+                          )
+{
+    int ret = 0;
+    long value;
+#ifdef HAVE_NCURSESW
+    wchar_t buffer[2];
+#endif
+
+    if (PyUnicode_Check(obj)) {
+#ifdef HAVE_NCURSESW
+        if (PyUnicode_AsWideChar(obj, buffer, 2) != 1) {
+            PyErr_Format(PyExc_TypeError,
+                         "expect bytes or str of length 1, or int, "
+                         "got a str of length %zi",
+                         PyUnicode_GET_LENGTH(obj));
+            return 0;
+        }
+        memset(wch->chars, 0, sizeof(wch->chars));
+        wch->chars[0] = buffer[0];
+        return 2;
+#else
+        return PyCurses_ConvertToChtype(win, obj, ch);
+#endif
+    }
+    else if(PyBytes_Check(obj) && PyBytes_Size(obj) == 1) {
+        value = (unsigned char)PyBytes_AsString(obj)[0];
+        ret = 1;
+    }
+    else if (PyLong_CheckExact(obj)) {
+        int overflow;
+        value = PyLong_AsLongAndOverflow(obj, &overflow);
+        if (overflow) {
+            PyErr_SetString(PyExc_OverflowError,
+                            "int doesn't fit in long");
+            return 0;
+        }
+#ifdef HAVE_NCURSESW
+        ret = 2;
+#else
+        ret = 1;
+#endif
+    }
+    else {
+        PyErr_Format(PyExc_TypeError,
+                     "expect bytes or str of length 1, or int, got %s",
+                     Py_TYPE(obj)->tp_name);
+        return 0;
+    }
+#ifdef HAVE_NCURSESW
+    if (ret == 2) {
+        memset(wch->chars, 0, sizeof(wch->chars));
+        wch->chars[0] = (wchar_t)value;
+        if ((long)wch->chars[0] != value) {
+            PyErr_Format(PyExc_OverflowError,
+                         "character doesn't fit in wchar_t");
+            return 0;
+        }
+    }
+    else
+#endif
+    {
+        *ch = (chtype)value;
+        if ((long)*ch != value || value < 0 || value > 255) {
+            PyErr_Format(PyExc_OverflowError,
+                         "byte doesn't fit in chtype");
+            return 0;
+        }
+    }
+    return ret;
+}
+
+/* Convert an object to a byte string (char*) or a wide character string
+   (wchar_t*). Return:
+
+    - 2 if obj is a character string (written into *wch)
+    - 1 if obj is a byte string (written into *bytes)
+    - 0 on error: raise an exception */
+static int
+PyCurses_ConvertToString(PyCursesWindowObject *win, PyObject *obj,
+                         PyObject **bytes, wchar_t **wstr)
+{
+    if (PyUnicode_Check(obj)) {
+#ifdef HAVE_NCURSESW
+        assert (wstr != NULL);
+        *wstr = PyUnicode_AsWideCharString(obj, NULL);
+        if (*wstr == NULL)
+            return 0;
+        return 2;
+#else
+        assert (wstr == NULL);
+        *bytes = PyUnicode_AsEncodedObject(obj, win->encoding, NULL);
+        if (*bytes == NULL)
+            return 0;
+        return 1;
+#endif
+    }
+    else if (PyBytes_Check(obj)) {
+        Py_INCREF(obj);
+        *bytes = obj;
+        return 1;
+    }
+
+    PyErr_Format(PyExc_TypeError, "expect bytes or str, got %s",
+                 Py_TYPE(obj)->tp_name);
+    return 0;
 }
 
 /* Function versions of the 3 functions for testing whether curses has been
@@ -357,13 +528,37 @@
 /* Allocation and deallocation of Window Objects */
 
 static PyObject *
-PyCursesWindow_New(WINDOW *win)
+PyCursesWindow_New(WINDOW *win, const char *encoding)
 {
     PyCursesWindowObject *wo;
 
+    if (encoding == NULL) {
+#if defined(MS_WINDOWS)
+        char *buffer[100];
+        UINT cp;
+        cp = GetConsoleOutputCP();
+        if (cp != 0) {
+            PyOS_snprintf(buffer, sizeof(buffer), "cp%u", cp);
+            encoding = buffer;
+        }
+#elif defined(CODESET)
+        const char *codeset = nl_langinfo(CODESET);
+        if (codeset != NULL && codeset[0] != 0)
+            encoding = codeset;
+#endif
+        if (encoding == NULL)
+            encoding = "utf-8";
+    }
+
     wo = PyObject_NEW(PyCursesWindowObject, &PyCursesWindow_Type);
     if (wo == NULL) return NULL;
     wo->win = win;
+    wo->encoding = strdup(encoding);
+    if (wo->encoding == NULL) {
+        Py_DECREF(wo);
+        PyErr_NoMemory();
+        return NULL;
+    }
     return (PyObject *)wo;
 }
 
@@ -371,6 +566,8 @@
 PyCursesWindow_Dealloc(PyCursesWindowObject *wo)
 {
     if (wo->win != stdscr) delwin(wo->win);
+    if (wo->encoding != NULL)
+        free(wo->encoding);
     PyObject_DEL(wo);
 }
 
@@ -380,29 +577,34 @@
 PyCursesWindow_AddCh(PyCursesWindowObject *self, PyObject *args)
 {
     int rtn, x, y, use_xy = FALSE;
-    PyObject *temp;
-    chtype ch = 0;
+    PyObject *chobj;
+    int type;
+    chtype ch;
+#ifdef HAVE_NCURSESW
+    cchar_t wch;
+#endif
     attr_t attr = A_NORMAL;
     long lattr;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 1:
-        if (!PyArg_ParseTuple(args, "O;ch or int", &temp))
+        if (!PyArg_ParseTuple(args, "O;ch or int", &chobj))
             return NULL;
         break;
     case 2:
-        if (!PyArg_ParseTuple(args, "Ol;ch or int,attr", &temp, &lattr))
+        if (!PyArg_ParseTuple(args, "Ol;ch or int,attr", &chobj, &lattr))
             return NULL;
         attr = lattr;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"iiO;y,x,ch or int", &y, &x, &temp))
+        if (!PyArg_ParseTuple(args,"iiO;y,x,ch or int", &y, &x, &chobj))
             return NULL;
         use_xy = TRUE;
         break;
     case 4:
         if (!PyArg_ParseTuple(args,"iiOl;y,x,ch or int, attr",
-                              &y, &x, &temp, &lattr))
+                              &y, &x, &chobj, &lattr))
             return NULL;
         attr = lattr;
         use_xy = TRUE;
@@ -412,17 +614,33 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 or 3 must be a ch or an int");
+#ifdef HAVE_NCURSESW
+    type = PyCurses_ConvertToCchar_t(self, chobj, &ch, &wch);
+    if (type == 2) {
+        funcname = "add_wch";
+        wch.attr = attr;
+        if (use_xy == TRUE)
+            rtn = mvwadd_wch(self->win,y,x, &wch);
+        else {
+            rtn = wadd_wch(self->win, &wch);
+        }
+    }
+    else
+#else
+    type = PyCurses_ConvertToCchar_t(self, chobj, &ch);
+#endif
+    if (type == 1) {
+        funcname = "addch";
+        if (use_xy == TRUE)
+            rtn = mvwaddch(self->win,y,x, ch | attr);
+        else {
+            rtn = waddch(self->win, ch | attr);
+        }
+    }
+    else {
         return NULL;
     }
-
-    if (use_xy == TRUE)
-        rtn = mvwaddch(self->win,y,x, ch | attr);
-    else {
-        rtn = waddch(self->win, ch | attr);
-    }
-    return PyCursesCheckERR(rtn, "addch");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
@@ -430,29 +648,34 @@
 {
     int rtn;
     int x, y;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 1:
-        if (!PyArg_ParseTuple(args,"s;str", &str))
+        if (!PyArg_ParseTuple(args,"O;str", &strobj))
             return NULL;
         break;
     case 2:
-        if (!PyArg_ParseTuple(args,"sl;str,attr", &str, &lattr))
+        if (!PyArg_ParseTuple(args,"Ol;str,attr", &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"iis;int,int,str", &y, &x, &str))
+        if (!PyArg_ParseTuple(args,"iiO;int,int,str", &y, &x, &strobj))
             return NULL;
         use_xy = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisl;int,int,str,attr", &y, &x, &str, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOl;int,int,str,attr", &y, &x, &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -461,47 +684,74 @@
         PyErr_SetString(PyExc_TypeError, "addstr requires 1 to 4 arguments");
         return NULL;
     }
-
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwaddstr(self->win,y,x,str);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "addwstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddwstr(self->win,y,x,wstr);
+        else
+            rtn = waddwstr(self->win,wstr);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = waddstr(self->win,str);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "addstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddstr(self->win,y,x,str);
+        else
+            rtn = waddstr(self->win,str);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "addstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
 PyCursesWindow_AddNStr(PyCursesWindowObject *self, PyObject *args)
 {
     int rtn, x, y, n;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 2:
-        if (!PyArg_ParseTuple(args,"si;str,n", &str, &n))
+        if (!PyArg_ParseTuple(args,"Oi;str,n", &strobj, &n))
             return NULL;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"sil;str,n,attr", &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"Oil;str,n,attr", &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisi;y,x,str,n", &y, &x, &str, &n))
+        if (!PyArg_ParseTuple(args,"iiOi;y,x,str,n", &y, &x, &strobj, &n))
             return NULL;
         use_xy = TRUE;
         break;
     case 5:
-        if (!PyArg_ParseTuple(args,"iisil;y,x,str,n,attr", &y, &x, &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOil;y,x,str,n,attr", &y, &x, &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -510,18 +760,41 @@
         PyErr_SetString(PyExc_TypeError, "addnstr requires 2 to 5 arguments");
         return NULL;
     }
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
 
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwaddnstr(self->win,y,x,str,n);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "addnwstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddnwstr(self->win,y,x,wstr,n);
+        else
+            rtn = waddnwstr(self->win,wstr,n);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = waddnstr(self->win,str,n);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "addnstr";
+        if (use_xy == TRUE)
+            rtn = mvwaddnstr(self->win,y,x,str,n);
+        else
+            rtn = waddnstr(self->win,str,n);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "addnstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
@@ -547,10 +820,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &bkgd)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 or 3 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &bkgd))
         return NULL;
-    }
 
     return PyCursesCheckERR(wbkgd(self->win, bkgd | attr), "bkgd");
 }
@@ -605,10 +876,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &bkgd)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &bkgd))
         return NULL;
-    }
 
     wbkgdset(self->win, bkgd | attr);
     return PyCursesCheckERR(0, "bkgdset");
@@ -633,11 +902,8 @@
         return NULL;
 
     for(i=0; i<8; i++) {
-        if (temp[i] != NULL && !PyCurses_ConvertToChtype(temp[i], &ch[i])) {
-            PyErr_Format(PyExc_TypeError,
-                         "argument %i must be a ch or an int", i+1);
+        if (temp[i] != NULL && !PyCurses_ConvertToChtype(self, temp[i], &ch[i]))
             return NULL;
-        }
     }
 
     wborder(self->win,
@@ -782,7 +1048,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -810,10 +1076,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument 1 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &ch))
         return NULL;
-    }
 
 #ifdef WINDOW_HAS_FLAGS
     if (self->win->_flags & _ISPAD)
@@ -1034,11 +1298,8 @@
     }
 
     if (code != ERR) {
-        if (!PyCurses_ConvertToChtype(temp, &ch)) {
-            PyErr_SetString(PyExc_TypeError,
-                            "argument 1 or 3 must be a ch or an int");
+        if (!PyCurses_ConvertToChtype(self, temp, &ch))
             return NULL;
-        }
         return PyCursesCheckERR(whline(self->win, ch | attr, n), "hline");
     } else
         return PyCursesCheckERR(code, "wmove");
@@ -1079,11 +1340,8 @@
         return NULL;
     }
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError,
-                        "argument 1 or 3 must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(self, temp, &ch))
         return NULL;
-    }
 
     if (use_xy == TRUE)
         rtn = mvwinsch(self->win,y,x, ch | attr);
@@ -1154,29 +1412,34 @@
 {
     int rtn;
     int x, y;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 1:
-        if (!PyArg_ParseTuple(args,"s;str", &str))
+        if (!PyArg_ParseTuple(args,"O;str", &strobj))
             return NULL;
         break;
     case 2:
-        if (!PyArg_ParseTuple(args,"sl;str,attr", &str, &lattr))
+        if (!PyArg_ParseTuple(args,"Ol;str,attr", &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"iis;y,x,str", &y, &x, &str))
+        if (!PyArg_ParseTuple(args,"iiO;y,x,str", &y, &x, &strobj))
             return NULL;
         use_xy = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisl;y,x,str,attr", &y, &x, &str, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOl;y,x,str,attr", &y, &x, &strobj, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -1186,46 +1449,75 @@
         return NULL;
     }
 
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
+
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwinsstr(self->win,y,x,str);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "inswstr";
+        if (use_xy == TRUE)
+            rtn = mvwins_wstr(self->win,y,x,wstr);
+        else
+            rtn = wins_wstr(self->win,wstr);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = winsstr(self->win,str);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "insstr";
+        if (use_xy == TRUE)
+            rtn = mvwinsstr(self->win,y,x,str);
+        else
+            rtn = winsstr(self->win,str);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "insstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
 PyCursesWindow_InsNStr(PyCursesWindowObject *self, PyObject *args)
 {
     int rtn, x, y, n;
-    char *str;
+    int strtype;
+    PyObject *strobj, *bytesobj;
+#ifdef HAVE_NCURSESW
+    wchar_t *wstr = NULL;
+#endif
     attr_t attr = A_NORMAL , attr_old = A_NORMAL;
     long lattr;
     int use_xy = FALSE, use_attr = FALSE;
+    const char *funcname;
 
     switch (PyTuple_Size(args)) {
     case 2:
-        if (!PyArg_ParseTuple(args,"si;str,n", &str, &n))
+        if (!PyArg_ParseTuple(args,"Oi;str,n", &strobj, &n))
             return NULL;
         break;
     case 3:
-        if (!PyArg_ParseTuple(args,"sil;str,n,attr", &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"Oil;str,n,attr", &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_attr = TRUE;
         break;
     case 4:
-        if (!PyArg_ParseTuple(args,"iisi;y,x,str,n", &y, &x, &str, &n))
+        if (!PyArg_ParseTuple(args,"iiOi;y,x,str,n", &y, &x, &strobj, &n))
             return NULL;
         use_xy = TRUE;
         break;
     case 5:
-        if (!PyArg_ParseTuple(args,"iisil;y,x,str,n,attr", &y, &x, &str, &n, &lattr))
+        if (!PyArg_ParseTuple(args,"iiOil;y,x,str,n,attr", &y, &x, &strobj, &n, &lattr))
             return NULL;
         attr = lattr;
         use_xy = use_attr = TRUE;
@@ -1235,17 +1527,41 @@
         return NULL;
     }
 
+#ifdef HAVE_NCURSESW
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, &wstr);
+#else
+    strtype = PyCurses_ConvertToString(self, strobj, &bytesobj, NULL);
+#endif
+    if (strtype == 0)
+        return NULL;
+
     if (use_attr == TRUE) {
         attr_old = getattrs(self->win);
         (void)wattrset(self->win,attr);
     }
-    if (use_xy == TRUE)
-        rtn = mvwinsnstr(self->win,y,x,str,n);
+#ifdef HAVE_NCURSESW
+    if (strtype == 2) {
+        funcname = "insn_wstr";
+        if (use_xy == TRUE)
+            rtn = mvwins_nwstr(self->win,y,x,wstr,n);
+        else
+            rtn = wins_nwstr(self->win,wstr,n);
+        PyMem_Free(wstr);
+    }
     else
-        rtn = winsnstr(self->win,str,n);
+#endif
+    {
+        char *str = PyBytes_AS_STRING(bytesobj);
+        funcname = "insnstr";
+        if (use_xy == TRUE)
+            rtn = mvwinsnstr(self->win,y,x,str,n);
+        else
+            rtn = winsnstr(self->win,str,n);
+        Py_DECREF(bytesobj);
+    }
     if (use_attr == TRUE)
         (void)wattrset(self->win,attr_old);
-    return PyCursesCheckERR(rtn, "insnstr");
+    return PyCursesCheckERR(rtn, funcname);
 }
 
 static PyObject *
@@ -1528,7 +1844,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, self->encoding);
 }
 
 static PyObject *
@@ -1604,16 +1920,51 @@
     }
 
     if (code != ERR) {
-        if (!PyCurses_ConvertToChtype(temp, &ch)) {
-            PyErr_SetString(PyExc_TypeError,
-                            "argument 1 or 3 must be a ch or an int");
+        if (!PyCurses_ConvertToChtype(self, temp, &ch))
             return NULL;
-        }
         return PyCursesCheckERR(wvline(self->win, ch | attr, n), "vline");
     } else
         return PyCursesCheckERR(code, "wmove");
 }
 
+static PyObject *
+PyCursesWindow_get_encoding(PyCursesWindowObject *self, void *closure)
+{
+    return PyUnicode_FromString(self->encoding);
+}
+
+static int
+PyCursesWindow_set_encoding(PyCursesWindowObject *self, PyObject *value)
+{
+    PyObject *ascii;
+    char *encoding;
+
+    /* It is illegal to del win.encoding */
+    if (value == NULL) {
+        PyErr_SetString(PyExc_TypeError,
+                        "encoding may not be deleted");
+        return -1;
+    }
+
+    if (!PyUnicode_Check(value)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "setting encoding to a non-string");
+        return -1;
+    }
+    ascii = PyUnicode_AsASCIIString(value);
+    if (ascii == NULL)
+        return -1;
+    encoding = strdup(PyBytes_AS_STRING(ascii));
+    if (encoding == NULL) {
+        PyErr_NoMemory();
+        return -1;
+    }
+    free(self->encoding);
+    self->encoding = encoding;
+    return 0;
+}
+
+
 static PyMethodDef PyCursesWindow_Methods[] = {
     {"addch",           (PyCFunction)PyCursesWindow_AddCh, METH_VARARGS},
     {"addnstr",         (PyCFunction)PyCursesWindow_AddNStr, METH_VARARGS},
@@ -1701,6 +2052,13 @@
     {NULL,                  NULL}   /* sentinel */
 };
 
+static PyGetSetDef PyCursesWindow_getsets[] = {
+    {"encoding",
+     (getter)PyCursesWindow_get_encoding,
+     (setter)PyCursesWindow_set_encoding,
+     "the typecode character used to create the array"}
+};
+
 /* -------------------------------------------------------*/
 
 PyTypeObject PyCursesWindow_Type = {
@@ -1733,6 +2091,8 @@
     0,                          /*tp_iter*/
     0,                          /*tp_iternext*/
     PyCursesWindow_Methods,     /*tp_methods*/
+    0,                          /* tp_members */
+    PyCursesWindow_getsets,     /* tp_getset */
 };
 
 /*********************************************************************
@@ -1956,7 +2316,7 @@
         PyErr_SetString(PyCursesError, catchall_NULL);
         return NULL;
     }
-    return PyCursesWindow_New(win);
+    return PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -2034,10 +2394,11 @@
 PyCurses_InitScr(PyObject *self)
 {
     WINDOW *win;
+    PyCursesWindowObject *winobj;
 
     if (initialised == TRUE) {
         wrefresh(stdscr);
-        return (PyObject *)PyCursesWindow_New(stdscr);
+        return (PyObject *)PyCursesWindow_New(stdscr, NULL);
     }
 
     win = initscr();
@@ -2129,7 +2490,9 @@
     SetDictInt("LINES", LINES);
     SetDictInt("COLS", COLS);
 
-    return (PyObject *)PyCursesWindow_New(win);
+    winobj = (PyCursesWindowObject *)PyCursesWindow_New(win, NULL);
+    screen_encoding = winobj->encoding;
+    return (PyObject *)winobj;
 }
 
 static PyObject *
@@ -2331,7 +2694,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -2363,7 +2726,7 @@
         return NULL;
     }
 
-    return (PyObject *)PyCursesWindow_New(win);
+    return (PyObject *)PyCursesWindow_New(win, NULL);
 }
 
 static PyObject *
@@ -2680,10 +3043,8 @@
 
     if (!PyArg_ParseTuple(args,"O;ch or int",&temp)) return NULL;
 
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument must be a ch or an int");
+    if (!PyCurses_ConvertToChtype(NULL, temp, &ch))
         return NULL;
-    }
 
     return PyBytes_FromString(unctrl(ch));
 }
@@ -2696,12 +3057,11 @@
 
     PyCursesInitialised;
 
-    if (!PyArg_ParseTuple(args,"O;ch or int",&temp)) return NULL;
-
-    if (!PyCurses_ConvertToChtype(temp, &ch)) {
-        PyErr_SetString(PyExc_TypeError, "argument must be a ch or an int");
+    if (!PyArg_ParseTuple(args,"O;ch or int",&temp))
         return NULL;
-    }
+
+    if (!PyCurses_ConvertToChtype(NULL, temp, &ch))
+        return NULL;
 
     return PyCursesCheckERR(ungetch(ch), "ungetch");
 }
diff --git a/Objects/funcobject.c b/Objects/funcobject.c
--- a/Objects/funcobject.c
+++ b/Objects/funcobject.c
@@ -6,7 +6,7 @@
 #include "structmember.h"
 
 PyObject *
-PyFunction_New(PyObject *code, PyObject *globals)
+PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
 {
     PyFunctionObject *op = PyObject_GC_New(PyFunctionObject,
                                         &PyFunction_Type);
@@ -54,6 +54,11 @@
             Py_INCREF(module);
             op->func_module = module;
         }
+        if (qualname)
+            op->func_qualname = qualname;
+        else
+            op->func_qualname = op->func_name;
+        Py_INCREF(op->func_qualname);
     }
     else
         return NULL;
@@ -62,6 +67,12 @@
 }
 
 PyObject *
+PyFunction_New(PyObject *code, PyObject *globals)
+{
+    return PyFunction_NewWithQualName(code, globals, NULL);
+}
+
+PyObject *
 PyFunction_GetCode(PyObject *op)
 {
     if (!PyFunction_Check(op)) {
@@ -334,6 +345,32 @@
 }
 
 static PyObject *
+func_get_qualname(PyFunctionObject *op)
+{
+    Py_INCREF(op->func_qualname);
+    return op->func_qualname;
+}
+
+static int
+func_set_qualname(PyFunctionObject *op, PyObject *value)
+{
+    PyObject *tmp;
+
+    /* Not legal to del f.__qualname__ or to set it to anything
+     * other than a string object. */
+    if (value == NULL || !PyUnicode_Check(value)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "__qualname__ must be set to a string object");
+        return -1;
+    }
+    tmp = op->func_qualname;
+    Py_INCREF(value);
+    op->func_qualname = value;
+    Py_DECREF(tmp);
+    return 0;
+}
+
+static PyObject *
 func_get_defaults(PyFunctionObject *op)
 {
     if (op->func_defaults == NULL) {
@@ -441,6 +478,7 @@
      (setter)func_set_annotations},
     {"__dict__", (getter)func_get_dict, (setter)func_set_dict},
     {"__name__", (getter)func_get_name, (setter)func_set_name},
+    {"__qualname__", (getter)func_get_qualname, (setter)func_set_qualname},
     {NULL} /* Sentinel */
 };
 
@@ -561,6 +599,7 @@
     Py_XDECREF(op->func_dict);
     Py_XDECREF(op->func_closure);
     Py_XDECREF(op->func_annotations);
+    Py_XDECREF(op->func_qualname);
     PyObject_GC_Del(op);
 }
 
@@ -568,7 +607,7 @@
 func_repr(PyFunctionObject *op)
 {
     return PyUnicode_FromFormat("<function %U at %p>",
-                               op->func_name, op);
+                               op->func_qualname, op);
 }
 
 static int
@@ -584,6 +623,7 @@
     Py_VISIT(f->func_dict);
     Py_VISIT(f->func_closure);
     Py_VISIT(f->func_annotations);
+    Py_VISIT(f->func_qualname);
     return 0;
 }
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -242,6 +242,19 @@
     }
 }
 
+static PyObject *
+type_qualname(PyTypeObject *type, void *context)
+{
+    if (type->tp_flags & Py_TPFLAGS_HEAPTYPE) {
+        PyHeapTypeObject* et = (PyHeapTypeObject*)type;
+        Py_INCREF(et->ht_qualname);
+        return et->ht_qualname;
+    }
+    else {
+        return type_name(type, context);
+    }
+}
+
 static int
 type_set_name(PyTypeObject *type, PyObject *value, void *context)
 {
@@ -286,6 +299,25 @@
     return 0;
 }
 
+static int
+type_set_qualname(PyTypeObject *type, PyObject *value, void *context)
+{
+    PyHeapTypeObject* et;
+
+    if (!PyUnicode_Check(value)) {
+        PyErr_Format(PyExc_TypeError,
+                     "can only assign string to %s.__qualname__, not '%s'",
+                     type->tp_name, Py_TYPE(value)->tp_name);
+        return -1;
+    }
+
+    et = (PyHeapTypeObject*)type;
+    Py_INCREF(value);
+    Py_DECREF(et->ht_qualname);
+    et->ht_qualname = value;
+    return 0;
+}
+
 static PyObject *
 type_module(PyTypeObject *type, void *context)
 {
@@ -631,6 +663,7 @@
 
 static PyGetSetDef type_getsets[] = {
     {"__name__", (getter)type_name, (setter)type_set_name, NULL},
+    {"__qualname__", (getter)type_qualname, (setter)type_set_qualname, NULL},
     {"__bases__", (getter)type_get_bases, (setter)type_set_bases, NULL},
     {"__module__", (getter)type_module, (setter)type_set_module, NULL},
     {"__abstractmethods__", (getter)type_abstractmethods,
@@ -652,7 +685,7 @@
         Py_DECREF(mod);
         mod = NULL;
     }
-    name = type_name(type, NULL);
+    name = type_qualname(type, NULL);
     if (name == NULL)
         return NULL;
 
@@ -1955,7 +1988,7 @@
 {
     PyObject *name, *bases, *dict;
     static char *kwlist[] = {"name", "bases", "dict", 0};
-    PyObject *slots, *tmp, *newslots;
+    PyObject *qualname, *slots, *tmp, *newslots;
     PyTypeObject *type, *base, *tmptype, *winner;
     PyHeapTypeObject *et;
     PyMemberDef *mp;
@@ -2032,6 +2065,18 @@
         return NULL;
     }
 
+    /* Check for a __qualname__ variable in dict */
+    qualname = PyDict_GetItemString(dict, "__qualname__");
+    if (qualname == NULL) {
+        qualname = name;
+    }
+    else {
+        if (PyDict_DelItemString(dict, "__qualname__") < 0) {
+            Py_DECREF(bases);
+            return NULL;
+        }
+    }
+
     /* Check for a __slots__ sequence variable in dict, and count it */
     slots = PyDict_GetItemString(dict, "__slots__");
     nslots = 0;
@@ -2185,7 +2230,9 @@
     /* Keep name and slots alive in the extended type object */
     et = (PyHeapTypeObject *)type;
     Py_INCREF(name);
+    Py_INCREF(qualname);
     et->ht_name = name;
+    et->ht_qualname = qualname;
     et->ht_slots = slots;
 
     /* Initialize tp_flags */
@@ -2369,6 +2416,8 @@
     res->ht_name = PyUnicode_FromString(spec->name);
     if (!res->ht_name)
         goto fail;
+    res->ht_qualname = res->ht_name;
+    Py_INCREF(res->ht_qualname);
     res->ht_type.tp_name = _PyUnicode_AsString(res->ht_name);
     if (!res->ht_type.tp_name)
         goto fail;
@@ -2568,6 +2617,7 @@
      */
     PyObject_Free((char *)type->tp_doc);
     Py_XDECREF(et->ht_name);
+    Py_XDECREF(et->ht_qualname);
     Py_XDECREF(et->ht_slots);
     Py_TYPE(type)->tp_free((PyObject *)type);
 }
@@ -2983,7 +3033,7 @@
         Py_DECREF(mod);
         mod = NULL;
     }
-    name = type_name(type, NULL);
+    name = type_qualname(type, NULL);
     if (name == NULL)
         return NULL;
     if (mod != NULL && PyUnicode_CompareWithASCIIString(mod, "builtins"))
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -8839,15 +8839,8 @@
                         char *output,
                         const char *errors)
 {
-    PyObject *errorHandler = NULL;
-    PyObject *exc = NULL;
     PyObject *unicode;
-    const char *encoding = "decimal";
-    const char *reason = "invalid decimal Unicode string";
-    /* the following variable is used for caching string comparisons
-     * -1=not initialized, 0=unknown, 1=strict, 2=replace, 3=ignore, 4=xmlcharrefreplace */
-    int known_errorHandler = -1;
-    Py_ssize_t i, j;
+    Py_ssize_t i;
     enum PyUnicode_Kind kind;
     void *data;
 
@@ -8860,15 +8853,20 @@
     if (unicode == NULL)
         return -1;
 
-    if (PyUnicode_READY(unicode) < 0)
-        goto onError;
+    if (PyUnicode_READY(unicode) < 0) {
+        Py_DECREF(unicode);
+        return -1;
+    }
     kind = PyUnicode_KIND(unicode);
     data = PyUnicode_DATA(unicode);
 
     for (i=0; i < length; ) {
-        Py_UCS4 ch = PyUnicode_READ(kind, data, i);
+        PyObject *exc;
+        Py_UCS4 ch;
         int decimal;
-        Py_ssize_t startpos, endpos;
+        Py_ssize_t startpos;
+
+        ch = PyUnicode_READ(kind, data, i);
 
         if (Py_UNICODE_ISSPACE(ch)) {
             *output++ = ' ';
@@ -8886,113 +8884,20 @@
             i++;
             continue;
         }
-        /* All other characters are considered unencodable */
+
         startpos = i;
-        endpos = i+1;
-        for (; endpos < length; endpos++) {
-            ch = PyUnicode_READ(kind, data, endpos);
-            if ((0 < ch && ch < 256) ||
-                Py_UNICODE_ISSPACE(ch) ||
-                0 <= Py_UNICODE_TODECIMAL(ch))
-                break;
-        }
-        /* cache callback name lookup
-         * (if not done yet, i.e. it's the first error) */
-        if (known_errorHandler==-1) {
-            if ((errors==NULL) || (!strcmp(errors, "strict")))
-                known_errorHandler = 1;
-            else if (!strcmp(errors, "replace"))
-                known_errorHandler = 2;
-            else if (!strcmp(errors, "ignore"))
-                known_errorHandler = 3;
-            else if (!strcmp(errors, "xmlcharrefreplace"))
-                known_errorHandler = 4;
-            else
-                known_errorHandler = 0;
-        }
-        switch (known_errorHandler) {
-        case 1: /* strict */
-            raise_encode_exception(&exc, encoding, unicode, startpos, endpos, reason);
-            goto onError;
-        case 2: /* replace */
-            for (j=startpos; j < endpos; j++)
-                *output++ = '?';
-            i = endpos;
-            break;
-        case 3: /* ignore */
-            i = endpos;
-            break;
-        case 4: /* xmlcharrefreplace */
-            /* generate replacement */
-            for (j=startpos; j < endpos; j++) {
-                ch = PyUnicode_READ(kind, data, i);
-                output += sprintf(output, "&#%d;", (int)ch);
-                i++;
-            }
-            break;
-        default:
-        {
-            PyObject *repunicode;
-            Py_ssize_t repsize, newpos, k;
-            enum PyUnicode_Kind repkind;
-            void *repdata;
-
-            repunicode = unicode_encode_call_errorhandler(errors, &errorHandler,
-                                                          encoding, reason, unicode, &exc,
-                                                          startpos, endpos, &newpos);
-            if (repunicode == NULL)
-                goto onError;
-            if (!PyUnicode_Check(repunicode)) {
-                /* Byte results not supported, since they have no decimal property. */
-                PyErr_SetString(PyExc_TypeError, "error handler should return unicode");
-                Py_DECREF(repunicode);
-                goto onError;
-            }
-            if (PyUnicode_READY(repunicode) < 0) {
-                Py_DECREF(repunicode);
-                goto onError;
-            }
-            repkind = PyUnicode_KIND(repunicode);
-            repdata = PyUnicode_DATA(repunicode);
-
-            /* generate replacement  */
-            repsize = PyUnicode_GET_SIZE(repunicode);
-            for (k=0; k<repsize; k++) {
-                ch = PyUnicode_READ(repkind, repdata, k);
-                if (Py_UNICODE_ISSPACE(ch))
-                    *output++ = ' ';
-                else {
-                    decimal = Py_UNICODE_TODECIMAL(ch);
-                    if (decimal >= 0)
-                        *output++ = '0' + decimal;
-                    else if (0 < ch && ch < 256)
-                        *output++ = (char)ch;
-                    else {
-                        Py_DECREF(repunicode);
-                        raise_encode_exception(&exc, encoding,
-                                               unicode, startpos, endpos,
-                                               reason);
-                        goto onError;
-                    }
-                }
-            }
-            i = newpos;
-            Py_DECREF(repunicode);
-        }
-        }
+        exc = NULL;
+        raise_encode_exception(&exc, "decimal", unicode,
+                               startpos, startpos+1,
+                               "invalid decimal Unicode string");
+        Py_XDECREF(exc);
+        Py_DECREF(unicode);
+        return -1;
     }
     /* 0-terminate the output string */
     *output++ = '\0';
-    Py_XDECREF(exc);
-    Py_XDECREF(errorHandler);
     Py_DECREF(unicode);
     return 0;
-
-  onError:
-    Py_XDECREF(exc);
-    Py_XDECREF(errorHandler);
-    Py_DECREF(unicode);
-    return -1;
 }
 
 /* --- Helpers ------------------------------------------------------------ */
diff --git a/Python/ceval.c b/Python/ceval.c
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -2687,9 +2687,11 @@
             int kwdefaults = (oparg>>8) & 0xff;
             int num_annotations = (oparg >> 16) & 0x7fff;
 
+            w = POP(); /* qualname */
             v = POP(); /* code object */
-            x = PyFunction_New(v, f->f_globals);
+            x = PyFunction_NewWithQualName(v, f->f_globals, w);
             Py_DECREF(v);
+            Py_DECREF(w);
 
             if (x != NULL && opcode == MAKE_CLOSURE) {
                 v = POP();
diff --git a/Python/compile.c b/Python/compile.c
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -90,6 +90,13 @@
     basicblock *fb_block;
 };
 
+enum {
+    COMPILER_SCOPE_MODULE,
+    COMPILER_SCOPE_CLASS,
+    COMPILER_SCOPE_FUNCTION,
+    COMPILER_SCOPE_COMPREHENSION,
+};
+
 /* The following items change on entry and exit of code blocks.
    They must be saved and restored when returning to a block.
 */
@@ -97,6 +104,9 @@
     PySTEntryObject *u_ste;
 
     PyObject *u_name;
+    PyObject *u_qualname;  /* dot-separated qualified name (lazy) */
+    int u_scope_type;
+
     /* The following fields are dicts that map objects to
        the index of them in co_XXX.      The index is used as
        the argument for opcodes that refer to those collections.
@@ -149,7 +159,7 @@
     PyArena *c_arena;            /* pointer to memory allocation arena */
 };
 
-static int compiler_enter_scope(struct compiler *, identifier, void *, int);
+static int compiler_enter_scope(struct compiler *, identifier, int, void *, int);
 static void compiler_free(struct compiler *);
 static basicblock *compiler_new_block(struct compiler *);
 static int compiler_next_instr(struct compiler *, basicblock *);
@@ -457,6 +467,7 @@
     }
     Py_CLEAR(u->u_ste);
     Py_CLEAR(u->u_name);
+    Py_CLEAR(u->u_qualname);
     Py_CLEAR(u->u_consts);
     Py_CLEAR(u->u_names);
     Py_CLEAR(u->u_varnames);
@@ -467,8 +478,8 @@
 }
 
 static int
-compiler_enter_scope(struct compiler *c, identifier name, void *key,
-                     int lineno)
+compiler_enter_scope(struct compiler *c, identifier name,
+                     int scope_type, void *key, int lineno)
 {
     struct compiler_unit *u;
 
@@ -479,6 +490,7 @@
         return 0;
     }
     memset(u, 0, sizeof(struct compiler_unit));
+    u->u_scope_type = scope_type;
     u->u_argcount = 0;
     u->u_kwonlyargcount = 0;
     u->u_ste = PySymtable_Lookup(c->c_st, key);
@@ -566,6 +578,59 @@
 
 }
 
+static PyObject *
+compiler_scope_qualname(struct compiler *c)
+{
+    Py_ssize_t stack_size, i;
+    _Py_static_string(dot, ".");
+    _Py_static_string(locals, "<locals>");
+    struct compiler_unit *u;
+    PyObject *capsule, *name, *seq, *dot_str, *locals_str;
+
+    u = c->u;
+    if (u->u_qualname != NULL) {
+        Py_INCREF(u->u_qualname);
+        return u->u_qualname;
+    }
+
+    seq = PyList_New(0);
+    if (seq == NULL)
+        return NULL;
+
+    stack_size = PyList_GET_SIZE(c->c_stack);
+    for (i = 0; i < stack_size; i++) {
+        capsule = PyList_GET_ITEM(c->c_stack, i);
+        u = (struct compiler_unit *)PyCapsule_GetPointer(capsule, COMPILER_CAPSULE_NAME_COMPILER_UNIT);
+        assert(u);
+        if (u->u_scope_type == COMPILER_SCOPE_MODULE)
+            continue;
+        if (PyList_Append(seq, u->u_name))
+            goto _error;
+        if (u->u_scope_type == COMPILER_SCOPE_FUNCTION) {
+            locals_str = _PyUnicode_FromId(&locals);
+            if (locals_str == NULL)
+                goto _error;
+            if (PyList_Append(seq, locals_str))
+                goto _error;
+        }
+    }
+    u = c->u;
+    if (PyList_Append(seq, u->u_name))
+        goto _error;
+    dot_str = _PyUnicode_FromId(&dot);
+    if (dot_str == NULL)
+        goto _error;
+    name = PyUnicode_Join(dot_str, seq);
+    Py_DECREF(seq);
+    u->u_qualname = name;
+    Py_XINCREF(name);
+    return name;
+
+_error:
+    Py_XDECREF(seq);
+    return NULL;
+}
+
 /* Allocate a new block and return a pointer to it.
    Returns NULL on error.
 */
@@ -862,9 +927,9 @@
         case CALL_FUNCTION_VAR_KW:
             return -NARGS(oparg)-2;
         case MAKE_FUNCTION:
-            return -NARGS(oparg) - ((oparg >> 16) & 0xffff);
+            return -1 -NARGS(oparg) - ((oparg >> 16) & 0xffff);
         case MAKE_CLOSURE:
-            return -1 - NARGS(oparg) - ((oparg >> 16) & 0xffff);
+            return -2 - NARGS(oparg) - ((oparg >> 16) & 0xffff);
 #undef NARGS
         case BUILD_SLICE:
             if (oparg == 3)
@@ -1194,7 +1259,7 @@
             return NULL;
     }
     /* Use 0 for firstlineno initially, will fixup in assemble(). */
-    if (!compiler_enter_scope(c, module, mod, 0))
+    if (!compiler_enter_scope(c, module, COMPILER_SCOPE_MODULE, mod, 0))
         return NULL;
     switch (mod->kind) {
     case Module_kind:
@@ -1270,11 +1335,15 @@
 }
 
 static int
-compiler_make_closure(struct compiler *c, PyCodeObject *co, int args)
+compiler_make_closure(struct compiler *c, PyCodeObject *co, int args, PyObject *qualname)
 {
     int i, free = PyCode_GetNumFree(co);
+    if (qualname == NULL)
+        qualname = co->co_name;
+
     if (free == 0) {
         ADDOP_O(c, LOAD_CONST, (PyObject*)co, consts);
+        ADDOP_O(c, LOAD_CONST, qualname, consts);
         ADDOP_I(c, MAKE_FUNCTION, args);
         return 1;
     }
@@ -1311,6 +1380,7 @@
     }
     ADDOP_I(c, BUILD_TUPLE, free);
     ADDOP_O(c, LOAD_CONST, (PyObject*)co, consts);
+    ADDOP_O(c, LOAD_CONST, qualname, consts);
     ADDOP_I(c, MAKE_CLOSURE, args);
     return 1;
 }
@@ -1452,7 +1522,7 @@
 compiler_function(struct compiler *c, stmt_ty s)
 {
     PyCodeObject *co;
-    PyObject *first_const = Py_None;
+    PyObject *qualname, *first_const = Py_None;
     arguments_ty args = s->v.FunctionDef.args;
     expr_ty returns = s->v.FunctionDef.returns;
     asdl_seq* decos = s->v.FunctionDef.decorator_list;
@@ -1478,7 +1548,8 @@
         return 0;
     assert((num_annotations & 0xFFFF) == num_annotations);
 
-    if (!compiler_enter_scope(c, s->v.FunctionDef.name, (void *)s,
+    if (!compiler_enter_scope(c, s->v.FunctionDef.name,
+                              COMPILER_SCOPE_FUNCTION, (void *)s,
                               s->lineno))
         return 0;
 
@@ -1500,14 +1571,19 @@
         VISIT_IN_SCOPE(c, stmt, st);
     }
     co = assemble(c, 1);
+    qualname = compiler_scope_qualname(c);
     compiler_exit_scope(c);
-    if (co == NULL)
+    if (qualname == NULL || co == NULL) {
+        Py_XDECREF(qualname);
+        Py_XDECREF(co);
         return 0;
+    }
 
     arglength = asdl_seq_LEN(args->defaults);
     arglength |= kw_default_count << 8;
     arglength |= num_annotations << 16;
-    compiler_make_closure(c, co, arglength);
+    compiler_make_closure(c, co, arglength, qualname);
+    Py_DECREF(qualname);
     Py_DECREF(co);
 
     /* decorators */
@@ -1542,7 +1618,8 @@
     */
 
     /* 1. compile the class body into a code object */
-    if (!compiler_enter_scope(c, s->v.ClassDef.name, (void *)s, s->lineno))
+    if (!compiler_enter_scope(c, s->v.ClassDef.name,
+                              COMPILER_SCOPE_CLASS, (void *)s, s->lineno))
         return 0;
     /* this block represents what we do in the new scope */
     {
@@ -1572,6 +1649,21 @@
             return 0;
         }
         Py_DECREF(str);
+        /* store the __qualname__ */
+        str = compiler_scope_qualname(c);
+        if (!str) {
+            compiler_exit_scope(c);
+            return 0;
+        }
+        ADDOP_O(c, LOAD_CONST, str, consts);
+        Py_DECREF(str);
+        str = PyUnicode_InternFromString("__qualname__");
+        if (!str || !compiler_nameop(c, str, Store)) {
+            Py_XDECREF(str);
+            compiler_exit_scope(c);
+            return 0;
+        }
+        Py_DECREF(str);
         /* compile the body proper */
         if (!compiler_body(c, s->v.ClassDef.body)) {
             compiler_exit_scope(c);
@@ -1608,7 +1700,7 @@
     ADDOP(c, LOAD_BUILD_CLASS);
 
     /* 3. load a function (or closure) made from the code object */
-    compiler_make_closure(c, co, 0);
+    compiler_make_closure(c, co, 0, NULL);
     Py_DECREF(co);
 
     /* 4. load class name */
@@ -1659,6 +1751,7 @@
 compiler_lambda(struct compiler *c, expr_ty e)
 {
     PyCodeObject *co;
+    PyObject *qualname;
     static identifier name;
     int kw_default_count = 0, arglength;
     arguments_ty args = e->v.Lambda.args;
@@ -1678,7 +1771,8 @@
     }
     if (args->defaults)
         VISIT_SEQ(c, expr, args->defaults);
-    if (!compiler_enter_scope(c, name, (void *)e, e->lineno))
+    if (!compiler_enter_scope(c, name, COMPILER_SCOPE_FUNCTION,
+                              (void *)e, e->lineno))
         return 0;
 
     /* Make None the first constant, so the lambda can't have a
@@ -1696,13 +1790,15 @@
         ADDOP_IN_SCOPE(c, RETURN_VALUE);
     }
     co = assemble(c, 1);
+    qualname = compiler_scope_qualname(c);
     compiler_exit_scope(c);
-    if (co == NULL)
+    if (qualname == NULL || co == NULL)
         return 0;
 
     arglength = asdl_seq_LEN(args->defaults);
     arglength |= kw_default_count << 8;
-    compiler_make_closure(c, co, arglength);
+    compiler_make_closure(c, co, arglength, qualname);
+    Py_DECREF(qualname);
     Py_DECREF(co);
 
     return 1;
@@ -2916,11 +3012,13 @@
 {
     PyCodeObject *co = NULL;
     expr_ty outermost_iter;
+    PyObject *qualname = NULL;
 
     outermost_iter = ((comprehension_ty)
                       asdl_seq_GET(generators, 0))->iter;
 
-    if (!compiler_enter_scope(c, name, (void *)e, e->lineno))
+    if (!compiler_enter_scope(c, name, COMPILER_SCOPE_COMPREHENSION,
+                              (void *)e, e->lineno))
         goto error;
 
     if (type != COMP_GENEXP) {
@@ -2953,12 +3051,14 @@
     }
 
     co = assemble(c, 1);
+    qualname = compiler_scope_qualname(c);
     compiler_exit_scope(c);
-    if (co == NULL)
+    if (qualname == NULL || co == NULL)
         goto error;
 
-    if (!compiler_make_closure(c, co, 0))
+    if (!compiler_make_closure(c, co, 0, qualname))
         goto error;
+    Py_DECREF(qualname);
     Py_DECREF(co);
 
     VISIT(c, expr, outermost_iter);
@@ -2968,6 +3068,7 @@
 error_in_scope:
     compiler_exit_scope(c);
 error:
+    Py_XDECREF(qualname);
     Py_XDECREF(co);
     return 0;
 }
diff --git a/Python/import.c b/Python/import.c
--- a/Python/import.c
+++ b/Python/import.c
@@ -103,6 +103,7 @@
                      tag: cpython-32
        Python 3.2a2  3180 (add DELETE_DEREF)
        Python 3.3a0  3190 __class__ super closure changed
+       Python 3.3a0  3200 (__qualname__ added)
 */
 
 /* MAGIC must change whenever the bytecode emitted by the compiler may no
@@ -115,7 +116,7 @@
 #define STRIFY(name) QUOTE(name)
 #define MAJOR STRIFY(PY_MAJOR_VERSION)
 #define MINOR STRIFY(PY_MINOR_VERSION)
-#define MAGIC (3190 | ((long)'\r'<<16) | ((long)'\n'<<24))
+#define MAGIC (3200 | ((long)'\r'<<16) | ((long)'\n'<<24))
 #define TAG "cpython-" MAJOR MINOR;
 #define CACHEDIR "__pycache__"
 /* Current magic word and string tag as globals. */

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 01:47:03 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 26 Nov 2011 01:47:03 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Some_libcs=27_snprintf_may_?=
 =?utf8?q?output_a_leading_minus_sign_when_=25p-formatting?=
Message-ID: <E1RU6Q3-0004y7-Bn@dinsdale.python.org>

http://hg.python.org/cpython/rev/eaddf32cd65c
changeset:   73749:eaddf32cd65c
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 26 01:42:03 2011 +0100
summary:
  Some libcs' snprintf may output a leading minus sign when %p-formatting
(should fix a failure on the OpenIndiana AMD64 buildbot)

files:
  Lib/test/test_gdb.py |  34 ++++++++++++++++----------------
  1 files changed, 17 insertions(+), 17 deletions(-)


diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py
--- a/Lib/test/test_gdb.py
+++ b/Lib/test/test_gdb.py
@@ -315,7 +315,7 @@
 foo = Foo()
 foo.an_int = 42
 id(foo)''')
-        m = re.match(r'<Foo\(an_int=42\) at remote 0x[0-9a-f]+>', gdb_repr)
+        m = re.match(r'<Foo\(an_int=42\) at remote 0x-?[0-9a-f]+>', gdb_repr)
         self.assertTrue(m,
                         msg='Unexpected new-style class rendering %r' % gdb_repr)
 
@@ -328,7 +328,7 @@
 foo += [1, 2, 3]
 foo.an_int = 42
 id(foo)''')
-        m = re.match(r'<Foo\(an_int=42\) at remote 0x[0-9a-f]+>', gdb_repr)
+        m = re.match(r'<Foo\(an_int=42\) at remote 0x-?[0-9a-f]+>', gdb_repr)
 
         self.assertTrue(m,
                         msg='Unexpected new-style class rendering %r' % gdb_repr)
@@ -343,7 +343,7 @@
 foo = Foo((1, 2, 3))
 foo.an_int = 42
 id(foo)''')
-        m = re.match(r'<Foo\(an_int=42\) at remote 0x[0-9a-f]+>', gdb_repr)
+        m = re.match(r'<Foo\(an_int=42\) at remote 0x-?[0-9a-f]+>', gdb_repr)
 
         self.assertTrue(m,
                         msg='Unexpected new-style class rendering %r' % gdb_repr)
@@ -370,7 +370,7 @@
 
         # Match anything for the type name; 0xDEADBEEF could point to
         # something arbitrary (see  http://bugs.python.org/issue8330)
-        pattern = '<.* at remote 0x[0-9a-f]+>'
+        pattern = '<.* at remote 0x-?[0-9a-f]+>'
 
         m = re.match(pattern, gdb_repr)
         if not m:
@@ -416,7 +416,7 @@
         #  http://bugs.python.org/issue8032#msg100537 )
         gdb_repr, gdb_output = self.get_gdb_repr('id(__builtins__.help)', import_site=True)
 
-        m = re.match(r'<_Helper at remote 0x[0-9a-f]+>', gdb_repr)
+        m = re.match(r'<_Helper at remote 0x-?[0-9a-f]+>', gdb_repr)
         self.assertTrue(m,
                         msg='Unexpected rendering %r' % gdb_repr)
 
@@ -447,7 +447,7 @@
 foo = Foo()
 foo.an_attr = foo
 id(foo)''')
-        self.assertTrue(re.match('<Foo\(an_attr=<\.\.\.>\) at remote 0x[0-9a-f]+>',
+        self.assertTrue(re.match('<Foo\(an_attr=<\.\.\.>\) at remote 0x-?[0-9a-f]+>',
                                  gdb_repr),
                         'Unexpected gdb representation: %r\n%s' % \
                             (gdb_repr, gdb_output))
@@ -460,7 +460,7 @@
 foo = Foo()
 foo.an_attr = foo
 id(foo)''')
-        self.assertTrue(re.match('<Foo\(an_attr=<\.\.\.>\) at remote 0x[0-9a-f]+>',
+        self.assertTrue(re.match('<Foo\(an_attr=<\.\.\.>\) at remote 0x-?[0-9a-f]+>',
                                  gdb_repr),
                         'Unexpected gdb representation: %r\n%s' % \
                             (gdb_repr, gdb_output))
@@ -474,7 +474,7 @@
 a.an_attr = b
 b.an_attr = a
 id(a)''')
-        self.assertTrue(re.match('<Foo\(an_attr=<Foo\(an_attr=<\.\.\.>\) at remote 0x[0-9a-f]+>\) at remote 0x[0-9a-f]+>',
+        self.assertTrue(re.match('<Foo\(an_attr=<Foo\(an_attr=<\.\.\.>\) at remote 0x-?[0-9a-f]+>\) at remote 0x-?[0-9a-f]+>',
                                  gdb_repr),
                         'Unexpected gdb representation: %r\n%s' % \
                             (gdb_repr, gdb_output))
@@ -509,7 +509,7 @@
 
     def test_builtin_method(self):
         gdb_repr, gdb_output = self.get_gdb_repr('import sys; id(sys.stdout.readlines)')
-        self.assertTrue(re.match('<built-in method readlines of _io.TextIOWrapper object at remote 0x[0-9a-f]+>',
+        self.assertTrue(re.match('<built-in method readlines of _io.TextIOWrapper object at remote 0x-?[0-9a-f]+>',
                                  gdb_repr),
                         'Unexpected gdb representation: %r\n%s' % \
                             (gdb_repr, gdb_output))
@@ -524,7 +524,7 @@
                                           breakpoint='builtin_id',
                                           cmds_after_breakpoint=['print (PyFrameObject*)(((PyCodeObject*)v)->co_zombieframe)']
                                           )
-        self.assertTrue(re.match('.*\s+\$1 =\s+Frame 0x[0-9a-f]+, for file <string>, line 3, in foo \(\)\s+.*',
+        self.assertTrue(re.match('.*\s+\$1 =\s+Frame 0x-?[0-9a-f]+, for file <string>, line 3, in foo \(\)\s+.*',
                                  gdb_output,
                                  re.DOTALL),
                         'Unexpected gdb representation: %r\n%s' % (gdb_output, gdb_output))
@@ -577,7 +577,7 @@
                                   cmds_after_breakpoint=['py-up'])
         self.assertMultilineMatches(bt,
                                     r'''^.*
-#[0-9]+ Frame 0x[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\)
+#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\)
     baz\(a, b, c\)
 $''')
 
@@ -604,9 +604,9 @@
                                   cmds_after_breakpoint=['py-up', 'py-down'])
         self.assertMultilineMatches(bt,
                                     r'''^.*
-#[0-9]+ Frame 0x[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\)
+#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\)
     baz\(a, b, c\)
-#[0-9]+ Frame 0x[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\)
+#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\)
     id\(42\)
 $''')
 
@@ -634,11 +634,11 @@
                                   cmds_after_breakpoint=['py-bt-full'])
         self.assertMultilineMatches(bt,
                                     r'''^.*
-#[0-9]+ Frame 0x[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\)
+#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\)
     baz\(a, b, c\)
-#[0-9]+ Frame 0x[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\)
+#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\)
     bar\(a, b, c\)
-#[0-9]+ Frame 0x[0-9a-f]+, for file .*gdb_sample.py, line 12, in <module> \(\)
+#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in <module> \(\)
 foo\(1, 2, 3\)
 ''')
 
@@ -667,7 +667,7 @@
         bt = self.get_stack_trace(script=self.get_sample_script(),
                                   cmds_after_breakpoint=['py-print len'])
         self.assertMultilineMatches(bt,
-                                    r".*\nbuiltin 'len' = <built-in method len of module object at remote 0x[0-9a-f]+>\n.*")
+                                    r".*\nbuiltin 'len' = <built-in method len of module object at remote 0x-?[0-9a-f]+>\n.*")
 
 class PyLocalsTests(DebuggerTests):
     def test_basic_command(self):

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sat Nov 26 05:31:56 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sat, 26 Nov 2011 05:31:56 +0100
Subject: [Python-checkins] Daily reference leaks (eaddf32cd65c): sum=0
Message-ID: <E1RU9vg-0002VY-3g@ap.vmr.nerim.net>

results for eaddf32cd65c on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogOqMJpf', '-x']

From python-checkins at python.org  Sat Nov 26 05:38:34 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 05:38:34 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMzgw?=
 =?utf8?q?=3A_add_an_internal_function_for_resetting_the_ctypes_caches?=
Message-ID: <E1RUA26-0002tq-Sp@dinsdale.python.org>

http://hg.python.org/cpython/rev/6783aa5c15ae
changeset:   73750:6783aa5c15ae
branch:      2.7
parent:      73745:e42be90eb9c5
user:        Meador Inge <meadori at gmail.com>
date:        Fri Nov 25 22:25:06 2011 -0600
summary:
  Issue #13380: add an internal function for resetting the ctypes caches

files:
  Lib/ctypes/__init__.py               |  28 +++++++++------
  Lib/ctypes/test/test_as_parameter.py |   1 +
  Lib/ctypes/test/test_functions.py    |   1 +
  Lib/test/regrtest.py                 |   7 ++++
  4 files changed, 26 insertions(+), 11 deletions(-)


diff --git a/Lib/ctypes/__init__.py b/Lib/ctypes/__init__.py
--- a/Lib/ctypes/__init__.py
+++ b/Lib/ctypes/__init__.py
@@ -262,6 +262,22 @@
 
 from _ctypes import POINTER, pointer, _pointer_type_cache
 
+def _reset_cache():
+    _pointer_type_cache.clear()
+    _c_functype_cache.clear()
+    if _os.name in ("nt", "ce"):
+        _win_functype_cache.clear()
+    # _SimpleCData.c_wchar_p_from_param
+    POINTER(c_wchar).from_param = c_wchar_p.from_param
+    # _SimpleCData.c_char_p_from_param
+    POINTER(c_char).from_param = c_char_p.from_param
+    _pointer_type_cache[None] = c_void_p
+    # XXX for whatever reasons, creating the first instance of a callback
+    # function is needed for the unittests on Win64 to succeed.  This MAY
+    # be a compiler bug, since the problem occurs only when _ctypes is
+    # compiled with the MS SDK compiler.  Or an uninitialized variable?
+    CFUNCTYPE(c_int)(lambda: None)
+
 try:
     from _ctypes import set_conversion_mode
 except ImportError:
@@ -278,8 +294,6 @@
     class c_wchar(_SimpleCData):
         _type_ = "u"
 
-    POINTER(c_wchar).from_param = c_wchar_p.from_param #_SimpleCData.c_wchar_p_from_param
-
     def create_unicode_buffer(init, size=None):
         """create_unicode_buffer(aString) -> character array
         create_unicode_buffer(anInteger) -> character array
@@ -298,8 +312,6 @@
             return buf
         raise TypeError(init)
 
-POINTER(c_char).from_param = c_char_p.from_param #_SimpleCData.c_char_p_from_param
-
 # XXX Deprecated
 def SetPointerType(pointer, cls):
     if _pointer_type_cache.get(cls, None) is not None:
@@ -458,8 +470,6 @@
             descr = FormatError(code).strip()
         return WindowsError(code, descr)
 
-_pointer_type_cache[None] = c_void_p
-
 if sizeof(c_uint) == sizeof(c_void_p):
     c_size_t = c_uint
     c_ssize_t = c_int
@@ -542,8 +552,4 @@
     elif sizeof(kind) == 8: c_uint64 = kind
 del(kind)
 
-# XXX for whatever reasons, creating the first instance of a callback
-# function is needed for the unittests on Win64 to succeed.  This MAY
-# be a compiler bug, since the problem occurs only when _ctypes is
-# compiled with the MS SDK compiler.  Or an uninitialized variable?
-CFUNCTYPE(c_int)(lambda: None)
+_reset_cache()
diff --git a/Lib/ctypes/test/test_as_parameter.py b/Lib/ctypes/test/test_as_parameter.py
--- a/Lib/ctypes/test/test_as_parameter.py
+++ b/Lib/ctypes/test/test_as_parameter.py
@@ -74,6 +74,7 @@
     def test_callbacks(self):
         f = dll._testfunc_callback_i_if
         f.restype = c_int
+        f.argtypes = None
 
         MyCallback = CFUNCTYPE(c_int, c_int)
 
diff --git a/Lib/ctypes/test/test_functions.py b/Lib/ctypes/test/test_functions.py
--- a/Lib/ctypes/test/test_functions.py
+++ b/Lib/ctypes/test/test_functions.py
@@ -250,6 +250,7 @@
     def test_callbacks(self):
         f = dll._testfunc_callback_i_if
         f.restype = c_int
+        f.argtypes = None
 
         MyCallback = CFUNCTYPE(c_int, c_int)
 
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1076,6 +1076,13 @@
     filecmp._cache.clear()
     struct._clearcache()
     doctest.master = None
+    try:
+        import ctypes
+    except ImportError:
+        # Don't worry about resetting the cache if ctypes is not supported
+        pass
+    else:
+        ctypes._reset_cache()
 
     # Collect cyclic trash.
     gc.collect()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 05:38:37 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 05:38:37 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzMzgw?=
 =?utf8?q?=3A_add_an_internal_function_for_resetting_the_ctypes_caches?=
Message-ID: <E1RUA29-0002tv-Oa@dinsdale.python.org>

http://hg.python.org/cpython/rev/fa59b3758b14
changeset:   73751:fa59b3758b14
branch:      3.2
parent:      73743:3c9ddd93c983
user:        Meador Inge <meadori at gmail.com>
date:        Fri Nov 25 22:33:32 2011 -0600
summary:
  Issue #13380: add an internal function for resetting the ctypes caches

files:
  Lib/ctypes/__init__.py               |  25 ++++++++++-----
  Lib/ctypes/test/test_as_parameter.py |   1 +
  Lib/ctypes/test/test_functions.py    |   1 +
  Lib/test/regrtest.py                 |   7 ++++
  4 files changed, 25 insertions(+), 9 deletions(-)


diff --git a/Lib/ctypes/__init__.py b/Lib/ctypes/__init__.py
--- a/Lib/ctypes/__init__.py
+++ b/Lib/ctypes/__init__.py
@@ -265,7 +265,21 @@
 class c_wchar(_SimpleCData):
     _type_ = "u"
 
-POINTER(c_wchar).from_param = c_wchar_p.from_param #_SimpleCData.c_wchar_p_from_param
+def _reset_cache():
+    _pointer_type_cache.clear()
+    _c_functype_cache.clear()
+    if _os.name in ("nt", "ce"):
+        _win_functype_cache.clear()
+    # _SimpleCData.c_wchar_p_from_param
+    POINTER(c_wchar).from_param = c_wchar_p.from_param
+    # _SimpleCData.c_char_p_from_param
+    POINTER(c_char).from_param = c_char_p.from_param
+    _pointer_type_cache[None] = c_void_p
+    # XXX for whatever reasons, creating the first instance of a callback
+    # function is needed for the unittests on Win64 to succeed.  This MAY
+    # be a compiler bug, since the problem occurs only when _ctypes is
+    # compiled with the MS SDK compiler.  Or an uninitialized variable?
+    CFUNCTYPE(c_int)(lambda: None)
 
 def create_unicode_buffer(init, size=None):
     """create_unicode_buffer(aString) -> character array
@@ -285,7 +299,6 @@
         return buf
     raise TypeError(init)
 
-POINTER(c_char).from_param = c_char_p.from_param #_SimpleCData.c_char_p_from_param
 
 # XXX Deprecated
 def SetPointerType(pointer, cls):
@@ -445,8 +458,6 @@
             descr = FormatError(code).strip()
         return WindowsError(code, descr)
 
-_pointer_type_cache[None] = c_void_p
-
 if sizeof(c_uint) == sizeof(c_void_p):
     c_size_t = c_uint
     c_ssize_t = c_int
@@ -529,8 +540,4 @@
     elif sizeof(kind) == 8: c_uint64 = kind
 del(kind)
 
-# XXX for whatever reasons, creating the first instance of a callback
-# function is needed for the unittests on Win64 to succeed.  This MAY
-# be a compiler bug, since the problem occurs only when _ctypes is
-# compiled with the MS SDK compiler.  Or an uninitialized variable?
-CFUNCTYPE(c_int)(lambda: None)
+_reset_cache()
diff --git a/Lib/ctypes/test/test_as_parameter.py b/Lib/ctypes/test/test_as_parameter.py
--- a/Lib/ctypes/test/test_as_parameter.py
+++ b/Lib/ctypes/test/test_as_parameter.py
@@ -74,6 +74,7 @@
     def test_callbacks(self):
         f = dll._testfunc_callback_i_if
         f.restype = c_int
+        f.argtypes = None
 
         MyCallback = CFUNCTYPE(c_int, c_int)
 
diff --git a/Lib/ctypes/test/test_functions.py b/Lib/ctypes/test/test_functions.py
--- a/Lib/ctypes/test/test_functions.py
+++ b/Lib/ctypes/test/test_functions.py
@@ -250,6 +250,7 @@
     def test_callbacks(self):
         f = dll._testfunc_callback_i_if
         f.restype = c_int
+        f.argtypes = None
 
         MyCallback = CFUNCTYPE(c_int, c_int)
 
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1255,6 +1255,13 @@
     filecmp._cache.clear()
     struct._clearcache()
     doctest.master = None
+    try:
+        import ctypes
+    except ImportError:
+        # Don't worry about resetting the cache if ctypes is not supported
+        pass
+    else:
+        ctypes._reset_cache()
 
     # Collect cyclic trash.
     gc.collect()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 05:38:38 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 05:38:38 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313380=3A_add_an_internal_function_for_resetting_the?=
 =?utf8?q?_ctypes_caches?=
Message-ID: <E1RUA2A-0002u0-F6@dinsdale.python.org>

http://hg.python.org/cpython/rev/963d861d0eb5
changeset:   73752:963d861d0eb5
parent:      73749:eaddf32cd65c
parent:      73751:fa59b3758b14
user:        Meador Inge <meadori at gmail.com>
date:        Fri Nov 25 22:37:44 2011 -0600
summary:
  Issue #13380: add an internal function for resetting the ctypes caches

files:
  Lib/ctypes/__init__.py               |  25 ++++++++++-----
  Lib/ctypes/test/test_as_parameter.py |   1 +
  Lib/ctypes/test/test_functions.py    |   1 +
  Lib/test/regrtest.py                 |   7 ++++
  4 files changed, 25 insertions(+), 9 deletions(-)


diff --git a/Lib/ctypes/__init__.py b/Lib/ctypes/__init__.py
--- a/Lib/ctypes/__init__.py
+++ b/Lib/ctypes/__init__.py
@@ -265,7 +265,21 @@
 class c_wchar(_SimpleCData):
     _type_ = "u"
 
-POINTER(c_wchar).from_param = c_wchar_p.from_param #_SimpleCData.c_wchar_p_from_param
+def _reset_cache():
+    _pointer_type_cache.clear()
+    _c_functype_cache.clear()
+    if _os.name in ("nt", "ce"):
+        _win_functype_cache.clear()
+    # _SimpleCData.c_wchar_p_from_param
+    POINTER(c_wchar).from_param = c_wchar_p.from_param
+    # _SimpleCData.c_char_p_from_param
+    POINTER(c_char).from_param = c_char_p.from_param
+    _pointer_type_cache[None] = c_void_p
+    # XXX for whatever reasons, creating the first instance of a callback
+    # function is needed for the unittests on Win64 to succeed.  This MAY
+    # be a compiler bug, since the problem occurs only when _ctypes is
+    # compiled with the MS SDK compiler.  Or an uninitialized variable?
+    CFUNCTYPE(c_int)(lambda: None)
 
 def create_unicode_buffer(init, size=None):
     """create_unicode_buffer(aString) -> character array
@@ -285,7 +299,6 @@
         return buf
     raise TypeError(init)
 
-POINTER(c_char).from_param = c_char_p.from_param #_SimpleCData.c_char_p_from_param
 
 # XXX Deprecated
 def SetPointerType(pointer, cls):
@@ -445,8 +458,6 @@
             descr = FormatError(code).strip()
         return WindowsError(code, descr)
 
-_pointer_type_cache[None] = c_void_p
-
 if sizeof(c_uint) == sizeof(c_void_p):
     c_size_t = c_uint
     c_ssize_t = c_int
@@ -529,8 +540,4 @@
     elif sizeof(kind) == 8: c_uint64 = kind
 del(kind)
 
-# XXX for whatever reasons, creating the first instance of a callback
-# function is needed for the unittests on Win64 to succeed.  This MAY
-# be a compiler bug, since the problem occurs only when _ctypes is
-# compiled with the MS SDK compiler.  Or an uninitialized variable?
-CFUNCTYPE(c_int)(lambda: None)
+_reset_cache()
diff --git a/Lib/ctypes/test/test_as_parameter.py b/Lib/ctypes/test/test_as_parameter.py
--- a/Lib/ctypes/test/test_as_parameter.py
+++ b/Lib/ctypes/test/test_as_parameter.py
@@ -74,6 +74,7 @@
     def test_callbacks(self):
         f = dll._testfunc_callback_i_if
         f.restype = c_int
+        f.argtypes = None
 
         MyCallback = CFUNCTYPE(c_int, c_int)
 
diff --git a/Lib/ctypes/test/test_functions.py b/Lib/ctypes/test/test_functions.py
--- a/Lib/ctypes/test/test_functions.py
+++ b/Lib/ctypes/test/test_functions.py
@@ -250,6 +250,7 @@
     def test_callbacks(self):
         f = dll._testfunc_callback_i_if
         f.restype = c_int
+        f.argtypes = None
 
         MyCallback = CFUNCTYPE(c_int, c_int)
 
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1401,6 +1401,13 @@
     filecmp._cache.clear()
     struct._clearcache()
     doctest.master = None
+    try:
+        import ctypes
+    except ImportError:
+        # Don't worry about resetting the cache if ctypes is not supported
+        pass
+    else:
+        ctypes._reset_cache()
 
     # Collect cyclic trash.
     gc.collect()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 06:45:30 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 06:45:30 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyNjE4?=
 =?utf8?q?=3A_create_unit_tests_for_the_py=5Fcompile_module?=
Message-ID: <E1RUB4s-0000M2-K3@dinsdale.python.org>

http://hg.python.org/cpython/rev/bcc7bf3963cc
changeset:   73753:bcc7bf3963cc
branch:      2.7
parent:      73750:6783aa5c15ae
user:        Meador Inge <meadori at gmail.com>
date:        Fri Nov 25 23:28:18 2011 -0600
summary:
  Issue #12618: create unit tests for the py_compile module

files:
  Lib/test/test_py_compile.py |  43 +++++++++++++++++++++++++
  1 files changed, 43 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
new file mode 100644
--- /dev/null
+++ b/Lib/test/test_py_compile.py
@@ -0,0 +1,43 @@
+import imp
+import os
+import py_compile
+import shutil
+import tempfile
+import unittest
+
+from test import test_support
+
+class PyCompileTests(unittest.TestCase):
+
+    def setUp(self):
+        self.directory = tempfile.mkdtemp()
+        self.source_path = os.path.join(self.directory, '_test.py')
+        self.pyc_path = self.source_path + 'c'
+        with open(self.source_path, 'w') as file:
+            file.write('x = 123\n')
+
+    def tearDown(self):
+        shutil.rmtree(self.directory)
+
+    def test_absolute_path(self):
+        py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+
+    def test_cwd(self):
+        cwd = os.getcwd()
+        os.chdir(self.directory)
+        py_compile.compile(os.path.basename(self.source_path),
+                           os.path.basename(self.pyc_path))
+        os.chdir(cwd)
+        self.assertTrue(os.path.exists(self.pyc_path))
+
+    def test_relative_path(self):
+        py_compile.compile(os.path.relpath(self.source_path),
+                           os.path.relpath(self.pyc_path))
+        self.assertTrue(os.path.exists(self.pyc_path))
+
+def test_main():
+    test_support.run_unittest(PyCompileTests)
+
+if __name__ == "__main__":
+    test_main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 06:45:31 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 06:45:31 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNjE4?=
 =?utf8?q?=3A_create_unit_tests_for_the_py=5Fcompile_module?=
Message-ID: <E1RUB4t-0000M6-AV@dinsdale.python.org>

http://hg.python.org/cpython/rev/2be3a2e63683
changeset:   73754:2be3a2e63683
branch:      3.2
parent:      73751:fa59b3758b14
user:        Meador Inge <meadori at gmail.com>
date:        Fri Nov 25 23:36:48 2011 -0600
summary:
  Issue #12618: create unit tests for the py_compile module

files:
  Lib/test/test_py_compile.py |  42 +++++++++++++++++++++++++
  1 files changed, 42 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
new file mode 100644
--- /dev/null
+++ b/Lib/test/test_py_compile.py
@@ -0,0 +1,42 @@
+import imp
+import os
+import py_compile
+import shutil
+import tempfile
+import unittest
+
+from test import support, script_helper
+
+class PyCompileTests(unittest.TestCase):
+
+    def setUp(self):
+        self.directory = tempfile.mkdtemp()
+        self.source_path = os.path.join(self.directory, '_test.py')
+        self.pyc_path = self.source_path + 'c'
+        self.cache_path = imp.cache_from_source(self.source_path)
+        with open(self.source_path, 'w') as file:
+            file.write('x = 123\n')
+
+    def tearDown(self):
+        shutil.rmtree(self.directory)
+
+    def test_absolute_path(self):
+        py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+
+    def test_cache_path(self):
+        py_compile.compile(self.source_path)
+        self.assertTrue(os.path.exists(self.cache_path))
+
+    def test_relative_path(self):
+        py_compile.compile(os.path.relpath(self.source_path),
+                           os.path.relpath(self.pyc_path))
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+
+def test_main():
+    support.run_unittest(PyCompileTests)
+
+if __name__ == "__main__":
+    test_main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 06:45:32 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 06:45:32 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312618=3A_create_unit_tests_for_the_py=5Fcompile_mod?=
 =?utf8?q?ule?=
Message-ID: <E1RUB4u-0000MA-0d@dinsdale.python.org>

http://hg.python.org/cpython/rev/f8f58db0715e
changeset:   73755:f8f58db0715e
parent:      73752:963d861d0eb5
parent:      73754:2be3a2e63683
user:        Meador Inge <meadori at gmail.com>
date:        Fri Nov 25 23:40:53 2011 -0600
summary:
  Issue #12618: create unit tests for the py_compile module

files:
  Lib/test/test_py_compile.py |  42 +++++++++++++++++++++++++
  1 files changed, 42 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
new file mode 100644
--- /dev/null
+++ b/Lib/test/test_py_compile.py
@@ -0,0 +1,42 @@
+import imp
+import os
+import py_compile
+import shutil
+import tempfile
+import unittest
+
+from test import support, script_helper
+
+class PyCompileTests(unittest.TestCase):
+
+    def setUp(self):
+        self.directory = tempfile.mkdtemp()
+        self.source_path = os.path.join(self.directory, '_test.py')
+        self.pyc_path = self.source_path + 'c'
+        self.cache_path = imp.cache_from_source(self.source_path)
+        with open(self.source_path, 'w') as file:
+            file.write('x = 123\n')
+
+    def tearDown(self):
+        shutil.rmtree(self.directory)
+
+    def test_absolute_path(self):
+        py_compile.compile(self.source_path, self.pyc_path)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+
+    def test_cache_path(self):
+        py_compile.compile(self.source_path)
+        self.assertTrue(os.path.exists(self.cache_path))
+
+    def test_relative_path(self):
+        py_compile.compile(os.path.relpath(self.source_path),
+                           os.path.relpath(self.pyc_path))
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+
+def test_main():
+    support.run_unittest(PyCompileTests)
+
+if __name__ == "__main__":
+    test_main()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 11:33:26 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sat, 26 Nov 2011 11:33:26 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313415=3A_Skip_test?=
 =?utf8?q?=5Fos=2Etest=5Funset=5Ferror_on_FreeBSD_=3C_7_and_OS_X_=3C_10=2E?=
 =?utf8?q?6?=
Message-ID: <E1RUFZW-0001hs-7L@dinsdale.python.org>

http://hg.python.org/cpython/rev/23ed66484ff2
changeset:   73756:23ed66484ff2
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sat Nov 26 11:32:46 2011 +0100
summary:
  Issue #13415: Skip test_os.test_unset_error on FreeBSD < 7 and OS X < 10.6
(where unsetenv() doesn't return a value).

files:
  Lib/test/test_os.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -447,6 +447,10 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    # On FreeBSD < 7 and OS X < 10.6, unsetenv() doesn't return a value (issue
+    # #13415).
+    @support.requires_freebsd_version(7)
+    @support.requires_mac_ver(10, 6)
     def test_unset_error(self):
         if sys.platform == "win32":
             # an environment variable is limited to 32,767 characters

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 12:17:47 2011
From: python-checkins at python.org (giampaolo.rodola)
Date: Sat, 26 Nov 2011 12:17:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_sched=2Epy=3A_fix_test=5Fqu?=
 =?utf8?q?eue_by_making_sure_that_queue_property_returns_an_orderd?=
Message-ID: <E1RUGGR-0004Fo-54@dinsdale.python.org>

http://hg.python.org/cpython/rev/fb5e78e0539c
changeset:   73757:fb5e78e0539c
user:        Giampaolo Rodola' <g.rodola at gmail.com>
date:        Sat Nov 26 12:17:42 2011 +0100
summary:
  sched.py: fix test_queue by making sure that queue property returns an orderd list of upcoming events

files:
  Lib/test/test_sched.py |  15 ++++++++-------
  1 files changed, 8 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_sched.py b/Lib/test/test_sched.py
--- a/Lib/test/test_sched.py
+++ b/Lib/test/test_sched.py
@@ -62,15 +62,16 @@
 
     def test_queue(self):
         l = []
-        events = []
         fun = lambda x: l.append(x)
         scheduler = sched.scheduler(time.time, time.sleep)
-        self.assertEqual(scheduler._queue, [])
-        for x in [0.05, 0.04, 0.03, 0.02, 0.01]:
-            events.append(scheduler.enterabs(x, 1, fun, (x,)))
-        self.assertEqual(scheduler._queue.sort(), events.sort())
-        scheduler.run()
-        self.assertEqual(scheduler._queue, [])
+        e5 = scheduler.enter(0.05, 1, fun)
+        e1 = scheduler.enter(0.01, 1, fun)
+        e2 = scheduler.enter(0.02, 1, fun)
+        e4 = scheduler.enter(0.04, 1, fun)
+        e3 = scheduler.enter(0.03, 1, fun)
+        # queue property is supposed to return an order list of
+        # upcoming events
+        self.assertEqual(list(scheduler.queue), [e1, e2, e3, e4, e5])
 
     def test_args_kwargs(self):
         flag = []

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 19:01:21 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 19:01:21 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEyNjE4?=
 =?utf8?q?=3A_fix_py=5Fcompile_unit_tests_to_handle_different_drives_on_Wi?=
 =?utf8?q?ndows?=
Message-ID: <E1RUMYz-0001qP-A7@dinsdale.python.org>

http://hg.python.org/cpython/rev/b23453530d5f
changeset:   73758:b23453530d5f
branch:      2.7
parent:      73753:bcc7bf3963cc
user:        Meador Inge <meadori at gmail.com>
date:        Sat Nov 26 11:30:21 2011 -0600
summary:
  Issue #12618: fix py_compile unit tests to handle different drives on Windows

files:
  Lib/test/test_py_compile.py |  11 +++++++++++
  1 files changed, 11 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
--- a/Lib/test/test_py_compile.py
+++ b/Lib/test/test_py_compile.py
@@ -13,11 +13,22 @@
         self.directory = tempfile.mkdtemp()
         self.source_path = os.path.join(self.directory, '_test.py')
         self.pyc_path = self.source_path + 'c'
+        self.cwd_drive = os.path.splitdrive(os.getcwd())[0]
+        # In these tests we compute relative paths.  When using Windows, the
+        # current working directory path and the 'self.source_path' might be
+        # on different drives.  Therefore we need to switch to the drive where
+        # the temporary source file lives.
+        drive = os.path.splitdrive(self.source_path)[0]
+        if drive:
+            os.chdir(drive)
+
         with open(self.source_path, 'w') as file:
             file.write('x = 123\n')
 
     def tearDown(self):
         shutil.rmtree(self.directory)
+        if self.cwd_drive:
+            os.chdir(self.cwd_drive)
 
     def test_absolute_path(self):
         py_compile.compile(self.source_path, self.pyc_path)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 19:01:22 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 19:01:22 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNjE4?=
 =?utf8?q?=3A_fix_py=5Fcompile_unit_tests_to_handle_different_drives_on_Wi?=
 =?utf8?q?ndows?=
Message-ID: <E1RUMZ0-0001qT-1B@dinsdale.python.org>

http://hg.python.org/cpython/rev/7097d52cacee
changeset:   73759:7097d52cacee
branch:      3.2
parent:      73754:2be3a2e63683
user:        Meador Inge <meadori at gmail.com>
date:        Sat Nov 26 11:37:02 2011 -0600
summary:
  Issue #12618: fix py_compile unit tests to handle different drives on Windows

files:
  Lib/test/test_py_compile.py |  10 ++++++++++
  1 files changed, 10 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
--- a/Lib/test/test_py_compile.py
+++ b/Lib/test/test_py_compile.py
@@ -14,11 +14,21 @@
         self.source_path = os.path.join(self.directory, '_test.py')
         self.pyc_path = self.source_path + 'c'
         self.cache_path = imp.cache_from_source(self.source_path)
+        self.cwd_drive = os.path.splitdrive(os.getcwd())[0]
+        # In these tests we compute relative paths.  When using Windows, the
+        # current working directory path and the 'self.source_path' might be
+        # on different drives.  Therefore we need to switch to the drive where
+        # the temporary source file lives.
+        drive = os.path.splitdrive(self.source_path)[0]
+        if drive:
+            os.chdir(drive)
         with open(self.source_path, 'w') as file:
             file.write('x = 123\n')
 
     def tearDown(self):
         shutil.rmtree(self.directory)
+        if self.cwd_drive:
+            os.chdir(self.cwd_drive)
 
     def test_absolute_path(self):
         py_compile.compile(self.source_path, self.pyc_path)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 19:01:22 2011
From: python-checkins at python.org (meador.inge)
Date: Sat, 26 Nov 2011 19:01:22 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312618=3A_fix_py=5Fcompile_unit_tests_to_handle_diff?=
 =?utf8?q?erent_drives_on_Windows?=
Message-ID: <E1RUMZ0-0001qX-T8@dinsdale.python.org>

http://hg.python.org/cpython/rev/5243752e19aa
changeset:   73760:5243752e19aa
parent:      73757:fb5e78e0539c
parent:      73759:7097d52cacee
user:        Meador Inge <meadori at gmail.com>
date:        Sat Nov 26 11:39:49 2011 -0600
summary:
  Issue #12618: fix py_compile unit tests to handle different drives on Windows

files:
  Lib/test/test_py_compile.py |  10 ++++++++++
  1 files changed, 10 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
--- a/Lib/test/test_py_compile.py
+++ b/Lib/test/test_py_compile.py
@@ -14,11 +14,21 @@
         self.source_path = os.path.join(self.directory, '_test.py')
         self.pyc_path = self.source_path + 'c'
         self.cache_path = imp.cache_from_source(self.source_path)
+        self.cwd_drive = os.path.splitdrive(os.getcwd())[0]
+        # In these tests we compute relative paths.  When using Windows, the
+        # current working directory path and the 'self.source_path' might be
+        # on different drives.  Therefore we need to switch to the drive where
+        # the temporary source file lives.
+        drive = os.path.splitdrive(self.source_path)[0]
+        if drive:
+            os.chdir(drive)
         with open(self.source_path, 'w') as file:
             file.write('x = 123\n')
 
     def tearDown(self):
         shutil.rmtree(self.directory)
+        if self.cwd_drive:
+            os.chdir(self.cwd_drive)
 
     def test_absolute_path(self):
         py_compile.compile(self.source_path, self.pyc_path)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 22:11:57 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 26 Nov 2011 22:11:57 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDQ0?=
 =?utf8?q?=3A_When_stdout_has_been_closed_explicitly=2C_we_should_not_atte?=
 =?utf8?q?mpt_to?=
Message-ID: <E1RUPXR-0000Cx-Vf@dinsdale.python.org>

http://hg.python.org/cpython/rev/150e096095e5
changeset:   73761:150e096095e5
branch:      3.2
parent:      73759:7097d52cacee
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 26 21:59:36 2011 +0100
summary:
  Issue #13444: When stdout has been closed explicitly, we should not attempt to flush it at shutdown and print an error.

This also adds a test for issue #5319, whose resolution introduced the issue.

files:
  Lib/test/test_cmd_line.py |  19 +++++++++++++++++++
  Misc/NEWS                 |   3 +++
  Python/pythonrun.c        |  20 ++++++++++++++++++--
  3 files changed, 40 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -272,6 +272,25 @@
         self.assertRegex(err.decode('ascii', 'ignore'), 'SyntaxError')
         self.assertEqual(b'', out)
 
+    def test_stdout_flush_at_shutdown(self):
+        # Issue #5319: if stdout.flush() fails at shutdown, an error should
+        # be printed out.
+        code = """if 1:
+            import os, sys
+            sys.stdout.write('x')
+            os.close(sys.stdout.fileno())"""
+        rc, out, err = assert_python_ok('-c', code)
+        self.assertEqual(b'', out)
+        self.assertRegex(err.decode('ascii', 'ignore'),
+                         'Exception IOError: .* ignored')
+
+    def test_closed_stdout(self):
+        # Issue #13444: if stdout has been explicitly closed, we should
+        # not attempt to flush it at shutdown.
+        code = "import sys; sys.stdout.close()"
+        rc, out, err = assert_python_ok('-c', code)
+        self.assertEqual(b'', err)
+
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -83,6 +83,9 @@
 Library
 -------
 
+- Issue #13444: When stdout has been closed explicitly, we should not attempt
+  to flush it at shutdown and print an error.
+
 - Issue #12856: Ensure child processes do not inherit the parent's random
   seed for filename generation in the tempfile module.  Patch by Brian
   Harring.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -332,6 +332,22 @@
 
 /* Flush stdout and stderr */
 
+static int
+file_is_closed(PyObject *fobj)
+{
+    int r;
+    PyObject *tmp = PyObject_GetAttrString(fobj, "closed");
+    if (tmp == NULL) {
+        PyErr_Clear();
+        return 0;
+    }
+    r = PyObject_IsTrue(tmp);
+    Py_DECREF(tmp);
+    if (r < 0)
+        PyErr_Clear();
+    return r > 0;
+}
+
 static void
 flush_std_files(void)
 {
@@ -339,7 +355,7 @@
     PyObject *ferr = PySys_GetObject("stderr");
     PyObject *tmp;
 
-    if (fout != NULL && fout != Py_None) {
+    if (fout != NULL && fout != Py_None && !file_is_closed(fout)) {
         tmp = PyObject_CallMethod(fout, "flush", "");
         if (tmp == NULL)
             PyErr_WriteUnraisable(fout);
@@ -347,7 +363,7 @@
             Py_DECREF(tmp);
     }
 
-    if (ferr != NULL && ferr != Py_None) {
+    if (ferr != NULL && ferr != Py_None && !file_is_closed(ferr)) {
         tmp = PyObject_CallMethod(ferr, "flush", "");
         if (tmp == NULL)
             PyErr_Clear();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sat Nov 26 22:11:58 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Sat, 26 Nov 2011 22:11:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2313444=3A_When_stdout_has_been_closed_explicitly=2C_?=
 =?utf8?q?we_should_not_attempt_to?=
Message-ID: <E1RUPXS-0000D5-Qu@dinsdale.python.org>

http://hg.python.org/cpython/rev/37300a1df7d7
changeset:   73762:37300a1df7d7
parent:      73760:5243752e19aa
parent:      73761:150e096095e5
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Sat Nov 26 22:02:29 2011 +0100
summary:
  Issue #13444: When stdout has been closed explicitly, we should not attempt to flush it at shutdown and print an error.

This also adds a test for issue #5319, whose resolution introduced the issue.

files:
  Lib/test/test_cmd_line.py |  19 +++++++++++++++++++
  Misc/NEWS                 |   3 +++
  Python/pythonrun.c        |  20 ++++++++++++++++++--
  3 files changed, 40 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -266,6 +266,25 @@
         self.assertRegex(err.decode('ascii', 'ignore'), 'SyntaxError')
         self.assertEqual(b'', out)
 
+    def test_stdout_flush_at_shutdown(self):
+        # Issue #5319: if stdout.flush() fails at shutdown, an error should
+        # be printed out.
+        code = """if 1:
+            import os, sys
+            sys.stdout.write('x')
+            os.close(sys.stdout.fileno())"""
+        rc, out, err = assert_python_ok('-c', code)
+        self.assertEqual(b'', out)
+        self.assertRegex(err.decode('ascii', 'ignore'),
+                         'Exception OSError: .* ignored')
+
+    def test_closed_stdout(self):
+        # Issue #13444: if stdout has been explicitly closed, we should
+        # not attempt to flush it at shutdown.
+        code = "import sys; sys.stdout.close()"
+        rc, out, err = assert_python_ok('-c', code)
+        self.assertEqual(b'', err)
+
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -395,6 +395,9 @@
 Library
 -------
 
+- Issue #13444: When stdout has been closed explicitly, we should not attempt
+  to flush it at shutdown and print an error.
+
 - Issue #12567: The curses module uses Unicode functions for Unicode arguments
   when it is linked to the ncurses library. It encodes also Unicode strings to
   the locale encoding instead of UTF-8.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -348,6 +348,22 @@
 
 /* Flush stdout and stderr */
 
+static int
+file_is_closed(PyObject *fobj)
+{
+    int r;
+    PyObject *tmp = PyObject_GetAttrString(fobj, "closed");
+    if (tmp == NULL) {
+        PyErr_Clear();
+        return 0;
+    }
+    r = PyObject_IsTrue(tmp);
+    Py_DECREF(tmp);
+    if (r < 0)
+        PyErr_Clear();
+    return r > 0;
+}
+
 static void
 flush_std_files(void)
 {
@@ -356,7 +372,7 @@
     PyObject *tmp;
     _Py_IDENTIFIER(flush);
 
-    if (fout != NULL && fout != Py_None) {
+    if (fout != NULL && fout != Py_None && !file_is_closed(fout)) {
         tmp = _PyObject_CallMethodId(fout, &PyId_flush, "");
         if (tmp == NULL)
             PyErr_WriteUnraisable(fout);
@@ -364,7 +380,7 @@
             Py_DECREF(tmp);
     }
 
-    if (ferr != NULL && ferr != Py_None) {
+    if (ferr != NULL && ferr != Py_None && !file_is_closed(ferr)) {
         tmp = _PyObject_CallMethodId(ferr, &PyId_flush, "");
         if (tmp == NULL)
             PyErr_Clear();

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 00:20:15 2011
From: python-checkins at python.org (victor.stinner)
Date: Sun, 27 Nov 2011 00:20:15 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313415=3A_Help_to_l?=
 =?utf8?q?ocate_curses=2Eh_when_=5Fcurses_module_is_linked_to_ncursesw?=
Message-ID: <E1RURXb-0003sx-UE@dinsdale.python.org>

http://hg.python.org/cpython/rev/919259054621
changeset:   73763:919259054621
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Sun Nov 27 00:19:53 2011 +0100
summary:
  Issue #13415: Help to locate curses.h when _curses module is linked to ncursesw

files:
  Lib/test/test_curses.py |  5 ++++-
  setup.py                |  4 ++++
  2 files changed, 8 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -268,7 +268,10 @@
     if not hasattr(curses, 'unget_wch'):
         return
     for ch in ('a', '\xe9', '\u20ac', '\U0010FFFF'):
-        curses.unget_wch(ch)
+        try:
+            curses.unget_wch(ch)
+        except Exception as err:
+            raise Exception("unget_wch(%a) failed: %s" % (ch, err))
         read = stdscr.get_wch()
         read = chr(read)
         if read != ch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1173,13 +1173,16 @@
         # Curses support, requiring the System V version of curses, often
         # provided by the ncurses library.
         panel_library = 'panel'
+        curses_includes = []
         if curses_library.startswith('ncurses'):
             if curses_library == 'ncursesw':
                 # Bug 1464056: If _curses.so links with ncursesw,
                 # _curses_panel.so must link with panelw.
                 panel_library = 'panelw'
+                curses_includes = ['/usr/include/ncursesw']
             curses_libs = [curses_library]
             exts.append( Extension('_curses', ['_cursesmodule.c'],
+                                   include_dirs=curses_includes,
                                    define_macros=curses_defines,
                                    libraries = curses_libs) )
         elif curses_library == 'curses' and platform != 'darwin':
@@ -1202,6 +1205,7 @@
         if (module_enabled(exts, '_curses') and
             self.compiler.find_library_file(lib_dirs, panel_library)):
             exts.append( Extension('_curses_panel', ['_curses_panel.c'],
+                                   include_dirs=curses_includes,
                                    libraries = [panel_library] + curses_libs) )
         else:
             missing.append('_curses_panel')

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 05:17:37 2011
From: python-checkins at python.org (jesus.cea)
Date: Sun, 27 Nov 2011 05:17:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Closes_issue_?=
 =?utf8?q?=2313488=3A_Some_old_preprocessors_have_problem_with_=23define_n?=
 =?utf8?q?ot_in?=
Message-ID: <E1RUWBN-0006OS-3G@dinsdale.python.org>

http://hg.python.org/cpython/rev/05979ae431fb
changeset:   73764:05979ae431fb
branch:      3.2
parent:      73761:150e096095e5
user:        Jesus Cea <jcea at jcea.es>
date:        Sun Nov 27 05:16:22 2011 +0100
summary:
  Closes issue #13488: Some old preprocessors have problem with #define not in the first column

files:
  Include/dynamic_annotations.h |  160 +++++++++++-----------
  1 files changed, 80 insertions(+), 80 deletions(-)


diff --git a/Include/dynamic_annotations.h b/Include/dynamic_annotations.h
--- a/Include/dynamic_annotations.h
+++ b/Include/dynamic_annotations.h
@@ -103,26 +103,26 @@
 
   /* Report that wait on the condition variable at address "cv" has succeeded
      and the lock at address "lock" is held. */
-  #define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) \
+#define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) \
     AnnotateCondVarWait(__FILE__, __LINE__, cv, lock)
 
   /* Report that wait on the condition variable at "cv" has succeeded.  Variant
      w/o lock. */
-  #define _Py_ANNOTATE_CONDVAR_WAIT(cv) \
+#define _Py_ANNOTATE_CONDVAR_WAIT(cv) \
     AnnotateCondVarWait(__FILE__, __LINE__, cv, NULL)
 
   /* Report that we are about to signal on the condition variable at address
      "cv". */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) \
+#define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) \
     AnnotateCondVarSignal(__FILE__, __LINE__, cv)
 
   /* Report that we are about to signal_all on the condition variable at "cv". */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) \
+#define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) \
     AnnotateCondVarSignalAll(__FILE__, __LINE__, cv)
 
   /* Annotations for user-defined synchronization mechanisms. */
-  #define _Py_ANNOTATE_HAPPENS_BEFORE(obj) _Py_ANNOTATE_CONDVAR_SIGNAL(obj)
-  #define _Py_ANNOTATE_HAPPENS_AFTER(obj)  _Py_ANNOTATE_CONDVAR_WAIT(obj)
+#define _Py_ANNOTATE_HAPPENS_BEFORE(obj) _Py_ANNOTATE_CONDVAR_SIGNAL(obj)
+#define _Py_ANNOTATE_HAPPENS_AFTER(obj)  _Py_ANNOTATE_CONDVAR_WAIT(obj)
 
   /* Report that the bytes in the range [pointer, pointer+size) are about
      to be published safely. The race checker will create a happens-before
@@ -131,7 +131,7 @@
      Note: this annotation may not work properly if the race detector uses
      sampling, i.e. does not observe all memory accesses.
      */
-  #define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(pointer, size) \
+#define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(pointer, size) \
     AnnotatePublishMemoryRange(__FILE__, __LINE__, pointer, size)
 
   /* Instruct the tool to create a happens-before arc between mu->Unlock() and
@@ -141,7 +141,7 @@
      This annotation makes sense only for hybrid race detectors. For pure
      happens-before detectors this is a no-op. For more details see
      http://code.google.com/p/data-race-test/wiki/PureHappensBeforeVsHybrid . */
-  #define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) \
+#define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) \
     AnnotateMutexIsUsedAsCondVar(__FILE__, __LINE__, mu)
 
   /* -------------------------------------------------------------
@@ -152,7 +152,7 @@
      This might be used when the memory has been retrieved from a free list and
      is about to be reused, or when a the locking discipline for a variable
      changes. */
-  #define _Py_ANNOTATE_NEW_MEMORY(address, size) \
+#define _Py_ANNOTATE_NEW_MEMORY(address, size) \
     AnnotateNewMemory(__FILE__, __LINE__, address, size)
 
   /* -------------------------------------------------------------
@@ -164,20 +164,20 @@
      be used only for FIFO queues.  For non-FIFO queues use
      _Py_ANNOTATE_HAPPENS_BEFORE (for put) and _Py_ANNOTATE_HAPPENS_AFTER (for
      get). */
-  #define _Py_ANNOTATE_PCQ_CREATE(pcq) \
+#define _Py_ANNOTATE_PCQ_CREATE(pcq) \
     AnnotatePCQCreate(__FILE__, __LINE__, pcq)
 
   /* Report that the queue at address "pcq" is about to be destroyed. */
-  #define _Py_ANNOTATE_PCQ_DESTROY(pcq) \
+#define _Py_ANNOTATE_PCQ_DESTROY(pcq) \
     AnnotatePCQDestroy(__FILE__, __LINE__, pcq)
 
   /* Report that we are about to put an element into a FIFO queue at address
      "pcq". */
-  #define _Py_ANNOTATE_PCQ_PUT(pcq) \
+#define _Py_ANNOTATE_PCQ_PUT(pcq) \
     AnnotatePCQPut(__FILE__, __LINE__, pcq)
 
   /* Report that we've just got an element from a FIFO queue at address "pcq". */
-  #define _Py_ANNOTATE_PCQ_GET(pcq) \
+#define _Py_ANNOTATE_PCQ_GET(pcq) \
     AnnotatePCQGet(__FILE__, __LINE__, pcq)
 
   /* -------------------------------------------------------------
@@ -189,13 +189,13 @@
      "sizeof(*(pointer))". "pointer" must be a non-void* pointer.  Insert at the
      point where "pointer" has been allocated, preferably close to the point
      where the race happens.  See also _Py_ANNOTATE_BENIGN_RACE_STATIC. */
-  #define _Py_ANNOTATE_BENIGN_RACE(pointer, description) \
+#define _Py_ANNOTATE_BENIGN_RACE(pointer, description) \
     AnnotateBenignRaceSized(__FILE__, __LINE__, pointer, \
                             sizeof(*(pointer)), description)
 
   /* Same as _Py_ANNOTATE_BENIGN_RACE(address, description), but applies to
      the memory range [address, address+size). */
-  #define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) \
+#define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) \
     AnnotateBenignRaceSized(__FILE__, __LINE__, address, size, description)
 
   /* Request the analysis tool to ignore all reads in the current thread
@@ -203,30 +203,30 @@
      Useful to ignore intentional racey reads, while still checking
      other reads and all writes.
      See also _Py_ANNOTATE_UNPROTECTED_READ. */
-  #define _Py_ANNOTATE_IGNORE_READS_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_READS_BEGIN() \
     AnnotateIgnoreReadsBegin(__FILE__, __LINE__)
 
   /* Stop ignoring reads. */
-  #define _Py_ANNOTATE_IGNORE_READS_END() \
+#define _Py_ANNOTATE_IGNORE_READS_END() \
     AnnotateIgnoreReadsEnd(__FILE__, __LINE__)
 
   /* Similar to _Py_ANNOTATE_IGNORE_READS_BEGIN, but ignore writes. */
-  #define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() \
     AnnotateIgnoreWritesBegin(__FILE__, __LINE__)
 
   /* Stop ignoring writes. */
-  #define _Py_ANNOTATE_IGNORE_WRITES_END() \
+#define _Py_ANNOTATE_IGNORE_WRITES_END() \
     AnnotateIgnoreWritesEnd(__FILE__, __LINE__)
 
   /* Start ignoring all memory accesses (reads and writes). */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() \
     do {\
       _Py_ANNOTATE_IGNORE_READS_BEGIN();\
       _Py_ANNOTATE_IGNORE_WRITES_BEGIN();\
     }while(0)\
 
   /* Stop ignoring all memory accesses. */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() \
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() \
     do {\
       _Py_ANNOTATE_IGNORE_WRITES_END();\
       _Py_ANNOTATE_IGNORE_READS_END();\
@@ -234,29 +234,29 @@
 
   /* Similar to _Py_ANNOTATE_IGNORE_READS_BEGIN, but ignore synchronization events:
      RWLOCK* and CONDVAR*. */
-  #define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() \
     AnnotateIgnoreSyncBegin(__FILE__, __LINE__)
 
   /* Stop ignoring sync events. */
-  #define _Py_ANNOTATE_IGNORE_SYNC_END() \
+#define _Py_ANNOTATE_IGNORE_SYNC_END() \
     AnnotateIgnoreSyncEnd(__FILE__, __LINE__)
 
 
   /* Enable (enable!=0) or disable (enable==0) race detection for all threads.
      This annotation could be useful if you want to skip expensive race analysis
      during some period of program execution, e.g. during initialization. */
-  #define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) \
+#define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) \
     AnnotateEnableRaceDetection(__FILE__, __LINE__, enable)
 
   /* -------------------------------------------------------------
      Annotations useful for debugging. */
 
   /* Request to trace every access to "address". */
-  #define _Py_ANNOTATE_TRACE_MEMORY(address) \
+#define _Py_ANNOTATE_TRACE_MEMORY(address) \
     AnnotateTraceMemory(__FILE__, __LINE__, address)
 
   /* Report the current thread name to a race detector. */
-  #define _Py_ANNOTATE_THREAD_NAME(name) \
+#define _Py_ANNOTATE_THREAD_NAME(name) \
     AnnotateThreadName(__FILE__, __LINE__, name)
 
   /* -------------------------------------------------------------
@@ -265,20 +265,20 @@
      The "lock" argument is a pointer to the lock object. */
 
   /* Report that a lock has been created at address "lock". */
-  #define _Py_ANNOTATE_RWLOCK_CREATE(lock) \
+#define _Py_ANNOTATE_RWLOCK_CREATE(lock) \
     AnnotateRWLockCreate(__FILE__, __LINE__, lock)
 
   /* Report that the lock at address "lock" is about to be destroyed. */
-  #define _Py_ANNOTATE_RWLOCK_DESTROY(lock) \
+#define _Py_ANNOTATE_RWLOCK_DESTROY(lock) \
     AnnotateRWLockDestroy(__FILE__, __LINE__, lock)
 
   /* Report that the lock at address "lock" has been acquired.
      is_w=1 for writer lock, is_w=0 for reader lock. */
-  #define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) \
+#define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) \
     AnnotateRWLockAcquired(__FILE__, __LINE__, lock, is_w)
 
   /* Report that the lock at address "lock" is about to be released. */
-  #define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) \
+#define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) \
     AnnotateRWLockReleased(__FILE__, __LINE__, lock, is_w)
 
   /* -------------------------------------------------------------
@@ -289,20 +289,20 @@
   /* Report that the "barrier" has been initialized with initial "count".
    If 'reinitialization_allowed' is true, initialization is allowed to happen
    multiple times w/o calling barrier_destroy() */
-  #define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) \
+#define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) \
     AnnotateBarrierInit(__FILE__, __LINE__, barrier, count, \
                         reinitialization_allowed)
 
   /* Report that we are about to enter barrier_wait("barrier"). */
-  #define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) \
+#define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) \
     AnnotateBarrierWaitBefore(__FILE__, __LINE__, barrier)
 
   /* Report that we just exited barrier_wait("barrier"). */
-  #define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) \
+#define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) \
     AnnotateBarrierWaitAfter(__FILE__, __LINE__, barrier)
 
   /* Report that the "barrier" has been destroyed. */
-  #define _Py_ANNOTATE_BARRIER_DESTROY(barrier) \
+#define _Py_ANNOTATE_BARRIER_DESTROY(barrier) \
     AnnotateBarrierDestroy(__FILE__, __LINE__, barrier)
 
   /* -------------------------------------------------------------
@@ -310,61 +310,61 @@
 
   /* Report that we expect a race on the variable at "address".
      Use only in unit tests for a race detector. */
-  #define _Py_ANNOTATE_EXPECT_RACE(address, description) \
+#define _Py_ANNOTATE_EXPECT_RACE(address, description) \
     AnnotateExpectRace(__FILE__, __LINE__, address, description)
 
   /* A no-op. Insert where you like to test the interceptors. */
-  #define _Py_ANNOTATE_NO_OP(arg) \
+#define _Py_ANNOTATE_NO_OP(arg) \
     AnnotateNoOp(__FILE__, __LINE__, arg)
 
   /* Force the race detector to flush its state. The actual effect depends on
    * the implementation of the detector. */
-  #define _Py_ANNOTATE_FLUSH_STATE() \
+#define _Py_ANNOTATE_FLUSH_STATE() \
     AnnotateFlushState(__FILE__, __LINE__)
 
 
 #else  /* DYNAMIC_ANNOTATIONS_ENABLED == 0 */
 
-  #define _Py_ANNOTATE_RWLOCK_CREATE(lock) /* empty */
-  #define _Py_ANNOTATE_RWLOCK_DESTROY(lock) /* empty */
-  #define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) /* empty */
-  #define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) /* empty */
-  #define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) /* */
-  #define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) /* empty */
-  #define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) /* empty */
-  #define _Py_ANNOTATE_BARRIER_DESTROY(barrier) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_WAIT(cv) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) /* empty */
-  #define _Py_ANNOTATE_HAPPENS_BEFORE(obj) /* empty */
-  #define _Py_ANNOTATE_HAPPENS_AFTER(obj) /* empty */
-  #define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(address, size) /* empty */
-  #define _Py_ANNOTATE_UNPUBLISH_MEMORY_RANGE(address, size)  /* empty */
-  #define _Py_ANNOTATE_SWAP_MEMORY_RANGE(address, size)  /* empty */
-  #define _Py_ANNOTATE_PCQ_CREATE(pcq) /* empty */
-  #define _Py_ANNOTATE_PCQ_DESTROY(pcq) /* empty */
-  #define _Py_ANNOTATE_PCQ_PUT(pcq) /* empty */
-  #define _Py_ANNOTATE_PCQ_GET(pcq) /* empty */
-  #define _Py_ANNOTATE_NEW_MEMORY(address, size) /* empty */
-  #define _Py_ANNOTATE_EXPECT_RACE(address, description) /* empty */
-  #define _Py_ANNOTATE_BENIGN_RACE(address, description) /* empty */
-  #define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) /* empty */
-  #define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) /* empty */
-  #define _Py_ANNOTATE_MUTEX_IS_USED_AS_CONDVAR(mu) /* empty */
-  #define _Py_ANNOTATE_TRACE_MEMORY(arg) /* empty */
-  #define _Py_ANNOTATE_THREAD_NAME(name) /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_END() /* empty */
-  #define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_WRITES_END() /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() /* empty */
-  #define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_SYNC_END() /* empty */
-  #define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) /* empty */
-  #define _Py_ANNOTATE_NO_OP(arg) /* empty */
-  #define _Py_ANNOTATE_FLUSH_STATE() /* empty */
+#define _Py_ANNOTATE_RWLOCK_CREATE(lock) /* empty */
+#define _Py_ANNOTATE_RWLOCK_DESTROY(lock) /* empty */
+#define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) /* empty */
+#define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) /* empty */
+#define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) /* */
+#define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) /* empty */
+#define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) /* empty */
+#define _Py_ANNOTATE_BARRIER_DESTROY(barrier) /* empty */
+#define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) /* empty */
+#define _Py_ANNOTATE_CONDVAR_WAIT(cv) /* empty */
+#define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) /* empty */
+#define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) /* empty */
+#define _Py_ANNOTATE_HAPPENS_BEFORE(obj) /* empty */
+#define _Py_ANNOTATE_HAPPENS_AFTER(obj) /* empty */
+#define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(address, size) /* empty */
+#define _Py_ANNOTATE_UNPUBLISH_MEMORY_RANGE(address, size)  /* empty */
+#define _Py_ANNOTATE_SWAP_MEMORY_RANGE(address, size)  /* empty */
+#define _Py_ANNOTATE_PCQ_CREATE(pcq) /* empty */
+#define _Py_ANNOTATE_PCQ_DESTROY(pcq) /* empty */
+#define _Py_ANNOTATE_PCQ_PUT(pcq) /* empty */
+#define _Py_ANNOTATE_PCQ_GET(pcq) /* empty */
+#define _Py_ANNOTATE_NEW_MEMORY(address, size) /* empty */
+#define _Py_ANNOTATE_EXPECT_RACE(address, description) /* empty */
+#define _Py_ANNOTATE_BENIGN_RACE(address, description) /* empty */
+#define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) /* empty */
+#define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) /* empty */
+#define _Py_ANNOTATE_MUTEX_IS_USED_AS_CONDVAR(mu) /* empty */
+#define _Py_ANNOTATE_TRACE_MEMORY(arg) /* empty */
+#define _Py_ANNOTATE_THREAD_NAME(name) /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_END() /* empty */
+#define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_WRITES_END() /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() /* empty */
+#define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_SYNC_END() /* empty */
+#define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) /* empty */
+#define _Py_ANNOTATE_NO_OP(arg) /* empty */
+#define _Py_ANNOTATE_FLUSH_STATE() /* empty */
 
 #endif  /* DYNAMIC_ANNOTATIONS_ENABLED */
 
@@ -477,7 +477,7 @@
     return res;
   }
   /* Apply _Py_ANNOTATE_BENIGN_RACE_SIZED to a static variable. */
-  #define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)        \
+#define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)        \
     namespace {                                                       \
       class static_var ## _annotator {                                \
        public:                                                        \
@@ -491,8 +491,8 @@
     }
 #else /* DYNAMIC_ANNOTATIONS_ENABLED == 0 */
 
-  #define _Py_ANNOTATE_UNPROTECTED_READ(x) (x)
-  #define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)  /* empty */
+#define _Py_ANNOTATE_UNPROTECTED_READ(x) (x)
+#define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)  /* empty */
 
 #endif /* DYNAMIC_ANNOTATIONS_ENABLED */
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 05:17:37 2011
From: python-checkins at python.org (jesus.cea)
Date: Sun, 27 Nov 2011 05:17:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_MERGE=3A_Closes_issue_=2313488=3A_Some_old_preprocessors_hav?=
 =?utf8?q?e_problem_with_=23define?=
Message-ID: <E1RUWBN-0006OW-RX@dinsdale.python.org>

http://hg.python.org/cpython/rev/3a44640682c3
changeset:   73765:3a44640682c3
parent:      73763:919259054621
parent:      73764:05979ae431fb
user:        Jesus Cea <jcea at jcea.es>
date:        Sun Nov 27 05:17:14 2011 +0100
summary:
  MERGE: Closes issue #13488: Some old preprocessors have problem with #define not in the first column

files:
  Include/dynamic_annotations.h |  160 +++++++++++-----------
  1 files changed, 80 insertions(+), 80 deletions(-)


diff --git a/Include/dynamic_annotations.h b/Include/dynamic_annotations.h
--- a/Include/dynamic_annotations.h
+++ b/Include/dynamic_annotations.h
@@ -103,26 +103,26 @@
 
   /* Report that wait on the condition variable at address "cv" has succeeded
      and the lock at address "lock" is held. */
-  #define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) \
+#define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) \
     AnnotateCondVarWait(__FILE__, __LINE__, cv, lock)
 
   /* Report that wait on the condition variable at "cv" has succeeded.  Variant
      w/o lock. */
-  #define _Py_ANNOTATE_CONDVAR_WAIT(cv) \
+#define _Py_ANNOTATE_CONDVAR_WAIT(cv) \
     AnnotateCondVarWait(__FILE__, __LINE__, cv, NULL)
 
   /* Report that we are about to signal on the condition variable at address
      "cv". */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) \
+#define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) \
     AnnotateCondVarSignal(__FILE__, __LINE__, cv)
 
   /* Report that we are about to signal_all on the condition variable at "cv". */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) \
+#define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) \
     AnnotateCondVarSignalAll(__FILE__, __LINE__, cv)
 
   /* Annotations for user-defined synchronization mechanisms. */
-  #define _Py_ANNOTATE_HAPPENS_BEFORE(obj) _Py_ANNOTATE_CONDVAR_SIGNAL(obj)
-  #define _Py_ANNOTATE_HAPPENS_AFTER(obj)  _Py_ANNOTATE_CONDVAR_WAIT(obj)
+#define _Py_ANNOTATE_HAPPENS_BEFORE(obj) _Py_ANNOTATE_CONDVAR_SIGNAL(obj)
+#define _Py_ANNOTATE_HAPPENS_AFTER(obj)  _Py_ANNOTATE_CONDVAR_WAIT(obj)
 
   /* Report that the bytes in the range [pointer, pointer+size) are about
      to be published safely. The race checker will create a happens-before
@@ -131,7 +131,7 @@
      Note: this annotation may not work properly if the race detector uses
      sampling, i.e. does not observe all memory accesses.
      */
-  #define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(pointer, size) \
+#define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(pointer, size) \
     AnnotatePublishMemoryRange(__FILE__, __LINE__, pointer, size)
 
   /* Instruct the tool to create a happens-before arc between mu->Unlock() and
@@ -141,7 +141,7 @@
      This annotation makes sense only for hybrid race detectors. For pure
      happens-before detectors this is a no-op. For more details see
      http://code.google.com/p/data-race-test/wiki/PureHappensBeforeVsHybrid . */
-  #define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) \
+#define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) \
     AnnotateMutexIsUsedAsCondVar(__FILE__, __LINE__, mu)
 
   /* -------------------------------------------------------------
@@ -152,7 +152,7 @@
      This might be used when the memory has been retrieved from a free list and
      is about to be reused, or when a the locking discipline for a variable
      changes. */
-  #define _Py_ANNOTATE_NEW_MEMORY(address, size) \
+#define _Py_ANNOTATE_NEW_MEMORY(address, size) \
     AnnotateNewMemory(__FILE__, __LINE__, address, size)
 
   /* -------------------------------------------------------------
@@ -164,20 +164,20 @@
      be used only for FIFO queues.  For non-FIFO queues use
      _Py_ANNOTATE_HAPPENS_BEFORE (for put) and _Py_ANNOTATE_HAPPENS_AFTER (for
      get). */
-  #define _Py_ANNOTATE_PCQ_CREATE(pcq) \
+#define _Py_ANNOTATE_PCQ_CREATE(pcq) \
     AnnotatePCQCreate(__FILE__, __LINE__, pcq)
 
   /* Report that the queue at address "pcq" is about to be destroyed. */
-  #define _Py_ANNOTATE_PCQ_DESTROY(pcq) \
+#define _Py_ANNOTATE_PCQ_DESTROY(pcq) \
     AnnotatePCQDestroy(__FILE__, __LINE__, pcq)
 
   /* Report that we are about to put an element into a FIFO queue at address
      "pcq". */
-  #define _Py_ANNOTATE_PCQ_PUT(pcq) \
+#define _Py_ANNOTATE_PCQ_PUT(pcq) \
     AnnotatePCQPut(__FILE__, __LINE__, pcq)
 
   /* Report that we've just got an element from a FIFO queue at address "pcq". */
-  #define _Py_ANNOTATE_PCQ_GET(pcq) \
+#define _Py_ANNOTATE_PCQ_GET(pcq) \
     AnnotatePCQGet(__FILE__, __LINE__, pcq)
 
   /* -------------------------------------------------------------
@@ -189,13 +189,13 @@
      "sizeof(*(pointer))". "pointer" must be a non-void* pointer.  Insert at the
      point where "pointer" has been allocated, preferably close to the point
      where the race happens.  See also _Py_ANNOTATE_BENIGN_RACE_STATIC. */
-  #define _Py_ANNOTATE_BENIGN_RACE(pointer, description) \
+#define _Py_ANNOTATE_BENIGN_RACE(pointer, description) \
     AnnotateBenignRaceSized(__FILE__, __LINE__, pointer, \
                             sizeof(*(pointer)), description)
 
   /* Same as _Py_ANNOTATE_BENIGN_RACE(address, description), but applies to
      the memory range [address, address+size). */
-  #define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) \
+#define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) \
     AnnotateBenignRaceSized(__FILE__, __LINE__, address, size, description)
 
   /* Request the analysis tool to ignore all reads in the current thread
@@ -203,30 +203,30 @@
      Useful to ignore intentional racey reads, while still checking
      other reads and all writes.
      See also _Py_ANNOTATE_UNPROTECTED_READ. */
-  #define _Py_ANNOTATE_IGNORE_READS_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_READS_BEGIN() \
     AnnotateIgnoreReadsBegin(__FILE__, __LINE__)
 
   /* Stop ignoring reads. */
-  #define _Py_ANNOTATE_IGNORE_READS_END() \
+#define _Py_ANNOTATE_IGNORE_READS_END() \
     AnnotateIgnoreReadsEnd(__FILE__, __LINE__)
 
   /* Similar to _Py_ANNOTATE_IGNORE_READS_BEGIN, but ignore writes. */
-  #define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() \
     AnnotateIgnoreWritesBegin(__FILE__, __LINE__)
 
   /* Stop ignoring writes. */
-  #define _Py_ANNOTATE_IGNORE_WRITES_END() \
+#define _Py_ANNOTATE_IGNORE_WRITES_END() \
     AnnotateIgnoreWritesEnd(__FILE__, __LINE__)
 
   /* Start ignoring all memory accesses (reads and writes). */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() \
     do {\
       _Py_ANNOTATE_IGNORE_READS_BEGIN();\
       _Py_ANNOTATE_IGNORE_WRITES_BEGIN();\
     }while(0)\
 
   /* Stop ignoring all memory accesses. */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() \
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() \
     do {\
       _Py_ANNOTATE_IGNORE_WRITES_END();\
       _Py_ANNOTATE_IGNORE_READS_END();\
@@ -234,29 +234,29 @@
 
   /* Similar to _Py_ANNOTATE_IGNORE_READS_BEGIN, but ignore synchronization events:
      RWLOCK* and CONDVAR*. */
-  #define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() \
+#define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() \
     AnnotateIgnoreSyncBegin(__FILE__, __LINE__)
 
   /* Stop ignoring sync events. */
-  #define _Py_ANNOTATE_IGNORE_SYNC_END() \
+#define _Py_ANNOTATE_IGNORE_SYNC_END() \
     AnnotateIgnoreSyncEnd(__FILE__, __LINE__)
 
 
   /* Enable (enable!=0) or disable (enable==0) race detection for all threads.
      This annotation could be useful if you want to skip expensive race analysis
      during some period of program execution, e.g. during initialization. */
-  #define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) \
+#define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) \
     AnnotateEnableRaceDetection(__FILE__, __LINE__, enable)
 
   /* -------------------------------------------------------------
      Annotations useful for debugging. */
 
   /* Request to trace every access to "address". */
-  #define _Py_ANNOTATE_TRACE_MEMORY(address) \
+#define _Py_ANNOTATE_TRACE_MEMORY(address) \
     AnnotateTraceMemory(__FILE__, __LINE__, address)
 
   /* Report the current thread name to a race detector. */
-  #define _Py_ANNOTATE_THREAD_NAME(name) \
+#define _Py_ANNOTATE_THREAD_NAME(name) \
     AnnotateThreadName(__FILE__, __LINE__, name)
 
   /* -------------------------------------------------------------
@@ -265,20 +265,20 @@
      The "lock" argument is a pointer to the lock object. */
 
   /* Report that a lock has been created at address "lock". */
-  #define _Py_ANNOTATE_RWLOCK_CREATE(lock) \
+#define _Py_ANNOTATE_RWLOCK_CREATE(lock) \
     AnnotateRWLockCreate(__FILE__, __LINE__, lock)
 
   /* Report that the lock at address "lock" is about to be destroyed. */
-  #define _Py_ANNOTATE_RWLOCK_DESTROY(lock) \
+#define _Py_ANNOTATE_RWLOCK_DESTROY(lock) \
     AnnotateRWLockDestroy(__FILE__, __LINE__, lock)
 
   /* Report that the lock at address "lock" has been acquired.
      is_w=1 for writer lock, is_w=0 for reader lock. */
-  #define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) \
+#define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) \
     AnnotateRWLockAcquired(__FILE__, __LINE__, lock, is_w)
 
   /* Report that the lock at address "lock" is about to be released. */
-  #define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) \
+#define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) \
     AnnotateRWLockReleased(__FILE__, __LINE__, lock, is_w)
 
   /* -------------------------------------------------------------
@@ -289,20 +289,20 @@
   /* Report that the "barrier" has been initialized with initial "count".
    If 'reinitialization_allowed' is true, initialization is allowed to happen
    multiple times w/o calling barrier_destroy() */
-  #define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) \
+#define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) \
     AnnotateBarrierInit(__FILE__, __LINE__, barrier, count, \
                         reinitialization_allowed)
 
   /* Report that we are about to enter barrier_wait("barrier"). */
-  #define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) \
+#define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) \
     AnnotateBarrierWaitBefore(__FILE__, __LINE__, barrier)
 
   /* Report that we just exited barrier_wait("barrier"). */
-  #define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) \
+#define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) \
     AnnotateBarrierWaitAfter(__FILE__, __LINE__, barrier)
 
   /* Report that the "barrier" has been destroyed. */
-  #define _Py_ANNOTATE_BARRIER_DESTROY(barrier) \
+#define _Py_ANNOTATE_BARRIER_DESTROY(barrier) \
     AnnotateBarrierDestroy(__FILE__, __LINE__, barrier)
 
   /* -------------------------------------------------------------
@@ -310,61 +310,61 @@
 
   /* Report that we expect a race on the variable at "address".
      Use only in unit tests for a race detector. */
-  #define _Py_ANNOTATE_EXPECT_RACE(address, description) \
+#define _Py_ANNOTATE_EXPECT_RACE(address, description) \
     AnnotateExpectRace(__FILE__, __LINE__, address, description)
 
   /* A no-op. Insert where you like to test the interceptors. */
-  #define _Py_ANNOTATE_NO_OP(arg) \
+#define _Py_ANNOTATE_NO_OP(arg) \
     AnnotateNoOp(__FILE__, __LINE__, arg)
 
   /* Force the race detector to flush its state. The actual effect depends on
    * the implementation of the detector. */
-  #define _Py_ANNOTATE_FLUSH_STATE() \
+#define _Py_ANNOTATE_FLUSH_STATE() \
     AnnotateFlushState(__FILE__, __LINE__)
 
 
 #else  /* DYNAMIC_ANNOTATIONS_ENABLED == 0 */
 
-  #define _Py_ANNOTATE_RWLOCK_CREATE(lock) /* empty */
-  #define _Py_ANNOTATE_RWLOCK_DESTROY(lock) /* empty */
-  #define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) /* empty */
-  #define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) /* empty */
-  #define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) /* */
-  #define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) /* empty */
-  #define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) /* empty */
-  #define _Py_ANNOTATE_BARRIER_DESTROY(barrier) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_WAIT(cv) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) /* empty */
-  #define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) /* empty */
-  #define _Py_ANNOTATE_HAPPENS_BEFORE(obj) /* empty */
-  #define _Py_ANNOTATE_HAPPENS_AFTER(obj) /* empty */
-  #define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(address, size) /* empty */
-  #define _Py_ANNOTATE_UNPUBLISH_MEMORY_RANGE(address, size)  /* empty */
-  #define _Py_ANNOTATE_SWAP_MEMORY_RANGE(address, size)  /* empty */
-  #define _Py_ANNOTATE_PCQ_CREATE(pcq) /* empty */
-  #define _Py_ANNOTATE_PCQ_DESTROY(pcq) /* empty */
-  #define _Py_ANNOTATE_PCQ_PUT(pcq) /* empty */
-  #define _Py_ANNOTATE_PCQ_GET(pcq) /* empty */
-  #define _Py_ANNOTATE_NEW_MEMORY(address, size) /* empty */
-  #define _Py_ANNOTATE_EXPECT_RACE(address, description) /* empty */
-  #define _Py_ANNOTATE_BENIGN_RACE(address, description) /* empty */
-  #define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) /* empty */
-  #define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) /* empty */
-  #define _Py_ANNOTATE_MUTEX_IS_USED_AS_CONDVAR(mu) /* empty */
-  #define _Py_ANNOTATE_TRACE_MEMORY(arg) /* empty */
-  #define _Py_ANNOTATE_THREAD_NAME(name) /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_END() /* empty */
-  #define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_WRITES_END() /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() /* empty */
-  #define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() /* empty */
-  #define _Py_ANNOTATE_IGNORE_SYNC_END() /* empty */
-  #define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) /* empty */
-  #define _Py_ANNOTATE_NO_OP(arg) /* empty */
-  #define _Py_ANNOTATE_FLUSH_STATE() /* empty */
+#define _Py_ANNOTATE_RWLOCK_CREATE(lock) /* empty */
+#define _Py_ANNOTATE_RWLOCK_DESTROY(lock) /* empty */
+#define _Py_ANNOTATE_RWLOCK_ACQUIRED(lock, is_w) /* empty */
+#define _Py_ANNOTATE_RWLOCK_RELEASED(lock, is_w) /* empty */
+#define _Py_ANNOTATE_BARRIER_INIT(barrier, count, reinitialization_allowed) /* */
+#define _Py_ANNOTATE_BARRIER_WAIT_BEFORE(barrier) /* empty */
+#define _Py_ANNOTATE_BARRIER_WAIT_AFTER(barrier) /* empty */
+#define _Py_ANNOTATE_BARRIER_DESTROY(barrier) /* empty */
+#define _Py_ANNOTATE_CONDVAR_LOCK_WAIT(cv, lock) /* empty */
+#define _Py_ANNOTATE_CONDVAR_WAIT(cv) /* empty */
+#define _Py_ANNOTATE_CONDVAR_SIGNAL(cv) /* empty */
+#define _Py_ANNOTATE_CONDVAR_SIGNAL_ALL(cv) /* empty */
+#define _Py_ANNOTATE_HAPPENS_BEFORE(obj) /* empty */
+#define _Py_ANNOTATE_HAPPENS_AFTER(obj) /* empty */
+#define _Py_ANNOTATE_PUBLISH_MEMORY_RANGE(address, size) /* empty */
+#define _Py_ANNOTATE_UNPUBLISH_MEMORY_RANGE(address, size)  /* empty */
+#define _Py_ANNOTATE_SWAP_MEMORY_RANGE(address, size)  /* empty */
+#define _Py_ANNOTATE_PCQ_CREATE(pcq) /* empty */
+#define _Py_ANNOTATE_PCQ_DESTROY(pcq) /* empty */
+#define _Py_ANNOTATE_PCQ_PUT(pcq) /* empty */
+#define _Py_ANNOTATE_PCQ_GET(pcq) /* empty */
+#define _Py_ANNOTATE_NEW_MEMORY(address, size) /* empty */
+#define _Py_ANNOTATE_EXPECT_RACE(address, description) /* empty */
+#define _Py_ANNOTATE_BENIGN_RACE(address, description) /* empty */
+#define _Py_ANNOTATE_BENIGN_RACE_SIZED(address, size, description) /* empty */
+#define _Py_ANNOTATE_PURE_HAPPENS_BEFORE_MUTEX(mu) /* empty */
+#define _Py_ANNOTATE_MUTEX_IS_USED_AS_CONDVAR(mu) /* empty */
+#define _Py_ANNOTATE_TRACE_MEMORY(arg) /* empty */
+#define _Py_ANNOTATE_THREAD_NAME(name) /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_END() /* empty */
+#define _Py_ANNOTATE_IGNORE_WRITES_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_WRITES_END() /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_READS_AND_WRITES_END() /* empty */
+#define _Py_ANNOTATE_IGNORE_SYNC_BEGIN() /* empty */
+#define _Py_ANNOTATE_IGNORE_SYNC_END() /* empty */
+#define _Py_ANNOTATE_ENABLE_RACE_DETECTION(enable) /* empty */
+#define _Py_ANNOTATE_NO_OP(arg) /* empty */
+#define _Py_ANNOTATE_FLUSH_STATE() /* empty */
 
 #endif  /* DYNAMIC_ANNOTATIONS_ENABLED */
 
@@ -477,7 +477,7 @@
     return res;
   }
   /* Apply _Py_ANNOTATE_BENIGN_RACE_SIZED to a static variable. */
-  #define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)        \
+#define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)        \
     namespace {                                                       \
       class static_var ## _annotator {                                \
        public:                                                        \
@@ -491,8 +491,8 @@
     }
 #else /* DYNAMIC_ANNOTATIONS_ENABLED == 0 */
 
-  #define _Py_ANNOTATE_UNPROTECTED_READ(x) (x)
-  #define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)  /* empty */
+#define _Py_ANNOTATE_UNPROTECTED_READ(x) (x)
+#define _Py_ANNOTATE_BENIGN_RACE_STATIC(static_var, description)  /* empty */
 
 #endif /* DYNAMIC_ANNOTATIONS_ENABLED */
 

-- 
Repository URL: http://hg.python.org/cpython

From solipsis at pitrou.net  Sun Nov 27 05:32:33 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 27 Nov 2011 05:32:33 +0100
Subject: [Python-checkins] Daily reference leaks (919259054621): sum=1
Message-ID: <E1RUWPp-0006gG-Gz@ap.vmr.nerim.net>

results for 919259054621 on branch "default"
--------------------------------------------

test_xmlrpc leaked [1, 0, 0] references, sum=1


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog0Pl11L', '-x']

From python-checkins at python.org  Sun Nov 27 12:51:34 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sun, 27 Nov 2011 12:51:34 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDE1?=
 =?utf8?q?=3A_Test_in_configure_if_unsetenv=28=29_has_a_return_value_or_no?=
 =?utf8?q?t=2E?=
Message-ID: <E1RUdGg-0004FX-9U@dinsdale.python.org>

http://hg.python.org/cpython/rev/26275a1c229c
changeset:   73766:26275a1c229c
branch:      3.2
parent:      73764:05979ae431fb
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sun Nov 27 12:41:06 2011 +0100
summary:
  Issue #13415: Test in configure if unsetenv() has a return value or not.

files:
  Modules/posixmodule.c |    6 +
  configure             |  625 +++++++++++++++--------------
  configure.in          |    9 +
  pyconfig.h.in         |    3 +
  4 files changed, 333 insertions(+), 310 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -6185,18 +6185,24 @@
 {
     PyObject *os1;
     char *s1;
+#ifndef HAVE_BROKEN_UNSETENV
     int err;
+#endif
 
     if (!PyArg_ParseTuple(args, "O&:unsetenv",
                           PyUnicode_FSConverter, &os1))
         return NULL;
     s1 = PyBytes_AsString(os1);
 
+#ifdef HAVE_BROKEN_UNSETENV
+    unsetenv(s1);
+#else
     err = unsetenv(s1);
     if (err) {
         Py_DECREF(os1);
         return posix_error();
     }
+#endif
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -1,7 +1,7 @@
 #! /bin/sh
 # From configure.in Revision.
 # Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.68 for python 3.2.
+# Generated by GNU Autoconf 2.67 for python 3.2.
 #
 # Report bugs to <http://bugs.python.org/>.
 #
@@ -92,7 +92,6 @@
 IFS=" ""	$as_nl"
 
 # Find who we are.  Look in the path if we contain no directory separator.
-as_myself=
 case $0 in #((
   *[\\/]* ) as_myself=$0 ;;
   *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -218,18 +217,11 @@
   # We cannot yet assume a decent shell, so we have to provide a
 	# neutralization value for shells without unset; and this also
 	# works around shells that cannot unset nonexistent variables.
-	# Preserve -v and -x to the replacement shell.
 	BASH_ENV=/dev/null
 	ENV=/dev/null
 	(unset BASH_ENV) >/dev/null 2>&1 && unset BASH_ENV ENV
 	export CONFIG_SHELL
-	case $- in # ((((
-	  *v*x* | *x*v* ) as_opts=-vx ;;
-	  *v* ) as_opts=-v ;;
-	  *x* ) as_opts=-x ;;
-	  * ) as_opts= ;;
-	esac
-	exec "$CONFIG_SHELL" $as_opts "$as_myself" ${1+"$@"}
+	exec "$CONFIG_SHELL" "$as_myself" ${1+"$@"}
 fi
 
     if test x$as_have_required = xno; then :
@@ -1186,7 +1178,7 @@
     $as_echo "$as_me: WARNING: you should use --build, --host, --target" >&2
     expr "x$ac_option" : ".*[^-._$as_cr_alnum]" >/dev/null &&
       $as_echo "$as_me: WARNING: invalid host type: $ac_option" >&2
-    : "${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}"
+    : ${build_alias=$ac_option} ${host_alias=$ac_option} ${target_alias=$ac_option}
     ;;
 
   esac
@@ -1524,7 +1516,7 @@
 if $ac_init_version; then
   cat <<\_ACEOF
 python configure 3.2
-generated by GNU Autoconf 2.68
+generated by GNU Autoconf 2.67
 
 Copyright (C) 2010 Free Software Foundation, Inc.
 This configure script is free software; the Free Software Foundation
@@ -1570,7 +1562,7 @@
 
 	ac_retval=1
 fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_compile
@@ -1616,7 +1608,7 @@
   # interfere with the next link command; also delete a directory that is
   # left behind by Apple's compiler.  We do this before executing the actions.
   rm -rf conftest.dSYM conftest_ipa8_conftest.oo
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_link
@@ -1653,7 +1645,7 @@
 
     ac_retval=1
 fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_cpp
@@ -1666,10 +1658,10 @@
 ac_fn_c_check_header_mongrel ()
 {
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
-  if eval \${$3+:} false; then :
+  if eval "test \"\${$3+set}\"" = set; then :
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 fi
 eval ac_res=\$$3
@@ -1736,7 +1728,7 @@
 esac
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=\$ac_header_compiler"
@@ -1745,7 +1737,7 @@
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
 fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_header_mongrel
 
@@ -1786,7 +1778,7 @@
        ac_retval=$ac_status
 fi
   rm -rf conftest.dSYM conftest_ipa8_conftest.oo
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_try_run
@@ -1800,7 +1792,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -1818,7 +1810,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_header_compile
 
@@ -1831,7 +1823,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1872,7 +1864,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_type
 
@@ -1885,7 +1877,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for uint$2_t" >&5
 $as_echo_n "checking for uint$2_t... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1925,7 +1917,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_find_uintX_t
 
@@ -1938,7 +1930,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for int$2_t" >&5
 $as_echo_n "checking for int$2_t... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   eval "$3=no"
@@ -1999,7 +1991,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_find_intX_t
 
@@ -2176,7 +2168,7 @@
 rm -f conftest.val
 
   fi
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
   as_fn_set_status $ac_retval
 
 } # ac_fn_c_compute_int
@@ -2189,7 +2181,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2" >&5
 $as_echo_n "checking for $2... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2244,7 +2236,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_func
 
@@ -2257,7 +2249,7 @@
   as_lineno=${as_lineno-"$1"} as_lineno_stack=as_lineno_stack=$as_lineno_stack
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $2.$3" >&5
 $as_echo_n "checking for $2.$3... " >&6; }
-if eval \${$4+:} false; then :
+if eval "test \"\${$4+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2301,7 +2293,7 @@
 eval ac_res=\$$4
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_member
 
@@ -2316,7 +2308,7 @@
   as_decl_use=`echo $2|sed -e 's/(/((/' -e 's/)/) 0&/' -e 's/,/) 0& (/g'`
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $as_decl_name is declared" >&5
 $as_echo_n "checking whether $as_decl_name is declared... " >&6; }
-if eval \${$3+:} false; then :
+if eval "test \"\${$3+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -2347,7 +2339,7 @@
 eval ac_res=\$$3
 	       { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_res" >&5
 $as_echo "$ac_res" >&6; }
-  eval $as_lineno_stack; ${as_lineno_stack:+:} unset as_lineno
+  eval $as_lineno_stack; test "x$as_lineno_stack" = x && { as_lineno=; unset as_lineno;}
 
 } # ac_fn_c_check_decl
 cat >config.log <<_ACEOF
@@ -2355,7 +2347,7 @@
 running configure, to aid debugging if configure makes a mistake.
 
 It was created by python $as_me 3.2, which was
-generated by GNU Autoconf 2.68.  Invocation command line was
+generated by GNU Autoconf 2.67.  Invocation command line was
 
   $ $0 $@
 
@@ -2613,7 +2605,7 @@
       || { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "failed to load site script $ac_site_file
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
   fi
 done
 
@@ -3203,7 +3195,7 @@
 set dummy ${ac_tool_prefix}gcc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3243,7 +3235,7 @@
 set dummy gcc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_ac_ct_CC+:} false; then :
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_CC"; then
@@ -3296,7 +3288,7 @@
 set dummy ${ac_tool_prefix}cc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3336,7 +3328,7 @@
 set dummy cc; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3395,7 +3387,7 @@
 set dummy $ac_tool_prefix$ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CC+:} false; then :
+if test "${ac_cv_prog_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CC"; then
@@ -3439,7 +3431,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_ac_ct_CC+:} false; then :
+if test "${ac_cv_prog_ac_ct_CC+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_CC"; then
@@ -3494,7 +3486,7 @@
 test -z "$CC" && { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "no acceptable C compiler found in \$PATH
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 
 # Provide some information about the compiler.
 $as_echo "$as_me:${as_lineno-$LINENO}: checking for C compiler version" >&5
@@ -3609,7 +3601,7 @@
 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "C compiler cannot create executables
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
 $as_echo "yes" >&6; }
@@ -3652,7 +3644,7 @@
   { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot compute suffix of executables: cannot compile and link
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 fi
 rm -f conftest conftest$ac_cv_exeext
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_exeext" >&5
@@ -3711,7 +3703,7 @@
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot run C compiled programs.
 If you meant to cross compile, use \`--host'.
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
     fi
   fi
 fi
@@ -3722,7 +3714,7 @@
 ac_clean_files=$ac_clean_files_save
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for suffix of object files" >&5
 $as_echo_n "checking for suffix of object files... " >&6; }
-if ${ac_cv_objext+:} false; then :
+if test "${ac_cv_objext+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -3763,7 +3755,7 @@
 { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "cannot compute suffix of object files: cannot compile
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 fi
 rm -f conftest.$ac_cv_objext conftest.$ac_ext
 fi
@@ -3773,7 +3765,7 @@
 ac_objext=$OBJEXT
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether we are using the GNU C compiler" >&5
 $as_echo_n "checking whether we are using the GNU C compiler... " >&6; }
-if ${ac_cv_c_compiler_gnu+:} false; then :
+if test "${ac_cv_c_compiler_gnu+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -3810,7 +3802,7 @@
 ac_save_CFLAGS=$CFLAGS
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -g" >&5
 $as_echo_n "checking whether $CC accepts -g... " >&6; }
-if ${ac_cv_prog_cc_g+:} false; then :
+if test "${ac_cv_prog_cc_g+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_c_werror_flag=$ac_c_werror_flag
@@ -3888,7 +3880,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $CC option to accept ISO C89" >&5
 $as_echo_n "checking for $CC option to accept ISO C89... " >&6; }
-if ${ac_cv_prog_cc_c89+:} false; then :
+if test "${ac_cv_prog_cc_c89+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_prog_cc_c89=no
@@ -4023,7 +4015,7 @@
 set dummy g++; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_CXX+:} false; then :
+if test "${ac_cv_path_CXX+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $CXX in
@@ -4064,7 +4056,7 @@
 set dummy c++; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_CXX+:} false; then :
+if test "${ac_cv_path_CXX+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $CXX in
@@ -4115,7 +4107,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_CXX+:} false; then :
+if test "${ac_cv_prog_CXX+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$CXX"; then
@@ -4216,7 +4208,7 @@
   CPP=
 fi
 if test -z "$CPP"; then
-  if ${ac_cv_prog_CPP+:} false; then :
+  if test "${ac_cv_prog_CPP+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
       # Double quotes because CPP needs to be expanded
@@ -4332,7 +4324,7 @@
   { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error $? "C preprocessor \"$CPP\" fails sanity check
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
 fi
 
 ac_ext=c
@@ -4344,7 +4336,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for grep that handles long lines and -e" >&5
 $as_echo_n "checking for grep that handles long lines and -e... " >&6; }
-if ${ac_cv_path_GREP+:} false; then :
+if test "${ac_cv_path_GREP+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -z "$GREP"; then
@@ -4407,7 +4399,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for egrep" >&5
 $as_echo_n "checking for egrep... " >&6; }
-if ${ac_cv_path_EGREP+:} false; then :
+if test "${ac_cv_path_EGREP+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if echo a | $GREP -E '(a|b)' >/dev/null 2>&1
@@ -4474,7 +4466,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }
-if ${ac_cv_header_stdc+:} false; then :
+if test "${ac_cv_header_stdc+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -4603,7 +4595,7 @@
 
 
   ac_fn_c_check_header_mongrel "$LINENO" "minix/config.h" "ac_cv_header_minix_config_h" "$ac_includes_default"
-if test "x$ac_cv_header_minix_config_h" = xyes; then :
+if test "x$ac_cv_header_minix_config_h" = x""yes; then :
   MINIX=yes
 else
   MINIX=
@@ -4625,7 +4617,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether it is safe to define __EXTENSIONS__" >&5
 $as_echo_n "checking whether it is safe to define __EXTENSIONS__... " >&6; }
-if ${ac_cv_safe_to_define___extensions__+:} false; then :
+if test "${ac_cv_safe_to_define___extensions__+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -4818,7 +4810,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inline" >&5
 $as_echo_n "checking for inline... " >&6; }
-if ${ac_cv_c_inline+:} false; then :
+if test "${ac_cv_c_inline+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_c_inline=no
@@ -5019,7 +5011,7 @@
 set dummy ${ac_tool_prefix}ranlib; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_RANLIB+:} false; then :
+if test "${ac_cv_prog_RANLIB+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$RANLIB"; then
@@ -5059,7 +5051,7 @@
 set dummy ranlib; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_ac_ct_RANLIB+:} false; then :
+if test "${ac_cv_prog_ac_ct_RANLIB+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$ac_ct_RANLIB"; then
@@ -5113,7 +5105,7 @@
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_AR+:} false; then :
+if test "${ac_cv_prog_AR+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$AR"; then
@@ -5163,7 +5155,7 @@
 set dummy svnversion; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_SVNVERSION+:} false; then :
+if test "${ac_cv_prog_SVNVERSION+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$SVNVERSION"; then
@@ -5214,7 +5206,7 @@
 set dummy hg; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_HAS_HG+:} false; then :
+if test "${ac_cv_prog_HAS_HG+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$HAS_HG"; then
@@ -5268,7 +5260,7 @@
 set dummy python; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_HAS_PYTHON+:} false; then :
+if test "${ac_cv_prog_HAS_PYTHON+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$HAS_PYTHON"; then
@@ -5362,7 +5354,7 @@
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for a BSD-compatible install" >&5
 $as_echo_n "checking for a BSD-compatible install... " >&6; }
 if test -z "$INSTALL"; then
-if ${ac_cv_path_install+:} false; then :
+if test "${ac_cv_path_install+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -5554,7 +5546,7 @@
      ac_save_cc="$CC"
      CC="$CC -fno-strict-aliasing"
      save_CFLAGS="$CFLAGS"
-     if ${ac_cv_no_strict_aliasing+:} false; then :
+     if test "${ac_cv_no_strict_aliasing+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -5810,7 +5802,7 @@
 # options before we can check whether -Kpthread improves anything.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether pthreads are available without options" >&5
 $as_echo_n "checking whether pthreads are available without options... " >&6; }
-if ${ac_cv_pthread_is_default+:} false; then :
+if test "${ac_cv_pthread_is_default+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -5863,7 +5855,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -Kpthread" >&5
 $as_echo_n "checking whether $CC accepts -Kpthread... " >&6; }
-if ${ac_cv_kpthread+:} false; then :
+if test "${ac_cv_kpthread+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5912,7 +5904,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -Kthread" >&5
 $as_echo_n "checking whether $CC accepts -Kthread... " >&6; }
-if ${ac_cv_kthread+:} false; then :
+if test "${ac_cv_kthread+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -5961,7 +5953,7 @@
 # function available.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC accepts -pthread" >&5
 $as_echo_n "checking whether $CC accepts -pthread... " >&6; }
-if ${ac_cv_thread+:} false; then :
+if test "${ac_cv_thread+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_save_cc="$CC"
@@ -6046,7 +6038,7 @@
 # checks for header files
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }
-if ${ac_cv_header_stdc+:} false; then :
+if test "${ac_cv_header_stdc+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6185,7 +6177,7 @@
   as_ac_Header=`$as_echo "ac_cv_header_dirent_$ac_hdr" | $as_tr_sh`
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_hdr that defines DIR" >&5
 $as_echo_n "checking for $ac_hdr that defines DIR... " >&6; }
-if eval \${$as_ac_Header+:} false; then :
+if eval "test \"\${$as_ac_Header+set}\"" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6225,7 +6217,7 @@
 if test $ac_header_dirent = dirent.h; then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing opendir" >&5
 $as_echo_n "checking for library containing opendir... " >&6; }
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -6259,11 +6251,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if ${ac_cv_search_opendir+:} false; then :
+  if test "${ac_cv_search_opendir+set}" = set; then :
   break
 fi
 done
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
 
 else
   ac_cv_search_opendir=no
@@ -6282,7 +6274,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing opendir" >&5
 $as_echo_n "checking for library containing opendir... " >&6; }
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -6316,11 +6308,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if ${ac_cv_search_opendir+:} false; then :
+  if test "${ac_cv_search_opendir+set}" = set; then :
   break
 fi
 done
-if ${ac_cv_search_opendir+:} false; then :
+if test "${ac_cv_search_opendir+set}" = set; then :
 
 else
   ac_cv_search_opendir=no
@@ -6340,7 +6332,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether sys/types.h defines makedev" >&5
 $as_echo_n "checking whether sys/types.h defines makedev... " >&6; }
-if ${ac_cv_header_sys_types_h_makedev+:} false; then :
+if test "${ac_cv_header_sys_types_h_makedev+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6368,7 +6360,7 @@
 
 if test $ac_cv_header_sys_types_h_makedev = no; then
 ac_fn_c_check_header_mongrel "$LINENO" "sys/mkdev.h" "ac_cv_header_sys_mkdev_h" "$ac_includes_default"
-if test "x$ac_cv_header_sys_mkdev_h" = xyes; then :
+if test "x$ac_cv_header_sys_mkdev_h" = x""yes; then :
 
 $as_echo "#define MAJOR_IN_MKDEV 1" >>confdefs.h
 
@@ -6378,7 +6370,7 @@
 
   if test $ac_cv_header_sys_mkdev_h = no; then
     ac_fn_c_check_header_mongrel "$LINENO" "sys/sysmacros.h" "ac_cv_header_sys_sysmacros_h" "$ac_includes_default"
-if test "x$ac_cv_header_sys_sysmacros_h" = xyes; then :
+if test "x$ac_cv_header_sys_sysmacros_h" = x""yes; then :
 
 $as_echo "#define MAJOR_IN_SYSMACROS 1" >>confdefs.h
 
@@ -6398,7 +6390,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_term_h" = xyes; then :
+if test "x$ac_cv_header_term_h" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_TERM_H 1
 _ACEOF
@@ -6420,7 +6412,7 @@
 #endif
 
 "
-if test "x$ac_cv_header_linux_netlink_h" = xyes; then :
+if test "x$ac_cv_header_linux_netlink_h" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LINUX_NETLINK_H 1
 _ACEOF
@@ -6586,7 +6578,7 @@
 
 # Type availability checks
 ac_fn_c_check_type "$LINENO" "mode_t" "ac_cv_type_mode_t" "$ac_includes_default"
-if test "x$ac_cv_type_mode_t" = xyes; then :
+if test "x$ac_cv_type_mode_t" = x""yes; then :
 
 else
 
@@ -6597,7 +6589,7 @@
 fi
 
 ac_fn_c_check_type "$LINENO" "off_t" "ac_cv_type_off_t" "$ac_includes_default"
-if test "x$ac_cv_type_off_t" = xyes; then :
+if test "x$ac_cv_type_off_t" = x""yes; then :
 
 else
 
@@ -6608,7 +6600,7 @@
 fi
 
 ac_fn_c_check_type "$LINENO" "pid_t" "ac_cv_type_pid_t" "$ac_includes_default"
-if test "x$ac_cv_type_pid_t" = xyes; then :
+if test "x$ac_cv_type_pid_t" = x""yes; then :
 
 else
 
@@ -6624,7 +6616,7 @@
 _ACEOF
 
 ac_fn_c_check_type "$LINENO" "size_t" "ac_cv_type_size_t" "$ac_includes_default"
-if test "x$ac_cv_type_size_t" = xyes; then :
+if test "x$ac_cv_type_size_t" = x""yes; then :
 
 else
 
@@ -6636,7 +6628,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for uid_t in sys/types.h" >&5
 $as_echo_n "checking for uid_t in sys/types.h... " >&6; }
-if ${ac_cv_type_uid_t+:} false; then :
+if test "${ac_cv_type_uid_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -6715,7 +6707,7 @@
 esac
 
 ac_fn_c_check_type "$LINENO" "ssize_t" "ac_cv_type_ssize_t" "$ac_includes_default"
-if test "x$ac_cv_type_ssize_t" = xyes; then :
+if test "x$ac_cv_type_ssize_t" = x""yes; then :
 
 $as_echo "#define HAVE_SSIZE_T 1" >>confdefs.h
 
@@ -6730,7 +6722,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of int" >&5
 $as_echo_n "checking size of int... " >&6; }
-if ${ac_cv_sizeof_int+:} false; then :
+if test "${ac_cv_sizeof_int+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (int))" "ac_cv_sizeof_int"        "$ac_includes_default"; then :
@@ -6740,7 +6732,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (int)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_int=0
    fi
@@ -6763,7 +6755,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long" >&5
 $as_echo_n "checking size of long... " >&6; }
-if ${ac_cv_sizeof_long+:} false; then :
+if test "${ac_cv_sizeof_long+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long))" "ac_cv_sizeof_long"        "$ac_includes_default"; then :
@@ -6773,7 +6765,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_long=0
    fi
@@ -6796,7 +6788,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of void *" >&5
 $as_echo_n "checking size of void *... " >&6; }
-if ${ac_cv_sizeof_void_p+:} false; then :
+if test "${ac_cv_sizeof_void_p+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (void *))" "ac_cv_sizeof_void_p"        "$ac_includes_default"; then :
@@ -6806,7 +6798,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (void *)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_void_p=0
    fi
@@ -6829,7 +6821,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of short" >&5
 $as_echo_n "checking size of short... " >&6; }
-if ${ac_cv_sizeof_short+:} false; then :
+if test "${ac_cv_sizeof_short+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (short))" "ac_cv_sizeof_short"        "$ac_includes_default"; then :
@@ -6839,7 +6831,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (short)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_short=0
    fi
@@ -6862,7 +6854,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of float" >&5
 $as_echo_n "checking size of float... " >&6; }
-if ${ac_cv_sizeof_float+:} false; then :
+if test "${ac_cv_sizeof_float+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (float))" "ac_cv_sizeof_float"        "$ac_includes_default"; then :
@@ -6872,7 +6864,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (float)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_float=0
    fi
@@ -6895,7 +6887,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of double" >&5
 $as_echo_n "checking size of double... " >&6; }
-if ${ac_cv_sizeof_double+:} false; then :
+if test "${ac_cv_sizeof_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (double))" "ac_cv_sizeof_double"        "$ac_includes_default"; then :
@@ -6905,7 +6897,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (double)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_double=0
    fi
@@ -6928,7 +6920,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of fpos_t" >&5
 $as_echo_n "checking size of fpos_t... " >&6; }
-if ${ac_cv_sizeof_fpos_t+:} false; then :
+if test "${ac_cv_sizeof_fpos_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (fpos_t))" "ac_cv_sizeof_fpos_t"        "$ac_includes_default"; then :
@@ -6938,7 +6930,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (fpos_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_fpos_t=0
    fi
@@ -6961,7 +6953,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of size_t" >&5
 $as_echo_n "checking size of size_t... " >&6; }
-if ${ac_cv_sizeof_size_t+:} false; then :
+if test "${ac_cv_sizeof_size_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (size_t))" "ac_cv_sizeof_size_t"        "$ac_includes_default"; then :
@@ -6971,7 +6963,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (size_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_size_t=0
    fi
@@ -6994,7 +6986,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of pid_t" >&5
 $as_echo_n "checking size of pid_t... " >&6; }
-if ${ac_cv_sizeof_pid_t+:} false; then :
+if test "${ac_cv_sizeof_pid_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (pid_t))" "ac_cv_sizeof_pid_t"        "$ac_includes_default"; then :
@@ -7004,7 +6996,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (pid_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_pid_t=0
    fi
@@ -7054,7 +7046,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long long" >&5
 $as_echo_n "checking size of long long... " >&6; }
-if ${ac_cv_sizeof_long_long+:} false; then :
+if test "${ac_cv_sizeof_long_long+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long long))" "ac_cv_sizeof_long_long"        "$ac_includes_default"; then :
@@ -7064,7 +7056,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long long)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_long_long=0
    fi
@@ -7115,7 +7107,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of long double" >&5
 $as_echo_n "checking size of long double... " >&6; }
-if ${ac_cv_sizeof_long_double+:} false; then :
+if test "${ac_cv_sizeof_long_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (long double))" "ac_cv_sizeof_long_double"        "$ac_includes_default"; then :
@@ -7125,7 +7117,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (long double)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_long_double=0
    fi
@@ -7177,7 +7169,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of _Bool" >&5
 $as_echo_n "checking size of _Bool... " >&6; }
-if ${ac_cv_sizeof__Bool+:} false; then :
+if test "${ac_cv_sizeof__Bool+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (_Bool))" "ac_cv_sizeof__Bool"        "$ac_includes_default"; then :
@@ -7187,7 +7179,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (_Bool)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof__Bool=0
    fi
@@ -7213,7 +7205,7 @@
         #include <inttypes.h>
         #endif
 "
-if test "x$ac_cv_type_uintptr_t" = xyes; then :
+if test "x$ac_cv_type_uintptr_t" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_UINTPTR_T 1
@@ -7225,7 +7217,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of uintptr_t" >&5
 $as_echo_n "checking size of uintptr_t... " >&6; }
-if ${ac_cv_sizeof_uintptr_t+:} false; then :
+if test "${ac_cv_sizeof_uintptr_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (uintptr_t))" "ac_cv_sizeof_uintptr_t"        "$ac_includes_default"; then :
@@ -7235,7 +7227,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (uintptr_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_uintptr_t=0
    fi
@@ -7261,7 +7253,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of off_t" >&5
 $as_echo_n "checking size of off_t... " >&6; }
-if ${ac_cv_sizeof_off_t+:} false; then :
+if test "${ac_cv_sizeof_off_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (off_t))" "ac_cv_sizeof_off_t"        "
@@ -7276,7 +7268,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (off_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_off_t=0
    fi
@@ -7320,7 +7312,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of time_t" >&5
 $as_echo_n "checking size of time_t... " >&6; }
-if ${ac_cv_sizeof_time_t+:} false; then :
+if test "${ac_cv_sizeof_time_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (time_t))" "ac_cv_sizeof_time_t"        "
@@ -7338,7 +7330,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (time_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_time_t=0
    fi
@@ -7395,7 +7387,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of pthread_t" >&5
 $as_echo_n "checking size of pthread_t... " >&6; }
-if ${ac_cv_sizeof_pthread_t+:} false; then :
+if test "${ac_cv_sizeof_pthread_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (pthread_t))" "ac_cv_sizeof_pthread_t"        "
@@ -7410,7 +7402,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (pthread_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_pthread_t=0
    fi
@@ -7842,7 +7834,7 @@
 # checks for libraries
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for dlopen in -ldl" >&5
 $as_echo_n "checking for dlopen in -ldl... " >&6; }
-if ${ac_cv_lib_dl_dlopen+:} false; then :
+if test "${ac_cv_lib_dl_dlopen+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7876,7 +7868,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_dl_dlopen" >&5
 $as_echo "$ac_cv_lib_dl_dlopen" >&6; }
-if test "x$ac_cv_lib_dl_dlopen" = xyes; then :
+if test "x$ac_cv_lib_dl_dlopen" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBDL 1
 _ACEOF
@@ -7887,7 +7879,7 @@
 	# Dynamic linking for SunOS/Solaris and SYSV
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for shl_load in -ldld" >&5
 $as_echo_n "checking for shl_load in -ldld... " >&6; }
-if ${ac_cv_lib_dld_shl_load+:} false; then :
+if test "${ac_cv_lib_dld_shl_load+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -7921,7 +7913,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_dld_shl_load" >&5
 $as_echo "$ac_cv_lib_dld_shl_load" >&6; }
-if test "x$ac_cv_lib_dld_shl_load" = xyes; then :
+if test "x$ac_cv_lib_dld_shl_load" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBDLD 1
 _ACEOF
@@ -7935,7 +7927,7 @@
 if test "$with_threads" = "yes" -o -z "$with_threads"; then
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sem_init" >&5
 $as_echo_n "checking for library containing sem_init... " >&6; }
-if ${ac_cv_search_sem_init+:} false; then :
+if test "${ac_cv_search_sem_init+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_func_search_save_LIBS=$LIBS
@@ -7969,11 +7961,11 @@
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext
-  if ${ac_cv_search_sem_init+:} false; then :
+  if test "${ac_cv_search_sem_init+set}" = set; then :
   break
 fi
 done
-if ${ac_cv_search_sem_init+:} false; then :
+if test "${ac_cv_search_sem_init+set}" = set; then :
 
 else
   ac_cv_search_sem_init=no
@@ -7996,7 +7988,7 @@
 # check if we need libintl for locale functions
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for textdomain in -lintl" >&5
 $as_echo_n "checking for textdomain in -lintl... " >&6; }
-if ${ac_cv_lib_intl_textdomain+:} false; then :
+if test "${ac_cv_lib_intl_textdomain+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8030,7 +8022,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_intl_textdomain" >&5
 $as_echo "$ac_cv_lib_intl_textdomain" >&6; }
-if test "x$ac_cv_lib_intl_textdomain" = xyes; then :
+if test "x$ac_cv_lib_intl_textdomain" = x""yes; then :
 
 $as_echo "#define WITH_LIBINTL 1" >>confdefs.h
 
@@ -8077,7 +8069,7 @@
 # Most SVR4 platforms (e.g. Solaris) need -lsocket and -lnsl.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for t_open in -lnsl" >&5
 $as_echo_n "checking for t_open in -lnsl... " >&6; }
-if ${ac_cv_lib_nsl_t_open+:} false; then :
+if test "${ac_cv_lib_nsl_t_open+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8111,13 +8103,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_nsl_t_open" >&5
 $as_echo "$ac_cv_lib_nsl_t_open" >&6; }
-if test "x$ac_cv_lib_nsl_t_open" = xyes; then :
+if test "x$ac_cv_lib_nsl_t_open" = x""yes; then :
   LIBS="-lnsl $LIBS"
 fi
  # SVR4
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for socket in -lsocket" >&5
 $as_echo_n "checking for socket in -lsocket... " >&6; }
-if ${ac_cv_lib_socket_socket+:} false; then :
+if test "${ac_cv_lib_socket_socket+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8151,7 +8143,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_socket_socket" >&5
 $as_echo "$ac_cv_lib_socket_socket" >&6; }
-if test "x$ac_cv_lib_socket_socket" = xyes; then :
+if test "x$ac_cv_lib_socket_socket" = x""yes; then :
   LIBS="-lsocket $LIBS"
 fi
  # SVR4 sockets
@@ -8177,7 +8169,7 @@
 set dummy ${ac_tool_prefix}pkg-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_PKG_CONFIG+:} false; then :
+if test "${ac_cv_path_PKG_CONFIG+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $PKG_CONFIG in
@@ -8220,7 +8212,7 @@
 set dummy pkg-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then :
+if test "${ac_cv_path_ac_pt_PKG_CONFIG+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   case $ac_pt_PKG_CONFIG in
@@ -8502,7 +8494,7 @@
     $as_echo "#define _REENTRANT 1" >>confdefs.h
 
     ac_fn_c_check_header_mongrel "$LINENO" "cthreads.h" "ac_cv_header_cthreads_h" "$ac_includes_default"
-if test "x$ac_cv_header_cthreads_h" = xyes; then :
+if test "x$ac_cv_header_cthreads_h" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     $as_echo "#define C_THREADS 1" >>confdefs.h
@@ -8515,7 +8507,7 @@
 else
 
     ac_fn_c_check_header_mongrel "$LINENO" "mach/cthreads.h" "ac_cv_header_mach_cthreads_h" "$ac_includes_default"
-if test "x$ac_cv_header_mach_cthreads_h" = xyes; then :
+if test "x$ac_cv_header_mach_cthreads_h" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     $as_echo "#define C_THREADS 1" >>confdefs.h
@@ -8559,7 +8551,7 @@
 
     LIBS=$_libs
     ac_fn_c_check_func "$LINENO" "pthread_detach" "ac_cv_func_pthread_detach"
-if test "x$ac_cv_func_pthread_detach" = xyes; then :
+if test "x$ac_cv_func_pthread_detach" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8568,7 +8560,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthreads" >&5
 $as_echo_n "checking for pthread_create in -lpthreads... " >&6; }
-if ${ac_cv_lib_pthreads_pthread_create+:} false; then :
+if test "${ac_cv_lib_pthreads_pthread_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8602,7 +8594,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthreads_pthread_create" >&5
 $as_echo "$ac_cv_lib_pthreads_pthread_create" >&6; }
-if test "x$ac_cv_lib_pthreads_pthread_create" = xyes; then :
+if test "x$ac_cv_lib_pthreads_pthread_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8612,7 +8604,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lc_r" >&5
 $as_echo_n "checking for pthread_create in -lc_r... " >&6; }
-if ${ac_cv_lib_c_r_pthread_create+:} false; then :
+if test "${ac_cv_lib_c_r_pthread_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8646,7 +8638,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_c_r_pthread_create" >&5
 $as_echo "$ac_cv_lib_c_r_pthread_create" >&6; }
-if test "x$ac_cv_lib_c_r_pthread_create" = xyes; then :
+if test "x$ac_cv_lib_c_r_pthread_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8656,7 +8648,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __pthread_create_system in -lpthread" >&5
 $as_echo_n "checking for __pthread_create_system in -lpthread... " >&6; }
-if ${ac_cv_lib_pthread___pthread_create_system+:} false; then :
+if test "${ac_cv_lib_pthread___pthread_create_system+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8690,7 +8682,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread___pthread_create_system" >&5
 $as_echo "$ac_cv_lib_pthread___pthread_create_system" >&6; }
-if test "x$ac_cv_lib_pthread___pthread_create_system" = xyes; then :
+if test "x$ac_cv_lib_pthread___pthread_create_system" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8700,7 +8692,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lcma" >&5
 $as_echo_n "checking for pthread_create in -lcma... " >&6; }
-if ${ac_cv_lib_cma_pthread_create+:} false; then :
+if test "${ac_cv_lib_cma_pthread_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8734,7 +8726,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_cma_pthread_create" >&5
 $as_echo "$ac_cv_lib_cma_pthread_create" >&6; }
-if test "x$ac_cv_lib_cma_pthread_create" = xyes; then :
+if test "x$ac_cv_lib_cma_pthread_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     posix_threads=yes
@@ -8766,7 +8758,7 @@
 
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking for usconfig in -lmpc" >&5
 $as_echo_n "checking for usconfig in -lmpc... " >&6; }
-if ${ac_cv_lib_mpc_usconfig+:} false; then :
+if test "${ac_cv_lib_mpc_usconfig+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8800,7 +8792,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_mpc_usconfig" >&5
 $as_echo "$ac_cv_lib_mpc_usconfig" >&6; }
-if test "x$ac_cv_lib_mpc_usconfig" = xyes; then :
+if test "x$ac_cv_lib_mpc_usconfig" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
     LIBS="$LIBS -lmpc"
@@ -8812,7 +8804,7 @@
     if test "$posix_threads" != "yes"; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: checking for thr_create in -lthread" >&5
 $as_echo_n "checking for thr_create in -lthread... " >&6; }
-if ${ac_cv_lib_thread_thr_create+:} false; then :
+if test "${ac_cv_lib_thread_thr_create+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -8846,7 +8838,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_thread_thr_create" >&5
 $as_echo "$ac_cv_lib_thread_thr_create" >&6; }
-if test "x$ac_cv_lib_thread_thr_create" = xyes; then :
+if test "x$ac_cv_lib_thread_thr_create" = x""yes; then :
   $as_echo "#define WITH_THREAD 1" >>confdefs.h
 
       LIBS="$LIBS -lthread"
@@ -8891,7 +8883,7 @@
 
       { $as_echo "$as_me:${as_lineno-$LINENO}: checking if PTHREAD_SCOPE_SYSTEM is supported" >&5
 $as_echo_n "checking if PTHREAD_SCOPE_SYSTEM is supported... " >&6; }
-      if ${ac_cv_pthread_system_supported+:} false; then :
+      if test "${ac_cv_pthread_system_supported+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -8934,7 +8926,7 @@
       for ac_func in pthread_sigmask
 do :
   ac_fn_c_check_func "$LINENO" "pthread_sigmask" "ac_cv_func_pthread_sigmask"
-if test "x$ac_cv_func_pthread_sigmask" = xyes; then :
+if test "x$ac_cv_func_pthread_sigmask" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_PTHREAD_SIGMASK 1
 _ACEOF
@@ -9326,7 +9318,7 @@
 $as_echo "$with_valgrind" >&6; }
 if test "$with_valgrind" != no; then
     ac_fn_c_check_header_mongrel "$LINENO" "valgrind/valgrind.h" "ac_cv_header_valgrind_valgrind_h" "$ac_includes_default"
-if test "x$ac_cv_header_valgrind_valgrind_h" = xyes; then :
+if test "x$ac_cv_header_valgrind_valgrind_h" = x""yes; then :
 
 $as_echo "#define WITH_VALGRIND 1" >>confdefs.h
 
@@ -9348,7 +9340,7 @@
 for ac_func in dlopen
 do :
   ac_fn_c_check_func "$LINENO" "dlopen" "ac_cv_func_dlopen"
-if test "x$ac_cv_func_dlopen" = xyes; then :
+if test "x$ac_cv_func_dlopen" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_DLOPEN 1
 _ACEOF
@@ -9675,7 +9667,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock declaration" >&5
 $as_echo_n "checking for flock declaration... " >&6; }
-if ${ac_cv_flock_decl+:} false; then :
+if test "${ac_cv_flock_decl+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -9705,7 +9697,7 @@
   for ac_func in flock
 do :
   ac_fn_c_check_func "$LINENO" "flock" "ac_cv_func_flock"
-if test "x$ac_cv_func_flock" = xyes; then :
+if test "x$ac_cv_func_flock" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_FLOCK 1
 _ACEOF
@@ -9713,7 +9705,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for flock in -lbsd" >&5
 $as_echo_n "checking for flock in -lbsd... " >&6; }
-if ${ac_cv_lib_bsd_flock+:} false; then :
+if test "${ac_cv_lib_bsd_flock+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9747,7 +9739,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_flock" >&5
 $as_echo "$ac_cv_lib_bsd_flock" >&6; }
-if test "x$ac_cv_lib_bsd_flock" = xyes; then :
+if test "x$ac_cv_lib_bsd_flock" = x""yes; then :
   $as_echo "#define HAVE_FLOCK 1" >>confdefs.h
 
 
@@ -9790,13 +9782,41 @@
 fi
 rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken unsetenv" >&5
+$as_echo_n "checking for broken unsetenv... " >&6; }
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+#include <stdlib.h>
+
+int
+main ()
+{
+int res = unsetenv("DUMMY")
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_compile "$LINENO"; then :
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+else
+
+$as_echo "#define HAVE_BROKEN_UNSETENV 1" >>confdefs.h
+
+   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+
 for ac_prog in true
 do
   # Extract the first word of "$ac_prog", so it can be a program name with args.
 set dummy $ac_prog; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
 $as_echo_n "checking for $ac_word... " >&6; }
-if ${ac_cv_prog_TRUE+:} false; then :
+if test "${ac_cv_prog_TRUE+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test -n "$TRUE"; then
@@ -9836,7 +9856,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inet_aton in -lc" >&5
 $as_echo_n "checking for inet_aton in -lc... " >&6; }
-if ${ac_cv_lib_c_inet_aton+:} false; then :
+if test "${ac_cv_lib_c_inet_aton+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9870,12 +9890,12 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_c_inet_aton" >&5
 $as_echo "$ac_cv_lib_c_inet_aton" >&6; }
-if test "x$ac_cv_lib_c_inet_aton" = xyes; then :
+if test "x$ac_cv_lib_c_inet_aton" = x""yes; then :
   $ac_cv_prog_TRUE
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inet_aton in -lresolv" >&5
 $as_echo_n "checking for inet_aton in -lresolv... " >&6; }
-if ${ac_cv_lib_resolv_inet_aton+:} false; then :
+if test "${ac_cv_lib_resolv_inet_aton+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -9909,7 +9929,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_resolv_inet_aton" >&5
 $as_echo "$ac_cv_lib_resolv_inet_aton" >&6; }
-if test "x$ac_cv_lib_resolv_inet_aton" = xyes; then :
+if test "x$ac_cv_lib_resolv_inet_aton" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBRESOLV 1
 _ACEOF
@@ -9926,7 +9946,7 @@
 # exit Python
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for chflags" >&5
 $as_echo_n "checking for chflags... " >&6; }
-if ${ac_cv_have_chflags+:} false; then :
+if test "${ac_cv_have_chflags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -9960,7 +9980,7 @@
 $as_echo "$ac_cv_have_chflags" >&6; }
 if test "$ac_cv_have_chflags" = cross ; then
   ac_fn_c_check_func "$LINENO" "chflags" "ac_cv_func_chflags"
-if test "x$ac_cv_func_chflags" = xyes; then :
+if test "x$ac_cv_func_chflags" = x""yes; then :
   ac_cv_have_chflags="yes"
 else
   ac_cv_have_chflags="no"
@@ -9975,7 +9995,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for lchflags" >&5
 $as_echo_n "checking for lchflags... " >&6; }
-if ${ac_cv_have_lchflags+:} false; then :
+if test "${ac_cv_have_lchflags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -10009,7 +10029,7 @@
 $as_echo "$ac_cv_have_lchflags" >&6; }
 if test "$ac_cv_have_lchflags" = cross ; then
   ac_fn_c_check_func "$LINENO" "lchflags" "ac_cv_func_lchflags"
-if test "x$ac_cv_func_lchflags" = xyes; then :
+if test "x$ac_cv_func_lchflags" = x""yes; then :
   ac_cv_have_lchflags="yes"
 else
   ac_cv_have_lchflags="no"
@@ -10033,7 +10053,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for inflateCopy in -lz" >&5
 $as_echo_n "checking for inflateCopy in -lz... " >&6; }
-if ${ac_cv_lib_z_inflateCopy+:} false; then :
+if test "${ac_cv_lib_z_inflateCopy+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10067,7 +10087,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_z_inflateCopy" >&5
 $as_echo "$ac_cv_lib_z_inflateCopy" >&6; }
-if test "x$ac_cv_lib_z_inflateCopy" = xyes; then :
+if test "x$ac_cv_lib_z_inflateCopy" = x""yes; then :
 
 $as_echo "#define HAVE_ZLIB_COPY 1" >>confdefs.h
 
@@ -10210,7 +10230,7 @@
 for ac_func in openpty
 do :
   ac_fn_c_check_func "$LINENO" "openpty" "ac_cv_func_openpty"
-if test "x$ac_cv_func_openpty" = xyes; then :
+if test "x$ac_cv_func_openpty" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_OPENPTY 1
 _ACEOF
@@ -10218,7 +10238,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty in -lutil" >&5
 $as_echo_n "checking for openpty in -lutil... " >&6; }
-if ${ac_cv_lib_util_openpty+:} false; then :
+if test "${ac_cv_lib_util_openpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10252,13 +10272,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_util_openpty" >&5
 $as_echo "$ac_cv_lib_util_openpty" >&6; }
-if test "x$ac_cv_lib_util_openpty" = xyes; then :
+if test "x$ac_cv_lib_util_openpty" = x""yes; then :
   $as_echo "#define HAVE_OPENPTY 1" >>confdefs.h
  LIBS="$LIBS -lutil"
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty in -lbsd" >&5
 $as_echo_n "checking for openpty in -lbsd... " >&6; }
-if ${ac_cv_lib_bsd_openpty+:} false; then :
+if test "${ac_cv_lib_bsd_openpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10292,7 +10312,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_openpty" >&5
 $as_echo "$ac_cv_lib_bsd_openpty" >&6; }
-if test "x$ac_cv_lib_bsd_openpty" = xyes; then :
+if test "x$ac_cv_lib_bsd_openpty" = x""yes; then :
   $as_echo "#define HAVE_OPENPTY 1" >>confdefs.h
  LIBS="$LIBS -lbsd"
 fi
@@ -10307,7 +10327,7 @@
 for ac_func in forkpty
 do :
   ac_fn_c_check_func "$LINENO" "forkpty" "ac_cv_func_forkpty"
-if test "x$ac_cv_func_forkpty" = xyes; then :
+if test "x$ac_cv_func_forkpty" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_FORKPTY 1
 _ACEOF
@@ -10315,7 +10335,7 @@
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for forkpty in -lutil" >&5
 $as_echo_n "checking for forkpty in -lutil... " >&6; }
-if ${ac_cv_lib_util_forkpty+:} false; then :
+if test "${ac_cv_lib_util_forkpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10349,13 +10369,13 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_util_forkpty" >&5
 $as_echo "$ac_cv_lib_util_forkpty" >&6; }
-if test "x$ac_cv_lib_util_forkpty" = xyes; then :
+if test "x$ac_cv_lib_util_forkpty" = x""yes; then :
   $as_echo "#define HAVE_FORKPTY 1" >>confdefs.h
  LIBS="$LIBS -lutil"
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for forkpty in -lbsd" >&5
 $as_echo_n "checking for forkpty in -lbsd... " >&6; }
-if ${ac_cv_lib_bsd_forkpty+:} false; then :
+if test "${ac_cv_lib_bsd_forkpty+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -10389,7 +10409,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_bsd_forkpty" >&5
 $as_echo "$ac_cv_lib_bsd_forkpty" >&6; }
-if test "x$ac_cv_lib_bsd_forkpty" = xyes; then :
+if test "x$ac_cv_lib_bsd_forkpty" = x""yes; then :
   $as_echo "#define HAVE_FORKPTY 1" >>confdefs.h
  LIBS="$LIBS -lbsd"
 fi
@@ -10406,7 +10426,7 @@
 for ac_func in memmove
 do :
   ac_fn_c_check_func "$LINENO" "memmove" "ac_cv_func_memmove"
-if test "x$ac_cv_func_memmove" = xyes; then :
+if test "x$ac_cv_func_memmove" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_MEMMOVE 1
 _ACEOF
@@ -10430,7 +10450,7 @@
 
 
 ac_fn_c_check_func "$LINENO" "dup2" "ac_cv_func_dup2"
-if test "x$ac_cv_func_dup2" = xyes; then :
+if test "x$ac_cv_func_dup2" = x""yes; then :
   $as_echo "#define HAVE_DUP2 1" >>confdefs.h
 
 else
@@ -10443,7 +10463,7 @@
 fi
 
 ac_fn_c_check_func "$LINENO" "getcwd" "ac_cv_func_getcwd"
-if test "x$ac_cv_func_getcwd" = xyes; then :
+if test "x$ac_cv_func_getcwd" = x""yes; then :
   $as_echo "#define HAVE_GETCWD 1" >>confdefs.h
 
 else
@@ -10456,7 +10476,7 @@
 fi
 
 ac_fn_c_check_func "$LINENO" "strdup" "ac_cv_func_strdup"
-if test "x$ac_cv_func_strdup" = xyes; then :
+if test "x$ac_cv_func_strdup" = x""yes; then :
   $as_echo "#define HAVE_STRDUP 1" >>confdefs.h
 
 else
@@ -10472,7 +10492,7 @@
 for ac_func in getpgrp
 do :
   ac_fn_c_check_func "$LINENO" "getpgrp" "ac_cv_func_getpgrp"
-if test "x$ac_cv_func_getpgrp" = xyes; then :
+if test "x$ac_cv_func_getpgrp" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETPGRP 1
 _ACEOF
@@ -10500,7 +10520,7 @@
 for ac_func in setpgrp
 do :
   ac_fn_c_check_func "$LINENO" "setpgrp" "ac_cv_func_setpgrp"
-if test "x$ac_cv_func_setpgrp" = xyes; then :
+if test "x$ac_cv_func_setpgrp" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_SETPGRP 1
 _ACEOF
@@ -10528,7 +10548,7 @@
 for ac_func in gettimeofday
 do :
   ac_fn_c_check_func "$LINENO" "gettimeofday" "ac_cv_func_gettimeofday"
-if test "x$ac_cv_func_gettimeofday" = xyes; then :
+if test "x$ac_cv_func_gettimeofday" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETTIMEOFDAY 1
 _ACEOF
@@ -10630,7 +10650,7 @@
 then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking getaddrinfo bug" >&5
 $as_echo_n "checking getaddrinfo bug... " >&6; }
-  if ${ac_cv_buggy_getaddrinfo+:} false; then :
+  if test "${ac_cv_buggy_getaddrinfo+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -10759,7 +10779,7 @@
 for ac_func in getnameinfo
 do :
   ac_fn_c_check_func "$LINENO" "getnameinfo" "ac_cv_func_getnameinfo"
-if test "x$ac_cv_func_getnameinfo" = xyes; then :
+if test "x$ac_cv_func_getnameinfo" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETNAMEINFO 1
 _ACEOF
@@ -10771,7 +10791,7 @@
 # checks for structures
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether time.h and sys/time.h may both be included" >&5
 $as_echo_n "checking whether time.h and sys/time.h may both be included... " >&6; }
-if ${ac_cv_header_time+:} false; then :
+if test "${ac_cv_header_time+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10806,7 +10826,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether struct tm is in sys/time.h or time.h" >&5
 $as_echo_n "checking whether struct tm is in sys/time.h or time.h... " >&6; }
-if ${ac_cv_struct_tm+:} false; then :
+if test "${ac_cv_struct_tm+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10843,7 +10863,7 @@
 #include <$ac_cv_struct_tm>
 
 "
-if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then :
+if test "x$ac_cv_member_struct_tm_tm_zone" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_TM_TM_ZONE 1
@@ -10859,7 +10879,7 @@
 else
   ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h>
 "
-if test "x$ac_cv_have_decl_tzname" = xyes; then :
+if test "x$ac_cv_have_decl_tzname" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -10871,7 +10891,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5
 $as_echo_n "checking for tzname... " >&6; }
-if ${ac_cv_var_tzname+:} false; then :
+if test "${ac_cv_var_tzname+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -10907,7 +10927,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_rdev" "ac_cv_member_struct_stat_st_rdev" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_rdev" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_rdev" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_RDEV 1
@@ -10917,7 +10937,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_blksize" "ac_cv_member_struct_stat_st_blksize" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_blksize" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_blksize" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BLKSIZE 1
@@ -10927,7 +10947,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_flags" "ac_cv_member_struct_stat_st_flags" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_flags" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_flags" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_FLAGS 1
@@ -10937,7 +10957,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_gen" "ac_cv_member_struct_stat_st_gen" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_gen" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_gen" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_GEN 1
@@ -10947,7 +10967,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_birthtime" "ac_cv_member_struct_stat_st_birthtime" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_birthtime" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_birthtime" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BIRTHTIME 1
@@ -10957,7 +10977,7 @@
 fi
 
 ac_fn_c_check_member "$LINENO" "struct stat" "st_blocks" "ac_cv_member_struct_stat_st_blocks" "$ac_includes_default"
-if test "x$ac_cv_member_struct_stat_st_blocks" = xyes; then :
+if test "x$ac_cv_member_struct_stat_st_blocks" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_STAT_ST_BLOCKS 1
@@ -10979,7 +10999,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for time.h that defines altzone" >&5
 $as_echo_n "checking for time.h that defines altzone... " >&6; }
-if ${ac_cv_header_time_altzone+:} false; then :
+if test "${ac_cv_header_time_altzone+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11043,7 +11063,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for addrinfo" >&5
 $as_echo_n "checking for addrinfo... " >&6; }
-if ${ac_cv_struct_addrinfo+:} false; then :
+if test "${ac_cv_struct_addrinfo+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11075,7 +11095,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sockaddr_storage" >&5
 $as_echo_n "checking for sockaddr_storage... " >&6; }
-if ${ac_cv_struct_sockaddr_storage+:} false; then :
+if test "${ac_cv_struct_sockaddr_storage+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11111,7 +11131,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether char is unsigned" >&5
 $as_echo_n "checking whether char is unsigned... " >&6; }
-if ${ac_cv_c_char_unsigned+:} false; then :
+if test "${ac_cv_c_char_unsigned+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11143,7 +11163,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for an ANSI C-conforming const" >&5
 $as_echo_n "checking for an ANSI C-conforming const... " >&6; }
-if ${ac_cv_c_const+:} false; then :
+if test "${ac_cv_c_const+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -11431,7 +11451,7 @@
 
 
 ac_fn_c_check_func "$LINENO" "gethostbyname_r" "ac_cv_func_gethostbyname_r"
-if test "x$ac_cv_func_gethostbyname_r" = xyes; then :
+if test "x$ac_cv_func_gethostbyname_r" = x""yes; then :
 
   $as_echo "#define HAVE_GETHOSTBYNAME_R 1" >>confdefs.h
 
@@ -11562,7 +11582,7 @@
   for ac_func in gethostbyname
 do :
   ac_fn_c_check_func "$LINENO" "gethostbyname" "ac_cv_func_gethostbyname"
-if test "x$ac_cv_func_gethostbyname" = xyes; then :
+if test "x$ac_cv_func_gethostbyname" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_GETHOSTBYNAME 1
 _ACEOF
@@ -11584,12 +11604,12 @@
 
 # Linux requires this for correct f.p. operations
 ac_fn_c_check_func "$LINENO" "__fpu_control" "ac_cv_func___fpu_control"
-if test "x$ac_cv_func___fpu_control" = xyes; then :
+if test "x$ac_cv_func___fpu_control" = x""yes; then :
 
 else
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __fpu_control in -lieee" >&5
 $as_echo_n "checking for __fpu_control in -lieee... " >&6; }
-if ${ac_cv_lib_ieee___fpu_control+:} false; then :
+if test "${ac_cv_lib_ieee___fpu_control+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -11623,7 +11643,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_ieee___fpu_control" >&5
 $as_echo "$ac_cv_lib_ieee___fpu_control" >&6; }
-if test "x$ac_cv_lib_ieee___fpu_control" = xyes; then :
+if test "x$ac_cv_lib_ieee___fpu_control" = x""yes; then :
   cat >>confdefs.h <<_ACEOF
 #define HAVE_LIBIEEE 1
 _ACEOF
@@ -11717,7 +11737,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are little-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are little-endian IEEE 754 binary64... " >&6; }
-if ${ac_cv_little_endian_double+:} false; then :
+if test "${ac_cv_little_endian_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11759,7 +11779,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are big-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are big-endian IEEE 754 binary64... " >&6; }
-if ${ac_cv_big_endian_double+:} false; then :
+if test "${ac_cv_big_endian_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11805,7 +11825,7 @@
 # conversions work.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether C doubles are ARM mixed-endian IEEE 754 binary64" >&5
 $as_echo_n "checking whether C doubles are ARM mixed-endian IEEE 754 binary64... " >&6; }
-if ${ac_cv_mixed_endian_double+:} false; then :
+if test "${ac_cv_mixed_endian_double+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -11975,7 +11995,7 @@
 
 ac_fn_c_check_decl "$LINENO" "isinf" "ac_cv_have_decl_isinf" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isinf" = xyes; then :
+if test "x$ac_cv_have_decl_isinf" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11986,7 +12006,7 @@
 _ACEOF
 ac_fn_c_check_decl "$LINENO" "isnan" "ac_cv_have_decl_isnan" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isnan" = xyes; then :
+if test "x$ac_cv_have_decl_isnan" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -11997,7 +12017,7 @@
 _ACEOF
 ac_fn_c_check_decl "$LINENO" "isfinite" "ac_cv_have_decl_isfinite" "#include <math.h>
 "
-if test "x$ac_cv_have_decl_isfinite" = xyes; then :
+if test "x$ac_cv_have_decl_isfinite" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -12012,7 +12032,7 @@
 # -0. on some architectures.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether tanh preserves the sign of zero" >&5
 $as_echo_n "checking whether tanh preserves the sign of zero... " >&6; }
-if ${ac_cv_tanh_preserves_zero_sign+:} false; then :
+if test "${ac_cv_tanh_preserves_zero_sign+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12060,7 +12080,7 @@
     # -0.  See issue #9920.
     { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether log1p drops the sign of negative zero" >&5
 $as_echo_n "checking whether log1p drops the sign of negative zero... " >&6; }
-    if ${ac_cv_log1p_drops_zero_sign+:} false; then :
+    if test "${ac_cv_log1p_drops_zero_sign+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12112,7 +12132,7 @@
 # sem_open results in a 'Signal 12' error.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether POSIX semaphores are enabled" >&5
 $as_echo_n "checking whether POSIX semaphores are enabled... " >&6; }
-if ${ac_cv_posix_semaphores_enabled+:} false; then :
+if test "${ac_cv_posix_semaphores_enabled+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -12163,7 +12183,7 @@
 # Multiprocessing check for broken sem_getvalue
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken sem_getvalue" >&5
 $as_echo_n "checking for broken sem_getvalue... " >&6; }
-if ${ac_cv_broken_sem_getvalue+:} false; then :
+if test "${ac_cv_broken_sem_getvalue+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -12228,7 +12248,7 @@
 15|30)
   ;;
 *)
-  as_fn_error $? "bad value $enable_big_digits for --enable-big-digits; value should be 15 or 30" "$LINENO" 5 ;;
+  as_fn_error $? "bad value $enable_big_digits for --enable-big-digits; value should be 15 or 30" "$LINENO" 5  ;;
 esac
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_big_digits" >&5
 $as_echo "$enable_big_digits" >&6; }
@@ -12246,7 +12266,7 @@
 
 # check for wchar.h
 ac_fn_c_check_header_mongrel "$LINENO" "wchar.h" "ac_cv_header_wchar_h" "$ac_includes_default"
-if test "x$ac_cv_header_wchar_h" = xyes; then :
+if test "x$ac_cv_header_wchar_h" = x""yes; then :
 
 
 $as_echo "#define HAVE_WCHAR_H 1" >>confdefs.h
@@ -12269,7 +12289,7 @@
 # This bug is HP SR number 8606223364.
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking size of wchar_t" >&5
 $as_echo_n "checking size of wchar_t... " >&6; }
-if ${ac_cv_sizeof_wchar_t+:} false; then :
+if test "${ac_cv_sizeof_wchar_t+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if ac_fn_c_compute_int "$LINENO" "(long int) (sizeof (wchar_t))" "ac_cv_sizeof_wchar_t"        "#include <wchar.h>
@@ -12280,7 +12300,7 @@
      { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5
 $as_echo "$as_me: error: in \`$ac_pwd':" >&2;}
 as_fn_error 77 "cannot compute sizeof (wchar_t)
-See \`config.log' for more details" "$LINENO" 5; }
+See \`config.log' for more details" "$LINENO" 5 ; }
    else
      ac_cv_sizeof_wchar_t=0
    fi
@@ -12335,7 +12355,7 @@
   # check whether wchar_t is signed or not
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether wchar_t is signed" >&5
 $as_echo_n "checking whether wchar_t is signed... " >&6; }
-  if ${ac_cv_wchar_t_signed+:} false; then :
+  if test "${ac_cv_wchar_t_signed+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12431,7 +12451,7 @@
 # check for endianness
  { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether byte ordering is bigendian" >&5
 $as_echo_n "checking whether byte ordering is bigendian... " >&6; }
-if ${ac_cv_c_bigendian+:} false; then :
+if test "${ac_cv_c_bigendian+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_cv_c_bigendian=unknown
@@ -12650,7 +12670,7 @@
      ;; #(
    *)
      as_fn_error $? "unknown endianness
- presetting ac_cv_c_bigendian=no (or yes) will help" "$LINENO" 5 ;;
+ presetting ac_cv_c_bigendian=no (or yes) will help" "$LINENO" 5  ;;
  esac
 
 
@@ -12722,7 +12742,7 @@
 # or fills with zeros (like the Cray J90, according to Tim Peters).
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether right shift extends the sign bit" >&5
 $as_echo_n "checking whether right shift extends the sign bit... " >&6; }
-if ${ac_cv_rshift_extends_sign+:} false; then :
+if test "${ac_cv_rshift_extends_sign+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12761,7 +12781,7 @@
 # check for getc_unlocked and related locking functions
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for getc_unlocked() and friends" >&5
 $as_echo_n "checking for getc_unlocked() and friends... " >&6; }
-if ${ac_cv_have_getc_unlocked+:} false; then :
+if test "${ac_cv_have_getc_unlocked+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -12859,7 +12879,7 @@
 # check for readline 2.1
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_callback_handler_install in -lreadline" >&5
 $as_echo_n "checking for rl_callback_handler_install in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_callback_handler_install+:} false; then :
+if test "${ac_cv_lib_readline_rl_callback_handler_install+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12893,7 +12913,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_callback_handler_install" >&5
 $as_echo "$ac_cv_lib_readline_rl_callback_handler_install" >&6; }
-if test "x$ac_cv_lib_readline_rl_callback_handler_install" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_callback_handler_install" = x""yes; then :
 
 $as_echo "#define HAVE_RL_CALLBACK 1" >>confdefs.h
 
@@ -12945,7 +12965,7 @@
 # check for readline 4.0
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_pre_input_hook in -lreadline" >&5
 $as_echo_n "checking for rl_pre_input_hook in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_pre_input_hook+:} false; then :
+if test "${ac_cv_lib_readline_rl_pre_input_hook+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -12979,7 +12999,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_pre_input_hook" >&5
 $as_echo "$ac_cv_lib_readline_rl_pre_input_hook" >&6; }
-if test "x$ac_cv_lib_readline_rl_pre_input_hook" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_pre_input_hook" = x""yes; then :
 
 $as_echo "#define HAVE_RL_PRE_INPUT_HOOK 1" >>confdefs.h
 
@@ -12989,7 +13009,7 @@
 # also in 4.0
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_completion_display_matches_hook in -lreadline" >&5
 $as_echo_n "checking for rl_completion_display_matches_hook in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_completion_display_matches_hook+:} false; then :
+if test "${ac_cv_lib_readline_rl_completion_display_matches_hook+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -13023,7 +13043,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_completion_display_matches_hook" >&5
 $as_echo "$ac_cv_lib_readline_rl_completion_display_matches_hook" >&6; }
-if test "x$ac_cv_lib_readline_rl_completion_display_matches_hook" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_completion_display_matches_hook" = x""yes; then :
 
 $as_echo "#define HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK 1" >>confdefs.h
 
@@ -13033,7 +13053,7 @@
 # check for readline 4.2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for rl_completion_matches in -lreadline" >&5
 $as_echo_n "checking for rl_completion_matches in -lreadline... " >&6; }
-if ${ac_cv_lib_readline_rl_completion_matches+:} false; then :
+if test "${ac_cv_lib_readline_rl_completion_matches+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
@@ -13067,7 +13087,7 @@
 fi
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_readline_rl_completion_matches" >&5
 $as_echo "$ac_cv_lib_readline_rl_completion_matches" >&6; }
-if test "x$ac_cv_lib_readline_rl_completion_matches" = xyes; then :
+if test "x$ac_cv_lib_readline_rl_completion_matches" = x""yes; then :
 
 $as_echo "#define HAVE_RL_COMPLETION_MATCHES 1" >>confdefs.h
 
@@ -13108,7 +13128,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken nice()" >&5
 $as_echo_n "checking for broken nice()... " >&6; }
-if ${ac_cv_broken_nice+:} false; then :
+if test "${ac_cv_broken_nice+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -13149,7 +13169,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken poll()" >&5
 $as_echo_n "checking for broken poll()... " >&6; }
-if ${ac_cv_broken_poll+:} false; then :
+if test "${ac_cv_broken_poll+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13204,7 +13224,7 @@
 #include <$ac_cv_struct_tm>
 
 "
-if test "x$ac_cv_member_struct_tm_tm_zone" = xyes; then :
+if test "x$ac_cv_member_struct_tm_tm_zone" = x""yes; then :
 
 cat >>confdefs.h <<_ACEOF
 #define HAVE_STRUCT_TM_TM_ZONE 1
@@ -13220,7 +13240,7 @@
 else
   ac_fn_c_check_decl "$LINENO" "tzname" "ac_cv_have_decl_tzname" "#include <time.h>
 "
-if test "x$ac_cv_have_decl_tzname" = xyes; then :
+if test "x$ac_cv_have_decl_tzname" = x""yes; then :
   ac_have_decl=1
 else
   ac_have_decl=0
@@ -13232,7 +13252,7 @@
 
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tzname" >&5
 $as_echo_n "checking for tzname... " >&6; }
-if ${ac_cv_var_tzname+:} false; then :
+if test "${ac_cv_var_tzname+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13271,7 +13291,7 @@
 # check tzset(3) exists and works like we expect it to
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for working tzset()" >&5
 $as_echo_n "checking for working tzset()... " >&6; }
-if ${ac_cv_working_tzset+:} false; then :
+if test "${ac_cv_working_tzset+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
 
@@ -13368,7 +13388,7 @@
 # Look for subsecond timestamps in struct stat
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tv_nsec in struct stat" >&5
 $as_echo_n "checking for tv_nsec in struct stat... " >&6; }
-if ${ac_cv_stat_tv_nsec+:} false; then :
+if test "${ac_cv_stat_tv_nsec+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13405,7 +13425,7 @@
 # Look for BSD style subsecond timestamps in struct stat
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for tv_nsec2 in struct stat" >&5
 $as_echo_n "checking for tv_nsec2 in struct stat... " >&6; }
-if ${ac_cv_stat_tv_nsec2+:} false; then :
+if test "${ac_cv_stat_tv_nsec2+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13442,7 +13462,7 @@
 # On HP/UX 11.0, mvwdelch is a block with a return statement
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether mvwdelch is an expression" >&5
 $as_echo_n "checking whether mvwdelch is an expression... " >&6; }
-if ${ac_cv_mvwdelch_is_expression+:} false; then :
+if test "${ac_cv_mvwdelch_is_expression+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13479,7 +13499,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether WINDOW has _flags" >&5
 $as_echo_n "checking whether WINDOW has _flags... " >&6; }
-if ${ac_cv_window_has_flags+:} false; then :
+if test "${ac_cv_window_has_flags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   cat confdefs.h - <<_ACEOF >conftest.$ac_ext
@@ -13627,7 +13647,7 @@
 then
   { $as_echo "$as_me:${as_lineno-$LINENO}: checking for %lld and %llu printf() format support" >&5
 $as_echo_n "checking for %lld and %llu printf() format support... " >&6; }
-  if ${ac_cv_have_long_long_format+:} false; then :
+  if test "${ac_cv_have_long_long_format+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13697,7 +13717,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for %zd printf() format support" >&5
 $as_echo_n "checking for %zd printf() format support... " >&6; }
-if ${ac_cv_have_size_t_format+:} false; then :
+if test "${ac_cv_have_size_t_format+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13770,7 +13790,7 @@
 #endif
 
 "
-if test "x$ac_cv_type_socklen_t" = xyes; then :
+if test "x$ac_cv_type_socklen_t" = x""yes; then :
 
 else
 
@@ -13781,7 +13801,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken mbstowcs" >&5
 $as_echo_n "checking for broken mbstowcs... " >&6; }
-if ${ac_cv_broken_mbstowcs+:} false; then :
+if test "${ac_cv_broken_mbstowcs+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13821,7 +13841,7 @@
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking whether $CC supports computed gotos" >&5
 $as_echo_n "checking whether $CC supports computed gotos... " >&6; }
-if ${ac_cv_computed_gotos+:} false; then :
+if test "${ac_cv_computed_gotos+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
   if test "$cross_compiling" = yes; then :
@@ -13901,11 +13921,11 @@
 
 
 case $ac_sys_system in
-  OSF*) as_fn_error $? "OSF* systems are deprecated unless somebody volunteers. Check http://bugs.python.org/issue8606" "$LINENO" 5 ;;
+  OSF*) as_fn_error $? "OSF* systems are deprecated unless somebody volunteers. Check http://bugs.python.org/issue8606" "$LINENO" 5  ;;
 esac
 
 ac_fn_c_check_func "$LINENO" "pipe2" "ac_cv_func_pipe2"
-if test "x$ac_cv_func_pipe2" = xyes; then :
+if test "x$ac_cv_func_pipe2" = x""yes; then :
 
 $as_echo "#define HAVE_PIPE2 1" >>confdefs.h
 
@@ -14000,21 +14020,10 @@
      :end' >>confcache
 if diff "$cache_file" confcache >/dev/null 2>&1; then :; else
   if test -w "$cache_file"; then
-    if test "x$cache_file" != "x/dev/null"; then
+    test "x$cache_file" != "x/dev/null" &&
       { $as_echo "$as_me:${as_lineno-$LINENO}: updating cache $cache_file" >&5
 $as_echo "$as_me: updating cache $cache_file" >&6;}
-      if test ! -f "$cache_file" || test -h "$cache_file"; then
-	cat confcache >"$cache_file"
-      else
-        case $cache_file in #(
-        */* | ?:*)
-	  mv -f confcache "$cache_file"$$ &&
-	  mv -f "$cache_file"$$ "$cache_file" ;; #(
-        *)
-	  mv -f confcache "$cache_file" ;;
-	esac
-      fi
-    fi
+    cat confcache >$cache_file
   else
     { $as_echo "$as_me:${as_lineno-$LINENO}: not updating unwritable cache $cache_file" >&5
 $as_echo "$as_me: not updating unwritable cache $cache_file" >&6;}
@@ -14047,7 +14056,7 @@
 
 
 
-: "${CONFIG_STATUS=./config.status}"
+: ${CONFIG_STATUS=./config.status}
 ac_write_fail=0
 ac_clean_files_save=$ac_clean_files
 ac_clean_files="$ac_clean_files $CONFIG_STATUS"
@@ -14148,7 +14157,6 @@
 IFS=" ""	$as_nl"
 
 # Find who we are.  Look in the path if we contain no directory separator.
-as_myself=
 case $0 in #((
   *[\\/]* ) as_myself=$0 ;;
   *) as_save_IFS=$IFS; IFS=$PATH_SEPARATOR
@@ -14456,7 +14464,7 @@
 # values after options handling.
 ac_log="
 This file was extended by python $as_me 3.2, which was
-generated by GNU Autoconf 2.68.  Invocation command line was
+generated by GNU Autoconf 2.67.  Invocation command line was
 
   CONFIG_FILES    = $CONFIG_FILES
   CONFIG_HEADERS  = $CONFIG_HEADERS
@@ -14480,8 +14488,8 @@
 
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
 # Files that config.status was made for.
-config_files="`echo $ac_config_files`"
-config_headers="`echo $ac_config_headers`"
+config_files="$ac_config_files"
+config_headers="$ac_config_headers"
 
 _ACEOF
 
@@ -14518,7 +14526,7 @@
 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
 ac_cs_version="\\
 python config.status 3.2
-configured by $0, generated by GNU Autoconf 2.68,
+configured by $0, generated by GNU Autoconf 2.67,
   with options \\"\$ac_cs_config\\"
 
 Copyright (C) 2010 Free Software Foundation, Inc.
@@ -14649,7 +14657,7 @@
     "Misc/python.pc") CONFIG_FILES="$CONFIG_FILES Misc/python.pc" ;;
     "Modules/ld_so_aix") CONFIG_FILES="$CONFIG_FILES Modules/ld_so_aix" ;;
 
-  *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5;;
+  *) as_fn_error $? "invalid argument: \`$ac_config_target'" "$LINENO" 5 ;;
   esac
 done
 
@@ -14671,10 +14679,9 @@
 # after its creation but before its name has been assigned to `$tmp'.
 $debug ||
 {
-  tmp= ac_tmp=
+  tmp=
   trap 'exit_status=$?
-  : "${ac_tmp:=$tmp}"
-  { test ! -d "$ac_tmp" || rm -fr "$ac_tmp"; } && exit $exit_status
+  { test -z "$tmp" || test ! -d "$tmp" || rm -fr "$tmp"; } && exit $exit_status
 ' 0
   trap 'as_fn_exit 1' 1 2 13 15
 }
@@ -14682,13 +14689,12 @@
 
 {
   tmp=`(umask 077 && mktemp -d "./confXXXXXX") 2>/dev/null` &&
-  test -d "$tmp"
+  test -n "$tmp" && test -d "$tmp"
 }  ||
 {
   tmp=./conf$$-$RANDOM
   (umask 077 && mkdir "$tmp")
 } || as_fn_error $? "cannot create a temporary directory in ." "$LINENO" 5
-ac_tmp=$tmp
 
 # Set up the scripts for CONFIG_FILES section.
 # No need to generate them if there are no CONFIG_FILES.
@@ -14710,7 +14716,7 @@
   ac_cs_awk_cr=$ac_cr
 fi
 
-echo 'BEGIN {' >"$ac_tmp/subs1.awk" &&
+echo 'BEGIN {' >"$tmp/subs1.awk" &&
 _ACEOF
 
 
@@ -14738,7 +14744,7 @@
 rm -f conf$$subs.sh
 
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
-cat >>"\$ac_tmp/subs1.awk" <<\\_ACAWK &&
+cat >>"\$tmp/subs1.awk" <<\\_ACAWK &&
 _ACEOF
 sed -n '
 h
@@ -14786,7 +14792,7 @@
 rm -f conf$$subs.awk
 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
 _ACAWK
-cat >>"\$ac_tmp/subs1.awk" <<_ACAWK &&
+cat >>"\$tmp/subs1.awk" <<_ACAWK &&
   for (key in S) S_is_set[key] = 1
   FS = ""
 
@@ -14818,7 +14824,7 @@
   sed "s/$ac_cr\$//; s/$ac_cr/$ac_cs_awk_cr/g"
 else
   cat
-fi < "$ac_tmp/subs1.awk" > "$ac_tmp/subs.awk" \
+fi < "$tmp/subs1.awk" > "$tmp/subs.awk" \
   || as_fn_error $? "could not setup config files machinery" "$LINENO" 5
 _ACEOF
 
@@ -14852,7 +14858,7 @@
 # No need to generate them if there are no CONFIG_HEADERS.
 # This happens for instance with `./config.status Makefile'.
 if test -n "$CONFIG_HEADERS"; then
-cat >"$ac_tmp/defines.awk" <<\_ACAWK ||
+cat >"$tmp/defines.awk" <<\_ACAWK ||
 BEGIN {
 _ACEOF
 
@@ -14864,8 +14870,8 @@
 # handling of long lines.
 ac_delim='%!_!# '
 for ac_last_try in false false :; do
-  ac_tt=`sed -n "/$ac_delim/p" confdefs.h`
-  if test -z "$ac_tt"; then
+  ac_t=`sed -n "/$ac_delim/p" confdefs.h`
+  if test -z "$ac_t"; then
     break
   elif $ac_last_try; then
     as_fn_error $? "could not make $CONFIG_HEADERS" "$LINENO" 5
@@ -14966,7 +14972,7 @@
   esac
   case $ac_mode$ac_tag in
   :[FHL]*:*);;
-  :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5;;
+  :L* | :C*:*) as_fn_error $? "invalid tag \`$ac_tag'" "$LINENO" 5 ;;
   :[FH]-) ac_tag=-:-;;
   :[FH]*) ac_tag=$ac_tag:$ac_tag.in;;
   esac
@@ -14985,7 +14991,7 @@
     for ac_f
     do
       case $ac_f in
-      -) ac_f="$ac_tmp/stdin";;
+      -) ac_f="$tmp/stdin";;
       *) # Look for the file first in the build tree, then in the source tree
 	 # (if the path is not absolute).  The absolute path cannot be DOS-style,
 	 # because $ac_f cannot contain `:'.
@@ -14994,7 +15000,7 @@
 	   [\\/$]*) false;;
 	   *) test -f "$srcdir/$ac_f" && ac_f="$srcdir/$ac_f";;
 	   esac ||
-	   as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5;;
+	   as_fn_error 1 "cannot find input file: \`$ac_f'" "$LINENO" 5 ;;
       esac
       case $ac_f in *\'*) ac_f=`$as_echo "$ac_f" | sed "s/'/'\\\\\\\\''/g"`;; esac
       as_fn_append ac_file_inputs " '$ac_f'"
@@ -15020,8 +15026,8 @@
     esac
 
     case $ac_tag in
-    *:-:* | *:-) cat >"$ac_tmp/stdin" \
-      || as_fn_error $? "could not create $ac_file" "$LINENO" 5 ;;
+    *:-:* | *:-) cat >"$tmp/stdin" \
+      || as_fn_error $? "could not create $ac_file" "$LINENO" 5  ;;
     esac
     ;;
   esac
@@ -15151,22 +15157,21 @@
 s&@INSTALL@&$ac_INSTALL&;t t
 $ac_datarootdir_hack
 "
-eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$ac_tmp/subs.awk" \
-  >$ac_tmp/out || as_fn_error $? "could not create $ac_file" "$LINENO" 5
+eval sed \"\$ac_sed_extra\" "$ac_file_inputs" | $AWK -f "$tmp/subs.awk" >$tmp/out \
+  || as_fn_error $? "could not create $ac_file" "$LINENO" 5
 
 test -z "$ac_datarootdir_hack$ac_datarootdir_seen" &&
-  { ac_out=`sed -n '/\${datarootdir}/p' "$ac_tmp/out"`; test -n "$ac_out"; } &&
-  { ac_out=`sed -n '/^[	 ]*datarootdir[	 ]*:*=/p' \
-      "$ac_tmp/out"`; test -z "$ac_out"; } &&
+  { ac_out=`sed -n '/\${datarootdir}/p' "$tmp/out"`; test -n "$ac_out"; } &&
+  { ac_out=`sed -n '/^[	 ]*datarootdir[	 ]*:*=/p' "$tmp/out"`; test -z "$ac_out"; } &&
   { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: $ac_file contains a reference to the variable \`datarootdir'
 which seems to be undefined.  Please make sure it is defined" >&5
 $as_echo "$as_me: WARNING: $ac_file contains a reference to the variable \`datarootdir'
 which seems to be undefined.  Please make sure it is defined" >&2;}
 
-  rm -f "$ac_tmp/stdin"
+  rm -f "$tmp/stdin"
   case $ac_file in
-  -) cat "$ac_tmp/out" && rm -f "$ac_tmp/out";;
-  *) rm -f "$ac_file" && mv "$ac_tmp/out" "$ac_file";;
+  -) cat "$tmp/out" && rm -f "$tmp/out";;
+  *) rm -f "$ac_file" && mv "$tmp/out" "$ac_file";;
   esac \
   || as_fn_error $? "could not create $ac_file" "$LINENO" 5
  ;;
@@ -15177,20 +15182,20 @@
   if test x"$ac_file" != x-; then
     {
       $as_echo "/* $configure_input  */" \
-      && eval '$AWK -f "$ac_tmp/defines.awk"' "$ac_file_inputs"
-    } >"$ac_tmp/config.h" \
+      && eval '$AWK -f "$tmp/defines.awk"' "$ac_file_inputs"
+    } >"$tmp/config.h" \
       || as_fn_error $? "could not create $ac_file" "$LINENO" 5
-    if diff "$ac_file" "$ac_tmp/config.h" >/dev/null 2>&1; then
+    if diff "$ac_file" "$tmp/config.h" >/dev/null 2>&1; then
       { $as_echo "$as_me:${as_lineno-$LINENO}: $ac_file is unchanged" >&5
 $as_echo "$as_me: $ac_file is unchanged" >&6;}
     else
       rm -f "$ac_file"
-      mv "$ac_tmp/config.h" "$ac_file" \
+      mv "$tmp/config.h" "$ac_file" \
 	|| as_fn_error $? "could not create $ac_file" "$LINENO" 5
     fi
   else
     $as_echo "/* $configure_input  */" \
-      && eval '$AWK -f "$ac_tmp/defines.awk"' "$ac_file_inputs" \
+      && eval '$AWK -f "$tmp/defines.awk"' "$ac_file_inputs" \
       || as_fn_error $? "could not create -" "$LINENO" 5
   fi
  ;;
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -2696,6 +2696,15 @@
   [AC_MSG_RESULT(no)
 ])
 
+AC_MSG_CHECKING(for broken unsetenv)
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+#include <stdlib.h>
+]], [[int res = unsetenv("DUMMY")]])],
+  [AC_MSG_RESULT(no)],
+  [AC_DEFINE(HAVE_BROKEN_UNSETENV, 1, Define if `unsetenv` does not return an int.)
+   AC_MSG_RESULT(yes)
+])
+
 dnl check for true
 AC_CHECK_PROGS(TRUE, true, /bin/true)
 
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -98,6 +98,9 @@
 /* define to 1 if your sem_getvalue is broken. */
 #undef HAVE_BROKEN_SEM_GETVALUE
 
+/* Define if `unsetenv` does not return an int. */
+#undef HAVE_BROKEN_UNSETENV
+
 /* Define this if you have the type _Bool. */
 #undef HAVE_C99_BOOL
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 12:51:35 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sun, 27 Nov 2011 12:51:35 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Null_merge=2E?=
Message-ID: <E1RUdGh-0004Fb-2x@dinsdale.python.org>

http://hg.python.org/cpython/rev/656ccd7e8749
changeset:   73767:656ccd7e8749
parent:      73765:3a44640682c3
parent:      73766:26275a1c229c
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sun Nov 27 12:42:30 2011 +0100
summary:
  Null merge.

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 12:51:35 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sun, 27 Nov 2011 12:51:35 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDE1?=
 =?utf8?q?=3A_Skip_test=5Fos=2Etest=5Funset=5Ferror_on_FreeBSD_and_OS_X=2E?=
Message-ID: <E1RUdGh-0004Fg-T5@dinsdale.python.org>

http://hg.python.org/cpython/rev/bceb6aea8554
changeset:   73768:bceb6aea8554
branch:      3.2
parent:      73766:26275a1c229c
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sun Nov 27 12:49:27 2011 +0100
summary:
  Issue #13415: Skip test_os.test_unset_error on FreeBSD and OS X.

files:
  Lib/test/test_os.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -423,6 +423,10 @@
         value_str = value.decode(sys.getfilesystemencoding(), 'surrogateescape')
         self.assertEqual(os.environ['bytes'], value_str)
 
+    # On FreeBSD < 7 and OS X < 10.6, unsetenv() doesn't return a value (issue
+    # #13415).
+    @unittest.skipIf(sys.platform.startswith(('freebsd', 'darwin')),
+                     "due to known OS bug: see issue #13415")
     def test_unset_error(self):
         if sys.platform == "win32":
             # an environment variable is limited to 32,767 characters

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 12:51:37 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sun, 27 Nov 2011 12:51:37 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Null_merge=2E?=
Message-ID: <E1RUdGj-0004GC-52@dinsdale.python.org>

http://hg.python.org/cpython/rev/f4e3ae988a70
changeset:   73769:f4e3ae988a70
parent:      73767:656ccd7e8749
parent:      73768:bceb6aea8554
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sun Nov 27 12:50:15 2011 +0100
summary:
  Null merge.

files:



-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 13:05:50 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sun, 27 Nov 2011 13:05:50 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzNDE1?=
 =?utf8?q?=3A_Test_in_configure_if_unsetenv=28=29_has_a_return_value_or_no?=
 =?utf8?q?t=2E?=
Message-ID: <E1RUdUU-0005bi-6F@dinsdale.python.org>

http://hg.python.org/cpython/rev/11bbeaf03894
changeset:   73770:11bbeaf03894
branch:      2.7
parent:      73758:b23453530d5f
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sun Nov 27 13:01:35 2011 +0100
summary:
  Issue #13415: Test in configure if unsetenv() has a return value or not.

files:
  Modules/posixmodule.c |   6 ++++++
  configure             |  28 ++++++++++++++++++++++++++++
  configure.in          |   9 +++++++++
  pyconfig.h.in         |   3 +++
  4 files changed, 46 insertions(+), 0 deletions(-)


diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -7044,14 +7044,20 @@
 posix_unsetenv(PyObject *self, PyObject *args)
 {
     char *s1;
+#ifndef HAVE_BROKEN_UNSETENV
     int err;
+#endif
 
     if (!PyArg_ParseTuple(args, "s:unsetenv", &s1))
         return NULL;
 
+#ifdef HAVE_BROKEN_UNSETENV
+    unsetenv(s1);
+#else
     err = unsetenv(s1);
     if (err)
         return posix_error();
+#endif
 
     /* Remove the key from posix_putenv_garbage;
      * this will cause it to be collected.  This has to
diff --git a/configure b/configure
--- a/configure
+++ b/configure
@@ -9924,6 +9924,34 @@
 fi
 rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for broken unsetenv" >&5
+$as_echo_n "checking for broken unsetenv... " >&6; }
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+#include <stdlib.h>
+
+int
+main ()
+{
+int res = unsetenv("DUMMY")
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_compile "$LINENO"; then :
+  { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
+$as_echo "no" >&6; }
+else
+
+$as_echo "#define HAVE_BROKEN_UNSETENV 1" >>confdefs.h
+
+   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
+$as_echo "yes" >&6; }
+
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+
 for ac_prog in true
 do
   # Extract the first word of "$ac_prog", so it can be a program name with args.
diff --git a/configure.in b/configure.in
--- a/configure.in
+++ b/configure.in
@@ -2831,6 +2831,15 @@
   [AC_MSG_RESULT(no)
 ])
 
+AC_MSG_CHECKING(for broken unsetenv)
+AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[
+#include <stdlib.h>
+]], [[int res = unsetenv("DUMMY")]])],
+  [AC_MSG_RESULT(no)],
+  [AC_DEFINE(HAVE_BROKEN_UNSETENV, 1, Define if `unsetenv` does not return an int.)
+   AC_MSG_RESULT(yes)
+])
+
 dnl check for true
 AC_CHECK_PROGS(TRUE, true, /bin/true)
 
diff --git a/pyconfig.h.in b/pyconfig.h.in
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -97,6 +97,9 @@
 /* define to 1 if your sem_getvalue is broken. */
 #undef HAVE_BROKEN_SEM_GETVALUE
 
+/* Define if `unsetenv` does not return an int. */
+#undef HAVE_BROKEN_UNSETENV
+
 /* Define this if you have the type _Bool. */
 #undef HAVE_C99_BOOL
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 13:05:50 2011
From: python-checkins at python.org (charles-francois.natali)
Date: Sun, 27 Nov 2011 13:05:50 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzNDE1?=
 =?utf8?q?=3A_Skip_test=5Fos=2Etest=5Funset=5Ferror_on_FreeBSD_and_OS_X=2E?=
Message-ID: <E1RUdUU-0005bn-OL@dinsdale.python.org>

http://hg.python.org/cpython/rev/99f5a0475ead
changeset:   73771:99f5a0475ead
branch:      2.7
user:        Charles-Fran?ois Natali <neologix at free.fr>
date:        Sun Nov 27 13:05:14 2011 +0100
summary:
  Issue #13415: Skip test_os.test_unset_error on FreeBSD and OS X.

files:
  Lib/test/test_os.py |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -361,6 +361,10 @@
                 value = popen.read().strip()
                 self.assertEqual(value, "World")
 
+    # On FreeBSD < 7 and OS X < 10.6, unsetenv() doesn't return a value (issue
+    # #13415).
+    @unittest.skipIf(sys.platform.startswith(('freebsd', 'darwin')),
+                     "due to known OS bug: see issue #13415")
     def test_unset_error(self):
         if sys.platform == "win32":
             # an environment variable is limited to 32,767 characters

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Sun Nov 27 23:42:26 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Sun, 27 Nov 2011 23:42:26 +0100 (CET)
Subject: [Python-checkins] r88926 -
	tracker/instances/python-dev/extensions/openid_login.py
Message-ID: <3Ss5kB69WHzNQq@mail.python.org>

Author: martin.v.loewis
Date: Sun Nov 27 23:42:26 2011
New Revision: 88926

Log:
Catch failed discovery of builtin providers.


Modified:
   tracker/instances/python-dev/extensions/openid_login.py

Modified: tracker/instances/python-dev/extensions/openid_login.py
==============================================================================
--- tracker/instances/python-dev/extensions/openid_login.py	(original)
+++ tracker/instances/python-dev/extensions/openid_login.py	Sun Nov 27 23:42:26 2011
@@ -161,7 +161,10 @@
         # results. However, the risk of login breaking if a provider does change
         # its service URL outweighs the cost of another HTTP request to perform
         # the discovery during login.
-        services, op_endpoint, op_local = openid2rp.discover(provider_id)
+        result = openid2rp.discover(provider_id)
+        if result is None:
+            self.client.error_message.append('Provider %s appears to be down' % providers[provider][0])
+        services, op_endpoint, op_local = result
         session = self.get_session(op_endpoint, services)
         try:
             session = self.get_session(op_endpoint, services)

From python-checkins at python.org  Mon Nov 28 00:47:43 2011
From: python-checkins at python.org (martin.v.loewis)
Date: Mon, 28 Nov 2011 00:47:43 +0100 (CET)
Subject: [Python-checkins] r88927 -
	tracker/instances/python-dev/extensions/openid_login.py
Message-ID: <3Ss79W3F55zNPH@mail.python.org>

Author: martin.v.loewis
Date: Mon Nov 28 00:47:42 2011
New Revision: 88927

Log:
Fix minor bug.


Modified:
   tracker/instances/python-dev/extensions/openid_login.py

Modified: tracker/instances/python-dev/extensions/openid_login.py
==============================================================================
--- tracker/instances/python-dev/extensions/openid_login.py	(original)
+++ tracker/instances/python-dev/extensions/openid_login.py	Mon Nov 28 00:47:42 2011
@@ -164,6 +164,7 @@
         result = openid2rp.discover(provider_id)
         if result is None:
             self.client.error_message.append('Provider %s appears to be down' % providers[provider][0])
+            return
         services, op_endpoint, op_local = result
         session = self.get_session(op_endpoint, services)
         try:

From solipsis at pitrou.net  Mon Nov 28 05:32:33 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 28 Nov 2011 05:32:33 +0100
Subject: [Python-checkins] Daily reference leaks (f4e3ae988a70): sum=0
Message-ID: <E1RUstN-00072t-81@ap.vmr.nerim.net>

results for f4e3ae988a70 on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog_1kzUX', '-x']

From python-checkins at python.org  Mon Nov 28 07:23:52 2011
From: python-checkins at python.org (victor.stinner)
Date: Mon, 28 Nov 2011 07:23:52 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=2313415=3A_test=5Fcu?=
 =?utf8?q?rses_skips_unencodable_characters?=
Message-ID: <E1RUud6-0005sB-Tx@dinsdale.python.org>

http://hg.python.org/cpython/rev/bf51e32b2a81
changeset:   73772:bf51e32b2a81
parent:      73769:f4e3ae988a70
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Mon Nov 28 07:26:19 2011 +0100
summary:
  Issue #13415: test_curses skips unencodable characters

files:
  Lib/test/test_curses.py |  9 ++++++++-
  1 files changed, 8 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -267,11 +267,18 @@
 def test_unget_wch(stdscr):
     if not hasattr(curses, 'unget_wch'):
         return
+    import locale
+    encoding = locale.getpreferredencoding()
     for ch in ('a', '\xe9', '\u20ac', '\U0010FFFF'):
         try:
+            ch.encode(encoding)
+        except UnicodeEncodeError:
+            continue
+        try:
             curses.unget_wch(ch)
         except Exception as err:
-            raise Exception("unget_wch(%a) failed: %s" % (ch, err))
+            raise Exception("unget_wch(%a) failed with locale encoding %s: %s"
+                            % (ch, encoding, err))
         read = stdscr.get_wch()
         read = chr(read)
         if read != ch:

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 15:08:28 2011
From: python-checkins at python.org (eric.araujo)
Date: Mon, 28 Nov 2011 15:08:28 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Add_a_make_dependency_on_PEP_i?=
 =?utf8?q?ndex_generator_code_to_generated_file?=
Message-ID: <E1RV1si-0005Ix-O7@dinsdale.python.org>

http://hg.python.org/peps/rev/1b7fed04108c
changeset:   4004:1b7fed04108c
user:        ?ric Araujo <merwok at netwok.org>
date:        Mon Nov 28 15:07:42 2011 +0100
summary:
  Add a make dependency on PEP index generator code to generated file

files:
  Makefile |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -18,7 +18,7 @@
 
 $(TARGETS): pep2html.py
 
-pep-0000.txt: $(wildcard pep-????.txt)
+pep-0000.txt: $(wildcard pep-????.txt) $(wildcard pep0/*.py)
 	$(PYTHON) genpepindex.py .
 
 install:

-- 
Repository URL: http://hg.python.org/peps

From python-checkins at python.org  Mon Nov 28 16:39:17 2011
From: python-checkins at python.org (meador.inge)
Date: Mon, 28 Nov 2011 16:39:17 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEyNjE4?=
 =?utf8?q?=3A_py=5Fcompile_cannot_create_files_in_current_directory?=
Message-ID: <E1RV3Ib-0002ph-Cz@dinsdale.python.org>

http://hg.python.org/cpython/rev/661fb211f220
changeset:   73773:661fb211f220
branch:      3.2
parent:      73768:bceb6aea8554
user:        Meador Inge <meadori at gmail.com>
date:        Mon Nov 28 09:27:32 2011 -0600
summary:
  Issue #12618: py_compile cannot create files in current directory

Initial patch by Sjoerd de Vries.

files:
  Lib/py_compile.py           |  4 +++-
  Lib/test/test_py_compile.py |  9 +++++++++
  Misc/ACKS                   |  1 +
  Misc/NEWS                   |  3 +++
  4 files changed, 16 insertions(+), 1 deletions(-)


diff --git a/Lib/py_compile.py b/Lib/py_compile.py
--- a/Lib/py_compile.py
+++ b/Lib/py_compile.py
@@ -130,7 +130,9 @@
         else:
             cfile = imp.cache_from_source(file)
     try:
-        os.makedirs(os.path.dirname(cfile))
+        dirname = os.path.dirname(cfile)
+        if dirname:
+            os.makedirs(dirname)
     except OSError as error:
         if error.errno != errno.EEXIST:
             raise
diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
--- a/Lib/test/test_py_compile.py
+++ b/Lib/test/test_py_compile.py
@@ -39,6 +39,15 @@
         py_compile.compile(self.source_path)
         self.assertTrue(os.path.exists(self.cache_path))
 
+    def test_cwd(self):
+        cwd = os.getcwd()
+        os.chdir(self.directory)
+        py_compile.compile(os.path.basename(self.source_path),
+                           os.path.basename(self.pyc_path))
+        os.chdir(cwd)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+
     def test_relative_path(self):
         py_compile.compile(os.path.relpath(self.source_path),
                            os.path.relpath(self.pyc_path))
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -941,6 +941,7 @@
 Kurt Vile
 Norman Vine
 Frank Visser
+Sjoerd de Vries
 Niki W. Waibel
 Wojtek Walczak
 Charles Waldman
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -83,6 +83,9 @@
 Library
 -------
 
+- Issue #12618: Fix a bug that prevented py_compile from creating byte
+  compiled files in the current directory.  Initial patch by Sjoerd de Vries.
+
 - Issue #13444: When stdout has been closed explicitly, we should not attempt
   to flush it at shutdown and print an error.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 16:39:18 2011
From: python-checkins at python.org (meador.inge)
Date: Mon, 28 Nov 2011 16:39:18 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=2312618=3A_py=5Fcompile_cannot_create_files_in_curren?=
 =?utf8?q?t_directory?=
Message-ID: <E1RV3Ic-0002qE-5p@dinsdale.python.org>

http://hg.python.org/cpython/rev/e3647275f468
changeset:   73774:e3647275f468
parent:      73772:bf51e32b2a81
parent:      73773:661fb211f220
user:        Meador Inge <meadori at gmail.com>
date:        Mon Nov 28 09:34:47 2011 -0600
summary:
  Issue #12618: py_compile cannot create files in current directory

Initial patch by Sjoerd de Vries.

files:
  Lib/py_compile.py           |  4 +++-
  Lib/test/test_py_compile.py |  9 +++++++++
  Misc/ACKS                   |  1 +
  Misc/NEWS                   |  3 +++
  4 files changed, 16 insertions(+), 1 deletions(-)


diff --git a/Lib/py_compile.py b/Lib/py_compile.py
--- a/Lib/py_compile.py
+++ b/Lib/py_compile.py
@@ -130,7 +130,9 @@
         else:
             cfile = imp.cache_from_source(file)
     try:
-        os.makedirs(os.path.dirname(cfile))
+        dirname = os.path.dirname(cfile)
+        if dirname:
+            os.makedirs(dirname)
     except OSError as error:
         if error.errno != errno.EEXIST:
             raise
diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
--- a/Lib/test/test_py_compile.py
+++ b/Lib/test/test_py_compile.py
@@ -39,6 +39,15 @@
         py_compile.compile(self.source_path)
         self.assertTrue(os.path.exists(self.cache_path))
 
+    def test_cwd(self):
+        cwd = os.getcwd()
+        os.chdir(self.directory)
+        py_compile.compile(os.path.basename(self.source_path),
+                           os.path.basename(self.pyc_path))
+        os.chdir(cwd)
+        self.assertTrue(os.path.exists(self.pyc_path))
+        self.assertFalse(os.path.exists(self.cache_path))
+
     def test_relative_path(self):
         py_compile.compile(os.path.relpath(self.source_path),
                            os.path.relpath(self.pyc_path))
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -1014,6 +1014,7 @@
 Kurt Vile
 Norman Vine
 Frank Visser
+Sjoerd de Vries
 Niki W. Waibel
 Wojtek Walczak
 Charles Waldman
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -395,6 +395,9 @@
 Library
 -------
 
+- Issue #12618: Fix a bug that prevented py_compile from creating byte
+  compiled files in the current directory.  Initial patch by Sjoerd de Vries.
+
 - Issue #13444: When stdout has been closed explicitly, we should not attempt
   to flush it at shutdown and print an error.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 19:16:23 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 28 Nov 2011 19:16:23 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzcxMTE6?=
 =?utf8?q?_Python_can_now_be_run_without_a_stdin=2C_stdout_or_stderr_strea?=
 =?utf8?q?m=2E?=
Message-ID: <E1RV5kd-0002b9-9v@dinsdale.python.org>

http://hg.python.org/cpython/rev/f15943505db0
changeset:   73775:f15943505db0
branch:      3.2
parent:      73773:661fb211f220
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 28 19:08:36 2011 +0100
summary:
  Issue #7111: Python can now be run without a stdin, stdout or stderr stream.

It was already the case with Python 2.  However, the corresponding
sys module entries are now set to None (instead of an unusable file object).

files:
  Lib/test/test_cmd_line.py |  39 +++++++++++++++++++++++++++
  Misc/NEWS                 |   4 ++
  Python/pythonrun.c        |  31 +++++++++++----------
  3 files changed, 59 insertions(+), 15 deletions(-)


diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -291,6 +291,45 @@
         rc, out, err = assert_python_ok('-c', code)
         self.assertEqual(b'', err)
 
+    # Issue #7111: Python should work without standard streams
+
+    @unittest.skipIf(os.name != 'posix', "test needs POSIX semantics")
+    def _test_no_stdio(self, streams):
+        code = """if 1:
+            import os, sys
+            for i, s in enumerate({streams}):
+                if getattr(sys, s) is not None:
+                    os._exit(i + 1)
+            os._exit(42)""".format(streams=streams)
+        def preexec():
+            if 'stdin' in streams:
+                os.close(0)
+            if 'stdout' in streams:
+                os.close(1)
+            if 'stderr' in streams:
+                os.close(2)
+        p = subprocess.Popen(
+            [sys.executable, "-E", "-c", code],
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            preexec_fn=preexec)
+        out, err = p.communicate()
+        self.assertEqual(test.support.strip_python_stderr(err), b'')
+        self.assertEqual(p.returncode, 42)
+
+    def test_no_stdin(self):
+        self._test_no_stdio(['stdin'])
+
+    def test_no_stdout(self):
+        self._test_no_stdio(['stdout'])
+
+    def test_no_stderr(self):
+        self._test_no_stdio(['stderr'])
+
+    def test_no_std_streams(self):
+        self._test_no_stdio(['stdin', 'stdout', 'stderr'])
+
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #7111: Python can now be run without a stdin, stdout or stderr
+  stream.  It was already the case with Python 2.  However, the corresponding
+  sys module entries are now set to None (instead of an unusable file object).
+
 - Issue #13436: Fix a bogus error message when an AST object was passed
   an invalid integer value.
 
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -892,6 +892,19 @@
     return NULL;
 }
 
+static int
+is_valid_fd(int fd)
+{
+    int dummy_fd;
+    if (fd < 0 || !_PyVerify_fd(fd))
+        return 0;
+    dummy_fd = dup(fd);
+    if (dummy_fd < 0)
+        return 0;
+    close(dummy_fd);
+    return 1;
+}
+
 /* Initialize sys.stdin, stdout, stderr and builtins.open */
 static int
 initstdio(void)
@@ -951,13 +964,9 @@
      * and fileno() may point to an invalid file descriptor. For example
      * GUI apps don't have valid standard streams by default.
      */
-    if (fd < 0) {
-#ifdef MS_WINDOWS
+    if (!is_valid_fd(fd)) {
         std = Py_None;
         Py_INCREF(std);
-#else
-        goto error;
-#endif
     }
     else {
         std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors);
@@ -970,13 +979,9 @@
 
     /* Set sys.stdout */
     fd = fileno(stdout);
-    if (fd < 0) {
-#ifdef MS_WINDOWS
+    if (!is_valid_fd(fd)) {
         std = Py_None;
         Py_INCREF(std);
-#else
-        goto error;
-#endif
     }
     else {
         std = create_stdio(iomod, fd, 1, "<stdout>", encoding, errors);
@@ -990,13 +995,9 @@
 #if 1 /* Disable this if you have trouble debugging bootstrap stuff */
     /* Set sys.stderr, replaces the preliminary stderr */
     fd = fileno(stderr);
-    if (fd < 0) {
-#ifdef MS_WINDOWS
+    if (!is_valid_fd(fd)) {
         std = Py_None;
         Py_INCREF(std);
-#else
-        goto error;
-#endif
     }
     else {
         std = create_stdio(iomod, fd, 1, "<stderr>", encoding, "backslashreplace");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 19:16:24 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 28 Nov 2011 19:16:24 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=237111=3A_Python_can_now_be_run_without_a_stdin=2C_st?=
 =?utf8?q?dout_or_stderr_stream=2E?=
Message-ID: <E1RV5ke-0002bD-34@dinsdale.python.org>

http://hg.python.org/cpython/rev/c86fb10eaf68
changeset:   73776:c86fb10eaf68
parent:      73774:e3647275f468
parent:      73775:f15943505db0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 28 19:09:45 2011 +0100
summary:
  Issue #7111: Python can now be run without a stdin, stdout or stderr stream.

It was already the case with Python 2.  However, the corresponding
sys module entries are now set to None (instead of an unusable file object).

files:
  Lib/test/test_cmd_line.py |  39 +++++++++++++++++++++++++++
  Misc/NEWS                 |   4 ++
  Python/pythonrun.c        |  31 +++++++++++----------
  3 files changed, 59 insertions(+), 15 deletions(-)


diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -285,6 +285,45 @@
         rc, out, err = assert_python_ok('-c', code)
         self.assertEqual(b'', err)
 
+    # Issue #7111: Python should work without standard streams
+
+    @unittest.skipIf(os.name != 'posix', "test needs POSIX semantics")
+    def _test_no_stdio(self, streams):
+        code = """if 1:
+            import os, sys
+            for i, s in enumerate({streams}):
+                if getattr(sys, s) is not None:
+                    os._exit(i + 1)
+            os._exit(42)""".format(streams=streams)
+        def preexec():
+            if 'stdin' in streams:
+                os.close(0)
+            if 'stdout' in streams:
+                os.close(1)
+            if 'stderr' in streams:
+                os.close(2)
+        p = subprocess.Popen(
+            [sys.executable, "-E", "-c", code],
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            preexec_fn=preexec)
+        out, err = p.communicate()
+        self.assertEqual(test.support.strip_python_stderr(err), b'')
+        self.assertEqual(p.returncode, 42)
+
+    def test_no_stdin(self):
+        self._test_no_stdio(['stdin'])
+
+    def test_no_stdout(self):
+        self._test_no_stdio(['stdout'])
+
+    def test_no_stderr(self):
+        self._test_no_stdio(['stderr'])
+
+    def test_no_std_streams(self):
+        self._test_no_stdio(['stdin', 'stdout', 'stderr'])
+
 
 def test_main():
     test.support.run_unittest(CmdLineTest)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,10 @@
 Core and Builtins
 -----------------
 
+- Issue #7111: Python can now be run without a stdin, stdout or stderr
+  stream.  It was already the case with Python 2.  However, the corresponding
+  sys module entries are now set to None (instead of an unusable file object).
+
 - Issue #11849: Ensure that free()d memory arenas are really released
   on POSIX systems supporting anonymous memory mappings.  Patch by
   Charles-Fran?ois Natali.
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -912,6 +912,19 @@
     return NULL;
 }
 
+static int
+is_valid_fd(int fd)
+{
+    int dummy_fd;
+    if (fd < 0 || !_PyVerify_fd(fd))
+        return 0;
+    dummy_fd = dup(fd);
+    if (dummy_fd < 0)
+        return 0;
+    close(dummy_fd);
+    return 1;
+}
+
 /* Initialize sys.stdin, stdout, stderr and builtins.open */
 static int
 initstdio(void)
@@ -971,13 +984,9 @@
      * and fileno() may point to an invalid file descriptor. For example
      * GUI apps don't have valid standard streams by default.
      */
-    if (fd < 0) {
-#ifdef MS_WINDOWS
+    if (!is_valid_fd(fd)) {
         std = Py_None;
         Py_INCREF(std);
-#else
-        goto error;
-#endif
     }
     else {
         std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors);
@@ -990,13 +999,9 @@
 
     /* Set sys.stdout */
     fd = fileno(stdout);
-    if (fd < 0) {
-#ifdef MS_WINDOWS
+    if (!is_valid_fd(fd)) {
         std = Py_None;
         Py_INCREF(std);
-#else
-        goto error;
-#endif
     }
     else {
         std = create_stdio(iomod, fd, 1, "<stdout>", encoding, errors);
@@ -1010,13 +1015,9 @@
 #if 1 /* Disable this if you have trouble debugging bootstrap stuff */
     /* Set sys.stderr, replaces the preliminary stderr */
     fd = fileno(stderr);
-    if (fd < 0) {
-#ifdef MS_WINDOWS
+    if (!is_valid_fd(fd)) {
         std = Py_None;
         Py_INCREF(std);
-#else
-        goto error;
-#endif
     }
     else {
         std = create_stdio(iomod, fd, 1, "<stderr>", encoding, "backslashreplace");

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 22:00:46 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 28 Nov 2011 22:00:46 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogSXNzdWUgIzEzNDM0?=
 =?utf8?q?=3A_skip_the_time=2Exmlrpc=2Ecom_test=2C_and_reenable_the_buildb?=
 =?utf8?q?ot_test=2E?=
Message-ID: <E1RV8Ji-00041r-C3@dinsdale.python.org>

http://hg.python.org/cpython/rev/da148f0d86bd
changeset:   73777:da148f0d86bd
branch:      3.2
parent:      73775:f15943505db0
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 28 21:12:24 2011 +0100
summary:
  Issue #13434: skip the time.xmlrpc.com test, and reenable the buildbot test.

files:
  Lib/test/test_xmlrpc_net.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py
--- a/Lib/test/test_xmlrpc_net.py
+++ b/Lib/test/test_xmlrpc_net.py
@@ -14,6 +14,7 @@
     def test_current_time(self):
         # Get the current time from xmlrpc.com.  This code exercises
         # the minimal HTTP functionality in xmlrpclib.
+        self.skipTest("time.xmlrpc.com is unreliable")
         server = xmlrpclib.ServerProxy("http://time.xmlrpc.com/RPC2")
         try:
             t0 = server.currentTime.getCurrentTime()
@@ -38,7 +39,6 @@
     def test_python_builders(self):
         # Get the list of builders from the XMLRPC buildbot interface at
         # python.org.
-        self.skipTest("XMLRPC interface removed in Buildbot 0.8.2")
         server = xmlrpclib.ServerProxy("http://www.python.org/dev/buildbot/all/xmlrpc/")
         try:
             builders = server.getAllBuilders()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 22:00:47 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 28 Nov 2011 22:00:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_resource_wa?=
 =?utf8?q?rning_in_test=5Fxmlrpc=5Fnet?=
Message-ID: <E1RV8Jj-00041v-3A@dinsdale.python.org>

http://hg.python.org/cpython/rev/d2e5bc1accfb
changeset:   73778:d2e5bc1accfb
branch:      3.2
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 28 21:14:46 2011 +0100
summary:
  Fix resource warning in test_xmlrpc_net

files:
  Lib/test/test_xmlrpc_net.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py
--- a/Lib/test/test_xmlrpc_net.py
+++ b/Lib/test/test_xmlrpc_net.py
@@ -45,6 +45,7 @@
         except socket.error as e:
             self.skipTest("network error: %s" % e)
             return
+        self.addCleanup(lambda: server('close')())
 
         # Perform a minimal sanity check on the result, just to be sure
         # the request means what we think it means.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Mon Nov 28 22:00:47 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 28 Nov 2011 22:00:47 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Merge_test=5Fxmlrpc=5Fnet_fixes_=28issue_=2313434=29?=
Message-ID: <E1RV8Jj-000420-Qf@dinsdale.python.org>

http://hg.python.org/cpython/rev/1330beac9ec9
changeset:   73779:1330beac9ec9
parent:      73776:c86fb10eaf68
parent:      73778:d2e5bc1accfb
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Nov 28 21:15:22 2011 +0100
summary:
  Merge test_xmlrpc_net fixes (issue #13434)

files:
  Lib/test/test_xmlrpc_net.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_xmlrpc_net.py b/Lib/test/test_xmlrpc_net.py
--- a/Lib/test/test_xmlrpc_net.py
+++ b/Lib/test/test_xmlrpc_net.py
@@ -14,6 +14,7 @@
     def test_current_time(self):
         # Get the current time from xmlrpc.com.  This code exercises
         # the minimal HTTP functionality in xmlrpclib.
+        self.skipTest("time.xmlrpc.com is unreliable")
         server = xmlrpclib.ServerProxy("http://time.xmlrpc.com/RPC2")
         try:
             t0 = server.currentTime.getCurrentTime()
@@ -38,13 +39,13 @@
     def test_python_builders(self):
         # Get the list of builders from the XMLRPC buildbot interface at
         # python.org.
-        self.skipTest("XMLRPC interface removed in Buildbot 0.8.2")
         server = xmlrpclib.ServerProxy("http://www.python.org/dev/buildbot/all/xmlrpc/")
         try:
             builders = server.getAllBuilders()
         except socket.error as e:
             self.skipTest("network error: %s" % e)
             return
+        self.addCleanup(lambda: server('close')())
 
         # Perform a minimal sanity check on the result, just to be sure
         # the request means what we think it means.

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 00:02:06 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 29 Nov 2011 00:02:06 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_curses_modu?=
 =?utf8?q?le_for_strict_SysV_implementation_=28without_has=5Fkey_function?=
 =?utf8?q?=29?=
Message-ID: <E1RVAD8-0007Fh-5J@dinsdale.python.org>

http://hg.python.org/cpython/rev/aba5cce30d71
changeset:   73780:aba5cce30d71
branch:      3.2
parent:      73778:d2e5bc1accfb
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 29 00:03:50 2011 +0100
summary:
  Fix curses module for strict SysV implementation (without has_key function)

files:
  Lib/curses/__init__.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/curses/__init__.py b/Lib/curses/__init__.py
--- a/Lib/curses/__init__.py
+++ b/Lib/curses/__init__.py
@@ -54,4 +54,4 @@
 try:
     has_key
 except NameError:
-    from has_key import has_key
+    from .has_key import has_key

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 00:03:43 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 29 Nov 2011 00:03:43 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_=28Merge_3=2E2=29_Fix_curses_module_for_strict_SysV_implemen?=
 =?utf8?q?tation_=28without_has=5Fkey?=
Message-ID: <E1RVAEh-0007Lf-E2@dinsdale.python.org>

http://hg.python.org/cpython/rev/bff927c8b410
changeset:   73781:bff927c8b410
parent:      73779:1330beac9ec9
parent:      73780:aba5cce30d71
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 29 00:06:10 2011 +0100
summary:
  (Merge 3.2) Fix curses module for strict SysV implementation (without has_key function)

files:
  Lib/curses/__init__.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/curses/__init__.py b/Lib/curses/__init__.py
--- a/Lib/curses/__init__.py
+++ b/Lib/curses/__init__.py
@@ -53,7 +53,7 @@
 try:
     has_key
 except NameError:
-    from has_key import has_key
+    from .has_key import has_key
 
 # Wrapper for the entire curses-based application.  Runs a function which
 # should be the rest of your curses-based application.  If the application

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 00:06:41 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 29 Nov 2011 00:06:41 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_cleanup_setup=2Epy_for_curs?=
	=?utf8?q?es_options?=
Message-ID: <E1RVAHZ-0007dx-Q4@dinsdale.python.org>

http://hg.python.org/cpython/rev/5d42f22189be
changeset:   73782:5d42f22189be
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 29 00:08:12 2011 +0100
summary:
  cleanup setup.py for curses options

files:
  setup.py |  17 ++++++++---------
  1 files changed, 8 insertions(+), 9 deletions(-)


diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1166,20 +1166,19 @@
         else:
             missing.extend(['nis', 'resource', 'termios'])
 
+        # Curses support, requiring the System V version of curses, often
+        # provided by the ncurses library.
         curses_defines = []
+        curses_includes = []
+        panel_library = 'panel'
         if curses_library == 'ncursesw':
             curses_defines.append(('HAVE_NCURSESW', '1'))
+            curses_includes.append('/usr/include/ncursesw')
+            # Bug 1464056: If _curses.so links with ncursesw,
+            # _curses_panel.so must link with panelw.
+            panel_library = 'panelw'
 
-        # Curses support, requiring the System V version of curses, often
-        # provided by the ncurses library.
-        panel_library = 'panel'
-        curses_includes = []
         if curses_library.startswith('ncurses'):
-            if curses_library == 'ncursesw':
-                # Bug 1464056: If _curses.so links with ncursesw,
-                # _curses_panel.so must link with panelw.
-                panel_library = 'panelw'
-                curses_includes = ['/usr/include/ncursesw']
             curses_libs = [curses_library]
             exts.append( Extension('_curses', ['_cursesmodule.c'],
                                    include_dirs=curses_includes,

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 00:50:42 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 29 Nov 2011 00:50:42 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzEzMDkz?=
 =?utf8?q?=3A_Fix_=5Ftestcapi=2Eunicode=5Fencodedecimal=28=29?=
Message-ID: <E1RVAyA-0002v7-S6@dinsdale.python.org>

http://hg.python.org/cpython/rev/3ecddf168f1f
changeset:   73783:3ecddf168f1f
branch:      2.7
parent:      73771:99f5a0475ead
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 29 00:53:09 2011 +0100
summary:
  Issue #13093: Fix _testcapi.unicode_encodedecimal()

_testcapimodule.c is not "ssize_t" safe in Python 2.7: the length argument type
is int, not Py_ssize_t.

files:
  Modules/_testcapimodule.c |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -1109,7 +1109,7 @@
 unicode_encodedecimal(PyObject *self, PyObject *args)
 {
     Py_UNICODE *unicode;
-    Py_ssize_t length;
+    int length;
     char *errors = NULL;
     PyObject *decimal;
     Py_ssize_t decimal_length, new_length;

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 04:50:52 2011
From: python-checkins at python.org (ezio.melotti)
Date: Tue, 29 Nov 2011 04:50:52 +0100 (CET)
Subject: [Python-checkins] r88928 - in
	tracker/instances/python-dev/extensions: local_replace.py
	test/local_replace_data.txt
Message-ID: <3SsrWc4FZPzM9K@mail.python.org>

Author: ezio.melotti
Date: Tue Nov 29 04:50:52 2011
New Revision: 88928

Log:
Make "rev. 12345" work too.

Modified:
   tracker/instances/python-dev/extensions/local_replace.py
   tracker/instances/python-dev/extensions/test/local_replace_data.txt

Modified: tracker/instances/python-dev/extensions/local_replace.py
==============================================================================
--- tracker/instances/python-dev/extensions/local_replace.py	(original)
+++ tracker/instances/python-dev/extensions/local_replace.py	Tue Nov 29 04:50:52 2011
@@ -79,8 +79,8 @@
     (re.compile(r'\b(?<![/?&;=])(?P<revision>[a-fA-F0-9]{12})\b'),
      r'<a href="http://hg.python.org/lookup/\g<revision>">\g<revision></a>'),
 
-    # r12345, r 12345, rev12345, rev 12345, revision12345, revision 12345
-    (re.compile(r'\b(?<![/?&;])(?P<revstr>r(ev(ision)?)?\s*)(?P<revision>\d{4,})'),
+    # r12345, r 12345, rev12345, rev. 12345, revision12345, revision 12345
+    (re.compile(r'\b(?<![/?&;])(?P<revstr>r\.?(ev\.?(ision)?)?\s*)(?P<revision>\d{4,})'),
      r'<a href="http://hg.python.org/lookup/r\g<revision>">\g<revstr>\g<revision></a>'),
 
     # Lib/somefile.py, Lib/somefile.py:123, Modules/somemodule.c:123, ...

Modified: tracker/instances/python-dev/extensions/test/local_replace_data.txt
==============================================================================
--- tracker/instances/python-dev/extensions/test/local_replace_data.txt	(original)
+++ tracker/instances/python-dev/extensions/test/local_replace_data.txt	Tue Nov 29 04:50:52 2011
@@ -8,10 +8,14 @@
  <a href="http://hg.python.org/lookup/r2222">r2222</a>
  r 2222
  <a href="http://hg.python.org/lookup/r2222">r 2222</a>
+ r. 2222
+ <a href="http://hg.python.org/lookup/r2222">r. 2222</a>
  rev2222
  <a href="http://hg.python.org/lookup/r2222">rev2222</a>
  rev  2222
  <a href="http://hg.python.org/lookup/r2222">rev  2222</a>
+ rev.  2222
+ <a href="http://hg.python.org/lookup/r2222">rev.  2222</a>
  revision2222
  <a href="http://hg.python.org/lookup/r2222">revision2222</a>
  revision 2222
@@ -62,6 +66,8 @@
  #1234567890123
 pyissue1000
 pyissue1000
+http://mail.python.org/pipermail/python-ideas/2011-October/thread.html#12459
+<a href="http://mail.python.org/pipermail/python-ideas/2011-October/thread.html#12459">http://mail.python.org/pipermail/python-ideas/2011-October/thread.html#12459</a>
 ##
 ## files - Lib/somefile.py, Modules/somemodule.c, Doc/somedocfile.rst, ...
 Lib/cgi.py

From solipsis at pitrou.net  Tue Nov 29 05:33:13 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 29 Nov 2011 05:33:13 +0100
Subject: [Python-checkins] Daily reference leaks (5d42f22189be): sum=0
Message-ID: <E1RVFNZ-0007JY-9r@ap.vmr.nerim.net>

results for 5d42f22189be on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogdwrSqD', '-x']

From python-checkins at python.org  Tue Nov 29 17:15:08 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:08 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Add_a_regrtest_?=
 =?utf8?q?cleanup_check_for_shutil_registries?=
Message-ID: <E1RVQKq-0001Lr-Pe@dinsdale.python.org>

http://hg.python.org/cpython/rev/7292a6c21fe5
changeset:   73786:7292a6c21fe5
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 16:45:07 2011 +0100
summary:
  Add a regrtest cleanup check for shutil registries

files:
  Lib/test/regrtest.py |  19 +++++++++++++++++++
  1 files changed, 19 insertions(+), 0 deletions(-)


diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -165,6 +165,7 @@
 import platform
 import random
 import re
+import shutil
 import sys
 import sysconfig
 import tempfile
@@ -887,6 +888,7 @@
                  'os.environ', 'sys.path', 'sys.path_hooks', '__import__',
                  'warnings.filters', 'asyncore.socket_map',
                  'logging._handlers', 'logging._handlerList',
+                 'shutil.archive_formats', 'shutil.unpack_formats',
                  'sys.warnoptions', 'threading._dangling',
                  'multiprocessing.process._dangling')
 
@@ -956,6 +958,23 @@
             asyncore.close_all(ignore_all=True)
             asyncore.socket_map.update(saved_map)
 
+    def get_shutil_archive_formats(self):
+        # we could call get_archives_formats() but that only returns the
+        # registry keys; we want to check the values too (the functions that
+        # are registered)
+        return shutil._ARCHIVE_FORMATS, shutil._ARCHIVE_FORMATS.copy()
+    def restore_shutil_archive_formats(self, saved):
+        shutil._ARCHIVE_FORMATS = saved[0]
+        shutil._ARCHIVE_FORMATS.clear()
+        shutil._ARCHIVE_FORMATS.update(saved[1])
+
+    def get_shutil_unpack_formats(self):
+        return shutil._UNPACK_FORMATS, shutil._UNPACK_FORMATS.copy()
+    def restore_shutil_unpack_formats(self, saved):
+        shutil._UNPACK_FORMATS = saved[0]
+        shutil._UNPACK_FORMATS.clear()
+        shutil._UNPACK_FORMATS.update(saved[1])
+
     def get_logging__handlers(self):
         # _handlers is a WeakValueDictionary
         return id(logging._handlers), logging._handlers, logging._handlers.copy()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 17:15:10 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Mark_up_constan?=
	=?utf8?q?ts_in_sockets_docs_as_such?=
Message-ID: <E1RVQKs-0001Lz-Lw@dinsdale.python.org>

http://hg.python.org/cpython/rev/5641a7c44f2b
changeset:   73788:5641a7c44f2b
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 16:46:38 2011 +0100
summary:
  Mark up constants in sockets docs as such

files:
  Doc/library/socket.rst |  16 ++++++++--------
  1 files changed, 8 insertions(+), 8 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -64,20 +64,20 @@
   tuple, and the fields depend on the address type. The general tuple form is
   ``(addr_type, v1, v2, v3 [, scope])``, where:
 
-  - *addr_type* is one of TIPC_ADDR_NAMESEQ, TIPC_ADDR_NAME, or
-    TIPC_ADDR_ID.
-  - *scope* is one of TIPC_ZONE_SCOPE, TIPC_CLUSTER_SCOPE, and
-    TIPC_NODE_SCOPE.
-  - If *addr_type* is TIPC_ADDR_NAME, then *v1* is the server type, *v2* is
+  - *addr_type* is one of :const:`TIPC_ADDR_NAMESEQ`, :const:`TIPC_ADDR_NAME`,
+    or :const:`TIPC_ADDR_ID`.
+  - *scope* is one of :const:`TIPC_ZONE_SCOPE`, :const:`TIPC_CLUSTER_SCOPE`, and
+    :const:`TIPC_NODE_SCOPE`.
+  - If *addr_type* is :const:`TIPC_ADDR_NAME`, then *v1* is the server type, *v2* is
     the port identifier, and *v3* should be 0.
 
-    If *addr_type* is TIPC_ADDR_NAMESEQ, then *v1* is the server type, *v2*
+    If *addr_type* is :const:`TIPC_ADDR_NAMESEQ`, then *v1* is the server type, *v2*
     is the lower port number, and *v3* is the upper port number.
 
-    If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the
+    If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the
     reference, and *v3* should be set to 0.
 
-    If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the
+    If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the
     reference, and *v3* should be set to 0.
 
 - Certain other address families (:const:`AF_BLUETOOTH`, :const:`AF_PACKET`)

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 17:15:06 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:06 +0100
Subject: [Python-checkins] =?utf8?b?Y3B5dGhvbiAoMy4yKTogRml4IHR5cG8gKCMx?=
	=?utf8?q?3467=29?=
Message-ID: <E1RVQKo-0001KT-1j@dinsdale.python.org>

http://hg.python.org/cpython/rev/a7d38a969132
changeset:   73784:a7d38a969132
branch:      3.2
parent:      73780:aba5cce30d71
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 16:34:58 2011 +0100
summary:
  Fix typo (#13467)

files:
  Doc/library/sysconfig.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst
--- a/Doc/library/sysconfig.rst
+++ b/Doc/library/sysconfig.rst
@@ -130,7 +130,7 @@
    one may call this function and get the default value.
 
    If *scheme* is provided, it must be a value from the list returned by
-   :func:`get_path_names`.  Otherwise, the default scheme for the current
+   :func:`get_scheme_names`.  Otherwise, the default scheme for the current
    platform is used.
 
    If *vars* is provided, it must be a dictionary of variables that will update

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 17:15:11 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_inspect=2Eg?=
 =?utf8?q?etmodule_to_use_a_copy_of_sys=2Emodules_for_iteration_=28=231348?=
 =?utf8?b?Nyku?=
Message-ID: <E1RVQKt-0001M4-HL@dinsdale.python.org>

http://hg.python.org/cpython/rev/2ef359d7a2e9
changeset:   73789:2ef359d7a2e9
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 16:58:53 2011 +0100
summary:
  Fix inspect.getmodule to use a copy of sys.modules for iteration (#13487).

This fixes a regression compared to 2.x, where sys.modules.items()
returns a copy, as indicated by a comment in the source.  Diagnosis and
patch by Erik Tollerud.

files:
  Lib/inspect.py |  2 +-
  Misc/ACKS      |  1 +
  Misc/NEWS      |  3 +++
  3 files changed, 5 insertions(+), 1 deletions(-)


diff --git a/Lib/inspect.py b/Lib/inspect.py
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -483,7 +483,7 @@
         return sys.modules.get(modulesbyfile[file])
     # Update the filename to module name cache and check yet again
     # Copy sys.modules in order to cope with changes while iterating
-    for modname, module in sys.modules.items():
+    for modname, module in list(sys.modules.items()):
         if ismodule(module) and hasattr(module, '__file__'):
             f = module.__file__
             if f == _filesbymodname.get(modname, None):
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -905,6 +905,7 @@
 Frank J. Tobin
 R Lindsay Todd
 Bennett Todd
+Erik Tollerud
 Matias Torchinsky
 Sandro Tosi
 Richard Townsend
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -87,6 +87,9 @@
 Library
 -------
 
+- Issue #13487: Make inspect.getmodule robust against changes done to
+  sys.modules while it is iterating over it.
+
 - Issue #12618: Fix a bug that prevented py_compile from creating byte
   compiled files in the current directory.  Initial patch by Sjoerd de Vries.
 

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 17:15:07 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:07 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_last_remain?=
 =?utf8?q?ing_references_to_ex-devguide?=
Message-ID: <E1RVQKp-0001L4-D8@dinsdale.python.org>

http://hg.python.org/cpython/rev/49d69219fa59
changeset:   73785:49d69219fa59
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 16:36:17 2011 +0100
summary:
  Fix last remaining references to ex-devguide

files:
  Lib/locale.py   |  3 +--
  Misc/python.man |  2 +-
  2 files changed, 2 insertions(+), 3 deletions(-)


diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -1598,8 +1598,7 @@
 # to include every locale up to Windows Vista.
 #
 # NOTE: this mapping is incomplete.  If your language is missing, please
-# submit a bug report to Python bug manager, which you can find via:
-#     http://www.python.org/dev/
+# submit a bug report to the Python bug tracker at http://bugs.python.org/
 # Make sure you include the missing language identifier and the suggested
 # locale code.
 #
diff --git a/Misc/python.man b/Misc/python.man
--- a/Misc/python.man
+++ b/Misc/python.man
@@ -420,7 +420,7 @@
 .br
 Documentation:  http://docs.python.org/py3k/
 .br
-Developer resources:  http://www.python.org/dev/
+Developer resources:  http://docs.python.org/devguide/
 .br
 Downloads:  http://python.org/download/
 .br

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 17:15:12 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:12 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
	=?utf8?q?=3A_Merge_3=2E2?=
Message-ID: <E1RVQKu-0001M8-Eh@dinsdale.python.org>

http://hg.python.org/cpython/rev/599763461943
changeset:   73790:599763461943
parent:      73782:5d42f22189be
parent:      73789:2ef359d7a2e9
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 17:14:27 2011 +0100
summary:
  Merge 3.2

files:
  Doc/library/socket.rst    |  16 ++++++++--------
  Doc/library/sysconfig.rst |   2 +-
  Lib/inspect.py            |   2 +-
  Lib/keyword.py            |   2 +-
  Lib/locale.py             |   3 +--
  Lib/symbol.py             |   2 +-
  Lib/test/regrtest.py      |  18 ++++++++++++++++++
  Lib/token.py              |   2 +-
  Misc/ACKS                 |   1 +
  Misc/NEWS                 |   3 +++
  Misc/python.man           |   2 +-
  11 files changed, 37 insertions(+), 16 deletions(-)


diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -64,20 +64,20 @@
   tuple, and the fields depend on the address type. The general tuple form is
   ``(addr_type, v1, v2, v3 [, scope])``, where:
 
-  - *addr_type* is one of TIPC_ADDR_NAMESEQ, TIPC_ADDR_NAME, or
-    TIPC_ADDR_ID.
-  - *scope* is one of TIPC_ZONE_SCOPE, TIPC_CLUSTER_SCOPE, and
-    TIPC_NODE_SCOPE.
-  - If *addr_type* is TIPC_ADDR_NAME, then *v1* is the server type, *v2* is
+  - *addr_type* is one of :const:`TIPC_ADDR_NAMESEQ`, :const:`TIPC_ADDR_NAME`,
+    or :const:`TIPC_ADDR_ID`.
+  - *scope* is one of :const:`TIPC_ZONE_SCOPE`, :const:`TIPC_CLUSTER_SCOPE`, and
+    :const:`TIPC_NODE_SCOPE`.
+  - If *addr_type* is :const:`TIPC_ADDR_NAME`, then *v1* is the server type, *v2* is
     the port identifier, and *v3* should be 0.
 
-    If *addr_type* is TIPC_ADDR_NAMESEQ, then *v1* is the server type, *v2*
+    If *addr_type* is :const:`TIPC_ADDR_NAMESEQ`, then *v1* is the server type, *v2*
     is the lower port number, and *v3* is the upper port number.
 
-    If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the
+    If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the
     reference, and *v3* should be set to 0.
 
-    If *addr_type* is TIPC_ADDR_ID, then *v1* is the node, *v2* is the
+    If *addr_type* is :const:`TIPC_ADDR_ID`, then *v1* is the node, *v2* is the
     reference, and *v3* should be set to 0.
 
 - A tuple ``(interface, )`` is used for the :const:`AF_CAN` address family,
diff --git a/Doc/library/sysconfig.rst b/Doc/library/sysconfig.rst
--- a/Doc/library/sysconfig.rst
+++ b/Doc/library/sysconfig.rst
@@ -130,7 +130,7 @@
    one may call this function and get the default value.
 
    If *scheme* is provided, it must be a value from the list returned by
-   :func:`get_path_names`.  Otherwise, the default scheme for the current
+   :func:`get_scheme_names`.  Otherwise, the default scheme for the current
    platform is used.
 
    If *vars* is provided, it must be a dictionary of variables that will update
diff --git a/Lib/inspect.py b/Lib/inspect.py
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -482,7 +482,7 @@
         return sys.modules.get(modulesbyfile[file])
     # Update the filename to module name cache and check yet again
     # Copy sys.modules in order to cope with changes while iterating
-    for modname, module in sys.modules.items():
+    for modname, module in list(sys.modules.items()):
         if ismodule(module) and hasattr(module, '__file__'):
             f = module.__file__
             if f == _filesbymodname.get(modname, None):
diff --git a/Lib/keyword.py b/Lib/keyword.py
--- a/Lib/keyword.py
+++ b/Lib/keyword.py
@@ -7,7 +7,7 @@
 To update the symbols in this file, 'cd' to the top directory of
 the python source tree after building the interpreter and run:
 
-    python Lib/keyword.py
+    ./python Lib/keyword.py
 """
 
 __all__ = ["iskeyword", "kwlist"]
diff --git a/Lib/locale.py b/Lib/locale.py
--- a/Lib/locale.py
+++ b/Lib/locale.py
@@ -1598,8 +1598,7 @@
 # to include every locale up to Windows Vista.
 #
 # NOTE: this mapping is incomplete.  If your language is missing, please
-# submit a bug report to Python bug manager, which you can find via:
-#     http://www.python.org/dev/
+# submit a bug report to the Python bug tracker at http://bugs.python.org/
 # Make sure you include the missing language identifier and the suggested
 # locale code.
 #
diff --git a/Lib/symbol.py b/Lib/symbol.py
--- a/Lib/symbol.py
+++ b/Lib/symbol.py
@@ -7,7 +7,7 @@
 #  To update the symbols in this file, 'cd' to the top directory of
 #  the python source tree after building the interpreter and run:
 #
-#    python Lib/symbol.py
+#    ./python Lib/symbol.py
 
 #--start constants--
 single_input = 256
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -177,6 +177,7 @@
 import platform
 import random
 import re
+import shutil
 import signal
 import sys
 import sysconfig
@@ -1042,6 +1043,23 @@
             asyncore.close_all(ignore_all=True)
             asyncore.socket_map.update(saved_map)
 
+    def get_shutil_archive_formats(self):
+        # we could call get_archives_formats() but that only returns the
+        # registry keys; we want to check the values too (the functions that
+        # are registered)
+        return shutil._ARCHIVE_FORMATS, shutil._ARCHIVE_FORMATS.copy()
+    def restore_shutil_archive_formats(self, saved):
+        shutil._ARCHIVE_FORMATS = saved[0]
+        shutil._ARCHIVE_FORMATS.clear()
+        shutil._ARCHIVE_FORMATS.update(saved[1])
+
+    def get_shutil_unpack_formats(self):
+        return shutil._UNPACK_FORMATS, shutil._UNPACK_FORMATS.copy()
+    def restore_shutil_unpack_formats(self, saved):
+        shutil._UNPACK_FORMATS = saved[0]
+        shutil._UNPACK_FORMATS.clear()
+        shutil._UNPACK_FORMATS.update(saved[1])
+
     def get_logging__handlers(self):
         # _handlers is a WeakValueDictionary
         return id(logging._handlers), logging._handlers, logging._handlers.copy()
diff --git a/Lib/token.py b/Lib/token.py
--- a/Lib/token.py
+++ b/Lib/token.py
@@ -7,7 +7,7 @@
 #  To update the symbols in this file, 'cd' to the top directory of
 #  the python source tree after building the interpreter and run:
 #
-#    python Lib/token.py
+#    ./python Lib/token.py
 
 #--start constants--
 ENDMARKER = 0
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -978,6 +978,7 @@
 Bennett Todd
 R Lindsay Todd
 Eugene Toder
+Erik Tollerud
 Matias Torchinsky
 Sandro Tosi
 Richard Townsend
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -399,6 +399,9 @@
 Library
 -------
 
+- Issue #13487: Make inspect.getmodule robust against changes done to
+  sys.modules while it is iterating over it.
+
 - Issue #12618: Fix a bug that prevented py_compile from creating byte
   compiled files in the current directory.  Initial patch by Sjoerd de Vries.
 
diff --git a/Misc/python.man b/Misc/python.man
--- a/Misc/python.man
+++ b/Misc/python.man
@@ -409,7 +409,7 @@
 .br
 Documentation:  http://docs.python.org/py3k/
 .br
-Developer resources:  http://www.python.org/dev/
+Developer resources:  http://docs.python.org/devguide/
 .br
 Downloads:  http://python.org/download/
 .br

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 17:15:09 2011
From: python-checkins at python.org (eric.araujo)
Date: Tue, 29 Nov 2011 17:15:09 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Fix_instruction?=
 =?utf8?q?s_on_how_to_rebuild_some_modules?=
Message-ID: <E1RVQKr-0001Lw-QC@dinsdale.python.org>

http://hg.python.org/cpython/rev/0f7253d6b1cc
changeset:   73787:0f7253d6b1cc
branch:      3.2
user:        ?ric Araujo <merwok at netwok.org>
date:        Tue Nov 29 16:45:34 2011 +0100
summary:
  Fix instructions on how to rebuild some modules

files:
  Lib/keyword.py |  2 +-
  Lib/symbol.py  |  2 +-
  Lib/token.py   |  2 +-
  3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Lib/keyword.py b/Lib/keyword.py
--- a/Lib/keyword.py
+++ b/Lib/keyword.py
@@ -7,7 +7,7 @@
 To update the symbols in this file, 'cd' to the top directory of
 the python source tree after building the interpreter and run:
 
-    python Lib/keyword.py
+    ./python Lib/keyword.py
 """
 
 __all__ = ["iskeyword", "kwlist"]
diff --git a/Lib/symbol.py b/Lib/symbol.py
--- a/Lib/symbol.py
+++ b/Lib/symbol.py
@@ -7,7 +7,7 @@
 #  To update the symbols in this file, 'cd' to the top directory of
 #  the python source tree after building the interpreter and run:
 #
-#    python Lib/symbol.py
+#    ./python Lib/symbol.py
 
 #--start constants--
 single_input = 256
diff --git a/Lib/token.py b/Lib/token.py
--- a/Lib/token.py
+++ b/Lib/token.py
@@ -7,7 +7,7 @@
 #  To update the symbols in this file, 'cd' to the top directory of
 #  the python source tree after building the interpreter and run:
 #
-#    python Lib/token.py
+#    ./python Lib/token.py
 
 #--start constants--
 ENDMARKER = 0

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 22:52:23 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 29 Nov 2011 22:52:23 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Add_subheaders_to_make_PEP_?=
	=?utf8?q?393_description_clearer?=
Message-ID: <E1RVVbD-0001IH-Cs@dinsdale.python.org>

http://hg.python.org/cpython/rev/3e295f2a47b9
changeset:   73791:3e295f2a47b9
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 29 22:45:07 2011 +0100
summary:
  Add subheaders to make PEP 393 description clearer

files:
  Doc/whatsnew/3.3.rst |  50 +++++++++++++++++--------------
  1 files changed, 28 insertions(+), 22 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -69,6 +69,9 @@
 a bit more memory, because Python may have to maintain two versions of each
 string (in the legacy format and in the new efficient storage).
 
+Functionality
+-------------
+
 Changes introduced by :pep:`393` are the following:
 
 * Python now always supports the full range of Unicode codepoints, including
@@ -76,28 +79,6 @@
   narrow and wide builds no longer exists and Python now behaves like a wide
   build, even under Windows.
 
-* The storage of Unicode strings now depends on the highest codepoint in the string:
-
-  * pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint;
-
-  * BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint;
-
-  * non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint.
-
-  The net effect is that for most applications, memory usage of string storage
-  should decrease significantly - especially compared to former wide unicode
-  builds - as, in many cases, strings will be pure ASCII even in international
-  contexts (because many strings store non-human language data, such as XML
-  fragments, HTTP headers, JSON-encoded data, etc.).  We also hope that it
-  will, for the same reasons, increase CPU cache efficiency on non-trivial
-  applications.
-
-   .. The memory usage of Python 3.3 is two to three times smaller than Python 3.2,
-      and a little bit better than Python 2.7, on a `Django benchmark
-      <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
-      XXX The result should be moved in the PEP and a link to the PEP should
-      be added here.
-
 * With the death of narrow builds, the problems specific to narrow builds have
   also been fixed, for example:
 
@@ -120,6 +101,31 @@
 
 * The :file:`./configure` flag ``--with-wide-unicode`` has been removed.
 
+Performance and resource usage
+------------------------------
+
+The storage of Unicode strings now depends on the highest codepoint in the string:
+
+* pure ASCII and Latin1 strings (``U+0000-U+00FF``) use 1 byte per codepoint;
+
+* BMP strings (``U+0000-U+FFFF``) use 2 bytes per codepoint;
+
+* non-BMP strings (``U+10000-U+10FFFF``) use 4 bytes per codepoint.
+
+The net effect is that for most applications, memory usage of string storage
+should decrease significantly - especially compared to former wide unicode
+builds - as, in many cases, strings will be pure ASCII even in international
+contexts (because many strings store non-human language data, such as XML
+fragments, HTTP headers, JSON-encoded data, etc.).  We also hope that it
+will, for the same reasons, increase CPU cache efficiency on non-trivial
+applications.
+
+.. The memory usage of Python 3.3 is two to three times smaller than Python 3.2,
+   and a little bit better than Python 2.7, on a `Django benchmark
+   <http://mail.python.org/pipermail/python-dev/2011-September/113714.html>`_.
+   XXX The result should be moved in the PEP and a link to the PEP should
+   be added here.
+
 
 PEP 3151: Reworking the OS and IO exception hierarchy
 =====================================================

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 22:52:24 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Tue, 29 Nov 2011 22:52:24 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Mention_hashable_memoryview?=
	=?utf8?q?s=2C_and_reformat=2E?=
Message-ID: <E1RVVbE-0001IR-D7@dinsdale.python.org>

http://hg.python.org/cpython/rev/1828e21832b3
changeset:   73792:1828e21832b3
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Tue Nov 29 22:47:11 2011 +0100
summary:
  Mention hashable memoryviews, and reformat.

files:
  Doc/whatsnew/3.3.rst |  26 +++++++++++++-------------
  1 files changed, 13 insertions(+), 13 deletions(-)


diff --git a/Doc/whatsnew/3.3.rst b/Doc/whatsnew/3.3.rst
--- a/Doc/whatsnew/3.3.rst
+++ b/Doc/whatsnew/3.3.rst
@@ -259,26 +259,26 @@
 
 Some smaller changes made to the core Python language are:
 
-* Stub
+* Added support for Unicode name aliases and named sequences.
+  Both :func:`unicodedata.lookup()` and ``'\N{...}'`` now resolve name aliases,
+  and :func:`unicodedata.lookup()` resolves named sequences too.
 
-Added support for Unicode name aliases and named sequences.
-Both :func:`unicodedata.lookup()` and ``'\N{...}'`` now resolve name aliases,
-and :func:`unicodedata.lookup()` resolves named sequences too.
+  (Contributed by Ezio Melotti in :issue:`12753`)
 
-(Contributed by Ezio Melotti in :issue:`12753`)
+* Equality comparisons on :func:`range` objects now return a result reflecting
+  the equality of the underlying sequences generated by those range objects.
 
+  (:issue:`13021`)
 
-Equality comparisons on :func:`range` objects now return a result reflecting
-the equality of the underlying sequences generated by those range objects.
+* The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()``
+  methods of :class:`bytes` and :class:`bytearray` objects now accept an
+  integer between 0 and 255 as their first argument.
 
-(:issue:`13021`)
+  (:issue:`12170`)
 
+* Memoryview objects are now hashable when the underlying object is hashable.
 
-The ``count()``, ``find()``, ``rfind()``, ``index()`` and ``rindex()``
-methods of :class:`bytes` and :class:`bytearray` objects now accept an
-integer between 0 and 255 as their first argument.
-
-(:issue:`12170`)
+  (Contributed by Antoine Pitrou in :issue:`13411`)
 
 
 New and Improved Modules

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 23:01:16 2011
From: python-checkins at python.org (victor.stinner)
Date: Tue, 29 Nov 2011 23:01:16 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Py=5FUNICODE=5FHIGH=5FSURRO?=
 =?utf8?q?GATE=28=29_and_Py=5FUNICODE=5FLOW=5FSURROGATE=28=29_macros?=
Message-ID: <E1RVVjo-0002F5-EU@dinsdale.python.org>

http://hg.python.org/cpython/rev/351cdf87bed0
changeset:   73793:351cdf87bed0
user:        Victor Stinner <victor.stinner at haypocalc.com>
date:        Tue Nov 29 22:58:13 2011 +0100
summary:
  Py_UNICODE_HIGH_SURROGATE() and Py_UNICODE_LOW_SURROGATE() macros

And use surrogates macros everywhere in unicodeobject.c

files:
  Include/unicodeobject.h |   4 ++
  Objects/unicodeobject.c |  45 ++++++++++++----------------
  2 files changed, 23 insertions(+), 26 deletions(-)


diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -187,6 +187,10 @@
 #define Py_UNICODE_JOIN_SURROGATES(high, low)  \
     (((((Py_UCS4)(high) & 0x03FF) << 10) |      \
       ((Py_UCS4)(low) & 0x03FF)) + 0x10000)
+/* high surrogate = top 10 bits added to D800 */
+#define Py_UNICODE_HIGH_SURROGATE(ch) (0xD800 | (((ch) - 0x10000) >> 10))
+/* low surrogate = bottom 10 bits added to DC00 */
+#define Py_UNICODE_LOW_SURROGATE(ch) (0xDC00 | (((ch) - 0x10000) & 0x3FF))
 
 /* Check if substring matches at given offset.  The offset must be
    valid, and the substring must not be empty. */
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -1077,10 +1077,11 @@
     for (iter = begin; iter < end; ) {
         assert(ucs4_out < (PyUnicode_4BYTE_DATA(unicode) +
                            _PyUnicode_GET_LENGTH(unicode)));
-        if (*iter >= 0xD800 && *iter <= 0xDBFF
-            && (iter+1) < end && iter[1] >= 0xDC00 && iter[1] <= 0xDFFF)
+        if (Py_UNICODE_IS_HIGH_SURROGATE(iter[0])
+            && (iter+1) < end
+            && Py_UNICODE_IS_LOW_SURROGATE(iter[1]))
         {
-            *ucs4_out++ = (((iter[0] & 0x3FF)<<10) | (iter[1] & 0x3FF)) + 0x10000;
+            *ucs4_out++ = Py_UNICODE_JOIN_SURROGATES(iter[0], iter[1]);
             iter += 2;
         }
         else {
@@ -3473,8 +3474,8 @@
                 if (*four_bytes > 0xFFFF) {
                     assert(*four_bytes <= 0x10FFFF);
                     /* encode surrogate pair in this case */
-                    *w++ = 0xD800 | ((*four_bytes - 0x10000) >> 10);
-                    *w   = 0xDC00 | ((*four_bytes - 0x10000) & 0x3FF);
+                    *w++ = Py_UNICODE_HIGH_SURROGATE(*four_bytes);
+                    *w   = Py_UNICODE_LOW_SURROGATE(*four_bytes);
                 }
                 else
                     *w = *four_bytes;
@@ -3906,9 +3907,8 @@
                     base64buffer &= (1 << base64bits) - 1; /* clear high bits */
                     if (surrogate) {
                         /* expecting a second surrogate */
-                        if (outCh >= 0xDC00 && outCh <= 0xDFFF) {
-                            Py_UCS4 ch2 = (((surrogate & 0x3FF)<<10)
-                                           | (outCh & 0x3FF)) + 0x10000;
+                        if (Py_UNICODE_IS_LOW_SURROGATE(outCh)) {
+                            Py_UCS4 ch2 = Py_UNICODE_JOIN_SURROGATES(surrogate, outCh);
                             if (unicode_putchar(&unicode, &outpos, ch2) < 0)
                                 goto onError;
                             surrogate = 0;
@@ -3920,7 +3920,7 @@
                             surrogate = 0;
                         }
                     }
-                    if (outCh >= 0xD800 && outCh <= 0xDBFF) {
+                    if (Py_UNICODE_IS_HIGH_SURROGATE(outCh)) {
                         /* first surrogate */
                         surrogate = outCh;
                     }
@@ -4128,7 +4128,7 @@
                 base64bits -= 6;
             }
             /* prepare second surrogate */
-            ch =  0xDC00 | ((ch-0x10000) & 0x3FF);
+            ch = Py_UNICODE_LOW_SURROGATE(ch);
         }
         base64bits += 16;
         base64buffer = (base64buffer << 16) | ch;
@@ -4720,15 +4720,8 @@
             *p++ = (wchar_t)ch;
 #else
             /*  compute and append the two surrogates: */
-
-            /*  translate from 10000..10FFFF to 0..FFFF */
-            ch -= 0x10000;
-
-            /*  high surrogate = top 10 bits added to D800 */
-            *p++ = (wchar_t)(0xD800 + (ch >> 10));
-
-            /*  low surrogate = bottom 10 bits added to DC00 */
-            *p++ = (wchar_t)(0xDC00 + (ch & 0x03FF));
+            *p++ = (wchar_t)Py_UNICODE_HIGH_SURROGATE(ch);
+            *p++ = (wchar_t)Py_UNICODE_LOW_SURROGATE(ch);
 #endif
             break;
         }
@@ -4819,7 +4812,7 @@
             /* Encode Latin-1 */
             *p++ = (char)(0xc0 | (ch >> 6));
             *p++ = (char)(0x80 | (ch & 0x3f));
-        } else if (0xD800 <= ch && ch <= 0xDFFF) {
+        } else if (Py_UNICODE_IS_SURROGATE(ch)) {
             Py_ssize_t newpos;
             Py_ssize_t repsize, k, startpos;
             startpos = i-1;
@@ -5367,7 +5360,7 @@
 
         q += 2;
 
-        if (ch < 0xD800 || ch > 0xDFFF) {
+        if (!Py_UNICODE_IS_SURROGATE(ch)) {
             if (unicode_putchar(&unicode, &outpos, ch) < 0)
                 goto onError;
             continue;
@@ -5538,8 +5531,8 @@
         Py_UCS4 ch = PyUnicode_READ(kind, data, i);
         Py_UCS4 ch2 = 0;
         if (ch >= 0x10000) {
-            ch2 = 0xDC00 | ((ch-0x10000) & 0x3FF);
-            ch  = 0xD800 | ((ch-0x10000) >> 10);
+            ch2 = Py_UNICODE_LOW_SURROGATE(ch);
+            ch  = Py_UNICODE_HIGH_SURROGATE(ch);
         }
         STORECHAR(ch);
         if (ch2)
@@ -6300,14 +6293,14 @@
 
         s += Py_UNICODE_SIZE;
 #ifndef Py_UNICODE_WIDE
-        if (ch >= 0xD800 && ch <= 0xDBFF && s < end)
+        if (Py_UNICODE_IS_HIGH_SURROGATE(ch) && s < end)
         {
             Py_UNICODE uch2;
             ((char *) &uch2)[0] = s[0];
             ((char *) &uch2)[1] = s[1];
-            if (uch2 >= 0xDC00 && uch2 <= 0xDFFF)
+            if (Py_UNICODE_IS_LOW_SURROGATE(uch2))
             {
-                ch = (((uch & 0x3FF)<<10) | (uch2 & 0x3FF)) + 0x10000;
+                ch = Py_UNICODE_JOIN_SURROGATES(uch, uch2);
                 s += Py_UNICODE_SIZE;
             }
         }

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 23:36:58 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Tue, 29 Nov 2011 23:36:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Issue_=236715=3A_Add_module?=
 =?utf8?q?_for_compression_using_the_LZMA_algorithm=2E?=
Message-ID: <E1RVWIM-0005fm-KH@dinsdale.python.org>

http://hg.python.org/cpython/rev/74d182cf0187
changeset:   73794:74d182cf0187
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Wed Nov 30 00:25:06 2011 +0200
summary:
  Issue #6715: Add module for compression using the LZMA algorithm.

files:
  Doc/library/archiving.rst          |     4 +-
  Doc/library/bz2.rst                |     2 +-
  Doc/library/gzip.rst               |     2 +-
  Doc/library/lzma.rst               |   344 ++++
  Doc/library/zipfile.rst            |     2 +-
  Doc/library/zlib.rst               |     4 +-
  Lib/lzma.py                        |   398 ++++
  Lib/test/test_lzma.py              |  1335 ++++++++++++++++
  Misc/NEWS                          |     2 +
  Modules/_lzmamodule.c              |  1106 +++++++++++++
  PCbuild/_lzma.vcproj               |   537 ++++++
  PCbuild/pcbuild.sln                |    21 +
  PCbuild/pyproject.vsprops          |     4 +
  PCbuild/readme.txt                 |     6 +
  Tools/buildbot/external-common.bat |     5 +
  setup.py                           |     7 +
  16 files changed, 3773 insertions(+), 6 deletions(-)


diff --git a/Doc/library/archiving.rst b/Doc/library/archiving.rst
--- a/Doc/library/archiving.rst
+++ b/Doc/library/archiving.rst
@@ -5,7 +5,8 @@
 ******************************
 
 The modules described in this chapter support data compression with the zlib,
-gzip, and bzip2 algorithms, and  the creation of ZIP- and tar-format archives.
+gzip, bzip2 and lzma algorithms, and the creation of ZIP- and tar-format
+archives.
 
 
 .. toctree::
@@ -13,5 +14,6 @@
    zlib.rst
    gzip.rst
    bz2.rst
+   lzma.rst
    zipfile.rst
    tarfile.rst
diff --git a/Doc/library/bz2.rst b/Doc/library/bz2.rst
--- a/Doc/library/bz2.rst
+++ b/Doc/library/bz2.rst
@@ -12,7 +12,7 @@
 This module provides a comprehensive interface for compressing and
 decompressing data using the bzip2 compression algorithm.
 
-For related file formats, see the :mod:`gzip`, :mod:`zipfile`, and
+For related file formats, see the :mod:`gzip`, :mod:`lzma`, :mod:`zipfile`, and
 :mod:`tarfile` modules.
 
 The :mod:`bz2` module contains:
diff --git a/Doc/library/gzip.rst b/Doc/library/gzip.rst
--- a/Doc/library/gzip.rst
+++ b/Doc/library/gzip.rst
@@ -21,7 +21,7 @@
 :program:`gzip` and :program:`gunzip` programs, such  as those produced by
 :program:`compress` and :program:`pack`, are not supported by this module.
 
-For other archive formats, see the :mod:`bz2`, :mod:`zipfile`, and
+For related file formats, see the :mod:`bz2`, :mod:`lzma`, :mod:`zipfile`, and
 :mod:`tarfile` modules.
 
 The module defines the following items:
diff --git a/Doc/library/lzma.rst b/Doc/library/lzma.rst
new file mode 100644
--- /dev/null
+++ b/Doc/library/lzma.rst
@@ -0,0 +1,344 @@
+:mod:`lzma` --- Compression using the LZMA algorithm
+====================================================
+
+.. module:: lzma
+   :synopsis: A Python wrapper for the liblzma compression library.
+.. moduleauthor:: Nadeem Vawda <nadeem.vawda at gmail.com>
+.. sectionauthor:: Nadeem Vawda <nadeem.vawda at gmail.com>
+
+.. versionadded:: 3.3
+
+
+This module provides classes and convenience functions for compressing and
+decompressing data using the LZMA compression algorithm. Also included is a file
+interface supporting the ``.xz`` and legacy ``.lzma`` file formats used by the
+:program:`xz` utility, as well as raw compressed streams.
+
+For related file formats, see the :mod:`bz2`, :mod:`gzip`, :mod:`zipfile`, and
+:mod:`tarfile` modules.
+
+The interface provided by this module is very similar to that of the :mod:`bz2`
+module. However, note that :class:`LZMAFile` is *not* thread-safe, unlike
+:class:`bz2.BZ2File`, so if you need to use a single :class:`LZMAFile` instance
+from multiple threads, it is necessary to protect it with a lock.
+
+
+.. exception:: LZMAError
+
+   This exception is raised when an error occurs during compression or
+   decompression, or while initializing the compressor/decompressor state.
+
+
+Reading and writing compressed files
+------------------------------------
+
+.. class:: LZMAFile(filename=None, mode="r", fileobj=None, format=None, check=-1, preset=None, filters=None)
+
+   Open an LZMA-compressed file.
+
+   An :class:`LZMAFile` can wrap an existing :term:`file object` (given by
+   *fileobj*), or operate directly on a named file (named by *filename*).
+   Exactly one of these two parameters should be provided. If *fileobj* is
+   provided, it is not closed when the :class:`LZMAFile` is closed.
+
+   The *mode* argument can be either ``"r"`` for reading (default), ``"w"`` for
+   overwriting, or ``"a"`` for appending. If *fileobj* is provided, a mode of
+   ``"w"`` does not truncate the file, and is instead equivalent to ``"a"``.
+
+   When opening a file for reading, the input file may be the concatenation of
+   multiple separate compressed streams. These are transparently decoded as a
+   single logical stream.
+
+   When opening a file for reading, the *format* and *filters* arguments have
+   the same meanings as for :class:`LZMADecompressor`. In this case, the *check*
+   and *preset* arguments should not be used.
+
+   When opening a file for writing, the *format*, *check*, *preset* and
+   *filters* arguments have the same meanings as for :class:`LZMACompressor`.
+
+   :class:`LZMAFile` supports all the members specified by
+   :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`.
+   Iteration and the :keyword:`with` statement are supported.
+
+   The following method is also provided:
+
+   .. method:: peek(size=-1)
+
+      Return buffered data without advancing the file position. At least one
+      byte of data will be returned, unless EOF has been reached. The exact
+      number of bytes returned is unspecified (the *size* argument is ignored).
+
+
+Compressing and decompressing data in memory
+--------------------------------------------
+
+.. class:: LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None)
+
+   Create a compressor object, which can be used to compress data incrementally.
+
+   For a more convenient way of compressing a single chunk of data, see
+   :func:`compress`.
+
+   The *format* argument specifies what container format should be used.
+   Possible values are:
+
+   * :const:`FORMAT_XZ`: The ``.xz`` container format.
+      This is the default format.
+
+   * :const:`FORMAT_ALONE`: The legacy ``.lzma`` container format.
+      This format is more limited than ``.xz`` -- it does not support integrity
+      checks or multiple filters.
+
+   * :const:`FORMAT_RAW`: A raw data stream, not using any container format.
+      This format specifier does not support integrity checks, and requires that
+      you always specify a custom filter chain (for both compression and
+      decompression). Additionally, data compressed in this manner cannot be
+      decompressed using :const:`FORMAT_AUTO` (see :class:`LZMADecompressor`).
+
+   The *check* argument specifies the type of integrity check to include in the
+   compressed data. This check is used when decompressing, to ensure that the
+   data has not been corrupted. Possible values are:
+
+   * :const:`CHECK_NONE`: No integrity check.
+     This is the default (and the only acceptable value) for
+     :const:`FORMAT_ALONE` and :const:`FORMAT_RAW`.
+
+   * :const:`CHECK_CRC32`: 32-bit Cyclic Redundancy Check.
+
+   * :const:`CHECK_CRC64`: 64-bit Cyclic Redundancy Check.
+     This is the default for :const:`FORMAT_XZ`.
+
+   * :const:`CHECK_SHA256`: 256-bit Secure Hash Algorithm.
+
+   If the specified check is not supported, an :class:`LZMAError` is raised.
+
+   The compression settings can be specified either as a preset compression
+   level (with the *preset* argument), or in detail as a custom filter chain
+   (with the *filters* argument).
+
+   The *preset* argument (if provided) should be an integer between ``0`` and
+   ``9`` (inclusive), optionally OR-ed with the constant
+   :const:`PRESET_EXTREME`. If neither *preset* nor *filters* are given, the
+   default behavior is to use :const:`PRESET_DEFAULT` (preset level ``6``).
+   Higher presets produce smaller output, but make compression more CPU- and
+   memory-intensive, and also increase the memory required for decompression.
+
+   The *filters* argument (if provided) should be a filter chain specifier.
+   See :ref:`filter-chain-specs` for details.
+
+   .. method:: compress(data)
+
+      Compress *data* (a :class:`bytes` object), returning a :class:`bytes`
+      object containing compressed data for at least part of the input. Some of
+      *data* may be buffered internally, for use in later calls to
+      :meth:`compress` and :meth:`flush`. The returned data should be
+      concatenated with the output of any previous calls to :meth:`compress`.
+
+   .. method:: flush()
+
+      Finish the compression process, returning a :class:`bytes` object
+      containing any data stored in the compressor's internal buffers.
+
+      The compressor cannot be used after this method has been called.
+
+
+.. class:: LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None)
+
+   Create a decompressor object, which can be used to decompress data
+   incrementally.
+
+   For a more convenient way of decompressing an entire compressed stream at
+   once, see :func:`decompress`.
+
+   The *format* argument specifies the container format that should be used. The
+   default is :const:`FORMAT_AUTO`, which can decompress both ``.xz`` and
+   ``.lzma`` files. Other possible values are :const:`FORMAT_XZ`,
+   :const:`FORMAT_ALONE`, and :const:`FORMAT_RAW`.
+
+   The *memlimit* argument specifies a limit (in bytes) on the amount of memory
+   that the decompressor can use. When this argument is used, decompression will
+   fail with an :class:`LZMAError` if it is not possible to decompress the input
+   within the given memory limit.
+
+   The *filters* argument specifies the filter chain that was used to create
+   the stream being decompressed. This argument is required if *format* is
+   :const:`FORMAT_RAW`, but should not be used for other formats.
+   See :ref:`filter-chain-specs` for more information about filter chains.
+
+   .. note::
+      This class does not transparently handle inputs containing multiple
+      compressed streams, unlike :func:`decompress` and :class:`LZMAFile`. To
+      decompress a multi-stream input with :class:`LZMADecompressor`, you must
+      create a new decompressor for each stream.
+
+   .. method:: decompress(data)
+
+      Decompress *data* (a :class:`bytes` object), returning a :class:`bytes`
+      object containing the decompressed data for at least part of the input.
+      Some of *data* may be buffered internally, for use in later calls to
+      :meth:`decompress`. The returned data should be concatenated with the
+      output of any previous calls to :meth:`decompress`.
+
+   .. attribute:: check
+
+      The ID of the integrity check used by the input stream. This may be
+      :const:`CHECK_UNKNOWN` until enough of the input has been decoded to
+      determine what integrity check it uses.
+
+   .. attribute:: eof
+
+      True if the end-of-stream marker has been reached.
+
+   .. attribute:: unused_data
+
+      Data found after the end of the compressed stream.
+
+      Before the end of the stream is reached, this will be ``b""``.
+
+
+.. function:: compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None)
+
+   Compress *data* (a :class:`bytes` object), returning the compressed data as a
+   :class:`bytes` object.
+
+   See :class:`LZMACompressor` above for a description of the *format*, *check*,
+   *preset* and *filters* arguments.
+
+
+.. function:: decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None)
+
+   Decompress *data* (a :class:`bytes` object), returning the uncompressed data
+   as a :class:`bytes` object.
+
+   If *data* is the concatenation of multiple distinct compressed streams,
+   decompress all of these streams, and return the concatenation of the results.
+
+   See :class:`LZMADecompressor` above for a description of the *format*,
+   *memlimit* and *filters* arguments.
+
+
+Miscellaneous
+-------------
+
+.. function:: check_is_supported(check)
+
+   Returns true if the given integrity check is supported on this system.
+
+   :const:`CHECK_NONE` and :const:`CHECK_CRC32` are always supported.
+   :const:`CHECK_CRC64` and :const:`CHECK_SHA256` may be unavailable if you are
+   using a version of :program:`liblzma` that was compiled with a limited
+   feature set.
+
+
+.. _filter-chain-specs:
+
+Specifying custom filter chains
+-------------------------------
+
+A filter chain specifier is a sequence of dictionaries, where each dictionary
+contains the ID and options for a single filter. Each dictionary must contain
+the key ``"id"``, and may contain additional keys to specify filter-dependent
+options. Valid filter IDs are as follows:
+
+* Compression filters:
+   * :const:`FILTER_LZMA1` (for use with :const:`FORMAT_ALONE`)
+   * :const:`FILTER_LZMA2` (for use with :const:`FORMAT_XZ` and :const:`FORMAT_RAW`)
+
+* Delta filter:
+   * :const:`FILTER_DELTA`
+
+* Branch-Call-Jump (BCJ) filters:
+   * :const:`FILTER_X86`
+   * :const:`FILTER_IA64`
+   * :const:`FILTER_ARM`
+   * :const:`FILTER_ARMTHUMB`
+   * :const:`FILTER_POWERPC`
+   * :const:`FILTER_SPARC`
+
+A filter chain can consist of up to 4 filters, and cannot be empty. The last
+filter in the chain must be a compression filter, and any other filters must be
+delta or BCJ filters.
+
+Compression filters support the following options (specified as additional
+entries in the dictionary representing the filter):
+
+   * ``preset``: A compression preset to use as a source of default values for
+     options that are not specified explicitly.
+   * ``dict_size``: Dictionary size in bytes. This should be between 4KiB and
+     1.5GiB (inclusive).
+   * ``lc``: Number of literal context bits.
+   * ``lp``: Number of literal position bits. The sum ``lc + lp`` must be at
+     most 4.
+   * ``pb``: Number of position bits; must be at most 4.
+   * ``mode``: :const:`MODE_FAST` or :const:`MODE_NORMAL`.
+   * ``nice_len``: What should be considered a "nice length" for a match.
+     This should be 273 or less.
+   * ``mf``: What match finder to use -- :const:`MF_HC3`, :const:`MF_HC4`,
+     :const:`MF_BT2`, :const:`MF_BT3`, or :const:`MF_BT4`.
+   * ``depth``: Maximum search depth used by match finder. 0 (default) means to
+     select automatically based on other filter options.
+
+The delta filter stores the differences between bytes, producing more repetitive
+input for the compressor in certain circumstances. It only supports a single
+The delta filter supports only one option, ``dist``. This indicates the distance
+between bytes to be subtracted. The default is 1, i.e. take the differences
+between adjacent bytes.
+
+The BCJ filters are intended to be applied to machine code. They convert
+relative branches, calls and jumps in the code to use absolute addressing, with
+the aim of increasing the redundancy that can be exploited by the compressor.
+These filters support one option, ``start_offset``. This specifies the address
+that should be mapped to the beginning of the input data. The default is 0.
+
+
+Examples
+--------
+
+Reading in a compressed file::
+
+   import lzma
+   with lzma.LZMAFile("file.xz") as f:
+      file_content = f.read()
+
+Creating a compressed file::
+
+   import lzma
+   data = b"Insert Data Here"
+   with lzma.LZMAFile("file.xz", "w") as f:
+      f.write(data)
+
+Compressing data in memory::
+
+   import lzma
+   data_in = b"Insert Data Here"
+   data_out = lzma.compress(data_in)
+
+Incremental compression::
+
+   import lzma
+   lzc = lzma.LZMACompressor()
+   out1 = lzc.compress(b"Some data\n")
+   out2 = lzc.compress(b"Another piece of data\n")
+   out3 = lzc.compress(b"Even more data\n")
+   out4 = lzc.flush()
+   # Concatenate all the partial results:
+   result = b"".join([out1, out2, out3, out4])
+
+Writing compressed data to an already-open file::
+
+   import lzma
+   with open("file.xz", "wb") as f:
+       f.write(b"This data will not be compressed\n")
+       with lzma.LZMAFile(fileobj=f, mode="w") as lzf:
+           lzf.write(b"This *will* be compressed\n")
+       f.write(b"Not compressed\n")
+
+Creating a compressed file using a custom filter chain::
+
+   import lzma
+   my_filters = [
+       {"id": lzma.FILTER_DELTA, "dist": 5},
+       {"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME},
+   ]
+   with lzma.LZMAFile("file.xz", "w", filters=my_filters) as f:
+       f.write(b"blah blah blah")
diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst
--- a/Doc/library/zipfile.rst
+++ b/Doc/library/zipfile.rst
@@ -23,7 +23,7 @@
 create an encrypted file.  Decryption is extremely slow as it is
 implemented in native Python rather than C.
 
-For other archive formats, see the :mod:`bz2`, :mod:`gzip`, and
+For related file formats, see the :mod:`bz2`, :mod:`gzip`, :mod:`lzma`, and
 :mod:`tarfile` modules.
 
 The module defines the following items:
diff --git a/Doc/library/zlib.rst b/Doc/library/zlib.rst
--- a/Doc/library/zlib.rst
+++ b/Doc/library/zlib.rst
@@ -18,8 +18,8 @@
 consult the zlib manual at http://www.zlib.net/manual.html for authoritative
 information.
 
-For reading and writing ``.gz`` files see the :mod:`gzip` module. For
-other related file formats, see the :mod:`bz2`, :mod:`zipfile`, and
+For reading and writing ``.gz`` files see the :mod:`gzip` module. For other
+related file formats, see the :mod:`bz2`, :mod:`lzma`, :mod:`zipfile`, and
 :mod:`tarfile` modules.
 
 The available exception and functions in this module are:
diff --git a/Lib/lzma.py b/Lib/lzma.py
new file mode 100644
--- /dev/null
+++ b/Lib/lzma.py
@@ -0,0 +1,398 @@
+"""Interface to the liblzma compression library.
+
+This module provides a class for reading and writing compressed files,
+classes for incremental (de)compression, and convenience functions for
+one-shot (de)compression.
+
+These classes and functions support both the XZ and legacy LZMA
+container formats, as well as raw compressed data streams.
+"""
+
+__all__ = [
+    "CHECK_NONE", "CHECK_CRC32", "CHECK_CRC64", "CHECK_SHA256",
+    "CHECK_ID_MAX", "CHECK_UNKNOWN",
+    "FILTER_LZMA1", "FILTER_LZMA2", "FILTER_DELTA", "FILTER_X86", "FILTER_IA64",
+    "FILTER_ARM", "FILTER_ARMTHUMB", "FILTER_POWERPC", "FILTER_SPARC",
+    "FORMAT_AUTO", "FORMAT_XZ", "FORMAT_ALONE", "FORMAT_RAW",
+    "MF_HC3", "MF_HC4", "MF_BT2", "MF_BT3", "MF_BT4",
+    "MODE_FAST", "MODE_NORMAL", "PRESET_DEFAULT", "PRESET_EXTREME",
+
+    "LZMACompressor", "LZMADecompressor", "LZMAFile", "LZMAError",
+    "compress", "decompress", "check_is_supported",
+]
+
+import io
+from _lzma import *
+
+
+_MODE_CLOSED   = 0
+_MODE_READ     = 1
+_MODE_READ_EOF = 2
+_MODE_WRITE    = 3
+
+_BUFFER_SIZE = 8192
+
+
+class LZMAFile(io.BufferedIOBase):
+
+    """A file object providing transparent LZMA (de)compression.
+
+    An LZMAFile can act as a wrapper for an existing file object, or
+    refer directly to a named file on disk.
+
+    Note that LZMAFile provides a *binary* file interface - data read
+    is returned as bytes, and data to be written must be given as bytes.
+    """
+
+    def __init__(self, filename=None, mode="r", *,
+                 fileobj=None, format=None, check=-1,
+                 preset=None, filters=None):
+        """Open an LZMA-compressed file.
+
+        If filename is given, open the named file. Otherwise, operate on
+        the file object given by fileobj. Exactly one of these two
+        parameters should be provided.
+
+        mode can be "r" for reading (default), "w" for (over)writing, or
+        "a" for appending.
+
+        format specifies the container format to use for the file.
+        If mode is "r", this defaults to FORMAT_AUTO. Otherwise, the
+        default is FORMAT_XZ.
+
+        check specifies the integrity check to use. This argument can
+        only be used when opening a file for writing. For FORMAT_XZ,
+        the default is CHECK_CRC64. FORMAT_ALONE and FORMAT_RAW do not
+        support integrity checks - for these formats, check must be
+        omitted, or be CHECK_NONE.
+
+        When opening a file for reading, the *preset* argument is not
+        meaningful, and should be omitted. The *filters* argument should
+        also be omitted, except when format is FORMAT_RAW (in which case
+        it is required).
+
+        When opening a file for writing, the settings used by the
+        compressor can be specified either as a preset compression
+        level (with the *preset* argument), or in detail as a custom
+        filter chain (with the *filters* argument). For FORMAT_XZ and
+        FORMAT_ALONE, the default is to use the PRESET_DEFAULT preset
+        level. For FORMAT_RAW, the caller must always specify a filter
+        chain; the raw compressor does not support preset compression
+        levels.
+
+        preset (if provided) should be an integer in the range 0-9,
+        optionally OR-ed with the constant PRESET_EXTREME.
+
+        filters (if provided) should be a sequence of dicts. Each dict
+        should have an entry for "id" indicating ID of the filter, plus
+        additional entries for options to the filter.
+        """
+        self._fp = None
+        self._closefp = False
+        self._mode = _MODE_CLOSED
+        self._pos = 0
+        self._size = -1
+
+        if mode == "r":
+            if check != -1:
+                raise ValueError("Cannot specify an integrity check "
+                                 "when opening a file for reading")
+            if preset is not None:
+                raise ValueError("Cannot specify a preset compression "
+                                 "level when opening a file for reading")
+            if format is None:
+                format = FORMAT_AUTO
+            mode_code = _MODE_READ
+            # Save the args to pass to the LZMADecompressor initializer.
+            # If the file contains multiple compressed streams, each
+            # stream will need a separate decompressor object.
+            self._init_args = {"format":format, "filters":filters}
+            self._decompressor = LZMADecompressor(**self._init_args)
+            self._buffer = None
+        elif mode in ("w", "a"):
+            if format is None:
+                format = FORMAT_XZ
+            mode_code = _MODE_WRITE
+            self._compressor = LZMACompressor(format=format, check=check,
+                                              preset=preset, filters=filters)
+        else:
+            raise ValueError("Invalid mode: {!r}".format(mode))
+
+        if filename is not None and fileobj is None:
+            mode += "b"
+            self._fp = open(filename, mode)
+            self._closefp = True
+            self._mode = mode_code
+        elif fileobj is not None and filename is None:
+            self._fp = fileobj
+            self._mode = mode_code
+        else:
+            raise ValueError("Must give exactly one of filename and fileobj")
+
+    def close(self):
+        """Flush and close the file.
+
+        May be called more than once without error. Once the file is
+        closed, any other operation on it will raise a ValueError.
+        """
+        if self._mode == _MODE_CLOSED:
+            return
+        try:
+            if self._mode in (_MODE_READ, _MODE_READ_EOF):
+                self._decompressor = None
+                self._buffer = None
+            elif self._mode == _MODE_WRITE:
+                self._fp.write(self._compressor.flush())
+                self._compressor = None
+        finally:
+            try:
+                if self._closefp:
+                    self._fp.close()
+            finally:
+                self._fp = None
+                self._closefp = False
+                self._mode = _MODE_CLOSED
+
+    @property
+    def closed(self):
+        """True if this file is closed."""
+        return self._mode == _MODE_CLOSED
+
+    def fileno(self):
+        """Return the file descriptor for the underlying file."""
+        self._check_not_closed()
+        return self._fp.fileno()
+
+    def seekable(self):
+        """Return whether the file supports seeking."""
+        return self.readable()
+
+    def readable(self):
+        """Return whether the file was opened for reading."""
+        self._check_not_closed()
+        return self._mode in (_MODE_READ, _MODE_READ_EOF)
+
+    def writable(self):
+        """Return whether the file was opened for writing."""
+        self._check_not_closed()
+        return self._mode == _MODE_WRITE
+
+    # Mode-checking helper functions.
+
+    def _check_not_closed(self):
+        if self.closed:
+            raise ValueError("I/O operation on closed file")
+
+    def _check_can_read(self):
+        if not self.readable():
+            raise io.UnsupportedOperation("File not open for reading")
+
+    def _check_can_write(self):
+        if not self.writable():
+            raise io.UnsupportedOperation("File not open for writing")
+
+    def _check_can_seek(self):
+        if not self.seekable():
+            raise io.UnsupportedOperation("Seeking is only supported "
+                                          "on files open for reading")
+
+    # Fill the readahead buffer if it is empty. Returns False on EOF.
+    def _fill_buffer(self):
+        if self._buffer:
+            return True
+
+        if self._decompressor.unused_data:
+            rawblock = self._decompressor.unused_data
+        else:
+            rawblock = self._fp.read(_BUFFER_SIZE)
+
+        if not rawblock:
+            if self._decompressor.eof:
+                self._mode = _MODE_READ_EOF
+                self._size = self._pos
+                return False
+            else:
+                raise EOFError("Compressed file ended before the "
+                               "end-of-stream marker was reached")
+
+        # Continue to next stream.
+        if self._decompressor.eof:
+            self._decompressor = LZMADecompressor(**self._init_args)
+
+        self._buffer = self._decompressor.decompress(rawblock)
+        return True
+
+    # Read data until EOF.
+    # If return_data is false, consume the data without returning it.
+    def _read_all(self, return_data=True):
+        blocks = []
+        while self._fill_buffer():
+            if return_data:
+                blocks.append(self._buffer)
+            self._pos += len(self._buffer)
+            self._buffer = None
+        if return_data:
+            return b"".join(blocks)
+
+    # Read a block of up to n bytes.
+    # If return_data is false, consume the data without returning it.
+    def _read_block(self, n, return_data=True):
+        blocks = []
+        while n > 0 and self._fill_buffer():
+            if n < len(self._buffer):
+                data = self._buffer[:n]
+                self._buffer = self._buffer[n:]
+            else:
+                data = self._buffer
+                self._buffer = None
+            if return_data:
+                blocks.append(data)
+            self._pos += len(data)
+            n -= len(data)
+        if return_data:
+            return b"".join(blocks)
+
+    def peek(self, size=-1):
+        """Return buffered data without advancing the file position.
+
+        Always returns at least one byte of data, unless at EOF.
+        The exact number of bytes returned is unspecified.
+        """
+        self._check_can_read()
+        if self._mode == _MODE_READ_EOF or not self._fill_buffer():
+            return b""
+        return self._buffer
+
+    def read(self, size=-1):
+        """Read up to size uncompressed bytes from the file.
+
+        If size is negative or omitted, read until EOF is reached.
+        Returns b"" if the file is already at EOF.
+        """
+        self._check_can_read()
+        if self._mode == _MODE_READ_EOF or size == 0:
+            return b""
+        elif size < 0:
+            return self._read_all()
+        else:
+            return self._read_block(size)
+
+    def read1(self, size=-1):
+        """Read up to size uncompressed bytes with at most one read
+        from the underlying stream.
+
+        Returns b"" if the file is at EOF.
+        """
+        self._check_can_read()
+        if (size == 0 or self._mode == _MODE_READ_EOF or
+            not self._fill_buffer()):
+            return b""
+        if 0 < size < len(self._buffer):
+            data = self._buffer[:size]
+            self._buffer = self._buffer[size:]
+        else:
+            data = self._buffer
+            self._buffer = None
+        self._pos += len(data)
+        return data
+
+    def write(self, data):
+        """Write a bytes object to the file.
+
+        Returns the number of uncompressed bytes written, which is
+        always len(data). Note that due to buffering, the file on disk
+        may not reflect the data written until close() is called.
+        """
+        self._check_can_write()
+        compressed = self._compressor.compress(data)
+        self._fp.write(compressed)
+        self._pos += len(data)
+        return len(data)
+
+    # Rewind the file to the beginning of the data stream.
+    def _rewind(self):
+        self._fp.seek(0, 0)
+        self._mode = _MODE_READ
+        self._pos = 0
+        self._decompressor = LZMADecompressor(**self._init_args)
+        self._buffer = None
+
+    def seek(self, offset, whence=0):
+        """Change the file position.
+
+        The new position is specified by offset, relative to the
+        position indicated by whence. Possible values for whence are:
+
+            0: start of stream (default): offset must not be negative
+            1: current stream position
+            2: end of stream; offset must not be positive
+
+        Returns the new file position.
+
+        Note that seeking is emulated, sp depending on the parameters,
+        this operation may be extremely slow.
+        """
+        self._check_can_seek()
+
+        # Recalculate offset as an absolute file position.
+        if whence == 0:
+            pass
+        elif whence == 1:
+            offset = self._pos + offset
+        elif whence == 2:
+            # Seeking relative to EOF - we need to know the file's size.
+            if self._size < 0:
+                self._read_all(return_data=False)
+            offset = self._size + offset
+        else:
+            raise ValueError("Invalid value for whence: {}".format(whence))
+
+        # Make it so that offset is the number of bytes to skip forward.
+        if offset < self._pos:
+            self._rewind()
+        else:
+            offset -= self._pos
+
+        # Read and discard data until we reach the desired position.
+        if self._mode != _MODE_READ_EOF:
+            self._read_block(offset, return_data=False)
+
+        return self._pos
+
+    def tell(self):
+        """Return the current file position."""
+        self._check_not_closed()
+        return self._pos
+
+
+def compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None):
+    """Compress a block of data.
+
+    Refer to LZMACompressor's docstring for a description of the
+    optional arguments *format*, *check*, *preset* and *filters*.
+
+    For incremental compression, use an LZMACompressor object instead.
+    """
+    comp = LZMACompressor(format, check, preset, filters)
+    return comp.compress(data) + comp.flush()
+
+
+def decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None):
+    """Decompress a block of data.
+
+    Refer to LZMADecompressor's docstring for a description of the
+    optional arguments *format*, *check* and *filters*.
+
+    For incremental decompression, use a LZMADecompressor object instead.
+    """
+    results = []
+    while True:
+        decomp = LZMADecompressor(format, memlimit, filters)
+        results.append(decomp.decompress(data))
+        if not decomp.eof:
+            raise LZMAError("Compressed data ended before the "
+                            "end-of-stream marker was reached")
+        if not decomp.unused_data:
+            return b"".join(results)
+        # There is unused data left over. Proceed to next stream.
+        data = decomp.unused_data
diff --git a/Lib/test/test_lzma.py b/Lib/test/test_lzma.py
new file mode 100644
--- /dev/null
+++ b/Lib/test/test_lzma.py
@@ -0,0 +1,1335 @@
+from io import BytesIO, UnsupportedOperation
+import os
+import random
+import unittest
+
+from test.support import (
+    _4G, TESTFN, import_module, bigmemtest, run_unittest, unlink
+)
+
+lzma = import_module("lzma")
+from lzma import LZMACompressor, LZMADecompressor, LZMAError, LZMAFile
+
+
+class CompressorDecompressorTestCase(unittest.TestCase):
+
+    # Test error cases.
+
+    def test_simple_bad_args(self):
+        self.assertRaises(TypeError, LZMACompressor, [])
+        self.assertRaises(TypeError, LZMACompressor, format=3.45)
+        self.assertRaises(TypeError, LZMACompressor, check="")
+        self.assertRaises(TypeError, LZMACompressor, preset="asdf")
+        self.assertRaises(TypeError, LZMACompressor, filters=3)
+        # Can't specify FORMAT_AUTO when compressing.
+        self.assertRaises(ValueError, LZMACompressor, format=lzma.FORMAT_AUTO)
+        # Can't specify a preset and a custom filter chain at the same time.
+        with self.assertRaises(ValueError):
+            LZMACompressor(preset=7, filters=[{"id": lzma.FILTER_LZMA2}])
+
+        self.assertRaises(TypeError, LZMADecompressor, ())
+        self.assertRaises(TypeError, LZMADecompressor, memlimit=b"qw")
+        with self.assertRaises(TypeError):
+            LZMADecompressor(lzma.FORMAT_RAW, filters="zzz")
+        # Cannot specify a memory limit with FILTER_RAW.
+        with self.assertRaises(ValueError):
+            LZMADecompressor(lzma.FORMAT_RAW, memlimit=0x1000000)
+        # Can only specify a custom filter chain with FILTER_RAW.
+        self.assertRaises(ValueError, LZMADecompressor, filters=FILTERS_RAW_1)
+        with self.assertRaises(ValueError):
+            LZMADecompressor(format=lzma.FORMAT_XZ, filters=FILTERS_RAW_1)
+        with self.assertRaises(ValueError):
+            LZMADecompressor(format=lzma.FORMAT_ALONE, filters=FILTERS_RAW_1)
+
+        lzc = LZMACompressor()
+        self.assertRaises(TypeError, lzc.compress)
+        self.assertRaises(TypeError, lzc.compress, b"foo", b"bar")
+        self.assertRaises(TypeError, lzc.flush, b"blah")
+        empty = lzc.flush()
+        self.assertRaises(ValueError, lzc.compress, b"quux")
+        self.assertRaises(ValueError, lzc.flush)
+
+        lzd = LZMADecompressor()
+        self.assertRaises(TypeError, lzd.decompress)
+        self.assertRaises(TypeError, lzd.decompress, b"foo", b"bar")
+        lzd.decompress(empty)
+        self.assertRaises(EOFError, lzd.decompress, b"quux")
+
+    def test_bad_filter_spec(self):
+        self.assertRaises(TypeError, LZMACompressor, filters=[b"wobsite"])
+        self.assertRaises(ValueError, LZMACompressor, filters=[{"xyzzy": 3}])
+        self.assertRaises(ValueError, LZMACompressor, filters=[{"id": 98765}])
+        with self.assertRaises(ValueError):
+            LZMACompressor(filters=[{"id": lzma.FILTER_LZMA2, "foo": 0}])
+        with self.assertRaises(ValueError):
+            LZMACompressor(filters=[{"id": lzma.FILTER_DELTA, "foo": 0}])
+        with self.assertRaises(ValueError):
+            LZMACompressor(filters=[{"id": lzma.FILTER_X86, "foo": 0}])
+
+    def test_decompressor_after_eof(self):
+        lzd = LZMADecompressor()
+        lzd.decompress(COMPRESSED_XZ)
+        self.assertRaises(EOFError, lzd.decompress, b"nyan")
+
+    def test_decompressor_memlimit(self):
+        lzd = LZMADecompressor(memlimit=1024)
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ)
+
+        lzd = LZMADecompressor(lzma.FORMAT_XZ, memlimit=1024)
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ)
+
+        lzd = LZMADecompressor(lzma.FORMAT_ALONE, memlimit=1024)
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_ALONE)
+
+    # Test LZMADecompressor on known-good input data.
+
+    def _test_decompressor(self, lzd, data, check, unused_data=b""):
+        self.assertFalse(lzd.eof)
+        out = lzd.decompress(data)
+        self.assertEqual(out, INPUT)
+        self.assertEqual(lzd.check, check)
+        self.assertTrue(lzd.eof)
+        self.assertEqual(lzd.unused_data, unused_data)
+
+    def test_decompressor_auto(self):
+        lzd = LZMADecompressor()
+        self._test_decompressor(lzd, COMPRESSED_XZ, lzma.CHECK_CRC64)
+
+        lzd = LZMADecompressor()
+        self._test_decompressor(lzd, COMPRESSED_ALONE, lzma.CHECK_NONE)
+
+    def test_decompressor_xz(self):
+        lzd = LZMADecompressor(lzma.FORMAT_XZ)
+        self._test_decompressor(lzd, COMPRESSED_XZ, lzma.CHECK_CRC64)
+
+    def test_decompressor_alone(self):
+        lzd = LZMADecompressor(lzma.FORMAT_ALONE)
+        self._test_decompressor(lzd, COMPRESSED_ALONE, lzma.CHECK_NONE)
+
+    def test_decompressor_raw_1(self):
+        lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_1)
+        self._test_decompressor(lzd, COMPRESSED_RAW_1, lzma.CHECK_NONE)
+
+    def test_decompressor_raw_2(self):
+        lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_2)
+        self._test_decompressor(lzd, COMPRESSED_RAW_2, lzma.CHECK_NONE)
+
+    def test_decompressor_raw_3(self):
+        lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_3)
+        self._test_decompressor(lzd, COMPRESSED_RAW_3, lzma.CHECK_NONE)
+
+    def test_decompressor_raw_4(self):
+        lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+        self._test_decompressor(lzd, COMPRESSED_RAW_4, lzma.CHECK_NONE)
+
+    def test_decompressor_chunks(self):
+        lzd = LZMADecompressor()
+        out = []
+        for i in range(0, len(COMPRESSED_XZ), 10):
+            self.assertFalse(lzd.eof)
+            out.append(lzd.decompress(COMPRESSED_XZ[i:i+10]))
+        out = b"".join(out)
+        self.assertEqual(out, INPUT)
+        self.assertEqual(lzd.check, lzma.CHECK_CRC64)
+        self.assertTrue(lzd.eof)
+        self.assertEqual(lzd.unused_data, b"")
+
+    def test_decompressor_unused_data(self):
+        lzd = LZMADecompressor()
+        extra = b"fooblibar"
+        self._test_decompressor(lzd, COMPRESSED_XZ + extra, lzma.CHECK_CRC64,
+                                unused_data=extra)
+
+    def test_decompressor_bad_input(self):
+        lzd = LZMADecompressor()
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_RAW_1)
+
+        lzd = LZMADecompressor(lzma.FORMAT_XZ)
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_ALONE)
+
+        lzd = LZMADecompressor(lzma.FORMAT_ALONE)
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ)
+
+        lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_1)
+        self.assertRaises(LZMAError, lzd.decompress, COMPRESSED_XZ)
+
+    # Test that LZMACompressor->LZMADecompressor preserves the input data.
+
+    def test_roundtrip_xz(self):
+        lzc = LZMACompressor()
+        cdata = lzc.compress(INPUT) + lzc.flush()
+        lzd = LZMADecompressor()
+        self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64)
+
+    def test_roundtrip_alone(self):
+        lzc = LZMACompressor(lzma.FORMAT_ALONE)
+        cdata = lzc.compress(INPUT) + lzc.flush()
+        lzd = LZMADecompressor()
+        self._test_decompressor(lzd, cdata, lzma.CHECK_NONE)
+
+    def test_roundtrip_raw(self):
+        lzc = LZMACompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+        cdata = lzc.compress(INPUT) + lzc.flush()
+        lzd = LZMADecompressor(lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+        self._test_decompressor(lzd, cdata, lzma.CHECK_NONE)
+
+    def test_roundtrip_chunks(self):
+        lzc = LZMACompressor()
+        cdata = []
+        for i in range(0, len(INPUT), 10):
+            cdata.append(lzc.compress(INPUT[i:i+10]))
+        cdata.append(lzc.flush())
+        cdata = b"".join(cdata)
+        lzd = LZMADecompressor()
+        self._test_decompressor(lzd, cdata, lzma.CHECK_CRC64)
+
+    # LZMADecompressor intentionally does not handle concatenated streams.
+
+    def test_decompressor_multistream(self):
+        lzd = LZMADecompressor()
+        self._test_decompressor(lzd, COMPRESSED_XZ + COMPRESSED_ALONE,
+                                lzma.CHECK_CRC64, unused_data=COMPRESSED_ALONE)
+
+    # Test with inputs larger than 4GiB.
+
+    @bigmemtest(size=_4G + 100, memuse=2)
+    def test_compressor_bigmem(self, size):
+        lzc = LZMACompressor()
+        cdata = lzc.compress(b"x" * size) + lzc.flush()
+        ddata = lzma.decompress(cdata)
+        try:
+            self.assertEqual(len(ddata), size)
+            self.assertEqual(len(ddata.strip(b"x")), 0)
+        finally:
+            ddata = None
+
+    @bigmemtest(size=_4G + 100, memuse=3)
+    def test_decompressor_bigmem(self, size):
+        lzd = LZMADecompressor()
+        blocksize = 10 * 1024 * 1024
+        block = random.getrandbits(blocksize * 8).to_bytes(blocksize, "little")
+        try:
+            input = block * (size // blocksize + 1)
+            cdata = lzma.compress(input)
+            ddata = lzd.decompress(cdata)
+            self.assertEqual(ddata, input)
+        finally:
+            input = cdata = ddata = None
+
+
+class CompressDecompressFunctionTestCase(unittest.TestCase):
+
+    # Test error cases:
+
+    def test_bad_args(self):
+        self.assertRaises(TypeError, lzma.compress)
+        self.assertRaises(TypeError, lzma.compress, [])
+        self.assertRaises(TypeError, lzma.compress, b"", format="xz")
+        self.assertRaises(TypeError, lzma.compress, b"", check="none")
+        self.assertRaises(TypeError, lzma.compress, b"", preset="blah")
+        self.assertRaises(TypeError, lzma.compress, b"", filters=1024)
+        # Can't specify a preset and a custom filter chain at the same time.
+        with self.assertRaises(ValueError):
+            lzma.compress(b"", preset=3, filters=[{"id": lzma.FILTER_LZMA2}])
+
+        self.assertRaises(TypeError, lzma.decompress)
+        self.assertRaises(TypeError, lzma.decompress, [])
+        self.assertRaises(TypeError, lzma.decompress, b"", format="lzma")
+        self.assertRaises(TypeError, lzma.decompress, b"", memlimit=7.3e9)
+        with self.assertRaises(TypeError):
+            lzma.decompress(b"", format=lzma.FORMAT_RAW, filters={})
+        # Cannot specify a memory limit with FILTER_RAW.
+        with self.assertRaises(ValueError):
+            lzma.decompress(b"", format=lzma.FORMAT_RAW, memlimit=0x1000000)
+        # Can only specify a custom filter chain with FILTER_RAW.
+        with self.assertRaises(ValueError):
+            lzma.decompress(b"", filters=FILTERS_RAW_1)
+        with self.assertRaises(ValueError):
+            lzma.decompress(b"", format=lzma.FORMAT_XZ, filters=FILTERS_RAW_1)
+        with self.assertRaises(ValueError):
+            lzma.decompress(
+                    b"", format=lzma.FORMAT_ALONE, filters=FILTERS_RAW_1)
+
+    def test_decompress_memlimit(self):
+        with self.assertRaises(LZMAError):
+            lzma.decompress(COMPRESSED_XZ, memlimit=1024)
+        with self.assertRaises(LZMAError):
+            lzma.decompress(
+                    COMPRESSED_XZ, format=lzma.FORMAT_XZ, memlimit=1024)
+        with self.assertRaises(LZMAError):
+            lzma.decompress(
+                    COMPRESSED_ALONE, format=lzma.FORMAT_ALONE, memlimit=1024)
+
+    # Test LZMADecompressor on known-good input data.
+
+    def test_decompress_good_input(self):
+        ddata = lzma.decompress(COMPRESSED_XZ)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(COMPRESSED_ALONE)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(COMPRESSED_XZ, lzma.FORMAT_XZ)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(COMPRESSED_ALONE, lzma.FORMAT_ALONE)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(
+                COMPRESSED_RAW_1, lzma.FORMAT_RAW, filters=FILTERS_RAW_1)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(
+                COMPRESSED_RAW_2, lzma.FORMAT_RAW, filters=FILTERS_RAW_2)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(
+                COMPRESSED_RAW_3, lzma.FORMAT_RAW, filters=FILTERS_RAW_3)
+        self.assertEqual(ddata, INPUT)
+
+        ddata = lzma.decompress(
+                COMPRESSED_RAW_4, lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+        self.assertEqual(ddata, INPUT)
+
+    def test_decompress_incomplete_input(self):
+        self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_XZ[:128])
+        self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_ALONE[:128])
+        self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_1[:128],
+                          format=lzma.FORMAT_RAW, filters=FILTERS_RAW_1)
+        self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_2[:128],
+                          format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2)
+        self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_3[:128],
+                          format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3)
+        self.assertRaises(LZMAError, lzma.decompress, COMPRESSED_RAW_4[:128],
+                          format=lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+
+    def test_decompress_bad_input(self):
+        with self.assertRaises(LZMAError):
+            lzma.decompress(COMPRESSED_RAW_1)
+        with self.assertRaises(LZMAError):
+            lzma.decompress(COMPRESSED_ALONE, format=lzma.FORMAT_XZ)
+        with self.assertRaises(LZMAError):
+            lzma.decompress(COMPRESSED_XZ, format=lzma.FORMAT_ALONE)
+        with self.assertRaises(LZMAError):
+            lzma.decompress(COMPRESSED_XZ, format=lzma.FORMAT_RAW,
+                            filters=FILTERS_RAW_1)
+
+    # Test that compress()->decompress() preserves the input data.
+
+    def test_roundtrip(self):
+        cdata = lzma.compress(INPUT)
+        ddata = lzma.decompress(cdata)
+        self.assertEqual(ddata, INPUT)
+
+        cdata = lzma.compress(INPUT, lzma.FORMAT_XZ)
+        ddata = lzma.decompress(cdata)
+        self.assertEqual(ddata, INPUT)
+
+        cdata = lzma.compress(INPUT, lzma.FORMAT_ALONE)
+        ddata = lzma.decompress(cdata)
+        self.assertEqual(ddata, INPUT)
+
+        cdata = lzma.compress(INPUT, lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+        ddata = lzma.decompress(cdata, lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
+        self.assertEqual(ddata, INPUT)
+
+    # Unlike LZMADecompressor, decompress() *does* handle concatenated streams.
+
+    def test_decompress_multistream(self):
+        ddata = lzma.decompress(COMPRESSED_XZ + COMPRESSED_ALONE)
+        self.assertEqual(ddata, INPUT * 2)
+
+
+class TempFile:
+    """Context manager - creates a file, and deletes it on __exit__."""
+
+    def __init__(self, filename, data=b""):
+        self.filename = filename
+        self.data = data
+
+    def __enter__(self):
+        with open(self.filename, "wb") as f:
+            f.write(self.data)
+
+    def __exit__(self, *args):
+        unlink(self.filename)
+
+
+class FileTestCase(unittest.TestCase):
+
+    def test_init(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            pass
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            pass
+        with LZMAFile(fileobj=BytesIO(), mode="a") as f:
+            pass
+
+    def test_init_with_filename(self):
+        with TempFile(TESTFN, COMPRESSED_XZ):
+            with LZMAFile(TESTFN) as f:
+                pass
+            with LZMAFile(TESTFN, "w") as f:
+                pass
+            with LZMAFile(TESTFN, "a") as f:
+                pass
+
+    def test_init_bad_mode(self):
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode=(3, "x"))
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="")
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="x")
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="rb")
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="r+")
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="wb")
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="w+")
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="rw")
+
+    def test_init_bad_check(self):
+        with self.assertRaises(TypeError):
+            LZMAFile(fileobj=BytesIO(), mode="w", check=b"asd")
+        # CHECK_UNKNOWN and anything above CHECK_ID_MAX should be invalid.
+        with self.assertRaises(LZMAError):
+            LZMAFile(fileobj=BytesIO(), mode="w", check=lzma.CHECK_UNKNOWN)
+        with self.assertRaises(LZMAError):
+            LZMAFile(fileobj=BytesIO(), mode="w", check=lzma.CHECK_ID_MAX + 3)
+        # Cannot specify a check with mode="r".
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_NONE)
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_CRC32)
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_CRC64)
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_SHA256)
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), check=lzma.CHECK_UNKNOWN)
+
+    def test_init_bad_preset(self):
+        with self.assertRaises(TypeError):
+            LZMAFile(fileobj=BytesIO(), mode="w", preset=4.39)
+        with self.assertRaises(LZMAError):
+            LZMAFile(fileobj=BytesIO(), mode="w", preset=10)
+        with self.assertRaises(LZMAError):
+            LZMAFile(fileobj=BytesIO(), mode="w", preset=23)
+        with self.assertRaises(OverflowError):
+            LZMAFile(fileobj=BytesIO(), mode="w", preset=-1)
+        with self.assertRaises(OverflowError):
+            LZMAFile(fileobj=BytesIO(), mode="w", preset=-7)
+        with self.assertRaises(TypeError):
+            LZMAFile(fileobj=BytesIO(), mode="w", preset="foo")
+        # Cannot specify a preset with mode="r".
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), preset=3)
+
+    def test_init_bad_filter_spec(self):
+        with self.assertRaises(TypeError):
+            LZMAFile(fileobj=BytesIO(), mode="w", filters=[b"wobsite"])
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(), mode="w", filters=[{"xyzzy": 3}])
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(), mode="w", filters=[{"id": 98765}])
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(), mode="w",
+                     filters=[{"id": lzma.FILTER_LZMA2, "foo": 0}])
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(), mode="w",
+                     filters=[{"id": lzma.FILTER_DELTA, "foo": 0}])
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(), mode="w",
+                     filters=[{"id": lzma.FILTER_X86, "foo": 0}])
+
+    def test_init_with_preset_and_filters(self):
+        with self.assertRaises(ValueError):
+            LZMAFile(fileobj=BytesIO(), mode="w", format=lzma.FORMAT_RAW,
+                     preset=6,  filters=FILTERS_RAW_1)
+
+    def test_init_with_filename_and_fileobj(self):
+        with self.assertRaises(ValueError):
+            LZMAFile("/dev/null", fileobj=BytesIO())
+
+    def test_close(self):
+        with BytesIO(COMPRESSED_XZ) as src:
+            f = LZMAFile(fileobj=src)
+            f.close()
+            # LZMAFile.close() should not close the underlying file object.
+            self.assertFalse(src.closed)
+            # Try closing an already-closed LZMAFile.
+            f.close()
+            self.assertFalse(src.closed)
+
+        # Test with a real file on disk, opened directly by LZMAFile.
+        with TempFile(TESTFN, COMPRESSED_XZ):
+            f = LZMAFile(TESTFN)
+            fp = f._fp
+            f.close()
+            # Here, LZMAFile.close() *should* close the underlying file object.
+            self.assertTrue(fp.closed)
+            # Try closing an already-closed LZMAFile.
+            f.close()
+
+    def test_closed(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        try:
+            self.assertFalse(f.closed)
+            f.read()
+            self.assertFalse(f.closed)
+        finally:
+            f.close()
+        self.assertTrue(f.closed)
+
+        f = LZMAFile(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertFalse(f.closed)
+        finally:
+            f.close()
+        self.assertTrue(f.closed)
+
+    def test_fileno(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        try:
+            self.assertRaises(UnsupportedOperation, f.fileno)
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.fileno)
+        with TempFile(TESTFN, COMPRESSED_XZ):
+            f = LZMAFile(TESTFN)
+            try:
+                self.assertEqual(f.fileno(), f._fp.fileno())
+                self.assertIsInstance(f.fileno(), int)
+            finally:
+                f.close()
+        self.assertRaises(ValueError, f.fileno)
+
+    def test_seekable(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        try:
+            self.assertTrue(f.seekable())
+            f.read()
+            self.assertTrue(f.seekable())
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.seekable)
+
+        f = LZMAFile(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertFalse(f.seekable())
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.seekable)
+
+    def test_readable(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        try:
+            self.assertTrue(f.readable())
+            f.read()
+            self.assertTrue(f.readable())
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.readable)
+
+        f = LZMAFile(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertFalse(f.readable())
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.readable)
+
+    def test_writable(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        try:
+            self.assertFalse(f.writable())
+            f.read()
+            self.assertFalse(f.writable())
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.writable)
+
+        f = LZMAFile(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertTrue(f.writable())
+        finally:
+            f.close()
+        self.assertRaises(ValueError, f.writable)
+
+    def test_read(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE)) as f:
+            self.assertEqual(f.read(), INPUT)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ),
+                      format=lzma.FORMAT_XZ) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE),
+                      format=lzma.FORMAT_ALONE) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_1),
+                      format=lzma.FORMAT_RAW, filters=FILTERS_RAW_1) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_2),
+                      format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_3),
+                      format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_4),
+                      format=lzma.FORMAT_RAW, filters=FILTERS_RAW_4) as f:
+            self.assertEqual(f.read(), INPUT)
+            self.assertEqual(f.read(), b"")
+
+    def test_read_0(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertEqual(f.read(0), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE)) as f:
+            self.assertEqual(f.read(0), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ),
+                      format=lzma.FORMAT_XZ) as f:
+            self.assertEqual(f.read(0), b"")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE),
+                      format=lzma.FORMAT_ALONE) as f:
+            self.assertEqual(f.read(0), b"")
+
+    def test_read_10(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            chunks = []
+            while True:
+                result = f.read(10)
+                if not result:
+                    break
+                self.assertLessEqual(len(result), 10)
+                chunks.append(result)
+            self.assertEqual(b"".join(chunks), INPUT)
+
+    def test_read_multistream(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 5)) as f:
+            self.assertEqual(f.read(), INPUT * 5)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ + COMPRESSED_ALONE)) as f:
+            self.assertEqual(f.read(), INPUT * 2)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_3 * 4),
+                      format=lzma.FORMAT_RAW, filters=FILTERS_RAW_3) as f:
+            self.assertEqual(f.read(), INPUT * 4)
+
+    def test_read_multistream_buffer_size_aligned(self):
+        # Test the case where a stream boundary coincides with the end
+        # of the raw read buffer.
+        saved_buffer_size = lzma._BUFFER_SIZE
+        lzma._BUFFER_SIZE = len(COMPRESSED_XZ)
+        try:
+            with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ *  5)) as f:
+                self.assertEqual(f.read(), INPUT * 5)
+        finally:
+            lzma._BUFFER_SIZE = saved_buffer_size
+
+    def test_read_from_file(self):
+        with TempFile(TESTFN, COMPRESSED_XZ):
+            with LZMAFile(TESTFN) as f:
+                self.assertEqual(f.read(), INPUT)
+                self.assertEqual(f.read(), b"")
+
+    def test_read_incomplete(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ[:128])) as f:
+            self.assertRaises(EOFError, f.read)
+
+    def test_read_bad_args(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        f.close()
+        self.assertRaises(ValueError, f.read)
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            self.assertRaises(ValueError, f.read)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertRaises(TypeError, f.read, None)
+
+    def test_read1(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            blocks = []
+            while True:
+                result = f.read1()
+                if not result:
+                    break
+                blocks.append(result)
+            self.assertEqual(b"".join(blocks), INPUT)
+            self.assertEqual(f.read1(), b"")
+
+    def test_read1_0(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertEqual(f.read1(0), b"")
+
+    def test_read1_10(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            blocks = []
+            while True:
+                result = f.read1(10)
+                if not result:
+                    break
+                blocks.append(result)
+            self.assertEqual(b"".join(blocks), INPUT)
+            self.assertEqual(f.read1(), b"")
+
+    def test_read1_multistream(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 5)) as f:
+            blocks = []
+            while True:
+                result = f.read1()
+                if not result:
+                    break
+                blocks.append(result)
+            self.assertEqual(b"".join(blocks), INPUT * 5)
+            self.assertEqual(f.read1(), b"")
+
+    def test_read1_bad_args(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        f.close()
+        self.assertRaises(ValueError, f.read1)
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            self.assertRaises(ValueError, f.read1)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertRaises(TypeError, f.read1, None)
+
+    def test_peek(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            result = f.peek()
+            self.assertGreater(len(result), 0)
+            self.assertTrue(INPUT.startswith(result))
+            self.assertEqual(f.read(), INPUT)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            result = f.peek(10)
+            self.assertGreater(len(result), 0)
+            self.assertTrue(INPUT.startswith(result))
+            self.assertEqual(f.read(), INPUT)
+
+    def test_peek_bad_args(self):
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            self.assertRaises(ValueError, f.peek)
+
+    def test_iterator(self):
+        with BytesIO(INPUT) as f:
+            lines = f.readlines()
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertListEqual(list(iter(f)), lines)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE)) as f:
+            self.assertListEqual(list(iter(f)), lines)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ),
+                      format=lzma.FORMAT_XZ) as f:
+            self.assertListEqual(list(iter(f)), lines)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_ALONE),
+                      format=lzma.FORMAT_ALONE) as f:
+            self.assertListEqual(list(iter(f)), lines)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_RAW_2),
+                      format=lzma.FORMAT_RAW, filters=FILTERS_RAW_2) as f:
+            self.assertListEqual(list(iter(f)), lines)
+
+    def test_readline(self):
+        with BytesIO(INPUT) as f:
+            lines = f.readlines()
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            for line in lines:
+                self.assertEqual(f.readline(), line)
+
+    def test_readlines(self):
+        with BytesIO(INPUT) as f:
+            lines = f.readlines()
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertListEqual(f.readlines(), lines)
+
+    def test_write(self):
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w") as f:
+                f.write(INPUT)
+            expected = lzma.compress(INPUT)
+            self.assertEqual(dst.getvalue(), expected)
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w", format=lzma.FORMAT_XZ) as f:
+                f.write(INPUT)
+            expected = lzma.compress(INPUT, format=lzma.FORMAT_XZ)
+            self.assertEqual(dst.getvalue(), expected)
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w", format=lzma.FORMAT_ALONE) as f:
+                f.write(INPUT)
+            expected = lzma.compress(INPUT, format=lzma.FORMAT_ALONE)
+            self.assertEqual(dst.getvalue(), expected)
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w", format=lzma.FORMAT_RAW,
+                          filters=FILTERS_RAW_2) as f:
+                f.write(INPUT)
+            expected = lzma.compress(INPUT, format=lzma.FORMAT_RAW,
+                                     filters=FILTERS_RAW_2)
+            self.assertEqual(dst.getvalue(), expected)
+
+    def test_write_10(self):
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w") as f:
+                for start in range(0, len(INPUT), 10):
+                    f.write(INPUT[start:start+10])
+            expected = lzma.compress(INPUT)
+            self.assertEqual(dst.getvalue(), expected)
+
+    def test_write_append(self):
+        part1 = INPUT[:1024]
+        part2 = INPUT[1024:1536]
+        part3 = INPUT[1536:]
+        expected = b"".join(lzma.compress(x) for x in (part1, part2, part3))
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w") as f:
+                f.write(part1)
+            with LZMAFile(fileobj=dst, mode="a") as f:
+                f.write(part2)
+            with LZMAFile(fileobj=dst, mode="a") as f:
+                f.write(part3)
+            self.assertEqual(dst.getvalue(), expected)
+
+    def test_write_to_file(self):
+        try:
+            with LZMAFile(TESTFN, "w") as f:
+                f.write(INPUT)
+            expected = lzma.compress(INPUT)
+            with open(TESTFN, "rb") as f:
+                self.assertEqual(f.read(), expected)
+        finally:
+            unlink(TESTFN)
+
+    def test_write_append_to_file(self):
+        part1 = INPUT[:1024]
+        part2 = INPUT[1024:1536]
+        part3 = INPUT[1536:]
+        expected = b"".join(lzma.compress(x) for x in (part1, part2, part3))
+        try:
+            with LZMAFile(TESTFN, "w") as f:
+                f.write(part1)
+            with LZMAFile(TESTFN, "a") as f:
+                f.write(part2)
+            with LZMAFile(TESTFN, "a") as f:
+                f.write(part3)
+            with open(TESTFN, "rb") as f:
+                self.assertEqual(f.read(), expected)
+        finally:
+            unlink(TESTFN)
+
+    def test_write_bad_args(self):
+        f = LZMAFile(fileobj=BytesIO(), mode="w")
+        f.close()
+        self.assertRaises(ValueError, f.write, b"foo")
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ), mode="r") as f:
+            self.assertRaises(ValueError, f.write, b"bar")
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            self.assertRaises(TypeError, f.write, None)
+            self.assertRaises(TypeError, f.write, "text")
+            self.assertRaises(TypeError, f.write, 789)
+
+    def test_writelines(self):
+        with BytesIO(INPUT) as f:
+            lines = f.readlines()
+        with BytesIO() as dst:
+            with LZMAFile(fileobj=dst, mode="w") as f:
+                f.writelines(lines)
+            expected = lzma.compress(INPUT)
+            self.assertEqual(dst.getvalue(), expected)
+
+    def test_seek_forward(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.seek(555)
+            self.assertEqual(f.read(), INPUT[555:])
+
+    def test_seek_forward_across_streams(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 2)) as f:
+            f.seek(len(INPUT) + 123)
+            self.assertEqual(f.read(), INPUT[123:])
+
+    def test_seek_forward_relative_to_current(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.read(100)
+            f.seek(1236, 1)
+            self.assertEqual(f.read(), INPUT[1336:])
+
+    def test_seek_forward_relative_to_end(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.seek(-555, 2)
+            self.assertEqual(f.read(), INPUT[-555:])
+
+    def test_seek_backward(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.read(1001)
+            f.seek(211)
+            self.assertEqual(f.read(), INPUT[211:])
+
+    def test_seek_backward_across_streams(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ * 2)) as f:
+            f.read(len(INPUT) + 333)
+            f.seek(737)
+            self.assertEqual(f.read(), INPUT[737:] + INPUT)
+
+    def test_seek_backward_relative_to_end(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.seek(-150, 2)
+            self.assertEqual(f.read(), INPUT[-150:])
+
+    def test_seek_past_end(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.seek(len(INPUT) + 9001)
+            self.assertEqual(f.tell(), len(INPUT))
+            self.assertEqual(f.read(), b"")
+
+    def test_seek_past_start(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            f.seek(-88)
+            self.assertEqual(f.tell(), 0)
+            self.assertEqual(f.read(), INPUT)
+
+    def test_seek_bad_args(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        f.close()
+        self.assertRaises(ValueError, f.seek, 0)
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            self.assertRaises(ValueError, f.seek, 0)
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            self.assertRaises(ValueError, f.seek, 0, 3)
+            self.assertRaises(ValueError, f.seek, 9, ())
+            self.assertRaises(TypeError, f.seek, None)
+            self.assertRaises(TypeError, f.seek, b"derp")
+
+    def test_tell(self):
+        with LZMAFile(fileobj=BytesIO(COMPRESSED_XZ)) as f:
+            pos = 0
+            while True:
+                self.assertEqual(f.tell(), pos)
+                result = f.read(183)
+                if not result:
+                    break
+                pos += len(result)
+            self.assertEqual(f.tell(), len(INPUT))
+        with LZMAFile(fileobj=BytesIO(), mode="w") as f:
+            for pos in range(0, len(INPUT), 144):
+                self.assertEqual(f.tell(), pos)
+                f.write(INPUT[pos:pos+144])
+            self.assertEqual(f.tell(), len(INPUT))
+
+    def test_tell_bad_args(self):
+        f = LZMAFile(fileobj=BytesIO(COMPRESSED_XZ))
+        f.close()
+        self.assertRaises(ValueError, f.tell)
+
+
+class MiscellaneousTestCase(unittest.TestCase):
+
+    def test_is_check_supported(self):
+        # CHECK_NONE and CHECK_CRC32 should always be supported,
+        # regardless of the options liblzma was compiled with.
+        self.assertTrue(lzma.check_is_supported(lzma.CHECK_NONE))
+        self.assertTrue(lzma.check_is_supported(lzma.CHECK_CRC32))
+
+        # The .xz format spec cannot store check IDs above this value.
+        self.assertFalse(lzma.check_is_supported(lzma.CHECK_ID_MAX + 1))
+
+        # This value should not be a valid check ID.
+        self.assertFalse(lzma.check_is_supported(lzma.CHECK_UNKNOWN))
+
+
+# Test data:
+
+INPUT = b"""
+LAERTES
+
+       O, fear me not.
+       I stay too long: but here my father comes.
+
+       Enter POLONIUS
+
+       A double blessing is a double grace,
+       Occasion smiles upon a second leave.
+
+LORD POLONIUS
+
+       Yet here, Laertes! aboard, aboard, for shame!
+       The wind sits in the shoulder of your sail,
+       And you are stay'd for. There; my blessing with thee!
+       And these few precepts in thy memory
+       See thou character. Give thy thoughts no tongue,
+       Nor any unproportioned thought his act.
+       Be thou familiar, but by no means vulgar.
+       Those friends thou hast, and their adoption tried,
+       Grapple them to thy soul with hoops of steel;
+       But do not dull thy palm with entertainment
+       Of each new-hatch'd, unfledged comrade. Beware
+       Of entrance to a quarrel, but being in,
+       Bear't that the opposed may beware of thee.
+       Give every man thy ear, but few thy voice;
+       Take each man's censure, but reserve thy judgment.
+       Costly thy habit as thy purse can buy,
+       But not express'd in fancy; rich, not gaudy;
+       For the apparel oft proclaims the man,
+       And they in France of the best rank and station
+       Are of a most select and generous chief in that.
+       Neither a borrower nor a lender be;
+       For loan oft loses both itself and friend,
+       And borrowing dulls the edge of husbandry.
+       This above all: to thine ownself be true,
+       And it must follow, as the night the day,
+       Thou canst not then be false to any man.
+       Farewell: my blessing season this in thee!
+
+LAERTES
+
+       Most humbly do I take my leave, my lord.
+
+LORD POLONIUS
+
+       The time invites you; go; your servants tend.
+
+LAERTES
+
+       Farewell, Ophelia; and remember well
+       What I have said to you.
+
+OPHELIA
+
+       'Tis in my memory lock'd,
+       And you yourself shall keep the key of it.
+
+LAERTES
+
+       Farewell.
+"""
+
+COMPRESSED_XZ = (
+    b"\xfd7zXZ\x00\x00\x04\xe6\xd6\xb4F\x02\x00!\x01\x16\x00\x00\x00t/\xe5\xa3"
+    b"\xe0\x07\x80\x03\xdf]\x00\x05\x14\x07bX\x19\xcd\xddn\x98\x15\xe4\xb4\x9d"
+    b"o\x1d\xc4\xe5\n\x03\xcc2h\xc7\\\x86\xff\xf8\xe2\xfc\xe7\xd9\xfe6\xb8("
+    b"\xa8wd\xc2\"u.n\x1e\xc3\xf2\x8e\x8d\x8f\x02\x17/\xa6=\xf0\xa2\xdf/M\x89"
+    b"\xbe\xde\xa7\x1cz\x18-]\xd5\xef\x13\x8frZ\x15\x80\x8c\xf8\x8do\xfa\x12"
+    b"\x9b#z/\xef\xf0\xfaF\x01\x82\xa3M\x8e\xa1t\xca6 BF$\xe5Q\xa4\x98\xee\xde"
+    b"l\xe8\x7f\xf0\x9d,bn\x0b\x13\xd4\xa8\x81\xe4N\xc8\x86\x153\xf5x2\xa2O"
+    b"\x13 at Q\xa1\x00/\xa5\xd0O\x97\xdco\xae\xf7z\xc4\xcdS\xb6t<\x16\xf2\x9cI#"
+    b"\x89ud\xc66Y\xd9\xee\xe6\xce\x12]\xe5\xf0\xaa\x96-Pe\xade:\x04\t\x1b\xf7"
+    b"\xdb7\n\x86\x1fp\xc8J\xba\xf4\xf0V\xa9\xdc\xf0\x02%G\xf9\xdf=?\x15\x1b"
+    b"\xe1(\xce\x82=\xd6I\xac3\x12\x0cR\xb7\xae\r\xb1i\x03\x95\x01\xbd\xbe\xfa"
+    b"\x02s\x01P\x9d\x96X\xb12j\xc8L\xa8\x84b\xf6\xc3\xd4c-H\x93oJl\xd0iQ\xe4k"
+    b"\x84\x0b\xc1\xb7\xbc\xb1\x17\x88\xb1\xca?@\xf6\x07\xea\xe6x\xf1H12P\x0f"
+    b"\x8a\xc9\xeauw\xe3\xbe\xaai\xa9W\xd0\x80\xcd#cb5\x99\xd8]\xa9d\x0c\xbd"
+    b"\xa2\xdcWl\xedUG\xbf\x89yF\xf77\x81v\xbd5\x98\xbeh8\x18W\x08\xf0\x1b\x99"
+    b"5:\x1a?rD\x96\xa1\x04\x0f\xae\xba\x85\xeb\x9d5@\xf5\x83\xd37\x83\x8ac"
+    b"\x06\xd4\x97i\xcdt\x16S\x82k\xf6K\x01vy\x88\x91\x9b6T\xdae\r\xfd]:k\xbal"
+    b"\xa9\xbba\xc34\xf9r\xeb}r\xdb\xc7\xdb*\x8f\x03z\xdc8h\xcc\xc9\xd3\xbcl"
+    b"\xa5-\xcb\xeaK\xa2\xc5\x15\xc0\xe3\xc1\x86Z\xfb\xebL\xe13\xcf\x9c\xe3"
+    b"\x1d\xc9\xed\xc2\x06\xcc\xce!\x92\xe5\xfe\x9c^\xa59w \x9bP\xa3PK\x08d"
+    b"\xf9\xe2Z}\xa7\xbf\xed\xeb%$\x0c\x82\xb8/\xb0\x01\xa9&,\xf7qh{Q\x96)\xf2"
+    b"q\x96\xc3\x80\xb4\x12\xb0\xba\xe6o\xf4!\xb4[\xd4\x8aw\x10\xf7t\x0c\xb3"
+    b"\xd9\xd5\xc3`^\x81\x11??\\\xa4\x99\x85R\xd4\x8e\x83\xc9\x1eX\xbfa\xf1"
+    b"\xac\xb0\xea\xea\xd7\xd0\xab\x18\xe2\xf2\xed\xe1\xb7\xc9\x18\xcbS\xe4>"
+    b"\xc9\x95H\xe8\xcb\t\r%\xeb\xc7$.o\xf1\xf3R\x17\x1db\xbb\xd8U\xa5^\xccS"
+    b"\x16\x01\x87\xf3/\x93\xd1\xf0v\xc0r\xd7\xcc\xa2Gkz\xca\x80\x0e\xfd\xd0"
+    b"\x8b\xbb\xd2Ix\xb3\x1ey\xca-0\xe3z^\xd6\xd6\x8f_\xf1\x9dP\x9fi\xa7\xd1"
+    b"\xe8\x90\x84\xdc\xbf\xcdky\x8e\xdc\x81\x7f\xa3\xb2+\xbf\x04\xef\xd8\\"
+    b"\xc4\xdf\xe1\xb0\x01\xe9\x93\xe3Y\xf1\x1dY\xe8h\x81\xcf\xf1w\xcc\xb4\xef"
+    b" \x8b|\x04\xea\x83ej\xbe\x1f\xd4z\x9c`\xd3\x1a\x92A\x06\xe5\x8f\xa9\x13"
+    b"\t\x9e=\xfa\x1c\xe5_\x9f%v\x1bo\x11ZO\xd8\xf4\t\xddM\x16-\x04\xfc\x18<\""
+    b"CM\xddg~b\xf6\xef\x8e\x0c\xd0\xde|\xa0'\x8a\x0c\xd6x\xae!J\xa6F\x88\x15u"
+    b"\x008\x17\xbc7y\xb3\xd8u\xac_\x85\x8d\xe7\xc1@\x9c\xecqc\xa3#\xad\xf1"
+    b"\x935\xb5)_\r\xec3]\x0fo]5\xd0my\x07\x9b\xee\x81\xb5\x0f\xcfK+\x00\xc0"
+    b"\xe4b\x10\xe4\x0c\x1a \x9b\xe0\x97t\xf6\xa1\x9e\x850\xba\x0c\x9a\x8d\xc8"
+    b"\x8f\x07\xd7\xae\xc8\xf9+i\xdc\xb9k\xb0>f\x19\xb8\r\xa8\xf8\x1f$\xa5{p"
+    b"\xc6\x880\xce\xdb\xcf\xca_\x86\xac\x88h6\x8bZ%'\xd0\n\xbf\x0f\x9c\"\xba"
+    b"\xe5\x86\x9f\x0f7X=mNX[\xcc\x19FU\xc9\x860\xbc\x90a+* \xae_$\x03\x1e\xd3"
+    b"\xcd_\xa0\x9c\xde\xaf46q\xa5\xc9\x92\xd7\xca\xe3`\x9d\x85}\xb4\xff\xb3"
+    b"\x83\xfb\xb6\xca\xae`\x0bw\x7f\xfc\xd8\xacVe\x19\xc8\x17\x0bZ\xad\x88"
+    b"\xeb#\x97\x03\x13\xb1d\x0f{\x0c\x04w\x07\r\x97\xbd\xd6\xc1\xc3B:\x95\x08"
+    b"^\x10V\xaeaH\x02\xd9\xe3\n\\\x01X\xf6\x9c\x8a\x06u#%\xbe*\xa1\x18v\x85"
+    b"\xec!\t4\x00\x00\x00\x00Vj?uLU\xf3\xa6\x00\x01\xfb\x07\x81\x0f\x00\x00tw"
+    b"\x99P\xb1\xc4g\xfb\x02\x00\x00\x00\x00\x04YZ"
+)
+
+COMPRESSED_ALONE = (
+    b"]\x00\x00\x80\x00\xff\xff\xff\xff\xff\xff\xff\xff\x00\x05\x14\x07bX\x19"
+    b"\xcd\xddn\x98\x15\xe4\xb4\x9do\x1d\xc4\xe5\n\x03\xcc2h\xc7\\\x86\xff\xf8"
+    b"\xe2\xfc\xe7\xd9\xfe6\xb8(\xa8wd\xc2\"u.n\x1e\xc3\xf2\x8e\x8d\x8f\x02"
+    b"\x17/\xa6=\xf0\xa2\xdf/M\x89\xbe\xde\xa7\x1cz\x18-]\xd5\xef\x13\x8frZ"
+    b"\x15\x80\x8c\xf8\x8do\xfa\x12\x9b#z/\xef\xf0\xfaF\x01\x82\xa3M\x8e\xa1t"
+    b"\xca6 BF$\xe5Q\xa4\x98\xee\xdel\xe8\x7f\xf0\x9d,bn\x0b\x13\xd4\xa8\x81"
+    b"\xe4N\xc8\x86\x153\xf5x2\xa2O\x13 at Q\xa1\x00/\xa5\xd0O\x97\xdco\xae\xf7z"
+    b"\xc4\xcdS\xb6t<\x16\xf2\x9cI#\x89ud\xc66Y\xd9\xee\xe6\xce\x12]\xe5\xf0"
+    b"\xaa\x96-Pe\xade:\x04\t\x1b\xf7\xdb7\n\x86\x1fp\xc8J\xba\xf4\xf0V\xa9"
+    b"\xdc\xf0\x02%G\xf9\xdf=?\x15\x1b\xe1(\xce\x82=\xd6I\xac3\x12\x0cR\xb7"
+    b"\xae\r\xb1i\x03\x95\x01\xbd\xbe\xfa\x02s\x01P\x9d\x96X\xb12j\xc8L\xa8"
+    b"\x84b\xf8\x1epl\xeajr\xd1=\t\x03\xdd\x13\x1b3!E\xf9vV\xdaF\xf3\xd7\xb4"
+    b"\x0c\xa9P~\xec\xdeE\xe37\xf6\x1d\xc6\xbb\xddc%\xb6\x0fI\x07\xf0;\xaf\xe7"
+    b"\xa0\x8b\xa7Z\x99(\xe9\xe2\xf0o\x18>`\xe1\xaa\xa8\xd9\xa1\xb2}\xe7\x8d"
+    b"\x834T\xb6\xef\xc1\xde\xe3\x98\xbcD\x03MA@\xd8\xed\xdc\xc8\x93\x03\x1a"
+    b"\x93\x0b\x7f\x94\x12\x0b\x02Sa\x18\xc9\xc5\x9bTJE}\xf6\xc8g\x17#ZV\x01"
+    b"\xc9\x9dc\x83\x0e>0\x16\x90S\xb8/\x03y_\x18\xfa(\xd7\x0br\xa2\xb0\xba?"
+    b"\x8c\xe6\x83@\x84\xdf\x02:\xc5z\x9e\xa6\x84\xc9\xf5BeyX\x83\x1a\xf1 :\t"
+    b"\xf7\x19\xfexD\\&G\xf3\x85Y\xa2J\xf9\x0bv{\x89\xf6\xe7)A\xaf\x04o\x00"
+    b"\x075\xd3\xe0\x7f\x97\x98F\x0f?v\x93\xedVtTf\xb5\x97\x83\xed\x19\xd7\x1a"
+    b"'k\xd7\xd9\xc5\\Y\xd1\xdc\x07\x15|w\xbc\xacd\x87\x08d\xec\xa7\xf6\x82"
+    b"\xfc\xb3\x93\xeb\xb9 \x8d\xbc ,\xb3X\xb0\xd2s\xd7\xd1\xffv\x05\xdf}\xa2"
+    b"\x96\xfb%\n\xdf\xa2\x7f\x08.\xa16\n\xe0\x19\x93\x7fh\n\x1c\x8c\x0f \x11"
+    b"\xc6Bl\x95\x19U}\xe4s\xb5\x10H\xea\x86pB\xe88\x95\xbe\x8cZ\xdb\xe4\x94A"
+    b"\x92\xb9;z\xaa\xa7{\x1c5!\xc0\xaf\xc1A\xf9\xda\xf0$\xb0\x02qg\xc8\xc7/|"
+    b"\xafr\x99^\x91\x88\xbf\x03\xd9=\xd7n\xda6{>8\n\xc7:\xa9'\xba.\x0b\xe2"
+    b"\xb5\x1d\x0e\n\x9a\x8e\x06\x8f:\xdd\x82'[\xc3\"wD$\xa7w\xecq\x8c,1\x93"
+    b"\xd0,\xae2w\x93\x12$Jd\x19mg\x02\x93\x9cA\x95\x9d&\xca8i\x9c\xb0;\xe7NQ"
+    b"\x1frh\x8beL;\xb0m\xee\x07Q\x9b\xc6\xd8\x03\xb5\xdeN\xd4\xfe\x98\xd0\xdc"
+    b"\x1a[\x04\xde\x1a\xf6\x91j\xf8EOli\x8eB^\x1d\x82\x07\xb2\xb5R]\xb7\xd7"
+    b"\xe9\xa6\xc3.\xfb\xf0-\xb4e\x9b\xde\x03\x88\xc6\xc1iN\x0e\x84wbQ\xdf~"
+    b"\xe9\xa4\x884\x96kM\xbc)T\xf3\x89\x97\x0f\x143\xe7)\xa0\xb3B\x00\xa8\xaf"
+    b"\x82^\xcb\xc7..\xdb\xc7\t\x9dH\xee5\xe9#\xe6NV\x94\xcb$Kk\xe3\x7f\r\xe3t"
+    b"\x12\xcf'\xefR\x8b\xf42\xcf-LH\xac\xe5\x1f0~?SO\xeb\xc1E\x1a\x1c]\xf2"
+    b"\xc4<\x11\x02\x10Z0a*?\xe4r\xff\xfb\xff\xf6\x14nG\xead^\xd6\xef8\xb6uEI"
+    b"\x99\nV\xe2\xb3\x95\x8e\x83\xf6i!\xb5&1F\xb1DP\xf4 SO3D!w\x99_G\x7f+\x90"
+    b".\xab\xbb]\x91>\xc9#h;\x0f5J\x91K\xf4^-[\x9e\x8a\\\x94\xca\xaf\xf6\x19"
+    b"\xd4\xa1\x9b\xc4\xb8p\xa1\xae\x15\xe9r\x84\xe0\xcar.l []\x8b\xaf+0\xf2g"
+    b"\x01aKY\xdfI\xcf,\n\xe8\xf0\xe7V\x80_#\xb2\xf2\xa9\x06\x8c>w\xe2W,\xf4"
+    b"\x8c\r\xf963\xf5J\xcc2\x05=kT\xeaUti\xe5_\xce\x1b\xfa\x8dl\x02h\xef\xa8"
+    b"\xfbf\x7f\xff\xf0\x19\xeax"
+)
+
+FILTERS_RAW_1 = [{"id": lzma.FILTER_LZMA2, "preset": 3}]
+COMPRESSED_RAW_1 = (
+    b"\xe0\x07\x80\x03\xfd]\x00\x05\x14\x07bX\x19\xcd\xddn\x96cyq\xa1\xdd\xee"
+    b"\xf8\xfam\xe3'\x88\xd3\xff\xe4\x9e \xceQ\x91\xa4\x14I\xf6\xb9\x9dVL8\x15"
+    b"_\x0e\x12\xc3\xeb\xbc\xa5\xcd\nW\x1d$=R;\x1d\xf8k8\t\xb1{\xd4\xc5+\x9d"
+    b"\x87c\xe5\xef\x98\xb4\xd7S3\xcd\xcc\xd2\xed\xa4\x0em\xe5\xf4\xdd\xd0b"
+    b"\xbe4*\xaa\x0b\xc5\x08\x10\x85+\x81.\x17\xaf9\xc9b\xeaZrA\xe20\x7fs\"r"
+    b"\xdaG\x81\xde\x90cu\xa5\xdb\xa9.A\x08l\xb0<\xf6\x03\xddOi\xd0\xc5\xb4"
+    b"\xec\xecg4t6\"\xa6\xb8o\xb5?\x18^}\xb6}\x03[:\xeb\x03\xa9\n[\x89l\x19g"
+    b"\x16\xc82\xed\x0b\xfb\x86n\xa2\x857@\x93\xcd6T\xc3u\xb0\t\xf9\x1b\x918"
+    b"\xfc[\x1b\x1e4\xb3\x14\x06PCV\xa8\"\xf5\x81x~\xe9\xb5N\x9cK\x9f\xc6\xc3%"
+    b"\xc8k:{6\xe7\xf7\xbd\x05\x02\xb4\xc4\xc3\xd3\xfd\xc3\xa8\\\xfc@\xb1F_"
+    b"\xc8\x90\xd9sU\x98\xad8\x05\x07\xde7J\x8bM\xd0\xb3;X\xec\x87\xef\xae\xb3"
+    b"eO,\xb1z,d\x11y\xeejlB\x02\x1d\xf28\x1f#\x896\xce\x0b\xf0\xf5\xa9PK\x0f"
+    b"\xb3\x13P\xd8\x88\xd2\xa1\x08\x04C?\xdb\x94_\x9a\"\xe9\xe3e\x1d\xde\x9b"
+    b"\xa1\xe8>H\x98\x10;\xc5\x03#\xb5\x9d4\x01\xe7\xc5\xba%v\xa49\x97A\xe0\""
+    b"\x8c\xc22\xe3i\xc1\x9d\xab3\xdf\xbe\xfdDm7\x1b\x9d\xab\xb5\x15o:J\x92"
+    b"\xdb\x816\x17\xc2O\x99\x1b\x0e\x8d\xf3\tQ\xed\x8e\x95S/\x16M\xb2S\x04"
+    b"\x0f\xc3J\xc6\xc7\xe4\xcb\xc5\xf4\xe7d\x14\xe4=^B\xfb\xd3E\xd3\x1e\xcd"
+    b"\x91\xa5\xd0G\x8f.\xf6\xf9\x0bb&\xd9\x9f\xc2\xfdj\xa2\x9e\xc4\\\x0e\x1dC"
+    b"v\xe8\xd2\x8a?^H\xec\xae\xeb>\xfe\xb8\xab\xd4IqY\x8c\xd4K7\x11\xf4D\xd0W"
+    b"\xa5\xbe\xeaO\xbf\xd0\x04\xfdl\x10\xae5\xd4U\x19\x06\xf9{\xaa\xe0\x81"
+    b"\x0f\xcf\xa3k{\x95\xbd\x19\xa2\xf8\xe4\xa3\x08O*\xf1\xf1B-\xc7(\x0eR\xfd"
+    b"@E\x9f\xd3\x1e:\xfdV\xb7\x04Y\x94\xeb]\x83\xc4\xa5\xd7\xc0gX\x98\xcf\x0f"
+    b"\xcd3\x00]n\x17\xec\xbd\xa3Y\x86\xc5\xf3u\xf6*\xbdT\xedA$A\xd9A\xe7\x98"
+    b"\xef\x14\x02\x9a\xfdiw\xec\xa0\x87\x11\xd9%\xc5\xeb\x8a=\xae\xc0\xc4\xc6"
+    b"D\x80\x8f\xa8\xd1\xbbq\xb2\xc0\xa0\xf5Cqp\xeeL\xe3\xe5\xdc \x84\"\xe9"
+    b"\x80t\x83\x05\xba\xf1\xc5~\x93\xc9\xf0\x01c\xceix\x9d\xed\xc5)l\x16)\xd1"
+    b"\x03 at l\x04\x7f\x87\xa5yn\x1b\x01D\xaa:\xd2\x96\xb4\xb3?\xb0\xf9\xce\x07"
+    b"\xeb\x81\x00\xe4\xc3\xf5%_\xae\xd4\xf9\xeb\xe2\rh\xb2#\xd67Q\x16D\x82hn"
+    b"\xd1\xa3_?q\xf0\xe2\xac\xf317\x9e\xd0_\x83|\xf1\xca\xb7\x95S\xabW\x12"
+    b"\xff\xddt\xf69L\x01\xf2|\xdaW\xda\xees\x98L\x18\xb8_\xe8$\x82\xea\xd6"
+    b"\xd1F\xd4\x0b\xcdk\x01vf\x88h\xc3\xae\xb91\xc7Q\x9f\xa5G\xd9\xcc\x1f\xe3"
+    b"5\xb1\xdcy\x7fI\x8bcw\x8e\x10rIp\x02:\x19p_\xc8v\xcea\"\xc1\xd9\x91\x03"
+    b"\xbfe\xbe\xa6\xb3\xa8\x14\x18\xc3\xabH*m}\xc2\xc1\x9a}>l%\xce\x84\x99"
+    b"\xb3d\xaf\xd3\x82\x15\xdf\xc1\xfc5fOg\x9b\xfc\x8e^&\t@\xce\x9f\x06J\xb8"
+    b"\xb5\x86\x1d\xda{\x9f\xae\xb0\xff\x02\x81r\x92z\x8cM\xb7ho\xc9^\x9c\xb6"
+    b"\x9c\xae\xd1\xc9\xf4\xdfU7\xd6\\!\xea\x0b\x94k\xb9Ud~\x98\xe7\x86\x8az"
+    b"\x10;\xe3\x1d\xe5PG\xf8\xa4\x12\x05w\x98^\xc4\xb1\xbb\xfb\xcf\xe0\x7f"
+    b"\x033Sf\x0c \xb1\xf6@\x94\xe5\xa3\xb2\xa7\x10\x9a\xc0\x14\xc3s\xb5xRD"
+    b"\xf4`W\xd9\xe5\xd3\xcf\x91\rTZ-X\xbe\xbf\xb5\xe2\xee|\x1a\xbf\xfb\x08"
+    b"\x91\xe1\xfc\x9a\x18\xa3\x8b\xd6^\x89\xf5[\xef\x87\xd1\x06\x1c7\xd6\xa2"
+    b"\t\tQ5/@S\xc05\xd2VhAK\x03VC\r\x9b\x93\xd6M\xf1xO\xaaO\xed\xb9<\x0c\xdae"
+    b"*\xd0\x07Hk6\x9fG+\xa1)\xcd\x9cl\x87\xdb\xe1\xe7\xefK}\x875\xab\xa0\x19u"
+    b"\xf6*F\xb32\x00\x00\x00"
+)
+
+FILTERS_RAW_2 = [{"id": lzma.FILTER_DELTA, "dist": 2},
+                 {"id": lzma.FILTER_LZMA2,
+                  "preset": lzma.PRESET_DEFAULT | lzma.PRESET_EXTREME}]
+COMPRESSED_RAW_2 = (
+    b"\xe0\x07\x80\x05\x91]\x00\x05\x14\x06-\xd4\xa8d?\xef\xbe\xafH\xee\x042"
+    b"\xcb.\xb5g\x8f\xfb\x14\xab\xa5\x9f\x025z\xa4\xdd\xd8\t[}W\xf8\x0c\x1dmH"
+    b"\xfa\x05\xfcg\xba\xe5\x01Q\x0b\x83R\xb6A\x885\xc0\xba\xee\n\x1cv~\xde:o"
+    b"\x06:J\xa7\x11Cc\xea\xf7\xe5*o\xf7\x83\\l\xbdE\x19\x1f\r\xa8\x10\xb42"
+    b"\x0caU{\xd7\xb8w\xdc\xbe\x1b\xfc8\xb4\xcc\xd38\\\xf6\x13\xf6\xe7\x98\xfa"
+    b"\xc7[\x17_9\x86%\xa8\xf8\xaa\xb8\x8dfs#\x1e=\xed<\x92\x10\\t\xff\x86\xfb"
+    b"=\x9e7\x18\x1dft\\\xb5\x01\x95Q\xc5\x19\xb38\xe0\xd4\xaa\x07\xc3\x7f\xd8"
+    b"\xa2\x00>-\xd3\x8e\xa1#\xfa\x83ArAm\xdbJ~\x93\xa3B\x82\xe0\xc7\xcc(\x08`"
+    b"WK\xad\x1b\x94kaj\x04 \xde\xfc\xe1\xed\xb0\x82\x91\xefS\x84%\x86\xfbi"
+    b"\x99X\xf1B\xe7\x90;E\xfde\x98\xda\xca\xd6T\xb4bg\xa4\n\x9aj\xd1\x83\x9e]"
+    b"\"\x7fM\xb5\x0fr\xd2\\\xa5j~P\x10GH\xbfN*Z\x10.\x81\tpE\x8a\x08\xbe1\xbd"
+    b"\xcd\xa9\xe1\x8d\x1f\x04\xf9\x0eH\xb9\xae\xd6\xc3\xc1\xa5\xa9\x95P\xdc~"
+    b"\xff\x01\x930\xa9\x04\xf6\x03\xfe\xb5JK\xc3]\xdd9\xb1\xd3\xd7F\xf5\xd1"
+    b"\x1e\xa0\x1c_\xed[\x0c\xae\xd4\x8b\x946\xeb\xbf\xbb\xe3$kS{\xb5\x80,f:Sj"
+    b"\x0f\x08z\x1c\xf5\xe8\xe6\xae\x98\xb0Q~r\x0f\xb0\x05?\xb6\x90\x19\x02&"
+    b"\xcb\x80\t\xc4\xea\x9c|x\xce\x10\x9c\xc5|\xcbdhh+\x0c'\xc5\x81\xc33\xb5"
+    b"\x14q\xd6\xc5\xe3`Z#\xdc\x8a\xab\xdd\xea\x08\xc2I\xe7\x02l{\xec\x196\x06"
+    b"\x91\x8d\xdc\xd5\xb3x\xe1hz%\xd1\xf8\xa5\xdd\x98!\x8c\x1c\xc1\x17RUa\xbb"
+    b"\x95\x0f\xe4X\xea1\x0c\xf1=R\xbe\xc60\xe3\xa4\x9a\x90bd\x97$]B\x01\xdd"
+    b"\x1f\xe3h2c\x1e\xa0L`4\xc6x\xa3Z\x8a\r\x14]T^\xd8\x89\x1b\x92\r;\xedY"
+    b"\x0c\xef\x8d9z\xf3o\xb6)f\xa9]$n\rp\x93\xd0\x10\xa4\x08\xb8\xb2\x8b\xb6"
+    b"\x8f\x80\xae;\xdcQ\xf1\xfa\x9a\x06\x8e\xa5\x0e\x8cK\x9c @\xaa:UcX\n!\xc6"
+    b"\x02\x12\xcb\x1b\"=\x16.\x1f\x176\xf2g=\xe1Wn\xe9\xe1\xd4\xf1O\xad\x15"
+    b"\x86\xe9\xa3T\xaf\xa9\xd7D\xb5\xd1W3pnt\x11\xc7VOj\xb7M\xc4i\xa1\xf1$3"
+    b"\xbb\xdc\x8af\xb0\xc5Y\r\xd1\xfb\xf2\xe7K\xe6\xc5hwO\xfe\x8c2^&\x07\xd5"
+    b"\x1fV\x19\xfd\r\x14\xd2i=yZ\xe6o\xaf\xc6\xb6\x92\x9d\xc4\r\xb3\xafw\xac%"
+    b"\xcfc\x1a\xf1`]\xf2\x1a\x9e\x808\xedm\xedQ\xb2\xfe\xe4h`[q\xae\xe0\x0f"
+    b"\xba0g\xb6\"N\xc3\xfb\xcfR\x11\xc5\x18)(\xc40\\\xa3\x02\xd9G!\xce\x1b"
+    b"\xc1\x96x\xb5\xc8z\x1f\x01\xb4\xaf\xde\xc2\xcd\x07\xe7H\xb3y\xa8M\n\\A\t"
+    b"ar\xddM\x8b\x9a\xea\x84\x9b!\xf1\x8d\xb1\xf1~\x1e\r\xa5H\xba\xf1\x84o"
+    b"\xda\x87\x01h\xe9\xa2\xbe\xbeqN\x9d\x84\x0b!WG\xda\xa1\xa5A\xb7\xc7`j"
+    b"\x15\xf2\xe9\xdd?\x015B\xd2~E\x06\x11\xe0\x91!\x05^\x80\xdd\xa8y\x15}"
+    b"\xa1)\xb1)\x81\x18\xf4\xf4\xf8\xc0\xefD\xe3\xdb2f\x1e\x12\xabu\xc9\x97"
+    b"\xcd\x1e\xa7\x0c\x02x4_6\x03\xc4$t\xf39\x94\x1d=\xcb\xbfv\\\xf5\xa3\x1d"
+    b"\x9d8jk\x95\x13)ff\xf9n\xc4\xa9\xe3\x01\xb8\xda\xfb\xab\xdfM\x99\xfb\x05"
+    b"\xe0\xe9\xb0I\xf4E\xab\xe2\x15\xa3\x035\xe7\xdeT\xee\x82p\xb4\x88\xd3"
+    b"\x893\x9c/\xc0\xd6\x8fou;\xf6\x95PR\xa9\xb2\xc1\xefFj\xe2\xa7$\xf7h\xf1"
+    b"\xdfK(\xc9c\xba7\xe8\xe3)\xdd\xb2,\x83\xfb\x84\x18.y\x18Qi\x88\xf8`h-"
+    b"\xef\xd5\xed\x8c\t\xd8\xc3^\x0f\x00\xb7\xd0[!\xafM\x9b\xd7.\x07\xd8\xfb"
+    b"\xd9\xe2-S+\xaa8,\xa0\x03\x1b \xea\xa8\x00\xc3\xab~\xd0$e\xa5\x7f\xf7"
+    b"\x95P]\x12\x19i\xd9\x7fo\x0c\xd8g^\rE\xa5\x80\x18\xc5\x01\x80\xaek`\xff~"
+    b"\xb6y\xe7+\xe5\x11^D\xa7\x85\x18\"!\xd6\xd2\xa7\xf4\x1eT\xdb\x02\xe15"
+    b"\x02Y\xbc\x174Z\xe7\x9cH\x1c\xbf\x0f\xc6\xe9f]\xcf\x8cx\xbc\xe5\x15\x94"
+    b"\xfc3\xbc\xa7TUH\xf1\x84\x1b\xf7\xa9y\xc07\x84\xf8X\xd8\xef\xfc \x1c\xd8"
+    b"( /\xf2\xb7\xec\xc1\\\x8c\xf6\x95\xa1\x03J\x83vP8\xe1\xe3\xbb~\xc24kA"
+    b"\x98y\xa1\xf2P\xe9\x9d\xc9J\xf8N\x99\xb4\xceaO\xde\x16\x1e\xc2\x19\xa7"
+    b"\x03\xd2\xe0\x8f:\x15\xf3\x84\x9e\xee\xe6e\xb8\x02q\xc7AC\x1emw\xfd\t"
+    b"\x9a\x1eu\xc1\xa9\xcaCwUP\x00\xa5\xf78L4w!\x91L2 \x87\xd0\xf2\x06\x81j"
+    b"\x80;\x03V\x06\x87\x92\xcb\x90lv at E\x8d\x8d\xa5\xa6\xe7Z[\xdf\xd6E\x03`>"
+    b"\x8f\xde\xa1bZ\x84\xd0\xa9`\x05\x0e{\x80;\xe3\xbef\x8d\x1d\xebk1.\xe3"
+    b"\xe9N\x15\xf7\xd4(\xfa\xbb\x15\xbdu\xf7\x7f\x86\xae!\x03L\x1d\xb5\xc1"
+    b"\xb9\x11\xdb\xd0\x93\xe4\x02\xe1\xd2\xcbBjc_\xe8}d\xdb\xc3\xa0Y\xbe\xc9/"
+    b"\x95\x01\xa3,\xe6bl@\x01\xdbp\xc2\xce\x14\x168\xc2q\xe3uH\x89X\xa4\xa9"
+    b"\x19\x1d\xc1}\x7fOX\x19\x9f\xdd\xbe\x85\x83\xff\x96\x1ee\x82O`CF=K\xeb$I"
+    b"\x17_\xefX\x8bJ'v\xde\x1f+\xd9.v\xf8Tv\x17\xf2\x9f5\x19\xe1\xb9\x91\xa8S"
+    b"\x86\xbd\x1a\"(\xa5x\x8dC\x03X\x81\x91\xa8\x11\xc4pS\x13\xbc\xf2'J\xae!"
+    b"\xef\xef\x84G\t\x8d\xc4\x10\x132\x00oS\x9e\xe0\xe4d\x8f\xb8y\xac\xa6\x9f"
+    b",\xb8f\x87\r\xdf\x9eE\x0f\xe1\xd0\\L\x00\xb2\xe1h\x84\xef}\x98\xa8\x11"
+    b"\xccW#\\\x83\x7fo\xbbz\x8f\x00"
+)
+
+FILTERS_RAW_3 = [{"id": lzma.FILTER_IA64, "start_offset": 0x100},
+                 {"id": lzma.FILTER_LZMA2}]
+COMPRESSED_RAW_3 = (
+    b"\xe0\x07\x80\x03\xdf]\x00\x05\x14\x07bX\x19\xcd\xddn\x98\x15\xe4\xb4\x9d"
+    b"o\x1d\xc4\xe5\n\x03\xcc2h\xc7\\\x86\xff\xf8\xe2\xfc\xe7\xd9\xfe6\xb8("
+    b"\xa8wd\xc2\"u.n\x1e\xc3\xf2\x8e\x8d\x8f\x02\x17/\xa6=\xf0\xa2\xdf/M\x89"
+    b"\xbe\xde\xa7\x1cz\x18-]\xd5\xef\x13\x8frZ\x15\x80\x8c\xf8\x8do\xfa\x12"
+    b"\x9b#z/\xef\xf0\xfaF\x01\x82\xa3M\x8e\xa1t\xca6 BF$\xe5Q\xa4\x98\xee\xde"
+    b"l\xe8\x7f\xf0\x9d,bn\x0b\x13\xd4\xa8\x81\xe4N\xc8\x86\x153\xf5x2\xa2O"
+    b"\x13 at Q\xa1\x00/\xa5\xd0O\x97\xdco\xae\xf7z\xc4\xcdS\xb6t<\x16\xf2\x9cI#"
+    b"\x89ud\xc66Y\xd9\xee\xe6\xce\x12]\xe5\xf0\xaa\x96-Pe\xade:\x04\t\x1b\xf7"
+    b"\xdb7\n\x86\x1fp\xc8J\xba\xf4\xf0V\xa9\xdc\xf0\x02%G\xf9\xdf=?\x15\x1b"
+    b"\xe1(\xce\x82=\xd6I\xac3\x12\x0cR\xb7\xae\r\xb1i\x03\x95\x01\xbd\xbe\xfa"
+    b"\x02s\x01P\x9d\x96X\xb12j\xc8L\xa8\x84b\xf6\xc3\xd4c-H\x93oJl\xd0iQ\xe4k"
+    b"\x84\x0b\xc1\xb7\xbc\xb1\x17\x88\xb1\xca?@\xf6\x07\xea\xe6x\xf1H12P\x0f"
+    b"\x8a\xc9\xeauw\xe3\xbe\xaai\xa9W\xd0\x80\xcd#cb5\x99\xd8]\xa9d\x0c\xbd"
+    b"\xa2\xdcWl\xedUG\xbf\x89yF\xf77\x81v\xbd5\x98\xbeh8\x18W\x08\xf0\x1b\x99"
+    b"5:\x1a?rD\x96\xa1\x04\x0f\xae\xba\x85\xeb\x9d5@\xf5\x83\xd37\x83\x8ac"
+    b"\x06\xd4\x97i\xcdt\x16S\x82k\xf6K\x01vy\x88\x91\x9b6T\xdae\r\xfd]:k\xbal"
+    b"\xa9\xbba\xc34\xf9r\xeb}r\xdb\xc7\xdb*\x8f\x03z\xdc8h\xcc\xc9\xd3\xbcl"
+    b"\xa5-\xcb\xeaK\xa2\xc5\x15\xc0\xe3\xc1\x86Z\xfb\xebL\xe13\xcf\x9c\xe3"
+    b"\x1d\xc9\xed\xc2\x06\xcc\xce!\x92\xe5\xfe\x9c^\xa59w \x9bP\xa3PK\x08d"
+    b"\xf9\xe2Z}\xa7\xbf\xed\xeb%$\x0c\x82\xb8/\xb0\x01\xa9&,\xf7qh{Q\x96)\xf2"
+    b"q\x96\xc3\x80\xb4\x12\xb0\xba\xe6o\xf4!\xb4[\xd4\x8aw\x10\xf7t\x0c\xb3"
+    b"\xd9\xd5\xc3`^\x81\x11??\\\xa4\x99\x85R\xd4\x8e\x83\xc9\x1eX\xbfa\xf1"
+    b"\xac\xb0\xea\xea\xd7\xd0\xab\x18\xe2\xf2\xed\xe1\xb7\xc9\x18\xcbS\xe4>"
+    b"\xc9\x95H\xe8\xcb\t\r%\xeb\xc7$.o\xf1\xf3R\x17\x1db\xbb\xd8U\xa5^\xccS"
+    b"\x16\x01\x87\xf3/\x93\xd1\xf0v\xc0r\xd7\xcc\xa2Gkz\xca\x80\x0e\xfd\xd0"
+    b"\x8b\xbb\xd2Ix\xb3\x1ey\xca-0\xe3z^\xd6\xd6\x8f_\xf1\x9dP\x9fi\xa7\xd1"
+    b"\xe8\x90\x84\xdc\xbf\xcdky\x8e\xdc\x81\x7f\xa3\xb2+\xbf\x04\xef\xd8\\"
+    b"\xc4\xdf\xe1\xb0\x01\xe9\x93\xe3Y\xf1\x1dY\xe8h\x81\xcf\xf1w\xcc\xb4\xef"
+    b" \x8b|\x04\xea\x83ej\xbe\x1f\xd4z\x9c`\xd3\x1a\x92A\x06\xe5\x8f\xa9\x13"
+    b"\t\x9e=\xfa\x1c\xe5_\x9f%v\x1bo\x11ZO\xd8\xf4\t\xddM\x16-\x04\xfc\x18<\""
+    b"CM\xddg~b\xf6\xef\x8e\x0c\xd0\xde|\xa0'\x8a\x0c\xd6x\xae!J\xa6F\x88\x15u"
+    b"\x008\x17\xbc7y\xb3\xd8u\xac_\x85\x8d\xe7\xc1@\x9c\xecqc\xa3#\xad\xf1"
+    b"\x935\xb5)_\r\xec3]\x0fo]5\xd0my\x07\x9b\xee\x81\xb5\x0f\xcfK+\x00\xc0"
+    b"\xe4b\x10\xe4\x0c\x1a \x9b\xe0\x97t\xf6\xa1\x9e\x850\xba\x0c\x9a\x8d\xc8"
+    b"\x8f\x07\xd7\xae\xc8\xf9+i\xdc\xb9k\xb0>f\x19\xb8\r\xa8\xf8\x1f$\xa5{p"
+    b"\xc6\x880\xce\xdb\xcf\xca_\x86\xac\x88h6\x8bZ%'\xd0\n\xbf\x0f\x9c\"\xba"
+    b"\xe5\x86\x9f\x0f7X=mNX[\xcc\x19FU\xc9\x860\xbc\x90a+* \xae_$\x03\x1e\xd3"
+    b"\xcd_\xa0\x9c\xde\xaf46q\xa5\xc9\x92\xd7\xca\xe3`\x9d\x85}\xb4\xff\xb3"
+    b"\x83\xfb\xb6\xca\xae`\x0bw\x7f\xfc\xd8\xacVe\x19\xc8\x17\x0bZ\xad\x88"
+    b"\xeb#\x97\x03\x13\xb1d\x0f{\x0c\x04w\x07\r\x97\xbd\xd6\xc1\xc3B:\x95\x08"
+    b"^\x10V\xaeaH\x02\xd9\xe3\n\\\x01X\xf6\x9c\x8a\x06u#%\xbe*\xa1\x18v\x85"
+    b"\xec!\t4\x00\x00\x00"
+)
+
+FILTERS_RAW_4 = [{"id": lzma.FILTER_DELTA, "dist": 4},
+                 {"id": lzma.FILTER_X86, "start_offset": 0x40},
+                 {"id": lzma.FILTER_LZMA2, "preset": 4, "lc": 2}]
+COMPRESSED_RAW_4 = (
+    b"\xe0\x07\x80\x06\x0e\\\x00\x05\x14\x07bW\xaah\xdd\x10\xdc'\xd6\x90,\xc6v"
+    b"Jq \x14l\xb7\x83xB\x0b\x97f=&fx\xba\n>Tn\xbf\x8f\xfb\x1dF\xca\xc3v_\xca?"
+    b"\xfbV<\x92#\xd4w\xa6\x8a\xeb\xf6\x03\xc0\x01\x94\xd8\x9e\x13\x12\x98\xd1"
+    b"*\xfa]c\xe8\x1e~\xaf\xb5]Eg\xfb\x9e\x01\"8\xb2\x90\x06=~\xe9\x91W\xcd"
+    b"\xecD\x12\xc7\xfa\xe1\x91\x06\xc7\x99\xb9\xe3\x901\x87\x19u\x0f\x869\xff"
+    b"\xc1\xb0hw|\xb0\xdcl\xcck\xb16o7\x85\xee{Y_b\xbf\xbc$\xf3=\x8d\x8bw\xe5Z"
+    b"\x08@\xc4kmE\xad\xfb\xf6*\xd8\xad\xa1\xfb\xc5{\xdej,)\x1emB\x1f<\xaeca"
+    b"\x80(\xee\x07 \xdf\xe9\xf8\xeb\x0e-\x97\x86\x90c\xf9\xea'B\xf7`\xd7\xb0"
+    b"\x92\xbd\xa0\x82]\xbd\x0e\x0eB\x19\xdc\x96\xc6\x19\xd86D\xf0\xd5\x831"
+    b"\x03\xb7\x1c\xf7&5\x1a\x8f PZ&j\xf8\x98\x1bo\xcc\x86\x9bS\xd3\xa5\xcdu"
+    b"\xf9$\xcc\x97o\xe5V~\xfb\x97\xb5\x0b\x17\x9c\xfdxW\x10\xfep4\x80\xdaHDY"
+    b"\xfa)\xfet\xb5\"\xd4\xd3F\x81\xf4\x13\x1f\xec\xdf\xa5\x13\xfc\"\x91x\xb7"
+    b"\x99\xce\xc8\x92\n\xeb[\x10l*Y\xd8\xb1@\x06\xc8o\x8d7r\xebu\xfd5\x0e\x7f"
+    b"\xf1$U{\t}\x1fQ\xcfxN\x9d\x9fXX\xe9`\x83\xc1\x06\xf4\x87v-f\x11\xdb/\\"
+    b"\x06\xff\xd7)B\xf3g\x06\x88#2\x1eB244\x7f4q\t\xc893?mPX\x95\xa6a\xfb)d"
+    b"\x9b\xfc\x98\x9aj\x04\xae\x9b\x9d\x19w\xba\xf92\xfaA\x11\\\x17\x97C3\xa4"
+    b"\xbc!\x88\xcdo[\xec:\x030\x91.\x85\xe0@\\4\x16\x12\x9d\xcaJv\x97\xb04"
+    b"\xack\xcbkf\xa3ss\xfc\x16^\x8ce\x85a\xa5=&\xecr\xb3p\xd1E\xd5\x80y\xc7"
+    b"\xda\xf6\xfek\xbcT\xbfH\xee\x15o\xc5\x8c\x830\xec\x1d\x01\xae\x0c-e\\"
+    b"\x91\x90\x94\xb2\xf8\x88\x91\xe8\x0b\xae\xa7>\x98\xf6\x9ck\xd2\xc6\x08"
+    b"\xe6\xab\t\x98\xf2!\xa0\x8c^\xacqA\x99<\x1cEG\x97\xc8\xf1\xb6\xb9\x82"
+    b"\x8d\xf7\x08s\x98a\xff\xe3\xcc\x92\x0e\xd2\xb6U\xd7\xd9\x86\x7fa\xe5\x1c"
+    b"\x8dTG@\t\x1e\x0e7*\xfc\xde\xbc]6N\xf7\xf1\x84\x9e\x9f\xcf\xe9\x1e\xb5'"
+    b"\xf4<\xdf\x99sq\xd0\x9d\xbd\x99\x0b\xb4%p4\xbf{\xbb\x8a\xd2\x0b\xbc=M"
+    b"\x94H:\xf5\xa8\xd6\xa4\xc90\xc2D\xb9\xd3\xa8\xb0S\x87 `\xa2\xeb\xf3W\xce"
+    b" 7\xf9N#\r\xe6\xbe\t\x9d\xe7\x811\xf9\x10\xc1\xc2\x14\xf6\xfc\xcba\xb7"
+    b"\xb1\x7f\x95l\xe4\tjA\xec:\x10\xe5\xfe\xc2\\=D\xe2\x0c\x0b3]\xf7\xc1\xf7"
+    b"\xbceZ\xb1A\xea\x16\xe5\xfddgFQ\xed\xaf\x04\xa3\xd3\xf8\xa2q\x19B\xd4r"
+    b"\xc5\x0c\x9a\x14\x94\xea\x91\xc4o\xe4\xbb\xb4\x99\xf4@\xd1\xe6\x0c\xe3"
+    b"\xc6d\xa0Q\n\xf2/\xd8\xb8S5\x8a\x18:\xb5g\xac\x95D\xce\x17\x07\xd4z\xda"
+    b"\x90\xe65\x07\x19H!\t\xfdu\x16\x8e\x0eR\x19\xf4\x8cl\x0c\xf9Q\xf1\x80"
+    b"\xe3\xbf\xd7O\xf8\x8c\x18\x0b\x9c\xf1\x1fb\xe1\tR\xb2\xf1\xe1A\xea \xcf-"
+    b"IGE\xf1\x14\x98$\x83\x15\xc9\xd8j\xbf\x19\x0f\xd5\xd1\xaa\xb3\xf3\xa5I2s"
+    b"\x8d\x145\xca\xd5\xd93\x9c\xb8D0\xe6\xaa%\xd0\xc0P}JO^h\x8e\x08\xadlV."
+    b"\x18\x88\x13\x05o\xb0\x07\xeaw\xe0\xb6\xa4\xd5*\xe4r\xef\x07G+\xc1\xbei["
+    b"w\xe8\xab at _\xef\x15y\xe5\x12\xc9W\x1b.\xad\x85-\xc2\xf7\xe3mU6g\x8eSA"
+    b"\x01(\xd3\xdb\x16\x13=\xde\x92\xf9,D\xb8\x8a\xb2\xb4\xc9\xc3\xefnE\xe8\\"
+    b"\xa6\xe2Y\xd2\xcf\xcb\x8c\xb6\xd5\xe9\x1d\x1e\x9a\x8b~\xe2\xa6\rE\x84uV"
+    b"\xed\xc6\x99\xddm<\x10[\x0fu\x1f\xc1\x1d1\n\xcfw\xb2%!\xf0[\xce\x87\x83B"
+    b"\x08\xaa,\x08%d\xcef\x94\"\xd9g.\xc83\xcbXY+4\xec\x85qA\n\x1d=9\xf0*\xb1"
+    b"\x1f/\xf3s\xd61b\x7f@\xfb\x9d\xe3FQ\\\xbd\x82\x1e\x00\xf3\xce\xd3\xe1"
+    b"\xca,E\xfd7[\xab\xb6\xb7\xac!mA}\xbd\x9d3R5\x9cF\xabH\xeb\x92)cc\x13\xd0"
+    b"\xbd\xee\xe9n{\x1dIJB\xa5\xeb\x11\xe8`w&`\x8b}@Oxe\t\x8a\x07\x02\x95\xf2"
+    b"\xed\xda|\xb1e\xbe\xaa\xbbg\x19@\xe1Y\x878\x84\x0f\x8c\xe3\xc98\xf2\x9e"
+    b"\xd5N\xb5J\xef\xab!\xe2\x8dq\xe1\xe5q\xc5\xee\x11W\xb7\xe4k*\x027\xa0"
+    b"\xa3J\xf4\xd8m\xd0q\x94\xcb\x07\n:\xb6`.\xe4\x9c\x15+\xc0)\xde\x80X\xd4"
+    b"\xcfQm\x01\xc2cP\x1cA\x85'\xc9\xac\x8b\xe6\xb2)\xe6\x84t\x1c\x92\xe4Z"
+    b"\x1cR\xb0\x9e\x96\xd1\xfb\x1c\xa6\x8b\xcb`\x10\x12]\xf2gR\x9bFT\xe0\xc8H"
+    b"S\xfb\xac<\x04\xc7\xc1\xe8\xedP\xf4\x16\xdb\xc0\xd7e\xc2\x17J^\x1f\xab"
+    b"\xff[\x08\x19\xb4\xf5\xfb\x19\xb4\x04\xe5c~']\xcb\xc2A\xec\x90\xd0\xed"
+    b"\x06,\xc5K{\x86\x03\xb1\xcdMx\xdeQ\x8c3\xf9\x8a\xea=\x89\xaba\xd2\xc89a"
+    b"\xd72\xf0\xc3\x19\x8a\xdfs\xd4\xfd\xbb\x81b\xeaE\"\xd8\xf4d\x0cD\xf7IJ!"
+    b"\xe5d\xbbG\xe9\xcam\xaa\x0f_r\x95\x91NBq\xcaP\xce\xa7\xa9\xb5\x10\x94eP!"
+    b"|\x856\xcd\xbfIir\xb8e\x9bjP\x97q\xabwS7\x1a\x0ehM\xe7\xca\x86?\xdeP}y~"
+    b"\x0f\x95I\xfc\x13\xe1<Q\x1b\x868\x1d\x11\xdf\x94\xf4\x82>r\xa9k\x88\xcb"
+    b"\xfd\xc3v\xe2\xb9\x8a\x02\x8eq\x92I\xf8\xf6\xf1\x03s\x9b\xb8\xe3\"\xe3"
+    b"\xa9\xa5>D\xb8\x96;\xe7\x92\xd133\xe8\xdd'e\xc9.\xdc;\x17\x1f\xf5H\x13q"
+    b"\xa4W\x0c\xdb~\x98\x01\xeb\xdf\xe32\x13\x0f\xddx\n6\xa0\t\x10\xb6\xbb"
+    b"\xb0\xc3\x18\xb6;\x9fj[\xd9\xd5\xc9\x06\x8a\x87\xcd\xe5\xee\xfc\x9c-%@"
+    b"\xee\xe0\xeb\xd2\xe3\xe8\xfb\xc0\x122\\\xc7\xaf\xc2\xa1Oth\xb3\x8f\x82"
+    b"\xb3\x18\xa8\x07\xd5\xee_\xbe\xe0\x1cA\x1e_\r\x9a\xb0\x17W&\xa2D\x91\x94"
+    b"\x1a\xb2\xef\xf2\xdc\x85;X\xb0,\xeb>-7S\xe5\xca\x07)\x1fp\x7f\xcaQBL\xca"
+    b"\xf3\xb9d\xfc\xb5su\xb0\xc8\x95\x90\xeb*)\xa0v\xe4\x9a{FW\xf4l\xde\xcdj"
+    b"\x00"
+)
+
+
+def test_main():
+    run_unittest(
+        CompressorDecompressorTestCase,
+        CompressDecompressFunctionTestCase,
+        FileTestCase,
+        MiscellaneousTestCase,
+    )
+
+if __name__ == "__main__":
+    test_main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -399,6 +399,8 @@
 Library
 -------
 
+- Issue #6715: Add a module 'lzma' for compression using the LZMA algorithm.
+
 - Issue #13487: Make inspect.getmodule robust against changes done to
   sys.modules while it is iterating over it.
 
diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c
new file mode 100644
--- /dev/null
+++ b/Modules/_lzmamodule.c
@@ -0,0 +1,1106 @@
+/* _lzma - Low-level Python interface to liblzma. */
+
+#define PY_SSIZE_T_CLEAN
+
+#include "Python.h"
+#include "structmember.h"
+#ifdef WITH_THREAD
+#include "pythread.h"
+#endif
+
+#include <stdarg.h>
+#include <string.h>
+
+#include <lzma.h>
+
+
+#ifndef PY_LONG_LONG
+#error "This module requires PY_LONG_LONG to be defined"
+#endif
+
+
+#ifdef WITH_THREAD
+#define ACQUIRE_LOCK(obj) do { \
+    if (!PyThread_acquire_lock((obj)->lock, 0)) { \
+        Py_BEGIN_ALLOW_THREADS \
+        PyThread_acquire_lock((obj)->lock, 1); \
+        Py_END_ALLOW_THREADS \
+    } } while (0)
+#define RELEASE_LOCK(obj) PyThread_release_lock((obj)->lock)
+#else
+#define ACQUIRE_LOCK(obj)
+#define RELEASE_LOCK(obj)
+#endif
+
+
+/* Container formats: */
+enum {
+    FORMAT_AUTO,
+    FORMAT_XZ,
+    FORMAT_ALONE,
+    FORMAT_RAW,
+};
+
+#define LZMA_CHECK_UNKNOWN (LZMA_CHECK_ID_MAX + 1)
+
+
+typedef struct {
+    PyObject_HEAD
+    lzma_stream lzs;
+    int flushed;
+#ifdef WITH_THREAD
+    PyThread_type_lock lock;
+#endif
+} Compressor;
+
+typedef struct {
+    PyObject_HEAD
+    lzma_stream lzs;
+    int check;
+    char eof;
+    PyObject *unused_data;
+#ifdef WITH_THREAD
+    PyThread_type_lock lock;
+#endif
+} Decompressor;
+
+/* LZMAError class object. */
+static PyObject *Error;
+
+/* An empty tuple, used by the filter specifier parsing code. */
+static PyObject *empty_tuple;
+
+
+/* Helper functions. */
+
+static int
+catch_lzma_error(lzma_ret lzret)
+{
+    switch (lzret) {
+        case LZMA_OK:
+        case LZMA_GET_CHECK:
+        case LZMA_NO_CHECK:
+        case LZMA_STREAM_END:
+            return 0;
+        case LZMA_UNSUPPORTED_CHECK:
+            PyErr_SetString(Error, "Unsupported integrity check");
+            return 1;
+        case LZMA_MEM_ERROR:
+            PyErr_NoMemory();
+            return 1;
+        case LZMA_MEMLIMIT_ERROR:
+            PyErr_SetString(Error, "Memory usage limit exceeded");
+            return 1;
+        case LZMA_FORMAT_ERROR:
+            PyErr_SetString(Error, "Input format not supported by decoder");
+            return 1;
+        case LZMA_OPTIONS_ERROR:
+            PyErr_SetString(Error, "Invalid or unsupported options");
+            return 1;
+        case LZMA_DATA_ERROR:
+            PyErr_SetString(Error, "Corrupt input data");
+            return 1;
+        case LZMA_BUF_ERROR:
+            PyErr_SetString(Error, "Insufficient buffer space");
+            return 1;
+        case LZMA_PROG_ERROR:
+            PyErr_SetString(Error, "Internal error");
+            return 1;
+        default:
+            PyErr_Format(Error, "Unrecognized error from liblzma: %d", lzret);
+            return 1;
+    }
+}
+
+#if BUFSIZ < 8192
+#define INITIAL_BUFFER_SIZE 8192
+#else
+#define INITIAL_BUFFER_SIZE BUFSIZ
+#endif
+
+static int
+grow_buffer(PyObject **buf)
+{
+    size_t size = PyBytes_GET_SIZE(*buf);
+    return _PyBytes_Resize(buf, size + (size >> 3) + 6);
+}
+
+
+/* Some custom type conversions for PyArg_ParseTupleAndKeywords(),
+   since the predefined conversion specifiers do not suit our needs:
+
+      uint32_t - the "I" (unsigned int) specifier is the right size, but
+      silently ignores overflows on conversion.
+
+      lzma_mode and lzma_match_finder - these are enumeration types, and
+      so the size of each is implementation-defined. Worse, different
+      enum types can be of different sizes within the same program, so
+      to be strictly correct, we need to define two separate converters.
+ */
+
+#define INT_TYPE_CONVERTER_FUNC(TYPE, FUNCNAME) \
+    static int \
+    FUNCNAME(PyObject *obj, void *ptr) \
+    { \
+        unsigned long val; \
+        \
+        val = PyLong_AsUnsignedLong(obj); \
+        if (PyErr_Occurred()) \
+            return 0; \
+        if ((unsigned long)(TYPE)val != val) { \
+            PyErr_SetString(PyExc_OverflowError, \
+                            "Value too large for " #TYPE " type"); \
+            return 0; \
+        } \
+        *(TYPE *)ptr = val; \
+        return 1; \
+    }
+
+INT_TYPE_CONVERTER_FUNC(uint32_t, uint32_converter)
+INT_TYPE_CONVERTER_FUNC(lzma_mode, lzma_mode_converter)
+INT_TYPE_CONVERTER_FUNC(lzma_match_finder, lzma_mf_converter)
+
+#undef INT_TYPE_CONVERTER_FUNC
+
+
+/* Filter specifier parsing functions. */
+
+static void *
+parse_filter_spec_lzma(PyObject *spec)
+{
+    static char *optnames[] = {"id", "preset", "dict_size", "lc", "lp",
+                               "pb", "mode", "nice_len", "mf", "depth", NULL};
+    PyObject *id;
+    PyObject *preset_obj;
+    uint32_t preset = LZMA_PRESET_DEFAULT;
+    lzma_options_lzma *options;
+
+    /* First, fill in default values for all the options using a preset.
+       Then, override the defaults with any values given by the caller. */
+
+    preset_obj = PyMapping_GetItemString(spec, "preset");
+    if (preset_obj == NULL) {
+        if (PyErr_ExceptionMatches(PyExc_KeyError))
+            PyErr_Clear();
+        else
+            return NULL;
+    } else {
+        int ok = uint32_converter(preset_obj, &preset);
+        Py_DECREF(preset_obj);
+        if (!ok)
+            return NULL;
+    }
+
+    options = (lzma_options_lzma *)PyMem_Malloc(sizeof *options);
+    if (options == NULL)
+        return PyErr_NoMemory();
+    memset(options, 0, sizeof *options);
+
+    if (lzma_lzma_preset(options, preset)) {
+        PyMem_Free(options);
+        PyErr_Format(Error, "lzma_lzma_preset() failed for preset %#x", preset);
+        return NULL;
+    }
+
+    if (!PyArg_ParseTupleAndKeywords(empty_tuple, spec,
+                                     "|OOO&O&O&O&O&O&O&O&", optnames,
+                                     &id, &preset_obj,
+                                     uint32_converter, &options->dict_size,
+                                     uint32_converter, &options->lc,
+                                     uint32_converter, &options->lp,
+                                     uint32_converter, &options->pb,
+                                     lzma_mode_converter, &options->mode,
+                                     uint32_converter, &options->nice_len,
+                                     lzma_mf_converter, &options->mf,
+                                     uint32_converter, &options->depth)) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Invalid filter specifier for LZMA filter");
+        PyMem_Free(options);
+        options = NULL;
+    }
+    return options;
+}
+
+static void *
+parse_filter_spec_delta(PyObject *spec)
+{
+    static char *optnames[] = {"id", "dist", NULL};
+    PyObject *id;
+    uint32_t dist = 1;
+    lzma_options_delta *options;
+
+    if (!PyArg_ParseTupleAndKeywords(empty_tuple, spec, "|OO&", optnames,
+                                     &id, uint32_converter, &dist)) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Invalid filter specifier for delta filter");
+        return NULL;
+    }
+
+    options = (lzma_options_delta *)PyMem_Malloc(sizeof *options);
+    if (options == NULL)
+        return PyErr_NoMemory();
+    memset(options, 0, sizeof *options);
+    options->type = LZMA_DELTA_TYPE_BYTE;
+    options->dist = dist;
+    return options;
+}
+
+static void *
+parse_filter_spec_bcj(PyObject *spec)
+{
+    static char *optnames[] = {"id", "start_offset", NULL};
+    PyObject *id;
+    uint32_t start_offset = 0;
+    lzma_options_bcj *options;
+
+    if (!PyArg_ParseTupleAndKeywords(empty_tuple, spec, "|OO&", optnames,
+                                     &id, uint32_converter, &start_offset)) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Invalid filter specifier for BCJ filter");
+        return NULL;
+    }
+
+    options = (lzma_options_bcj *)PyMem_Malloc(sizeof *options);
+    if (options == NULL)
+        return PyErr_NoMemory();
+    memset(options, 0, sizeof *options);
+    options->start_offset = start_offset;
+    return options;
+}
+
+static void *
+parse_filter_spec(lzma_filter *f, PyObject *spec)
+{
+    PyObject *id_obj;
+
+    if (!PyMapping_Check(spec)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "Filter specifier must be a dict or dict-like object");
+        return NULL;
+    }
+    id_obj = PyMapping_GetItemString(spec, "id");
+    if (id_obj == NULL) {
+        if (PyErr_ExceptionMatches(PyExc_KeyError))
+            PyErr_SetString(PyExc_ValueError,
+                            "Filter specifier must have an \"id\" entry");
+        return NULL;
+    }
+    f->id = PyLong_AsUnsignedLongLong(id_obj);
+    Py_DECREF(id_obj);
+    if (PyErr_Occurred())
+        return NULL;
+
+    switch (f->id) {
+        case LZMA_FILTER_LZMA1:
+        case LZMA_FILTER_LZMA2:
+            f->options = parse_filter_spec_lzma(spec);
+            return f->options;
+        case LZMA_FILTER_DELTA:
+            f->options = parse_filter_spec_delta(spec);
+            return f->options;
+        case LZMA_FILTER_X86:
+        case LZMA_FILTER_POWERPC:
+        case LZMA_FILTER_IA64:
+        case LZMA_FILTER_ARM:
+        case LZMA_FILTER_ARMTHUMB:
+        case LZMA_FILTER_SPARC:
+            f->options = parse_filter_spec_bcj(spec);
+            return f->options;
+        default:
+            PyErr_Format(PyExc_ValueError, "Invalid filter ID: %llu", f->id);
+            return NULL;
+    }
+}
+
+static void
+free_filter_chain(lzma_filter filters[])
+{
+    int i;
+
+    for (i = 0; filters[i].id != LZMA_VLI_UNKNOWN; i++)
+        PyMem_Free(filters[i].options);
+}
+
+static int
+parse_filter_chain_spec(lzma_filter filters[], PyObject *filterspecs)
+{
+    Py_ssize_t i, num_filters;
+
+    num_filters = PySequence_Length(filterspecs);
+    if (num_filters == -1)
+        return -1;
+    if (num_filters > LZMA_FILTERS_MAX) {
+        PyErr_Format(PyExc_ValueError,
+                     "Too many filters - liblzma supports a maximum of %d",
+                     LZMA_FILTERS_MAX);
+        return -1;
+    }
+
+    for (i = 0; i < num_filters; i++) {
+        int ok = 1;
+        PyObject *spec = PySequence_GetItem(filterspecs, i);
+        if (spec == NULL || parse_filter_spec(&filters[i], spec) == NULL)
+            ok = 0;
+        Py_XDECREF(spec);
+        if (!ok) {
+            filters[i].id = LZMA_VLI_UNKNOWN;
+            free_filter_chain(filters);
+            return -1;
+        }
+    }
+    filters[num_filters].id = LZMA_VLI_UNKNOWN;
+    return 0;
+}
+
+
+/* LZMACompressor class. */
+
+static PyObject *
+compress(Compressor *c, uint8_t *data, size_t len, lzma_action action)
+{
+    size_t data_size = 0;
+    PyObject *result;
+
+    result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE);
+    if (result == NULL)
+        return NULL;
+    c->lzs.next_in = data;
+    c->lzs.avail_in = len;
+    c->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result);
+    c->lzs.avail_out = PyBytes_GET_SIZE(result);
+    for (;;) {
+        lzma_ret lzret;
+
+        Py_BEGIN_ALLOW_THREADS
+        lzret = lzma_code(&c->lzs, action);
+        data_size = (char *)c->lzs.next_out - PyBytes_AS_STRING(result);
+        Py_END_ALLOW_THREADS
+        if (catch_lzma_error(lzret))
+            goto error;
+        if ((action == LZMA_RUN && c->lzs.avail_in == 0) ||
+            (action == LZMA_FINISH && lzret == LZMA_STREAM_END)) {
+            break;
+        } else if (c->lzs.avail_out == 0) {
+            if (grow_buffer(&result) == -1)
+                goto error;
+            c->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size;
+            c->lzs.avail_out = PyBytes_GET_SIZE(result) - data_size;
+        }
+    }
+    if (data_size != PyBytes_GET_SIZE(result))
+        if (_PyBytes_Resize(&result, data_size) == -1)
+            goto error;
+    return result;
+
+error:
+    Py_XDECREF(result);
+    return NULL;
+}
+
+PyDoc_STRVAR(Compressor_compress_doc,
+"compress(data) -> bytes\n"
+"\n"
+"Provide data to the compressor object. Returns a chunk of\n"
+"compressed data if possible, or b\"\" otherwise.\n"
+"\n"
+"When you have finished providing data to the compressor, call the\n"
+"flush() method to finish the conversion process.\n");
+
+static PyObject *
+Compressor_compress(Compressor *self, PyObject *args)
+{
+    Py_buffer buffer;
+    PyObject *result = NULL;
+
+    if (!PyArg_ParseTuple(args, "y*:compress", &buffer))
+        return NULL;
+
+    ACQUIRE_LOCK(self);
+    if (self->flushed)
+        PyErr_SetString(PyExc_ValueError, "Compressor has been flushed");
+    else
+        result = compress(self, buffer.buf, buffer.len, LZMA_RUN);
+    RELEASE_LOCK(self);
+    PyBuffer_Release(&buffer);
+    return result;
+}
+
+PyDoc_STRVAR(Compressor_flush_doc,
+"flush() -> bytes\n"
+"\n"
+"Finish the compression process. Returns the compressed data left\n"
+"in internal buffers.\n"
+"\n"
+"The compressor object cannot be used after this method is called.\n");
+
+static PyObject *
+Compressor_flush(Compressor *self, PyObject *noargs)
+{
+    PyObject *result = NULL;
+
+    ACQUIRE_LOCK(self);
+    if (self->flushed) {
+        PyErr_SetString(PyExc_ValueError, "Repeated call to flush()");
+    } else {
+        self->flushed = 1;
+        result = compress(self, NULL, 0, LZMA_FINISH);
+    }
+    RELEASE_LOCK(self);
+    return result;
+}
+
+static int
+Compressor_init_xz(lzma_stream *lzs, int check, uint32_t preset,
+                   PyObject *filterspecs)
+{
+    lzma_ret lzret;
+
+    if (filterspecs == Py_None) {
+        lzret = lzma_easy_encoder(lzs, preset, check);
+    } else {
+        lzma_filter filters[LZMA_FILTERS_MAX + 1];
+
+        if (parse_filter_chain_spec(filters, filterspecs) == -1)
+            return -1;
+        lzret = lzma_stream_encoder(lzs, filters, check);
+        free_filter_chain(filters);
+    }
+    if (catch_lzma_error(lzret))
+        return -1;
+    else
+        return 0;
+}
+
+static int
+Compressor_init_alone(lzma_stream *lzs, uint32_t preset, PyObject *filterspecs)
+{
+    lzma_ret lzret;
+
+    if (filterspecs == Py_None) {
+        lzma_options_lzma options;
+
+        if (lzma_lzma_preset(&options, preset)) {
+            PyErr_Format(Error, "Invalid compression preset: %#x", preset);
+            return -1;
+        }
+        lzret = lzma_alone_encoder(lzs, &options);
+    } else {
+        lzma_filter filters[LZMA_FILTERS_MAX + 1];
+
+        if (parse_filter_chain_spec(filters, filterspecs) == -1)
+            return -1;
+        if (filters[0].id == LZMA_FILTER_LZMA1 &&
+            filters[1].id == LZMA_VLI_UNKNOWN) {
+            lzret = lzma_alone_encoder(lzs, filters[0].options);
+        } else {
+            PyErr_SetString(PyExc_ValueError,
+                            "Invalid filter chain for FORMAT_ALONE - "
+                            "must be a single LZMA1 filter");
+            lzret = LZMA_PROG_ERROR;
+        }
+        free_filter_chain(filters);
+    }
+    if (PyErr_Occurred() || catch_lzma_error(lzret))
+        return -1;
+    else
+        return 0;
+}
+
+static int
+Compressor_init_raw(lzma_stream *lzs, PyObject *filterspecs)
+{
+    lzma_filter filters[LZMA_FILTERS_MAX + 1];
+    lzma_ret lzret;
+
+    if (filterspecs == Py_None) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Must specify filters for FORMAT_RAW");
+        return -1;
+    }
+    if (parse_filter_chain_spec(filters, filterspecs) == -1)
+        return -1;
+    lzret = lzma_raw_encoder(lzs, filters);
+    free_filter_chain(filters);
+    if (catch_lzma_error(lzret))
+        return -1;
+    else
+        return 0;
+}
+
+static int
+Compressor_init(Compressor *self, PyObject *args, PyObject *kwargs)
+{
+    static char *arg_names[] = {"format", "check", "preset", "filters", NULL};
+    int format = FORMAT_XZ;
+    int check = -1;
+    uint32_t preset = LZMA_PRESET_DEFAULT;
+    PyObject *preset_obj = Py_None;
+    PyObject *filterspecs = Py_None;
+
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs,
+                                     "|iiOO:LZMACompressor", arg_names,
+                                     &format, &check, &preset_obj,
+                                     &filterspecs))
+        return -1;
+
+    if (format != FORMAT_XZ && check != -1 && check != LZMA_CHECK_NONE) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Integrity checks are only supported by FORMAT_XZ");
+        return -1;
+    }
+
+    if (preset_obj != Py_None && filterspecs != Py_None) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Cannot specify both preset and filter chain");
+        return -1;
+    }
+
+    if (preset_obj != Py_None)
+        if (!uint32_converter(preset_obj, &preset))
+            return -1;
+
+#ifdef WITH_THREAD
+    self->lock = PyThread_allocate_lock();
+    if (self->lock == NULL) {
+        PyErr_SetString(PyExc_MemoryError, "Unable to allocate lock");
+        return -1;
+    }
+#endif
+
+    self->flushed = 0;
+    switch (format) {
+        case FORMAT_XZ:
+            if (check == -1)
+                check = LZMA_CHECK_CRC64;
+            if (Compressor_init_xz(&self->lzs, check, preset, filterspecs) != 0)
+                break;
+            return 0;
+
+        case FORMAT_ALONE:
+            if (Compressor_init_alone(&self->lzs, preset, filterspecs) != 0)
+                break;
+            return 0;
+
+        case FORMAT_RAW:
+            if (Compressor_init_raw(&self->lzs, filterspecs) != 0)
+                break;
+            return 0;
+
+        default:
+            PyErr_Format(PyExc_ValueError,
+                         "Invalid container format: %d", format);
+            break;
+    }
+
+#ifdef WITH_THREAD
+    PyThread_free_lock(self->lock);
+    self->lock = NULL;
+#endif
+    return -1;
+}
+
+static void
+Compressor_dealloc(Compressor *self)
+{
+    lzma_end(&self->lzs);
+#ifdef WITH_THREAD
+    if (self->lock != NULL)
+        PyThread_free_lock(self->lock);
+#endif
+    Py_TYPE(self)->tp_free((PyObject *)self);
+}
+
+static PyMethodDef Compressor_methods[] = {
+    {"compress", (PyCFunction)Compressor_compress, METH_VARARGS,
+     Compressor_compress_doc},
+    {"flush", (PyCFunction)Compressor_flush, METH_NOARGS,
+     Compressor_flush_doc},
+    {NULL}
+};
+
+PyDoc_STRVAR(Compressor_doc,
+"LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None)\n"
+"\n"
+"Create a compressor object for compressing data incrementally.\n"
+"\n"
+"format specifies the container format to use for the output. This can\n"
+"be FORMAT_XZ (default), FORMAT_ALONE, or FORMAT_RAW.\n"
+"\n"
+"check specifies the integrity check to use. For FORMAT_XZ, the default\n"
+"is CHECK_CRC64. FORMAT_ALONE and FORMAT_RAW do not suport integrity\n"
+"checks; for these formats, check must be omitted, or be CHECK_NONE.\n"
+"\n"
+"The settings used by the compressor can be specified either as a\n"
+"preset compression level (with the 'preset' argument), or in detail\n"
+"as a custom filter chain (with the 'filters' argument). For FORMAT_XZ\n"
+"and FORMAT_ALONE, the default is to use the PRESET_DEFAULT preset\n"
+"level. For FORMAT_RAW, the caller must always specify a filter chain;\n"
+"the raw compressor does not support preset compression levels.\n"
+"\n"
+"preset (if provided) should be an integer in the range 0-9, optionally\n"
+"OR-ed with the constant PRESET_EXTREME.\n"
+"\n"
+"filters (if provided) should be a sequence of dicts. Each dict should\n"
+"have an entry for \"id\" indicating the ID of the filter, plus\n"
+"additional entries for options to the filter.\n"
+"\n"
+"For one-shot compression, use the compress() function instead.\n");
+
+static PyTypeObject Compressor_type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "_lzma.LZMACompressor",             /* tp_name */
+    sizeof(Compressor),                 /* tp_basicsize */
+    0,                                  /* tp_itemsize */
+    (destructor)Compressor_dealloc,     /* tp_dealloc */
+    0,                                  /* tp_print */
+    0,                                  /* tp_getattr */
+    0,                                  /* tp_setattr */
+    0,                                  /* tp_reserved */
+    0,                                  /* tp_repr */
+    0,                                  /* tp_as_number */
+    0,                                  /* tp_as_sequence */
+    0,                                  /* tp_as_mapping */
+    0,                                  /* tp_hash */
+    0,                                  /* tp_call */
+    0,                                  /* tp_str */
+    0,                                  /* tp_getattro */
+    0,                                  /* tp_setattro */
+    0,                                  /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
+    Compressor_doc,                     /* tp_doc */
+    0,                                  /* tp_traverse */
+    0,                                  /* tp_clear */
+    0,                                  /* tp_richcompare */
+    0,                                  /* tp_weaklistoffset */
+    0,                                  /* tp_iter */
+    0,                                  /* tp_iternext */
+    Compressor_methods,                 /* tp_methods */
+    0,                                  /* tp_members */
+    0,                                  /* tp_getset */
+    0,                                  /* tp_base */
+    0,                                  /* tp_dict */
+    0,                                  /* tp_descr_get */
+    0,                                  /* tp_descr_set */
+    0,                                  /* tp_dictoffset */
+    (initproc)Compressor_init,          /* tp_init */
+    0,                                  /* tp_alloc */
+    PyType_GenericNew,                  /* tp_new */
+};
+
+
+/* LZMADecompressor class. */
+
+static PyObject *
+decompress(Decompressor *d, uint8_t *data, size_t len)
+{
+    size_t data_size = 0;
+    PyObject *result;
+
+    result = PyBytes_FromStringAndSize(NULL, INITIAL_BUFFER_SIZE);
+    if (result == NULL)
+        return NULL;
+    d->lzs.next_in = data;
+    d->lzs.avail_in = len;
+    d->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result);
+    d->lzs.avail_out = PyBytes_GET_SIZE(result);
+    for (;;) {
+        lzma_ret lzret;
+
+        Py_BEGIN_ALLOW_THREADS
+        lzret = lzma_code(&d->lzs, LZMA_RUN);
+        data_size = (char *)d->lzs.next_out - PyBytes_AS_STRING(result);
+        Py_END_ALLOW_THREADS
+        if (catch_lzma_error(lzret))
+            goto error;
+        if (lzret == LZMA_GET_CHECK || lzret == LZMA_NO_CHECK)
+            d->check = lzma_get_check(&d->lzs);
+        if (lzret == LZMA_STREAM_END) {
+            d->eof = 1;
+            if (d->lzs.avail_in > 0) {
+                Py_CLEAR(d->unused_data);
+                d->unused_data = PyBytes_FromStringAndSize(
+                        (char *)d->lzs.next_in, d->lzs.avail_in);
+                if (d->unused_data == NULL)
+                    goto error;
+            }
+            break;
+        } else if (d->lzs.avail_in == 0) {
+            break;
+        } else if (d->lzs.avail_out == 0) {
+            if (grow_buffer(&result) == -1)
+                goto error;
+            d->lzs.next_out = (uint8_t *)PyBytes_AS_STRING(result) + data_size;
+            d->lzs.avail_out = PyBytes_GET_SIZE(result) - data_size;
+        }
+    }
+    if (data_size != PyBytes_GET_SIZE(result))
+        if (_PyBytes_Resize(&result, data_size) == -1)
+            goto error;
+    return result;
+
+error:
+    Py_XDECREF(result);
+    return NULL;
+}
+
+PyDoc_STRVAR(Decompressor_decompress_doc,
+"decompress(data) -> bytes\n"
+"\n"
+"Provide data to the decompressor object. Returns a chunk of\n"
+"decompressed data if possible, or b\"\" otherwise.\n"
+"\n"
+"Attempting to decompress data after the end of the stream is\n"
+"reached raises an EOFError. Any data found after the end of the\n"
+"stream is ignored, and saved in the unused_data attribute.\n");
+
+static PyObject *
+Decompressor_decompress(Decompressor *self, PyObject *args)
+{
+    Py_buffer buffer;
+    PyObject *result = NULL;
+
+    if (!PyArg_ParseTuple(args, "y*:decompress", &buffer))
+        return NULL;
+
+    ACQUIRE_LOCK(self);
+    if (self->eof)
+        PyErr_SetString(PyExc_EOFError, "Already at end of stream");
+    else
+        result = decompress(self, buffer.buf, buffer.len);
+    RELEASE_LOCK(self);
+    PyBuffer_Release(&buffer);
+    return result;
+}
+
+static int
+Decompressor_init_raw(lzma_stream *lzs, PyObject *filterspecs)
+{
+    lzma_filter filters[LZMA_FILTERS_MAX + 1];
+    lzma_ret lzret;
+
+    if (parse_filter_chain_spec(filters, filterspecs) == -1)
+        return -1;
+    lzret = lzma_raw_decoder(lzs, filters);
+    free_filter_chain(filters);
+    if (catch_lzma_error(lzret))
+        return -1;
+    else
+        return 0;
+}
+
+static int
+Decompressor_init(Decompressor *self, PyObject *args, PyObject *kwargs)
+{
+    static char *arg_names[] = {"format", "memlimit", "filters", NULL};
+    const uint32_t decoder_flags = LZMA_TELL_ANY_CHECK | LZMA_TELL_NO_CHECK;
+    int format = FORMAT_AUTO;
+    uint64_t memlimit = UINT64_MAX;
+    PyObject *memlimit_obj = Py_None;
+    PyObject *filterspecs = Py_None;
+    lzma_ret lzret;
+
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs,
+                                     "|iOO:LZMADecompressor", arg_names,
+                                     &format, &memlimit_obj, &filterspecs))
+        return -1;
+
+    if (memlimit_obj != Py_None) {
+        if (format == FORMAT_RAW) {
+            PyErr_SetString(PyExc_ValueError,
+                            "Cannot specify memory limit with FORMAT_RAW");
+            return -1;
+        }
+        memlimit = PyLong_AsUnsignedLongLong(memlimit_obj);
+        if (PyErr_Occurred())
+            return -1;
+    }
+
+    if (format == FORMAT_RAW && filterspecs == Py_None) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Must specify filters for FORMAT_RAW");
+        return -1;
+    } else if (format != FORMAT_RAW && filterspecs != Py_None) {
+        PyErr_SetString(PyExc_ValueError,
+                        "Cannot specify filters except with FORMAT_RAW");
+        return -1;
+    }
+
+#ifdef WITH_THREAD
+    self->lock = PyThread_allocate_lock();
+    if (self->lock == NULL) {
+        PyErr_SetString(PyExc_MemoryError, "Unable to allocate lock");
+        return -1;
+    }
+#endif
+
+    self->check = LZMA_CHECK_UNKNOWN;
+    self->unused_data = PyBytes_FromStringAndSize(NULL, 0);
+    if (self->unused_data == NULL)
+        goto error;
+
+    switch (format) {
+        case FORMAT_AUTO:
+            lzret = lzma_auto_decoder(&self->lzs, memlimit, decoder_flags);
+            if (catch_lzma_error(lzret))
+                break;
+            return 0;
+
+        case FORMAT_XZ:
+            lzret = lzma_stream_decoder(&self->lzs, memlimit, decoder_flags);
+            if (catch_lzma_error(lzret))
+                break;
+            return 0;
+
+        case FORMAT_ALONE:
+            self->check = LZMA_CHECK_NONE;
+            lzret = lzma_alone_decoder(&self->lzs, memlimit);
+            if (catch_lzma_error(lzret))
+                break;
+            return 0;
+
+        case FORMAT_RAW:
+            self->check = LZMA_CHECK_NONE;
+            if (Decompressor_init_raw(&self->lzs, filterspecs) == -1)
+                break;
+            return 0;
+
+        default:
+            PyErr_Format(PyExc_ValueError,
+                         "Invalid container format: %d", format);
+            break;
+    }
+
+error:
+    Py_CLEAR(self->unused_data);
+#ifdef WITH_THREAD
+    PyThread_free_lock(self->lock);
+    self->lock = NULL;
+#endif
+    return -1;
+}
+
+static void
+Decompressor_dealloc(Decompressor *self)
+{
+    lzma_end(&self->lzs);
+    Py_CLEAR(self->unused_data);
+#ifdef WITH_THREAD
+    if (self->lock != NULL)
+        PyThread_free_lock(self->lock);
+#endif
+    Py_TYPE(self)->tp_free((PyObject *)self);
+}
+
+static PyMethodDef Decompressor_methods[] = {
+    {"decompress", (PyCFunction)Decompressor_decompress, METH_VARARGS,
+     Decompressor_decompress_doc},
+    {NULL}
+};
+
+PyDoc_STRVAR(Decompressor_check_doc,
+"ID of the integrity check used by the input stream.");
+
+PyDoc_STRVAR(Decompressor_eof_doc,
+"True if the end-of-stream marker has been reached.");
+
+PyDoc_STRVAR(Decompressor_unused_data_doc,
+"Data found after the end of the compressed stream.");
+
+static PyMemberDef Decompressor_members[] = {
+    {"check", T_INT, offsetof(Decompressor, check), READONLY,
+     Decompressor_check_doc},
+    {"eof", T_BOOL, offsetof(Decompressor, eof), READONLY,
+     Decompressor_eof_doc},
+    {"unused_data", T_OBJECT_EX, offsetof(Decompressor, unused_data), READONLY,
+     Decompressor_unused_data_doc},
+    {NULL}
+};
+
+PyDoc_STRVAR(Decompressor_doc,
+"LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None)\n"
+"\n"
+"Create a decompressor object for decompressing data incrementally.\n"
+"\n"
+"format specifies the container format of the input stream. If this is\n"
+"FORMAT_AUTO (the default), the decompressor will automatically detect\n"
+"whether the input is FORMAT_XZ or FORMAT_ALONE. Streams created with\n"
+"FORMAT_RAW cannot be autodetected.\n"
+"\n"
+"memlimit can be specified to limit the amount of memory used by the\n"
+"decompressor. This will cause decompression to fail if the input\n"
+"cannot be decompressed within the given limit.\n"
+"\n"
+"filters specifies a custom filter chain. This argument is required for\n"
+"FORMAT_RAW, and not accepted with any other format. When provided,\n"
+"this should be a sequence of dicts, each indicating the ID and options\n"
+"for a single filter.\n"
+"\n"
+"For one-shot decompression, use the decompress() function instead.\n");
+
+static PyTypeObject Decompressor_type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "_lzma.LZMADecompressor",           /* tp_name */
+    sizeof(Decompressor),               /* tp_basicsize */
+    0,                                  /* tp_itemsize */
+    (destructor)Decompressor_dealloc,   /* tp_dealloc */
+    0,                                  /* tp_print */
+    0,                                  /* tp_getattr */
+    0,                                  /* tp_setattr */
+    0,                                  /* tp_reserved */
+    0,                                  /* tp_repr */
+    0,                                  /* tp_as_number */
+    0,                                  /* tp_as_sequence */
+    0,                                  /* tp_as_mapping */
+    0,                                  /* tp_hash */
+    0,                                  /* tp_call */
+    0,                                  /* tp_str */
+    0,                                  /* tp_getattro */
+    0,                                  /* tp_setattro */
+    0,                                  /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT,                 /* tp_flags */
+    Decompressor_doc,                   /* tp_doc */
+    0,                                  /* tp_traverse */
+    0,                                  /* tp_clear */
+    0,                                  /* tp_richcompare */
+    0,                                  /* tp_weaklistoffset */
+    0,                                  /* tp_iter */
+    0,                                  /* tp_iternext */
+    Decompressor_methods,               /* tp_methods */
+    Decompressor_members,               /* tp_members */
+    0,                                  /* tp_getset */
+    0,                                  /* tp_base */
+    0,                                  /* tp_dict */
+    0,                                  /* tp_descr_get */
+    0,                                  /* tp_descr_set */
+    0,                                  /* tp_dictoffset */
+    (initproc)Decompressor_init,        /* tp_init */
+    0,                                  /* tp_alloc */
+    PyType_GenericNew,                  /* tp_new */
+};
+
+
+/* Module-level functions. */
+
+PyDoc_STRVAR(check_is_supported_doc,
+"check_is_supported(check_id) -> bool\n"
+"\n"
+"Test whether the given integrity check is supported.\n"
+"\n"
+"Always returns True for CHECK_NONE and CHECK_CRC32.\n");
+
+static PyObject *
+check_is_supported(PyObject *self, PyObject *args)
+{
+    int check_id;
+
+    if (!PyArg_ParseTuple(args, "i:check_is_supported", &check_id))
+        return NULL;
+
+    return PyBool_FromLong(lzma_check_is_supported(check_id));
+}
+
+
+/* Module initialization. */
+
+static PyMethodDef module_methods[] = {
+    {"check_is_supported", (PyCFunction)check_is_supported,
+     METH_VARARGS, check_is_supported_doc},
+    {NULL}
+};
+
+static PyModuleDef _lzmamodule = {
+    PyModuleDef_HEAD_INIT,
+    "_lzma",
+    NULL,
+    -1,
+    module_methods,
+    NULL,
+    NULL,
+    NULL,
+    NULL,
+};
+
+/* Some of our constants are more than 32 bits wide, so PyModule_AddIntConstant
+   would not work correctly on platforms with 32-bit longs. */
+static int
+module_add_int_constant(PyObject *m, const char *name, PY_LONG_LONG value)
+{
+    PyObject *o = PyLong_FromLongLong(value);
+    if (o == NULL)
+        return -1;
+    if (PyModule_AddObject(m, name, o) == 0)
+        return 0;
+    Py_DECREF(o);
+    return -1;
+}
+
+#define ADD_INT_PREFIX_MACRO(m, macro) \
+    module_add_int_constant(m, #macro, LZMA_ ## macro)
+
+PyMODINIT_FUNC
+PyInit__lzma(void)
+{
+    PyObject *m;
+
+    empty_tuple = PyTuple_New(0);
+    if (empty_tuple == NULL)
+        return NULL;
+
+    m = PyModule_Create(&_lzmamodule);
+    if (m == NULL)
+        return NULL;
+
+    if (PyModule_AddIntMacro(m, FORMAT_AUTO) == -1 ||
+        PyModule_AddIntMacro(m, FORMAT_XZ) == -1 ||
+        PyModule_AddIntMacro(m, FORMAT_ALONE) == -1 ||
+        PyModule_AddIntMacro(m, FORMAT_RAW) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, CHECK_NONE) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, CHECK_CRC32) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, CHECK_CRC64) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, CHECK_SHA256) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, CHECK_ID_MAX) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, CHECK_UNKNOWN) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_LZMA1) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_LZMA2) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_DELTA) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_X86) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_IA64) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_ARM) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_ARMTHUMB) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_SPARC) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, FILTER_POWERPC) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MF_HC3) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MF_HC4) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MF_BT2) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MF_BT3) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MF_BT4) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MODE_FAST) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, MODE_NORMAL) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, PRESET_DEFAULT) == -1 ||
+        ADD_INT_PREFIX_MACRO(m, PRESET_EXTREME) == -1)
+        return NULL;
+
+    Error = PyErr_NewExceptionWithDoc(
+            "_lzma.LZMAError", "Call to liblzma failed.", NULL, NULL);
+    if (Error == NULL)
+        return NULL;
+    Py_INCREF(Error);
+    if (PyModule_AddObject(m, "LZMAError", Error) == -1)
+        return NULL;
+
+    if (PyType_Ready(&Compressor_type) == -1)
+        return NULL;
+    Py_INCREF(&Compressor_type);
+    if (PyModule_AddObject(m, "LZMACompressor",
+                           (PyObject *)&Compressor_type) == -1)
+        return NULL;
+
+    if (PyType_Ready(&Decompressor_type) == -1)
+        return NULL;
+    Py_INCREF(&Decompressor_type);
+    if (PyModule_AddObject(m, "LZMADecompressor",
+                           (PyObject *)&Decompressor_type) == -1)
+        return NULL;
+
+    return m;
+}
diff --git a/PCbuild/_lzma.vcproj b/PCbuild/_lzma.vcproj
new file mode 100644
--- /dev/null
+++ b/PCbuild/_lzma.vcproj
@@ -0,0 +1,537 @@
+<?xml version="1.0" encoding="Windows-1252"?>
+<VisualStudioProject
+	ProjectType="Visual C++"
+	Version="9,00"
+	Name="_lzma"
+	ProjectGUID="{F9D71780-F393-11E0-BE50-0800200C9A66}"
+	RootNamespace="lzma"
+	Keyword="Win32Proj"
+	TargetFrameworkVersion="196613"
+	>
+	<Platforms>
+		<Platform
+			Name="Win32"
+		/>
+		<Platform
+			Name="x64"
+		/>
+	</Platforms>
+	<ToolFiles>
+	</ToolFiles>
+	<Configurations>
+		<Configuration
+			Name="Debug|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd_d.vsprops"
+			CharacterSet="0"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Debug|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd_d.vsprops;.\x64.vsprops"
+			CharacterSet="0"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Release|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="Release|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGInstrument|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\pginstrument.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGInstrument|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pginstrument.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a"
+				TargetMachine="17"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGUpdate|Win32"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\pgupdate.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_i486\liblzma.a"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+		<Configuration
+			Name="PGUpdate|x64"
+			ConfigurationType="2"
+			InheritedPropertySheets=".\pyd.vsprops;.\x64.vsprops;.\pgupdate.vsprops"
+			CharacterSet="0"
+			WholeProgramOptimization="1"
+			>
+			<Tool
+				Name="VCPreBuildEventTool"
+			/>
+			<Tool
+				Name="VCCustomBuildTool"
+			/>
+			<Tool
+				Name="VCXMLDataGeneratorTool"
+			/>
+			<Tool
+				Name="VCWebServiceProxyGeneratorTool"
+			/>
+			<Tool
+				Name="VCMIDLTool"
+				TargetEnvironment="3"
+			/>
+			<Tool
+				Name="VCCLCompilerTool"
+				AdditionalIncludeDirectories="$(lzmaDir)\include"
+				PreprocessorDefinitions="WIN32;_FILE_OFFSET_BITS=64;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;LZMA_API_STATIC"
+			/>
+			<Tool
+				Name="VCManagedResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCResourceCompilerTool"
+			/>
+			<Tool
+				Name="VCPreLinkEventTool"
+			/>
+			<Tool
+				Name="VCLinkerTool"
+				AdditionalDependencies="$(lzmaDir)\bin_x86-64\liblzma.a"
+				TargetMachine="17"
+			/>
+			<Tool
+				Name="VCALinkTool"
+			/>
+			<Tool
+				Name="VCManifestTool"
+			/>
+			<Tool
+				Name="VCXDCMakeTool"
+			/>
+			<Tool
+				Name="VCBscMakeTool"
+			/>
+			<Tool
+				Name="VCFxCopTool"
+			/>
+			<Tool
+				Name="VCAppVerifierTool"
+			/>
+			<Tool
+				Name="VCPostBuildEventTool"
+			/>
+		</Configuration>
+	</Configurations>
+	<References>
+	</References>
+	<Files>
+		<Filter
+			Name="Source Files"
+			>
+			<File
+				RelativePath="..\Modules\_lzmamodule.c"
+				>
+			</File>
+		</Filter>
+	</Files>
+	<Globals>
+	</Globals>
+</VisualStudioProject>
diff --git a/PCbuild/pcbuild.sln b/PCbuild/pcbuild.sln
--- a/PCbuild/pcbuild.sln
+++ b/PCbuild/pcbuild.sln
@@ -92,6 +92,11 @@
 		{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}
 	EndProjectSection
 EndProject
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "_lzma", "_lzma.vcproj", "{F9D71780-F393-11E0-BE50-0800200C9A66}"
+	ProjectSection(ProjectDependencies) = postProject
+		{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}
+	EndProjectSection
+EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "select", "select.vcproj", "{18CAE28C-B454-46C1-87A0-493D91D97F03}"
 	ProjectSection(ProjectDependencies) = postProject
 		{CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26} = {CF7AC3D1-E2DF-41D2-BEA6-1E2556CDEA26}
@@ -421,6 +426,22 @@
 		{73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|Win32.Build.0 = Release|Win32
 		{73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|x64.ActiveCfg = Release|x64
 		{73FCD2BD-F133-46B7-8EC1-144CD82A59D5}.Release|x64.Build.0 = Release|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|Win32.ActiveCfg = Debug|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|Win32.Build.0 = Debug|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|x64.ActiveCfg = Debug|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Debug|x64.Build.0 = Debug|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|Win32.ActiveCfg = PGInstrument|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|Win32.Build.0 = PGInstrument|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|x64.ActiveCfg = PGInstrument|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGInstrument|x64.Build.0 = PGInstrument|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|Win32.ActiveCfg = PGUpdate|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|Win32.Build.0 = PGUpdate|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|x64.ActiveCfg = PGUpdate|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.PGUpdate|x64.Build.0 = PGUpdate|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Release|Win32.ActiveCfg = Release|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Release|Win32.Build.0 = Release|Win32
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Release|x64.ActiveCfg = Release|x64
+		{F9D71780-F393-11E0-BE50-0800200C9A66}.Release|x64.Build.0 = Release|x64
 		{18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.ActiveCfg = Debug|Win32
 		{18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|Win32.Build.0 = Debug|Win32
 		{18CAE28C-B454-46C1-87A0-493D91D97F03}.Debug|x64.ActiveCfg = Debug|x64
diff --git a/PCbuild/pyproject.vsprops b/PCbuild/pyproject.vsprops
--- a/PCbuild/pyproject.vsprops
+++ b/PCbuild/pyproject.vsprops
@@ -57,6 +57,10 @@
 		Value="$(externalsDir)\bzip2-1.0.5"
 	/>
 	<UserMacro
+		Name="lzmaDir"
+		Value="$(externalsDir)\xz-5.0.3"
+	/>
+	<UserMacro
 		Name="opensslDir"
 		Value="$(externalsDir)\openssl-1.0.0a"
 	/>
diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt
--- a/PCbuild/readme.txt
+++ b/PCbuild/readme.txt
@@ -133,6 +133,12 @@
 
     All of this managed to build libbz2.lib in 
     bzip2-1.0.5\$platform-$configuration\, which the Python project links in.
+_lzma
+    Python wrapper for the liblzma compression library.
+
+    Download the pre-built Windows binaries from http://tukaani.org/xz/, and
+    extract to ..\xz-5.0.3. If you are using a more recent version of liblzma,
+    it will be necessary to rename the directory from xz-<VERSION> to xz-5.0.3.
 
 _ssl
     Python wrapper for the secure sockets library.
diff --git a/Tools/buildbot/external-common.bat b/Tools/buildbot/external-common.bat
--- a/Tools/buildbot/external-common.bat
+++ b/Tools/buildbot/external-common.bat
@@ -41,3 +41,8 @@
   rd /s/q sqlite-source-3.6.21
   svn export http://svn.python.org/projects/external/sqlite-3.7.4
 )
+
+ at rem lzma
+if not exist xz-5.0.3 (
+  svn export http://svn.python.org/projects/external/xz-5.0.3
+)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1279,6 +1279,13 @@
         else:
             missing.append('_bz2')
 
+        # LZMA compression support.
+        if self.compiler.find_library_file(lib_dirs, 'lzma'):
+            exts.append( Extension('_lzma', ['_lzmamodule.c'],
+                                   libraries = ['lzma']) )
+        else:
+            missing.append('_lzma')
+
         # Interface to the Expat XML parser
         #
         # Expat was written by James Clark and is now maintained by a group of

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Tue Nov 29 23:37:05 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Tue, 29 Nov 2011 23:37:05 +0100
Subject: [Python-checkins] =?utf8?q?devguide=3A_Add_entry_for_new_lzma_mod?=
	=?utf8?q?ule_to_experts=2Erst=2E?=
Message-ID: <E1RVWIT-0005fz-Hv@dinsdale.python.org>

http://hg.python.org/devguide/rev/3f6ae39917cd
changeset:   464:3f6ae39917cd
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Wed Nov 30 00:08:59 2011 +0200
summary:
  Add entry for new lzma module to experts.rst.

files:
  experts.rst |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/experts.rst b/experts.rst
--- a/experts.rst
+++ b/experts.rst
@@ -137,6 +137,7 @@
 linecache
 locale                loewis, lemburg
 logging               vinay.sajip
+lzma                  nadeem.vawda
 macpath
 mailbox
 mailcap

-- 
Repository URL: http://hg.python.org/devguide

From python-checkins at python.org  Wed Nov 30 03:35:36 2011
From: python-checkins at python.org (jesus.cea)
Date: Wed, 30 Nov 2011 03:35:36 +0100
Subject: [Python-checkins] =?utf8?q?peps=3A_Typo_in_PEP_393?=
Message-ID: <E1RVa1I-0006uJ-Lz@dinsdale.python.org>

http://hg.python.org/peps/rev/37ae67559050
changeset:   4005:37ae67559050
user:        Jesus Cea <jcea at jcea.es>
date:        Wed Nov 30 03:35:05 2011 +0100
summary:
  Typo in PEP 393

files:
  pep-0393.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-0393.txt b/pep-0393.txt
--- a/pep-0393.txt
+++ b/pep-0393.txt
@@ -189,7 +189,7 @@
 gives the void pointer to the data. Access to individual characters
 should use PyUnicode_{READ|WRITE}[_CHAR]:
 
- - PyUnciode_READ(kind, data, index)
+ - PyUnicode_READ(kind, data, index)
  - PyUnicode_WRITE(kind, data, index, value)
  - PyUnicode_READ_CHAR(unicode, index)
 

-- 
Repository URL: http://hg.python.org/peps

From solipsis at pitrou.net  Wed Nov 30 05:32:38 2011
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 30 Nov 2011 05:32:38 +0100
Subject: [Python-checkins] Daily reference leaks (5d42f22189be): sum=0
Message-ID: <E1RVbqY-0001WT-MU@ap.vmr.nerim.net>

results for 5d42f22189be on branch "default"
--------------------------------------------



Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/refloghzzVTc', '-x']

From python-checkins at python.org  Wed Nov 30 16:26:53 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 30 Nov 2011 16:26:53 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Typos?=
Message-ID: <E1RVm3h-0000Zh-Dm@dinsdale.python.org>

http://hg.python.org/distutils2/rev/5281c4d4075f
changeset:   1268:5281c4d4075f
parent:      1266:3f8b40403811
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Nov 30 13:24:44 2011 +0100
summary:
  Typos

files:
  distutils2/pypi/dist.py   |  2 +-
  distutils2/pypi/simple.py |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/distutils2/pypi/dist.py b/distutils2/pypi/dist.py
--- a/distutils2/pypi/dist.py
+++ b/distutils2/pypi/dist.py
@@ -430,7 +430,7 @@
         """Sort the results with the given properties.
 
         The `prefer_final` argument can be used to specify if final
-        distributions (eg. not dev, bet or alpha) would be prefered or not.
+        distributions (eg. not dev, beta or alpha) would be preferred or not.
 
         Results can be inverted by using `reverse`.
 
diff --git a/distutils2/pypi/simple.py b/distutils2/pypi/simple.py
--- a/distutils2/pypi/simple.py
+++ b/distutils2/pypi/simple.py
@@ -273,7 +273,7 @@
     def _register_release(self, release=None, release_info={}):
         """Register a new release.
 
-        Both a release or a dict of release_info can be provided, the prefered
+        Both a release or a dict of release_info can be provided, the preferred
         way (eg. the quicker) is the dict one.
 
         Return the list of existing releases for the given project.

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Nov 30 16:26:53 2011
From: python-checkins at python.org (eric.araujo)
Date: Wed, 30 Nov 2011 16:26:53 +0100
Subject: [Python-checkins] =?utf8?q?distutils2=3A_Update_list_of_trove_cla?=
	=?utf8?q?ssifiers?=
Message-ID: <E1RVm3h-0000Zq-Ey@dinsdale.python.org>

http://hg.python.org/distutils2/rev/42342803970e
changeset:   1269:42342803970e
user:        ?ric Araujo <merwok at netwok.org>
date:        Wed Nov 30 13:30:20 2011 +0100
summary:
  Update list of trove classifiers

files:
  distutils2/_trove.py |  12 ++++++++++++
  1 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/distutils2/_trove.py b/distutils2/_trove.py
--- a/distutils2/_trove.py
+++ b/distutils2/_trove.py
@@ -47,6 +47,12 @@
 'Framework :: IDLE',
 'Framework :: Paste',
 'Framework :: Plone',
+'Framework :: Plone :: 3.2',
+'Framework :: Plone :: 3.3',
+'Framework :: Plone :: 4.0',
+'Framework :: Plone :: 4.1',
+'Framework :: Plone :: 4.2',
+'Framework :: Plone :: 4.3',
 'Framework :: Pylons',
 'Framework :: Setuptools Plugin',
 'Framework :: Trac',
@@ -261,6 +267,12 @@
 'Programming Language :: Python :: 3.0',
 'Programming Language :: Python :: 3.1',
 'Programming Language :: Python :: 3.2',
+'Programming Language :: Python :: Implementation',
+'Programming Language :: Python :: Implementation :: CPython',
+'Programming Language :: Python :: Implementation :: IronPython',
+'Programming Language :: Python :: Implementation :: Jython',
+'Programming Language :: Python :: Implementation :: PyPy',
+'Programming Language :: Python :: Implementation :: Stackless',
 'Programming Language :: REBOL',
 'Programming Language :: Rexx',
 'Programming Language :: Ruby',

-- 
Repository URL: http://hg.python.org/distutils2

From python-checkins at python.org  Wed Nov 30 16:46:58 2011
From: python-checkins at python.org (nadeem.vawda)
Date: Wed, 30 Nov 2011 16:46:58 +0100
Subject: [Python-checkins] =?utf8?q?cpython=3A_Make_error_handling_in_BZ2F?=
 =?utf8?q?ile=2E=7Breadable=2Cseekable=2Cwritable=2Cfileno=7D_consistent?=
Message-ID: <E1RVmN8-0002mi-Dx@dinsdale.python.org>

http://hg.python.org/cpython/rev/354dc460cba5
changeset:   73795:354dc460cba5
user:        Nadeem Vawda <nadeem.vawda at gmail.com>
date:        Wed Nov 30 17:39:30 2011 +0200
summary:
  Make error handling in BZ2File.{readable,seekable,writable,fileno} consistent with TextIOWrapper.

Also, add tests for these methods.

files:
  Lib/bz2.py           |   6 +-
  Lib/test/test_bz2.py |  57 +++++++++++++++++++++++++++++++-
  2 files changed, 59 insertions(+), 4 deletions(-)


diff --git a/Lib/bz2.py b/Lib/bz2.py
--- a/Lib/bz2.py
+++ b/Lib/bz2.py
@@ -125,6 +125,7 @@
 
     def fileno(self):
         """Return the file descriptor for the underlying file."""
+        self._check_not_closed()
         return self._fp.fileno()
 
     def seekable(self):
@@ -133,10 +134,12 @@
 
     def readable(self):
         """Return whether the file was opened for reading."""
+        self._check_not_closed()
         return self._mode in (_MODE_READ, _MODE_READ_EOF)
 
     def writable(self):
         """Return whether the file was opened for writing."""
+        self._check_not_closed()
         return self._mode == _MODE_WRITE
 
     # Mode-checking helper functions.
@@ -147,17 +150,14 @@
 
     def _check_can_read(self):
         if not self.readable():
-            self._check_not_closed()
             raise io.UnsupportedOperation("File not open for reading")
 
     def _check_can_write(self):
         if not self.writable():
-            self._check_not_closed()
             raise io.UnsupportedOperation("File not open for writing")
 
     def _check_can_seek(self):
         if not self.seekable():
-            self._check_not_closed()
             raise io.UnsupportedOperation("Seeking is only supported "
                                           "on files open for reading")
 
diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py
--- a/Lib/test/test_bz2.py
+++ b/Lib/test/test_bz2.py
@@ -348,8 +348,63 @@
     def testFileno(self):
         self.createTempFile()
         with open(self.filename, 'rb') as rawf:
-            with BZ2File(fileobj=rawf) as bz2f:
+            bz2f = BZ2File(fileobj=rawf)
+            try:
                 self.assertEqual(bz2f.fileno(), rawf.fileno())
+            finally:
+                bz2f.close()
+        self.assertRaises(ValueError, bz2f.fileno)
+
+    def testSeekable(self):
+        bz2f = BZ2File(fileobj=BytesIO(self.DATA))
+        try:
+            self.assertTrue(bz2f.seekable())
+            bz2f.read()
+            self.assertTrue(bz2f.seekable())
+        finally:
+            bz2f.close()
+        self.assertRaises(ValueError, bz2f.seekable)
+
+        bz2f = BZ2File(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertFalse(bz2f.seekable())
+        finally:
+            bz2f.close()
+        self.assertRaises(ValueError, bz2f.seekable)
+
+    def testReadable(self):
+        bz2f = BZ2File(fileobj=BytesIO(self.DATA))
+        try:
+            self.assertTrue(bz2f.readable())
+            bz2f.read()
+            self.assertTrue(bz2f.readable())
+        finally:
+            bz2f.close()
+        self.assertRaises(ValueError, bz2f.readable)
+
+        bz2f = BZ2File(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertFalse(bz2f.readable())
+        finally:
+            bz2f.close()
+        self.assertRaises(ValueError, bz2f.readable)
+
+    def testWritable(self):
+        bz2f = BZ2File(fileobj=BytesIO(self.DATA))
+        try:
+            self.assertFalse(bz2f.writable())
+            bz2f.read()
+            self.assertFalse(bz2f.writable())
+        finally:
+            bz2f.close()
+        self.assertRaises(ValueError, bz2f.writable)
+
+        bz2f = BZ2File(fileobj=BytesIO(), mode="w")
+        try:
+            self.assertTrue(bz2f.writable())
+        finally:
+            bz2f.close()
+        self.assertRaises(ValueError, bz2f.writable)
 
     def testOpenDel(self):
         self.createTempFile()

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 30 21:25:10 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Wed, 30 Nov 2011 21:25:10 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=283=2E2=29=3A_Issue_=23104043?=
 =?utf8?q?9=3A_better_document_how_to_compile_and_link_an_embedded_Python?=
Message-ID: <E1RVqiM-0006E7-Ic@dinsdale.python.org>

http://hg.python.org/cpython/rev/2c05b8a6cdd1
changeset:   73796:2c05b8a6cdd1
branch:      3.2
parent:      73789:2ef359d7a2e9
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Wed Nov 30 21:19:21 2011 +0100
summary:
  Issue #1040439: better document how to compile and link an embedded Python interpreter.
Still lacks docs for Windows (anyone?).

files:
  Doc/extending/embedding.rst |  72 +++++++++++++++---------
  1 files changed, 44 insertions(+), 28 deletions(-)


diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst
--- a/Doc/extending/embedding.rst
+++ b/Doc/extending/embedding.rst
@@ -133,8 +133,9 @@
 
 This code loads a Python script using ``argv[1]``, and calls the function named
 in ``argv[2]``.  Its integer arguments are the other values of the ``argv``
-array.  If you compile and link this program (let's call the finished executable
-:program:`call`), and use it to execute a Python script, such as::
+array.  If you :ref:`compile and link <compiling>` this program (let's call
+the finished executable :program:`call`), and use it to execute a Python
+script, such as::
 
    def multiply(a,b):
        print("Will compute", a, "times", b)
@@ -257,37 +258,52 @@
 program.  There is no need to recompile Python itself using C++.
 
 
-.. _link-reqs:
+.. _compiling:
 
-Linking Requirements
-====================
+Compiling and Linking under Unix-like systems
+=============================================
 
-While the :program:`configure` script shipped with the Python sources will
-correctly build Python to export the symbols needed by dynamically linked
-extensions, this is not automatically inherited by applications which embed the
-Python library statically, at least on Unix.  This is an issue when the
-application is linked to the static runtime library (:file:`libpython.a`) and
-needs to load dynamic extensions (implemented as :file:`.so` files).
+It is not necessarily trivial to find the right flags to pass to your
+compiler (and linker) in order to embed the Python interpreter into your
+application, particularly because Python needs to load library modules
+implemented as C dynamic extensions (:file:`.so` files) linked against
+it.
 
-The problem is that some entry points are defined by the Python runtime solely
-for extension modules to use.  If the embedding application does not use any of
-these entry points, some linkers will not include those entries in the symbol
-table of the finished executable.  Some additional options are needed to inform
-the linker not to remove these symbols.
+To find out the required compiler and linker flags, you can execute the
+:file:`python{X.Y}-config` script which is generated as part of the
+installation process (a generic :file:`python3-config` script is also
+available).  This script has several options, of which the following will
+be directly useful to you:
 
-Determining the right options to use for any given platform can be quite
-difficult, but fortunately the Python configuration already has those values.
-To retrieve them from an installed Python interpreter, start an interactive
-interpreter and have a short session like this::
+* ``pythonX.Y-config --cflags`` will give you the recommended flags when
+  compiling::
 
-   >>> import distutils.sysconfig
-   >>> distutils.sysconfig.get_config_var('LINKFORSHARED')
+   $ /opt/bin/python3.2-config --cflags
+   -I/opt/include/python3.2m -I/opt/include/python3.2m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
+
+* ``pythonX.Y-config --ldflags`` will give you the recommended flags when
+  linking::
+
+   $ /opt/bin/python3.2-config --ldflags
+   -I/opt/lib/python3.2/config-3.2m -lpthread -ldl -lutil -lm -lpython3.2m -Xlinker -export-dynamic
+
+.. note::
+   To avoid confusion between several Python installations (and especially
+   between the system Python and your own compiled Python), it is recommended
+   that you use the absolute path to :file:`python{X.Y}-config`, as in the above
+   example.
+
+If this procedure doesn't work for you (it is not guaranteed to work for
+all Unix-like platforms; however, we welcome bug reports at
+http://bugs.python.org),  you will have to read your system's documentation
+about dynamic linking and/or examine Python's Makefile and compilation
+options.  In this case, the :mod:`sysconfig` module is a useful tool to
+programmatically extract the configuration values that you will want to
+combine together::
+
+   >>> import sysconfig
+   >>> sysconfig.get_config_var('LINKFORSHARED')
    '-Xlinker -export-dynamic'
 
-.. index:: module: distutils.sysconfig
 
-The contents of the string presented will be the options that should be used.
-If the string is empty, there's no need to add any additional options.  The
-:const:`LINKFORSHARED` definition corresponds to the variable of the same name
-in Python's top-level :file:`Makefile`.
-
+.. XXX similar documentation for Windows missing

-- 
Repository URL: http://hg.python.org/cpython

From python-checkins at python.org  Wed Nov 30 21:25:11 2011
From: python-checkins at python.org (antoine.pitrou)
Date: Wed, 30 Nov 2011 21:25:11 +0100
Subject: [Python-checkins] =?utf8?q?cpython_=28merge_3=2E2_-=3E_default=29?=
 =?utf8?q?=3A_Issue_=231040439=3A_better_document_how_to_compile_and_link_?=
 =?utf8?q?an_embedded_Python?=
Message-ID: <E1RVqiN-0006Ed-AO@dinsdale.python.org>

http://hg.python.org/cpython/rev/528abe272856
changeset:   73797:528abe272856
parent:      73795:354dc460cba5
parent:      73796:2c05b8a6cdd1
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Wed Nov 30 21:20:01 2011 +0100
summary:
  Issue #1040439: better document how to compile and link an embedded Python interpreter.
Still lacks docs for Windows (anyone?).

files:
  Doc/extending/embedding.rst |  72 +++++++++++++++---------
  1 files changed, 44 insertions(+), 28 deletions(-)


diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst
--- a/Doc/extending/embedding.rst
+++ b/Doc/extending/embedding.rst
@@ -133,8 +133,9 @@
 
 This code loads a Python script using ``argv[1]``, and calls the function named
 in ``argv[2]``.  Its integer arguments are the other values of the ``argv``
-array.  If you compile and link this program (let's call the finished executable
-:program:`call`), and use it to execute a Python script, such as::
+array.  If you :ref:`compile and link <compiling>` this program (let's call
+the finished executable :program:`call`), and use it to execute a Python
+script, such as::
 
    def multiply(a,b):
        print("Will compute", a, "times", b)
@@ -257,37 +258,52 @@
 program.  There is no need to recompile Python itself using C++.
 
 
-.. _link-reqs:
+.. _compiling:
 
-Linking Requirements
-====================
+Compiling and Linking under Unix-like systems
+=============================================
 
-While the :program:`configure` script shipped with the Python sources will
-correctly build Python to export the symbols needed by dynamically linked
-extensions, this is not automatically inherited by applications which embed the
-Python library statically, at least on Unix.  This is an issue when the
-application is linked to the static runtime library (:file:`libpython.a`) and
-needs to load dynamic extensions (implemented as :file:`.so` files).
+It is not necessarily trivial to find the right flags to pass to your
+compiler (and linker) in order to embed the Python interpreter into your
+application, particularly because Python needs to load library modules
+implemented as C dynamic extensions (:file:`.so` files) linked against
+it.
 
-The problem is that some entry points are defined by the Python runtime solely
-for extension modules to use.  If the embedding application does not use any of
-these entry points, some linkers will not include those entries in the symbol
-table of the finished executable.  Some additional options are needed to inform
-the linker not to remove these symbols.
+To find out the required compiler and linker flags, you can execute the
+:file:`python{X.Y}-config` script which is generated as part of the
+installation process (a generic :file:`python3-config` script is also
+available).  This script has several options, of which the following will
+be directly useful to you:
 
-Determining the right options to use for any given platform can be quite
-difficult, but fortunately the Python configuration already has those values.
-To retrieve them from an installed Python interpreter, start an interactive
-interpreter and have a short session like this::
+* ``pythonX.Y-config --cflags`` will give you the recommended flags when
+  compiling::
 
-   >>> import distutils.sysconfig
-   >>> distutils.sysconfig.get_config_var('LINKFORSHARED')
+   $ /opt/bin/python3.2-config --cflags
+   -I/opt/include/python3.2m -I/opt/include/python3.2m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
+
+* ``pythonX.Y-config --ldflags`` will give you the recommended flags when
+  linking::
+
+   $ /opt/bin/python3.2-config --ldflags
+   -I/opt/lib/python3.2/config-3.2m -lpthread -ldl -lutil -lm -lpython3.2m -Xlinker -export-dynamic
+
+.. note::
+   To avoid confusion between several Python installations (and especially
+   between the system Python and your own compiled Python), it is recommended
+   that you use the absolute path to :file:`python{X.Y}-config`, as in the above
+   example.
+
+If this procedure doesn't work for you (it is not guaranteed to work for
+all Unix-like platforms; however, we welcome bug reports at
+http://bugs.python.org),  you will have to read your system's documentation
+about dynamic linking and/or examine Python's Makefile and compilation
+options.  In this case, the :mod:`sysconfig` module is a useful tool to
+programmatically extract the configuration values that you will want to
+combine together::
+
+   >>> import sysconfig
+   >>> sysconfig.get_config_var('LINKFORSHARED')
    '-Xlinker -export-dynamic'
 
-.. index:: module: distutils.sysconfig
 
-The contents of the string presented will be the options that should be used.
-If the string is empty, there's no need to add any additional options.  The
-:const:`LINKFORSHARED` definition corresponds to the variable of the same name
-in Python's top-level :file:`Makefile`.
-
+.. XXX similar documentation for Windows missing

-- 
Repository URL: http://hg.python.org/cpython